235 lines
11 KiB
Markdown
235 lines
11 KiB
Markdown
# Design: enterprise-audio-analysis-beat
|
||
|
||
**Change ID**: enterprise-audio-analysis-beat
|
||
**Package**: `@suaveplan/audio-analysis-beat`
|
||
|
||
`@suaveplan/audio-analysis-beat` is a universal-tier Analysis Op plugin for the `@suaveplan/audio-*`
|
||
family. It performs tempo (BPM) detection, beat-grid construction, and onset detection over
|
||
`AudioDocument` streams. This document carries the **package-specific design** — algorithm
|
||
rationale, code sketches, file layout, integration notes, and trade-offs. It deliberately does not
|
||
restate the cross-cutting ecosystem model.
|
||
|
||
---
|
||
|
||
## 1. Relationship to the ecosystem architecture
|
||
|
||
The cross-cutting `@suaveplan/audio-*` architecture — the three-layer separation, the six plugin
|
||
slots, the zero-duplication policy over genesis primitives, and the universal-purity rule — is
|
||
defined once in `enterprise-audio-core/design.md` §1–§4 and is **referenced, not duplicated** here.
|
||
|
||
Two facts from that document anchor this package:
|
||
|
||
- **Analysis Op is plugin slot 3.** Per `enterprise-audio-core/design.md` §1.3, an Analysis Op
|
||
"reads audio and emits features (BPM, key, spectrum); runs as a sidecar scan, not part of the
|
||
render pipeline." `BeatDetector` is exactly such a sidecar: it never registers in an `OpRegistry`
|
||
plan or process slot, never mutates the `AudioDocument`, and runs concurrently with playback or
|
||
as a standalone batch scan.
|
||
- **The `AnalysisOp` interface and `AnalysisResult` shape** are defined by the audio ecosystem and
|
||
consumed here unchanged. `BeatDetector implements AnalysisOp`; `BeatResult extends
|
||
AnalysisResult`.
|
||
|
||
Everything below is specific to beat/tempo/onset detection.
|
||
|
||
---
|
||
|
||
## 2. Detection pipeline
|
||
|
||
Beat detection is a three-stage pipeline regardless of which algorithm is selected:
|
||
|
||
```text
|
||
AudioDocument ──► [1] Onset Detection Function ──► [2] Tempo Estimation ──► [3] Beat Tracking
|
||
(per-block ODF curve) (dominant period) (phase + grid)
|
||
```
|
||
|
||
### 2.1. Stage 1 — Onset Detection Function (ODF)
|
||
|
||
The ODF is a 1-D curve, one value per analysis block, that peaks where the signal's energy or
|
||
spectral content rises sharply — i.e. at note onsets. Two ODF algorithms ship; a third algorithm
|
||
(`comb`) reuses the energy ODF and differs only in stage 2.
|
||
|
||
**`energy` (default).** The cheapest ODF. It consumes the per-block energy statistics already
|
||
produced by `@suaveplan/audio-core` and takes the half-wave-rectified first difference:
|
||
|
||
```ts
|
||
// energy-odf.ts — sketch
|
||
export function energyOdf(blockEnergies: Float64Array): Float64Array {
|
||
const odf = new Float64Array(blockEnergies.length)
|
||
for (let i = 1; i < blockEnergies.length; i++) {
|
||
const d = blockEnergies[i] - blockEnergies[i - 1]
|
||
odf[i] = d > 0 ? d : 0 // half-wave rectify: onsets are energy increases
|
||
}
|
||
return odf
|
||
}
|
||
```
|
||
|
||
It is fast and allocation-light but blurs onsets that are loud in timbre yet quiet in total energy
|
||
(e.g. a hi-hat over a sustained pad).
|
||
|
||
**`spectral-flux`.** Higher precision and the right default for streaming. A hand-rolled real FFT
|
||
(no `AnalyserNode`, no Node bindings — universal purity, see the spec delta) produces a magnitude
|
||
spectrum per frame; the ODF is the sum of positive bin-to-bin differences:
|
||
|
||
```ts
|
||
// spectral-flux.ts — sketch
|
||
export function spectralFlux(frame: Float32Array, prevMag: Float32Array, fft: RealFft): number {
|
||
const mag = fft.magnitude(frame) // |X[k]|
|
||
let flux = 0
|
||
for (let k = 0; k < mag.length; k++) {
|
||
const d = mag[k] - prevMag[k]
|
||
if (d > 0) flux += d // only rising bins count toward an onset
|
||
prevMag[k] = mag[k]
|
||
}
|
||
return flux
|
||
}
|
||
```
|
||
|
||
Spectral flux catches timbral onsets the energy ODF misses, at the cost of an FFT per frame.
|
||
|
||
In both cases, peaks in the ODF above an adaptive threshold (a moving median plus a fixed margin)
|
||
are reported as `onsets` — timestamps in seconds, strictly increasing.
|
||
|
||
### 2.2. Stage 2 — Tempo estimation
|
||
|
||
Given the ODF, the dominant periodicity is the tempo. Two strategies:
|
||
|
||
- **Autocorrelation** (used by `energy` and `spectral-flux`). Autocorrelate the ODF; the lag of the
|
||
strongest peak whose implied BPM falls inside `[minBpm, maxBpm]` is the period. Octave errors
|
||
(detecting half- or double-tempo) are resolved by preferring the candidate closest to the
|
||
perceptual centre (~120 BPM) when two peaks are within a small ratio.
|
||
- **Comb filter** (the `comb` algorithm). A bank of comb filters, one per candidate BPM in the
|
||
range, is convolved with the ODF; the filter with the highest resonance output wins. This is more
|
||
robust on syncopated material than raw autocorrelation but costs `O(range × odfLength)`.
|
||
|
||
```ts
|
||
// comb-tempo.ts — sketch
|
||
export function combTempo(odf: Float64Array, sr: number, hop: number, lo: number, hi: number) {
|
||
let bestBpm = lo, bestScore = -Infinity
|
||
for (let bpm = lo; bpm <= hi; bpm++) {
|
||
const period = Math.round((60 / bpm) * (sr / hop)) // ODF samples per beat
|
||
let score = 0
|
||
for (let i = period; i < odf.length; i++) score += odf[i] * odf[i - period]
|
||
if (score > bestScore) { bestScore = score; bestBpm = bpm }
|
||
}
|
||
return { bpm: bestBpm, score: bestScore }
|
||
}
|
||
```
|
||
|
||
### 2.3. Stage 3 — Beat tracking and grid construction
|
||
|
||
With a period known, `beat-grid.ts` finds the phase: the offset that, when a `60 / bpm`-spaced grid
|
||
is laid down, minimises total onset-to-nearest-gridline distance. The output `beats` array is that
|
||
phase-aligned grid clipped to the document duration. `confidence` is derived from how tightly the
|
||
detected onsets cluster around the grid (a sharp cluster → high confidence; scattered onsets → low).
|
||
|
||
---
|
||
|
||
## 3. The three entry points
|
||
|
||
`BeatDetector` exposes the full `AnalysisOp` surface; each entry point reuses the same three-stage
|
||
pipeline at a different granularity.
|
||
|
||
- **`analyze(doc, opts?)`** — batch scan. Walks the whole document through stages 1–3 and returns a
|
||
single settled `BeatResult`. Wrapped in the `audio.analysis.beat` OTel span.
|
||
- **`analyzeStream(doc, opts?)`** — incremental scan. Runs stage 1 block-by-block and re-runs
|
||
stages 2–3 on the ODF accumulated so far, yielding a refining `BeatResult` after each window. For
|
||
a stable-tempo signal, `confidence` is non-decreasing across yields and the final yield's `bpm`
|
||
equals `analyze`'s.
|
||
- **`analyzeBlock(block, ctx)`** — real-time. Computes the ODF value for a single block, updates a
|
||
running tempo estimate, and returns synchronously. Allocation-light enough to call once per
|
||
audio block from a visualiser's render loop.
|
||
|
||
```ts
|
||
// beat-detector.ts — analyze sketch
|
||
async analyze(doc: AudioDocument, opts?: BeatDetectorOptions): Promise<BeatResult> {
|
||
const o = BeatDetectorOptionsSchema.parse({ ...this.opts, ...opts })
|
||
if (o.signal?.aborted) throw new AudioError('AUDIO_ANALYSIS_BEAT.ABORTED')
|
||
if (doc.length === 0) throw new AudioError('AUDIO_ANALYSIS_BEAT.EMPTY_DOCUMENT')
|
||
return tracer.startActiveSpan('audio.analysis.beat', async (span) => {
|
||
const odf = await this.buildOdf(doc, o) // stage 1
|
||
const onsets = pickPeaks(odf, doc.sampleRate)
|
||
if (onsets.length === 0) throw new AudioError('AUDIO_ANALYSIS_BEAT.NO_BEATS_DETECTED')
|
||
const { bpm } = this.estimateTempo(odf, o) // stage 2
|
||
const { beats, confidence } = buildBeatGrid(onsets, bpm, doc.duration) // stage 3
|
||
runsCounter.add(1); durationHist.record(span.duration)
|
||
return { op: 'beat-detector', timestamp: 0, duration: doc.duration, bpm, beats, onsets, confidence }
|
||
})
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 4. File layout
|
||
|
||
```text
|
||
packages/media/audio/audio-analysis-beat/
|
||
package.json, tsconfig.json, vitest.config.ts
|
||
src/
|
||
index.ts
|
||
error-codes.ts error-codes.test.ts
|
||
detector/
|
||
beat-detector.ts (+ .md .types.ts .test.ts) · beat-detector.schemas.ts · index.ts
|
||
algorithms/
|
||
energy-odf.ts (+ .md .test.ts)
|
||
spectral-flux.ts (+ .md .test.ts)
|
||
comb-tempo.ts (+ .md .test.ts)
|
||
beat-grid.ts (+ .md .test.ts)
|
||
index.ts
|
||
__benchmarks__/ beat.bench.ts
|
||
tests/
|
||
integration.test.ts property.test.ts purity.test.ts exports.test.ts
|
||
```
|
||
|
||
`detector/` owns the `BeatDetector` class, its `BeatDetectorOptions` / `BeatResult` types, and the
|
||
Zod schemas. `algorithms/` holds the four pure numeric modules — each independently importable via a
|
||
subpath export so a consumer that only wants the comb-filter estimator does not pull the whole
|
||
package.
|
||
|
||
---
|
||
|
||
## 5. Integration notes
|
||
|
||
- **`@suaveplan/audio-core`** — `BeatDetector` consumes `AudioDocument` (page reads via the core's
|
||
streaming API, never direct page poking), `BlockContext` for `analyzeBlock`, and the `AnalysisOp`
|
||
interface. The `energy` ODF reuses the core's per-block energy statistics rather than recomputing
|
||
them. The package never registers an op in an `OpRegistry`.
|
||
- **`@suaveplan/error`** — `AudioError` is the base for every throw; the `AUDIO_ANALYSIS_*` codes
|
||
are registered through `@suaveplan/error`'s `defineCodes` registry.
|
||
- **`@suaveplan/opentelemetry`** — supplies the tracer, counter, and histogram. Instrumentation is
|
||
inert without an exporter and never runs at import time.
|
||
- **`@suaveplan/types`** — `LoggerLike` for the optional injected logger.
|
||
- **Downstream consumers** — application/timeline code registers `BeatDetector` as a sidecar
|
||
analysis op; it is not depended on by any other package in this 13-change batch.
|
||
|
||
---
|
||
|
||
## 6. Benchmarks
|
||
|
||
- `beat.bench.ts` — `analyze` over a synthetic 1-minute 48 kHz stereo track, once per algorithm.
|
||
- `beat.bench.ts` — `analyze` over a synthetic 10-minute track to confirm linear scaling and bound
|
||
per-minute scan cost.
|
||
|
||
---
|
||
|
||
## 7. Decisions and trade-offs
|
||
|
||
- **`energy` is the default, not `spectral-flux`.** The energy ODF reuses statistics `audio-core`
|
||
already computes, so the default path adds almost no cost over a plain document scan. Callers who
|
||
need timbral-onset precision opt into `spectral-flux` explicitly. This keeps the common case
|
||
cheap and the precise case available.
|
||
- **Hand-rolled FFT, no `AnalyserNode`.** The raw source material suggested a browser
|
||
`AnalyserNode` for the FFT. That would break universal-tier purity (the package must run
|
||
identically under Bun, Node, Deno, and browsers). The `spectral-flux` algorithm therefore ships a
|
||
hand-rolled real FFT over `Float32Array` — slightly slower in a browser than the native node, but
|
||
one code path everywhere and zero platform branching.
|
||
- **Comb filter is a selectable algorithm, not the default.** Comb-filter tempo estimation is more
|
||
robust on syncopated material but costs `O(bpmRange × odfLength)`. It is offered as
|
||
`algorithm: 'comb'` rather than forced on every caller.
|
||
- **No-beats is an error, not a zero-confidence result.** A scan that finds no onsets above
|
||
threshold throws `AUDIO_ANALYSIS_BEAT.NO_BEATS_DETECTED` rather than returning `{ bpm: 0, confidence:
|
||
0 }`. A silent or non-rhythmic input is a caller-visible condition, and a thrown code is
|
||
unmissable where a zero-confidence struct is easy to ignore.
|
||
- **Octave-error resolution favours the perceptual centre.** Autocorrelation routinely detects
|
||
half- or double-tempo. When two candidate periods are within a small ratio, the one whose BPM is
|
||
closest to ~120 wins — a cheap heuristic that matches human tapping far more often than picking
|
||
the raw strongest peak.
|