201 lines
9.7 KiB
Markdown
201 lines
9.7 KiB
Markdown
# Design: enterprise-audio-analysis-spectrum
|
||
|
||
**Change ID**: enterprise-audio-analysis-spectrum
|
||
**Package**: `@suaveplan/audio-analysis-spectrum`
|
||
|
||
`@suaveplan/audio-analysis-spectrum` is a universal-tier analysis plugin in the `@suaveplan/audio-*`
|
||
family. It computes magnitude spectra, spectrograms, MFCCs, and the scalar spectral features
|
||
(centroid, rolloff, flux) that downstream packages — beat, pitch, and visualisation layers — build
|
||
on. This document carries the **package-specific design**: the algorithm rationale, code sketches,
|
||
file layout, integration notes, benchmarks, and the decisions behind them.
|
||
|
||
---
|
||
|
||
## 1. Relationship to the ecosystem architecture
|
||
|
||
The cross-cutting `@suaveplan/audio-*` architecture — the three-layer separation, the six plugin
|
||
slots, the zero-duplication rule, and the universal-tier purity constraint — lives in
|
||
`enterprise-audio-core/design.md` §1–§4 and is **referenced, not duplicated** here.
|
||
|
||
The only slot this package occupies is **Analysis Op (plugin slot 3)**: a sidecar processor that
|
||
reads an `AudioDocument` and emits features. It is *not* part of the render pipeline — it never
|
||
appears in a `CompiledPlan` or a `ProcessPipeline`. `SpectrumAnalyzer`, `Spectrogram`, and `MFCC`
|
||
each implement the `AnalysisOp` contract defined by `@suaveplan/audio-core` and consume the
|
||
`AudioDocument`, `BlockProcessor`, and `BlockContext` types from there.
|
||
|
||
---
|
||
|
||
## 2. Algorithm rationale
|
||
|
||
### 2.1. Real FFT (radix-2)
|
||
|
||
The FFT is the hot path for every analyzer, so it is hand-rolled rather than pulled from a
|
||
dependency: a portable radix-2 Cooley–Tukey decimation-in-time transform restricted to power-of-two
|
||
frame sizes (`256`–`32768`). Restricting to power-of-two keeps the implementation small and
|
||
branch-free and matches the natural `frameSize` choices for audio (`2048` default). Twiddle-factor
|
||
tables (`cos`/`sin` lookup) are precomputed once per `frameSize` and cached, so a multi-frame scan
|
||
allocates them exactly once. Because the input is real, only `frameSize / 2 + 1` bins are unique;
|
||
the analyzer keeps just that half-spectrum. Determinism is a hard requirement — the same input frame
|
||
always yields byte-identical magnitudes — which is what makes the property tests (sine → single
|
||
peak) stable.
|
||
|
||
The earlier draft mentioned delegating to a browser `AnalyserNode`. That path is **dropped from the
|
||
universal core**: `AnalyserNode` is a browser global and would break universal-tier purity. If a
|
||
browser engine wants hardware-accelerated FFT it injects an adapter; the package's own code stays
|
||
portable.
|
||
|
||
### 2.2. Windowing
|
||
|
||
Each frame is multiplied by a window before the FFT to suppress spectral leakage. Three windows
|
||
ship — Hann (default), Hamming, Blackman — selectable per call. Hann is the default because it is
|
||
the standard general-purpose choice for music analysis: a good main-lobe/side-lobe trade-off.
|
||
Window arrays are pure functions of `(type, size)` and are cached, so repeated same-shape frames
|
||
reuse one `Float32Array`.
|
||
|
||
```ts
|
||
function hann(size: number): Float32Array {
|
||
const w = new Float32Array(size)
|
||
for (let n = 0; n < size; n++) w[n] = 0.5 - 0.5 * Math.cos((2 * Math.PI * n) / (size - 1))
|
||
return w
|
||
}
|
||
```
|
||
|
||
### 2.3. Frame slicing
|
||
|
||
`SpectrumAnalyzer`, `Spectrogram`, and `MFCC` share one framing model: slide a `frameSize` window
|
||
over the document advancing by `hopSize`. The frame count for a document of length `L` is
|
||
`floor((L - frameSize) / hopSize) + 1`, and a document shorter than one frame yields an empty
|
||
result rather than throwing — callers analysing very short clips should not have to guard.
|
||
|
||
### 2.4. Spectral features
|
||
|
||
- **Centroid** — magnitude-weighted mean frequency; the "brightness" of the frame.
|
||
- **Rolloff** — the frequency below which 85% of the spectral energy lies.
|
||
- **Flux** — summed positive frame-to-frame magnitude difference; the raw onset signal that
|
||
`@suaveplan/audio-analysis-beat` consumes.
|
||
|
||
All three are cheap reductions over the magnitude spectrum already computed for the frame, so they
|
||
are produced as a by-product rather than a second pass.
|
||
|
||
### 2.5. Mel scale and the triangular filterbank
|
||
|
||
The mel scale is a perceptual frequency warping — roughly linear below 1 kHz, logarithmic above.
|
||
The filterbank is a set of `melBands` triangular filters evenly spaced on the mel scale across
|
||
`[minHz, maxHz]`; applying it to a magnitude spectrum collapses `frameSize / 2 + 1` FFT bins into
|
||
`melBands` perceptually-spaced energies. The filter matrix is a pure function of
|
||
`(frameSize, sampleRate, melBands, minHz, maxHz)` and is cached.
|
||
|
||
```ts
|
||
const hzToMel = (hz: number) => 2595 * Math.log10(1 + hz / 700)
|
||
const melToHz = (mel: number) => 700 * (10 ** (mel / 2595) - 1)
|
||
```
|
||
|
||
### 2.6. MFCC — DCT-II on the log mel-spectrum
|
||
|
||
MFCCs are the standard compact timbre descriptor. The pipeline per frame is: magnitude spectrum →
|
||
mel filterbank → natural log of mel energies (floored to avoid `log(0)`) → type-II DCT → keep the
|
||
first `coefficients` values (default `13`). The DCT decorrelates the log-mel vector so the low-order
|
||
coefficients capture the spectral envelope. `coefficients` must not exceed `melBands` — asking for
|
||
more cepstral coefficients than mel bands is meaningless and is rejected at the boundary.
|
||
|
||
---
|
||
|
||
## 3. Public surface and code sketch
|
||
|
||
```ts
|
||
class SpectrumAnalyzer implements AnalysisOp {
|
||
readonly name = 'spectrum-analyzer'
|
||
readonly description = 'Magnitude spectrum + spectral centroid/rolloff/flux'
|
||
|
||
async analyze(doc: AudioDocument, opts?: SpectrumAnalyzerOptions): Promise<SpectrumResult> {
|
||
const o = SpectrumAnalyzerOptionsSchema.parse(opts ?? {})
|
||
const win = window(o.window, o.frameSize)
|
||
let prevMag: Float32Array | undefined
|
||
// slide frames, fft(win · frame), reduce to bins + centroid/rolloff/flux
|
||
}
|
||
|
||
analyzeStream(doc: AudioDocument, opts?: SpectrumAnalyzerOptions): AsyncGenerator<SpectrumResult>
|
||
analyzeBlock(block: Float32Array[], ctx: BlockContext): SpectrumResult
|
||
}
|
||
```
|
||
|
||
`Spectrogram` and `MFCC` follow the same shape; `Spectrogram` keeps every per-frame magnitude
|
||
vector, `MFCC` runs the mel + log + DCT chain per frame. All option validation goes through the
|
||
Zod schemas in `*.schemas.ts`; all errors are `AudioError` instances carrying an
|
||
`AUDIO_ANALYSIS_*` code.
|
||
|
||
---
|
||
|
||
## 4. File layout
|
||
|
||
```text
|
||
packages/media/audio/audio-analysis-spectrum/
|
||
package.json, tsconfig.json, vitest.config.ts
|
||
src/
|
||
index.ts
|
||
error-codes.ts
|
||
analyzer/
|
||
spectrum-analyzer.ts (+ .md .types.ts .schemas.ts .test.ts)
|
||
spectrogram.ts (+ .md .types.ts .schemas.ts .test.ts)
|
||
mfcc.ts (+ .md .types.ts .schemas.ts .test.ts)
|
||
index.ts
|
||
algorithms/
|
||
fft.ts (+ .md .test.ts)
|
||
window.ts (+ .md .test.ts)
|
||
mel-scale.ts (+ .md .test.ts)
|
||
dct.ts (+ .md .test.ts)
|
||
index.ts
|
||
__benchmarks__/
|
||
spectrum.bench.ts
|
||
tests/
|
||
integration.test.ts property.test.ts purity.test.ts exports.test.ts
|
||
```
|
||
|
||
---
|
||
|
||
## 5. Integration with genesis packages
|
||
|
||
- **`@suaveplan/audio-core`** — supplies `AudioDocument` (the scan input), the `AnalysisOp`
|
||
contract, and `BlockProcessor` / `BlockContext` for `analyzeBlock`. The analyzers iterate the
|
||
document with `doc.stream()` / `doc.read()` rather than touching pages directly.
|
||
- **`@suaveplan/error`** — `AudioError` base class; this package registers the `AUDIO_ANALYSIS`
|
||
code namespace (`AUDIO_ANALYSIS_SPECTRUM.INVALID_OPTIONS`, `AUDIO_ANALYSIS_SPECTRUM.INVALID_FRAME`,
|
||
`AUDIO_ANALYSIS_SPECTRUM.ABORTED`).
|
||
- **`@suaveplan/opentelemetry`** — spans, counters, and histograms (`audio.analysis.runs_total`,
|
||
`audio.analysis.duration_ms`, `audio.analysis.block_duration_ms`); inert with no exporter.
|
||
- **`@suaveplan/types`** — `LoggerLike` for optional injected logging.
|
||
|
||
Downstream, `@suaveplan/audio-analysis-beat` consumes the spectral-flux signal, and
|
||
`@suaveplan/audio-analysis-pitch` builds its chromagram from the magnitude spectrum — but those are
|
||
separate changes; this package exposes the surface, it does not import them.
|
||
|
||
---
|
||
|
||
## 6. Benchmarks
|
||
|
||
- `spectrum.bench.ts` — spectrogram of a 1-minute 48 kHz track at `frameSize` `2048` / `hopSize`
|
||
`512`, and MFCC of the same track at `13` coefficients / `40` mel bands. Establishes the
|
||
per-frame FFT + reduction cost and the mel + DCT overhead.
|
||
|
||
---
|
||
|
||
## 7. Decisions and trade-offs
|
||
|
||
- **No `AnalyserNode` in the universal core.** The draft suggested delegating FFT to the browser's
|
||
`AnalyserNode`. That is a browser global and breaks universal-tier purity, so the package ships
|
||
one portable radix-2 FFT used everywhere. A browser engine wanting hardware acceleration injects
|
||
an adapter; the universal path never references a platform global.
|
||
- **Power-of-two frame sizes only.** Restricting `frameSize` to `256`–`32768` powers of two keeps
|
||
the FFT small, branch-free, and deterministic, and matches the natural audio frame sizes. A
|
||
non-power-of-two request is a boundary error (`AUDIO_ANALYSIS_SPECTRUM.INVALID_FRAME`), not a slow
|
||
Bluestein fallback.
|
||
- **Three analyzers, one framing model.** `SpectrumAnalyzer`, `Spectrogram`, and `MFCC` share frame
|
||
slicing, windowing, and the FFT; they differ only in the reduction. Splitting them into three
|
||
`AnalysisOp` classes (rather than one mega-analyzer with flags) keeps each registerable
|
||
independently and keeps each result type tight.
|
||
- **Cached pure tables.** Window arrays, FFT twiddle factors, and mel filterbank matrices are all
|
||
pure functions of their shape parameters and are cached. A long scan allocates each exactly once;
|
||
this is the single biggest constant-factor win and is why the benchmark targets a full minute.
|
||
- **Features as a by-product.** Centroid, rolloff, and flux are computed from the magnitude
|
||
spectrum already in hand, not in a second pass — keeping `SpectrumAnalyzer` a single-pass scan.
|