# Design: enterprise-audio-analysis-spectrum **Change ID**: enterprise-audio-analysis-spectrum **Package**: `@suaveplan/audio-analysis-spectrum` `@suaveplan/audio-analysis-spectrum` is a universal-tier analysis plugin in the `@suaveplan/audio-*` family. It computes magnitude spectra, spectrograms, MFCCs, and the scalar spectral features (centroid, rolloff, flux) that downstream packages — beat, pitch, and visualisation layers — build on. This document carries the **package-specific design**: the algorithm rationale, code sketches, file layout, integration notes, benchmarks, and the decisions behind them. --- ## 1. Relationship to the ecosystem architecture The cross-cutting `@suaveplan/audio-*` architecture — the three-layer separation, the six plugin slots, the zero-duplication rule, and the universal-tier purity constraint — lives in `enterprise-audio-core/design.md` §1–§4 and is **referenced, not duplicated** here. The only slot this package occupies is **Analysis Op (plugin slot 3)**: a sidecar processor that reads an `AudioDocument` and emits features. It is *not* part of the render pipeline — it never appears in a `CompiledPlan` or a `ProcessPipeline`. `SpectrumAnalyzer`, `Spectrogram`, and `MFCC` each implement the `AnalysisOp` contract defined by `@suaveplan/audio-core` and consume the `AudioDocument`, `BlockProcessor`, and `BlockContext` types from there. --- ## 2. Algorithm rationale ### 2.1. Real FFT (radix-2) The FFT is the hot path for every analyzer, so it is hand-rolled rather than pulled from a dependency: a portable radix-2 Cooley–Tukey decimation-in-time transform restricted to power-of-two frame sizes (`256`–`32768`). Restricting to power-of-two keeps the implementation small and branch-free and matches the natural `frameSize` choices for audio (`2048` default). Twiddle-factor tables (`cos`/`sin` lookup) are precomputed once per `frameSize` and cached, so a multi-frame scan allocates them exactly once. Because the input is real, only `frameSize / 2 + 1` bins are unique; the analyzer keeps just that half-spectrum. Determinism is a hard requirement — the same input frame always yields byte-identical magnitudes — which is what makes the property tests (sine → single peak) stable. The earlier draft mentioned delegating to a browser `AnalyserNode`. That path is **dropped from the universal core**: `AnalyserNode` is a browser global and would break universal-tier purity. If a browser engine wants hardware-accelerated FFT it injects an adapter; the package's own code stays portable. ### 2.2. Windowing Each frame is multiplied by a window before the FFT to suppress spectral leakage. Three windows ship — Hann (default), Hamming, Blackman — selectable per call. Hann is the default because it is the standard general-purpose choice for music analysis: a good main-lobe/side-lobe trade-off. Window arrays are pure functions of `(type, size)` and are cached, so repeated same-shape frames reuse one `Float32Array`. ```ts function hann(size: number): Float32Array { const w = new Float32Array(size) for (let n = 0; n < size; n++) w[n] = 0.5 - 0.5 * Math.cos((2 * Math.PI * n) / (size - 1)) return w } ``` ### 2.3. Frame slicing `SpectrumAnalyzer`, `Spectrogram`, and `MFCC` share one framing model: slide a `frameSize` window over the document advancing by `hopSize`. The frame count for a document of length `L` is `floor((L - frameSize) / hopSize) + 1`, and a document shorter than one frame yields an empty result rather than throwing — callers analysing very short clips should not have to guard. ### 2.4. Spectral features - **Centroid** — magnitude-weighted mean frequency; the "brightness" of the frame. - **Rolloff** — the frequency below which 85% of the spectral energy lies. - **Flux** — summed positive frame-to-frame magnitude difference; the raw onset signal that `@suaveplan/audio-analysis-beat` consumes. All three are cheap reductions over the magnitude spectrum already computed for the frame, so they are produced as a by-product rather than a second pass. ### 2.5. Mel scale and the triangular filterbank The mel scale is a perceptual frequency warping — roughly linear below 1 kHz, logarithmic above. The filterbank is a set of `melBands` triangular filters evenly spaced on the mel scale across `[minHz, maxHz]`; applying it to a magnitude spectrum collapses `frameSize / 2 + 1` FFT bins into `melBands` perceptually-spaced energies. The filter matrix is a pure function of `(frameSize, sampleRate, melBands, minHz, maxHz)` and is cached. ```ts const hzToMel = (hz: number) => 2595 * Math.log10(1 + hz / 700) const melToHz = (mel: number) => 700 * (10 ** (mel / 2595) - 1) ``` ### 2.6. MFCC — DCT-II on the log mel-spectrum MFCCs are the standard compact timbre descriptor. The pipeline per frame is: magnitude spectrum → mel filterbank → natural log of mel energies (floored to avoid `log(0)`) → type-II DCT → keep the first `coefficients` values (default `13`). The DCT decorrelates the log-mel vector so the low-order coefficients capture the spectral envelope. `coefficients` must not exceed `melBands` — asking for more cepstral coefficients than mel bands is meaningless and is rejected at the boundary. --- ## 3. Public surface and code sketch ```ts class SpectrumAnalyzer implements AnalysisOp { readonly name = 'spectrum-analyzer' readonly description = 'Magnitude spectrum + spectral centroid/rolloff/flux' async analyze(doc: AudioDocument, opts?: SpectrumAnalyzerOptions): Promise { const o = SpectrumAnalyzerOptionsSchema.parse(opts ?? {}) const win = window(o.window, o.frameSize) let prevMag: Float32Array | undefined // slide frames, fft(win · frame), reduce to bins + centroid/rolloff/flux } analyzeStream(doc: AudioDocument, opts?: SpectrumAnalyzerOptions): AsyncGenerator analyzeBlock(block: Float32Array[], ctx: BlockContext): SpectrumResult } ``` `Spectrogram` and `MFCC` follow the same shape; `Spectrogram` keeps every per-frame magnitude vector, `MFCC` runs the mel + log + DCT chain per frame. All option validation goes through the Zod schemas in `*.schemas.ts`; all errors are `AudioError` instances carrying an `AUDIO_ANALYSIS_*` code. --- ## 4. File layout ```text packages/media/audio/audio-analysis-spectrum/ package.json, tsconfig.json, vitest.config.ts src/ index.ts error-codes.ts analyzer/ spectrum-analyzer.ts (+ .md .types.ts .schemas.ts .test.ts) spectrogram.ts (+ .md .types.ts .schemas.ts .test.ts) mfcc.ts (+ .md .types.ts .schemas.ts .test.ts) index.ts algorithms/ fft.ts (+ .md .test.ts) window.ts (+ .md .test.ts) mel-scale.ts (+ .md .test.ts) dct.ts (+ .md .test.ts) index.ts __benchmarks__/ spectrum.bench.ts tests/ integration.test.ts property.test.ts purity.test.ts exports.test.ts ``` --- ## 5. Integration with genesis packages - **`@suaveplan/audio-core`** — supplies `AudioDocument` (the scan input), the `AnalysisOp` contract, and `BlockProcessor` / `BlockContext` for `analyzeBlock`. The analyzers iterate the document with `doc.stream()` / `doc.read()` rather than touching pages directly. - **`@suaveplan/error`** — `AudioError` base class; this package registers the `AUDIO_ANALYSIS` code namespace (`AUDIO_ANALYSIS_SPECTRUM.INVALID_OPTIONS`, `AUDIO_ANALYSIS_SPECTRUM.INVALID_FRAME`, `AUDIO_ANALYSIS_SPECTRUM.ABORTED`). - **`@suaveplan/opentelemetry`** — spans, counters, and histograms (`audio.analysis.runs_total`, `audio.analysis.duration_ms`, `audio.analysis.block_duration_ms`); inert with no exporter. - **`@suaveplan/types`** — `LoggerLike` for optional injected logging. Downstream, `@suaveplan/audio-analysis-beat` consumes the spectral-flux signal, and `@suaveplan/audio-analysis-pitch` builds its chromagram from the magnitude spectrum — but those are separate changes; this package exposes the surface, it does not import them. --- ## 6. Benchmarks - `spectrum.bench.ts` — spectrogram of a 1-minute 48 kHz track at `frameSize` `2048` / `hopSize` `512`, and MFCC of the same track at `13` coefficients / `40` mel bands. Establishes the per-frame FFT + reduction cost and the mel + DCT overhead. --- ## 7. Decisions and trade-offs - **No `AnalyserNode` in the universal core.** The draft suggested delegating FFT to the browser's `AnalyserNode`. That is a browser global and breaks universal-tier purity, so the package ships one portable radix-2 FFT used everywhere. A browser engine wanting hardware acceleration injects an adapter; the universal path never references a platform global. - **Power-of-two frame sizes only.** Restricting `frameSize` to `256`–`32768` powers of two keeps the FFT small, branch-free, and deterministic, and matches the natural audio frame sizes. A non-power-of-two request is a boundary error (`AUDIO_ANALYSIS_SPECTRUM.INVALID_FRAME`), not a slow Bluestein fallback. - **Three analyzers, one framing model.** `SpectrumAnalyzer`, `Spectrogram`, and `MFCC` share frame slicing, windowing, and the FFT; they differ only in the reduction. Splitting them into three `AnalysisOp` classes (rather than one mega-analyzer with flags) keeps each registerable independently and keeps each result type tight. - **Cached pure tables.** Window arrays, FFT twiddle factors, and mel filterbank matrices are all pure functions of their shape parameters and are cached. A long scan allocates each exactly once; this is the single biggest constant-factor win and is why the benchmark targets a full minute. - **Features as a by-product.** Centroid, rolloff, and flux are computed from the magnitude spectrum already in hand, not in a second pass — keeping `SpectrumAnalyzer` a single-pass scan.