Design: enterprise-audio-analysis-spectrum

Change ID: enterprise-audio-analysis-spectrum Package: @suaveplan/audio-analysis-spectrum

@suaveplan/audio-analysis-spectrum is a universal-tier analysis plugin in the @suaveplan/audio-* family. It computes magnitude spectra, spectrograms, MFCCs, and the scalar spectral features (centroid, rolloff, flux) that downstream packages — beat, pitch, and visualisation layers — build on. This document carries the package-specific design: the algorithm rationale, code sketches, file layout, integration notes, benchmarks, and the decisions behind them.

1. Relationship to the ecosystem architecture

The cross-cutting @suaveplan/audio-* architecture — the three-layer separation, the six plugin slots, the zero-duplication rule, and the universal-tier purity constraint — lives in enterprise-audio-core/design.md §1–§4 and is referenced, not duplicated here.

The only slot this package occupies is Analysis Op (plugin slot 3): a sidecar processor that reads an AudioDocument and emits features. It is not part of the render pipeline — it never appears in a CompiledPlan or a ProcessPipeline. SpectrumAnalyzer, Spectrogram, and MFCC each implement the AnalysisOp contract defined by @suaveplan/audio-core and consume the AudioDocument, BlockProcessor, and BlockContext types from there.

2. Algorithm rationale

2.1. Real FFT (radix-2)

The FFT is the hot path for every analyzer, so it is hand-rolled rather than pulled from a dependency: a portable radix-2 Cooley–Tukey decimation-in-time transform restricted to power-of-two frame sizes (256–32768). Restricting to power-of-two keeps the implementation small and branch-free and matches the natural frameSize choices for audio (2048 default). Twiddle-factor tables (cos/sin lookup) are precomputed once per frameSize and cached, so a multi-frame scan allocates them exactly once. Because the input is real, only frameSize / 2 + 1 bins are unique; the analyzer keeps just that half-spectrum. Determinism is a hard requirement — the same input frame always yields byte-identical magnitudes — which is what makes the property tests (sine → single peak) stable.

The earlier draft mentioned delegating to a browser AnalyserNode. That path is dropped from the universal core: AnalyserNode is a browser global and would break universal-tier purity. If a browser engine wants hardware-accelerated FFT it injects an adapter; the package's own code stays portable.

2.2. Windowing

Each frame is multiplied by a window before the FFT to suppress spectral leakage. Three windows ship — Hann (default), Hamming, Blackman — selectable per call. Hann is the default because it is the standard general-purpose choice for music analysis: a good main-lobe/side-lobe trade-off. Window arrays are pure functions of (type, size) and are cached, so repeated same-shape frames reuse one Float32Array.

function hann(size: number): Float32Array {
  const w = new Float32Array(size)
  for (let n = 0; n < size; n++) w[n] = 0.5 - 0.5 * Math.cos((2 * Math.PI * n) / (size - 1))
  return w
}

2.3. Frame slicing

SpectrumAnalyzer, Spectrogram, and MFCC share one framing model: slide a frameSize window over the document advancing by hopSize. The frame count for a document of length L is floor((L - frameSize) / hopSize) + 1, and a document shorter than one frame yields an empty result rather than throwing — callers analysing very short clips should not have to guard.

2.4. Spectral features

Centroid — magnitude-weighted mean frequency; the "brightness" of the frame.
Rolloff — the frequency below which 85% of the spectral energy lies.
Flux — summed positive frame-to-frame magnitude difference; the raw onset signal that @suaveplan/audio-analysis-beat consumes.

All three are cheap reductions over the magnitude spectrum already computed for the frame, so they are produced as a by-product rather than a second pass.

2.5. Mel scale and the triangular filterbank

The mel scale is a perceptual frequency warping — roughly linear below 1 kHz, logarithmic above. The filterbank is a set of melBands triangular filters evenly spaced on the mel scale across [minHz, maxHz]; applying it to a magnitude spectrum collapses frameSize / 2 + 1 FFT bins into melBands perceptually-spaced energies. The filter matrix is a pure function of (frameSize, sampleRate, melBands, minHz, maxHz) and is cached.

const hzToMel = (hz: number) => 2595 * Math.log10(1 + hz / 700)
const melToHz = (mel: number) => 700 * (10 ** (mel / 2595) - 1)

2.6. MFCC — DCT-II on the log mel-spectrum

MFCCs are the standard compact timbre descriptor. The pipeline per frame is: magnitude spectrum → mel filterbank → natural log of mel energies (floored to avoid log(0)) → type-II DCT → keep the first coefficients values (default 13). The DCT decorrelates the log-mel vector so the low-order coefficients capture the spectral envelope. coefficients must not exceed melBands — asking for more cepstral coefficients than mel bands is meaningless and is rejected at the boundary.

3. Public surface and code sketch

class SpectrumAnalyzer implements AnalysisOp {
  readonly name = 'spectrum-analyzer'
  readonly description = 'Magnitude spectrum + spectral centroid/rolloff/flux'

  async analyze(doc: AudioDocument, opts?: SpectrumAnalyzerOptions): Promise<SpectrumResult> {
    const o = SpectrumAnalyzerOptionsSchema.parse(opts ?? {})
    const win = window(o.window, o.frameSize)
    let prevMag: Float32Array | undefined
    // slide frames, fft(win · frame), reduce to bins + centroid/rolloff/flux
  }

  analyzeStream(doc: AudioDocument, opts?: SpectrumAnalyzerOptions): AsyncGenerator<SpectrumResult>
  analyzeBlock(block: Float32Array[], ctx: BlockContext): SpectrumResult
}

Spectrogram and MFCC follow the same shape; Spectrogram keeps every per-frame magnitude vector, MFCC runs the mel + log + DCT chain per frame. All option validation goes through the Zod schemas in *.schemas.ts; all errors are AudioError instances carrying an AUDIO_ANALYSIS_* code.

4. File layout

packages/media/audio/audio-analysis-spectrum/
  package.json, tsconfig.json, vitest.config.ts
  src/
    index.ts
    error-codes.ts
    analyzer/
      spectrum-analyzer.ts (+ .md .types.ts .schemas.ts .test.ts)
      spectrogram.ts        (+ .md .types.ts .schemas.ts .test.ts)
      mfcc.ts               (+ .md .types.ts .schemas.ts .test.ts)
      index.ts
    algorithms/
      fft.ts        (+ .md .test.ts)
      window.ts     (+ .md .test.ts)
      mel-scale.ts  (+ .md .test.ts)
      dct.ts        (+ .md .test.ts)
      index.ts
    __benchmarks__/
      spectrum.bench.ts
  tests/
    integration.test.ts  property.test.ts  purity.test.ts  exports.test.ts

5. Integration with genesis packages

@suaveplan/audio-core — supplies AudioDocument (the scan input), the AnalysisOp contract, and BlockProcessor / BlockContext for analyzeBlock. The analyzers iterate the document with doc.stream() / doc.read() rather than touching pages directly.
@suaveplan/error — AudioError base class; this package registers the AUDIO_ANALYSIS code namespace (AUDIO_ANALYSIS_SPECTRUM.INVALID_OPTIONS, AUDIO_ANALYSIS_SPECTRUM.INVALID_FRAME, AUDIO_ANALYSIS_SPECTRUM.ABORTED).
@suaveplan/opentelemetry — spans, counters, and histograms (audio.analysis.runs_total, audio.analysis.duration_ms, audio.analysis.block_duration_ms); inert with no exporter.
@suaveplan/types — LoggerLike for optional injected logging.

Downstream, @suaveplan/audio-analysis-beat consumes the spectral-flux signal, and @suaveplan/audio-analysis-pitch builds its chromagram from the magnitude spectrum — but those are separate changes; this package exposes the surface, it does not import them.

6. Benchmarks

spectrum.bench.ts — spectrogram of a 1-minute 48 kHz track at frameSize 2048 / hopSize 512, and MFCC of the same track at 13 coefficients / 40 mel bands. Establishes the per-frame FFT + reduction cost and the mel + DCT overhead.

7. Decisions and trade-offs

No AnalyserNode in the universal core. The draft suggested delegating FFT to the browser's AnalyserNode. That is a browser global and breaks universal-tier purity, so the package ships one portable radix-2 FFT used everywhere. A browser engine wanting hardware acceleration injects an adapter; the universal path never references a platform global.
Power-of-two frame sizes only. Restricting frameSize to 256–32768 powers of two keeps the FFT small, branch-free, and deterministic, and matches the natural audio frame sizes. A non-power-of-two request is a boundary error (AUDIO_ANALYSIS_SPECTRUM.INVALID_FRAME), not a slow Bluestein fallback.
Three analyzers, one framing model. SpectrumAnalyzer, Spectrogram, and MFCC share frame slicing, windowing, and the FFT; they differ only in the reduction. Splitting them into three AnalysisOp classes (rather than one mega-analyzer with flags) keeps each registerable independently and keeps each result type tight.
Cached pure tables. Window arrays, FFT twiddle factors, and mel filterbank matrices are all pure functions of their shape parameters and are cached. A long scan allocates each exactly once; this is the single biggest constant-factor win and is why the benchmark targets a full minute.
Features as a by-product. Centroid, rolloff, and flux are computed from the magnitude spectrum already in hand, not in a second pass — keeping SpectrumAnalyzer a single-pass scan.

9.7 KiB Raw Permalink Blame History Unescape Escape