Files

2026-05-27 23:29:53 +01:00

10 KiB

Raw Permalink Blame History

Design: enterprise-audio-analysis-pitch

Change ID: enterprise-audio-analysis-pitch Package: @suaveplan/audio-analysis-pitch

@suaveplan/audio-analysis-pitch is a universal-tier Analysis Op package in the @suaveplan/audio-* family. It detects fundamental frequency (pitch), musical key, and chord progressions from AudioDocument streams. This document carries the package-specific design: algorithm rationale, code sketches, file layout, and trade-offs.

The cross-cutting architecture for the whole audio ecosystem — the three-layer separation, the six plugin slots, the AnalysisOp contract, genesis-package reuse, and the AudioError hierarchy — lives in enterprise-audio-core/design.md §1–§4 and is referenced, not duplicated here. In particular: an Analysis Op is plugin slot 3 — a sidecar processor that reads audio and emits features, running concurrently with playback or as a standalone scan, never part of the render pipeline and never mutating the AudioDocument.

Scope note. Pitch detection / analysis lives here. Pitch shifting — DSP that changes the pitch of audio — is not in this package; it ships as the PitchShifterPlugin in @suaveplan/audio-plugins-fx (an FX plugin, slot 6). This package only measures pitch; it never alters it.

1. Package responsibilities

Three Analysis Op plugins, each implementing the AnalysisOp interface from @suaveplan/audio-core:

Plugin	`name`	Output	Core algorithms
`PitchDetector`	`pitch-detector`	`PitchResult` (`pitch`, `midiNote`, `confidence`)	YIN, McLeod (MPM), cepstrum
`KeyDetector`	`key-detector`	`KeyResult` (`key`, `confidence`, `chromagram`)	chromagram + Krumhansl-Schmuckler / Temperley
`ChordDetector`	`chord-detector`	`ChordResult` (`chords: Chord[]`)	per-frame chromagram + template matching + smoothing

All three support the full AnalysisOp triad where it makes sense: analyze (whole-document, blocking), analyzeStream (async generator, refines as audio is consumed), and — for PitchDetector — analyzeBlock (synchronous, single block, for real-time visualisation). Key and chord estimation need a longer observation window than a single block to be meaningful, so KeyDetector and ChordDetector expose analyze + analyzeStream only.

2. Pitch detection algorithms

2.1. YIN (default)

YIN (de Cheveigné & Kawahara, 2002) is the default because it is robust, has low octave-error rates, and needs no FFT. The pipeline over a frame of samples:

Difference function d(τ) = Σ (x[j] − x[j+τ])² for lag τ in the range implied by [minFreq, maxFreq].
Cumulative mean normalised difference d'(τ) = d(τ) / ((1/τ) Σ d(j)), with d'(0) = 1.
Absolute threshold — pick the first τ where d'(τ) drops below ~0.1; if none, take the global minimum.
Parabolic interpolation around the chosen τ for sub-sample precision.
pitch = sampleRate / τ; confidence = 1 − d'(τ).

function yinPitch(frame: Float32Array, sampleRate: number, minF: number, maxF: number): PitchEstimate {
  const tauMin = Math.floor(sampleRate / maxF)
  const tauMax = Math.min(Math.ceil(sampleRate / minF), frame.length >> 1)
  const dprime = cumulativeMeanNormalisedDifference(frame, tauMin, tauMax)
  const tau = absoluteThreshold(dprime, 0.1)
  if (tau < 0) return { pitch: 0, confidence: 0 }
  const refined = parabolicInterpolate(dprime, tau)
  return { pitch: sampleRate / refined, confidence: 1 - dprime[tau] }
}

2.2. McLeod Pitch Method (MPM)

MPM uses the normalised square difference function (NSDF) and key-maxima picking. It is stronger than YIN on signals with strong harmonics and slightly better at clean musical instrument tones, at higher cost. Selected via { algorithm: "mcleod" }.

2.3. Cepstrum

The cepstrum method takes the inverse FFT of the log magnitude spectrum and finds the quefrency peak. It is cheap when an FFT is already on hand and resilient to formant structure (useful for voice), but has coarser resolution than YIN/MPM. Selected via { algorithm: "cepstrum" }. The package ships its own small radix-2 real FFT for this path so it carries no hard dependency on @suaveplan/audio-analysis-spectrum.

2.4. MIDI mapping and the no-pitch case

midiNote = round(69 + 12 * log2(pitch / 440)). When no periodicity clears the algorithm's confidence floor (silence, broadband noise), the detector returns pitch: 0, midiNote: -1, confidence: 0 rather than guessing.

3. Key detection

3.1. Chromagram

A chromagram folds spectral energy into 12 pitch classes. Per analysis frame: window → real FFT → magnitude spectrum → map each bin's frequency to a pitch class via pitchClass = round(12 * log2(freq / C0)) mod 12 → accumulate magnitude into the bin. Frames are summed (or averaged) across the analysed range to produce one aggregate 12-bin Float32Array.

3.2. Krumhansl-Schmuckler key-finding

The aggregate chromagram is correlated against all 24 major/minor key profiles (each profile is a 12-vector of perceptual pitch-class weights, rotated to each tonic). The default profile set is Krumhansl-Schmuckler; { profile: "temperley" } swaps in Temperley's revised weights, which reduce relative-major/minor confusion. The best Pearson correlation wins; confidence is the normalised margin between the top score and the runner-up.

function detectKey(chroma: Float32Array, profiles: KeyProfileSet): KeyEstimate {
  let best = { key: "", score: -Infinity }
  let second = -Infinity
  for (const { name, vector } of profiles.all()) {
    const score = pearson(chroma, vector)
    if (score > best.score) { second = best.score; best = { key: name, score } }
    else if (score > second) { second = score }
  }
  return { key: best.key, confidence: clamp01((best.score - second) / Math.max(best.score, 1e-6)) }
}

3.3. Chord progression

ChordDetector computes a chromagram per frame (no aggregation), template-matches each frame against major / minor / diminished / augmented triad templates rotated to each root (48 templates), and applies temporal smoothing — a median/Viterbi-style pass over the frame labels — so a single noisy frame cannot spawn a spurious chord. Runs of identical labels are merged into one Chord record { timestamp, duration, root, quality, confidence }; no emitted chord is shorter than the smoothing window.

4. Streaming refinement

analyzeStream for all three detectors consumes doc.stream() block-by-block. Pitch yields a result per frame. Key and chord detectors maintain a running chromagram (or running frame buffer) and re-run the estimator on each yield, so confidence is non-decreasing in expectation as more audio arrives. An AbortSignal is threaded through the options object: pre-aborted → throw AUDIO_ANALYSIS_PITCH.ABORTED before any block work; aborted mid-stream → throw at the next block boundary. This mirrors the AbortSignal contract audio-core defines for AudioDocument.stream().

5. File layout

packages/media/audio/audio-analysis-pitch/
  package.json, tsconfig.json, vitest.config.ts
  src/
    index.ts
    error-codes.ts            (+ .md .test.ts)
    detector/
      pitch-detector.ts       (+ .md .types.ts .schemas.ts .test.ts)
      key-detector.ts         (+ .md .types.ts .schemas.ts .test.ts)
      chord-detector.ts       (+ .md .types.ts .schemas.ts .test.ts)
      index.ts
    algorithms/
      yin.ts                  (+ .md .test.ts)   — YIN + McLeod + cepstrum estimators
      chromagram.ts           (+ .md .test.ts)   — pitch-class folding + small real FFT
      key-finding.ts          (+ .md .test.ts)   — Krumhansl-Schmuckler + Temperley profiles
      chord-matching.ts       (+ .md .test.ts)   — triad templates + temporal smoothing
      index.ts
    __benchmarks__/
      pitch.bench.ts
  tests/
    integration.test.ts  property.test.ts  purity.test.ts  exports.test.ts

6. Benchmarks

pitch.bench.ts — PitchDetector.analyze over a 1-minute 48 kHz mono track (all three algorithms), and KeyDetector.analyze + ChordDetector.analyze over the same track.

7. Integration notes

@suaveplan/audio-core — provides AudioDocument, the AnalysisOp / AnalysisResult contract, BlockContext, and the AudioError base. PitchResult, KeyResult, and ChordResult all extend AnalysisResult.
@suaveplan/audio-analysis-spectrum — optional peer. When present, callers may feed its FFT output in; this package never imports it at runtime and ships its own FFT so it stays a leaf.
@suaveplan/opentelemetry — spans, counters, and histograms per detector run.
Consumers — application/timeline code registers these detectors as analysis ops; e.g. audio-plugins-fx auto-tune reads PitchResult, a React player surfaces KeyResult.key.

8. Decisions and trade-offs

YIN is the default, three algorithms ship. YIN is the best general-purpose default (no FFT, low octave error). MPM and cepstrum are offered because they win on specific material (harmonically rich instruments, voice). Per-call selection avoids forcing one trade-off on every caller.
Own FFT, no hard audio-analysis-spectrum dependency. A pitch package that could not run without the spectrum package would couple two leaves unnecessarily. The cepstrum path and the chromagram need only a small radix-2 real FFT, which this package vendors. audio-analysis-spectrum stays a sibling, not a prerequisite.
Pitch shift is explicitly out of scope. Detection and shifting are different concerns in different plugin slots. PitchShifterPlugin (phase-vocoder, stateful) belongs in audio-plugins-fx; this package is read-only over audio.
Chord detection ships here, not in a separate package. Key and chord estimation share the chromagram machinery and pitch-class theory; splitting them would duplicate that core. They live together behind separate AnalysisOp plugins.
Temporal smoothing on chords is mandatory, not optional. Raw per-frame chord labels are too noisy to be useful; the smoothing pass is part of the algorithm, not a tuning knob, so every caller gets a usable progression.

10 KiB Raw Permalink Blame History Unescape Escape