# Design: enterprise-audio-analysis-pitch **Change ID**: enterprise-audio-analysis-pitch **Package**: `@suaveplan/audio-analysis-pitch` `@suaveplan/audio-analysis-pitch` is a universal-tier Analysis Op package in the `@suaveplan/audio-*` family. It detects fundamental frequency (pitch), musical key, and chord progressions from `AudioDocument` streams. This document carries the **package-specific design**: algorithm rationale, code sketches, file layout, and trade-offs. The **cross-cutting architecture** for the whole audio ecosystem — the three-layer separation, the six plugin slots, the `AnalysisOp` contract, genesis-package reuse, and the `AudioError` hierarchy — lives in `enterprise-audio-core/design.md` §1–§4 and is **referenced, not duplicated** here. In particular: an Analysis Op is **plugin slot 3** — a sidecar processor that reads audio and emits features, running concurrently with playback or as a standalone scan, never part of the render pipeline and never mutating the `AudioDocument`. > **Scope note.** Pitch *detection / analysis* lives here. Pitch *shifting* — DSP that changes the > pitch of audio — is **not** in this package; it ships as the `PitchShifterPlugin` in > `@suaveplan/audio-plugins-fx` (an FX plugin, slot 6). This package only measures pitch; it never > alters it. --- ## 1. Package responsibilities Three Analysis Op plugins, each implementing the `AnalysisOp` interface from `@suaveplan/audio-core`: | Plugin | `name` | Output | Core algorithms | |---|---|---|---| | `PitchDetector` | `pitch-detector` | `PitchResult` (`pitch`, `midiNote`, `confidence`) | YIN, McLeod (MPM), cepstrum | | `KeyDetector` | `key-detector` | `KeyResult` (`key`, `confidence`, `chromagram`) | chromagram + Krumhansl-Schmuckler / Temperley | | `ChordDetector` | `chord-detector` | `ChordResult` (`chords: Chord[]`) | per-frame chromagram + template matching + smoothing | All three support the full `AnalysisOp` triad where it makes sense: `analyze` (whole-document, blocking), `analyzeStream` (async generator, refines as audio is consumed), and — for `PitchDetector` — `analyzeBlock` (synchronous, single block, for real-time visualisation). Key and chord estimation need a longer observation window than a single block to be meaningful, so `KeyDetector` and `ChordDetector` expose `analyze` + `analyzeStream` only. --- ## 2. Pitch detection algorithms ### 2.1. YIN (default) YIN (de Cheveigné & Kawahara, 2002) is the default because it is robust, has low octave-error rates, and needs no FFT. The pipeline over a frame of samples: 1. **Difference function** `d(τ) = Σ (x[j] − x[j+τ])²` for lag `τ` in the range implied by `[minFreq, maxFreq]`. 2. **Cumulative mean normalised difference** `d'(τ) = d(τ) / ((1/τ) Σ d(j))`, with `d'(0) = 1`. 3. **Absolute threshold** — pick the first `τ` where `d'(τ)` drops below ~0.1; if none, take the global minimum. 4. **Parabolic interpolation** around the chosen `τ` for sub-sample precision. 5. `pitch = sampleRate / τ`; `confidence = 1 − d'(τ)`. ```ts function yinPitch(frame: Float32Array, sampleRate: number, minF: number, maxF: number): PitchEstimate { const tauMin = Math.floor(sampleRate / maxF) const tauMax = Math.min(Math.ceil(sampleRate / minF), frame.length >> 1) const dprime = cumulativeMeanNormalisedDifference(frame, tauMin, tauMax) const tau = absoluteThreshold(dprime, 0.1) if (tau < 0) return { pitch: 0, confidence: 0 } const refined = parabolicInterpolate(dprime, tau) return { pitch: sampleRate / refined, confidence: 1 - dprime[tau] } } ``` ### 2.2. McLeod Pitch Method (MPM) MPM uses the normalised square difference function (NSDF) and key-maxima picking. It is stronger than YIN on signals with strong harmonics and slightly better at clean musical instrument tones, at higher cost. Selected via `{ algorithm: "mcleod" }`. ### 2.3. Cepstrum The cepstrum method takes the inverse FFT of the log magnitude spectrum and finds the quefrency peak. It is cheap when an FFT is already on hand and resilient to formant structure (useful for voice), but has coarser resolution than YIN/MPM. Selected via `{ algorithm: "cepstrum" }`. The package ships its own small radix-2 real FFT for this path so it carries **no hard dependency** on `@suaveplan/audio-analysis-spectrum`. ### 2.4. MIDI mapping and the no-pitch case `midiNote = round(69 + 12 * log2(pitch / 440))`. When no periodicity clears the algorithm's confidence floor (silence, broadband noise), the detector returns `pitch: 0`, `midiNote: -1`, `confidence: 0` rather than guessing. --- ## 3. Key detection ### 3.1. Chromagram A chromagram folds spectral energy into 12 pitch classes. Per analysis frame: window → real FFT → magnitude spectrum → map each bin's frequency to a pitch class via `pitchClass = round(12 * log2(freq / C0)) mod 12` → accumulate magnitude into the bin. Frames are summed (or averaged) across the analysed range to produce one aggregate 12-bin `Float32Array`. ### 3.2. Krumhansl-Schmuckler key-finding The aggregate chromagram is correlated against all 24 major/minor **key profiles** (each profile is a 12-vector of perceptual pitch-class weights, rotated to each tonic). The default profile set is Krumhansl-Schmuckler; `{ profile: "temperley" }` swaps in Temperley's revised weights, which reduce relative-major/minor confusion. The best Pearson correlation wins; `confidence` is the normalised margin between the top score and the runner-up. ```ts function detectKey(chroma: Float32Array, profiles: KeyProfileSet): KeyEstimate { let best = { key: "", score: -Infinity } let second = -Infinity for (const { name, vector } of profiles.all()) { const score = pearson(chroma, vector) if (score > best.score) { second = best.score; best = { key: name, score } } else if (score > second) { second = score } } return { key: best.key, confidence: clamp01((best.score - second) / Math.max(best.score, 1e-6)) } } ``` ### 3.3. Chord progression `ChordDetector` computes a chromagram **per frame** (no aggregation), template-matches each frame against major / minor / diminished / augmented triad templates rotated to each root (48 templates), and applies temporal smoothing — a median/Viterbi-style pass over the frame labels — so a single noisy frame cannot spawn a spurious chord. Runs of identical labels are merged into one `Chord` record `{ timestamp, duration, root, quality, confidence }`; no emitted chord is shorter than the smoothing window. --- ## 4. Streaming refinement `analyzeStream` for all three detectors consumes `doc.stream()` block-by-block. Pitch yields a result per frame. Key and chord detectors maintain a running chromagram (or running frame buffer) and re-run the estimator on each yield, so `confidence` is non-decreasing in expectation as more audio arrives. An `AbortSignal` is threaded through the options object: pre-aborted → throw `AUDIO_ANALYSIS_PITCH.ABORTED` before any block work; aborted mid-stream → throw at the next block boundary. This mirrors the `AbortSignal` contract `audio-core` defines for `AudioDocument.stream()`. --- ## 5. File layout ```text packages/media/audio/audio-analysis-pitch/ package.json, tsconfig.json, vitest.config.ts src/ index.ts error-codes.ts (+ .md .test.ts) detector/ pitch-detector.ts (+ .md .types.ts .schemas.ts .test.ts) key-detector.ts (+ .md .types.ts .schemas.ts .test.ts) chord-detector.ts (+ .md .types.ts .schemas.ts .test.ts) index.ts algorithms/ yin.ts (+ .md .test.ts) — YIN + McLeod + cepstrum estimators chromagram.ts (+ .md .test.ts) — pitch-class folding + small real FFT key-finding.ts (+ .md .test.ts) — Krumhansl-Schmuckler + Temperley profiles chord-matching.ts (+ .md .test.ts) — triad templates + temporal smoothing index.ts __benchmarks__/ pitch.bench.ts tests/ integration.test.ts property.test.ts purity.test.ts exports.test.ts ``` --- ## 6. Benchmarks - `pitch.bench.ts` — `PitchDetector.analyze` over a 1-minute 48 kHz mono track (all three algorithms), and `KeyDetector.analyze` + `ChordDetector.analyze` over the same track. --- ## 7. Integration notes - **`@suaveplan/audio-core`** — provides `AudioDocument`, the `AnalysisOp` / `AnalysisResult` contract, `BlockContext`, and the `AudioError` base. `PitchResult`, `KeyResult`, and `ChordResult` all extend `AnalysisResult`. - **`@suaveplan/audio-analysis-spectrum`** — *optional* peer. When present, callers may feed its FFT output in; this package never imports it at runtime and ships its own FFT so it stays a leaf. - **`@suaveplan/opentelemetry`** — spans, counters, and histograms per detector run. - **Consumers** — application/timeline code registers these detectors as analysis ops; e.g. `audio-plugins-fx` auto-tune reads `PitchResult`, a React player surfaces `KeyResult.key`. --- ## 8. Decisions and trade-offs - **YIN is the default, three algorithms ship.** YIN is the best general-purpose default (no FFT, low octave error). MPM and cepstrum are offered because they win on specific material (harmonically rich instruments, voice). Per-call selection avoids forcing one trade-off on every caller. - **Own FFT, no hard `audio-analysis-spectrum` dependency.** A pitch package that could not run without the spectrum package would couple two leaves unnecessarily. The cepstrum path and the chromagram need only a small radix-2 real FFT, which this package vendors. `audio-analysis-spectrum` stays a *sibling*, not a *prerequisite*. - **Pitch shift is explicitly out of scope.** Detection and shifting are different concerns in different plugin slots. `PitchShifterPlugin` (phase-vocoder, stateful) belongs in `audio-plugins-fx`; this package is read-only over audio. - **Chord detection ships here, not in a separate package.** Key and chord estimation share the chromagram machinery and pitch-class theory; splitting them would duplicate that core. They live together behind separate `AnalysisOp` plugins. - **Temporal smoothing on chords is mandatory, not optional.** Raw per-frame chord labels are too noisy to be useful; the smoothing pass is part of the algorithm, not a tuning knob, so every caller gets a usable progression.