Initial Commit
This commit is contained in:
@@ -0,0 +1,209 @@
|
||||
# Design: enterprise-audio-analysis-pitch
|
||||
|
||||
**Change ID**: enterprise-audio-analysis-pitch
|
||||
**Package**: `@suaveplan/audio-analysis-pitch`
|
||||
|
||||
`@suaveplan/audio-analysis-pitch` is a universal-tier Analysis Op package in the `@suaveplan/audio-*`
|
||||
family. It detects fundamental frequency (pitch), musical key, and chord progressions from
|
||||
`AudioDocument` streams. This document carries the **package-specific design**: algorithm rationale,
|
||||
code sketches, file layout, and trade-offs.
|
||||
|
||||
The **cross-cutting architecture** for the whole audio ecosystem — the three-layer separation, the
|
||||
six plugin slots, the `AnalysisOp` contract, genesis-package reuse, and the `AudioError` hierarchy —
|
||||
lives in `enterprise-audio-core/design.md` §1–§4 and is **referenced, not duplicated** here. In
|
||||
particular: an Analysis Op is **plugin slot 3** — a sidecar processor that reads audio and emits
|
||||
features, running concurrently with playback or as a standalone scan, never part of the render
|
||||
pipeline and never mutating the `AudioDocument`.
|
||||
|
||||
> **Scope note.** Pitch *detection / analysis* lives here. Pitch *shifting* — DSP that changes the
|
||||
> pitch of audio — is **not** in this package; it ships as the `PitchShifterPlugin` in
|
||||
> `@suaveplan/audio-plugins-fx` (an FX plugin, slot 6). This package only measures pitch; it never
|
||||
> alters it.
|
||||
|
||||
---
|
||||
|
||||
## 1. Package responsibilities
|
||||
|
||||
Three Analysis Op plugins, each implementing the `AnalysisOp` interface from `@suaveplan/audio-core`:
|
||||
|
||||
| Plugin | `name` | Output | Core algorithms |
|
||||
|---|---|---|---|
|
||||
| `PitchDetector` | `pitch-detector` | `PitchResult` (`pitch`, `midiNote`, `confidence`) | YIN, McLeod (MPM), cepstrum |
|
||||
| `KeyDetector` | `key-detector` | `KeyResult` (`key`, `confidence`, `chromagram`) | chromagram + Krumhansl-Schmuckler / Temperley |
|
||||
| `ChordDetector` | `chord-detector` | `ChordResult` (`chords: Chord[]`) | per-frame chromagram + template matching + smoothing |
|
||||
|
||||
All three support the full `AnalysisOp` triad where it makes sense: `analyze` (whole-document,
|
||||
blocking), `analyzeStream` (async generator, refines as audio is consumed), and — for
|
||||
`PitchDetector` — `analyzeBlock` (synchronous, single block, for real-time visualisation). Key and
|
||||
chord estimation need a longer observation window than a single block to be meaningful, so
|
||||
`KeyDetector` and `ChordDetector` expose `analyze` + `analyzeStream` only.
|
||||
|
||||
---
|
||||
|
||||
## 2. Pitch detection algorithms
|
||||
|
||||
### 2.1. YIN (default)
|
||||
|
||||
YIN (de Cheveigné & Kawahara, 2002) is the default because it is robust, has low octave-error rates,
|
||||
and needs no FFT. The pipeline over a frame of samples:
|
||||
|
||||
1. **Difference function** `d(τ) = Σ (x[j] − x[j+τ])²` for lag `τ` in the range implied by
|
||||
`[minFreq, maxFreq]`.
|
||||
2. **Cumulative mean normalised difference** `d'(τ) = d(τ) / ((1/τ) Σ d(j))`, with `d'(0) = 1`.
|
||||
3. **Absolute threshold** — pick the first `τ` where `d'(τ)` drops below ~0.1; if none, take the
|
||||
global minimum.
|
||||
4. **Parabolic interpolation** around the chosen `τ` for sub-sample precision.
|
||||
5. `pitch = sampleRate / τ`; `confidence = 1 − d'(τ)`.
|
||||
|
||||
```ts
|
||||
function yinPitch(frame: Float32Array, sampleRate: number, minF: number, maxF: number): PitchEstimate {
|
||||
const tauMin = Math.floor(sampleRate / maxF)
|
||||
const tauMax = Math.min(Math.ceil(sampleRate / minF), frame.length >> 1)
|
||||
const dprime = cumulativeMeanNormalisedDifference(frame, tauMin, tauMax)
|
||||
const tau = absoluteThreshold(dprime, 0.1)
|
||||
if (tau < 0) return { pitch: 0, confidence: 0 }
|
||||
const refined = parabolicInterpolate(dprime, tau)
|
||||
return { pitch: sampleRate / refined, confidence: 1 - dprime[tau] }
|
||||
}
|
||||
```
|
||||
|
||||
### 2.2. McLeod Pitch Method (MPM)
|
||||
|
||||
MPM uses the normalised square difference function (NSDF) and key-maxima picking. It is stronger than
|
||||
YIN on signals with strong harmonics and slightly better at clean musical instrument tones, at higher
|
||||
cost. Selected via `{ algorithm: "mcleod" }`.
|
||||
|
||||
### 2.3. Cepstrum
|
||||
|
||||
The cepstrum method takes the inverse FFT of the log magnitude spectrum and finds the quefrency peak.
|
||||
It is cheap when an FFT is already on hand and resilient to formant structure (useful for voice), but
|
||||
has coarser resolution than YIN/MPM. Selected via `{ algorithm: "cepstrum" }`. The package ships its
|
||||
own small radix-2 real FFT for this path so it carries **no hard dependency** on
|
||||
`@suaveplan/audio-analysis-spectrum`.
|
||||
|
||||
### 2.4. MIDI mapping and the no-pitch case
|
||||
|
||||
`midiNote = round(69 + 12 * log2(pitch / 440))`. When no periodicity clears the algorithm's
|
||||
confidence floor (silence, broadband noise), the detector returns `pitch: 0`, `midiNote: -1`,
|
||||
`confidence: 0` rather than guessing.
|
||||
|
||||
---
|
||||
|
||||
## 3. Key detection
|
||||
|
||||
### 3.1. Chromagram
|
||||
|
||||
A chromagram folds spectral energy into 12 pitch classes. Per analysis frame: window → real FFT →
|
||||
magnitude spectrum → map each bin's frequency to a pitch class via
|
||||
`pitchClass = round(12 * log2(freq / C0)) mod 12` → accumulate magnitude into the bin. Frames are
|
||||
summed (or averaged) across the analysed range to produce one aggregate 12-bin `Float32Array`.
|
||||
|
||||
### 3.2. Krumhansl-Schmuckler key-finding
|
||||
|
||||
The aggregate chromagram is correlated against all 24 major/minor **key profiles** (each profile is a
|
||||
12-vector of perceptual pitch-class weights, rotated to each tonic). The default profile set is
|
||||
Krumhansl-Schmuckler; `{ profile: "temperley" }` swaps in Temperley's revised weights, which reduce
|
||||
relative-major/minor confusion. The best Pearson correlation wins; `confidence` is the normalised
|
||||
margin between the top score and the runner-up.
|
||||
|
||||
```ts
|
||||
function detectKey(chroma: Float32Array, profiles: KeyProfileSet): KeyEstimate {
|
||||
let best = { key: "", score: -Infinity }
|
||||
let second = -Infinity
|
||||
for (const { name, vector } of profiles.all()) {
|
||||
const score = pearson(chroma, vector)
|
||||
if (score > best.score) { second = best.score; best = { key: name, score } }
|
||||
else if (score > second) { second = score }
|
||||
}
|
||||
return { key: best.key, confidence: clamp01((best.score - second) / Math.max(best.score, 1e-6)) }
|
||||
}
|
||||
```
|
||||
|
||||
### 3.3. Chord progression
|
||||
|
||||
`ChordDetector` computes a chromagram **per frame** (no aggregation), template-matches each frame
|
||||
against major / minor / diminished / augmented triad templates rotated to each root (48 templates),
|
||||
and applies temporal smoothing — a median/Viterbi-style pass over the frame labels — so a single
|
||||
noisy frame cannot spawn a spurious chord. Runs of identical labels are merged into one `Chord`
|
||||
record `{ timestamp, duration, root, quality, confidence }`; no emitted chord is shorter than the
|
||||
smoothing window.
|
||||
|
||||
---
|
||||
|
||||
## 4. Streaming refinement
|
||||
|
||||
`analyzeStream` for all three detectors consumes `doc.stream()` block-by-block. Pitch yields a result
|
||||
per frame. Key and chord detectors maintain a running chromagram (or running frame buffer) and re-run
|
||||
the estimator on each yield, so `confidence` is non-decreasing in expectation as more audio arrives.
|
||||
An `AbortSignal` is threaded through the options object: pre-aborted → throw `AUDIO_ANALYSIS_PITCH.ABORTED`
|
||||
before any block work; aborted mid-stream → throw at the next block boundary. This mirrors the
|
||||
`AbortSignal` contract `audio-core` defines for `AudioDocument.stream()`.
|
||||
|
||||
---
|
||||
|
||||
## 5. File layout
|
||||
|
||||
```text
|
||||
packages/media/audio/audio-analysis-pitch/
|
||||
package.json, tsconfig.json, vitest.config.ts
|
||||
src/
|
||||
index.ts
|
||||
error-codes.ts (+ .md .test.ts)
|
||||
detector/
|
||||
pitch-detector.ts (+ .md .types.ts .schemas.ts .test.ts)
|
||||
key-detector.ts (+ .md .types.ts .schemas.ts .test.ts)
|
||||
chord-detector.ts (+ .md .types.ts .schemas.ts .test.ts)
|
||||
index.ts
|
||||
algorithms/
|
||||
yin.ts (+ .md .test.ts) — YIN + McLeod + cepstrum estimators
|
||||
chromagram.ts (+ .md .test.ts) — pitch-class folding + small real FFT
|
||||
key-finding.ts (+ .md .test.ts) — Krumhansl-Schmuckler + Temperley profiles
|
||||
chord-matching.ts (+ .md .test.ts) — triad templates + temporal smoothing
|
||||
index.ts
|
||||
__benchmarks__/
|
||||
pitch.bench.ts
|
||||
tests/
|
||||
integration.test.ts property.test.ts purity.test.ts exports.test.ts
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Benchmarks
|
||||
|
||||
- `pitch.bench.ts` — `PitchDetector.analyze` over a 1-minute 48 kHz mono track (all three
|
||||
algorithms), and `KeyDetector.analyze` + `ChordDetector.analyze` over the same track.
|
||||
|
||||
---
|
||||
|
||||
## 7. Integration notes
|
||||
|
||||
- **`@suaveplan/audio-core`** — provides `AudioDocument`, the `AnalysisOp` / `AnalysisResult`
|
||||
contract, `BlockContext`, and the `AudioError` base. `PitchResult`, `KeyResult`, and `ChordResult`
|
||||
all extend `AnalysisResult`.
|
||||
- **`@suaveplan/audio-analysis-spectrum`** — *optional* peer. When present, callers may feed its FFT
|
||||
output in; this package never imports it at runtime and ships its own FFT so it stays a leaf.
|
||||
- **`@suaveplan/opentelemetry`** — spans, counters, and histograms per detector run.
|
||||
- **Consumers** — application/timeline code registers these detectors as analysis ops; e.g.
|
||||
`audio-plugins-fx` auto-tune reads `PitchResult`, a React player surfaces `KeyResult.key`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Decisions and trade-offs
|
||||
|
||||
- **YIN is the default, three algorithms ship.** YIN is the best general-purpose default (no FFT,
|
||||
low octave error). MPM and cepstrum are offered because they win on specific material
|
||||
(harmonically rich instruments, voice). Per-call selection avoids forcing one trade-off on every
|
||||
caller.
|
||||
- **Own FFT, no hard `audio-analysis-spectrum` dependency.** A pitch package that could not run
|
||||
without the spectrum package would couple two leaves unnecessarily. The cepstrum path and the
|
||||
chromagram need only a small radix-2 real FFT, which this package vendors. `audio-analysis-spectrum`
|
||||
stays a *sibling*, not a *prerequisite*.
|
||||
- **Pitch shift is explicitly out of scope.** Detection and shifting are different concerns in
|
||||
different plugin slots. `PitchShifterPlugin` (phase-vocoder, stateful) belongs in
|
||||
`audio-plugins-fx`; this package is read-only over audio.
|
||||
- **Chord detection ships here, not in a separate package.** Key and chord estimation share the
|
||||
chromagram machinery and pitch-class theory; splitting them would duplicate that core. They live
|
||||
together behind separate `AnalysisOp` plugins.
|
||||
- **Temporal smoothing on chords is mandatory, not optional.** Raw per-frame chord labels are too
|
||||
noisy to be useful; the smoothing pass is part of the algorithm, not a tuning knob, so every
|
||||
caller gets a usable progression.
|
||||
Reference in New Issue
Block a user