Design — enterprise-react-audio-fingerprint

@suaveplan/react-audio-fingerprint is a focused package: client-side audio fingerprinting + matching, exposed as React hooks and components. Its job is to take a MediaSource | URL | ArrayBuffer | AudioBuffer, compute a constellation fingerprint, and either store / match it against a corpus. Every piece of generic audio handling — decode, FFT, spectrogram — is delegated to existing audio-family packages so this package only owns the fingerprinting algorithm and its UI surface.

1. The constellation algorithm

The package implements the Shazam-style algorithm:

Decode the input audio to PCM via audio-engine-browser's decode path.
Window + FFT through @suaveplan/audio-analysis-spectrum's FFT primitives (4096-point windows, Hann window, 75 % overlap by default).
2D local-peak detection in the resulting spectrogram (per-frame top-N peaks above a noise floor; configurable peaksPerFrame and noiseFloorDb).
Constellation pair hashing: for each peak, look at the next fanout peaks within a targetZoneSec time window; emit a hash (f1, f2, dt) quantised into a 32-bit integer.
Output is a sorted list of (hash: number, timeOffsetSec: number) pairs — the Fingerprint shape.

This sits in src/fingerprint/audio-fingerprinter.ts and src/fingerprint/constellation.ts. Tests verify deterministic output for identical inputs, partial-match resilience under noise / pitch-shift / time-stretch (within tolerance), and stable output across browsers (FFT result equivalence guaranteed by audio-analysis-spectrum).

2. Matching

FingerprintMatcher exposes:

class FingerprintMatcher {
  match(query: Fingerprint, corpus: Iterable<{ id: string; fingerprint: Fingerprint }>): MatchResult[]
  matchOne(query: Fingerprint, candidate: Fingerprint): { score: number; offsetSec: number }
}

The basic match uses time-offset histogramming: for each (query, candidate) pair, compute the time-offset for each shared hash, bin into 10 ms buckets, and the maximum bucket count is the match score. The offset of that bucket gives the time alignment. This is robust to partial matches (the query is a clip of the full reference) and outperforms naive Jaccard.

matchOne is exposed for direct two-fingerprint comparison; match iterates the corpus. For corpora over ~10k, the package recommends the optional python-audio-fingerprint backend via the injectable FingerprintBackend interface — the same match() API, different storage and indexing internals.

3. Corpus storage

The package ships two corpus implementations:

MemoryCorpus — Map<string, Fingerprint> for tests / small corpora.
IndexedDbCorpus — IndexedDB-backed for browser sessions; persists across page reloads.

Both implement Corpus:

interface Corpus {
  add(id: string, fingerprint: Fingerprint): Promise<void>
  remove(id: string): Promise<void>
  get(id: string): Promise<Fingerprint | undefined>
  values(): AsyncIterable<{ id: string; fingerprint: Fingerprint }>
  size(): Promise<number>
}

Consumers can implement their own (Postgres-via-API, S3, etc.).

4. Backend abstraction

The FingerprintBackend interface lets consumers swap out the local fingerprinting algorithm for a remote service:

interface FingerprintBackend {
  fingerprint(source: AudioBuffer): Promise<Fingerprint>
  match(query: Fingerprint, corpusRef: string): Promise<MatchResult[]>
}

LocalBackend (default) runs the algorithm in-browser. PythonBackend (optional subpath; requires @suaveplan/python-audio-fingerprint peer) delegates both fingerprint and match to a backend service over HTTP/WebSocket. Hooks accept a backend?: FingerprintBackend prop and default to LocalBackend.

5. Hooks

useAudioFingerprint(source: AudioSource, opts?: { backend?: FingerprintBackend, signal?: AbortSignal }): {
  fingerprint: Fingerprint | undefined
  state: 'idle' | 'decoding' | 'fingerprinting' | 'done' | 'error'
  error: ReactAudioFingerprintError | undefined
  progress: number  // 0..1
}

useFingerprintMatcher(corpus: Corpus, opts?: { backend?: FingerprintBackend }): {
  match(query: Fingerprint): Promise<MatchResult[]>
  matching: boolean
  lastResults: MatchResult[]
}

useAudioFingerprint is concurrent-rendering safe (useSyncExternalStore-backed). It uses AbortSignal so navigating away cancels long-running fingerprints cleanly.

6. Components

<AudioFingerprintUploader onFingerprint={fp => ...}> — accepts drag-drop, file picker, URL paste. Renders progress via useAudioFingerprint. Errors surface in an accessible toast.
<FingerprintMatchDisplay results> — ranked list of { id, score, offsetSec } with confidence bars (score normalised). Click-to-play if the corpus entry provides a playable URL.
<FingerprintVisualizer fingerprint> — overlays constellation peaks on a spectrogram (via audio-analysis-spectrum's spectrogram render). Useful for debugging / explainability.

All three are keyboard-accessible per the react-audio accessibility requirement.

7. Decisions and trade-offs

FFT is delegated, not duplicated. One FFT implementation across the audio family.
Time-offset histogram for matching, not raw Jaccard. Robust to partial matches; matches the published Shazam method.
Backend abstraction. Local default; server-delegation opt-in via subpath. Same hook API both ways.
Two corpus implementations + an injectable interface. Memory for tests, IndexedDB for sessions, custom for production.
AbortSignal everywhere. Long-running fingerprints don't leak on unmount.
Visualizer reuses spectrogram render from audio-analysis-spectrum. No new render path.

5.7 KiB Raw Permalink Blame History