suaveplan/openspec/archive/2026-05-23-enterprise-audio-codec-ffmpeg-wasm/design.md

Design: @suaveplan/audio-codec-ffmpeg-wasm

Non-normative design notes. The normative contract lives in ./specs/audio-codec-ffmpeg-wasm/spec.md.

---

Why wasm + a Web Worker

@suaveplan/audio-codecs defines a tier-agnostic plugin contract. The server-tier audio-codec-ffmpeg-native package wraps a system ffmpeg binary via subprocess. The browser cannot spawn subprocesses, and there is no ffmpeg analogue baked into Web Audio. The only credible way to reach the same format coverage in a sandboxed page is ffmpeg.wasm — ffmpeg compiled to WebAssembly and exposed via the @ffmpeg/ffmpeg JS façade.

Two non-negotiable design choices follow from that:

1. The wasm core runs inside a Worker. Decoding a 5-minute MP3 takes hundreds of milliseconds of solid CPU; doing that on the main thread blocks input handling, layout, and the compositor. @ffmpeg/ffmpeg already supports a worker boundary; we always use it. The worker is the unit of cancellation: AbortSignal.abort translates to worker.terminate(), which is the only reliable way to interrupt wasm.
2. The wasm payload is lazy-loaded. @ffmpeg/core is ~30 MB; eagerly fetching it at import time would crater page load. The codec stays inert at module load and pulls the core only on the first decode/encode/createDecoder/createEncoder call. A second call reuses the loaded core; dispose() tears it down.

Lazy load strategy

The core-loader module exposes loadCore(opts): Promise<LoadedCore> and memoises the result per FfmpegWasmCodec instance. Resolution order for the core URL:

1. opts.corePath (explicit local URL) — preferred for self-hosted, COEP-correct serving.
2. opts.coreUrl (explicit CDN URL).
3. The package's own bundled default (unpkg.com/@ffmpeg/core@x.y.z/dist/ffmpeg-core.js) — documented but not the recommended path; only used when neither option is set.

When opts.enableMultithread === true:

• The loader inspects globalThis.crossOriginIsolated. If it is not true, the loader throws AUDIO_CODEC_FFMPEG_WASM.CROSS_ORIGIN_ISOLATION_REQUIRED with a context payload explaining the COOP/COEP requirement (see "Cross-origin isolation" below). This is the only path that surfaces that code — when enableMultithread is unset or false the loader falls back to the single-thread core silently.
• When COI is true, the loader dynamic-imports @ffmpeg/core-mt (declared in optionalDependencies so single-thread consumers do not pay the install cost). If the dynamic import rejects (not installed), the loader emits a warning via opts.logger and falls back to @ffmpeg/core rather than failing — multi-thread is an optimisation, not a correctness boundary.

Cross-origin isolation

The multi-threaded core uses SharedArrayBuffer, which browsers gate behind cross-origin isolation:

• The page must be served with Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: require-corp (or credentialless).
• All loaded resources must opt-in via Cross-Origin-Resource-Policy or be crossorigin="anonymous" with proper CORS.
• globalThis.crossOriginIsolated is true only when both headers are present.

The codec never silently downgrades correctness because of COI. The flag controls performance only:

enableMultithread	crossOriginIsolated	Result
false (default)	any	Single-thread core loads
true	true	@ffmpeg/core-mt loads (with fallback to single-thread if optional dep missing)
true	false	Throws AUDIO_CODEC_FFMPEG_WASM.CROSS_ORIGIN_ISOLATION_REQUIRED on first use

The "throw" case is the only way that error code is surfaced, satisfying §2c (every registered code reachable from a real call site).

Worker isolation

@ffmpeg/ffmpeg's FFmpeg class spawns a dedicated worker. We do not subclass FFmpeg; the worker-host module wraps it to add:

• AbortSignal plumbing → worker.terminate() on abort plus a Promise.reject with AUDIO_CODEC_FFMPEG_WASM.ABORTED.
• Timeout watchdog (default 60s for decode, 120s for encode; both configurable via EncodeOpts/DecodeOpts signal plus a wrapping AbortSignal.timeout) → AUDIO_CODEC_FFMPEG_WASM.TIMEOUT.
• A progress event listener that bridges into opts.progressCallback.
• Crash detection: an unexpected worker error event maps to AUDIO_CODEC_FFMPEG_WASM.WORKER_SPAWN_FAILED if it fires before load() resolved, else to AUDIO_CODEC_FFMPEG_WASM.DECODE_FAILED / _ENCODE_FAILED depending on the active call.
• Memory-pressure detection: wasm linear memory is capped at 4 GB; ffmpeg.wasm surfaces OutOfMemory via stderr or an Aborted() JS exception. The host parses this and re-emits AUDIO_CODEC_FFMPEG_WASM.MEMORY_LIMIT_EXCEEDED with the requested allocation size in context.

Test seam: opts.workerFactory overrides new Worker(coreUrl, …) so tests can supply a controllable shim. The shim is exercised in tests/ under happy-dom plus a hand-rolled Worker polyfill — see worker-host.test.ts.

Decode pipeline

Per the AudioCodec contract, decode(bytes) returns DecodeResult = { channelData: Float32Array[]; sampleRate: number; format: string }. The ffmpeg command we run is:

ffmpeg -hide_banner -nostdin -i input.<ext> -f f32le -ac <preserve> -ar <preserve> output.raw

Steps:

1. Inspect bytes magic to pick the input container/extension hint (re-using @suaveplan/audio-metadata's sniffer through audio-codecs). Failure to detect → AUDIO_CODEC_FFMPEG_WASM.UNSUPPORTED_CODEC.
2. ffmpeg.writeFile('input.' + ext, bytes) — writes into MEMFS (in-memory virtual FS).
3. Run a probe pass with ffprobe-equivalent flags to learn the source sampleRate and channels. (@ffmpeg/ffmpeg exposes this as a JSON log line.)
4. Run the f32le extraction; ffmpeg writes interleaved Float32 samples.
5. ffmpeg.readFile('output.raw') → Uint8Array.
6. Deinterleave into Float32Array[] per channel.
7. ffmpeg.deleteFile('input.' + ext) and ffmpeg.deleteFile('output.raw') to free MEMFS.

Any ffmpeg non-zero exit → AUDIO_CODEC_FFMPEG_WASM.DECODE_FAILED with the stderr tail in context.

Encode pipeline

encode(pcm, opts) accepts Float32Array[] per channel plus an EncodeOpts with at minimum format, sampleRate, and a format-specific bitrate / quality. Steps:

1. Interleave channels into a single Float32Array, then new Uint8Array(buffer) of the underlying bytes.

2. ffmpeg.writeFile('input.raw', interleavedBytes).

3. Run a format-specific command, for example for MP3:

   ffmpeg -f f32le -ar <rate> -ac <ch> -i input.raw -c:a libmp3lame -b:a <bitrate> output.mp3
   

4. ffmpeg.readFile('output.<ext>') → Uint8Array returned to caller.

5. Cleanup both MEMFS entries.

Each supported format owns a small command-builder function in encode-pipeline.ts. A format outside the supported set → AUDIO_CODEC_FFMPEG_WASM.UNSUPPORTED_CODEC before any wasm call.

Streaming "emulation" — be honest

@ffmpeg/ffmpeg is not naturally streaming. Its MEMFS surface is whole-file in, whole-file out. We expose StreamingDecoder / StreamingEncoder because the AudioCodec contract asks for it, but the implementation is buffered emulation, and the spec scenario calls this out plainly. The behaviour:

• StreamingDecoder.write(chunk): append chunk to an internal Uint8Array buffer; do not call ffmpeg. Always returns null (no decoded pages yet).
• StreamingDecoder.flush(): write the buffered bytes to MEMFS, run the same decode pipeline as decode(), and return the resulting DecodeResult. Subsequent flush() calls on the same decoder are no-ops.
• Progress events fire during the flush() call only, not during write().
• StreamingEncoder is symmetric.

This is documented in the spec scenario "Streaming is buffered emulation, not chunked decode" so callers do not assume true streaming behaviour. Real streaming would require a fork of @ffmpeg/ffmpeg that pipes stdout/stdin through MessagePort — out of scope.

Format support matrix

Format	Decode	Encode	Encoder cmd flag
wav	✅	✅	-c:a pcm_s16le (default)
mp3	✅	✅	-c:a libmp3lame
ogg	✅	✅	-c:a libvorbis
opus	✅	✅	-c:a libopus
flac	✅	✅	-c:a flac
aac	✅	✅	-c:a aac
webm	✅	✅	-c:a libopus -f webm

The exact codec availability depends on which @ffmpeg/core build is loaded. The default @ffmpeg/core build ships libmp3lame, libvorbis, libopus, flac, aac, pcm_*. A runtime probe (ffmpeg -encoders parsed once at load) checks availability; missing entries get pruned from extensions/mimeTypes rather than silently failing later.

Trade-offs vs the native server-tier counterpart

Axis	audio-codec-ffmpeg-native (server)	audio-codec-ffmpeg-wasm (browser)
Throughput	Hardware SIMD, full CPU	10–50× slower; wasm SIMD only when COI
Startup cost	Negligible (system binary)	~30 MB wasm fetch + ~200ms instantiate
Memory ceiling	OS-bound	4 GB hard wasm cap
Sandbox	None	Page sandbox + Worker isolation
Streaming	True streaming via pipes	Buffered emulation
Cancellation	child.kill()	worker.terminate()
Format reach	Whatever the system ffmpeg supports	Whatever the loaded core ships

The two are interchangeable at the registry level: applications register the appropriate codec per tier; consumers of CodecRegistry.decode(source) see the same AudioDocument shape regardless.

OTel instrumentation

The codec wraps every public method in a span:

• audio-codec-ffmpeg-wasm.decode — attributes: format, bytes, channels, sampleRate, multithread.
• audio-codec-ffmpeg-wasm.encode — attributes: format, samples, channels, sampleRate, bitrate.
• Counter audio_codec_ffmpeg_wasm_decode_total (labels: format, outcome).
• Counter audio_codec_ffmpeg_wasm_encode_total (labels: format, outcome).
• Histogram audio_codec_ffmpeg_wasm_decode_duration_ms (labels: format).
• Histogram audio_codec_ffmpeg_wasm_encode_duration_ms (labels: format).

All instrumentation is a no-op when no exporter is registered; nothing executes at module import. The injectable Clock drives the histogram measurements so tests are deterministic.

Browser-tier purity

CLAUDE.md rule "Universal: ZERO browser/React deps | Browser: React/DOM ok | Server: Node.js ok" makes the constraint explicit. This package is browser-tier — DOM and Worker APIs are allowed; Node-only APIs are forbidden even via dynamic import. The build is configured "browser": true in package.json and the test in tests/purity.test.ts greps the built dist/ for node:, process., Buffer., and require( to confirm.

Source bytes arrive as ArrayBuffer / Uint8Array / ReadableStream<Uint8Array> — never as a file path. File-path sources are the registry's problem and resolved by the universal audio-codecs source-resolution code paths before bytes ever reach this codec.