225 lines
11 KiB
Markdown
225 lines
11 KiB
Markdown
# Design: `@suaveplan/audio-codec-ffmpeg-wasm`
|
||
|
||
Non-normative design notes. The normative contract lives in
|
||
`./specs/audio-codec-ffmpeg-wasm/spec.md`.
|
||
|
||
---
|
||
|
||
## Why wasm + a Web Worker
|
||
|
||
`@suaveplan/audio-codecs` defines a tier-agnostic plugin contract. The server-tier
|
||
`audio-codec-ffmpeg-native` package wraps a system `ffmpeg` binary via subprocess. The
|
||
browser cannot spawn subprocesses, and there is no `ffmpeg` analogue baked into Web Audio.
|
||
The only credible way to reach the same format coverage in a sandboxed page is `ffmpeg.wasm`
|
||
— ffmpeg compiled to WebAssembly and exposed via the `@ffmpeg/ffmpeg` JS façade.
|
||
|
||
Two non-negotiable design choices follow from that:
|
||
|
||
1. **The wasm core runs inside a `Worker`.** Decoding a 5-minute MP3 takes hundreds of
|
||
milliseconds of solid CPU; doing that on the main thread blocks input handling, layout,
|
||
and the compositor. `@ffmpeg/ffmpeg` already supports a worker boundary; we always use
|
||
it. The worker is the unit of cancellation: `AbortSignal.abort` translates to
|
||
`worker.terminate()`, which is the only reliable way to interrupt wasm.
|
||
2. **The wasm payload is lazy-loaded.** `@ffmpeg/core` is ~30 MB; eagerly fetching it at
|
||
`import` time would crater page load. The codec stays inert at module load and pulls the
|
||
core only on the first `decode`/`encode`/`createDecoder`/`createEncoder` call. A second
|
||
call reuses the loaded core; `dispose()` tears it down.
|
||
|
||
## Lazy load strategy
|
||
|
||
The `core-loader` module exposes `loadCore(opts): Promise<LoadedCore>` and memoises the
|
||
result per `FfmpegWasmCodec` instance. Resolution order for the core URL:
|
||
|
||
1. `opts.corePath` (explicit local URL) — preferred for self-hosted, COEP-correct serving.
|
||
2. `opts.coreUrl` (explicit CDN URL).
|
||
3. The package's own bundled default (`unpkg.com/@ffmpeg/core@x.y.z/dist/ffmpeg-core.js`)
|
||
— documented but not the recommended path; only used when neither option is set.
|
||
|
||
When `opts.enableMultithread === true`:
|
||
|
||
- The loader inspects `globalThis.crossOriginIsolated`. If it is not `true`, the loader
|
||
throws `AUDIO_CODEC_FFMPEG_WASM.CROSS_ORIGIN_ISOLATION_REQUIRED` with a context payload
|
||
explaining the COOP/COEP requirement (see "Cross-origin isolation" below). This is the
|
||
only path that surfaces that code — when `enableMultithread` is unset or `false` the
|
||
loader falls back to the single-thread core silently.
|
||
- When COI is true, the loader dynamic-imports `@ffmpeg/core-mt` (declared in
|
||
`optionalDependencies` so single-thread consumers do not pay the install cost). If the
|
||
dynamic import rejects (not installed), the loader emits a warning via `opts.logger` and
|
||
falls back to `@ffmpeg/core` rather than failing — multi-thread is an optimisation, not a
|
||
correctness boundary.
|
||
|
||
## Cross-origin isolation
|
||
|
||
The multi-threaded core uses `SharedArrayBuffer`, which browsers gate behind cross-origin
|
||
isolation:
|
||
|
||
- The page must be served with `Cross-Origin-Opener-Policy: same-origin` and
|
||
`Cross-Origin-Embedder-Policy: require-corp` (or `credentialless`).
|
||
- All loaded resources must opt-in via `Cross-Origin-Resource-Policy` or be
|
||
`crossorigin="anonymous"` with proper CORS.
|
||
- `globalThis.crossOriginIsolated` is `true` only when both headers are present.
|
||
|
||
The codec **never silently downgrades correctness** because of COI. The flag controls
|
||
performance only:
|
||
|
||
| `enableMultithread` | `crossOriginIsolated` | Result |
|
||
|---|---|---|
|
||
| `false` (default) | any | Single-thread core loads |
|
||
| `true` | `true` | `@ffmpeg/core-mt` loads (with fallback to single-thread if optional dep missing) |
|
||
| `true` | `false` | Throws `AUDIO_CODEC_FFMPEG_WASM.CROSS_ORIGIN_ISOLATION_REQUIRED` on first use |
|
||
|
||
The "throw" case is the only way that error code is surfaced, satisfying §2c (every
|
||
registered code reachable from a real call site).
|
||
|
||
## Worker isolation
|
||
|
||
`@ffmpeg/ffmpeg`'s `FFmpeg` class spawns a dedicated worker. We do not subclass `FFmpeg`;
|
||
the `worker-host` module wraps it to add:
|
||
|
||
- `AbortSignal` plumbing → `worker.terminate()` on abort plus a `Promise.reject` with
|
||
`AUDIO_CODEC_FFMPEG_WASM.ABORTED`.
|
||
- Timeout watchdog (default 60s for decode, 120s for encode; both configurable via
|
||
`EncodeOpts`/`DecodeOpts` `signal` plus a wrapping `AbortSignal.timeout`) →
|
||
`AUDIO_CODEC_FFMPEG_WASM.TIMEOUT`.
|
||
- A `progress` event listener that bridges into `opts.progressCallback`.
|
||
- Crash detection: an unexpected worker `error` event maps to
|
||
`AUDIO_CODEC_FFMPEG_WASM.WORKER_SPAWN_FAILED` if it fires before `load()` resolved, else
|
||
to `AUDIO_CODEC_FFMPEG_WASM.DECODE_FAILED` / `_ENCODE_FAILED` depending on the active
|
||
call.
|
||
- Memory-pressure detection: wasm linear memory is capped at 4 GB; ffmpeg.wasm surfaces
|
||
`OutOfMemory` via stderr or an `Aborted()` JS exception. The host parses this and re-emits
|
||
`AUDIO_CODEC_FFMPEG_WASM.MEMORY_LIMIT_EXCEEDED` with the requested allocation size in
|
||
`context`.
|
||
|
||
Test seam: `opts.workerFactory` overrides `new Worker(coreUrl, …)` so tests can supply a
|
||
controllable shim. The shim is exercised in `tests/` under `happy-dom` plus a hand-rolled
|
||
`Worker` polyfill — see `worker-host.test.ts`.
|
||
|
||
## Decode pipeline
|
||
|
||
Per the `AudioCodec` contract, `decode(bytes)` returns
|
||
`DecodeResult = { channelData: Float32Array[]; sampleRate: number; format: string }`. The
|
||
ffmpeg command we run is:
|
||
|
||
```text
|
||
ffmpeg -hide_banner -nostdin -i input.<ext> -f f32le -ac <preserve> -ar <preserve> output.raw
|
||
```
|
||
|
||
Steps:
|
||
|
||
1. Inspect `bytes` magic to pick the input container/extension hint (re-using
|
||
`@suaveplan/audio-metadata`'s sniffer through `audio-codecs`). Failure to detect →
|
||
`AUDIO_CODEC_FFMPEG_WASM.UNSUPPORTED_CODEC`.
|
||
2. `ffmpeg.writeFile('input.' + ext, bytes)` — writes into MEMFS (in-memory virtual FS).
|
||
3. Run a probe pass with `ffprobe`-equivalent flags to learn the source `sampleRate` and
|
||
`channels`. (`@ffmpeg/ffmpeg` exposes this as a JSON log line.)
|
||
4. Run the f32le extraction; ffmpeg writes interleaved `Float32` samples.
|
||
5. `ffmpeg.readFile('output.raw')` → `Uint8Array`.
|
||
6. Deinterleave into `Float32Array[]` per channel.
|
||
7. `ffmpeg.deleteFile('input.' + ext)` and `ffmpeg.deleteFile('output.raw')` to free MEMFS.
|
||
|
||
Any ffmpeg non-zero exit → `AUDIO_CODEC_FFMPEG_WASM.DECODE_FAILED` with the stderr tail in
|
||
`context`.
|
||
|
||
## Encode pipeline
|
||
|
||
`encode(pcm, opts)` accepts `Float32Array[]` per channel plus an `EncodeOpts` with at
|
||
minimum `format`, `sampleRate`, and a format-specific `bitrate` / `quality`. Steps:
|
||
|
||
1. Interleave channels into a single `Float32Array`, then `new Uint8Array(buffer)` of the
|
||
underlying bytes.
|
||
2. `ffmpeg.writeFile('input.raw', interleavedBytes)`.
|
||
3. Run a format-specific command, for example for MP3:
|
||
|
||
```text
|
||
ffmpeg -f f32le -ar <rate> -ac <ch> -i input.raw -c:a libmp3lame -b:a <bitrate> output.mp3
|
||
```
|
||
|
||
4. `ffmpeg.readFile('output.<ext>')` → `Uint8Array` returned to caller.
|
||
5. Cleanup both MEMFS entries.
|
||
|
||
Each supported format owns a small command-builder function in `encode-pipeline.ts`. A
|
||
`format` outside the supported set → `AUDIO_CODEC_FFMPEG_WASM.UNSUPPORTED_CODEC` *before*
|
||
any wasm call.
|
||
|
||
## Streaming "emulation" — be honest
|
||
|
||
`@ffmpeg/ffmpeg` is not naturally streaming. Its MEMFS surface is whole-file in, whole-file
|
||
out. We expose `StreamingDecoder` / `StreamingEncoder` because the `AudioCodec` contract
|
||
asks for it, but the implementation is **buffered emulation**, and the spec scenario calls
|
||
this out plainly. The behaviour:
|
||
|
||
- `StreamingDecoder.write(chunk)`: append `chunk` to an internal `Uint8Array` buffer; do
|
||
**not** call ffmpeg. Always returns `null` (no decoded pages yet).
|
||
- `StreamingDecoder.flush()`: write the buffered bytes to MEMFS, run the same decode
|
||
pipeline as `decode()`, and return the resulting `DecodeResult`. Subsequent `flush()`
|
||
calls on the same decoder are no-ops.
|
||
- Progress events fire during the `flush()` call only, not during `write()`.
|
||
- `StreamingEncoder` is symmetric.
|
||
|
||
This is documented in the spec scenario "Streaming is buffered emulation, not chunked
|
||
decode" so callers do not assume true streaming behaviour. Real streaming would require a
|
||
fork of `@ffmpeg/ffmpeg` that pipes stdout/stdin through `MessagePort` — out of scope.
|
||
|
||
## Format support matrix
|
||
|
||
| Format | Decode | Encode | Encoder cmd flag |
|
||
|---|---|---|---|
|
||
| `wav` | ✅ | ✅ | `-c:a pcm_s16le` (default) |
|
||
| `mp3` | ✅ | ✅ | `-c:a libmp3lame` |
|
||
| `ogg` | ✅ | ✅ | `-c:a libvorbis` |
|
||
| `opus` | ✅ | ✅ | `-c:a libopus` |
|
||
| `flac` | ✅ | ✅ | `-c:a flac` |
|
||
| `aac` | ✅ | ✅ | `-c:a aac` |
|
||
| `webm` | ✅ | ✅ | `-c:a libopus -f webm` |
|
||
|
||
The exact codec availability depends on which `@ffmpeg/core` build is loaded. The default
|
||
`@ffmpeg/core` build ships `libmp3lame`, `libvorbis`, `libopus`, `flac`, `aac`, `pcm_*`. A
|
||
runtime probe (`ffmpeg -encoders` parsed once at load) checks availability; missing entries
|
||
get pruned from `extensions`/`mimeTypes` rather than silently failing later.
|
||
|
||
## Trade-offs vs the native server-tier counterpart
|
||
|
||
| Axis | `audio-codec-ffmpeg-native` (server) | `audio-codec-ffmpeg-wasm` (browser) |
|
||
|---|---|---|
|
||
| Throughput | Hardware SIMD, full CPU | 10–50× slower; wasm SIMD only when COI |
|
||
| Startup cost | Negligible (system binary) | ~30 MB wasm fetch + ~200ms instantiate |
|
||
| Memory ceiling | OS-bound | 4 GB hard wasm cap |
|
||
| Sandbox | None | Page sandbox + Worker isolation |
|
||
| Streaming | True streaming via pipes | Buffered emulation |
|
||
| Cancellation | `child.kill()` | `worker.terminate()` |
|
||
| Format reach | Whatever the system ffmpeg supports | Whatever the loaded core ships |
|
||
|
||
The two are interchangeable at the registry level: applications register the appropriate
|
||
codec per tier; consumers of `CodecRegistry.decode(source)` see the same `AudioDocument`
|
||
shape regardless.
|
||
|
||
## OTel instrumentation
|
||
|
||
The codec wraps every public method in a span:
|
||
|
||
- `audio-codec-ffmpeg-wasm.decode` — attributes: `format`, `bytes`, `channels`,
|
||
`sampleRate`, `multithread`.
|
||
- `audio-codec-ffmpeg-wasm.encode` — attributes: `format`, `samples`, `channels`,
|
||
`sampleRate`, `bitrate`.
|
||
- Counter `audio_codec_ffmpeg_wasm_decode_total` (labels: `format`, `outcome`).
|
||
- Counter `audio_codec_ffmpeg_wasm_encode_total` (labels: `format`, `outcome`).
|
||
- Histogram `audio_codec_ffmpeg_wasm_decode_duration_ms` (labels: `format`).
|
||
- Histogram `audio_codec_ffmpeg_wasm_encode_duration_ms` (labels: `format`).
|
||
|
||
All instrumentation is a no-op when no exporter is registered; nothing executes at module
|
||
import. The injectable `Clock` drives the histogram measurements so tests are
|
||
deterministic.
|
||
|
||
## Browser-tier purity
|
||
|
||
`CLAUDE.md` rule "Universal: ZERO browser/React deps | Browser: React/DOM ok | Server:
|
||
Node.js ok" makes the constraint explicit. This package is browser-tier — DOM and Worker
|
||
APIs are allowed; Node-only APIs are forbidden **even via dynamic import**. The build is
|
||
configured `"browser": true` in `package.json` and the test in `tests/purity.test.ts`
|
||
greps the built `dist/` for `node:`, `process.`, `Buffer.`, and `require(` to confirm.
|
||
|
||
Source bytes arrive as `ArrayBuffer` / `Uint8Array` / `ReadableStream<Uint8Array>` —
|
||
never as a file path. File-path sources are the registry's problem and resolved by the
|
||
universal `audio-codecs` source-resolution code paths before bytes ever reach this codec.
|