# Design: `@suaveplan/audio-codec-ffmpeg-wasm`

Non-normative design notes. The normative contract lives in
`./specs/audio-codec-ffmpeg-wasm/spec.md`.

---

## Why wasm + a Web Worker

`@suaveplan/audio-codecs` defines a tier-agnostic plugin contract. The server-tier
`audio-codec-ffmpeg-native` package wraps a system `ffmpeg` binary via subprocess. The
browser cannot spawn subprocesses, and there is no `ffmpeg` analogue baked into Web Audio.
The only credible way to reach the same format coverage in a sandboxed page is `ffmpeg.wasm`
— ffmpeg compiled to WebAssembly and exposed via the `@ffmpeg/ffmpeg` JS façade.

Two non-negotiable design choices follow from that:

1. **The wasm core runs inside a `Worker`.** Decoding a 5-minute MP3 takes hundreds of
   milliseconds of solid CPU; doing that on the main thread blocks input handling, layout,
   and the compositor. `@ffmpeg/ffmpeg` already supports a worker boundary; we always use
   it. The worker is the unit of cancellation: `AbortSignal.abort` translates to
   `worker.terminate()`, which is the only reliable way to interrupt wasm.
2. **The wasm payload is lazy-loaded.** `@ffmpeg/core` is ~30 MB; eagerly fetching it at
   `import` time would crater page load. The codec stays inert at module load and pulls the
   core only on the first `decode`/`encode`/`createDecoder`/`createEncoder` call. A second
   call reuses the loaded core; `dispose()` tears it down.

## Lazy load strategy

The `core-loader` module exposes `loadCore(opts): Promise<LoadedCore>` and memoises the
result per `FfmpegWasmCodec` instance. Resolution order for the core URL:

1. `opts.corePath` (explicit local URL) — preferred for self-hosted, COEP-correct serving.
2. `opts.coreUrl` (explicit CDN URL).
3. The package's own bundled default (`unpkg.com/@ffmpeg/core@x.y.z/dist/ffmpeg-core.js`)
   — documented but not the recommended path; only used when neither option is set.

When `opts.enableMultithread === true`:

- The loader inspects `globalThis.crossOriginIsolated`. If it is not `true`, the loader
  throws `AUDIO_CODEC_FFMPEG_WASM.CROSS_ORIGIN_ISOLATION_REQUIRED` with a context payload
  explaining the COOP/COEP requirement (see "Cross-origin isolation" below). This is the
  only path that surfaces that code — when `enableMultithread` is unset or `false` the
  loader falls back to the single-thread core silently.
- When COI is true, the loader dynamic-imports `@ffmpeg/core-mt` (declared in
  `optionalDependencies` so single-thread consumers do not pay the install cost). If the
  dynamic import rejects (not installed), the loader emits a warning via `opts.logger` and
  falls back to `@ffmpeg/core` rather than failing — multi-thread is an optimisation, not a
  correctness boundary.

## Cross-origin isolation

The multi-threaded core uses `SharedArrayBuffer`, which browsers gate behind cross-origin
isolation:

- The page must be served with `Cross-Origin-Opener-Policy: same-origin` and
  `Cross-Origin-Embedder-Policy: require-corp` (or `credentialless`).
- All loaded resources must opt-in via `Cross-Origin-Resource-Policy` or be
  `crossorigin="anonymous"` with proper CORS.
- `globalThis.crossOriginIsolated` is `true` only when both headers are present.

The codec **never silently downgrades correctness** because of COI. The flag controls
performance only:

| `enableMultithread` | `crossOriginIsolated` | Result |
|---|---|---|
| `false` (default) | any | Single-thread core loads |
| `true` | `true` | `@ffmpeg/core-mt` loads (with fallback to single-thread if optional dep missing) |
| `true` | `false` | Throws `AUDIO_CODEC_FFMPEG_WASM.CROSS_ORIGIN_ISOLATION_REQUIRED` on first use |

The "throw" case is the only way that error code is surfaced, satisfying §2c (every
registered code reachable from a real call site).

## Worker isolation

`@ffmpeg/ffmpeg`'s `FFmpeg` class spawns a dedicated worker. We do not subclass `FFmpeg`;
the `worker-host` module wraps it to add:

- `AbortSignal` plumbing → `worker.terminate()` on abort plus a `Promise.reject` with
  `AUDIO_CODEC_FFMPEG_WASM.ABORTED`.
- Timeout watchdog (default 60s for decode, 120s for encode; both configurable via
  `EncodeOpts`/`DecodeOpts` `signal` plus a wrapping `AbortSignal.timeout`) →
  `AUDIO_CODEC_FFMPEG_WASM.TIMEOUT`.
- A `progress` event listener that bridges into `opts.progressCallback`.
- Crash detection: an unexpected worker `error` event maps to
  `AUDIO_CODEC_FFMPEG_WASM.WORKER_SPAWN_FAILED` if it fires before `load()` resolved, else
  to `AUDIO_CODEC_FFMPEG_WASM.DECODE_FAILED` / `_ENCODE_FAILED` depending on the active
  call.
- Memory-pressure detection: wasm linear memory is capped at 4 GB; ffmpeg.wasm surfaces
  `OutOfMemory` via stderr or an `Aborted()` JS exception. The host parses this and re-emits
  `AUDIO_CODEC_FFMPEG_WASM.MEMORY_LIMIT_EXCEEDED` with the requested allocation size in
  `context`.

Test seam: `opts.workerFactory` overrides `new Worker(coreUrl, …)` so tests can supply a
controllable shim. The shim is exercised in `tests/` under `happy-dom` plus a hand-rolled
`Worker` polyfill — see `worker-host.test.ts`.

## Decode pipeline

Per the `AudioCodec` contract, `decode(bytes)` returns
`DecodeResult = { channelData: Float32Array[]; sampleRate: number; format: string }`. The
ffmpeg command we run is:

```text
ffmpeg -hide_banner -nostdin -i input.<ext> -f f32le -ac <preserve> -ar <preserve> output.raw
```

Steps:

1. Inspect `bytes` magic to pick the input container/extension hint (re-using
   `@suaveplan/audio-metadata`'s sniffer through `audio-codecs`). Failure to detect →
   `AUDIO_CODEC_FFMPEG_WASM.UNSUPPORTED_CODEC`.
2. `ffmpeg.writeFile('input.' + ext, bytes)` — writes into MEMFS (in-memory virtual FS).
3. Run a probe pass with `ffprobe`-equivalent flags to learn the source `sampleRate` and
   `channels`. (`@ffmpeg/ffmpeg` exposes this as a JSON log line.)
4. Run the f32le extraction; ffmpeg writes interleaved `Float32` samples.
5. `ffmpeg.readFile('output.raw')` → `Uint8Array`.
6. Deinterleave into `Float32Array[]` per channel.
7. `ffmpeg.deleteFile('input.' + ext)` and `ffmpeg.deleteFile('output.raw')` to free MEMFS.

Any ffmpeg non-zero exit → `AUDIO_CODEC_FFMPEG_WASM.DECODE_FAILED` with the stderr tail in
`context`.

## Encode pipeline

`encode(pcm, opts)` accepts `Float32Array[]` per channel plus an `EncodeOpts` with at
minimum `format`, `sampleRate`, and a format-specific `bitrate` / `quality`. Steps:

1. Interleave channels into a single `Float32Array`, then `new Uint8Array(buffer)` of the
   underlying bytes.
2. `ffmpeg.writeFile('input.raw', interleavedBytes)`.
3. Run a format-specific command, for example for MP3:

   ```text
   ffmpeg -f f32le -ar <rate> -ac <ch> -i input.raw -c:a libmp3lame -b:a <bitrate> output.mp3
   ```

4. `ffmpeg.readFile('output.<ext>')` → `Uint8Array` returned to caller.
5. Cleanup both MEMFS entries.

Each supported format owns a small command-builder function in `encode-pipeline.ts`. A
`format` outside the supported set → `AUDIO_CODEC_FFMPEG_WASM.UNSUPPORTED_CODEC` *before*
any wasm call.

## Streaming "emulation" — be honest

`@ffmpeg/ffmpeg` is not naturally streaming. Its MEMFS surface is whole-file in, whole-file
out. We expose `StreamingDecoder` / `StreamingEncoder` because the `AudioCodec` contract
asks for it, but the implementation is **buffered emulation**, and the spec scenario calls
this out plainly. The behaviour:

- `StreamingDecoder.write(chunk)`: append `chunk` to an internal `Uint8Array` buffer; do
  **not** call ffmpeg. Always returns `null` (no decoded pages yet).
- `StreamingDecoder.flush()`: write the buffered bytes to MEMFS, run the same decode
  pipeline as `decode()`, and return the resulting `DecodeResult`. Subsequent `flush()`
  calls on the same decoder are no-ops.
- Progress events fire during the `flush()` call only, not during `write()`.
- `StreamingEncoder` is symmetric.

This is documented in the spec scenario "Streaming is buffered emulation, not chunked
decode" so callers do not assume true streaming behaviour. Real streaming would require a
fork of `@ffmpeg/ffmpeg` that pipes stdout/stdin through `MessagePort` — out of scope.

## Format support matrix

| Format | Decode | Encode | Encoder cmd flag |
|---|---|---|---|
| `wav` | ✅ | ✅ | `-c:a pcm_s16le` (default) |
| `mp3` | ✅ | ✅ | `-c:a libmp3lame` |
| `ogg` | ✅ | ✅ | `-c:a libvorbis` |
| `opus` | ✅ | ✅ | `-c:a libopus` |
| `flac` | ✅ | ✅ | `-c:a flac` |
| `aac` | ✅ | ✅ | `-c:a aac` |
| `webm` | ✅ | ✅ | `-c:a libopus -f webm` |

The exact codec availability depends on which `@ffmpeg/core` build is loaded. The default
`@ffmpeg/core` build ships `libmp3lame`, `libvorbis`, `libopus`, `flac`, `aac`, `pcm_*`. A
runtime probe (`ffmpeg -encoders` parsed once at load) checks availability; missing entries
get pruned from `extensions`/`mimeTypes` rather than silently failing later.

## Trade-offs vs the native server-tier counterpart

| Axis | `audio-codec-ffmpeg-native` (server) | `audio-codec-ffmpeg-wasm` (browser) |
|---|---|---|
| Throughput | Hardware SIMD, full CPU | 10–50× slower; wasm SIMD only when COI |
| Startup cost | Negligible (system binary) | ~30 MB wasm fetch + ~200ms instantiate |
| Memory ceiling | OS-bound | 4 GB hard wasm cap |
| Sandbox | None | Page sandbox + Worker isolation |
| Streaming | True streaming via pipes | Buffered emulation |
| Cancellation | `child.kill()` | `worker.terminate()` |
| Format reach | Whatever the system ffmpeg supports | Whatever the loaded core ships |

The two are interchangeable at the registry level: applications register the appropriate
codec per tier; consumers of `CodecRegistry.decode(source)` see the same `AudioDocument`
shape regardless.

## OTel instrumentation

The codec wraps every public method in a span:

- `audio-codec-ffmpeg-wasm.decode` — attributes: `format`, `bytes`, `channels`,
  `sampleRate`, `multithread`.
- `audio-codec-ffmpeg-wasm.encode` — attributes: `format`, `samples`, `channels`,
  `sampleRate`, `bitrate`.
- Counter `audio_codec_ffmpeg_wasm_decode_total` (labels: `format`, `outcome`).
- Counter `audio_codec_ffmpeg_wasm_encode_total` (labels: `format`, `outcome`).
- Histogram `audio_codec_ffmpeg_wasm_decode_duration_ms` (labels: `format`).
- Histogram `audio_codec_ffmpeg_wasm_encode_duration_ms` (labels: `format`).

All instrumentation is a no-op when no exporter is registered; nothing executes at module
import. The injectable `Clock` drives the histogram measurements so tests are
deterministic.

## Browser-tier purity

`CLAUDE.md` rule "Universal: ZERO browser/React deps | Browser: React/DOM ok | Server:
Node.js ok" makes the constraint explicit. This package is browser-tier — DOM and Worker
APIs are allowed; Node-only APIs are forbidden **even via dynamic import**. The build is
configured `"browser": true` in `package.json` and the test in `tests/purity.test.ts`
greps the built `dist/` for `node:`, `process.`, `Buffer.`, and `require(` to confirm.

Source bytes arrive as `ArrayBuffer` / `Uint8Array` / `ReadableStream<Uint8Array>` —
never as a file path. File-path sources are the registry's problem and resolved by the
universal `audio-codecs` source-resolution code paths before bytes ever reach this codec.