Initial Commit
This commit is contained in:
@@ -0,0 +1,224 @@
|
||||
# Design: `@suaveplan/audio-codec-ffmpeg-wasm`
|
||||
|
||||
Non-normative design notes. The normative contract lives in
|
||||
`./specs/audio-codec-ffmpeg-wasm/spec.md`.
|
||||
|
||||
---
|
||||
|
||||
## Why wasm + a Web Worker
|
||||
|
||||
`@suaveplan/audio-codecs` defines a tier-agnostic plugin contract. The server-tier
|
||||
`audio-codec-ffmpeg-native` package wraps a system `ffmpeg` binary via subprocess. The
|
||||
browser cannot spawn subprocesses, and there is no `ffmpeg` analogue baked into Web Audio.
|
||||
The only credible way to reach the same format coverage in a sandboxed page is `ffmpeg.wasm`
|
||||
— ffmpeg compiled to WebAssembly and exposed via the `@ffmpeg/ffmpeg` JS façade.
|
||||
|
||||
Two non-negotiable design choices follow from that:
|
||||
|
||||
1. **The wasm core runs inside a `Worker`.** Decoding a 5-minute MP3 takes hundreds of
|
||||
milliseconds of solid CPU; doing that on the main thread blocks input handling, layout,
|
||||
and the compositor. `@ffmpeg/ffmpeg` already supports a worker boundary; we always use
|
||||
it. The worker is the unit of cancellation: `AbortSignal.abort` translates to
|
||||
`worker.terminate()`, which is the only reliable way to interrupt wasm.
|
||||
2. **The wasm payload is lazy-loaded.** `@ffmpeg/core` is ~30 MB; eagerly fetching it at
|
||||
`import` time would crater page load. The codec stays inert at module load and pulls the
|
||||
core only on the first `decode`/`encode`/`createDecoder`/`createEncoder` call. A second
|
||||
call reuses the loaded core; `dispose()` tears it down.
|
||||
|
||||
## Lazy load strategy
|
||||
|
||||
The `core-loader` module exposes `loadCore(opts): Promise<LoadedCore>` and memoises the
|
||||
result per `FfmpegWasmCodec` instance. Resolution order for the core URL:
|
||||
|
||||
1. `opts.corePath` (explicit local URL) — preferred for self-hosted, COEP-correct serving.
|
||||
2. `opts.coreUrl` (explicit CDN URL).
|
||||
3. The package's own bundled default (`unpkg.com/@ffmpeg/core@x.y.z/dist/ffmpeg-core.js`)
|
||||
— documented but not the recommended path; only used when neither option is set.
|
||||
|
||||
When `opts.enableMultithread === true`:
|
||||
|
||||
- The loader inspects `globalThis.crossOriginIsolated`. If it is not `true`, the loader
|
||||
throws `AUDIO_CODEC_FFMPEG_WASM.CROSS_ORIGIN_ISOLATION_REQUIRED` with a context payload
|
||||
explaining the COOP/COEP requirement (see "Cross-origin isolation" below). This is the
|
||||
only path that surfaces that code — when `enableMultithread` is unset or `false` the
|
||||
loader falls back to the single-thread core silently.
|
||||
- When COI is true, the loader dynamic-imports `@ffmpeg/core-mt` (declared in
|
||||
`optionalDependencies` so single-thread consumers do not pay the install cost). If the
|
||||
dynamic import rejects (not installed), the loader emits a warning via `opts.logger` and
|
||||
falls back to `@ffmpeg/core` rather than failing — multi-thread is an optimisation, not a
|
||||
correctness boundary.
|
||||
|
||||
## Cross-origin isolation
|
||||
|
||||
The multi-threaded core uses `SharedArrayBuffer`, which browsers gate behind cross-origin
|
||||
isolation:
|
||||
|
||||
- The page must be served with `Cross-Origin-Opener-Policy: same-origin` and
|
||||
`Cross-Origin-Embedder-Policy: require-corp` (or `credentialless`).
|
||||
- All loaded resources must opt-in via `Cross-Origin-Resource-Policy` or be
|
||||
`crossorigin="anonymous"` with proper CORS.
|
||||
- `globalThis.crossOriginIsolated` is `true` only when both headers are present.
|
||||
|
||||
The codec **never silently downgrades correctness** because of COI. The flag controls
|
||||
performance only:
|
||||
|
||||
| `enableMultithread` | `crossOriginIsolated` | Result |
|
||||
|---|---|---|
|
||||
| `false` (default) | any | Single-thread core loads |
|
||||
| `true` | `true` | `@ffmpeg/core-mt` loads (with fallback to single-thread if optional dep missing) |
|
||||
| `true` | `false` | Throws `AUDIO_CODEC_FFMPEG_WASM.CROSS_ORIGIN_ISOLATION_REQUIRED` on first use |
|
||||
|
||||
The "throw" case is the only way that error code is surfaced, satisfying §2c (every
|
||||
registered code reachable from a real call site).
|
||||
|
||||
## Worker isolation
|
||||
|
||||
`@ffmpeg/ffmpeg`'s `FFmpeg` class spawns a dedicated worker. We do not subclass `FFmpeg`;
|
||||
the `worker-host` module wraps it to add:
|
||||
|
||||
- `AbortSignal` plumbing → `worker.terminate()` on abort plus a `Promise.reject` with
|
||||
`AUDIO_CODEC_FFMPEG_WASM.ABORTED`.
|
||||
- Timeout watchdog (default 60s for decode, 120s for encode; both configurable via
|
||||
`EncodeOpts`/`DecodeOpts` `signal` plus a wrapping `AbortSignal.timeout`) →
|
||||
`AUDIO_CODEC_FFMPEG_WASM.TIMEOUT`.
|
||||
- A `progress` event listener that bridges into `opts.progressCallback`.
|
||||
- Crash detection: an unexpected worker `error` event maps to
|
||||
`AUDIO_CODEC_FFMPEG_WASM.WORKER_SPAWN_FAILED` if it fires before `load()` resolved, else
|
||||
to `AUDIO_CODEC_FFMPEG_WASM.DECODE_FAILED` / `_ENCODE_FAILED` depending on the active
|
||||
call.
|
||||
- Memory-pressure detection: wasm linear memory is capped at 4 GB; ffmpeg.wasm surfaces
|
||||
`OutOfMemory` via stderr or an `Aborted()` JS exception. The host parses this and re-emits
|
||||
`AUDIO_CODEC_FFMPEG_WASM.MEMORY_LIMIT_EXCEEDED` with the requested allocation size in
|
||||
`context`.
|
||||
|
||||
Test seam: `opts.workerFactory` overrides `new Worker(coreUrl, …)` so tests can supply a
|
||||
controllable shim. The shim is exercised in `tests/` under `happy-dom` plus a hand-rolled
|
||||
`Worker` polyfill — see `worker-host.test.ts`.
|
||||
|
||||
## Decode pipeline
|
||||
|
||||
Per the `AudioCodec` contract, `decode(bytes)` returns
|
||||
`DecodeResult = { channelData: Float32Array[]; sampleRate: number; format: string }`. The
|
||||
ffmpeg command we run is:
|
||||
|
||||
```text
|
||||
ffmpeg -hide_banner -nostdin -i input.<ext> -f f32le -ac <preserve> -ar <preserve> output.raw
|
||||
```
|
||||
|
||||
Steps:
|
||||
|
||||
1. Inspect `bytes` magic to pick the input container/extension hint (re-using
|
||||
`@suaveplan/audio-metadata`'s sniffer through `audio-codecs`). Failure to detect →
|
||||
`AUDIO_CODEC_FFMPEG_WASM.UNSUPPORTED_CODEC`.
|
||||
2. `ffmpeg.writeFile('input.' + ext, bytes)` — writes into MEMFS (in-memory virtual FS).
|
||||
3. Run a probe pass with `ffprobe`-equivalent flags to learn the source `sampleRate` and
|
||||
`channels`. (`@ffmpeg/ffmpeg` exposes this as a JSON log line.)
|
||||
4. Run the f32le extraction; ffmpeg writes interleaved `Float32` samples.
|
||||
5. `ffmpeg.readFile('output.raw')` → `Uint8Array`.
|
||||
6. Deinterleave into `Float32Array[]` per channel.
|
||||
7. `ffmpeg.deleteFile('input.' + ext)` and `ffmpeg.deleteFile('output.raw')` to free MEMFS.
|
||||
|
||||
Any ffmpeg non-zero exit → `AUDIO_CODEC_FFMPEG_WASM.DECODE_FAILED` with the stderr tail in
|
||||
`context`.
|
||||
|
||||
## Encode pipeline
|
||||
|
||||
`encode(pcm, opts)` accepts `Float32Array[]` per channel plus an `EncodeOpts` with at
|
||||
minimum `format`, `sampleRate`, and a format-specific `bitrate` / `quality`. Steps:
|
||||
|
||||
1. Interleave channels into a single `Float32Array`, then `new Uint8Array(buffer)` of the
|
||||
underlying bytes.
|
||||
2. `ffmpeg.writeFile('input.raw', interleavedBytes)`.
|
||||
3. Run a format-specific command, for example for MP3:
|
||||
|
||||
```text
|
||||
ffmpeg -f f32le -ar <rate> -ac <ch> -i input.raw -c:a libmp3lame -b:a <bitrate> output.mp3
|
||||
```
|
||||
|
||||
4. `ffmpeg.readFile('output.<ext>')` → `Uint8Array` returned to caller.
|
||||
5. Cleanup both MEMFS entries.
|
||||
|
||||
Each supported format owns a small command-builder function in `encode-pipeline.ts`. A
|
||||
`format` outside the supported set → `AUDIO_CODEC_FFMPEG_WASM.UNSUPPORTED_CODEC` *before*
|
||||
any wasm call.
|
||||
|
||||
## Streaming "emulation" — be honest
|
||||
|
||||
`@ffmpeg/ffmpeg` is not naturally streaming. Its MEMFS surface is whole-file in, whole-file
|
||||
out. We expose `StreamingDecoder` / `StreamingEncoder` because the `AudioCodec` contract
|
||||
asks for it, but the implementation is **buffered emulation**, and the spec scenario calls
|
||||
this out plainly. The behaviour:
|
||||
|
||||
- `StreamingDecoder.write(chunk)`: append `chunk` to an internal `Uint8Array` buffer; do
|
||||
**not** call ffmpeg. Always returns `null` (no decoded pages yet).
|
||||
- `StreamingDecoder.flush()`: write the buffered bytes to MEMFS, run the same decode
|
||||
pipeline as `decode()`, and return the resulting `DecodeResult`. Subsequent `flush()`
|
||||
calls on the same decoder are no-ops.
|
||||
- Progress events fire during the `flush()` call only, not during `write()`.
|
||||
- `StreamingEncoder` is symmetric.
|
||||
|
||||
This is documented in the spec scenario "Streaming is buffered emulation, not chunked
|
||||
decode" so callers do not assume true streaming behaviour. Real streaming would require a
|
||||
fork of `@ffmpeg/ffmpeg` that pipes stdout/stdin through `MessagePort` — out of scope.
|
||||
|
||||
## Format support matrix
|
||||
|
||||
| Format | Decode | Encode | Encoder cmd flag |
|
||||
|---|---|---|---|
|
||||
| `wav` | ✅ | ✅ | `-c:a pcm_s16le` (default) |
|
||||
| `mp3` | ✅ | ✅ | `-c:a libmp3lame` |
|
||||
| `ogg` | ✅ | ✅ | `-c:a libvorbis` |
|
||||
| `opus` | ✅ | ✅ | `-c:a libopus` |
|
||||
| `flac` | ✅ | ✅ | `-c:a flac` |
|
||||
| `aac` | ✅ | ✅ | `-c:a aac` |
|
||||
| `webm` | ✅ | ✅ | `-c:a libopus -f webm` |
|
||||
|
||||
The exact codec availability depends on which `@ffmpeg/core` build is loaded. The default
|
||||
`@ffmpeg/core` build ships `libmp3lame`, `libvorbis`, `libopus`, `flac`, `aac`, `pcm_*`. A
|
||||
runtime probe (`ffmpeg -encoders` parsed once at load) checks availability; missing entries
|
||||
get pruned from `extensions`/`mimeTypes` rather than silently failing later.
|
||||
|
||||
## Trade-offs vs the native server-tier counterpart
|
||||
|
||||
| Axis | `audio-codec-ffmpeg-native` (server) | `audio-codec-ffmpeg-wasm` (browser) |
|
||||
|---|---|---|
|
||||
| Throughput | Hardware SIMD, full CPU | 10–50× slower; wasm SIMD only when COI |
|
||||
| Startup cost | Negligible (system binary) | ~30 MB wasm fetch + ~200ms instantiate |
|
||||
| Memory ceiling | OS-bound | 4 GB hard wasm cap |
|
||||
| Sandbox | None | Page sandbox + Worker isolation |
|
||||
| Streaming | True streaming via pipes | Buffered emulation |
|
||||
| Cancellation | `child.kill()` | `worker.terminate()` |
|
||||
| Format reach | Whatever the system ffmpeg supports | Whatever the loaded core ships |
|
||||
|
||||
The two are interchangeable at the registry level: applications register the appropriate
|
||||
codec per tier; consumers of `CodecRegistry.decode(source)` see the same `AudioDocument`
|
||||
shape regardless.
|
||||
|
||||
## OTel instrumentation
|
||||
|
||||
The codec wraps every public method in a span:
|
||||
|
||||
- `audio-codec-ffmpeg-wasm.decode` — attributes: `format`, `bytes`, `channels`,
|
||||
`sampleRate`, `multithread`.
|
||||
- `audio-codec-ffmpeg-wasm.encode` — attributes: `format`, `samples`, `channels`,
|
||||
`sampleRate`, `bitrate`.
|
||||
- Counter `audio_codec_ffmpeg_wasm_decode_total` (labels: `format`, `outcome`).
|
||||
- Counter `audio_codec_ffmpeg_wasm_encode_total` (labels: `format`, `outcome`).
|
||||
- Histogram `audio_codec_ffmpeg_wasm_decode_duration_ms` (labels: `format`).
|
||||
- Histogram `audio_codec_ffmpeg_wasm_encode_duration_ms` (labels: `format`).
|
||||
|
||||
All instrumentation is a no-op when no exporter is registered; nothing executes at module
|
||||
import. The injectable `Clock` drives the histogram measurements so tests are
|
||||
deterministic.
|
||||
|
||||
## Browser-tier purity
|
||||
|
||||
`CLAUDE.md` rule "Universal: ZERO browser/React deps | Browser: React/DOM ok | Server:
|
||||
Node.js ok" makes the constraint explicit. This package is browser-tier — DOM and Worker
|
||||
APIs are allowed; Node-only APIs are forbidden **even via dynamic import**. The build is
|
||||
configured `"browser": true` in `package.json` and the test in `tests/purity.test.ts`
|
||||
greps the built `dist/` for `node:`, `process.`, `Buffer.`, and `require(` to confirm.
|
||||
|
||||
Source bytes arrive as `ArrayBuffer` / `Uint8Array` / `ReadableStream<Uint8Array>` —
|
||||
never as a file path. File-path sources are the registry's problem and resolved by the
|
||||
universal `audio-codecs` source-resolution code paths before bytes ever reach this codec.
|
||||
Reference in New Issue
Block a user