275 lines
12 KiB
Markdown
275 lines
12 KiB
Markdown
# Design: enterprise-audio-codecs
|
||
|
||
**Change ID**: enterprise-audio-codecs
|
||
**Package**: `@suaveplan/audio-codecs`
|
||
|
||
`@suaveplan/audio-codecs` is the universal format decode/encode layer of the `@suaveplan/audio-*`
|
||
family. It turns any audio source — file path, URL, buffer, or byte stream — into an
|
||
`AudioDocument`, and turns an `AudioDocument` back into encoded bytes in any supported format. This
|
||
document carries the **package-specific design** for the codec layer.
|
||
|
||
---
|
||
|
||
## 1. Relationship to the ecosystem architecture
|
||
|
||
The cross-cutting `@suaveplan/audio-*` architecture — the three-layer separation (data model /
|
||
timeline-transport / rendering engine), the six plugin slots, the "zero duplication of genesis
|
||
primitives" rule, and the universal-tier purity rule — lives in
|
||
`enterprise-audio-core/design.md` §1–§4. It is **referenced here, not duplicated**.
|
||
|
||
`audio-codecs` occupies **plugin slot 4 (Codec)** from that architecture: it decodes / encodes a
|
||
specific format and registers codecs in a `CodecRegistry`. It sits directly above `audio-core` in
|
||
the dependency graph and below every rendering engine and CLI. Its output is the universal
|
||
`AudioDocument` defined by `audio-core` — the codec layer never owns its own audio container.
|
||
|
||
The `Segment` shape referenced anywhere in this package is the single family-wide definition from
|
||
`enterprise-audio-core/design.md` §2.2: `{ from, count, to, rate?, ref?, interp? }`. Codecs do not
|
||
introduce segment variants — they produce `AudioPage` PCM, and editing/segmentation is `audio-core`'s
|
||
job.
|
||
|
||
---
|
||
|
||
## 2. Decode model
|
||
|
||
### 2.1. Source resolution
|
||
|
||
A `Source` is one of `string | URL | ArrayBuffer | Uint8Array | ReadableStream<Uint8Array>`.
|
||
Resolution rules:
|
||
|
||
- `string` starting with `http`/`https`/`data`/`blob`, or any `URL` → `fetch()`.
|
||
- `string` starting with `file:` → dynamic `import('node:fs/promises')`.
|
||
- other bare `string` → `fs/promises.readFile` on a server runtime, `fetch()` in a browser.
|
||
- `ArrayBuffer` / `Uint8Array` / `ReadableStream<Uint8Array>` → consumed directly, no platform call.
|
||
|
||
The `node:fs/promises` import is **always** inside the runtime-gated branch — never a top-level
|
||
import — so the package stays universal-tier per `enterprise-audio-core/design.md` §1.5.
|
||
|
||
### 2.2. Format detection
|
||
|
||
```ts
|
||
const HEADER_SIZE = 256 * 1024
|
||
|
||
async function detectSource(source: Source) {
|
||
if (typeof source === 'string' && !source.startsWith('http')) {
|
||
const { open } = await import('node:fs/promises')
|
||
const fh = await open(source, 'r')
|
||
const hdr = new Uint8Array(HEADER_SIZE)
|
||
await fh.read(hdr, 0, HEADER_SIZE, 0)
|
||
await fh.close()
|
||
return { format: detectFormat(hdr), path: source }
|
||
}
|
||
const bytes = await resolveToUint8Array(source)
|
||
return { format: detectFormat(bytes.subarray(0, HEADER_SIZE)), bytes }
|
||
}
|
||
```
|
||
|
||
`detectFormat` delegates to `@suaveplan/audio-metadata`'s magic-byte sniffing. Detection never
|
||
throws for unknown content — it returns `format: undefined`, and the caller falls back to an
|
||
explicit `opts.format` hint or throws `AUDIO_CODECS.UNKNOWN_FORMAT`. Reading only the first 256 KiB
|
||
keeps detection cheap even for multi-gigabyte files.
|
||
|
||
### 2.3. Progressive streaming decode
|
||
|
||
The decoder fills pages progressively so the document is editable before decode completes:
|
||
|
||
```ts
|
||
async function progressiveDecode(source, opts, registry) {
|
||
const { format, reader, bytes } = await detectSource(source)
|
||
const codec = registry.get(format ?? opts.format)
|
||
const dec = codec.createDecoder?.() ?? null
|
||
|
||
const acc = pageAccumulator({
|
||
pages: opts.pages ?? [],
|
||
notify: opts.notify,
|
||
ondata: opts.ondata,
|
||
})
|
||
|
||
if (reader) {
|
||
for await (const chunk of reader) {
|
||
const result = dec ? await dec.write(chunk) : await codec.decode(chunk)
|
||
if (result?.channelData.length) acc.push(result.channelData, result.sampleRate)
|
||
}
|
||
if (dec) {
|
||
const flushed = await dec.flush()
|
||
if (flushed?.channelData.length) acc.push(flushed.channelData, flushed.sampleRate)
|
||
}
|
||
}
|
||
|
||
const final = acc.done()
|
||
return createDocument(acc.pages, acc.sampleRate, acc.channels, final.length, opts, final.stats)
|
||
}
|
||
```
|
||
|
||
**Key property:** the returned `AudioDocument` is usable immediately. Because `audio-core` edits are
|
||
metadata (`enterprise-audio-core/design.md` §2.1), an edit chained onto the partial document applies
|
||
to whatever pages exist and extends automatically as the accumulator pushes more pages. `notify()`
|
||
fires when the first page is ready; `ondata(delta)` fires per decoded block with running stats.
|
||
|
||
### 2.4. Memory budget
|
||
|
||
When `opts.budget` is set, `decodeSource` / `progressiveDecode` project required memory as
|
||
`sampleRate × channels × durationSamples × 4` (bytes per `Float32` sample) and throw
|
||
`AUDIO_CODECS.BUDGET_EXCEEDED` before allocating any page when the projection exceeds the budget. The
|
||
projected and budgeted byte counts go into `error.context` so the caller can size a retry.
|
||
|
||
---
|
||
|
||
## 3. Encode model
|
||
|
||
`encodeTo(doc, destination, format, opts?)` renders the document's PCM through `audio-core`, encodes
|
||
it via the codec registered for `format`, then writes the bytes to the `Destination`:
|
||
|
||
- `string` file path → dynamic `import('node:fs/promises')`, `writeFile`.
|
||
- `WritableStream<Uint8Array>` → obtain a writer, write, close.
|
||
- `(chunk: Uint8Array) => void` callback → invoke per chunk (single chunk for non-streaming codecs).
|
||
|
||
Streaming encode is exposed two ways: `CodecRegistry.encodeStream(doc, format)` returns a
|
||
`ReadableStream<Uint8Array>`, and `createEncodeTransform(format)` returns a `@suaveplan/streams`
|
||
`TransformStream<Float32Array[], Uint8Array>` so a codec stage drops into a stream pipeline with
|
||
native backpressure.
|
||
|
||
---
|
||
|
||
## 4. Codec plugin contract
|
||
|
||
Every codec implements `AudioCodec`:
|
||
|
||
```ts
|
||
interface AudioCodec {
|
||
readonly name: string
|
||
readonly extensions: string[]
|
||
readonly mimeTypes: string[]
|
||
canDecode(bytes: Uint8Array): boolean
|
||
decode(bytes: Uint8Array | ArrayBuffer): Promise<DecodeResult>
|
||
encode(pcm: Float32Array[], opts: EncodeOpts): Promise<Uint8Array>
|
||
createDecoder?(): StreamingDecoder
|
||
createEncoder?(): StreamingEncoder
|
||
}
|
||
```
|
||
|
||
The `CodecRegistry` indexes each codec by `name`, every `extension`, and every `mimeType`.
|
||
`detect(bytes)` returns the first codec whose `canDecode` accepts the header. Codecs that do not
|
||
support encoding throw `AUDIO_CODECS.NOT_SUPPORTED` from `encode`; codecs that do not support
|
||
streaming **omit** `createDecoder`/`createEncoder` rather than ship a non-functional stub — the
|
||
registry checks for the factory's presence before offering a streaming path.
|
||
|
||
### 4.1. Built-in codecs
|
||
|
||
| Format | Decode | Encode | Streaming | Backing |
|
||
|---|---|---|---|---|
|
||
| WAV | yes | yes | yes | native, no deps |
|
||
| AIFF | yes | no | no | native, no deps |
|
||
| CAF | yes | no | no | native, no deps |
|
||
| MP3 | yes | yes | yes | WASM `@audio/decode-mp3` |
|
||
| FLAC | yes | yes | yes | WASM `@audio/decode-flac` |
|
||
| Ogg Vorbis | yes | no | yes | WASM |
|
||
| Opus | yes | yes | yes | WASM |
|
||
| AAC | yes | no | no | WASM |
|
||
| WebM | yes | no | no | WASM |
|
||
| QOA | yes | yes | yes | WASM |
|
||
|
||
Native codecs (WAV/AIFF/CAF) parse their container chunk structures directly and require zero
|
||
external dependencies. WASM codecs wrap a WebAssembly module loaded via dynamic `import()` of an
|
||
`optionalDependency`, so consumers install only the formats they use. The WASM module loads lazily
|
||
on first `decode`/`encode`/`createDecoder` — never at package import time — and a missing optional
|
||
dependency throws `AUDIO_CODECS.NOT_SUPPORTED` naming the absent module.
|
||
|
||
### 4.2. `@suaveplan/streams` integration
|
||
|
||
```ts
|
||
import { TransformStream } from '@suaveplan/streams'
|
||
|
||
function createDecodeTransform(format: string): TransformStream<Uint8Array, AudioPage> {
|
||
const codec = registry.get(format)
|
||
const dec = codec.createDecoder()!
|
||
return new TransformStream({
|
||
async transform(chunk, controller) {
|
||
const result = await dec.write(chunk)
|
||
if (result?.channelData.length) controller.enqueue(createPage(result.channelData))
|
||
},
|
||
async flush(controller) {
|
||
const result = await dec.flush()
|
||
if (result?.channelData.length) controller.enqueue(createPage(result.channelData))
|
||
},
|
||
})
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 5. Error hierarchy
|
||
|
||
`audio-codecs` registers its own `AUDIO_CODEC_*` namespace with `@suaveplan/error`, parallel to
|
||
`audio-core`'s `AUDIO_CORE_*` namespace:
|
||
|
||
```ts
|
||
defineCodes('AUDIO_CODEC', [
|
||
{ code: 'AUDIO_CODECS.UNKNOWN_FORMAT', severity: 'error', retryable: false },
|
||
{ code: 'AUDIO_CODECS.DECODE_FAILED', severity: 'error', retryable: false },
|
||
{ code: 'AUDIO_CODECS.ENCODE_FAILED', severity: 'error', retryable: false },
|
||
{ code: 'AUDIO_CODECS.STREAM_ERROR', severity: 'warning', retryable: true },
|
||
{ code: 'AUDIO_CODECS.NOT_SUPPORTED', severity: 'error', retryable: false },
|
||
{ code: 'AUDIO_CODECS.INVALID_SOURCE', severity: 'error', retryable: false },
|
||
{ code: 'AUDIO_CODECS.BUDGET_EXCEEDED', severity: 'error', retryable: false },
|
||
])
|
||
```
|
||
|
||
Every code is reachable from a real call site exercised by tests — `error-codes.test.ts` asserts
|
||
each metadata entry is thrown somewhere.
|
||
|
||
---
|
||
|
||
## 6. File layout
|
||
|
||
```text
|
||
packages/media/audio/audio-codecs/
|
||
package.json, tsconfig.json, vitest.config.ts
|
||
src/
|
||
index.ts
|
||
error-codes.ts
|
||
registry/ registry.ts .md .types.ts .test.ts index.ts
|
||
decode/ decode-source.ts .md .types.ts .test.ts index.ts
|
||
encode/ encode-to.ts .md .types.ts .test.ts index.ts
|
||
detect/ detect-format.ts .md .test.ts index.ts
|
||
stream/ stream-decoder.ts (+ .md .test.ts) · stream-encoder.ts (+ .md .test.ts) · index.ts
|
||
codecs/ wav.ts (+ .md .test.ts) · aiff.ts · caf.ts · mp3.ts (+ .md .test.ts) · flac.ts (+ .md .test.ts) · ogg-vorbis.ts · opus.ts · aac.ts · webm.ts · qoa.ts · index.ts
|
||
__benchmarks__/ decode.bench.ts encode.bench.ts stream-decode.bench.ts
|
||
tests/
|
||
integration.test.ts property.test.ts purity.test.ts exports.test.ts
|
||
```
|
||
|
||
---
|
||
|
||
## 7. Benchmarks
|
||
|
||
- `decode.bench.ts` — decode a 5-minute stereo WAV (native) and a 5-minute MP3 (WASM).
|
||
- `encode.bench.ts` — encode 5 minutes of stereo PCM to WAV and to FLAC.
|
||
- `stream-decode.bench.ts` — progressive stream decode of a 5-minute MP3, measuring time-to-first-page.
|
||
|
||
---
|
||
|
||
## 8. Decisions and trade-offs
|
||
|
||
- **Codecs are plugins, not a fixed enum.** A `CodecRegistry` rather than a hardcoded `switch`
|
||
means consumers can register custom or proprietary codecs, and the built-in set can shrink to just
|
||
the native WAV/AIFF/CAF codecs for a minimal install. This matches plugin slot 4 in the ecosystem
|
||
architecture.
|
||
- **WASM codecs are `optionalDependencies`.** Bundling every WASM decoder would bloat every install
|
||
with formats most consumers never touch. Declaring them optional and loading via dynamic `import()`
|
||
keeps the base package small and lets the dependency graph stay universal-tier — the import is
|
||
lazy and runtime-gated, so a browser build that never decodes FLAC never pulls the FLAC WASM.
|
||
- **`node:fs` is dynamic-import-only.** The package must run in browsers. Resolving a file path is
|
||
the one server-only operation, so it lives behind a runtime-gated `import('node:fs/promises')`
|
||
rather than a top-level import — preserving the universal-tier purity rule from
|
||
`enterprise-audio-core/design.md` §1.5.
|
||
- **Progressive decode returns the document early.** Because `audio-core` edits are metadata, the
|
||
document can be handed back before decode finishes and edits chained onto it extend as pages
|
||
arrive. This is the whole reason the codec layer produces `AudioPage` PCM rather than a monolithic
|
||
buffer — it lets a UI start editing a long file the instant the first page lands.
|
||
- **Memory budget is opt-in.** Projecting and checking allocation size on every decode would add
|
||
overhead to the common small-file case. `budget` is opt-in for callers decoding untrusted or
|
||
unbounded sources, where an OOM would otherwise crash the host.
|
||
- **The architecture doc is not re-stated here.** Per the same convention `audio-core` uses, the
|
||
cross-cutting model lives once in `enterprise-audio-core/design.md` §1–§4 and sibling proposals
|
||
reference it. Duplicating it would create drift.
|