suaveplan/openspec/changes/enterprise-audio-codecs/design.md

Design: enterprise-audio-codecs

Change ID: enterprise-audio-codecs Package: @suaveplan/audio-codecs

@suaveplan/audio-codecs is the universal format decode/encode layer of the @suaveplan/audio-* family. It turns any audio source — file path, URL, buffer, or byte stream — into an AudioDocument, and turns an AudioDocument back into encoded bytes in any supported format. This document carries the package-specific design for the codec layer.

---

1. Relationship to the ecosystem architecture

The cross-cutting @suaveplan/audio-* architecture — the three-layer separation (data model / timeline-transport / rendering engine), the six plugin slots, the "zero duplication of genesis primitives" rule, and the universal-tier purity rule — lives in enterprise-audio-core/design.md §1–§4. It is referenced here, not duplicated.

audio-codecs occupies plugin slot 4 (Codec) from that architecture: it decodes / encodes a specific format and registers codecs in a CodecRegistry. It sits directly above audio-core in the dependency graph and below every rendering engine and CLI. Its output is the universal AudioDocument defined by audio-core — the codec layer never owns its own audio container.

The Segment shape referenced anywhere in this package is the single family-wide definition from enterprise-audio-core/design.md §2.2: { from, count, to, rate?, ref?, interp? }. Codecs do not introduce segment variants — they produce AudioPage PCM, and editing/segmentation is audio-core's job.

---

2. Decode model

2.1. Source resolution

A Source is one of string | URL | ArrayBuffer | Uint8Array | ReadableStream<Uint8Array>. Resolution rules:

• string starting with http/https/data/blob, or any URL → fetch().
• string starting with file: → dynamic import('node:fs/promises').
• other bare string → fs/promises.readFile on a server runtime, fetch() in a browser.
• ArrayBuffer / Uint8Array / ReadableStream<Uint8Array> → consumed directly, no platform call.

The node:fs/promises import is always inside the runtime-gated branch — never a top-level import — so the package stays universal-tier per enterprise-audio-core/design.md §1.5.

2.2. Format detection

const HEADER_SIZE = 256 * 1024

async function detectSource(source: Source) {
  if (typeof source === 'string' && !source.startsWith('http')) {
    const { open } = await import('node:fs/promises')
    const fh = await open(source, 'r')
    const hdr = new Uint8Array(HEADER_SIZE)
    await fh.read(hdr, 0, HEADER_SIZE, 0)
    await fh.close()
    return { format: detectFormat(hdr), path: source }
  }
  const bytes = await resolveToUint8Array(source)
  return { format: detectFormat(bytes.subarray(0, HEADER_SIZE)), bytes }
}

detectFormat delegates to @suaveplan/audio-metadata's magic-byte sniffing. Detection never throws for unknown content — it returns format: undefined, and the caller falls back to an explicit opts.format hint or throws AUDIO_CODECS.UNKNOWN_FORMAT. Reading only the first 256 KiB keeps detection cheap even for multi-gigabyte files.

2.3. Progressive streaming decode

The decoder fills pages progressively so the document is editable before decode completes:

async function progressiveDecode(source, opts, registry) {
  const { format, reader, bytes } = await detectSource(source)
  const codec = registry.get(format ?? opts.format)
  const dec = codec.createDecoder?.() ?? null

  const acc = pageAccumulator({
    pages: opts.pages ?? [],
    notify: opts.notify,
    ondata: opts.ondata,
  })

  if (reader) {
    for await (const chunk of reader) {
      const result = dec ? await dec.write(chunk) : await codec.decode(chunk)
      if (result?.channelData.length) acc.push(result.channelData, result.sampleRate)
    }
    if (dec) {
      const flushed = await dec.flush()
      if (flushed?.channelData.length) acc.push(flushed.channelData, flushed.sampleRate)
    }
  }

  const final = acc.done()
  return createDocument(acc.pages, acc.sampleRate, acc.channels, final.length, opts, final.stats)
}

Key property: the returned AudioDocument is usable immediately. Because audio-core edits are metadata (enterprise-audio-core/design.md §2.1), an edit chained onto the partial document applies to whatever pages exist and extends automatically as the accumulator pushes more pages. notify() fires when the first page is ready; ondata(delta) fires per decoded block with running stats.

2.4. Memory budget

When opts.budget is set, decodeSource / progressiveDecode project required memory as sampleRate × channels × durationSamples × 4 (bytes per Float32 sample) and throw AUDIO_CODECS.BUDGET_EXCEEDED before allocating any page when the projection exceeds the budget. The projected and budgeted byte counts go into error.context so the caller can size a retry.

---

3. Encode model

encodeTo(doc, destination, format, opts?) renders the document's PCM through audio-core, encodes it via the codec registered for format, then writes the bytes to the Destination:

• string file path → dynamic import('node:fs/promises'), writeFile.
• WritableStream<Uint8Array> → obtain a writer, write, close.
• (chunk: Uint8Array) => void callback → invoke per chunk (single chunk for non-streaming codecs).

Streaming encode is exposed two ways: CodecRegistry.encodeStream(doc, format) returns a ReadableStream<Uint8Array>, and createEncodeTransform(format) returns a @suaveplan/streams TransformStream<Float32Array[], Uint8Array> so a codec stage drops into a stream pipeline with native backpressure.

---

4. Codec plugin contract

Every codec implements AudioCodec:

interface AudioCodec {
  readonly name: string
  readonly extensions: string[]
  readonly mimeTypes: string[]
  canDecode(bytes: Uint8Array): boolean
  decode(bytes: Uint8Array | ArrayBuffer): Promise<DecodeResult>
  encode(pcm: Float32Array[], opts: EncodeOpts): Promise<Uint8Array>
  createDecoder?(): StreamingDecoder
  createEncoder?(): StreamingEncoder
}

The CodecRegistry indexes each codec by name, every extension, and every mimeType. detect(bytes) returns the first codec whose canDecode accepts the header. Codecs that do not support encoding throw AUDIO_CODECS.NOT_SUPPORTED from encode; codecs that do not support streaming omit createDecoder/createEncoder rather than ship a non-functional stub — the registry checks for the factory's presence before offering a streaming path.

4.1. Built-in codecs

Format	Decode	Encode	Streaming	Backing
WAV	yes	yes	yes	native, no deps
AIFF	yes	no	no	native, no deps
CAF	yes	no	no	native, no deps
MP3	yes	yes	yes	WASM @audio/decode-mp3
FLAC	yes	yes	yes	WASM @audio/decode-flac
Ogg Vorbis	yes	no	yes	WASM
Opus	yes	yes	yes	WASM
AAC	yes	no	no	WASM
WebM	yes	no	no	WASM
QOA	yes	yes	yes	WASM

Native codecs (WAV/AIFF/CAF) parse their container chunk structures directly and require zero external dependencies. WASM codecs wrap a WebAssembly module loaded via dynamic import() of an optionalDependency, so consumers install only the formats they use. The WASM module loads lazily on first decode/encode/createDecoder — never at package import time — and a missing optional dependency throws AUDIO_CODECS.NOT_SUPPORTED naming the absent module.

4.2. @suaveplan/streams integration

import { TransformStream } from '@suaveplan/streams'

function createDecodeTransform(format: string): TransformStream<Uint8Array, AudioPage> {
  const codec = registry.get(format)
  const dec = codec.createDecoder()!
  return new TransformStream({
    async transform(chunk, controller) {
      const result = await dec.write(chunk)
      if (result?.channelData.length) controller.enqueue(createPage(result.channelData))
    },
    async flush(controller) {
      const result = await dec.flush()
      if (result?.channelData.length) controller.enqueue(createPage(result.channelData))
    },
  })
}

---

5. Error hierarchy

audio-codecs registers its own AUDIO_CODEC_* namespace with @suaveplan/error, parallel to audio-core's AUDIO_CORE_* namespace:

defineCodes('AUDIO_CODEC', [
  { code: 'AUDIO_CODECS.UNKNOWN_FORMAT',  severity: 'error',   retryable: false },
  { code: 'AUDIO_CODECS.DECODE_FAILED',   severity: 'error',   retryable: false },
  { code: 'AUDIO_CODECS.ENCODE_FAILED',   severity: 'error',   retryable: false },
  { code: 'AUDIO_CODECS.STREAM_ERROR',    severity: 'warning', retryable: true  },
  { code: 'AUDIO_CODECS.NOT_SUPPORTED',   severity: 'error',   retryable: false },
  { code: 'AUDIO_CODECS.INVALID_SOURCE',  severity: 'error',   retryable: false },
  { code: 'AUDIO_CODECS.BUDGET_EXCEEDED', severity: 'error',   retryable: false },
])

Every code is reachable from a real call site exercised by tests — error-codes.test.ts asserts each metadata entry is thrown somewhere.

---

6. File layout

packages/media/audio/audio-codecs/
  package.json, tsconfig.json, vitest.config.ts
  src/
    index.ts
    error-codes.ts
    registry/    registry.ts .md .types.ts .test.ts index.ts
    decode/      decode-source.ts .md .types.ts .test.ts index.ts
    encode/      encode-to.ts .md .types.ts .test.ts index.ts
    detect/      detect-format.ts .md .test.ts index.ts
    stream/      stream-decoder.ts (+ .md .test.ts) · stream-encoder.ts (+ .md .test.ts) · index.ts
    codecs/      wav.ts (+ .md .test.ts) · aiff.ts · caf.ts · mp3.ts (+ .md .test.ts) · flac.ts (+ .md .test.ts) · ogg-vorbis.ts · opus.ts · aac.ts · webm.ts · qoa.ts · index.ts
    __benchmarks__/  decode.bench.ts encode.bench.ts stream-decode.bench.ts
  tests/
    integration.test.ts  property.test.ts  purity.test.ts  exports.test.ts

---

7. Benchmarks

• decode.bench.ts — decode a 5-minute stereo WAV (native) and a 5-minute MP3 (WASM).
• encode.bench.ts — encode 5 minutes of stereo PCM to WAV and to FLAC.
• stream-decode.bench.ts — progressive stream decode of a 5-minute MP3, measuring time-to-first-page.

---

8. Decisions and trade-offs

• Codecs are plugins, not a fixed enum. A CodecRegistry rather than a hardcoded switch means consumers can register custom or proprietary codecs, and the built-in set can shrink to just the native WAV/AIFF/CAF codecs for a minimal install. This matches plugin slot 4 in the ecosystem architecture.
• WASM codecs are optionalDependencies. Bundling every WASM decoder would bloat every install with formats most consumers never touch. Declaring them optional and loading via dynamic import() keeps the base package small and lets the dependency graph stay universal-tier — the import is lazy and runtime-gated, so a browser build that never decodes FLAC never pulls the FLAC WASM.
• node:fs is dynamic-import-only. The package must run in browsers. Resolving a file path is the one server-only operation, so it lives behind a runtime-gated import('node:fs/promises') rather than a top-level import — preserving the universal-tier purity rule from enterprise-audio-core/design.md §1.5.
• Progressive decode returns the document early. Because audio-core edits are metadata, the document can be handed back before decode finishes and edits chained onto it extend as pages arrive. This is the whole reason the codec layer produces AudioPage PCM rather than a monolithic buffer — it lets a UI start editing a long file the instant the first page lands.
• Memory budget is opt-in. Projecting and checking allocation size on every decode would add overhead to the common small-file case. budget is opt-in for callers decoding untrusted or unbounded sources, where an OOM would otherwise crash the host.
• The architecture doc is not re-stated here. Per the same convention audio-core uses, the cross-cutting model lives once in enterprise-audio-core/design.md §1–§4 and sibling proposals reference it. Duplicating it would create drift.