Initial Commit

openspec/archive/2026-05-14-enterprise-audio-core/design.md

@@ -0,0 +1,357 @@
+# Design: enterprise-audio-core
+
+**Change ID**: enterprise-audio-core
+**Package**: `@suaveplan/audio-core`
+
+`@suaveplan/audio-core` is the foundation of the `@suaveplan/audio-*` family. This document carries
+both the **cross-cutting architecture** for the whole audio ecosystem (because `audio-core` defines
+the abstractions every sibling package builds on) and the **package-specific design** for the core
+itself. Sibling change proposals reference §1–§4 of this document for the shared model.
+
+---
+
+## 1. Ecosystem architecture
+
+### 1.1. Goal
+
+A cross-platform audio processing ecosystem that runs identically in browsers (Web Audio API),
+servers (Node/Bun), and offline render pipelines. The **universal data model** (pages, plans, edits)
+is separated from **platform-specific rendering engines** (browser AudioNodes, server block
+processors) so every package above the core is environment-agnostic.
+
+The ecosystem is:
+
+- **Plugin-first** — every capability, from a gain boost to beat detection to a full DAW timeline,
+  is a plugin.
+- **Stream-native** — audio flows through `TransformStream` pipelines using `@suaveplan/streams`.
+- **Non-destructive** — edits are metadata; materialisation is deferred until read/stream/render.
+- **Feature-rich** — capable of everything from format conversion to a working DAW with beat
+  detection, auto-mixing, pitch/tempo adjustment, tempo/key detection, and chainable effects.
+
+### 1.2. Three-layer separation
+
+| Layer | Responsibility | Environments |
+|---|---|---|
+| **Data Model** (`audio-core`) | PCM pages, edit plans, block processors, op registry | Universal |
+| **Timeline / Transport** (`audio-timeline`, `audio-transport`) | Tracks, clips, musical time, scheduling, undo/redo | Universal |
+| **Rendering Engine** (`audio-engine-*`) | Node graphs, speakers, file I/O, MIDI | Browser / Server / Offline |
+
+### 1.3. The six plugin slots
+
+The core defines six plugin slots; every audio feature occupies one of them.
+
+1. **Plan Op** — transforms segment lists (changes length / rate / channels).
+2. **Process Op** — block-level DSP on `Float32Array[]`.
+3. **Analysis Op** — reads audio and emits features (BPM, key, spectrum); runs as a sidecar scan,
+   not part of the render pipeline.
+4. **Codec** — decodes / encodes a specific format; registered in a `CodecRegistry`.
+5. **Timeline Adapter** — renders timeline state to a platform backend.
+6. **FX Plugin** — dual-mode effect with both `processBlock()` (universal) and optional
+   `createAudioNode()` (browser).
+
+### 1.4. Zero duplication of genesis primitives
+
+Audio packages reuse existing genesis packages rather than re-implementing them.
+
+| Capability | Genesis package | Usage in audio |
+|---|---|---|
+| Streams / backpressure | `@suaveplan/streams` | Audio `TransformStream` pipelines, progressive decode/encode |
+| Scheduling / clocks | `@suaveplan/timing` | Transport lookahead scheduler, tempo clock |
+| Event bus | `@suaveplan/events` | Document lifecycle, engine, transport, per-track events |
+| OTel metrics / tracing | `@suaveplan/events-otel`, `@suaveplan/opentelemetry` | Plugin latency histograms, CPU counters, transport spans |
+| Undo / redo | `@suaveplan/undo` | Timeline undo/redo with branching history and command merging |
+| State machines | `@suaveplan/fsm` | Transport playback state, engine lifecycle, MIDI controller state |
+| Serialization | `@suaveplan/serialize` | Session save/load (edit lists, transport state, plugin presets) |
+| Binary packing | `@suaveplan/bitpacker`, `@suaveplan/packer` | Compressed sprite maps, custom binary formats, state compression |
+| Metadata extraction | `@suaveplan/audio-metadata` | Format detection, ID3/MP4/FLAC tags during decode |
+| Voice interfaces | `@suaveplan/voice` | TTS/STT integration points in the timeline |
+| Error substrate | `@suaveplan/error` | `AudioError` base class, typed `AUDIO.*` codes |
+| Schema registry | `@suaveplan/schema-registry` | Plugin preset Zod schemas, descriptor registration |
+| Validation | `@suaveplan/validation` | Runtime boundary validation where Zod is unavailable |
+| Testing | `@suaveplan/testing` | `FakeClock`, shared runner primitives |
+
+### 1.5. No platform assumptions in universal packages
+
+Universal packages (`audio-core`, `audio-timeline`, `audio-transport`, `audio-analysis-*`,
+`audio-plugins-fx`) import zero browser APIs (`window`, `document`, `navigator`, `AudioContext`) and
+zero Node-only APIs (`fs`, `process`) except via injected adapters, and pass all unit tests under
+`bun test` (server) and vitest with a `happy-dom` shim (browser).
+
+### 1.6. Architecture diagram
+
+```text
+┌─────────────────────────────────────────────────────────────────────┐
+│                         APPLICATION LAYER                            │
+│  @suaveplan/react-audio*  |  @suaveplan/r3f-audio  |  audio-cli       │
+└─────────────────────────────────────────────────────────────────────┘
+                              │
+┌─────────────────────────────────────────────────────────────────────┐
+│                         TIMELINE / TRANSPORT                         │
+│  @suaveplan/audio-timeline  ──TimelineAdapter──► rendering engine     │
+│  @suaveplan/audio-transport ──TransportScheduler──► lookahead loop    │
+└─────────────────────────────────────────────────────────────────────┘
+                              │
+┌─────────────────────────────────────────────────────────────────────┐
+│                         ANALYSIS + FX PLUGINS                        │
+│  @suaveplan/audio-analysis-*  |  @suaveplan/audio-plugins-fx          │
+│  (BlockProcessor + AnalysisOp interfaces)                            │
+└─────────────────────────────────────────────────────────────────────┘
+                              │
+┌─────────────────────────────────────────────────────────────────────┐
+│                         UNIVERSAL CORE — @suaveplan/audio-core       │
+│  AudioDocument (pages, edits, plans) · PlanEngine (segment compile)  │
+│  ProcessPipeline (block DSP) · OpRegistry (plan + process ops)       │
+└─────────────────────────────────────────────────────────────────────┘
+                              │
+        ┌─────────────────────┼─────────────────────┐
+        ▼                     ▼                     ▼
+  BROWSER ENGINE        SERVER I/O ONLY        OFFLINE RENDER
+  AudioContext /        fs read/write          audio-core
+  AudioWorklet /        file streams           .renderPlan()
+  AudioNodes
+```
+
+---
+
+## 2. Core abstractions
+
+### 2.1. `AudioDocument`
+
+The universal audio container — replaces the browser-only `AudioBuffer`. PCM is stored as paged
+`Float32Array[]` data so the model maps cleanly to both browser memory limits and server stream
+chunks. `length`, `duration`, and the stats are virtual: they read cached values invalidated by
+`version` bumps, so editing is O(1) (push to `edits[]`) and materialisation is deferred to `read()`
+or `stream()`.
+
+### 2.2. `Segment` — the canonical definition
+
+A `CompiledPlan` is a list of `Segment` records describing how output samples map back to source
+samples. **There is exactly one `Segment` shape across the whole audio family:**
+
+```ts
+interface Segment {
+  from: number                 // source sample start
+  count: number                // source sample count
+  to: number                   // output sample start
+  rate?: number                // playback rate (1 = normal); affects resampling
+  ref?: AudioDocument | null    // external source reference (null = silence segment)
+  interp?: Interpolator         // custom resampler for non-unity rate
+}
+```
+
+Plan ops receive and return `Segment[]`; process ops run per-block over the rendered output of the
+segment list. The earlier draft variants (`{ srcStart, srcEnd, dstOffset }` and the positional
+tuple `[from, count, to]`) are superseded by this record form — they are not used anywhere.
+
+### 2.3. `PlanEngine`
+
+Compiles `AudioEdit[]` → `CompiledPlan` in two passes:
+
+1. **Resolve pass** — expand high-level edits into primitive edits using source stats
+   (`op.resolve`).
+2. **Plan pass** — transform the `Segment[]` for each edit (`op.plan`); ops without a `plan`
+   function are appended to `pipeline` as process-stage ops.
+
+```ts
+function compilePlan(doc: AudioDocument, final: boolean): CompiledPlan {
+  let segs: Segment[] = [{ from: 0, count: doc.sourceLength, to: 0 }]
+  let pipeline: AudioEdit[] = []
+  let limit = doc.sourceLength
+  let sr = doc.sampleRate
+  let ch = doc.channels
+
+  for (const edit of doc.edits) {
+    const op = registry.get(edit.type)
+    if (!op) throw new AudioError('AUDIO_PLUGIN_NOT_FOUND', { context: { type: edit.type } })
+
+    if (op.resolve) {
+      const resolved = op.resolve({ stats: doc.stats, sampleRate: sr, channels: ch, opts: edit.opts })
+      if (resolved === false) continue
+      // resolved edits are spliced in and recursed over
+    }
+
+    if (op.plan) {
+      segs = op.plan(segs, mkPlanCtx(planLen(segs), sr, edit.opts))
+      if (!final) limit = adjustLimit(limit, edit.type)
+    } else {
+      pipeline.push(edit)
+    }
+
+    if (op.sr) sr = op.sr(sr, edit.opts) ?? sr
+    if (op.ch) ch = op.ch(ch, edit.opts) ?? ch
+  }
+
+  return { segs, pipeline, totalLen: planLen(segs), sampleRate: sr, limit: Math.max(0, limit) }
+}
+```
+
+### 2.4. `ProcessPipeline`
+
+Runs process ops per `BLOCK_SIZE` (1024 samples) using double-buffer rotation. `init` allocates
+`ProcessState[]`; `patch` hot-swaps parameter values without re-allocation; `process` runs each op
+over a block. On a **structural** change (op added/removed, detected via a pipeline signature) the
+pipeline reinitialises affected processors and crossfades over ~20 ms to avoid clicks.
+
+### 2.5. `BlockProcessor` — the universal DSP interface
+
+Every DSP plugin (FX, analysis, codec post-processing) implements:
+
+```ts
+interface BlockProcessor {
+  readonly name: string
+  setup(config: ProcessorConfig): void
+  process(input: Float32Array[], output: Float32Array[], ctx: BlockContext): void
+  reset(): void
+  dispose(): void
+}
+```
+
+This is the interface that works everywhere: browser worklets, server batch pipelines, offline
+renders, and WebAssembly modules.
+
+### 2.6. Error isolation
+
+A crashing process op must not corrupt the stream. `ProcessPipeline.process` wraps each op:
+
+```ts
+try {
+  op.process(input, output, ctx)
+} catch (e) {
+  emit(doc, 'plugin:error', { plugin: op.name, error: e })
+  for (let c = 0; c < input.length; c++) output[c].set(input[c])  // pass-through
+}
+```
+
+---
+
+## 3. Integration with genesis packages
+
+- **`@suaveplan/streams`** — `AudioDocument.stream()` and `PlanEngine.streamPlan()` yield blocks
+  composable with `TransformStream`; a helper wraps a compiled plan + process pipeline as a
+  `TransformStream<Float32Array[], Float32Array[]>` so downstream FX get native backpressure.
+  Progressive decode (in `audio-codecs`) returns a `ReadableStream<AudioPage>`; the document is
+  editable before decode completes because edits are metadata.
+- **`@suaveplan/events`** — backs the typed `AudioDocumentEventMap` (`edit`, `undo`,
+  `plugin:error`, `page:loaded`). The interceptor API is the attachment point for OTel metrics,
+  schema validation, and cross-context bridges in the engine packages.
+- **`@suaveplan/serialize`** — `AudioDocument.toJSON()` round-trips through `serialize`/
+  `deserialize`, preserving `Float32Array` page data, `edits`, `version`, `source`, `metadata`.
+- **`@suaveplan/audio-metadata`** — supplies the `AudioMetadata` type carried on
+  `AudioDocument.metadata`. `audio-core` depends on `@suaveplan/audio-metadata` for this type only;
+  population happens in `audio-codecs` during decode.
+
+---
+
+## 4. Error hierarchy
+
+Every audio package extends `@suaveplan/error`. `audio-core` registers the foundation codes:
+
+```ts
+defineCodes('AUDIO_CORE', [
+  { code: 'AUDIO_PLUGIN_NOT_FOUND',  severity: 'error',   retryable: false },
+  { code: 'AUDIO_INVALID_EDIT',      severity: 'error',   retryable: false },
+  { code: 'AUDIO_STREAM_ABORTED',    severity: 'warning', retryable: true  },
+  { code: 'AUDIO_PAGE_OUT_OF_RANGE', severity: 'error',   retryable: false },
+  { code: 'AUDIO_PROCESS_ERROR',     severity: 'error',   retryable: true  },
+])
+```
+
+Sibling packages register their own `AUDIO_<DOMAIN>` namespaces (engine, codec, analysis, etc.).
+
+---
+
+## 5. Package-specific design
+
+### 5.1. Paged memory model
+
+```ts
+const PAGE_SIZE = 1024 * 1024  // samples (~23 MB stereo @ 48 kHz)
+const BLOCK_SIZE = 1024        // samples per process block
+```
+
+`AudioDocument` owns an `Array<AudioPage | null>` where `null` marks an evicted page that can be
+reloaded from cache. Pages are allocated from a pool to reduce GC pressure. Primitives:
+
+- `walkPages(doc, channel, startSample, count, visitor)` — iterate overlapping pages.
+- `copyPages(doc, channel, startSample, count, target, targetOffset)` — copy into a pre-allocated
+  buffer without allocating.
+- `readPages(doc, offset?, duration?)` — read a range as `Float32Array[]`.
+
+### 5.2. Streaming
+
+`AudioDocument.stream()` is an async generator that compiles a non-final plan, initialises the
+process pipeline, and yields `Float32Array[]` blocks of at most `BLOCK_SIZE` samples:
+
+```ts
+async function* stream(opts?: StreamOptions) {
+  const plan = planEngine.compile(doc, false)
+  const procs = processPipeline.init(plan.pipeline, plan.totalLen, plan.sampleRate, doc.channels)
+  let outPos = opts?.startSample ?? 0
+  const end = opts?.endSample ?? plan.totalLen
+  while (outPos < end) {
+    if (opts?.signal?.aborted) throw new AudioError('AUDIO_STREAM_ABORTED')
+    const blockLen = Math.min(BLOCK_SIZE, end - outPos)
+    renderBlock(doc, plan.segs, outPos, blockLen, bufA)
+    const out = procs.length
+      ? processPipeline.process(bufA, bufB, procs, outPos, plan.sampleRate)
+      : bufA
+    yield out.map((b) => b.subarray(0, blockLen))
+    outPos += blockLen
+  }
+}
+```
+
+### 5.3. Built-in operations
+
+Plan ops: `trim`, `crop`, `speed`, `stretch`, `reverse`, `insert`, `remove`, `repeat`, `pad`,
+`resample`. Process ops: `gain`, `normalize`, `fade`, `pan`, `mix`, `dither`, `transform`.
+
+`stretch` is dual-stage — a plan op that changes duration plus a process op that preserves pitch.
+Pitch *shifting* without a duration change is **not** a core built-in: it ships as the
+`PitchShifterPlugin` in `@suaveplan/audio-plugins-fx`. (The earlier draft's `registry/built-in/
+pitch.ts` entry is dropped — there is no core `pitch` op.)
+
+### 5.4. File layout
+
+```text
+packages/audio/audio-core/
+  package.json, tsconfig.json, vitest.config.ts
+  src/
+    index.ts
+    error-codes.ts
+    document/    document.ts .md .types.ts .test.ts index.ts
+    page/        page.ts .md .types.ts .test.ts index.ts
+    plan/        plan-engine.ts (+ .md .types.ts .test.ts) · segment.ts (+ .md .types.ts .test.ts) · index.ts
+    process/     process-pipeline.ts (+ .md .types.ts .test.ts) · block-processor.ts (+ .md .types.ts .test.ts) · index.ts
+    registry/    op-registry.ts (+ .md .types.ts .test.ts) · built-in/{trim,crop,speed,stretch,reverse,insert,remove,repeat,pad,resample,gain,normalize,fade,pan,mix,dither,transform}.ts · index.ts
+    stream/      stream.ts .md .types.ts .test.ts index.ts
+    render/      render.ts .md .test.ts index.ts
+    stats/       stats.ts .md .types.ts .test.ts index.ts
+    __benchmarks__/  plan.bench.ts stream.bench.ts process.bench.ts
+  tests/
+    integration.test.ts  property.test.ts  purity.test.ts  exports.test.ts
+```
+
+### 5.5. Benchmarks
+
+- `plan.bench.ts` — compile 100 edits into a plan (1000 iterations).
+- `stream.bench.ts` — stream 5 minutes of 48 kHz stereo through 10 process ops.
+- `process.bench.ts` — run `normalize` + `fade` + `pan` over 1M samples.
+
+---
+
+## 6. Decisions and trade-offs
+
+- **One `Segment` shape, everywhere.** The richer record form (`from`/`count`/`to` + optional
+  `rate`/`ref`/`interp`) is canonical because `ref` enables `insert`/`mix` from other documents and
+  `rate`/`interp` enable `speed`/`resample` without a separate segment variant. Picking one shape
+  now avoids translation layers between core and the timeline/engine packages.
+- **`audio-core` carries the ecosystem architecture.** Rather than a free-standing architecture
+  spec under `openspec/specs/`, the cross-cutting model lives in this design doc because every
+  abstraction it describes (`OpRegistry`, `BlockProcessor`, the six plugin slots, the error base) is
+  *defined* by `audio-core`. Sibling proposals reference §1–§4 here.
+- **Pitch shift is a plugin, not a core op.** Core built-ins are limited to operations expressible
+  as pure segment transforms or simple per-block DSP. Phase-vocoder pitch shifting is stateful and
+  belongs in `audio-plugins-fx`.
+- **No design.md for the React/r3f tier yet.** `react-audio*` and `r3f-audio` are out of scope for
+  this 13-change batch; they land in a later wave once the universal and platform tiers are green.