# Design: enterprise-audio-core **Change ID**: enterprise-audio-core **Package**: `@suaveplan/audio-core` `@suaveplan/audio-core` is the foundation of the `@suaveplan/audio-*` family. This document carries both the **cross-cutting architecture** for the whole audio ecosystem (because `audio-core` defines the abstractions every sibling package builds on) and the **package-specific design** for the core itself. Sibling change proposals reference §1–§4 of this document for the shared model. --- ## 1. Ecosystem architecture ### 1.1. Goal A cross-platform audio processing ecosystem that runs identically in browsers (Web Audio API), servers (Node/Bun), and offline render pipelines. The **universal data model** (pages, plans, edits) is separated from **platform-specific rendering engines** (browser AudioNodes, server block processors) so every package above the core is environment-agnostic. The ecosystem is: - **Plugin-first** — every capability, from a gain boost to beat detection to a full DAW timeline, is a plugin. - **Stream-native** — audio flows through `TransformStream` pipelines using `@suaveplan/streams`. - **Non-destructive** — edits are metadata; materialisation is deferred until read/stream/render. - **Feature-rich** — capable of everything from format conversion to a working DAW with beat detection, auto-mixing, pitch/tempo adjustment, tempo/key detection, and chainable effects. ### 1.2. Three-layer separation | Layer | Responsibility | Environments | |---|---|---| | **Data Model** (`audio-core`) | PCM pages, edit plans, block processors, op registry | Universal | | **Timeline / Transport** (`audio-timeline`, `audio-transport`) | Tracks, clips, musical time, scheduling, undo/redo | Universal | | **Rendering Engine** (`audio-engine-*`) | Node graphs, speakers, file I/O, MIDI | Browser / Server / Offline | ### 1.3. The six plugin slots The core defines six plugin slots; every audio feature occupies one of them. 1. **Plan Op** — transforms segment lists (changes length / rate / channels). 2. **Process Op** — block-level DSP on `Float32Array[]`. 3. **Analysis Op** — reads audio and emits features (BPM, key, spectrum); runs as a sidecar scan, not part of the render pipeline. 4. **Codec** — decodes / encodes a specific format; registered in a `CodecRegistry`. 5. **Timeline Adapter** — renders timeline state to a platform backend. 6. **FX Plugin** — dual-mode effect with both `processBlock()` (universal) and optional `createAudioNode()` (browser). ### 1.4. Zero duplication of genesis primitives Audio packages reuse existing genesis packages rather than re-implementing them. | Capability | Genesis package | Usage in audio | |---|---|---| | Streams / backpressure | `@suaveplan/streams` | Audio `TransformStream` pipelines, progressive decode/encode | | Scheduling / clocks | `@suaveplan/timing` | Transport lookahead scheduler, tempo clock | | Event bus | `@suaveplan/events` | Document lifecycle, engine, transport, per-track events | | OTel metrics / tracing | `@suaveplan/events-otel`, `@suaveplan/opentelemetry` | Plugin latency histograms, CPU counters, transport spans | | Undo / redo | `@suaveplan/undo` | Timeline undo/redo with branching history and command merging | | State machines | `@suaveplan/fsm` | Transport playback state, engine lifecycle, MIDI controller state | | Serialization | `@suaveplan/serialize` | Session save/load (edit lists, transport state, plugin presets) | | Binary packing | `@suaveplan/bitpacker`, `@suaveplan/packer` | Compressed sprite maps, custom binary formats, state compression | | Metadata extraction | `@suaveplan/audio-metadata` | Format detection, ID3/MP4/FLAC tags during decode | | Voice interfaces | `@suaveplan/voice` | TTS/STT integration points in the timeline | | Error substrate | `@suaveplan/error` | `AudioError` base class, typed `AUDIO.*` codes | | Schema registry | `@suaveplan/schema-registry` | Plugin preset Zod schemas, descriptor registration | | Validation | `@suaveplan/validation` | Runtime boundary validation where Zod is unavailable | | Testing | `@suaveplan/testing` | `FakeClock`, shared runner primitives | ### 1.5. No platform assumptions in universal packages Universal packages (`audio-core`, `audio-timeline`, `audio-transport`, `audio-analysis-*`, `audio-plugins-fx`) import zero browser APIs (`window`, `document`, `navigator`, `AudioContext`) and zero Node-only APIs (`fs`, `process`) except via injected adapters, and pass all unit tests under `bun test` (server) and vitest with a `happy-dom` shim (browser). ### 1.6. Architecture diagram ```text ┌─────────────────────────────────────────────────────────────────────┐ │ APPLICATION LAYER │ │ @suaveplan/react-audio* | @suaveplan/r3f-audio | audio-cli │ └─────────────────────────────────────────────────────────────────────┘ │ ┌─────────────────────────────────────────────────────────────────────┐ │ TIMELINE / TRANSPORT │ │ @suaveplan/audio-timeline ──TimelineAdapter──► rendering engine │ │ @suaveplan/audio-transport ──TransportScheduler──► lookahead loop │ └─────────────────────────────────────────────────────────────────────┘ │ ┌─────────────────────────────────────────────────────────────────────┐ │ ANALYSIS + FX PLUGINS │ │ @suaveplan/audio-analysis-* | @suaveplan/audio-plugins-fx │ │ (BlockProcessor + AnalysisOp interfaces) │ └─────────────────────────────────────────────────────────────────────┘ │ ┌─────────────────────────────────────────────────────────────────────┐ │ UNIVERSAL CORE — @suaveplan/audio-core │ │ AudioDocument (pages, edits, plans) · PlanEngine (segment compile) │ │ ProcessPipeline (block DSP) · OpRegistry (plan + process ops) │ └─────────────────────────────────────────────────────────────────────┘ │ ┌─────────────────────┼─────────────────────┐ ▼ ▼ ▼ BROWSER ENGINE SERVER I/O ONLY OFFLINE RENDER AudioContext / fs read/write audio-core AudioWorklet / file streams .renderPlan() AudioNodes ``` --- ## 2. Core abstractions ### 2.1. `AudioDocument` The universal audio container — replaces the browser-only `AudioBuffer`. PCM is stored as paged `Float32Array[]` data so the model maps cleanly to both browser memory limits and server stream chunks. `length`, `duration`, and the stats are virtual: they read cached values invalidated by `version` bumps, so editing is O(1) (push to `edits[]`) and materialisation is deferred to `read()` or `stream()`. ### 2.2. `Segment` — the canonical definition A `CompiledPlan` is a list of `Segment` records describing how output samples map back to source samples. **There is exactly one `Segment` shape across the whole audio family:** ```ts interface Segment { from: number // source sample start count: number // source sample count to: number // output sample start rate?: number // playback rate (1 = normal); affects resampling ref?: AudioDocument | null // external source reference (null = silence segment) interp?: Interpolator // custom resampler for non-unity rate } ``` Plan ops receive and return `Segment[]`; process ops run per-block over the rendered output of the segment list. The earlier draft variants (`{ srcStart, srcEnd, dstOffset }` and the positional tuple `[from, count, to]`) are superseded by this record form — they are not used anywhere. ### 2.3. `PlanEngine` Compiles `AudioEdit[]` → `CompiledPlan` in two passes: 1. **Resolve pass** — expand high-level edits into primitive edits using source stats (`op.resolve`). 2. **Plan pass** — transform the `Segment[]` for each edit (`op.plan`); ops without a `plan` function are appended to `pipeline` as process-stage ops. ```ts function compilePlan(doc: AudioDocument, final: boolean): CompiledPlan { let segs: Segment[] = [{ from: 0, count: doc.sourceLength, to: 0 }] let pipeline: AudioEdit[] = [] let limit = doc.sourceLength let sr = doc.sampleRate let ch = doc.channels for (const edit of doc.edits) { const op = registry.get(edit.type) if (!op) throw new AudioError('AUDIO_PLUGIN_NOT_FOUND', { context: { type: edit.type } }) if (op.resolve) { const resolved = op.resolve({ stats: doc.stats, sampleRate: sr, channels: ch, opts: edit.opts }) if (resolved === false) continue // resolved edits are spliced in and recursed over } if (op.plan) { segs = op.plan(segs, mkPlanCtx(planLen(segs), sr, edit.opts)) if (!final) limit = adjustLimit(limit, edit.type) } else { pipeline.push(edit) } if (op.sr) sr = op.sr(sr, edit.opts) ?? sr if (op.ch) ch = op.ch(ch, edit.opts) ?? ch } return { segs, pipeline, totalLen: planLen(segs), sampleRate: sr, limit: Math.max(0, limit) } } ``` ### 2.4. `ProcessPipeline` Runs process ops per `BLOCK_SIZE` (1024 samples) using double-buffer rotation. `init` allocates `ProcessState[]`; `patch` hot-swaps parameter values without re-allocation; `process` runs each op over a block. On a **structural** change (op added/removed, detected via a pipeline signature) the pipeline reinitialises affected processors and crossfades over ~20 ms to avoid clicks. ### 2.5. `BlockProcessor` — the universal DSP interface Every DSP plugin (FX, analysis, codec post-processing) implements: ```ts interface BlockProcessor { readonly name: string setup(config: ProcessorConfig): void process(input: Float32Array[], output: Float32Array[], ctx: BlockContext): void reset(): void dispose(): void } ``` This is the interface that works everywhere: browser worklets, server batch pipelines, offline renders, and WebAssembly modules. ### 2.6. Error isolation A crashing process op must not corrupt the stream. `ProcessPipeline.process` wraps each op: ```ts try { op.process(input, output, ctx) } catch (e) { emit(doc, 'plugin:error', { plugin: op.name, error: e }) for (let c = 0; c < input.length; c++) output[c].set(input[c]) // pass-through } ``` --- ## 3. Integration with genesis packages - **`@suaveplan/streams`** — `AudioDocument.stream()` and `PlanEngine.streamPlan()` yield blocks composable with `TransformStream`; a helper wraps a compiled plan + process pipeline as a `TransformStream` so downstream FX get native backpressure. Progressive decode (in `audio-codecs`) returns a `ReadableStream`; the document is editable before decode completes because edits are metadata. - **`@suaveplan/events`** — backs the typed `AudioDocumentEventMap` (`edit`, `undo`, `plugin:error`, `page:loaded`). The interceptor API is the attachment point for OTel metrics, schema validation, and cross-context bridges in the engine packages. - **`@suaveplan/serialize`** — `AudioDocument.toJSON()` round-trips through `serialize`/ `deserialize`, preserving `Float32Array` page data, `edits`, `version`, `source`, `metadata`. - **`@suaveplan/audio-metadata`** — supplies the `AudioMetadata` type carried on `AudioDocument.metadata`. `audio-core` depends on `@suaveplan/audio-metadata` for this type only; population happens in `audio-codecs` during decode. --- ## 4. Error hierarchy Every audio package extends `@suaveplan/error`. `audio-core` registers the foundation codes: ```ts defineCodes('AUDIO_CORE', [ { code: 'AUDIO_PLUGIN_NOT_FOUND', severity: 'error', retryable: false }, { code: 'AUDIO_INVALID_EDIT', severity: 'error', retryable: false }, { code: 'AUDIO_STREAM_ABORTED', severity: 'warning', retryable: true }, { code: 'AUDIO_PAGE_OUT_OF_RANGE', severity: 'error', retryable: false }, { code: 'AUDIO_PROCESS_ERROR', severity: 'error', retryable: true }, ]) ``` Sibling packages register their own `AUDIO_` namespaces (engine, codec, analysis, etc.). --- ## 5. Package-specific design ### 5.1. Paged memory model ```ts const PAGE_SIZE = 1024 * 1024 // samples (~23 MB stereo @ 48 kHz) const BLOCK_SIZE = 1024 // samples per process block ``` `AudioDocument` owns an `Array` where `null` marks an evicted page that can be reloaded from cache. Pages are allocated from a pool to reduce GC pressure. Primitives: - `walkPages(doc, channel, startSample, count, visitor)` — iterate overlapping pages. - `copyPages(doc, channel, startSample, count, target, targetOffset)` — copy into a pre-allocated buffer without allocating. - `readPages(doc, offset?, duration?)` — read a range as `Float32Array[]`. ### 5.2. Streaming `AudioDocument.stream()` is an async generator that compiles a non-final plan, initialises the process pipeline, and yields `Float32Array[]` blocks of at most `BLOCK_SIZE` samples: ```ts async function* stream(opts?: StreamOptions) { const plan = planEngine.compile(doc, false) const procs = processPipeline.init(plan.pipeline, plan.totalLen, plan.sampleRate, doc.channels) let outPos = opts?.startSample ?? 0 const end = opts?.endSample ?? plan.totalLen while (outPos < end) { if (opts?.signal?.aborted) throw new AudioError('AUDIO_STREAM_ABORTED') const blockLen = Math.min(BLOCK_SIZE, end - outPos) renderBlock(doc, plan.segs, outPos, blockLen, bufA) const out = procs.length ? processPipeline.process(bufA, bufB, procs, outPos, plan.sampleRate) : bufA yield out.map((b) => b.subarray(0, blockLen)) outPos += blockLen } } ``` ### 5.3. Built-in operations Plan ops: `trim`, `crop`, `speed`, `stretch`, `reverse`, `insert`, `remove`, `repeat`, `pad`, `resample`. Process ops: `gain`, `normalize`, `fade`, `pan`, `mix`, `dither`, `transform`. `stretch` is dual-stage — a plan op that changes duration plus a process op that preserves pitch. Pitch *shifting* without a duration change is **not** a core built-in: it ships as the `PitchShifterPlugin` in `@suaveplan/audio-plugins-fx`. (The earlier draft's `registry/built-in/ pitch.ts` entry is dropped — there is no core `pitch` op.) ### 5.4. File layout ```text packages/audio/audio-core/ package.json, tsconfig.json, vitest.config.ts src/ index.ts error-codes.ts document/ document.ts .md .types.ts .test.ts index.ts page/ page.ts .md .types.ts .test.ts index.ts plan/ plan-engine.ts (+ .md .types.ts .test.ts) · segment.ts (+ .md .types.ts .test.ts) · index.ts process/ process-pipeline.ts (+ .md .types.ts .test.ts) · block-processor.ts (+ .md .types.ts .test.ts) · index.ts registry/ op-registry.ts (+ .md .types.ts .test.ts) · built-in/{trim,crop,speed,stretch,reverse,insert,remove,repeat,pad,resample,gain,normalize,fade,pan,mix,dither,transform}.ts · index.ts stream/ stream.ts .md .types.ts .test.ts index.ts render/ render.ts .md .test.ts index.ts stats/ stats.ts .md .types.ts .test.ts index.ts __benchmarks__/ plan.bench.ts stream.bench.ts process.bench.ts tests/ integration.test.ts property.test.ts purity.test.ts exports.test.ts ``` ### 5.5. Benchmarks - `plan.bench.ts` — compile 100 edits into a plan (1000 iterations). - `stream.bench.ts` — stream 5 minutes of 48 kHz stereo through 10 process ops. - `process.bench.ts` — run `normalize` + `fade` + `pan` over 1M samples. --- ## 6. Decisions and trade-offs - **One `Segment` shape, everywhere.** The richer record form (`from`/`count`/`to` + optional `rate`/`ref`/`interp`) is canonical because `ref` enables `insert`/`mix` from other documents and `rate`/`interp` enable `speed`/`resample` without a separate segment variant. Picking one shape now avoids translation layers between core and the timeline/engine packages. - **`audio-core` carries the ecosystem architecture.** Rather than a free-standing architecture spec under `openspec/specs/`, the cross-cutting model lives in this design doc because every abstraction it describes (`OpRegistry`, `BlockProcessor`, the six plugin slots, the error base) is *defined* by `audio-core`. Sibling proposals reference §1–§4 here. - **Pitch shift is a plugin, not a core op.** Core built-ins are limited to operations expressible as pure segment transforms or simple per-block DSP. Phase-vocoder pitch shifting is stateful and belongs in `audio-plugins-fx`. - **No design.md for the React/r3f tier yet.** `react-audio*` and `r3f-audio` are out of scope for this 13-change batch; they land in a later wave once the universal and platform tiers are green.