358 lines
18 KiB
Markdown
358 lines
18 KiB
Markdown
# Design: enterprise-audio-core
|
||
|
||
**Change ID**: enterprise-audio-core
|
||
**Package**: `@suaveplan/audio-core`
|
||
|
||
`@suaveplan/audio-core` is the foundation of the `@suaveplan/audio-*` family. This document carries
|
||
both the **cross-cutting architecture** for the whole audio ecosystem (because `audio-core` defines
|
||
the abstractions every sibling package builds on) and the **package-specific design** for the core
|
||
itself. Sibling change proposals reference §1–§4 of this document for the shared model.
|
||
|
||
---
|
||
|
||
## 1. Ecosystem architecture
|
||
|
||
### 1.1. Goal
|
||
|
||
A cross-platform audio processing ecosystem that runs identically in browsers (Web Audio API),
|
||
servers (Node/Bun), and offline render pipelines. The **universal data model** (pages, plans, edits)
|
||
is separated from **platform-specific rendering engines** (browser AudioNodes, server block
|
||
processors) so every package above the core is environment-agnostic.
|
||
|
||
The ecosystem is:
|
||
|
||
- **Plugin-first** — every capability, from a gain boost to beat detection to a full DAW timeline,
|
||
is a plugin.
|
||
- **Stream-native** — audio flows through `TransformStream` pipelines using `@suaveplan/streams`.
|
||
- **Non-destructive** — edits are metadata; materialisation is deferred until read/stream/render.
|
||
- **Feature-rich** — capable of everything from format conversion to a working DAW with beat
|
||
detection, auto-mixing, pitch/tempo adjustment, tempo/key detection, and chainable effects.
|
||
|
||
### 1.2. Three-layer separation
|
||
|
||
| Layer | Responsibility | Environments |
|
||
|---|---|---|
|
||
| **Data Model** (`audio-core`) | PCM pages, edit plans, block processors, op registry | Universal |
|
||
| **Timeline / Transport** (`audio-timeline`, `audio-transport`) | Tracks, clips, musical time, scheduling, undo/redo | Universal |
|
||
| **Rendering Engine** (`audio-engine-*`) | Node graphs, speakers, file I/O, MIDI | Browser / Server / Offline |
|
||
|
||
### 1.3. The six plugin slots
|
||
|
||
The core defines six plugin slots; every audio feature occupies one of them.
|
||
|
||
1. **Plan Op** — transforms segment lists (changes length / rate / channels).
|
||
2. **Process Op** — block-level DSP on `Float32Array[]`.
|
||
3. **Analysis Op** — reads audio and emits features (BPM, key, spectrum); runs as a sidecar scan,
|
||
not part of the render pipeline.
|
||
4. **Codec** — decodes / encodes a specific format; registered in a `CodecRegistry`.
|
||
5. **Timeline Adapter** — renders timeline state to a platform backend.
|
||
6. **FX Plugin** — dual-mode effect with both `processBlock()` (universal) and optional
|
||
`createAudioNode()` (browser).
|
||
|
||
### 1.4. Zero duplication of genesis primitives
|
||
|
||
Audio packages reuse existing genesis packages rather than re-implementing them.
|
||
|
||
| Capability | Genesis package | Usage in audio |
|
||
|---|---|---|
|
||
| Streams / backpressure | `@suaveplan/streams` | Audio `TransformStream` pipelines, progressive decode/encode |
|
||
| Scheduling / clocks | `@suaveplan/timing` | Transport lookahead scheduler, tempo clock |
|
||
| Event bus | `@suaveplan/events` | Document lifecycle, engine, transport, per-track events |
|
||
| OTel metrics / tracing | `@suaveplan/events-otel`, `@suaveplan/opentelemetry` | Plugin latency histograms, CPU counters, transport spans |
|
||
| Undo / redo | `@suaveplan/undo` | Timeline undo/redo with branching history and command merging |
|
||
| State machines | `@suaveplan/fsm` | Transport playback state, engine lifecycle, MIDI controller state |
|
||
| Serialization | `@suaveplan/serialize` | Session save/load (edit lists, transport state, plugin presets) |
|
||
| Binary packing | `@suaveplan/bitpacker`, `@suaveplan/packer` | Compressed sprite maps, custom binary formats, state compression |
|
||
| Metadata extraction | `@suaveplan/audio-metadata` | Format detection, ID3/MP4/FLAC tags during decode |
|
||
| Voice interfaces | `@suaveplan/voice` | TTS/STT integration points in the timeline |
|
||
| Error substrate | `@suaveplan/error` | `AudioError` base class, typed `AUDIO.*` codes |
|
||
| Schema registry | `@suaveplan/schema-registry` | Plugin preset Zod schemas, descriptor registration |
|
||
| Validation | `@suaveplan/validation` | Runtime boundary validation where Zod is unavailable |
|
||
| Testing | `@suaveplan/testing` | `FakeClock`, shared runner primitives |
|
||
|
||
### 1.5. No platform assumptions in universal packages
|
||
|
||
Universal packages (`audio-core`, `audio-timeline`, `audio-transport`, `audio-analysis-*`,
|
||
`audio-plugins-fx`) import zero browser APIs (`window`, `document`, `navigator`, `AudioContext`) and
|
||
zero Node-only APIs (`fs`, `process`) except via injected adapters, and pass all unit tests under
|
||
`bun test` (server) and vitest with a `happy-dom` shim (browser).
|
||
|
||
### 1.6. Architecture diagram
|
||
|
||
```text
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ APPLICATION LAYER │
|
||
│ @suaveplan/react-audio* | @suaveplan/r3f-audio | audio-cli │
|
||
└─────────────────────────────────────────────────────────────────────┘
|
||
│
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ TIMELINE / TRANSPORT │
|
||
│ @suaveplan/audio-timeline ──TimelineAdapter──► rendering engine │
|
||
│ @suaveplan/audio-transport ──TransportScheduler──► lookahead loop │
|
||
└─────────────────────────────────────────────────────────────────────┘
|
||
│
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ ANALYSIS + FX PLUGINS │
|
||
│ @suaveplan/audio-analysis-* | @suaveplan/audio-plugins-fx │
|
||
│ (BlockProcessor + AnalysisOp interfaces) │
|
||
└─────────────────────────────────────────────────────────────────────┘
|
||
│
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ UNIVERSAL CORE — @suaveplan/audio-core │
|
||
│ AudioDocument (pages, edits, plans) · PlanEngine (segment compile) │
|
||
│ ProcessPipeline (block DSP) · OpRegistry (plan + process ops) │
|
||
└─────────────────────────────────────────────────────────────────────┘
|
||
│
|
||
┌─────────────────────┼─────────────────────┐
|
||
▼ ▼ ▼
|
||
BROWSER ENGINE SERVER I/O ONLY OFFLINE RENDER
|
||
AudioContext / fs read/write audio-core
|
||
AudioWorklet / file streams .renderPlan()
|
||
AudioNodes
|
||
```
|
||
|
||
---
|
||
|
||
## 2. Core abstractions
|
||
|
||
### 2.1. `AudioDocument`
|
||
|
||
The universal audio container — replaces the browser-only `AudioBuffer`. PCM is stored as paged
|
||
`Float32Array[]` data so the model maps cleanly to both browser memory limits and server stream
|
||
chunks. `length`, `duration`, and the stats are virtual: they read cached values invalidated by
|
||
`version` bumps, so editing is O(1) (push to `edits[]`) and materialisation is deferred to `read()`
|
||
or `stream()`.
|
||
|
||
### 2.2. `Segment` — the canonical definition
|
||
|
||
A `CompiledPlan` is a list of `Segment` records describing how output samples map back to source
|
||
samples. **There is exactly one `Segment` shape across the whole audio family:**
|
||
|
||
```ts
|
||
interface Segment {
|
||
from: number // source sample start
|
||
count: number // source sample count
|
||
to: number // output sample start
|
||
rate?: number // playback rate (1 = normal); affects resampling
|
||
ref?: AudioDocument | null // external source reference (null = silence segment)
|
||
interp?: Interpolator // custom resampler for non-unity rate
|
||
}
|
||
```
|
||
|
||
Plan ops receive and return `Segment[]`; process ops run per-block over the rendered output of the
|
||
segment list. The earlier draft variants (`{ srcStart, srcEnd, dstOffset }` and the positional
|
||
tuple `[from, count, to]`) are superseded by this record form — they are not used anywhere.
|
||
|
||
### 2.3. `PlanEngine`
|
||
|
||
Compiles `AudioEdit[]` → `CompiledPlan` in two passes:
|
||
|
||
1. **Resolve pass** — expand high-level edits into primitive edits using source stats
|
||
(`op.resolve`).
|
||
2. **Plan pass** — transform the `Segment[]` for each edit (`op.plan`); ops without a `plan`
|
||
function are appended to `pipeline` as process-stage ops.
|
||
|
||
```ts
|
||
function compilePlan(doc: AudioDocument, final: boolean): CompiledPlan {
|
||
let segs: Segment[] = [{ from: 0, count: doc.sourceLength, to: 0 }]
|
||
let pipeline: AudioEdit[] = []
|
||
let limit = doc.sourceLength
|
||
let sr = doc.sampleRate
|
||
let ch = doc.channels
|
||
|
||
for (const edit of doc.edits) {
|
||
const op = registry.get(edit.type)
|
||
if (!op) throw new AudioError('AUDIO_PLUGIN_NOT_FOUND', { context: { type: edit.type } })
|
||
|
||
if (op.resolve) {
|
||
const resolved = op.resolve({ stats: doc.stats, sampleRate: sr, channels: ch, opts: edit.opts })
|
||
if (resolved === false) continue
|
||
// resolved edits are spliced in and recursed over
|
||
}
|
||
|
||
if (op.plan) {
|
||
segs = op.plan(segs, mkPlanCtx(planLen(segs), sr, edit.opts))
|
||
if (!final) limit = adjustLimit(limit, edit.type)
|
||
} else {
|
||
pipeline.push(edit)
|
||
}
|
||
|
||
if (op.sr) sr = op.sr(sr, edit.opts) ?? sr
|
||
if (op.ch) ch = op.ch(ch, edit.opts) ?? ch
|
||
}
|
||
|
||
return { segs, pipeline, totalLen: planLen(segs), sampleRate: sr, limit: Math.max(0, limit) }
|
||
}
|
||
```
|
||
|
||
### 2.4. `ProcessPipeline`
|
||
|
||
Runs process ops per `BLOCK_SIZE` (1024 samples) using double-buffer rotation. `init` allocates
|
||
`ProcessState[]`; `patch` hot-swaps parameter values without re-allocation; `process` runs each op
|
||
over a block. On a **structural** change (op added/removed, detected via a pipeline signature) the
|
||
pipeline reinitialises affected processors and crossfades over ~20 ms to avoid clicks.
|
||
|
||
### 2.5. `BlockProcessor` — the universal DSP interface
|
||
|
||
Every DSP plugin (FX, analysis, codec post-processing) implements:
|
||
|
||
```ts
|
||
interface BlockProcessor {
|
||
readonly name: string
|
||
setup(config: ProcessorConfig): void
|
||
process(input: Float32Array[], output: Float32Array[], ctx: BlockContext): void
|
||
reset(): void
|
||
dispose(): void
|
||
}
|
||
```
|
||
|
||
This is the interface that works everywhere: browser worklets, server batch pipelines, offline
|
||
renders, and WebAssembly modules.
|
||
|
||
### 2.6. Error isolation
|
||
|
||
A crashing process op must not corrupt the stream. `ProcessPipeline.process` wraps each op:
|
||
|
||
```ts
|
||
try {
|
||
op.process(input, output, ctx)
|
||
} catch (e) {
|
||
emit(doc, 'plugin:error', { plugin: op.name, error: e })
|
||
for (let c = 0; c < input.length; c++) output[c].set(input[c]) // pass-through
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 3. Integration with genesis packages
|
||
|
||
- **`@suaveplan/streams`** — `AudioDocument.stream()` and `PlanEngine.streamPlan()` yield blocks
|
||
composable with `TransformStream`; a helper wraps a compiled plan + process pipeline as a
|
||
`TransformStream<Float32Array[], Float32Array[]>` so downstream FX get native backpressure.
|
||
Progressive decode (in `audio-codecs`) returns a `ReadableStream<AudioPage>`; the document is
|
||
editable before decode completes because edits are metadata.
|
||
- **`@suaveplan/events`** — backs the typed `AudioDocumentEventMap` (`edit`, `undo`,
|
||
`plugin:error`, `page:loaded`). The interceptor API is the attachment point for OTel metrics,
|
||
schema validation, and cross-context bridges in the engine packages.
|
||
- **`@suaveplan/serialize`** — `AudioDocument.toJSON()` round-trips through `serialize`/
|
||
`deserialize`, preserving `Float32Array` page data, `edits`, `version`, `source`, `metadata`.
|
||
- **`@suaveplan/audio-metadata`** — supplies the `AudioMetadata` type carried on
|
||
`AudioDocument.metadata`. `audio-core` depends on `@suaveplan/audio-metadata` for this type only;
|
||
population happens in `audio-codecs` during decode.
|
||
|
||
---
|
||
|
||
## 4. Error hierarchy
|
||
|
||
Every audio package extends `@suaveplan/error`. `audio-core` registers the foundation codes:
|
||
|
||
```ts
|
||
defineCodes('AUDIO_CORE', [
|
||
{ code: 'AUDIO_PLUGIN_NOT_FOUND', severity: 'error', retryable: false },
|
||
{ code: 'AUDIO_INVALID_EDIT', severity: 'error', retryable: false },
|
||
{ code: 'AUDIO_STREAM_ABORTED', severity: 'warning', retryable: true },
|
||
{ code: 'AUDIO_PAGE_OUT_OF_RANGE', severity: 'error', retryable: false },
|
||
{ code: 'AUDIO_PROCESS_ERROR', severity: 'error', retryable: true },
|
||
])
|
||
```
|
||
|
||
Sibling packages register their own `AUDIO_<DOMAIN>` namespaces (engine, codec, analysis, etc.).
|
||
|
||
---
|
||
|
||
## 5. Package-specific design
|
||
|
||
### 5.1. Paged memory model
|
||
|
||
```ts
|
||
const PAGE_SIZE = 1024 * 1024 // samples (~23 MB stereo @ 48 kHz)
|
||
const BLOCK_SIZE = 1024 // samples per process block
|
||
```
|
||
|
||
`AudioDocument` owns an `Array<AudioPage | null>` where `null` marks an evicted page that can be
|
||
reloaded from cache. Pages are allocated from a pool to reduce GC pressure. Primitives:
|
||
|
||
- `walkPages(doc, channel, startSample, count, visitor)` — iterate overlapping pages.
|
||
- `copyPages(doc, channel, startSample, count, target, targetOffset)` — copy into a pre-allocated
|
||
buffer without allocating.
|
||
- `readPages(doc, offset?, duration?)` — read a range as `Float32Array[]`.
|
||
|
||
### 5.2. Streaming
|
||
|
||
`AudioDocument.stream()` is an async generator that compiles a non-final plan, initialises the
|
||
process pipeline, and yields `Float32Array[]` blocks of at most `BLOCK_SIZE` samples:
|
||
|
||
```ts
|
||
async function* stream(opts?: StreamOptions) {
|
||
const plan = planEngine.compile(doc, false)
|
||
const procs = processPipeline.init(plan.pipeline, plan.totalLen, plan.sampleRate, doc.channels)
|
||
let outPos = opts?.startSample ?? 0
|
||
const end = opts?.endSample ?? plan.totalLen
|
||
while (outPos < end) {
|
||
if (opts?.signal?.aborted) throw new AudioError('AUDIO_STREAM_ABORTED')
|
||
const blockLen = Math.min(BLOCK_SIZE, end - outPos)
|
||
renderBlock(doc, plan.segs, outPos, blockLen, bufA)
|
||
const out = procs.length
|
||
? processPipeline.process(bufA, bufB, procs, outPos, plan.sampleRate)
|
||
: bufA
|
||
yield out.map((b) => b.subarray(0, blockLen))
|
||
outPos += blockLen
|
||
}
|
||
}
|
||
```
|
||
|
||
### 5.3. Built-in operations
|
||
|
||
Plan ops: `trim`, `crop`, `speed`, `stretch`, `reverse`, `insert`, `remove`, `repeat`, `pad`,
|
||
`resample`. Process ops: `gain`, `normalize`, `fade`, `pan`, `mix`, `dither`, `transform`.
|
||
|
||
`stretch` is dual-stage — a plan op that changes duration plus a process op that preserves pitch.
|
||
Pitch *shifting* without a duration change is **not** a core built-in: it ships as the
|
||
`PitchShifterPlugin` in `@suaveplan/audio-plugins-fx`. (The earlier draft's `registry/built-in/
|
||
pitch.ts` entry is dropped — there is no core `pitch` op.)
|
||
|
||
### 5.4. File layout
|
||
|
||
```text
|
||
packages/audio/audio-core/
|
||
package.json, tsconfig.json, vitest.config.ts
|
||
src/
|
||
index.ts
|
||
error-codes.ts
|
||
document/ document.ts .md .types.ts .test.ts index.ts
|
||
page/ page.ts .md .types.ts .test.ts index.ts
|
||
plan/ plan-engine.ts (+ .md .types.ts .test.ts) · segment.ts (+ .md .types.ts .test.ts) · index.ts
|
||
process/ process-pipeline.ts (+ .md .types.ts .test.ts) · block-processor.ts (+ .md .types.ts .test.ts) · index.ts
|
||
registry/ op-registry.ts (+ .md .types.ts .test.ts) · built-in/{trim,crop,speed,stretch,reverse,insert,remove,repeat,pad,resample,gain,normalize,fade,pan,mix,dither,transform}.ts · index.ts
|
||
stream/ stream.ts .md .types.ts .test.ts index.ts
|
||
render/ render.ts .md .test.ts index.ts
|
||
stats/ stats.ts .md .types.ts .test.ts index.ts
|
||
__benchmarks__/ plan.bench.ts stream.bench.ts process.bench.ts
|
||
tests/
|
||
integration.test.ts property.test.ts purity.test.ts exports.test.ts
|
||
```
|
||
|
||
### 5.5. Benchmarks
|
||
|
||
- `plan.bench.ts` — compile 100 edits into a plan (1000 iterations).
|
||
- `stream.bench.ts` — stream 5 minutes of 48 kHz stereo through 10 process ops.
|
||
- `process.bench.ts` — run `normalize` + `fade` + `pan` over 1M samples.
|
||
|
||
---
|
||
|
||
## 6. Decisions and trade-offs
|
||
|
||
- **One `Segment` shape, everywhere.** The richer record form (`from`/`count`/`to` + optional
|
||
`rate`/`ref`/`interp`) is canonical because `ref` enables `insert`/`mix` from other documents and
|
||
`rate`/`interp` enable `speed`/`resample` without a separate segment variant. Picking one shape
|
||
now avoids translation layers between core and the timeline/engine packages.
|
||
- **`audio-core` carries the ecosystem architecture.** Rather than a free-standing architecture
|
||
spec under `openspec/specs/`, the cross-cutting model lives in this design doc because every
|
||
abstraction it describes (`OpRegistry`, `BlockProcessor`, the six plugin slots, the error base) is
|
||
*defined* by `audio-core`. Sibling proposals reference §1–§4 here.
|
||
- **Pitch shift is a plugin, not a core op.** Core built-ins are limited to operations expressible
|
||
as pure segment transforms or simple per-block DSP. Phase-vocoder pitch shifting is stateful and
|
||
belongs in `audio-plugins-fx`.
|
||
- **No design.md for the React/r3f tier yet.** `react-audio*` and `r3f-audio` are out of scope for
|
||
this 13-change batch; they land in a later wave once the universal and platform tiers are green.
|