suaveplan/openspec/archive/2026-05-14-enterprise-audio-core/design.md

Design: enterprise-audio-core

Change ID: enterprise-audio-core Package: @suaveplan/audio-core

@suaveplan/audio-core is the foundation of the @suaveplan/audio-* family. This document carries both the cross-cutting architecture for the whole audio ecosystem (because audio-core defines the abstractions every sibling package builds on) and the package-specific design for the core itself. Sibling change proposals reference §1–§4 of this document for the shared model.

---

1. Ecosystem architecture

1.1. Goal

A cross-platform audio processing ecosystem that runs identically in browsers (Web Audio API), servers (Node/Bun), and offline render pipelines. The universal data model (pages, plans, edits) is separated from platform-specific rendering engines (browser AudioNodes, server block processors) so every package above the core is environment-agnostic.

The ecosystem is:

• Plugin-first — every capability, from a gain boost to beat detection to a full DAW timeline, is a plugin.
• Stream-native — audio flows through TransformStream pipelines using @suaveplan/streams.
• Non-destructive — edits are metadata; materialisation is deferred until read/stream/render.
• Feature-rich — capable of everything from format conversion to a working DAW with beat detection, auto-mixing, pitch/tempo adjustment, tempo/key detection, and chainable effects.

1.2. Three-layer separation

Layer	Responsibility	Environments
Data Model (audio-core)	PCM pages, edit plans, block processors, op registry	Universal
Timeline / Transport (audio-timeline, audio-transport)	Tracks, clips, musical time, scheduling, undo/redo	Universal
Rendering Engine (audio-engine-*)	Node graphs, speakers, file I/O, MIDI	Browser / Server / Offline

1.3. The six plugin slots

The core defines six plugin slots; every audio feature occupies one of them.

1. Plan Op — transforms segment lists (changes length / rate / channels).
2. Process Op — block-level DSP on Float32Array[].
3. Analysis Op — reads audio and emits features (BPM, key, spectrum); runs as a sidecar scan, not part of the render pipeline.
4. Codec — decodes / encodes a specific format; registered in a CodecRegistry.
5. Timeline Adapter — renders timeline state to a platform backend.
6. FX Plugin — dual-mode effect with both processBlock() (universal) and optional createAudioNode() (browser).

1.4. Zero duplication of genesis primitives

Audio packages reuse existing genesis packages rather than re-implementing them.

Capability	Genesis package	Usage in audio
Streams / backpressure	@suaveplan/streams	Audio TransformStream pipelines, progressive decode/encode
Scheduling / clocks	@suaveplan/timing	Transport lookahead scheduler, tempo clock
Event bus	@suaveplan/events	Document lifecycle, engine, transport, per-track events
OTel metrics / tracing	@suaveplan/events-otel, @suaveplan/opentelemetry	Plugin latency histograms, CPU counters, transport spans
Undo / redo	@suaveplan/undo	Timeline undo/redo with branching history and command merging
State machines	@suaveplan/fsm	Transport playback state, engine lifecycle, MIDI controller state
Serialization	@suaveplan/serialize	Session save/load (edit lists, transport state, plugin presets)
Binary packing	@suaveplan/bitpacker, @suaveplan/packer	Compressed sprite maps, custom binary formats, state compression
Metadata extraction	@suaveplan/audio-metadata	Format detection, ID3/MP4/FLAC tags during decode
Voice interfaces	@suaveplan/voice	TTS/STT integration points in the timeline
Error substrate	@suaveplan/error	AudioError base class, typed AUDIO.* codes
Schema registry	@suaveplan/schema-registry	Plugin preset Zod schemas, descriptor registration
Validation	@suaveplan/validation	Runtime boundary validation where Zod is unavailable
Testing	@suaveplan/testing	FakeClock, shared runner primitives

1.5. No platform assumptions in universal packages

Universal packages (audio-core, audio-timeline, audio-transport, audio-analysis-*, audio-plugins-fx) import zero browser APIs (window, document, navigator, AudioContext) and zero Node-only APIs (fs, process) except via injected adapters, and pass all unit tests under bun test (server) and vitest with a happy-dom shim (browser).

1.6. Architecture diagram

┌─────────────────────────────────────────────────────────────────────┐
│                         APPLICATION LAYER                            │
│  @suaveplan/react-audio*  |  @suaveplan/r3f-audio  |  audio-cli       │
└─────────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────────────┐
│                         TIMELINE / TRANSPORT                         │
│  @suaveplan/audio-timeline  ──TimelineAdapter──► rendering engine     │
│  @suaveplan/audio-transport ──TransportScheduler──► lookahead loop    │
└─────────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────────────┐
│                         ANALYSIS + FX PLUGINS                        │
│  @suaveplan/audio-analysis-*  |  @suaveplan/audio-plugins-fx          │
│  (BlockProcessor + AnalysisOp interfaces)                            │
└─────────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────────────────────────────────────────────┐
│                         UNIVERSAL CORE — @suaveplan/audio-core       │
│  AudioDocument (pages, edits, plans) · PlanEngine (segment compile)  │
│  ProcessPipeline (block DSP) · OpRegistry (plan + process ops)       │
└─────────────────────────────────────────────────────────────────────┘
                              │
        ┌─────────────────────┼─────────────────────┐
        ▼                     ▼                     ▼
  BROWSER ENGINE        SERVER I/O ONLY        OFFLINE RENDER
  AudioContext /        fs read/write          audio-core
  AudioWorklet /        file streams           .renderPlan()
  AudioNodes

---

2. Core abstractions

2.1. AudioDocument

The universal audio container — replaces the browser-only AudioBuffer. PCM is stored as paged Float32Array[] data so the model maps cleanly to both browser memory limits and server stream chunks. length, duration, and the stats are virtual: they read cached values invalidated by version bumps, so editing is O(1) (push to edits[]) and materialisation is deferred to read() or stream().

2.2. Segment — the canonical definition

A CompiledPlan is a list of Segment records describing how output samples map back to source samples. There is exactly one Segment shape across the whole audio family:

interface Segment {
  from: number                 // source sample start
  count: number                // source sample count
  to: number                   // output sample start
  rate?: number                // playback rate (1 = normal); affects resampling
  ref?: AudioDocument | null    // external source reference (null = silence segment)
  interp?: Interpolator         // custom resampler for non-unity rate
}

Plan ops receive and return Segment[]; process ops run per-block over the rendered output of the segment list. The earlier draft variants ({ srcStart, srcEnd, dstOffset } and the positional tuple [from, count, to]) are superseded by this record form — they are not used anywhere.

2.3. PlanEngine

Compiles AudioEdit[] → CompiledPlan in two passes:

1. Resolve pass — expand high-level edits into primitive edits using source stats (op.resolve).
2. Plan pass — transform the Segment[] for each edit (op.plan); ops without a plan function are appended to pipeline as process-stage ops.

function compilePlan(doc: AudioDocument, final: boolean): CompiledPlan {
  let segs: Segment[] = [{ from: 0, count: doc.sourceLength, to: 0 }]
  let pipeline: AudioEdit[] = []
  let limit = doc.sourceLength
  let sr = doc.sampleRate
  let ch = doc.channels

  for (const edit of doc.edits) {
    const op = registry.get(edit.type)
    if (!op) throw new AudioError('AUDIO_PLUGIN_NOT_FOUND', { context: { type: edit.type } })

    if (op.resolve) {
      const resolved = op.resolve({ stats: doc.stats, sampleRate: sr, channels: ch, opts: edit.opts })
      if (resolved === false) continue
      // resolved edits are spliced in and recursed over
    }

    if (op.plan) {
      segs = op.plan(segs, mkPlanCtx(planLen(segs), sr, edit.opts))
      if (!final) limit = adjustLimit(limit, edit.type)
    } else {
      pipeline.push(edit)
    }

    if (op.sr) sr = op.sr(sr, edit.opts) ?? sr
    if (op.ch) ch = op.ch(ch, edit.opts) ?? ch
  }

  return { segs, pipeline, totalLen: planLen(segs), sampleRate: sr, limit: Math.max(0, limit) }
}

2.4. ProcessPipeline

Runs process ops per BLOCK_SIZE (1024 samples) using double-buffer rotation. init allocates ProcessState[]; patch hot-swaps parameter values without re-allocation; process runs each op over a block. On a structural change (op added/removed, detected via a pipeline signature) the pipeline reinitialises affected processors and crossfades over ~20 ms to avoid clicks.

2.5. BlockProcessor — the universal DSP interface

Every DSP plugin (FX, analysis, codec post-processing) implements:

interface BlockProcessor {
  readonly name: string
  setup(config: ProcessorConfig): void
  process(input: Float32Array[], output: Float32Array[], ctx: BlockContext): void
  reset(): void
  dispose(): void
}

This is the interface that works everywhere: browser worklets, server batch pipelines, offline renders, and WebAssembly modules.

2.6. Error isolation

A crashing process op must not corrupt the stream. ProcessPipeline.process wraps each op:

try {
  op.process(input, output, ctx)
} catch (e) {
  emit(doc, 'plugin:error', { plugin: op.name, error: e })
  for (let c = 0; c < input.length; c++) output[c].set(input[c])  // pass-through
}

---

3. Integration with genesis packages

• @suaveplan/streams — AudioDocument.stream() and PlanEngine.streamPlan() yield blocks composable with TransformStream; a helper wraps a compiled plan + process pipeline as a TransformStream<Float32Array[], Float32Array[]> so downstream FX get native backpressure. Progressive decode (in audio-codecs) returns a ReadableStream<AudioPage>; the document is editable before decode completes because edits are metadata.
• @suaveplan/events — backs the typed AudioDocumentEventMap (edit, undo, plugin:error, page:loaded). The interceptor API is the attachment point for OTel metrics, schema validation, and cross-context bridges in the engine packages.
• @suaveplan/serialize — AudioDocument.toJSON() round-trips through serialize/ deserialize, preserving Float32Array page data, edits, version, source, metadata.
• @suaveplan/audio-metadata — supplies the AudioMetadata type carried on AudioDocument.metadata. audio-core depends on @suaveplan/audio-metadata for this type only; population happens in audio-codecs during decode.

---

4. Error hierarchy

Every audio package extends @suaveplan/error. audio-core registers the foundation codes:

defineCodes('AUDIO_CORE', [
  { code: 'AUDIO_PLUGIN_NOT_FOUND',  severity: 'error',   retryable: false },
  { code: 'AUDIO_INVALID_EDIT',      severity: 'error',   retryable: false },
  { code: 'AUDIO_STREAM_ABORTED',    severity: 'warning', retryable: true  },
  { code: 'AUDIO_PAGE_OUT_OF_RANGE', severity: 'error',   retryable: false },
  { code: 'AUDIO_PROCESS_ERROR',     severity: 'error',   retryable: true  },
])

Sibling packages register their own AUDIO_<DOMAIN> namespaces (engine, codec, analysis, etc.).

---

5. Package-specific design

5.1. Paged memory model

const PAGE_SIZE = 1024 * 1024  // samples (~23 MB stereo @ 48 kHz)
const BLOCK_SIZE = 1024        // samples per process block

AudioDocument owns an Array<AudioPage | null> where null marks an evicted page that can be reloaded from cache. Pages are allocated from a pool to reduce GC pressure. Primitives:

• walkPages(doc, channel, startSample, count, visitor) — iterate overlapping pages.
• copyPages(doc, channel, startSample, count, target, targetOffset) — copy into a pre-allocated buffer without allocating.
• readPages(doc, offset?, duration?) — read a range as Float32Array[].

5.2. Streaming

AudioDocument.stream() is an async generator that compiles a non-final plan, initialises the process pipeline, and yields Float32Array[] blocks of at most BLOCK_SIZE samples:

async function* stream(opts?: StreamOptions) {
  const plan = planEngine.compile(doc, false)
  const procs = processPipeline.init(plan.pipeline, plan.totalLen, plan.sampleRate, doc.channels)
  let outPos = opts?.startSample ?? 0
  const end = opts?.endSample ?? plan.totalLen
  while (outPos < end) {
    if (opts?.signal?.aborted) throw new AudioError('AUDIO_STREAM_ABORTED')
    const blockLen = Math.min(BLOCK_SIZE, end - outPos)
    renderBlock(doc, plan.segs, outPos, blockLen, bufA)
    const out = procs.length
      ? processPipeline.process(bufA, bufB, procs, outPos, plan.sampleRate)
      : bufA
    yield out.map((b) => b.subarray(0, blockLen))
    outPos += blockLen
  }
}

5.3. Built-in operations

Plan ops: trim, crop, speed, stretch, reverse, insert, remove, repeat, pad, resample. Process ops: gain, normalize, fade, pan, mix, dither, transform.

stretch is dual-stage — a plan op that changes duration plus a process op that preserves pitch. Pitch shifting without a duration change is not a core built-in: it ships as the PitchShifterPlugin in @suaveplan/audio-plugins-fx. (The earlier draft's registry/built-in/ pitch.ts entry is dropped — there is no core pitch op.)

5.4. File layout

packages/audio/audio-core/
  package.json, tsconfig.json, vitest.config.ts
  src/
    index.ts
    error-codes.ts
    document/    document.ts .md .types.ts .test.ts index.ts
    page/        page.ts .md .types.ts .test.ts index.ts
    plan/        plan-engine.ts (+ .md .types.ts .test.ts) · segment.ts (+ .md .types.ts .test.ts) · index.ts
    process/     process-pipeline.ts (+ .md .types.ts .test.ts) · block-processor.ts (+ .md .types.ts .test.ts) · index.ts
    registry/    op-registry.ts (+ .md .types.ts .test.ts) · built-in/{trim,crop,speed,stretch,reverse,insert,remove,repeat,pad,resample,gain,normalize,fade,pan,mix,dither,transform}.ts · index.ts
    stream/      stream.ts .md .types.ts .test.ts index.ts
    render/      render.ts .md .test.ts index.ts
    stats/       stats.ts .md .types.ts .test.ts index.ts
    __benchmarks__/  plan.bench.ts stream.bench.ts process.bench.ts
  tests/
    integration.test.ts  property.test.ts  purity.test.ts  exports.test.ts

5.5. Benchmarks

• plan.bench.ts — compile 100 edits into a plan (1000 iterations).
• stream.bench.ts — stream 5 minutes of 48 kHz stereo through 10 process ops.
• process.bench.ts — run normalize + fade + pan over 1M samples.

---

6. Decisions and trade-offs

• One Segment shape, everywhere. The richer record form (from/count/to + optional rate/ref/interp) is canonical because ref enables insert/mix from other documents and rate/interp enable speed/resample without a separate segment variant. Picking one shape now avoids translation layers between core and the timeline/engine packages.
• audio-core carries the ecosystem architecture. Rather than a free-standing architecture spec under openspec/specs/, the cross-cutting model lives in this design doc because every abstraction it describes (OpRegistry, BlockProcessor, the six plugin slots, the error base) is defined by audio-core. Sibling proposals reference §1–§4 here.
• Pitch shift is a plugin, not a core op. Core built-ins are limited to operations expressible as pure segment transforms or simple per-block DSP. Phase-vocoder pitch shifting is stateful and belongs in audio-plugins-fx.
• No design.md for the React/r3f tier yet. react-audio* and r3f-audio are out of scope for this 13-change batch; they land in a later wave once the universal and platform tiers are green.