262 lines
13 KiB
Markdown
262 lines
13 KiB
Markdown
# Design: enterprise-audio-engine-browser
|
||
|
||
**Change ID**: enterprise-audio-engine-browser
|
||
**Package**: `@suaveplan/audio-engine-browser`
|
||
|
||
`@suaveplan/audio-engine-browser` is the **browser rendering engine** of the audio family's
|
||
three-layer model — the layer that turns universal timeline state into actual sound through the
|
||
Web Audio API. It is the only browser-only package in the audio core: everything below it
|
||
(`audio-core`, `audio-timeline`, `audio-transport`, `audio-plugins-fx`) is universal, and everything
|
||
above it (`react-audio*`, `r3f-audio`) is application-tier. This document carries the
|
||
**package-specific design**; the **cross-cutting architecture** for the whole `@suaveplan/audio-*`
|
||
ecosystem lives in `enterprise-audio-core/design.md` §1–§4 (the three-layer separation, the six
|
||
plugin slots, the genesis-package reuse table, the universal `Segment` shape, and the error
|
||
hierarchy) and is referenced here, not duplicated.
|
||
|
||
---
|
||
|
||
## 1. Position in the three-layer model
|
||
|
||
Per `enterprise-audio-core/design.md` §1.2, the audio ecosystem separates the **data model**
|
||
(`audio-core`) from the **timeline/transport** layer (`audio-timeline`, `audio-transport`) from the
|
||
**rendering engines** (`audio-engine-*`). This package is one of those rendering engines — the
|
||
browser one. Its sibling `audio-engine-server` renders the same timeline state to Node/Bun file I/O
|
||
and speakers; both implement the same `TimelineAdapter` interface (plugin slot 5) so a
|
||
`TimelineEngine` is engine-agnostic.
|
||
|
||
The design consequence: **this package contains no timeline logic.** Track ordering, clip
|
||
placement, musical-time conversion, undo/redo, and tempo maps all live in `@suaveplan/audio-timeline`.
|
||
`BrowserTimelineAdapter` is a pure translation layer — it receives already-resolved `Track[]` and
|
||
playback commands and maps them onto Web Audio `AudioNode` graphs. This is what makes the legacy
|
||
"browser-only" audio openspecs obsolete: they tangled timeline state into the rendering code; the
|
||
adapter-based architecture keeps them apart.
|
||
|
||
---
|
||
|
||
## 2. Why an injectable context seam
|
||
|
||
Every browser audio API the package touches — `AudioContext`, `OfflineAudioContext`,
|
||
`AudioWorklet`, `AudioNode` — is reached through one of two seams:
|
||
|
||
1. The `context` option on `BrowserAdapterOptions` / the `WebAudioGraph` and `AudioWorkletManager`
|
||
constructors. When supplied, the package uses it verbatim and never reads `globalThis`.
|
||
2. `standardized-audio-context` when no context is injected — a cross-browser shim that smooths
|
||
over Safari/Firefox/Chrome `AudioContext` differences and, critically, provides a real
|
||
`OfflineAudioContext` implementation usable under `happy-dom`.
|
||
|
||
This is the same testability discipline `audio-core` applies to its adapters (`enterprise-audio-core`
|
||
"Universal-tier purity" requirement), adapted for the browser tier. The package **may** name browser
|
||
APIs — that is its job — but it must be drivable in CI without a real browser. Every test injects an
|
||
`OfflineAudioContext`; `globalThis.AudioContext` is never read in the test path.
|
||
|
||
**Tier purity:** the package must not import Node-only APIs (`node:fs`, `node:path`, `process`,
|
||
`Buffer`). There is no filesystem in this layer — file I/O belongs to `audio-engine-server`. The
|
||
purity test greps the built `dist/` for those identifiers and asserts zero hits.
|
||
|
||
---
|
||
|
||
## 3. WebAudioGraph build strategy
|
||
|
||
`WebAudioGraph` is a thin, id-keyed registry over a `BaseAudioContext`. It exists so the adapter can
|
||
address nodes by stable string id (e.g. `track:42:gain`) rather than juggling live `AudioNode`
|
||
references, and so a graph can be **snapshotted** to a plain serializable object and **restored**.
|
||
|
||
```ts
|
||
class WebAudioGraph {
|
||
private readonly nodes = new Map<string, AudioNode>()
|
||
private readonly edges: Connection[] = []
|
||
|
||
connect(sourceId: string, destinationId: string, outputIndex = 0, inputIndex = 0): void {
|
||
const source = this.nodes.get(sourceId)
|
||
const destination = this.nodes.get(destinationId)
|
||
if (!source) throw new BrowserAudioError('AUDIO_ENGINE_BROWSER.NODE_NOT_FOUND', { context: { id: sourceId } })
|
||
if (!destination) throw new BrowserAudioError('AUDIO_ENGINE_BROWSER.NODE_NOT_FOUND', { context: { id: destinationId } })
|
||
source.connect(destination as AudioNode, outputIndex, inputIndex)
|
||
this.edges.push({ source: sourceId, destination: destinationId, outputIndex, inputIndex })
|
||
}
|
||
|
||
snapshot(): AudioGraphSnapshot {
|
||
const nodes: Record<string, AudioNodeType> = {}
|
||
for (const [id, node] of this.nodes) nodes[id] = classifyNode(node)
|
||
return { nodes, connections: this.edges.map((e) => ({ ...e })) }
|
||
}
|
||
}
|
||
```
|
||
|
||
`snapshot()` records **node types**, not node instances — an `AudioGraphSnapshot` is JSON-safe so it
|
||
can be persisted in a session file or diffed for debugging. `restore()` rebuilds typed nodes against
|
||
the graph's context and replays the edge list. The snapshot is Zod-validated on `restore` so a
|
||
corrupt session file fails fast at the boundary rather than mid-render.
|
||
|
||
---
|
||
|
||
## 4. Per-track subgraph
|
||
|
||
Each `Track` becomes a fixed-shape subgraph. The chain is deliberately the same for every track so
|
||
that mute/solo, volume, and pan map to predictable nodes:
|
||
|
||
```text
|
||
AudioBufferSourceNode (per clip)
|
||
→ GainNode (clip volume)
|
||
→ GainNode (track volume) ← setTrackVolume / mute / solo write here
|
||
→ StereoPannerNode (track pan) ← setTrackPan writes here
|
||
→ DynamicsCompressorNode (limiter)
|
||
→ GainNode (master mix) ← setMasterVolume writes here
|
||
→ AnalyserNode (master analysis)
|
||
→ context.destination
|
||
```
|
||
|
||
**Solo semantics.** DAW solo is a whole-mix operation, not a per-track flag. When *any* track is
|
||
soloed, every non-soloed track's track-volume `GainNode` is set to `0`; when the last solo is
|
||
cleared, each track's gain reverts to its mute-governed value. The adapter keeps a `Set<soloed>` and
|
||
recomputes effective gains on every `setTrackMute` / `setTrackSolo` call — this is the one piece of
|
||
real state the adapter owns, and it is small and easy to test against an `OfflineAudioContext`.
|
||
|
||
`removeTrack` / `updateTrack` / the per-track setters throw `AUDIO_ENGINE_BROWSER.TRACK_NOT_FOUND`
|
||
for an unknown id rather than silently no-op'ing — a missing track id is a caller bug, not a
|
||
recoverable condition.
|
||
|
||
---
|
||
|
||
## 5. FX plugin hosting — three paths
|
||
|
||
An `AudioFxPlugin` from `@suaveplan/audio-fx-plugin-base` is dual-mode by design (plugin slot 6): it
|
||
always exposes a universal `processBlock`, and *optionally* a browser-native `createAudioNode`. The
|
||
host picks the best available path:
|
||
|
||
```ts
|
||
function createFxNode(plugin: AudioFxPlugin, context: BaseAudioContext, worklets: AudioWorkletManager): AudioNode {
|
||
// 1. AudioNode path — native, lowest latency, runs on the audio thread
|
||
if (plugin.createAudioNode) return plugin.createAudioNode(context)
|
||
|
||
// 2. Worklet path — universal processBlock compiled into an AudioWorkletProcessor
|
||
if (context.audioWorklet && plugin.getWorkletProcessor) {
|
||
worklets.addModule(plugin.getWorkletProcessor())
|
||
return worklets.createNode(plugin.name)
|
||
}
|
||
|
||
// 3. Fallback path — ScriptProcessorNode on the main thread (deprecated, but universal)
|
||
const sp = context.createScriptProcessor(BLOCK_SIZE, 2, 2)
|
||
sp.onaudioprocess = (e) => {
|
||
const input = [e.inputBuffer.getChannelData(0), e.inputBuffer.getChannelData(1)]
|
||
const output = [e.outputBuffer.getChannelData(0), e.outputBuffer.getChannelData(1)]
|
||
plugin.processBlock(input, output, { sampleRate: context.sampleRate, blockSize: BLOCK_SIZE })
|
||
}
|
||
return sp
|
||
}
|
||
```
|
||
|
||
Path 1 is preferred because a native chain runs entirely on the audio rendering thread. Path 2 keeps
|
||
DSP off the main thread when the plugin only ships universal block code. Path 3 exists so *every*
|
||
FX plugin works *somewhere* — `ScriptProcessorNode` is deprecated but universally available, and the
|
||
fallback is honest about its main-thread cost. Whichever node is chosen, it is inserted into the
|
||
track subgraph through `WebAudioGraph` so it sits in the signal path between the clip gain and the
|
||
track gain.
|
||
|
||
---
|
||
|
||
## 6. Offline render
|
||
|
||
`renderOffline` is the bridge to non-realtime export. It wires a `BrowserTimelineAdapter` to an
|
||
`OfflineAudioContext` and lets the Web Audio engine render faster than realtime:
|
||
|
||
```ts
|
||
async function renderOffline(timeline: TimelineEngine, duration: number, sampleRate: number): Promise<AudioBuffer> {
|
||
const offline = new OfflineAudioContext(2, Math.ceil(duration * sampleRate), sampleRate)
|
||
const adapter = new BrowserTimelineAdapter({ context: offline })
|
||
await adapter.init()
|
||
timeline.setAdapter(adapter)
|
||
timeline.play(0)
|
||
return offline.startRendering()
|
||
}
|
||
```
|
||
|
||
Because the adapter takes an injected context, offline render reuses the *exact same* rendering code
|
||
path as realtime playback — there is no separate offline renderer to keep in sync. This is also why
|
||
the test suite can exercise the whole adapter against an `OfflineAudioContext`: the "tests" and the
|
||
"export feature" are the same seam.
|
||
|
||
---
|
||
|
||
## 7. Html5AudioFallback
|
||
|
||
For environments with no Web Audio API at all (locked-down embedded webviews, very old browsers),
|
||
`Html5AudioFallback` implements the same `TimelineAdapter` interface using a bare `HTMLAudioElement`.
|
||
It supports only what an `<audio>` element can honour — `play` / `pause` / `stop` / `seek` /
|
||
`getCurrentTime` / `setLoop` / `setMasterVolume`. Operations it cannot perform (per-track
|
||
volume/mute/solo/pan, `setTempo`, FX hosting, sample-accurate scheduling) are defined no-ops or throw
|
||
`BrowserAudioError`; they never silently corrupt state. The fallback never constructs an
|
||
`AudioContext`. It is a graceful-degradation path, not a feature-equivalent engine, and the spec is
|
||
explicit about that boundary.
|
||
|
||
---
|
||
|
||
## 8. File layout
|
||
|
||
```text
|
||
packages/audio/audio-engine-browser/
|
||
package.json, tsconfig.json, vitest.config.ts
|
||
src/
|
||
index.ts
|
||
error-codes.ts (+ .test.ts)
|
||
adapter/ browser-adapter.ts (+ .md .types.ts .test.ts) · fx-host.ts (+ .md .test.ts) · index.ts
|
||
graph/ web-audio-graph.ts (+ .md .types.ts .test.ts) · index.ts
|
||
worklet/ worklet-manager.ts (+ .md .types.ts .test.ts) · index.ts
|
||
fallback/ html5-fallback.ts (+ .md .test.ts) · index.ts
|
||
offline/ offline-render.ts (+ .md .test.ts) · index.ts
|
||
__benchmarks__/ graph.bench.ts worklet.bench.ts
|
||
tests/
|
||
integration.test.ts purity.test.ts exports.test.ts
|
||
```
|
||
|
||
Each public options object (`BrowserAdapterOptions`, `WorkletManagerOptions`, `WorkletModuleOptions`,
|
||
`AudioGraphSnapshot`) gets a sibling `*.schemas.ts` with a Zod schema, validated at the boundary.
|
||
|
||
---
|
||
|
||
## 9. Integration notes
|
||
|
||
- **`@suaveplan/audio-timeline`** — supplies the `TimelineAdapter` interface (plugin slot 5),
|
||
`TimelineEngine`, `Track`, and `Clip`. `BrowserTimelineAdapter` and `Html5AudioFallback` both
|
||
implement `TimelineAdapter`; the adapter receives resolved `Track[]` and never re-derives timeline
|
||
state.
|
||
- **`@suaveplan/audio-transport`** — supplies the `Transport` musical clock; `lookAhead` and `ppqn`
|
||
on the adapter feed the transport's lookahead scheduler.
|
||
- **`@suaveplan/audio-fx-plugin-base`** — supplies the `AudioFxPlugin` interface (plugin slot 6) hosted
|
||
by `createFxNode`.
|
||
- **`@suaveplan/audio-core`** — supplies shared audio primitives (`BLOCK_SIZE`, the `BlockProcessor`
|
||
contract that `processBlock` satisfies) used by the `ScriptProcessorNode` fallback path.
|
||
- **`@suaveplan/events`** — backs the adapter's typed event surface.
|
||
- **`standardized-audio-context`** — the cross-browser `AudioContext` / `OfflineAudioContext` shim;
|
||
the sole non-`@suaveplan` runtime dependency beyond `@opentelemetry/api`.
|
||
|
||
---
|
||
|
||
## 10. Benchmarks
|
||
|
||
- `graph.bench.ts` — add 100 nodes, connect 200 edges, then `snapshot()` + `restore()`.
|
||
- `worklet.bench.ts` — load 10 worklet modules and create 50 `AudioWorkletNode`s.
|
||
|
||
---
|
||
|
||
## 11. Decisions and trade-offs
|
||
|
||
- **Adapter, not engine.** The package is a `TimelineAdapter` implementation with zero timeline
|
||
logic. This makes it swappable with `audio-engine-server` and obsoletes the legacy browser-only
|
||
audio openspecs that tangled timeline state into rendering code.
|
||
- **One injectable context seam.** Every browser API goes through the `context` option or
|
||
`standardized-audio-context`. The cost is one extra constructor parameter; the payoff is that the
|
||
whole package — including offline render — is testable under `happy-dom` and that offline export
|
||
and realtime playback share one code path.
|
||
- **Fixed-shape per-track subgraph.** Every track gets the identical node chain. A uniform shape
|
||
makes mute/solo/volume/pan map to predictable nodes and keeps `snapshot()`/`restore()` trivial,
|
||
at the cost of a few unused nodes on a track with no pan or compression — acceptable, since
|
||
`GainNode`/`StereoPannerNode` at unity are effectively free.
|
||
- **Three FX hosting paths, preference-ordered.** Native `createAudioNode` first, worklet second,
|
||
`ScriptProcessorNode` last. `ScriptProcessorNode` is deprecated but kept as the universal fallback
|
||
so no FX plugin is ever unhostable; the spec is explicit that it runs on the main thread.
|
||
- **Html5AudioFallback is degradation, not parity.** It implements `TimelineAdapter` so callers
|
||
need no branching, but unsupported operations throw or no-op rather than pretending to work.
|
||
- **No design.md for the React/r3f tier yet.** `react-audio*` and `r3f-audio` consume this package
|
||
but are out of scope for this 13-change batch; they land in a later wave.
|