Files

2026-05-27 23:29:53 +01:00

24 KiB

Raw Blame History

Design — enterprise-audio-plugins-fx

@suaveplan/audio-plugins-fx is the FX tier of the audio family — the layer between the universal data/timeline/transport packages below it and the rendering engines above. Its job is not to render audio; it is to express what audio processing happens, in a way that is universal (runs anywhere a Float32 buffer can flow), composable (plugins chain into ordered FX slots), introspectable (chains expose their full structure to plugins, hosts, and serializers), and extensible (third-party plugins implement one interface).

This document carries the package-specific design. The cross-cutting audio architecture (three-layer model, plugin slot taxonomy, AudioError hierarchy, universal Segment shape) lives in openspec/archive/2026-05-14-enterprise-audio-core/design.md and is referenced here, not duplicated.

1. Position in the audio family

                    universal — runs anywhere
┌─────────────────────────────────────────────────────────────┐
│ audio-core         AudioDocument · PlanEngine · BlockProcessor│
│ audio-timeline     TimelineEngine · TimelineAdapter port      │
│ audio-transport    Transport (musical clock)                  │
│ audio-plugins-fx   ← this package                             │
│                    AudioFxPlugin + PluginChain + AmbientState │
└──────────────────────────┬──────────────────────────────────┘
                           │
                ┌──────────┴───────────┐
                │                      │
       audio-engine-browser    audio-engine-server
       (TimelineAdapter)       (TimelineAdapter)
       owns: BusRegistry,      owns: BusRegistry,
              hosts chains            hosts chains

The engines host chains. Each track's FX slot list is one PluginChain instance. The engine pushes AmbientState into the chain (track gain/pan/mute, transport BPM/state, listener position) so plugins see the truth of the project without coupling to engine types. The engine owns the project-level BusRegistry; chains read from / write to it via ctx.buses.

This means one chain implementation services every render mode: live browser playback, offline-render via OfflineAudioContext, headless server render to file. The same PluginChain.process() runs in all of them.

2. Why a `PluginChain` primitive belongs here

Three forces pull PluginChain into this package and not into the engines:

One implementation across engines. If each engine implemented chain composition, channel-validation, snapshot/restore, and PDC, the three would drift. Putting the chain in audio-plugins-fx means the engines reduce to "render the chain's output into my backend."
Universal use case beyond engines. A test suite, a CLI batch processor, a worker that pre-renders a stem, an offline export job, a server-side mastering pipeline — all of these need plugin chaining without an AudioContext or node:fs. The chain is universal because the plugin contract is universal.
Round-trip serialization is a chain-level concern. A chain knows its plugins, their order, their parameters, their bypass state, and which buses they read/write. Snapshotting that to JSON and restoring it (the "track preset" / "FX chain preset" feature in every DAW) belongs to the chain. The engine just renders the result.

The chain is insert-only / serial. Parallel processing (parallel compression, send effects) is solved with project-level buses: a track writes its post-FX output to a named bus; another chain reads from that bus. This is exactly how Pro Tools / Logic / Ableton / Reaper handle parallel routing — there is no per-chain parallel branching.

3. The `AmbientState` propagation model — your "config tree"

A plugin needs to know things the chain knows: where in the chain it sits, what plugins came before, what the track's pan setting is, what BPM the transport is at, whether the project is in playback or recording state, where the 3D listener is, what sidechain buses are available.

Without explicit propagation, plugins reach for:

module-level globals (poisons isolation, breaks testing),
per-plugin constructor injection (re-binds on every chain change, leaks engine types into plugins),
post-hoc setters (plugin.setTrackPan(pan)) — the upstream @suaveplan/audio-engine did this and it produced a mess.

The clean answer is the React-context pattern adapted to DSP. setup(config, chain, opts) hands every plugin a ChainContext. The plugin captures the reference and reads from it during process() (the reference is stable for the chain's lifetime; the contents update through controlled channels). Reads are O(1) on tracked properties; ambient updates emit ambient-changed events so plugins can recompute derived state lazily.

Concrete cases this design enables:

Plugin	What it reads from `ChainContext`
`limiter`	`predecessors[i].getParameter('gain')` to compute makeup-gain headroom
`compressor` (mid/side mode)	`ambient.track.layout` to decide whether to convert to mid/side
`compressor` (sidechain)	`buses.get(this.sidechainBusName)` then `bus.read(scratchBuffer)`
`reverb`	`ambient.track.pan` to skew the stereo image of the early reflections
`delay` (BPM-sync)	`ambient.transport.tempo` + `setParameterAt(transportTimeSec + 1 bar)`
`spatial-positional`	`ambient.listener.position` + `ambient.listener.orientation`
`proximity-voice`	`ambient.listener.position` to attenuate distant voices
`music-layer-mixer`	`ambient.extension.gameState` (custom domain state via the open extension key)
Auto-gain compensation	walks `predecessors` summing `getParameter('gain')` to set inverse compensation

The pattern is uniform: read once at the top of process(), compute derived state if needed, run DSP. No global state, no engine coupling.

Crucially, PluginHandle.getParameter is read-only. Peers can read each other's parameters but cannot mutate them — that would create non-local effects no one could reason about. Mutation always goes through the owning plugin's own setParameter / setParameterCurve.

4. `PluginIo` — channel topology as a contract

Every plugin declares its IO shape:

class GainPlugin {
  readonly io = { inputs: [{ count: 'preserve' }], outputs: [{ count: 'preserve' }] }
}
class MonoToStereoUpmixer {
  readonly io = { inputs: [{ count: 1, layout: 'mono' }], outputs: [{ count: 2, layout: 'stereo' }] }
}
class FivePointOneToStereoDownmixer {
  readonly io = { inputs: [{ count: 6, layout: '5.1' }], outputs: [{ count: 2, layout: 'stereo' }] }
}
class SpectrumAnalyser {
  readonly io = { inputs: [{ count: 'any' }], outputs: [{ count: 'preserve' }] }
}

PluginChain.add(plugin) validates compatibility at insertion time. If the predecessor's effective output channels are 2 and the new plugin's input is { count: 1 } and not 'any', the chain refuses with AUDIO_FX.CHAIN_IO_INCOMPATIBLE and names the mismatching pair. Insertion fails closed — better an early throw than silent channel-count drift mid-render.

The chain computes the effective output channel count per slot at insertion time and re-validates on move, replace, and remove. The engine reads chain.getEffectiveOutputChannels() to wire its post-chain bus correctly.

5. `PluginCapabilities` — explicit feature flags

Engines need to make routing decisions per plugin:

Does it have a createAudioNode() fast path? → wire as a native Web Audio node.
Does it have a worklet processor? → fall back to AudioWorkletNode.
Is it sample-accurate? → safe to ramp parameters mid-block.
Is it stateful? → must call reset() on transport seek.
Does it have async setup? → engine must await before first process().
Does it have a fixed or dynamic latency? → latencySamples plus onLatencyChanged() subscription for PDC recompute.

Putting these in a typed capabilities record (rather than detecting via optional-method presence) makes the dispatch matrix legible:

const fxNode =
  cap.hasAudioNode             ? plugin.createAudioNode!(ctx, chainCtx)
  : cap.hasWorklet             ? wrapInWorklet(plugin)
  :                              wrapInScriptProcessor(plugin)

Capabilities are static per plugin instance; they describe the contract the plugin honors, not runtime state. Bypass and dry/wet are separate (chain wrapper + per-plugin parameter, respectively).

6. Plugin delay compensation

Latency accrues. A pitch shifter introduces FFT-frame latency. A lookahead limiter introduces lookahead samples. A convolution reverb introduces its impulse-response length minus one. If track A has a pitch shifter and track B is dry, B arrives early and the mix smears.

Every plugin reports getLatencySamples(): number. PluginChain.getEffectiveLatencySamples() sums latencies across non-bypassed plugins. The engine reads each chain's effective latency, finds the maximum across all chains, and inserts a compensating delay on every chain shorter than the max. This is identical to how Pro Tools / Logic / Cubase handle PDC.

Dynamic latency (lookahead limiters that vary lookahead, look-ahead transient designers) is supported via onLatencyChanged(listener). The chain re-emits as a chain-level latency-changed event; the engine recomputes per-chain compensation. Latency changes during playback can cause a brief mix-realignment glitch; that's normal in every DAW and is documented as a known cost.

7. Bypass and dry/wet — matching enterprise DAW conventions

Surveyed: Pro Tools, Logic Pro, Ableton Live, Cubase, Studio One, Reaper, FL Studio.

Bypass is universally a host (slot) feature. Every chain insert has a bypass toggle. When bypassed, the plugin's process() is not called and the chain pipes input straight to output for that slot (no allocation; the chain re-uses the input buffer as the slot output). Bypass is therefore a PluginChain wrapper concern — the plugin doesn't implement it.

Dry/wet is universally a plugin feature where it makes musical sense (reverb, delay, modulation, distortion); it is not a host wrapper in any major DAW because not every plugin has a dry/wet meaning (an EQ's "wet" is just the EQ; a compressor's "wet" is parallel compression which deserves its own bus, not a knob). Plugins that want dry/wet expose a wet parameter (0..1) on their parameters array and handle the mix internally. The chain does NOT auto-wrap it.

Ableton-style "Audio Effect Racks" with chain-level dry/wet are an opt-in container, not a wrapper on every plugin. If we ever add ChainGroup (rack-style sub-chains), chain-level dry/wet lives there. Not in v1.

8. Async setup with `AbortSignal`

Reverb impulse responses, sampler files, worklet module source — these need to load before process() can run. The contract: setup(config, chain, opts?): Promise<void> where opts.signal is an optional AbortSignal. Synchronous plugins return Promise.resolve(). The engine awaits chain.setup(config, ambient) before the first process(); chain.setup() awaits every plugin's setup in parallel via Promise.all. A signal aborted mid-setup propagates to every in-flight plugin via the same signal handle; setup rejects with AUDIO_FX.ABORTED.

capabilities.asyncSetup === true is advisory; engines should treat all setups as potentially async.

9. Transport-synced parameter automation

Two methods:

setParameterCurve(id, curve) — ramp over curve.duration seconds, starting at curve.startTime ?? now. UI knobs, hand-drawn envelopes.
setParameterAt(id, value, transportTimeSec) — set the parameter to value at musical-time transportTimeSec (resolved against the active Transport). DAW automation lanes, MIDI clock events.

The chain's ambient.transport.transportTimeSec updates every block from the engine; the plugin's base class computes the per-block parameter value from the schedule queue using sample-accurate interpolation. Schedules persist across reset() only if the schedule's transportTime is in the future relative to the current transport position; past events are dropped on reset (matching DAW behavior on stop-rewind).

10. Project-level `BusRegistry`

Owned by the engine. Each engine adapter (audio-engine-browser / audio-engine-server) constructs one BusRegistry instance at init time and passes it into every chain it hosts via the chain's ChainContext.buses. Tracks publish their post-FX output to named buses (registry.publish('kick', 2)); plugins on other tracks consume them (ctx.buses.get('kick')?.read(scratch)).

The registry pattern matches Pro Tools "internal buses", Logic's "Bus 1..256", Ableton's "send tracks", Cubase's "group/FX channels", Reaper's "receives". The chain itself never instantiates a bus — that would scope sidechain to single-chain reach.

SidechainBus.read(into) is block-aligned. The engine guarantees that publishers write before subscribers read in the same block by ordering chain processing topologically (publishers first, subscribers after). Cycles in the bus graph are detected and rejected by the engine.

11. Capability dispatch table

The engine adapters use this matrix to choose how to host each plugin:

Plugin path	`hasAudioNode`	`hasWorklet`	Engine renders via
`eq`	✅	–	`BiquadFilterNode` chain
`filter`	✅	–	`BiquadFilterNode`
`compressor`	✅	–	`DynamicsCompressorNode` (browser-native)
`limiter`	–	–	`BlockProcessor` only (worklet wrapper)
`gate`	–	–	`BlockProcessor` only
`delay`	✅	–	`DelayNode` chain
`reverb`	✅	–	`ConvolverNode` (browser); `BlockProcessor` (server)
`pitch-shifter`	–	✅	`AudioWorkletNode` (browser); `BlockProcessor` (server)
`synthesizer`	✅	–	`OscillatorNode` voice pool
`sampler`	✅	–	`AudioBufferSourceNode` pool
`mixer`	✅	–	`GainNode` matrix + `ChannelSplitterNode` / `ChannelMergerNode`
`spatial-positional`	✅	–	`PannerNode` (HRTF) browser; `BlockProcessor` server
All others	–	–	`BlockProcessor` only

The matrix is documented in the plugin spec; the engine's FX-host requirement just states the dispatch order (AudioNode → Worklet → BlockProcessor) and a fallback contract.

12. New plugins from upstream

microphone — a complete mic processing chain modeled on the upstream 494-line plugin: frequency response shaping, resonance control, distortion, noise generation (pink/white/hum/hiss), dynamics (compressor + gate), character (wow & flutter, vintage warmth, tape saturation), output gain. Useful for podcast / voice-acting / vocal-effects workflows. Stateful (envelope followers, delay lines for wow/flutter); preserves channel count; no Web Audio fast path (BlockProcessor-only on browser via worklet).

glados — character vocoder. Phase-vocoder + formant-shift + ring-modulator pipeline approximating the GLaDOS character voice. Real workload, mid latency. Worklet-only on the browser path.

audio-player — a track-style audio playback plugin, distinct from @suaveplan/audio-sprite-player:

audio-sprite-player triggers short cues from a sprite map (game SFX, UI sounds).
audio-player is a track-style plugin: loads a single source (URL or buffer), exposes transport-style controls (play/pause/seek/getDuration), reports playback position, and renders into the chain's output as the chain's source (used as the first plugin in a track's chain).

It implements AudioFxPlugin because uniform treatment with FX is desirable — the chain can have an audio-player as plugin 0 (source) and [eq, compressor, limiter] as plugins 1..3 in any order. The chain validates that exactly one source-shaped plugin (io.inputs[0].count === 0) sits at index 0.

Game-audio building blocks:

spatial-positional — 3D positional audio: HRTF panning (browser), amplitude panning + distance attenuation (universal). Reads ambient.listener for position/orientation.
proximity-voice — voice-chat attenuation by listener distance, optional spatial blend, VAD threshold. Reads ambient.listener.
music-layer-mixer — N music stems, each with a per-stem level driven by a vector in ambient.extension.gameState (e.g. { tension, exploration, combat, stealth }). Crossfades smoothly between game states.
voice-transform — character voice transformation (pitch, formant, distortion, mod). General-purpose; glados is a preset/specialization.

r3f-audio composes these four building blocks into the React-Three-Fiber surface. Non-R3F games consume them directly.

13. File layout

packages/audio/audio-plugins-fx/
  package.json, tsconfig.json, vitest.config.ts
  src/
    index.ts                              ← explicit named re-exports only
    error-codes.ts (+ .test.ts)           ← AUDIO_FX.* codes via defineCodes
    base/
      audio-fx-plugin.ts (+ .md .types.ts .test.ts)         ← interface + AbstractAudioFxPlugin
      capabilities.ts (+ .md .types.ts .test.ts)            ← PluginCapabilities + helpers
      plugin-io.ts (+ .md .types.ts .test.ts)               ← PluginIo + compatibility check
      parameter-schedule.ts (+ .md .test.ts)                ← curve + at-transport-time evaluator
      preset.ts (+ .md .types.ts .test.ts)                  ← snapshot/restore w/ Zod
      index.ts
    chain/
      plugin-chain.ts (+ .md .types.ts .test.ts)            ← PluginChain class
      chain-context.ts (+ .md .types.ts .test.ts)           ← ChainContext factory
      ambient-state.ts (+ .md .types.ts .test.ts)           ← AmbientState shape + reducer
      chain-snapshot.ts (+ .md .test.ts)                    ← Zod-validated session save/restore
      pdc.ts (+ .md .test.ts)                               ← latency summing + recompute
      bypass-wrapper.ts (+ .md .test.ts)                    ← chain-level bypass; passthrough buffer reuse
      bus-registry.ts (+ .md .types.ts .test.ts)            ← interface + DefaultBusRegistry impl
      index.ts
    registry/
      plugin-registry.ts (+ .md .types.ts .test.ts)         ← id → factory for session restore
      index.ts
    plugins/
      eq/                eq.ts .md .types.ts .schemas.ts .test.ts index.ts
      filter/            …
      compressor/        … (sidechain + mid/side)
      limiter/           …
      gate/              …
      reverb/            … (IR + algorithmic + presets/)
      delay/             … (ping-pong + BPM-sync)
      chorus/            …
      phaser/            …
      flanger/           …
      tremolo/           …
      auto-filter/       …
      distortion/        … (waveshaper/bitcrusher/overdrive)
      pitch-shifter/     … (FFT vocoder + worklet)
      synthesizer/       … (oscillator/envelope/voice-pool/portamento)
      sampler/           … (load + voice pool)
      mixer/             … (channels + sends + returns + master + VU)
      widener/           … (mid/side width)
      spatial-positional/ …
      proximity-voice/   …
      music-layer-mixer/ …
      voice-transform/   …
      microphone/        …
      glados/            …
      audio-player/      …
    presets/
      reverb/   cathedrals.ts  rooms.ts  plates.ts
      synth/    leads.ts  pads.ts  basses.ts
      glados/   defaults.ts
    __benchmarks__/
      eq.bench.ts  reverb.bench.ts  synth.bench.ts  chain.bench.ts
  tests/
    integration.test.ts        ← end-to-end chain w/ ambient + bus
    purity.test.ts             ← processing path = zero browser/Node globals
    exports.test.ts            ← every subpath exported; barrels use named exports
    bundle-size/
      eq.size.test.ts  reverb.size.test.ts  synth.size.test.ts  chain.size.test.ts

Five files per feature module: <name>.ts, <name>.md, <name>.types.ts, <name>.schemas.ts (Zod), <name>.test.ts. Barrel index.ts uses explicit named re-exports (§6a).

14. Integration with sibling packages

@suaveplan/audio-core — supplies BlockProcessor, BlockContext, ProcessorConfig, and the AudioError base. Every plugin extends BlockProcessor.
@suaveplan/audio-transport — supplies the Transport type read by chains for ambient.transport.*. Used by setParameterAt to resolve musical time.
@suaveplan/audio-engine-browser — hosts chains via the chain's process(). Owns the project BusRegistry. Uses capabilities.hasAudioNode / hasWorklet to choose render path per plugin.
@suaveplan/audio-engine-server — hosts chains via the chain's process() (server tier always uses BlockProcessor). Owns the project BusRegistry for offline render with sidechain.
@suaveplan/audio-controllers — binds MIDI CCs to plugin.parameterIndex.get(id) lookups. Controllers do not depend on individual plugins; they depend on the AudioFxPlugin interface from ./base.
@suaveplan/schema-registry — registers per-plugin Zod schemas for presets; centralizes versioning and migration.
@suaveplan/events — backs the chain's typed event bus.

15. Benchmarks

eq.bench.ts — process 1 second of stereo audio through an 8-band EQ.
reverb.bench.ts — process 1 second through algorithmic + IR reverb (10 s cathedral).
synth.bench.ts — render 32-voice polyphonic synth.
chain.bench.ts — chain composition overhead: 8-plugin serial chain vs. raw BlockProcessor loop; expected ≤ 5 % overhead from the chain wrapper.

Published in README with bun --cpu-prof-md output.

16. Decisions and trade-offs

Chain primitive in audio-plugins-fx, not in the engines. One implementation, used by both engines and by direct universal callers (tests, CLIs, workers). The engines reduce to "render the chain's output."
AmbientState is read-only and engine-pushed. No plugin can mutate ambient state. Reduces reasoning load to local state per plugin plus an immutable view of context.
PluginHandle.getParameter is read-only. Peers can read; mutation goes through the owning plugin. Prevents non-local effects.
Bypass at chain level, dry/wet per plugin. Matches every enterprise DAW surveyed. No exceptions, no special cases.
Project-level BusRegistry on the engine, not the chain. Cross-chain sidechain only works when buses are project-scoped — matches Pro Tools / Logic / Ableton / Reaper / Cubase.
Async setup with Promise<void> return. Simpler than a separate loadAsync; engines always await. Sync plugins return resolved promises (zero-cost).
Latency reporting on the plugin, not the chain. The plugin knows its latency; the chain sums. Dynamic latency via onLatencyChanged propagates up through latency-changed chain events.
Game-audio decomposed into building blocks. Reusable in non-R3F contexts; r3f-audio is one composition. Avoids a 700-line monolith.
No chain-internal parallel branches. Parallel routing uses the bus registry. Matches every major DAW; keeps the chain primitive small and predictable.
PluginIo declares topology before insertion. Validation at insert is loud and early; no silent channel-count drift during render.
Zod schema as source of truth for parameter shape. One declaration; types, validators, and preset shape all derive from it.
No third-party plugin base package extraction in v1. Third-party plugins implement @suaveplan/audio-plugins-fx/base. If a real third-party ecosystem materializes, we extract audio-plugins-fx-base then — non-breaking move.

24 KiB Raw Blame History Unescape Escape