329 lines
20 KiB
Markdown
329 lines
20 KiB
Markdown
# OpenSpec Change: enterprise-audio-plugins-fx
|
||
|
||
- **Change ID**: enterprise-audio-plugins-fx
|
||
- **Package**: `@suaveplan/audio-plugins-fx`
|
||
- **Category**: universal (with optional browser-native fast paths)
|
||
- **Target path**: `packages/audio/audio-plugins-fx`
|
||
- **Status**: PROPOSED
|
||
- **Date**: 2026-05-15
|
||
- **Design**: `./design.md`
|
||
- **Spec delta**: `./specs/audio-plugins-fx/spec.md`
|
||
|
||
---
|
||
|
||
## Summary
|
||
|
||
Deliver `@suaveplan/audio-plugins-fx` on the genesis branch from scratch at enterprise quality. The package provides:
|
||
|
||
1. **A future-proof `AudioFxPlugin` contract** built on `@suaveplan/audio-core`'s `BlockProcessor` — universal DSP that runs identically in Node/Bun/server, browser worklets, and offline render, with an optional `createAudioNode()` fast path for browser-native Web Audio rendering.
|
||
2. **A `PluginChain` universal primitive** — the missing piece. A first-class, headless, ordered plugin list with insertion-time channel validation, plugin delay compensation, hot-swap, snapshot/restore, an event bus, and chain-level bypass wrapping. The chain is what enterprise DAWs use as the "insert FX slot list"; it lives in `audio-plugins-fx` so every engine adapter (browser, server, offline) reuses one implementation.
|
||
3. **A `ChainContext` / `AmbientState` propagation system** — every plugin in a chain receives a read-only view of the chain's structure (predecessors, successors, total length), of ambient state (track gain/pan/mute/solo/layout, transport BPM/time/state, listener position for 3D), and of project-level sidechain buses. This is the "config tree plugins can traverse" requirement: a late-chain compressor can read `ctx.ambient.track.pan`, a tempo-synced delay can read `ctx.ambient.transport.tempo`, a sidechain compressor can pull from `ctx.buses.get('kick')`. No global state, no prop drilling, fully typed.
|
||
4. **22 first-party plugins** — EQ, filter, compressor, limiter, gate, reverb, delay, chorus, phaser, flanger, tremolo, auto-filter, distortion, pitch shifter, synthesizer, sampler, mixer, stereo widener, plus four newly-added plugins from upstream (`microphone`, `glados`, `audio-player`) and the building blocks for game audio (`spatial-positional`, `proximity-voice`, `music-layer-mixer`, `voice-transform`).
|
||
5. **Subpath exports per plugin** — `sideEffects: false`, one `package.json` `exports` entry per plugin so importing `./eq` does not bundle `./synthesizer`.
|
||
|
||
The package is universal-tier: zero browser globals, zero Node globals on the `BlockProcessor` path. Browser-native rendering and worklet hosting are optional surfaces that engine adapters consume via capability flags.
|
||
|
||
---
|
||
|
||
## Current-state audit (genesis)
|
||
|
||
The package does not exist on genesis yet — this is a fresh build. The upstream `packages/browser/audio-plugins-fx/` is browser-only and built on the obsolete monolithic `@suaveplan/audio-engine`; it is being replaced with this universal architecture.
|
||
|
||
---
|
||
|
||
## Parity Checklist (vs upstream `../suaveplan/packages/browser/audio-plugins-fx/`)
|
||
|
||
| Capability | Upstream | Migrated | Notes |
|
||
|---|:---:|:---:|---|
|
||
| `AudioPlugin` base class | ✅ | ⬆️ | Replaced by `AudioFxPlugin` interface extending `BlockProcessor` from `@suaveplan/audio-core`. Now universal — runs in browser, server, offline, worklet contexts identically. Optional `createAudioNode` / `getWorkletProcessor` for browser-native fast paths. |
|
||
| `EQPlugin` (8-band parametric) | ✅ | ✅ | Same surface; now extends `AbstractAudioFxPlugin`; presets validated via Zod. |
|
||
| `CompressorPlugin` | ✅ | ⬆️ | Adds sidechain via `setSidechain(bus)` consuming the project-level `BusRegistry`; adds mid/side mode reading `ChainContext.ambient.track.layout`. |
|
||
| `ReverbPlugin` (IR + algorithmic) | ✅ | ⬆️ | `loadImpulseResponse` now accepts `AbortSignal`. |
|
||
| `DelayPlugin` (ping-pong + BPM sync) | ✅ | ⬆️ | BPM sync now reads `ChainContext.ambient.transport.tempo` instead of a separate transport reference. |
|
||
| `PitchShifterPlugin` (phase vocoder + worklet) | ✅ | ⬆️ | Reports FFT-frame `latencySamples` via the new `onLatencyChanged` event so chain PDC stays accurate. |
|
||
| `SynthesizerPlugin` (oscillator + ADSR + voice pool + portamento) | ✅ | ✅ | Source plugin (`io.inputs[0].count === 0`); validates note range with `AUDIO_FX.INVALID_NOTE`. |
|
||
| `MixerPlugin` (channel strips + sends + returns + master + VU) | ✅ | ✅ | Same surface; integrated with the `PluginChain`'s ambient state for solo across the project. |
|
||
| `GLaDOSPlugin` (character vocoder) | ✅ | ✅ | Preset envelope validated against the plugin's Zod schema. |
|
||
| `MicrophonePlugin` (full mic processing chain) | ✅ | ✅ | Worklet path is the browser fast path; universal `BlockProcessor` is the server path. |
|
||
| `AudioPlayerPlugin` (track-style playback) | ✅ | ✅ | Source plugin; distinct from `@suaveplan/audio-sprite-player` (cue triggering). |
|
||
| `AudioAnalyzerPlugin` (FFT / tempo / pitch / spectral) | ✅ | 🚫 | Scoped out: replaced by dedicated `@suaveplan/audio-analysis-{beat,pitch,loudness,spectrum}` packages so analysis plugins stay headless and don't depend on the FX-plugin surface. The plugin's old features are 100 % covered by the four split packages. |
|
||
| `GameAudioPlugin` (spatial + voice-chat + music layers + voice transform) | ✅ | ⬆️ | Decomposed into four building blocks (`spatial-positional`, `proximity-voice`, `music-layer-mixer`, `voice-transform`) so non-R3F games can consume them directly; `r3f-audio` composes them for the R3F surface. |
|
||
| New plugins (`filter`, `limiter`, `gate`, `chorus`, `phaser`, `flanger`, `tremolo`, `auto-filter`, `distortion`, `sampler`, `widener`) | – | ⬆️ | Genesis additions covering the standard FX surface every DAW expects. |
|
||
| Chain primitive (`PluginChain`) | – | ⬆️ | Genesis addition — universal insert list with insertion-time IO validation, chain-level bypass, ambient-state propagation, plugin delay compensation, snapshot/restore, typed event surface. |
|
||
| Project-level `BusRegistry` | – | ⬆️ | Genesis addition — sidechain routing crossing tracks/chains. |
|
||
| Parameter automation curves + transport-time scheduling | partial (per-plugin) | ⬆️ | Universal `ParameterSchedule` evaluator owns wall-clock curves + transport-time events with shared queue; transport events override in-flight curves. |
|
||
| Preset system | partial | ⬆️ | Universal `PresetSnapshot` envelope with `schemaVersion`, Zod validation at the boundary, atomic apply. |
|
||
| Subpath exports per plugin | partial | ⬆️ | One subpath per plugin + `./base` + `./chain` + `./registry`; bundle-size tests prove tree-shaking. |
|
||
| Unit tests / coverage | partial | ⬆️ | 100/100/100/100 across every module; property-based tests; per-plugin and chain benchmarks. |
|
||
|
||
One 🚫 row, justified: `audio-analyzer` is replaced by the dedicated `audio-analysis-*` family so analysis stays headless and composable with non-FX consumers (CLIs, server pipelines).
|
||
|
||
---
|
||
|
||
## Available workspace deps on genesis
|
||
|
||
| Package | Role |
|
||
|---|---|
|
||
| `@suaveplan/audio-core` (`workspace:^`) | `BlockProcessor`, `BlockContext`, `ProcessorConfig`, `AudioError` base |
|
||
| `@suaveplan/audio-transport` (`workspace:^`) | `Transport` reference for `setParameterAt(transportTime)` resolution |
|
||
| `@suaveplan/schema-registry` (`workspace:^`) | Plugin preset Zod schema registration |
|
||
| `@suaveplan/events` (`workspace:^`) | Typed event bus for chain events |
|
||
| `@suaveplan/error` (`workspace:^`) | `defineCodes`, `SuaveplanError` |
|
||
| `@suaveplan/types` (`workspace:^`) | `LoggerLike`, `Clock` |
|
||
| `@suaveplan/opentelemetry` (`workspace:^`) | OTel spans, counters, histograms |
|
||
| `zod` (`catalog:forms`) | Source of truth for parameter shapes + preset schemas |
|
||
|
||
Dev deps: `vitest`, `@vitest/coverage-istanbul`, `fast-check`, `mitata` (all `catalog:testing`).
|
||
|
||
---
|
||
|
||
## Public API
|
||
|
||
### Core contract
|
||
|
||
```ts
|
||
interface AudioFxPlugin extends BlockProcessor {
|
||
readonly metadata: PluginMetadata
|
||
readonly schema: ZodType<unknown> // source of truth for options + preset
|
||
readonly parameters: ReadonlyArray<PluginParameter> // declaration order, for UI iteration
|
||
readonly parameterIndex: ReadonlyMap<string, PluginParameter> // O(1) lookup
|
||
readonly io: PluginIo // declared channel topology
|
||
readonly capabilities: PluginCapabilities // explicit feature flags
|
||
|
||
// Lifecycle
|
||
setup(config: ProcessorConfig, chain: ChainContext, opts?: { signal?: AbortSignal }): Promise<void>
|
||
process(input: Float32Array[], output: Float32Array[], ctx: BlockContext): void
|
||
reset(): void
|
||
dispose(): void
|
||
getLatencySamples(): number // for plugin delay compensation
|
||
|
||
// Parameters
|
||
getParameter(id: string): number
|
||
setParameter(id: string, value: number): void
|
||
setParameterCurve(id: string, curve: ParameterCurve): void // ramp over seconds
|
||
setParameterAt(id: string, value: number, transportTime: number): void // ramp to musical-time
|
||
onLatencyChanged(listener: (samples: number) => void): UnsubscribeFn
|
||
|
||
// Atomic snapshots
|
||
getPreset(): PresetSnapshot
|
||
applyPreset(p: PresetSnapshot): void
|
||
|
||
// Optional browser-native fast paths
|
||
createAudioNode?(ctx: BaseAudioContext, chain: ChainContext): AudioNode
|
||
getWorkletProcessor?(): string
|
||
|
||
// Optional sidechain consumer
|
||
setSidechain?(bus: SidechainBus | undefined): void
|
||
}
|
||
|
||
interface PluginIo {
|
||
readonly inputs: ReadonlyArray<ChannelSpec>
|
||
readonly outputs: ReadonlyArray<ChannelSpec>
|
||
}
|
||
|
||
interface ChannelSpec {
|
||
readonly count: number | 'preserve' | 'any'
|
||
readonly layout?: ChannelLayout // 'mono' | 'stereo' | 'mid-side' | '5.1' | '7.1' | 'ambisonic-1'
|
||
}
|
||
|
||
interface PluginCapabilities {
|
||
readonly hasAudioNode: boolean
|
||
readonly hasWorklet: boolean
|
||
readonly hasSidechain: boolean
|
||
readonly sampleAccurate: boolean
|
||
readonly stateful: boolean
|
||
readonly asyncSetup: boolean
|
||
readonly preservesChannelCount: boolean
|
||
readonly latencySamples: number // initial; dynamic via getLatencySamples()
|
||
}
|
||
|
||
interface PluginParameter {
|
||
readonly id: string
|
||
readonly name: string
|
||
readonly value: number
|
||
readonly min: number
|
||
readonly max: number
|
||
readonly default: number
|
||
readonly unit?: ParameterUnit // 'db' | 'hz' | 'ms' | 'percent' | 'semitones' | …
|
||
readonly scale?: 'linear' | 'log' | 'exponential'
|
||
readonly automatable?: boolean
|
||
}
|
||
|
||
interface ParameterCurve {
|
||
readonly type: 'linear' | 'exponential' | 'target'
|
||
readonly value: number
|
||
readonly duration: number
|
||
readonly startTime?: number
|
||
}
|
||
```
|
||
|
||
### The PluginChain primitive
|
||
|
||
```ts
|
||
class PluginChain {
|
||
constructor(opts?: PluginChainOptions) // { id, name, channels, sampleRate, blockSize, buses?, ambient? }
|
||
|
||
// Composition (insertion-time IO validation; events emitted)
|
||
add(plugin: AudioFxPlugin, at?: number): void
|
||
remove(idOrIndex: string | number): void
|
||
move(from: number, to: number): void
|
||
replace(idOrIndex, plugin, opts?: { crossfadeMs?: number }): Promise<void>
|
||
bypass(idOrIndex, bypassed: boolean): void // chain-level wrapper; standard DAW behavior
|
||
// Note: dry/wet is per-plugin (matching Pro Tools / Logic / Cubase / Reaper / Studio One);
|
||
// the chain does NOT auto-wrap dry/wet. Plugins that need it expose a `wet` parameter.
|
||
|
||
// Lifecycle
|
||
setup(config: ProcessorConfig, ambient: AmbientState, opts?: { signal?: AbortSignal }): Promise<void>
|
||
process(input: Float32Array[], output: Float32Array[], ctx: BlockContext): void
|
||
reset(): void
|
||
dispose(): void
|
||
|
||
// Inspection — the "config tree plugins can traverse"
|
||
readonly plugins: ReadonlyArray<AudioFxPlugin>
|
||
findById(id: string): { plugin: AudioFxPlugin; index: number } | undefined
|
||
getEffectiveLatencySamples(): number // sum across non-bypassed plugins (PDC)
|
||
snapshot(): ChainSnapshot // Zod-validated session save
|
||
restore(s: ChainSnapshot): Promise<void>
|
||
|
||
// Ambient updates pushed from track / transport / scene
|
||
updateAmbient(patch: Partial<AmbientState>): void
|
||
|
||
// Events
|
||
on(event: ChainEvent, listener): UnsubscribeFn
|
||
// events: 'plugin-added' | 'plugin-removed' | 'plugin-moved' | 'plugin-replaced'
|
||
// | 'parameter-changed' | 'preset-applied' | 'bypass-changed'
|
||
// | 'ambient-changed' | 'latency-changed'
|
||
}
|
||
```
|
||
|
||
### Ambient state and ChainContext
|
||
|
||
```ts
|
||
interface ChainContext {
|
||
readonly chain: ChainHandle // read-only view of the chain
|
||
readonly position: { readonly index: number; readonly total: number }
|
||
readonly predecessors: ReadonlyArray<PluginHandle> // upstream peers
|
||
readonly successors: ReadonlyArray<PluginHandle> // downstream peers
|
||
readonly ambient: AmbientState
|
||
readonly buses: BusRegistry // project-level (engine-owned)
|
||
}
|
||
|
||
interface PluginHandle {
|
||
readonly id: string
|
||
readonly metadata: PluginMetadata
|
||
readonly io: PluginIo
|
||
readonly capabilities: PluginCapabilities
|
||
readonly bypassed: boolean
|
||
getParameter(id: string): number // READ-ONLY view of peer parameters
|
||
getLatencySamples(): number
|
||
}
|
||
|
||
interface AmbientState {
|
||
readonly track: {
|
||
readonly id: string
|
||
readonly gainDb: number
|
||
readonly pan: number // -1 (full L) .. +1 (full R) — the LR-balance case
|
||
readonly mute: boolean
|
||
readonly solo: boolean
|
||
readonly layout: ChannelLayout
|
||
}
|
||
readonly transport: {
|
||
readonly tempo: number // BPM
|
||
readonly timeSignature: TimeSignature
|
||
readonly transportTimeSec: number
|
||
readonly state: 'stopped' | 'playing' | 'paused' | 'recording'
|
||
readonly ppqn: number
|
||
}
|
||
readonly listener?: { readonly position: Vec3; readonly orientation: Quat; readonly velocity: Vec3 }
|
||
readonly extension: Readonly<Record<string, unknown>> // open extension point
|
||
}
|
||
```
|
||
|
||
### Project-level bus registry (engine-owned, chain references it)
|
||
|
||
```ts
|
||
interface BusRegistry {
|
||
get(name: string): SidechainBus | undefined
|
||
list(): ReadonlyArray<string>
|
||
publish(name: string, channels: number): SidechainPublisher // create or get
|
||
subscribe(name: string, listener: (bus: SidechainBus | undefined) => void): UnsubscribeFn
|
||
}
|
||
|
||
interface SidechainBus {
|
||
readonly name: string
|
||
readonly channels: number
|
||
read(into: Float32Array[]): void // block-aligned read
|
||
}
|
||
|
||
interface SidechainPublisher {
|
||
write(samples: Float32Array[]): void // chain writes its post-fx output
|
||
close(): void
|
||
}
|
||
```
|
||
|
||
The chain reads from `ctx.buses` via its `ChainContext`. The actual `BusRegistry` instance lives on the engine adapter (`audio-engine-browser` / `audio-engine-server`) so cross-track / cross-chain routing works across the whole project.
|
||
|
||
### Subpath exports (one per plugin + the chain + base)
|
||
|
||
```
|
||
@suaveplan/audio-plugins-fx
|
||
├── ./base AudioFxPlugin, PluginIo, PluginCapabilities, ParameterCurve, PresetSnapshot, AbstractAudioFxPlugin
|
||
├── ./chain PluginChain, ChainContext, AmbientState, BusRegistry, ChainSnapshot, ChainEvent
|
||
├── ./registry PluginRegistry (id → factory; for session restore)
|
||
├── ./eq, ./filter, ./compressor, ./limiter, ./gate
|
||
├── ./reverb, ./delay, ./chorus, ./phaser, ./flanger
|
||
├── ./tremolo, ./auto-filter
|
||
├── ./distortion, ./pitch-shifter
|
||
├── ./synthesizer, ./sampler
|
||
├── ./mixer, ./widener
|
||
├── ./spatial-positional ← building block
|
||
├── ./proximity-voice ← building block
|
||
├── ./music-layer-mixer ← building block
|
||
├── ./voice-transform ← building block
|
||
├── ./microphone ← new from upstream
|
||
├── ./glados ← new from upstream
|
||
├── ./audio-player ← new from upstream
|
||
└── ./presets/* lazy preset packs
|
||
```
|
||
|
||
---
|
||
|
||
## New additions for genesis (beyond upstream)
|
||
|
||
1. **Universal `BlockProcessor` base** — every plugin runs headless. Browser-native rendering is optional.
|
||
2. **`PluginChain` primitive** with chain-level bypass, snapshot/restore, hot-swap, events, PDC computation.
|
||
3. **`ChainContext` + `AmbientState`** — read-only ambient propagation for track / transport / listener / sidechain.
|
||
4. **`PluginIo` channel topology** — explicit input/output channel specs validated at chain insertion; supports upmix, downmix, channel-count change.
|
||
5. **`PluginCapabilities` flags** — explicit feature detection; replaces opacity of optional methods.
|
||
6. **Project-level `BusRegistry`** — first-class sidechain routing.
|
||
7. **Plugin delay compensation** — every plugin reports `getLatencySamples()`; chains sum it; engines apply per-track compensation.
|
||
8. **Dynamic latency** — `onLatencyChanged` event for lookahead limiters etc.
|
||
9. **Transport-synced parameter automation** — `setParameterAt(transportTime)` for DAW automation lanes.
|
||
10. **Async setup with `AbortSignal`** — reverb IRs, sampler files, worklet modules load via `setup(): Promise<void>` cancellable mid-load.
|
||
11. **Zod-typed parameter schemas** — single source of truth; parameter list + preset shape derived from one schema.
|
||
12. **Game-audio decomposed into building blocks** — `spatial-positional`, `proximity-voice`, `music-layer-mixer`, `voice-transform` — usable in `r3f-audio` and non-R3F games alike.
|
||
13. **Three plugins ported from upstream** — `microphone`, `glados`, `audio-player`.
|
||
14. **OTel instrumentation per-plugin and per-chain** — latency histograms, block counters, parameter-change counters, preset-apply counters.
|
||
15. **Property-based tests** — gain plugin: 0 dB → identity, -∞ dB → silence; bypass → bit-identity; chain order invariance for commutative pairs.
|
||
16. **Per-plugin and chain benchmark suite** — block throughput, chain composition overhead.
|
||
|
||
---
|
||
|
||
## Impact
|
||
|
||
**Breaking changes**: none — fresh build, package does not yet exist on genesis.
|
||
|
||
**Prerequisites**: `@suaveplan/audio-core`, `@suaveplan/audio-transport`, `@suaveplan/schema-registry`, `@suaveplan/error`, `@suaveplan/types`, `@suaveplan/opentelemetry`, `@suaveplan/events` published to Verdaccio.
|
||
|
||
### Affected dependents
|
||
|
||
`@suaveplan/audio-plugins-fx` sits in the FX tier. At the time of this change none of its dependents exist on genesis yet — they land in later changes in this same batch — so there is no existing dependent surface to test or migrate. Each dependent is created against this package's spec delta, not retrofitted.
|
||
|
||
- `@suaveplan/audio-engine-browser` — [ ] tests run / [ ] docs reviewed / [x] no action needed (reason: does not exist yet; created in `enterprise-audio-engine-browser`. It hosts one `PluginChain` per track via the chain's `process()` method, owns the project `BusRegistry`, and maps each plugin's `createAudioNode()` into its node graph where present.)
|
||
- `@suaveplan/audio-engine-server` — [ ] tests run / [ ] docs reviewed / [x] no action needed (reason: does not exist yet; created in `enterprise-audio-engine-server`. It hosts the chain's `BlockProcessor` path per track and owns the project `BusRegistry`.)
|
||
- `@suaveplan/audio-controllers` — [ ] tests run / [ ] docs reviewed / [x] no action needed (reason: does not exist yet; created in `enterprise-audio-controllers`. It binds MIDI CCs to `plugin.parameterIndex.get(id)` lookups for live parameter control.)
|
||
- `@suaveplan/audio-cli` — [ ] tests run / [ ] docs reviewed / [x] no action needed (reason: does not exist yet; created in `enterprise-audio-cli`.)
|
||
- `@suaveplan/r3f-audio` — [ ] tests run / [ ] docs reviewed / [x] no action needed (reason: requires re-authoring per `openspec/changes/REACT_AUDIO_WAVE_REWRITE_COMPLETE.md`. The rewritten version composes the four `spatial-positional` / `proximity-voice` / `music-layer-mixer` / `voice-transform` building blocks from this package.)
|
||
|
||
Once landed, any subsequent change to this package's public surface MUST run `bunx turbo test --filter=...@suaveplan/audio-plugins-fx` per `openspec/AGENTS.md` §13 step 10.
|