# Design — enterprise-gaming-r3f-audio `@suaveplan/r3f-audio` is a **bridge package**. Its only job is to translate Three.js scene-graph state (camera pose, object pose, raycaster results) into the universal `AmbientState` and into parameter writes on game-audio plugins. All audio behaviour — distance attenuation, HRTF panning, music-layer crossfading, voice transformation, sidechain ducking — lives in `@suaveplan/audio-plugins-fx` and runs identically without R3F (in tests, in headless renders, in non-R3F games). R3F is the **input source**, not the audio. Cross-cutting audio architecture: see `openspec/changes/enterprise-audio-plugins-fx/design.md`. This document covers the R3F-specific bridge. --- ## 1. Position in the audio family ``` R3F game scene │ ▼ @suaveplan/r3f-audio ← this package (scene-graph → AmbientState bridge, ref-to-plugin parameter wiring) │ ▼ @suaveplan/audio-plugins-fx spatial-positional + proximity-voice + music-layer-mixer + voice-transform │ ▼ @suaveplan/audio-engine-browser (rendering) ``` The package contains **no `BaseAudioContext` reference**. It does not construct, own, or dispose audio objects. Every audio behaviour is delegated to plugins; this package only writes the plugins' inputs every frame. --- ## 2. The scene-graph-to-ambient bridge Three.js gives us a transform per `Object3D` updated every render frame. The universal `AmbientState.listener` carries `{ position, orientation, velocity }`. The bridge is: ```ts function useAudioListener(objectRef: RefObject) { const { engine } = useAudioEngine() useFrame(() => { const obj = objectRef.current if (!obj) return const pose: ListenerState = { position: vec3FromThree(obj.getWorldPosition(scratchVec)), orientation: quatFromThree(obj.getWorldQuaternion(scratchQuat)), velocity: computeVelocity(obj, scratchVec, scratchPrev), // discrete diff per frame } engine.updateAmbient({ listener: pose }) }) } ``` - `useFrame` runs once per R3F render tick (typically 60 Hz). - The bridge calls `engine.updateAmbient({ listener })`, which the engine forwards to every active chain. Every chain emits a single `ambient-changed` event; plugins read the new listener on their next `process()`. - Scratch vectors and quaternions are allocated once at hook construction (no per-frame allocation). `` is a thin component that mounts `useAudioListener` against `useThree().camera` by default. Users can opt into a different `Object3D` (e.g. first-person controller) by passing `target`. --- ## 3. `` — plugin per source Each `` mounts one `SpatialPositionalPlugin` instance via the provider's `PluginRegistry`, inserts it into a designated `PluginChain` (default: a project-level "spatial sources" chain owned by the provider), and writes its `position`/`orientation` parameters from a Three.js `Object3D` ref every frame: ```tsx function PositionalAudio({ src, position, audio, options }: PositionalAudioProps) { const ref = useRef(null) const plugin = useSpatialPositionalPlugin(audio, options) useFrame(() => { const obj = ref.current if (!obj || !plugin) return plugin.setParameter('positionX', obj.position.x) plugin.setParameter('positionY', obj.position.y) plugin.setParameter('positionZ', obj.position.z) }) return } ``` Distance attenuation, HRTF panning, occlusion compensation, and (browser tier) the `PannerNode` wiring all happen inside the plugin. The component owns only the pose write loop. --- ## 4. Occlusion `` exposes a per-source occlusion factor by raycasting from listener to source against a configurable list of occluder meshes: ```ts const hits = raycaster.intersectObjects(occluders, true) const occlusion = hits.length > 0 ? sumAttenuation(hits) : 0 // dB plugin.setParameterCurve('occlusionDb', { type: 'linear', value: occlusion, duration: 0.05 }) ``` The 50 ms `setParameterCurve` smooths discrete frame-to-frame occlusion changes into an audible curve — without it, walking past a wall produces audible stepping. The raycaster pool reuses one `Raycaster` instance across all positional-audio sources to avoid per-frame allocation. --- ## 5. `` — game-state to layer weights `MusicLayerMixerPlugin` reads `AmbientState.extension.gameState`. The R3F component binds a React-controlled prop to that ambient slot: ```tsx function MusicLayerMixer({ layers, gameState }: MusicLayerMixerProps) { const { engine } = useAudioEngine() useEffect(() => { engine.updateAmbient({ extension: { gameState } }) }, [engine, gameState]) // mounts/loads each layer once via plugin.loadLayer } ``` A consumer renders `` and the plugin's crossfade lands on the next process block. The crossfade duration is set on the plugin (`transitionMs`), not the component. --- ## 6. `` and `` Voice plugins wrap inputs from a `MediaStream` (peer audio over WebRTC, captured mic, etc.). The R3F components handle: - attaching the stream as a source to the plugin (via the plugin's `setSourceStream` API), - writing the participant's world-space position to the plugin's `voicePosition` parameter every frame (proximity voice), - writing transform parameters from React props (voice transform). The streaming itself is owned by upstream packages (`react-audio-stream` for WebRTC, browser `getUserMedia` for local capture). --- ## 7. Voice pooling and lifecycle `` mounts and unmounts as the React tree changes. To avoid plugin churn (constructor + destructor per dialogue line is expensive), the provider exposes a **voice pool** for short-lived sources: ```ts const handle = pool.acquire('spatial-positional') // gets a pooled plugin instance playSound(handle, buffer) pool.release(handle) // returns to pool, resets state ``` Long-lived sources (continuous emitters, music layers) bypass the pool and mount their plugin for the component's lifetime. --- ## 8. Decisions and trade-offs - **No new audio surface in this package.** Everything is in `audio-plugins-fx`. R3F is the input source. - **One bridge per concept**: listener → ambient, positional source → plugin parameter writes, music layers → ambient extension, occlusion → parameter curve. - **Scene-graph reads via `useFrame`**, never via React state. React renders are too slow for 60 Hz pose updates. - **All vector/quaternion math uses scratch allocations** captured at hook construction. Per-frame allocation is the easiest way to tank a 60 fps game. - **Voice pool for short-lived sources, dedicated mount for long-lived ones.** Construction cost is real for FFT-based plugins. - **Occlusion is smoothed via `setParameterCurve`, not raw writes.** Discrete frame-to-frame stepping is audible; 50 ms ramps are not. - **No coupling to a specific game state shape**: `gameState` is a generic `Record` written into `AmbientState.extension.gameState`. The plugin reads what it knows; unknown keys are ignored. - **R3F is a peer dep, not a runtime dep.** The package never imports R3F transitively — it only imports types — so non-R3F games that consume the same building blocks pay zero R3F cost.