Initial Commit

openspec/changes/enterprise-gaming-r3f-audio/design.md

@@ -0,0 +1,151 @@
+# Design — enterprise-gaming-r3f-audio
+
+`@suaveplan/r3f-audio` is a **bridge package**. Its only job is to translate Three.js scene-graph state (camera pose, object pose, raycaster results) into the universal `AmbientState` and into parameter writes on game-audio plugins. All audio behaviour — distance attenuation, HRTF panning, music-layer crossfading, voice transformation, sidechain ducking — lives in `@suaveplan/audio-plugins-fx` and runs identically without R3F (in tests, in headless renders, in non-R3F games). R3F is the **input source**, not the audio.
+
+Cross-cutting audio architecture: see `openspec/changes/enterprise-audio-plugins-fx/design.md`. This document covers the R3F-specific bridge.
+
+---
+
+## 1. Position in the audio family
+
+```
+        R3F game scene
+                │
+                ▼
+        @suaveplan/r3f-audio   ← this package
+          (scene-graph → AmbientState bridge,
+           ref-to-plugin parameter wiring)
+                │
+                ▼
+        @suaveplan/audio-plugins-fx
+          spatial-positional + proximity-voice
+          + music-layer-mixer + voice-transform
+                │
+                ▼
+        @suaveplan/audio-engine-browser
+          (rendering)
+```
+
+The package contains **no `BaseAudioContext` reference**. It does not construct, own, or dispose audio objects. Every audio behaviour is delegated to plugins; this package only writes the plugins' inputs every frame.
+
+---
+
+## 2. The scene-graph-to-ambient bridge
+
+Three.js gives us a transform per `Object3D` updated every render frame. The universal `AmbientState.listener` carries `{ position, orientation, velocity }`. The bridge is:
+
+```ts
+function useAudioListener(objectRef: RefObject<Object3D>) {
+  const { engine } = useAudioEngine()
+  useFrame(() => {
+    const obj = objectRef.current
+    if (!obj) return
+    const pose: ListenerState = {
+      position: vec3FromThree(obj.getWorldPosition(scratchVec)),
+      orientation: quatFromThree(obj.getWorldQuaternion(scratchQuat)),
+      velocity: computeVelocity(obj, scratchVec, scratchPrev), // discrete diff per frame
+    }
+    engine.updateAmbient({ listener: pose })
+  })
+}
+```
+
+- `useFrame` runs once per R3F render tick (typically 60 Hz).
+- The bridge calls `engine.updateAmbient({ listener })`, which the engine forwards to every active chain. Every chain emits a single `ambient-changed` event; plugins read the new listener on their next `process()`.
+- Scratch vectors and quaternions are allocated once at hook construction (no per-frame allocation).
+
+`<AudioListener>` is a thin component that mounts `useAudioListener` against `useThree().camera` by default. Users can opt into a different `Object3D` (e.g. first-person controller) by passing `target`.
+
+---
+
+## 3. `<PositionalAudio>` — plugin per source
+
+Each `<PositionalAudio>` mounts one `SpatialPositionalPlugin` instance via the provider's `PluginRegistry`, inserts it into a designated `PluginChain` (default: a project-level "spatial sources" chain owned by the provider), and writes its `position`/`orientation` parameters from a Three.js `Object3D` ref every frame:
+
+```tsx
+function PositionalAudio({ src, position, audio, options }: PositionalAudioProps) {
+  const ref = useRef<Group>(null)
+  const plugin = useSpatialPositionalPlugin(audio, options)
+  useFrame(() => {
+    const obj = ref.current
+    if (!obj || !plugin) return
+    plugin.setParameter('positionX', obj.position.x)
+    plugin.setParameter('positionY', obj.position.y)
+    plugin.setParameter('positionZ', obj.position.z)
+  })
+  return <group ref={ref} position={position} />
+}
+```
+
+Distance attenuation, HRTF panning, occlusion compensation, and (browser tier) the `PannerNode` wiring all happen inside the plugin. The component owns only the pose write loop.
+
+---
+
+## 4. Occlusion
+
+`<OcclusionProvider>` exposes a per-source occlusion factor by raycasting from listener to source against a configurable list of occluder meshes:
+
+```ts
+const hits = raycaster.intersectObjects(occluders, true)
+const occlusion = hits.length > 0 ? sumAttenuation(hits) : 0  // dB
+plugin.setParameterCurve('occlusionDb', { type: 'linear', value: occlusion, duration: 0.05 })
+```
+
+The 50 ms `setParameterCurve` smooths discrete frame-to-frame occlusion changes into an audible curve — without it, walking past a wall produces audible stepping. The raycaster pool reuses one `Raycaster` instance across all positional-audio sources to avoid per-frame allocation.
+
+---
+
+## 5. `<MusicLayerMixer>` — game-state to layer weights
+
+`MusicLayerMixerPlugin` reads `AmbientState.extension.gameState`. The R3F component binds a React-controlled prop to that ambient slot:
+
+```tsx
+function MusicLayerMixer({ layers, gameState }: MusicLayerMixerProps) {
+  const { engine } = useAudioEngine()
+  useEffect(() => {
+    engine.updateAmbient({ extension: { gameState } })
+  }, [engine, gameState])
+  // mounts/loads each layer once via plugin.loadLayer
+}
+```
+
+A consumer renders `<MusicLayerMixer gameState={{ tension: 0.8, exploration: 0.1, combat: 0.6 }} />` and the plugin's crossfade lands on the next process block. The crossfade duration is set on the plugin (`transitionMs`), not the component.
+
+---
+
+## 6. `<ProximityVoiceChat>` and `<VoiceTransform>`
+
+Voice plugins wrap inputs from a `MediaStream` (peer audio over WebRTC, captured mic, etc.). The R3F components handle:
+
+- attaching the stream as a source to the plugin (via the plugin's `setSourceStream` API),
+- writing the participant's world-space position to the plugin's `voicePosition` parameter every frame (proximity voice),
+- writing transform parameters from React props (voice transform).
+
+The streaming itself is owned by upstream packages (`react-audio-stream` for WebRTC, browser `getUserMedia` for local capture).
+
+---
+
+## 7. Voice pooling and lifecycle
+
+`<PositionalAudio>` mounts and unmounts as the React tree changes. To avoid plugin churn (constructor + destructor per dialogue line is expensive), the provider exposes a **voice pool** for short-lived sources:
+
+```ts
+const handle = pool.acquire('spatial-positional')   // gets a pooled plugin instance
+playSound(handle, buffer)
+pool.release(handle)                                // returns to pool, resets state
+```
+
+Long-lived sources (continuous emitters, music layers) bypass the pool and mount their plugin for the component's lifetime.
+
+---
+
+## 8. Decisions and trade-offs
+
+- **No new audio surface in this package.** Everything is in `audio-plugins-fx`. R3F is the input source.
+- **One bridge per concept**: listener → ambient, positional source → plugin parameter writes, music layers → ambient extension, occlusion → parameter curve.
+- **Scene-graph reads via `useFrame`**, never via React state. React renders are too slow for 60 Hz pose updates.
+- **All vector/quaternion math uses scratch allocations** captured at hook construction. Per-frame allocation is the easiest way to tank a 60 fps game.
+- **Voice pool for short-lived sources, dedicated mount for long-lived ones.** Construction cost is real for FFT-based plugins.
+- **Occlusion is smoothed via `setParameterCurve`, not raw writes.** Discrete frame-to-frame stepping is audible; 50 ms ramps are not.
+- **No coupling to a specific game state shape**: `gameState` is a generic `Record<string, number>` written into `AmbientState.extension.gameState`. The plugin reads what it knows; unknown keys are ignored.
+- **R3F is a peer dep, not a runtime dep.** The package never imports R3F transitively — it only imports types — so non-R3F games that consume the same building blocks pay zero R3F cost.