suaveplan/openspec/changes/enterprise-gaming-r3f-audio/design.md

Design — enterprise-gaming-r3f-audio

@suaveplan/r3f-audio is a bridge package. Its only job is to translate Three.js scene-graph state (camera pose, object pose, raycaster results) into the universal AmbientState and into parameter writes on game-audio plugins. All audio behaviour — distance attenuation, HRTF panning, music-layer crossfading, voice transformation, sidechain ducking — lives in @suaveplan/audio-plugins-fx and runs identically without R3F (in tests, in headless renders, in non-R3F games). R3F is the input source, not the audio.

Cross-cutting audio architecture: see openspec/changes/enterprise-audio-plugins-fx/design.md. This document covers the R3F-specific bridge.

---

1. Position in the audio family

        R3F game scene
                │
                ▼
        @suaveplan/r3f-audio   ← this package
          (scene-graph → AmbientState bridge,
           ref-to-plugin parameter wiring)
                │
                ▼
        @suaveplan/audio-plugins-fx
          spatial-positional + proximity-voice
          + music-layer-mixer + voice-transform
                │
                ▼
        @suaveplan/audio-engine-browser
          (rendering)

The package contains no BaseAudioContext reference. It does not construct, own, or dispose audio objects. Every audio behaviour is delegated to plugins; this package only writes the plugins' inputs every frame.

---

2. The scene-graph-to-ambient bridge

Three.js gives us a transform per Object3D updated every render frame. The universal AmbientState.listener carries { position, orientation, velocity }. The bridge is:

function useAudioListener(objectRef: RefObject<Object3D>) {
  const { engine } = useAudioEngine()
  useFrame(() => {
    const obj = objectRef.current
    if (!obj) return
    const pose: ListenerState = {
      position: vec3FromThree(obj.getWorldPosition(scratchVec)),
      orientation: quatFromThree(obj.getWorldQuaternion(scratchQuat)),
      velocity: computeVelocity(obj, scratchVec, scratchPrev), // discrete diff per frame
    }
    engine.updateAmbient({ listener: pose })
  })
}

• useFrame runs once per R3F render tick (typically 60 Hz).
• The bridge calls engine.updateAmbient({ listener }), which the engine forwards to every active chain. Every chain emits a single ambient-changed event; plugins read the new listener on their next process().
• Scratch vectors and quaternions are allocated once at hook construction (no per-frame allocation).

<AudioListener> is a thin component that mounts useAudioListener against useThree().camera by default. Users can opt into a different Object3D (e.g. first-person controller) by passing target.

---

3. <PositionalAudio> — plugin per source

Each <PositionalAudio> mounts one SpatialPositionalPlugin instance via the provider's PluginRegistry, inserts it into a designated PluginChain (default: a project-level "spatial sources" chain owned by the provider), and writes its position/orientation parameters from a Three.js Object3D ref every frame:

function PositionalAudio({ src, position, audio, options }: PositionalAudioProps) {
  const ref = useRef<Group>(null)
  const plugin = useSpatialPositionalPlugin(audio, options)
  useFrame(() => {
    const obj = ref.current
    if (!obj || !plugin) return
    plugin.setParameter('positionX', obj.position.x)
    plugin.setParameter('positionY', obj.position.y)
    plugin.setParameter('positionZ', obj.position.z)
  })
  return <group ref={ref} position={position} />
}

Distance attenuation, HRTF panning, occlusion compensation, and (browser tier) the PannerNode wiring all happen inside the plugin. The component owns only the pose write loop.

---

4. Occlusion

<OcclusionProvider> exposes a per-source occlusion factor by raycasting from listener to source against a configurable list of occluder meshes:

const hits = raycaster.intersectObjects(occluders, true)
const occlusion = hits.length > 0 ? sumAttenuation(hits) : 0  // dB
plugin.setParameterCurve('occlusionDb', { type: 'linear', value: occlusion, duration: 0.05 })

The 50 ms setParameterCurve smooths discrete frame-to-frame occlusion changes into an audible curve — without it, walking past a wall produces audible stepping. The raycaster pool reuses one Raycaster instance across all positional-audio sources to avoid per-frame allocation.

---

5. <MusicLayerMixer> — game-state to layer weights

MusicLayerMixerPlugin reads AmbientState.extension.gameState. The R3F component binds a React-controlled prop to that ambient slot:

function MusicLayerMixer({ layers, gameState }: MusicLayerMixerProps) {
  const { engine } = useAudioEngine()
  useEffect(() => {
    engine.updateAmbient({ extension: { gameState } })
  }, [engine, gameState])
  // mounts/loads each layer once via plugin.loadLayer
}

A consumer renders <MusicLayerMixer gameState={{ tension: 0.8, exploration: 0.1, combat: 0.6 }} /> and the plugin's crossfade lands on the next process block. The crossfade duration is set on the plugin (transitionMs), not the component.

---

6. <ProximityVoiceChat> and <VoiceTransform>

Voice plugins wrap inputs from a MediaStream (peer audio over WebRTC, captured mic, etc.). The R3F components handle:

• attaching the stream as a source to the plugin (via the plugin's setSourceStream API),
• writing the participant's world-space position to the plugin's voicePosition parameter every frame (proximity voice),
• writing transform parameters from React props (voice transform).

The streaming itself is owned by upstream packages (react-audio-stream for WebRTC, browser getUserMedia for local capture).

---

7. Voice pooling and lifecycle

<PositionalAudio> mounts and unmounts as the React tree changes. To avoid plugin churn (constructor + destructor per dialogue line is expensive), the provider exposes a voice pool for short-lived sources:

const handle = pool.acquire('spatial-positional')   // gets a pooled plugin instance
playSound(handle, buffer)
pool.release(handle)                                // returns to pool, resets state

Long-lived sources (continuous emitters, music layers) bypass the pool and mount their plugin for the component's lifetime.

---

8. Decisions and trade-offs

• No new audio surface in this package. Everything is in audio-plugins-fx. R3F is the input source.
• One bridge per concept: listener → ambient, positional source → plugin parameter writes, music layers → ambient extension, occlusion → parameter curve.
• Scene-graph reads via useFrame, never via React state. React renders are too slow for 60 Hz pose updates.
• All vector/quaternion math uses scratch allocations captured at hook construction. Per-frame allocation is the easiest way to tank a 60 fps game.
• Voice pool for short-lived sources, dedicated mount for long-lived ones. Construction cost is real for FFT-based plugins.
• Occlusion is smoothed via setParameterCurve, not raw writes. Discrete frame-to-frame stepping is audible; 50 ms ramps are not.
• No coupling to a specific game state shape: gameState is a generic Record<string, number> written into AmbientState.extension.gameState. The plugin reads what it knows; unknown keys are ignored.
• R3F is a peer dep, not a runtime dep. The package never imports R3F transitively — it only imports types — so non-R3F games that consume the same building blocks pay zero R3F cost.