152 lines
7.4 KiB
Markdown
152 lines
7.4 KiB
Markdown
# Design — enterprise-gaming-r3f-audio
|
|
|
|
`@suaveplan/r3f-audio` is a **bridge package**. Its only job is to translate Three.js scene-graph state (camera pose, object pose, raycaster results) into the universal `AmbientState` and into parameter writes on game-audio plugins. All audio behaviour — distance attenuation, HRTF panning, music-layer crossfading, voice transformation, sidechain ducking — lives in `@suaveplan/audio-plugins-fx` and runs identically without R3F (in tests, in headless renders, in non-R3F games). R3F is the **input source**, not the audio.
|
|
|
|
Cross-cutting audio architecture: see `openspec/changes/enterprise-audio-plugins-fx/design.md`. This document covers the R3F-specific bridge.
|
|
|
|
---
|
|
|
|
## 1. Position in the audio family
|
|
|
|
```
|
|
R3F game scene
|
|
│
|
|
▼
|
|
@suaveplan/r3f-audio ← this package
|
|
(scene-graph → AmbientState bridge,
|
|
ref-to-plugin parameter wiring)
|
|
│
|
|
▼
|
|
@suaveplan/audio-plugins-fx
|
|
spatial-positional + proximity-voice
|
|
+ music-layer-mixer + voice-transform
|
|
│
|
|
▼
|
|
@suaveplan/audio-engine-browser
|
|
(rendering)
|
|
```
|
|
|
|
The package contains **no `BaseAudioContext` reference**. It does not construct, own, or dispose audio objects. Every audio behaviour is delegated to plugins; this package only writes the plugins' inputs every frame.
|
|
|
|
---
|
|
|
|
## 2. The scene-graph-to-ambient bridge
|
|
|
|
Three.js gives us a transform per `Object3D` updated every render frame. The universal `AmbientState.listener` carries `{ position, orientation, velocity }`. The bridge is:
|
|
|
|
```ts
|
|
function useAudioListener(objectRef: RefObject<Object3D>) {
|
|
const { engine } = useAudioEngine()
|
|
useFrame(() => {
|
|
const obj = objectRef.current
|
|
if (!obj) return
|
|
const pose: ListenerState = {
|
|
position: vec3FromThree(obj.getWorldPosition(scratchVec)),
|
|
orientation: quatFromThree(obj.getWorldQuaternion(scratchQuat)),
|
|
velocity: computeVelocity(obj, scratchVec, scratchPrev), // discrete diff per frame
|
|
}
|
|
engine.updateAmbient({ listener: pose })
|
|
})
|
|
}
|
|
```
|
|
|
|
- `useFrame` runs once per R3F render tick (typically 60 Hz).
|
|
- The bridge calls `engine.updateAmbient({ listener })`, which the engine forwards to every active chain. Every chain emits a single `ambient-changed` event; plugins read the new listener on their next `process()`.
|
|
- Scratch vectors and quaternions are allocated once at hook construction (no per-frame allocation).
|
|
|
|
`<AudioListener>` is a thin component that mounts `useAudioListener` against `useThree().camera` by default. Users can opt into a different `Object3D` (e.g. first-person controller) by passing `target`.
|
|
|
|
---
|
|
|
|
## 3. `<PositionalAudio>` — plugin per source
|
|
|
|
Each `<PositionalAudio>` mounts one `SpatialPositionalPlugin` instance via the provider's `PluginRegistry`, inserts it into a designated `PluginChain` (default: a project-level "spatial sources" chain owned by the provider), and writes its `position`/`orientation` parameters from a Three.js `Object3D` ref every frame:
|
|
|
|
```tsx
|
|
function PositionalAudio({ src, position, audio, options }: PositionalAudioProps) {
|
|
const ref = useRef<Group>(null)
|
|
const plugin = useSpatialPositionalPlugin(audio, options)
|
|
useFrame(() => {
|
|
const obj = ref.current
|
|
if (!obj || !plugin) return
|
|
plugin.setParameter('positionX', obj.position.x)
|
|
plugin.setParameter('positionY', obj.position.y)
|
|
plugin.setParameter('positionZ', obj.position.z)
|
|
})
|
|
return <group ref={ref} position={position} />
|
|
}
|
|
```
|
|
|
|
Distance attenuation, HRTF panning, occlusion compensation, and (browser tier) the `PannerNode` wiring all happen inside the plugin. The component owns only the pose write loop.
|
|
|
|
---
|
|
|
|
## 4. Occlusion
|
|
|
|
`<OcclusionProvider>` exposes a per-source occlusion factor by raycasting from listener to source against a configurable list of occluder meshes:
|
|
|
|
```ts
|
|
const hits = raycaster.intersectObjects(occluders, true)
|
|
const occlusion = hits.length > 0 ? sumAttenuation(hits) : 0 // dB
|
|
plugin.setParameterCurve('occlusionDb', { type: 'linear', value: occlusion, duration: 0.05 })
|
|
```
|
|
|
|
The 50 ms `setParameterCurve` smooths discrete frame-to-frame occlusion changes into an audible curve — without it, walking past a wall produces audible stepping. The raycaster pool reuses one `Raycaster` instance across all positional-audio sources to avoid per-frame allocation.
|
|
|
|
---
|
|
|
|
## 5. `<MusicLayerMixer>` — game-state to layer weights
|
|
|
|
`MusicLayerMixerPlugin` reads `AmbientState.extension.gameState`. The R3F component binds a React-controlled prop to that ambient slot:
|
|
|
|
```tsx
|
|
function MusicLayerMixer({ layers, gameState }: MusicLayerMixerProps) {
|
|
const { engine } = useAudioEngine()
|
|
useEffect(() => {
|
|
engine.updateAmbient({ extension: { gameState } })
|
|
}, [engine, gameState])
|
|
// mounts/loads each layer once via plugin.loadLayer
|
|
}
|
|
```
|
|
|
|
A consumer renders `<MusicLayerMixer gameState={{ tension: 0.8, exploration: 0.1, combat: 0.6 }} />` and the plugin's crossfade lands on the next process block. The crossfade duration is set on the plugin (`transitionMs`), not the component.
|
|
|
|
---
|
|
|
|
## 6. `<ProximityVoiceChat>` and `<VoiceTransform>`
|
|
|
|
Voice plugins wrap inputs from a `MediaStream` (peer audio over WebRTC, captured mic, etc.). The R3F components handle:
|
|
|
|
- attaching the stream as a source to the plugin (via the plugin's `setSourceStream` API),
|
|
- writing the participant's world-space position to the plugin's `voicePosition` parameter every frame (proximity voice),
|
|
- writing transform parameters from React props (voice transform).
|
|
|
|
The streaming itself is owned by upstream packages (`react-audio-stream` for WebRTC, browser `getUserMedia` for local capture).
|
|
|
|
---
|
|
|
|
## 7. Voice pooling and lifecycle
|
|
|
|
`<PositionalAudio>` mounts and unmounts as the React tree changes. To avoid plugin churn (constructor + destructor per dialogue line is expensive), the provider exposes a **voice pool** for short-lived sources:
|
|
|
|
```ts
|
|
const handle = pool.acquire('spatial-positional') // gets a pooled plugin instance
|
|
playSound(handle, buffer)
|
|
pool.release(handle) // returns to pool, resets state
|
|
```
|
|
|
|
Long-lived sources (continuous emitters, music layers) bypass the pool and mount their plugin for the component's lifetime.
|
|
|
|
---
|
|
|
|
## 8. Decisions and trade-offs
|
|
|
|
- **No new audio surface in this package.** Everything is in `audio-plugins-fx`. R3F is the input source.
|
|
- **One bridge per concept**: listener → ambient, positional source → plugin parameter writes, music layers → ambient extension, occlusion → parameter curve.
|
|
- **Scene-graph reads via `useFrame`**, never via React state. React renders are too slow for 60 Hz pose updates.
|
|
- **All vector/quaternion math uses scratch allocations** captured at hook construction. Per-frame allocation is the easiest way to tank a 60 fps game.
|
|
- **Voice pool for short-lived sources, dedicated mount for long-lived ones.** Construction cost is real for FFT-based plugins.
|
|
- **Occlusion is smoothed via `setParameterCurve`, not raw writes.** Discrete frame-to-frame stepping is audible; 50 ms ramps are not.
|
|
- **No coupling to a specific game state shape**: `gameState` is a generic `Record<string, number>` written into `AmbientState.extension.gameState`. The plugin reads what it knows; unknown keys are ignored.
|
|
- **R3F is a peer dep, not a runtime dep.** The package never imports R3F transitively — it only imports types — so non-R3F games that consume the same building blocks pay zero R3F cost.
|