142 lines
6.8 KiB
Markdown
142 lines
6.8 KiB
Markdown
# Design — enterprise-react-audio-stream
|
|
|
|
`@suaveplan/react-audio-stream` provides real-time audio streaming over the unified architecture. Its core insight: an incoming peer audio stream is just another **source plugin** on a dedicated chain, indistinguishable to the rest of the system from a local `AudioPlayerPlugin`. That uniformity gets us FX chains per peer, sidechain ducking from peers, analyser readouts per peer, and snapshot/restore of "studio sessions" for free — without the streaming code having to know about any of it.
|
|
|
|
---
|
|
|
|
## 1. Position in the audio family
|
|
|
|
```
|
|
consumer React app
|
|
│
|
|
▼
|
|
react-audio-stream ← this package
|
|
(StreamSession + per-peer source plugin + transports)
|
|
│
|
|
▼
|
|
react-audio (provider + hooks)
|
|
│
|
|
▼
|
|
audio-engine-browser (TimelineAdapter + chains)
|
|
```
|
|
|
|
Per-peer sources are inserted on dedicated tracks in the engine's timeline. Track ids follow the convention `peer:<participantId>`. The chain owns FX; the source plugin owns playback of the incoming `MediaStream`.
|
|
|
|
---
|
|
|
|
## 2. `StreamSession` — transport-agnostic core
|
|
|
|
```ts
|
|
class StreamSession {
|
|
constructor(opts: StreamSessionOptions)
|
|
|
|
connect(roomId: string): Promise<void>
|
|
join(): Promise<void>
|
|
leave(): Promise<void>
|
|
disconnect(): Promise<void>
|
|
|
|
readonly state: SessionState // FSM via @suaveplan/fsm
|
|
readonly peers: ReadonlyMap<string, PeerChannel>
|
|
|
|
on(event: SessionEvent, listener): UnsubscribeFn
|
|
// 'state-changed' | 'peer-joined' | 'peer-left' | 'peer-stream-added'
|
|
// | 'peer-stream-removed' | 'transport-share-request' | 'transport-share-granted'
|
|
// | 'interruption' | 'reconnect-attempt' | 'reconnect-failed'
|
|
}
|
|
```
|
|
|
|
The session does not speak any specific transport. It delegates negotiation, ICE, signalling, and stream attachment to an injected `StreamTransport` adapter (`WebRTCTransport`, `WebSocketTransport`, `TrysteroTransport`). The same session can run identically against any transport.
|
|
|
|
---
|
|
|
|
## 3. Per-peer source plugin
|
|
|
|
When a peer adds a stream, the session emits `'peer-stream-added'` with `{ participantId, stream: MediaStream }`. The provider listens, instantiates a `StreamSourcePlugin` (an `AudioFxPlugin` source — `io.inputs[0].count === 0`, captures from a `MediaStream`), inserts it into the chain of a new `peer:<participantId>` track, and exposes it via `usePeerStreams()`.
|
|
|
|
```ts
|
|
class StreamSourcePlugin implements AudioFxPlugin {
|
|
readonly io = { inputs: [{ count: 0 }], outputs: [{ count: 2, layout: 'stereo' }] }
|
|
// browser path: createAudioNode wraps MediaStreamSourceNode
|
|
// server path: not applicable (server doesn't have MediaStream)
|
|
setSourceStream(stream: MediaStream): void
|
|
process(_in, output, ctx) {
|
|
// browser worklet path mirrors AudioNode path's behaviour
|
|
}
|
|
}
|
|
```
|
|
|
|
The plugin is just like every other source plugin — `<PluginChainView>` renders it generically, `useParameter` writes its volume, the chain can have FX after it, the bus registry can read its post-FX output.
|
|
|
|
---
|
|
|
|
## 4. FSM session lifecycle
|
|
|
|
`@suaveplan/fsm` drives the session through:
|
|
|
|
```
|
|
idle → connecting → connected (idle in room) → joining → joined (streaming)
|
|
│ │
|
|
└─→ disconnecting ─┴─→ disconnected
|
|
↑
|
|
(reconnect attempt brings us back to connecting)
|
|
```
|
|
|
|
Transitions:
|
|
|
|
- `connect(roomId)` from `idle` → `connecting` → `connected`
|
|
- `join()` from `connected` → `joining` → `joined`
|
|
- `leave()` from `joined` → `connected`
|
|
- `disconnect()` from any → `disconnecting` → `disconnected`
|
|
- Network failure from `connected`/`joined` → `disconnected` (auto-reconnect triggers)
|
|
|
|
Illegal transitions throw `AudioError` with code `REACT_AUDIO_STREAM.INVALID_STATE`.
|
|
|
|
---
|
|
|
|
## 5. Auto-reconnect
|
|
|
|
Backed by `@suaveplan/retry` (exponential back-off with jitter, capped at 30 s). On disconnect, the session classifies the interruption (see §6), and if the class is `network` or `signalling` it auto-reconnects unless `autoReconnect: false`. `media-permission` and explicit-disconnect do not auto-reconnect. Each attempt emits `'reconnect-attempt'`; success emits `'state-changed'`; final failure emits `'reconnect-failed'`.
|
|
|
|
---
|
|
|
|
## 6. Interruption classification
|
|
|
|
When a stream drops, classify to one of:
|
|
|
|
- `network` — ICE failure, DNS fail, no route.
|
|
- `media-permission` — `getUserMedia` revoked, OS audio device removed.
|
|
- `signalling` — signalling-server disconnect.
|
|
- `peer` — peer left explicitly, or peer's media track ended.
|
|
- `unknown` — anything else.
|
|
|
|
The classifier reads the underlying error / event and applies a rule set. Classification feeds both the reconnect decision and the UI toast.
|
|
|
|
---
|
|
|
|
## 7. Transport-share mode
|
|
|
|
In a collaborative listening session a designated peer drives transport state for everyone:
|
|
|
|
- Designation happens via `session.requestTransportShare(peerId)` from any client; the target peer accepts with `session.grantTransportShare()`.
|
|
- Once granted, the local transport's `play`/`pause`/`seek`/`setTempo` events are mirrored to a control channel on each peer's `PeerChannel`.
|
|
- Each non-driving peer's local transport receives the events and applies them; their UI controls become advisory (no-op `play`/`pause` calls).
|
|
- A driver disconnect transfers control back to "local"; UI surfaces this via toast.
|
|
|
|
---
|
|
|
|
## 8. WebSocket transport (server-relay fallback)
|
|
|
|
WebRTC is the default, but corporate networks and embedded contexts often need WebSocket-relayed audio. The `WebSocketTransport` packs raw PCM frames (or opus-encoded frames if `@suaveplan/audio-codec-ffmpeg-wasm` is available) and ships them through the `network-transport-websockets`. Latency is higher than WebRTC but it works through NATs that block UDP.
|
|
|
|
The transport is interchangeable; the session and source plugin do not care which.
|
|
|
|
---
|
|
|
|
## 9. Decisions and trade-offs
|
|
|
|
- **Per-peer source plugin = uniform consumer experience.** No special-case streaming code in `<PluginChainView>`, `useParameter`, `usePreset`, etc.
|
|
- **Transport-agnostic session.** Same code drives WebRTC, WebSocket, or Trystero.
|
|
- **FSM session lifecycle.** Avoids race conditions from imperative `if (state === 'X') …` checks; the FSM enforces legal transitions.
|
|
- **Interruption classification before reconnect decision.** Re-permission prompts are user-driven; we don't endlessly retry through them.
|
|
- **Transport-share via control channel, not engine state.** The driving peer's `Transport.on('play')` listener fires a control-channel message; receivers apply it locally. No special "shared transport" type — it's the local transport, just driven remotely.
|