Design — enterprise-react-audio-stream

@suaveplan/react-audio-stream provides real-time audio streaming over the unified architecture. Its core insight: an incoming peer audio stream is just another source plugin on a dedicated chain, indistinguishable to the rest of the system from a local AudioPlayerPlugin. That uniformity gets us FX chains per peer, sidechain ducking from peers, analyser readouts per peer, and snapshot/restore of "studio sessions" for free — without the streaming code having to know about any of it.

1. Position in the audio family

        consumer React app
                │
                ▼
        react-audio-stream         ← this package
          (StreamSession + per-peer source plugin + transports)
                │
                ▼
        react-audio                (provider + hooks)
                │
                ▼
        audio-engine-browser       (TimelineAdapter + chains)

Per-peer sources are inserted on dedicated tracks in the engine's timeline. Track ids follow the convention peer:<participantId>. The chain owns FX; the source plugin owns playback of the incoming MediaStream.

2. `StreamSession` — transport-agnostic core

class StreamSession {
  constructor(opts: StreamSessionOptions)

  connect(roomId: string): Promise<void>
  join(): Promise<void>
  leave(): Promise<void>
  disconnect(): Promise<void>

  readonly state: SessionState                // FSM via @suaveplan/fsm
  readonly peers: ReadonlyMap<string, PeerChannel>

  on(event: SessionEvent, listener): UnsubscribeFn
  // 'state-changed' | 'peer-joined' | 'peer-left' | 'peer-stream-added'
  // | 'peer-stream-removed' | 'transport-share-request' | 'transport-share-granted'
  // | 'interruption' | 'reconnect-attempt' | 'reconnect-failed'
}

The session does not speak any specific transport. It delegates negotiation, ICE, signalling, and stream attachment to an injected StreamTransport adapter (WebRTCTransport, WebSocketTransport, TrysteroTransport). The same session can run identically against any transport.

3. Per-peer source plugin

When a peer adds a stream, the session emits 'peer-stream-added' with { participantId, stream: MediaStream }. The provider listens, instantiates a StreamSourcePlugin (an AudioFxPlugin source — io.inputs[0].count === 0, captures from a MediaStream), inserts it into the chain of a new peer:<participantId> track, and exposes it via usePeerStreams().

class StreamSourcePlugin implements AudioFxPlugin {
  readonly io = { inputs: [{ count: 0 }], outputs: [{ count: 2, layout: 'stereo' }] }
  // browser path: createAudioNode wraps MediaStreamSourceNode
  // server path: not applicable (server doesn't have MediaStream)
  setSourceStream(stream: MediaStream): void
  process(_in, output, ctx) {
    // browser worklet path mirrors AudioNode path's behaviour
  }
}

The plugin is just like every other source plugin — <PluginChainView> renders it generically, useParameter writes its volume, the chain can have FX after it, the bus registry can read its post-FX output.

4. FSM session lifecycle

@suaveplan/fsm drives the session through:

idle → connecting → connected (idle in room) → joining → joined (streaming)
                                  │                  │
                                  └─→ disconnecting ─┴─→ disconnected
                                                          ↑
                                          (reconnect attempt brings us back to connecting)

Transitions:

connect(roomId) from idle → connecting → connected
join() from connected → joining → joined
leave() from joined → connected
disconnect() from any → disconnecting → disconnected
Network failure from connected/joined → disconnected (auto-reconnect triggers)

Illegal transitions throw AudioError with code REACT_AUDIO_STREAM.INVALID_STATE.

5. Auto-reconnect

Backed by @suaveplan/retry (exponential back-off with jitter, capped at 30 s). On disconnect, the session classifies the interruption (see §6), and if the class is network or signalling it auto-reconnects unless autoReconnect: false. media-permission and explicit-disconnect do not auto-reconnect. Each attempt emits 'reconnect-attempt'; success emits 'state-changed'; final failure emits 'reconnect-failed'.

6. Interruption classification

When a stream drops, classify to one of:

network — ICE failure, DNS fail, no route.
media-permission — getUserMedia revoked, OS audio device removed.
signalling — signalling-server disconnect.
peer — peer left explicitly, or peer's media track ended.
unknown — anything else.

The classifier reads the underlying error / event and applies a rule set. Classification feeds both the reconnect decision and the UI toast.

In a collaborative listening session a designated peer drives transport state for everyone:

Designation happens via session.requestTransportShare(peerId) from any client; the target peer accepts with session.grantTransportShare().
Once granted, the local transport's play/pause/seek/setTempo events are mirrored to a control channel on each peer's PeerChannel.
Each non-driving peer's local transport receives the events and applies them; their UI controls become advisory (no-op play/pause calls).
A driver disconnect transfers control back to "local"; UI surfaces this via toast.

8. WebSocket transport (server-relay fallback)

WebRTC is the default, but corporate networks and embedded contexts often need WebSocket-relayed audio. The WebSocketTransport packs raw PCM frames (or opus-encoded frames if @suaveplan/audio-codec-ffmpeg-wasm is available) and ships them through the network-transport-websockets. Latency is higher than WebRTC but it works through NATs that block UDP.

The transport is interchangeable; the session and source plugin do not care which.

9. Decisions and trade-offs

Per-peer source plugin = uniform consumer experience. No special-case streaming code in <PluginChainView>, useParameter, usePreset, etc.
Transport-agnostic session. Same code drives WebRTC, WebSocket, or Trystero.
FSM session lifecycle. Avoids race conditions from imperative if (state === 'X') … checks; the FSM enforces legal transitions.
Interruption classification before reconnect decision. Re-permission prompts are user-driven; we don't endlessly retry through them.
Transport-share via control channel, not engine state. The driving peer's Transport.on('play') listener fires a control-channel message; receivers apply it locally. No special "shared transport" type — it's the local transport, just driven remotely.

6.8 KiB Raw Permalink Blame History