suaveplan/openspec/archive/2026-05-02-voice/proposal.md

# OpenSpec Change: voice

- **Change ID**: voice
- **Package**: `@suaveplan/voice`
- **Category**: universal
- **Target path**: `universal/media/voice`
- **Status**: PROPOSED
- **Date**: 2026-04-24
- **Reference spec**: none — new capability

---

## Summary

Deliver `@suaveplan/voice` on the genesis branch from scratch at enterprise quality. The package is the provider-agnostic voice core: abstract interfaces, shared types, Zod schemas, and a structured error hierarchy for speech synthesis (TTS), speech recognition (STT), voice management, sound-effect generation, and voice isolation. No provider-specific code, no HTTP client, no API keys. Concrete provider adapters (ElevenLabs, OpenAI, Google Cloud, Azure, Amazon Polly) live in sibling `voice-*` packages.

This package enables any downstream package — API plugins, React components, browser audio engines, or game narrative systems — to depend on voice contracts without pulling in provider SDKs or networking libraries.

---

## Current-state audit (genesis)

The package does not exist on genesis yet — this is a fresh build.

---

## Available workspace deps on genesis

| Package | Role |
|---|---|
| `@suaveplan/error` (`workspace:^`) | `VoiceError`; `VOICE.*` error codes |
| `@suaveplan/types` (`workspace:^`) | `Clock`, `LoggerLike`, `CancellableOptions` |
| `@suaveplan/streams` (`workspace:^`) | `ReadableStream` / `WritableStream` helpers for audio streaming |
| `zod` (`catalog:forms`) | Schema validation |
| `tslib` (`catalog:build`) | Runtime helpers |

Dev deps: `vitest`, `@vitest/coverage-istanbul`, `fast-check` (all `catalog:testing`).

---

## Public API surface

### Error hierarchy

`VoiceError extends SuaveplanError` with codes:

- `VOICE.TTS.SYNTHESIS_FAILED`
- `VOICE.TTS.STREAM_INTERRUPTED`
- `VOICE.STT.TRANSCRIPTION_FAILED`
- `VOICE.STT.UNSUPPORTED_FORMAT`
- `VOICE.VOICE.NOT_FOUND`
- `VOICE.VOICE.CLONE_REJECTED`
- `VOICE.EFFECTS.GENERATION_FAILED`
- `VOICE.ISOLATION.FAILED`
- `VOICE.PROVIDER.NOT_CONFIGURED`
- `VOICE.PROVIDER.RATE_LIMITED`
- `VOICE.VALIDATION.INVALID_REQUEST`
- `VOICE.AUDIO.FORMAT_UNSUPPORTED`

### Shared types (schemas + TypeScript)

```ts
// types/common
AudioFormat: 'mp3_22050_32' | 'mp3_44100_128' | 'mp3_44100_192' | 'pcm_16000' | 'pcm_22050' | 'pcm_44100' | 'ulaw_8000' | 'wav_44100' | 'opus_48000' | 'aac_44100' | 'flac_44100'
LatencyMode: 0 | 1 | 2 | 3 | 4
VoiceCategory: 'premade' | 'cloned' | 'generated' | 'professional'
VoiceGender: 'male' | 'female' | 'neutral'
VoiceAge: 'young' | 'middle' | 'old'

VoiceSettings {
  stability?: number        // 0–1
  similarityBoost?: number  // 0–1
  style?: number            // 0–1
  speakerBoost?: boolean
}

Voice {
  id: string
  name: string
  category: VoiceCategory
  gender?: VoiceGender
  accent?: string
  age?: VoiceAge
  languageCodes?: string[]   // ISO 639-1
  description?: string
  previewUrl?: string
  settings?: VoiceSettings
  metadata?: Record<string, unknown>
}

ListVoicesOptions {
  category?: VoiceCategory
  languageCode?: string
  search?: string
  pageToken?: string
  pageSize?: number          // 1–100
}

VoiceList {
  voices: Voice[]
  nextPageToken?: string
  hasMore: boolean
}

AudioSource = Uint8Array | ReadableStream<Uint8Array> | string // URL or file path (provider decides)
```

### TTS engine interface

```ts
interface VoiceSynthesizer {
  readonly providerName: string
  synthesize(options: SynthesisRequest): Promise<SynthesisResult>
  synthesizeStream(options: SynthesisRequest): ReadableStream<Uint8Array>
}

SynthesisRequest {
  text: string
  voiceId: string
  modelId?: string
  languageCode?: string
  voiceSettings?: VoiceSettings
  outputFormat?: AudioFormat
  latencyMode?: LatencyMode
  previousText?: string
  nextText?: string
  seed?: number
  pronunciationDictionaryIds?: string[]
  signal?: AbortSignal
}

SynthesisResult {
  audio: Uint8Array
  format: AudioFormat
  durationMs?: number
  charactersUsed?: number
  requestId?: string
}
```

### STT engine interface

```ts
interface TranscriptionEngine {
  readonly providerName: string
  transcribe(options: TranscriptionRequest): Promise<TranscriptionResult>
}

TranscriptionRequest {
  audio: AudioSource
  modelId?: string
  languageCode?: string
  tagAudioEvents?: boolean
  numSpeakers?: number       // 1–32
  timestampsGranularity?: 'word' | 'character'
  diarize?: boolean
  diarizationThreshold?: number
  useMultiChannel?: boolean
  entityDetection?: string[] | 'all'
  entityRedaction?: string[] | 'all'
  keyterms?: string[]
  temperature?: number       // 0–2
  seed?: number
  signal?: AbortSignal
}

WordTimestamp {
  word: string
  startMs: number
  endMs: number
  speakerId?: string
  channelIndex?: number
  confidence?: number
}

SpeakerSegment {
  speakerId: string
  startMs: number
  endMs: number
}

DetectedEntity {
  text: string
  type: string
  startChar: number
  endChar: number
}

TranscriptionResult {
  text: string
  words?: WordTimestamp[]
  speakers?: SpeakerSegment[]
  entities?: DetectedEntity[]
  language?: string
  durationMs?: number
  requestId?: string
}
```

### Voice repository interface

```ts
interface VoiceRepository {
  readonly providerName: string
  list(options?: ListVoicesOptions): Promise<VoiceList>
  get(voiceId: string): Promise<Voice | null>
  // Cloning is intentionally NOT in the base interface;
  // provider-specific clone APIs vary too widely.
}
```

### Sound-effect engine interface

```ts
interface SoundEffectEngine {
  readonly providerName: string
  generate(options: SoundEffectRequest): Promise<SoundEffectResult>
}

SoundEffectRequest {
  text: string
  durationSeconds?: number   // 0.5–30
  loop?: boolean
  promptInfluence?: number   // 0–1
  modelId?: string
  outputFormat?: AudioFormat
  signal?: AbortSignal
}

SoundEffectResult {
  audio: Uint8Array
  format: AudioFormat
  durationMs?: number
  requestId?: string
}
```

### Voice-isolation engine interface

```ts
interface VoiceIsolationEngine {
  readonly providerName: string
  isolate(options: IsolationRequest): Promise<IsolationResult>
}

IsolationRequest {
  audio: AudioSource
  outputFormat?: AudioFormat
  signal?: AbortSignal
}

IsolationResult {
  audio: Uint8Array
  format: AudioFormat
  requestId?: string
}
```

---

## Design decisions

1. **No provider-specific types leak into the core.** `AudioSource` is intentionally loose (`Uint8Array | ReadableStream | string`) so providers can accept URLs, file paths, or raw bytes. The provider validates and narrows at runtime.
2. **Streaming is TTS-only in the base interface.** STT streaming, real-time conversational AI, and duplex WebSocket audio are provider-specific extensions that will ship in per-provider `/streaming` subpaths, not in the core contract.
3. **Voice cloning is excluded from `VoiceRepository`.** Clone APIs (sample upload, verification, fine-tuning status) vary radically across providers. A `VoiceCloner` interface may be added later in a provider-specific subpath or a future core revision.
4. **Voice changing is excluded from the core.** It is an ElevenLabs-specific feature with no clear generic equivalent in other providers.
5. **Conversational AI agents are excluded from the core.** They are a product-level abstraction above raw TTS/STT and will ship as a separate package if needed.
6. **`@suaveplan/types` stays pure.** Voice types live in `@suaveplan/voice/types/common` so API plugins can depend on the voice contract without circular issues, while `@suaveplan/types` remains a zero-runtime foundation.
7. **All async entrypoints accept `AbortSignal`.** Every engine method propagates cancellation to provider adapters.
8. **All engines expose `readonly providerName`.** This enables observability tags and debugging without `instanceof` checks.