SuavePlan/suaveplan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
suaveplan/openspec/archive/2026-05-02-voice/proposal.md
admin fbc70f96e6 Initial Commit
2026-05-27 23:29:53 +01:00

7.6 KiB
Raw Permalink Blame History

OpenSpec Change: voice

  • Change ID: voice
  • Package: @suaveplan/voice
  • Category: universal
  • Target path: universal/media/voice
  • Status: PROPOSED
  • Date: 2026-04-24
  • Reference spec: none — new capability

Summary

Deliver @suaveplan/voice on the genesis branch from scratch at enterprise quality. The package is the provider-agnostic voice core: abstract interfaces, shared types, Zod schemas, and a structured error hierarchy for speech synthesis (TTS), speech recognition (STT), voice management, sound-effect generation, and voice isolation. No provider-specific code, no HTTP client, no API keys. Concrete provider adapters (ElevenLabs, OpenAI, Google Cloud, Azure, Amazon Polly) live in sibling voice-* packages.

This package enables any downstream package — API plugins, React components, browser audio engines, or game narrative systems — to depend on voice contracts without pulling in provider SDKs or networking libraries.

Current-state audit (genesis)

The package does not exist on genesis yet — this is a fresh build.

Available workspace deps on genesis

Package Role
@suaveplan/error (workspace:^) VoiceError; VOICE.* error codes
@suaveplan/types (workspace:^) Clock, LoggerLike, CancellableOptions
@suaveplan/streams (workspace:^) ReadableStream / WritableStream helpers for audio streaming
zod (catalog:forms) Schema validation
tslib (catalog:build) Runtime helpers

Dev deps: vitest, @vitest/coverage-istanbul, fast-check (all catalog:testing).

Public API surface

Error hierarchy

VoiceError extends SuaveplanError with codes:

  • VOICE.TTS.SYNTHESIS_FAILED
  • VOICE.TTS.STREAM_INTERRUPTED
  • VOICE.STT.TRANSCRIPTION_FAILED
  • VOICE.STT.UNSUPPORTED_FORMAT
  • VOICE.VOICE.NOT_FOUND
  • VOICE.VOICE.CLONE_REJECTED
  • VOICE.EFFECTS.GENERATION_FAILED
  • VOICE.ISOLATION.FAILED
  • VOICE.PROVIDER.NOT_CONFIGURED
  • VOICE.PROVIDER.RATE_LIMITED
  • VOICE.VALIDATION.INVALID_REQUEST
  • VOICE.AUDIO.FORMAT_UNSUPPORTED

Shared types (schemas + TypeScript)

// types/common
AudioFormat: 'mp3_22050_32' | 'mp3_44100_128' | 'mp3_44100_192' | 'pcm_16000' | 'pcm_22050' | 'pcm_44100' | 'ulaw_8000' | 'wav_44100' | 'opus_48000' | 'aac_44100' | 'flac_44100'
LatencyMode: 0 | 1 | 2 | 3 | 4
VoiceCategory: 'premade' | 'cloned' | 'generated' | 'professional'
VoiceGender: 'male' | 'female' | 'neutral'
VoiceAge: 'young' | 'middle' | 'old'

VoiceSettings {
  stability?: number        // 0–1
  similarityBoost?: number  // 0–1
  style?: number            // 0–1
  speakerBoost?: boolean
}

Voice {
  id: string
  name: string
  category: VoiceCategory
  gender?: VoiceGender
  accent?: string
  age?: VoiceAge
  languageCodes?: string[]   // ISO 639-1
  description?: string
  previewUrl?: string
  settings?: VoiceSettings
  metadata?: Record<string, unknown>
}

ListVoicesOptions {
  category?: VoiceCategory
  languageCode?: string
  search?: string
  pageToken?: string
  pageSize?: number          // 1–100
}

VoiceList {
  voices: Voice[]
  nextPageToken?: string
  hasMore: boolean
}

AudioSource = Uint8Array | ReadableStream<Uint8Array> | string // URL or file path (provider decides)

TTS engine interface

interface VoiceSynthesizer {
  readonly providerName: string
  synthesize(options: SynthesisRequest): Promise<SynthesisResult>
  synthesizeStream(options: SynthesisRequest): ReadableStream<Uint8Array>
}

SynthesisRequest {
  text: string
  voiceId: string
  modelId?: string
  languageCode?: string
  voiceSettings?: VoiceSettings
  outputFormat?: AudioFormat
  latencyMode?: LatencyMode
  previousText?: string
  nextText?: string
  seed?: number
  pronunciationDictionaryIds?: string[]
  signal?: AbortSignal
}

SynthesisResult {
  audio: Uint8Array
  format: AudioFormat
  durationMs?: number
  charactersUsed?: number
  requestId?: string
}

STT engine interface

interface TranscriptionEngine {
  readonly providerName: string
  transcribe(options: TranscriptionRequest): Promise<TranscriptionResult>
}

TranscriptionRequest {
  audio: AudioSource
  modelId?: string
  languageCode?: string
  tagAudioEvents?: boolean
  numSpeakers?: number       // 1–32
  timestampsGranularity?: 'word' | 'character'
  diarize?: boolean
  diarizationThreshold?: number
  useMultiChannel?: boolean
  entityDetection?: string[] | 'all'
  entityRedaction?: string[] | 'all'
  keyterms?: string[]
  temperature?: number       // 0–2
  seed?: number
  signal?: AbortSignal
}

WordTimestamp {
  word: string
  startMs: number
  endMs: number
  speakerId?: string
  channelIndex?: number
  confidence?: number
}

SpeakerSegment {
  speakerId: string
  startMs: number
  endMs: number
}

DetectedEntity {
  text: string
  type: string
  startChar: number
  endChar: number
}

TranscriptionResult {
  text: string
  words?: WordTimestamp[]
  speakers?: SpeakerSegment[]
  entities?: DetectedEntity[]
  language?: string
  durationMs?: number
  requestId?: string
}

Voice repository interface

interface VoiceRepository {
  readonly providerName: string
  list(options?: ListVoicesOptions): Promise<VoiceList>
  get(voiceId: string): Promise<Voice | null>
  // Cloning is intentionally NOT in the base interface;
  // provider-specific clone APIs vary too widely.
}

Sound-effect engine interface

interface SoundEffectEngine {
  readonly providerName: string
  generate(options: SoundEffectRequest): Promise<SoundEffectResult>
}

SoundEffectRequest {
  text: string
  durationSeconds?: number   // 0.5–30
  loop?: boolean
  promptInfluence?: number   // 0–1
  modelId?: string
  outputFormat?: AudioFormat
  signal?: AbortSignal
}

SoundEffectResult {
  audio: Uint8Array
  format: AudioFormat
  durationMs?: number
  requestId?: string
}

Voice-isolation engine interface

interface VoiceIsolationEngine {
  readonly providerName: string
  isolate(options: IsolationRequest): Promise<IsolationResult>
}

IsolationRequest {
  audio: AudioSource
  outputFormat?: AudioFormat
  signal?: AbortSignal
}

IsolationResult {
  audio: Uint8Array
  format: AudioFormat
  requestId?: string
}

Design decisions

  1. No provider-specific types leak into the core. AudioSource is intentionally loose (Uint8Array | ReadableStream | string) so providers can accept URLs, file paths, or raw bytes. The provider validates and narrows at runtime.
  2. Streaming is TTS-only in the base interface. STT streaming, real-time conversational AI, and duplex WebSocket audio are provider-specific extensions that will ship in per-provider /streaming subpaths, not in the core contract.
  3. Voice cloning is excluded from VoiceRepository. Clone APIs (sample upload, verification, fine-tuning status) vary radically across providers. A VoiceCloner interface may be added later in a provider-specific subpath or a future core revision.
  4. Voice changing is excluded from the core. It is an ElevenLabs-specific feature with no clear generic equivalent in other providers.
  5. Conversational AI agents are excluded from the core. They are a product-level abstraction above raw TTS/STT and will ship as a separate package if needed.
  6. @suaveplan/types stays pure. Voice types live in @suaveplan/voice/types/common so API plugins can depend on the voice contract without circular issues, while @suaveplan/types remains a zero-runtime foundation.
  7. All async entrypoints accept AbortSignal. Every engine method propagates cancellation to provider adapters.
  8. All engines expose readonly providerName. This enables observability tags and debugging without instanceof checks.
Reference in New Issue View Git Blame Copy Permalink