SuavePlan/suaveplan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
suaveplan/openspec/archive/2026-05-22-voice-elevenlabs/proposal.md
admin fbc70f96e6 Initial Commit
2026-05-27 23:29:53 +01:00

14 KiB
Raw Permalink Blame History

OpenSpec Change: voice-elevenlabs

  • Change ID: voice-elevenlabs
  • Package: @suaveplan/voice-elevenlabs
  • Category: universal
  • Target path: packages/audio/voice-elevenlabs
  • Status: PROPOSED (rewritten 2026-05-02 — supersedes the 2026-04-24 fused-HTTP design)
  • Date: 2026-05-02
  • Depends on:
    • openspec/changes/voice/ (abstract core interfaces)
    • openspec/changes/third-party-elevenlabs/ (typed thin client; Wave 1 voice surface)
  • Reference spec: none — new capability (genesis-original)

Summary

Deliver @suaveplan/voice-elevenlabs on the genesis branch as a thin domain adapter that implements every @suaveplan/voice engine interface (VoiceSynthesizer, TranscriptionEngine, VoiceRepository, SoundEffectEngine, VoiceIsolationEngine) by delegating to @suaveplan/third-party-elevenlabs. It owns no HTTP, no auth, no retry, no rate-limit handling, no OTel transport spans, and no Zod schemas for ElevenLabs DTOs — those all live in the third-party package. Its sole responsibilities are:

  1. DTO ↔ domain mapping. ElevenLabs-shaped DTOs from @suaveplan/third-party-elevenlabs become @suaveplan/voice types (Voice, SynthesisResult, TranscriptionResult, SoundEffectResult, IsolationResult).
  2. Error remapping. ElevenLabsApiError subclasses are remapped to VoiceError codes per the voice-domain semantics (e.g. THIRD_PARTY_ELEVENLABS.REQUEST.NOT_FOUND from voices.get(id) → VOICE.VOICE.NOT_FOUND).
  3. Multipart and stream plumbing for the voice surface. STT audio sources (Uint8Array | ReadableStream | string) are converted to the MultipartFile envelope expected by @suaveplan/third-party-elevenlabs; streaming TTS bodies pass through unchanged.
  4. Composite factory. createElevenLabsVoice(options) returns the five engines plus the underlying ElevenLabsClient for callers who need to escape to lower levels.

Why this design supersedes the 2026-04-24 proposal

The original proposal fused HTTP plumbing into the adapter. That design failed three forward-looking constraints:

  1. ElevenLabs surfaces non-voice domains. Music, Dubbing, Studio, Audio Native, Conversational AI, Pronunciation Dictionaries, History, Workspace administration. A voice-only adapter cannot host those.
  2. Every future ElevenLabs adapter would re-implement the same fetch wrapper, retry policy, rate-limit parsing, and OTel instrumentation. That is duplication on a scale we are not willing to accept.
  3. The 2026-04-24 implementation diverged from its own spec on multiple load-bearing points. No @suaveplan/retry integration despite the dep being declared. No X-RateLimit-Remaining parsing. 5xx incorrectly mapped to ProviderNotConfiguredError. No OTel spans/counters/histograms. The fused design encouraged these omissions because the package was implementing too much at once.

The rewrite splits responsibilities: @suaveplan/third-party-elevenlabs owns transport correctness; @suaveplan/voice-elevenlabs owns domain mapping. Both are testable independently.

Out of scope

  • Any HTTP, auth, retry, OTel transport spans, or rate-limit logic. Those live in @suaveplan/third-party-elevenlabs.
  • Voice cloning. ElevenLabs clone APIs (instant-clone, professional-clone, voice-design) ship in Wave 2 of the third-party client (third-party-elevenlabs-voice-extensions); the corresponding voice-domain adapter will land via a follow-up change once @suaveplan/voice adds a VoiceCloner interface.
  • Conversational AI. Ships under a separate package @suaveplan/conversational-elevenlabs against a future @suaveplan/conversational core.
  • Music, Dubbing, Studio, Audio Native. Each gets its own domain core + provider adapter pair when the domain is in scope.
  • Speech-to-speech (voice changer). The third-party client exposes it in Wave 1, but @suaveplan/voice does not yet have a SpeechConverter interface. When it does, this adapter gains a convert() method.

Available workspace deps on genesis

Package Role
@suaveplan/voice (workspace:^) Abstract interfaces, types, schemas, VoiceError codes
@suaveplan/third-party-elevenlabs (workspace:^) Typed REST/WebSocket client, ElevenLabs DTOs, transport-level errors
@suaveplan/error (workspace:^) SuaveplanError base + cause-chain preservation when remapping
@suaveplan/types (workspace:^) Clock, LoggerLike
@suaveplan/streams (workspace:^) Stream helpers (re-exported from third-party for convenience-free access)
zod (catalog:forms) Mapping-result validation in tests
tslib (catalog:build) Runtime helpers

Dev deps: @suaveplan/testing (workspace:^, dev-only), vitest, @vitest/coverage-istanbul, fast-check, mitata (all catalog:testing).

@opentelemetry/api is not a direct dep of this package. Transport-level OTel instrumentation lives in @suaveplan/third-party-elevenlabs. The adapter does not emit its own spans because it adds no I/O on top of the client; if a future scenario justifies adapter-level spans, they will be added under a follow-up change with explicit motivation.

Public API surface

Engine implementations

class ElevenLabsTtsEngine implements VoiceSynthesizer {
  readonly providerName: 'elevenlabs'
  constructor(deps: { tts: ElevenLabsTts }, options?: ElevenLabsTtsAdapterOptions)
  synthesize(request: SynthesisRequest): Promise<SynthesisResult>
  synthesizeStream(request: SynthesisRequest): ReadableStream<Uint8Array>
}

class ElevenLabsSttEngine implements TranscriptionEngine {
  readonly providerName: 'elevenlabs'
  constructor(deps: { stt: ElevenLabsStt }, options?: ElevenLabsSttAdapterOptions)
  transcribe(request: TranscriptionRequest): Promise<TranscriptionResult>
}

class ElevenLabsVoiceRepository implements VoiceRepository {
  readonly providerName: 'elevenlabs'
  constructor(deps: { voices: ElevenLabsVoices })
  list(options?: ListVoicesOptions): Promise<VoiceList>
  get(voiceId: string): Promise<Voice | null>
}

class ElevenLabsSoundEffectEngine implements SoundEffectEngine {
  readonly providerName: 'elevenlabs'
  constructor(deps: { soundGeneration: ElevenLabsSoundGeneration }, options?: ElevenLabsSoundEffectAdapterOptions)
  generate(request: SoundEffectRequest): Promise<SoundEffectResult>
}

class ElevenLabsVoiceIsolationEngine implements VoiceIsolationEngine {
  readonly providerName: 'elevenlabs'
  constructor(deps: { audioIsolation: ElevenLabsAudioIsolation })
  isolate(request: IsolationRequest): Promise<IsolationResult>
}

ElevenLabsTtsAdapterOptions carries voice-specific defaults (e.g. defaultModelId: 'eleven_multilingual_v2', defaultVoiceSettings: VoiceSettings). The adapter never owns the API key or the base URL — those belong to the ElevenLabsClient injected upstream.

Composite factory

function createElevenLabsVoice(options: CreateElevenLabsVoiceOptions): {
  client: ElevenLabsClient
  tts: ElevenLabsTtsEngine
  stt: ElevenLabsSttEngine
  voices: ElevenLabsVoiceRepository
  effects: ElevenLabsSoundEffectEngine
  isolation: ElevenLabsVoiceIsolationEngine
}

interface CreateElevenLabsVoiceOptions {
  // Either an existing client...
  client?: ElevenLabsClient
  // ...or the inputs needed to construct one (forwarded to ElevenLabsClient unchanged)
  apiKey?: string
  baseUrl?: string
  timeoutMs?: number
  retryPolicy?: RetryPolicy
  fetchImpl?: typeof fetch
  webSocketImpl?: WebSocketImpl
  clock?: Clock
  logger?: LoggerLike
  events?: EventBusLike<ElevenLabsClientEvents>
  userAgent?: string
  defaultHeaders?: Record<string, string>

  // Adapter-specific defaults
  defaultModelId?: string
  defaultVoiceSettings?: VoiceSettings
  defaultSoundEffectModelId?: string
}

If client is provided, apiKey and the other client options are ignored; otherwise the factory constructs a fresh ElevenLabsClient. Either way, the adapter classes share the same client instance — so retry, rate-limit accounting, and OTel are unified across engines.

ElevenLabs API mapping

Voice interface Third-party module Endpoint
VoiceSynthesizer.synthesize tts.synthesize(voiceId, body) POST /v1/text-to-speech/{id}
VoiceSynthesizer.synthesizeStream tts.synthesizeStream(voiceId, body) POST /v1/text-to-speech/{id}/stream
TranscriptionEngine.transcribe stt.transcribe(body) POST /v1/speech-to-text (multipart)
VoiceRepository.list voices.list(params) GET /v1/voices
VoiceRepository.get voices.get(voiceId) GET /v1/voices/{id}
SoundEffectEngine.generate soundGeneration.generate(body) POST /v1/sound-generation
VoiceIsolationEngine.isolate audioIsolation.isolate(body) POST /v1/audio-isolation

Error remapping table

The adapter remaps every ElevenLabsApiError subclass to a VoiceError code and preserves the original via cause:

Third-party code Voice domain code Notes
AUTH.UNAUTHORIZED VOICE.PROVIDER.NOT_CONFIGURED The API key is invalid or missing
AUTH.FORBIDDEN VOICE.PROVIDER.NOT_CONFIGURED Key valid but lacks scope
REQUEST.NOT_FOUND (from voices.get) VOICE.VOICE.NOT_FOUND With context.voiceId populated
REQUEST.NOT_FOUND (other endpoints) VOICE.VALIDATION.INVALID_REQUEST Generic resource-missing
REQUEST.VALIDATION_FAILED VOICE.VALIDATION.INVALID_REQUEST Body preserved on cause.context
REQUEST.INVALID VOICE.VALIDATION.INVALID_REQUEST
RATE_LIMIT.EXCEEDED VOICE.PROVIDER.RATE_LIMITED retryAfterMs preserved
RATE_LIMIT.QUOTA_EXHAUSTED VOICE.PROVIDER.RATE_LIMITED
SERVER.TRANSIENT (TTS endpoints) VOICE.TTS.SYNTHESIS_FAILED
SERVER.TRANSIENT (STT endpoints) VOICE.STT.TRANSCRIPTION_FAILED
SERVER.TRANSIENT (sound-generation) VOICE.EFFECTS.GENERATION_FAILED
SERVER.TRANSIENT (audio-isolation) VOICE.ISOLATION.FAILED
SERVER.UNAVAILABLE Same as SERVER.TRANSIENT per endpoint
STREAM.INTERRUPTED VOICE.TTS.STREAM_INTERRUPTED
RESPONSE.MALFORMED VOICE.PROVIDER.NOT_CONFIGURED Indicates an ElevenLabs API contract change
RESPONSE.UNSUPPORTED_FORMAT VOICE.AUDIO.FORMAT_UNSUPPORTED
NETWORK.* VOICE.PROVIDER.NOT_CONFIGURED (timeout/unreachable) or VOICE.TTS.STREAM_INTERRUPTED (during streams)

Every remapped error preserves the original via cause so callers can inspect the underlying transport detail when needed.

DTO mapping decisions

  1. VoiceCategory mapping. ElevenLabs returns category: 'premade' | 'cloned' | 'generated' | 'professional' directly compatible with @suaveplan/voice enum — no remapping needed.
  2. Voice.languageCodes. Aggregated from verified_languages (BCP-47) and labels.language if present. De-duplicated and ISO-639-normalised.
  3. Voice.gender / Voice.age / Voice.accent. Read from labels.gender, labels.age, labels.accent. Unrecognised values fall back to metadata.
  4. SynthesisResult.charactersUsed. Read from response metadata.characterCost (third-party client surfaces X-Character-Cost header).
  5. TranscriptionResult.words. Mapped from ElevenLabs words[] array, dropping non-word entries (type !== 'word' enters entities instead).
  6. TranscriptionResult.speakers. Derived by collapsing consecutive words[] entries with the same speaker_id into {startMs, endMs, speakerId} segments.
  7. AudioFormat round-trip. All @suaveplan/voice AudioFormat strings (mp3_22050_32, pcm_16000, etc.) map to ElevenLabs output_format values 1:1 — verified by property test.

Parity Checklist

Genesis-original package — the prior packages/audio/voice-elevenlabs implementation (2026-05-01) is being deleted and rewritten, not migrated, because the architecture changes substantially. The deleted code is superseded by the splits described in ## Why this design supersedes the 2026-04-24 proposal. No upstream @suaveplan/voice-elevenlabs exists in suaveplan main.

Capability Prior fused build This adapter Notes
TTS synthesize ✅ partial ✅ Now via tts.synthesize
TTS streaming ✅ partial ✅ Now via tts.synthesizeStream
STT transcribe ✅ partial ✅ Now via stt.transcribe
Voice list/get ✅ partial ✅ Now via voices.list / voices.get
Sound generation ✅ partial ✅ Now via soundGeneration.generate
Audio isolation ✅ partial ✅ Now via audioIsolation.isolate
Retry / rate-limit 🚫 ✅ Was missing in prior build; now provided by third-party client
OTel 🚫 ✅ Was missing in prior build; now provided by third-party client
X-RateLimit-Remaining 🚫 ✅ Was missing in prior build; now provided by third-party client
X-Character-Cost 🚫 ✅ Was missing in prior build; now provided by third-party client
5xx mapped to transient 🚫 (incorrectly mapped to PROVIDER.NOT_CONFIGURED) ✅ Per error remapping table

Impact

Affected dependents

Run bunx turbo ls --filter=...@suaveplan/voice-elevenlabs to enumerate. Currently no shipped genesis package consumes @suaveplan/voice-elevenlabs (the rebuild lands before any downstream takes a dep on it).

  • Future @suaveplan/api-plugin-voice — [ ] tests run / [ ] docs reviewed / [x] no action needed (does not yet exist on genesis)
  • Future @suaveplan/react-voice — [ ] tests run / [ ] docs reviewed / [x] no action needed (does not yet exist on genesis)

Migration

The 2026-05-01 fused-HTTP build is deleted in this change. Any in-progress consumer code that imported from @suaveplan/voice-elevenlabs must:

  • Old: import { ElevenLabsClient } from '@suaveplan/voice-elevenlabs/client/...'

  • New: import { ElevenLabsClient } from '@suaveplan/third-party-elevenlabs'

  • Old: import { createElevenLabsVoice } from '@suaveplan/voice-elevenlabs/factory/factory'

  • New: Same path, same return shape — createElevenLabsVoice(options) is preserved with identical signature except that apiKey / baseUrl / timeoutMs are now forwarded to ElevenLabsClient rather than consumed locally, and the returned object now includes client: ElevenLabsClient.

Search command for downstream consumers:

grep -rln "@suaveplan/voice-elevenlabs/client" packages/ | grep -v "/dist/\|/coverage/\|/voice-elevenlabs/"

If any hits exist, update them to import from @suaveplan/third-party-elevenlabs before merging.

Reference in New Issue View Git Blame Copy Permalink