Initial Commit
This commit is contained in:
245
openspec/archive/2026-05-22-voice-elevenlabs/proposal.md
Normal file
245
openspec/archive/2026-05-22-voice-elevenlabs/proposal.md
Normal file
@@ -0,0 +1,245 @@
|
||||
# OpenSpec Change: voice-elevenlabs
|
||||
|
||||
- **Change ID**: voice-elevenlabs
|
||||
- **Package**: `@suaveplan/voice-elevenlabs`
|
||||
- **Category**: universal
|
||||
- **Target path**: `packages/audio/voice-elevenlabs`
|
||||
- **Status**: PROPOSED (rewritten 2026-05-02 — supersedes the 2026-04-24 fused-HTTP design)
|
||||
- **Date**: 2026-05-02
|
||||
- **Depends on**:
|
||||
- `openspec/changes/voice/` (abstract core interfaces)
|
||||
- `openspec/changes/third-party-elevenlabs/` (typed thin client; Wave 1 voice surface)
|
||||
- **Reference spec**: none — new capability (genesis-original)
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Deliver `@suaveplan/voice-elevenlabs` on the genesis branch as a **thin domain adapter** that implements every `@suaveplan/voice` engine interface (`VoiceSynthesizer`, `TranscriptionEngine`, `VoiceRepository`, `SoundEffectEngine`, `VoiceIsolationEngine`) by delegating to `@suaveplan/third-party-elevenlabs`. It owns no HTTP, no auth, no retry, no rate-limit handling, no OTel transport spans, and no Zod schemas for ElevenLabs DTOs — those all live in the third-party package. Its sole responsibilities are:
|
||||
|
||||
1. **DTO ↔ domain mapping.** ElevenLabs-shaped DTOs from `@suaveplan/third-party-elevenlabs` become `@suaveplan/voice` types (`Voice`, `SynthesisResult`, `TranscriptionResult`, `SoundEffectResult`, `IsolationResult`).
|
||||
2. **Error remapping.** `ElevenLabsApiError` subclasses are remapped to `VoiceError` codes per the voice-domain semantics (e.g. `THIRD_PARTY_ELEVENLABS.REQUEST.NOT_FOUND` from `voices.get(id)` → `VOICE.VOICE.NOT_FOUND`).
|
||||
3. **Multipart and stream plumbing for the voice surface.** STT audio sources (`Uint8Array | ReadableStream | string`) are converted to the `MultipartFile` envelope expected by `@suaveplan/third-party-elevenlabs`; streaming TTS bodies pass through unchanged.
|
||||
4. **Composite factory.** `createElevenLabsVoice(options)` returns the five engines plus the underlying `ElevenLabsClient` for callers who need to escape to lower levels.
|
||||
|
||||
---
|
||||
|
||||
## Why this design supersedes the 2026-04-24 proposal
|
||||
|
||||
The original proposal fused HTTP plumbing into the adapter. That design failed three forward-looking constraints:
|
||||
|
||||
1. **ElevenLabs surfaces non-voice domains.** Music, Dubbing, Studio, Audio Native, Conversational AI, Pronunciation Dictionaries, History, Workspace administration. A voice-only adapter cannot host those.
|
||||
2. **Every future ElevenLabs adapter would re-implement the same `fetch` wrapper, retry policy, rate-limit parsing, and OTel instrumentation.** That is duplication on a scale we are not willing to accept.
|
||||
3. **The 2026-04-24 implementation diverged from its own spec on multiple load-bearing points.** No `@suaveplan/retry` integration despite the dep being declared. No `X-RateLimit-Remaining` parsing. 5xx incorrectly mapped to `ProviderNotConfiguredError`. No OTel spans/counters/histograms. The fused design encouraged these omissions because the package was implementing too much at once.
|
||||
|
||||
The rewrite splits responsibilities: `@suaveplan/third-party-elevenlabs` owns transport correctness; `@suaveplan/voice-elevenlabs` owns domain mapping. Both are testable independently.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Any HTTP, auth, retry, OTel transport spans, or rate-limit logic. Those live in `@suaveplan/third-party-elevenlabs`.
|
||||
- Voice cloning. ElevenLabs clone APIs (instant-clone, professional-clone, voice-design) ship in Wave 2 of the third-party client (`third-party-elevenlabs-voice-extensions`); the corresponding voice-domain adapter will land via a follow-up change once `@suaveplan/voice` adds a `VoiceCloner` interface.
|
||||
- Conversational AI. Ships under a separate package `@suaveplan/conversational-elevenlabs` against a future `@suaveplan/conversational` core.
|
||||
- Music, Dubbing, Studio, Audio Native. Each gets its own domain core + provider adapter pair when the domain is in scope.
|
||||
- Speech-to-speech (voice changer). The third-party client exposes it in Wave 1, but `@suaveplan/voice` does not yet have a `SpeechConverter` interface. When it does, this adapter gains a `convert()` method.
|
||||
|
||||
---
|
||||
|
||||
## Available workspace deps on genesis
|
||||
|
||||
| Package | Role |
|
||||
|---|---|
|
||||
| `@suaveplan/voice` (`workspace:^`) | Abstract interfaces, types, schemas, `VoiceError` codes |
|
||||
| `@suaveplan/third-party-elevenlabs` (`workspace:^`) | Typed REST/WebSocket client, ElevenLabs DTOs, transport-level errors |
|
||||
| `@suaveplan/error` (`workspace:^`) | `SuaveplanError` base + cause-chain preservation when remapping |
|
||||
| `@suaveplan/types` (`workspace:^`) | `Clock`, `LoggerLike` |
|
||||
| `@suaveplan/streams` (`workspace:^`) | Stream helpers (re-exported from third-party for convenience-free access) |
|
||||
| `zod` (`catalog:forms`) | Mapping-result validation in tests |
|
||||
| `tslib` (`catalog:build`) | Runtime helpers |
|
||||
|
||||
Dev deps: `@suaveplan/testing` (`workspace:^`, dev-only), `vitest`, `@vitest/coverage-istanbul`, `fast-check`, `mitata` (all `catalog:testing`).
|
||||
|
||||
`@opentelemetry/api` is **not** a direct dep of this package. Transport-level OTel instrumentation lives in `@suaveplan/third-party-elevenlabs`. The adapter does not emit its own spans because it adds no I/O on top of the client; if a future scenario justifies adapter-level spans, they will be added under a follow-up change with explicit motivation.
|
||||
|
||||
---
|
||||
|
||||
## Public API surface
|
||||
|
||||
### Engine implementations
|
||||
|
||||
```ts
|
||||
class ElevenLabsTtsEngine implements VoiceSynthesizer {
|
||||
readonly providerName: 'elevenlabs'
|
||||
constructor(deps: { tts: ElevenLabsTts }, options?: ElevenLabsTtsAdapterOptions)
|
||||
synthesize(request: SynthesisRequest): Promise<SynthesisResult>
|
||||
synthesizeStream(request: SynthesisRequest): ReadableStream<Uint8Array>
|
||||
}
|
||||
|
||||
class ElevenLabsSttEngine implements TranscriptionEngine {
|
||||
readonly providerName: 'elevenlabs'
|
||||
constructor(deps: { stt: ElevenLabsStt }, options?: ElevenLabsSttAdapterOptions)
|
||||
transcribe(request: TranscriptionRequest): Promise<TranscriptionResult>
|
||||
}
|
||||
|
||||
class ElevenLabsVoiceRepository implements VoiceRepository {
|
||||
readonly providerName: 'elevenlabs'
|
||||
constructor(deps: { voices: ElevenLabsVoices })
|
||||
list(options?: ListVoicesOptions): Promise<VoiceList>
|
||||
get(voiceId: string): Promise<Voice | null>
|
||||
}
|
||||
|
||||
class ElevenLabsSoundEffectEngine implements SoundEffectEngine {
|
||||
readonly providerName: 'elevenlabs'
|
||||
constructor(deps: { soundGeneration: ElevenLabsSoundGeneration }, options?: ElevenLabsSoundEffectAdapterOptions)
|
||||
generate(request: SoundEffectRequest): Promise<SoundEffectResult>
|
||||
}
|
||||
|
||||
class ElevenLabsVoiceIsolationEngine implements VoiceIsolationEngine {
|
||||
readonly providerName: 'elevenlabs'
|
||||
constructor(deps: { audioIsolation: ElevenLabsAudioIsolation })
|
||||
isolate(request: IsolationRequest): Promise<IsolationResult>
|
||||
}
|
||||
```
|
||||
|
||||
`ElevenLabsTtsAdapterOptions` carries voice-specific defaults (e.g. `defaultModelId: 'eleven_multilingual_v2'`, `defaultVoiceSettings: VoiceSettings`). The adapter never owns the API key or the base URL — those belong to the `ElevenLabsClient` injected upstream.
|
||||
|
||||
### Composite factory
|
||||
|
||||
```ts
|
||||
function createElevenLabsVoice(options: CreateElevenLabsVoiceOptions): {
|
||||
client: ElevenLabsClient
|
||||
tts: ElevenLabsTtsEngine
|
||||
stt: ElevenLabsSttEngine
|
||||
voices: ElevenLabsVoiceRepository
|
||||
effects: ElevenLabsSoundEffectEngine
|
||||
isolation: ElevenLabsVoiceIsolationEngine
|
||||
}
|
||||
|
||||
interface CreateElevenLabsVoiceOptions {
|
||||
// Either an existing client...
|
||||
client?: ElevenLabsClient
|
||||
// ...or the inputs needed to construct one (forwarded to ElevenLabsClient unchanged)
|
||||
apiKey?: string
|
||||
baseUrl?: string
|
||||
timeoutMs?: number
|
||||
retryPolicy?: RetryPolicy
|
||||
fetchImpl?: typeof fetch
|
||||
webSocketImpl?: WebSocketImpl
|
||||
clock?: Clock
|
||||
logger?: LoggerLike
|
||||
events?: EventBusLike<ElevenLabsClientEvents>
|
||||
userAgent?: string
|
||||
defaultHeaders?: Record<string, string>
|
||||
|
||||
// Adapter-specific defaults
|
||||
defaultModelId?: string
|
||||
defaultVoiceSettings?: VoiceSettings
|
||||
defaultSoundEffectModelId?: string
|
||||
}
|
||||
```
|
||||
|
||||
If `client` is provided, `apiKey` and the other client options are ignored; otherwise the factory constructs a fresh `ElevenLabsClient`. Either way, the adapter classes share the same client instance — so retry, rate-limit accounting, and OTel are unified across engines.
|
||||
|
||||
---
|
||||
|
||||
## ElevenLabs API mapping
|
||||
|
||||
| Voice interface | Third-party module | Endpoint |
|
||||
|---|---|---|
|
||||
| `VoiceSynthesizer.synthesize` | `tts.synthesize(voiceId, body)` | `POST /v1/text-to-speech/{id}` |
|
||||
| `VoiceSynthesizer.synthesizeStream` | `tts.synthesizeStream(voiceId, body)` | `POST /v1/text-to-speech/{id}/stream` |
|
||||
| `TranscriptionEngine.transcribe` | `stt.transcribe(body)` | `POST /v1/speech-to-text` (multipart) |
|
||||
| `VoiceRepository.list` | `voices.list(params)` | `GET /v1/voices` |
|
||||
| `VoiceRepository.get` | `voices.get(voiceId)` | `GET /v1/voices/{id}` |
|
||||
| `SoundEffectEngine.generate` | `soundGeneration.generate(body)` | `POST /v1/sound-generation` |
|
||||
| `VoiceIsolationEngine.isolate` | `audioIsolation.isolate(body)` | `POST /v1/audio-isolation` |
|
||||
|
||||
---
|
||||
|
||||
## Error remapping table
|
||||
|
||||
The adapter remaps every `ElevenLabsApiError` subclass to a `VoiceError` code and preserves the original via `cause`:
|
||||
|
||||
| Third-party code | Voice domain code | Notes |
|
||||
|---|---|---|
|
||||
| `AUTH.UNAUTHORIZED` | `VOICE.PROVIDER.NOT_CONFIGURED` | The API key is invalid or missing |
|
||||
| `AUTH.FORBIDDEN` | `VOICE.PROVIDER.NOT_CONFIGURED` | Key valid but lacks scope |
|
||||
| `REQUEST.NOT_FOUND` (from `voices.get`) | `VOICE.VOICE.NOT_FOUND` | With `context.voiceId` populated |
|
||||
| `REQUEST.NOT_FOUND` (other endpoints) | `VOICE.VALIDATION.INVALID_REQUEST` | Generic resource-missing |
|
||||
| `REQUEST.VALIDATION_FAILED` | `VOICE.VALIDATION.INVALID_REQUEST` | Body preserved on `cause.context` |
|
||||
| `REQUEST.INVALID` | `VOICE.VALIDATION.INVALID_REQUEST` | |
|
||||
| `RATE_LIMIT.EXCEEDED` | `VOICE.PROVIDER.RATE_LIMITED` | `retryAfterMs` preserved |
|
||||
| `RATE_LIMIT.QUOTA_EXHAUSTED` | `VOICE.PROVIDER.RATE_LIMITED` | |
|
||||
| `SERVER.TRANSIENT` (TTS endpoints) | `VOICE.TTS.SYNTHESIS_FAILED` | |
|
||||
| `SERVER.TRANSIENT` (STT endpoints) | `VOICE.STT.TRANSCRIPTION_FAILED` | |
|
||||
| `SERVER.TRANSIENT` (sound-generation) | `VOICE.EFFECTS.GENERATION_FAILED` | |
|
||||
| `SERVER.TRANSIENT` (audio-isolation) | `VOICE.ISOLATION.FAILED` | |
|
||||
| `SERVER.UNAVAILABLE` | Same as `SERVER.TRANSIENT` per endpoint | |
|
||||
| `STREAM.INTERRUPTED` | `VOICE.TTS.STREAM_INTERRUPTED` | |
|
||||
| `RESPONSE.MALFORMED` | `VOICE.PROVIDER.NOT_CONFIGURED` | Indicates an ElevenLabs API contract change |
|
||||
| `RESPONSE.UNSUPPORTED_FORMAT` | `VOICE.AUDIO.FORMAT_UNSUPPORTED` | |
|
||||
| `NETWORK.*` | `VOICE.PROVIDER.NOT_CONFIGURED` (timeout/unreachable) or `VOICE.TTS.STREAM_INTERRUPTED` (during streams) | |
|
||||
|
||||
Every remapped error preserves the original via `cause` so callers can inspect the underlying transport detail when needed.
|
||||
|
||||
---
|
||||
|
||||
## DTO mapping decisions
|
||||
|
||||
1. **`VoiceCategory` mapping.** ElevenLabs returns `category: 'premade' | 'cloned' | 'generated' | 'professional'` directly compatible with `@suaveplan/voice` enum — no remapping needed.
|
||||
2. **`Voice.languageCodes`.** Aggregated from `verified_languages` (BCP-47) and `labels.language` if present. De-duplicated and ISO-639-normalised.
|
||||
3. **`Voice.gender` / `Voice.age` / `Voice.accent`.** Read from `labels.gender`, `labels.age`, `labels.accent`. Unrecognised values fall back to `metadata`.
|
||||
4. **`SynthesisResult.charactersUsed`.** Read from response `metadata.characterCost` (third-party client surfaces `X-Character-Cost` header).
|
||||
5. **`TranscriptionResult.words`.** Mapped from ElevenLabs `words[]` array, dropping non-word entries (`type !== 'word'` enters `entities` instead).
|
||||
6. **`TranscriptionResult.speakers`.** Derived by collapsing consecutive `words[]` entries with the same `speaker_id` into `{startMs, endMs, speakerId}` segments.
|
||||
7. **`AudioFormat` round-trip.** All `@suaveplan/voice` `AudioFormat` strings (`mp3_22050_32`, `pcm_16000`, etc.) map to ElevenLabs `output_format` values 1:1 — verified by property test.
|
||||
|
||||
---
|
||||
|
||||
## Parity Checklist
|
||||
|
||||
Genesis-original package — the prior `packages/audio/voice-elevenlabs` implementation (2026-05-01) is being **deleted and rewritten**, not migrated, because the architecture changes substantially. The deleted code is superseded by the splits described in `## Why this design supersedes the 2026-04-24 proposal`. No upstream `@suaveplan/voice-elevenlabs` exists in suaveplan main.
|
||||
|
||||
| Capability | Prior fused build | This adapter | Notes |
|
||||
|---|---|---|---|
|
||||
| TTS synthesize | ✅ partial | ✅ | Now via `tts.synthesize` |
|
||||
| TTS streaming | ✅ partial | ✅ | Now via `tts.synthesizeStream` |
|
||||
| STT transcribe | ✅ partial | ✅ | Now via `stt.transcribe` |
|
||||
| Voice list/get | ✅ partial | ✅ | Now via `voices.list` / `voices.get` |
|
||||
| Sound generation | ✅ partial | ✅ | Now via `soundGeneration.generate` |
|
||||
| Audio isolation | ✅ partial | ✅ | Now via `audioIsolation.isolate` |
|
||||
| Retry / rate-limit | 🚫 | ✅ | Was missing in prior build; now provided by third-party client |
|
||||
| OTel | 🚫 | ✅ | Was missing in prior build; now provided by third-party client |
|
||||
| `X-RateLimit-Remaining` | 🚫 | ✅ | Was missing in prior build; now provided by third-party client |
|
||||
| `X-Character-Cost` | 🚫 | ✅ | Was missing in prior build; now provided by third-party client |
|
||||
| 5xx mapped to transient | 🚫 (incorrectly mapped to `PROVIDER.NOT_CONFIGURED`) | ✅ | Per error remapping table |
|
||||
|
||||
---
|
||||
|
||||
## Impact
|
||||
|
||||
### Affected dependents
|
||||
|
||||
Run `bunx turbo ls --filter=...@suaveplan/voice-elevenlabs` to enumerate. Currently no shipped genesis package consumes `@suaveplan/voice-elevenlabs` (the rebuild lands before any downstream takes a dep on it).
|
||||
|
||||
- Future `@suaveplan/api-plugin-voice` — [ ] tests run / [ ] docs reviewed / [x] no action needed (does not yet exist on genesis)
|
||||
- Future `@suaveplan/react-voice` — [ ] tests run / [ ] docs reviewed / [x] no action needed (does not yet exist on genesis)
|
||||
|
||||
### Migration
|
||||
|
||||
The 2026-05-01 fused-HTTP build is deleted in this change. Any in-progress consumer code that imported from `@suaveplan/voice-elevenlabs` must:
|
||||
|
||||
- **Old:** `import { ElevenLabsClient } from '@suaveplan/voice-elevenlabs/client/...'`
|
||||
- **New:** `import { ElevenLabsClient } from '@suaveplan/third-party-elevenlabs'`
|
||||
|
||||
- **Old:** `import { createElevenLabsVoice } from '@suaveplan/voice-elevenlabs/factory/factory'`
|
||||
- **New:** Same path, same return shape — `createElevenLabsVoice(options)` is preserved with identical signature except that `apiKey` / `baseUrl` / `timeoutMs` are now forwarded to `ElevenLabsClient` rather than consumed locally, and the returned object now includes `client: ElevenLabsClient`.
|
||||
|
||||
Search command for downstream consumers:
|
||||
|
||||
```bash
|
||||
grep -rln "@suaveplan/voice-elevenlabs/client" packages/ | grep -v "/dist/\|/coverage/\|/voice-elevenlabs/"
|
||||
```
|
||||
|
||||
If any hits exist, update them to import from `@suaveplan/third-party-elevenlabs` before merging.
|
||||
Reference in New Issue
Block a user