Initial Commit
This commit is contained in:
224
openspec/archive/2026-05-22-third-party-elevenlabs/proposal.md
Normal file
224
openspec/archive/2026-05-22-third-party-elevenlabs/proposal.md
Normal file
@@ -0,0 +1,224 @@
|
||||
# OpenSpec Change: third-party-elevenlabs
|
||||
|
||||
- **Change ID**: third-party-elevenlabs
|
||||
- **Package**: `@suaveplan/third-party-elevenlabs`
|
||||
- **Category**: universal
|
||||
- **Target path**: `packages/audio/third-party-elevenlabs`
|
||||
- **Status**: PROPOSED
|
||||
- **Date**: 2026-05-02
|
||||
- **Reference spec**: none — new capability (genesis-original)
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Deliver `@suaveplan/third-party-elevenlabs` on the genesis branch from scratch at enterprise quality. The package is a **typed thin client for the entire ElevenLabs platform API** — REST and WebSocket. It owns HTTP plumbing (auth, base URL, fetch wrapper), retry + circuit breaker via `@suaveplan/retry`, rate-limit-header parsing, structured error mapping, OpenTelemetry instrumentation, and Zod-validated request/response DTOs for every documented endpoint.
|
||||
|
||||
It is **not** a voice abstraction. It does not implement `@suaveplan/voice` interfaces. It is the substrate that domain adapter packages — `@suaveplan/voice-elevenlabs`, future `@suaveplan/conversational-elevenlabs`, `@suaveplan/music-elevenlabs`, `@suaveplan/dubbing-elevenlabs` — depend on. Domain adapters consume DTOs from this package and re-shape them into provider-agnostic domain types.
|
||||
|
||||
This split exists because the ElevenLabs API surface spans far beyond TTS/STT — Music, Dubbing, Studio, Audio Native, Conversational AI agents, History, Pronunciation Dictionaries, Workspace administration. Fusing HTTP plumbing into a single voice adapter would force every future surface to reinvent auth, retry, rate-limiting, OTel, and error mapping.
|
||||
|
||||
---
|
||||
|
||||
## Why a separate package
|
||||
|
||||
1. **DRY HTTP plumbing.** Auth (`xi-api-key` header), base URL handling, idempotent retry, `Retry-After` and `X-RateLimit-Remaining` parsing, 401/422/429/5xx → `SuaveplanError` mapping, OTel spans/counters/histograms — all live in one place.
|
||||
2. **Layering matches the genesis precedent.** `payment` + `payment-providers`, `currency` + `currency-providers`, `sms` + `sms-providers` already separate domain abstraction from provider HTTP. ElevenLabs is the inverse — the *vendor* surfaces multiple domains, so the vendor client is the shared layer.
|
||||
3. **Independent versioning of vendor changes.** ElevenLabs ships rapid API changes (new models, new headers, new endpoints). The thin client absorbs them; downstream adapters only update when domain semantics shift.
|
||||
4. **Reusable across workspaces.** The same client backs voice, music, dubbing, conversational-AI, and any future surface. No two adapters re-implement the same `fetch` wrapper.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Provider-agnostic abstractions. The client returns ElevenLabs-shaped DTOs (validated via Zod). Mapping to `@suaveplan/voice` types happens in `@suaveplan/voice-elevenlabs`.
|
||||
- Voice-domain factories or composite engines. Those live in `@suaveplan/voice-elevenlabs`.
|
||||
- API key management or secret storage. The caller injects `apiKey` at construction time; this package never reads from `process.env`.
|
||||
- Rate-limit *enforcement*. The client surfaces rate-limit metadata via headers + errors; quota planning is the caller's responsibility.
|
||||
|
||||
---
|
||||
|
||||
## Available workspace deps on genesis
|
||||
|
||||
| Package | Role |
|
||||
|---|---|
|
||||
| `@suaveplan/error` (`workspace:^`) | `SuaveplanError` base + `defineCodes` for `THIRD_PARTY_ELEVENLABS.*` codes |
|
||||
| `@suaveplan/types` (`workspace:^`) | `Clock`, `LoggerLike`, `EventBusLike`, `CancellableOptions` |
|
||||
| `@suaveplan/streams` (`workspace:^`) | `toBytes` for buffering `ReadableStream` audio uploads |
|
||||
| `@suaveplan/retry` (`workspace:^`) | Retry policies, circuit breaker, backoff with jitter |
|
||||
| `@suaveplan/logger` (`workspace:^`) | Optional structured logging hook |
|
||||
| `@suaveplan/serialize` (`workspace:^`) | Multipart form-data helpers and binary-safe encoders if needed |
|
||||
| `zod` (`catalog:forms`) | DTO schemas for every endpoint |
|
||||
| `@opentelemetry/api` (`catalog:observability`) | Spans, counters, histograms |
|
||||
| `tslib` (`catalog:build`) | Runtime helpers |
|
||||
|
||||
Dev deps: `@suaveplan/testing` (`workspace:^`, dev-only), `vitest`, `@vitest/coverage-istanbul`, `fast-check`, `mitata` (all `catalog:testing`).
|
||||
|
||||
---
|
||||
|
||||
## Public API surface
|
||||
|
||||
### Errors
|
||||
|
||||
```ts
|
||||
class ElevenLabsApiError extends SuaveplanError<ElevenLabsErrorCode> {}
|
||||
```
|
||||
|
||||
Codes (under namespace `THIRD_PARTY_ELEVENLABS`):
|
||||
|
||||
- `AUTH.UNAUTHORIZED` — 401, invalid/missing API key
|
||||
- `AUTH.FORBIDDEN` — 403, key valid but lacks scope/quota
|
||||
- `REQUEST.INVALID` — 4xx (excluding 401/403/404/422/429), client-side request shape error
|
||||
- `REQUEST.VALIDATION_FAILED` — 422, ElevenLabs validation rejection
|
||||
- `REQUEST.NOT_FOUND` — 404
|
||||
- `RATE_LIMIT.EXCEEDED` — 429 (with `retryAfterMs` context)
|
||||
- `RATE_LIMIT.QUOTA_EXHAUSTED` — quota header indicates exhaustion (subclass for caller branch)
|
||||
- `SERVER.TRANSIENT` — 5xx, retryable
|
||||
- `SERVER.UNAVAILABLE` — 503 / circuit-open
|
||||
- `NETWORK.TIMEOUT` — request timeout
|
||||
- `NETWORK.ABORTED` — caller-supplied `AbortSignal` aborted
|
||||
- `NETWORK.UNREACHABLE` — DNS / connect failure
|
||||
- `RESPONSE.MALFORMED` — JSON parse failure or Zod schema mismatch
|
||||
- `RESPONSE.UNSUPPORTED_FORMAT` — server returned content-type the client cannot handle
|
||||
- `STREAM.INTERRUPTED` — server closed the streaming response mid-flight
|
||||
- `WEBSOCKET.CONNECTION_FAILED` — handshake failure
|
||||
- `WEBSOCKET.PROTOCOL` — server sent unexpected frame / close code
|
||||
|
||||
Each code has a typed subclass (`UnauthorizedError`, `RateLimitExceededError`, etc.) so callers can branch with `instanceof`.
|
||||
|
||||
### Client
|
||||
|
||||
```ts
|
||||
class ElevenLabsClient {
|
||||
constructor(options: ElevenLabsClientOptions)
|
||||
|
||||
fetchJson<TBody>(req: JsonRequest, schema: ZodSchema<TBody>): Promise<TBody>
|
||||
fetchBinary(req: BinaryRequest): Promise<BinaryResponse>
|
||||
fetchStream(req: StreamRequest): Promise<ReadableStream<Uint8Array>>
|
||||
fetchMultipart<TBody>(req: MultipartRequest, schema: ZodSchema<TBody>): Promise<TBody>
|
||||
|
||||
openSocket(req: SocketRequest): Promise<ElevenLabsSocket>
|
||||
}
|
||||
|
||||
interface ElevenLabsClientOptions {
|
||||
apiKey: string
|
||||
baseUrl?: string
|
||||
timeoutMs?: number
|
||||
retryPolicy?: RetryPolicy
|
||||
circuitBreaker?: CircuitBreaker
|
||||
fetchImpl?: typeof fetch
|
||||
webSocketImpl?: WebSocketImpl
|
||||
clock?: Clock
|
||||
logger?: LoggerLike
|
||||
events?: EventBusLike<ElevenLabsClientEvents>
|
||||
userAgent?: string
|
||||
defaultHeaders?: Record<string, string>
|
||||
}
|
||||
```
|
||||
|
||||
### Feature modules (REST)
|
||||
|
||||
Each ElevenLabs API surface gets its own subpath module. Every module exports a class taking `ElevenLabsClient` in its constructor; Zod schemas in `*.schemas.ts`; TypeScript types inferred from Zod; sibling `.md`.
|
||||
|
||||
| Module | Subpath | ElevenLabs endpoints |
|
||||
|---|---|---|
|
||||
| `tts` | `./tts/tts` | `POST /v1/text-to-speech/{voice_id}`, `/stream`, `/with-timestamps`, `/stream/with-timestamps` |
|
||||
| `tts-websocket` | `./tts-websocket/tts-websocket` | `wss://api.elevenlabs.io/v1/text-to-speech/{voice_id}/stream-input` |
|
||||
| `stt` | `./stt/stt` | `POST /v1/speech-to-text` (multipart) |
|
||||
| `voices` | `./voices/voices` | `GET /v1/voices`, `/v1/voices/{id}`, `/v1/shared-voices`, settings get/edit, default settings |
|
||||
| `voice-cloning` | `./voice-cloning/voice-cloning` | instant clone, edit, delete, noise-reduced add |
|
||||
| `voice-design` | `./voice-design/voice-design` | generate-voice, create-voice, design, create-from-design |
|
||||
| `speech-to-speech` | `./speech-to-speech/speech-to-speech` | speech-to-speech (sync + stream) |
|
||||
| `sound-generation` | `./sound-generation/sound-generation` | `POST /v1/sound-generation` |
|
||||
| `audio-isolation` | `./audio-isolation/audio-isolation` | `POST /v1/audio-isolation`, `/stream` |
|
||||
| `music` | `./music/music` | music, music/stream, composition-plan, music/detailed |
|
||||
| `dubbing` | `./dubbing/dubbing` | project, asset, transcript, language audio, dubbing-resource CRUD |
|
||||
| `studio` | `./studio/studio` | projects + chapters + snapshots |
|
||||
| `audio-native` | `./audio-native/audio-native` | settings, content-update |
|
||||
| `history` | `./history/history` | list, get, audio, delete, download |
|
||||
| `user` | `./user/user` | user info, subscription |
|
||||
| `workspace` | `./workspace/workspace` | invites, members, groups, secrets, webhooks |
|
||||
| `pronunciation-dictionaries` | `./pronunciation-dictionaries/pronunciation-dictionaries` | CRUD + version listing |
|
||||
| `conversational-ai` | `./conversational-ai/conversational-ai` | agents, conversations, knowledge bases, tools, phone numbers, secrets, batch calls, dashboard |
|
||||
| `models` | `./models/models` | `GET /v1/models` |
|
||||
|
||||
### WebSocket transport
|
||||
|
||||
```ts
|
||||
interface ElevenLabsSocket {
|
||||
readonly url: string
|
||||
readonly readyState: 'connecting' | 'open' | 'closing' | 'closed'
|
||||
send(frame: SocketFrame): void
|
||||
close(code?: number, reason?: string): void
|
||||
onMessage(handler: (frame: SocketFrame) => void): () => void
|
||||
onClose(handler: (event: SocketCloseEvent) => void): () => void
|
||||
onError(handler: (err: ElevenLabsApiError) => void): () => void
|
||||
}
|
||||
```
|
||||
|
||||
WebSocket modules (`tts-websocket`, `conversational-ai`) expose typed wrappers that emit/parse domain frames over this transport. The transport is responsible for ping/pong, surfacing protocol errors as `WEBSOCKET.PROTOCOL` errors, and clean shutdown.
|
||||
|
||||
---
|
||||
|
||||
## Design decisions
|
||||
|
||||
1. **Native `fetch` only.** No `axios`, `node-fetch`, `undici` direct dep. Bun's `fetch` supports streaming bodies, multipart uploads, `AbortSignal`, and `ReadableStream` responses out of the box.
|
||||
2. **DTOs validated at the boundary.** Every JSON response is parsed through a Zod schema before reaching the caller. Malformed responses throw `RESPONSE.MALFORMED` with the raw body in `context` (truncated for logging).
|
||||
3. **Retry only on documented retryable codes.** 429, 502, 503, 504 and `NETWORK.UNREACHABLE` retry by default. 5xx that aren't gateway/throttling errors do not auto-retry — domain code decides whether to escalate.
|
||||
4. **`Retry-After` strict parsing.** Both seconds and HTTP-date formats. Surface in `RateLimitExceededError.context.retryAfterMs`.
|
||||
5. **`X-RateLimit-Remaining` and `X-Character-Cost` exposed via response metadata.** Every successful call returns a `metadata` envelope alongside the body so adapters can attribute usage.
|
||||
6. **Streams are never buffered server-side.** `fetchStream` returns `response.body` as `ReadableStream<Uint8Array>` directly. Callers can pipe into Web Audio, MediaSource, or buffer on demand via `@suaveplan/streams`.
|
||||
7. **Multipart uses `FormData`.** Audio uploads accept `Uint8Array | Blob | ReadableStream | string` (URL). `ReadableStream` is buffered to bytes via `@suaveplan/streams.toBytes` because ElevenLabs STT requires a complete file. URLs are fetched first.
|
||||
8. **OTel on every call.** Span name `elevenlabs.<feature>.<operation>`, attributes `elevenlabs.endpoint`, `elevenlabs.method`, `http.response.status_code`, `elevenlabs.model_id` (where present). Counter `third_party_elevenlabs.requests_total{feature, operation, status}`. Histogram `third_party_elevenlabs.request_duration_ms{feature, operation}`.
|
||||
9. **Versioned base path.** Each module is constructed with the `ElevenLabsClient`; the path version (`/v1/`) is owned by the module so a v2 migration touches only the affected module.
|
||||
10. **No singleton state.** No global registry. Every feature class is constructed per-`ElevenLabsClient` instance.
|
||||
11. **Structured error context preservation.** Every error carries `context.requestId`, `context.endpoint`, `context.statusCode` (where applicable), and `context.retryAfterMs` (for 429). The Zod-parsed body is carried under `context.body` for `VALIDATION_FAILED` only.
|
||||
12. **No conditional `process.env` reads.** This package is universal-tier and must work in browsers and edge runtimes; environment variables are the caller's concern.
|
||||
|
||||
---
|
||||
|
||||
## Implementation phases
|
||||
|
||||
The full ElevenLabs surface is large. Implementation lands in waves; the package ships at version `0.1.0` between waves.
|
||||
|
||||
| Wave | Scope | Status target |
|
||||
|---|---|---|
|
||||
| **Wave 1 — voice surface** | client, errors, schemas, OTel, retry, `tts`, `stt`, `voices`, `sound-generation`, `audio-isolation`, `speech-to-speech`, `models` | **This OpenSpec change ships Wave 1**; required by `@suaveplan/voice-elevenlabs` v1 |
|
||||
| Wave 2 — voice extensions | `tts-websocket`, `voice-cloning`, `voice-design`, `pronunciation-dictionaries` | Follow-up `third-party-elevenlabs-voice-extensions` |
|
||||
| Wave 3 — Conversational AI | `conversational-ai`, agent WebSocket frames | Follow-up `third-party-elevenlabs-conversational` |
|
||||
| Wave 4 — content ops | `music`, `dubbing`, `studio`, `audio-native` | Follow-up `third-party-elevenlabs-content-ops` |
|
||||
| Wave 5 — admin + history | `history`, `user`, `workspace` | Follow-up `third-party-elevenlabs-admin` |
|
||||
|
||||
Each subsequent wave gets its own `openspec/changes/<id>/` folder with delta `## ADDED Requirements` against `openspec/specs/third-party-elevenlabs/spec.md`.
|
||||
|
||||
---
|
||||
|
||||
## Parity Checklist
|
||||
|
||||
Genesis-original package — no upstream `@suaveplan/third-party-elevenlabs` exists in suaveplan main. Reference for behaviour: ElevenLabs API documentation at <https://third-party-elevenlabs.io/docs/api-reference/>.
|
||||
|
||||
| Capability | Upstream | This proposal | Notes |
|
||||
|---|---|---|---|
|
||||
| Typed REST client | n/a | ✅ Wave 1 | |
|
||||
| Streaming TTS | n/a | ✅ Wave 1 | `ReadableStream<Uint8Array>` |
|
||||
| WebSocket transport | n/a | ⏳ Wave 2 + Wave 3 | Plumbing in Wave 2 (tts-websocket); agents in Wave 3 |
|
||||
| Retry/backoff/circuit breaker | n/a | ✅ Wave 1 | via `@suaveplan/retry` |
|
||||
| OTel instrumentation | n/a | ✅ Wave 1 | spans + counters + histograms |
|
||||
| Rate-limit header parsing | n/a | ✅ Wave 1 | `Retry-After`, `X-RateLimit-Remaining`, `X-Character-Cost` |
|
||||
| All ElevenLabs endpoints | n/a | ✅ planned across waves 1–5 | this proposal lists every documented endpoint |
|
||||
|
||||
---
|
||||
|
||||
## Impact
|
||||
|
||||
### Affected dependents
|
||||
|
||||
This is a new package — no existing dependents. After Wave 1 ships, the only direct consumer will be `@suaveplan/voice-elevenlabs` (rebuilt as a thin adapter under change `voice-elevenlabs`).
|
||||
|
||||
- `@suaveplan/voice-elevenlabs` — [x] tests run / [x] docs reviewed / [ ] no action needed (the rebuild lands together; see change `voice-elevenlabs`)
|
||||
|
||||
Future changes that consume this package: `@suaveplan/conversational-elevenlabs`, `@suaveplan/music-elevenlabs`, `@suaveplan/dubbing-elevenlabs`, `@suaveplan/studio-elevenlabs` — each will land via its own OpenSpec change.
|
||||
|
||||
### Migration
|
||||
|
||||
This is the first version of the package; no migration notes apply. When endpoints change across ElevenLabs API revisions, follow-up OpenSpec changes will document the migration in their `## Migration` sections.
|
||||
Reference in New Issue
Block a user