suaveplan/openspec/archive/2026-05-22-third-party-elevenlabs/proposal.md

# OpenSpec Change: third-party-elevenlabs

- **Change ID**: third-party-elevenlabs
- **Package**: `@suaveplan/third-party-elevenlabs`
- **Category**: universal
- **Target path**: `packages/audio/third-party-elevenlabs`
- **Status**: PROPOSED
- **Date**: 2026-05-02
- **Reference spec**: none — new capability (genesis-original)

---

## Summary

Deliver `@suaveplan/third-party-elevenlabs` on the genesis branch from scratch at enterprise quality. The package is a **typed thin client for the entire ElevenLabs platform API** — REST and WebSocket. It owns HTTP plumbing (auth, base URL, fetch wrapper), retry + circuit breaker via `@suaveplan/retry`, rate-limit-header parsing, structured error mapping, OpenTelemetry instrumentation, and Zod-validated request/response DTOs for every documented endpoint.

It is **not** a voice abstraction. It does not implement `@suaveplan/voice` interfaces. It is the substrate that domain adapter packages — `@suaveplan/voice-elevenlabs`, future `@suaveplan/conversational-elevenlabs`, `@suaveplan/music-elevenlabs`, `@suaveplan/dubbing-elevenlabs` — depend on. Domain adapters consume DTOs from this package and re-shape them into provider-agnostic domain types.

This split exists because the ElevenLabs API surface spans far beyond TTS/STT — Music, Dubbing, Studio, Audio Native, Conversational AI agents, History, Pronunciation Dictionaries, Workspace administration. Fusing HTTP plumbing into a single voice adapter would force every future surface to reinvent auth, retry, rate-limiting, OTel, and error mapping.

---

## Why a separate package

1. **DRY HTTP plumbing.** Auth (`xi-api-key` header), base URL handling, idempotent retry, `Retry-After` and `X-RateLimit-Remaining` parsing, 401/422/429/5xx → `SuaveplanError` mapping, OTel spans/counters/histograms — all live in one place.
2. **Layering matches the genesis precedent.** `payment` + `payment-providers`, `currency` + `currency-providers`, `sms` + `sms-providers` already separate domain abstraction from provider HTTP. ElevenLabs is the inverse — the *vendor* surfaces multiple domains, so the vendor client is the shared layer.
3. **Independent versioning of vendor changes.** ElevenLabs ships rapid API changes (new models, new headers, new endpoints). The thin client absorbs them; downstream adapters only update when domain semantics shift.
4. **Reusable across workspaces.** The same client backs voice, music, dubbing, conversational-AI, and any future surface. No two adapters re-implement the same `fetch` wrapper.

---

## Out of scope

- Provider-agnostic abstractions. The client returns ElevenLabs-shaped DTOs (validated via Zod). Mapping to `@suaveplan/voice` types happens in `@suaveplan/voice-elevenlabs`.
- Voice-domain factories or composite engines. Those live in `@suaveplan/voice-elevenlabs`.
- API key management or secret storage. The caller injects `apiKey` at construction time; this package never reads from `process.env`.
- Rate-limit *enforcement*. The client surfaces rate-limit metadata via headers + errors; quota planning is the caller's responsibility.

---

## Available workspace deps on genesis

| Package | Role |
|---|---|
| `@suaveplan/error` (`workspace:^`) | `SuaveplanError` base + `defineCodes` for `THIRD_PARTY_ELEVENLABS.*` codes |
| `@suaveplan/types` (`workspace:^`) | `Clock`, `LoggerLike`, `EventBusLike`, `CancellableOptions` |
| `@suaveplan/streams` (`workspace:^`) | `toBytes` for buffering `ReadableStream` audio uploads |
| `@suaveplan/retry` (`workspace:^`) | Retry policies, circuit breaker, backoff with jitter |
| `@suaveplan/logger` (`workspace:^`) | Optional structured logging hook |
| `@suaveplan/serialize` (`workspace:^`) | Multipart form-data helpers and binary-safe encoders if needed |
| `zod` (`catalog:forms`) | DTO schemas for every endpoint |
| `@opentelemetry/api` (`catalog:observability`) | Spans, counters, histograms |
| `tslib` (`catalog:build`) | Runtime helpers |

Dev deps: `@suaveplan/testing` (`workspace:^`, dev-only), `vitest`, `@vitest/coverage-istanbul`, `fast-check`, `mitata` (all `catalog:testing`).

---

## Public API surface

### Errors

```ts
class ElevenLabsApiError extends SuaveplanError<ElevenLabsErrorCode> {}
```

Codes (under namespace `THIRD_PARTY_ELEVENLABS`):

- `AUTH.UNAUTHORIZED` — 401, invalid/missing API key
- `AUTH.FORBIDDEN` — 403, key valid but lacks scope/quota
- `REQUEST.INVALID` — 4xx (excluding 401/403/404/422/429), client-side request shape error
- `REQUEST.VALIDATION_FAILED` — 422, ElevenLabs validation rejection
- `REQUEST.NOT_FOUND` — 404
- `RATE_LIMIT.EXCEEDED` — 429 (with `retryAfterMs` context)
- `RATE_LIMIT.QUOTA_EXHAUSTED` — quota header indicates exhaustion (subclass for caller branch)
- `SERVER.TRANSIENT` — 5xx, retryable
- `SERVER.UNAVAILABLE` — 503 / circuit-open
- `NETWORK.TIMEOUT` — request timeout
- `NETWORK.ABORTED` — caller-supplied `AbortSignal` aborted
- `NETWORK.UNREACHABLE` — DNS / connect failure
- `RESPONSE.MALFORMED` — JSON parse failure or Zod schema mismatch
- `RESPONSE.UNSUPPORTED_FORMAT` — server returned content-type the client cannot handle
- `STREAM.INTERRUPTED` — server closed the streaming response mid-flight
- `WEBSOCKET.CONNECTION_FAILED` — handshake failure
- `WEBSOCKET.PROTOCOL` — server sent unexpected frame / close code

Each code has a typed subclass (`UnauthorizedError`, `RateLimitExceededError`, etc.) so callers can branch with `instanceof`.

### Client

```ts
class ElevenLabsClient {
  constructor(options: ElevenLabsClientOptions)

  fetchJson<TBody>(req: JsonRequest, schema: ZodSchema<TBody>): Promise<TBody>
  fetchBinary(req: BinaryRequest): Promise<BinaryResponse>
  fetchStream(req: StreamRequest): Promise<ReadableStream<Uint8Array>>
  fetchMultipart<TBody>(req: MultipartRequest, schema: ZodSchema<TBody>): Promise<TBody>

  openSocket(req: SocketRequest): Promise<ElevenLabsSocket>
}

interface ElevenLabsClientOptions {
  apiKey: string
  baseUrl?: string
  timeoutMs?: number
  retryPolicy?: RetryPolicy
  circuitBreaker?: CircuitBreaker
  fetchImpl?: typeof fetch
  webSocketImpl?: WebSocketImpl
  clock?: Clock
  logger?: LoggerLike
  events?: EventBusLike<ElevenLabsClientEvents>
  userAgent?: string
  defaultHeaders?: Record<string, string>
}
```

### Feature modules (REST)

Each ElevenLabs API surface gets its own subpath module. Every module exports a class taking `ElevenLabsClient` in its constructor; Zod schemas in `*.schemas.ts`; TypeScript types inferred from Zod; sibling `.md`.

| Module | Subpath | ElevenLabs endpoints |
|---|---|---|
| `tts` | `./tts/tts` | `POST /v1/text-to-speech/{voice_id}`, `/stream`, `/with-timestamps`, `/stream/with-timestamps` |
| `tts-websocket` | `./tts-websocket/tts-websocket` | `wss://api.elevenlabs.io/v1/text-to-speech/{voice_id}/stream-input` |
| `stt` | `./stt/stt` | `POST /v1/speech-to-text` (multipart) |
| `voices` | `./voices/voices` | `GET /v1/voices`, `/v1/voices/{id}`, `/v1/shared-voices`, settings get/edit, default settings |
| `voice-cloning` | `./voice-cloning/voice-cloning` | instant clone, edit, delete, noise-reduced add |
| `voice-design` | `./voice-design/voice-design` | generate-voice, create-voice, design, create-from-design |
| `speech-to-speech` | `./speech-to-speech/speech-to-speech` | speech-to-speech (sync + stream) |
| `sound-generation` | `./sound-generation/sound-generation` | `POST /v1/sound-generation` |
| `audio-isolation` | `./audio-isolation/audio-isolation` | `POST /v1/audio-isolation`, `/stream` |
| `music` | `./music/music` | music, music/stream, composition-plan, music/detailed |
| `dubbing` | `./dubbing/dubbing` | project, asset, transcript, language audio, dubbing-resource CRUD |
| `studio` | `./studio/studio` | projects + chapters + snapshots |
| `audio-native` | `./audio-native/audio-native` | settings, content-update |
| `history` | `./history/history` | list, get, audio, delete, download |
| `user` | `./user/user` | user info, subscription |
| `workspace` | `./workspace/workspace` | invites, members, groups, secrets, webhooks |
| `pronunciation-dictionaries` | `./pronunciation-dictionaries/pronunciation-dictionaries` | CRUD + version listing |
| `conversational-ai` | `./conversational-ai/conversational-ai` | agents, conversations, knowledge bases, tools, phone numbers, secrets, batch calls, dashboard |
| `models` | `./models/models` | `GET /v1/models` |

### WebSocket transport

```ts
interface ElevenLabsSocket {
  readonly url: string
  readonly readyState: 'connecting' | 'open' | 'closing' | 'closed'
  send(frame: SocketFrame): void
  close(code?: number, reason?: string): void
  onMessage(handler: (frame: SocketFrame) => void): () => void
  onClose(handler: (event: SocketCloseEvent) => void): () => void
  onError(handler: (err: ElevenLabsApiError) => void): () => void
}
```

WebSocket modules (`tts-websocket`, `conversational-ai`) expose typed wrappers that emit/parse domain frames over this transport. The transport is responsible for ping/pong, surfacing protocol errors as `WEBSOCKET.PROTOCOL` errors, and clean shutdown.

---

## Design decisions

1. **Native `fetch` only.** No `axios`, `node-fetch`, `undici` direct dep. Bun's `fetch` supports streaming bodies, multipart uploads, `AbortSignal`, and `ReadableStream` responses out of the box.
2. **DTOs validated at the boundary.** Every JSON response is parsed through a Zod schema before reaching the caller. Malformed responses throw `RESPONSE.MALFORMED` with the raw body in `context` (truncated for logging).
3. **Retry only on documented retryable codes.** 429, 502, 503, 504 and `NETWORK.UNREACHABLE` retry by default. 5xx that aren't gateway/throttling errors do not auto-retry — domain code decides whether to escalate.
4. **`Retry-After` strict parsing.** Both seconds and HTTP-date formats. Surface in `RateLimitExceededError.context.retryAfterMs`.
5. **`X-RateLimit-Remaining` and `X-Character-Cost` exposed via response metadata.** Every successful call returns a `metadata` envelope alongside the body so adapters can attribute usage.
6. **Streams are never buffered server-side.** `fetchStream` returns `response.body` as `ReadableStream<Uint8Array>` directly. Callers can pipe into Web Audio, MediaSource, or buffer on demand via `@suaveplan/streams`.
7. **Multipart uses `FormData`.** Audio uploads accept `Uint8Array | Blob | ReadableStream | string` (URL). `ReadableStream` is buffered to bytes via `@suaveplan/streams.toBytes` because ElevenLabs STT requires a complete file. URLs are fetched first.
8. **OTel on every call.** Span name `elevenlabs.<feature>.<operation>`, attributes `elevenlabs.endpoint`, `elevenlabs.method`, `http.response.status_code`, `elevenlabs.model_id` (where present). Counter `third_party_elevenlabs.requests_total{feature, operation, status}`. Histogram `third_party_elevenlabs.request_duration_ms{feature, operation}`.
9. **Versioned base path.** Each module is constructed with the `ElevenLabsClient`; the path version (`/v1/`) is owned by the module so a v2 migration touches only the affected module.
10. **No singleton state.** No global registry. Every feature class is constructed per-`ElevenLabsClient` instance.
11. **Structured error context preservation.** Every error carries `context.requestId`, `context.endpoint`, `context.statusCode` (where applicable), and `context.retryAfterMs` (for 429). The Zod-parsed body is carried under `context.body` for `VALIDATION_FAILED` only.
12. **No conditional `process.env` reads.** This package is universal-tier and must work in browsers and edge runtimes; environment variables are the caller's concern.

---

## Implementation phases

The full ElevenLabs surface is large. Implementation lands in waves; the package ships at version `0.1.0` between waves.

| Wave | Scope | Status target |
|---|---|---|
| **Wave 1 — voice surface** | client, errors, schemas, OTel, retry, `tts`, `stt`, `voices`, `sound-generation`, `audio-isolation`, `speech-to-speech`, `models` | **This OpenSpec change ships Wave 1**; required by `@suaveplan/voice-elevenlabs` v1 |
| Wave 2 — voice extensions | `tts-websocket`, `voice-cloning`, `voice-design`, `pronunciation-dictionaries` | Follow-up `third-party-elevenlabs-voice-extensions` |
| Wave 3 — Conversational AI | `conversational-ai`, agent WebSocket frames | Follow-up `third-party-elevenlabs-conversational` |
| Wave 4 — content ops | `music`, `dubbing`, `studio`, `audio-native` | Follow-up `third-party-elevenlabs-content-ops` |
| Wave 5 — admin + history | `history`, `user`, `workspace` | Follow-up `third-party-elevenlabs-admin` |

Each subsequent wave gets its own `openspec/changes/<id>/` folder with delta `## ADDED Requirements` against `openspec/specs/third-party-elevenlabs/spec.md`.

---

## Parity Checklist

Genesis-original package — no upstream `@suaveplan/third-party-elevenlabs` exists in suaveplan main. Reference for behaviour: ElevenLabs API documentation at <https://third-party-elevenlabs.io/docs/api-reference/>.

| Capability | Upstream | This proposal | Notes |
|---|---|---|---|
| Typed REST client | n/a | ✅ Wave 1 | |
| Streaming TTS | n/a | ✅ Wave 1 | `ReadableStream<Uint8Array>` |
| WebSocket transport | n/a | ⏳ Wave 2 + Wave 3 | Plumbing in Wave 2 (tts-websocket); agents in Wave 3 |
| Retry/backoff/circuit breaker | n/a | ✅ Wave 1 | via `@suaveplan/retry` |
| OTel instrumentation | n/a | ✅ Wave 1 | spans + counters + histograms |
| Rate-limit header parsing | n/a | ✅ Wave 1 | `Retry-After`, `X-RateLimit-Remaining`, `X-Character-Cost` |
| All ElevenLabs endpoints | n/a | ✅ planned across waves 1–5 | this proposal lists every documented endpoint |

---

## Impact

### Affected dependents

This is a new package — no existing dependents. After Wave 1 ships, the only direct consumer will be `@suaveplan/voice-elevenlabs` (rebuilt as a thin adapter under change `voice-elevenlabs`).

- `@suaveplan/voice-elevenlabs` — [x] tests run / [x] docs reviewed / [ ] no action needed (the rebuild lands together; see change `voice-elevenlabs`)

Future changes that consume this package: `@suaveplan/conversational-elevenlabs`, `@suaveplan/music-elevenlabs`, `@suaveplan/dubbing-elevenlabs`, `@suaveplan/studio-elevenlabs` — each will land via its own OpenSpec change.

### Migration

This is the first version of the package; no migration notes apply. When endpoints change across ElevenLabs API revisions, follow-up OpenSpec changes will document the migration in their `## Migration` sections.