SuavePlan/suaveplan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9b28db724f7c99ad304963ee7d8a876190dd4167
suaveplan/openspec/archive/2026-05-22-third-party-elevenlabs/proposal.md
admin fbc70f96e6 Initial Commit
2026-05-27 23:29:53 +01:00

14 KiB
Raw Blame History

OpenSpec Change: third-party-elevenlabs

  • Change ID: third-party-elevenlabs
  • Package: @suaveplan/third-party-elevenlabs
  • Category: universal
  • Target path: packages/audio/third-party-elevenlabs
  • Status: PROPOSED
  • Date: 2026-05-02
  • Reference spec: none — new capability (genesis-original)

Summary

Deliver @suaveplan/third-party-elevenlabs on the genesis branch from scratch at enterprise quality. The package is a typed thin client for the entire ElevenLabs platform API — REST and WebSocket. It owns HTTP plumbing (auth, base URL, fetch wrapper), retry + circuit breaker via @suaveplan/retry, rate-limit-header parsing, structured error mapping, OpenTelemetry instrumentation, and Zod-validated request/response DTOs for every documented endpoint.

It is not a voice abstraction. It does not implement @suaveplan/voice interfaces. It is the substrate that domain adapter packages — @suaveplan/voice-elevenlabs, future @suaveplan/conversational-elevenlabs, @suaveplan/music-elevenlabs, @suaveplan/dubbing-elevenlabs — depend on. Domain adapters consume DTOs from this package and re-shape them into provider-agnostic domain types.

This split exists because the ElevenLabs API surface spans far beyond TTS/STT — Music, Dubbing, Studio, Audio Native, Conversational AI agents, History, Pronunciation Dictionaries, Workspace administration. Fusing HTTP plumbing into a single voice adapter would force every future surface to reinvent auth, retry, rate-limiting, OTel, and error mapping.

Why a separate package

  1. DRY HTTP plumbing. Auth (xi-api-key header), base URL handling, idempotent retry, Retry-After and X-RateLimit-Remaining parsing, 401/422/429/5xx → SuaveplanError mapping, OTel spans/counters/histograms — all live in one place.
  2. Layering matches the genesis precedent. payment + payment-providers, currency + currency-providers, sms + sms-providers already separate domain abstraction from provider HTTP. ElevenLabs is the inverse — the vendor surfaces multiple domains, so the vendor client is the shared layer.
  3. Independent versioning of vendor changes. ElevenLabs ships rapid API changes (new models, new headers, new endpoints). The thin client absorbs them; downstream adapters only update when domain semantics shift.
  4. Reusable across workspaces. The same client backs voice, music, dubbing, conversational-AI, and any future surface. No two adapters re-implement the same fetch wrapper.

Out of scope

  • Provider-agnostic abstractions. The client returns ElevenLabs-shaped DTOs (validated via Zod). Mapping to @suaveplan/voice types happens in @suaveplan/voice-elevenlabs.
  • Voice-domain factories or composite engines. Those live in @suaveplan/voice-elevenlabs.
  • API key management or secret storage. The caller injects apiKey at construction time; this package never reads from process.env.
  • Rate-limit enforcement. The client surfaces rate-limit metadata via headers + errors; quota planning is the caller's responsibility.

Available workspace deps on genesis

Package Role
@suaveplan/error (workspace:^) SuaveplanError base + defineCodes for THIRD_PARTY_ELEVENLABS.* codes
@suaveplan/types (workspace:^) Clock, LoggerLike, EventBusLike, CancellableOptions
@suaveplan/streams (workspace:^) toBytes for buffering ReadableStream audio uploads
@suaveplan/retry (workspace:^) Retry policies, circuit breaker, backoff with jitter
@suaveplan/logger (workspace:^) Optional structured logging hook
@suaveplan/serialize (workspace:^) Multipart form-data helpers and binary-safe encoders if needed
zod (catalog:forms) DTO schemas for every endpoint
@opentelemetry/api (catalog:observability) Spans, counters, histograms
tslib (catalog:build) Runtime helpers

Dev deps: @suaveplan/testing (workspace:^, dev-only), vitest, @vitest/coverage-istanbul, fast-check, mitata (all catalog:testing).

Public API surface

Errors

class ElevenLabsApiError extends SuaveplanError<ElevenLabsErrorCode> {}

Codes (under namespace THIRD_PARTY_ELEVENLABS):

  • AUTH.UNAUTHORIZED — 401, invalid/missing API key
  • AUTH.FORBIDDEN — 403, key valid but lacks scope/quota
  • REQUEST.INVALID — 4xx (excluding 401/403/404/422/429), client-side request shape error
  • REQUEST.VALIDATION_FAILED — 422, ElevenLabs validation rejection
  • REQUEST.NOT_FOUND — 404
  • RATE_LIMIT.EXCEEDED — 429 (with retryAfterMs context)
  • RATE_LIMIT.QUOTA_EXHAUSTED — quota header indicates exhaustion (subclass for caller branch)
  • SERVER.TRANSIENT — 5xx, retryable
  • SERVER.UNAVAILABLE — 503 / circuit-open
  • NETWORK.TIMEOUT — request timeout
  • NETWORK.ABORTED — caller-supplied AbortSignal aborted
  • NETWORK.UNREACHABLE — DNS / connect failure
  • RESPONSE.MALFORMED — JSON parse failure or Zod schema mismatch
  • RESPONSE.UNSUPPORTED_FORMAT — server returned content-type the client cannot handle
  • STREAM.INTERRUPTED — server closed the streaming response mid-flight
  • WEBSOCKET.CONNECTION_FAILED — handshake failure
  • WEBSOCKET.PROTOCOL — server sent unexpected frame / close code

Each code has a typed subclass (UnauthorizedError, RateLimitExceededError, etc.) so callers can branch with instanceof.

Client

class ElevenLabsClient {
  constructor(options: ElevenLabsClientOptions)

  fetchJson<TBody>(req: JsonRequest, schema: ZodSchema<TBody>): Promise<TBody>
  fetchBinary(req: BinaryRequest): Promise<BinaryResponse>
  fetchStream(req: StreamRequest): Promise<ReadableStream<Uint8Array>>
  fetchMultipart<TBody>(req: MultipartRequest, schema: ZodSchema<TBody>): Promise<TBody>

  openSocket(req: SocketRequest): Promise<ElevenLabsSocket>
}

interface ElevenLabsClientOptions {
  apiKey: string
  baseUrl?: string
  timeoutMs?: number
  retryPolicy?: RetryPolicy
  circuitBreaker?: CircuitBreaker
  fetchImpl?: typeof fetch
  webSocketImpl?: WebSocketImpl
  clock?: Clock
  logger?: LoggerLike
  events?: EventBusLike<ElevenLabsClientEvents>
  userAgent?: string
  defaultHeaders?: Record<string, string>
}

Feature modules (REST)

Each ElevenLabs API surface gets its own subpath module. Every module exports a class taking ElevenLabsClient in its constructor; Zod schemas in *.schemas.ts; TypeScript types inferred from Zod; sibling .md.

Module Subpath ElevenLabs endpoints
tts ./tts/tts POST /v1/text-to-speech/{voice_id}, /stream, /with-timestamps, /stream/with-timestamps
tts-websocket ./tts-websocket/tts-websocket wss://api.elevenlabs.io/v1/text-to-speech/{voice_id}/stream-input
stt ./stt/stt POST /v1/speech-to-text (multipart)
voices ./voices/voices GET /v1/voices, /v1/voices/{id}, /v1/shared-voices, settings get/edit, default settings
voice-cloning ./voice-cloning/voice-cloning instant clone, edit, delete, noise-reduced add
voice-design ./voice-design/voice-design generate-voice, create-voice, design, create-from-design
speech-to-speech ./speech-to-speech/speech-to-speech speech-to-speech (sync + stream)
sound-generation ./sound-generation/sound-generation POST /v1/sound-generation
audio-isolation ./audio-isolation/audio-isolation POST /v1/audio-isolation, /stream
music ./music/music music, music/stream, composition-plan, music/detailed
dubbing ./dubbing/dubbing project, asset, transcript, language audio, dubbing-resource CRUD
studio ./studio/studio projects + chapters + snapshots
audio-native ./audio-native/audio-native settings, content-update
history ./history/history list, get, audio, delete, download
user ./user/user user info, subscription
workspace ./workspace/workspace invites, members, groups, secrets, webhooks
pronunciation-dictionaries ./pronunciation-dictionaries/pronunciation-dictionaries CRUD + version listing
conversational-ai ./conversational-ai/conversational-ai agents, conversations, knowledge bases, tools, phone numbers, secrets, batch calls, dashboard
models ./models/models GET /v1/models

WebSocket transport

interface ElevenLabsSocket {
  readonly url: string
  readonly readyState: 'connecting' | 'open' | 'closing' | 'closed'
  send(frame: SocketFrame): void
  close(code?: number, reason?: string): void
  onMessage(handler: (frame: SocketFrame) => void): () => void
  onClose(handler: (event: SocketCloseEvent) => void): () => void
  onError(handler: (err: ElevenLabsApiError) => void): () => void
}

WebSocket modules (tts-websocket, conversational-ai) expose typed wrappers that emit/parse domain frames over this transport. The transport is responsible for ping/pong, surfacing protocol errors as WEBSOCKET.PROTOCOL errors, and clean shutdown.

Design decisions

  1. Native fetch only. No axios, node-fetch, undici direct dep. Bun's fetch supports streaming bodies, multipart uploads, AbortSignal, and ReadableStream responses out of the box.
  2. DTOs validated at the boundary. Every JSON response is parsed through a Zod schema before reaching the caller. Malformed responses throw RESPONSE.MALFORMED with the raw body in context (truncated for logging).
  3. Retry only on documented retryable codes. 429, 502, 503, 504 and NETWORK.UNREACHABLE retry by default. 5xx that aren't gateway/throttling errors do not auto-retry — domain code decides whether to escalate.
  4. Retry-After strict parsing. Both seconds and HTTP-date formats. Surface in RateLimitExceededError.context.retryAfterMs.
  5. X-RateLimit-Remaining and X-Character-Cost exposed via response metadata. Every successful call returns a metadata envelope alongside the body so adapters can attribute usage.
  6. Streams are never buffered server-side. fetchStream returns response.body as ReadableStream<Uint8Array> directly. Callers can pipe into Web Audio, MediaSource, or buffer on demand via @suaveplan/streams.
  7. Multipart uses FormData. Audio uploads accept Uint8Array | Blob | ReadableStream | string (URL). ReadableStream is buffered to bytes via @suaveplan/streams.toBytes because ElevenLabs STT requires a complete file. URLs are fetched first.
  8. OTel on every call. Span name elevenlabs.<feature>.<operation>, attributes elevenlabs.endpoint, elevenlabs.method, http.response.status_code, elevenlabs.model_id (where present). Counter third_party_elevenlabs.requests_total{feature, operation, status}. Histogram third_party_elevenlabs.request_duration_ms{feature, operation}.
  9. Versioned base path. Each module is constructed with the ElevenLabsClient; the path version (/v1/) is owned by the module so a v2 migration touches only the affected module.
  10. No singleton state. No global registry. Every feature class is constructed per-ElevenLabsClient instance.
  11. Structured error context preservation. Every error carries context.requestId, context.endpoint, context.statusCode (where applicable), and context.retryAfterMs (for 429). The Zod-parsed body is carried under context.body for VALIDATION_FAILED only.
  12. No conditional process.env reads. This package is universal-tier and must work in browsers and edge runtimes; environment variables are the caller's concern.

Implementation phases

The full ElevenLabs surface is large. Implementation lands in waves; the package ships at version 0.1.0 between waves.

Wave Scope Status target
Wave 1 — voice surface client, errors, schemas, OTel, retry, tts, stt, voices, sound-generation, audio-isolation, speech-to-speech, models This OpenSpec change ships Wave 1; required by @suaveplan/voice-elevenlabs v1
Wave 2 — voice extensions tts-websocket, voice-cloning, voice-design, pronunciation-dictionaries Follow-up third-party-elevenlabs-voice-extensions
Wave 3 — Conversational AI conversational-ai, agent WebSocket frames Follow-up third-party-elevenlabs-conversational
Wave 4 — content ops music, dubbing, studio, audio-native Follow-up third-party-elevenlabs-content-ops
Wave 5 — admin + history history, user, workspace Follow-up third-party-elevenlabs-admin

Each subsequent wave gets its own openspec/changes/<id>/ folder with delta ## ADDED Requirements against openspec/specs/third-party-elevenlabs/spec.md.

Parity Checklist

Genesis-original package — no upstream @suaveplan/third-party-elevenlabs exists in suaveplan main. Reference for behaviour: ElevenLabs API documentation at https://third-party-elevenlabs.io/docs/api-reference/.

Capability Upstream This proposal Notes
Typed REST client n/a ✅ Wave 1
Streaming TTS n/a ✅ Wave 1 ReadableStream<Uint8Array>
WebSocket transport n/a ⏳ Wave 2 + Wave 3 Plumbing in Wave 2 (tts-websocket); agents in Wave 3
Retry/backoff/circuit breaker n/a ✅ Wave 1 via @suaveplan/retry
OTel instrumentation n/a ✅ Wave 1 spans + counters + histograms
Rate-limit header parsing n/a ✅ Wave 1 Retry-After, X-RateLimit-Remaining, X-Character-Cost
All ElevenLabs endpoints n/a ✅ planned across waves 1–5 this proposal lists every documented endpoint

Impact

Affected dependents

This is a new package — no existing dependents. After Wave 1 ships, the only direct consumer will be @suaveplan/voice-elevenlabs (rebuilt as a thin adapter under change voice-elevenlabs).

  • @suaveplan/voice-elevenlabs — [x] tests run / [x] docs reviewed / [ ] no action needed (the rebuild lands together; see change voice-elevenlabs)

Future changes that consume this package: @suaveplan/conversational-elevenlabs, @suaveplan/music-elevenlabs, @suaveplan/dubbing-elevenlabs, @suaveplan/studio-elevenlabs — each will land via its own OpenSpec change.

Migration

This is the first version of the package; no migration notes apply. When endpoints change across ElevenLabs API revisions, follow-up OpenSpec changes will document the migration in their ## Migration sections.

Reference in New Issue View Git Blame Copy Permalink