first commit
This commit is contained in:
88
docs/media/audio/voice/README.md
Normal file
88
docs/media/audio/voice/README.md
Normal file
@@ -0,0 +1,88 @@
|
||||
# @suaveplan/voice - Provider-agnostic voice core: abstract interfaces, shared types, Zod schemas, and structured errors for TTS, STT, voice management, sound effects, and voice isolation.
|
||||
|
||||
## Summary
|
||||
|
||||
Provider-agnostic voice core for Suaveplan. Defines the abstract
|
||||
interfaces, shared types, Zod schemas, and structured error hierarchy that
|
||||
every voice provider adapter (`@suaveplan/voice-elevenlabs`, future
|
||||
`-openai`, `-google`, `-azure`, `-polly`) implements. This package contains
|
||||
zero provider-specific code, no HTTP, no API keys — it is the contract.
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
## Documentation tree
|
||||
|
||||
- [effects](./src/effects/effects.md) — The text-to-sound-effect engine interface for `@suaveplan/voice`. Defines the `SoundEffectEngine` contract every sound-…
|
||||
- [effects](./src/effects/effects.md) — The text-to-sound-effect engine interface for `@suaveplan/voice`. Defines the `SoundEffectEngine` contract every sound-…
|
||||
- [errors](./src/errors/errors.md) — Stable error class hierarchy and code registry for `@suaveplan/voice`. Every typed error in this module extends `VoiceE…
|
||||
- [errors](./src/errors/errors.md) — Stable error class hierarchy and code registry for `@suaveplan/voice`. Every typed error in this module extends `VoiceE…
|
||||
- [isolation](./src/isolation/isolation.md) — The voice-isolation engine interface for `@suaveplan/voice`. Defines the `VoiceIsolationEngine` contract every voice-is…
|
||||
- [isolation](./src/isolation/isolation.md) — The voice-isolation engine interface for `@suaveplan/voice`. Defines the `VoiceIsolationEngine` contract every voice-is…
|
||||
- [stt](./src/stt/stt.md) — The speech-to-text (STT) engine interface for `@suaveplan/voice`. Defines the `TranscriptionEngine` contract every STT…
|
||||
- [stt](./src/stt/stt.md) — The speech-to-text (STT) engine interface for `@suaveplan/voice`. Defines the `TranscriptionEngine` contract every STT…
|
||||
- [tts](./src/tts/tts.md) — The text-to-speech (TTS) engine interface for `@suaveplan/voice`. Defines the `VoiceSynthesizer` contract every TTS pro…
|
||||
- [tts](./src/tts/tts.md) — The text-to-speech (TTS) engine interface for `@suaveplan/voice`. Defines the `VoiceSynthesizer` contract every TTS pro…
|
||||
- [types](./src/types/types.md) — Shared types and Zod schemas used by every voice engine interface in `@suaveplan/voice`. The module is the single sourc…
|
||||
- [types](./src/types/types.md) — Shared types and Zod schemas used by every voice engine interface in `@suaveplan/voice`. The module is the single sourc…
|
||||
- [voices](./src/voices/voices.md) — The voice repository interface for `@suaveplan/voice` — read-only listing and lookup of voices available to the active…
|
||||
- [voices](./src/voices/voices.md) — The voice repository interface for `@suaveplan/voice` — read-only listing and lookup of voices available to the active…
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
bun add @suaveplan/voice
|
||||
```
|
||||
|
||||
## Quick start
|
||||
|
||||
Consume the contract via a provider adapter (here, ElevenLabs). The adapter
|
||||
is responsible for transport; this package owns the shape:
|
||||
|
||||
```ts
|
||||
import type { VoiceSynthesizer } from "@suaveplan/voice";
|
||||
import { createElevenLabsVoice } from "@suaveplan/voice-elevenlabs";
|
||||
|
||||
const { tts } = createElevenLabsVoice({ apiKey: process.env.ELEVENLABS_API_KEY! });
|
||||
|
||||
const synth: VoiceSynthesizer = tts;
|
||||
const result = await synth.synthesize({
|
||||
text: "Hello, world.",
|
||||
voiceId: "21m00Tcm4TlvDq8ikWAM",
|
||||
});
|
||||
// result.audio is a Uint8Array, result.format is an AudioFormat
|
||||
```
|
||||
|
||||
## Modules
|
||||
|
||||
- [`errors/errors`](src/errors/errors.md) — `VoiceError` base class and 12 typed subclasses with stable `VOICE.*` codes.
|
||||
- [`types/types`](src/types/types.md) — `AudioFormat`, `Voice`, `VoiceSettings`, `WordTimestamp`, `SpeakerSegment`, `DetectedEntity` and their Zod schemas.
|
||||
- [`tts/tts`](src/tts/tts.md) — `VoiceSynthesizer` interface, `SynthesisRequest`, `SynthesisResult`.
|
||||
- [`stt/stt`](src/stt/stt.md) — `TranscriptionEngine` interface, `TranscriptionRequest`, `TranscriptionResult`.
|
||||
- [`voices/voices`](src/voices/voices.md) — `VoiceRepository` interface for listing and looking up voices.
|
||||
- [`effects/effects`](src/effects/effects.md) — `SoundEffectEngine` interface for text-to-sound-effect generation.
|
||||
- [`isolation/isolation`](src/isolation/isolation.md) — `VoiceIsolationEngine` interface for extracting a voice from mixed audio.
|
||||
|
||||
## Design
|
||||
|
||||
`@suaveplan/voice` is a pure abstraction layer:
|
||||
|
||||
1. **No transport.** This package contains no `fetch`, no `WebSocket`, no
|
||||
retry logic, no rate-limit handling. Provider adapters own all of that.
|
||||
2. **No factories tied to providers.** The package never knows which adapter
|
||||
is plugged in. Composition happens at the adapter site
|
||||
(`createElevenLabsVoice`, `createOpenAIVoice`, etc.).
|
||||
3. **One canonical type per concept.** `Voice`, `AudioFormat`,
|
||||
`WordTimestamp`, `SynthesisRequest`, etc. — adapters map their native
|
||||
shapes onto these at the package boundary.
|
||||
4. **Stable error codes.** Every adapter remaps its transport errors onto
|
||||
the 12 codes in `VOICE_CODES`. Callers branch on `error.code` or
|
||||
`instanceof TtsSynthesisFailedError` and never touch upstream details.
|
||||
5. **Zod schemas mirror every public type.** Inputs validate at API
|
||||
boundaries; outputs validate when crossing process or service borders.
|
||||
`AbortSignal` and binary `audio` fields are excluded from schemas (they
|
||||
have no canonical encoding) and validate structurally at runtime.
|
||||
|
||||
Provider-specific extensions (voice cloning, conversational AI, music,
|
||||
dubbing, voice changing) ship in their own packages under separate
|
||||
abstractions when those domains stabilise across two or more vendors.
|
||||
|
||||
98
docs/media/audio/voice/src/effects/effects.md
Normal file
98
docs/media/audio/voice/src/effects/effects.md
Normal file
@@ -0,0 +1,98 @@
|
||||
# `effects/effects`
|
||||
|
||||
The text-to-sound-effect engine interface for `@suaveplan/voice`. Defines the
|
||||
`SoundEffectEngine` contract every sound-effect provider adapter must
|
||||
implement, plus `SoundEffectRequest` / `SoundEffectResult` shapes, Zod
|
||||
schemas for both, and a runtime type guard.
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Sound-effect generation is a distinct capability from speech synthesis:
|
||||
prompts describe non-vocal audio (rain, footsteps, magical chimes, engine
|
||||
roars), durations are typically short and bounded, and looping semantics
|
||||
matter for game and ambient applications. This module models that surface
|
||||
explicitly rather than overloading the TTS interface.
|
||||
|
||||
## Features
|
||||
|
||||
- `SoundEffectEngine` interface with `providerName` and `generate`.
|
||||
- `SoundEffectRequest` covering text prompt, optional duration (`0.5..30`
|
||||
seconds), loop flag, prompt-influence weight, model id, and output format.
|
||||
- `SoundEffectResult` containing audio bytes, format, optional duration, and
|
||||
request id.
|
||||
- Zod schemas excluding `AbortSignal`.
|
||||
- Runtime type guard `isSoundEffectEngine`.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import type { SoundEffectEngine } from "@suaveplan/voice/effects/effects";
|
||||
|
||||
async function generateExplosion(
|
||||
effects: SoundEffectEngine,
|
||||
): Promise<Uint8Array> {
|
||||
const result = await effects.generate({ text: "explosion" });
|
||||
return result.audio;
|
||||
}
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
Generate a 10-second looping ocean ambience with a strong prompt-influence
|
||||
bias and a specific output format:
|
||||
|
||||
```ts
|
||||
import type {
|
||||
SoundEffectEngine,
|
||||
SoundEffectRequest,
|
||||
} from "@suaveplan/voice/effects/effects";
|
||||
|
||||
async function ambientOcean(effects: SoundEffectEngine, signal: AbortSignal) {
|
||||
const req: SoundEffectRequest = {
|
||||
text: "calm ocean waves rolling onto a sandy beach at dawn",
|
||||
durationSeconds: 10,
|
||||
loop: true,
|
||||
promptInfluence: 0.85,
|
||||
outputFormat: "opus_48000",
|
||||
signal,
|
||||
};
|
||||
return effects.generate(req);
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `SoundEffectEngine` — interface with:
|
||||
- `providerName: string` — stable provider identifier.
|
||||
- `generate(request: SoundEffectRequest): Promise<SoundEffectResult>`.
|
||||
- `SoundEffectRequest` — TypeScript interface with:
|
||||
- `text` — required, non-empty prompt.
|
||||
- `durationSeconds?` — bounded to `[0.5, 30]`.
|
||||
- `loop?`, `promptInfluence?` (0..1), `modelId?`, `outputFormat?`,
|
||||
`signal?`.
|
||||
- `SoundEffectResult` — TypeScript interface with:
|
||||
- `audio` — `Uint8Array`.
|
||||
- `format` — `AudioFormat`.
|
||||
- `durationMs?`, `requestId?`.
|
||||
- `SoundEffectRequestSchema` / `SoundEffectResultSchema` — Zod parsers.
|
||||
- `isSoundEffectEngine(value): value is SoundEffectEngine` — duck-type
|
||||
guard.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Duration upper-bound of 30 seconds is the published ElevenLabs maximum and
|
||||
matches typical industry limits. Providers with shorter limits should
|
||||
reject at their boundary with `SoundEffectGenerationFailedError`.
|
||||
- `loop` is a hint that the produced audio should be loop-compatible
|
||||
(start/end alignment, no silent tails). Providers may ignore the flag if
|
||||
unsupported.
|
||||
- `promptInfluence` controls how strictly the output adheres to the prompt —
|
||||
higher values produce more literal interpretations at the cost of variety.
|
||||
- Streaming sound effects is not part of this base interface. Most
|
||||
providers return the full audio in a single response; a future extension
|
||||
may add a `generateStream` method when a use case justifies it.
|
||||
- The runtime guard checks `providerName: string` and `generate: function`.
|
||||
122
docs/media/audio/voice/src/errors/errors.md
Normal file
122
docs/media/audio/voice/src/errors/errors.md
Normal file
@@ -0,0 +1,122 @@
|
||||
# `errors/errors`
|
||||
|
||||
Stable error class hierarchy and code registry for `@suaveplan/voice`. Every
|
||||
typed error in this module extends `VoiceError`, which in turn extends
|
||||
`SuaveplanError` from `@suaveplan/error`. All twelve voice-domain reasons are
|
||||
declared once in `VOICE_CODES` so consumers can compare codes across providers
|
||||
without coupling to a specific transport (ElevenLabs, OpenAI, Google Cloud,
|
||||
etc.).
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Voice provider adapters call out to remote APIs whose failure modes vary —
|
||||
ElevenLabs returns 429 with `Retry-After`, Google Cloud returns specific gRPC
|
||||
status codes, Azure returns its own envelopes. Without a stable domain error
|
||||
hierarchy, every consumer of an `@suaveplan/voice` engine would have to learn
|
||||
the upstream's quirks. This module collapses those into twelve codes that every
|
||||
adapter must remap onto. Callers branch on `error.code` or
|
||||
`error instanceof TtsSynthesisFailedError` and never touch transport details
|
||||
directly.
|
||||
|
||||
## Features
|
||||
|
||||
- 12-code stable registry under namespace `VOICE.*` via
|
||||
`defineCodes("VOICE", [...])`.
|
||||
- `VoiceError` base class extending `SuaveplanError<VoiceErrorCode>` —
|
||||
preserves the cause-chain, structured context, traceId, and JSON /
|
||||
binary wire format.
|
||||
- 12 typed subclasses, one per code, with sensible default messages.
|
||||
- Convenience options: `VoiceNotFoundError({ voiceId })`,
|
||||
`ProviderRateLimitedError({ retryAfterMs })`,
|
||||
`AudioFormatUnsupportedError({ format })`,
|
||||
`SttUnsupportedFormatError({ format })` — these populate
|
||||
`context` automatically.
|
||||
- Caller-supplied `context` is merged on top of derived context so adapters can
|
||||
layer their own attribution (e.g. `tenantId`).
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { VoiceNotFoundError } from "@suaveplan/voice/errors/errors";
|
||||
|
||||
throw new VoiceNotFoundError({ voiceId: "v1" });
|
||||
// → error.code === "VOICE.VOICE.NOT_FOUND"
|
||||
// → error.message === 'Voice not found: "v1"'
|
||||
// → error.context === { voiceId: "v1" }
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
Adapters remap upstream errors onto `VoiceError` codes while preserving the
|
||||
original via `cause`, so a downstream observer can still inspect the transport
|
||||
failure when needed:
|
||||
|
||||
```ts
|
||||
import {
|
||||
ProviderRateLimitedError,
|
||||
TtsSynthesisFailedError,
|
||||
} from "@suaveplan/voice/errors/errors";
|
||||
|
||||
try {
|
||||
await tts.synthesize(req);
|
||||
} catch (transportError) {
|
||||
if (isUpstreamRateLimit(transportError)) {
|
||||
throw new ProviderRateLimitedError({
|
||||
retryAfterMs: transportError.retryAfterMs,
|
||||
cause: transportError,
|
||||
context: { tenantId: ctx.tenantId },
|
||||
});
|
||||
}
|
||||
throw new TtsSynthesisFailedError({ cause: transportError });
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `VOICE_CODES` — frozen record `{ "<reason>": "VOICE.<reason>" }` for all 12
|
||||
reasons. Used as the single source of truth.
|
||||
- `VoiceErrorCode` — TypeScript union of every value in `VOICE_CODES`.
|
||||
- `VoiceError` — abstract base extending `SuaveplanError<VoiceErrorCode>`. Do
|
||||
not instantiate directly; use a typed subclass.
|
||||
- `TtsSynthesisFailedError(opts?)` — code `VOICE.TTS.SYNTHESIS_FAILED`.
|
||||
- `TtsStreamInterruptedError(opts?)` — code `VOICE.TTS.STREAM_INTERRUPTED`.
|
||||
- `SttTranscriptionFailedError(opts?)` — code `VOICE.STT.TRANSCRIPTION_FAILED`.
|
||||
- `SttUnsupportedFormatError(opts?)` — code `VOICE.STT.UNSUPPORTED_FORMAT`;
|
||||
`opts.format` is folded into `context.format` when present.
|
||||
- `VoiceNotFoundError(opts?)` — code `VOICE.VOICE.NOT_FOUND`; `opts.voiceId` is
|
||||
folded into both the default message and `context.voiceId`.
|
||||
- `VoiceCloneRejectedError(opts?)` — code `VOICE.VOICE.CLONE_REJECTED`.
|
||||
- `SoundEffectGenerationFailedError(opts?)` — code
|
||||
`VOICE.EFFECTS.GENERATION_FAILED`.
|
||||
- `VoiceIsolationFailedError(opts?)` — code `VOICE.ISOLATION.FAILED`.
|
||||
- `ProviderNotConfiguredError(opts?)` — code `VOICE.PROVIDER.NOT_CONFIGURED`.
|
||||
- `ProviderRateLimitedError(opts?)` — code `VOICE.PROVIDER.RATE_LIMITED`;
|
||||
`opts.retryAfterMs` is folded into `context.retryAfterMs`.
|
||||
- `VoiceValidationError(opts?)` — code `VOICE.VALIDATION.INVALID_REQUEST`.
|
||||
- `AudioFormatUnsupportedError(opts?)` — code `VOICE.AUDIO.FORMAT_UNSUPPORTED`;
|
||||
`opts.format` is folded into `context.format`.
|
||||
|
||||
Every `opts` object accepts `{ message?, cause?, context?, traceId? }`. The
|
||||
caller's `context` is shallow-merged on top of any derived context, so caller
|
||||
keys win on collision.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- The 12 codes are intentionally narrow. Provider adapters that need finer
|
||||
granularity (e.g. distinguishing voice-clone-quota from voice-clone-policy)
|
||||
should preserve the original error via `cause` and add provider-specific
|
||||
attributes to `context`.
|
||||
- `VoiceError` is abstract because the constructor on `SuaveplanError` is
|
||||
`protected`; instantiating it would lose the discriminant identity that lets
|
||||
consumers branch on `error.constructor` or `instanceof`.
|
||||
- All subclasses default to a stable English message. Real localisation
|
||||
flows through the `MessageResolver` hook on `@suaveplan/error` — no message
|
||||
text in this file is part of the wire format.
|
||||
- `context` is wire-serialised (subject to redaction) by `SuaveplanError.toJSON`
|
||||
/ `toBinary`. Adapters MUST NOT put high-volume payloads (full request
|
||||
bodies, full audio bytes) in `context` — keep it to identifiers and small
|
||||
scalars.
|
||||
96
docs/media/audio/voice/src/isolation/isolation.md
Normal file
96
docs/media/audio/voice/src/isolation/isolation.md
Normal file
@@ -0,0 +1,96 @@
|
||||
# `isolation/isolation`
|
||||
|
||||
The voice-isolation engine interface for `@suaveplan/voice`. Defines the
|
||||
`VoiceIsolationEngine` contract every voice-isolation provider adapter must
|
||||
implement, plus `IsolationRequest` / `IsolationResult` shapes, Zod schemas
|
||||
for both, and a runtime type guard.
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Voice isolation removes background noise, music, and other audio from a
|
||||
mixed recording, leaving only the dominant voice. Use cases include
|
||||
podcast clean-up, meeting recording extraction, and pre-processing for
|
||||
downstream STT. This module locks the canonical request / result so the
|
||||
caller is independent of the active backend.
|
||||
|
||||
## Features
|
||||
|
||||
- `VoiceIsolationEngine` interface with `providerName` and `isolate`.
|
||||
- `IsolationRequest` accepting `audio` (`Uint8Array | ReadableStream | URL`),
|
||||
optional output format, and `AbortSignal`.
|
||||
- `IsolationResult` containing isolated audio bytes, format, and optional
|
||||
request id.
|
||||
- Zod schemas excluding `audio` and `signal`.
|
||||
- Runtime type guard `isVoiceIsolationEngine`.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import type { VoiceIsolationEngine } from "@suaveplan/voice/isolation/isolation";
|
||||
|
||||
async function cleanup(isolation: VoiceIsolationEngine, audio: Uint8Array) {
|
||||
const result = await isolation.isolate({ audio });
|
||||
return result.audio;
|
||||
}
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
Stream a podcast file through isolation directly into a downstream STT
|
||||
engine, without buffering the intermediate result:
|
||||
|
||||
```ts
|
||||
import type {
|
||||
TranscriptionEngine,
|
||||
} from "@suaveplan/voice/stt/stt";
|
||||
import type {
|
||||
VoiceIsolationEngine,
|
||||
} from "@suaveplan/voice/isolation/isolation";
|
||||
|
||||
async function transcribeWithIsolation(
|
||||
isolation: VoiceIsolationEngine,
|
||||
stt: TranscriptionEngine,
|
||||
audio: ReadableStream<Uint8Array>,
|
||||
signal: AbortSignal,
|
||||
) {
|
||||
const isolated = await isolation.isolate({
|
||||
audio,
|
||||
outputFormat: "wav_44100",
|
||||
signal,
|
||||
});
|
||||
return stt.transcribe({ audio: isolated.audio, signal });
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `VoiceIsolationEngine` — interface with:
|
||||
- `providerName: string`.
|
||||
- `isolate(request: IsolationRequest): Promise<IsolationResult>`.
|
||||
- `IsolationRequest` — TypeScript interface with:
|
||||
- `audio` — `Uint8Array | ReadableStream<Uint8Array> | string` (URL).
|
||||
- `outputFormat?`, `signal?`.
|
||||
- `IsolationResult` — TypeScript interface with:
|
||||
- `audio` — `Uint8Array`.
|
||||
- `format` — `AudioFormat`.
|
||||
- `requestId?`.
|
||||
- `IsolationRequestSchema` / `IsolationResultSchema` — Zod parsers.
|
||||
- `isVoiceIsolationEngine(value): value is VoiceIsolationEngine` — duck-type
|
||||
guard.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Streaming isolation is not part of this base interface. The dominant
|
||||
pattern in production is full-file processing because isolation models
|
||||
often need a complete audio context to detect voice activity. Adapters
|
||||
that support streaming may expose it via a provider-specific subpath.
|
||||
- `audio` accepts the same three shapes as STT: in-memory bytes, a stream
|
||||
(which adapters typically buffer), or a URL string.
|
||||
- `outputFormat` defaults are provider-specific. Most isolation services
|
||||
return PCM or WAV by default to avoid lossy compression artefacts on top
|
||||
of the model's reconstruction.
|
||||
- The runtime guard checks `providerName: string` and `isolate: function`.
|
||||
125
docs/media/audio/voice/src/stt/stt.md
Normal file
125
docs/media/audio/voice/src/stt/stt.md
Normal file
@@ -0,0 +1,125 @@
|
||||
# `stt/stt`
|
||||
|
||||
The speech-to-text (STT) engine interface for `@suaveplan/voice`. Defines the
|
||||
`TranscriptionEngine` contract every STT provider adapter must implement,
|
||||
along with `TranscriptionRequest` / `TranscriptionResult` shapes, Zod
|
||||
schemas for both, and a runtime type guard.
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
STT providers (ElevenLabs Scribe, OpenAI Whisper, Google Cloud Speech-to-Text,
|
||||
Azure Speech) accept different request shapes for diarization, entity
|
||||
detection, key-term boosting, and timestamp granularity. This module locks
|
||||
the canonical request and result so consumers never depend on a specific
|
||||
provider's wire format. Adapters map their native bodies onto these shapes
|
||||
at the package boundary.
|
||||
|
||||
## Features
|
||||
|
||||
- `TranscriptionEngine` interface with `providerName` and `transcribe`.
|
||||
- `TranscriptionRequest` covering audio source, optional model, language,
|
||||
audio-event tagging, speaker count, timestamp granularity (word /
|
||||
character), diarization on/off + threshold, multi-channel support,
|
||||
entity-detection / entity-redaction lists or `'all'`, key-term hints,
|
||||
temperature, deterministic seed, and an `AbortSignal`.
|
||||
- `TranscriptionResult` containing transcript text plus optional
|
||||
`WordTimestamp[]`, `SpeakerSegment[]`, `DetectedEntity[]`, language code,
|
||||
duration, and request id.
|
||||
- Zod schemas excluding `audio` (binary / streaming) and `signal` — those
|
||||
validate structurally at runtime.
|
||||
- Runtime type guard `isTranscriptionEngine` for plugin / config-driven
|
||||
scenarios.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import type { TranscriptionEngine } from "@suaveplan/voice/stt/stt";
|
||||
|
||||
async function transcribeBytes(
|
||||
stt: TranscriptionEngine,
|
||||
audio: Uint8Array,
|
||||
): Promise<string> {
|
||||
const result = await stt.transcribe({ audio });
|
||||
return result.text;
|
||||
}
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
Diarized transcription with entity detection and a key-term hint:
|
||||
|
||||
```ts
|
||||
import type {
|
||||
TranscriptionEngine,
|
||||
TranscriptionRequest,
|
||||
} from "@suaveplan/voice/stt/stt";
|
||||
|
||||
async function transcribeMeeting(
|
||||
stt: TranscriptionEngine,
|
||||
audio: ReadableStream<Uint8Array>,
|
||||
signal: AbortSignal,
|
||||
) {
|
||||
const req: TranscriptionRequest = {
|
||||
audio,
|
||||
modelId: "scribe_v2",
|
||||
diarize: true,
|
||||
diarizationThreshold: 0.6,
|
||||
timestampsGranularity: "word",
|
||||
entityDetection: ["person", "organization"],
|
||||
keyterms: ["suaveplan", "OpenSpec"],
|
||||
signal,
|
||||
};
|
||||
const result = await stt.transcribe(req);
|
||||
return {
|
||||
transcript: result.text,
|
||||
speakers: result.speakers ?? [],
|
||||
entities: result.entities ?? [],
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `TranscriptionEngine` — interface with:
|
||||
- `providerName: string`.
|
||||
- `transcribe(request: TranscriptionRequest): Promise<TranscriptionResult>`.
|
||||
- `TimestampGranularity` — `'word' | 'character'`.
|
||||
- `TranscriptionRequest` — TypeScript interface with:
|
||||
- `audio` — `Uint8Array | ReadableStream<Uint8Array> | string` (URL).
|
||||
- `modelId?`, `languageCode?`, `tagAudioEvents?`, `numSpeakers?` (1..32),
|
||||
`timestampsGranularity?`, `diarize?`, `diarizationThreshold?` (0..1),
|
||||
`useMultiChannel?`, `entityDetection?`, `entityRedaction?`, `keyterms?`,
|
||||
`temperature?` (0..2), `seed?`, `signal?`.
|
||||
- `TranscriptionResult` — TypeScript interface with:
|
||||
- `text` — full transcript.
|
||||
- `words?`, `speakers?`, `entities?`, `language?`, `durationMs?`,
|
||||
`requestId?`.
|
||||
- `TranscriptionRequestSchema` / `TranscriptionResultSchema` — Zod parsers
|
||||
excluding `audio` and `signal`.
|
||||
- `isTranscriptionEngine(value): value is TranscriptionEngine` — duck-type
|
||||
guard.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- `audio` is excluded from the Zod request schema because `Uint8Array`,
|
||||
`ReadableStream`, and URL string have no useful canonical encoding for
|
||||
comparison. Adapters validate the runtime shape via `instanceof` or
|
||||
`typeof` checks.
|
||||
- `numSpeakers` is bounded to `[1, 32]`. Providers without diarization should
|
||||
reject `diarize: true` with `SttUnsupportedFormatError` or similar at the
|
||||
adapter boundary, not silently ignore it.
|
||||
- `entityDetection` / `entityRedaction` accept either an explicit array of
|
||||
entity types or the literal `'all'`. The provider's set of recognised
|
||||
entity types is provider-specific.
|
||||
- `keyterms` is for biasing recognition toward domain-specific phrases — not
|
||||
every provider supports it; those that do not should ignore the field
|
||||
silently.
|
||||
- `temperature` is bounded to `[0, 2]` matching the OpenAI Whisper convention.
|
||||
Providers with different scales remap at the boundary.
|
||||
- The runtime guard checks that `providerName` is a string and `transcribe`
|
||||
is a function. Stronger validation requires invoking the engine against a
|
||||
known fixture.
|
||||
109
docs/media/audio/voice/src/tts/tts.md
Normal file
109
docs/media/audio/voice/src/tts/tts.md
Normal file
@@ -0,0 +1,109 @@
|
||||
# `tts/tts`
|
||||
|
||||
The text-to-speech (TTS) engine interface for `@suaveplan/voice`. Defines the
|
||||
`VoiceSynthesizer` contract every TTS provider adapter must implement, plus
|
||||
the `SynthesisRequest` / `SynthesisResult` shapes, Zod schemas for both, and a
|
||||
runtime type guard.
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Voice synthesis providers — ElevenLabs, OpenAI, Google Cloud, Azure, Amazon
|
||||
Polly — surface different request shapes, parameter scales, and streaming
|
||||
semantics. This module locks the canonical request, the canonical result,
|
||||
and the canonical streaming surface so consumers depend on one stable type
|
||||
regardless of which adapter is plugged in. Adapters are responsible for
|
||||
mapping their native request bodies onto `SynthesisRequest` and their native
|
||||
responses onto `SynthesisResult`.
|
||||
|
||||
## Features
|
||||
|
||||
- `VoiceSynthesizer` interface with `providerName`, `synthesize`, and
|
||||
`synthesizeStream`.
|
||||
- `SynthesisRequest` covering text, voice id, optional model, language,
|
||||
voice settings, output format, latency mode, contextual `previousText` /
|
||||
`nextText`, deterministic `seed`, pronunciation dictionary ids, and an
|
||||
`AbortSignal`.
|
||||
- `SynthesisResult` containing audio bytes, format, optional duration,
|
||||
characters used, and request id.
|
||||
- Zod schemas `SynthesisRequestSchema` / `SynthesisResultSchema` that exclude
|
||||
`AbortSignal` (it cannot be serialised).
|
||||
- Runtime type guard `isVoiceSynthesizer` for plugin / config-driven
|
||||
scenarios where the engine is loaded dynamically.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import type { VoiceSynthesizer } from "@suaveplan/voice/tts/tts";
|
||||
|
||||
async function speak(tts: VoiceSynthesizer, text: string, voiceId: string) {
|
||||
const result = await tts.synthesize({ text, voiceId });
|
||||
return result.audio;
|
||||
}
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
Stream audio into a Web Audio `MediaSource` while honouring an `AbortSignal`,
|
||||
and capture per-character usage from the result envelope returned by a
|
||||
non-streaming follow-up call:
|
||||
|
||||
```ts
|
||||
import type {
|
||||
SynthesisRequest,
|
||||
VoiceSynthesizer,
|
||||
} from "@suaveplan/voice/tts/tts";
|
||||
|
||||
async function pipeIntoMediaSource(
|
||||
tts: VoiceSynthesizer,
|
||||
req: SynthesisRequest,
|
||||
source: MediaSourceHandle,
|
||||
signal: AbortSignal,
|
||||
) {
|
||||
const stream = tts.synthesizeStream({ ...req, signal });
|
||||
await stream.pipeTo(source.writable, { signal });
|
||||
// Optionally fetch a non-streaming response to capture charactersUsed:
|
||||
const final = await tts.synthesize({ ...req, signal });
|
||||
source.attribution = { charactersUsed: final.charactersUsed };
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `VoiceSynthesizer` — interface with:
|
||||
- `providerName: string` — stable identifier (e.g. `'elevenlabs'`).
|
||||
- `synthesize(request: SynthesisRequest): Promise<SynthesisResult>`.
|
||||
- `synthesizeStream(request: SynthesisRequest): ReadableStream<Uint8Array>`.
|
||||
- `SynthesisRequest` — TypeScript interface with:
|
||||
- `text` — required, non-empty.
|
||||
- `voiceId` — required, non-empty.
|
||||
- `modelId?`, `languageCode?`, `voiceSettings?`, `outputFormat?`,
|
||||
`latencyMode?`, `previousText?`, `nextText?`, `seed?`,
|
||||
`pronunciationDictionaryIds?`, `signal?`.
|
||||
- `SynthesisResult` — TypeScript interface with:
|
||||
- `audio` — `Uint8Array`.
|
||||
- `format` — `AudioFormat`.
|
||||
- `durationMs?`, `charactersUsed?`, `requestId?`.
|
||||
- `SynthesisRequestSchema` / `SynthesisResultSchema` — Zod parsers.
|
||||
- `isVoiceSynthesizer(value): value is VoiceSynthesizer` — duck-type guard.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- `synthesizeStream` returns a `ReadableStream` synchronously rather than a
|
||||
`Promise<ReadableStream>` so callers can immediately wire the output into a
|
||||
`pipeTo`/`pipeThrough` chain. Adapters that need to await an HTTP handshake
|
||||
do that work inside the stream's `start()` callback.
|
||||
- `seed` is constrained to non-negative integers in the Zod schema. Providers
|
||||
that take signed seeds map at the boundary.
|
||||
- `AbortSignal` is excluded from the Zod schema — there is no canonical wire
|
||||
representation. Adapters thread the signal end-to-end into their HTTP
|
||||
client.
|
||||
- The runtime guard checks the three required members but does not enforce
|
||||
arity or argument types on the methods. Callers that need stronger checks
|
||||
should wrap the result of `isVoiceSynthesizer` with an additional sanity
|
||||
call (`await tts.synthesize(...)` against a known-good fixture).
|
||||
- `pronunciationDictionaryIds` is an array of opaque ids; the lookup
|
||||
resolution is provider-specific.
|
||||
121
docs/media/audio/voice/src/types/types.md
Normal file
121
docs/media/audio/voice/src/types/types.md
Normal file
@@ -0,0 +1,121 @@
|
||||
# `types/types`
|
||||
|
||||
Shared types and Zod schemas used by every voice engine interface in
|
||||
`@suaveplan/voice`. The module is the single source of truth for
|
||||
audio-format strings, voice metadata, request/response timing primitives, and
|
||||
listing pagination shapes — provider adapters import from here so consumers
|
||||
get the same `Voice`, `WordTimestamp`, and `AudioFormat` regardless of which
|
||||
backend is plugged in.
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Voice engines speak audio in many encodings, surface speaker information in
|
||||
many shapes, and page voice catalogues differently. Without a normalised set
|
||||
of types, callers would learn one shape per provider. This module locks the
|
||||
canonical shape: every adapter MUST accept the documented `AudioFormat`
|
||||
strings, return `Voice` objects matching `VoiceSchema`, and emit transcription
|
||||
results made of `WordTimestamp[]` and derived `SpeakerSegment[]`.
|
||||
|
||||
## Features
|
||||
|
||||
- Frozen `AudioFormatSchema` enum covering 12 documented codec/sample-rate/
|
||||
bitrate combinations spanning MP3, PCM, WAV, μ-law, Opus, AAC, FLAC.
|
||||
- `LatencyMode` literal union `0..4` for streaming-latency optimisation.
|
||||
- Voice taxonomy: `VoiceCategorySchema`, `VoiceGenderSchema`, `VoiceAgeSchema`.
|
||||
- `VoiceSettingsSchema` normalising stability, similarity-boost, style to
|
||||
`[0, 1]` plus a boolean speaker-boost flag.
|
||||
- `VoiceSchema` covering id, name, category, optional gender / accent / age,
|
||||
language codes, description, preview URL, settings, free-form metadata.
|
||||
- Pagination types `ListVoicesOptionsSchema` and `VoiceListSchema`.
|
||||
- `AudioSource = Uint8Array | ReadableStream<Uint8Array> | string` — the
|
||||
three input shapes every audio-consuming engine accepts.
|
||||
- Transcription primitives: `WordTimestampSchema`, `SpeakerSegmentSchema`,
|
||||
`DetectedEntitySchema`.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import {
|
||||
AudioFormatSchema,
|
||||
type Voice,
|
||||
VoiceSchema,
|
||||
} from "@suaveplan/voice/types/types";
|
||||
|
||||
const v: Voice = VoiceSchema.parse({
|
||||
id: "v1",
|
||||
name: "Demo",
|
||||
category: "premade",
|
||||
});
|
||||
|
||||
// Validate a format string at the boundary:
|
||||
const fmt = AudioFormatSchema.parse("mp3_44100_128");
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
Build a `VoiceList` cursor pager around any provider's list response:
|
||||
|
||||
```ts
|
||||
import {
|
||||
type ListVoicesOptions,
|
||||
type VoiceList,
|
||||
VoiceListSchema,
|
||||
} from "@suaveplan/voice/types/types";
|
||||
|
||||
async function* paginate(
|
||||
list: (opts: ListVoicesOptions) => Promise<VoiceList>,
|
||||
): AsyncIterable<Voice> {
|
||||
let token: string | undefined;
|
||||
while (true) {
|
||||
const opts: ListVoicesOptions = { pageSize: 100 };
|
||||
if (token !== undefined) opts.pageToken = token;
|
||||
const page = VoiceListSchema.parse(await list(opts));
|
||||
for (const v of page.voices) yield v;
|
||||
if (!page.hasMore || page.nextPageToken === undefined) break;
|
||||
token = page.nextPageToken;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `AudioFormatSchema` / `AudioFormat` — Zod enum + inferred type covering all
|
||||
12 documented format strings.
|
||||
- `LatencyModeSchema` / `LatencyMode` — `0 | 1 | 2 | 3 | 4`.
|
||||
- `VoiceCategorySchema` / `VoiceCategory` —
|
||||
`'premade' | 'cloned' | 'generated' | 'professional'`.
|
||||
- `VoiceGenderSchema` / `VoiceGender` — `'male' | 'female' | 'neutral'`.
|
||||
- `VoiceAgeSchema` / `VoiceAge` — `'young' | 'middle' | 'old'`.
|
||||
- `VoiceSettingsSchema` / `VoiceSettings` — `{ stability?, similarityBoost?,
|
||||
style?, speakerBoost? }`; floats clamped to `[0, 1]`.
|
||||
- `VoiceSchema` / `Voice` — full voice record with required `id`/`name`/
|
||||
`category` and optional taxonomy + metadata.
|
||||
- `ListVoicesOptionsSchema` / `ListVoicesOptions` — paging and filter
|
||||
parameters; `pageSize` clamped to `[1, 100]`.
|
||||
- `VoiceListSchema` / `VoiceList` — `{ voices, nextPageToken?, hasMore }`.
|
||||
- `AudioSource` — input-shape union for audio-consuming engines.
|
||||
- `WordTimestampSchema` / `WordTimestamp` — single-word timing, optional
|
||||
`speakerId`, `channelIndex`, `confidence`.
|
||||
- `SpeakerSegmentSchema` / `SpeakerSegment` — collapsed contiguous runs of
|
||||
same-speaker words.
|
||||
- `DetectedEntitySchema` / `DetectedEntity` — entity-detection annotation.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- `AudioFormat` strings deliberately follow the `<codec>_<sampleRate>[_<bitrate>]`
|
||||
convention so adapters can parse them without a per-format lookup table.
|
||||
- `previewUrl` is validated by Zod's `.url()` check — provider adapters must
|
||||
ensure the URL is reachable; this package does not fetch it.
|
||||
- `pageSize` upper bound is `100`; providers may request smaller pages but
|
||||
must never exceed this without an explicit OpenSpec change.
|
||||
- `AudioSource` is a TypeScript type, not a Zod schema — `ReadableStream`
|
||||
cannot be deeply parsed by Zod without consuming the stream. Adapters
|
||||
validate the shape at runtime via `instanceof` checks.
|
||||
- `confidence` is optional: not every provider surfaces it. When present it
|
||||
is normalised to `[0, 1]`.
|
||||
- `metadata` is a free-form `Record<string, unknown>` so adapters can attach
|
||||
provider-specific labels without polluting the canonical surface.
|
||||
111
docs/media/audio/voice/src/voices/voices.md
Normal file
111
docs/media/audio/voice/src/voices/voices.md
Normal file
@@ -0,0 +1,111 @@
|
||||
# `voices/voices`
|
||||
|
||||
The voice repository interface for `@suaveplan/voice` — read-only listing and
|
||||
lookup of voices available to the active provider. Defines the
|
||||
`VoiceRepository` contract every voice provider adapter must implement, plus
|
||||
the supporting Zod schemas and a runtime type guard.
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
|
||||
## Purpose
|
||||
|
||||
Provider voice catalogues differ in pagination scheme (cursor / page-token /
|
||||
offset), filter taxonomy, and "shared" vs "owned" voice splits. This module
|
||||
locks the canonical surface — `list(options?)` returning a `VoiceList`
|
||||
envelope and `get(voiceId)` returning `Voice | null`. Provider adapters
|
||||
collapse their native responses onto these shapes, so consumers can browse
|
||||
voices without learning each backend's quirks.
|
||||
|
||||
## Features
|
||||
|
||||
- `VoiceRepository` interface with `providerName`, `list`, and `get`.
|
||||
- `list(options?)` returns a `VoiceList` envelope (`voices`, `nextPageToken`,
|
||||
`hasMore`).
|
||||
- `get(voiceId)` resolves to `Voice | null` — adapters MUST map upstream
|
||||
not-found codes to `null` rather than throwing.
|
||||
- Zod schemas re-exporting `ListVoicesOptionsSchema` (as
|
||||
`VoiceRepositoryListOptionsSchema`), `VoiceListSchema` (as
|
||||
`VoiceRepositoryListResultSchema`), and a `Voice | null` union for the
|
||||
`get` result.
|
||||
- Runtime type guard `isVoiceRepository`.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import type { VoiceRepository } from "@suaveplan/voice/voices/voices";
|
||||
|
||||
async function findVoiceByName(
|
||||
voices: VoiceRepository,
|
||||
name: string,
|
||||
) {
|
||||
const page = await voices.list({ search: name, pageSize: 25 });
|
||||
return page.voices[0] ?? null;
|
||||
}
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
Async-iterate every voice across all pages, with a guarded language filter:
|
||||
|
||||
```ts
|
||||
import type {
|
||||
ListVoicesOptions,
|
||||
Voice,
|
||||
VoiceRepository,
|
||||
} from "@suaveplan/voice";
|
||||
|
||||
async function* allEnglishVoices(
|
||||
voices: VoiceRepository,
|
||||
): AsyncIterable<Voice> {
|
||||
let token: string | undefined;
|
||||
while (true) {
|
||||
const opts: ListVoicesOptions = { languageCode: "en", pageSize: 100 };
|
||||
if (token !== undefined) opts.pageToken = token;
|
||||
const page = await voices.list(opts);
|
||||
for (const v of page.voices) yield v;
|
||||
if (!page.hasMore || page.nextPageToken === undefined) break;
|
||||
token = page.nextPageToken;
|
||||
}
|
||||
}
|
||||
|
||||
async function lookupOrFail(voices: VoiceRepository, id: string): Promise<Voice> {
|
||||
const v = await voices.get(id);
|
||||
if (v === null) throw new Error(`voice ${id} not found`);
|
||||
return v;
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- `VoiceRepository` — interface with:
|
||||
- `providerName: string` — stable identifier (e.g. `'elevenlabs'`).
|
||||
- `list(options?: ListVoicesOptions): Promise<VoiceList>`.
|
||||
- `get(voiceId: string): Promise<Voice | null>`.
|
||||
- `VoiceRepositoryListOptionsSchema` — re-export of `ListVoicesOptionsSchema`
|
||||
for ergonomic discovery.
|
||||
- `VoiceRepositoryListResultSchema` — alias of `VoiceListSchema`.
|
||||
- `VoiceRepositoryGetResultSchema` — Zod union of `VoiceSchema` and
|
||||
`z.null()`.
|
||||
- `isVoiceRepository(value): value is VoiceRepository` — duck-type guard.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- The `null`-on-not-found contract for `get` is deliberate: callers commonly
|
||||
check existence before falling back to a default voice. Throwing would
|
||||
force a try/catch around every probe. Adapters that receive a 404 from
|
||||
upstream MUST map it to `null`; any other error is propagated as a
|
||||
`VoiceError` subclass.
|
||||
- Voice cloning is intentionally absent. ElevenLabs supports instant /
|
||||
professional cloning; OpenAI does not at all; Azure has its own
|
||||
"Custom Neural Voice" workflow with a multi-week verification process.
|
||||
A future `VoiceCloner` interface will land in a dedicated subpath when the
|
||||
design is settled and ≥ 2 providers are mapped onto it.
|
||||
- The schemas in `voices.schemas.ts` are re-exports / aliases of the
|
||||
authoritative schemas in `types/types.ts`. They exist to give callers
|
||||
ergonomic discovery without forcing them to know that `ListVoicesOptions`
|
||||
and `VoiceList` are shared with non-repository consumers.
|
||||
- The runtime guard checks the three required members but does not enforce
|
||||
argument shapes on the methods. Callers needing stronger validation should
|
||||
call `await voices.list({})` against a fixture and assert the response.
|
||||
Reference in New Issue
Block a user