feat(#83): outbound voice vertical slice — Harness text to TTS audio track - #125
Conversation
…track Smallest Universal Voice Agent slice (#83), entirely inside agent-runtime: a resident Agent turns its local Harness response text into ordered playable audio written to an injectable outbound track writer. - speech/types.ts: TTS seam next to STT (StreamingTtsSession/Provider, TtsAudioChunk, descriptor createTtsProvider) - voice/chunking.ts: pure incremental sentence/phrase chunker (CJK + latin terminators, decimal guard, clause-level emergency splits) - voice/speaker.ts: VoiceSpeaker with strict FIFO ordering, bounded pending queue (backpressure at the sink write), epoch-based stale cancellation (newest turn wins; late provider frames never reach the sink), lazy single-sink lifecycle with dead-track recreation - voice/providers/openaiCompatible.ts: reference BYOK adapter for any OpenAI-compatible /audio/speech endpoint returning raw PCM; injectable fetch, status-only HTTP errors, credential-redacted transport errors; registered in the production registry as "openai-compatible" - voice/ttsProvider.ts: resolveConfiguredTtsProvider reusing the local BYOK credential store (FREE4CHAT_TTS_PROVIDER override) - core/runtime.ts: optional createVoiceOutput wiring — cancel stale speech when a new addressed turn starts, speak after sendText, close on cleanup; voice failures never affect the text path No Worker/DO/MCP changes: the RoomSession authorization boundary is untouched and agent SFU sessions remain subscribe-only. Production room-audible playback lands with the follow-up that extends the publish boundary; until then the sink seam ships with deterministic fakes. Tests (34 new): chunking determinism, ordering, backpressure, cancellation/stale-frame filtering, truncation bound, sink lifecycle, provider mapping without secret leakage, resolution matrix, and an end-to-end runtime test driving Harness turns through chunker -> TTS -> fake TrackLocal with mid-flight interruption. Gates: agent-runtime build/lint/format/test (168 passing) + app lint/type-check/build all green.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 95e07a10b5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| export function productionSpeechRegistry(): SpeechProviderRegistry { | ||
| const registry = new MutableSpeechProviderRegistry() | ||
| registry.register(doubaoSpeechProvider) | ||
| registry.register(openAiCompatibleSpeechProvider) |
There was a problem hiding this comment.
When an operator runs free4chat-agent speech setup openai-compatible, registering this TTS-only descriptor makes it available to the existing speech CLI, whose LocalSpeechStore.saveProvider() unconditionally writes the selected ID to speech.stt.provider. Subsequent createConfiguredSpeechTranscriber() calls therefore return null because this descriptor has no createSttProvider, disabling a previously configured Meeting Notes transcriber merely by setting up TTS. Persist a separate TTS selection or keep TTS-only providers out of the shared STT setup path.
Useful? React with 👍 / 👎.
| const endpoint = `${this.config.baseUrl.replace(/\/+$/, "")}/audio/speech` | ||
| let response: Response | ||
| try { | ||
| response = await this.fetchImpl(endpoint, { |
There was a problem hiding this comment.
If /audio/speech stalls, this fetch has neither an AbortSignal nor a timeout, and the session's close() is a no-op. VoiceSpeaker.cancel() cannot start the newer turn until the old async iterator unwinds, while VoiceSpeaker.close() waits on the same drain promise, so a hung provider can indefinitely block subsequent speech and prevent runtime cleanup from reaching leave/adapter/client shutdown. Track the active request with an AbortController and abort it from close().
Useful? React with 👍 / 👎.
| }) | ||
| } catch (error) { | ||
| if (this.stopped || myEpoch !== this.epoch) return | ||
| this.sinkBroken = true |
There was a problem hiding this comment.
When synthesis or writeAudio() fails after the current chunk has been shifted, this catch reports turnFailed but leaves the remaining chunks in pending and keeps the same epoch. The drainPromise.finally() callback consequently starts another drain immediately, causing later sentences from the failed turn to be spoken and potentially emitting turnFinished after turnFailed. Clear the pending chunks or invalidate the epoch when failing a turn.
Useful? React with 👍 / 👎.
|
|
||
| private async ensureSink(): Promise<OutboundVoiceSink> { | ||
| if (this.sink && !this.sinkBroken) return this.sink | ||
| this.sink = await this.createSink() |
There was a problem hiding this comment.
After any drain error sets sinkBroken, the next call replaces this.sink without closing the previous sink. This leaks the old outbound track and also abandons a healthy sink when the error originated in TTS rather than writeAudio; repeated provider failures can therefore accumulate live track resources. Close and detach the old sink before creating its replacement, and only mark it broken for sink failures.
Useful? React with 👍 / 👎.
P1 from the exact-head review of PR #125: TTS resolution previously fell back to config speech.stt.provider, and speech setup always wrote that slot — so configuring a TTS-only provider could silently displace an existing Doubao STT selection. - storage: SpeechConfig gains speech.tts.provider; saveProvider accepts { slot: "stt" | "tts" } with the historical "stt" default so existing callers and config files behave exactly as before - speech setup routes the slot by capability: stt-capable providers keep using the stt slot; TTS-only providers activate through the tts slot - voice/ttsProvider.ts reads only FREE4CHAT_TTS_PROVIDER or speech.tts.provider (never the stt slot) - openai-compatible provider/session gain a toJSON guard: TypeScript private fields are runtime-enumerable, so an accidental stringify of a resolved provider would otherwise emit the API key Regression coverage: doubao STT + openai-compatible TTS both stay selected/resolvable side by side; a tts-only `speech setup` run leaves the existing stt selection, its credentials, and Doubao STT resolution untouched; the stt slot alone never activates tts; provider/session serialization never exposes the credential. Gates: agent-runtime lint/format/build/pack + tests 172/172 green; app lint/type-check/build green.
… interface
Replaces the placeholder openai-compatible provider with the real product
provider: 豆包语音合成模型 2.0 via the official V3 output-unidirectional
HTTP interface (POST /api/v3/tts/unidirectional, chunked NDJSON stream).
Protocol (per official docs and cross-checked OSS clients):
- X-Api-Key auth — the same console key family as Doubao ASR; no
AppId/AccessToken; DOUBAO_API_KEY is reused for both capabilities
- X-Api-Resource-Id: seed-tts-2.0; speaker defaults to the 2.0 voice
zh_female_shuangkuaisisi_uranus_bigtts, overridable with DOUBAO_TTS_VOICE
- request: user.uid + req_params{text, speaker,
audio_params:{format:"pcm", sample_rate:24000}}
- response objects {code,message?,data?}: code 0 carries one base64 PCM
chunk (raw s16le/24kHz/mono), code 20000000 terminates, anything else is
an error; brace-aware incremental scanner tolerates newline-split or
concatenated objects across network chunk boundaries
Robustness: connect timeout, per-chunk idle timeout, deterministic cancel
(abort raced against pending reads so close() always settles promptly),
truncated-stream failure (missing terminator fails loudly), empty-text
short-circuit, and key-redacted error surfaces on every path.
Selection/storage: saveProvider now activates a provider in every
capability slot it supports ({slots:[...]}, default ["stt"] unchanged);
`speech setup doubao` therefore powers speech.stt.provider AND
speech.tts.provider from one credential while slots stay independent.
The doubao descriptor now advertises stt+tts with createSttProvider and
createTtsProvider.
Local real-audio entry point (room SFU playback remains unwired):
free4chat-agent speech speak-tts --text "..." --out out.pcm [--wav]
resolves the configured TTS provider exactly like the runtime wiring and
writes provider audio to a file without ever printing the key.
Removed: voice/providers/openaiCompatible* and their tests. Tests: new
doubaoTts protocol suite (headers/URL/body, boundary-crossing stream
parse, in-stream + HTTP errors, truncation, timeout, cancellation, key-leak
guards, WAV header) and resolver/CLI regressions proving one
DOUBAO_API_KEY resolves Doubao STT + Doubao TTS side by side.
Gates: agent-runtime lint/type-check/build/pack + tests 179/179 green;
app lint/type-check/build green (public docs updated).
…etion P2 from the exact-head review of PR #125: classifyTtsStreamObject mapped unparsable JSON and code-0 objects without audio data to a lenient shape that DoubaoTtsSession.consumeObject treated as a normal stream end, so a malformed or truncated response could silently complete spoken answers. - only code 20000000 classifies as kind:end; balanced-but-unexpected objects and code-0 objects lacking non-empty data classify as invalid - consumeObject maps invalid to a tts_invalid_stream_object protocol error instead of completion - deterministic tests: malformed object fails, code-0-without-data fails, terminator-only stream completes normally, plus direct classifier coverage of every shape Doubao TTS 2.0 V3 X-Api-Key / seed-tts-2.0 unidirectional implementation, selection slots, CLI probe, and SFU boundary unchanged. Gates: agent-runtime lint/type-check/build/pack + tests 183/183 green.
…tream Live V3 verification against /api/v3/tts/unidirectional showed official streams interleave code-0 frames carrying data:null plus a sentence metadata object (phonemes/text/words) between audio chunks; the strict classifier rejected them as invalid and failed real synthesis. - code-0 frames with a plain-object sentence field are now classified as metadata and safely ignored while waiting for the terminator - everything else stays fail-closed: malformed JSON, unexpected shapes, and code-0 objects without data AND without sentence metadata still raise tts_invalid_stream_object; only 20000000 ends a stream - deterministic tests: full realistic stream (audio x2 + metadata + terminator) completes with both chunks intact; data:null without sentence metadata still fails closed; classifier covers the metadata shape directly Verified live with the locally configured DOUBAO_API_KEY via `speech speak-tts --wav`: 221288 PCM bytes written as a valid RIFF/24kHz/mono WAV (key never printed). Gates: agent-runtime lint/type-check/build/pack + tests 185/185 green.
2 participants
Refs #83 — vertical slice only; this prerequisite does not implement the umbrella issue.
What this adds
This PR implements the local resident-Agent voice pipeline:
Harness response text → bounded sentence/chunk splitting → Doubao Speech Synthesis 2.0 → VoiceSpeaker ordering/backpressure/cancellation → OutboundVoiceSink.
The TTS provider is the real Doubao Speech Synthesis 2.0 integration:
The Doubao provider exposes both STT and TTS capabilities while keeping speech.stt.provider and speech.tts.provider as independent activation slots. speech setup doubao activates both capabilities from the same API key.
Local real-audio check
After building the runtime:
The command writes local PCM/WAV output and never prints the API key. Use a 2.0 speaker available to the account if overriding DOUBAO_TTS_VOICE.
Scope boundary
This PR deliberately does not change Worker/DO/MCP authorization or enable Agent SFU publishing. The output terminates at the injectable OutboundVoiceSink; room-audible delivery still requires a separate grant-gated publish boundary and real Pion/werift TrackLocal writer.
Verification
Follow-ups tracked under #83