fix(voice): spoken audio vs transcript misalignment#6404
Draft
Bobronium wants to merge 1 commit into
Draft
Conversation
…r turn The false-interruption resume timer raced the turn machinery: when a caller barged into a reply and their STT final landed later than false_interruption_timeout (2s default) after VAD end-of-speech, the timer fired first and resumed the stale reply out loud over the caller's already-finished turn. Each interim transcript re-paused and re-armed the timer, so the outdated tail kept bleeding out in bursts until the final arrived — the transcript/audio divergence observed in staging sim runs SR_ye3sFKLN2ZYz and SR_4UpFo6cNC7wA (chat commits reply B while the audio in its window is stale reply A). Resuming is now subordinate to turn evidence: - VAD segments longer than the new interruption.false_interruption_max_speech_duration (default 1.5s) are treated as real speech whose transcript is in flight: the resume timer is not armed; an interrupt-watchdog (endpointing.max_delay) bounds the pause in case the final never arrives. - Any non-empty interim/final transcript marks the pause as a real turn (transcript_seen); from then on the only exits are turn commit or interrupt — never resume. - Pause bookkeeping now has a single exit point (_release_paused_speech) that maintains the invariant "output paused iff a _PausedSpeechInfo owns it": the timer's abandon paths and _cancel_speech_pause no longer leave the shared audio output paused without an owner. Short VAD-only noise blips (no transcript) keep the existing 2s pause-then-resume behavior. tests/test_transcript_audio_divergence.py reproduces the recorded run with amplitude-tagged TTS audio (live sink + recorder channel): its regression tests failed before this fix, and whitebox tests pin the pause-ownership invariant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generated by Claude
What this run's timeline becomes
transcript_seen, keep pausedThe caller hears ≤1.4s of A (pre-barge-in), silence, then reply B — audio matches the transcript. My three repro tests and the sibling stale-resume test go green; the control and the pinned noise-blip test stay green.
The accepted trade-off: a long noise burst (>1.5s of VAD speech that never produces a transcript) now ends in an interrupt after
endpointing.max_delayinstead of a resume at 2s. That's the right side of the asymmetry — a wrongly-silenced agent gets re-prompted; a wrongly-resumed one talks over the caller and corrupts the transcript/audio record for every downstream consumer (LLM context, sims, insights). I'd reject the two cheaper alternatives: raisingfalse_interruption_timeoutglobally freezes noise recovery for everyone, and an adaptive per-provider timeout adds machinery without removing the race.tests/test_transcript_audio_divergence.py reproduces the recorded run with amplitude-tagged TTS audio (live sink + recorder channel): its regression tests failed before this fix, and whitebox tests pin the pause-ownership invariant.