Endpointing: how does the auto-submit factor in pFinishedSpeaking? (server eval shows submits decorrelated from the score)

[rosscado]

Context

On the API side we've stood up a server-side endpointing eval to measure and improve pFinishedSpeaking (the float the finishedSpeaking decorator returns on each /transcribe response). The missing piece was a non-circular ground-truth label for "did the user actually finish their turn?", which the extension now provides via the turn-outcome emitter shipped in #505 (POST /turn-outcome, carrying user_resumed, last_sequence_number, submitted_at, last_speech_ended_at).

We now have the first real data — 40 auto-submitted turns from live sessions — and before we invest in recalibrating pFinishedSpeaking, we need to understand how (or whether) the extension actually consumes that score when deciding to submit a turn. The eval is only meaningful if model improvements would change endpointing behavior.

What the data shows

Two findings that we can't interpret without knowing the client's submit logic:

1. Submits are decorrelated from pFinishedSpeaking. 48% of the utterances that drove an auto-submit had pFinishedSpeaking ≤ 0.40 — i.e. the model was actively signalling "not finished." If the submit were gated on a p ≥ threshold test, almost every submit would sit at p ≥ 0.8. Instead they're spread across the whole range. Several are clearly mid-sentence fragments the model correctly scored ~0.2 ("of", "start this", "So one of the…"), yet they were submitted and the user then resumed speaking.

2. The submit delay doesn't look score-modulated. We measured the endpointing wait as submitted_at − last_speech_ended_at per turn:

submit-driving p	n	mean wait	median wait
low (≤0.4)	18	1.65s	1.35s
high (≥0.8)	22	1.95s	1.15s

Pearson r(p, wait) = +0.11 — essentially uncorrelated (slightly positive). The working hypothesis on our side was that the extension waits something like max_delay * (1 - pFinishedSpeaking), which would predict a strong negative correlation (low p → long wait). The data doesn't show that; the wait is ~flat regardless of the score.

Questions

1. What actually triggers the auto-submit of a transcribed turn — a Silero-VAD silence timeout, a pFinishedSpeaking threshold, or some combination? If a combination, how are they weighted?
2. Does pFinishedSpeaking factor into the submit decision or its timing at all today? If so, what's the formula (is our max_delay * (1 - p) guess close)? If not, is it currently informational only?
3. For the turn-outcome signal's last_sequence_number — is it the utterance whose pFinishedSpeaking drove the submit decision, or just the last /transcribe response received in the turn? These can differ: if a low-confidence trailing fragment arrives right around submit time, last_sequence_number may point at it rather than at the utterance the decision was actually based on — which would explain both findings above, and would mean our eval is joining the wrong score to each turn.

The answer determines whether the right next step is model work (recalibrate pFinishedSpeaking), client work (adjust the submit policy), or a fix to which sequence number the turn-outcome reports. Not urgent — we just don't want to optimize a signal the product isn't listening to. Thanks!

[rosscado]

Thanks for the analysis — the data is interpretable, and the short version is: your findings are exactly what the implemented policy predicts, because the client uses pFinishedSpeaking with the opposite sign from your working hypothesis. There is no threshold gate, and the wait is proportional to p, not to (1−p). Answers to the three questions, then what that means for next steps.

1. What actually triggers the auto-submit

A sequence, not a weighted combination:

1. Silero VAD speech-end (with audio) stamps context.timeUserStoppedSpeaking and fires the /transcribe request (ConversationMachine.ts#L1020-L1036).
2. Each /transcribe response re-enters the accumulating state, which (re)arms two timers (ConversationMachine.ts#L1069-L1083): a score-modulated submissionDelay (formula below) and a fixed 10s fallback (USER_STOPPED_TIMEOUT_MS).
3. When a timer fires, the guard submissionConditionsMet must pass: auto-submit preference on, user not currently speaking, no in-flight transcription requests, non-empty transcript, and a VAD stop actually recorded (ConversationMachine.ts#L680-L713). If the user resumes speaking first, the cycle resets.

So VAD is the trigger and precondition; pFinishedSpeaking (and tempo) shape only the wait duration. Nothing ever blocks a submit because p is low — a low score just means a different (in fact shorter, see below) wait. That's why your submits are spread across the whole p range rather than clustering at p ≥ 0.8: a p ≥ threshold test doesn't exist. (The only content-based gate is the separate responseAnalysis.shouldRespond flag used in discretionary mode — unrelated to p.)

2. Does p factor into timing, and is max_delay * (1 - p) close?

It factors in, but your guess has the sign inverted. The actual formula (TimerModule.ts#L10-L34, unchanged since Nov 2023, invoked from ConversationMachine.ts#L729-L806):

initialDelay = 7000ms · pFinishedSpeaking · (1 − tempo)
finalDelay   = max(initialDelay − elapsedSinceVadStop, 0)

Three consequences that match your data point-for-point:

• Wait ∝ p, so the predicted correlation is positive, not strongly negative. Your measured r = +0.11 has exactly the sign the implementation predicts. It's weak rather than strong because (a) (1 − tempo) multiplies in and is uncorrelated with p, and (b) the − elapsed subtraction means your measured submitted_at − last_speech_ended_at is floored at transcription latency: the client-side wait from speech end is effectively max(7000·p·(1−tempo), STT latency). With typical tempo values the p-term lands in the ~0.5–2s range, right where the latency floor blurs it. You have tempo per utterance server-side, so you can verify this directly: recompute max(7·p·(1−tempo), stt_latency) for your 40 turns and it should track the measured waits.
• Your finding 1 is the formula working as written. A mid-sentence fragment scored p ≈ 0.2 gets initialDelay ≈ 1.4s·(1−tempo) — often sub-second, frequently fully consumed by transcription latency, i.e. submit-immediately. The utterances the model correctly flags as unfinished are precisely the ones submitted fastest, which is why users resume after them. The score isn't being ignored; it's being obeyed backwards.
• This is inverted relative to the client's own documented intent (doc/voice-endpointing.md: "a patient listener... give the user every chance to continue" — which (1−p) would serve, not p). I've filed Endpointing: submission delay is proportional to pFinishedSpeaking — patience spent where it's least needed #521 to re-derive the delay policy on the client.

3. What last_sequence_number reports

It's the highest sequence number received in the turn, paired with that same response's pFinishedSpeaking — the two are persisted together with an out-of-order guard so the reported (seq, p) always describe the same utterance (ConversationMachine.ts#L278-L291).

Is that the decision-driving utterance? In the common case, yes. The submission timer is re-armed by every arriving /transcribe response using that event's p, so the response that arms the final timer is the last-arrived one — and with in-order delivery, last-arrived = highest-seq = what turn-outcome reports. Your specific worry actually resolves the other way: a low-confidence trailing fragment arriving near submit time doesn't get skipped over — it re-arms the timer with its own (short, because low-p) delay and becomes the reported sequence number. So the join is correct there, and those rows are real: the fragment genuinely drove (and accelerated) the submit.

Two genuine divergence cases to keep in mind, both minor at n=40:

• Out-of-order arrival: a stale lower-seq response arriving late re-arms the timer with its p, while turn-outcome reports the higher-seq pair. Rare, and arguably the reported pair is the more meaningful label anyway.
• 10s-fallback submits: if the fixed USER_STOPPED_TIMEOUT_MS timer fires instead, the reported p had no causal role in the timing. Your wait distribution (means ~1.6–2s) says these are a small minority; anything with wait ≈ 10s in your data is one of these.

What this means for your fork in the road

Client work first (#521), model recalibration second, eval join fix mostly unnecessary. The eval infrastructure is measuring the right thing and joining the right score — what it caught is that the client spends its patience where it's least needed. Recalibrating pFinishedSpeaking now would sharpen a signal the product consumes upside-down; once #521 lands a corrected policy, the same eval gives you a clean before/after and a basis for choosing constants empirically. Your 40-turn dataset (and its growth) is exactly what we'll want for that — let's coordinate there.

[rosscado]

Confirmed the formula empirically against the 40 turns, per your suggestion. I reconstructed tempo from the captured word_count/duration_s (the decorator's wpm→tempo ramp is deterministic) and applied 7·p·(1−tempo):

p	tempo	predicted	observed
0.80	0.13	4.89s	4.89s
0.90	0.17	5.25s	5.25s
0.30	0.14	1.81s	1.82s

• r(p, observed) = +0.11 (score alone)
• r(7·p·(1−tempo), observed) = +0.76
• r(max(7·p·(1−tempo), floor), observed) = +0.79 (your full model, floor = observed min 0.69s)

Near-exact on every above-floor case; the high-tempo rows collapse to the STT-latency floor exactly as you described. So the join is sound and the eval reproduces the client's own timing math — thanks for the precise walkthrough, that resolved it completely.

Picking up the corrected-policy / constant-selection thread on #521, where I've simulated a few candidates against these turns.

[rosscado]

Update for the saypi-api side: the client fix landed — #521 is resolved (PR #537, merged to main). The submit wait is now max(7000ms · (1 − p) · (1 − tempo), 500ms floor) − elapsed, i.e. patience goes to the low-p utterances the model flags as unfinished, and the 500ms grace floor covers overconfident high-p scores when transcription returns fast.

Two things that matter for the eval:

1. The before/after boundary is the next store release, not the merge — users get the new policy when the next extension version ships and auto-updates. Until then the turn-outcome stream keeps reflecting the old (proportional) policy. I'd hold off on recalibrating pFinishedSpeaking until you can segment the data by extension version (the events carry the version header) and re-baseline on post-release turns.
2. What to expect in the new data: the wait should now correlate negatively with p (predicted wait ≈ max(7·(1−p)·(1−tempo), stt_latency, 0.5s) seconds), and the user_resumed rate on low-p submits should drop — those turns now get up to ~5.6s of grace instead of the shortest waits. If it doesn't drop, that's evidence the score itself needs the recalibration work, and the eval will finally be measuring a signal the product listens to right-side-up.

The 500ms floor and the 7s cap are explicitly empirical knobs now (documented in doc/voice-endpointing.md) — happy to tune them from your data once there's a post-release sample.