Fix stripping of Gemma reasoning markers in streamed output#6398
Fix stripping of Gemma reasoning markers in streamed output#6398C0deRatoR wants to merge 2 commits into
Conversation
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| THINK_TAG_END = "</think>" | ||
| THINK_TAG_PAIRS = ( | ||
| (THINK_TAG_START, THINK_TAG_END), | ||
| ("<|channel>thought", "<channel|>"), |
There was a problem hiding this comment.
since this is gemma only, can we make the start and end tags configurable per model, so it applies to only gamma. for future models we can set that tag directly, and if not set fallback to the default
There was a problem hiding this comment.
Done in 68b535d. ThinkingTokenFilter now accepts configurable start/end tags; the Inference path maps only google/gemma-4-31b-it to the Gemma markers and falls back to <think>/</think> for every other model. I also added a regression test confirming Gemma markers remain untouched without explicit tag configuration.
| return call_chunk | ||
|
|
||
| delta.content = llm_utils.strip_thinking_tokens(delta.content, thinking) | ||
| delta.content = llm_utils.strip_thinking_tokens( |
There was a problem hiding this comment.
maybe move strip to the top of _parse_choice so it also applied for the text content alongside a tool call?
There was a problem hiding this comment.
Done in 68b535d. Filtering now runs immediately after the delta is None guard, before tool-call handling, so text emitted alongside a tool call is filtered too. Added a regression covering visible text followed by an empty Gemma thought block on a tool-call chunk.
3 participants
Fixes #6375
Problem
Gemma channel-thought markers and their hidden reasoning pass through the shared thinking-token filter into customer-visible output, including TTS, transcripts, and chat history.
Root cause
strip_thinking_tokensonly recognizes<think>blocks and uses anasyncio.Event, which records whether the stream is inside a reasoning block but cannot retain marker fragments split across chunks. It also does not handle complete or multiple reasoning blocks within one chunk.Change
<think>and Gemma<|channel>thought/<channel|>blocksTests
uv run --frozen pytest tests/test_llm_utils.py --unit -q— 11 passeduv run --frozen pytest --unit --no-concurrent --ignore=tests/test_room.py -q— 1116 passed, 2 skippedmake check— formatting, Ruff, and mypy passedtests/test_room.pycould not run locally because this environment has no Docker executable; its nine failures were fixture setup errors before test execution.Scope
No new dependencies, provider calls, or unrelated inference changes.