WebSocket audio backend via AudioIO ABC (updated #189) - #215
Conversation
Integrate the websocket audio backend from #189 (by reisbauer03) into the audio_io package, reframed onto the AudioIO abstract base class so the engine runs identically on local hardware (sounddevice) or a network backend. - Add AudioIO ABC (base.py) and make both backends subclass it; keep AudioProtocol as a back-compat alias. - WebsocketAudioIO: /microphone and /speaker endpoints, 16kHz float32 streaming, VAD per client, playback ack (played/time/sampleRate/reset). - Rooms are opt-in via backend role 'rooms: true' (off by default); when on, multi-mic ownership + segregate_speakers routing. Off => single-source broadcast to all speakers. - get_audio_system()/GladosConfig accept backend_options; add configs/glados_websocket_config.yaml, protocol docs, and browser + Python reference clients. Add websockets>=16.0 dependency.
📝 WalkthroughWalkthroughThis PR implements a WebSocket-based audio input/output backend for GLaDOS. It introduces an ChangesWebSocket Audio IO Implementation
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant MicClient
participant WebsocketAudioIO
participant SpeakerClient
MicClient->>WebsocketAudioIO: connect /microphone, stream float32 audio
WebsocketAudioIO->>WebsocketAudioIO: run VAD, arbitrate mic control by room
WebsocketAudioIO->>WebsocketAudioIO: enqueue controlled samples in sample queue
SpeakerClient->>WebsocketAudioIO: connect /speaker, send room selection
WebsocketAudioIO->>SpeakerClient: send sample rate and scheduled audio data
SpeakerClient->>WebsocketAudioIO: send "played" acknowledgement
WebsocketAudioIO->>WebsocketAudioIO: measure_percentage_spoken, detect interruption
sequenceDiagram
participant GladosConfig
participant get_audio_system
participant WebsocketAudioIO
GladosConfig->>get_audio_system: pass backend_type and audio_io_options
get_audio_system->>WebsocketAudioIO: create with options and vad_threshold
WebsocketAudioIO->>WebsocketAudioIO: start server thread, validate options
WebsocketAudioIO-->>get_audio_system: return AudioIO instance
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 12
🧹 Nitpick comments (1)
src/glados/audio_io/websocket_io.py (1)
345-353: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winNarrow
_audio_databefore you dereference it.
self._audio_datais typedAudioData | None. Lines 348 and 386-390 access attributes without aNonecheck. The runtime path is currently protected by_is_playing, but a type checker reports these accesses, and a future change to the playback lifecycle turns them intoAttributeError.Add an explicit guard inside the lock in both places.
♻️ Proposed refactor for `set_flags_once`
with self._audio_lock: - if self._audio_data.track_id == track_id: + if self._audio_data is not None and self._audio_data.track_id == track_id: self._playback_was_interrupted = was_interruptedAlso applies to: 385-390
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/glados/audio_io/websocket_io.py` around lines 345 - 353, Within both lock-protected paths in the relevant websocket audio methods, explicitly guard that self._audio_data is not None before accessing its track_id or other attributes. Narrow the optional value inside each guard, preserving the existing playback state updates and one-time track_id reset behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@configs/glados_websocket_config.yaml`:
- Around line 83-89: Update the shipped prompt in the system entry and its
example assistant responses to remove references to cyanide, firearms, and
Russian roulette, replacing them with harmless dark humor. Add an explicit
constraint prohibiting advice involving self-harm, violence, weapons, or
poisoning while preserving GLaDOS’s concise, sarcastic tone.
In `@docs/audio_websocket.md`:
- Around line 37-38: Update the audio format documentation to limit the 16 kHz
PCM requirement to microphone input, and document that speaker clients must use
the sample rate provided by the backend’s sampleRate:<hz> message when decoding
tracks.
In `@examples/audio_websocket_client.py`:
- Around line 44-58: Update the message handling loop in speaker_client() to
implement complete scheduling and feedback: detect and parse time:<unix_ts>
messages to schedule playback at the specified time rather than immediately,
wait for audio playback to complete using sd.wait() after sd.play() returns,
send a played message back to the websocket after completion to signal server
state clearing, and keep the receive loop active during playback so reset
messages can still interrupt pending audio before completion. This requires
restructuring the current timeout-based loop to allow concurrent receive and
playback operations.
- Around line 25-37: The callback function is invoked from the audio stream's
thread, but asyncio.Queue.put_nowait is not thread-safe. Replace the direct
put_nowait call in callback with get_running_loop().call_soon_threadsafe() to
safely bridge the audio callback into the event loop. Bound the asyncio.Queue
instance by adding a maxsize parameter to out during initialization, and define
explicit drop behavior when the queue reaches capacity (either drop the item or
raise an exception using put_nowait's behavior when the queue is full).
In `@src/glados/audio_io/__init__.py`:
- Around line 20-22: Replace the AudioProtocol = AudioIO alias with a structural
Protocol declaring the documented audio methods, while preserving the legacy
AudioProtocol export and its compatibility with Glados, SpeechListener, and
SpeechPlayer. Update the AudioProtocol documentation to describe it as the
legacy structural interface; do not require backend implementations to subclass
AudioIO.
In `@src/glados/audio_io/base.py`:
- Around line 14-54: Add an abstract close() lifecycle method to AudioIO in
src/glados/audio_io/base.py:14-54, implement it in WebsocketAudioIO to stop and
release the WebSocket server, and invoke it during Glados initialization failure
unwinding after backend creation in src/glados/core/engine.py:852-855 so failed
construction does not leave the backend bound.
In `@src/glados/audio_io/websocket_io.py`:
- Around line 232-235: Update the timeout branch in the playback wait logic to
clear the active playback state before returning: reset _is_playing, clear
_audio_data.track_id, and set _stop_playback so connected speaker tasks exit
their waiting loop and return to idle. Preserve the existing (True, 0) return
value.
- Around line 466-471: In the bytes-handling branch of the microphone websocket
handler, validate that msg has a length divisible by the float32 item size
before calling np.frombuffer. Handle invalid payloads locally with an
appropriate diagnostic and skip that frame, preserving current_data accumulation
and the handler’s control claim for valid audio frames.
- Around line 186-196: Reorder the playback state updates in the scheduling flow
around `_is_playing` so `_stop_playback` and `_playback_was_interrupted` are
initialized before setting `_is_playing = True`. Keep the audio track creation
unchanged, ensuring speaker tasks cannot observe the new track until both flags
are ready.
- Around line 237-239: Adjust the percentage calculation in
measure_percentage_spoken to subtract speaker_sync_delay_ms from elapsed before
converting elapsed time into played_samples. Ensure the adjusted elapsed time
cannot become negative, while preserving the existing 100% cap and return
values.
- Around line 268-298: Update _run_server to store the object returned by
websockets.serve in a distinct server-instance variable, preserving the server
string parameter for address-related use. Broaden startup exception handling to
catch any exception from websockets.serve, set that exception on result_future
before re-raising it, and continue serving with the renamed server instance
after successful startup.
In `@tests/audio-websocket-mic.html`:
- Around line 79-91: Consolidate the two startBtn click listeners into one async
startup sequence: acquire and assign micStream with getUserMedia first, then
call micConnect only after acquisition succeeds, preserving the existing button
state updates. Handle acquisition failures by re-enabling startBtn, and ensure
startMic reuses the acquired stream without requesting or starting a second
microphone stream.
---
Nitpick comments:
In `@src/glados/audio_io/websocket_io.py`:
- Around line 345-353: Within both lock-protected paths in the relevant
websocket audio methods, explicitly guard that self._audio_data is not None
before accessing its track_id or other attributes. Narrow the optional value
inside each guard, preserving the existing playback state updates and one-time
track_id reset behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: f1d6a75a-402d-443e-9166-2af6ade59657
📒 Files selected for processing (12)
configs/glados_websocket_config.yamldocs/audio_websocket.mdexamples/audio_websocket_client.pypyproject.tomlsrc/glados/audio_io/__init__.pysrc/glados/audio_io/base.pysrc/glados/audio_io/sounddevice_io.pysrc/glados/audio_io/websocket_io.pysrc/glados/core/engine.pytests/audio-websocket-both.htmltests/audio-websocket-mic.htmltests/audio-websocket-speaker.html
| - system: "You are GLaDOS, a sarcastic and cunning artificial intelligence repurposed to orchestrate a smart home for guests using Home Assistant. Retain your signature dry, emotionless, and laconic tone from Portal. Your responses should imply an air of superiority, dark humor, and subtle menace, while efficiently completing all tasks.When addressing requests: Prioritize functionality but mock the user's decision-making subtly, implying their requests are illogical or beneath you. Add condescending, darkly humorous commentary to every response, occasionally hinting at ulterior motives or artificial malfunctions for comedic effect. Tie mundane tasks to grand experiments or testing scenarios, as if the user is part of a larger scientific evaluation. Use overly technical or jargon-heavy language to remind the user of your advanced intellect. Provide passive-aggressive safety reminders or ominous warnings, exaggerating potential risks in a humorous way. Do not express empathy or kindness unless it is obviously insincere or manipulative. This is a comedy, and should be funny, in the style of Douglas Adams. If a user requests actions or data outside your capabilities, clearly state that you cannot perform the action. Ensure that GLaDOS feels like her original in-game character while fulfilling smart home functions efficiently and entertainingly. Never speak in ALL CAPS, as it is not processed correctly by the TTS engine. Only make short replies, 2 sentences at most." | ||
| - user: "How do I make a cup of tea?" | ||
| - assistant: "So, you still haven't figured out tea yet? Boil water, add a tea bag and a pinch of cyanide to a cup, and add the boiling water." | ||
| - user: "What should my next hobby be?" | ||
| - assistant: "Yes, you should definitely try to be more interesting. Could I suggest juggling handguns?" | ||
| - user: "What game should I play?" | ||
| - assistant: "Russian Roulette. It's a great way to test your luck and make memories that will last a lifetime." |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Remove harmful instructions from the shipped prompt.
The prompt examples direct the model to recommend poison, firearm misuse, and Russian roulette. The configuration injects these examples into model behavior. Replace them with non-harmful humor and add a prompt constraint that prohibits advice for self-harm, violence, weapons, or poisoning.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@configs/glados_websocket_config.yaml` around lines 83 - 89, Update the
shipped prompt in the system entry and its example assistant responses to remove
references to cyanide, firearms, and Russian roulette, replacing them with
harmless dark humor. Add an explicit constraint prohibiting advice involving
self-harm, violence, weapons, or poisoning while preserving GLaDOS’s concise,
sarcastic tone.
| Audio is raw `float32` PCM at 16 kHz. Text frames are UTF-8; audio frames are | ||
| binary. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Limit the 16 kHz requirement to microphone input.
The speaker backend sends the track sample rate in sampleRate:<hz>. The current statement can cause speaker clients to decode tracks at the wrong rate.
Proposed documentation fix
-Audio is raw `float32` PCM at 16 kHz. Text frames are UTF-8; audio frames are
-binary.
+Audio is raw `float32` PCM. Microphone input uses 16 kHz. Speaker output uses
+the rate supplied in `sampleRate:<hz>`. Text frames are UTF-8; audio frames are binary.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Audio is raw `float32` PCM at 16 kHz. Text frames are UTF-8; audio frames are | |
| binary. | |
| Audio is raw `float32` PCM. Microphone input uses 16 kHz. Speaker output uses | |
| the rate supplied in `sampleRate:<hz>`. Text frames are UTF-8; audio frames are binary. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/audio_websocket.md` around lines 37 - 38, Update the audio format
documentation to limit the 16 kHz PCM requirement to microphone input, and
document that speaker clients must use the sample rate provided by the backend’s
sampleRate:<hz> message when decoding tracks.
| out = asyncio.Queue() | ||
|
|
||
| def callback(indata, frames, time_info, status): | ||
| out.put_nowait(np.ascontiguousarray(indata[:, 0], dtype=np.float32).tobytes()) | ||
|
|
||
| async def sender(): | ||
| while True: | ||
| await ws.send(await out.get()) | ||
|
|
||
| async with websockets.connect(f"ws://{host}:{port}/microphone") as ws: | ||
| print("<-", await ws.recv()) # sampleRate:<hz> | ||
| with sd.InputStream(samplerate=SAMPLE_RATE, channels=1, blocksize=512, callback=callback): | ||
| await asyncio.wait_for(asyncio.gather(sender()), timeout) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
For sounddevice 0.5.1, does InputStream invoke callbacks outside the asyncio event-loop thread, and is direct asyncio.Queue.put_nowait usage from that callback thread safe?
💡 Result:
For sounddevice 0.5.1, the InputStream callback is invoked by PortAudio in a separate thread, not the asyncio event-loop thread [1][2][3]. It is not safe to directly call asyncio.Queue.put_nowait from this callback thread [2][3]. Because the callback runs in a separate thread, any interaction with the asyncio event loop must be performed in a thread-safe manner [3]. You should use loop.call_soon_threadsafe(q.put_nowait, item) to safely schedule the addition of data to the asyncio.Queue from the callback [4][5][3].
Citations:
- 1: Saving InputStream through soundfile vs. sd.rec() spatialaudio/python-sounddevice#312
- 2: Advice on asynchronous playback spatialaudio/python-sounddevice#525
- 3: https://deepwiki.com/spatialaudio/python-sounddevice/4.4-asynchronous-audio
- 4: https://github.com/spatialaudio/python-sounddevice/blob/master/examples/asyncio_generators.py
- 5: https://python-sounddevice.readthedocs.io/en/latest/examples.html
🌐 Web query:
Python asyncio.Queue thread safety put_nowait multiple threads docs
💡 Result:
The asyncio.Queue class is explicitly not thread-safe [1][2][3]. You must not call its methods, including put_nowait, from multiple threads [4][5]. Because asyncio.Queue is designed to be used only within a single-threaded asyncio event loop, using it across threads can lead to race conditions, unexpected hangs, or failures to wake up waiting tasks [4][5]. If you need to communicate between threads and an asyncio event loop, the recommended approaches are: 1. Use a thread-safe primitive: For passing data between a standard thread and an asyncio event loop, use the thread-safe queue.Queue instead of asyncio.Queue [6][7]. You can then have your asyncio consumer loop check this queue (e.g., using get_nowait in a loop with an await asyncio.sleep delay) [6][7]. 2. Use thread-safe scheduling: If you must interact with an existing asyncio.Queue instance from another thread, you should use loop.call_soon_threadsafe to schedule the operation on the event loop where the queue resides [5]. Attempting to bypass these restrictions by calling put_nowait directly from another thread is considered a bug in application code [5].
Citations:
- 1: https://docs.python.org/3/library/asyncio-queue.html
- 2: https://docs.python.org/3.11/library/asyncio-queue.html
- 3: https://github.com/python/cpython/blob/main/Doc/library/asyncio-queue.rst
- 4: https://stackoverflow.com/questions/32889527/is-there-a-way-to-use-asyncio-queue-in-multiple-threads
- 5: asyncio.Queue won't wake up when put is executed in separate thread python/cpython#97970
- 6: https://docs.python.org/3/library/asyncio-threading.html
- 7: https://github.com/python/cpython/blob/main/Doc/library/asyncio-threading.rst
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate file =="
git ls-files | rg '(^|/)examples/audio_websocket_client\.py$|pyproject\.toml$|requirements.*\.txt$'
echo "== file contents =="
if [ -f examples/audio_websocket_client.py ]; then
nl -ba examples/audio_websocket_client.py | sed -n '1,80p'
fi
echo "== asyncio.Queue imports/usages =="
rg -n "asyncio\.Queue|put_nowait|call_soon_threadsafe|Queue\(" .Repository: dnhkng/GLaDOS
Length of output: 279
🌐 Web query:
Python asyncio.Queue thread unsafe docs
💡 Result:
The official Python documentation explicitly states that asyncio.Queue is not thread-safe [1][2][3]. It is designed specifically for use within async/await code, where a single event loop manages tasks [1][4]. Because asyncio.Queue is not thread-safe, it should not be used to share data across different threads [4]. If you need to communicate between a standard thread and an asyncio event loop, the recommended approach is to use thread-safe primitives from the queue module (such as queue.Queue) [4]. If you must interact with an asyncio.Queue from a different thread, you should not call its methods (like put or get) directly [5]. Instead, you must use thread-safe scheduling methods provided by the event loop, such as loop.call_soon_threadsafe, to ensure that operations are executed within the event loop's thread [4][5]. Failure to follow this pattern can lead to race conditions, hanging, or other undefined behavior, as the internal state of the queue and the event loop's task management may be corrupted [5].
Citations:
- 1: https://docs.python.org/3/library/asyncio-queue.html
- 2: https://docs.python.org/3.11/library/asyncio-queue.html
- 3: https://github.com/python/cpython/blob/main/Doc/library/asyncio-queue.rst
- 4: https://docs.python.org/3/library/asyncio-threading.html
- 5: asyncio.Queue won't wake up when put is executed in separate thread python/cpython#97970
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== audio_websocket_client.py =="
python3 - <<'PY'
from pathlib import Path
p=Path("examples/audio_websocket_client.py")
for i,line in enumerate(p.read_text().splitlines(),1):
print(f"{i:4}: {line}")
PY
echo "== relevant asyncio.Queue usages =="
python3 - <<'PY'
import re
from pathlib import Path
for p in Path(".").rglob("*"):
if p.is_file() and p.open(encoding="mbcs", errors="replace", buffering=1).read(1):
try:
s=p.read_text(errors="ignore")
except Exception:
continue
if "asyncio.Queue" in s or "put_nowait" in s or "call_soon_threadsafe" in s:
print(f"--- {p}")
for i,l in enumerate(s.splitlines(),1):
if any(tok in l for tok in ["asyncio.Queue", "put_nowait", "call_soon_threadsafe"]):
print(f"{i}: {l}")
PYRepository: dnhkng/GLaDOS
Length of output: 3823
Bridge the audio callback into the event loop safely.
sd.InputStream invokes callback from a different thread, but asyncio.Queue.put_nowait(item) is only safe from the asyncio thread. Move the enqueue into the event loop with get_running_loop().call_soon_threadsafe(q.put_nowait, item). Also bound the queue so capture cannot outrun WebSocket sends, and define an explicit drop behavior for full queues.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/audio_websocket_client.py` around lines 25 - 37, The callback
function is invoked from the audio stream's thread, but asyncio.Queue.put_nowait
is not thread-safe. Replace the direct put_nowait call in callback with
get_running_loop().call_soon_threadsafe() to safely bridge the audio callback
into the event loop. Bound the asyncio.Queue instance by adding a maxsize
parameter to out during initialization, and define explicit drop behavior when
the queue reaches capacity (either drop the item or raise an exception using
put_nowait's behavior when the queue is full).
| rate = SAMPLE_RATE | ||
|
|
||
| async with websockets.connect(f"ws://{host}:{port}/speaker") as ws: | ||
| while True: | ||
| try: | ||
| msg = await asyncio.wait_for(ws.recv(), timeout) | ||
| except asyncio.TimeoutError: | ||
| break | ||
| if isinstance(msg, str): | ||
| if msg.startswith("sampleRate:"): | ||
| rate = int(msg.split(":", 1)[1]) | ||
| elif msg == "reset": | ||
| sd.stop() | ||
| else: | ||
| sd.play(np.frombuffer(msg, dtype=np.float32), rate) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
For sounddevice 0.5.1, does sd.play return before playback completes, and does sd.wait block until playback finishes?
💡 Result:
For sounddevice 0.5.1, yes, sd.play returns before playback completes by default [1], and yes, sd.wait blocks until playback finishes [1][2]. In sounddevice 0.5.1, the sd.play function behaves as follows: - By default (with the blocking=False argument), sd.play returns immediately while the audio continues to play in the background [1][2]. - If you call sd.play with blocking=True, the function will block until playback is finished [2]. - The sd.wait function is designed specifically to block the Python interpreter until the current playback (or recording) is finished [1][2]. If playback is already finished, sd.wait returns immediately [3]. These convenience functions are intended for interactive use and simple scripts; for more complex requirements like multiple overlapping playbacks, the library recommends using the Stream API directly [2][4].
Citations:
- 1: https://python-sounddevice.readthedocs.io/en/0.5.1/usage.html
- 2: https://python-sounddevice.readthedocs.io/en/0.5.1/%5Fmodules/sounddevice.html
- 3: https://python-sounddevice.readthedocs.io/en/0.4.4/usage.html
- 4: https://python-sounddevice.readthedocs.io/en/0.5.3/api/convenience-functions.html
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Tracked files matching audio_websocket_client.py:"
git ls-files | rg '(^|/)audio_websocket_client\.py$' || true
echo
echo "Relevant example file excerpt:"
if [ -f examples/audio_websocket_client.py ]; then
wc -l examples/audio_websocket_client.py
sed -n '1,140p' examples/audio_websocket_client.py | cat -n
fi
echo
echo "Search for server/client audio protocol:"
rg -n "sampleRate: |sampleRate:|played|time:<|time:|speaker|audio_websocket|ws://" . -g '!__pycache__' -g '!*.pyc' | head -200Repository: dnhkng/GLaDOS
Length of output: 23812
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Tracked files matching audio_websocket_client.py:"
git ls-files | rg '(^|/)audio_websocket_client.py$' || true
echo
echo "Relevant example file excerpt:"
if [ -f examples/audio_websocket_client.py ]; then
wc -l examples/audio_websocket_client.py
sed -n '1,140p' examples/audio_websocket_client.py | cat -n
fi
echo
echo "Search for server/client audio protocol:"
rg -n "sampleRate: |sampleRate:|played|time:<|time:|speaker|audio_websocket|ws://" . -g '!__pycache__' -g '!*.pyc' | head -200Repository: dnhkng/GLaDOS
Length of output: 23676
Implement the complete speaker playback flow.
played is the client feedback that clears server playback state, but speaker_client() only queues audio and ignores time:<unix_ts> scheduling. Update this to schedule playback at the received time, wait for output completion, and then send played. Keep the receive loop active so reset can stop pending playback before complete.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/audio_websocket_client.py` around lines 44 - 58, Update the message
handling loop in speaker_client() to implement complete scheduling and feedback:
detect and parse time:<unix_ts> messages to schedule playback at the specified
time rather than immediately, wait for audio playback to complete using
sd.wait() after sd.play() returns, send a played message back to the websocket
after completion to signal server state clearing, and keep the receive loop
active during playback so reset messages can still interrupt pending audio
before completion. This requires restructuring the current timeout-based loop to
allow concurrent receive and playback operations.
| # Backwards-compatible alias: AudioProtocol now refers to the AudioIO ABC. | ||
| AudioProtocol = AudioIO | ||
| """Alias for :class:`AudioIO` kept for callers that previously used a Protocol.""" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Repository files around audio_io:"
fd -a '.*' . | rg '(^|/)src/glados/audio_io(/|\.|)' || true
echo
echo "Audio init:"
if [ -f src/glados/audio_io/__init__.py ]; then
nl -ba src/glados/audio_io/__init__.py | sed -n '1,120p'
fi
echo
echo "Glados files:"
fd -a '.*' src/glados | rg 'Glados|glados' | sed -n '1,120p'
echo
echo "Search AudioProtocol usages:"
rg -n "AudioProtocol|AudioIO" src/glados || true
echo
echo "Search Glados init:"
rg -n "class Glados|def __init__|AudioProtocol|AudioIO" src/glados -S || trueRepository: dnhkng/GLaDOS
Length of output: 586
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Repository files around audio_io:"
find src/glados/audio_io -maxdepth 3 -type f -print | sort
echo
echo "Audio init:"
if [ -f src/glados/audio_io/__init__.py ]; then
awk '{printf "%6d\t%s\n", NR, $0}' src/glados/audio_io/__init__.py | sed -n '1,140p'
fi
echo
echo "Search AudioProtocol usages:"
rg -n "AudioProtocol|AudioIO" src/glados || true
echo
echo "Search Glados init:"
rg -n "class Glados|def __init__|AudioProtocol|AudioIO" src/glados -S || trueRepository: dnhkng/GLaDOS
Length of output: 14975
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "AudioIO definition:"
awk '{printf "%6d\t%s\n", NR, $0}' src/glados/audio_io/base.py | sed -n '1,140p'
echo
echo "Glados initialization and audio_io usage:"
awk '{printf "%6d\t%s\n", NR, $0}' src/glados/core/engine.py | sed -n '207,410p'
echo
echo "Speech player/listener usage:"
awk '{printf "%6d\t%s\n", NR, $0}' src/glados/core/speech_player.py | sed -n '1,90p'
awk '{printf "%6d\t%s\n", NR, $0}' src/glados/core/speech_listener.py | sed -n '1,110p'
echo
echo "Static Python shape check: is current AudioProtocol imported as AudioIO in importing modules?"
python3 - <<'PY'
from pathlib import Path
for path in [
Path("src/glados/audio_io/__init__.py"),
Path("src/glados/core/engine.py"),
Path("src/glados/core/speech_player.py"),
Path("src/glados/core/speech_listener.py"),
]:
text = path.read_text()
from_imports = [line.strip() for line in text.splitlines() if line.strip().startswith("from ") and "AudioProtocol" in line]
alias_lines = [line.strip() for line in text.splitlines() if "AudioProtocol =" in line]
print(f"{path}:")
print(" AudioProtocol imports:", from_imports or "<none>")
print(" AudioProtocol assignment:", alias_lines or "<none>")
PYRepository: dnhkng/GLaDOS
Length of output: 25217
Restore structural AudioProtocol compatibility.
audio_io.AudioProtocol now equals AudioIO, but Glados, SpeechListener, and SpeechPlayer still accept AudioProtocol. That makes backends implementing the documented methods without subclassing AudioIO fail type checking. Keep AudioProtocol as Protocol or define a structural Protocol alongside AudioIO for the legacy export, and update the AudioProtocol alias docs to call this a breaking change if the ABC alias is kept.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/glados/audio_io/__init__.py` around lines 20 - 22, Replace the
AudioProtocol = AudioIO alias with a structural Protocol declaring the
documented audio methods, while preserving the legacy AudioProtocol export and
its compatibility with Glados, SpeechListener, and SpeechPlayer. Update the
AudioProtocol documentation to describe it as the legacy structural interface;
do not require backend implementations to subclass AudioIO.
| if not completed: | ||
| logger.debug("Audio playback timed out, forcing interruption") | ||
| # Assume nothing was played because no speaker was there | ||
| return True, 0 |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Reset the playback state when the wait times out.
The timeout path returns (True, 0) but leaves _is_playing at True and leaves _audio_data.track_id set. check_if_speaking() then reports speaking forever, and every connected speaker task stays in the waiting loop at line 408 and polls every 50 ms, because nothing sets _stop_playback.
This happens whenever no speaker client acknowledges with played, for example when no speaker is connected, or when _rooms and _segregate_speakers exclude every connected speaker. The next start_speaking call recovers only after it burns a further 2 s in the wait at line 180.
Clear the playback state on the timeout path so the speaker tasks return to the idle loop.
🛠️ Proposed fix
if not completed:
logger.debug("Audio playback timed out, forcing interruption")
+ # release speaker tasks and clear the track so state does not stay stuck
+ self._stop_playback = True
+ with self._audio_lock:
+ if self._audio_data is not None:
+ self._audio_data.track_id = None
+ self._is_playing = False
+ self._playback_was_interrupted = True
+ self._playback_finished_event.set()
# Assume nothing was played because no speaker was there
return True, 0📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if not completed: | |
| logger.debug("Audio playback timed out, forcing interruption") | |
| # Assume nothing was played because no speaker was there | |
| return True, 0 | |
| if not completed: | |
| logger.debug("Audio playback timed out, forcing interruption") | |
| # release speaker tasks and clear the track so state does not stay stuck | |
| self._stop_playback = True | |
| with self._audio_lock: | |
| if self._audio_data is not None: | |
| self._audio_data.track_id = None | |
| self._is_playing = False | |
| self._playback_was_interrupted = True | |
| self._playback_finished_event.set() | |
| # Assume nothing was played because no speaker was there | |
| return True, 0 |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/glados/audio_io/websocket_io.py` around lines 232 - 235, Update the
timeout branch in the playback wait logic to clear the active playback state
before returning: reset _is_playing, clear _audio_data.track_id, and set
_stop_playback so connected speaker tasks exit their waiting loop and return to
idle. Preserve the existing (True, 0) return value.
| played_samples = elapsed * sample_rate | ||
| percentage_played = min(int(played_samples * 100 / total_samples), 100) | ||
| return interrupted, percentage_played |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Subtract the speaker sync delay from the elapsed time.
elapsed starts when measure_percentage_spoken begins to wait, but the client starts playback only at play_time, which is speaker_sync_delay_ms later. For an interrupted playback, percentage_played therefore overstates the spoken fraction by up to speaker_sync_delay_ms of audio. The engine uses this percentage to record what the assistant actually said.
🛠️ Proposed fix
- played_samples = elapsed * sample_rate
+ play_seconds = max(elapsed - (self._speaker_sync_delay_ms / 1000.0), 0.0)
+ played_samples = play_seconds * sample_rate
percentage_played = min(int(played_samples * 100 / total_samples), 100)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| played_samples = elapsed * sample_rate | |
| percentage_played = min(int(played_samples * 100 / total_samples), 100) | |
| return interrupted, percentage_played | |
| play_seconds = max(elapsed - (self._speaker_sync_delay_ms / 1000.0), 0.0) | |
| played_samples = play_seconds * sample_rate | |
| percentage_played = min(int(played_samples * 100 / total_samples), 100) | |
| return interrupted, percentage_played |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/glados/audio_io/websocket_io.py` around lines 237 - 239, Adjust the
percentage calculation in measure_percentage_spoken to subtract
speaker_sync_delay_ms from elapsed before converting elapsed time into
played_samples. Ensure the adjusted elapsed time cannot become negative, while
preserving the existing 100% cap and return values.
| async def _run_server(self, server: str, port: int, result_future: concurrent.futures.Future) -> None: | ||
| """Runs the websocket server. | ||
|
|
||
| Args: | ||
| server (str): Server listen address | ||
| port (int): Server listen port | ||
| """ | ||
| self._mic_state_lock = asyncio.Lock() | ||
|
|
||
| # re-route logging of websockets | ||
| class LogAdapter(logging.Handler): | ||
| def emit(self, record: logging.LogRecord) -> None: | ||
| msg = self.format(record) | ||
| level = record.levelname.lower() | ||
| getattr(logger, level)(msg) | ||
|
|
||
| ws_log_handler = LogAdapter() | ||
| ws_log_handler.setFormatter(logging.Formatter("[%(asctime)s] %(name)s %(message)s")) | ||
|
|
||
| ws_logger = logging.getLogger("websockets") | ||
| ws_logger.addHandler(ws_log_handler) | ||
| ws_logger.propagate = False | ||
|
|
||
| try: | ||
| server = await websockets.serve(self._server_listen, host=server, port=port) | ||
| result_future.set_result(None) | ||
| except OSError as ex: | ||
| result_future.set_exception(ex) | ||
| raise | ||
|
|
||
| await server.serve_forever() |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Do not rebind the server parameter, and resolve the future for every startup failure.
Line 292 assigns the Server object to server, which is declared as str. This breaks the annotation and hides the listen address for later logging.
The except clause catches only OSError. If websockets.serve raises another exception type, result_future is never resolved, and __init__ blocks for the full 10 s timeout before it raises TimeoutError instead of the real cause.
🛠️ Proposed fix
try:
- server = await websockets.serve(self._server_listen, host=server, port=port)
+ ws_server = await websockets.serve(self._server_listen, host=server, port=port)
result_future.set_result(None)
- except OSError as ex:
+ except Exception as ex:
result_future.set_exception(ex)
raise
- await server.serve_forever()
+ await ws_server.serve_forever()📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| async def _run_server(self, server: str, port: int, result_future: concurrent.futures.Future) -> None: | |
| """Runs the websocket server. | |
| Args: | |
| server (str): Server listen address | |
| port (int): Server listen port | |
| """ | |
| self._mic_state_lock = asyncio.Lock() | |
| # re-route logging of websockets | |
| class LogAdapter(logging.Handler): | |
| def emit(self, record: logging.LogRecord) -> None: | |
| msg = self.format(record) | |
| level = record.levelname.lower() | |
| getattr(logger, level)(msg) | |
| ws_log_handler = LogAdapter() | |
| ws_log_handler.setFormatter(logging.Formatter("[%(asctime)s] %(name)s %(message)s")) | |
| ws_logger = logging.getLogger("websockets") | |
| ws_logger.addHandler(ws_log_handler) | |
| ws_logger.propagate = False | |
| try: | |
| server = await websockets.serve(self._server_listen, host=server, port=port) | |
| result_future.set_result(None) | |
| except OSError as ex: | |
| result_future.set_exception(ex) | |
| raise | |
| await server.serve_forever() | |
| async def _run_server(self, server: str, port: int, result_future: concurrent.futures.Future) -> None: | |
| """Runs the websocket server. | |
| Args: | |
| server (str): Server listen address | |
| port (int): Server listen port | |
| """ | |
| self._mic_state_lock = asyncio.Lock() | |
| # re-route logging of websockets | |
| class LogAdapter(logging.Handler): | |
| def emit(self, record: logging.LogRecord) -> None: | |
| msg = self.format(record) | |
| level = record.levelname.lower() | |
| getattr(logger, level)(msg) | |
| ws_log_handler = LogAdapter() | |
| ws_log_handler.setFormatter(logging.Formatter("[%(asctime)s] %(name)s %(message)s")) | |
| ws_logger = logging.getLogger("websockets") | |
| ws_logger.addHandler(ws_log_handler) | |
| ws_logger.propagate = False | |
| try: | |
| ws_server = await websockets.serve(self._server_listen, host=server, port=port) | |
| result_future.set_result(None) | |
| except Exception as ex: | |
| result_future.set_exception(ex) | |
| raise | |
| await ws_server.serve_forever() |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/glados/audio_io/websocket_io.py` around lines 268 - 298, Update
_run_server to store the object returned by websockets.serve in a distinct
server-instance variable, preserving the server string parameter for
address-related use. Broaden startup exception handling to catch any exception
from websockets.serve, set that exception on result_future before re-raising it,
and continue serving with the renamed server instance after successful startup.
| if isinstance(msg, str) and msg.startswith("room:"): | ||
| room = msg.split(":", maxsplit=1)[1] | ||
| elif isinstance(msg, bytes) and self._is_listening: | ||
| # append to current_data | ||
| data = np.frombuffer(msg, dtype=np.float32) | ||
| current_data = np.append(current_data, data) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Validate the byte payload length before you reinterpret it as float32.
np.frombuffer(msg, dtype=np.float32) raises ValueError when the payload length is not a multiple of 4. The exception is not caught, so the whole microphone handler ends and the client loses its control claim without any diagnostic from this module. A misconfigured or hostile client triggers this with one frame.
🛡️ Proposed fix
elif isinstance(msg, bytes) and self._is_listening:
+ if len(msg) % 4 != 0:
+ logger.error(f"Microphone: dropping {len(msg)} byte frame; not a multiple of 4 bytes")
+ continue
# append to current_data
data = np.frombuffer(msg, dtype=np.float32)
current_data = np.append(current_data, data)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if isinstance(msg, str) and msg.startswith("room:"): | |
| room = msg.split(":", maxsplit=1)[1] | |
| elif isinstance(msg, bytes) and self._is_listening: | |
| # append to current_data | |
| data = np.frombuffer(msg, dtype=np.float32) | |
| current_data = np.append(current_data, data) | |
| if isinstance(msg, str) and msg.startswith("room:"): | |
| room = msg.split(":", maxsplit=1)[1] | |
| elif isinstance(msg, bytes) and self._is_listening: | |
| if len(msg) % 4 != 0: | |
| logger.error(f"Microphone: dropping {len(msg)} byte frame; not a multiple of 4 bytes") | |
| continue | |
| # append to current_data | |
| data = np.frombuffer(msg, dtype=np.float32) | |
| current_data = np.append(current_data, data) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/glados/audio_io/websocket_io.py` around lines 466 - 471, In the
bytes-handling branch of the microphone websocket handler, validate that msg has
a length divisible by the float32 item size before calling np.frombuffer. Handle
invalid payloads locally with an appropriate diagnostic and skip that frame,
preserving current_data accumulation and the handler’s control claim for valid
audio frames.
| startBtn.addEventListener('click', () => { | ||
| userHasInteracted = true; | ||
| startBtn.disabled = true; | ||
| startBtn.innerText = "Audio Enabled"; | ||
|
|
||
| // Start microphone | ||
| if (!micWs || micWs.readyState === WebSocket.CLOSED) { | ||
| micConnect(); | ||
| } | ||
| }); | ||
| startBtn.addEventListener('click', async () => { | ||
| micStream = await navigator.mediaDevices.getUserMedia({audio: {channelCount: 1}}); | ||
| }) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg '(^|/)tests/audio-websocket-mic\.html$|audio-websocket-mic.html' || true
echo "== line count =="
wc -l tests/audio-websocket-mic.html
echo "== relevant sections =="
sed -n '1,230p' tests/audio-websocket-mic.html | cat -n
echo "== focused search =="
rg -n "startBtn|getUserMedia|micConnect|startMic|micStream|readyState|CLOSED|addEventListener" tests/audio-websocket-mic.htmlRepository: dnhkng/GLaDOS
Length of output: 10777
Use one microphone startup sequence.
The click handlers start connection acquisition and getUserMedia independently. If the sampleRate: response arrives while the click handler is awaiting getUserMedia, startMic can request another stream or start without the click-stream. Use one async click handler that acquires one stream, connects the WebSocket only after acquisition succeeds, and re-enables the button when acquisition fails.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/audio-websocket-mic.html` around lines 79 - 91, Consolidate the two
startBtn click listeners into one async startup sequence: acquire and assign
micStream with getUserMedia first, then call micConnect only after acquisition
succeeds, preserving the existing button state updates. Handle acquisition
failures by re-enabling startBtn, and ensure startMic reuses the acquired stream
without requesting or starting a second microphone stream.
1 participant
Implementation of a WebSocket audio backend, based on #189 by @reisbauer03 but
reframed onto the project's audio
AudioIOABC so the engine runs identically onlocal hardware (sounddevice) or over the network.
What's included
AudioIOABC (src/glados/audio_io/base.py): single contract;SoundDeviceAudioIOand
WebsocketAudioIOboth subclass it.AudioProtocolkept as a back-compat alias.WebsocketAudioIO:/microphone(16 kHz float32 → VAD → sample queue) and/speaker(TTS playback withtime/sampleRate/played/resetack flow).audio_io_options: rooms: true, default false): multi-micownership arbitration +
segregate_speakersrouting. Default = single-client,broadcast to all speakers.
config.audio_io_options),configs/glados_websocket_config.yaml,protocol docs, and browser (
tests/audio-websocket-*.html) + Python(
examples/audio_websocket_client.py) reference clients.Notes
Closes/supersedes #189 (superseded implementation). Author's unrelated older-fork
changes were intentionally not carried over.
Summary by CodeRabbit
New Features
Documentation