Skip to content

voice.md

Latest commit

 

History

History
158 lines (116 loc) · 11.7 KB

voice.md

File metadata and controls

158 lines (116 loc) · 11.7 KB
Raw

Voice Mode

PortOS includes an optional voice assistant with support for fully local operation. Supports push-to-talk and hands-free continuous mode with barge-in (start speaking over a reply to interrupt it).

Privacy note: LLM, TTS, and pipeline orchestration always run locally. For speech-to-text, the default Web Speech API uses your browser's built-in recognition service — in Chrome/Chromium this forwards audio to a Google cloud endpoint. Switch stt.engine to whisper (whisper.cpp, local HTTP server) under Settings → Voice for fully-offline STT with no audio leaving the machine.

Stack

Stage Default engine Alternatives Local?
Speech-to-text Browser Web Speech API (default — note: Chromium browsers forward audio to a vendor cloud speech service) or whisper.cpp via whisper-server (HTTP :5562, fully local) — ✅ (whisper) / ⚠️ (web-speech)
LLM LM Studio (/v1/chat/completions) OpenAI-compatible local server ✅
Text-to-speech Kokoro-82M via kokoro-js (in-process) Piper (CLI) ✅
Voice activity AudioWorklet + RMS VAD (hands-free) or MediaRecorder (push-to-talk) — Web Speech mode bypasses server audio and posts final text via voice:text — ✅

The TTS engine is selectable in Settings → Voice → TTS engine.

FaceTime Audio control plane

FaceTime Audio controls are off by default and remain machine-local. On macOS, run npm run setup:facetime, grant the installed helper Accessibility permission in System Settings, choose the configured BlackHole devices in FaceTime, then enable FaceTime Audio in Settings → Features. Set a target name and E.164 phone number or email in Settings → Voice, save, and use Probe, Test call, or Hang up. The helper refuses ambiguous FaceTime surfaces and never uses coordinate clicks.

Why Kokoro is the default

Kokoro is a 82M-parameter frontier TTS model that runs in-process via ONNX Runtime + transformers.js — no Python, no extra binaries, cross-platform. Quality is significantly higher than Piper (more natural prosody, expressive pacing). First synthesis after server start has a 2–3 s cold start as the model loads; warm calls are 200–500 ms per sentence on CPU.

Use Piper instead if you need lower latency per call (~100 ms cold start), are on a memory-constrained machine, or want a particular pre-trained voice from rhasspy's catalogue.

First-time setup

  1. Open PortOS → Settings → Voice.
  2. Pick your TTS engine (default: Kokoro), Whisper model size, and CoreML toggle (macOS).
  3. Toggle Enable voice mode and click Save & Reconcile.

PortOS will:

  • Install whisper-cpp via Homebrew if missing (the whisper.cpp binary + its base models)
  • Download the selected Whisper ggml .bin model into ~/.portos/voice/models/
  • On macOS with CoreML enabled, download the matching <model>-encoder.mlmodelc (2–3× faster STT on Apple Silicon, requires a custom whisper.cpp build)
  • If Piper is selected, download the pre-built Piper binary + phonemize libs from rhasspy/piper and rhasspy/piper-phonemize GitHub Releases (Piper is not on Homebrew), then fetch the selected voice .onnx into ~/.portos/voice/voices/
  • Start portos-whisper under PM2

Kokoro models live under ~/.cache/huggingface/hub/ and download lazily on first synthesis.

You can also run the bootstrap script directly:

TTS_ENGINE=kokoro INSTALL_COREML=1 bash scripts/setup-voice.sh
TTS_ENGINE=piper VOICE_NAME=en_US-ryan-high bash scripts/setup-voice.sh
MODEL_NAME=ggml-small.en.bin bash scripts/setup-voice.sh

Configuration options

All options live in data/settings.json under voice (Settings UI patches this file).

Option Default Notes
enabled false Master toggle. Triggers reconcile on change.
hotkey Space Held to talk. Ignored while typing in inputs.
stt.model base.en tiny.en · base.en · small.en · medium.en · large-v3
stt.coreml false Optional on macOS. Enable to use the CoreML encoder companion (requires a custom whisper.cpp build with -DWHISPER_COREML=1).
stt.endpoint http://127.0.0.1:5562 whisper-server listen address (whisper engine only).
tts.engine kokoro kokoro or piper
tts.rate 1.0 Speech rate, 0.5–2.0
tts.kokoro.voice af_heart See Kokoro voices — A-grade are af_heart, af_bella.
tts.kokoro.dtype q8 q4 · q8 · fp16 · fp32 (size/quality trade-off)
tts.kokoro.modelId onnx-community/Kokoro-82M-v1.0-ONNX HuggingFace repo id
tts.piper.voice en_GB-jenny_dioco-medium Piper voice id (path-encoded)
tts.piper.voicePath ~/.portos/voice/voices/<voice>.onnx ONNX file location
llm.model auto auto picks first loaded LM Studio model
llm.systemPrompt (concise voice prompt) Edit to change personality
llm.fastPath.enabled false Fast-resolution cascade (see below). Off = every turn runs on the server LLM.
llm.fastPath.triggers true Tier 1: resolve navigation commands ("go to tasks") in the browser, no LLM.
llm.fastPath.browserLlm true Tier 2: answer simple/conversational turns with Chrome's on-device Gemini Nano.
llm.fastPath.browser.temperature 0.7 Nano sampling temperature (0–2).
llm.fastPath.browser.topK 3 Nano sampling top-K (1–128).

Using voice mode

Two modes, toggle with the headset icon in the voice widget:

  • Push-to-talk: click/hold the mic or hold the configured hotkey. Release to send. The hotkey ignores keypresses while an input/textarea is focused.
  • Hands-free: the mic stays live; an AudioWorklet computes an RMS envelope and auto-submits once you've been silent for vad.endOfSpeechMs (default 700 ms). Ambient noise is calibrated at session start.

Either mode: start speaking while the assistant is replying to interrupt it (barge-in). Click the square button to stop current TTS playback without sending a new turn.

With stt.engine = 'web-speech' the browser's SpeechRecognition handles STT entirely client-side; PortOS sends the final transcript as voice:text instead of shipping audio, which skips whisper.cpp and avoids a 250–800 ms server-side round-trip.

Fast-resolution cascade (lower latency)

The server LLM is the slowest part of a turn — a local model with tools attached can take several seconds. When llm.fastPath.enabled is on, the client triages each turn through faster tiers first and only falls through to the server LLM when it has to:

  1. Trigger (fastPath.triggers) — deterministic, offline. A navigation command ("go to tasks", "open the daily log") is matched against the ⌘K palette nav manifest and navigates immediately. No LLM.
  2. Browser LLM (fastPath.browserLlm) — Chrome's on-device Gemini Nano (the Prompt API: window.LanguageModel / legacy self.ai.languageModel) answers simple/conversational turns entirely in the browser (fast, private, offline). Nano is also asked to reply ESCALATE for anything that needs a real action.
  3. Server — the configured provider/model (recommend Ollama with a small model) via the existing pipeline. Handles every tool/action turn, personal-data retrieval, dictation, confirmations, and anything the fast tiers decline or can't run.

Notes:

  • The cascade only applies to client-produced transcripts — Web Speech STT or typed input. Whisper / hands-free audio turns have no client transcript to triage, so they stay fully server-driven. Set stt.engine = 'web-speech' to get the benefit.
  • Trigger/Nano replies are spoken through the server's configured TTS (POST /api/voice/public/synthesize), reusing the normal playback queue and echo-suppression, so barge-in keeps working.
  • Fast-tier turns are handled without a server round-trip, so they are not added to the server-side conversation history; a later server turn won't have that chit-chat in its context. Action turns (which always hit the server) are unaffected.
  • Nano availability is surfaced in Settings → Voice → Fast resolution. When it isn't downloaded/enabled (chrome://flags/#prompt-api-for-gemini-nano + #optimization-guide-on-device-model), tier 2 transparently falls through to the server.
  • Nothing here runs a model the user hasn't triggered — Nano only executes on a real spoken/typed turn, consistent with the no-cold-bootstrap AI policy.

Client modules: client/src/services/browserLlm.js (Nano client), client/src/services/voiceFastPath.js (the routing decision), wired into VoiceWidget.jsx.

Architecture

browser mic → MediaRecorder (PTT) OR AudioWorklet + RMS VAD (hands-free)
  → Socket.IO 'voice:turn' (audio) OR 'voice:text' (Web Speech final)
  → whisper.cpp /inference          (STT for audio path)
  → LM Studio /v1/chat (streaming)  (LLM)
  → sentence-boundary TTS dispatch  (Kokoro in-process | Piper CLI)
  → Socket.IO 'voice:tts:audio'     → Web Audio playback queue

Pipeline orchestration: server/services/voice/pipeline.js. The pipeline emits events as it runs:

  • voice:transcript — STT result
  • voice:llm:delta — each token delta from LM Studio
  • voice:llm:done — full assistant reply
  • voice:tts:audio — one WAV per sentence as soon as TTS finishes it
  • voice:idle — turn complete (or interrupted)

Barge-in works by aborting the shared AbortController tied to the current turn — the LLM stream is torn down and any queued TTS is discarded.

Endpoints

Method Path Purpose
GET /api/voice/status Health probes + active engines + binary/model presence
GET /api/voice/config Current merged voice config
PUT /api/voice/config Deep-merge patch; triggers PM2 + setup reconcile
GET /api/voice/voices Voices for the active TTS engine
POST /api/voice/test Body { text }, returns WAV bytes — verifies TTS

Socket events are documented in server/sockets/voice.js.

Troubleshooting

  • Whisper badge red — brew install whisper-cpp, then which whisper-server.
  • CoreML missing — re-run INSTALL_COREML=1 bash scripts/setup-voice.sh (or toggle voice off/on after enabling CoreML).
  • Kokoro shows lazy — model loads on first synthesis. Hit "Test voice" to warm it up.
  • Kokoro slow on first call — first call after server start downloads model (~80 MB for q8) and initializes the runtime. Subsequent calls are 200–500 ms.
  • Piper spawn fails — which piper and check voice file at ~/.portos/voice/voices/<name>.onnx.
  • LM Studio red — start LM Studio and load a chat model; the voice pipeline uses /v1/chat/completions.
  • No audio playback — browsers require a user gesture before AudioContext can play. Click the page once or press the mic button.

Performance notes

Engine Cold start Warm latency (per sentence) Quality
Kokoro q8 (CPU) 2–3 s 200–500 ms High
Kokoro fp32 (CPU) 3–5 s 400–900 ms Highest
Piper ~100 ms (CLI spawn) ~100 ms Mid
Whisper base.en (no CoreML) 0 (server resident) 400–800 ms / 2 s of audio Good
Whisper base.en + CoreML 0 150–300 ms / 2 s of audio Good
Whisper small.en + CoreML 0 300–600 ms / 2 s of audio Better