🎙️ claude-voice-memo

Turn any voice recording into a clean, speaker-aware transcript + a key briefing — fully local, private, and free — using Claude Code and on-device Whisper.

English · 한국어

---

A Claude Code skill that takes a raw voice memo and produces two things in one file:

1. 📋 Key briefing — TL;DR, per-speaker messages, decisions/action items, open questions.
2. 📝 Corrected transcript — proper-noun fixes, speaker labels, hallucination cleanup.

Your audio never leaves your machine. No cloud STT, no API bills, no length limits. Optional precise speaker separation downloads a pyannote model, then runs diarization locally.

Why local?

	claude-voice-memo (local)	Cloud STT apps	Raw whisper CLI
Privacy	✅ audio never leaves device	❌ uploaded to a server	✅ local
Cost	✅ free	💸 per-minute / subscription	✅ free
Length limits	✅ none	⚠️ often capped	✅ none
Context correction	✅ glossary + memory	⚠️ generic	❌ none
Speaker labels	✅ inference or pyannote diarization	⚠️ varies	❌ none
Briefing / summary	✅ built in	⚠️ sometimes	❌ none
Hallucination control	✅ tuned options	❓ opaque	⚠️ defaults drift

How it works

  audio file              ffmpeg            Whisper (GPU)         Claude
  (.m4a/.qta/...)  ──▶  16kHz mono wav  ──▶  local transcript  ──▶  context-correct
                                             + anti-hallucination     + speaker labels
                                                                      + briefing
                                                                          │
                                                                          ▼
                                                            📋 briefing + 📝 transcript

The key idea: gather context before transcription (who is speaking, the purpose, the domain) so the glossary and speaker priors are right from the start — then clean up only the residual ambiguities afterward.

Platform support

Layer	Tested ✅	Community / adaptable 🌱
Engine	Apple Silicon → mlx-whisper (Metal GPU)	NVIDIA (Win/Linux) → faster-whisper · CPU → openai-whisper / whisper.cpp
Input	iPhone Voice Memos (iCloud path, .qta/.m4a)	Android (.m4a/.opus/.amr, moved via Drive/USB) · any audio file path (all OS)

The skill auto-detects your OS/GPU and picks the engine. The "tested" column is what the author runs daily; the rest is documented in docs/platforms.md — contributions welcome.

Modes & options

Everything beyond the basics is opt-in — sensible defaults, toggles for those who want them.

Option	Default	What it does
Speaker separation	context inference	off / inference / precise (pyannote) for 3+ speakers
Recurring-meeting profile	off	Reuse participants/glossary/series-note for regular meetings
Glossary + correction dictionary	memory only	Domain glossaries + persistent typo→fix mappings
Low-confidence flagging	off	Mark uncertain segments ⚠️ instead of silently guessing

See docs/configuration.md for details.

Requirements

• Claude Code
• ffmpeg
• An STT engine for your platform:
  • Apple Silicon: pip install mlx-whisper
  • NVIDIA: pip install faster-whisper
  • CPU fallback: pip install openai-whisper
• Optional precise speaker separation:
  • From this repo: python3 -m pip install -e ".[diarization]"
  • Accept the pyannote/speaker-diarization-community-1 model terms on Hugging Face.
  • Export HF_TOKEN at runtime, or store it in macOS Keychain as huggingface-hf-token; do not store it in config files.

Install

Drop the skill into your Claude Code skills directory:

git clone https://github.com/jayjoolee/claude-voice-memo
cp -r claude-voice-memo/skills/voice-memo ~/.claude/skills/
# optional: copy the config templates and edit them
cp claude-voice-memo/config/*.example.md ~/.claude/skills/voice-memo/

Optional local CLI for acoustic diarization merge:

cd claude-voice-memo
python3 -m pip install -e .
# add ".[diarization]" when you want pyannote support

Usage

Just talk to Claude Code:

organize my last recording

or invoke it explicitly:

/voice-memo

Claude detects your platform, finds the recording, asks a few up-front context questions (only what it can't infer), transcribes locally, and saves a briefing + transcript.

If you need voice-based speaker separation, run Whisper with JSON timestamps first, then merge pyannote's acoustic speaker ranges into the transcript:

voice-memo-diarize \
  --audio /tmp/voicememo_stt/audio.wav \
  --whisper-json /tmp/voicememo_stt/audio.json \
  --num-speakers 2 \
  --speaker-map "SPEAKER_00=Me,SPEAKER_01=Guest" \
  --diarization-out /tmp/voicememo_stt/speakers.tsv \
  --out /tmp/voicememo_stt/speaker-transcript.md

On macOS, the CLI also checks Keychain when HF_TOKEN is not set:

security add-generic-password -a "$USER" -s "huggingface-hf-token" -w "$(pbpaste)" -U

Customize

• Output paths — edit the save locations in SKILL.md (~/path/to/notes).
• Glossaries — add your domain terms in config/glossaries.example.md.
• Language — pass any Whisper-supported language; defaults are easy to change.

Multilingual

Whisper supports 90+ languages. Set the --language flag (or let it auto-detect). The context-intake and briefing work in any language Claude handles.

Roadmap

• First-class faster-whisper path verification (Windows/Linux + NVIDIA)
• Android input helper
• Optional pyannote diarization merge CLI

License

MIT © jayjoolee