ClipBot

A chatbot that replies entirely in movie/TV dialogue clips from your own collection. Built per the project spec, in the order it prescribes: Phase 0 (index) → Phase 1 (first working prototype) are done and tested below. Phases 2–5 are scaffolded with clear next steps.

What's here

clipbot/
├── indexer.py      # Phase 0: clips/ folder -> word/phrase -> clip vocabulary
├── generator.py     # Phase 1a: constrained reply generation (Ollama, w/ offline fallback)
├── assembler.py     # Phase 1b: reply text -> stitched video w/ glitch transitions + fallback
├── main.py           # CLI entrypoint tying it all together
├── clips/            # put your real .mp4s here (synthetic demo clips included for now)
├── cache/             # vocabulary.json + intermediate segments
└── output/            # assembled reply videos land here

Quickstart (works right now, zero real footage needed)

cd clipbot
python3 indexer.py --clips-dir clips --build-demo-library   # fabricates ~27 tiny test clips
python3 main.py "hello robot, tell me about the future"
# -> output/reply.mp4

This proves the whole Phase 1 loop end to end: text in, a vocabulary-constrained reply generated, matching clips looked up, glitch transitions applied between them, missing words replaced with glitch-text + robotic voice (espeak-ng), all concatenated into one video.

Pointing it at your real archive

1. Copy your real .mp4s into clips/ (delete the synthetic demo ones first, or use a different --clips-dir).

2. The naming-convention parser is a guess — I don't have your HowINamedMyClips.md, so indexer.py's FILENAME_RE implements a reasonable reading of what the spec described:

   • Amazing.mp4 → dialogue word "amazing"
   • Amazing (1).mp4 → same word, duplicate/alt version 1
   • Great scott.mp4 → dialogue phrase "great scott"
   • No x3.mp4 → word "no", repetition marker 3
   • [wink].mp4 → action tag (excluded from sentence vocabulary)
   • [42].mp4 / [$20].mp4 → number/money tags

   Send me the actual naming doc and I'll adjust parse_filename() to match exactly — everything downstream (generator, assembler) only depends on the ClipEntry.kind/key fields it produces, not on the regex itself, so this is a small, isolated change.

3. Rebuild the index: python3 indexer.py --clips-dir clips --out cache/vocabulary.json

4. Run python3 main.py "your message".

Connecting real constrained generation (Ollama)

Right now, since this sandbox has no network access to install Ollama, generator.py auto-detects that no local server is reachable and falls back to a small dependency-free constrained picker (OfflineFallbackGenerator) — good enough to prove the pipeline, but not smart or funny.

On your own machine:

# install Ollama: https://ollama.com
ollama pull llama3.2
ollama serve   # usually runs automatically
python3 main.py "your message" --ollama-model llama3.2

generator.py will detect the running server automatically and use it — no code changes needed. The system prompt in SYSTEM_PROMPT_TEMPLATE is what enforces "only use these exact words" (the vocabulary-constrained-generation idea from the spec) — tune it there if replies feel too repetitive or off-tone.

Design notes / where spec decisions were made concretely

• Tokenization (assembler.tokenize_against_vocabulary) does greedy longest-phrase matching against your vocabulary, so multi-word entries like "great scott" are preferred over stitching "great" + "scott" separately, per §4 of the spec.
• Duplicate versions (Amazing (0).mp4 / (1).mp4) are picked randomly per-use (assembler.pick_clip) for variety, per §4.
• Glitch transitions are three ffmpeg filter styles (stutter, flicker, rgb_split) chosen randomly between segments — swap/extend the filters dict in assembler.add_glitch_transition for more variety in Phase 5.
• Glitch-text fallback renders large on-screen text with a noise overlay and reads it with espeak-ng (installed via apt in this environment) — matches §3's "system voice cut in for one word" framing rather than looking like an error state.
• Action tags ([wink], [chuckles]) are indexed but never selected by the current sentence assembler — they're kept separate per §4, ready for Phase 5's "sprinkle in for flavor" idea.

Continuing through the remaining phases

• Phase 2 — voice input. Add a stt.py using openai-whisper or faster-whisper (both installable via pip once you have network access to PyPI — this sandbox does): transcribe to text, then feed straight into the existing main.reply() — the rest of the pipeline is already input-agnostic per §6.
• Phase 3 — "thinking" mirror animation. While generator.generate() + assembler.build_reply_video() run, play back a distorted loop of the user's own input (typed text glitch-animated the same way as make_glitch_text_segment, or their recorded voice audio pitched/stuttered) in the UI, then cut to the real reply once build_reply_video() returns. This is a UI-layer concern once you pick a front end (web app / native).
• Phase 4 — word extraction from longer clips. Run whisper with word-level timestamps over every clip already in clips/, and for any vocabulary word that's only available embedded in a longer line, use ffmpeg -ss <start> -to <end> to cut a standalone sub-clip and add it to the index as an additional ClipEntry. This plugs directly into indexer.py as a second pass after the filename-based scan.
• Phase 5 — polish. Tone/energy-aware clip selection, volume normalization (ffmpeg loudnorm filter) between consecutive segments, multi-turn memory (just keep a running conversation string fed into generator.generate()), and a real interface (this whole thing works as-is behind a simple Flask/ FastAPI endpoint if you want a browser UI next).

Requirements

• ffmpeg / ffprobe (already used above)
• espeak-ng (robotic-voice fallback)
• Python 3.10+, stdlib only for Phases 0–1 (no pip installs needed)
• Later: ollama (Phase 1 real generation), openai-whisper/faster-whisper (Phases 2 & 4)