Point at any window, then let a coding agent drive it — click, type, screenshot, and control another app's window.
No API key: the agent (Claude Code, Codex, Cursor, …) is the loop, through a small CLI of window-control primitives.
- macOS — the polished path: per-window capture, live-overlay picker.
- Windows / Linux (X11) — portable backend (mss + pynput + pywinctl);
select windows with
pick --name/pick --listinstead of the overlay. - Wayland — unsupported (it blocks synthetic input and capture by design).
Requires Python ≥3.10.
# 1. Put the `planchette` CLI on PATH
curl -LsSf https://raw.githubusercontent.com/shahriarshm/planchette/main/install.sh | sh
# (or, if you already have uv: `uv tool install planchette`)Then wire it to your agent:
# 2a. Claude Code: install as a plugin
# /plugin marketplace add shahriarshm/planchette
# /plugin install planchette@planchette-marketplace
# 2b. Any MCP-capable agent (Codex, Cursor, …): register `planchette mcp`
# — see "MCP mode" below.
# 2c. Any other agent tool: point it at AGENTS.md in this repo —
# it documents the capture → look → act loop.
planchette mcp serves the same primitives over MCP (stdio). Screenshots
come back inline in the tool result, so the agent needs no file access —
this is the recommended door for Codex (no sandbox bypass needed), Cursor,
and anything else that speaks MCP.
| agent | register with |
|---|---|
| Claude Code | claude mcp add planchette -- planchette mcp |
| Codex | codex mcp add planchette -- planchette mcp |
| Gemini CLI | gemini mcp add planchette planchette mcp |
| Cursor | add to .cursor/mcp.json: {"mcpServers": {"planchette": {"command": "planchette", "args": ["mcp"]}}} |
| opencode | add to opencode.json: {"mcp": {"planchette": {"type": "local", "command": ["planchette", "mcp"]}}} |
Then ask the agent to pick a window (pick_window, or pick_window name="Telegram") and drive it. macOS permissions (Screen Recording +
Accessibility) must be granted to whatever app hosts the agent.
The skills/control-window/ folder is a standard Agent Skill: the same
folder works in Claude Code, Codex, Cursor, and other SKILL.md readers.
| agent | install |
|---|---|
| Claude Code | /plugin → add this repo (also brings the /planchette command) |
| Codex | cp -r skills/control-window ~/.codex/skills/ (or .codex/skills/ in a project) |
| other SKILL.md readers | copy skills/control-window/ into the agent's skills directory |
| anything else | point the agent at AGENTS.md — it documents the whole loop |
Grant these to the app you run your agent from (Terminal, iTerm, VS Code…), in System Settings → Privacy & Security:
- Screen Recording — to capture the target window (else screenshots are black).
- Accessibility — to move the mouse, click, type, and raise windows.
After granting, fully quit and reopen the terminal app.
In Claude Code:
/planchette open a new tab and search for the weather
…or just ask in chat: "control my Telegram window and message Saved Messages 'hi'." Claude runs the picker (hover + click the window you want), then loops: screenshot → decide → act, until the goal is done.
- Background control (macOS): input is delivered straight to the target app's process — the window needn't be frontmost, your focus and cursor stay put, and you can keep chatting with Claude in the terminal while it drives. (Exceptions: modifier combos briefly activate the target and hand focus back; scroll hops the cursor there and back.) On Windows/Linux input follows real focus — don't fight it while it runs.
- One display in v1.
planchette pick # live-overlay picker (macOS) → select a window
planchette pick --list # print candidate windows
planchette pick --name "Telegram" # select frontmost match by app/title substring
planchette capture [--out PATH] # screenshot it → PNG, prints "PATH WIDTHxHEIGHT"
planchette capture --crop X Y W H # native-res zoom of a region (for small text)
planchette click X Y [--double] [--right] # X Y are captured-image pixels
planchette drag X1 Y1 X2 Y2 # select text, sliders, drag & drop
planchette move X Y
planchette type "text"
planchette key "cmd+t" [--repeat N]
planchette scroll X Y up|down [N]
planchette raise
planchette status
planchette agent [--agent claude|gemini] # macOS: pick a window → floating panel drives it
planchette mcp # serve window control over MCP (stdio)
planchette hotkey [--combo C] # macOS: global hotkey (default ctrl+cmd+p) → agent panel
planchette hotkey --install # …as a login LaunchAgent; --uninstall removes it
Window bounds are re-read from the live window before every action, so a
window that moved after pick still gets clicked in the right place.
planchette hotkey --install registers ctrl+cmd+P system-wide (like the ⌘⇧5
screenshot overlay) and keeps it across logins. Pressing it runs
planchette agent: pick a window under the overlay, then a small floating
panel appears — type a goal ("reply ok to the last message") and a headless
agent session (claude or gemini, whichever is installed — force one with
planchette agent --agent gemini) drives that window, streaming its actions
into the panel. Enter again for follow-ups in the same session; Esc closes it.
A dropdown on the input row lists the installed agent CLIs — switch anytime
between turns; switching starts a fresh conversation for the new agent.
The spawned session is scoped to window control only (claude:
--allowedTools "Bash(planchette:*)" Read; gemini: a generated
~/.planchette/.gemini/settings.json capping its tools the same way). The
agent CLI and planchette must be installed; the panel finds them on PATH
plus the usual install dirs (~/.local/bin, /opt/homebrew/bin,
/usr/local/bin). Gemini also needs auth the spawned process can see, e.g.
GEMINI_API_KEY in ~/.gemini/.env.
Note: the LaunchAgent runs outside your terminal, so macOS asks for Accessibility again — grant it to the listed Python in System Settings → Privacy & Security; the daemon retries on its own. Pre-picking for a terminal session still works: pick via the hotkey, then just Esc the panel.