idasql-inspired analysis layer: SQLite index, SQL query, Ghidra daemon

[Night1099]

Summary

Adds an idasql-inspired analysis layer to the RE toolkit: a per-game SQLite
index, a read-only SQL front-end, a consolidated kb.h parser, and a warm
per-project Ghidra daemon with transparent routing. Static findings that
used to live only in a flat kb.h are now also queryable as structured data,
and repeat Ghidra work (decompile / export / kb-apply) skips the ~3s JVM cold
start.

The whole branch was reviewed end-to-end and then validated on a real binary
(Mount & Blade: Warband mb_warband.exe, 32-bit DX9) — which surfaced and fixed
a real kb-apply transaction bug that only appears against actual Ghidra.

What's new

retools/index.py — per-game SQLite index (GameIndex)

• Schema for funcs/names/xrefs/strings/imports/entries/segments/blocks, plus
  callers / callees / grep views.
• WAL-with-DELETE fallback, schema-version guard, bit-63 address guard.
• Source-priority upsert: authoritative Ghidra rows overwrite provisional
  bootstrap rows, and re-running bootstrap can never downgrade them back.
• index status <Game> CLI.

retools/query.py — read-only SQL front-end

• query <Game> "SELECT …" over index.db via a file:…?mode=ro connection
  (can never mutate the index). Text + --json envelope, --list-tables,
  --schema TABLE.

retools/kb.py — one kb.h parser

• Single source of truth for the kb.h grammar; the three former duplicate
  parsers (decompiler, context, bootstrap) now delegate to it.

retools/pyghidra_backend.py — export + kb-apply

• export seeds index.db with authoritative source='ghidra'
  funcs/names/xrefs/blocks.
• kb-apply pushes kb.h names/labels/prototypes into the Ghidra project.
• All Ghidra imports are lazy so the module loads with no JDK.

retools/ghidra_server.py + ghidra_client.py — per-project daemon

• Warm Ghidra program on port 27043 (livetools owns 27042). decompile,
  export, and kb-apply route through a live daemon automatically;
  RETOOLS_GHIDRA_COLD=1 / --cold forces cold. Tracks itself in
  patches/<Game>/ghidra/.state.json.

bootstrap.py / context.py

• Bootstrap seeds index.db (source='bootstrap') from pefile data.
• context assemble resolves callees from index.db when present, falling
  back to the disassembly scan otherwise.

Real-world validation (Warband)

Ran the full pipeline on mb_warband.exe:

Step	Result
bootstrap	seeds index.db (9,648 funcs, 35,201 strings, 376 imports) + kb.h
Ghidra analyze	~96s, reusable project
export (via daemon)	funcs=10,243, names=10,243, xrefs=220,237, blocks=137,368
source-priority	funcs = 12,502 = 10,243 Ghidra + 2,258 surviving bootstrap (no downgrade)
warm decompile	0.46s → 0.15s (cached DecompInterface) vs 3.3s cold
kb-apply	2,890 functions renamed from kb.h, persisted + re-exported
context assemble	real named callee edges from the index

master vs this branch, same task: master reaches the same answer but with
16 modules (no index/query/ghidra_server), a bootstrap that produces no
index.db, ~4s cold decompiles with no warm path, and context callees
limited to edx: indirect call placeholders. The improvements don't change the
answer — they make the toolkit queryable and fast to iterate (SQL over
220k xrefs, 0.15s warm decompiles, named cross-references).

Review findings fixed

Correctness

• Ghidra daemon cross-project routing hardened: pid-first liveness check +
  server-side game-identity handshake + dropped SO_REUSEADDR.
• export default --db derives from --project (was the binary stem →
  wrong dir); _iter_blocks keys blocks by containing-function entry.
• _route_daemon propagates socket.timeout instead of silently cold-retrying
  into a mid-operation daemon; sends absolute paths (cwd-independent).
• context.assemble: verifies the start is a known function before trusting an
  empty callee result, falls back on a schemaless index.db (was crashing),
  recovers indirect-call markers, and prefers fresh kb.h names.
• index.db path anchoring unified; schema-version repaired on interrupted
  first open; decompiler.py script-mode import fixed.
• kb-apply transaction bug (found on real Ghidra): removed the manual
  program.save() that fails with "Unable to lock due to active transaction"
  inside pyghidra's open_program context — persist via project.save on exit
  (cold) / program close (daemon); @ entries at data addresses become labels,
  not bogus functions.

Cleanup / conventions

• ix_xrefs_from_func index; narrowed catch-all excepts; single-threaded daemon
  loop (removes the lock, close-variant split, thread list, idle self-connect);
  reused framing; error response on malformed frames; cached DecompInterface;
  parse kb.h once per assemble; shared resolve_db; one bootstrap string
  sweep; extracted shared _kb_apply_txn.

Tests

123 passed / 1 skipped across all changed modules, including new coverage for
the daemon lifecycle + identity handshake, source-priority, empty-xrefs
fallback, and the kb-apply label path.

Still needs manual verification on a Ghidra-equipped host

The real Ghidra export / kb-apply class paths and the daemon warm-program
lifecycle were exercised on Warband in this branch; a second independent host
run is worthwhile before merge.

🤖 Generated with Claude Code