Cerebro

One connectable, provable brain out of a pile of records about people and money.

📋 Feature catalog →  ·  🧮 Calculations & algorithms →  ·  🕸️ GraphRAG architecture →  ·  📈 Forecasting & workforce statistics →  ·  🖼️ Screenshot gallery → — every feature, the exact math behind every metric, and every surface in pictures.

Cerebro is a local-first investigative-analytics platform. Feed it heterogeneous records about an organization and its people — documents, payroll rosters, insurance claims, database tables, emails, web pages — and it turns them into a brain: a searchable index where every person and organization is a first-class entity, linked by the records they appear in. Then you interrogate it — full-text search, an entity-connection graph, and a Workbench of exact-money and statistical analysis — entirely on your own machine. No cloud, no account, no per-document cost.

What makes it more than search over your documents: the entity is the unit of value, and it persists across every source. Because entity ids are content-derived (type:name), the same person is one identity whether they appear in a memo, on a payroll line, or in a claims record. So you follow the people and the money across sources — and Cerebro proves what it tells you, to the cent.

The model — one engine: adapters in, lenses out

• One engine. Ingest records → build a brain → search / graph / analyze / prove.
• Input adapters — the ways a brain gets filled: local files, URLs, RSS, SQL Server, email, calendar, and web-crawl. One roof: cerebro/ingest.py.
• Analytical lenses — what you do with a brain: the document-connection graph (Explore), exact-money / roster analytics (the Roster Suite), and claims-irregularity screening (the FI lens). Each runs over the same brain, through the same DataSource seam.

Where the lenses compose — the payoff

Because every lens keys to the same entity identity, they join. The cross-brain rollup (/panels/rollup) takes every entity that appears in two or more brains and puts each lens's facts on one row — a person's document mentions beside their pay beside any claims money. "Who is discussed and paid and flagged" becomes a single query instead of a week of manual cross-referencing.

Validated at scale. Two payroll brains of 176,000 employees each (352,000 records total) sharing 20% of their people — the rollup surfaces all 35,200 people who appear on both payrolls in ~3 seconds, each with their salary at each employer side by side (e.g. one person drawing ~$258k at Roster A and ~$258k at Roster B — $518k combined). It finds every match, not a top-N sample, because it intersects the full entity-id sets and only sums money for the shared set. Reproduce it with generate_overlap_rosters.py. (Synthetic data — a scale/correctness demonstration, not a production figure.)

Built vs. growing (in plain terms): the engine, the three lenses, and the cross-brain join ship today and are covered by tests (and by the 352k-record scale run above). The rollup's density scales with overlap — it lights up the moment two brains concern the same people (a company's documents and that company's payroll). That shared-subject data is the part that grows over time; the machinery to compose it is already here.

It proves its own math

Cerebro's differentiator is not that it's clever — it's that it's checkable. Money is counted to the cent, aggregations are pushed down into the query (population-complete, no row caps), and the flagship roster report is re-derived line-for-line from the raw source by an independent auditor (~1,000 checks) before it counts as done — a failed audit fails the command. For output that has to survive a court, a regulator, or an auditor, the tool re-derives every number and refuses to lie.

---

Why it scales where a notes app doesn't

A personal-notes app loads every file at startup, so it falls over on a corpus of hundreds of thousands of documents. Cerebro never does that. Search is an indexed lookup, and the graph is always a bounded neighborhood around whatever you're looking at — it renders the slice you clicked, never the whole universe. The same design serves 600 documents or 6 million; only the index on disk grows.

Three design choices keep it flat at scale:

• Semantic search is ANN, not brute force. Related-notes and "find similar" queries hit an HNSW approximate-nearest-neighbor index (hnswlib, cosine) — O(log N) per query, and the embedding matrix is never loaded into memory. The old path scanned every vector per query and cached the full matrix.
• The import is streaming + append-only. Each file goes through the full pipeline individually and its result is appended to a resume log — a crash at document 400,000 of 500,000 resumes at 400,001, never restarts. Nothing is held in memory that grows with the corpus, and the checkpoint is never rewritten wholesale.
• The graph renders a bounded neighborhood, so a 6-million-connection index draws just as fast as a small one.
• Opening a document is an O(1) lookup. The full text is stored in a primary-key-indexed documents.body column, so the reader fetches it directly instead of scanning the search index. A single write_body() chokepoint keeps that copy and the FTS index in lockstep — they cannot drift, and it raises rather than silently orphan an index row.

---

The Explore workspace

Open https://localhost:8800/ after building an index. The center stage is a live neural brain — a glowing cyan filament field firing from a bright pulsing core (canvas, additive-bloom, not a stock image), framed by a heads-up display.

• Left — search documents, people, orgs, and places; filter entities by type.
• Center — the neural brain. In the overview it's just the brain (clean, no clutter). Search or click an entity and its connections light up on the brain: colored nodes are entities (blue people, gold orgs, green places, purple emails, orange phones), cyan dots are documents, and lines mean "this entity appears in this document." Drag to rearrange, scroll to zoom. A HUD shows live INDEX TOTALS and a CORTEX STATUS panel (your real entity-type counts).
• Right — a document reader: extracted text and the original PDF side-by-side, with clickable entity chips that jump you to that entity's connections.

The canvas throttles itself on weaker GPUs (fewer threads, softer glow) so it stays smooth on modest hardware.

---

The Data Dissection Workbench

Drilling into a dataset opens the Workbench — a surgical-tray graph that is the analysis surface. Right-click any jar (dataset) or cabinet (collection) on the shelf and Send to Workbench. A project holds a working set of datasets that may span multiple brains (separate indexes / SQL Server databases); they merge into one cross-brain graph. Because entity ids are name-derived, the same entity unifies across brains automatically, and any entity that bridges two brains is flagged.

A Calculations panel runs, on-demand and cross-brain, over the working set:

Tab	What it answers
Profile	docs, distinct entities, mentions, avg mentions/doc, empty-snippet rate, date span, by-type breakdown
Joins	entities present in both / only one / either of two dataset groups (inner, left-anti, union — the "complex left joins")
Overlap	pairwise shared-entity count + Jaccard similarity between datasets
Aggregate	SQL-style GROUP BY entity type or dataset, with rollups
Statistics	mentions-per-entity distribution (mean/median/percentiles), log-scale histogram, z-score & IQR outliers
Path	the shortest co-occurrence path between two entities (entity → shared doc → entity …)
Communities	clusters of entities that co-occur in the same documents (connected components)
Documents	full-text search across the whole working set
Watchlist	match a list of persons-of-interest (typed or imported from a file; exact + fuzzy)
Timeline	documents over time (day/month/year)
Quality	reconciliation (docs with vs without extracted entities) + duplicate-entity candidates to merge

Every result is exportable as CSV, carries the generated SQL behind it (View SQL → copy/save .sql), and entity names link straight into the graph. The whole working-set graph exports as GraphML for Gephi/Cytoscape. Calculations stay responsive on a 250k-entity index via per-set caching and bounded sampling (no hub-sized joins).

📋 Full feature catalog: FEATURES.md — every calculation, federation, collaboration, and ingestion capability at a glance.

---

Financial ledger analytics (the Roster Suite)

Cerebro distinguishes two kinds of money data, because their math differs:

• Corpus brains (the default) hold documents that mention money — the same figure in many documents is usually the same money, so money calculations dedup repeated amounts.
• Ledger brains hold documents that are financial records — a payroll roster line, a claim, a transaction. Identical amounts on different records are different dollars, and every calculation counts each record's amount exactly, to the cent.

That exactness is enforced at the engine level: money aggregations are pushed down into the query (population-complete, no row caps), and calcs that must assemble rows in Python count the true population first and lift their caps to it — a truncated total can never present itself as a complete one. On a 532,000-row multi-month brain, every published total equals a direct SQL SUM over all rows (design).

The Roster Suite is the ledger side, built around position-control-number (PCN) payroll rosters — a row is a funded position line, not a person; lines without employee IDs are vacancies, subcontracts, and sessional pools carrying placeholder names. The whole loop runs in the app:

• ＋ Load snapshot — upload a roster .xlsx + its as-of date; the atomic loader validates while building (duplicate PCNs abort, computed totals reconcile against the export's own grand-total columns) and registers one brain per snapshot, grouped in a cabinet.

• Audited reports — every generated report is re-derived line-for-line from the raw spreadsheet by an independent auditor (~1,000 checks on the flagship report) before it counts as done; a failed audit fails the command. Analyst "why" notes carry their proof arithmetic.

• Plausibility screens — record-vs-peer-median pay outliers and $/FTE rate outliers (high and low side) with a consensus rule across every org group a record belongs to, job titles on every flag, terminated-record handling, and identical-amount cluster callouts (four records at exactly $19,000 are one stipend policy, not four anomalies).

• Vacancy aging — every funded-but-unfilled line bucketed by how long it has been open, with the 1-year+ money called out.

• ⇄ Movement — PCN-joined snapshot-to-snapshot comparison: added / removed / changed lines with a cross-foot proof (added − removed + changed must equal the total-to-total difference), every row linked to its source record.

• ⇆ Payroll map — upload a payroll extract and map real people onto the pool/sessional budget lines by PCN, with per-pool coverage and an optional apply that writes the mapped people into the brain's graph.

📖 Full guide: docs/ROSTER_SUITE.md — the exactness guarantee and its boundary, the verification chain, and every screen.

---

Claims-irregularity screening (the FI lens)

The third lens runs over ledger brains of health-insurance claims (build_claims_brain.py). It scores claims and providers for irregularities that warrant human review — never a verdict, never the word "fraud" — in tiers: per-record and cross-record rules (duplicate/unbundled/impossible-day billing), statistical soft signals (Benford, peer-outlier, concentration, templated-narrative reuse), a referral-ring graph, and an optional supervised model. It shares nothing with e-discovery except the substrate: the same brain, the same DataSource seam, the same entity identity — which is exactly why a flagged provider can also surface in the cross-brain rollup beside their document mentions and money.

📖 Guides: docs/FINANCIAL_IRREGULARITIES.md · docs/FI_USAGE.md. The "irregularities, not fraud" rule is enforced in code (a unit test forbids the word in the package).

---

Collaboration (real-time)

Workbench projects are shared server-side, so multiple browser sessions see and edit the same investigation — modeled on Databricks Omnigent's collaboration layer. The server binds loopback only (127.0.0.1, hard-pinned — the LAN scaffolding was deleted in the 0.3 hardening), so "shared" means shared on this machine; add multi-user access with the optional local-accounts layer (CEREBRO_AUTH=1, below).

• Live presence — who's in the Workbench right now.
• Comments & findings — discuss a project; pin a calculation's result as a finding the whole team sees.
• Share & invite — copy a link to a project, or generate a single-use invite.
• Investigation report — compile the working set + findings + discussion into a downloadable Markdown brief.
• Live updates — edits, comments, findings, and presence push over Server-Sent Events; a change merges into your graph in place without resetting your view.

Edits are atomic (delta add/remove of datasets), so two sessions editing the same project never clobber each other.

---

Automation (scheduled SQL polling)

Cerebro can keep an index growing from live SQL sources. The Automation panel registers a SQL source (connection string → OS keyring, table, watermark column, target brain) — with a Test connection dry-run — and a background scheduler incrementally polls it:

• Watermark-based change data capture — each poll pulls only rows newer than the last watermark, ingests them (text + entities), and advances the checkpoint. A crash resumes from the last value, never a full re-ingest.
• Opt-in & safe — auto-polling is gated on a per-source flag that is off by default (separate from enabled), so a one-time backfill source is never re-ingested behind your back; each source routes to its own index.
• Index caches rebuild on a slower cadence, only when new rows actually arrived, and the Workbench Live toggle shows the graph grow.

---

Background enrichment (the Process menu)

Heavier per-document analyses are kept out of the fast text index so the core stays quick, and run on demand from the Process menu (top-right of Explore) as background jobs:

• OCR scanned documents, image captions (BLIP), visual Q&A, face detection, video/audio transcription, similarity linking.
• Every job has Start, a live progress bar, and Stop. Stop is cooperative and commits partial work before ending — nothing already processed is lost, and re-running resumes where it left off.
• Results are written back into the index and become searchable/visible. GPU jobs are best run when the GPUs are free.

---

How records get in — the input adapters

A brain gets filled through one of several input adapters, gathered under cerebro/ingest.py so "how do records get in?" has one answer. Web-crawl is one input among many, not a separate product:

Adapter	Source	Target
pipeline.from_files / from_url / from_rss	local files, web pages, feeds	notes → index
ingest.from_sqlite / from_connection	SQL rows (SQLite / SQL Server)	directly into a brain
sql_poller	live SQL, incremental (watermark CDC)	directly into a brain
ingest.from_crawl	a crawled site	directly into a brain
ingest_email / ingest_calendar	mail + calendar	notes → index

The document path (cerebro/pipeline.py, streaming + checkpointed — resumes after any crash) processes each file individually:

• Text is extracted directly from PDFs that have a text layer (most of them).
• Scanned / image-only PDFs are OCR'd in memory so they're still searchable — nothing is left as a black hole in search.
• The source PDF is embedded (hardlinked when on the same drive — zero extra disk) rather than exploded into one image per page, so the document count stays manageable and imports stay fast.
• Entities — people, organizations, locations, emails, and phone numbers — are extracted with spaCy + regex and become the nodes of the graph, keyed on type:name so the same entity is one identity in every brain. (Dates and money are detected during extraction but are not graphed.)

Full per-page image enrichment (BLIP captioning, VQA, face detection) is off by default. Turn it on for a run with CEREBRO_ENRICH_IMAGES=1 — it uses the GPU.

---

Quick start

Python: run on 3.12 if you have an NVIDIA GPU (the CUDA PyTorch wheels only support 3.12); any newer Python (e.g. 3.14) is fine on a machine with no GPU — it runs CPU-only. The startup check (cerebro/check_env.py) enforces exactly this: it aborts only when a GPU is present on a non-3.12 interpreter. From the project root:

# 1. Build a search/graph index from a folder of PDFs (resume-safe).
python -m cerebro.index build --source "D:\Raw Data\MyDocs" --limit 20000

# 2. Launch the app.
python -m cerebro.web
# → open https://localhost:8800/  (self-signed cert; accept the warning)

# Inspect the index any time:
python -m cerebro.index stats

The index lives in index/cerebro.db. Point the app at a different one with the CEREBRO_INDEX environment variable.

To import documents into the older vault/notes pipeline (Markdown notes + embedded PDFs), use the import UI at /import or the CLI:

python -m cerebro --help

---

Requirements

• Python — 3.12 with a GPU (CUDA PyTorch wheels are 3.12-only), or any newer Python with no GPU (CPU-only). GPU-gated at startup in cerebro/check_env.py.
• PyMuPDF (fitz) for PDF text/image extraction.
• spaCy with en_core_web_sm for entity extraction.
• sentence-transformers + hnswlib for semantic search and related-notes (the HNSW ANN index).
• Tesseract on PATH for OCR of scanned PDFs (optional but recommended).
• CUDA PyTorch only if you want GPU image enrichment; the app and search need no GPU.

Dependencies are checked at startup (cerebro/ensure_deps.py) and installed if missing — GPU-aware: with no GPU it installs the plain CPU wheels (torch, onnxruntime) instead of the CUDA builds.

---

Glossary — the metaphor stack, in one place

New readers meet several nested metaphors at once; each maps to exactly one concept:

	Term	What it actually is
🧠	Brain	One searchable index (a SQLite index file, or a live SQL Server database). The federation registry (brains.py) can serve many at once.
📚	Constellation	The home visualization: every brain drawn as a bookshelf cabinet you can drill into.
🗄️	Cabinet / shelf / jar	Levels inside the constellation: a cabinet is a brain, shelves group datasets, a jar is one dataset.
📁	Library (code: vault_dir)	A folder of Markdown notes + attachments that imports write into. "Vault" in code identifiers is the legacy internal spelling of the same thing.
🔬	Workbench	The analysis workspace: set operations, money flow, watchlists, dossiers over the selected datasets.
🫙	Dataset	One imported corpus inside a brain (e.g. one document production).
📝	Note	One Markdown file: frontmatter identity (source, sha256) + extracted body + generated sections.

---

Architecture

cerebro/
├── index.py          ← SQLite FTS5 search + PK-indexed body + entity→document graph
├── datasource.py     ← DataSource seam: SqliteSource + SqlServerSource (live SQL Server brain)
├── sources.py        ← ONE brain→DataSource resolver, shared by the web + workbench layers
├── brains.py         ← federation registry — serve many indexes at once
├── ingest.py         ← input-adapter roof: from_crawl (+ re-exported SQL adapters)
├── workbench_ops.py  ← cross-brain calculations facade (joins, overlap, stats, …)
├── workbench/        ← calculation engine: sources seam, corpus money, ledger money
├── fi/               ← claims-irregularity lens: tiered detectors, referral graph, model
├── roster_common.py  ← shared roster invariants (PCN, person/not-person rule, pay map)
├── collab.py         ← shared projects, comments, findings, presence, invites, SSE
├── poll_scheduler.py ← automation backbone — scheduled incremental SQL polling
├── sql_poller.py     ← watermark CDC: pull-newer-than, ingest, checkpoint
├── crawler.py        ← bounded web crawler (an input adapter, via ingest.from_crawl)
├── graph_ops.py      ← path-finding, centrality, communities, cross_brain_rollup
├── pipeline.py       ← streaming, checkpointed import (text + OCR + embedded PDF)
├── streaming.py      ← per-file processing, batch NER, batch finish
├── extractors.py     ← PDF text/image extraction, in-memory OCR
├── entities.py       ← spaCy + regex entity extraction (entity_key: type:name)
├── ner_gpu.py        ← GPU-batched transformer NER (fast path for big builds)
├── device.py         ← GPU-first / CPU-fallback selector
└── web/
    ├── app.py        ← Flask app (security, import/admin API)
    ├── explore.py    ← Explore + Workbench + collab + automation APIs
    ├── panels.py     ← HTMX panel surface (search, stats, money, timeline, rollup)
    ├── templates/    ← explore.html + panels/*.html fragments
    └── static/       ← brain-graph engine, Workbench UI, panels-state.js

GraphRAG, the honest kind. Cerebro is a graph-grounded retrieval system: entities are extracted at ingest (spaCy/GPU-BERT + regex), the mentions table is the entity↔document edge set, and retrieval runs as BM25/full-text over that graph — no vector store, no retrieve-then-generate. Synthesis is deterministic and proof-carrying (every figure re-derives from source), with one strictly-grounded local-LLM narrator that never invents a number. The full stage-by-stage mapping, a pipeline diagram, and an explicit list of what it does not do are in docs/GRAPHRAG.md.

The federation model: each brain is its own source behind the DataSource seam, and the Workbench merges them cross-brain in app code. A brain can be a local SqliteSource index or a SqlServerSource — a live SQL Server database (carrying the Cerebro schema) served in place, nothing copied. Register one with brains.register_brain(id, name, kind="sqlserver", connection=...) and it appears in the constellation, drills into a live graph, and is searchable like any other brain. Every Workbench calculation routes through this seam, so a single working set can mix a local SQLite index and a live SQL Server database and every calculation (joins, overlap, statistics, path-finding, communities, money flow, …) runs across both at once. See CALCULATIONS.md for the exact formula behind each one.

Logs go to logs/ (rotating; INFO+ to console, DEBUG+ to file, WARNING+ to a separate error file). Tests: py -3.12 -m pytest for the Python suite (incl. test_real_dataset.py, which validates the full calc suite against the real brain and skips when absent), and node test_js/run.js for the front-end formatting/escaping unit tests.

Optional auth. Set CEREBRO_AUTH=1 to require local accounts (salted PBKDF2-HMAC-SHA256 hashes in .cerebro/auth.db, expiring sessions, lockout after 5 failed logins). Off by default. Per-brain read and write ACLs are enforced at every source-resolution chokepoint — not just the constellation picker, so the API cannot serve a restricted brain's data by id. Sensitive actions (view/search/import/export/production) are recorded in a hash-chained access log. Bootstrap the first (admin) account via POST /api/auth/bootstrap, then /api/auth/{login,account,grant}.

---

Status

Cerebro began as a document import pipeline and grew into a platform: one engine (ingest → brain → analyze → prove), several input adapters, and several analytical lenses, all keyed on a shared entity identity. The Explore workspace (the neural-brain front door) is the way in; the Data Dissection Workbench is where the analysis happens; the legacy import/admin UI lives at /import.

Built and running at scale:

• A 1,000,000+ document e-discovery index (≈450k entities, 6.5M+ connections), built with the sharded, resume-safe builder (python -m cerebro import --shard N/M per shard, then cerebro.index.merge), plus a second brain served from SQL Server (331k documents) — the two analyzed together, cross-brain, in the Workbench. Every Workbench calculation stays under ~8 seconds on that 860k-document / 458k-entity working set.
• The Roster Suite (the money lens) runs against an 8,800-line / $1.76B payroll roster; its flagship report passes a 1,000-check line-for-line audit against the raw spreadsheet on every generation.
• The FI lens and the cross-brain rollup ship with test coverage.

Shipped in v1.0.0 (2026-07-14): the launch hardening — loopback-only bind (LAN scaffolding deleted), optional local accounts with per-brain read/write ACLs and login lockout, and a hash-chained access log. Since then, a full-codebase audit closed ~30 correctness/perf/security findings, document opens became an O(1) lookup (PK-indexed documents.body), and DuckDB joined SQLite and SQL Server as a brain backend — see docs/CHANGELOG.md.

Growing / forward-looking: the rollup's payoff scales with how much your brains overlap in subject — it is densest when, for example, a company's documents and that company's payroll are both loaded. The composition machinery is built; the shared-subject data that lights it up is what accrues over time.