⚖️ CodeCouncil

Multi-agent code review where the vote is an auction, not a show of hands.

CodeCouncil puts several coding agents — Claude Code, Codex, Cline, Ollama, or your own — through a generate → review → debate → vote loop, and settles their disagreements using auction theory: the branch of mechanism design, itself a subfield of game theory, that studies how to design rules which still produce good outcomes when the participants are self-interested and their private information cannot be verified.

The problem it solves

Every review carries a self-reported confidence between 0 and 1. Average those and the loudest reviewer wins: an agent that says 0.95 about everything outvotes a well-calibrated one that says 0.6 when it means 0.6. Nothing checks the claim, and language models are not calibrated by default.

That is not a bug to patch — it is a mechanism design problem with decades of theory behind it. Confidence is a bid. The tally is an auction. CodeCouncil ships four mechanisms for it:

Vickrey	Caps a reviewer's influence at the second-highest confidence in the round, so one loud voice cannot run the table.
VCG	Weights by pivotality — would removing this reviewer change the outcome? Echoing the majority earns almost nothing.
Double auction	Reviews become quality bids that must clear the author's asking price. An explicit, tunable bar.
Trust-weighted	Weight = confidence × reputation earned over past rounds. The council learns whom to believe.

Behind them sits a persistent reputation ledger. Agreeing with the outcome earns reputation; disagreeing costs 1.2× more than agreeing pays, so confidence cannot be farmed by voting loudly and often. Over time the council converges on trusting the reviewers that have actually been right.

What it does not claim: these are not strategy-proof, despite the names — there is no payment, so a reviewer can still gain by inflating, just not without bound. The honest analysis, with numbers, is in Voting & Mechanism Design, and the behaviour is pinned in tests/test_auction_incentives.py.

The loop

1. Generate — One or all agents write code for a task
2. Review — Agents cross-review each other's work
3. Debate — Optionally, agents see each other's reviews and may change position
4. Vote — A voting mechanism settles it (majority, or one of the four above)
5. Revise — Author incorporates feedback and resubmits
6. Repeat — Until the council approves or max rounds are hit

🚀 Quick Start

# Install with all agents
pip install "codecouncil[all-agents]"

# Run a task
codecouncil run "Build a REST API for a todo app"

# Parallel mode — all agents compete
codecouncil run "Implement a binary search tree" --parallel

# Use specific agents and voting strategy (repeat -a per agent)
codecouncil run "Write a rate limiter" -a claude -a codex --strategy supermajority

# Only have one provider? Seat it several times at different models. Each is a
# separate session, and debate needs 3+ (an author never reviews its own work).
codecouncil run "Write a rate limiter" \
  -a claude:claude-opus-5 \
  -a claude:claude-sonnet-5 \
  -a claude:claude-haiku-4-5-20251001 \
  --debate-rounds 2

# Launch the dashboard
codecouncil dashboard --port 8080

# Start the MCP server
codecouncil mcp

📦 Installation

# Minimal (bring your own agent adapters)
pip install codecouncil

# With specific agents
pip install "codecouncil[anthropic]"     # Claude Code
pip install "codecouncil[openai]"        # Codex
pip install "codecouncil[all-agents]"    # Everything

# With dashboard & MCP server
pip install "codecouncil[all-agents,dashboard,mcp]"

# Development
pip install -e ".[dev]"

⚖️ Voting & Mechanism Design

Most multi-agent review tools average the reviewers and ship the majority opinion. That quietly assumes every reviewer is equally good and equally honest about how sure they are. Neither holds.

The intellectual lineage here is specific: game theory studies strategic interaction; mechanism design inverts it, asking what rules produce a desired outcome when players act in their own interest; auction theory is its most developed branch. Three of the strategies below come straight from it — Vickrey (1961), VCG, and the double auction — while trust_weighted is reputation weighting rather than an auction proper.

Vickrey, Clarke and Groves are the canonical answers to "how do you aggregate private valuations from parties who have no reason to be honest?" — exactly the question a review council faces when every confidence score is self-reported and unverifiable. The lineage is decorated: Vickrey took the 1996 Nobel, Hurwicz, Maskin and Myerson the 2007 prize for mechanism design, and Milgrom and Wilson the 2020 prize for auction theory.

The strategies

Pick one with --strategy / -s:

Strategy	How it decides	Reach for it when
simple_majority	One agent, one vote. Confidence ignored.	You want a plain headcount.
weighted (default)	Sum of confidence per option.	Reviewers are roughly equally trustworthy.
supermajority	2/3 must agree, else needs_changes.	Merging costs more than iterating.
vickrey	Weight is capped at the second-highest confidence in the round.	One agent tends to be loudly overconfident.
vcg	Weight by pivotality — does removing this reviewer change the outcome?	Several agents echo each other and you want to discount the chorus.
double_auction	Each review becomes a quality bid (approve → confidence, reject → 1−confidence). The median bid clears against the author's ask, default 0.8.	You want an explicit, tunable quality bar rather than a popularity contest.
trust_weighted	Weight = confidence × reputation earned over past rounds.	You have history and some agents have proven better than others.

Verify the mechanisms on your install at any time:

codecouncil auction-check

Reputation: the part that learns

TrustLedger scores each agent 0–100, starting at 50. After every round, a reviewer that voted with the final outcome gains, and one that voted against it loses. The update is deliberately asymmetric — losing costs 1.2× what winning pays — so reputation cannot be farmed by voting confidently and often:

reviewer votes approve @0.9, council approves  ->  50.0 + 4.5 = 54.5
reviewer votes approve @0.9, council rejects   ->  50.0 - 5.4 = 44.6

Persist it across runs and the council gets better at knowing whom to believe:

codecouncil run "Add retry logic" \
  -a claude:claude-opus-5 -a claude:claude-sonnet-5 -a ollama:deepseek-coder \
  --strategy trust_weighted \
  --reputation-file .codecouncil/reputation.json

Reputation is recorded under every strategy, so you can run weighted for a while and switch to trust_weighted later with a warm ledger. Without --reputation-file it is in-memory only and resets each run — which means trust_weighted on a cold ledger behaves like weighted, since every agent sits at the same starting reputation.

What these mechanisms do and don't guarantee

This is the honest part, and it matters if you are choosing a strategy.

They are not strategy-proof, despite the names. In a sealed-bid Vickrey auction truthful bidding is dominant because the winner pays the second price. Here nothing is paid. Effective weight is min(bid, second_price), which rises with the bid until it saturates:

truly believes 0.30, another reviewer at 0.90
  reports 0.30 -> effective weight 0.30
  reports 0.70 -> effective weight 0.70     <- lying paid
  reports 0.90 -> effective weight 0.90     <- capped here
  reports 0.99 -> effective weight 0.90     <- lying stops paying

So the cap bounds how much a reviewer gains by inflating, not whether it gains. trust_weighted is manipulable in the same direction, and VCG's weights also shift with reported confidence. What actually punishes inflation is the ledger's asymmetry, applied across rounds — reputation, not the tally. These properties are pinned in tests/test_auction_incentives.py; if you change a mechanism and those tests fail, the incentives moved.

Ties never ship code. An exact tie resolves to the cautious option (needs_changes, then reject, and only then approve) rather than to whichever review happened to be first in the list. This is not hypothetical: a Vickrey tally over exactly two reviewers always ties, because both are capped at the lower confidence — and a three-agent council produces exactly two reviews per artifact.

The bids come from language models. The mechanisms assume a bid means something. LLM confidence is typically miscalibrated and often sycophantic, so in practice these are better understood as robustness measures against miscalibration than as defences against a strategic adversary. The trust ledger is the piece that adapts to how wrong a given model's self-assessment actually is.

Same-provider councils correlate. Three instances of one model agree on things three different providers would have argued about. The vote is only as informative as the diversity behind it.

🔌 Plugin System

Add custom agents by implementing BaseAgent and registering via entry points:

# In your plugin's pyproject.toml
[project.entry-points."codecouncil.agents"]
my_agent = "my_package:MyCustomAgent"

from codecouncil.agents.base import BaseAgent, AgentRole

class MyCustomAgent(BaseAgent):
    def __init__(self):
        super().__init__(
            agent_id="my_agent",
            name="My Agent",
            supported_roles=[AgentRole.GENERATOR, AgentRole.REVIEWER],
        )

    async def generate_code(self, task):
        ...

🏗️ Architecture

┌─────────────────────────────────────────────┐
│              CLI / Dashboard / MCP           │
├─────────────────────────────────────────────┤
│              Orchestrator                    │
│  ┌────────────┐  ┌─────────┐  ┌──────────┐ │
│  │ Task Queue  │  │ Voting  │  │ History  │ │
│  └────────────┘  └─────────┘  └──────────┘ │
├─────────────────────────────────────────────┤
│              Plugin Registry                 │
│  ┌───────┐  ┌───────┐  ┌───────┐  ┌─────┐ │
│  │Claude │  │ Codex │  │ Cline │  │ ... │ │
│  └───────┘  └───────┘  └───────┘  └─────┘ │
└─────────────────────────────────────────────┘

🔒 Security

Read this before pointing CodeCouncil at anything you care about.

Agents execute autonomously. The Codex agent runs the CLI with --full-auto and the Claude agent runs claude -p. Both act without per-step confirmation. Run the council on code you are willing to have modified, ideally in a container or a scratch checkout.

Scanned code is untrusted input. Whatever you point --repo at is fed verbatim into agent prompts, so comments or strings inside it can attempt to steer the agents — and the Codex agent acts on what it is told without asking. Treat scanning a repository you do not control as running its contents, and sandbox accordingly.

Repository contents leave your machine. codecouncil run --repo reads matching source files and sends them to whichever provider backs the configured agent (Anthropic, OpenAI, or a local Ollama host — note OLLAMA_HOST may point off-box). Files whose names look like credential carriers — .env*, *secret*, *credential*, *.pem, *.key, *.tfvars, kubeconfig, and more — are excluded by SENSITIVE_FILE_PATTERNS in repo_scanner.py, and .gitignore is honored including nested files. The same exclusion gates the agent file tools described below, so both routes into model context apply one rule. That is a safety net, not a guarantee: review what you scan.

The dashboard has no authentication. codecouncil dashboard binds to 127.0.0.1 by default. POST /api/tasks will spawn agents and spend API credits for anyone who can reach it, so only pass --host 0.0.0.0 behind a trusted network or an authenticating reverse proxy.

Agent file access is confined to the repo. The Ollama agent explores via read_file, list_directory, search_code, and write_file rather than a context dump. All four resolve paths and reject anything outside the scanned repository — .. traversal, sibling directories, and symlink escapes — and all four refuse credential filenames, so a secret can be neither read out through a tool call nor overwritten by an injected instruction. They also refuse to run at all without an explicit repository root, so they never fall back to the working directory.

One agent can fetch a CLI from npm. If codex is not already on your PATH, the Codex agent falls back to npx @openai/codex — a real package published by OpenAI — which npm will download and execute. That is the only registry fetch CodeCouncil performs; no other agent shells out to npm. Install the Codex CLI yourself if you would rather it never reach a registry at all.

API keys are read only from the environment (ANTHROPIC_API_KEY, OPENAI_API_KEY) and are never logged or written to disk. Copy .env.example to .env — which is gitignored — rather than hardcoding them.

Found a vulnerability? See SECURITY.md — report it through a private advisory, not a public issue. That file also lists what is in scope and what is documented behaviour.

🤝 Contributing

Contributions welcome. CONTRIBUTING.md covers setup, the checks CI enforces (make test, make lint), how to add an agent plugin, and the areas where a careless change breaks a security guarantee.

pip install -e ".[dev]"
pre-commit install
make test && make lint

By participating you agree to the Code of Conduct.

📄 License

MIT — see LICENSE.