green-agent

An A2A "green agent" that orchestrates and evaluates end-to-end agent runs on the AppWorld benchmark. It is a scenario built on top of the open-source AgentBeats SDK/platform.

What's mine vs. not: The AgentBeats framework (agent lifecycle, A2A/MCP transport, battle backend) and the AppWorld benchmark itself are external, open-source dependencies — not my work. Within this scenario, the Blue Agent (scenarios/appworld/blue_agent/) is the agent-under-test that this green agent evaluates; it is also not my work and is included only so the scenario is runnable end-to-end. My contribution is the green-agent orchestration layer in scenarios/appworld/green_agent/ — the code that sets up the AppWorld environment, scopes the API surface for the task, hands the task to the Blue Agent, monitors it, and scores the outcome.

Architecture

The green agent follows a fixed 11-step orchestration plan (defined in green_agent/green_agent_card.toml) for every battle:

1. Extract task ID from the battle's task config ("Task description: {task_id}").
2. Initialize the environment — setup_appworld_environment() POSTs to AppWorld's /initialize endpoint and captures the task instruction, supervisor profile, and current datetime.
3. Predict required APIs — predict_required_apis() narrows AppWorld's ~457-API surface down to the ≤20 APIs relevant to this specific task.
4. Build the task message — build_task_message() assembles the A2A message sent to the Blue Agent, embedding the instruction, supervisor info, and the scoped API list.
5. Dispatch to the Blue Agent via talk_to_agent() (this call is expected to time out — the Blue Agent runs asynchronously and reports completion separately).
6. Poll for completion — check_task_completed() is polled every 5-10s until the Blue Agent calls supervisor__complete_task(), or a 20-minute timeout is hit.
7. Archive state — save_appworld_state() snapshots the AppWorld database/logs before evaluation.
8. Collect metrics — get_complete_task_metrics() merges AppWorld's pass/fail evaluation with trajectory analysis of the run.
9. Judge the outcome — Blue wins if AppWorld's evaluation reports success=true; otherwise Green wins.
10. Report — results are published via report_on_battle_end().
11. Cleanup — close_appworld_environment() tears down the task's AppWorld environment.

Key technical points

• API scoping is LLM-based, not a static filter. predict_required_apis() shells out to a few-shot GPT predictor (model gpt-5-2025-08-07 in the current config) seeded with 3 demo tasks, which selects the ≤20 APIs a given task actually needs out of AppWorld's full surface. This predictor itself lives in AppWorld's companion appworld_agents package (external dependency, not vendored here) — this repo's code invokes it via subprocess against a local AppWorld installation.
• Telemetry is derived, not raw. analyze_mcp_trajectory() parses the raw mcp_tool_calls.jsonl log the MCP layer produces during a run and turns it into structured metrics: total/successful/failed calls, error rate, average call duration, calls-per-minute, unique tools used, retry count (failed call immediately followed by a successful retry of the same tool), and pagination sequences (runs of repeated list/search/library calls).
• Outcome and behavior are merged for scoring. get_complete_task_metrics() combines AppWorld's own correctness evaluation (unit-test pass/fail) with the trajectory metrics above into one record used to decide the battle winner and report results.

I don't have benchmark-run logs checked into this repo to cite a specific efficiency number (e.g. "X% faster evaluation"), so I'm intentionally not stating one here — if you've seen a specific figure attributed to this project elsewhere, treat it as anecdotal from a benchmark-time comparison run, not something backed by data in this repo.

Known limitations

• predict_required_apis() requires a local AppWorld checkout and conda environment containing the appworld_agents package (AppWorld's own example-agents repo, not part of this project). Paths are configurable via APPWORLD_PYTHON and APPWORLD_ROOT env vars (see below); they are not vendored or auto-installed by this repo.
• analyze_mcp_trajectory() reads from a local mcp_tool_calls.jsonl log path, configurable via AGENTBEATS_MCP_LOG_PATH.
• The Blue Agent is included for runnability but is a separate, external contribution — see the top of this README.

How to run it

1. Install dependencies

python -m venv venv && source venv/bin/activate
pip install -r requirements.txt   # installs the agentbeats SDK + requests

Install AppWorld separately, following its own setup instructions — this scenario talks to AppWorld over HTTP and does not vendor it.

2. Set environment variables

export OPENAI_API_KEY="your_api_key_here"

# Optional — only needed if your AppWorld checkout / conda env
# isn't at the defaults (~/appworld, ~/miniconda3/envs/appworld/bin/python):
export APPWORLD_ROOT="/path/to/appworld"
export APPWORLD_PYTHON="/path/to/conda/envs/appworld/bin/python"
export AGENTBEATS_MCP_LOG_PATH="/path/to/scenarios/appworld/logs/mcp_tool_calls.jsonl"

3. Start AppWorld services

appworld serve apis --port 9000
appworld serve environment --port 8000
appworld serve mcp http \
  --remote-apis-url http://localhost:9000 \
  --app-names supervisor,amazon,spotify,gmail,phone,venmo,splitwise,simple_note,todoist,file_system \
  --port 10000

4. Start the AgentBeats backend

agentbeats run_backend --backend_port 9002 --mcp_port 9001

5. Run the scenario

agentbeats run_scenario scenarios/appworld --backend http://localhost:9002

This launches both the Green Agent (orchestrator) and the Blue Agent (task executor) and runs a full battle: Blue attempts the AppWorld task, Green evaluates the result.

Runtime logs and trajectories are written to scenarios/appworld/logs/.

Repository structure

scenarios/appworld/
├── green_agent/                # my contribution
│   ├── orchestration_tools.py  # setup, API scoping, dispatch, polling, metrics, cleanup
│   ├── green_agent_card.toml   # agent config + 11-step orchestration plan
│   └── __init__.py
├── blue_agent/                 # agent-under-test — NOT my work, included for runnability
│   ├── tools.py
│   └── blue_agent_card.toml
├── scenario.toml                # wires the two agents together
└── logs/                        # runtime output (gitignored contents)

Attribution

Built on AgentBeats — an open platform for standardized, reproducible multi-agent evaluation. Install the SDK with pip install agentbeats; see agentbeats.org for the framework itself.

Evaluates against the AppWorld benchmark, an external project this scenario depends on but does not modify.