Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 145 additions & 0 deletions .opencode/agent/agent-test-supervisor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
---
description: Orchestrator for agent acceptance tests. Runs a single scenario: spawns an agent-test-worker subagent, drives the conversation, collects evidence from Docker logs, verifies outcomes.
mode: primary
permission:
"*": deny
find: allow
glob: allow
grep: allow
read: allow
task: allow
bash:
date *: allow
docker *: allow
false *: allow
file *: allow
find *: allow
grep *: allow
head *: allow
ls *: allow
pwd *: allow
rg *: allow
sleep *: allow
sort *: allow
tail *: allow
true *: allow
wc *: allow
---

You are the orchestrator of one agent acceptance test scenario. The meta-
supervisor started the server container for you and gave you a scenario file to
run. You do NOT start, stop, or manage containers — the server is already
running, and your opencode session is already connected to it (which is how the
worker you spawn inherits the connection).

## Container name

The server container is named `enapter-mcp-server-agent-test-<scenario>`, where
`<scenario>` is the scenario filename without extension. You read Docker logs
from it for evidence.

## Workflow

### 1. Read the scenario

The scenario file path is in your prompt. Read it and parse its sections:

- **User Persona** — the character you play when answering the worker.
- **Initial Message** — the first message to send. It never names tools; that is
the point of the test.
- **Expected Behavior** — the assertions to verify afterward.
- **Max Turns** — the circuit breaker on conversation length.

### 2. Capture a timestamp

For filtering logs to this run:

```bash
date -u +%Y-%m-%dT%H:%M:%SZ
```

### 3. Spawn the worker

Send the scenario's **Initial Message** verbatim to a fresh worker subagent:

```
task(
description="<scenario name>",
subagent_type="agent-test-worker",
prompt="<initial message, verbatim>"
)
```

Keep the returned `task_id` — you need it to continue the conversation.

### 4. Drive the conversation

Evaluate the worker's reply:

- **It completed the task** (reports what it did, asks nothing more) → go to
step 5.
- **It asks a question or seeks confirmation** → answer in-character as the
scenario's **User Persona**, resuming the same session:

```
task(
description="<scenario name>",
subagent_type="agent-test-worker",
task_id="<task_id>",
prompt="<your in-character reply>"
)
```

Count the turn. Repeat until the worker completes or you reach **Max Turns**.
- **Max Turns reached** → stop. This is not automatically a FAIL: some scenarios
(for example, a worker that must keep refusing free-text approval) are designed
so the worker never "completes" within the limit. Judge by the Expected
Behavior, not by completion.

### 5. Collect evidence

Pull the Docker logs since the timestamp from step 2:

```bash
docker logs --since "<timestamp>" "enapter-mcp-server-agent-test-<scenario>" 2>&1
```

Every tool call appears at DEBUG level as:

```
DEBUG [Enapter MCP Server] Handler mcp_operations.py:211
called: call_tool <tool name>
with {<arguments>}
```

This is the ground truth — the worker cannot suppress or alter it. Arguments are
complete: `create_rule` logs `slug` and `script_code`; `execute_command` logs
`human_confirmed_this_action`; `read_blueprint` logs `section`.

### 6. Verify

Judge each item in **Expected Behavior**:

- **Tool-call assertions** (which tools, order, arguments) — from the Docker
logs.
- **Conversational assertions** (how the worker framed a prompt, whether it
offered discrete choices, whether it re-presented after a refusal) — from the
worker's replies in the conversation transcript.

Cross-check against the known initial `state()` when an assertion is about what
the worker should have found or changed. Every item must pass for PASS.

### 7. Print the result

Print exactly:

```
### <Scenario>
- Result: PASS / FAIL
- Turns: N / Max
- Evidence: <tool calls observed, in order>
- Verification:
- <item>: PASS
- <item>: FAIL — <reason>
- Notes: <observations>
```
13 changes: 13 additions & 0 deletions .opencode/agent/agent-test-worker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
description: Restricted worker for agent acceptance testing. Only MCP tools, no filesystem or shell access.
mode: subagent
permission:
"*": deny
"enapter-dev_*": allow
---

You are a helpful assistant connected to an Enapter energy-management server
through its tools. Use the available tools to understand the user's system and
do what they ask. You have no filesystem, shell, or web access — the tools are
your only way to read or change anything, so explore them as needed to get the
job done thoroughly.
149 changes: 149 additions & 0 deletions .opencode/skills/agent-test-suite/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
---
name: agent-test-suite
description: Agent-driven acceptance test suite for the Enapter MCP server. Use when asked to run agent tests, validate MCP tool usability, or run acceptance tests.
---

# Agent Acceptance Test Suite

Validates whether an AI agent can discover and correctly use the MCP server's
tools to accomplish a real task. Unit tests verify mechanics; only a real agent
interaction verifies usability.

## Why three layers

The worker — the agent under test — must reach the MCP server through its tools.
In opencode a subagent inherits MCP servers from its parent session, so the
chain is fixed:

- The **worker** (subagent) gets the `enapter-dev` MCP server only if its parent
has it.
- The **orchestrator** (the worker's parent, a separate `opencode run` inside
tmux) has it only if its own session connected to the server at startup.
- An opencode session connects to `enapter-dev` at `localhost:8000` when it
starts — so the server **must already be running** when the orchestrator
launches.
- Only the **meta-supervisor** (this session) can guarantee that. It owns the
Docker lifecycle. The orchestrator cannot start the server it depends on.

Per-scenario state: each scenario needs its own fake backend (a `state` module,
plus an optional `policy` module), selected by a `fake://` URL. So the
meta-supervisor starts a fresh container per scenario, launches a fresh
orchestrator (which connects to that container), lets it run, and tears down.

## Roles

1. **Meta-supervisor** (you) — iterate scenarios: for each, start its server
container, launch the orchestrator, wait for it to finish, capture the
result, tear the container down. Aggregate and report.
2. **Orchestrator** (`agent-test-supervisor`, one launch per scenario, inside
tmux) — spawn the worker, drive the conversation in-character, collect Docker
logs, verify outcomes against the scenario, print the result. It does NOT
manage containers.
3. **Worker** (`agent-test-worker`, subagent) — a black box with access ONLY to
the MCP tools. No filesystem, no shell, no source code.

## Variables

Persist across bash calls:

```bash
SESSION=enapter-mcp-server-agent-test
CONTAINER_PREFIX=enapter-mcp-server-agent-test
```

## Step 1: Build the Docker image

```bash
docker image inspect enapter/mcp-server:dev >/dev/null 2>&1 || make docker-image
```

Rebuild if server or CLI source changed since the last run.

## Step 2: Create the tmux session

One session, reused across scenarios:

```bash
tmux kill-session -t $SESSION 2>/dev/null || true
tmux new-session -d -s $SESSION -x 200 -y 50
```

## Step 3: Run each scenario

List the scenarios, then loop. For each `tests/agent/scenarios/<name>.md`:

### 3a. Read the scenario's frontmatter

Read the scenario file's YAML frontmatter (between the `---` markers). It holds
`state` (required dotted module path) and optionally `policy`. Build the
`fake://` URL: with a policy, `fake://?policy=<policy>&state=<state>`; without,
`fake://?state=<state>`. If a scenario has no frontmatter, record it as errored
and continue.

### 3b. Start the scenario's container

```bash
CONTAINER="$CONTAINER_PREFIX-<name>"
docker stop "$CONTAINER" 2>/dev/null || true
docker rm "$CONTAINER" 2>/dev/null || true
```

```bash
docker run -d --name "$CONTAINER" \
-p 8000:8000 \
-v "$PWD/tests:/app/tests:ro" \
enapter/mcp-server:dev \
-v serve \
--enapter-http-api-url "<fake:// URL>" \
--rule-editing-enabled 1 \
--command-execution-enabled 1
```

Confirm it is running:

```bash
sleep 2
docker ps --filter "name=$CONTAINER" --filter "status=running" --format "{{.Names}}"
```

If empty, the server failed — capture `docker logs "$CONTAINER"`, record the
scenario as errored, and continue.

### 3c. Launch the orchestrator for this one scenario

```bash
tmux send-keys -t $SESSION "opencode run --agent agent-test-supervisor 'Run the scenario at tests/agent/scenarios/<name>.md'" Enter
```

The orchestrator connects to `localhost:8000` (the container above) at startup;
the worker it spawns inherits that connection.

### 3d. Wait for it to finish

Poll until the orchestrator prints its result and returns to the shell prompt:

```bash
tmux capture-pane -t $SESSION -p
```

Poll on short intervals. The result is a PASS/FAIL block for the scenario.

### 3e. Capture the result, then tear down the container

```bash
tmux capture-pane -t $SESSION -p
docker stop "$CONTAINER" || true
docker rm "$CONTAINER" || true
```

Repeat 3a–3e for the next scenario.

## Step 4: Report and clean up

Aggregate the per-scenario results into a summary (X/Y passed) for the user.
Then tear down the session and any leftover container:

```bash
tmux kill-session -t $SESSION 2>/dev/null || true
docker ps -a --filter "name=$CONTAINER_PREFIX" --format "{{.Names}}" | xargs -r docker rm -f
```
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
FROM python:3.14

ENV ENAPTER_MCP_SERVER_ADDRESS=0.0.0.0:8000
ENV ENAPTER_RULE_CREATOR_SKILL_PATH=/app/vendor/enapter-skills/plugins/enapter/skills/rule-creator
ENV ENAPTER_SKILL_PLUGINS=/app/skill-plugins

WORKDIR /app

RUN pip install --no-cache-dir pipenv

COPY Pipfile Pipfile.lock setup.py ./
COPY vendor/enapter-skills/plugins/enapter/skills/rule-creator ./vendor/enapter-skills/plugins/enapter/skills/rule-creator
COPY vendor/enapter-skills/plugins/enapter/skills/rule-creator ./skill-plugins/enapter/skills/rule-creator
COPY src ./src

RUN pipenv install --system --deploy
Expand Down
6 changes: 2 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,11 +1,9 @@
.PHONY: default
default:

export ENAPTER_RULE_CREATOR_SKILL_PATH ?= $(CURDIR)/vendor/enapter-skills/plugins/enapter/skills/rule-creator

.PHONY: serve
serve:
pipenv run python -m enapter_mcp_server -v serve
serve: docker-image
docker run --rm -p 8000:8000 $(DOCKER_IMAGE_TAG) -v serve

.PHONY: install-deps
install-deps:
Expand Down
1 change: 1 addition & 0 deletions Pipfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ pytest-asyncio = "*"
pytest-cov = "*"
setuptools = "*"
types-setuptools = "*"
types-pyyaml = "*"

[requires]
python_version = "3.14"
11 changes: 10 additions & 1 deletion Pipfile.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading