Skip to content

fix: thread broker_token_issuer into the aggregator - #184

Merged
kratsg merged 1 commit into
mainfrom
fix/aggregator-broker-token-issuer
Aug 19, 2026
Merged

fix: thread broker_token_issuer into the aggregator#184
kratsg merged 1 commit into
mainfrom
fix/aggregator-broker-token-issuer

Conversation

@kratsg

@kratsg kratsg commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Symptom (root-caused live on the UChicago AF)

Every auth_type: x509 backend (ami-mcp) was permanently dropped from tools/list as "unavailable":

aggregator.backend_list_failed backend=ami-mcp reason=unavailable
error="Client error '401 Unauthorized' for url 'http://ami-mcp.mcp.svc.cluster.local/mcp'"

even though broker_token_issuer.loaded appeared at startup. An authorized tools/call fared no better — it raised the ToolError claiming "no signing key is configured (chart: broker.identityToken.existingSigningKeySecret)", which was misleading: the key WAS configured, loaded onto application.state.broker_token_issuer, and used successfully by the /v1 redeem endpoint.

Root cause

app.py's lifespan calls populate_aggregator(...) to push the freshly loaded registry/policy/settings into the eagerly built aggregator — but it omitted the broker_token_issuer kwarg. populate_aggregator threads that issuer into _make_client_factory, whose x509 branch mints the AF Broker Identity Token injected as Authorization: Bearer. With the kwarg omitted, the deployed aggregator's _x509_factory always saw broker_token_issuer is None:

  • tools/list: silently connects with no Authorization header → backend 401s → classified "unavailable", tools never listed.
  • authorized tools/call: raises the misleading "no signing key is configured" ToolError.

The fix is one line: pass broker_token_issuer=broker_token_issuer in the populate_aggregator(...) call.

Why tests missed it

Every existing x509-injection test constructed build_aggregator(...) directly, passing the issuer itself — nothing exercised the app lifespan's populate_aggregator wiring, so the gap between "issuer loaded onto app.state" and "issuer threaded into the aggregator's client factories" was uncovered.

Regression test added

test_lifespan_threads_issuer_into_x509_client_factory (broker/tests/test_broker_issued_app.py): boots the real app via app_client_factory with a signing key configured, fishes the shipped ami (x509) backend's provider out of the app's aggregator, invokes its client factory in a list-time context for an entitled principal, and asserts the resulting connection carries an Authorization: Bearer header that verifies against the exact issuer the lifespan loaded (aud=ami). Written TDD: it failed on main with "x509 list-time connection carried no identity token" and passes with the fix.

Full broker suite: 889 passed; ruff, mypy, and pre-commit clean.

🤖 Generated with Claude Code

The lifespan's populate_aggregator() call never passed the
broker_token_issuer it had just loaded, so the aggregator's x509 client
factories always saw None. In production every auth_type: x509 backend
(ami-mcp) then connected with no Authorization header at list time --
the backend 401'd and was dropped as "unavailable", its tools never
listed -- and an authorized tools/call raised a ToolError claiming no
signing key was configured, even though the key had loaded fine onto
app.state.broker_token_issuer and the /v1 redeem endpoint used it
without issue.

Existing tests constructed build_aggregator() directly with the issuer
passed, so this app-level wiring gap was uncovered. The new regression
test boots the real app with a signing key configured and asserts the
x509 client factory the lifespan wired up mints a list-time identity
header with the issuer the lifespan loaded.

Assisted-by: Claude (Anthropic)
@kratsg
kratsg merged commit 7b7aaea into main Aug 19, 2026
7 checks passed
@kratsg
kratsg deleted the fix/aggregator-broker-token-issuer branch August 19, 2026 06:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant