You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've sent a run of PRs over the past weeks and they probably look disconnected from the outside — a
file permission here, a cache key there, a log message somewhere else. They aren't. They came out of
one exercise, and I think the whole reads better than the parts, so this issue is the index and the
reasoning behind them.
Short version: we run Executor with real credentials, in a place where it matters to us if one
leaks. So I went looking, systematically, for every spot where a credential lives longer than it
needs to, travels further than it needs to, or ends up somewhere nobody meant it to go. Fourteen of the
PRs are what I found. The fifteenth (#1564) is a small capability change that came out of the same
look.
None of this is a criticism of the project. Most of what I found is the ordinary kind of thing that
accumulates in any codebase handling secrets — a value that was fine where it was written and became
a problem two hops away. I found them because I was specifically hunting for them, with a fairly
paranoid threat model, not because they were lying around obviously.
What I'd like from you
Putting this first, because the rest is context and you may not want all of it:
Review them individually, not as a batch. Each is independent and mergeable alone. If some are
wrong or unwanted, I'd genuinely rather they were closed than merged out of politeness.
If approving CI runs for the fork is easy on your side, that would help. None of these have had
checks run, because workflow runs from a fork need a maintainer's approval — so you're currently
being asked to take my word that the tests pass, which isn't a reasonable thing to ask. If there's a
reason not to, that's fine; I'd just stop expecting checks to appear.
Nothing here is urgent, and none of it is a report of something broken in production.
The direction behind it, briefly
Worth saying, because it explains the shape of the whole set. Where we're going is running
Executor with credentials held in a TEE — a hardware-isolated environment the host process
can't read into. What that buys is a credential the software can use without ever holding:
if the machine is compromised, what an attacker finds is ciphertext and a handle, not the key.
"We can't read it" is a much stronger thing to be able to say than "we promise not to look".
#1564 is the only PR that moves toward that directly — it lets a provider perform the OAuth
refresh itself, so the host never has to be handed the refresh token in order to spend it.
The other fourteen are the groundwork, and they matter more than they look. A sealed store is worth
very little while the same credential is also sitting in a cache key, an error message, a
world-readable file, or a browser's localStorage — a secret that never leaves the enclave through
the front door is no safer if it left through a log line an hour earlier. Most of what I found is
exactly that: not the credential store failing, but copies of the credential accumulating around
it.
That's also why I'd rather these were judged as ordinary fixes than as a strategy. Each one is
worth doing whether or not anyone ever puts a TEE behind it.
One thing worth adding, since it changes how much weight to put on the above: the TEE half is not
hypothetical. Executor's credential paths have been exercised against a real hardware-attested
enclave — GCP Confidential Space, production image, attestation verified against Google's live JWKS
rather than a fixture — and a few of these findings came out of running it there rather than from
reading. That is also why several of them are about what an operator can observe: in that setting
the interesting failure is not a credential leaking, it is a control that quietly did not engage
while everything still looked fine.
The question I was asking
For each credential Executor touches, I asked three things:
Where does it end up? Not where it's meant to go — where it actually ends up. Error
messages, cache keys, log lines, browser storage, module-level globals, files on disk.
How long does it stay? A secret that's correct to hold for one call is a different thing when
something keeps it for the lifetime of the process.
Who can read it once it's there? File modes, and what a person with access to the machine or
the logs would see.
That framing is worth stating because it explains why the PRs look scattered. They're scattered
because credentials are scattered; the question was the same every time.
Credit where it's due
A good part of the thinking behind this — particularly the idea that a credential should be usable
without ever being held, and that "we simply cannot read it" is a stronger promise than "we promise
not to look" — came out of conversations with @alexboone29. The framing is his. The bugs are
mine to have found and, where I got something wrong in a PR, mine to have got wrong.
The pull requests
A capability, and the only one that's a design question rather than a fix:
Let a credential provider own the OAuth refresh grant #1564 — let a credential provider perform the OAuth refresh exchange itself, instead of handing
the refresh token to Executor to spend. Optional; providers that don't implement it are completely
unaffected. This is the one that needs a direction from you, and the rest don't depend on it.
Credentials ending up somewhere they weren't meant to:
Stop reporting a rejected credential as a missing tool #1586 — when a connection's credential is rejected upstream it discovers no tools, and
asking for one reported that the tool did not exist. The reader goes looking for a renamed
or removed tool, which is the one thing that is not wrong.
Refuse an empty credential value instead of dialing with it #1587 — a credential input supplied as "" read as present, so the request went out
with an empty value and the upstream's 401 became the error the operator saw. The OpenAPI
backing already treated "" as missing, so this was an inconsistency rather than a choice.
What I'm not asking for
I'm not asking for these to be taken as a batch. Each one is independent and mergeable on its own,
and if some are wrong or unwanted I'd rather they were closed than merged out of politeness. #1564 is
the only one that's really a discussion.
One practical note: none of these have had CI run on them, because workflow runs from a fork need a
maintainer's approval. I'd rather you saw green checks than took my word that the tests pass, so if
approving runs for the fork is easy on your side it would help — and if there's a reason not to,
that's fine, I'd just stop expecting checks to appear.
Happy to split, rebase, or rework any of it. Thanks for building this — it's genuinely good software,
which is why we're using it somewhere that matters.
Why I opened fifteen pull requests
I've sent a run of PRs over the past weeks and they probably look disconnected from the outside — a
file permission here, a cache key there, a log message somewhere else. They aren't. They came out of
one exercise, and I think the whole reads better than the parts, so this issue is the index and the
reasoning behind them.
Short version: we run Executor with real credentials, in a place where it matters to us if one
leaks. So I went looking, systematically, for every spot where a credential lives longer than it
needs to, travels further than it needs to, or ends up somewhere nobody meant it to go. Fourteen of the
PRs are what I found. The fifteenth (#1564) is a small capability change that came out of the same
look.
None of this is a criticism of the project. Most of what I found is the ordinary kind of thing that
accumulates in any codebase handling secrets — a value that was fine where it was written and became
a problem two hops away. I found them because I was specifically hunting for them, with a fairly
paranoid threat model, not because they were lying around obviously.
What I'd like from you
Putting this first, because the rest is context and you may not want all of it:
wrong or unwanted, I'd genuinely rather they were closed than merged out of politeness.
checks run, because workflow runs from a fork need a maintainer's approval — so you're currently
being asked to take my word that the tests pass, which isn't a reasonable thing to ask. If there's a
reason not to, that's fine; I'd just stop expecting checks to appear.
Nothing here is urgent, and none of it is a report of something broken in production.
The direction behind it, briefly
Worth saying, because it explains the shape of the whole set. Where we're going is running
Executor with credentials held in a TEE — a hardware-isolated environment the host process
can't read into. What that buys is a credential the software can use without ever holding:
if the machine is compromised, what an attacker finds is ciphertext and a handle, not the key.
"We can't read it" is a much stronger thing to be able to say than "we promise not to look".
#1564 is the only PR that moves toward that directly — it lets a provider perform the OAuth
refresh itself, so the host never has to be handed the refresh token in order to spend it.
The other fourteen are the groundwork, and they matter more than they look. A sealed store is worth
very little while the same credential is also sitting in a cache key, an error message, a
world-readable file, or a browser's localStorage — a secret that never leaves the enclave through
the front door is no safer if it left through a log line an hour earlier. Most of what I found is
exactly that: not the credential store failing, but copies of the credential accumulating around
it.
That's also why I'd rather these were judged as ordinary fixes than as a strategy. Each one is
worth doing whether or not anyone ever puts a TEE behind it.
One thing worth adding, since it changes how much weight to put on the above: the TEE half is not
hypothetical. Executor's credential paths have been exercised against a real hardware-attested
enclave — GCP Confidential Space, production image, attestation verified against Google's live JWKS
rather than a fixture — and a few of these findings came out of running it there rather than from
reading. That is also why several of them are about what an operator can observe: in that setting
the interesting failure is not a credential leaking, it is a control that quietly did not engage
while everything still looked fine.
The question I was asking
For each credential Executor touches, I asked three things:
messages, cache keys, log lines, browser storage, module-level globals, files on disk.
something keeps it for the lifetime of the process.
the logs would see.
That framing is worth stating because it explains why the PRs look scattered. They're scattered
because credentials are scattered; the question was the same every time.
Credit where it's due
A good part of the thinking behind this — particularly the idea that a credential should be usable
without ever being held, and that "we simply cannot read it" is a stronger promise than "we promise
not to look" — came out of conversations with @alexboone29. The framing is his. The bugs are
mine to have found and, where I got something wrong in a PR, mine to have got wrong.
The pull requests
A capability, and the only one that's a design question rather than a fix:
the refresh token to Executor to spend. Optional; providers that don't implement it are completely
unaffected. This is the one that needs a direction from you, and the rest don't depend on it.
Credentials ending up somewhere they weren't meant to:
200from a token endpoint put an access token inside an error object,where anything logging that error would write the token out.
key is kept for the pool's lifetime rather than the call's.
each call. The smallest of the set.
the query string landed in the error message and went to the log on any transport failure.
localStorageas a fallback and never cleared it.Credentials outliving what they were for:
invisible, including the long-lived refresh token.
the database indefinitely.
session nobody dialled again stayed open and authenticated.
it, so a rollback could leave the row alive with its credential already deleted.
Files anyone on the machine could read:
~/.executor/server-connections.jsonheld a bearer token, or an OAuth access andrefresh token, and was created world-readable. Also revives four tests in that file that were never
actually running.
Telling the truth about state:
wrong answer that hides the other kind.
asking for one reported that the tool did not exist. The reader goes looking for a renamed
or removed tool, which is the one thing that is not wrong.
""read as present, so the request went outwith an empty value and the upstream's 401 became the error the operator saw. The OpenAPI
backing already treated
""as missing, so this was an inconsistency rather than a choice.What I'm not asking for
I'm not asking for these to be taken as a batch. Each one is independent and mergeable on its own,
and if some are wrong or unwanted I'd rather they were closed than merged out of politeness. #1564 is
the only one that's really a discussion.
One practical note: none of these have had CI run on them, because workflow runs from a fork need a
maintainer's approval. I'd rather you saw green checks than took my word that the tests pass, so if
approving runs for the fork is easy on your side it would help — and if there's a reason not to,
that's fine, I'd just stop expecting checks to appear.
Happy to split, rebase, or rework any of it. Thanks for building this — it's genuinely good software,
which is why we're using it somewhere that matters.