Files
zeekay 9baabd6022 Invalidate the token that failed, not a key nothing is stored under
The token cache is keyed on sha256(clientID:clientSecret@host). The reconciler
invalidated it with `InvalidateToken(host, "", "")`, which hashes to a key
nothing was ever stored under — so the delete matched nothing and the stale
entry survived. Every later reconcile reused a token the server had stopped
accepting.

Measured in production: 133 of 147 KMSSecrets failing, every read answering
403 "authentication required", for weeks. The credentials were never the
problem — logging in by hand with the same clientId and secret returns a token,
and a GET with that token returns 200. The operator was simply presenting a
dead one. The token's exp minus iat is 604800 seconds, and both operator pods
were seven days old.

The reconciler cannot pass the credentials because it does not have them:
AuthenticationDetails carries BearerToken and Host and deliberately not the
client secret, which is the right shape and should stay. So invalidation now
takes what the caller does hold — InvalidateBearer(host, token) drops whichever
entry issued that token.

Three tests. One pins that the old empty-credential call is a no-op, so the bug
cannot come back wearing its original face. One proves InvalidateBearer forces a
fresh login. One proves an unknown token evicts nothing, because an invalidation
that clears live entries is its own outage.

The existing InvalidateToken test passed throughout: it called the function with
real credentials while production called it with empty ones. A function proven
correct and a call site that never used it that way.
2026-08-20 17:44:42 -07:00
..