# RFC 0002: Refreshable secrets - Status: Implemented - Date: 2026-08-04 - Scope: vm-proxy, vm-cli config, vm-sdk public types ## Summary Let a secret be backed by a command instead of a host environment variable. The proxy runs that command to mint the real value, caches it in memory, and re-runs it as the value expires. The placeholder in the guest never changes, so a rotation is invisible inside the VM. This makes short lived credentials usable for long running sandbox work. The motivating case is a GitHub App installation token, which dies after one hour and currently strands any agent task that outlives it. It also narrows where substitution happens. Secrets are now substituted in the request head, meaning the request line and any header value, and no longer anywhere in the byte stream. Appendix A works through why. ## Motivation Secrets resolve from the host environment of the hanzo-vm process, which is frozen at exec. A VM that runs for ninety minutes against a sixty minute token fails with an auth error after most of the work is already done. The only workaround today is to split the task across VM boots using checkpoints. The alternative users reach for otherwise is a long lived personal access token, which is strictly worse: broader scope, no expiry, and a much larger blast radius if the host is compromised. Refreshing on the host side preserves the property the proxy exists to provide. Real credentials stay outside the VM, the guest holds a placeholder it cannot redeem anywhere except the bound hosts, and the guest cannot retain anything useful even in the window where it holds a live session. ## Non-goals - A control socket or `hanzo-vm secret set` command. That needs a per VM control plane, which does not exist and should not be motivated by secrets alone. - File watching. Secrets at rest plus partial write races, for no benefit over a command. - A CLI flag for command backed secrets. The `--secret NAME=ENV@hosts` syntax does not extend to argv without quoting hazards. Config file only. - Refreshing on an upstream 401. Expiry driven refresh covers the reported case; reactive refresh is future work. - Substituting inside request bodies. See appendix A. - Redacting secrets out of responses. See appendix B. ## Background Secrets flow through three points. `ProxyConfig.secrets` maps a guest visible env var name to a `SecretConfig` with `from` (host env var), `hosts` (domains where substitution is allowed), and an optional literal `value`. At proxy start, one random placeholder per secret is generated and handed to the guest as the value of that env var at exec time. At TLS connection setup the SNI is matched against the bound hosts, and the resulting placeholder to value pairs are applied to the guest to upstream stream. The value was already re-read per connection rather than cached at launch. It simply never changed, because the process environment cannot change. Mutating it is not an option either: `std::env::set_var` is unsafe in a multithreaded process under the 2024 edition, and the proxy runs on its own threads. Prior art converges on one answer for refresh. AWS `credential_process` runs a command that prints JSON with an `Expiration` field and re-runs it when the clock passes that time. Git credential helpers added `password_expiry_utc` in 2.41 for exactly the GitHub App case. Docker credential helpers pass the target host in on stdin but have no expiry concept, so they pay a subprocess spawn on every operation. This design takes the AWS shape, since the bound hosts are already known from config and there is nothing to pass in. ## Design ### Config surface `SecretEntry` gains `command` and `ttl`, and `from` becomes optional: ```json { "allow_net": true, "secrets": { "GITHUB_TOKEN": { "command": ["./scripts/mint-installation-token.sh"], "hosts": ["api.github.com", "github.com"] }, "API_KEY": { "from": "OPENAI_API_KEY", "hosts": ["api.openai.com"] } } } ``` Exactly one of `from` or `command` must be set. Both, or neither, is a config error reported at load time rather than at first request. `command` is an argv array, not a shell string. No shell sits between hanzo-vm and a credential, and there is no quoting hazard for paths with spaces. `ttl` is an optional duration string (`"45m"`, `"3600s"`) used only when the command reports no expiry of its own. ### Provider contract The command runs with stdin closed, inheriting the host environment of the hanzo-vm process, with the working directory set to the directory holding the resolved config file so relative paths behave the same wherever hanzo-vm is invoked from. On success it exits zero having written one JSON object to stdout: ```json { "version": 1, "value": "ghs_...", "expires_at": "2026-08-04T18:36:00Z" } ``` - `version` must be 1. Unknown versions are a resolve failure, which reserves room to change the shape later. - `value` is the credential, used verbatim. A trailing newline is trimmed. - `expires_at` is optional RFC3339. Present means refresh at that time. Absent falls back to `ttl`. Absent with no `ttl` means fetch once and never refresh, matching `credential_process`. A non-zero exit, unparseable stdout, an empty `value`, or a timeout is a resolve failure. The timeout is 10 seconds and the process is killed on expiry. Stderr is captured, truncated to 4 KB, and surfaced only in the resolve failure message. Stdout is never logged at any level, since it holds the credential, and `Minted` deliberately has no `Debug` implementation so a credential cannot reach a panic message. ### Resolution and caching `SecretResolver` owns what `ProxyConfig` cannot, since the engine holds the config as an immutable `Arc`: ```rust pub struct SecretResolver { config: Arc, placeholders: HashMap, cache: Mutex>, } ``` Env backed secrets keep reading `std::env::var` with no caching, which is free and preserves earlier behaviour exactly. Command backed secrets resolve against the cache. A cached entry with no expiry, or an expiry more than the refresh window away, is returned as is; otherwise the command runs and the result is stored. The refresh window is 60 seconds, so a value is re-minted shortly before it dies rather than at the first request that fails. Concurrent connections must not each spawn a process. The cache mutex is held across the fetch, so the first caller runs the command and the rest wait and observe the fresh entry. The mutex is per resolver rather than per secret, which serialises two different secrets in the rare case they lapse together. That is acceptable against a 10 second worst case and avoids a map of in-flight futures. ### Where substitution applies Substitution is scoped to the request head: the request line and every header value. It is scoped by position, not by header name, so any header carries a secret, including one this project has never heard of. A user writing `X-My-Vendor-Token: $API_KEY` needs no configuration and there is no provider table to maintain. The request line covers credentials passed as query parameters, such as the Google `?key=` form. Bodies stream through untouched. This is what makes the relay deterministic. At an arbitrary byte offset, "no more bytes yet" and "the request ended" are indistinguishable without a clock. A head that has not reached its terminating CRLF CRLF is unambiguously an unfinished request, so holding it is correct rather than a guess, and the upstream is waiting on those bytes regardless. Framing is read, never rewritten, and only to find where a body ends and the next head begins on a reused connection. Bodies are never modified, so no declared length is ever invalidated and nothing has to be recomputed. Framing follows the RFC 9112 section 6.3 algorithm, and anything that section calls out as a possible smuggling attempt is treated as ambiguous rather than resolved: both `Content-Length` and `Transfer-Encoding` present, conflicting `Content-Length` values, an unparseable head, `CONNECT`, or an `Upgrade`. Ambiguity stops substitution for the rest of the connection and degrades to a plain byte tunnel, which is the pre-existing behaviour rather than a guessed boundary. An intermediary that never re-frames cannot desynchronise its upstream by re-framing wrongly, so the smuggling exposure of a rewriting proxy does not arise. A head is bounded at 64 KiB. A larger one is released and the connection tunnelled, so nothing buffers without bound. This is also less work per byte than replacing over the whole stream, which scanned every byte of every body once per secret. Only heads are scanned now: roughly 200x less on an 80 KB upload, 2500x less on 1 MB. ### Refresh granularity Values resolve at TLS connection setup, and a live relay re-resolves every 5 seconds as it forwards, so a rotation reaches a connection that is already open rather than waiting for it to reconnect. Fresh values apply from the next request head. The relay can simply await the resolver, because it is already an async loop. An earlier draft assumed this needed a lock free read plus a background refresh task; it does not. While the cached value is live the re-resolve is a mutex and a map lookup, and a re-mint costs that connection what it would have cost at setup, bounded by the same provider timeout. ## Failure handling | Condition | Behaviour | | --- | --- | | Both or neither of `from` and `command` set | Config error at load, names the secret | | Command fails, no cached value | Fail the connection with a clear error | | Command fails, unexpired value cached | Serve the cached value, log under warn | | Command fails, cached value already expired | Fail the connection | | Env var missing (`from` secrets) | Skip substitution, unchanged behaviour | | Ambiguous or unparseable HTTP framing | Stop substituting, tunnel the connection | Command backed secrets fail closed. Forwarding a placeholder to an upstream produces a 401 that reads like a scope problem, and there is no reason to spend a request on a value known to be wrong. The failure is logged at warn with the provider stderr, since the guest only sees a dropped TLS connection. Env backed secrets keep the existing skip behaviour so nothing that works today starts failing. ## Security The credential still never enters the VM by way of the proxy. The placeholder is generated once per proxy and injected at exec, so a refresh changes only the host side substitution table and the guest observes nothing. Refreshing narrows the window. A guest that exfiltrates a live session gains nothing durable, because the value it was transiting expires on schedule and the replacement is minted outside the VM. Narrowing substitution to the head also removes a way for a guest to have the proxy write a credential into arbitrary data it controls: a placeholder appearing in a body is now left alone. New surface is one subprocess spawned by the host, from an argv array in a config file the user controls, with no guest input reaching it. A hostile `vm.json` can already run arbitrary code on `hanzo-vm run`, so this adds no capability that did not exist. Substitution is best effort against a cooperative guest and is not a containment boundary. A guest that base64s the placeholder, splits it across fields, or gzips the body defeats literal matching by construction. The host allowlist is what actually contains the credential. Appendix B records a related defect that is not addressed here. ## Testing Unit, in `secrets.rs`: expiry math and the refresh window, the `expires_at` over `ttl` over never precedence, provider output parsing including version mismatch and garbage stdout, single flight, fail closed and fallback to a live cached value, and that a parse error never quotes stdout. Unit, in `substitute.rs`: substitution in arbitrary header names and in the request line, at every chunk split point, bodies forwarded byte for byte, a second and third request on a reused connection, chunked bodies with extensions and trailers, and that each ambiguous framing case degrades to a tunnel with bytes released intact. Verified end to end against a live endpoint that echoes the request: - A length-changing secret substituted into a header, which previously hung when the same value sat in a body. - A 40 KB POST body forwarded untouched with the header substituted. - A placeholder in a body carried through verbatim, returning 200 rather than hanging. - Five requests on one connection, five substitutions, no placeholder leaked. - A rotation picked up mid-connection: 614 tokens took the first minted value and 386 the second, on one connection. ## Rollout Additive for configuration. Existing configs using `from` behave as before. `SecretConfig` is re-exported from vm-sdk, so the field changes are breaking for SDK consumers building it as a struct literal. It gains `Default`, with `SecretConfig::from_env` and `SecretConfig::from_command` as the constructors. vm-proxy goes to 0.3.0 and vm-sdk to 0.4.0. Behaviour change: a secret placed in a request body is no longer substituted. Appendix A shows this never worked for any real credential length, so no working configuration changes, but the failure mode moves from a hung request to a clean 401. ## Appendix A: why substitution is head scoped ### Body substitution never worked Byte substitution arrived with proxy based networking in c2b046f (v0.3.0, 2026-03-10) as a blind replace over the guest to upstream stream, commented only "Replace placeholder tokens with real values". No design named bodies as a target, and no documentation ever mentioned them. It is therefore a side effect rather than a feature, and it never functioned, because placeholders are a fixed 30 bytes and framing breaks the moment the replacement is any other length. Nothing real is 30 bytes: a GitHub token is 40, a legacy OpenAI key 51, an Anthropic key about 108, an AWS access key id 20 and its secret 40, a Stripe key about 107. Measured through a real VM, holding everything constant except the length of the minted value: | Request shape | Length equals placeholder | Length differs | | --- | --- | --- | | Secret in a header | 200 | 200 | | Body with `Content-Length` | 200 | fails | | Body with `Transfer-Encoding: chunked` | 200 | fails | | Body over 1 KB, so `Expect: 100-continue` | 200 | fails | Headers are immune because framing describes the body, not the head. Every body framing fails once a rewrite changes the byte count: with `Content-Length` the upstream waits for bytes that never arrive, and with chunked the emitted chunk size no longer matches its data. ### No comparable proxy offers it - Modal, in its credential-injection recipe, does not substitute at all. The proxy composes the header itself, via a Caddy `header_up x-api-key {env.ANTHROPIC_API_KEY}`. - Hermes iron-proxy substitutes "the proxy token wherever it appears in a matched location", and enumerates them: `Authorization`, `x-api-key`, `api-key`, `x-goog-api-key`, and the Google `?key=` form. Headers and query parameters, never bodies. The tools that do rewrite bodies are tokenization vaults such as VGS, where that is the entire product and it is done with structured payload operations over a fully parsed message. mitmproxy documents the underlying constraint plainly: with streaming enabled "the response body cannot be modified by the usual means", because "if the transfer encoding is not chunked, you cannot simply change the content length", and "it is recommended not to stream messages you need to modify". ### Options rejected - **A timeout on held-back fragments.** Shipped briefly. Any timer can be beaten by a slow enough stream, since "no bytes for N seconds" is not the same fact as "the request ended". Measured: at 50ms a 4 KB/s throttled upload released 6 of 1000 placeholders early; at 2s, none. Tuned to observed throughput rather than derived, and it does nothing for framing. - **Scaling that wait to fragment length.** Narrows both failure modes and is still guessing at the same missing fact. - **Re-framing everything to chunked.** Legal per RFC 9112 section 7.1.4, and it makes length changes free, but it rewrites requests that did not ask for it and breaks any upstream requiring an exact `Content-Length`, notably S3 style signed uploads. - **A buffering relay that rewrites `Content-Length`.** Rejected on cost. It introduces request smuggling exposure, which RFC 9112 section 11.2 describes as exploiting "differences in protocol parsing among various recipients"; it gives an untrusted guest a memory lever; it turns uploads into store and forward; and its over-cap path is incoherent, since a placeholder detected after earlier bytes are already upstream leaves a truncated request sent, which for a non-idempotent POST may have committed a side effect. Even done perfectly it still fails for signed and compressed bodies. All of that to preserve something that never worked. - **Length-matched placeholders**, generated to be exactly as long as the secret so substitution preserves framing by construction. Cheap and tempting, but entropy falls with the secret: 80 bits today, 40 against a 20 byte AWS key id, and under 10 bytes there is no room for the `hanzo_tok_` marker at all. A short placeholder starts colliding with ordinary request text, and a collision writes a real credential into unrelated data, which is worse than not substituting. Viable only with a length floor. ## Appendix B: a separate defect, not fixed here Substitution is one directional. The proxy replaces the placeholder with the real value on the way out and does nothing on the way back, so an upstream that reflects the request hands the credential to the guest: ``` guest env holds: hanzo_tok_18c8ae1dd13ae8680000 "authorization":"Bearer sk-live-REAL-CREDENTIAL-DO-NOT-LEAK" ``` The guest printed the real value. This weakens the claim that the real secret never enters the VM, with the precondition that a host the secret is bound to must reflect it, which a debug endpoint or a verbose error response can do. Scoping substitution to the head does not address it, since a reflected header carries the value just the same. The fix is response side redaction, replacing the real value with the placeholder on the way back, and it inherits the framing problem in the response direction. This wants its own issue and its own design. ## Future work - Reactive refresh on an upstream 401, covering providers that revoke early. - Policy driven header injection, where the config states that a host gets `Authorization: Bearer ` and the guest never holds a placeholder at all. It has none of the framing risk and none of the transformation limits, because the proxy composes the header rather than finding and rewriting something the guest wrote. Complementary to substitution rather than a replacement, since it does not cover query parameters or bodies. - A `hanzo-vm secret set` control command, once a general per VM control plane exists for port exposure and network policy changes.