size the build's disk request to a node that can hold it

Every native build has been Pending since 2026-08-02 -- six of them, five days,
across three repos -- and nothing anywhere failed to say so. A runner-pool-32g
node allocates ~88Gi of ephemeral storage and the buildkitd DaemonSet reserves
48Gi on each, leaving 40Gi. The build asked for 50Gi, so the scheduler could not
place it on any node in the pool ("0/34 nodes are available ... 3 Insufficient
ephemeral-storage") and the autoscaler declined to help because no larger node
matched the selector either. hanzoai/iam published its last image at v1.34.26
and six tags after it produced nothing.

The request itself is right and stays: a best-effort pod is placed on a full
node and evicted first, which is the failure it was added for. It just has to be
a number the pool can satisfy. 32Gi leaves headroom inside the 40Gi a node has
free beside buildkitd, and is close to what cloud's own builds already request
and schedule with today. The 80Gi limit is untouched, so a large build still
bursts; only the reservation changes.

LLM.md

@@ -5363,3 +5363,83 @@ from under its owner costs their work.
 ours names this pod", so a second cloud pointed at the same `SANDBOX_NAMESPACE`
 would read the first one's live sandboxes as orphans. Same invariant the stores
 already have; stated where breaking it deletes something.
+
+## Metasearch: an engine that says nothing is either empty or BLIND
+
+The browser service is REAL and the code said it was not. `apps/crawl` opened
+with "that service was named in config but did not exist —
+crawl.hanzo.svc.cluster.local was NXDOMAIN", and browser.go promised escalation
+was best-effort "while the browser is not deployed, which is the state this ships
+in". Measured: `svc/crawl` 10.124.54.223:11235, pod Running, `/health` 200,
+image `ghcr.io/hanzoai/crawl:sha-7b8dc59`. It has been up for days. It needs the
+bearer token that reaches both pods from KMS as `crawl-secrets/CRAWL_API_TOKEN` —
+unauthenticated it answers `{"detail":"Authentication required"}`, so a missing
+token is not a degraded render, it is no render.
+
+**The defect that hid everything else: zero results was not a value.**
+`fetchEngine` returned `([]webResult, error)` and a bot-challenge page came back
+`(nil, nil)` — the same value as a query the web has no answer for. An engine
+could stop working entirely and the only symptom was a slightly shorter page.
+That is how Brave was dropped rather than fixed and how DDG sat in the default
+set contributing nothing. `outcome.go` gives an engine's turn three states:
+`answered`, `blind` (the fetch succeeded and the parser read NOTHING), `failed`
+(never reached, so it says nothing about the parser).
+
+**Zero is blind per engine, always, because no engine can be trusted to report
+its own emptiness.** Bing has no zero state at all: asked three distinct nonsense
+strings it returned ten results each time — Edmonton property tax, Bastille Day,
+Microsoft support — and for a fourth, pornography. DDG and Mojeek do return zero,
+but also return zero when serving a captcha, which is the case worth catching.
+The genuine-empty case is recovered where its evidence actually lives, ACROSS
+engines: blind while a sibling answered the same query is proof the query has
+results and that engine cannot see them.
+
+Two instruments, deliberately different widths. The counter
+`hanzo_websearch_engine_total{engine,outcome}` records every turn (nine series,
+bounded by construction — the query is never an attribute); an operator reads the
+RATIO. The Warn line fires only on the confirmed fault, so it stays worth
+reading. `browsed` rides along because it decides who is woken: false is
+configuration, true means a real browser drew the page and our parser still read
+nothing — selector rot, the loudest signal this package has.
+
+**DDG's challenge is an HTTP 202, and that one fact cost us the engine.** The
+fetch accepted only 200, so the challenge became a transport error, the transport
+error short-circuited past the escalation, and the engine the browser was
+deployed to rescue was the one that could never reach it — `ddg=failed` on every
+live query. Any 2xx is now parsed, and there is ONE remedy for "nothing readable
+came back" whichever way it happened: render it. Live, from cluster egress, that
+turned `[bing=answered(10) ddg=failed(0) mojeek=answered(20)]` into all three
+answering, and pushed `post.ca.gov/Training` off the top of "post quantum
+cryptography lattice" (now redhat's lattice-based-cryptography, then Wikipedia).
+
+**GOOGLE IS NOT VIABLE FROM THIS NETWORK. Do not add it.** Every path returns the
+`/sorry/` interstitial — ~6KB, 19 captcha markers, "unusual traffic", zero
+results: headless Crawl with stealth on, `&udm=14`, `&gbv=1`, and a HEADFUL
+Chrome 150 on a real X display (`bot-browser`) from a second egress IP. The
+control rules out the technique — that same headful browser reads DDG's ten
+results and loads google.com's homepage normally (268KB, title "Google", no
+captcha). Only `/search` is refused, from two different node IPs, headless and
+headful alike. That is reputation attached to datacenter addresses, so no browser
+flag reaches it; the fix would be residential egress, which is a different
+decision than a parser.
+
+**crawl, not bot-browser, is the instrument.** They measured IDENTICALLY on both
+engines, so bot-browser buys nothing on capability and costs concurrency (one
+shared headful Chrome behind a single CDP endpoint), tenancy (one cookie jar for
+every caller) and a websocket client cloud does not have. bot-browser is an
+interactive instrument — VNC 5900 / noVNC 6080 exist so a person can watch a bot
+session — and driving search through it would serialize every query through one
+browser.
+
+Costs, measured from cluster egress: a browser render of a DDG result page is
+**~1.13s** (1119/1137/1160ms, 10 results each), against 400ms–1.4s for a whole
+static three-engine blend. Escalation is worth paying on zero and not before,
+which is what render.go already did. A REFUSAL is still worth a render because
+the browser leaves the cluster from a different node than cloud does — but it
+will not rescue a refusal aimed at the browser's own address (asked to render a
+URL that had just answered it 403, Crawl returned 0 bytes).
+
+Watch for `WEBSEARCH_ENGINES` in universe: it is set to `bing,ddg`, which
+excludes Mojeek — an independent index and, with DDG, one of the two engines that
+honours `site:` (`site:x.com openai` → 20 real x.com URLs on Mojeek, 10 on DDG,
+`blind` on Bing). The in-code default is now all three.

apps/agents/agents.go

@@ -919,9 +919,18 @@ func runAgent(s *cloud.Service[state], ctx context.Context, a Agent, input, acto
 	// row and on the money, which is the whole of "drill into this run".
 	id, _ := genID("run")
 
+	// hanzo.org, not a name of this package's own, because the TRACE PLANE reads
+	// exactly this key: apps/o11y/planesink.go planeOrg files each row under
+	// attrs["hanzo.org"] and falls back to the PLATFORM's org when it is absent.
+	// It is a per-span read — OTel children do not inherit a parent's attributes —
+	// so the correctly stamped HTTP span above this one buys the run nothing. A
+	// run that named its tenant in a private spelling was stored as the platform's
+	// own telemetry: invisible to the org-scoped read the console issues, and
+	// sitting in the platform's bucket with this tenant's tool names and users in
+	// it. One tenant attribute, the one the plane already reads.
 	span.SetAttributes(
 		attribute.String("hanzo.agent.name", a.Name),
-		attribute.String("hanzo.agent.org", a.Org),
+		attribute.String("hanzo.org", a.Org),
 		attribute.String("gen_ai.request.model", a.Model),
 		attribute.String("hanzo.agent.run_id", id),
 	)
@@ -1042,6 +1051,11 @@ func executeRun(ctx context.Context, ai types.AIClient, org, actor string, a Age
 	span.SetAttributes(
 		attribute.String("gen_ai.request.model", a.Model),
 		attribute.String("hanzo.agent.run_id", runID),
+		// The tenant, on this span too. A step that named no org was filed under
+		// the platform's, which put the middle of every run's waterfall in a
+		// bucket the tenant cannot read — the run above it and the tool calls
+		// below it were visible and the step joining them was not.
+		attribute.String("hanzo.org", org),
 	)
 
 	prompt := a.Instructions
@@ -1122,9 +1136,23 @@ func completeWithFailover(ctx context.Context, ai types.AIClient, req *types.Cha
 		models = append(models, f)
 	}
 	var lastErr error
-	for _, m := range models {
+	for i, m := range models {
 		req.Model = m
-		resp, err := completeWithRetry(ctx, ai, req)
+		resp, attempts, err := completeWithRetry(ctx, ai, req)
+		// A model call that did not succeed first time, said out loud. Retries and
+		// failovers were previously invisible AS SUCH: each attempt produced its own
+		// chat span, so three attempts looked like three unrelated calls and the
+		// switch to the reliable model looked like an agent that had simply asked
+		// for a different one. The waterfall showed the cost and never the reason.
+		//
+		// It is an EVENT, not an attribute, because completeWithFailover runs once
+		// per ROUND of the tool loop — an attribute would be overwritten by every
+		// later round and the span would report only the last one, while events
+		// accumulate. And nothing at all is recorded for the ordinary case, so the
+		// presence of one of these always means something happened.
+		if attempts > 1 || i > 0 {
+			noteRetry(ctx, m, attempts, i > 0, err)
+		}
 		if err == nil {
 			return resp, m, nil
 		}
@@ -1141,25 +1169,48 @@ func completeWithFailover(ctx context.Context, ai types.AIClient, req *types.Cha
 // completeWithRetry calls the completion up to maxAttempts times, retrying ONLY a
 // transient upstream overload (types.ErrUpstreamBusy) with jittered backoff and
 // respecting context cancellation. A non-transient error returns immediately.
-func completeWithRetry(ctx context.Context, ai types.AIClient, req *types.ChatRequest) (*types.ChatResponse, error) {
+// It returns how many attempts it MADE alongside the outcome, so the caller can
+// record a retry as a retry. Counting inside is the only place the number is
+// known — from outside, three attempts and three unrelated calls look identical.
+func completeWithRetry(ctx context.Context, ai types.AIClient, req *types.ChatRequest) (*types.ChatResponse, int, error) {
 	var lastErr error
 	for attempt := 0; attempt < maxAttempts; attempt++ {
 		resp, err := ai.ChatCompletion(ctx, req)
 		if err == nil {
-			return resp, nil
+			return resp, attempt + 1, nil
 		}
 		lastErr = err
 		if !errors.Is(err, types.ErrUpstreamBusy) {
-			return nil, err // permanent — do not burn retries repeating it
+			return nil, attempt + 1, err // permanent — do not burn retries repeating it
 		}
 		if attempt == maxAttempts-1 {
 			break
 		}
 		if err := sleepBackoff(ctx, attempt); err != nil {
-			return nil, err // context cancelled/expired mid-backoff
+			return nil, attempt + 1, err // context cancelled/expired mid-backoff
 		}
 	}
-	return nil, lastErr
+	return nil, maxAttempts, lastErr
+}
+
+// noteRetry records one model call that needed more than a first attempt, on
+// whichever span is current — the step for a plain run, the same step for every
+// round of a tool loop.
+//
+// The reason is carried when there is one: "it retried three times" and "it
+// retried three times because the gateway kept answering 429" are different
+// facts, and only the second tells an operator whether to look at us or at the
+// upstream.
+func noteRetry(ctx context.Context, model string, attempts int, failover bool, err error) {
+	attrs := []attribute.KeyValue{
+		attribute.String("gen_ai.request.model", model),
+		attribute.Int("hanzo.agent.model_attempts", attempts),
+		attribute.Bool("hanzo.agent.failover", failover),
+	}
+	if err != nil {
+		attrs = append(attrs, attribute.String("hanzo.agent.retry_reason", err.Error()))
+	}
+	trace.SpanFromContext(ctx).AddEvent("model retry", trace.WithAttributes(attrs...))
 }
 
 // sleepBackoff waits an exponential, equal-jittered delay before the next

apps/agents/observability_test.go

@@ -24,6 +24,7 @@ import (
 	"sync"
 	"sync/atomic"
 	"testing"
+	"unicode/utf8"
 
 	"github.com/hanzoai/cloud/clients"
 	"github.com/hanzoai/cloud/types"
@@ -134,6 +135,9 @@ type stubPlane struct {
 	offer  []string
 	fail   map[string]bool
 	called []string
+	// result is what a named tool returns; anything unnamed returns a default so
+	// the common case stays a one-line fixture.
+	result map[string]string
 }
 
 func (p *stubPlane) catalog(context.Context, string, string, []string) []types.ToolDef {
@@ -151,9 +155,34 @@ func (p *stubPlane) call(_ context.Context, _, _, name, _ string) (string, error
 	if p.fail[name] {
 		return "", fmt.Errorf("upstream refused %s", name)
 	}
+	if out, ok := p.result[name]; ok {
+		return out, nil
+	}
 	return "result of " + name, nil
 }
 
+// toolGatewayArgs answers with ONE tool call carrying the given arguments
+// verbatim, then a final answer — so a test can pin what a dispatch records about
+// arguments it did not choose.
+func toolGatewayArgs(t *testing.T, name, args string) *httptest.Server {
+	t.Helper()
+	return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		body, _ := io.ReadAll(r.Body)
+		w.Header().Set("Content-Type", "application/json")
+		if strings.Contains(string(body), `"role":"tool"`) || !strings.Contains(string(body), `"tools":[`) {
+			fmt.Fprint(w, `{"id":"c2","model":"gpt-4o-mini","choices":[{"index":0,"finish_reason":"stop",`+
+				`"message":{"role":"assistant","content":"the answer"}}],`+
+				`"usage":{"prompt_tokens":40,"completion_tokens":9,"total_tokens":49}}`)
+			return
+		}
+		call, _ := json.Marshal(args) // the arguments ride as a JSON *string*, as the wire has them
+		fmt.Fprintf(w, `{"id":"c1","model":"gpt-4o-mini","choices":[{"index":0,"finish_reason":"tool_calls",`+
+			`"message":{"role":"assistant","tool_calls":[`+
+			`{"id":"tc0","type":"function","function":{"name":%q,"arguments":%s}}]}}],`+
+			`"usage":{"prompt_tokens":11,"completion_tokens":22,"total_tokens":33}}`, name, call)
+	}))
+}
+
 // toolGateway answers the REQUEST rather than a call counter: a turn already
 // holding tool results gets the final answer; a turn offered tools asks for all of
 // them at once. Answering a counter makes the fixture depend on whoever happened
@@ -251,7 +280,7 @@ func TestOneRunIsObservableEndToEnd(t *testing.T) {
 	if got := attr(root, "hanzo.user"); got != "u-acme" {
 		t.Fatalf("run span must name the user, got %q", got)
 	}
-	if got := attr(root, "hanzo.agent.org"); got != "acme" {
+	if got := attr(root, "hanzo.org"); got != "acme" {
 		t.Fatalf("run span must name the org, got %q", got)
 	}
 
@@ -451,3 +480,202 @@ func TestToolSubsystemReadsTheNameNotAnIndex(t *testing.T) {
 		}
 	}
 }
+
+// TestToolCallRecordsWhatItDidWithoutItsCredential is the "exactly what steps it
+// took" contract, and the one place where getting observability right and getting
+// security right are the same edit.
+//
+// A dispatch used to record WHICH tool ran and nothing about what it ran WITH, so
+// a trace could say an agent called post_v1_exec_run six times and never say what
+// it executed. Recording the arguments closes that. But a tool argument routinely
+// carries a live credential — a clone URL with a token in its userinfo is the
+// ordinary shape apps/coding builds — and a span store is built to be queried and
+// kept, so a trace that records one is worse than no trace.
+//
+// Both halves are asserted here together, because either alone is a bug: capture
+// with no redaction leaks, redaction with no capture is the silence we started
+// from.
+func TestToolCallRecordsWhatItDidWithoutItsCredential(t *testing.T) {
+	sink := traced(t)
+
+	const token = "ghp_LIVE_TOKEN_VALUE"
+	const args = `{"cloneUrl":"https://x-access-token:` + token + `@github.com/acme/repo","branch":"main"}`
+
+	// The tool hands back a credential too — a result is as capable of carrying one
+	// as an argument, and it is recorded on the same span.
+	plane := &stubPlane{offer: []string{"post_v1_exec_run"}, result: map[string]string{
+		"post_v1_exec_run": `{"ok":true,"remote":"https://deploy:s3cr3t@git.example.com/x"}`,
+	}}
+	old := runTools
+	runTools = plane
+	t.Cleanup(func() { runTools = old })
+
+	gw := toolGatewayArgs(t, "post_v1_exec_run", args)
+	defer gw.Close()
+
+	app := mountApp(t, clients.AIHTTPAt(gw.URL+"/v1", "k", "gpt-4o-mini"))
+	do(t, app, http.MethodPost, "/v1/agents", "acme", map[string]any{
+		"name": "a", "model": "gpt-4o-mini", "instructions": "x", "tools": []string{"post_v1_exec_run"}})
+	if code, body := do(t, app, http.MethodPost, "/v1/agents/a/run", "acme", map[string]any{"input": "go"}); code != http.StatusOK {
+		t.Fatalf("run want 200, got %d (%s)", code, body)
+	}
+
+	sp, ok := sink.find("agent.tool post_v1_exec_run")
+	if !ok {
+		t.Fatal("the dispatch produced no span")
+	}
+
+	// CAPTURED: the call is legible as a call.
+	gotArgs := attr(sp, "gen_ai.tool.call.arguments")
+	if gotArgs == "" {
+		t.Fatal("the dispatch recorded no arguments — the trace cannot say what the tool was called with")
+	}
+	if !strings.Contains(gotArgs, "github.com/acme/repo") || !strings.Contains(gotArgs, "main") {
+		t.Fatalf("the recorded arguments lost the detail that makes them worth reading: %s", gotArgs)
+	}
+	gotResult := attr(sp, "gen_ai.tool.call.result")
+	if gotResult == "" {
+		t.Fatal("the dispatch recorded no result — the trace cannot say what came back")
+	}
+
+	// REDACTED: no credential reached the span, from either half.
+	for _, secret := range []string{token, "s3cr3t"} {
+		for _, where := range []struct{ name, v string }{{"arguments", gotArgs}, {"result", gotResult}} {
+			if strings.Contains(where.v, secret) {
+				t.Fatalf("a live credential reached the span in %s: %s", where.name, where.v)
+			}
+		}
+	}
+	// The non-secret half of the userinfo survives — it names HOW the call
+	// authenticated, which is worth reading and is not the secret.
+	if !strings.Contains(gotArgs, "x-access-token") {
+		t.Fatalf("redaction removed the authenticating user, not just its secret: %s", gotArgs)
+	}
+}
+
+// TestRecordedValueIsCutAndSaysSo: a tool argument larger than a span records is
+// bounded, and the value itself states that it was — a silently clipped argument
+// reads as the whole one, which is how an operator concludes a tool was called
+// with something it never saw.
+func TestRecordedValueIsCutAndSaysSo(t *testing.T) {
+	big := `{"blob":"` + strings.Repeat("x", maxRecorded*2) + `"}`
+	got := recordable(big)
+	if len(got) > maxRecorded+64 {
+		t.Fatalf("recorded value is %d bytes, want it bounded near %d", len(got), maxRecorded)
+	}
+	if !strings.Contains(got, "cut") {
+		t.Fatalf("a cut value must say it was cut, got tail %q", got[max(0, len(got)-40):])
+	}
+	if !utf8.ValidString(got) {
+		t.Fatal("cutting produced invalid UTF-8")
+	}
+	// A value that fits is returned whole, with no marker inviting a reader to
+	// wonder what is missing.
+	if small := recordable(`{"a":"b"}`); small != `{"a":"b"}` {
+		t.Fatalf("a value within the bound must pass through unchanged, got %q", small)
+	}
+}
+
+// TestEverySpanOfARunIsFiledUnderItsTenant is the one that decides whether any of
+// the rest is visible.
+//
+// The trace plane files each row under ONE attribute — hanzo.org — read per span
+// by apps/o11y/planesink.go planeOrg, which falls back to the PLATFORM's own org
+// ("hanzo") when it is absent. OTel children do not inherit a parent's
+// attributes, so a correctly stamped HTTP server span above the run buys its
+// children nothing: every span states its own tenant or is filed under the
+// platform.
+//
+// A run whose spans miss it fails twice over. The tenant opens the console and
+// sees a trace with a hole where its agent was, because the org-scoped read never
+// returns those rows; and the tenant's tool names, run ids and user subjects sit
+// in the platform's bucket instead. That is why this asserts on EVERY exported
+// span rather than on the root: one unstamped span is one missing branch of the
+// waterfall.
+func TestEverySpanOfARunIsFiledUnderItsTenant(t *testing.T) {
+	sink := traced(t)
+
+	plane := &stubPlane{offer: []string{"post_v1_search_query"}}
+	old := runTools
+	runTools = plane
+	t.Cleanup(func() { runTools = old })
+
+	gw := toolGateway(t, []string{"post_v1_search_query"})
+	defer gw.Close()
+
+	app := mountApp(t, clients.AIHTTPAt(gw.URL+"/v1", "k", "gpt-4o-mini"))
+	do(t, app, http.MethodPost, "/v1/agents", "acme", map[string]any{
+		"name": "a", "model": "gpt-4o-mini", "instructions": "x", "tools": []string{"post_v1_search_query"}})
+	if code, body := do(t, app, http.MethodPost, "/v1/agents/a/run", "acme", map[string]any{"input": "hi"}); code != http.StatusOK {
+		t.Fatalf("run want 200, got %d (%s)", code, body)
+	}
+
+	// The spans a run IS. Named explicitly so a span that stops being produced
+	// fails here rather than silently shrinking the set under assertion.
+	want := []string{"agent.run a", "agent.step", "agent.tool post_v1_search_query", "chat gpt-4o-mini"}
+	for _, name := range want {
+		spans := sink.all(name)
+		if len(spans) == 0 {
+			t.Fatalf("no %q span was exported", name)
+		}
+		for _, sp := range spans {
+			if got := attr(sp, "hanzo.org"); got != "acme" {
+				t.Fatalf("%s carries hanzo.org=%q, want \"acme\" — the plane files this span "+
+					"under the platform org, so the tenant cannot see its own run", name, got)
+			}
+		}
+	}
+}
+
+// TestOneOrgsRunNeverCarriesAnothersTenant proves the org stamp PARTITIONS the
+// spans: two tenants running the same agent name against the same process produce
+// two disjoint sets of rows, and neither names the other.
+//
+// Tenancy on the trace plane is exactly this attribute — the org-scoped read binds
+// it as its first predicate — so a run that stamped the wrong tenant, or stamped
+// none and fell back to the platform, would disclose one org's tool calls and user
+// subjects to a reader scoped to another. The gate is fail-closed downstream; this
+// asserts the value it closes on is right at the source.
+func TestOneOrgsRunNeverCarriesAnothersTenant(t *testing.T) {
+	sink := traced(t)
+
+	plane := &stubPlane{offer: []string{"post_v1_search_query"}}
+	old := runTools
+	runTools = plane
+	t.Cleanup(func() { runTools = old })
+
+	gw := toolGateway(t, []string{"post_v1_search_query"})
+	defer gw.Close()
+
+	app := mountApp(t, clients.AIHTTPAt(gw.URL+"/v1", "k", "gpt-4o-mini"))
+	for _, org := range []string{"acme", "globex"} {
+		do(t, app, http.MethodPost, "/v1/agents", org, map[string]any{
+			"name": "a", "model": "gpt-4o-mini", "instructions": "x", "tools": []string{"post_v1_search_query"}})
+		if code, body := do(t, app, http.MethodPost, "/v1/agents/a/run", org, map[string]any{"input": "hi"}); code != http.StatusOK {
+			t.Fatalf("%s run want 200, got %d (%s)", org, code, body)
+		}
+	}
+
+	// Every span belongs to exactly one of the two tenants, and the user subject it
+	// names is that tenant's own. A span filed under the platform default is
+	// counted as neither and fails the total below.
+	seen := map[string]int{}
+	for _, name := range []string{"agent.run a", "agent.step", "agent.tool post_v1_search_query", "chat gpt-4o-mini"} {
+		for _, sp := range sink.all(name) {
+			org := attr(sp, "hanzo.org")
+			seen[org]++
+			if org != "acme" && org != "globex" {
+				t.Fatalf("%s is filed under %q — neither tenant ran it", name, org)
+			}
+			// The person is scoped to the same tenant. "acme" carrying globex's
+			// subject would be a cross-tenant disclosure inside a correctly
+			// stamped row, which the org assertion alone cannot catch.
+			if sub := attr(sp, "hanzo.user"); sub != "" && sub != "u-"+org {
+				t.Fatalf("%s is filed under org %q but names user %q", name, org, sub)
+			}
+		}
+	}
+	if seen["acme"] == 0 || seen["globex"] == 0 {
+		t.Fatalf("want spans from both tenants, got acme=%d globex=%d", seen["acme"], seen["globex"])
+	}
+}

apps/agents/tools.go

@@ -56,8 +56,10 @@ import (
 	"fmt"
 	"strings"
 	"time"
+	"unicode/utf8"
 
 	"github.com/hanzoai/cloud/apps/tools"
+	"github.com/hanzoai/cloud/audit"
 	"github.com/hanzoai/cloud/types"
 	"go.opentelemetry.io/otel/attribute"
 	"go.opentelemetry.io/otel/codes"
@@ -85,6 +87,14 @@ const (
 	// maxToolArgs bounds the arguments a model may emit for one call, before they
 	// are ever parsed.
 	maxToolArgs = 32 * 1024
+	// maxRecorded bounds what ONE tool call may put on its span — its arguments or
+	// its result. A trace is for READING, not for replay, and this is deliberately
+	// far tighter than maxToolResult: that bound is paid once as prompt tokens,
+	// this one is stored for every call of every run and kept for the life of the
+	// trace. 4 KiB is a page of evidence. What exceeds it is cut and SAID to be
+	// cut — a silently clipped value reads as the whole argument, which is how an
+	// operator concludes a tool was called with something it never saw.
+	maxRecorded = 4 * 1024
 	// maxAgentDepth bounds how deep AGENTS may nest, which is a different bound
 	// from maxToolRounds and is not covered by it: an agent is itself a tool
 	// (agentToolProvider, agents.go:399), so A calling B calling A is a cycle in
@@ -243,6 +253,33 @@ func actorSub(org, actor string) string {
 	return actor
 }
 
+// recordable is what a tool call may put on a span: credentials stripped, size
+// bounded, and the bound stated.
+//
+// Both halves are load-bearing. A tool ARGUMENT routinely carries a live
+// credential — a clone URL with a token in the userinfo is the ordinary shape
+// (apps/coding builds exactly that) — and a trace that records one is worse than
+// no trace at all, because the token outlives the sandbox that used it and sits
+// in a store built to be queried. audit.RedactText is the fleet's ONE redactor;
+// this adds no second policy, it just applies it at the moment of recording.
+//
+// The order matters: redact FIRST, then cut. Cutting first can split a credential
+// and leave the front half of it on the span, which is both a leak and unreadable.
+func recordable(s string) string {
+	s = audit.RedactText(s)
+	if len(s) <= maxRecorded {
+		return s
+	}
+	// Cut on a rune boundary so the value stays valid UTF-8; a span store that
+	// rejects or mangles invalid UTF-8 would lose the whole attribute over the
+	// last byte of a multi-byte character.
+	cut := maxRecorded
+	for cut > 0 && !utf8.RuneStart(s[cut]) {
+		cut--
+	}
+	return s[:cut] + "\u2026[cut: longer than a span records]"
+}
+
 // renderToolResult turns whatever a tool returned into the text the model reads.
 // A string is already text; anything else is its JSON, which is the shape the
 // tool's own schema describes. A value that will not marshal is reported as that
@@ -346,10 +383,18 @@ func dispatchOne(ctx context.Context, org, actor string, tc types.ToolCall, runI
 	span.SetAttributes(
 		attribute.String("gen_ai.tool.name", tc.Name),
 		attribute.String("gen_ai.tool.call.id", tc.ID),
-		attribute.String("hanzo.agent.org", org),
+		// The tenant under the key the trace plane files rows by (planeOrg,
+		// apps/o11y/planesink.go) — see the note on the run span in agents.go.
+		attribute.String("hanzo.org", org),
 		attribute.String("hanzo.agent.run_id", runID),
 		attribute.String("hanzo.agent.tool_subsystem", toolSubsystem(tc.Name)),
 		attribute.Int("hanzo.agent.tool_round", round),
+		// WHAT it was called with. Without this a trace says an agent called
+		// "post_v1_exec_run" six times and cannot say what it ran — which is the
+		// difference between knowing a run touched a tool and knowing what it did.
+		// The convention's own name for this (opt-in precisely because it can carry
+		// user data), redacted and bounded by recordable.
+		attribute.String("gen_ai.tool.call.arguments", recordable(tc.Arguments)),
 	)
 	if sub := actorSub(org, actor); sub != "" {
 		span.SetAttributes(attribute.String("hanzo.user", sub))
@@ -376,6 +421,10 @@ func dispatchOne(ctx context.Context, org, actor string, tc types.ToolCall, runI
 		span.SetStatus(codes.Error, "tool call failed")
 		return "error: " + err.Error()
 	}
+	// WHAT came back — the other half of the dispatch, and the half that explains
+	// what the model did next. A tool that "succeeded" while returning an error
+	// document is invisible without it.
+	span.SetAttributes(attribute.String("gen_ai.tool.call.result", recordable(out)))
 	span.SetStatus(codes.Ok, "")
 	if strings.TrimSpace(out) == "" {
 		// An empty result and a failure look identical to a model reading a blank

apps/analytics/tag.js

@@ -181,5 +181,69 @@
   window.__hanzoEvent = true
   window.hanzo = { track: track, identify: identify, page: page, error: error, flush: flush }
 
+  // ── config-driven tag injection (the hosted twin of track.js) ─────────────
+  // Fetch the SITE's connected browser pixels (/v1/tags, dual-resolved per site by
+  // this key or the host) and inject them first-party. Every event then also fires
+  // the native pixel — translated through the SAME taxonomy the server-side CAPI uses
+  // (destinations/translate.go), stamped with a shared event_id carried on the
+  // /v1/event row too — so the browser pixel and the server CAPI DEDUPLICATE instead
+  // of double-counting. Keyless already returned above, so this only runs when keyed.
+  var STD = {
+    $pageview: 'page_view', pricing_viewed: 'view_content', signup_viewed: 'view_content',
+    product_viewed: 'view_content', plan_clicked: 'lead', signup_submitted: 'lead',
+    waitlist_joined: 'lead', referral_used: 'lead', signup_completed: 'signup',
+    product_added: 'add_to_cart', add_to_cart: 'add_to_cart', checkout_started: 'start_checkout',
+    begin_checkout: 'start_checkout', order_completed: 'purchase', purchase: 'purchase'
+  }
+  var NATIVE = {
+    ga: { page_view: 'page_view', view_content: 'view_item', add_to_cart: 'add_to_cart', lead: 'generate_lead', signup: 'sign_up', start_checkout: 'begin_checkout', purchase: 'purchase' },
+    meta: { page_view: 'PageView', view_content: 'ViewContent', add_to_cart: 'AddToCart', lead: 'Lead', signup: 'CompleteRegistration', start_checkout: 'InitiateCheckout', purchase: 'Purchase' },
+    tiktok: { page_view: 'Pageview', view_content: 'ViewContent', add_to_cart: 'AddToCart', lead: 'SubmitForm', signup: 'CompleteRegistration', start_checkout: 'InitiateCheckout', purchase: 'CompletePayment' },
+    x: { page_view: 'PageView', view_content: 'ViewContent', add_to_cart: 'AddToCart', lead: 'Lead', signup: 'SignUp', start_checkout: 'InitiateCheckout', purchase: 'Purchase' }
+  }
+  function nativeName(t, n) { var s = STD[n]; return s ? (NATIVE[t] || {})[s] || null : null }
+  function loadJS(u) { var e = document.createElement('script'); e.async = true; e.src = u; document.head.appendChild(e) }
+  function eid() { try { return crypto.randomUUID() } catch (e) { return 'e-' + Date.now().toString(36) + Math.random().toString(36).slice(2, 10) } }
+  var INJECT = {
+    ga: {
+      load: function (id) { loadJS('https://www.googletagmanager.com/gtag/js?id=' + encodeURIComponent(id)); window.dataLayer = window.dataLayer || []; window.gtag = window.gtag || function () { dataLayer.push(arguments) }; gtag('js', new Date()); gtag('config', id) },
+      fire: function (n, p, id) { if (!window.gtag) return; var name = nativeName('ga', n) || n; var o = {}; for (var k in p) o[k] = p[k]; o.event_id = id; if (STD[n] === 'purchase') o.transaction_id = id; gtag('event', name, o) }
+    },
+    meta: {
+      load: function (id) { if (!window.fbq) { var n = window.fbq = function () { n.callMethod ? n.callMethod.apply(n, arguments) : n.queue.push(arguments) }; if (!window._fbq) window._fbq = n; n.push = n; n.loaded = true; n.version = '2.0'; n.queue = []; loadJS('https://connect.facebook.net/en_US/fbevents.js') } fbq('init', id); fbq('track', 'PageView') },
+      fire: function (n, p, id) { if (!window.fbq) return; var name = nativeName('meta', n); if (name) fbq('track', name, p || {}, { eventID: id }); else fbq('trackCustom', n, p || {}) }
+    },
+    tiktok: {
+      load: function (id) { var q = window.ttq = window.ttq || []; if (!q.methods) { q.methods = ['page', 'track', 'identify', 'instances', 'debug', 'on', 'off', 'once', 'ready', 'alias', 'group', 'enableCookie', 'disableCookie']; q.setAndDefer = function (t, e) { t[e] = function () { t.push([e].concat(Array.prototype.slice.call(arguments, 0))) } }; for (var i = 0; i < q.methods.length; i++) q.setAndDefer(q, q.methods[i]); loadJS('https://analytics.tiktok.com/i18n/pixel/events.js?sdkid=' + encodeURIComponent(id) + '&lib=ttq') } if (q.load) q.load(id); if (q.page) q.page() },
+      fire: function (n, p, id) { if (!window.ttq || !window.ttq.track) return; var name = nativeName('tiktok', n) || n; window.ttq.track(name, p || {}, { event_id: id }) }
+    },
+    x: {
+      load: function (id) { if (!window.twq) { var s = window.twq = function () { s.exe ? s.exe.apply(s, arguments) : s.queue.push(arguments) }; s.version = '1.1'; s.queue = []; loadJS('https://static.ads-twitter.com/uwt.js') } twq('config', id) },
+      fire: function (n, p, id) { if (!window.twq) return; var name = nativeName('x', n) || n; var o = {}; for (var k in p) o[k] = p[k]; o.conversion_id = id; twq('event', name, o) }
+    }
+  }
+  var active = []
+  try {
+    fetch(src.origin + '/v1/tags?key=' + encodeURIComponent(key))
+      .then(function (r) { return r.ok ? r.json() : { tags: [] } })
+      .then(function (cfg) {
+        var tags = (cfg && cfg.tags) || []
+        for (var i = 0; i < tags.length; i++) {
+          var t = tags[i]
+          if (INJECT[t.type]) { try { INJECT[t.type].load(t.id); active.push(t) } catch (e) {} }
+        }
+        if (active.length) {
+          var base = window.hanzo.track
+          window.hanzo.track = function (n, p) {
+            var id = eid()
+            for (var j = 0; j < active.length; j++) { try { INJECT[active[j].type].fire(n, p, id) } catch (e) {} }
+            var o = {}; if (p) for (var k in p) o[k] = p[k]; o.event_id = id
+            base(n, o)
+          }
+        }
+      })
+      .catch(function () {})
+  } catch (e) {}
+
   page()
 })()

apps/analytics/tag_test.js

@@ -118,12 +118,14 @@ function anonOf(r) {
   assert.strictEqual(ev.event, undefined, 'naming is resolveEventName server-side')
 }
 
-// 4. Without sendBeacon the fetch path carries the key as a bearer.
+// 4. Without sendBeacon the fetch path carries the key as a bearer. (A keyed tag also
+//    fetches /v1/tags for its browser pixels; the /v1/event POST is the one asserted.)
 {
   const r = run({ 'data-key': 'pk-live-abc' }, { noBeacon: true })
   r.fire('pagehide')
-  assert.strictEqual(r.sent.fetch.length, 1)
-  const init = r.sent.fetch[0].init
+  const posts = r.sent.fetch.filter((f) => f.url.indexOf('/v1/event') !== -1)
+  assert.strictEqual(posts.length, 1)
+  const init = posts[0].init
   assert.strictEqual(init.headers.authorization, 'Bearer pk-live-abc')
   assert.strictEqual(init.keepalive, true)
   assert.ok(JSON.parse(init.body).batch.length === 1)
@@ -199,4 +201,22 @@ function anonOf(r) {
   assert.strictEqual(err.error.handled, false)
 }
 
-console.log('tag.js: 10/10 behavioral checks passed')
+// 11. Config-driven injection: a keyed tag fetches its site's browser pixels from
+//     /v1/tags (dual-resolved per site by this key or the host), so the hosted
+//     one-liner injects GA/Meta/TikTok/X first-party. The pixels + the deduping
+//     event_id are unit-tested in track.js; here we prove the hosted tag makes the
+//     request — and that a keyless tag never does.
+{
+  const r = run({ 'data-key': 'pk-live-abc' })
+  assert.ok(
+    r.sent.fetch.some((f) => f.url.indexOf('/v1/tags?key=pk-live-abc') !== -1),
+    'keyed tag fetches /v1/tags for its browser pixels'
+  )
+  const keyless = run({})
+  assert.ok(
+    !keyless.sent.fetch.some((f) => f.url.indexOf('/v1/tags') !== -1),
+    'keyless tag never fetches /v1/tags'
+  )
+}
+
+console.log('tag.js: 11/11 behavioral checks passed')

apps/answer/answer.go

@@ -202,6 +202,93 @@ func (e Engine) Serve(c *zip.Ctx, in Request, q string) error {
 	})
 }
 
+// Report is one researched answer held as a VALUE: the grounded prose and the
+// sources it cites. It carries what Serve writes as JSON minus the two keys that
+// only mean anything to the advisor contract — `figures`, which the web domain
+// leaves empty by construction, and `domain`, which is always "web" here.
+type Report struct {
+	// Answer is the grounded prose, with inline markdown citations. Every link in
+	// it points at a page in Sources: the citation check runs on the text before
+	// it leaves the engine, so a cited URL is one THIS call fetched.
+	Answer string `json:"answer"`
+	// Sources are the pages the answer was written from, deduplicated and ranked.
+	// Always an array, never null.
+	Sources []Source `json:"sources"`
+	// FollowUps are the questions worth asking next. Best-effort — an empty list
+	// is a normal outcome, not a fault.
+	FollowUps []string `json:"follow_ups"`
+	// Mode is the profile that ran: search, news, research or deep.
+	Mode string `json:"mode"`
+	// Model is the model that synthesized the answer.
+	Model string `json:"model"`
+}
+
+// Answer runs the SAME bounded loop Serve runs and hands back its outcome as a
+// value instead of writing it to a response.
+//
+// It exists because the loop's only door was an HTTP handler, and a handler is
+// the one shape an agent cannot reach: a typed op — and so an MCP tool, a CLI
+// command and an SDK method — is invoked with a context and no request at all.
+// So the engine that already separated Run (transport-free) from Serve (HTTP)
+// gains its second transport-free door rather than a second engine. Serve keeps
+// streaming; this returns one value; Run is still the only loop.
+//
+// The identity facts Serve reads off the request are read off the CONTEXT here,
+// which is where cloud.Bridge parks the server-minted ones. There is no ledger
+// claim on a context, so the caller's own org pays — the same fallback Serve
+// takes when no billing org was minted.
+func (e Engine) Answer(ctx context.Context, in Request, q string) (*Report, error) {
+	org, _ := principal.OrgFrom(ctx)
+	project := principal.ProjectFrom(ctx)
+
+	m := resolveMode(in.Mode)
+	fee := feeCents(m.name, m.feeCents)
+
+	// MONEY GATE — the same refusal Serve makes, before any work, so an
+	// out-of-funds caller is told so rather than handed a half answer.
+	capValidated := principal.ValidatedFrom(ctx) && !principal.IsDefaultProject(project)
+	if err := e.Bill.Gate(ctx, org, project, capValidated, "web", fee); err != nil {
+		return nil, err
+	}
+
+	models := synthModels(in.Model, m, e.Model)
+	p := Params{
+		q:            q,
+		webQuery:     buildQuery(q, m, in.Sources),
+		mode:         m,
+		model:        models[0],
+		fallbacks:    models[1:],
+		language:     strings.TrimSpace(in.Language),
+		maxSources:   clampPositive(in.MaxSources, m.maxSources),
+		maxQueries:   clampPositive(in.MaxQueries, m.maxQueries),
+		readTop:      m.readTop,
+		rounds:       min(m.rounds, maxRounds),
+		hostCap:      m.hostCap,
+		deadline:     m.deadline,
+		tokenCeiling: m.tokenCeiling,
+		followUps:    in.FollowUps == nil || *in.FollowUps,
+		system:       pickSystem(in.System, m.system),
+		dataOrg:      org,
+		payer:        org,
+		project:      project,
+		projectScope: project,
+		fee:          fee,
+	}
+
+	rctx, cancel := context.WithTimeout(ctx, p.deadline)
+	defer cancel()
+	buf := &bufferSink{}
+	e.Run(rctx, p, buf)
+
+	return &Report{
+		Answer:    buf.answer,
+		Sources:   nonNilSrc(buf.srcs),
+		FollowUps: nonNilStr(buf.follow),
+		Mode:      p.mode.name,
+		Model:     p.model,
+	}, nil
+}
+
 // Run is the bounded loop, parameterized by mode — the ONE code path for
 // search/news/research/deep. It emits the SearchEvent envelope through out, then
 // meters the caller ONCE. A failed step degrades (fewer sources, snippets instead

apps/answer/answer_test.go

@@ -777,3 +777,28 @@ func TestSynthesisPromptFencesCrawledPages(t *testing.T) {
 		t.Fatal("the page must still be present — fencing contains it, it does not drop it")
 	}
 }
+
+// A source hint must SCOPE the search, not decorate it. The hints were appended
+// as bare words — `x` turned "openai news" into "openai news x", which is the
+// original query plus a noise token and reaches X/Twitter not at all. Mojeek
+// honours `site:` (measured: site:x.com → 10 results; Bing → 0), so a hint that
+// names a site has to become one.
+func TestSourceHintsScopeToTheirSite(t *testing.T) {
+	q := buildQuery("openai", modes["search"], []string{"x"})
+	if !strings.Contains(q, "site:x.com") {
+		t.Fatalf("buildQuery with the x hint = %q, want it scoped to site:x.com", q)
+	}
+	multi := buildQuery("openai", modes["search"], []string{"github", "reddit"})
+	if !strings.Contains(multi, "site:github.com") || !strings.Contains(multi, "site:reddit.com") {
+		t.Fatalf("two site hints = %q, want both sites", multi)
+	}
+	// Two sites are alternatives, never both-at-once: `site:a site:b` is an AND
+	// no page can satisfy, and would answer zero.
+	if !strings.Contains(multi, " OR ") {
+		t.Fatalf("two site hints = %q, want them joined as alternatives", multi)
+	}
+	// A TOPICAL hint names no site and must stay a plain word.
+	if s := buildQuery("openai", modes["search"], []string{"web"}); strings.Contains(s, "site:") {
+		t.Fatalf("topical hint = %q, want no site scope", s)
+	}
+}

apps/answer/mode.go

@@ -261,15 +261,45 @@ func buildQuery(q string, m mode, sources []string) string {
 	if m.newsBias {
 		wq = q + " latest news " + strconv.Itoa(time.Now().Year())
 	}
-	var hints []string
+	var sites, words []string
 	for _, s := range sources {
 		t := strings.ToLower(strings.TrimSpace(s))
-		if knownSourceHints[t] {
-			hints = append(hints, t)
+		if !knownSourceHints[t] {
+			continue
 		}
+		if host := siteFor[t]; host != "" {
+			sites = append(sites, "site:"+host)
+			continue
+		}
+		words = append(words, t)
 	}
-	if len(hints) > 0 {
-		wq = wq + " " + strings.Join(hints, " ")
+	if len(words) > 0 {
+		wq = wq + " " + strings.Join(words, " ")
+	}
+	switch len(sites) {
+	case 0:
+	case 1:
+		wq = wq + " " + sites[0]
+	default:
+		// ALTERNATIVES, never both at once. `site:a site:b` reads as an AND and no
+		// page is on two hosts, so the naive join answers zero every time.
+		wq = wq + " (" + strings.Join(sites, " OR ") + ")"
 	}
 	return wq
 }
+
+// siteFor maps a source hint to the host it scopes the search to. A hint that
+// names a place becomes a `site:` operator; a hint that names a TOPIC (web,
+// news) has no host and stays a plain word, because there is no one site that
+// is "the news".
+//
+// This is what makes the `x` hint reach X at all. It was appended as the bare
+// word "x", so asking for X sources turned "openai" into "openai x" — the same
+// open-web query plus a token that matches nothing in particular. The scoped
+// form lands on Mojeek, which honours `site:` where Bing does not.
+var siteFor = map[string]string{
+	"academic": "arxiv.org",
+	"github":   "github.com",
+	"reddit":   "reddit.com",
+	"x":        "x.com",
+}

apps/ask/ask.go

@@ -163,6 +163,10 @@ func Mount(app cloud.Router, deps cloud.Deps) error {
 		model:    cloud.DefaultModel,
 	}}
 	app.Post("/v1/ask", cloud.Handle(svc, askHandler))
+	// The SAME answer engine, at an address an agent can speak. See web.go.
+	if err := mountWeb(app, svc.State, b); err != nil {
+		return err
+	}
 	b.Log.Info("ask mounted", "prefix", "/v1/ask", "domains", "books,projects,git,web", "web_modes", "search,news,research,deep")
 	return nil
 }

apps/ask/web.go

@@ -0,0 +1,147 @@
+// The researched answer, as a typed op — the door an agent can actually reach.
+//
+// The loop itself is not new and none of it lives here: apps/answer has run the
+// bounded plan → search → read → rank → synthesize → cite pass since it was
+// written, behind POST /v1/ask with a `mode`. What it did not have was a door a
+// MODEL could open. /v1/ask is untyped for three wire reasons that are all still
+// true (see the init in ask.go), and an untyped route reaches REST and nothing
+// else — no MCP tool, no CLI command, no SDK method. So the fleet's deep-research
+// capability was complete, correct, metered, and invisible to every agent in it.
+//
+// This is the same engine offered at an address the agent can speak, which is
+// exactly what apps/websearch already did for search: a compat door the model
+// cannot use, and beside it a native typed op running the SAME code. One engine,
+// two doors, no second implementation to drift.
+package ask
+
+import (
+	"context"
+	"strings"
+	"sync/atomic"
+
+	"github.com/hanzoai/cloud"
+	"github.com/hanzoai/cloud/apps/answer"
+	"github.com/hanzoai/cloud/apps/principal"
+	"github.com/zap-proto/zip"
+)
+
+// Go drops comments at compile time, so cmd/zipdoc is the ONLY path from the
+// handler's prose to the published document, the SDKs and the MCP tool
+// description. Its output is committed; `make zipdoc-check` fails on drift.
+//
+//go:generate go run github.com/zap-proto/zip/cmd/zipdoc
+
+// webPath is the researched answer's own address, under the prefix this app
+// already owns so the composed deployment routes it here without a manifest
+// change. /v1/research is NOT free — it belongs to the grants subsystem.
+const webPath = "/v1/ask/web"
+
+// engine holds what the typed op runs on. A typed handler is handed a context
+// and an In and nothing else, so the engine has to be reachable from package
+// scope; Mount stores it and the handler loads it. Same shape apps/sandbox uses
+// for its own typed ops, and for the same reason — a named handler, never a
+// closure, is what zipdoc can lift prose from.
+var engine atomic.Pointer[answer.Engine]
+
+// webQuestion is the POST /v1/ask/web body: the question, and the few knobs that
+// change how hard the engine looks for the answer.
+type webQuestion struct {
+	// Q is the question, in plain language. Required.
+	Q string `json:"q"`
+	// Mode is how much work to do: `search` (fast, one pass), `news` (recency
+	// biased), `research` (a plan and several rounds) or `deep` (the widest
+	// survey). Empty means research.
+	Mode string `json:"mode,omitempty"`
+	// Sources narrows where the evidence comes from: any of `web`, `news`,
+	// `academic`, `github`, `reddit`, `x`. Each becomes a site-scoped search, so
+	// `["x"]` researches X/Twitter posts rather than the open web.
+	Sources []string `json:"sources,omitempty"`
+	// Language narrows the search to a locale, BCP-47-ish ("en", "ja"). Empty
+	// means no narrowing.
+	Language string `json:"language,omitempty"`
+	// MaxSources caps how many pages are read. Empty means the mode's own budget.
+	MaxSources int `json:"max_sources,omitempty"`
+}
+
+// defaultMode is what an unspecified mode means. `research` and not `search`,
+// because a caller who reached for THIS op rather than search_web has already
+// said they want the reading done for them; a one-pass answer is what the other
+// op is.
+const defaultMode = "research"
+
+// researchWeb researches a question on the live web and answers it with its
+// sources cited.
+//
+// This is the DEEP one. It plans the question into topics, runs several web
+// searches, FETCHES AND READS the pages it finds, ranks them, and writes a
+// grounded answer with inline markdown citations. Use it for anything that needs
+// evidence, comparison or current fact — "what changed in X", "compare A and B",
+// "is this claim true". For a plain list of links, use search_web instead; for
+// one page you already have the URL of, use read_page.
+//
+// `mode` buys depth: `search` is a single fast pass, `news` biases to recency,
+// `research` plans and iterates, `deep` surveys widest. `sources` narrows the
+// evidence to `web`, `news`, `academic`, `github`, `reddit` or `x` — each becomes
+// a site-scoped search, which is how this reaches X/Twitter posts.
+//
+// EVERY CITATION IS A PAGE THIS CALL FETCHED. That is a property of the text and
+// not an instruction to the model: each source is fenced with a per-request nonce
+// so a crawled page cannot print itself a source number, and every markdown link
+// in the answer is checked against the gathered set before it is returned. So a
+// link in `answer` always appears in `sources`, and a page that was not read
+// cannot be cited.
+//
+// It is BOUNDED and it degrades rather than failing: a mode's rounds, wall clock
+// and token ceiling all cap it, and a search that finds little or a page that
+// will not load yields a thinner answer, never an error. A validated principal is
+// required, and the answer is billed once to that principal's org.
+func researchWeb(ctx context.Context, in *webQuestion) (*answer.Report, error) {
+	if !principal.ValidatedFrom(ctx) {
+		return nil, zip.ErrUnauthorized("sign in to research the web")
+	}
+	e := engine.Load()
+	if e == nil {
+		return nil, zip.Errorf(503, "ask: the answer engine is not mounted")
+	}
+	q := strings.TrimSpace(in.Q)
+	if q == "" {
+		return nil, zip.ErrBadRequest("q required")
+	}
+	if len(q) > maxQuestion {
+		q = q[:maxQuestion]
+	}
+	mode := strings.TrimSpace(in.Mode)
+	if mode == "" {
+		mode = defaultMode
+	}
+	if !answer.IsMode(mode) {
+		return nil, zip.ErrBadRequest("mode must be one of search, news, research, deep")
+	}
+	return e.Answer(ctx, answer.Request{
+		Mode:       mode,
+		Sources:    in.Sources,
+		Language:   in.Language,
+		MaxSources: in.MaxSources,
+	}, q)
+}
+
+// mountWeb stores the engine the typed op runs on and registers it.
+//
+// The registration is on the *zip.App and with an ABSOLUTE path, because that is
+// what zipdoc can resolve statically — it cannot follow a cloud.Router interface
+// to a prefix.
+func mountWeb(app cloud.Router, s *state, b cloud.Base) error {
+	engine.Store(&answer.Engine{Base: b, AI: s.ai, Model: s.model})
+	reg := cloud.ZipApp(app)
+	if reg == nil {
+		return nil // single-binary hosts without a typed registry keep the REST door
+	}
+	// Named, not derived. A POST to this path derives `create_ask_web`, which
+	// reads as "make an ask web" — a resource nothing here has. `research_web` is
+	// the verb over the noun, and it sits beside `search_web` so the two web verbs
+	// read as the pair they are: search returns links, research returns an answer.
+	zip.Post(reg, webPath, researchWeb,
+		zip.WithOperationID("research_web"),
+		zip.WithSummary("Research a question on the live web and answer it with sources cited"))
+	return nil
+}

apps/ask/zipdoc_gen.go

@@ -0,0 +1,25 @@
+// Code generated by zipdoc; DO NOT EDIT.
+
+package ask
+
+import (
+	"github.com/zap-proto/zip"
+)
+
+func init() {
+	zip.Describe("POST /v1/ask/web", zip.Doc{
+		Description: "Researches a question on the live web and answers it with its\nsources cited.\n\nThis is the DEEP one. It plans the question into topics, runs several web\nsearches, FETCHES AND READS the pages it finds, ranks them, and writes a\ngrounded answer with inline markdown citations. Use it for anything that needs\nevidence, comparison or current fact — \"what changed in X\", \"compare A and B\",\n\"is this claim true\". For a plain list of links, use search_web instead; for\none page you already have the URL of, use read_page.\n\n`mode` buys depth: `search` is a single fast pass, `news` biases to recency,\n`research` plans and iterates, `deep` surveys widest. `sources` narrows the\nevidence to `web`, `news`, `academic`, `github`, `reddit` or `x` — each becomes\na site-scoped search, which is how this reaches X/Twitter posts.\n\nEVERY CITATION IS A PAGE THIS CALL FETCHED. That is a property of the text and\nnot an instruction to the model: each source is fenced with a per-request nonce\nso a crawled page cannot print itself a source number, and every markdown link\nin the answer is checked against the gathered set before it is returned. So a\nlink in `answer` always appears in `sources`, and a page that was not read\ncannot be cited.\n\nIt is BOUNDED and it degrades rather than failing: a mode's rounds, wall clock\nand token ceiling all cap it, and a search that finds little or a page that\nwill not load yields a thinner answer, never an error. A validated principal is\nrequired, and the answer is billed once to that principal's org.",
+		Fields: map[string]string{
+			"Report.answer":           "Answer is the grounded prose, with inline markdown citations. Every link in\nit points at a page in Sources: the citation check runs on the text before\nit leaves the engine, so a cited URL is one THIS call fetched.",
+			"Report.follow_ups":       "FollowUps are the questions worth asking next. Best-effort — an empty list\nis a normal outcome, not a fault.",
+			"Report.mode":             "Mode is the profile that ran: search, news, research or deep.",
+			"Report.model":            "Model is the model that synthesized the answer.",
+			"Report.sources":          "Sources are the pages the answer was written from, deduplicated and ranked.\nAlways an array, never null.",
+			"webQuestion.language":    "Language narrows the search to a locale, BCP-47-ish (\"en\", \"ja\"). Empty\nmeans no narrowing.",
+			"webQuestion.max_sources": "MaxSources caps how many pages are read. Empty means the mode's own budget.",
+			"webQuestion.mode":        "Mode is how much work to do: `search` (fast, one pass), `news` (recency\nbiased), `research` (a plan and several rounds) or `deep` (the widest\nsurvey). Empty means research.",
+			"webQuestion.q":           "Q is the question, in plain language. Required.",
+			"webQuestion.sources":     "Sources narrows where the evidence comes from: any of `web`, `news`,\n`academic`, `github`, `reddit`, `x`. Each becomes a site-scoped search, so\n`[\"x\"]` researches X/Twitter posts rather than the open web.",
+		},
+	})
+}

apps/coding/adapters.go

@@ -54,7 +54,7 @@ const seamTimeout = 20 * time.Second
 func NewDispatcher(log func(msg string, kv ...any)) Dispatcher {
 	d := Dispatcher{
 		Sessions: planeSessions{},
-		Tracker:  planeTracker{},
+		PR:       planePR{},
 		// The SANDBOX runner. The docker-CLI path it replaces could not run at all:
 		// bot-gateway has neither that binary nor a socket, so every dispatch 503d
 		// while the chain read as configured. Same gVisor/Kata boundary, asked of the
@@ -118,10 +118,17 @@ func (planeSessions) Close(ctx context.Context, org, sessionID, status string) e
 	return err
 }
 
-// planeTracker files the native PR work item, in the tracker process.
-type planeTracker struct{}
+// planePR opens the pull request: the work item on our board (tracker), and the
+// proposal where the code lives (git — a GitHub pull request for a repository
+// that mirrors there, the branch's page here otherwise).
+//
+// Two peers, ONE seam. The board row is ours and is filed for every run, whatever
+// host the code is on; the address is a property of the host and only git can
+// answer it. Splitting them at the caller would put the choice of backend in the
+// orchestration, which is exactly where it must not be.
+type planePR struct{}
 
-func (planeTracker) CreatePR(ctx context.Context, in PRInput) (PRRef, error) {
+func (planePR) Open(ctx context.Context, in PRInput) (PRRef, error) {
 	ctx, cancel := bounded(ctx)
 	defer cancel()
 	// The org travels in the ENVELOPE (cloud.For), not in the body. Both spell the
@@ -130,7 +137,8 @@ func (planeTracker) CreatePR(ctx context.Context, in PRInput) (PRRef, error) {
 	// crossing the plane cannot be granted an org key the boundary would refuse.
 	// A body field would arrive unchecked — which is what made this a
 	// cross-tenant write. Same shape as cloud.UpsertIssue's Ask.
-	out, err := plane.Ask[plane.AgentPRIn, plane.AgentPROut](plane.For(ctx, in.Org), trackerApp, plane.TrackerAgentPR,
+	ctx = plane.For(ctx, in.Org)
+	out, err := plane.Ask[plane.AgentPRIn, plane.AgentPROut](ctx, trackerApp, plane.TrackerAgentPR,
 		&plane.AgentPRIn{
 			Project: in.Project, Repo: in.Repo, Base: in.Base,
 			Head: in.Head, Title: in.Title, Body: in.Body, Assignee: in.Assignee,
@@ -141,7 +149,24 @@ func (planeTracker) CreatePR(ctx context.Context, in PRInput) (PRRef, error) {
 	if out == nil {
 		return PRRef{}, fmt.Errorf("coding: tracker filed no PR")
 	}
-	return PRRef{Identifier: out.Identifier, ProjectKey: out.ProjectKey, Number: out.Number}, nil
+	ref := PRRef{Identifier: out.Identifier, ProjectKey: out.ProjectKey, Number: out.Number}
+
+	// The address. Its failure is returned BESIDE the row rather than instead of
+	// it: the work is pushed and tracked either way, and a run that could not
+	// reach GitHub must say so out loud instead of quietly answering with a forge
+	// link that is not where the review will happen.
+	p, perr := plane.Ask[plane.ProposeIn, plane.Proposed](ctx, gitApp, plane.GitPropose,
+		&plane.ProposeIn{
+			Project: in.Project, Repo: in.Repo, Base: in.Base,
+			Head: in.Head, Title: in.Title, Body: in.Body,
+		})
+	if perr != nil {
+		return ref, perr
+	}
+	if p != nil {
+		ref.URL = p.URL
+	}
+	return ref, nil
 }
 
 // planeCloneURL asks git for the org's clone URL. An error is an EMPTY url,

apps/coding/coding.go

@@ -5,7 +5,7 @@
 //
 // It is a LIBRARY, not an app: no route, no plugin, no manifest row. Its one
 // caller is apps/integrations (the Slack `code:` trigger). It touches its
-// collaborators only through interface seams (Sessions, Tracker, Runner) plus two
+// collaborators only through interface seams (Sessions, PR, Runner) plus two
 // git functions (CloneURL, VerifyRef), so the whole orchestration is unit-testable
 // with fakes and — critically — coding does NOT import apps/git: git imports
 // apps/integrations, integrations calls coding, so coding->git would cycle.
@@ -83,9 +83,16 @@ type Sessions interface {
 	Close(ctx context.Context, org, sessionID, status string) error
 }
 
-// Tracker is the work-item seam (clients/tracker in-process).
-type Tracker interface {
-	CreatePR(ctx context.Context, in PRInput) (PRRef, error)
+// PR opens the pull request for a finished run and says where it can be read.
+//
+// It is ONE seam with two backends behind it, because a repository can live in
+// two places and a run must not care which: the work item lands on our board
+// either way, and the address comes back from the forge for a repository that
+// lives only here or from GitHub for one that mirrors there. A caller that had to
+// ask which host it was talking to would be two orchestrations pretending to be
+// one, and they would drift.
+type PR interface {
+	Open(ctx context.Context, in PRInput) (PRRef, error)
 }
 
 // Runner is the bot-gateway coding-task seam (clients/bot in-process client).
@@ -110,6 +117,12 @@ type PRRef struct {
 	Identifier string
 	ProjectKey string
 	Number     int
+	// URL is where the proposal is READ — the GitHub pull request when the
+	// repository mirrors there, the branch's page in the forge when it does not.
+	// It is the only part of a finished run a person can click, so it travels all
+	// the way out to the chat thread; empty is a real answer (a project-scoped
+	// repository has no browsable page) and costs a link, never a run.
+	URL string
 }
 
 // RunRequest / Step / RunResult mirror the bot coding contract.
@@ -174,6 +187,11 @@ type Req struct {
 	CredToken      string
 	TimeoutSeconds int
 	TargetID       string // when set, route to this registered machine instead of the sandbox
+	// Tool / Desktop are the caller's choice of harness and whether it needs a
+	// screen. They travel to RunRequest unchanged; classFor and argvFor over in
+	// sandboxrunner.go are the only two things that read them.
+	Tool    string
+	Desktop bool
 	// SessionID adopts an ALREADY-OPEN session instead of opening one. The door
 	// (Start) opens it so it can answer with a real handle the moment the run is
 	// admitted, rather than an empty promise the caller cannot watch. Empty keeps
@@ -240,7 +258,7 @@ const (
 // cancelled with the run.
 type Dispatcher struct {
 	Sessions  Sessions
-	Tracker   Tracker
+	PR        PR
 	Runner    Runner
 	CloneURL  func(ctx context.Context, org, repo string) string
 	VerifyRef func(ctx context.Context, org, repo, branch string) (string, bool)
@@ -348,6 +366,7 @@ func (d Dispatcher) Run(ctx context.Context, req Req) Result {
 
 	// 2. Dispatch to the sandbox runtime, mirroring every progress line live.
 	runReq := RunRequest{
+		Tool: req.Tool, Desktop: req.Desktop,
 		CloneURL: cloneURL, BaseBranch: strings.TrimSpace(req.Base), Branch: branch,
 		Prompt: prompt, SessionID: sessionID, RunTimeoutSeconds: timeoutOr(req.TimeoutSeconds),
 		CredUser: req.CredUser, CredToken: req.CredToken,
@@ -372,7 +391,7 @@ func (d Dispatcher) Run(ctx context.Context, req Req) Result {
 	// cloud, named after a session the sandbox did not choose — and adopting the
 	// sandbox's self-report let a compromised one answer `main`, which then flowed
 	// into VerifyRef (which only asks whether a ref EXISTS, and main does) and out
-	// as CreatePR{Head: "main"}. A PR headed at the trunk, filed by us, from a run
+	// as PR.Open{Head: "main"}. A PR headed at the trunk, filed by us, from a run
 	// that never had permission to write there.
 	//
 	// The sandbox has nothing to report here: it was TOLD which branch to push,
@@ -431,19 +450,22 @@ func (d Dispatcher) completeChanged(ctx context.Context, c completion, res Resul
 			res.CommitSha = sha // authoritative tip from our own storage
 		}
 	}
-	pr, perr := d.Tracker.CreatePR(ctx, PRInput{
+	pr, perr := d.PR.Open(ctx, PRInput{
 		Org: c.org, Project: strings.TrimSpace(c.project), Repo: c.repo,
 		Base: baseOr(c.base), Head: c.branch, Title: codingTitle(c.repo, c.prompt),
 		Body: prBody(c.prompt, c.base, c.branch, res.CommitSha, c.diffstat, c.sessionID), Assignee: c.agentRef,
 	})
+	// Whatever came back is kept, even beside an error: opening a PR is two acts
+	// in two places, and a run that filed its work item but could not reach GitHub
+	// has a real handle to report. Discarding it would hide the half that worked.
+	res.PR = pr
 	if perr != nil {
-		d.logf("coding: tracker PR create failed", "org", c.org, "repo", c.repo, "err", perr)
-		d.mirror(ctx, c.org, c.sessionID, c.actor, kindLog, map[string]any{"message": "tracker PR not created: " + perr.Error()})
-	} else {
-		res.PR = pr
+		d.logf("coding: PR open failed", "org", c.org, "repo", c.repo, "err", perr)
+		d.mirror(ctx, c.org, c.sessionID, c.actor, kindLog, map[string]any{"message": "pull request: " + perr.Error()})
 	}
 	d.mirror(ctx, c.org, c.sessionID, c.actor, kindStatus, map[string]any{
-		"status": "done", "changed": true, "branch": c.branch, "commit": res.CommitSha, "pr": pr.Identifier,
+		"status": "done", "changed": true, "branch": c.branch,
+		"commit": res.CommitSha, "pr": pr.Identifier, "url": pr.URL,
 	})
 	_ = d.Sessions.Close(ctx, c.org, c.sessionID, statusDone)
 	res.OK = true

apps/coding/coding_test.go

@@ -56,14 +56,14 @@ func (f *fakeSessions) Close(_ context.Context, org, session, status string) err
 	return nil
 }
 
-type fakeTracker struct {
+type fakePR struct {
 	mu     sync.Mutex
 	inputs []PRInput
 	ref    PRRef
 	err    error
 }
 
-func (f *fakeTracker) CreatePR(_ context.Context, in PRInput) (PRRef, error) {
+func (f *fakePR) Open(_ context.Context, in PRInput) (PRRef, error) {
 	f.mu.Lock()
 	defer f.mu.Unlock()
 	f.inputs = append(f.inputs, in)
@@ -94,10 +94,10 @@ func (f *fakeRunner) Run(_ context.Context, org, userID string, req RunRequest,
 // dispatcherFor builds a Dispatcher whose git seams are org-aware fakes: CloneURL
 // echoes the (org, repo) so a test can assert the sandbox is pointed only at the
 // caller's namespace; VerifyRef succeeds for a configured branch.
-func dispatcherFor(sess *fakeSessions, tr *fakeTracker, run *fakeRunner, verifyOK bool) (Dispatcher, *[]string) {
+func dispatcherFor(sess *fakeSessions, tr *fakePR, run *fakeRunner, verifyOK bool) (Dispatcher, *[]string) {
 	var cloneCalls []string
 	d := Dispatcher{
-		Sessions: sess, Tracker: tr, Runner: run,
+		Sessions: sess, PR: tr, Runner: run,
 		CloneURL: func(_ context.Context, org, repo string) string {
 			cloneCalls = append(cloneCalls, org+"/"+repo)
 			return "https://git.test/v1/git/" + org + "/" + repo + ".git"
@@ -125,7 +125,7 @@ func baseReq() Req {
 
 func TestRun_HappyPath_PushVerifyPR_NoCredentialLeak(t *testing.T) {
 	sess := &fakeSessions{id: "sess_abc123def456"}
-	tr := &fakeTracker{ref: PRRef{Identifier: "API-1", ProjectKey: "API", Number: 1}}
+	tr := &fakePR{ref: PRRef{Identifier: "API-1", ProjectKey: "API", Number: 1}}
 	run := &fakeRunner{
 		steps:  []Step{{Type: "step", Step: "clone", Status: "ok"}, {Type: "log", Message: "editing handler.go"}, {Type: "step", Step: "push", Status: "ok"}},
 		result: RunResult{Branch: "agent/abc123def456", CommitSha: "deadbeef", Diffstat: "1 file changed", Changed: true, OK: true},
@@ -188,7 +188,7 @@ func TestRun_EventVocabulary_StepIsToolCall_LogIsLog(t *testing.T) {
 		steps:  []Step{{Type: "step", Step: "clone"}, {Type: "log", Message: "hi"}},
 		result: RunResult{Branch: "agent/x", Changed: true, OK: true},
 	}
-	d, _ := dispatcherFor(sess, &fakeTracker{}, run, true)
+	d, _ := dispatcherFor(sess, &fakePR{}, run, true)
 	d.Run(context.Background(), baseReq())
 
 	var toolCalls, logs, statuses int
@@ -214,7 +214,7 @@ func TestRun_EventVocabulary_StepIsToolCall_LogIsLog(t *testing.T) {
 
 func TestRun_NoChanges_NoPR_DoneNotError(t *testing.T) {
 	sess := &fakeSessions{id: "sess_x"}
-	tr := &fakeTracker{}
+	tr := &fakePR{}
 	run := &fakeRunner{result: RunResult{Changed: false, OK: true}}
 	d, _ := dispatcherFor(sess, tr, run, true)
 
@@ -232,7 +232,7 @@ func TestRun_NoChanges_NoPR_DoneNotError(t *testing.T) {
 
 func TestRun_RunnerError_ClosesError_NoPR(t *testing.T) {
 	sess := &fakeSessions{id: "sess_x"}
-	tr := &fakeTracker{}
+	tr := &fakePR{}
 	run := &fakeRunner{err: context.DeadlineExceeded}
 	d, _ := dispatcherFor(sess, tr, run, true)
 
@@ -250,7 +250,7 @@ func TestRun_RunnerError_ClosesError_NoPR(t *testing.T) {
 
 func TestRun_VerifyRefFails_FailsClosed_NoPR(t *testing.T) {
 	sess := &fakeSessions{id: "sess_x"}
-	tr := &fakeTracker{}
+	tr := &fakePR{}
 	run := &fakeRunner{result: RunResult{Branch: "agent/x", Changed: true, OK: true}}
 	d, _ := dispatcherFor(sess, tr, run, false) // verify fails
 
@@ -268,7 +268,7 @@ func TestRun_VerifyRefFails_FailsClosed_NoPR(t *testing.T) {
 
 func TestRun_MissingCredential_FailsBeforeOpeningSession(t *testing.T) {
 	sess := &fakeSessions{id: "sess_x"}
-	d, _ := dispatcherFor(sess, &fakeTracker{}, &fakeRunner{}, true)
+	d, _ := dispatcherFor(sess, &fakePR{}, &fakeRunner{}, true)
 
 	req := baseReq()
 	req.CredToken = ""
@@ -282,7 +282,7 @@ func TestRun_MissingCredential_FailsBeforeOpeningSession(t *testing.T) {
 }
 
 func TestRun_MissingRepoOrPrompt_FailsClosed(t *testing.T) {
-	d, _ := dispatcherFor(&fakeSessions{id: "s"}, &fakeTracker{}, &fakeRunner{}, true)
+	d, _ := dispatcherFor(&fakeSessions{id: "s"}, &fakePR{}, &fakeRunner{}, true)
 	if r := d.Run(context.Background(), Req{Org: "acme", CredToken: "x", Prompt: "do it"}); r.OK || r.Error == "" {
 		t.Fatalf("missing repo must fail: %+v", r)
 	}
@@ -291,9 +291,9 @@ func TestRun_MissingRepoOrPrompt_FailsClosed(t *testing.T) {
 	}
 }
 
-func TestRun_TrackerFailure_DoesNotFailRun(t *testing.T) {
+func TestRun_PRFailure_DoesNotFailRun(t *testing.T) {
 	sess := &fakeSessions{id: "sess_x"}
-	tr := &fakeTracker{err: context.Canceled}
+	tr := &fakePR{err: context.Canceled}
 	run := &fakeRunner{result: RunResult{Branch: "agent/x", Changed: true, OK: true}}
 	d, _ := dispatcherFor(sess, tr, run, true)
 

apps/coding/key.go

@@ -0,0 +1,98 @@
+package coding
+
+// The credential the agent inside a sandbox answers the gateway with.
+//
+// Without one the whole product stops one step short of working: the box leases,
+// the repo clones, `dev` starts, hanzo-mcp answers — and then the model call
+// returns `Missing environment variable: HANZO_API_KEY`, or, with a key that
+// names nobody, the gateway's 402 `a billable tenant is required (no anonymous
+// usage)`. Every part was built and nothing could finish a task.
+//
+// IT IS OUR OWN MACHINE IDENTITY, MINTED PER RUN, and it is minted the way the
+// rest of the fleet mints: client_credentials against IAM, through the same
+// golang.org/x/oauth2 config clients/aihttp.go uses for inference. There is no
+// second notion of "who a sandbox is" — a run authenticates as the deployment
+// that started it, which is the identity the gateway already prices and meters.
+//
+// A STATIC KEY WAS THE OTHER OPTION AND IS WORSE IN EVERY DIRECTION. A long-lived
+// hk- key would have to be stored, rotated, and handed to a box that is about to
+// execute a model's output; this token expires on its own, is scoped to one
+// identity, and nothing has to remember to revoke it.
+//
+// The token URL is not resolved here. IAMBaseURL is the fleet's split-horizon
+// policy — the public issuer is Cloudflare-fronted and 403s an in-cluster
+// server-side POST (edge 1006), so an in-cluster address must win — and that
+// policy having exactly one home is why the agent runner works at all.
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"strings"
+	"sync"
+
+	"github.com/hanzoai/cloud"
+	"golang.org/x/oauth2"
+	"golang.org/x/oauth2/clientcredentials"
+)
+
+// source is built once and reused: the oauth2 client_credentials source caches a
+// token until it expires, so a burst of runs mints one token rather than one
+// each. Nil when the deployment carries no machine identity, which is the honest
+// answer for a dev box and reads as "no credential" at the call site rather than
+// as an empty string that looks like one.
+var source = sync.OnceValue(func() oauth2.TokenSource {
+	id := strings.TrimSpace(os.Getenv("IAM_CLIENT_ID"))
+	secret := strings.TrimSpace(os.Getenv("IAM_CLIENT_SECRET"))
+	base := cloud.IAMBaseURL(cloud.IAMIssuer())
+	if id == "" || secret == "" || base == "" {
+		return nil
+	}
+	cc := &clientcredentials.Config{
+		ClientID:     id,
+		ClientSecret: secret,
+		TokenURL:     base + "/v1/iam/oauth/token",
+		// hanzo.id reads the credentials from the form body, not Basic auth —
+		// the same style clients/aihttp.go proved against this issuer.
+		AuthStyle: oauth2.AuthStyleInParams,
+	}
+	return cc.TokenSource(context.Background())
+})
+
+// key returns the bearer a run authenticates with, or "" when this deployment
+// holds no machine identity. An error is returned only when an identity IS
+// configured and minting failed, because that is a broken deployment rather than
+// an unconfigured one and the difference is what a caller needs to report.
+func key(ctx context.Context) (string, error) {
+	ts := source()
+	if ts == nil {
+		return "", nil
+	}
+	tok, err := ts.Token()
+	if err != nil {
+		return "", fmt.Errorf("mint sandbox credential: %w", err)
+	}
+	return tok.AccessToken, nil
+}
+
+// keyed wraps an agent's argv so the credential arrives on STDIN and becomes an
+// environment variable inside the box, and is never an argument.
+//
+// ARGV IS PUBLIC AND STDIN IS NOT. Every process in the pod can read another's
+// command line out of /proc, the argv is what a run echoes into its own audit
+// line and its session narration, and the process we are handing it to is about
+// to execute a language model's output. Stdin reaches this one process and
+// nothing else, the exec stream already carries it, and the value never touches
+// a file, a layer, or the pod spec.
+//
+// `IFS= read -r` takes exactly the first line and leaves the rest of stdin for
+// the agent, so this does not consume input a tool might want. `exec "$@"`
+// replaces the shell, so the agent keeps PID-of-the-command and signals and exit
+// codes travel unchanged — the wrapper is gone by the time the agent runs.
+func keyed(argv []string) []string {
+	return append([]string{
+		"sh", "-c",
+		`IFS= read -r HANZO_API_KEY; export HANZO_API_KEY; exec "$@"`,
+		"sh",
+	}, argv...)
+}

apps/coding/key_test.go

@@ -0,0 +1,75 @@
+package coding
+
+// Where the sandbox credential is allowed to appear.
+//
+// The run hands a bearer to a process that is about to execute a language
+// model's output, in a pod that also holds the model's own tools. Argv is the
+// one place it must never be: /proc makes another process's command line
+// readable, and this argv is echoed into the run's session narration and its
+// audit line, so a credential there is published three ways at once.
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestTheCredentialIsNeverAnArgument(t *testing.T) {
+	const secret = "eyJhbGciOiJSUzI1NiJ9.THE-BEARER.sig"
+	got := keyed([]string{"dev", "exec", "--full-auto", "--", "fix the bug"})
+	for i, a := range got {
+		if strings.Contains(a, secret) {
+			t.Fatalf("argv[%d] carries the credential: %q", i, a)
+		}
+	}
+	// It is read from stdin into the environment, and the agent still runs.
+	if !strings.Contains(got[2], "read -r HANZO_API_KEY") {
+		t.Errorf("the wrapper does not read the credential: %q", got[2])
+	}
+	if !strings.Contains(got[2], "export HANZO_API_KEY") {
+		t.Errorf("the credential is read but never exported: %q", got[2])
+	}
+	// `exec` matters: without it the shell outlives the agent and swallows its
+	// signals and exit code, so a cancelled run would report the shell's status.
+	if !strings.Contains(got[2], `exec "$@"`) {
+		t.Errorf("the wrapper does not exec the agent, so it keeps the process: %q", got[2])
+	}
+}
+
+func TestTheAgentArgvSurvivesIntact(t *testing.T) {
+	// A wrapper that drops or reorders the agent's own arguments would change the
+	// task being run. `$0` is spent on the shell's name, so the agent's argv must
+	// begin immediately after it.
+	argv := []string{"dev", "exec", "--full-auto", "--skip-git-repo-check", "--", "a prompt with spaces"}
+	got := keyed(argv)
+	if len(got) != len(argv)+4 {
+		t.Fatalf("got %d words, want %d: %v", len(got), len(argv)+4, got)
+	}
+	if got[0] != "sh" || got[1] != "-c" || got[3] != "sh" {
+		t.Fatalf("wrapper prefix is not `sh -c <script> sh`: %v", got[:4])
+	}
+	for i, want := range argv {
+		if got[i+4] != want {
+			t.Errorf("argv[%d] = %q, want %q", i, got[i+4], want)
+		}
+	}
+}
+
+func TestNoIdentityIsNotAnError(t *testing.T) {
+	// A dev box holds no machine identity. That must read as "no credential" and
+	// let the run proceed to the harness's own honest failure, not as a broken
+	// deployment — the lease, the clone and the branch all still happened.
+	t.Setenv("IAM_CLIENT_ID", "")
+	t.Setenv("IAM_CLIENT_SECRET", "")
+	if got := source(); got != nil {
+		// source() is memoized per process, so this only asserts the shape when
+		// this test is the one that built it.
+		t.Skip("a machine identity was already resolved in this process")
+	}
+	k, err := key(t.Context())
+	if err != nil {
+		t.Fatalf("no identity reported an error: %v", err)
+	}
+	if k != "" {
+		t.Fatalf("no identity produced a credential: %q", k)
+	}
+}

apps/coding/plane_test.go

@@ -32,11 +32,12 @@ import (
 // what ARRIVED on the far side rather than what was sent.
 type peers struct {
 	sessions *fakeSessions
-	tracker  *fakeTracker
+	tracker  *fakePR
 	clone    string
 	tip      string
 	found    bool
 	gated    []string
+	proposed string
 }
 
 func servePeers(t *testing.T, p *peers) {
@@ -82,13 +83,18 @@ func servePeers(t *testing.T, p *peers) {
 		func(_ context.Context, _ *plane.RefIn) (*plane.RefTip, error) {
 			return &plane.RefTip{SHA: p.tip, Found: p.found}, nil
 		}, zip.WithOperationID(plane.GitVerifyRef))
+	zip.Post[plane.ProposeIn, plane.Proposed](gitApp, "/git/propose",
+		func(_ context.Context, in *plane.ProposeIn) (*plane.Proposed, error) {
+			p.proposed = in.Head
+			return &plane.Proposed{URL: "https://git.test/git/acme/api?ref=" + in.Head}, nil
+		}, zip.WithOperationID(plane.GitPropose))
 
 	trackerApp := zip.New(zip.Config{AppName: "tracker", DisableStartupMessage: true})
 	zip.Post[plane.AgentPRIn, plane.AgentPROut](trackerApp, "/tracker/agent-pr",
 		func(ctx context.Context, in *plane.AgentPRIn) (*plane.AgentPROut, error) {
 			// No in.Org: the org is the caller's plane identity, mirroring the real
 			// handler (plugin/tracker/seams.go) after the cross-tenant write was closed.
-			ref, err := p.tracker.CreatePR(ctx, PRInput{
+			ref, err := p.tracker.Open(ctx, PRInput{
 				Org: zip.CallerOf(ctx).Org, Project: in.Project, Repo: in.Repo, Base: in.Base,
 				Head: in.Head, Title: in.Title, Body: in.Body, Assignee: in.Assignee,
 			})
@@ -123,7 +129,7 @@ func waitListening(t *testing.T, app string) {
 // runner faked — the runner was always an HTTP client and never had a boundary.
 func planeDispatcher(run *fakeRunner) Dispatcher {
 	return Dispatcher{
-		Sessions: planeSessions{}, Tracker: planeTracker{}, Runner: run,
+		Sessions: planeSessions{}, PR: planePR{}, Runner: run,
 		CloneURL: planeCloneURL, VerifyRef: planeVerifyRef, TargetGate: planeTargetGate,
 	}
 }
@@ -132,7 +138,7 @@ func planeDispatcher(run *fakeRunner) Dispatcher {
 func TestRun_OverThePlane_CompletesAcrossProcesses(t *testing.T) {
 	p := &peers{
 		sessions: &fakeSessions{id: "sess_abc123def456"},
-		tracker:  &fakeTracker{ref: PRRef{Identifier: "API-7", ProjectKey: "API", Number: 7}},
+		tracker:  &fakePR{ref: PRRef{Identifier: "API-7", ProjectKey: "API", Number: 7}},
 		tip:      "verifiedsha", found: true,
 	}
 	servePeers(t, p)
@@ -152,6 +158,15 @@ func TestRun_OverThePlane_CompletesAcrossProcesses(t *testing.T) {
 	if res.PR.Identifier != "API-7" {
 		t.Fatalf("the PR must be filed through tracker's door, got %q", res.PR.Identifier)
 	}
+	// ONE seam, both backends: the row landed on the board AND git answered where
+	// the proposal is read. A run whose result carries no address gives a person
+	// in a thread nothing to click.
+	if p.proposed != "agent/abc123def456" {
+		t.Fatalf("git was never asked to propose the branch, got %q", p.proposed)
+	}
+	if res.PR.URL != "https://git.test/git/acme/api?ref=agent/abc123def456" {
+		t.Fatalf("the address did not come back with the run: %q", res.PR.URL)
+	}
 	if res.CommitSha != "verifiedsha" {
 		t.Fatalf("the tip must come from git's own storage, got %q", res.CommitSha)
 	}
@@ -192,7 +207,7 @@ func TestRun_OverThePlane_CompletesAcrossProcesses(t *testing.T) {
 func TestRun_OverThePlane_UnverifiedRefFilesNoPR(t *testing.T) {
 	p := &peers{
 		sessions: &fakeSessions{id: "sess_abc123def456"},
-		tracker:  &fakeTracker{ref: PRRef{Identifier: "API-8"}},
+		tracker:  &fakePR{ref: PRRef{Identifier: "API-8"}},
 		found:    false,
 	}
 	servePeers(t, p)

apps/coding/progress.go

@@ -104,6 +104,7 @@ func renderLine(kind string, payload []byte) (line string, terminal bool) {
 		Status  string `json:"status"`
 		Branch  string `json:"branch"`
 		PR      string `json:"pr"`
+		URL     string `json:"url"`
 		Error   string `json:"error"`
 		Changed bool   `json:"changed"`
 	}
@@ -128,6 +129,14 @@ func renderLine(kind string, payload []byte) (line string, terminal bool) {
 		if e.PR != "" {
 			l += " · PR `" + esc(e.PR) + "`"
 		}
+		// The address goes out BARE. Slack turns a plain URL into a link on its
+		// own, so nothing here has to build `<url|text>` — which is the one markup
+		// element that carries an arbitrary destination, and therefore the one
+		// esc() exists to stop a run from writing. A link nobody had to construct
+		// cannot be constructed by a prompt.
+		if e.URL != "" {
+			l += " " + esc(e.URL)
+		}
 		return l, true
 	case e.Status == "started":
 		return ":hourglass_flowing_sand: Working `" + esc(e.Branch) + "`…", false

apps/coding/routed_test.go

@@ -47,7 +47,7 @@ func (g *fakeGate) gate(_ context.Context, org, target string) error {
 func routedDispatcher(sess *fakeSessions, run *fakeRunner, router *fakeRouter, gate *fakeGate) (Dispatcher, *[]string) {
 	var cloneCalls []string
 	d := Dispatcher{
-		Sessions: sess, Tracker: &fakeTracker{}, Runner: run,
+		Sessions: sess, PR: &fakePR{}, Runner: run,
 		CloneURL: func(_ context.Context, org, repo string) string {
 			cloneCalls = append(cloneCalls, org+"/"+repo)
 			return "https://git.test/v1/git/" + org + "/" + repo + ".git"
@@ -220,7 +220,7 @@ func TestRun_RoutedButRoutingUnwired_FailsClosed(t *testing.T) {
 	sess := &fakeSessions{id: "sess_x"}
 	run := &fakeRunner{}
 	// No Route / TargetGate seams.
-	d := Dispatcher{Sessions: sess, Tracker: &fakeTracker{}, Runner: run,
+	d := Dispatcher{Sessions: sess, PR: &fakePR{}, Runner: run,
 		CloneURL: func(_ context.Context, org, repo string) string {
 			return "https://git.test/v1/git/" + org + "/" + repo + ".git"
 		}}
@@ -236,9 +236,9 @@ func TestRun_RoutedButRoutingUnwired_FailsClosed(t *testing.T) {
 
 // ---- routed completion parity (#48 I2): verify + PR + session close ----
 
-func finalizeDispatcher(sess *fakeSessions, tr *fakeTracker, verifyOK bool) Dispatcher {
+func finalizeDispatcher(sess *fakeSessions, tr *fakePR, verifyOK bool) Dispatcher {
 	return Dispatcher{
-		Sessions: sess, Tracker: tr,
+		Sessions: sess, PR: tr,
 		VerifyRef: func(_ context.Context, _, _, _ string) (string, bool) {
 			if verifyOK {
 				return "verifiedsha", true
@@ -252,7 +252,7 @@ func finalizeDispatcher(sess *fakeSessions, tr *fakeTracker, verifyOK bool) Disp
 // side completion as the local path: the PR is filed and the session is closed done.
 func TestFinalizeRouted_ChangedVerifyPasses_FilesPR_ClosesDone(t *testing.T) {
 	sess := &fakeSessions{}
-	tr := &fakeTracker{ref: PRRef{Identifier: "API-9"}}
+	tr := &fakePR{ref: PRRef{Identifier: "API-9"}}
 	d := finalizeDispatcher(sess, tr, true)
 	in := RoutedRun{Org: "acme", SessionID: "sess_r", Repo: "api", Base: "main", Branch: "agent/r", Prompt: "add a test", Actor: "u-1", AgentRef: "hanzo"}
 
@@ -284,7 +284,7 @@ func TestFinalizeRouted_ChangedVerifyPasses_FilesPR_ClosesDone(t *testing.T) {
 // — trust the tips we can read, not the machine's self-report.
 func TestFinalizeRouted_VerifyFails_NoPR_ClosesError(t *testing.T) {
 	sess := &fakeSessions{}
-	tr := &fakeTracker{}
+	tr := &fakePR{}
 	d := finalizeDispatcher(sess, tr, false) // verify fails
 	in := RoutedRun{Org: "acme", SessionID: "s", Repo: "api", Branch: "agent/r"}
 
@@ -301,7 +301,7 @@ func TestFinalizeRouted_VerifyFails_NoPR_ClosesError(t *testing.T) {
 // A routed run that reported NO changes closes the session done with no PR.
 func TestFinalizeRouted_NoChanges_NoPR_ClosesDone(t *testing.T) {
 	sess := &fakeSessions{}
-	tr := &fakeTracker{}
+	tr := &fakePR{}
 	d := finalizeDispatcher(sess, tr, true)
 	d.finalizeRouted(context.Background(), RoutedRun{Org: "acme", SessionID: "s", Repo: "api"}, RoutedResult{OK: true, Changed: false})
 
@@ -317,9 +317,9 @@ func TestFinalizeRouted_NoChanges_NoPR_ClosesDone(t *testing.T) {
 // even VerifyRef is never consulted (there is nothing to verify).
 func TestFinalizeRouted_ReportedError_ClosesError_NoPR(t *testing.T) {
 	sess := &fakeSessions{}
-	tr := &fakeTracker{}
+	tr := &fakePR{}
 	verifyCalled := false
-	d := Dispatcher{Sessions: sess, Tracker: tr, VerifyRef: func(context.Context, string, string, string) (string, bool) {
+	d := Dispatcher{Sessions: sess, PR: tr, VerifyRef: func(context.Context, string, string, string) (string, bool) {
 		verifyCalled = true
 		return "", true
 	}}
@@ -363,7 +363,7 @@ func TestNewDispatcher_WiresRoutedFinalizer(t *testing.T) {
 // the attribution the completion needs.
 func TestFinalizeRoutedDurable_BridgesFields(t *testing.T) {
 	sess := &fakeSessions{}
-	tr := &fakeTracker{ref: PRRef{Identifier: "API-1"}}
+	tr := &fakePR{ref: PRRef{Identifier: "API-1"}}
 	d := finalizeDispatcher(sess, tr, true)
 	d.finalizeRoutedDurable(context.Background(),
 		agents.RoutedRun{Org: "acme", SessionID: "s", Repo: "api", Branch: "agent/b", AgentRef: "hanzo", Actor: "u-9"},

apps/coding/sandboxrunner.go

@@ -43,7 +43,9 @@ package coding
 
 import (
 	"context"
+	"encoding/base64"
 	"fmt"
+	"sort"
 	"strings"
 	"time"
 
@@ -77,24 +79,86 @@ func classFor(tool string, desktop bool) string {
 
 // argvFor is the command that runs inside the sandbox.
 //
-// The runtime owns the real name→argv table (bot's TOOLS), and this is the
-// minimum cloud has to know to start one. It is deliberately not a second table
-// of flags: everything a harness needs beyond its name travels in the prompt.
+// Each harness names its OWN non-interactive verb, and they do not agree. That
+// is the whole content of this table: `-p` means print-the-answer to claude and
+// --profile to dev, so one spelling copied across both is not a style choice,
+// it is a run that never starts.
+//
+// Measured in a dev-class pod (oci.hanzo.ai/hanzoai/sandbox 1.0.1, dev 0.6.91):
+//
+//	dev -p "say hi"     ->  Error loading configuration: config profile `say hi` not found
+//	dev exec "say hi"    ->  runs
+//
+// Every coding run cloud has ever dispatched took the first line. The prompt
+// was read as the name of a config profile, no profile by that name existed,
+// and the process was gone before a model was ever asked anything.
+//
+// TWO FLAGS, AND NEITHER IS A PREFERENCE.
+//
+//   - --full-auto is `-a on-failure --sandbox workspace-write`: nobody is at a
+//     terminal to approve a command, so a run that asks is a run that hangs.
+//   - --skip-git-repo-check because CloneURL is optional. dev refuses to run
+//     outside a git repo to protect a laptop from itself; the pod is not a
+//     laptop, and the checkout is the thing it was given rather than the thing
+//     it wandered into.
+//
+// dev keeps its own workspace-write sandbox on TOP of the pod's isolation. It
+// is not the boundary — runtimeClass is — but it costs nothing, it was measured
+// to work under gVisor, and a harness that confines its own writes is one fewer
+// way for a bad turn to reach the checkout.
+//
+// `--` ENDS THE OPTIONS, AND IT IS THE SECURITY-CARRYING TOKEN HERE.
+//
+// The prompt is caller text in an argv position, so a prompt that BEGINS WITH A
+// DASH is not a prompt — it is a flag to the harness. Measured in the same pod,
+// all three agree:
+//
+//	dev exec --full-auto "--version"  ->  hanzo dev-exec 0.6.83   (printed, exited)
+//	codex exec "--version"            ->  codex-cli-exec 0.146.1
+//	claude -p "--version"             ->  2.1.223 (Claude Code)
+//
+// None of those ran a task. The reachable end of that is worse than a wasted
+// run: dev's `-c key=value` overrides its own config, including the sandbox
+// policy it was just given, so the text a caller sends could choose how much of
+// the box the model is allowed to touch. With `--` the same string is the
+// prompt and nothing else. python3 -c / node -e need none: there the prompt is
+// the VALUE of a flag, which is already a position no parser reinterprets.
 func argvFor(tool, prompt string) []string {
 	switch strings.TrimSpace(tool) {
 	case "claude":
-		return []string{"claude", "-p", prompt}
+		return []string{"claude", "-p", "--", prompt}
 	case "codex":
-		return []string{"codex", "exec", prompt}
+		return []string{"codex", "exec", "--", prompt}
 	case "python":
 		return []string{"python3", "-c", prompt}
 	case "node":
 		return []string{"node", "-e", prompt}
 	default:
-		return []string{"dev", "-p", prompt}
+		return []string{"dev", "exec", "--full-auto", "--skip-git-repo-check", "--", prompt}
 	}
 }
 
+// tools is the closed set a caller may name, and it is closed for the reason
+// every other set in this package is: `default:` in the two switches above
+// reads dev, so an unknown name did not fail — it silently ran a DIFFERENT
+// harness than the one that was asked for, and answered as though it had run
+// the right one. A typo is refused at the door instead.
+var tools = map[string]bool{"dev": true, "claude": true, "codex": true, "python": true, "node": true}
+
+// CheckTool refuses a harness we do not carry. Empty is not a request, so it is
+// not an error — it is dev, which is what the whole default path already is.
+func CheckTool(tool string) error {
+	if tool = strings.TrimSpace(tool); tool == "" || tools[tool] {
+		return nil
+	}
+	names := make([]string, 0, len(tools))
+	for t := range tools {
+		names = append(names, t)
+	}
+	sort.Strings(names)
+	return fmt.Errorf("coding: %q is not a harness we run (%s)", tool, strings.Join(names, ", "))
+}
+
 // Run leases a sandbox, does the work in it, and ends the lease.
 //
 // The lease is ended on EVERY exit including a panic-free error path, because a
@@ -108,6 +172,17 @@ func (sandboxRunner) Run(ctx context.Context, org, userID string, req RunRequest
 		}
 	}
 
+	// THE ONE REF THIS RUN MAY WRITE, checked before anything is leased. The name
+	// is cloud's (BranchFor, a pure function of the session), the forge refuses
+	// anything else independently (apps/git/refpolicy.go), and this is the third
+	// statement of the same rule at the only other place that could break it — the
+	// process holding the credential. A run handed `main` stops here rather than
+	// discovering at push time that it was never allowed.
+	branch := strings.TrimSpace(req.Branch)
+	if strings.TrimSpace(req.CloneURL) != "" && !strings.HasPrefix(branch, agentPrefix) {
+		return RunResult{}, fmt.Errorf("coding: %q is not an agent branch", branch)
+	}
+
 	class := classFor(req.Tool, req.Desktop)
 	ttl := req.RunTimeoutSeconds
 	if ttl <= 0 {
@@ -121,8 +196,30 @@ func (sandboxRunner) Run(ctx context.Context, org, userID string, req RunRequest
 	}
 
 	step("lease", "leasing a "+class+" sandbox", "running")
+	// NO PROJECT, and that is the whole of it.
+	//
+	// A project NAMES A DISK. apps/sandbox keys one 20Gi PVC per (org, project) so a
+	// second lease on the same project finds the first disk and a checkout survives
+	// between sessions — which is right, and which is why a disk is KEPT unless
+	// somebody explicitly purges it.
+	//
+	// This run used to pass its SESSION id as the project. A session is opened once
+	// per dispatch and never reopened, so that name could not be asked for twice:
+	// every run minted a disk that was by construction unreusable, and the
+	// deliberate keep-by-default then made it immortal. 20Gi a run, and DO bills a
+	// volume whether or not anything is attached to it.
+	//
+	// The fix is not to purge harder. A purge on the way out still strands the disk
+	// of any run killed, OOM'd, or evicted between the lease and the goodbye. The
+	// fix is to stop asking for the wrong primitive: a run wants scratch space that
+	// dies with its pod, and that is EXACTLY what an emptyDir is — which is what
+	// runtime.go mounts whenever there is no volume. Its lifetime is the pod's by
+	// construction, so there is nothing to remember to delete and nothing to leak.
+	//
+	// The work does not live on that disk anyway: deliver() commits it and pushes it
+	// to the run's ref, which is the only reason anything survives a run at all.
 	leased, err := plane.Ask[plane.LeaseIn, plane.Leased](ctx, "sandboxes", plane.SandboxLease,
-		&plane.LeaseIn{Class: class, Project: req.SessionID, TTLSec: ttl})
+		&plane.LeaseIn{Class: class, TTLSec: ttl})
 	if err != nil {
 		return RunResult{}, fmt.Errorf("coding: lease sandbox: %w", err)
 	}
@@ -131,6 +228,12 @@ func (sandboxRunner) Run(ctx context.Context, org, userID string, req RunRequest
 	}
 	id := leased.ID
 
+	// THE SANDBOX'S ID IS SAID OUT LOUD, because it is the handle for the only two
+	// things a person watching a run can do to it: stop the work (stop_run) or
+	// release it (end_sandbox). A run that never named its sandbox could be watched
+	// and not touched.
+	step("leased", "sandbox "+id+" ("+class+")", "running")
+
 	// END ON EVERY PATH. A detached context because the caller's may already be
 	// cancelled by the time we unwind — the lease still has to be released, and
 	// releasing it is not the caller's deadline to spend.
@@ -139,60 +242,311 @@ func (sandboxRunner) Run(ctx context.Context, org, userID string, req RunRequest
 		defer cancel()
 		_, _ = plane.Ask[plane.EndIn, struct{}](end, "sandboxes", plane.SandboxEnd,
 			&plane.EndIn{ID: id})
+		// Said AFTER the lease is actually gone, so "ended" means the pod is
+		// released and not that we intended to release it.
+		step("ended", "released sandbox "+id, "running")
 	}()
 
 	// The checkout, when there is one. No repo means no clone and no credential —
 	// the request shape already guarantees the second (CredToken must be empty
 	// when CloneURL is), so there is nothing to strip here.
+	in := sandbox{ctx: ctx, id: id, ttl: ttl, session: req.SessionID,
+		token: req.CredToken, basic: basicOf(req)}
+	base := ""
 	if u := strings.TrimSpace(req.CloneURL); u != "" {
 		step("clone", "cloning "+u, "running")
-		if _, err := runIn(ctx, id, cloneArgv(req), ttl); err != nil {
-			return RunResult{}, fmt.Errorf("coding: clone: %w", err)
+		// The clone narrates into the session like everything else. It is the step
+		// that most often hangs — a big repo, a slow forge — and a watcher seeing
+		// git count objects knows the difference between slow and stuck.
+		if _, err := in.do(cloneArgv(req), "clone"); err != nil {
+			return RunResult{}, err
+		}
+		// The run's branch exists BEFORE the first edit, so a tool that commits for
+		// itself commits onto the run's branch and never onto the base.
+		if _, err := in.do([]string{"git", "switch", "-c", branch}, "branch"); err != nil {
+			return RunResult{}, err
+		}
+		var err error
+		if base, err = in.tip(); err != nil {
+			return RunResult{}, err
 		}
 	}
 
 	step(req.Tool, "running the task", "running")
-	ran, err := runIn(ctx, id, argvFor(req.Tool, req.Prompt), ttl)
+	// The credential the agent answers the gateway with. Absent on a deployment
+	// that holds no machine identity, and then the run proceeds without one and
+	// fails at the model call with the harness's own message — which is a truer
+	// report than refusing here would be, because everything up to that point
+	// (lease, clone, branch) still happened and is still worth showing.
+	cred, err := key(ctx)
+	if err != nil {
+		return RunResult{}, fmt.Errorf("coding: %w", err)
+	}
+	argv, stdin := argvFor(req.Tool, req.Prompt), ""
+	if cred != "" {
+		argv, stdin = keyed(argv), cred+"\n"
+	}
+	ran, err := runIn(ctx, id, argv, ttl, req.SessionID, stdin)
 	if err != nil {
 		return RunResult{}, fmt.Errorf("coding: run: %w", err)
 	}
 
+	// Branch is deliberately NOT reported. coding.Run does not read it — the branch
+	// is the one cloud issued, and a sandbox that answers a different one is
+	// reporting something it was never asked — so filling it in would only offer a
+	// value nobody may trust.
 	out := RunResult{
 		OK:      ran.ExitCode == 0,
-		LogTail: tail(ran.Stdout, ran.Stderr),
+		LogTail: in.scrub(tail(ran.Stdout, ran.Stderr)),
 	}
 	if !out.OK {
+		// A tool that failed produced nothing anyone should review, so its checkout
+		// stays here and dies with the lease. Pushing it would file work against a
+		// branch whose own run says it did not finish.
 		out.Error = fmt.Sprintf("exit %d", ran.ExitCode)
+		step("done", "finished", statusOf(false))
+		return out, nil
+	}
+	if base != "" {
+		if err := in.deliver(req, branch, base, &out, step); err != nil {
+			return RunResult{}, err
+		}
 	}
 	step("done", "finished", statusOf(out.OK))
 	return out, nil
 }
 
-// cloneArgv checks out the repo with the credential in the URL rather than in a
-// config file, so nothing survives the process that used it. The sandbox is
-// per-run and reaped, but a token written to .git/config would still be readable
-// by every later step of the same run, which is a wider window than the clone.
-func cloneArgv(req RunRequest) []string {
-	url := req.CloneURL
-	if req.CredToken != "" {
-		user := req.CredUser
-		if user == "" {
-			user = "x-access-token"
-		}
-		if rest, ok := strings.CutPrefix(url, "https://"); ok {
-			url = "https://" + user + ":" + req.CredToken + "@" + rest
-		}
+// ── the work leaves the sandbox ──────────────────────────────────────────────
+//
+// Everything above is how a run HAPPENS; this is how it SURVIVES. Without it the
+// rest is a model talking to itself: the sandbox is per-run and reaped, so an
+// edit that stays in the checkout is deleted by the lease ending. The Runner did
+// lease → clone → run → end and threw the work away.
+//
+// It is four facts and one rule.
+//
+//	the branch   is the one cloud issued for the session, created before the
+//	             first edit.
+//	the change   is the difference between two object ids — what we checked out
+//	             and what is there now. Not a status parse and not a guess: it is
+//	             the only reading that stays true both when a tool leaves the tree
+//	             dirty and when it commits for itself, and a run that reports "no
+//	             changes" because it never looked is the failure this file was.
+//	the tip      is what rev-parse says AFTER the commit, so the sha we report is
+//	             one that exists.
+//	the diffstat is measured over exactly that range, so what a reviewer reads and
+//	             what was pushed are the same thing.
+//
+//	THE RULE: the credential is applied to ONE INVOCATION and never recorded.
+
+// agentPrefix is the machine namespace, spelled as coding's own BranchFor builds
+// it. apps/git states the same rule over full refs (refpolicy.go agentRefPrefix)
+// and the two are deliberately independent: this one keeps an honest run inside
+// its lane, that one keeps a compromised one there.
+const agentPrefix = "agent/"
+
+// Who the commit is by. An agent is not a person and does not borrow one: the
+// author of a run's commit is the run's harness, and the human who asked for it
+// is on the PR and in the session.
+const (
+	authorName  = "hanzo-agent"
+	authorEmail = "agent@hanzo.ai"
+)
+
+// sandbox is one leased sandbox, addressed. It carries what every command of a
+// run needs — where, for how long, who is watching, and the grant that must never
+// come back out — so no call site repeats them and no error path can forget the
+// last one.
+type sandbox struct {
+	ctx     context.Context
+	id      string
+	ttl     int
+	session string
+	// Both spellings of the grant, because both exist: the token as the forge
+	// issued it, and the base64 git presents it as. Scrubbing one and not the
+	// other would leave the credential in the output in the only form that
+	// actually travelled.
+	token, basic string
+}
+
+// deliver commits what the tool left behind, pushes it to the run's ref, and
+// fills in what actually happened.
+//
+// The commit's EXIT CODE IS NOT READ. `git commit` with nothing staged fails, and
+// that failure is indistinguishable from a tool that had already committed — so
+// the question is answered by the sha instead, which is a fact rather than an
+// opinion about one. When the tip has not moved, nothing changed, and nothing is
+// pushed: a PR against a branch with no commits teaches a reviewer to ignore the
+// agent, which is worse than the missing PR it replaces.
+func (in sandbox) deliver(req RunRequest, branch, base string, out *RunResult, step func(name, msg, status string)) error {
+	if _, err := in.do([]string{"git", "add", "-A"}, "stage"); err != nil {
+		return err
 	}
-	argv := []string{"git", "clone", "--depth", "1"}
+	// The identity rides the invocation for the same reason the credential does:
+	// `git -c` before a subcommand is not written into the repository's config.
+	if _, err := runIn(in.ctx, in.id, []string{"git",
+		"-c", "user.name=" + authorName, "-c", "user.email=" + authorEmail,
+		"commit", "--quiet", "-m", message(req.Prompt, in.session)}, in.ttl, in.session, ""); err != nil {
+		return fmt.Errorf("coding: commit: %w", err)
+	}
+
+	tip, err := in.tip()
+	if err != nil {
+		return err
+	}
+	if tip == base {
+		step("done", "no changes were needed", "running")
+		return nil // and Changed stays false, which is now a measurement
+	}
+
+	stat, err := in.do([]string{"git", "diff", "--stat", base, tip}, "diff")
+	if err != nil {
+		return err
+	}
+	step("push", "pushing "+branch, "running")
+	if _, err := in.do(pushArgv(req, branch), "push"); err != nil {
+		return err
+	}
+	out.Changed, out.CommitSha, out.Diffstat = true, tip, strings.TrimSpace(stat.Stdout)
+	return nil
+}
+
+// cloneArgv checks out the repo. The credential is a per-invocation rewrite, so
+// what git records as the remote is the plain URL and the grant is gone the
+// moment the process is.
+func cloneArgv(req RunRequest) []string {
+	argv := append(gitAs(req), "clone", "--depth", "1")
 	if b := strings.TrimSpace(req.BaseBranch); b != "" {
 		argv = append(argv, "-b", b)
 	}
-	return append(argv, url, ".")
+	return append(argv, req.CloneURL, ".")
 }
 
-func runIn(ctx context.Context, id string, argv []string, ttl int) (*plane.Ran, error) {
+// pushArgv writes ONE ref, named in full, and never forces.
+//
+// The refspec is explicit rather than `push origin <branch>` because the local
+// name and the remote ref are then the same string we checked and the grant
+// names, with no configured remote and no push.default in between to resolve it
+// into something else.
+func pushArgv(req RunRequest, branch string) []string {
+	return append(gitAs(req), "push", req.CloneURL, "HEAD:refs/heads/"+branch)
+}
+
+// gitAs is `git`, carrying the grant when there is one — as an Authorization
+// header CONFINED to the one URL it is for, applied to this invocation alone.
+//
+// Three properties, each of which cost a bug to learn. All three were verified
+// against git 2.43 with a server that answers like the forge.
+//
+// IT ARRIVES. The credential used to ride the clone URL, and git does not send a
+// URL-embedded credential until it is CHALLENGED: it makes an anonymous request
+// first and waits for 401 WWW-Authenticate. The forge answers a caller it does
+// not recognise with 403 (smart_http.go resolvePackRepo), never 401 — so the
+// grant was never sent at all, and a clone of any private repo could only 403.
+// Presented as a header it is on the FIRST request, which is the same thing
+// apps/git's own wire test does and the reason that test passes.
+//
+// IT DOES NOT PERSIST. `git -c` before the subcommand is not written into the new
+// repository's config — unlike `git clone -c`, which is — so the checkout the
+// model then edits holds no credential. The URL form wrote one into .git/config
+// verbatim, where every later step of the run could read it, including the step
+// that executes untrusted model output.
+//
+// IT GOES NOWHERE ELSE. The key is scoped to the URL, so git attaches the header
+// to that repository and to nothing else. A bare http.extraHeader is sent to
+// WHATEVER the command reaches — a redirect, another host — and what a repository
+// makes git reach is chosen by whoever wrote the repository.
+//
+// One function, because clone and push are the same act — reach that URL as this
+// bearer — and a second spelling is a second place to get it wrong.
+func gitAs(req RunRequest) []string {
+	if req.CredToken == "" || !strings.HasPrefix(req.CloneURL, "https://") {
+		return []string{"git"}
+	}
+	return []string{"git", "-c", "http." + req.CloneURL + ".extraHeader=Authorization: Basic " + basicOf(req)}
+}
+
+// basicOf is the grant as git presents it: base64 of user:token. The user is
+// ignored by the forge — the password is the whole credential — but basic auth
+// has a shape and it has to be filled.
+func basicOf(req RunRequest) string {
+	user := req.CredUser
+	if user == "" {
+		user = "x-access-token"
+	}
+	return base64.StdEncoding.EncodeToString([]byte(user + ":" + req.CredToken))
+}
+
+// tip reads the checkout's current commit.
+func (in sandbox) tip() (string, error) {
+	ran, err := in.do([]string{"git", "rev-parse", "HEAD"}, "read the tip")
+	if err != nil {
+		return "", err
+	}
+	return strings.TrimSpace(ran.Stdout), nil
+}
+
+// do is a command whose failure is the RUN's failure.
+//
+// runIn answers a non-zero exit as data, deliberately — "the tests failed" and
+// "the sandbox is broken" are different facts. For the commands here they are the
+// same fact: a clone that did not clone or a push that did not push leaves
+// nothing to report, so the exit code is read and the run stops.
+func (in sandbox) do(argv []string, what string) (*plane.Ran, error) {
+	ran, err := runIn(in.ctx, in.id, argv, in.ttl, in.session, "")
+	if err != nil {
+		return nil, fmt.Errorf("coding: %s: %s", what, in.scrub(err.Error()))
+	}
+	if ran.ExitCode != 0 {
+		return nil, fmt.Errorf("coding: %s: exit %d: %s", what, ran.ExitCode,
+			in.scrub(strings.TrimSpace(tail(ran.Stdout, ran.Stderr))))
+	}
+	return ran, nil
+}
+
+// message is what the commit says: the task, and the run that did it.
+//
+// The session is in the trailer because the commit outlives every other record of
+// the run — the Slack thread scrolls away, the sandbox is reaped — and a commit
+// nobody can trace back to the conversation that asked for it is an orphan in
+// somebody's history.
+func message(prompt, session string) string {
+	m := firstLine(prompt)
+	if len(m) > maxTitlePrompt {
+		m = strings.TrimSpace(m[:maxTitlePrompt]) + "…"
+	}
+	if m == "" {
+		m = "agent changes"
+	}
+	return m + "\n\nRun: " + session + "\n"
+}
+
+// scrub takes the grant out of anything that leaves the sandbox.
+//
+// git redacts the userinfo from the URLs it prints — verified, including through
+// an insteadOf rewrite — and this does not rely on that. Everything here becomes a
+// session event, a Slack line, a log field and a span; a credential is one string
+// away from all four, and being sure costs four lines. It hangs off the sandbox
+// rather than sitting loose so that every path out of a command goes through it
+// without anyone having to remember.
+func (in sandbox) scrub(s string) string {
+	if strings.TrimSpace(in.token) == "" {
+		return s
+	}
+	return strings.NewReplacer(in.token, redacted, in.basic, redacted).Replace(s)
+}
+
+const redacted = "[redacted]"
+
+// runIn runs one command in the sandbox and narrates it into the run's session.
+//
+// The session travels with the command rather than the output travelling back
+// here, because the bytes are IN THE SANDBOX and this call does not return until
+// the command is over. Streamed from where they are produced, a twenty-five
+// minute agent edit loop is watchable; collected here, it is a silence.
+func runIn(ctx context.Context, id string, argv []string, ttl int, session, stdin string) (*plane.Ran, error) {
 	ran, err := plane.Ask[plane.RunIn, plane.Ran](ctx, "sandboxes", plane.SandboxRun,
-		&plane.RunIn{ID: id, Argv: argv, TimeoutSec: ttl})
+		&plane.RunIn{ID: id, Argv: argv, TimeoutSec: ttl, Session: session, Stdin: stdin})
 	if err != nil {
 		return nil, err
 	}

apps/coding/sandboxrunner_test.go

@@ -0,0 +1,506 @@
+package coding
+
+// sandboxrunner_test.go drives the REAL sandboxRunner against a fake sandbox on a
+// real socket, and asserts the exact sequence of commands a run sends into the
+// pod.
+//
+// That sequence IS the product. A run whose edits never leave the checkout is a
+// model talking to itself: the sandbox is reaped when the lease ends and the work
+// dies with it. So the assertions here are about commands and not about a return
+// value — a runner can be made to answer Changed:true with no test noticing that
+// nothing was ever pushed.
+//
+// The worse of the two failures is the FALSE POSITIVE. A run that reports "no
+// changes" when it changed something is a lost afternoon; a run that reports
+// changes when it made none files a PR against an empty branch, and a reviewer
+// learns to distrust every PR the agent opens. Both are covered, and the
+// no-changes case is first.
+
+import (
+	"context"
+	"encoding/base64"
+	"strings"
+	"sync"
+	"testing"
+
+	"github.com/hanzoai/cloud/plane"
+	"github.com/zap-proto/zip"
+)
+
+// theGrant is the push credential a run holds. Spelled once so every assertion
+// about where it may and may not appear is asking about the same string.
+const theGrant = "hgg_LIVEGRANTVALUE"
+
+const (
+	cleanURL = "https://git.test/v1/git/acme/api.git"
+	theSHA   = "1111111111111111111111111111111111111111"
+	newSHA   = "2222222222222222222222222222222222222222"
+)
+
+// pod is a fake sandbox that records every command and answers whatever the test
+// scripts. It records the argv VERBATIM, because the questions worth asking of
+// this file are all questions about arguments.
+type pod struct {
+	mu     sync.Mutex
+	ran    [][]string
+	answer func(argv []string) plane.Ran
+	// leased is every lease the run asked for, verbatim. A lease is a REQUEST for
+	// resources — a class, a ttl, and whether a disk is wanted — and the only place
+	// that request is visible is here, before it reaches a cluster.
+	leased []plane.LeaseIn
+}
+
+func (p *pod) exec(argv []string) plane.Ran {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	p.ran = append(p.ran, append([]string(nil), argv...))
+	if p.answer == nil {
+		return plane.Ran{}
+	}
+	return p.answer(argv)
+}
+
+// lines renders every command as one string, which is how a human reads a
+// transcript and how these assertions read it too.
+func (p *pod) lines() []string {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	out := make([]string, 0, len(p.ran))
+	for _, argv := range p.ran {
+		out = append(out, strings.Join(argv, " "))
+	}
+	return out
+}
+
+// did returns the first command containing want, and whether there was one.
+func (p *pod) did(want string) (string, bool) {
+	for _, l := range p.lines() {
+		if strings.Contains(l, want) {
+			return l, true
+		}
+	}
+	return "", false
+}
+
+// gitVerb reports whether any command asked git to do verb — matched on the
+// argv, so "push" the verb is not confused with "push" inside a URL or a prompt.
+func (p *pod) gitVerb(verb string) bool {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	for _, argv := range p.ran {
+		for i, a := range argv {
+			if a == verb && i > 0 && argv[0] == "git" {
+				return true
+			}
+		}
+	}
+	return false
+}
+
+// servePod stands up the two peers a sandbox run reaches — the sandbox itself and
+// the prepaid gate — on real unix sockets, so the runner under test is the
+// production one with a production transport.
+func servePod(t *testing.T, p *pod) {
+	t.Helper()
+	t.Setenv("ZIP_RUNTIME_DIR", t.TempDir())
+
+	sandboxes := zip.New(zip.Config{AppName: "sandboxes", DisableStartupMessage: true})
+	zip.Post[plane.LeaseIn, plane.Leased](sandboxes, "/sandbox/lease",
+		func(_ context.Context, in *plane.LeaseIn) (*plane.Leased, error) {
+			p.mu.Lock()
+			p.leased = append(p.leased, *in)
+			p.mu.Unlock()
+			return &plane.Leased{ID: "sbx_1", Class: in.Class, Status: "ready", Workdir: "/work"}, nil
+		}, zip.WithOperationID(plane.SandboxLease))
+	zip.Post[plane.RunIn, plane.Ran](sandboxes, "/sandbox/run",
+		func(_ context.Context, in *plane.RunIn) (*plane.Ran, error) {
+			r := p.exec(in.Argv)
+			return &r, nil
+		}, zip.WithOperationID(plane.SandboxRun))
+	zip.Post[plane.EndIn, struct{}](sandboxes, "/sandbox/end",
+		func(_ context.Context, _ *plane.EndIn) (*struct{}, error) {
+			return &struct{}{}, nil
+		}, zip.WithOperationID(plane.SandboxEnd))
+
+	commerce := zip.New(zip.Config{AppName: "commerce", DisableStartupMessage: true})
+	zip.Post[plane.AuthorizeIn, plane.Verdict](commerce, "/commerce/authorize",
+		func(_ context.Context, _ *plane.AuthorizeIn) (*plane.Verdict, error) {
+			return &plane.Verdict{OK: true}, nil
+		}, zip.WithOperationID(plane.FinanceAuthorize))
+
+	for name, app := range map[string]*zip.App{"sandboxes": sandboxes, "commerce": commerce} {
+		app := app
+		plane.Bind()
+		go func(path string) { _ = app.Listen(path) }(zip.SocketPath(name))
+		t.Cleanup(func() { _ = app.Shutdown() })
+		waitListening(t, name)
+	}
+}
+
+// aRun is the request a real dispatch builds: a checkout, the branch cloud issued
+// for the session, and the grant that may write exactly that one ref.
+func aRun() RunRequest {
+	return RunRequest{
+		CloneURL: cleanURL, BaseBranch: "main", Branch: "agent/abc123def456",
+		Prompt: "fix the flake", SessionID: "sess_abc123def456",
+		RunTimeoutSeconds: 60, CredUser: "x-access-token", CredToken: theGrant,
+	}
+}
+
+// steps collects what the run said out loud, which is what a person watching in
+// Slack actually reads.
+func steps(out *[]string) func(Step) {
+	return func(s Step) { *out = append(*out, s.Step+" "+s.Message+" "+s.Status) }
+}
+
+// A RUN ASKS FOR NO PROJECT, so it leaves no disk behind.
+//
+// A project names a per-(org, project) 20Gi PVC that apps/sandbox deliberately
+// KEEPS when a lease ends, so a checkout survives between sessions. A run used to
+// pass its SESSION id there — a name minted per dispatch and never reopened — so
+// each run created a disk that could never be addressed again and was then kept
+// forever. That is the whole mechanism of the leak, and it is visible only here,
+// in what the lease ASKS for; the run's own result looks identical either way.
+//
+// The assertion is on Project and not on some later cleanup on purpose. A purge on
+// the way out cannot cover a run that is killed, OOM'd, or evicted before it says
+// goodbye, whereas a lease that never asks for a disk cannot strand one. The pod's
+// emptyDir already has exactly the lifetime a run wants.
+func TestSandboxRun_LeasesNoDiskBecauseARunIsNotAProject(t *testing.T) {
+	p := &pod{answer: func(argv []string) plane.Ran {
+		if has(argv, "rev-parse") {
+			return plane.Ran{Stdout: theSHA + "\n"}
+		}
+		return plane.Ran{}
+	}}
+	servePod(t, p)
+
+	if _, err := (sandboxRunner{}).Run(context.Background(), "acme", "u_1", aRun(), nil); err != nil {
+		t.Fatalf("run: %v", err)
+	}
+
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	if len(p.leased) != 1 {
+		t.Fatalf("a run took %d leases, want exactly 1", len(p.leased))
+	}
+	if got := p.leased[0].Project; got != "" {
+		t.Fatalf("the run asked for project %q, which mints a 20Gi disk keyed to a name "+
+			"no later lease can ever reuse — a run must ask for no project at all", got)
+	}
+}
+
+// A run that edited NOTHING says so, and touches no ref. This is the first test
+// because the false PR is the worse failure: a branch with no commits, filed as
+// work, teaches a reviewer to ignore the agent.
+func TestSandboxRun_NoEditsReportNoChangesAndWriteNoRef(t *testing.T) {
+	p := &pod{answer: func(argv []string) plane.Ran {
+		switch {
+		case has(argv, "rev-parse"):
+			return plane.Ran{Stdout: theSHA + "\n"} // the tip never moves
+		case has(argv, "commit"):
+			return plane.Ran{ExitCode: 1, Stdout: "nothing to commit, working tree clean\n"}
+		}
+		return plane.Ran{}
+	}}
+	servePod(t, p)
+
+	var said []string
+	res, err := sandboxRunner{}.Run(context.Background(), "acme", "u_1", aRun(), steps(&said))
+	if err != nil {
+		t.Fatalf("a clean run is not an error: %v", err)
+	}
+	if !res.OK {
+		t.Fatalf("a run that changed nothing still succeeded: %+v", res)
+	}
+	if res.Changed || res.CommitSha != "" || res.Diffstat != "" {
+		t.Fatalf(`"no changes" must mean no changes: %+v`, res)
+	}
+	if p.gitVerb("push") {
+		t.Fatalf("nothing changed and something was pushed:\n%s", strings.Join(p.lines(), "\n"))
+	}
+}
+
+// A run that edited something commits it, pushes it to its own ref, and reports
+// what it actually did — the tip it created and the diffstat it measured.
+func TestSandboxRun_EditsAreCommittedPushedAndReportedHonestly(t *testing.T) {
+	tip := theSHA
+	p := &pod{}
+	p.answer = func(argv []string) plane.Ran {
+		switch {
+		case has(argv, "commit"):
+			tip = newSHA // the commit is what moves it
+			return plane.Ran{}
+		case has(argv, "rev-parse"):
+			return plane.Ran{Stdout: tip + "\n"}
+		case has(argv, "diff"):
+			return plane.Ran{Stdout: " 2 files changed, 9 insertions(+), 1 deletion(-)\n"}
+		}
+		return plane.Ran{}
+	}
+	servePod(t, p)
+
+	var said []string
+	res, err := sandboxRunner{}.Run(context.Background(), "acme", "u_1", aRun(), steps(&said))
+	if err != nil {
+		t.Fatalf("run: %v", err)
+	}
+	if !res.OK || !res.Changed {
+		t.Fatalf("the run changed something and must say so: %+v", res)
+	}
+	if res.CommitSha != newSHA {
+		t.Fatalf("the reported tip is not the one that was created: %q", res.CommitSha)
+	}
+	if !strings.Contains(res.Diffstat, "2 files changed") {
+		t.Fatalf("the diffstat was not measured: %q", res.Diffstat)
+	}
+
+	// THE BRANCH IS THE RUN'S OWN, and it is created before the first edit so
+	// nothing is ever committed onto the base.
+	if _, ok := p.did("switch -c agent/abc123def456"); !ok {
+		t.Fatalf("the run never left the base branch:\n%s", strings.Join(p.lines(), "\n"))
+	}
+	push, ok := p.did(" push ")
+	if !ok {
+		t.Fatalf("nothing was pushed:\n%s", strings.Join(p.lines(), "\n"))
+	}
+	// ONE ref, named in full, never a branch name git could resolve to something
+	// else and never a force.
+	if !strings.HasSuffix(push, "HEAD:refs/heads/agent/abc123def456") {
+		t.Fatalf("the push does not name exactly one ref: %q", push)
+	}
+	for _, forbidden := range []string{"--force", "-f ", "--mirror", "--all", "--delete"} {
+		if strings.Contains(push, forbidden) {
+			t.Fatalf("the push carries %q: %q", forbidden, push)
+		}
+	}
+}
+
+// The branch is the one cloud issued, and the runner will not write anywhere
+// else even when it is handed a name that says otherwise. The forge refuses this
+// too (apps/git/refpolicy.go); neither is the other's backstop.
+func TestSandboxRun_RefusesToWriteOutsideTheAgentNamespace(t *testing.T) {
+	p := &pod{}
+	servePod(t, p)
+
+	req := aRun()
+	req.Branch = "main"
+	r := sandboxRunner{}
+	if _, err := r.Run(context.Background(), "acme", "u_1", req, nil); err == nil {
+		t.Fatal("a run pointed at main must be refused")
+	}
+	if len(p.lines()) != 0 {
+		t.Fatalf("a refused run still did work:\n%s", strings.Join(p.lines(), "\n"))
+	}
+}
+
+// A tool that failed has produced nothing anyone should review, so its checkout
+// stays in the sandbox and dies there.
+func TestSandboxRun_AFailedToolWritesNoRef(t *testing.T) {
+	p := &pod{answer: func(argv []string) plane.Ran {
+		if has(argv, "rev-parse") {
+			return plane.Ran{Stdout: theSHA + "\n"}
+		}
+		if has(argv, "dev") {
+			return plane.Ran{ExitCode: 3, Stderr: "the model gave up\n"}
+		}
+		return plane.Ran{}
+	}}
+	servePod(t, p)
+
+	res, err := sandboxRunner{}.Run(context.Background(), "acme", "u_1", aRun(), nil)
+	if err != nil {
+		t.Fatalf("a failed tool is a reported failure, not a transport error: %v", err)
+	}
+	if res.OK || res.Changed {
+		t.Fatalf("a failed run must not report work: %+v", res)
+	}
+	if p.gitVerb("push") {
+		t.Fatalf("a failed run pushed:\n%s", strings.Join(p.lines(), "\n"))
+	}
+}
+
+// THE GRANT NEVER TOUCHES DISK AND IS NEVER SAID OUT LOUD.
+//
+// It may appear in exactly one place — the git option that applies it to a single
+// invocation — and nowhere else: not as a URL operand (git writes those into
+// .git/config), not in a config write, and not in a step, a log line or the tail
+// a run hands back for a person to read.
+func TestSandboxRun_TheGrantStaysOffDiskAndOutOfWhatIsSaid(t *testing.T) {
+	p := &pod{answer: func(argv []string) plane.Ran {
+		if has(argv, "rev-parse") {
+			return plane.Ran{Stdout: theSHA + "\n"}
+		}
+		return plane.Ran{Stdout: "done\n"}
+	}}
+	servePod(t, p)
+
+	var said []string
+	res, err := sandboxRunner{}.Run(context.Background(), "acme", "u_1", aRun(), steps(&said))
+	if err != nil {
+		t.Fatalf("run: %v", err)
+	}
+
+	// The grant travels as base64, which is how git presents basic auth. Both
+	// spellings are asked about: the literal must appear NOWHERE, and the one
+	// that actually travels must appear only where it is confined.
+	basic := base64.StdEncoding.EncodeToString([]byte("x-access-token:" + theGrant))
+	wantOpt := "http." + cleanURL + ".extraHeader=Authorization: Basic " + basic
+
+	for _, argv := range p.ran {
+		for i, a := range argv {
+			if strings.Contains(a, theGrant) {
+				t.Fatalf("the grant appears verbatim in argv[%d]: %q", i, strings.Join(argv, " "))
+			}
+			if !strings.Contains(a, basic) {
+				continue
+			}
+			// Exactly one shape: an option applied to THIS invocation, scoped to the
+			// ONE url it is for. `git -c` before the subcommand is not written into
+			// the new repository's config, so the checkout the model then edits holds
+			// no credential; the url scope is what stops git attaching it to a
+			// redirect or another host the repository's own contents point at.
+			if i == 0 || argv[i-1] != "-c" || a != wantOpt {
+				t.Fatalf("the credential is in argv[%d] as %q, want the confined option %q", i, a, wantOpt)
+			}
+		}
+		// Nothing may persist it. A `git config` write, a credential helper with a
+		// store behind it, or a redirect into a file each survive the command.
+		line := strings.Join(argv, " ")
+		for _, persists := range []string{"git config", "credential.helper=store", "credential.helper=cache", ".git/config"} {
+			if strings.Contains(line, persists) {
+				t.Fatalf("a command persists a credential: %q", line)
+			}
+		}
+	}
+
+	// The URL OPERAND — what git records as the remote — is the clean one.
+	for _, argv := range p.ran {
+		for _, a := range argv {
+			if strings.HasPrefix(a, "https://") && strings.Contains(a, "@") {
+				t.Fatalf("a credential rode a URL operand: %q", a)
+			}
+		}
+	}
+
+	// Nothing a person reads carries it, in either spelling.
+	for _, s := range said {
+		if strings.Contains(s, theGrant) || strings.Contains(s, basic) {
+			t.Fatalf("the grant was narrated: %q", s)
+		}
+	}
+	if strings.Contains(res.LogTail, theGrant) || strings.Contains(res.LogTail, basic) {
+		t.Fatalf("the grant came back in the log tail: %q", res.LogTail)
+	}
+}
+
+// A command's output is scrubbed of BOTH spellings on the way out, so a git that
+// ever echoed what it was given cannot put a live credential in a session event,
+// a span or a Slack thread.
+func TestSandboxRun_AnEchoedCredentialIsScrubbedOnTheWayOut(t *testing.T) {
+	basic := base64.StdEncoding.EncodeToString([]byte("x-access-token:" + theGrant))
+	p := &pod{answer: func(argv []string) plane.Ran {
+		if has(argv, "rev-parse") {
+			return plane.Ran{Stdout: theSHA + "\n"}
+		}
+		if has(argv, "dev") {
+			return plane.Ran{Stdout: "used " + theGrant + " and header " + basic + "\n"}
+		}
+		return plane.Ran{}
+	}}
+	servePod(t, p)
+
+	res, err := sandboxRunner{}.Run(context.Background(), "acme", "u_1", aRun(), nil)
+	if err != nil {
+		t.Fatalf("run: %v", err)
+	}
+	if strings.Contains(res.LogTail, theGrant) || strings.Contains(res.LogTail, basic) {
+		t.Fatalf("an echoed credential survived: %q", res.LogTail)
+	}
+	if !strings.Contains(res.LogTail, "[redacted]") {
+		t.Fatalf("nothing was scrubbed: %q", res.LogTail)
+	}
+}
+
+func has(argv []string, want string) bool {
+	for _, a := range argv {
+		if a == want {
+			return true
+		}
+	}
+	return false
+}
+
+// The three properties of argvFor that a run actually dies of, each pinned
+// against what the harness was MEASURED to do in a dev-class pod
+// (oci.hanzo.ai/hanzoai/sandbox 1.0.1: dev 0.6.91, claude 2.1.223, codex 0.146.1).
+func TestArgvForNonInteractiveVerb(t *testing.T) {
+	// `-p` is --profile to dev, not "prompt". `dev -p "<task>"` answered
+	// `config profile "<task>" not found` and exited before asking a model
+	// anything — which is what every coding run cloud dispatched used to do.
+	argv := argvFor("", "add a README")
+	if len(argv) < 2 || argv[0] != "dev" || argv[1] != "exec" {
+		t.Fatalf("the default harness must start with `dev exec`, got %q", argv)
+	}
+}
+
+func TestArgvForEndsOptionsBeforeThePrompt(t *testing.T) {
+	// A prompt is caller text in an argv position. Without `--`, a prompt that
+	// begins with a dash is a FLAG: all three agent harnesses printed their own
+	// version and exited when handed "--version" as the task.
+	for _, tool := range []string{"", "dev", "claude", "codex"} {
+		argv := argvFor(tool, "--version")
+		last := argv[len(argv)-1]
+		if last != "--version" {
+			t.Fatalf("%s: prompt must be last, got %q", tool, argv)
+		}
+		if argv[len(argv)-2] != "--" {
+			t.Fatalf("%s: prompt must be preceded by `--` or it parses as a flag, got %q", tool, argv)
+		}
+	}
+	// python3 -c / node -e take the prompt as a flag VALUE, which no parser
+	// reinterprets, so they neither need nor take a separator.
+	for tool, flag := range map[string]string{"python": "-c", "node": "-e"} {
+		argv := argvFor(tool, "--version")
+		if len(argv) != 3 || argv[1] != flag || argv[2] != "--version" {
+			t.Fatalf("%s: want [%s %s --version], got %q", tool, argv[0], flag, argv)
+		}
+	}
+}
+
+func TestCheckToolRefusesAHarnessWeDoNotCarry(t *testing.T) {
+	// Empty is dev, which is the whole default path, so it is not an error.
+	if err := CheckTool(""); err != nil {
+		t.Fatalf("empty tool must be accepted as dev: %v", err)
+	}
+	for _, ok := range []string{"dev", "claude", "codex", "python", "node"} {
+		if err := CheckTool(ok); err != nil {
+			t.Fatalf("%s is carried by the dev image: %v", ok, err)
+		}
+	}
+	// Without this, `default:` in classFor and argvFor read a typo as dev and
+	// answered as though the asked-for harness had run.
+	if err := CheckTool("clade"); err == nil {
+		t.Fatal("an unknown harness must be refused, not silently run as dev")
+	}
+}
+
+func TestClassForCarriesTheBinaryTheToolNeeds(t *testing.T) {
+	// Measured `command -v` in each published 1.0.1 class: exec carries dev,
+	// hanzo and hanzo-mcp but NOT claude/codex/gh; dev and desktop carry all.
+	for tool, want := range map[string]string{
+		"": "dev", "dev": "dev", "claude": "dev", "codex": "dev",
+		"python": "exec", "node": "exec",
+	} {
+		if got := classFor(tool, false); got != want {
+			t.Fatalf("classFor(%q): want %s, got %s", tool, want, got)
+		}
+	}
+	// A screen is an image variant, so it wins over every harness — desktop is
+	// dev plus an X server and therefore carries every binary dev does.
+	for _, tool := range []string{"", "dev", "claude", "python"} {
+		if got := classFor(tool, true); got != "desktop" {
+			t.Fatalf("classFor(%q, desktop): want desktop, got %s", tool, got)
+		}
+	}
+}

apps/coding/start.go

@@ -162,6 +162,9 @@ func Start(ctx context.Context, org string, in plane.CodingStartIn, log func(msg
 	if project != "" && !RepoRE.MatchString(project) {
 		return Accepted{}, fmt.Errorf("coding: %q is not a project name", project)
 	}
+	if err := CheckTool(in.Tool); err != nil {
+		return Accepted{}, err
+	}
 	if len(prompt) > maxPromptLen {
 		prompt = prompt[:maxPromptLen]
 	}
@@ -180,7 +183,7 @@ func Start(ctx context.Context, org string, in plane.CodingStartIn, log func(msg
 
 	req := Req{
 		Org: org, UserID: subject, AgentRef: in.AgentRef, Repo: repo,
-		Project: project, Base: base,
+		Project: project, Base: base, Tool: strings.TrimSpace(in.Tool), Desktop: in.Desktop,
 		Prompt: prompt, TimeoutSeconds: in.TimeoutSeconds, TargetID: strings.TrimSpace(in.TargetID),
 	}
 

apps/coding/start_tenant_test.go

@@ -188,6 +188,27 @@ func TestEveryTerminalGetsTheLastWord(t *testing.T) {
 	}
 }
 
+// The last word carries the ADDRESS. A run that pushed a branch and filed a PR
+// is finished work, and a thread that names it without saying where to read it
+// leaves the person who asked to go hunting for their own change.
+func TestTheLastWordSaysWhereToReadIt(t *testing.T) {
+	line, _ := renderLine(kindStatus, []byte(
+		`{"status":"done","changed":true,"branch":"agent/abc","pr":"ENG-1","url":"https://github.com/hanzo-inc/api/pull/7"}`))
+	if !strings.Contains(line, "https://github.com/hanzo-inc/api/pull/7") {
+		t.Fatalf("the address never reached the thread: %q", line)
+	}
+	// BARE, so Slack links it on its own. A `<url|text>` element is the one piece
+	// of mrkdwn that carries an arbitrary destination, and nothing a run emits may
+	// construct one.
+	if strings.Contains(line, "<") || strings.Contains(line, ">") {
+		t.Fatalf("the address was wrapped in a link element: %q", line)
+	}
+	// And a run with no address still gets its last word.
+	if l, _ := renderLine(kindStatus, []byte(`{"status":"done","changed":true,"branch":"agent/abc"}`)); !strings.Contains(l, "agent/abc") {
+		t.Fatalf("a run without an address lost its ending: %q", l)
+	}
+}
+
 // The sink reads untrusted payloads. A malformed one must be ignored, never
 // panic a run that has already pushed a branch.
 func TestProgressSurvivesAHostilePayload(t *testing.T) {

apps/coding/task.go

@@ -1,161 +0,0 @@
-package coding
-
-import (
-	"context"
-	"encoding/json"
-	"errors"
-	"fmt"
-	"os"
-	"strings"
-
-	"github.com/hanzoai/cloud/apps/bots"
-)
-
-// task.go is coding's WIRE CONTRACT with the bot runtime — the stub behind the
-// Runner seam. It says WHAT coding asks the runtime to do (run one coding job)
-// and how a progress line is shaped; how the bytes get there is runtime's problem.
-//
-// This file is the ONE place in clients/coding that knows the runtime exists.
-// coding.go (the orchestrator) is pure and never sees it.
-//
-// The runtime answers a stream: one message per step/log as the job progresses
-// (clone → dev exec → commit → push), then a terminal result (or error). Coding
-// mirrors every step into the agent session live, so the run is watchable at
-// GET /v1/agents/sessions/:id/stream, and returns the terminal outcome.
-//
-// CREDENTIAL CUSTODY: the per-org agent git credential travels in the request
-// BODY (never a URL, never argv, never a log). That is why the call declares
-// Secret — the transport then refuses to carry it over a cleartext hop.
-
-// taskOp addresses the runtime's sandbox-run operation.
-const taskOp = "/v1/coding-tasks"
-
-// sandboxURL is WHERE A RUN GOES, and it is deliberately not the bot address.
-//
-// A sandbox is not the bot. Coding, deep research and bare exec all want the
-// same thing — a computer to run something in — and none of them wants the
-// service that runs Slack channels. BOT_GATEWAY_URL is the right name for bot
-// traffic and stays that; this is the name for a sandbox.
-//
-// The old name is accepted for ONE release so a deploy cannot half-land, then it
-// is deleted. Not a permanent alias: two live names for one address is how the
-// two ends stop agreeing about where a run went, with nothing in a log to say so.
-// Empty here means the transport's own default, which today is the same pod.
-func sandboxURL() string {
-	for _, k := range []string{"SANDBOX_URL", "BOT_GATEWAY_URL"} {
-		if v := strings.TrimSpace(os.Getenv(k)); v != "" {
-			return v
-		}
-	}
-	return ""
-}
-
-// credential is the per-org agent git credential the sandbox presents to native
-// git. Token is the secret (an sk- key); Username is the basic-auth user label.
-// Encoded into the request body only — never logged.
-type credential struct {
-	Username string `json:"username"`
-	Token    string `json:"token"`
-}
-
-// taskRequest is the cloud→runtime body for one sandbox run.
-//
-// EVERY GIT FIELD IS omitempty, AND THAT IS THE CONTRACT, NOT A TIDINESS
-// PREFERENCE. A run with no repo must put NO credential on the wire at all —
-// not an empty one. `credential` is a pointer for the same reason: a value type
-// would always marshal, so "no repo" would still ship a `credential` object and
-// the runtime could not tell an absent grant from a blank one.
-type taskRequest struct {
-	Prompt            string      `json:"prompt"`               // the task
-	Tool              string      `json:"tool,omitempty"`       // dev|claude|codex|python|node (default dev)
-	Desktop           bool        `json:"desktop,omitempty"`    // select the xvfb image variant
-	SessionID         string      `json:"sessionId"`            // cloud session id (correlation)
-	RunTimeoutSeconds int         `json:"runTimeoutSeconds"`    // sandbox run budget
-	CloneURL          string      `json:"cloneUrl,omitempty"`   // https://<domain>/v1/git/<org>/<repo>.git
-	BaseBranch        string      `json:"baseBranch,omitempty"` // branch to start from (default repo default)
-	Branch            string      `json:"branch,omitempty"`     // branch to create + push (e.g. agent/<sessionid>)
-	Credential        *credential `json:"credential,omitempty"` // agent git credential (write-only); nil when there is no repo
-}
-
-// message is the discriminated shape of one streamed line: step/log while the job
-// runs, result/error to end it.
-type message struct {
-	Type      string `json:"type"` // step | log | result | error
-	Step      string `json:"step,omitempty"`
-	Message   string `json:"message,omitempty"`
-	Status    string `json:"status,omitempty"`
-	Branch    string `json:"branch,omitempty"`
-	CommitSha string `json:"commitSha,omitempty"`
-	Diffstat  string `json:"diffstat,omitempty"`
-	Changed   bool   `json:"changed,omitempty"`
-	OK        bool   `json:"ok,omitempty"`
-	LogTail   string `json:"logTail,omitempty"`
-}
-
-// runner is the Runner seam over the real runtime. Its fake twin in coding_test.go
-// is what the orchestrator is tested against.
-type runner struct{}
-
-// Run hands one coding job to the runtime and streams its progress, invoking
-// onStep for each line as it arrives, then returns the terminal result. A
-// transport failure, or a stream that ends without a terminal message, is an
-// error — no partial success is fabricated. org/userID are the tenant context the
-// runtime trusts AFTER its own bearer gate.
-func (runner) Run(ctx context.Context, org, userID string, req RunRequest, onStep func(Step)) (RunResult, error) {
-	var out RunResult
-	var terminal bool
-	body := taskRequest{
-		Prompt: req.Prompt, Tool: req.Tool, Desktop: req.Desktop,
-		SessionID: req.SessionID, RunTimeoutSeconds: req.RunTimeoutSeconds,
-	}
-	// The git half travels together or not at all — there is no path here that
-	// puts a credential on the wire without the repo it belongs to. A caller
-	// that supplies one anyway is REFUSED rather than quietly trimmed: silently
-	// dropping a secret hides the bug that minted it, and the runtime says the
-	// same thing at its own boundary, so the two ends agree.
-	if req.CloneURL == "" && (req.CredToken != "" || req.CredUser != "") {
-		return out, errors.New("coding: a credential without a repo cannot be used, and must not be sent")
-	}
-	if req.CloneURL != "" {
-		body.CloneURL, body.BaseBranch, body.Branch = req.CloneURL, req.BaseBranch, req.Branch
-		body.Credential = &credential{Username: req.CredUser, Token: req.CredToken}
-	}
-	err := bots.Stream(ctx, bots.Call{
-		Op:   taskOp,
-		Org:  org,
-		User: userID,
-		Base: sandboxURL(),
-		Body: body,
-		// Secret ONLY when the body actually carries the org's git credential.
-		// A run with no repo has no secret to protect, so it must not be refused
-		// by the cleartext guard that exists to protect one.
-		Secret: body.Credential != nil,
-	}, func(msg []byte) {
-		var m message
-		if json.Unmarshal(msg, &m) != nil {
-			return // skip a malformed message rather than abort the whole run
-		}
-		switch m.Type {
-		case "result":
-			out = RunResult{
-				Branch: m.Branch, CommitSha: m.CommitSha, Diffstat: m.Diffstat,
-				Changed: m.Changed, OK: m.OK, LogTail: m.LogTail,
-			}
-			terminal = true
-		case "error":
-			out = RunResult{OK: false, LogTail: m.LogTail, Error: nonEmpty(m.Message, "coding task failed")}
-			terminal = true
-		default: // step | log — mirror live
-			if onStep != nil {
-				onStep(Step{Type: m.Type, Step: m.Step, Message: m.Message, Status: m.Status})
-			}
-		}
-	})
-	if err != nil {
-		return RunResult{}, fmt.Errorf("coding: run task: %w", err)
-	}
-	if !terminal {
-		return RunResult{}, fmt.Errorf("coding: stream ended without a result")
-	}
-	return out, nil
-}

apps/coding/task_test.go

@@ -1,206 +0,0 @@
-package coding
-
-import (
-	"context"
-	"encoding/json"
-	"io"
-	"net/http"
-	"net/http/httptest"
-	"strings"
-	"testing"
-)
-
-// These pin coding's wire contract with the runtime through the REAL transport
-// (apps/bots' transport) against a stub server — so the credential-custody and
-// fail-closed properties are proven end to end over the seam, not against a fake
-// of it.
-
-// ndjsonServer streams the given lines as application/x-ndjson and records the
-// request it received, so a test can assert the wire contract (path, headers,
-// body) AND that the credential travels only in the body.
-func ndjsonServer(t *testing.T, lines []string, capture *http.Request, capBody *[]byte) *httptest.Server {
-	t.Helper()
-	return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		if capture != nil {
-			*capture = *r
-		}
-		if capBody != nil {
-			b, _ := io.ReadAll(r.Body)
-			*capBody = b
-		}
-		w.Header().Set("Content-Type", "application/x-ndjson")
-		w.WriteHeader(http.StatusOK)
-		for _, ln := range lines {
-			_, _ = io.WriteString(w, ln+"\n")
-		}
-	}))
-}
-
-func TestTask_StreamsStepsAndResult(t *testing.T) {
-	lines := []string{
-		`{"type":"step","step":"clone","status":"ok"}`,
-		`{"type":"log","message":"editing handler.go"}`,
-		`{"type":"step","step":"push","status":"ok"}`,
-		`{"type":"result","branch":"agent/x","commitSha":"deadbeef","diffstat":"1 file changed","changed":true,"ok":true,"logTail":"done"}`,
-	}
-	var gotReq http.Request
-	var gotBody []byte
-	srv := ndjsonServer(t, lines, &gotReq, &gotBody)
-	defer srv.Close()
-	t.Setenv("BOT_GATEWAY_URL", srv.URL)
-	t.Setenv("BOT_GATEWAY_ALLOW_PLAINTEXT", "1") // httptest is http://; assert-plaintext guard tested separately
-	t.Setenv("BOT_GATEWAY_TOKEN", "svc-token-xyz")
-
-	var steps []Step
-	res, err := runner{}.Run(context.Background(), "acme", "u-1", RunRequest{
-		CloneURL: "https://git.test/v1/git/acme/api.git", Branch: "agent/x", Prompt: "fix",
-		CredUser: "x-access-token", CredToken: "sk-SECRETtoken",
-	}, func(s Step) { steps = append(steps, s) })
-	if err != nil {
-		t.Fatalf("run: %v", err)
-	}
-
-	// Result parsed.
-	if !res.OK || !res.Changed || res.Branch != "agent/x" || res.CommitSha != "deadbeef" {
-		t.Fatalf("result wrong: %+v", res)
-	}
-	// Steps mirrored (2 steps + 1 log; result/terminal not delivered as a step).
-	if len(steps) != 3 {
-		t.Fatalf("want 3 progress steps, got %d: %+v", len(steps), steps)
-	}
-	if steps[0].Type != "step" || steps[0].Step != "clone" || steps[1].Type != "log" {
-		t.Fatalf("step vocabulary wrong: %+v", steps)
-	}
-
-	// Wire contract: POST /v1/coding-tasks, identity + bearer headers set.
-	if gotReq.Method != http.MethodPost || !strings.HasSuffix(gotReq.URL.Path, "/v1/coding-tasks") {
-		t.Fatalf("bad request line: %s %s", gotReq.Method, gotReq.URL.Path)
-	}
-	if gotReq.Header.Get("X-Org-Id") != "acme" || gotReq.Header.Get("X-User-Id") != "u-1" {
-		t.Fatalf("identity headers wrong: %v", gotReq.Header)
-	}
-	if gotReq.Header.Get("Authorization") != "Bearer svc-token-xyz" {
-		t.Fatalf("service bearer missing/wrong: %q", gotReq.Header.Get("Authorization"))
-	}
-	// The credential travels in the BODY only (never a URL/header).
-	if strings.Contains(gotReq.URL.RawQuery, "SECRETtoken") || strings.Contains(gotReq.Header.Get("Authorization"), "SECRETtoken") {
-		t.Fatal("credential leaked onto URL/header")
-	}
-	var body taskRequest
-	if err := json.Unmarshal(gotBody, &body); err != nil {
-		t.Fatalf("body decode: %v", err)
-	}
-	if body.Credential.Token != "sk-SECRETtoken" || body.Credential.Username != "x-access-token" {
-		t.Fatalf("credential not carried in body: %+v", body.Credential)
-	}
-}
-
-func TestTask_ErrorLine(t *testing.T) {
-	srv := ndjsonServer(t, []string{`{"type":"error","message":"dev exec failed","logTail":"boom"}`}, nil, nil)
-	defer srv.Close()
-	t.Setenv("BOT_GATEWAY_URL", srv.URL)
-	t.Setenv("BOT_GATEWAY_ALLOW_PLAINTEXT", "1") // httptest is http://; assert-plaintext guard tested separately
-
-	res, err := runner{}.Run(context.Background(), "acme", "u", RunRequest{}, nil)
-	if err != nil {
-		t.Fatalf("a clean error line is a terminal result, not a transport error: %v", err)
-	}
-	if res.OK || res.Error != "dev exec failed" || res.LogTail != "boom" {
-		t.Fatalf("error result wrong: %+v", res)
-	}
-}
-
-func TestTask_Non2xxIsError(t *testing.T) {
-	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
-		http.Error(w, "unauthorized", http.StatusUnauthorized)
-	}))
-	defer srv.Close()
-	t.Setenv("BOT_GATEWAY_URL", srv.URL)
-	t.Setenv("BOT_GATEWAY_ALLOW_PLAINTEXT", "1") // httptest is http://; assert-plaintext guard tested separately
-	if _, err := (runner{}).Run(context.Background(), "acme", "u", RunRequest{}, nil); err == nil {
-		t.Fatal("a 401 must be an error")
-	}
-}
-
-func TestTask_NoTerminalIsError(t *testing.T) {
-	srv := ndjsonServer(t, []string{`{"type":"step","step":"clone"}`}, nil, nil)
-	defer srv.Close()
-	t.Setenv("BOT_GATEWAY_URL", srv.URL)
-	t.Setenv("BOT_GATEWAY_ALLOW_PLAINTEXT", "1") // httptest is http://; assert-plaintext guard tested separately
-	if _, err := (runner{}).Run(context.Background(), "acme", "u", RunRequest{}, nil); err == nil {
-		t.Fatal("a stream with no result/error line must be an error (no fabricated success)")
-	}
-}
-
-func TestTask_RefusesCleartextByDefault(t *testing.T) {
-	// The credential-bearing POST must not go cleartext without an explicit mesh
-	// opt-in: an http target with BOT_GATEWAY_ALLOW_PLAINTEXT unset fails closed and
-	// never dials.
-	srv := ndjsonServer(t, []string{`{"type":"result","ok":true}`}, nil, nil)
-	defer srv.Close()
-	t.Setenv("BOT_GATEWAY_URL", srv.URL) // http://
-	t.Setenv("BOT_GATEWAY_ALLOW_PLAINTEXT", "")
-	// The repo is what makes this POST credential-bearing: with no CloneURL the
-	// credential never reaches the wire, so there would be no secret for the
-	// cleartext guard to protect and nothing for this test to prove.
-	_, err := runner{}.Run(context.Background(), "acme", "u", RunRequest{
-		CloneURL: "https://git.hanzo.ai/v1/git/acme/api.git", Branch: "agent/abc123",
-		CredUser: "x", CredToken: "sk-SECRET",
-	}, nil)
-	if err == nil {
-		t.Fatal("cleartext coding POST must fail closed by default")
-	}
-	if strings.Contains(err.Error(), "sk-SECRET") {
-		t.Fatalf("error must not leak the credential: %v", err)
-	}
-}
-
-// A run with NO repo carries no credential, so it is not a "secret" call and the
-// cleartext guard must not refuse it — otherwise every research and bare-exec run
-// is blocked by a rule written to protect a git token that is not there.
-func TestTask_NoRepoRunIsNotSecret(t *testing.T) {
-	var raw []byte
-	srv := ndjsonServer(t, []string{`{"type":"result","ok":true}`}, nil, &raw)
-	defer srv.Close()
-	t.Setenv("BOT_GATEWAY_URL", srv.URL) // http://
-	t.Setenv("BOT_GATEWAY_ALLOW_PLAINTEXT", "")
-	res, err := runner{}.Run(context.Background(), "acme", "u", RunRequest{
-		Prompt: "read the docs and summarise", Tool: "python", Desktop: true,
-	}, nil)
-	if err != nil {
-		t.Fatalf("a repo-less run must not be refused as cleartext-secret: %v", err)
-	}
-	if !res.OK {
-		t.Fatal("expected the terminal result to carry through")
-	}
-	// The wire says what the run is, and says nothing about git.
-	var got map[string]any
-	if err := json.Unmarshal(raw, &got); err != nil {
-		t.Fatalf("body was not JSON: %v", err)
-	}
-	if got["tool"] != "python" || got["desktop"] != true {
-		t.Fatalf("tool/desktop did not reach the runtime: %#v", got)
-	}
-	for _, k := range []string{"credential", "cloneUrl", "branch", "baseBranch"} {
-		if _, ok := got[k]; ok {
-			t.Fatalf("a repo-less run must not put %q on the wire: %#v", k, got)
-		}
-	}
-}
-
-// A credential with no repo is a caller bug, and it is refused at the door rather
-// than trimmed in silence.
-func TestTask_RefusesCredentialWithoutRepo(t *testing.T) {
-	srv := ndjsonServer(t, []string{`{"type":"result","ok":true}`}, nil, nil)
-	defer srv.Close()
-	t.Setenv("BOT_GATEWAY_URL", strings.Replace(srv.URL, "http://", "https://", 1))
-	_, err := runner{}.Run(context.Background(), "acme", "u", RunRequest{
-		Prompt: "x", CredUser: "x", CredToken: "sk-SECRET",
-	}, nil)
-	if err == nil {
-		t.Fatal("a credential with no repo must be refused")
-	}
-	if strings.Contains(err.Error(), "sk-SECRET") {
-		t.Fatalf("error must not leak the credential: %v", err)
-	}
-}

apps/coding/untrusted_test.go

@@ -68,17 +68,17 @@ func TestStartRefusesAProjectThatIsAPath(t *testing.T) {
 // The branch was cloud's to decide — BranchFor(sessionID), named after a session
 // the sandbox did not choose. Adopting the sandbox's self-report meant the claim
 // flowed into VerifyRef, which only asks whether a ref EXISTS (and main does),
-// and out the other side as CreatePR{Head: "main"}: a pull request headed at the
+// and out the other side as PR.Open{Head: "main"}: a pull request headed at the
 // trunk, filed by us, on behalf of a run that never had permission to write
 // there.
 func TestACompromisedSandboxCannotRenameItsOwnBranch(t *testing.T) {
 	sessions := &fakeSessions{id: "sess_abc123def456"}
-	tracker := &fakeTracker{ref: PRRef{Identifier: "API-1"}}
+	tracker := &fakePR{ref: PRRef{Identifier: "API-1"}}
 	verified := map[string]bool{}
 
 	d := Dispatcher{
 		Sessions: sessions,
-		Tracker:  tracker,
+		PR:       tracker,
 		Runner: &fakeRunner{result: RunResult{
 			OK: true, Changed: true, CommitSha: "deadbeef",
 			Branch: "main", // the lie
@@ -113,10 +113,10 @@ func TestACompromisedSandboxCannotRenameItsOwnBranch(t *testing.T) {
 // rather than our sandbox — a strictly less trusted place.
 func TestARoutedMachineCannotRenameItsOwnBranch(t *testing.T) {
 	sessions := &fakeSessions{id: "sess_abc123def456"}
-	tracker := &fakeTracker{ref: PRRef{Identifier: "API-2"}}
+	tracker := &fakePR{ref: PRRef{Identifier: "API-2"}}
 	d := Dispatcher{
 		Sessions:  sessions,
-		Tracker:   tracker,
+		PR:        tracker,
 		VerifyRef: func(context.Context, string, string, string) (string, bool) { return "deadbeef", true },
 	}
 	issuedBranch := BranchFor("sess_abc123def456")
@@ -133,7 +133,7 @@ func TestARoutedMachineCannotRenameItsOwnBranch(t *testing.T) {
 	t.Logf("the machine said %q; the PR is headed at %q", "main", issuedBranch)
 }
 
-func head(f *fakeTracker) string {
+func head(f *fakePR) string {
 	f.mu.Lock()
 	defer f.mu.Unlock()
 	return f.inputs[0].Head

apps/commerce/ledger.go

@@ -39,6 +39,13 @@ var _ creditledger.CreditLedger = ledger{}
 // with (account.Payer, via principal), so a credit and the spend it funds land on one
 // wallet by construction. An org whose members share one balance names no subject and
 // is byte-for-byte unchanged.
+//
+// AND Test IS THE THIRD PART OF THAT ADDRESS. finance keeps sandbox money in a
+// separate file per org, so which books to write is not a mode this adapter can leave
+// unsaid — it was saying "live" for every credit that came through here, which put a
+// sandbox tenant's grant in the books the gate spends real inference from. It is the
+// input's, unchanged, because the caller is the one that knows whether the charge it
+// is crediting was a sandbox charge.
 func (ledger) Credit(ctx context.Context, in creditledger.CreditInput) (string, int64, error) {
 	fin := finance.Current()
 	if fin == nil {
@@ -64,22 +71,31 @@ func (ledger) Credit(ctx context.Context, in creditledger.CreditInput) (string,
 		Notes:    in.Reason,
 		Tags:     tag,
 		Ref:      in.IdempotencyKey,
+		Test:     in.Test,
 	})
 	if err != nil {
 		return "", 0, err
 	}
-	// Read back the account that was CREDITED, never the pool — reporting a pool
-	// balance after crediting a member is how a caller concludes the grant vanished.
-	bal, berr := fin.Balance(ctx, in.Org, subject, cur, false)
+	// Read back the account that was CREDITED, never the pool and never the other
+	// books — reporting a pool balance after crediting a member, or a live balance
+	// after crediting the sandbox, is how a caller concludes the grant vanished. The
+	// read repeats the deposit's whole address, one value at a time.
+	bal, berr := fin.Balance(ctx, in.Org, subject, cur, in.Test)
 	if berr != nil {
 		return id, 0, berr
 	}
 	return id, bal.Cents(), nil
 }
 
-// Balance returns the org pool's available balance in cents for currency — the
-// same read the AI gate performs, so GET /v1/billing/balance and the gate agree.
-func (ledger) Balance(ctx context.Context, org, currency string) (int64, error) {
+// Balance returns the available balance in cents held at (org, subject) in currency,
+// from the sandbox books when test — the same read the AI gate performs, so
+// GET /v1/billing/balance and the gate agree.
+//
+// It takes the whole address for the reason Credit writes it: an empty subject is the
+// org's pool, and a named one is the member's own wallet. Answering the pool for a
+// member's read is answering about a different account, and it answers without
+// erroring — which is the shape of a balance bug nobody notices.
+func (ledger) Balance(ctx context.Context, org, subject, currency string, test bool) (int64, error) {
 	fin := finance.Current()
 	if fin == nil {
 		return 0, fmt.Errorf("commerce balance: no finance ledger co-resident")
@@ -87,7 +103,10 @@ func (ledger) Balance(ctx context.Context, org, currency string) (int64, error)
 	if currency == "" {
 		currency = "usd"
 	}
-	bal, err := fin.Balance(ctx, org, org, currency, false)
+	if subject == "" {
+		subject = org // pooled org: the slug IS the pool account the gate reads
+	}
+	bal, err := fin.Balance(ctx, org, subject, currency, test)
 	if err != nil {
 		return 0, err
 	}

apps/commerce/ledger_test.go

@@ -0,0 +1,218 @@
+// Copyright © 2026 Hanzo AI. MIT License.
+
+package commerce
+
+// ledger_test.go pins what the creditledger ADAPTER translates: commerce names an
+// address, and this is the one place that address becomes a finance call.
+//
+// The address has three parts — (Org, Subject, Test) — and the adapter used to drop
+// the third on the floor. `Test` was never passed to Deposit and both balance reads
+// hardcoded `false`, so every credit commerce minted through this seam landed in the
+// LIVE books whatever the caller meant, and a sandbox tenant's grant became money the
+// AI spend gate buys real inference with. finance keeps sandbox money in a separate
+// file per org, so this is not a flag being ignored — it is a different ledger.
+//
+// It asserts against a RECORDING finance client rather than a real one on purpose.
+// What changed here is the translation, and a recording seam states the arguments
+// exactly; that finance then routes (org, test) to the right file is finance's own
+// property and is covered where it lives (apps/finance).
+
+import (
+	"context"
+	"testing"
+
+	"github.com/hanzoai/commerce/billing/creditledger"
+
+	"github.com/hanzoai/cloud/apps/finance"
+	"github.com/hanzoai/cloud/money"
+	"github.com/hanzoai/cloud/types"
+)
+
+// recorder is a finance client that answers plausibly and remembers exactly what it
+// was asked. Balances are held per WHOLE address, so a deposit into one book is not
+// visible from a read of the other — the property the adapter has to get right.
+type recorder struct {
+	deposits []types.DepositInput
+	reads    []readArgs
+	bal      map[readArgs]int64
+}
+
+// readArgs is one balance read's whole address. It is also the deposits' key, so the
+// two can only meet when every component agrees.
+type readArgs struct {
+	org      string
+	subject  string
+	currency string
+	test     bool
+}
+
+var _ types.FinanceClient = (*recorder)(nil)
+
+func newRecorder() *recorder { return &recorder{bal: map[readArgs]int64{}} }
+
+func (r *recorder) Deposit(_ context.Context, in types.DepositInput) (string, error) {
+	r.deposits = append(r.deposits, in)
+	k := readArgs{in.Org, in.Subject, in.Currency, in.Test}
+	r.bal[k] += in.Amount.Cents()
+	return "dep_1", nil
+}
+
+func (r *recorder) Balance(_ context.Context, org, subject, currency string, test bool) (money.Amount, error) {
+	k := readArgs{org, subject, currency, test}
+	r.reads = append(r.reads, k)
+	return money.FromCents(r.bal[k]), nil
+}
+
+func (r *recorder) RecordUsage(context.Context, types.UsageInput) error { return nil }
+
+func (r *recorder) SumUsageSince(context.Context, string, bool, int64) (int64, error) {
+	return 0, nil
+}
+
+// seam is the adapter under test, bound once. A composite literal cannot start an
+// if-statement's expression, and naming it also says what it is: the ONE value
+// commerce is handed as its credit ledger.
+var seam = ledger{}
+
+// recording publishes the recorder as the process finance client for one test.
+func recording(t *testing.T) *recorder {
+	t.Helper()
+	r := newRecorder()
+	prev := finance.Current()
+	finance.Publish(r)
+	t.Cleanup(func() { finance.Publish(prev) })
+	return r
+}
+
+// A sandbox credit writes the SANDBOX books, and a live credit the live ones — the
+// deposit AND the balance read the caller is answered with.
+//
+// The read-back matters as much as the write: it is the number the credit door hands
+// back to its caller, and reading the live books after crediting the sandbox ones
+// reports a balance that has nothing to do with the credit just made. That was the
+// state of both call sites.
+func TestLedgerAdapter_TestRoutesToTheSandboxBooks(t *testing.T) {
+	for _, tc := range []struct {
+		name string
+		test bool
+	}{
+		{"a sandbox credit", true},
+		{"a live credit", false},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			rec := recording(t)
+
+			_, cents, err := seam.Credit(context.Background(), creditledger.CreditInput{
+				Org:            "acme",
+				Subject:        "acme/alice",
+				Currency:       "usd",
+				AmountCents:    2500,
+				IdempotencyKey: "pay_1",
+				Test:           tc.test,
+			})
+			if err != nil {
+				t.Fatalf("credit: %v", err)
+			}
+
+			if len(rec.deposits) != 1 {
+				t.Fatalf("deposits=%d, want 1", len(rec.deposits))
+			}
+			if got := rec.deposits[0].Test; got != tc.test {
+				t.Errorf("DepositInput.Test=%v, want %v — the credit went into the wrong books", got, tc.test)
+			}
+			if got := rec.deposits[0].Org; got != "acme" {
+				t.Errorf("DepositInput.Org=%q, want acme", got)
+			}
+			if got := rec.deposits[0].Subject; got != "acme/alice" {
+				t.Errorf("DepositInput.Subject=%q, want acme/alice", got)
+			}
+
+			// The read-back repeats the deposit's WHOLE address.
+			if len(rec.reads) != 1 {
+				t.Fatalf("balance reads=%d, want 1", len(rec.reads))
+			}
+			want := readArgs{"acme", "acme/alice", "usd", tc.test}
+			if rec.reads[0] != want {
+				t.Errorf("balance read %+v, want %+v — the credit was read back from a different account", rec.reads[0], want)
+			}
+			if cents != 2500 {
+				t.Errorf("returned balance=%d, want 2500 — the caller was answered from books it did not credit", cents)
+			}
+		})
+	}
+}
+
+// The two books do not see each other. A sandbox credit leaves the live balance at
+// zero, which is the whole reason `test` is part of the address and not a flag.
+func TestLedgerAdapter_SandboxMoneyIsNotSpendable(t *testing.T) {
+	recording(t)
+
+	if _, _, err := seam.Credit(context.Background(), creditledger.CreditInput{
+		Org: "acme", Subject: "acme/alice", Currency: "usd",
+		AmountCents: 9900, IdempotencyKey: "pay_sandbox", Test: true,
+	}); err != nil {
+		t.Fatalf("credit: %v", err)
+	}
+
+	live, err := seam.Balance(context.Background(), "acme", "acme/alice", "usd", false)
+	if err != nil {
+		t.Fatalf("live balance: %v", err)
+	}
+	if live != 0 {
+		t.Fatalf("LIVE balance=%d after a SANDBOX credit, want 0 — sandbox money became spendable", live)
+	}
+
+	sandbox, err := seam.Balance(context.Background(), "acme", "acme/alice", "usd", true)
+	if err != nil {
+		t.Fatalf("sandbox balance: %v", err)
+	}
+	if sandbox != 9900 {
+		t.Fatalf("sandbox balance=%d, want 9900", sandbox)
+	}
+}
+
+// Balance reads the account it is ASKED for, and an empty subject is the org's pool —
+// the same default Credit applies, so the two halves address one wallet.
+//
+// Its signature used to be (org, currency) with the org passed for the subject too,
+// which meant a member-scoped read opened a LEDGER named after the member: a
+// different file, holding nothing, answering a funded customer with a confident zero.
+func TestLedgerAdapter_BalanceReadsTheAccountItIsAsked(t *testing.T) {
+	rec := recording(t)
+
+	// The pool holds one amount and a member another, in the same org and books.
+	for _, seed := range []struct {
+		subject string
+		cents   int64
+	}{{"acme", 100}, {"acme/alice", 700}} {
+		if _, _, err := seam.Credit(context.Background(), creditledger.CreditInput{
+			Org: "acme", Subject: seed.subject, Currency: "usd",
+			AmountCents: seed.cents, IdempotencyKey: "seed:" + seed.subject,
+		}); err != nil {
+			t.Fatalf("seed %s: %v", seed.subject, err)
+		}
+	}
+	rec.reads = nil
+
+	pool, err := seam.Balance(context.Background(), "acme", "", "usd", false)
+	if err != nil {
+		t.Fatalf("pool balance: %v", err)
+	}
+	if pool != 100 {
+		t.Errorf("pool balance=%d, want 100 — an empty subject must be the org's own account", pool)
+	}
+	if got := rec.reads[0]; got.subject != "acme" {
+		t.Errorf("pool read addressed subject %q, want acme", got.subject)
+	}
+
+	member, err := seam.Balance(context.Background(), "acme", "acme/alice", "usd", false)
+	if err != nil {
+		t.Fatalf("member balance: %v", err)
+	}
+	if member != 700 {
+		t.Errorf("member balance=%d, want 700 — the read answered a different account than the credit funded", member)
+	}
+	if got := rec.reads[1]; got.org != "acme" || got.subject != "acme/alice" {
+		t.Errorf("member read addressed (%q,%q), want (acme, acme/alice) — the subject was used as the ledger", got.org, got.subject)
+	}
+}

apps/commerce/settle.go

@@ -112,11 +112,14 @@ package commerce
 // in the same one — a sandbox charge can never fund live inference, and a live
 // charge is never parked in books no gate reads.
 //
-// This is why the credit is posted through finance's own Deposit rather than
-// through commerce's injected creditledger adapter (ledger.go): CreditInput has no
-// test field, so that seam cannot express which books to write, and routing a
-// sandbox settlement through it would put unspendable sandbox money in the live
-// ledger. Same ledger, same idempotency, one field the seam cannot carry.
+// The credit is posted through finance's own Deposit rather than through commerce's
+// injected creditledger adapter (ledger.go) because of the ADDRESS, not the books:
+// the seam carries the test bit now, but it is commerce's door onto its own credit,
+// and the address this file deposits at is (p.ledger, p.subject) — the payer the
+// SCREEN resolved from the request's principal, which is a value commerce cannot
+// compute for itself. Going through the adapter would mean handing it back the
+// answer it exists to ask for. Same ledger, same idempotency, one address resolved
+// where the identity is.
 
 import (
 	"context"

apps/crawl/browser.go

@@ -9,10 +9,14 @@ package crawl
 // than one that fails, because nothing upstream can tell.
 //
 // So: static first, and if what came back is too thin to be the page, ask Hanzo
-// Crawl (headless Chromium, ghcr.io/hanzoai/crawl) for the rendered version.
-// Escalation is one-way and best-effort — if the browser is absent, slow or
-// unhappy, the static Page still stands. That keeps a working crawl working
-// while the browser is not deployed, which is the state this ships in.
+// Crawl (headless Chromium, ghcr.io/hanzoai/crawl) for the rendered version. It
+// runs at crawl.hanzo.svc:11235 — the address browserEndpoint returns — and it
+// requires the bearer token below.
+//
+// Escalation is one-way and best-effort: if the browser is slow or unhappy, the
+// static Page still stands. That is a property worth keeping even though the
+// service is up, because "the render failed" must never turn a page we already
+// have into an error.
 
 import (
 	"bytes"
@@ -183,8 +187,15 @@ func browse(ctx context.Context, raw string) (*Page, error) {
 		return nil, err
 	}
 	req.Header.Set("Content-Type", "application/json")
-	// Token if the deployment has one; the network policy is the real boundary,
-	// so its absence is not a reason to skip rendering.
+	// The service REQUIRES this: unauthenticated it answers
+	// {"detail":"Authentication required"}, so a missing token is not a degraded
+	// render, it is no render at all. It reaches both pods from KMS as the
+	// crawl-secrets/CRAWL_API_TOKEN key.
+	//
+	// Sent when present rather than demanded up front because the failure is
+	// already well reported — the request returns a non-2xx, escalate() keeps the
+	// static Page, and websearch counts the engine blind. One boundary, one
+	// refusal, read in one place.
 	if tok := strings.TrimSpace(os.Getenv("CRAWL_API_TOKEN")); tok != "" {
 		req.Header.Set("Authorization", "Bearer "+tok)
 	}

apps/crawl/crawl.go

@@ -3,14 +3,16 @@
 // Hanzo Crawl: fetch one URL and return its readable content as markdown,
 // in-process, in Go.
 //
-// It replaces the dial to a separate Crawl4AI deployment. That service was named
-// in config but did not exist — crawl.hanzo.svc.cluster.local was NXDOMAIN — so
-// every scrape returned {success:false, "no such host"} while the surface in front
-// of it answered 200. This package is the same move already made for the SEARCH
-// half in clients/websearch/search.go, which replaced a SearXNG pod with in-process
-// Go: one fewer non-Go dependency, one fewer thing that can be down, and no network
+// The FETCH is in-process Go rather than a dial to a separate service, which is
+// the same move already made for the SEARCH half in apps/websearch/search.go:
+// one fewer non-Go dependency, one fewer thing that can be down, and no network
 // hop for work that is a fetch and a parse.
 //
+// A browser still lives out of process, because rendering is not a fetch and a
+// parse. `crawl` (ghcr.io/hanzoai/crawl, headless Chromium) is deployed at
+// crawl.hanzo.svc:11235 and browser.go escalates to it for the pages this file
+// cannot read alone.
+//
 // The engine is three orthogonal steps, each in its own file and each testable
 // without the others:
 //

apps/crawl/mount.go

@@ -181,7 +181,14 @@ func Mount(app cloud.Router, deps cloud.Deps) error {
 	// ONE registration, at the address this API is called on. Declaring the whole
 	// path here — rather than a leaf on a Group — is the fix LLM.md prescribes for
 	// the trailing-slash class described on [Path].
+	// Named, not derived. A POST to /v1/crawl derives `create_crawl`, which reads
+	// as "make a crawl" — a job this surface does not have and cannot start. What
+	// it does is read ONE page that is already addressed, and a model choosing
+	// from an `op` enum picks by that name before it reads any description: the
+	// derived name offered a crawler and the op is a reader. It is the verb over
+	// the noun, beside search_web and research_web.
 	zip.Post(reg, Path, fetch,
+		zip.WithOperationID("read_page"),
 		zip.WithSummary("Fetch one URL and read it back as markdown"),
 		// 400 is DECLARED because this op answers it with its OWN body rather than
 		// with zip's error envelope. See [crawlResult.StatusCode].

apps/crawl/typed_wire_test.go

@@ -170,17 +170,17 @@ func TestCrawlProjectsAsATool(t *testing.T) {
 		t.Fatalf("POST /mcp did not answer MCP: %d — %.200s", resp.StatusCode, raw)
 	}
 	for _, tool := range env.Result.Tools {
-		if tool.Name != "post_v1_crawl" {
+		if tool.Name != "read_page" {
 			continue
 		}
 		if tool.Description == "" {
-			t.Fatal("post_v1_crawl projects with NO description — a tool a model cannot " +
+			t.Fatal("read_page projects with NO description — a tool a model cannot " +
 				"read is a tool it will not call; run `go generate -run zipdoc ./...`")
 		}
-		t.Logf("post_v1_crawl projects: %.90s…", tool.Description)
+		t.Logf("read_page projects: %.90s…", tool.Description)
 		return
 	}
-	t.Fatalf("post_v1_crawl is not in this subsystem's tools/list — %.300s", raw)
+	t.Fatalf("read_page is not in this subsystem's tools/list — %.300s", raw)
 }
 
 // TestCrawlPublishesItsBodies holds the document to account. It used to assert

apps/destinations/destinations.go

@@ -163,12 +163,8 @@ func routes(app cloud.Router, zapp *zip.App, s *cloud.Service[state]) {
 	g.Post("/:platform", cloud.Handle(s, connect))
 	zip.Delete(zapp, "/v1/destinations/:platform", o.disconnect)
 	zip.Post(zapp, "/v1/destinations/:platform/test", o.test)
-
-	// GET /v1/tags — the PUBLIC, pk--keyed PER-SITE browser-tag config the hosted tag
-	// fetches. A raw net/http handler (like analytics' /v1/event.js) so it sets CORS +
-	// cache directly; it dual-resolves the SITE (projects.TagsFor: by key, else by host)
-	// and requires no session.
-	app.Get("/v1/tags", zip.AdaptNetHTTP(http.HandlerFunc(serveTags)))
+	// GET /v1/tags moved to the projects app — it must be served by the process that
+	// owns the project store (production runs ~25 single-app processes).
 }
 
 // The one untyped route DECLARES what it carries. Its request is the map the

apps/destinations/tags_test.go

@@ -1,115 +0,0 @@
-package destinations
-
-import (
-	"encoding/json"
-	"net/http"
-	"net/http/httptest"
-	"testing"
-)
-
-func TestBuildTags(t *testing.T) {
-	// A site's per-project tag config: platform → non-secret pixel id.
-	tags := buildTags(map[string]string{
-		"ga4":        "G-ABC",
-		"meta":       "123",
-		"x":          "o1",
-		"reddit":     "a2", // server-side only → no browser pixel
-		"google-ads": "9",  // server-side only
-		"tiktok":     "",   // connected but no id → omitted
-	})
-	if len(tags) != 3 {
-		t.Fatalf("want 3 injectable tags (ga4/meta/x), got %d: %+v", len(tags), tags)
-	}
-	by := map[string]browserTagOut{}
-	for _, tg := range tags {
-		by[tg.Platform] = tg
-	}
-	if by["ga4"].Type != "ga" || by["ga4"].ID != "G-ABC" {
-		t.Errorf("ga4 → %+v, want {ga, G-ABC}", by["ga4"])
-	}
-	if by["meta"].Type != "meta" || by["meta"].ID != "123" {
-		t.Errorf("meta → %+v", by["meta"])
-	}
-	if by["x"].Type != "x" || by["x"].ID != "o1" {
-		t.Errorf("x → %+v", by["x"])
-	}
-	if _, has := by["reddit"]; has {
-		t.Error("reddit has no browser pixel and must be omitted")
-	}
-	if _, has := by["tiktok"]; has {
-		t.Error("a platform with no id must be omitted")
-	}
-}
-
-func TestTagsKey(t *testing.T) {
-	mk := func(url, auth string) *http.Request {
-		r := httptest.NewRequest(http.MethodGet, url, nil)
-		if auth != "" {
-			r.Header.Set("Authorization", auth)
-		}
-		return r
-	}
-	if k := tagsKey(mk("/v1/tags?key=pk-q", "")); k != "pk-q" {
-		t.Errorf("?key= → %q", k)
-	}
-	if k := tagsKey(mk("/v1/tags?ingest_key=pk-i", "")); k != "pk-i" {
-		t.Errorf("?ingest_key= → %q", k)
-	}
-	if k := tagsKey(mk("/v1/tags", "Bearer pk-b")); k != "pk-b" {
-		t.Errorf("Bearer → %q", k)
-	}
-	if k := tagsKey(mk("/v1/tags?key=pk-q", "Bearer pk-b")); k != "pk-b" {
-		t.Errorf("Bearer must win over query, got %q", k)
-	}
-	if k := tagsKey(mk("/v1/tags", "")); k != "" {
-		t.Errorf("no key → %q", k)
-	}
-}
-
-func TestTagsHost(t *testing.T) {
-	mk := func(q, origin, referer string) *http.Request {
-		r := httptest.NewRequest(http.MethodGet, "/v1/tags"+q, nil)
-		if origin != "" {
-			r.Header.Set("Origin", origin)
-		}
-		if referer != "" {
-			r.Header.Set("Referer", referer)
-		}
-		return r
-	}
-	if h := tagsHost(mk("?host=hanzo.ai", "", "")); h != "hanzo.ai" {
-		t.Errorf("?host= → %q", h)
-	}
-	if h := tagsHost(mk("", "https://hanzo.chat", "")); h != "hanzo.chat" {
-		t.Errorf("Origin → %q", h)
-	}
-	if h := tagsHost(mk("", "", "https://hanzo.app/pricing?x=1")); h != "hanzo.app" {
-		t.Errorf("Referer → %q", h)
-	}
-	if h := tagsHost(mk("", "", "")); h != "" {
-		t.Errorf("none → %q", h)
-	}
-}
-
-// TestServeTagsNoSite proves the fail-safe: with projects unmounted (TagsFor ⇒ false),
-// the tag door answers 200 with an empty set and permissive CORS — a page never breaks.
-func TestServeTagsNoSite(t *testing.T) {
-	req := httptest.NewRequest(http.MethodGet, "/v1/tags?key=pk-x", nil)
-	w := httptest.NewRecorder()
-	serveTags(w, req)
-
-	res := w.Result()
-	if res.StatusCode != http.StatusOK {
-		t.Fatalf("status = %d, want 200", res.StatusCode)
-	}
-	if res.Header.Get("Access-Control-Allow-Origin") != "*" {
-		t.Error("the tag is loaded cross-origin — must allow any origin")
-	}
-	var out tagConfig
-	if err := json.NewDecoder(res.Body).Decode(&out); err != nil {
-		t.Fatalf("decode: %v", err)
-	}
-	if len(out.Tags) != 0 {
-		t.Errorf("no resolvable site must yield an empty tag set, got %+v", out.Tags)
-	}
-}

apps/git/build_on_push.go

@@ -81,21 +81,29 @@ type pipelineImage struct {
 	Context    string `yaml:"context"`    // build context dir (default ".")
 	Dockerfile string `yaml:"dockerfile"` // default "<context>/Dockerfile"
 	TagSuffix  string `yaml:"tag-suffix"` // default = name
+	// Args are `--build-arg` values for THIS image. They are what makes several
+	// entries off ONE Dockerfile mean different things: hanzoai/bot declares
+	// three sandbox classes as three entries that differ only by
+	// `args: {STAGE: exec|dev|desktop}`. Dropped on the floor, the three tags are
+	// three copies of whatever stage the Dockerfile defaults to — an `exec` tag
+	// carrying a whole desktop, published under a name that says otherwise.
+	Args map[string]string `yaml:"args"`
 }
 
 // enqueueReq is platform's /v1/runner body (EnqueueBody). Field-for-field the
 // same shape the `hanzoai/ci mode:delegate` step POSTs, so a native-push build and a
 // delegated GitHub-Actions build are byte-identical downstream — one build path.
 type enqueueReq struct {
-	Repo       string `json:"repo"`  // GitHub owner/repo — BuildKit's clone context
-	SHA        string `json:"sha"`   // full commit the build pins to
-	Image      string `json:"image"` // full pushed ref repo:tag (we own the tag)
-	Branch     string `json:"branch,omitempty"`
-	Ref        string `json:"ref,omitempty"`
-	Dockerfile string `json:"dockerfile,omitempty"`
-	Context    string `json:"context,omitempty"`
-	OS         string `json:"os,omitempty"`
-	Arch       string `json:"arch,omitempty"`
+	Repo       string            `json:"repo"`  // GitHub owner/repo — BuildKit's clone context
+	SHA        string            `json:"sha"`   // full commit the build pins to
+	Image      string            `json:"image"` // full pushed ref repo:tag (we own the tag)
+	Branch     string            `json:"branch,omitempty"`
+	Ref        string            `json:"ref,omitempty"`
+	Dockerfile string            `json:"dockerfile,omitempty"`
+	Context    string            `json:"context,omitempty"`
+	OS         string            `json:"os,omitempty"`
+	Arch       string            `json:"arch,omitempty"`
+	Args       map[string]string `json:"args,omitempty"`
 }
 
 // nativeCICDEnabled reports whether the orchestrator is armed: the enable flag is
@@ -262,6 +270,7 @@ func enqueueBody(img pipelineImage, ghRepo, branch, sha string) *enqueueReq {
 		Context:    ctxDir,
 		OS:         "linux",
 		Arch:       "amd64",
+		Args:       img.Args,
 	}
 }
 

apps/git/mirror_out.go

@@ -4,7 +4,6 @@ import (
 	"context"
 	"encoding/base64"
 	"fmt"
-	"net/url"
 	"os"
 	"strconv"
 	"strings"
@@ -216,17 +215,9 @@ func mirrorPushAuthHeader(host string) string {
 	return base64.StdEncoding.EncodeToString([]byte(mirrorBasicUser(host) + ":" + tok))
 }
 
-// githubOwnerOf reads the account from a GitHub remote — the first path segment of
-// https://github.com/<owner>/<repo>.git. Empty when the URL names none, which lets
-// the single-connection case resolve as before.
+// githubOwnerOf reads the account from a GitHub remote. The full read lives in
+// propose.go, which needs the repository half too; one parser, two callers.
 func githubOwnerOf(remote string) string {
-	u, err := url.Parse(strings.TrimSpace(remote))
-	if err != nil {
-		return ""
-	}
-	parts := strings.Split(strings.Trim(u.Path, "/"), "/")
-	if len(parts) == 0 {
-		return ""
-	}
-	return parts[0]
+	owner, _ := githubRepoOf(remote)
+	return owner
 }

apps/git/propose.go

@@ -0,0 +1,179 @@
+package git
+
+// propose.go answers the one question left when a coding run finishes: where
+// does a person go to read what it did?
+//
+// There are two answers, and which one applies is a property of WHERE THE CODE
+// LIVES, never of who is asking:
+//
+//	github.com  the repository mirrors into a GitHub account, so the proposal is
+//	            a real pull request there, opened against the base branch.
+//	here        the repository lives only in the forge, which has no pull request
+//	            of its own. The branch's page is where the work is read; the
+//	            tracker row filed beside it is where it is tracked.
+//
+// One seam, two backends. A caller asks for the address and never for the host,
+// which is what keeps "paste a link from either and it works" from becoming two
+// orchestrations that drift apart.
+//
+// # The head is pushed here and not left to the mirror
+//
+// mirror_out.go already replicates a landed branch to every target — but it is a
+// best-effort subscriber to a lifecycle event, running on a one-slot semaphore
+// with a five-minute ceiling, so "the branch is on GitHub" is not true at any
+// particular moment. GitHub refuses a pull request whose head it cannot see, so
+// this pushes the head itself, through the SAME function, before asking. A force
+// push of one already-identical ref is a no-op, so doing it twice costs nothing
+// and removes a race that would otherwise fail a fraction of runs for no reason a
+// user could act on.
+//
+// # Credentials
+//
+// The GitHub App installation token is minted per call and never stored, exactly
+// as the outbound mirror mints it (mirror_out.go outboundAuthHeader). It rides an
+// Authorization header — never argv, never a log line, never the returned error.
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/url"
+	"strings"
+	"time"
+
+	"github.com/hanzoai/cloud"
+	"github.com/hanzoai/cloud/apps/integrations"
+	"github.com/hanzoai/cloud/brand"
+)
+
+// proposeTimeout bounds the API call. The mirror push before it carries its own,
+// much larger, ceiling; this one is a single small POST.
+const proposeTimeout = 20 * time.Second
+
+// Propose offers head for merging into base and returns where to read it.
+//
+// It is fail-closed on the GitHub path — an org whose repository mirrors to
+// GitHub and whose pull request could not be opened gets an error, not a quiet
+// forge link, because the two are different outcomes and only one of them is
+// "your change is waiting for review over there".
+func Propose(ctx context.Context, org, project, repo, base, head, title, body string) (string, error) {
+	s := mounted.Load()
+	if s == nil {
+		return "", fmt.Errorf("git: the forge is not mounted")
+	}
+	if !branchRE.MatchString(head) {
+		return "", fmt.Errorf("git: %q is not a branch", head)
+	}
+	store, err := storeFor(s, org)
+	if err != nil {
+		return "", fmt.Errorf("git: open store: %w", err)
+	}
+	targets, err := store.ListMirrors(ctx, org, project, repo)
+	if err != nil {
+		return "", fmt.Errorf("git: read mirrors: %w", err)
+	}
+	for _, t := range targets {
+		if !strings.EqualFold(strings.TrimSpace(t.Host), "github.com") {
+			continue
+		}
+		owner, name := githubRepoOf(t.URL)
+		if owner == "" || name == "" {
+			return "", fmt.Errorf("github: %q names no repository", t.URL)
+		}
+		// The credential is resolved BEFORE the head is pushed. Not an
+		// optimisation: a mirror push holds the forge's single outbound slot for up
+		// to five minutes, and spending it to replicate a branch we then cannot
+		// propose is a cost paid for nothing.
+		tok, err := integrations.InstallationToken(ctx, org, owner)
+		if err != nil || strings.TrimSpace(tok) == "" {
+			return "", fmt.Errorf("github: no installation for %s", owner)
+		}
+		bare := s.State.storage.absRepoPath(org, project, repo)
+		if err := pushBranchToMirror(ctx, org, bare, t, head); err != nil {
+			return "", fmt.Errorf("git: mirror %s: %s", t.Host, sanitizeGitErr(err.Error()))
+		}
+		return pullRequest(ctx, tok, owner, name, base, head, title, body)
+	}
+	return branchPage(s, org, project, repo, head), nil
+}
+
+// pullRequest opens the pull request on GitHub and returns its address.
+//
+// A 422 is GitHub's answer for "one already exists for this head", which for an
+// agent branch can only mean a retry of a run that already proposed — so it is
+// reported as what it is rather than dressed up as a failure.
+func pullRequest(ctx context.Context, tok, owner, name, base, head, title, body string) (string, error) {
+	payload, err := json.Marshal(struct {
+		Title string `json:"title"`
+		Head  string `json:"head"`
+		Base  string `json:"base"`
+		Body  string `json:"body,omitempty"`
+	}{Title: title, Head: head, Base: base, Body: body})
+	if err != nil {
+		return "", err
+	}
+
+	ctx, cancel := context.WithTimeout(ctx, proposeTimeout)
+	defer cancel()
+	endpoint := fmt.Sprintf("%s/repos/%s/%s/pulls", api, url.PathEscape(owner), url.PathEscape(name))
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(payload))
+	if err != nil {
+		return "", err
+	}
+	req.Header.Set("Authorization", "Bearer "+tok)
+	req.Header.Set("Accept", "application/vnd.github+json")
+	req.Header.Set("X-GitHub-Api-Version", "2022-11-28")
+	req.Header.Set("Content-Type", "application/json")
+
+	resp, err := (&http.Client{Timeout: proposeTimeout}).Do(req)
+	if err != nil {
+		return "", fmt.Errorf("github: open pull request: %w", err)
+	}
+	defer func() { _ = resp.Body.Close() }()
+
+	var out struct {
+		HTMLURL string `json:"html_url"`
+		Message string `json:"message"`
+	}
+	_ = json.NewDecoder(resp.Body).Decode(&out)
+	if resp.StatusCode == http.StatusCreated && out.HTMLURL != "" {
+		return out.HTMLURL, nil
+	}
+	return "", fmt.Errorf("github: %s/%s pull request: status %d: %s", owner, name, resp.StatusCode, out.Message)
+}
+
+// branchPage is where a branch is read when the repository lives only here.
+//
+// Empty for a project-scoped repository: the browse routes address an org and a
+// repo and nothing else (ui.go uiRoutes), so there is no page to point at, and an
+// address that 404s is worse than none.
+func branchPage(s *cloud.Service[state], org, project, repo, branch string) string {
+	if strings.TrimSpace(project) != "" {
+		return ""
+	}
+	host := s.Domain
+	if host == "" {
+		host = brand.APIHost(brand.Default)
+	}
+	return fmt.Sprintf("https://%s/git/%s/%s?ref=%s", host, org, repo, url.QueryEscape(branch))
+}
+
+// githubRepoOf reads the account and repository out of a GitHub remote —
+// https://github.com/<owner>/<repo>.git. Empty when the URL names neither, which
+// lets the single-connection case resolve as it always did.
+func githubRepoOf(remote string) (owner, name string) {
+	u, err := url.Parse(strings.TrimSpace(remote))
+	if err != nil {
+		return "", ""
+	}
+	parts := strings.Split(strings.Trim(u.Path, "/"), "/")
+	if len(parts) == 0 {
+		return "", ""
+	}
+	if len(parts) == 1 {
+		return parts[0], ""
+	}
+	return parts[0], strings.TrimSuffix(parts[1], ".git")
+}

apps/git/propose_test.go

@@ -0,0 +1,114 @@
+package git
+
+// propose_test.go proves the ONE thing that makes "paste a link from either host
+// and it works" true: which backend answers is decided by where the code lives,
+// and a repository that lives on GitHub never quietly gets a forge link instead
+// of the pull request its reviewers are waiting on.
+
+import (
+	"context"
+	"net/http"
+	"strings"
+	"testing"
+	"time"
+)
+
+// A repository that lives only in the forge is read at its branch's own page.
+func TestProposeAnswersTheBranchPageForAForgeRepo(t *testing.T) {
+	app := mountApp(t)
+	if code, b := do(t, app, http.MethodPost, "/v1/git/repos", "acme", map[string]any{"name": "api"}); code != 201 {
+		t.Fatalf("create repo: %d %s", code, b)
+	}
+
+	url, err := Propose(context.Background(), "acme", "", "api", "main",
+		"agent/abc123def456", "api: fix the flake", "body")
+	if err != nil {
+		t.Fatalf("propose: %v", err)
+	}
+	const want = "https://api.hanzo.test/git/acme/api?ref=agent%2Fabc123def456"
+	if url != want {
+		t.Fatalf("branch page = %q, want %q", url, want)
+	}
+}
+
+// A repository that mirrors into GitHub is proposed THERE, and when that cannot
+// happen the run is told so.
+//
+// The failure is the assertion. There is no GitHub App connection in a test, so
+// the credential cannot be minted — and the answer must be an error rather than
+// the forge link the previous case returns. A run whose reviewers are on GitHub,
+// handed a forge URL and told everything is fine, is the quiet wrong answer this
+// seam exists to make impossible.
+func TestProposeWillNotSubstituteAForgeLinkForAGitHubRepo(t *testing.T) {
+	app := mountApp(t)
+	if code, b := do(t, app, http.MethodPost, "/v1/git/repos", "acme", map[string]any{"name": "api"}); code != 201 {
+		t.Fatalf("create repo: %d %s", code, b)
+	}
+	store, err := storeFor(mounted.Load(), "acme")
+	if err != nil {
+		t.Fatalf("store: %v", err)
+	}
+	must(t, store.CreateMirror(context.Background(), MirrorTarget{
+		ID: "m1", Org: "acme", Repo: "api", Host: "github.com",
+		URL: "https://github.com/hanzo-inc/api.git", CreatedAt: time.Now().Unix(),
+	}))
+
+	url, err := Propose(context.Background(), "acme", "", "api", "main",
+		"agent/abc123def456", "api: fix the flake", "body")
+	if err == nil {
+		t.Fatalf("a GitHub repo answered without a pull request: %q", url)
+	}
+	if url != "" {
+		t.Fatalf("a failed proposal still returned an address: %q", url)
+	}
+	if !strings.Contains(err.Error(), "hanzo-inc") {
+		t.Fatalf("the error does not name the account it could not reach: %v", err)
+	}
+}
+
+// A head is checked as a BRANCH before it reaches a refspec.
+//
+// The load-bearing half is the first character class: the value lands in a `git
+// push` argument position, where anything beginning with '-' is a flag rather
+// than a branch and `--upload-pack=` is a command of the caller's choosing on
+// this machine. Length and whitespace matter too; the dash is the one that turns
+// data into a program.
+func TestProposeRefusesAHeadThatIsNotABranch(t *testing.T) {
+	mountApp(t)
+	for _, head := range []string{"", "--upload-pack=touch /tmp/x", "-x", "a b", "a\nb", strings.Repeat("a", 200)} {
+		if _, err := Propose(context.Background(), "acme", "", "api", "main", head, "t", "b"); err == nil {
+			t.Fatalf("propose accepted %q as a branch", head)
+		}
+	}
+}
+
+// A project-scoped repository has no browsable page, and says so rather than
+// pointing at one that would 404.
+func TestProposeHasNoPageForAProjectScopedRepo(t *testing.T) {
+	app := mountApp(t)
+	if code, b := do(t, app, http.MethodPost, "/v1/git/repos", "acme", map[string]any{"name": "api"}); code != 201 {
+		t.Fatalf("create repo: %d %s", code, b)
+	}
+	url, err := Propose(context.Background(), "acme", "web", "api", "main", "agent/abc123def456", "t", "b")
+	if err != nil {
+		t.Fatalf("propose: %v", err)
+	}
+	if url != "" {
+		t.Fatalf("a project-scoped repo answered with %q, which is not a page", url)
+	}
+}
+
+func TestGitHubRepoOf(t *testing.T) {
+	for remote, want := range map[string][2]string{
+		"https://github.com/hanzo-inc/cloud.git": {"hanzo-inc", "cloud"},
+		"https://github.com/hanzoai/cloud":       {"hanzoai", "cloud"},
+		"https://github.com/hanzoai/":            {"hanzoai", ""},
+		"https://github.com/":                    {"", ""},
+		"::not a url::":                          {"", ""},
+	} {
+		owner, name := githubRepoOf(remote)
+		if owner != want[0] || name != want[1] {
+			t.Fatalf("githubRepoOf(%q) = %q,%q want %q,%q", remote, owner, name, want[0], want[1])
+		}
+	}
+}

apps/git/refpolicy_desync_test.go

@@ -69,6 +69,39 @@ func TestAGrantMayNotWriteNothing(t *testing.T) {
 	}
 }
 
+// The sentence the whole coding path leans on, stated head-on: a run's grant is
+// for its own branch, and refs/heads/main is not it.
+//
+// apps/coding pushes with this grant now (sandboxrunner.go), so this is no longer
+// a claim about a credential nobody uses — it is the rule that stands between a
+// prompt-injected model and the trunk. It is asserted for a CREATE, which is the
+// friendliest thing a push can ask for and therefore the last shape anyone would
+// think to refuse: main already exists, so a create against it would be refused
+// anyway, and the point is that the GRANT refuses it first and would refuse it in
+// a repository where main did not exist yet.
+func TestAGrantMayNotWriteTheTrunk(t *testing.T) {
+	for _, ref := range []string{"refs/heads/main", "refs/heads/master", "refs/heads/release/2.1", "refs/tags/v1"} {
+		cmds := []refCommand{{
+			Old: "0000000000000000000000000000000000000000",
+			New: "1111111111111111111111111111111111111111",
+			Ref: ref,
+		}}
+		if err := checkRefPolicy(cmds, "main", "refs/heads/agent/abc123def456"); err == nil {
+			t.Fatalf("a grant confined to an agent branch wrote %s", ref)
+		}
+	}
+	// The control that gives it meaning: its OWN ref still goes through, or the
+	// rule is just an outage.
+	own := []refCommand{{
+		Old: "0000000000000000000000000000000000000000",
+		New: "1111111111111111111111111111111111111111",
+		Ref: "refs/heads/agent/abc123def456",
+	}}
+	if err := checkRefPolicy(own, "main", "refs/heads/agent/abc123def456"); err != nil {
+		t.Fatalf("a run cannot write its own branch: %v", err)
+	}
+}
+
 // The same emptiness stays harmless for a principal: an empty push is a no-op
 // and always was. Refusing it would break ordinary clients for no gain.
 func TestAPrincipalMayPushNothing(t *testing.T) {

apps/git/refwriters_wire_test.go

@@ -499,3 +499,84 @@ func doAuth(t *testing.T, app *zip.App, method, path, token string, body any) (i
 	out, _ := io.ReadAll(resp.Body)
 	return resp.StatusCode, out
 }
+
+// TestAGrantAuthenticatesTheWayTheSandboxPresentsIt is the end-to-end proof for
+// the shape apps/coding actually builds, and it exists because the shape it used
+// to build could not work.
+//
+// A run's clone and push carried the grant as URL userinfo —
+// https://x-access-token:hgg_…@host/… — on the reasoning that a credential in a
+// URL leaves nothing behind. Two things were wrong with it, and the second is
+// fatal rather than untidy:
+//
+//   - git writes the URL it was given into .git/config verbatim, so the grant sat
+//     readable in the checkout that then executes untrusted model output.
+//   - git does not SEND a URL-embedded credential until it is challenged. It makes
+//     an anonymous request and waits for 401 WWW-Authenticate. resolvePackRepo
+//     answers a caller it cannot place with 403, never 401 — so the grant was
+//     never presented at all.
+//
+// The header form is what the rest of the forge already uses (mirror_out.go), and
+// this pins that a URL-SCOPED one — which git attaches to this repository and to
+// no other host the run might be steered at — authenticates the same way.
+func TestAGrantAuthenticatesTheWayTheSandboxPresentsIt(t *testing.T) {
+	app := mountApp(t)
+	base := liveServer(t, app)
+	if code, b := do(t, app, http.MethodPost, "/v1/git/repos", "acme", map[string]any{"name": "code"}); code != 201 {
+		t.Fatalf("create repo: %d %s", code, b)
+	}
+	if code, b := do(t, app, http.MethodPost, "/v1/git/repos/code/push", "acme", map[string]any{
+		"branch": "main", "message": "seed",
+		"files": []map[string]any{{"path": "README.md", "content": "# seed\n"}},
+	}); code != 200 {
+		t.Fatalf("seed main: %d %s", code, b)
+	}
+
+	branch := "agent/abc123def456"
+	tok, _, err := issued.issue(grant{org: "acme", repo: "code", ref: "refs/heads/" + branch}, 0)
+	if err != nil {
+		t.Fatalf("issue: %v", err)
+	}
+	url := base + "/v1/git/acme/code.git"
+	// EXACTLY what apps/coding builds (sandboxrunner.go gitAs): one -c, scoped to
+	// this url, applied to this invocation.
+	auth := []string{"-c", "http." + url + ".extraHeader=Authorization: Basic " +
+		base64.StdEncoding.EncodeToString([]byte("x-access-token:"+tok))}
+
+	work := filepath.Join(t.TempDir(), "clone")
+	gitRun(t, "", append(auth, "clone", "-q", "--depth", "1", "-b", "main", url, work)...)
+	t.Log("the grant cloned its repo with a url-scoped header")
+
+	// AND IT LEFT NOTHING BEHIND. The checkout the model edits holds no credential:
+	// a top-level -c is not written into the new repository's config.
+	cfg, err := os.ReadFile(filepath.Join(work, ".git", "config"))
+	if err != nil {
+		t.Fatalf("read config: %v", err)
+	}
+	for _, leak := range []string{tok, "extraHeader", "Authorization"} {
+		if strings.Contains(string(cfg), leak) {
+			t.Fatalf("the clone persisted %q into .git/config:\n%s", leak, cfg)
+		}
+	}
+	t.Log("nothing was written to .git/config")
+
+	gitRun(t, work, "switch", "-q", "-c", branch)
+	write(t, work, "feature.txt", "agent work\n")
+	gitRun(t, work, "add", "-A")
+	gitRun(t, work, "commit", "-q", "-m", "agent change")
+	if out, err := gitTestCmd(work, append(auth, "push", url, "HEAD:refs/heads/"+branch)...).CombinedOutput(); err != nil {
+		t.Fatalf("the run must be able to push its own branch: %v\n%s", err, out)
+	}
+	t.Log("the grant pushed the one ref it names")
+
+	// The trunk is still refused with the credential presented THIS way, so the
+	// change of form did not change what the form is allowed to do.
+	out, err := gitTestCmd(work, append(auth, "push", url, "HEAD:refs/heads/main")...).CombinedOutput()
+	if err == nil {
+		t.Fatalf("A GRANT WROTE THE TRUNK\n%s", out)
+	}
+	if !strings.Contains(string(out), "may only write") {
+		t.Fatalf("refused, but not by the ref policy:\n%s", out)
+	}
+	t.Logf("REFUSED: %s", firstRejectLine(string(out)))
+}

apps/integrations/bridge.go

@@ -14,6 +14,9 @@ import (
 	"github.com/hanzoai/cloud"
 	"github.com/hanzoai/cloud/apps/kms"
 	"github.com/hanzoai/cloud/plane"
+	"go.opentelemetry.io/otel"
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/trace"
 )
 
 // bridge.go is the ONE ChatBridge core: the platform-agnostic @hanzo front-door
@@ -38,6 +41,10 @@ import (
 
 // ── normalized inbound + reply seam ─────────────────────────────────────────
 
+// bridgeTracer emits the per-turn chat span — one tracer for every platform this
+// package bridges, because a turn is the same event whichever one it arrived on.
+var bridgeTracer = otel.Tracer("hanzo.ai/cloud/integrations")
+
 // Inbound is the normalized inbound chat event — ONE shape for every platform. An
 // adapter produces it AFTER it has authenticated the request and parsed the
 // payload. The core never sees a raw platform payload.
@@ -188,7 +195,49 @@ func (l *orgLimiter) release(org string) {
 func runBridgeTurn(s *cloud.Service[state], org string, in Inbound, reply replyFunc) {
 	ctx, cancel := context.WithTimeout(context.Background(), bridgeAgentTimeout)
 	defer cancel()
-	text, ephemeral := bridgeReply(s, org, in.Provider, in.ExternalID, in.User, in.Text)
+
+	// The turn's own span, and the one record that ties a CONVERSATION to a run.
+	//
+	// A chat turn had no telemetry at all: the webhook's request span ended when we
+	// answered the platform 200, and everything that matters happens after that, in
+	// the goroutine below. So "someone asked @hanzo something in this thread and a
+	// run happened" was two facts with nothing in common — the run knew its own id
+	// and never knew which thread caused it, and the thread knew nothing.
+	//
+	// It carries the run's id (returned by bridgeReply, below) precisely because the
+	// trace cannot be relied on to carry it here: the turn runs on a DETACHED
+	// context by necessity, and the framework forwards trace context only for a
+	// call that has an inbound request behind it — so on a split deployment the
+	// run's spans are in a trace of their own. The id is the join that holds
+	// regardless: thread -> this span -> run_id -> the run row -> its trace_id ->
+	// its spans. One chain, whether the fleet runs fused or as separate processes.
+	//
+	// It is opened here rather than per adapter because this is the one body all
+	// four platforms dispatch — Slack, Teams, Discord and Telegram — so a turn is
+	// observed the same way on every one of them.
+	ctx, span := bridgeTracer.Start(ctx, "agent.turn "+in.Provider, trace.WithSpanKind(trace.SpanKindInternal))
+	defer span.End()
+	span.SetAttributes(
+		// hanzo.org is the key the trace plane files a row under (apps/o11y
+		// planesink.go planeOrg); without it this span is the platform's, not the
+		// tenant's, and the tenant cannot read its own conversation.
+		attribute.String("hanzo.org", org),
+		attribute.String("hanzo.chat.provider", in.Provider),
+	)
+	// The thread, which is how a human arrives here: they are looking at a Slack
+	// conversation and want the run behind it. Absent on a platform or a turn that
+	// has none, rather than empty — a DM is legitimately unthreaded.
+	if in.Channel != "" {
+		span.SetAttributes(attribute.String("hanzo.chat.channel", in.Channel))
+	}
+	if in.ThreadID != "" {
+		span.SetAttributes(attribute.String("hanzo.chat.thread", in.ThreadID))
+	}
+
+	text, ephemeral, runID := bridgeReply(s, org, in.Provider, in.ExternalID, in.User, in.Text)
+	if runID != "" {
+		span.SetAttributes(attribute.String("hanzo.agent.run_id", runID))
+	}
 	if text == "" {
 		return
 	}
@@ -232,10 +281,16 @@ func bridgeRunContext(org string) (context.Context, context.CancelFunc) {
 // to pass the webhook's — which both cancels the run when we answer Slack and
 // silently discards the org. Removing the parameter is what makes that unavailable
 // rather than merely discouraged.
-func bridgeReply(s *cloud.Service[state], org, provider, externalID, user, text string) (reply string, ephemeral bool) {
+// It returns the RUN's id when a run happened, so the turn above can record which
+// run answered this conversation. The id was already in hand and was thrown away
+// everywhere except one failure log, which is why a thread and the run behind it
+// had no value in common.
+func bridgeReply(s *cloud.Service[state], org, provider, externalID, user, text string) (reply string, ephemeral bool, runID string) {
 	link, say, ephemeral := bridgeIdentity(s, org, provider, externalID, user)
 	if say != "" {
-		return say, ephemeral
+		// No run happened — an unlinked user gets a prompt, not an agent turn — so
+		// there is no id to name, and saying so with "" is the honest answer.
+		return say, ephemeral, ""
 	}
 	// On-behalf-of run over the PLANE (ZAP/UDS): org is the isolation gate, tenant
 	// and balance; the linked user's Hanzo subject drives attribution. No bearer,
@@ -285,7 +340,7 @@ func bridgeReply(s *cloud.Service[state], org, provider, externalID, user, text
 	}
 	if rerr != nil {
 		s.Log.Warn("bridge: agent run", "provider", provider, "org", org, "err", rerr) // never logs a token
-		return "Sorry — the agent hit an error handling that. Please try again.", false
+		return "Sorry — the agent hit an error handling that. Please try again.", false, run.RunID
 	}
 	if run.Status != "ok" {
 		// SAID, not just returned. A run that EXECUTED and whose model failed comes
@@ -297,12 +352,12 @@ func bridgeReply(s *cloud.Service[state], org, provider, externalID, user, text
 		// a day of looking. The run id is here so the row is findable.
 		s.Log.Warn("bridge: agent run did not succeed", "provider", provider, "org", org,
 			"status", run.Status, "run_id", run.RunID)
-		return "Sorry — the agent hit an error handling that. Please try again.", false
+		return "Sorry — the agent hit an error handling that. Please try again.", false, run.RunID
 	}
 	if strings.TrimSpace(run.Output) == "" {
-		return "(the agent returned an empty response)", false
+		return "(the agent returned an empty response)", false, run.RunID
 	}
-	return run.Output, false
+	return run.Output, false, run.RunID
 }
 
 // bridgeIdentity resolves the caller's linked Hanzo account, or the sentence to

apps/integrations/bridge_trace_test.go

@@ -0,0 +1,126 @@
+package integrations
+
+// A chat turn had no telemetry at all, and the reason is easy to miss: the
+// webhook's request span ends when we answer the platform 200, and everything
+// worth observing happens AFTER that, in a detached goroutine. So "someone asked
+// @hanzo something in this thread" and "a run happened" were two facts with
+// nothing in common.
+//
+// These pin the span that joins them, and the identity it has to carry.
+
+import (
+	"context"
+	"testing"
+
+	"github.com/hanzoai/cloud"
+	luxlog "github.com/luxfi/log"
+	sdktrace "go.opentelemetry.io/otel/sdk/trace"
+	"go.opentelemetry.io/otel/sdk/trace/tracetest"
+)
+
+// recordTurns rebinds the package tracer to an in-memory recorder, which is the
+// only honest place to assert from: a span that is created and never exported is
+// the failure this file exists to catch.
+func recordTurns(t *testing.T) *tracetest.SpanRecorder {
+	t.Helper()
+	sr := tracetest.NewSpanRecorder()
+	tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))
+	prev := bridgeTracer
+	bridgeTracer = tp.Tracer("test")
+	t.Cleanup(func() { bridgeTracer = prev; _ = tp.Shutdown(context.Background()) })
+	return sr
+}
+
+func turnAttr(sp sdktrace.ReadOnlySpan, key string) string {
+	for _, kv := range sp.Attributes() {
+		if string(kv.Key) == key {
+			return kv.Value.Emit()
+		}
+	}
+	return ""
+}
+
+// TestTurnRecordsTheConversationItCameFrom: the span a turn emits names the
+// tenant and the exact thread, so a human looking at a Slack conversation has a
+// value to search telemetry by.
+//
+// It asserts on a turn whose run does NOT succeed (there is no agents plane
+// here), which is deliberate: a failed turn is precisely when someone goes
+// looking, and instrumentation that only records the happy path is absent when it
+// is needed.
+func TestTurnRecordsTheConversationItCameFrom(t *testing.T) {
+	sr := recordTurns(t)
+	bridgeReady()
+
+	s := &cloud.Service[state]{Base: cloud.Base{Log: luxlog.New("test")}, State: state{}}
+	in := Inbound{
+		Provider:   "slack",
+		ExternalID: "T0FAKE",
+		User:       "U0FAKE",
+		Channel:    "C0FAKE",
+		ThreadID:   "1699999999.000100",
+		Text:       "what broke last night",
+	}
+	delivered := 0
+	runBridgeTurn(s, "acme", in, func(context.Context, string, bool) error {
+		delivered++
+		return nil
+	})
+
+	spans := sr.Ended()
+	if len(spans) != 1 {
+		t.Fatalf("a turn must emit exactly one span, got %d", len(spans))
+	}
+	sp := spans[0]
+	if sp.Name() != "agent.turn slack" {
+		t.Fatalf("span name = %q, want %q", sp.Name(), "agent.turn slack")
+	}
+	// The tenant, under the key the trace plane files rows by. Without it the
+	// conversation is filed as the platform's telemetry, not the tenant's, and the
+	// org that owns the workspace cannot read it.
+	if got := turnAttr(sp, "hanzo.org"); got != "acme" {
+		t.Fatalf("turn span must name its tenant, got %q", got)
+	}
+	// The thread — the way in. This is the whole point of the span.
+	for _, want := range []struct{ key, value string }{
+		{"hanzo.chat.provider", "slack"},
+		{"hanzo.chat.channel", "C0FAKE"},
+		{"hanzo.chat.thread", "1699999999.000100"},
+	} {
+		if got := turnAttr(sp, want.key); got != want.value {
+			t.Fatalf("turn span %s = %q, want %q", want.key, got, want.value)
+		}
+	}
+	// The turn still answered the person, which is what makes the failed-run case
+	// worth recording rather than dropping.
+	if delivered != 1 {
+		t.Fatalf("the turn delivered %d replies, want 1", delivered)
+	}
+}
+
+// TestUnthreadedTurnSaysNothingRatherThanEmpty: a DM legitimately has no thread,
+// and absence is a different fact from an empty one. An attribute stored blank is
+// a value every query returns and none can explain.
+func TestUnthreadedTurnSaysNothingRatherThanEmpty(t *testing.T) {
+	sr := recordTurns(t)
+	bridgeReady()
+
+	s := &cloud.Service[state]{Base: cloud.Base{Log: luxlog.New("test")}, State: state{}}
+	runBridgeTurn(s, "acme", Inbound{Provider: "slack", ExternalID: "T0", User: "U0", Channel: "D0", Text: "hi"},
+		func(context.Context, string, bool) error { return nil })
+
+	spans := sr.Ended()
+	if len(spans) != 1 {
+		t.Fatalf("got %d spans, want 1", len(spans))
+	}
+	for _, kv := range spans[0].Attributes() {
+		if string(kv.Key) == "hanzo.chat.thread" {
+			t.Fatalf("an unthreaded turn recorded a thread attribute (%q) — absent is the honest answer", kv.Value.Emit())
+		}
+	}
+	// The channel is still there: a DM has one, and it is how the conversation is
+	// found.
+	if got := turnAttr(spans[0], "hanzo.chat.channel"); got != "D0" {
+		t.Fatalf("channel = %q, want D0", got)
+	}
+}

apps/integrations/github_app.go

@@ -177,6 +177,325 @@ func githubConnection(org, owner string) (Connection, error) {
 	}
 }
 
+// githubInstallation is one place the App is installed: a GitHub user or org.
+type githubInstallation struct {
+	ID      int64  `json:"id"`
+	Login   string `json:"login"`
+	Type    string `json:"type"`
+	HTMLURL string `json:"htmlUrl,omitempty"`
+	// Grant is how much of the account the install covers: "all" repositories or
+	// only "selected" ones. GitHub returns it with the installation, so the reach
+	// of an install is known without spending a token to list its repositories.
+	Grant string `json:"grant,omitempty"`
+}
+
+// appInstallations lists every account the GitHub App is installed on.
+//
+// This is the App's OWN view — signed with the App JWT, not an installation
+// token — so it answers a question no installation-scoped call can: WHICH
+// accounts granted us anything. Without it, cloud could only speak about
+// installations it had already been told about at connect time, so an App
+// installed across a dozen orgs looked like nothing at all until somebody
+// pasted an installation id. That is the gap this closes: the console can
+// offer the real list to connect, and an agent asked "which of my GitHub orgs
+// can you see" has something true to answer with.
+//
+// It is NOT tenant data and must never be returned raw to a tenant — the App is
+// installed across every customer, so the full list is every customer's name.
+// githubInstallations below is the org-scoped projection, and it is what the
+// route serves.
+func appInstallations(ctx context.Context) ([]githubInstallation, error) {
+	tr, err := ghApp.transport()
+	if err != nil {
+		return nil, err
+	}
+	client := &http.Client{Transport: tr, Timeout: 20 * time.Second}
+	const perPage = 100
+	const maxPages = 20 // 2k installations; far past any real App
+	var all []githubInstallation
+	for page := 1; page <= maxPages; page++ {
+		endpoint := fmt.Sprintf("%s/app/installations?per_page=%d&page=%d",
+			strings.TrimRight(githubAPIBase, "/"), perPage, page)
+		req, rerr := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
+		if rerr != nil {
+			return nil, rerr
+		}
+		req.Header.Set("Accept", "application/vnd.github+json")
+		resp, derr := client.Do(req)
+		if derr != nil {
+			return nil, fmt.Errorf("github call: %w", derr)
+		}
+		body, _ := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
+		_ = resp.Body.Close()
+		if resp.StatusCode/100 != 2 {
+			return nil, fmt.Errorf("github http %d: %s", resp.StatusCode, truncateBody(body))
+		}
+		var pageResp []struct {
+			ID      int64 `json:"id"`
+			Account struct {
+				Login   string `json:"login"`
+				Type    string `json:"type"`
+				HTMLURL string `json:"html_url"`
+			} `json:"account"`
+			Selection string `json:"repository_selection"`
+		}
+		if uerr := json.Unmarshal(body, &pageResp); uerr != nil {
+			return nil, fmt.Errorf("github decode: %w", uerr)
+		}
+		for _, it := range pageResp {
+			all = append(all, githubInstallation{
+				ID: it.ID, Login: it.Account.Login,
+				Type: it.Account.Type, HTMLURL: it.Account.HTMLURL,
+				Grant: it.Selection,
+			})
+		}
+		if len(pageResp) < perPage {
+			break
+		}
+	}
+	return all, nil
+}
+
+// githubInstallationView is one installable account as an org may see it.
+type githubInstallationView struct {
+	// Login is the GitHub account name — the org or user the App is installed on.
+	Login string `json:"login"`
+	// Type is "Organization" or "User".
+	Type string `json:"type,omitempty"`
+	// Connected reports whether THIS org has already bound this account.
+	Connected bool `json:"connected"`
+	// HTMLURL is the account's page on GitHub.
+	HTMLURL string `json:"htmlUrl,omitempty"`
+	// Grant is "all" or "selected" — how many of the account's repositories the
+	// install covers. A reader deciding what to import needs the reach, not just
+	// the name.
+	Grant string `json:"grant,omitempty"`
+}
+
+// githubInstallationsOut is what the caller's org may see of the App's installs.
+type githubInstallationsOut struct {
+	// Installations is every account the caller may see: the ones its own org has
+	// bound, or — for a super admin — every account the App is installed on.
+	// Never null; [] when none.
+	Installations []githubInstallationView `json:"installations"`
+	// InstallURL is where to grant a new account, so a UI with an empty list has
+	// somewhere to send the reader instead of a dead end.
+	InstallURL string `json:"installUrl,omitempty"`
+}
+
+// githubInstallations lists the GitHub accounts the caller may see the App
+// installed on, each confirmed against the App's own list, plus where to add
+// another.
+//
+// The confirmation is the point. A connection row holds an installation id, and
+// an id whose installation was since removed on GitHub is a row that mints
+// nothing — every list and import against it fails with a token error, which
+// reads as "our git integration is broken" rather than "that install is gone".
+// Checking the App's view turns that into a fact the caller can act on.
+//
+// ORG-SCOPED for a tenant, deliberately. The App is installed across every
+// customer, so the raw list is the customer list; a tenant sees only accounts its
+// own org has bound. It discovers a NEW account by installing it (InstallURL),
+// which is GitHub's own consent screen — not by reading ours.
+//
+// A SUPER ADMIN sees the App's whole install list, because that list is the
+// platform's own inventory rather than any one tenant's data, and platform sudo
+// is the single cross-tenant scope this house has. Without it an App installed
+// out-of-band — granted straight from GitHub, so no connect flow ever ran and no
+// connection row exists — is invisible to everyone: the console card reads "not
+// connected" and an operator asked "which GitHub orgs do you see" can only
+// answer for accounts already bound, which is precisely the accounts that were
+// never the question.
+//
+// Response: {"installations":[{"login":"hanzoai","type":"Organization","connected":true,"htmlUrl":"https://github.com/hanzoai","grant":"all"}],"installUrl":"https://github.com/apps/hanzo/installations/new"}
+func (o ops) githubInstallations(ctx context.Context, _ *noArgs) (*githubInstallationsOut, error) {
+	org, err := authed(ctx, principalRequired)
+	if err != nil {
+		return nil, err
+	}
+	out := &githubInstallationsOut{
+		Installations: []githubInstallationView{},
+		InstallURL:    githubInstallURL(),
+	}
+	// What this org has bound. For a tenant this is the authority on WHICH
+	// accounts to show; for either caller it is the authority on `connected`.
+	conns := Connections(org, "github")
+	bound := make(map[int64]bool, len(conns))
+	for _, c := range conns {
+		if id, perr := strconv.ParseInt(strings.TrimSpace(c.ExternalID), 10, 64); perr == nil {
+			bound[id] = true
+		}
+	}
+	if superAdmin(ctx) {
+		// Here the App's list IS the answer, so a failed call is an error rather
+		// than a degraded view — reporting zero installs to the one caller who
+		// asked for the whole inventory would be a lie in the shape of a success.
+		all, aerr := appInstallations(ctx)
+		if aerr != nil {
+			return nil, zip.Errorf(http.StatusBadGateway, "list github installations: %v", aerr)
+		}
+		for _, in := range all {
+			out.Installations = append(out.Installations, githubInstallationView{
+				Login: in.Login, Type: in.Type, HTMLURL: in.HTMLURL,
+				Grant: in.Grant, Connected: bound[in.ID],
+			})
+		}
+		return out, nil
+	}
+	if len(conns) == 0 {
+		return out, nil
+	}
+	live := map[int64]githubInstallation{}
+	if all, aerr := appInstallations(ctx); aerr == nil {
+		for _, in := range all {
+			live[in.ID] = in
+		}
+	}
+	// A failed App call leaves `live` empty, and every row then reports
+	// connected=false rather than vanishing: an unreachable GitHub must not read
+	// as "you have no integrations".
+	for _, c := range conns {
+		v := githubInstallationView{Login: c.Owner, Connected: false}
+		if id, perr := strconv.ParseInt(strings.TrimSpace(c.ExternalID), 10, 64); perr == nil {
+			if in, ok := live[id]; ok {
+				v.Connected = true
+				if in.Login != "" {
+					v.Login = in.Login // GitHub is authoritative on a renamed account
+				}
+				v.Type, v.HTMLURL, v.Grant = in.Type, in.HTMLURL, in.Grant
+			}
+		}
+		out.Installations = append(out.Installations, v)
+	}
+	return out, nil
+}
+
+// githubClaimIn selects which held installations to bind. Name accounts or pass
+// all — neither is a 400, because "bind nothing" is not a request worth making.
+type githubClaimIn struct {
+	// Accounts names GitHub logins the App is installed on ("hanzoai"). Matched
+	// case-insensitively, since GitHub logins are. Ignored when all is true.
+	Accounts []string `json:"accounts"`
+	// All binds every account the App holds, instead of naming them.
+	All bool `json:"all"`
+}
+
+// githubClaimOut reports what the call bound and what was already bound, so a
+// second run is visibly a no-op rather than silently indistinguishable from the
+// first.
+type githubClaimOut struct {
+	// Claimed are the accounts this call bound. Never null; [] when none.
+	Claimed []string `json:"claimed"`
+	// Already were bound before the call and are unchanged by it.
+	Already []string `json:"already"`
+}
+
+// githubClaim binds installations the App ALREADY holds to the org the caller is
+// acting in — the reconciliation for a grant that happened outside our connect
+// flow.
+//
+// An installation IS the grant: GitHub recorded the consent when the App was
+// installed, and our connection row is bookkeeping that never got written because
+// nobody came through our callback. This writes that row from the App's own view,
+// so 23 accounts granted straight from GitHub stop reading as nothing.
+//
+// The org is taken from the VALIDATED PRINCIPAL and never from the body, because
+// it is the one part GitHub cannot tell us. An installation carries an account
+// login, a type and a repository selection — nothing that names a Hanzo org. So
+// the binding cannot be DERIVED, only asserted, and the only unforgeable assertion
+// available is the org the caller is already acting in. Inferring one from the
+// account name would be a guess the store cannot catch: its key is
+// (org,provider,owner), so a wrong org is a valid row, and a valid row is a
+// mirror pointed at the wrong tenant.
+//
+// SUPER ADMIN only, for that same reason. A tenant's proof that an account is
+// theirs is GitHub's own consent screen — the connect flow — and without it any
+// org could claim any account the App holds. Platform sudo is already the scope
+// that reads the whole install list, so it is the scope that may bind from it;
+// giving a tenant this verb would hand it every other tenant's repositories.
+//
+// Idempotent: the row is keyed (org,provider,owner) and connected_at survives an
+// upsert, so claiming twice rebinds the same account to the same org and reports
+// it under `already`. Re-claiming also REFRESHES the installation id, so an
+// account reinstalled on GitHub — new id, same login — self-heals instead of
+// minting tokens against a dead installation.
+//
+// Response: {"claimed":["hanzoai","luxfi"],"already":["zooai"]}
+func (o ops) githubClaim(ctx context.Context, in *githubClaimIn) (*githubClaimOut, error) {
+	org, err := authed(ctx, principalRequired)
+	if err != nil {
+		return nil, err
+	}
+	if !superAdmin(ctx) {
+		return nil, zip.Errorf(http.StatusForbidden,
+			"claiming an installation is platform sudo; connect the account to grant it to this org")
+	}
+	if !in.All && len(in.Accounts) == 0 {
+		return nil, zip.ErrBadRequest("name accounts to claim, or pass all")
+	}
+	held, err := appInstallations(ctx)
+	if err != nil {
+		return nil, zip.Errorf(http.StatusBadGateway, "list github installations: %v", err)
+	}
+	byLogin := make(map[string]githubInstallation, len(held))
+	for _, ins := range held {
+		byLogin[strings.ToLower(ins.Login)] = ins
+	}
+
+	want := held
+	if !in.All {
+		want = nil
+		var absent []string
+		for _, name := range in.Accounts {
+			ins, ok := byLogin[strings.ToLower(strings.TrimSpace(name))]
+			if !ok {
+				absent = append(absent, name)
+				continue
+			}
+			want = append(want, ins)
+		}
+		// Refused WHOLE, before any write. A named account the App does not hold
+		// is a caller's mistake, and binding the rest of the list around it would
+		// leave a half-applied request whose result depends on argument order.
+		if len(absent) > 0 {
+			return nil, zip.Errorf(http.StatusBadRequest,
+				"the app holds no installation for %s", strings.Join(absent, ", "))
+		}
+	}
+
+	out := &githubClaimOut{Claimed: []string{}, Already: []string{}}
+	for _, ins := range want {
+		_, bound, gerr := o.s.State.store.Get(ctx, org, "github", ins.Login)
+		if gerr != nil {
+			return nil, gerr
+		}
+		// Written even when bound, so a reinstalled account's new id lands.
+		if uerr := o.s.State.store.Upsert(ctx, Connection{
+			Org: org, Provider: "github", Owner: ins.Login,
+			ExternalID: strconv.FormatInt(ins.ID, 10), AccountLabel: ins.Login,
+		}); uerr != nil {
+			return nil, uerr
+		}
+		if bound {
+			out.Already = append(out.Already, ins.Login)
+			continue
+		}
+		out.Claimed = append(out.Claimed, ins.Login)
+	}
+	return out, nil
+}
+
+// githubInstallURL is where a reader grants the App another account. The App's
+// public slug is the one piece a deployment configures; without it the console
+// still renders the list and simply offers no "add" link.
+func githubInstallURL() string {
+	slug := strings.TrimSpace(os.Getenv("GITHUB_APP_SLUG"))
+	if slug == "" {
+		return ""
+	}
+	return "https://github.com/apps/" + slug + "/installations/new"
+}
+
 // githubInstallationAccount fetches an installation's account login via the App JWT,
 // which both VALIDATES the installation (the App can see it) and yields the human
 // label. Used by githubExchange at connect time.

apps/integrations/github_claim_test.go

@@ -0,0 +1,227 @@
+package integrations
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/zap-proto/zip"
+)
+
+// github_claim_test.go proves the binding verb: who may bind an installation the
+// App already holds, to WHICH org, and that running it twice changes nothing.
+//
+// The property that must never be wrong is isolation. The store's key is
+// (org,provider,owner), so a binding to the wrong org is a perfectly valid row —
+// nothing downstream can catch it, and the row points a repository mirror at the
+// wrong tenant. The gate is therefore the only thing standing between a tenant
+// and every other tenant's repositories.
+
+// postJSON sends a typed op an In body, optionally as platform sudo.
+func postJSON(t *testing.T, app *zip.App, path, org string, super bool, body any) httpResult {
+	t.Helper()
+	b, _ := json.Marshal(body)
+	rq := httptest.NewRequest(http.MethodPost, path, bytes.NewReader(b))
+	rq.Header.Set("Content-Type", "application/json")
+	if org != "" {
+		rq.Header.Set("X-Org-Id", org)
+		rq.Header.Set("X-User-Id", "u-"+org)
+	}
+	if super {
+		rq.Header.Set("X-User-IsAdmin", "true")
+	}
+	resp, err := app.Test(rq)
+	if err != nil {
+		t.Fatalf("Test POST %s: %v", path, err)
+	}
+	defer func() { _ = resp.Body.Close() }()
+	raw, _ := io.ReadAll(resp.Body)
+	return httpResult{Code: resp.StatusCode, Body: raw}
+}
+
+func claimOut(t *testing.T, body []byte) githubClaimOut {
+	t.Helper()
+	var out githubClaimOut
+	if err := json.Unmarshal(body, &out); err != nil {
+		t.Fatalf("json: %v (%s)", err, body)
+	}
+	return out
+}
+
+// TestTenantCannotClaim is the isolation guard. A plain org — even an admin of its
+// OWN org — may not bind an account the App holds; its route to a grant is GitHub's
+// consent screen. Nothing may be written by the refused call.
+func TestTenantCannotClaim(t *testing.T) {
+	withGithubApp(t, mockInstallations(t, twoAccounts()))
+	app := newApp(t, newKMS(t))
+
+	for _, name := range []string{"plain member", "own-org admin"} {
+		rq := httptest.NewRequest(http.MethodPost, "/v1/integrations/github/claim",
+			bytes.NewReader([]byte(`{"all":true}`)))
+		rq.Header.Set("Content-Type", "application/json")
+		rq.Header.Set("X-Org-Id", "acme")
+		rq.Header.Set("X-User-Id", "u-acme")
+		if name == "own-org admin" {
+			// The own-org admin bit must NOT be mistaken for platform sudo.
+			rq.Header.Set("X-User-IsOrgAdmin", "true")
+		}
+		resp, err := app.Test(rq)
+		if err != nil {
+			t.Fatalf("%s: %v", name, err)
+		}
+		_ = resp.Body.Close()
+		if resp.StatusCode != http.StatusForbidden {
+			t.Fatalf("%s claiming want 403, got %d", name, resp.StatusCode)
+		}
+	}
+
+	// Refused means NOTHING was written: acme holds no connection, so it still
+	// cannot reach a single repository.
+	if conns := Connections("acme", "github"); len(conns) != 0 {
+		t.Fatalf("refused claim must write no rows, got %+v", conns)
+	}
+	if r := req(t, app, http.MethodGet, "/v1/integrations/github/repos", "acme", nil); r.Code != http.StatusConflict {
+		t.Fatalf("acme should still be unconnected (409), got %d (%s)", r.Code, r.Body)
+	}
+}
+
+// TestClaimBindsToCallerOrgOnly proves the target org comes from the validated
+// principal, so one caller's claim can never land in another org's rows.
+func TestClaimBindsToCallerOrgOnly(t *testing.T) {
+	withGithubApp(t, mockInstallations(t, twoAccounts()))
+	app := newApp(t, newKMS(t))
+
+	r := postJSON(t, app, "/v1/integrations/github/claim", "hanzo", true, map[string]any{"all": true})
+	if r.Code != http.StatusOK {
+		t.Fatalf("claim want 200, got %d (%s)", r.Code, r.Body)
+	}
+	if got := claimOut(t, r.Body); len(got.Claimed) != 2 {
+		t.Fatalf("want both accounts claimed, got %+v", got)
+	}
+	if conns := Connections("hanzo", "github"); len(conns) != 2 {
+		t.Fatalf("hanzo should hold 2 connections, got %d", len(conns))
+	}
+	// The org that did not ask holds nothing.
+	if conns := Connections("acme", "github"); len(conns) != 0 {
+		t.Fatalf("a claim must not touch another org, acme holds %+v", conns)
+	}
+}
+
+// TestClaimIsIdempotent proves a second run changes nothing: the same bindings,
+// reported under `already`, with connected_at preserved.
+func TestClaimIsIdempotent(t *testing.T) {
+	withGithubApp(t, mockInstallations(t, twoAccounts()))
+	app := newApp(t, newKMS(t))
+
+	first := claimOut(t, postJSON(t, app, "/v1/integrations/github/claim", "hanzo", true,
+		map[string]any{"all": true}).Body)
+	if len(first.Claimed) != 2 || len(first.Already) != 0 {
+		t.Fatalf("first claim should bind both, got %+v", first)
+	}
+	before, _, err := mounted.State.store.Get(context.Background(), "hanzo", "github", "hanzoai")
+	if err != nil {
+		t.Fatalf("get: %v", err)
+	}
+
+	second := claimOut(t, postJSON(t, app, "/v1/integrations/github/claim", "hanzo", true,
+		map[string]any{"all": true}).Body)
+	if len(second.Claimed) != 0 || len(second.Already) != 2 {
+		t.Fatalf("second claim should bind nothing, got %+v", second)
+	}
+	if conns := Connections("hanzo", "github"); len(conns) != 2 {
+		t.Fatalf("re-claiming must not duplicate rows, got %d", len(conns))
+	}
+	after, _, err := mounted.State.store.Get(context.Background(), "hanzo", "github", "hanzoai")
+	if err != nil {
+		t.Fatalf("get: %v", err)
+	}
+	if after.ConnectedAt != before.ConnectedAt {
+		t.Fatalf("connected since must survive a re-claim: %d -> %d", before.ConnectedAt, after.ConnectedAt)
+	}
+	if after.ExternalID != "111" {
+		t.Fatalf("installation id should still be 111, got %q", after.ExternalID)
+	}
+}
+
+// TestClaimNamedAccounts proves a named subset binds only what was named, and that
+// a name the App does not hold refuses the WHOLE call rather than half-applying it.
+func TestClaimNamedAccounts(t *testing.T) {
+	withGithubApp(t, mockInstallations(t, twoAccounts()))
+	app := newApp(t, newKMS(t))
+
+	// Case-insensitive, since GitHub logins are.
+	got := claimOut(t, postJSON(t, app, "/v1/integrations/github/claim", "lux", true,
+		map[string]any{"accounts": []string{"LuxFi"}}).Body)
+	if len(got.Claimed) != 1 || got.Claimed[0] != "luxfi" {
+		t.Fatalf("want [luxfi] claimed, got %+v", got)
+	}
+	if conns := Connections("lux", "github"); len(conns) != 1 {
+		t.Fatalf("only the named account binds, got %d", len(conns))
+	}
+
+	// One unknown name refuses everything — no partial write.
+	r := postJSON(t, app, "/v1/integrations/github/claim", "lux", true,
+		map[string]any{"accounts": []string{"hanzoai", "nope"}})
+	if r.Code != http.StatusBadRequest {
+		t.Fatalf("unknown account want 400, got %d (%s)", r.Code, r.Body)
+	}
+	if conns := Connections("lux", "github"); len(conns) != 1 {
+		t.Fatalf("a refused claim must write nothing, got %d rows", len(conns))
+	}
+
+	// Naming nothing at all is a 400, not a silent success.
+	if r := postJSON(t, app, "/v1/integrations/github/claim", "lux", true,
+		map[string]any{}); r.Code != http.StatusBadRequest {
+		t.Fatalf("empty claim want 400, got %d (%s)", r.Code, r.Body)
+	}
+}
+
+// TestClaimRefreshesReinstalledAccount proves a stale binding self-heals: an
+// account removed and reinstalled on GitHub keeps its login but gets a new
+// installation id, and a row still holding the dead id mints nothing.
+func TestClaimRefreshesReinstalledAccount(t *testing.T) {
+	withGithubApp(t, mockInstallations(t, twoAccounts()))
+	app := newApp(t, newKMS(t))
+	if err := mounted.State.store.Upsert(context.Background(), Connection{
+		Org: "hanzo", Provider: "github", Owner: "hanzoai",
+		ExternalID: "999", AccountLabel: "hanzoai", // the dead installation
+	}); err != nil {
+		t.Fatalf("upsert: %v", err)
+	}
+
+	got := claimOut(t, postJSON(t, app, "/v1/integrations/github/claim", "hanzo", true,
+		map[string]any{"accounts": []string{"hanzoai"}}).Body)
+	if len(got.Already) != 1 {
+		t.Fatalf("an existing binding reports already, got %+v", got)
+	}
+	after, _, err := mounted.State.store.Get(context.Background(), "hanzo", "github", "hanzoai")
+	if err != nil {
+		t.Fatalf("get: %v", err)
+	}
+	if after.ExternalID != "111" {
+		t.Fatalf("claim should refresh the installation id to 111, got %q", after.ExternalID)
+	}
+}
+
+// TestClaimThenInstallationsReadConnected closes the loop: after claiming, the
+// list the operator reads reports the accounts as connected.
+func TestClaimThenInstallationsReadConnected(t *testing.T) {
+	withGithubApp(t, mockInstallations(t, twoAccounts()))
+	app := newApp(t, newKMS(t))
+
+	postJSON(t, app, "/v1/integrations/github/claim", "hanzo", true, map[string]any{"all": true})
+
+	got := installations(t, superReq(t, app, http.MethodGet, "/v1/integrations/github/installations", "hanzo").Body)
+	if len(got) != 2 {
+		t.Fatalf("want 2 installations, got %d", len(got))
+	}
+	for _, v := range got {
+		if !v.Connected {
+			t.Fatalf("claimed account should read connected: %+v", v)
+		}
+	}
+}

apps/integrations/github_installations_test.go

@@ -0,0 +1,183 @@
+package integrations
+
+import (
+	"context"
+	"encoding/json"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/zap-proto/zip"
+)
+
+// github_installations_test.go proves who may see WHICH GitHub accounts the App is
+// installed on. The App is installed across every customer, so the raw list is the
+// customer list: a tenant must see only what its own org bound, and only platform
+// sudo may read the whole inventory.
+//
+// The case that motivated it: an App granted straight from GitHub runs no connect
+// flow, so no connection row exists and every org-scoped surface reports nothing —
+// the console card reads "not connected" and an agent asked "which of my GitHub
+// orgs do you see" has no true answer to give.
+
+// mockInstallations serves GET /app/installations (paginated like GitHub's) plus
+// the access-token mint, so the App-JWT path under test is the production one.
+func mockInstallations(t *testing.T, accounts []map[string]any) *httptest.Server {
+	t.Helper()
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch {
+		case r.URL.Path == "/app/installations":
+			// Page 1 carries the set; later pages are empty, which ends the walk.
+			if r.URL.Query().Get("page") != "1" {
+				_ = json.NewEncoder(w).Encode([]map[string]any{})
+				return
+			}
+			_ = json.NewEncoder(w).Encode(accounts)
+		case strings.HasSuffix(r.URL.Path, "/access_tokens"):
+			_ = json.NewEncoder(w).Encode(map[string]any{"token": "ghs_x"})
+		default:
+			w.WriteHeader(http.StatusNotFound)
+		}
+	}))
+	t.Cleanup(srv.Close)
+	return srv
+}
+
+// superReq is req with the platform-sudo header the identity boundary mints only
+// for a validated owner == "admin" — what principal.IsSuperAdmin reads.
+func superReq(t *testing.T, app *zip.App, method, path, org string) httpResult {
+	t.Helper()
+	rq := httptest.NewRequest(method, path, nil)
+	rq.Header.Set("X-Org-Id", org)
+	rq.Header.Set("X-User-Id", "u-"+org)
+	rq.Header.Set("X-User-IsAdmin", "true")
+	resp, err := app.Test(rq)
+	if err != nil {
+		t.Fatalf("Test %s %s: %v", method, path, err)
+	}
+	defer func() { _ = resp.Body.Close() }()
+	b, _ := io.ReadAll(resp.Body)
+	return httpResult{Code: resp.StatusCode, Location: resp.Header.Get("Location"), Body: b}
+}
+
+// twoAccounts is an App installed on two orgs, neither bound to any Hanzo org —
+// the out-of-band shape.
+func twoAccounts() []map[string]any {
+	return []map[string]any{
+		{"id": 111, "repository_selection": "all", "account": map[string]any{
+			"login": "hanzoai", "type": "Organization", "html_url": "https://github.com/hanzoai"}},
+		{"id": 222, "repository_selection": "selected", "account": map[string]any{
+			"login": "luxfi", "type": "Organization", "html_url": "https://github.com/luxfi"}},
+	}
+}
+
+func installations(t *testing.T, body []byte) []githubInstallationView {
+	t.Helper()
+	var out githubInstallationsOut
+	if err := json.Unmarshal(body, &out); err != nil {
+		t.Fatalf("json: %v (%s)", err, body)
+	}
+	return out.Installations
+}
+
+// TestSuperAdminSeesUnboundInstallations is the regression: platform sudo reads the
+// App's whole install list even when nothing has been connected. Before the fix the
+// handler returned early on len(conns)==0 and answered [] for every caller, so an
+// App installed out-of-band was invisible to the platform that owns it.
+func TestSuperAdminSeesUnboundInstallations(t *testing.T) {
+	withGithubApp(t, mockInstallations(t, twoAccounts()))
+	app := newApp(t, newKMS(t))
+
+	r := superReq(t, app, http.MethodGet, "/v1/integrations/github/installations", "admin")
+	if r.Code != http.StatusOK {
+		t.Fatalf("super admin installations want 200, got %d (%s)", r.Code, r.Body)
+	}
+	got := installations(t, r.Body)
+	if len(got) != 2 {
+		t.Fatalf("super admin should see both installs, got %d: %+v", len(got), got)
+	}
+	if got[0].Login != "hanzoai" || got[1].Login != "luxfi" {
+		t.Fatalf("want [hanzoai luxfi], got %+v", got)
+	}
+	// Reach comes back with the account, so a reader knows what an import covers.
+	if got[0].Grant != "all" || got[1].Grant != "selected" {
+		t.Fatalf("want grants [all selected], got %q %q", got[0].Grant, got[1].Grant)
+	}
+	// Nothing is bound, so nothing claims to be connected.
+	for _, v := range got {
+		if v.Connected {
+			t.Fatalf("unbound install must not report connected: %+v", v)
+		}
+	}
+}
+
+// TestSuperAdminConnectedReflectsBinding proves `connected` still answers about the
+// CALLER's org, not about the install existing.
+func TestSuperAdminConnectedReflectsBinding(t *testing.T) {
+	withGithubApp(t, mockInstallations(t, twoAccounts()))
+	app := newApp(t, newKMS(t))
+	if err := mounted.State.store.Upsert(context.Background(), Connection{
+		Org: "admin", Provider: "github", ExternalID: "111", AccountLabel: "hanzoai",
+	}); err != nil {
+		t.Fatalf("upsert: %v", err)
+	}
+
+	got := installations(t, superReq(t, app, http.MethodGet, "/v1/integrations/github/installations", "admin").Body)
+	if len(got) != 2 {
+		t.Fatalf("want 2 installs, got %d", len(got))
+	}
+	if !got[0].Connected {
+		t.Fatalf("bound install should report connected: %+v", got[0])
+	}
+	if got[1].Connected {
+		t.Fatalf("unbound install must not report connected: %+v", got[1])
+	}
+}
+
+// TestTenantNeverSeesOtherAccounts is the isolation guard: a plain org reads only
+// what it bound, never the App's inventory. This is the property the super-admin
+// breadth must not have widened.
+func TestTenantNeverSeesOtherAccounts(t *testing.T) {
+	withGithubApp(t, mockInstallations(t, twoAccounts()))
+	app := newApp(t, newKMS(t))
+	if err := mounted.State.store.Upsert(context.Background(), Connection{
+		Org: "acme", Provider: "github", ExternalID: "111", AccountLabel: "hanzoai",
+	}); err != nil {
+		t.Fatalf("upsert: %v", err)
+	}
+
+	// acme bound one account → it sees exactly that one, never luxfi.
+	got := installations(t, req(t, app, http.MethodGet, "/v1/integrations/github/installations", "acme", nil).Body)
+	if len(got) != 1 || got[0].Login != "hanzoai" {
+		t.Fatalf("tenant should see only its bound account, got %+v", got)
+	}
+
+	// beta bound nothing → it sees nothing, though two installs exist.
+	if got := installations(t, req(t, app, http.MethodGet, "/v1/integrations/github/installations", "beta", nil).Body); len(got) != 0 {
+		t.Fatalf("unbound tenant must see no installs, got %+v", got)
+	}
+
+	// No principal → 403, same as every org-scoped op here.
+	if r := req(t, app, http.MethodGet, "/v1/integrations/github/installations", "", nil); r.Code != http.StatusForbidden {
+		t.Fatalf("no-principal installations want 403, got %d", r.Code)
+	}
+}
+
+// TestSuperAdminInstallationsUpstreamFailure proves the platform view fails loudly.
+// For a tenant the connection rows are the answer and the App call only annotates
+// them, so an outage degrades; for a super admin the App call IS the answer, and
+// answering [] would read as "the App is installed nowhere".
+func TestSuperAdminInstallationsUpstreamFailure(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusInternalServerError)
+	}))
+	t.Cleanup(srv.Close)
+	withGithubApp(t, srv)
+	app := newApp(t, newKMS(t))
+
+	if r := superReq(t, app, http.MethodGet, "/v1/integrations/github/installations", "admin"); r.Code != http.StatusBadGateway {
+		t.Fatalf("super admin installations on upstream failure want 502, got %d (%s)", r.Code, r.Body)
+	}
+}

apps/integrations/integrations.go

@@ -863,6 +863,9 @@ func routes(app cloud.Router, zapp *zip.App, s *cloud.Service[state]) {
 	// It stays raw because it speaks GitHub's webhook protocol: the HMAC covers
 	// the raw body, which an op handed the decoded In could not re-verify.
 	app.Post("/v1/connector/github/webhook", cloud.Terminal(cloud.Handle(s, githubWebhook)))
+	zip.Get(zapp, "/v1/integrations/github/installations", o.githubInstallations)
+	// Bind installations the App already holds to the org the caller acts in.
+	zip.Post(zapp, "/v1/integrations/github/claim", o.githubClaim)
 	zip.Get(zapp, "/v1/integrations/github/repos", o.githubRepos)
 	// 202: the import runs in a bounded background worker, so the op DECLARES the
 	// status it has always answered rather than setting it per request.

apps/integrations/ops.go

@@ -70,6 +70,11 @@ type facts struct {
 	// admin is principal.IsOrgAdmin — admin OF ONE'S OWN org, the AdminOnly
 	// connector gate. NOT SuperAdmin (see principal's two-predicate note).
 	admin bool
+	// super is principal.IsSuperAdmin — platform sudo, the one CROSS-TENANT
+	// scope. Kept as its own fact rather than folded into admin: an org admin
+	// who inherited platform breadth is a privilege escalation, so the two
+	// answers stay two values.
+	super bool
 	// user is the validated principal's user id, the per-USER connector plane's
 	// row key. CLONED at the bridge: c.User() is a zero-copy view into the reused
 	// fasthttp request buffer, and this value keys rows and KMS paths that outlive
@@ -88,6 +93,7 @@ type factsKey struct{}
 func bridgeFacts(c *zip.Ctx) error {
 	c.SetContext(context.WithValue(c.Context(), factsKey{}, facts{
 		admin: principal.IsOrgAdmin(c),
+		super: principal.IsSuperAdmin(c),
 		user:  strings.Clone(strings.TrimSpace(c.User())),
 	}))
 	return c.Next()
@@ -101,6 +107,10 @@ func factsOf(ctx context.Context) facts {
 // orgAdmin reports the bridged own-org admin bit.
 func orgAdmin(ctx context.Context) bool { return factsOf(ctx).admin }
 
+// superAdmin reports the bridged platform-sudo bit. Absence of a bridge reads
+// false, so an unbridged request is never platform-privileged.
+func superAdmin(ctx context.Context) bool { return factsOf(ctx).super }
+
 // caller resolves the validated (org,user) pair the per-USER connector plane keys
 // every row by: authed's two steps plus the user id, refused 400 when it could not
 // be a path segment. Unchanged from the raw form it replaces — missing identity is

apps/integrations/ops_projection_test.go

@@ -34,11 +34,13 @@ var typedOps = []string{
 	"POST /v1/connectors/:provider/device/:flow/poll",
 	"GET /v1/integrations",
 	"GET /v1/integrations/:provider",
+	"GET /v1/integrations/github/installations",
 	"GET /v1/integrations/github/repos",
 	"GET /v1/integrations/github/repos/:repo/pages",
 	"POST /v1/integrations/:provider/connect",
 	"POST /v1/integrations/:provider/disconnect",
 	"POST /v1/integrations/:provider/verify",
+	"POST /v1/integrations/github/claim",
 	"POST /v1/integrations/github/issues/backfill",
 	"POST /v1/integrations/github/repos/:repo/pages",
 	"POST /v1/integrations/github/repos/:repo/pages/builds",
@@ -56,10 +58,10 @@ var typedOps = []string{
 // "202 Accepted" was a third until zip v1.18.2 gave WithStatus a vocabulary for
 // it; /repos/import and /pages/builds are typed ops now.
 var rawRoutes = map[string]string{
-	"GET /v1/integrations/:provider/callback":     "302 to the console",
-	"GET /v1/integrations/discord/link":           "302",
-	"GET /v1/integrations/discord/link/callback":  "302",
-	"GET /v1/integrations/discord/link/discord":   "302",
+	"GET /v1/integrations/:provider/callback":    "302 to the console",
+	"GET /v1/integrations/discord/link":          "302",
+	"GET /v1/integrations/discord/link/callback": "302",
+	"GET /v1/integrations/discord/link/discord":  "302",
 	// The Marketplace / "Add to Slack" entry point. Unauthenticated BY DESIGN —
 	// the person clicking Install in Slack's directory has no Hanzo session — and
 	// it reveals nothing: the consent URL it 302s to carries only the PUBLIC

apps/integrations/slack_events.go

@@ -14,6 +14,8 @@ import (
 
 	"github.com/hanzoai/cloud"
 	"github.com/zap-proto/zip"
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/trace"
 )
 
 // slack_events.go is the SLACK ADAPTER of the ChatBridge core (bridge.go): the Slack
@@ -333,7 +335,13 @@ func slackSlashTurn(s *cloud.Service[state], org string, in Inbound, responseURL
 			return
 		}
 	}
-	text, ephemeral := bridgeReply(s, org, in.Provider, in.ExternalID, in.User, in.Text)
+	// A slash command answers synchronously on the request, which already has a
+	// span; the run id is recorded on it for the same reason the async turn records
+	// one — so this invocation can be joined to the run it caused.
+	text, ephemeral, runID := bridgeReply(s, org, in.Provider, in.ExternalID, in.User, in.Text)
+	if runID != "" {
+		trace.SpanFromContext(ctx).SetAttributes(attribute.String("hanzo.agent.run_id", runID))
+	}
 	slackSlashReply(s, ctx, in, responseURL, text, ephemeral)
 }
 

apps/integrations/zipdoc_gen.go

@@ -105,6 +105,19 @@ func init() {
 		Example:  json.RawMessage(`{"provider":"slack"}`),
 		Response: json.RawMessage(`{"id":"slack","name":"Slack","description":"Connect your workspace.","category":"Communication","available":true,"connected":false}`),
 	})
+	zip.Describe("GET /v1/integrations/github/installations", zip.Doc{
+		Description: "Lists the GitHub accounts the caller may see the App\ninstalled on, each confirmed against the App's own list, plus where to add\nanother.\n\nThe confirmation is the point. A connection row holds an installation id, and\nan id whose installation was since removed on GitHub is a row that mints\nnothing — every list and import against it fails with a token error, which\nreads as \"our git integration is broken\" rather than \"that install is gone\".\nChecking the App's view turns that into a fact the caller can act on.\n\nORG-SCOPED for a tenant, deliberately. The App is installed across every\ncustomer, so the raw list is the customer list; a tenant sees only accounts its\nown org has bound. It discovers a NEW account by installing it (InstallURL),\nwhich is GitHub's own consent screen — not by reading ours.\n\nA SUPER ADMIN sees the App's whole install list, because that list is the\nplatform's own inventory rather than any one tenant's data, and platform sudo\nis the single cross-tenant scope this house has. Without it an App installed\nout-of-band — granted straight from GitHub, so no connect flow ever ran and no\nconnection row exists — is invisible to everyone: the console card reads \"not\nconnected\" and an operator asked \"which GitHub orgs do you see\" can only\nanswer for accounts already bound, which is precisely the accounts that were\nnever the question.",
+		Fields: map[string]string{
+			"githubInstallationView.connected":     "Connected reports whether THIS org has already bound this account.",
+			"githubInstallationView.grant":         "Grant is \"all\" or \"selected\" — how many of the account's repositories the\ninstall covers. A reader deciding what to import needs the reach, not just\nthe name.",
+			"githubInstallationView.htmlUrl":       "HTMLURL is the account's page on GitHub.",
+			"githubInstallationView.login":         "Login is the GitHub account name — the org or user the App is installed on.",
+			"githubInstallationView.type":          "Type is \"Organization\" or \"User\".",
+			"githubInstallationsOut.installUrl":    "InstallURL is where to grant a new account, so a UI with an empty list has\nsomewhere to send the reader instead of a dead end.",
+			"githubInstallationsOut.installations": "Installations is every account the caller may see: the ones its own org has\nbound, or — for a super admin — every account the App is installed on.\nNever null; [] when none.",
+		},
+		Response: json.RawMessage(`{"installations":[{"login":"hanzoai","type":"Organization","connected":true,"htmlUrl":"https://github.com/hanzoai","grant":"all"}],"installUrl":"https://github.com/apps/hanzo/installations/new"}`),
+	})
 	zip.Describe("GET /v1/integrations/github/repos", zip.Doc{
 		Description: "Lists the org's granted GitHub repositories, each annotated with its\nnative import + sync status from the git object plane. Org-authed: the org comes\nfrom the validated principal, and the granted set is bounded to THAT org's\ninstallation token — an org can never enumerate another org's repos. The console\npolls it to watch an import flip a repo to imported.",
 		Fields: map[string]string{
@@ -260,6 +273,16 @@ func init() {
 		Example:  json.RawMessage(`{"provider":"cloudflare"}`),
 		Response: json.RawMessage(`{"provider":"cloudflare","active":true,"account":"Acme","externalId":"a1b2c3","scopes":["zone:read"]}`),
 	})
+	zip.Describe("POST /v1/integrations/github/claim", zip.Doc{
+		Description: "Binds installations the App ALREADY holds to the org the caller is\nacting in — the reconciliation for a grant that happened outside our connect\nflow.\n\nAn installation IS the grant: GitHub recorded the consent when the App was\ninstalled, and our connection row is bookkeeping that never got written because\nnobody came through our callback. This writes that row from the App's own view,\nso 23 accounts granted straight from GitHub stop reading as nothing.\n\nThe org is taken from the VALIDATED PRINCIPAL and never from the body, because\nit is the one part GitHub cannot tell us. An installation carries an account\nlogin, a type and a repository selection — nothing that names a Hanzo org. So\nthe binding cannot be DERIVED, only asserted, and the only unforgeable assertion\navailable is the org the caller is already acting in. Inferring one from the\naccount name would be a guess the store cannot catch: its key is\n(org,provider,owner), so a wrong org is a valid row, and a valid row is a\nmirror pointed at the wrong tenant.\n\nSUPER ADMIN only, for that same reason. A tenant's proof that an account is\ntheirs is GitHub's own consent screen — the connect flow — and without it any\norg could claim any account the App holds. Platform sudo is already the scope\nthat reads the whole install list, so it is the scope that may bind from it;\ngiving a tenant this verb would hand it every other tenant's repositories.\n\nIdempotent: the row is keyed (org,provider,owner) and connected_at survives an\nupsert, so claiming twice rebinds the same account to the same org and reports\nit under `already`. Re-claiming also REFRESHES the installation id, so an\naccount reinstalled on GitHub — new id, same login — self-heals instead of\nminting tokens against a dead installation.",
+		Fields: map[string]string{
+			"githubClaimIn.accounts": "Accounts names GitHub logins the App is installed on (\"hanzoai\"). Matched\ncase-insensitively, since GitHub logins are. Ignored when all is true.",
+			"githubClaimIn.all":      "All binds every account the App holds, instead of naming them.",
+			"githubClaimOut.already": "Already were bound before the call and are unchanged by it.",
+			"githubClaimOut.claimed": "Claimed are the accounts this call bound. Never null; [] when none.",
+		},
+		Response: json.RawMessage(`{"claimed":["hanzoai","luxfi"],"already":["zooai"]}`),
+	})
 	zip.Describe("POST /v1/integrations/github/issues/backfill", zip.Doc{
 		Description: "Seeds the native tracker with the EXISTING issues across the\norg's granted repos (default state=open); the webhook keeps them live thereafter.\nOrg-scoped by the validated principal — a caller only ever backfills its OWN org.\nSynchronous + bounded (a total time budget and an issue cap) so it returns the\ncounts directly; idempotent by ExtRef, so a re-run continues where a truncated\npass left off and never duplicates.",
 		Fields: map[string]string{

apps/k8s/k8s.go

@@ -42,6 +42,14 @@ var Pods = schema.GroupVersionResource{Version: "v1", Resource: "pods"}
 // copy of the tenant's data. Nothing here deletes one.
 var Volumes = schema.GroupVersionResource{Version: "v1", Resource: "persistentvolumeclaims"}
 
+// RuntimeClasses is the isolation boundary a pod may name (node.k8s.io/v1), and
+// the object that says WHERE a pod naming it may land: `scheduling.nodeSelector`
+// and `scheduling.tolerations` are merged by the apiserver into every such pod.
+// That merge is why the sandbox reads this rather than writing a nodeSelector of
+// its own — the class already holds the fact, and a second copy in a pod spec is
+// one that goes stale the day the pool moves.
+var RuntimeClasses = schema.GroupVersionResource{Group: "node.k8s.io", Version: "v1", Resource: "runtimeclasses"}
+
 // CDApplications is Hanzo CD's Application CR (apps.hanzo.ai/v1alpha1) — the
 // GitOps plane's own record of a tracked git source: the revision it last applied,
 // its sync verdict, and its deploy history. A different fact from Apps: an App is

apps/o11y/planesink_agentseam_test.go

@@ -0,0 +1,103 @@
+package o11y
+
+// The seam between what an agent RUN emits and what this package files it under.
+//
+// planeOrg reads one key — hanzo.org — per span, and falls back to the platform's
+// own org when it is absent. The agent spans used to name their tenant
+// hanzo.agent.org, a spelling nothing here reads, so every run/step/tool span was
+// stored as PLATFORM telemetry: absent from the org-scoped read the console
+// issues, and sitting in the platform's bucket with that tenant's tool names and
+// user subjects in it. Both halves looked correct in isolation, which is exactly
+// why the seam needs a test of its own.
+//
+// The other half lives in apps/agents (TestEverySpanOfARunIsFiledUnderItsTenant),
+// which asserts every span a run produces carries this key. Together they are the
+// contract; either alone is the bug that shipped.
+
+import (
+	"context"
+	"testing"
+
+	"go.opentelemetry.io/otel/attribute"
+	sdktrace "go.opentelemetry.io/otel/sdk/trace"
+	"go.opentelemetry.io/otel/sdk/trace/tracetest"
+	"go.opentelemetry.io/otel/trace"
+)
+
+// TestAgentRunSpansAreFiledUnderTheirTenant renders the exact attribute set an
+// agent run emits and asserts each span lands on the tenant's rows — never the
+// platform's.
+func TestAgentRunSpansAreFiledUnderTheirTenant(t *testing.T) {
+	sr := tracetest.NewSpanRecorder()
+	tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))
+	tr := tp.Tracer("hanzo.ai/cloud/agents")
+
+	// The three spans a run is, each with the attributes apps/agents sets on it.
+	ctx, run := tr.Start(context.Background(), "agent.run helper", trace.WithSpanKind(trace.SpanKindInternal))
+	run.SetAttributes(
+		attribute.String("hanzo.agent.name", "helper"),
+		attribute.String("hanzo.org", "acme"),
+		attribute.String("hanzo.agent.run_id", "run_123"),
+		attribute.String("hanzo.user", "u-acme"),
+	)
+	ctx, step := tr.Start(ctx, "agent.step", trace.WithSpanKind(trace.SpanKindInternal))
+	step.SetAttributes(
+		attribute.String("hanzo.agent.run_id", "run_123"),
+		attribute.String("hanzo.org", "acme"),
+	)
+	_, tool := tr.Start(ctx, "agent.tool post_v1_search_query", trace.WithSpanKind(trace.SpanKindInternal))
+	tool.SetAttributes(
+		attribute.String("gen_ai.tool.name", "post_v1_search_query"),
+		attribute.String("hanzo.org", "acme"),
+		attribute.String("hanzo.agent.run_id", "run_123"),
+	)
+	tool.End()
+	step.End()
+	run.End()
+
+	rows := sdkSpanRowsOf(sr.Ended())
+	if len(rows) != 3 {
+		t.Fatalf("got %d rows, want 3 (run, step, tool)", len(rows))
+	}
+	for _, row := range rows {
+		name := spanCol(t, row, "name")
+		if got := spanCol(t, row, "org"); got != "acme" {
+			t.Fatalf("%v is filed under %v, want acme — a run's span stored as %q is "+
+				"invisible to the tenant that ran it", name, got, platformOrg)
+		}
+	}
+
+	// The trace SUMMARY is keyed (org, trace_id), so a run whose spans disagreed
+	// about their tenant would be split into two partials — the trace list would
+	// then report a fraction of the spans and the wrong duration for each half.
+	// One tenant, one summary row.
+	partials := traceRowsOf(rows)
+	if len(partials) != 1 {
+		t.Fatalf("got %d trace summary rows, want 1 — the run's spans disagree about their tenant", len(partials))
+	}
+	if got := partials[0][0]; got != "acme" {
+		t.Fatalf("the trace summary is filed under %v, want acme", got)
+	}
+	if got := partials[0][4]; got != uint64(3) {
+		t.Fatalf("the summary counts %v spans, want 3", got)
+	}
+}
+
+// TestAnAgentSpanMissingItsTenantIsThePlatforms is the negative that names the
+// failure: this is precisely what every run looked like before the attribute was
+// corrected, and it is what a future rename would silently restore.
+func TestAnAgentSpanMissingItsTenantIsThePlatforms(t *testing.T) {
+	sr := tracetest.NewSpanRecorder()
+	tp := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))
+	_, span := tp.Tracer("hanzo.ai/cloud/agents").Start(context.Background(), "agent.run helper")
+	// The old spelling: a tenant named in a key nothing reads.
+	span.SetAttributes(attribute.String("hanzo.agent.org", "acme"))
+	span.End()
+
+	rows := sdkSpanRowsOf(sr.Ended())
+	if got := spanCol(t, rows[0], "org"); got != platformOrg {
+		t.Fatalf("org = %v; a span naming its tenant under an unread key must fall "+
+			"back to %q — if this now resolves the tenant, planeOrg reads more than one key",
+			got, platformOrg)
+	}
+}

apps/platform/apps.go

@@ -308,7 +308,7 @@ func launchDeclareBuild(s *cloud.Service[state], c *zip.Ctx, req declareReq, org
 	// Charged to the ORG that asked, not to the fabric pool: the concurrency
 	// ceiling is per-org, so one org looping deploys can only exhaust its own
 	// share instead of locking every other org out of building (red F3).
-	job, err := s.State.k8s.launchDirectBuild(c.Context(), org, url, ref, image, dockerfile, id)
+	job, err := s.State.k8s.launchDirectBuild(c.Context(), org, url, ref, image, dockerfile, id, nil)
 	if err != nil {
 		s.Log.Error("declare build failed to launch", "app", name, "err", err)
 		return nil, zip.Errorf(http.StatusBadGateway, "could not launch the build: %v", err)

apps/platform/buildargs_test.go

@@ -0,0 +1,119 @@
+package platform
+
+// The `--build-arg`s an image declares.
+//
+// These exist because three `images:` entries off ONE Dockerfile, differing only
+// by `args: {STAGE: exec|dev|desktop}`, were all built as the stage the
+// Dockerfile happens to default to. Nothing failed: three tags were published,
+// each a copy of the same image, under three names that said otherwise. An
+// `exec` tag — the class that is supposed to be a minimal, volumeless
+// interpreter box — carried a whole X server.
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestBuildArgsAreSortedSoOneCommitIsOneCacheKey(t *testing.T) {
+	// A map has no order. Emitting in range order makes two builds of the SAME
+	// commit two different argv, hence two cache keys and two digests — which is
+	// the property the digest-pinned lane is built on.
+	got, err := buildArgs(map[string]string{"ZULU": "1", "ALPHA": "2", "MIKE": "3"})
+	if err != nil {
+		t.Fatalf("buildArgs: %v", err)
+	}
+	want := []string{
+		"--opt", "build-arg:ALPHA=2",
+		"--opt", "build-arg:MIKE=3",
+		"--opt", "build-arg:ZULU=1",
+	}
+	if len(got) != len(want) {
+		t.Fatalf("got %d opts, want %d: %v", len(got), len(want), got)
+	}
+	for i := range want {
+		if got[i] != want[i] {
+			t.Errorf("opt %d = %v, want %q", i, got[i], want[i])
+		}
+	}
+}
+
+func TestBuildArgsRefusesWhatWouldSplitAnOpt(t *testing.T) {
+	for name, args := range map[string]map[string]string{
+		"a name carrying = is two opts": {"A=B": "x"},
+		"a name that reads as a flag":   {"--opt": "x"},
+		"an empty name":                 {"": "x"},
+		"a name with a space":           {"A B": "x"},
+		"a value carrying a newline":    {"A": "x\ny"},
+		"a value carrying a NUL":        {"A": "x\x00y"},
+		"a name starting with a digit":  {"1A": "x"},
+	} {
+		if _, err := buildArgs(args); err == nil {
+			t.Errorf("%s: accepted %v, want refusal", name, args)
+		}
+	}
+	// The ordinary case still passes — a refusal that refuses everything is not
+	// a check, it is an outage.
+	if _, err := buildArgs(map[string]string{"STAGE": "desktop", "NODE_MAJOR": "24"}); err != nil {
+		t.Errorf("refused a plain declaration: %v", err)
+	}
+}
+
+func TestDeclaredArgsCannotForgeTheReceipts(t *testing.T) {
+	// VERSION and REVISION are derived from the tag and the commit. If a repo's
+	// own hanzo.yml could set them, an image could name a commit it was not built
+	// from — and those two values are exactly what "which of these is the
+	// release" is answered with.
+	cmd, err := buildFrontendCmdArgs(
+		"https://github.com/hanzoai/bot.git#refs/heads/main", "Dockerfile.box",
+		"oci.hanzo.ai/hanzoai/sandbox:1.0.1-dev", "fedeb0e5d0b3f7a9b171823a4f3ecf33686e5b1a",
+		map[string]string{"STAGE": "dev", "VERSION": "9.9.9", "REVISION": "0000000000000000000000000000000000000000"},
+	)
+	if err != nil {
+		t.Fatalf("buildFrontendCmdArgs: %v", err)
+	}
+	var got []string
+	for _, a := range cmd {
+		if s, ok := a.(string); ok && strings.HasPrefix(s, "build-arg:") {
+			got = append(got, s)
+		}
+	}
+	joined := strings.Join(got, " ")
+	if !strings.Contains(joined, "build-arg:STAGE=dev") {
+		t.Errorf("STAGE was dropped; the class selector never reached the build: %v", got)
+	}
+	if strings.Contains(joined, "VERSION=9.9.9") {
+		t.Errorf("a declared VERSION overwrote the tag-derived one: %v", got)
+	}
+	if strings.Contains(joined, "REVISION=0000000000000000000000000000000000000000") {
+		t.Errorf("a declared REVISION overwrote the commit: %v", got)
+	}
+	if !strings.Contains(joined, "build-arg:REVISION=fedeb0e5d0b3f7a9b171823a4f3ecf33686e5b1a") {
+		t.Errorf("the true commit is missing: %v", got)
+	}
+}
+
+func TestOurOwnRegistryIsNameableByItsCanonicalHost(t *testing.T) {
+	// oci.hanzo.ai and registry.hanzo.ai are ONE store. Refusing the canonical
+	// name while accepting the deprecated alias is a rule about spelling, not
+	// about reach.
+	for _, image := range []string{
+		"oci.hanzo.ai/hanzoai/sandbox:1.0.1-dev",
+		"registry.hanzo.ai/hanzoai/sandbox:1.0.1-dev",
+		"ghcr.io/hanzoai/cloud:v1",
+	} {
+		if !imageAllowed(image) {
+			t.Errorf("imageAllowed(%q) = false, want true", image)
+		}
+	}
+	// And the bound still holds: a host we do not operate, and a namespace we do
+	// not own on a host we do.
+	for _, image := range []string{
+		"docker.io/library/node:22",
+		"evil.example.com/hanzoai/sandbox:1",
+		"oci.hanzo.ai/globex/private:v1",
+	} {
+		if imageAllowed(image) {
+			t.Errorf("imageAllowed(%q) = true, want false", image)
+		}
+	}
+}

apps/platform/k8s.go

@@ -921,6 +921,31 @@ func buildFrontendCmd(buildCtx, dockerfile, image string) []any {
 	return buildFrontendCmdRev(buildCtx, dockerfile, image, "")
 }
 
+// buildFrontendCmdArgs is buildFrontendCmdRev plus the image's declared
+// `--build-arg`s. They are appended AFTER the derived ones, and that order is
+// the policy: buildkit takes the LAST value for a repeated build-arg, so a
+// declaration cannot overwrite... which is exactly why VERSION and REVISION are
+// filtered out of the declared set instead of relying on position. Those two are
+// receipts — the tag the image is published under and the commit it was built
+// from — and an image that can name a different commit than it was built from is
+// the one lie the whole digest-pinned lane exists to prevent.
+func buildFrontendCmdArgs(buildCtx, dockerfile, image, revision string, args map[string]string) ([]any, error) {
+	cmd := buildFrontendCmdRev(buildCtx, dockerfile, image, revision)
+	declared := make(map[string]string, len(args))
+	for k, v := range args {
+		switch k {
+		case "VERSION", "GIT_VERSION", "REVISION":
+			continue
+		}
+		declared[k] = v
+	}
+	extra, err := buildArgs(declared)
+	if err != nil {
+		return nil, err
+	}
+	return append(cmd, extra...), nil
+}
+
 // buildFrontendCmdRev is buildFrontendCmd plus the commit being built, which is
 // the difference between an image that can be traced back to source and one that
 // cannot.
@@ -1292,7 +1317,23 @@ func (k *k8sClient) buildJobSpec(jobName, org, app, pushSecret string, command [
 						// scheduler avoid a full node; the limit bounds a runaway build
 						// instead of letting it take the node down for everything else.
 						"resources": map[string]any{
-							"requests": map[string]any{"ephemeral-storage": "50Gi"},
+							// 32Gi, not 50: the request has to FIT beside buildkitd. Each
+							// runner-pool-32g node allocates ~88Gi of ephemeral storage and
+							// the buildkitd DaemonSet reserves 48Gi of it on every one, so a
+							// 50Gi request left no node able to hold the pod and every build
+							// sat Pending forever — "0/34 nodes are available … 3 Insufficient
+							// ephemeral-storage", with the autoscaler declining to scale up
+							// because no larger node matched either. Six builds queued for
+							// five days and a whole repo stopped publishing images, with
+							// nothing failing anywhere to say so.
+							//
+							// The reason the request exists is unchanged and still right: a
+							// best-effort pod gets placed on a full node and is evicted first.
+							// It just has to be a number the pool can actually satisfy. 32Gi
+							// leaves headroom on the 40Gi a node has free beside buildkitd,
+							// and matches what cloud's own builds already request and
+							// schedule with. The limit stays 80Gi so a big build still bursts.
+							"requests": map[string]any{"ephemeral-storage": "32Gi"},
 							"limits":   map[string]any{"ephemeral-storage": "80Gi"},
 						},
 						"securityContext": map[string]any{
@@ -1370,7 +1411,7 @@ func (k *k8sClient) buildJobSpec(jobName, org, app, pushSecret string, command [
 // looping deploys locked every other org out of building, with no attribution in
 // the Job labels to see it by. /v1/runner still passes platformBuildOrg (its
 // builds ARE the fabric's); a tenant deploy passes the tenant.
-func (k *k8sClient) launchDirectBuild(ctx context.Context, org, repoURL, ref, image, dockerfile, buildID string) (string, error) {
+func (k *k8sClient) launchDirectBuild(ctx context.Context, org, repoURL, ref, image, dockerfile, buildID string, args map[string]string) (string, error) {
 	if strings.TrimSpace(org) == "" {
 		return "", fmt.Errorf("a build must be attributed to an org")
 	}
@@ -1411,7 +1452,10 @@ func (k *k8sClient) launchDirectBuild(ctx context.Context, org, repoURL, ref, im
 	}
 	jobName := truncate("pf-runner-"+jobIDSuffix(buildID), 63)
 	buildCtx := strings.TrimSuffix(cleanURL, ".git") + ".git#" + cleanRef
-	command := buildFrontendCmdRev(buildCtx, cleanDockerfile, image, cleanRef)
+	command, err := buildFrontendCmdArgs(buildCtx, cleanDockerfile, image, cleanRef, args)
+	if err != nil {
+		return "", fmt.Errorf("invalid build input: %w", err)
+	}
 	pushSecret, err := buildPushSecret(image)
 	if err != nil {
 		return "", err

apps/platform/release.go

@@ -386,7 +386,7 @@ func launchRelease(s *cloud.Service[state], ctx context.Context, ref, repo, dock
 func releaseFor(s *cloud.Service[state], repoURL, sha, image, tag, dockerfile, bldID string) releasePlan {
 	return releasePlan{
 		build: func(ctx context.Context) error {
-			job, err := s.State.k8s.launchDirectBuild(ctx, platformBuildOrg, repoURL, sha, image, dockerfile, bldID)
+			job, err := s.State.k8s.launchDirectBuild(ctx, platformBuildOrg, repoURL, sha, image, dockerfile, bldID, nil)
 			if err != nil {
 				return fmt.Errorf("launch build: %w", err)
 			}

apps/platform/runner.go

@@ -51,6 +51,13 @@ type runnerBuildReq struct {
 	Context string `json:"context,omitempty" url:"-"`
 	// DockerTarget is the multi-stage build target to stop at.
 	DockerTarget string `json:"dockerTarget,omitempty" url:"-"`
+	// Args are --build-arg values. They are what lets several images off ONE
+	// Dockerfile mean different things — the sandbox classes are three entries
+	// differing only by STAGE. Validated at the k8s choke point, with VERSION and
+	// REVISION taking precedence: those are receipts the builder derives from the
+	// tag and the commit, and a caller that could overwrite them could make an
+	// image lie about which commit it is.
+	Args map[string]string `json:"args,omitempty" url:"-"`
 	// OS is the target operating system for the artifact lane.
 	OS string `json:"os,omitempty" url:"-"`
 	// Arch is the target architecture for the artifact lane.
@@ -101,7 +108,11 @@ type runnerBuildResp struct {
 // other host is NEVER allowed on the privileged build path. registry.hanzo.ai is
 // the self-hosted fleet registry (the native CI/CD home); ghcr stays during the
 // migration as the public mirror.
-var ownedRegistryHosts = []string{"registry.hanzo.ai", "ghcr.io"}
+// oci.hanzo.ai and registry.hanzo.ai are ONE store behind one Traefik router,
+// not two registries: naming the canonical host here grants no reach the
+// deprecated alias did not already have, and omitting it refused the very name
+// the fleet is told to write.
+var ownedRegistryHosts = []string{"oci.hanzo.ai", "registry.hanzo.ai", "ghcr.io"}
 
 // orgRegistryNamespaces maps an IAM org (the validated `owner` claim) to the
 // registry namespace(s) that org OWNS. Only the three brands that own a registry
@@ -414,7 +425,7 @@ func (o ops) runnerBuild(ctx context.Context, body *runnerBuildReq) (*runnerBuil
 		return nil, zip.Errorf(http.StatusInternalServerError, "rng: %v", err)
 	}
 
-	jobName, err := s.State.k8s.launchDirectBuild(ctx, platformBuildOrg, req.Repo, ref, req.Image, strings.TrimSpace(req.Dockerfile), bldID)
+	jobName, err := s.State.k8s.launchDirectBuild(ctx, platformBuildOrg, req.Repo, ref, req.Image, strings.TrimSpace(req.Dockerfile), bldID, req.Args)
 	if err != nil {
 		return nil, zip.Errorf(deployErrStatus(err), "launch build: %v", err)
 	}

apps/platform/validate.go

@@ -27,6 +27,7 @@ import (
 	"fmt"
 	"net/url"
 	"regexp"
+	"sort"
 	"strconv"
 	"strings"
 
@@ -226,6 +227,54 @@ func validateDockerfile(raw string) (string, error) {
 	return s, nil
 }
 
+// buildArgNameRE is a Dockerfile `ARG` name: the C-identifier shape docker
+// itself accepts, and nothing else. A name is half of one `build-arg:K=V` argv
+// element, so anything that could read as a second `=` or a leading dash is
+// simply not a name.
+var buildArgNameRE = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]{0,63}$`)
+
+// buildArgs turns a declared `args:` map into ordered `build-arg:K=V` opts.
+//
+// SORTED, because a map has no order and two builds of one commit that pass the
+// same args in two orders are two different cache keys — the thing the whole
+// digest-pinned lane exists to prevent. Sorting makes the argv a function of the
+// declaration alone.
+//
+// These are NOT a privilege boundary and the limits here do not pretend to be
+// one: `args:` lives in hanzo.yml, in the repo, beside the Dockerfile that
+// declares which ARGs exist. Whoever can add an arg can already edit the
+// Dockerfile that reads it. What the limits close is the argv itself — a name
+// carrying `=` or a leading `-`, or a value carrying a NUL or a newline, is how
+// one opt becomes two.
+func buildArgs(args map[string]string) ([]any, error) {
+	if len(args) == 0 {
+		return nil, nil
+	}
+	if len(args) > 64 {
+		return nil, fmt.Errorf("too many build args (%d, max 64)", len(args))
+	}
+	names := make([]string, 0, len(args))
+	for k := range args {
+		names = append(names, k)
+	}
+	sort.Strings(names)
+	out := make([]any, 0, 2*len(names))
+	for _, k := range names {
+		if !buildArgNameRE.MatchString(k) {
+			return nil, fmt.Errorf("build arg %q is not a valid ARG name", k)
+		}
+		v := args[k]
+		if len(v) > 1024 {
+			return nil, fmt.Errorf("build arg %q value is too long (%d, max 1024)", k, len(v))
+		}
+		if strings.ContainsAny(v, "\x00\n\r") {
+			return nil, fmt.Errorf("build arg %q value contains a control character", k)
+		}
+		out = append(out, "--opt", "build-arg:"+k+"="+v)
+	}
+	return out, nil
+}
+
 // validateGitRef enforces a safe git ref/commit that can be placed into the
 // `#<ref>` fragment of a buildctl git context without breaking it or smuggling a
 // flag/shell token. Empty is allowed by the caller (it defaults to the branch);

apps/projects/projects.go

@@ -309,6 +309,11 @@ func Mount(app cloud.Router, deps cloud.Deps) error {
 
 	routes(app, s)
 
+	// The public per-site tag door (GET /v1/tags), served by THIS process because it
+	// reads THIS process's project store in-process — the same reason the key/site/scope
+	// resolvers above are registered here rather than reached across the plane.
+	mountTagDoor(app, s)
+
 	// The site edge must hand the browser the bytes we published, unedited: a
 	// Cloudflare zone with an HTML rewriter on breaks every hydrating app the
 	// plane serves (see sites.rewriters). Assert it off-thread so a slow or

apps/projects/tagdoor.go

@@ -1,4 +1,4 @@
-package destinations
+package projects
 
 import (
 	"encoding/json"
@@ -7,41 +7,41 @@ import (
 	"sort"
 	"strings"
 
-	"github.com/hanzoai/cloud/apps/projects"
+	"github.com/hanzoai/cloud"
 	"github.com/hanzoai/cloud/openapi"
+	"github.com/zap-proto/zip"
 )
 
-// tags.go — GET /v1/tags, the PUBLIC per-site browser-tag config the hosted tag
-// (track.js) fetches to know which client-side pixels to inject.
+// tagdoor.go — GET /v1/tags, the PUBLIC per-site browser-tag config the hosted tag
+// (track.js / /v1/event.js) fetches to know which client-side pixels to inject.
 //
-// PER SITE, not per org: a project IS a site and carries its own pixel ids
-// (projects.Project.Tags); track.js is dual-resolved to the right site — by the
-// publishable KEY when it is a per-site project key, else by the request HOST (an
-// org-level key + where the tag runs). So hanzo.ai and hanzo.chat inject different pixels
-// under one org, and the server-side CAPI reads the same per-site ids. Public + pk--keyed
-// like /v1/event(.js); non-secret ids only; FAIL-SAFE to an empty set so a page never
-// breaks on its tag config.
+// It lives HERE, in the projects app, because a project IS a site and carries its own
+// Project.Tags — and this is the process that OWNS the project store. (It first lived in
+// the destinations app and read empty in production: destinations runs in a different
+// process, so its cross-app reach for the store resolved to nil. The rule the key
+// resolver already follows: the door that reads the project store must be served by the
+// process that holds it.)
 //
-// GATEWAY: like /v1/event(.js) this must be reachable without a validated session — the
-// browser presents only the publishable key — so it rides the same public allowlist.
+// Dual-resolved per site: by the publishable KEY when it is a per-site project key
+// (ResolveKey), else by the request HOST (ResolveHost) for an org-level key — so
+// hanzo.ai and hanzo.chat inject different pixels under one org. Public + pk--keyed like
+// /v1/event(.js); NON-SECRET ids only; FAIL-SAFE to an empty set so a page never breaks.
 
 // browserTags names the platforms with a client-side pixel track.js can inject, and the
 // injector `type` it dispatches on. A platform absent here forwards server-side only.
 var browserTags = map[string]string{
-	"ga4":    "ga",     // gtag('config', G-…)
-	"meta":   "meta",   // fbq('init', …)
-	"tiktok": "tiktok", // ttq.load(…)
-	"x":      "x",      // twq('config', …)
+	"ga4":    "ga",
+	"meta":   "meta",
+	"tiktok": "tiktok",
+	"x":      "x",
 }
 
-// browserTagOut is one injectable tag on the wire: the platform, the injector type, the id.
 type browserTagOut struct {
 	Platform string `json:"platform"`
 	Type     string `json:"type"`
 	ID       string `json:"id"`
 }
 
-// tagConfig is the /v1/tags response: the site's injectable browser tags. Never a secret.
 type tagConfig struct {
 	Tags []browserTagOut `json:"tags"`
 }
@@ -58,6 +58,14 @@ func init() {
 			"site it answers an empty set at 200 — a page never breaks on its tag config.")
 }
 
+// mountTagDoor registers GET /v1/tags as a public, raw net/http handler (like analytics'
+// /v1/event.js) that reads THIS process's project store directly.
+func mountTagDoor(app cloud.Router, s *cloud.Service[state]) {
+	app.Get("/v1/tags", zip.AdaptNetHTTP(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		serveTags(s, w, r)
+	})))
+}
+
 // buildTags maps a site's tag config (platform → non-secret pixel id) to injectable
 // browser tags, in stable platform order. A platform with no client pixel, or an empty
 // id, is omitted.
@@ -71,7 +79,7 @@ func buildTags(tags map[string]string) []browserTagOut {
 	for _, platform := range platforms {
 		typ, ok := browserTags[platform]
 		if !ok {
-			continue // server-side-only destination — no browser pixel to inject
+			continue
 		}
 		if id := strings.TrimSpace(tags[platform]); id != "" {
 			out = append(out, browserTagOut{Platform: platform, Type: typ, ID: id})
@@ -80,8 +88,7 @@ func buildTags(tags map[string]string) []browserTagOut {
 	return out
 }
 
-// tagsKey lifts the publishable key: Authorization: Bearer first, then ?key= (the tag's
-// data-key), then the retiring ?ingest_key=.
+// tagsKey lifts the publishable key: Authorization: Bearer first, then ?key=, then ?ingest_key=.
 func tagsKey(r *http.Request) string {
 	if b := r.Header.Get("Authorization"); strings.HasPrefix(b, "Bearer ") {
 		if k := strings.TrimSpace(strings.TrimPrefix(b, "Bearer ")); k != "" {
@@ -89,7 +96,10 @@ func tagsKey(r *http.Request) string {
 		}
 	}
 	q := r.URL.Query()
-	return firstNonEmpty(strings.TrimSpace(q.Get("key")), strings.TrimSpace(q.Get("ingest_key")))
+	if k := strings.TrimSpace(q.Get("key")); k != "" {
+		return k
+	}
+	return strings.TrimSpace(q.Get("ingest_key"))
 }
 
 // tagsHost lifts the site host for the org-key derivation path: ?host= first, else the
@@ -97,15 +107,14 @@ func tagsKey(r *http.Request) string {
 func tagsHost(r *http.Request) string {
 	q := r.URL.Query()
 	for _, raw := range []string{q.Get("host"), r.Header.Get("Origin"), r.Header.Get("Referer")} {
-		if h := hostname(raw); h != "" {
+		if h := hostnameOf(raw); h != "" {
 			return h
 		}
 	}
 	return ""
 }
 
-// hostname reduces an origin/URL/bare-host to its hostname ("" if none).
-func hostname(raw string) string {
+func hostnameOf(raw string) string {
 	raw = strings.TrimSpace(raw)
 	if raw == "" {
 		return ""
@@ -124,16 +133,33 @@ func hostname(raw string) string {
 	return raw
 }
 
-// serveTags writes the site's browser tag config. Public + cross-origin (the tag loads
-// from every property), non-secret, fail-safe. A raw net/http handler so it sets CORS +
-// cache directly, exactly like /v1/event.js. Dual-resolves the site via projects.TagsFor.
-func serveTags(w http.ResponseWriter, r *http.Request) {
+// resolveSiteTags dual-resolves the site and returns its Project.Tags, reading THIS
+// process's store directly (in-process; no cross-process reach). nil,false when nothing
+// resolves — the door then answers an empty set.
+func resolveSiteTags(s *cloud.Service[state], r *http.Request) (map[string]string, bool) {
+	ctx := r.Context()
+	if key := tagsKey(r); key != "" {
+		if p, err := s.State.store.ResolveKey(ctx, key); err == nil {
+			return p.Tags, true
+		}
+	}
+	if host := tagsHost(r); host != "" {
+		if p, err := s.State.store.ResolveHost(ctx, host); err == nil {
+			return p.Tags, true
+		}
+	}
+	return nil, false
+}
+
+// serveTags writes the site's browser tag config. Public + cross-origin, non-secret,
+// fail-safe. A raw net/http handler so it sets CORS + cache directly, like /v1/event.js.
+func serveTags(s *cloud.Service[state], w http.ResponseWriter, r *http.Request) {
 	w.Header().Set("Content-Type", "application/json; charset=utf-8")
 	w.Header().Set("Access-Control-Allow-Origin", "*")
 	w.Header().Set("Cache-Control", "public, max-age=60")
 
 	out := tagConfig{Tags: []browserTagOut{}}
-	if tags, ok := projects.TagsFor(r.Context(), tagsKey(r), tagsHost(r)); ok {
+	if tags, ok := resolveSiteTags(s, r); ok {
 		out.Tags = buildTags(tags)
 	}
 	body, err := json.Marshal(out)

apps/projects/tagdoor_test.go

@@ -0,0 +1,97 @@
+package projects
+
+import (
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/hanzoai/cloud"
+)
+
+func TestBuildTags(t *testing.T) {
+	tags := buildTags(map[string]string{
+		"ga4": "G-ABC", "meta": "123", "x": "o1",
+		"reddit": "a2", "google-ads": "9", "tiktok": "",
+	})
+	if len(tags) != 3 {
+		t.Fatalf("want 3 (ga4/meta/x), got %d: %+v", len(tags), tags)
+	}
+	by := map[string]browserTagOut{}
+	for _, tg := range tags {
+		by[tg.Platform] = tg
+	}
+	if by["ga4"].Type != "ga" || by["ga4"].ID != "G-ABC" {
+		t.Errorf("ga4 → %+v", by["ga4"])
+	}
+	if by["meta"].Type != "meta" || by["meta"].ID != "123" {
+		t.Errorf("meta → %+v", by["meta"])
+	}
+	if by["x"].Type != "x" || by["x"].ID != "o1" {
+		t.Errorf("x → %+v", by["x"])
+	}
+	if _, has := by["reddit"]; has {
+		t.Error("reddit has no browser pixel and must be omitted")
+	}
+	if _, has := by["tiktok"]; has {
+		t.Error("empty id must be omitted")
+	}
+}
+
+func TestTagsKeyHost(t *testing.T) {
+	mk := func(u, auth, origin, ref string) *http.Request {
+		r := httptest.NewRequest(http.MethodGet, u, nil)
+		if auth != "" {
+			r.Header.Set("Authorization", auth)
+		}
+		if origin != "" {
+			r.Header.Set("Origin", origin)
+		}
+		if ref != "" {
+			r.Header.Set("Referer", ref)
+		}
+		return r
+	}
+	if k := tagsKey(mk("/v1/tags?key=pk-q", "", "", "")); k != "pk-q" {
+		t.Errorf("?key= → %q", k)
+	}
+	if k := tagsKey(mk("/v1/tags", "Bearer pk-b", "", "")); k != "pk-b" {
+		t.Errorf("Bearer → %q", k)
+	}
+	if k := tagsKey(mk("/v1/tags?key=pk-q", "Bearer pk-b", "", "")); k != "pk-b" {
+		t.Errorf("Bearer must win → %q", k)
+	}
+	if h := tagsHost(mk("/v1/tags?host=hanzo.ai", "", "", "")); h != "hanzo.ai" {
+		t.Errorf("?host= → %q", h)
+	}
+	if h := tagsHost(mk("/v1/tags", "", "https://hanzo.chat", "")); h != "hanzo.chat" {
+		t.Errorf("Origin → %q", h)
+	}
+	if h := tagsHost(mk("/v1/tags", "", "", "https://hanzo.app/x?y=1")); h != "hanzo.app" {
+		t.Errorf("Referer → %q", h)
+	}
+}
+
+// TestServeTagsFailSafe: no key + no host ⇒ resolveSiteTags never touches the store ⇒
+// an empty set at 200 with permissive CORS, so a page never breaks on its tag config.
+func TestServeTagsFailSafe(t *testing.T) {
+	s := &cloud.Service[state]{}
+	req := httptest.NewRequest(http.MethodGet, "/v1/tags", nil)
+	w := httptest.NewRecorder()
+	serveTags(s, w, req)
+
+	res := w.Result()
+	if res.StatusCode != http.StatusOK {
+		t.Fatalf("status = %d, want 200", res.StatusCode)
+	}
+	if res.Header.Get("Access-Control-Allow-Origin") != "*" {
+		t.Error("the tag loads cross-origin — must allow any origin")
+	}
+	var out tagConfig
+	if err := json.NewDecoder(res.Body).Decode(&out); err != nil {
+		t.Fatalf("decode: %v", err)
+	}
+	if len(out.Tags) != 0 {
+		t.Errorf("no resolvable site must yield an empty tag set, got %+v", out.Tags)
+	}
+}

apps/projects/tags.go

@@ -1,39 +0,0 @@
-package projects
-
-import (
-	"context"
-	"strings"
-)
-
-// tags.go — TagsFor, the seam the destinations tag door (GET /v1/tags) reads to answer
-// which browser pixels a SITE injects. A project IS a site, and it carries its own
-// Project.Tags (platform → non-secret pixel id); this resolves the right site TWO ways so
-// one org's many sites never share a tag config:
-//
-//   - by the publishable KEY, when it is a per-site project key (ResolveKey);
-//   - else by the request HOST, when the caller presented an org-level key and the site
-//     is decided by where the tag runs (ResolveHost) — "derive from site which to use".
-//
-// The API SECRET never travels this path — only the non-secret ids, which ship in the
-// page anyway. Fails SOFT: unmounted, or no project, ⇒ (nil, false), so the tag door
-// answers an empty set and a page never breaks on its config.
-
-// TagsFor returns a site's browser tag config for (key, host). host must already be a
-// bare hostname (the caller normalizes the Origin/Referer). Key wins over host.
-func TagsFor(ctx context.Context, key, host string) (map[string]string, bool) {
-	s := mounted
-	if s == nil || s.State.store == nil {
-		return nil, false
-	}
-	if k := strings.TrimSpace(key); k != "" {
-		if p, err := s.State.store.ResolveKey(ctx, k); err == nil {
-			return p.Tags, true
-		}
-	}
-	if h := strings.TrimSpace(host); h != "" {
-		if p, err := s.State.store.ResolveHost(ctx, h); err == nil {
-			return p.Tags, true
-		}
-	}
-	return nil, false
-}

apps/sandbox/api.go

@@ -46,12 +46,18 @@ type Spec struct {
 	Class   string
 	Project string
 	Image   string
-	// RuntimeClass is the isolation boundary, per sandbox. It was deployment-wide
-	// (one env var read at startup), which made it impossible to run the same
-	// task on two runtimes and compare — and impossible for a caller to choose.
-	// Empty means the deployment's default.
-	RuntimeClass string
-	TTLSec       int
+	// Runtime is the isolation boundary the caller ASKS FOR, per sandbox. It was
+	// deployment-wide (one env var read at startup), which made it impossible to
+	// run the same task on two runtimes and compare — and impossible for a caller
+	// to choose. Empty means the deployment's default.
+	//
+	// Asking is not getting. The server decides, in runtimeFor, and it answers a
+	// request it cannot honour with a refusal rather than with a different
+	// runtime — so what the sandbox got comes back on Sandbox.Runtime and the two
+	// can be compared. `RuntimeClass` is the Kubernetes spelling and it stays in
+	// runtime.go, where Kubernetes is spoken.
+	Runtime string
+	TTLSec  int
 }
 
 // Cmd is one command to run inside a sandbox. Argv is the honest form; Command is
@@ -62,6 +68,11 @@ type Cmd struct {
 	Stdin      string
 	Dir        string
 	TimeoutSec int
+	// Session is where this command NARRATES. Its output is appended to that
+	// session's live log as the program produces it, so a surface watching the run
+	// sees the work happen instead of a blank pause with a verdict at the end.
+	// Empty means nothing is watching, and then nothing is sent — see work.go.
+	Session string
 }
 
 // Entry is what a path IS: a file's bytes, or a directory's entries. One read
@@ -87,7 +98,16 @@ const dirExit = 10
 // Resuming is checked against the STORE and against the row's status, so a session
 // whose pod the reaper already ended comes back as a fresh sandbox rather than as a
 // 502 from the first command sent into a pod that is gone.
-func Lease(s *Service, ctx context.Context, org string, spec Spec) (Sandbox, error) {
+//
+// `super` is the caller's platform sudo, and it is a PARAMETER rather than a field
+// on Spec or a read off the context. Both of the alternatives were worse in the
+// same way. On Spec it would sit beside Class and Project — things the caller asks
+// for — one refactor away from being bound off a request body, which is the whole
+// hazard trust_test.go exists to catch. Read from the context it would make this
+// core read identity, and the reason every function in this file takes `org` as an
+// argument is that none of them may. So it arrives the way org does: named by the
+// adapter that knows it, from the one predicate that answers it.
+func Lease(s *Service, ctx context.Context, org string, super bool, spec Spec) (Sandbox, error) {
 	if strings.TrimSpace(org) == "" {
 		return Sandbox{}, zip.ErrForbidden("org required")
 	}
@@ -122,9 +142,6 @@ func Lease(s *Service, ctx context.Context, org string, spec Spec) (Sandbox, err
 	if err := checkImage(org, spec.Image); err != nil {
 		return Sandbox{}, zip.ErrBadRequest(err.Error())
 	}
-	if err := checkRuntime(spec.RuntimeClass); err != nil {
-		return Sandbox{}, zip.ErrBadRequest(err.Error())
-	}
 	project := slug(spec.Project)
 	if class != "exec" && project == "" {
 		return Sandbox{}, zip.ErrBadRequest("project required for class " + class)
@@ -176,13 +193,25 @@ func Lease(s *Service, ctx context.Context, org string, spec Spec) (Sandbox, err
 	now := time.Now().Unix()
 	m := Sandbox{
 		ID: id, Org: org, Kind: KindSandbox, Class: class, Project: project,
-		Image: firstNonEmpty(spec.Image, s.State.rt.imageFor(class)),
+		Image: firstNonEmpty(spec.Image, s.State.rt.imageFor(class, super)),
 		Pod:   podName(id), Status: "pending",
 		CreatedAt: now, LastUsedAt: now,
 	}
 	if project != "" {
 		m.Volume = volumeName(org, project)
 	}
+	// THE ISOLATION BOUNDARY, derived from the volume the line above just decided.
+	// Asked here rather than at start() so that a request which cannot be honoured
+	// leaves nothing behind — no row, no PVC, no pod.
+	//
+	// The ANSWER is recorded, not the question. A caller that asked for one
+	// runtime and can only have another must be able to see which it got, or the
+	// two are indistinguishable from the outside and a comparison between them
+	// measures nothing.
+	m.Runtime, err = s.State.rt.runtimeFor(m, spec.Runtime)
+	if err != nil {
+		return Sandbox{}, zip.ErrBadRequest(err.Error())
+	}
 	// The lease. Unbounded is not an option for a sandbox running submitted code on
 	// our nodes, so an unset ttl takes the class default rather than forever.
 	ttl := spec.TTLSec
@@ -200,7 +229,7 @@ func Lease(s *Service, ctx context.Context, org string, spec Spec) (Sandbox, err
 	// A failure to start is RECORDED on the row and answered 503 — the row stays so
 	// an operator can see what was asked for and why it did not happen, rather than
 	// the request vanishing with the evidence.
-	if err := s.State.rt.start(ctx, m, spec.RuntimeClass); err != nil {
+	if err := s.State.rt.start(ctx, m); err != nil {
 		m.Status, m.Error = "error", err.Error()
 		_ = store.Put(ctx, m)
 		return Sandbox{}, zip.Errorf(http.StatusServiceUnavailable, "start sandbox: %v", err)
@@ -254,7 +283,21 @@ func Run(s *Service, ctx context.Context, org, id string, cmd Cmd) (ExecResult,
 		argv = append([]string{"sh", "-c", "cd " + shellQuote(cmd.Dir) + " && exec \"$@\"", "sh"}, argv...)
 	}
 	touched(ctx, store, m) // a call is in flight — see touched
-	r, err := s.State.rt.exec(ctx, m, argv, strings.NewReader(cmd.Stdin), cmd.TimeoutSec)
+
+	// THE COMMAND BECOMES ADDRESSABLE the moment it starts, in the two ways a
+	// caller needs it to be: its cancel is held under the sandbox so Stop can
+	// reach it, and its output is narrated into the session that asked to watch.
+	// Both end when the command does. See work.go.
+	ctx, stop := context.WithCancel(ctx)
+	defer stop()
+	forget := s.State.work.start(m.ID, stop)
+	defer forget()
+	say := newTell(org, cmd.Session)
+
+	r, err := s.State.rt.exec(ctx, m, argv, strings.NewReader(cmd.Stdin), cmd.TimeoutSec, say)
+	// The last word is said on EVERY path, including the one a stop took. A run
+	// that vanishes mid-sentence leaves a watcher reading "working…" forever.
+	say.done(r.ExitCode, err)
 	if err != nil {
 		return ExecResult{}, zip.Errorf(http.StatusBadGateway, "exec: %v", err)
 	}
@@ -262,6 +305,29 @@ func Run(s *Service, ctx context.Context, org, id string, cmd Cmd) (ExecResult,
 	return r, nil
 }
 
+// Stop interrupts whatever the caller's sandbox is running right now, and leaves
+// the sandbox leased.
+//
+// It is a different act from End, deliberately: STOP ENDS THE WORK, END ENDS THE
+// RESOURCE. A run that has gone wrong is usually one somebody still wants to look
+// at — the checkout, the logs, the half-written file are all in there — and a
+// stop that also deleted the pod would take the evidence with it. Two verbs, no
+// overlap, and nothing that has to be undone.
+//
+// It answers HOW MANY commands it interrupted, and zero is a true answer rather
+// than a failure: a command that finished a moment ago is one there is nothing
+// left to stop. What a caller must be able to tell apart is "already over" from
+// "not yours", and the second is a 404 out of find below — the same org gate
+// every other operation here walks through, applied before the in-flight set is
+// consulted at all.
+func Stop(s *Service, ctx context.Context, org, id string) (int, error) {
+	m, _, err := find(s, ctx, org, id)
+	if err != nil {
+		return 0, err
+	}
+	return s.State.work.stop(m.ID), nil
+}
+
 // Read reads one file, or lists a directory when the path names one. Both go through
 // the same exec channel — there is no second protocol and no daemon in the pod to
 // speak one.
@@ -276,7 +342,7 @@ func Read(s *Service, ctx context.Context, org, id, path string) (Entry, error)
 	}
 	q := shellQuote(p)
 	r, err := s.State.rt.exec(ctx, m, []string{"sh", "-c",
-		"if [ -d " + q + " ]; then ls -1A -- " + q + "; exit " + strconv.Itoa(dirExit) + "; fi; cat -- " + q}, nil, 0)
+		"if [ -d " + q + " ]; then ls -1A -- " + q + "; exit " + strconv.Itoa(dirExit) + "; fi; cat -- " + q}, nil, 0, nil)
 	if err != nil {
 		return Entry{}, zip.Errorf(http.StatusBadGateway, "fs read: %v", err)
 	}
@@ -304,7 +370,7 @@ func Write(s *Service, ctx context.Context, org, id, path string, data []byte) (
 	}
 	q := shellQuote(p)
 	r, err := s.State.rt.exec(ctx, m, []string{"sh", "-c",
-		"mkdir -p -- \"$(dirname -- " + q + ")\" && cat > " + q}, bytes.NewReader(data), 0)
+		"mkdir -p -- \"$(dirname -- " + q + ")\" && cat > " + q}, bytes.NewReader(data), 0, nil)
 	if err != nil {
 		return "", 0, zip.Errorf(http.StatusBadGateway, "fs write: %v", err)
 	}

apps/sandbox/desktop_test.go

@@ -0,0 +1,101 @@
+package sandbox
+
+// What a desktop's container is told to run.
+//
+// A desktop image's CMD starts an X server, a window manager and the VNC/noVNC
+// pair, and only then becomes the same `sleep infinity` the other classes run
+// outright. Stating a command in the pod spec replaces that script — so the one
+// class that exists to have a display came up with none, indistinguishable from
+// `dev` except by a label, and healthy-looking the whole time. A sleeping pod
+// answers every probe; only an X client ever notices.
+//
+// These cases are that difference, stated where it cannot drift back.
+
+import "testing"
+
+func podFor(t *testing.T, class string) map[string]any {
+	t.Helper()
+	r := &runtime{ns: "hanzo-sandboxes", image: "oci.hanzo.ai/hanzoai/sandbox"}
+	u := r.podSpec(Sandbox{ID: "m_1", Class: class, Pod: "m-1", Org: "acme", Image: "img"})
+	spec, ok := u.Object["spec"].(map[string]any)
+	if !ok {
+		t.Fatalf("%s: no spec", class)
+	}
+	cs, ok := spec["containers"].([]any)
+	if !ok || len(cs) != 1 {
+		t.Fatalf("%s: want one container, got %v", class, spec["containers"])
+	}
+	c, ok := cs[0].(map[string]any)
+	if !ok {
+		t.Fatalf("%s: container is not an object", class)
+	}
+	return c
+}
+
+func TestDesktopRunsItsImageAndTheOthersSleep(t *testing.T) {
+	// THE REGRESSION. A command here shadows the image's CMD, and the desktop
+	// entrypoint is the only thing that starts Xvfb. Its absence is not an error
+	// anywhere: the pod runs, exec answers, and the class is silently `dev`.
+	if cmd, stated := podFor(t, "desktop")["command"]; stated {
+		t.Errorf("desktop states a command (%v), which replaces the image CMD that starts its screen", cmd)
+	}
+	// The other two are a place to run commands, not a program. They must keep
+	// sleeping — deferring to the image would make the pod's lifetime depend on
+	// whatever CMD an operator-named image happens to carry.
+	for _, class := range []string{"exec", "dev"} {
+		cmd, stated := podFor(t, class)["command"]
+		if !stated {
+			t.Errorf("%s: no command, so its lifetime is the image's to decide", class)
+			continue
+		}
+		got, ok := cmd.([]any)
+		if !ok || len(got) != 2 || got[0] != "sleep" || got[1] != "infinity" {
+			t.Errorf("%s: command = %v, want [sleep infinity]", class, cmd)
+		}
+	}
+}
+
+func TestOnlyADesktopDeclaresAScreen(t *testing.T) {
+	ports, stated := podFor(t, "desktop")["ports"]
+	if !stated {
+		t.Fatal("desktop declares no ports, so nothing names the screen it serves")
+	}
+	got := map[string]int64{}
+	for _, p := range ports.([]any) {
+		m := p.(map[string]any)
+		got[m["name"].(string)] = m["containerPort"].(int64)
+	}
+	for name, want := range map[string]int64{"vnc": 5900, "novnc": 6080} {
+		if got[name] != want {
+			t.Errorf("desktop port %q = %d, want %d", name, got[name], want)
+		}
+	}
+	// A port on a class with nothing listening is a claim the image does not
+	// keep. exec and dev have no X server and no VNC bridge in them at all.
+	for _, class := range []string{"exec", "dev"} {
+		if p, stated := podFor(t, class)["ports"]; stated {
+			t.Errorf("%s declares ports %v, but nothing in that image listens", class, p)
+		}
+	}
+}
+
+func TestEveryClassStillDropsEverythingAndRunsAsNobody(t *testing.T) {
+	// The desktop's exemption is its COMMAND and nothing else. A class that runs
+	// its own process is exactly the one somebody would be tempted to hand a
+	// capability or a root uid to, so the shared floor is asserted per class
+	// rather than assumed from the one that has no process of its own.
+	for _, class := range []string{"exec", "dev", "desktop"} {
+		sc, ok := podFor(t, class)["securityContext"].(map[string]any)
+		if !ok {
+			t.Fatalf("%s: no container securityContext", class)
+		}
+		if sc["allowPrivilegeEscalation"] != false {
+			t.Errorf("%s: allowPrivilegeEscalation = %v, want false", class, sc["allowPrivilegeEscalation"])
+		}
+		caps, _ := sc["capabilities"].(map[string]any)
+		drop, _ := caps["drop"].([]any)
+		if len(drop) != 1 || drop[0] != "ALL" {
+			t.Errorf("%s: capabilities.drop = %v, want [ALL]", class, caps["drop"])
+		}
+	}
+}

apps/sandbox/image.go

@@ -67,23 +67,8 @@ func isOurs(host string) bool {
 	return false
 }
 
-// runtimes are the isolation boundaries a caller may ask for. It is a CLOSED
-// set, not free text, because runtimeClassName is passed to the apiserver and an
-// unknown value is a pod that never schedules — a caller typo would become a
-// sandbox stuck Pending with no explanation.
-//
-// The empty string is the deployment's own default (SANDBOX_RUNTIME_CLASS), and
-// it is what a caller naming nothing gets.
-var runtimes = map[string]bool{"gvisor": true, "kata-fc": true, "kata-clh": true}
-
-// checkRuntime refuses a runtime we do not run. Naming one that is not installed
-// on any node is the same failure with a slower clock, so this is only half the
-// check — the RuntimeClass has to exist in the cluster too, and the apiserver is
-// the one that knows.
-func checkRuntime(rc string) error {
-	rc = strings.TrimSpace(rc)
-	if rc == "" || runtimes[rc] {
-		return nil
-	}
-	return fmt.Errorf("runtime %q is not one we run (gvisor, kata-fc, kata-clh)", rc)
-}
+// The runtime a caller may ask for is checked in runtime.go, beside the table
+// that says what each runtime can do — see runtimeFor. It used to be a second
+// half-check here (is the name in the set?) with the real decision elsewhere,
+// which is how a caller could name a runtime that exists and still get a
+// sandbox that silently dropped its volume.

apps/sandbox/image_check_test.go

@@ -33,19 +33,7 @@ func TestCheckImageRefusesAnotherOrgsNamespaceOnOurRegistry(t *testing.T) {
 	}
 }
 
-// A runtime is passed to the apiserver as runtimeClassName, so an unknown value
-// is a pod that never schedules. Refusing it here turns a silent Pending into a
-// 400 that says which runtimes exist.
-func TestCheckRuntimeIsAClosedSet(t *testing.T) {
-	for _, c := range []struct {
-		rc      string
-		wantErr bool
-	}{
-		{"", false}, {"gvisor", false}, {"kata-fc", false}, {"kata-clh", false},
-		{"runsc", true}, {"gVisor", true}, {"anything", true},
-	} {
-		if err := checkRuntime(c.rc); (err != nil) != c.wantErr {
-			t.Fatalf("checkRuntime(%q) err=%v, wantErr=%v", c.rc, err, c.wantErr)
-		}
-	}
-}
+// The closed set of runtimes moved to runtime_test.go, beside the derivation
+// that reads it. Being in the set is only half the question a caller's runtime
+// has to answer; the other half is whether it can hold that sandbox's volume,
+// and splitting the two is what let a valid name lose a checkout.

apps/sandbox/image_test.go

@@ -8,11 +8,15 @@ package sandbox
 // SANDBOX_IMAGE_DIGEST would have handed every class the same image, which
 // looks correct in every log line it produces.
 
-import "testing"
+import (
+	"strings"
+	"testing"
+)
 
 func TestImageForResolvesTheTagThePublisherWrote(t *testing.T) {
 	for _, c := range []struct {
 		name, repo, tag, class, want string
+		super                        bool
 		digest                       map[string]string
 	}{{
 		name: "version first, class second — the order CI publishes",
@@ -43,19 +47,71 @@ func TestImageForResolvesTheTagThePublisherWrote(t *testing.T) {
 		repo: "oci.hanzo.ai/hanzoai/sandbox", tag: "2026.6.7", class: "exec",
 		digest: map[string]string{"DEV": "sha256:2baf7ede"},
 		want:   "oci.hanzo.ai/hanzoai/sandbox:2026.6.7-exec",
+	}, {
+		// WHO ASKS decides which bytes, for one class and one identity.
+		name: "a SuperAdmin's dev sandbox runs the admin image",
+		repo: "oci.hanzo.ai/hanzoai/sandbox", tag: "1.1.0", class: "dev", super: true,
+		want: "oci.hanzo.ai/hanzoai/sandbox:1.1.0-admin",
+	}, {
+		// The substitution follows the BUILD: `admin` is layered on dev, so it
+		// stands in for dev and for nothing else. Swapping exec would put a
+		// bigger image behind every fifteen-minute tool call this identity makes.
+		name: "a SuperAdmin's exec sandbox is the ordinary exec image",
+		repo: "oci.hanzo.ai/hanzoai/sandbox", tag: "1.1.0", class: "exec", super: true,
+		want: "oci.hanzo.ai/hanzoai/sandbox:1.1.0-exec",
+	}, {
+		// `admin` has no X server; desktop's whole reason for existing is one.
+		name: "a SuperAdmin's desktop sandbox keeps its screen",
+		repo: "oci.hanzo.ai/hanzoai/sandbox", tag: "1.1.0", class: "desktop", super: true,
+		want: "oci.hanzo.ai/hanzoai/sandbox:1.1.0-desktop",
+	}, {
+		// The admin image is a fourth PUBLISHED image, so it pins like the other
+		// three — by its own digest, under its own name. Reading _DEV here would
+		// be the neighbour's-digest bug wearing a new class.
+		name: "the admin image takes the admin digest, not dev's",
+		repo: "oci.hanzo.ai/hanzoai/sandbox", tag: "1.1.0", class: "dev", super: true,
+		digest: map[string]string{"DEV": "sha256:2baf7ede", "ADMIN": "sha256:9c1d0f42"},
+		want:   "oci.hanzo.ai/hanzoai/sandbox@sha256:9c1d0f42",
 	}} {
 		t.Run(c.name, func(t *testing.T) {
 			for k, v := range c.digest {
 				t.Setenv("SANDBOX_IMAGE_DIGEST_"+k, v)
 			}
 			r := &runtime{image: c.repo, tag: c.tag}
-			if got := r.imageFor(c.class); got != c.want {
-				t.Fatalf("imageFor(%q) = %q, want %q", c.class, got, c.want)
+			if got := r.imageFor(c.class, c.super); got != c.want {
+				t.Fatalf("imageFor(%q, super=%v) = %q, want %q", c.class, c.super, got, c.want)
 			}
 		})
 	}
 }
 
+// NOBODY BUT A SUPERADMIN IS EVER HANDED THE ADMIN IMAGE, whatever else is set.
+//
+// The rule is one line in imageFor, which is exactly why it is worth an
+// invariant rather than a case: a line that reads `super && class == "dev"` is
+// one edit away from `super || class == "dev"`, and the resulting defect is
+// invisible in every log — a tenant's sandbox that works, with three binaries in
+// it that nobody there asked for and no error anywhere.
+//
+// The env matrix is the other half. `SANDBOX_IMAGE_TAG_ADMIN` and
+// `SANDBOX_IMAGE_DIGEST_ADMIN` are read by name, so a deployment that pins the
+// admin image must not thereby serve it to anyone: pinning WHICH bytes and
+// deciding WHO gets them are two questions, and only one of them is a setting.
+func TestOnlyASuperAdminIsHandedTheAdminImage(t *testing.T) {
+	const repo = "oci.hanzo.ai/hanzoai/sandbox"
+	t.Setenv("SANDBOX_IMAGE_TAG_ADMIN", "1.1.0")
+	t.Setenv("SANDBOX_IMAGE_DIGEST_ADMIN", "sha256:9c1d0f42")
+	for _, tag := range []string{"", "1.1.0"} {
+		for _, class := range []string{"exec", "dev", "desktop"} {
+			r := &runtime{image: repo, tag: tag}
+			if got := r.imageFor(class, false); strings.Contains(got, "admin") {
+				t.Fatalf("imageFor(%q, super=false) with tag %q = %q — an ordinary "+
+					"caller was handed the operator's image", class, tag, got)
+			}
+		}
+	}
+}
+
 // The bare tag can never come back, for ANY class, pinned or not.
 //
 // The table above proves the `dev` case with no tag. This is the same rule
@@ -79,11 +135,14 @@ func TestImageForNeverComposesTheBareClassTag(t *testing.T) {
 	const repo = "oci.hanzo.ai/hanzoai/sandbox"
 	for _, tag := range []string{"", "2026.6.7", "1.0.0"} {
 		for _, class := range []string{"exec", "dev", "desktop"} {
-			r := &runtime{image: repo, tag: tag}
-			if got := r.imageFor(class); got == repo+":"+class {
-				t.Fatalf("imageFor(%q) with tag %q = %q — nothing in the fleet "+
-					"publishes that tag, so whatever answers it was written by hand",
-					class, tag, got)
+			for _, super := range []bool{false, true} {
+				r := &runtime{image: repo, tag: tag}
+				got := r.imageFor(class, super)
+				if got == repo+":"+class || got == repo+":admin" {
+					t.Fatalf("imageFor(%q, super=%v) with tag %q = %q — nothing in the "+
+						"fleet publishes that tag, so whatever answers it was written by hand",
+						class, super, tag, got)
+				}
 			}
 		}
 	}

apps/sandbox/live_test.go

@@ -54,7 +54,11 @@ func TestLiveSandboxRunsRealCode(t *testing.T) {
 	defer cancel()
 
 	t.Logf("starting %s image=%s ns=%s runtimeClass=%q", m.Pod, m.Image, r.ns, r.runtimeClass)
-	if err := r.start(ctx, m, ""); err != nil {
+	// The isolation boundary comes from the SAME derivation production uses, so the
+	// live proof runs under whatever the fleet is set to rather than under the
+	// node default. want is empty, and runtimeFor cannot refuse an empty ask.
+	m.Runtime, _ = r.runtimeFor(m, "")
+	if err := r.start(ctx, m); err != nil {
 		t.Fatalf("start: %v", err)
 	}
 	// Always clean up: a leaked pod on a shared cluster is somebody else's
@@ -83,7 +87,7 @@ func TestLiveSandboxRunsRealCode(t *testing.T) {
 	wd := workdirFor(m.Class)
 
 	// 1. It runs code at all.
-	res, err := r.exec(ctx, m, []string{"node", "-e", "console.log('SANDBOX-RUNS-CODE', process.version)"}, nil, 60)
+	res, err := r.exec(ctx, m, []string{"node", "-e", "console.log('SANDBOX-RUNS-CODE', process.version)"}, nil, 60, nil)
 	if err != nil {
 		t.Fatalf("exec: %v", err)
 	}
@@ -97,10 +101,10 @@ func TestLiveSandboxRunsRealCode(t *testing.T) {
 	//    inside one call is not a filesystem.
 	const src = "export const answer = 42; // edited by the agent\n"
 	if res, err = r.exec(ctx, m, []string{"sh", "-c", "mkdir -p " + wd + "/src && cat > " + wd + "/src/app.js"},
-		strings.NewReader(src), 60); err != nil || res.ExitCode != 0 {
+		strings.NewReader(src), 60, nil); err != nil || res.ExitCode != 0 {
 		t.Fatalf("write: err=%v exit=%d stderr=%q", err, res.ExitCode, res.Stderr)
 	}
-	if res, err = r.exec(ctx, m, []string{"cat", wd + "/src/app.js"}, nil, 60); err != nil {
+	if res, err = r.exec(ctx, m, []string{"cat", wd + "/src/app.js"}, nil, 60, nil); err != nil {
 		t.Fatalf("read back: %v", err)
 	}
 	if res.Stdout != src {
@@ -112,7 +116,7 @@ func TestLiveSandboxRunsRealCode(t *testing.T) {
 	//    executing it proves the edit reached the same filesystem the runtime
 	//    uses, which is the thing an agent depends on.
 	if res, err = r.exec(ctx, m, []string{"node", "-e",
-		"import('" + wd + "/src/app.js').then(m=>console.log('ANSWER='+m.answer))"}, nil, 60); err != nil {
+		"import('" + wd + "/src/app.js').then(m=>console.log('ANSWER='+m.answer))"}, nil, 60, nil); err != nil {
 		t.Fatalf("run edited code: %v", err)
 	}
 	if !strings.Contains(res.Stdout, "ANSWER=42") {
@@ -123,7 +127,7 @@ func TestLiveSandboxRunsRealCode(t *testing.T) {
 	// 4. A failing command is DATA, not an error. An agent has to be able to see
 	//    a test suite fail without the call itself failing, or it cannot tell
 	//    "your code is broken" from "the sandbox is broken".
-	if res, err = r.exec(ctx, m, []string{"sh", "-c", "echo to-stderr >&2; exit 3"}, nil, 60); err != nil {
+	if res, err = r.exec(ctx, m, []string{"sh", "-c", "echo to-stderr >&2; exit 3"}, nil, 60, nil); err != nil {
 		t.Fatalf("a non-zero exit must not be a transport error: %v", err)
 	}
 	if res.ExitCode != 3 || !strings.Contains(res.Stderr, "to-stderr") {
@@ -149,7 +153,11 @@ func TestLiveSandboxDoesGit(t *testing.T) {
 	}
 	ctx, cancel := context.WithTimeout(context.Background(), 4*time.Minute)
 	defer cancel()
-	if err := r.start(ctx, m, ""); err != nil {
+	// The isolation boundary comes from the SAME derivation production uses, so the
+	// live proof runs under whatever the fleet is set to rather than under the
+	// node default. want is empty, and runtimeFor cannot refuse an empty ask.
+	m.Runtime, _ = r.runtimeFor(m, "")
+	if err := r.start(ctx, m); err != nil {
 		t.Fatalf("start: %v", err)
 	}
 	defer func() {
@@ -158,7 +166,7 @@ func TestLiveSandboxDoesGit(t *testing.T) {
 	}()
 
 	// git present at all — an image without it cannot host a coding agent.
-	res, err := r.exec(ctx, m, []string{"git", "--version"}, nil, 60)
+	res, err := r.exec(ctx, m, []string{"git", "--version"}, nil, 60, nil)
 	if err != nil || res.ExitCode != 0 {
 		t.Fatalf("git missing from the image: err=%v exit=%d stderr=%q", err, res.ExitCode, res.Stderr)
 	}
@@ -180,7 +188,7 @@ printf 'edited\n' > file.txt
 git add file.txt
 git commit -q -m "the agent committed this"
 git log --oneline -1`
-	if res, err = r.exec(ctx, m, []string{"sh", "-c", script}, nil, 120); err != nil {
+	if res, err = r.exec(ctx, m, []string{"sh", "-c", script}, nil, 120, nil); err != nil {
 		t.Fatalf("git flow: %v", err)
 	}
 	if res.ExitCode != 0 {
@@ -192,6 +200,6 @@ git log --oneline -1`
 	// separates "the network allows it" from "we have a credential". Those are
 	// different gaps and conflating them sends someone to fix the wrong one.
 	res, _ = r.exec(ctx, m, []string{"sh", "-c",
-		"git ls-remote https://git.hanzo.ai/hanzo/universe HEAD 2>&1 | head -2"}, nil, 60)
+		"git ls-remote https://git.hanzo.ai/hanzo/universe HEAD 2>&1 | head -2"}, nil, 60, nil)
 	t.Logf("FORGE REACHABLE: exit=%d out=%q", res.ExitCode, strings.TrimSpace(res.Stdout+res.Stderr))
 }

apps/sandbox/plane.go

@@ -52,6 +52,9 @@ func expose() {
 	zip.Post[plane.WriteIn, plane.Wrote](p, "/sandbox/write", planeWrite,
 		zip.WithOperationID(plane.SandboxWrite),
 		zip.WithSummary("Write a file in a sandbox"))
+	zip.Post[plane.StopIn, plane.Stopped](p, "/sandbox/stop", planeStop,
+		zip.WithOperationID(plane.SandboxStop),
+		zip.WithSummary("Interrupt what a sandbox is running"))
 	zip.Post[plane.EndIn, struct{}](p, "/sandbox/end", planeEnd,
 		zip.WithOperationID(plane.SandboxEnd),
 		zip.WithSummary("End a sandbox's lease"))
@@ -78,29 +81,61 @@ func planeLease(ctx context.Context, in *plane.LeaseIn) (*plane.Leased, error) {
 	if err != nil {
 		return nil, err
 	}
-	m, err := Lease(s, ctx, org, Spec{ID: in.ID, Class: in.Class, Project: in.Project, TTLSec: in.TTLSec})
+	// The same fact the HTTP door reads off the principal, read here off the caller
+	// the plane already carries — cloud.Who is how this door spells identity, and
+	// zip.Caller.Admin is the same attestation X-User-IsAdmin carries. One rule,
+	// stated once in imageFor; each door names the caller in its own vocabulary.
+	m, err := Lease(s, ctx, org, cloud.Who(ctx).Admin, Spec{ID: in.ID, Class: in.Class,
+		Project: in.Project, Runtime: in.Runtime, TTLSec: in.TTLSec})
 	if err != nil {
 		return nil, err
 	}
-	return &plane.Leased{ID: m.ID, Class: m.Class, Status: m.Status, Workdir: workdirFor(m.Class)}, nil
+	return &plane.Leased{ID: m.ID, Class: m.Class, Runtime: m.Runtime, Status: m.Status,
+		Workdir: workdirFor(m.Class)}, nil
 }
 
 // planeRun runs one command inside the caller's sandbox and answers its exit code,
 // stdout and stderr. A non-zero exit is a successful call carrying a failed
 // program, so it comes back as data and not as an error.
+//
+// Name a `session` and the command NARRATES INTO IT: its output is appended to
+// that session's live log as the program produces it, so anything watching the
+// session — GET /v1/agents/sessions/stream, scoped to one run with ?root= —
+// watches the work happen rather than waiting for the verdict. Without it the
+// call is what it always was: silent until it returns, which for an agentic run
+// is twenty-five minutes of blank screen.
+//
+// The session is named; the TENANT is not. It is the org the caller already
+// proved, so a session belonging to somebody else is absent from the org this
+// call acts for and the append is refused there.
 func planeRun(ctx context.Context, in *plane.RunIn) (*plane.Ran, error) {
 	s, org, err := live(ctx)
 	if err != nil {
 		return nil, err
 	}
 	r, err := Run(s, ctx, org, in.ID, Cmd{Argv: in.Argv, Command: in.Command,
-		Stdin: in.Stdin, Dir: in.Dir, TimeoutSec: in.TimeoutSec})
+		Stdin: in.Stdin, Dir: in.Dir, TimeoutSec: in.TimeoutSec, Session: in.Session})
 	if err != nil {
 		return nil, err
 	}
 	return &plane.Ran{ExitCode: r.ExitCode, Stdout: r.Stdout, Stderr: r.Stderr}, nil
 }
 
+// planeStop interrupts whatever the caller's sandbox is running and answers how
+// many commands it ended. The sandbox stays leased — stop ends the WORK, end ends
+// the RESOURCE — so whoever stopped a run can still read what it left behind.
+func planeStop(ctx context.Context, in *plane.StopIn) (*plane.Stopped, error) {
+	s, org, err := live(ctx)
+	if err != nil {
+		return nil, err
+	}
+	n, err := Stop(s, ctx, org, in.ID)
+	if err != nil {
+		return nil, err
+	}
+	return &plane.Stopped{Stopped: n}, nil
+}
+
 // planeRead reads one path in the caller's sandbox: a file's bytes, or a
 // directory's entries when the path names one.
 func planeRead(ctx context.Context, in *plane.PathIn) (*plane.Blob, error) {

apps/sandbox/runtime.go

@@ -2,12 +2,18 @@
 // one channel into it.
 //
 // A sandbox is a Pod, and its isolation boundary is the RUNTIME that pod names
-// — one field, `SANDBOX_RUNTIME_CLASS`, holding `gvisor` or `kata-fc` or
-// `kata-clh` or nothing. It is the ONLY boundary claimed here: no uid juggling,
-// no process groups, no daemon inside the pod deciding what it is allowed to
-// run. Everything the predecessor built to approximate that boundary in Go is
-// deleted rather than kept "for defence in depth", because a second
-// half-boundary is a second thing to keep true.
+// — one field, whose value comes from `runtimes` and nowhere else. It is the
+// ONLY boundary claimed here: no uid juggling, no process groups, no daemon
+// inside the pod deciding what it is allowed to run. Everything the predecessor
+// built to approximate that boundary in Go is deleted rather than kept "for
+// defence in depth", because a second half-boundary is a second thing to keep
+// true.
+//
+// WHICH boundary is a derivation and not a setting — see runtimeFor, which is
+// the one place that decides it, over two facts a sandbox already states about
+// itself: whose code it runs, and whether it keeps anything.
+// `SANDBOX_RUNTIME_CLASS` states the deployment's preference among them and is
+// checked against the table at startup.
 //
 // Empty is honest, not a hole: it means the node's default runtime, which is
 // the containment a normal pod gets. It ships that way because runsc has to be
@@ -24,15 +30,18 @@ import (
 	"fmt"
 	"io"
 	"os"
+	"sort"
 	"strings"
 	"time"
 
+	"github.com/hanzoai/authz"
 	"github.com/hanzoai/cloud/apps/k8s"
 
 	corev1 "k8s.io/api/core/v1"
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/client-go/dynamic"
 	"k8s.io/client-go/kubernetes/scheme"
 	"k8s.io/client-go/rest"
@@ -68,8 +77,23 @@ const (
 	labSandbox = "hanzo.ai/sandbox"
 	labOrg     = "hanzo.ai/sandbox-org"
 	labClass   = "hanzo.ai/sandbox-class"
+	labProject = "hanzo.ai/sandbox-project"
 )
 
+// annLeased is the DATE a disk was last leased, UTC, on the disk itself.
+//
+// A disk outlives every object that could account for it — the pod by an hour,
+// the row by the lease — so "is this still wanted?" has no other place to be
+// asked. Without it the answer has to come from a hash: the name ends in
+// sha256(org, project), which identifies a disk only to whoever already knows
+// the project that made it, and that is nobody a month later.
+//
+// A DATE and not a timestamp, so the write is idempotent within a day: a disk
+// leased forty times before lunch is patched once. The question it answers is
+// counted in days, so the precision that would cost a write per lease would buy
+// nothing.
+const annLeased = "hanzo.ai/sandbox-leased"
+
 // defaultTTL is the lease a class gets when the caller names none. Unbounded is
 // not an option for a pod running submitted code on our nodes.
 var defaultTTL = map[string]int{"exec": 900, "dev": 14400, "desktop": 14400}
@@ -111,6 +135,12 @@ type runtime struct {
 	image        string // oci.hanzo.ai/hanzoai/sandbox, without a tag
 	tag          string
 	runtimeClass string
+	// bare is the boundary our OWN code takes — the one with no kernel of its
+	// own — and it is empty until the cluster keeps that boundary to nodes of its
+	// own. Resolved once, against the cluster, because containment is the
+	// cluster's fact and a setting that claimed it would be the one thing here
+	// worth lying about. See confine.
+	bare         string
 	startTimeout time.Duration
 	execTimeout  time.Duration
 	dyn          dynamic.Interface
@@ -143,6 +173,25 @@ func newRuntime() *runtime {
 	}
 	r.bound = b
 
+	// SANDBOX_RUNTIME_CLASS IS CHECKED AGAINST THE TABLE, AT STARTUP.
+	//
+	// It was read straight through to the pod spec, so a value the table had
+	// never heard of arrived at the apiserver as a runtimeClassName and the
+	// sandbox sat Pending with nothing said — the same silence the caller-facing
+	// refusal exists to end, one level up and with nobody watching for it. Worse,
+	// it only did that for a volumeless sandbox: the old derivation short-
+	// circuited on `m.Volume == ""` and never consulted the table at all, so the
+	// one path that skipped validation was also the common one.
+	//
+	// A deployment's runtime is decided once, so it is checked once, here, where
+	// the reason has somewhere to go. Everything downstream may then take a
+	// non-empty r.runtimeClass as a name we run.
+	if _, ok := runtimes[r.runtimeClass]; r.runtimeClass != "" && !ok {
+		r.initErr = fmt.Sprintf("SANDBOX_RUNTIME_CLASS=%q is not one we run (%s)",
+			r.runtimeClass, runtimeNames())
+		return r
+	}
+
 	cfg, cerr := rest.InClusterConfig()
 	if cerr != nil {
 		// KUBECONFIG fallback for local development — identical to every other
@@ -161,6 +210,12 @@ func newRuntime() *runtime {
 		return r
 	}
 	r.dyn, r.str = dyn, &spdy{cfg: cfg}
+	// Asked once, at startup, and only ever able to REMOVE a boundary from what
+	// this deployment offers — so a cluster that cannot answer costs a little
+	// speed and never an outage.
+	if n := bare(); r.confine(context.Background(), n) {
+		r.bare = n
+	}
 	return r
 }
 
@@ -174,10 +229,41 @@ func (r *runtime) ready() error {
 	return nil
 }
 
-// imageFor is the tag chain: one image, three tags. The deployment pins the
+// imageFor is the tag chain: one image, four tags. The deployment pins the
 // version; nothing here resolves `latest`, because an image decided by WHEN the
 // pod started rather than by what was shipped is not a deployment.
-func (r *runtime) imageFor(class string) string {
+//
+// `super` IS THE ONE PLACE AN IDENTITY REACHES THE IMAGE, and it reaches it for
+// exactly one class. A SuperAdmin's `dev` sandbox runs the `admin` image — dev
+// plus zsh, kubectl and doctl (hanzoai/bot Dockerfile.box) — and every other
+// caller's `dev` sandbox runs `dev`, byte for byte what it ran before.
+//
+// IT IS A SUBSTITUTION AND NOT A FOURTH CLASS. A class is a word in the request:
+// `classes` is the closed set a caller may ask for, and it stays three. Which
+// bytes a caller is handed is a fact about the CALLER, so it is answered where
+// the caller is known and never offered as a field. The row still says class
+// `dev`, the pod still carries the `dev` label, and only Image differs — which
+// is the honest record of what happened.
+//
+// ONLY `dev`, because `admin` is BUILT from dev and a substitute has to be a
+// superset of what it replaces or the swap quietly changes what the class means.
+// `exec` stays the throwaway a tool call spends fifteen minutes in — swapping it
+// would put a bigger image behind every function invocation this identity makes
+// — and `desktop` keeps the screen `admin` has no X server for.
+//
+// THE ADMIN IMAGE CARRIES NO CREDENTIAL, which is what keeps this one line
+// rather than a gate. kubectl with no kubeconfig and doctl with no token are
+// argument parsers; they reach nothing until a POD is handed something, and what
+// a pod is handed is decided by the identity that leased it, not by the bytes it
+// booted. So there is nothing here to defend against a caller who names the
+// admin image by hand — which checkImage already permits for every platform
+// image, for precisely this reason. That invariant is load-bearing: the day a
+// kubeconfig is wired in, it arrives at the pod from the identity, and it must
+// never arrive in a layer.
+func (r *runtime) imageFor(class string, super bool) string {
+	if super && class == "dev" {
+		class = "admin"
+	}
 	if d := r.digestFor(class); d != "" {
 		return r.image + "@" + d
 	}
@@ -220,14 +306,260 @@ func (r *runtime) imageFor(class string) string {
 //
 // SANDBOX_IMAGE_DIGEST is therefore honoured ahead of any tag: `repo@sha256:…`
 // names bytes, and bytes do not change under a running fleet.
-// It is PER CLASS, because the three classes are three different images and one
-// digest names one of them. A single SANDBOX_IMAGE_DIGEST would have quietly
-// given every class the exec image — the same shape of bug as a tag that looks
-// pinned and is not, which is what this function exists to end.
+// It is PER IMAGE — SANDBOX_IMAGE_DIGEST_EXEC, _DEV, _DESKTOP, _ADMIN — because
+// those are four different images and one digest names one of them. A single
+// SANDBOX_IMAGE_DIGEST would have quietly given every class the exec image: the
+// same shape of bug as a tag that looks pinned and is not, which is what this
+// function exists to end.
 func (r *runtime) digestFor(class string) string {
 	return strings.TrimSpace(os.Getenv("SANDBOX_IMAGE_DIGEST_" + strings.ToUpper(class)))
 }
 
+// runtimes are the isolation boundaries we run, and for each the TWO facts that
+// decide which sandboxes may take it.
+//
+//	kernel — the sandbox gets a kernel of its own, so a process that breaks out
+//	         of its container has not broken out onto the node.
+//	shares — the sandbox can back a PERSISTENT VOLUME.
+//
+// Firecracker has no virtio-fs. Read it on the node rather than take it on
+// faith: `configuration-fc.toml` carries no `shared_fs` key at all, where
+// `configuration-clh.toml:130` sets `shared_fs = "virtio-fs"`. A kata-fc guest
+// therefore cannot mount a directory from the host — and Kubernetes does not
+// fail that mount. The sandbox gets a ~599M tmpfs standing exactly where the
+// volume should be, every write SUCCEEDS into it, and the VM takes the bytes
+// with it when it exits. A `dev` sandbox would look perfect and lose the
+// checkout.
+//
+// runc has no kernel of its own because it IS the node's kernel, which is both
+// why it is the fastest thing here and why only code of ours may take it. See
+// runtimeFor for who that is, and confine for the topology that has to hold
+// before it is offered at all.
+//
+// That is why this is a table and not a comment: neither failure has a symptom
+// at the point it happens — a lost volume looks like a successful write, and a
+// shared kernel looks like a fast sandbox — so both are refused here, where the
+// facts are written down once.
+var runtimes = map[string]struct{ kernel, shares bool }{
+	"runc":     {shares: true},
+	"gvisor":   {kernel: true, shares: true},
+	"kata-clh": {kernel: true, shares: true},
+	"kata-fc":  {kernel: true},
+}
+
+// shared is the boundary that meets BOTH facts at once, and so the answer
+// whenever the one a deployment stated does not. gVisor isolates and shares a
+// filesystem, and it is already what the fleet runs, so this names the existing
+// behaviour rather than adding one.
+const shared = "gvisor"
+
+// fits reports whether this deployment may put a sandbox on a boundary — the
+// ONE predicate, asked by all three paths into runtimeFor.
+//
+// It is one and not three because it was two, briefly, and the third path was
+// the bug: the derivation consulted the topology, the by-name request did not,
+// and the deployment's own setting did not either — so `SANDBOX_RUNTIME_CLASS=runc`
+// put our sandboxes on the node's kernel wherever the scheduler felt like,
+// which is the entire thing confine exists to prevent. A fact that only some
+// callers consult is a fact that has already drifted.
+//
+// An unknown name fits nothing, which is what makes an answer fall to `shared`
+// rather than to a string the apiserver has never heard of.
+func (r *runtime) fits(name string, kernel, shares bool) bool {
+	b, ok := runtimes[name]
+	if !ok || (shares && !b.shares) {
+		return false
+	}
+	// A boundary with a kernel of its own suits anybody. The one without suits
+	// only code of ours, and only where the cluster keeps it to nodes of its own.
+	return b.kernel || (!kernel && name == r.bare)
+}
+
+// bare names the boundary with NO KERNEL OF ITS OWN — a container on the node's
+// kernel, which is the fastest thing we run and the only one reserved to code of
+// ours. Derived from the table rather than written down a second time, so adding
+// or removing a boundary stays a table entry.
+func bare() string {
+	for _, name := range sorted() {
+		if !runtimes[name].kernel {
+			return name
+		}
+	}
+	return ""
+}
+
+// runtimeFor is the ONE place that decides a sandbox's isolation boundary. Not
+// a fork in the code and not a knob per class — ONE LOOKUP over two facts the
+// sandbox already states about itself:
+//
+//	WHO OWNS THE CODE decides how much isolation is needed. Ours runs on our own
+//	kernel; everybody else's gets a kernel of its own.
+//	WHETHER IT KEEPS ANYTHING decides which boundaries can serve it. A project
+//	volume needs a boundary that can share a filesystem.
+//
+// NEITHER FACT CAN BE HANDED IN, and that is the whole security of it. The org
+// is the caller's identity — api.go takes it from principal.Org, which refuses
+// an org that arrived without a validated principal, and no body field reaches
+// it — while the volume was set two lines earlier from the project. A caller
+// that could name its own org could name its own kernel.
+//
+// Keying on m.Volume and not on m.Class is the same point made about the other
+// fact. CLASS DOES NOT DECIDE THIS — `project` does (api.go). `dev` and
+// `desktop` are refused without one so they always carry a volume, but `exec`
+// is optional: an exec sandbox NAMING A PROJECT gets a volume too. A table of
+// class→runtime would have read `exec → the fast one` and silently thrown that
+// org's project away. The volume is the thing that matters, so the volume is
+// what is asked.
+//
+// The two callers are answered differently ON PURPOSE:
+//
+//   - The DEPLOYMENT states a preference, so it is derived down. A fleet set to
+//     the fast runtime still has to run dev sandboxes, and refusing them would
+//     make the setting unusable.
+//   - A CALLER states a request, so a contradiction is refused rather than
+//     corrected. Handing back a runtime nobody asked for is the same silence
+//     this table exists to end, one level up.
+func (r *runtime) runtimeFor(m Sandbox, want string) (string, error) {
+	// THE TWO FACTS. authz.AdminOrg is the reserved platform org — the issuer's
+	// own constant, the same predicate admin-guard and the audit trail read, so
+	// there is no second notion of "ours" here to drift from IAM's.
+	kernel, shares := m.Org != authz.AdminOrg, m.Volume != ""
+
+	if want = strings.TrimSpace(want); want != "" {
+		b, known := runtimes[want]
+		switch {
+		case !known:
+			// A runtimeClassName the cluster has never heard of is a pod that
+			// waits Pending with no explanation, so a typo stops here instead.
+			return "", fmt.Errorf("runtime %q is not one we run (%s)", want, runtimeNames())
+		case kernel && !b.kernel:
+			return "", fmt.Errorf(
+				"runtime %q is the node's own kernel, which is a boundary only for code of ours — "+
+					"ask for %s", want, shared)
+		case !b.kernel && want != r.bare:
+			// EVEN FOR US. Asking by name must go through the same topology the
+			// derivation does, or the one caller allowed to name runc is the one
+			// caller who can put it on a pool it does not own — and on a cluster
+			// with no such class at all, on a pod that waits Pending forever.
+			return "", fmt.Errorf(
+				"runtime %q is the node's own kernel and this cluster does not keep it to a pool "+
+					"of its own, so nothing may take it — ask for %s", want, shared)
+		case shares && !b.shares:
+			return "", fmt.Errorf(
+				"runtime %q has no shared filesystem, so it cannot mount project volume %q — "+
+					"the write would succeed into a tmpfs and be lost when the sandbox ends; "+
+					"ask for %s, or drop the project for a sandbox that keeps nothing",
+				want, m.Volume, shared)
+		}
+		return want, nil
+	}
+	// OUR OWN CODE takes the boundary the cluster keeps to itself. There is no
+	// test for who the caller is here, and there does not need to be: the
+	// boundary has no kernel of its own, so it fits nothing that needs one and
+	// the table refuses it to everybody else. r.bare is empty until the topology
+	// holds (confine), and an empty name fits nothing at all.
+	if r.fits(r.bare, kernel, shares) {
+		return r.bare, nil
+	}
+	// Empty stays empty: that is the node's default runtime, which is a
+	// different request from any named class, and a cluster with no gVisor
+	// installed must not be handed one. Every named value has been through the
+	// table at startup, so what reaches a pod spec here is a name we run.
+	if r.runtimeClass == "" || r.fits(r.runtimeClass, kernel, shares) {
+		return r.runtimeClass, nil
+	}
+	return shared, nil
+}
+
+// sorted is the closed set, in one order. A Go map range would give a different
+// one every time, and both an error message and a derived choice that change on
+// their own are ones nobody can grep for or reproduce.
+func sorted() []string {
+	n := make([]string, 0, len(runtimes))
+	for k := range runtimes {
+		n = append(n, k)
+	}
+	sort.Strings(n)
+	return n
+}
+
+func runtimeNames() string { return strings.Join(sorted(), ", ") }
+
+// confine asks the CLUSTER whether it keeps a boundary to NODES OF ITS OWN, and
+// it is what stands between "runc is the fastest thing we run" and "somebody's
+// model output is on the node's kernel next to another tenant".
+//
+// Our own agent is not our own binary. Its commands are written by a model that
+// just read a repository, a web page or a user's prompt, so the realistic threat
+// is that somebody else wrote them — and the node's kernel is one bug away. That
+// is survivable only if the blast radius is drawn by TOPOLOGY rather than by the
+// runtime: nothing else on the pool, and nothing on the pool worth taking.
+//
+// A RuntimeClass already draws it. `scheduling.nodeSelector` and
+// `scheduling.tolerations` are merged into every pod that names the class, so
+// the selector says which pool our sandboxes go to and the toleration says that
+// pool is tainted against everything else. BOTH are required, because either
+// alone is half a pool: a selector with no taint puts our sandboxes on nodes
+// anything may join, and a taint with no selector leaves them free to land
+// anywhere. And the pool must be the boundary's OWN — sharing one with a
+// boundary other tenants take would put a kernel-sharing sandbox on the same
+// node as their gVisor ones, which is the radius we just went to the trouble of
+// drawing.
+//
+// It is READ and never configured. A boundary that merely says it is contained
+// is not, and this is precisely the fact an attacker would like taken on faith.
+// Every no-answer is a NO — no such class, no client, no permission — so the
+// boundary is simply not offered and our code takes the same one as everybody
+// else's. That is the failure this can afford to have.
+func (r *runtime) confine(ctx context.Context, name string) bool {
+	if r.dyn == nil || name == "" {
+		return false
+	}
+	ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
+	defer cancel()
+	all, err := r.dyn.Resource(k8s.RuntimeClasses).List(ctx, metav1.ListOptions{})
+	if err != nil {
+		return false
+	}
+	pool := map[string]any{}
+	for _, rc := range all.Items {
+		if rc.GetName() != name {
+			continue
+		}
+		sel, _, _ := unstructured.NestedMap(rc.Object, "scheduling", "nodeSelector")
+		tol, _, _ := unstructured.NestedSlice(rc.Object, "scheduling", "tolerations")
+		if len(sel) == 0 || len(tol) == 0 {
+			return false
+		}
+		pool = sel
+	}
+	if len(pool) == 0 {
+		return false
+	}
+	for _, rc := range all.Items {
+		sel, _, _ := unstructured.NestedMap(rc.Object, "scheduling", "nodeSelector")
+		if rc.GetName() != name && samePool(sel, pool) {
+			return false
+		}
+	}
+	return true
+}
+
+// samePool reports whether two node selectors choose the same nodes. An empty
+// selector chooses every node, so it is never "the same pool" as one that
+// chooses some — it is a superset, and the caller has already refused it.
+func samePool(a, b map[string]any) bool {
+	if len(a) == 0 || len(a) != len(b) {
+		return false
+	}
+	for k, v := range a {
+		if b[k] != v {
+			return false
+		}
+	}
+	return true
+}
+
 func (r *runtime) pods() dynamic.ResourceInterface {
 	return r.dyn.Resource(k8s.Pods).Namespace(r.ns)
 }
@@ -235,7 +567,7 @@ func (r *runtime) pods() dynamic.ResourceInterface {
 // start creates the sandbox's volume (if it has one) and its pod, and waits for
 // the pod to be running. A create that returns before the sandbox can answer is
 // a create that hands the caller a 502 on its very next call.
-func (r *runtime) start(ctx context.Context, m Sandbox, rc string) error {
+func (r *runtime) start(ctx context.Context, m Sandbox) error {
 	if err := r.ready(); err != nil {
 		return err
 	}
@@ -244,7 +576,7 @@ func (r *runtime) start(ctx context.Context, m Sandbox, rc string) error {
 			return err
 		}
 	}
-	if _, err := r.pods().Create(ctx, r.podSpec(m, rc), metav1.CreateOptions{}); err != nil {
+	if _, err := r.pods().Create(ctx, r.podSpec(m), metav1.CreateOptions{}); err != nil {
 		if !apierrors.IsAlreadyExists(err) {
 			return fmt.Errorf("create pod: %w", err)
 		}
@@ -256,9 +588,29 @@ func (r *runtime) start(ctx context.Context, m Sandbox, rc string) error {
 // OUTLIVES the sandbox that mounts it — it holds the checkout and the dependency
 // caches, which is the whole reason a second session is cheap — so this creates
 // and never deletes. Only an explicit purge does that.
+//
+// Because it outlives everything else, it is also the only place its own
+// accounting can live, so every lease stamps the disk with the day it was
+// wanted (annLeased) and every disk is born saying which project it belongs to.
+// Nothing here reclaims anything — this writes the facts a reclaim would need,
+// which is the part that was missing: a disk whose project is a hash and whose
+// last use is unrecorded cannot be shown to be dead, so it is kept forever by
+// default. Measured 2026-08-07: 15 disks, 300GiB, every one of them in exactly
+// that state.
 func (r *runtime) ensureVolume(ctx context.Context, m Sandbox) error {
 	vols := r.dyn.Resource(k8s.Volumes).Namespace(r.ns)
-	if _, err := vols.Get(ctx, m.Volume, metav1.GetOptions{}); err == nil {
+	today := time.Now().UTC().Format(time.DateOnly)
+	if have, err := vols.Get(ctx, m.Volume, metav1.GetOptions{}); err == nil {
+		if have.GetAnnotations()[annLeased] == today {
+			return nil
+		}
+		// A merge patch of the one key, so dating the disk cannot clobber a
+		// concurrent write of anything else on it. The error is dropped on purpose:
+		// a tenant's sandbox does not owe its existence to our bookkeeping, and a
+		// lease that failed because a disk could not be dated would be the rare
+		// outage caused entirely by the thing meant to save money.
+		_, _ = vols.Patch(ctx, m.Volume, types.MergePatchType,
+			[]byte(`{"metadata":{"annotations":{"`+annLeased+`":"`+today+`"}}}`), metav1.PatchOptions{})
 		return nil
 	} else if !apierrors.IsNotFound(err) {
 		return fmt.Errorf("get volume: %w", err)
@@ -267,9 +619,10 @@ func (r *runtime) ensureVolume(ctx context.Context, m Sandbox) error {
 		"apiVersion": "v1",
 		"kind":       "PersistentVolumeClaim",
 		"metadata": map[string]any{
-			"name":      m.Volume,
-			"namespace": r.ns,
-			"labels":    map[string]any{labOrg: slug(m.Org)},
+			"name":        m.Volume,
+			"namespace":   r.ns,
+			"labels":      map[string]any{labOrg: slug(m.Org), labProject: slug(m.Project)},
+			"annotations": map[string]any{annLeased: today},
 		},
 		"spec": map[string]any{
 			"accessModes": []any{"ReadWriteOnce"},
@@ -286,14 +639,10 @@ func (r *runtime) ensureVolume(ctx context.Context, m Sandbox) error {
 }
 
 // podSpec is the sandbox, stated once.
-func (r *runtime) podSpec(m Sandbox, rc string) *unstructured.Unstructured {
+func (r *runtime) podSpec(m Sandbox) *unstructured.Unstructured {
 	c := map[string]any{
-		"name":  container,
-		"image": m.Image,
-		// `sleep infinity` and nothing else. The pod is a place to run commands,
-		// not a program — every lifetime, from a one-shot invoke to a week-long
-		// session, is the same pod entered through the same channel.
-		"command":    []any{"sleep", "infinity"},
+		"name":       container,
+		"image":      m.Image,
 		"workingDir": workdirFor(m.Class),
 		// EPHEMERAL STORAGE IS REQUESTED AND LIMITED, both, and it is not
 		// optional. A pod that requests less than it uses is permanently first in
@@ -319,6 +668,32 @@ func (r *runtime) podSpec(m Sandbox, rc string) *unstructured.Unstructured {
 			"capabilities":             map[string]any{"drop": []any{"ALL"}},
 		},
 	}
+	// `sleep infinity` and nothing else — the pod is a place to run commands, not
+	// a program. Every lifetime, from a one-shot invoke to a week-long session, is
+	// the same pod entered through the same channel.
+	//
+	// EXCEPT A DESKTOP, WHOSE SCREEN IS ITS PROCESS. The desktop image's CMD
+	// starts Xvfb, a window manager and the VNC/noVNC pair and only then becomes
+	// the same `sleep infinity`; stating a command here replaced that script
+	// outright, so the class that exists to have a display came up with no X
+	// server at all — byte-identical to `dev` but for a label, and silent about
+	// it, because a pod that sleeps looks perfectly healthy.
+	//
+	// Deferring to the image is not a second way to start a sandbox. Work still
+	// arrives only through the exec subresource, for all three classes; the
+	// desktop simply also has something of its own to run first.
+	if m.Class != "desktop" {
+		c["command"] = []any{"sleep", "infinity"}
+	} else {
+		// Declared so the screen is addressable by name rather than by a number
+		// somebody has to look up. Ports are how a reader learns a desktop has a
+		// display; they do not open anything the entrypoint has not bound, and it
+		// binds loopback.
+		c["ports"] = []any{
+			map[string]any{"name": "vnc", "containerPort": int64(5900)},
+			map[string]any{"name": "novnc", "containerPort": int64(6080)},
+		}
+	}
 	spec := map[string]any{
 		// No token, ever. A sandbox runs somebody else's code; a projected
 		// service-account token in its filesystem is an API credential handed to
@@ -390,16 +765,19 @@ func (r *runtime) podSpec(m Sandbox, rc string) *unstructured.Unstructured {
 	// names it — so stating it again here would be a second copy that goes stale
 	// the day the runsc node pool moves, and a wrong copy pins pods Pending
 	// forever with a message that blames the wrong object.
-	// PER SANDBOX, falling back to the deployment default. It was
-	// deployment-wide, which meant the same task could not be run on two
-	// runtimes and compared without a rollout — and a caller could not choose.
-	// The row does NOT record it: the pod is the source of truth for what a
-	// sandbox is actually running, and a second copy could only go stale.
-	if rc == "" {
-		rc = r.runtimeClass
-	}
-	if rc != "" {
-		spec["runtimeClassName"] = rc
+	// PER SANDBOX, and read from the sandbox rather than passed beside it. It was
+	// deployment-wide, which meant the same task could not be run on two runtimes
+	// and compared without a rollout — and a caller could not choose.
+	//
+	// m.Runtime is what runtimeFor ANSWERED, resolved once in Lease. There is no
+	// second fallback here on purpose: a `rc == "" then use r.runtimeClass` line
+	// stood here, and it is precisely how the row and the pod come to disagree —
+	// the row would say "the node default" while the pod ran gvisor, and the
+	// person comparing two runtimes would be reading the wrong label on the right
+	// experiment. One derivation, one place, and the field the row reports is the
+	// same field the pod is built from.
+	if m.Runtime != "" {
+		spec["runtimeClassName"] = m.Runtime
 	}
 	// THE WORKDIR IS MOUNTED EITHER WAY, and until now only one of the two ways
 	// existed. A `dev` sandbox gets its project PVC at /work; an `exec` sandbox has
@@ -569,7 +947,11 @@ func (r *runtime) purge(ctx context.Context, m Sandbox) error {
 // arrive at a pod belonging to somebody else — a name is minted once per sandbox
 // and never reused, so the recycled-address break the predecessor defended
 // against with a header cannot be spelled here.
-func (r *runtime) exec(ctx context.Context, m Sandbox, argv []string, stdin io.Reader, timeoutSec int) (ExecResult, error) {
+// say, when a caller named a session, is handed THE SAME BYTES on their way to
+// the buffers, so a watcher reads the program's output while it is still being
+// written. It is a pass-through and never a second read: a tap that re-ran the
+// command to observe it would be observing a different command.
+func (r *runtime) exec(ctx context.Context, m Sandbox, argv []string, stdin io.Reader, timeoutSec int, say *tell) (ExecResult, error) {
 	if err := r.ready(); err != nil {
 		return ExecResult{}, err
 	}
@@ -584,7 +966,13 @@ func (r *runtime) exec(ctx context.Context, m Sandbox, argv []string, stdin io.R
 	defer cancel()
 
 	var out, errb capped
-	err := r.str.stream(ctx, r.ns, m.Pod, argv, stdin, &out, &errb)
+	so, se := io.Writer(&out), io.Writer(&errb)
+	if say != nil {
+		// BOTH streams, through one tell. A failing command says why on stderr, and
+		// a watcher that only saw stdout would watch the silence.
+		so, se = io.MultiWriter(&out, say), io.MultiWriter(&errb, say)
+	}
+	err := r.str.stream(ctx, r.ns, m.Pod, argv, stdin, so, se)
 	res := ExecResult{Stdout: out.String(), Stderr: errb.String()}
 	if err == nil {
 		return res, nil

apps/sandbox/runtime_test.go

@@ -0,0 +1,286 @@
+package sandbox
+
+// The derivation that picks a sandbox's isolation boundary.
+//
+// These cases are a DATA-LOSS boundary, not input hygiene. kata-fc has no
+// shared filesystem, so a sandbox that mounts a project volume under it writes
+// into a tmpfs the VM destroys on exit — and nothing in Kubernetes reports
+// that. The refusal has to happen here because there is no later moment at
+// which it could.
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"strings"
+	"testing"
+
+	"github.com/hanzoai/authz"
+	"github.com/hanzoai/cloud"
+	luxlog "github.com/luxfi/log"
+	"github.com/zap-proto/zip"
+)
+
+func TestRuntimeForAsksTheVolumeNotTheClass(t *testing.T) {
+	// api.go sets Volume from the PROJECT, never from the class. These three are
+	// the shapes that reach runtimeFor.
+	dev := Sandbox{ID: "m_1", Class: "dev", Project: "p", Volume: "m-acme-p-abc"}
+	// THE CASE A class→runtime TABLE WOULD HAVE LOST. `exec` is the class the
+	// fast runtime exists for, and an exec sandbox that names a project carries
+	// a volume exactly like a dev one. Keyed on class, this row silently threw
+	// the org's checkout away.
+	execVol := Sandbox{ID: "m_2", Class: "exec", Project: "p", Volume: "m-acme-p-abc"}
+	keeps := Sandbox{ID: "m_3", Class: "exec"}
+
+	for _, c := range []struct {
+		name string
+		// deploy is SANDBOX_RUNTIME_CLASS: what the deployment states.
+		deploy string
+		// ask is the caller's explicit request, empty for none.
+		ask     string
+		m       Sandbox
+		want    string
+		wantErr string
+	}{
+		// The deployment states a preference, so it is derived DOWN. Refusing a
+		// dev sandbox on a fleet set to kata-fc would make the setting unusable.
+		{"volumeless takes the fast runtime", "kata-fc", "", keeps, "kata-fc", ""},
+		{"a volume forces the shared one", "kata-fc", "", dev, "gvisor", ""},
+		{"an exec WITH a project is a volume", "kata-fc", "", execVol, "gvisor", ""},
+
+		// Rollback is one string: everything lands on gvisor, nothing derives.
+		{"rollback: volumeless", "gvisor", "", keeps, "gvisor", ""},
+		{"rollback: volume", "gvisor", "", dev, "gvisor", ""},
+
+		// Empty is the node's DEFAULT runtime, which is a different request from
+		// any named class. A cluster with no gVisor installed must not be handed
+		// one because a sandbox happened to have a volume.
+		{"unset stays unset, volumeless", "", "", keeps, "", ""},
+		{"unset stays unset, with a volume", "", "", dev, "", ""},
+
+		// kata-clh DOES share a filesystem (configuration-clh.toml sets
+		// shared_fs = "virtio-fs"), so it holds a volume and is not derived away.
+		{"clh shares a filesystem", "kata-clh", "", dev, "kata-clh", ""},
+
+		// A CALLER states a request, so a contradiction is refused rather than
+		// corrected — answering with a runtime nobody asked for is the same
+		// silence this table exists to end.
+		{"forcing fc onto a volume is refused", "gvisor", "kata-fc", dev, "",
+			"cannot mount project volume"},
+		{"forcing fc onto an exec volume is refused", "kata-fc", "kata-fc", execVol, "",
+			"cannot mount project volume"},
+		{"a caller may still choose fc for a volumeless sandbox", "gvisor", "kata-fc", keeps, "kata-fc", ""},
+		{"a caller may choose the shared one", "kata-fc", "gvisor", dev, "gvisor", ""},
+
+		// A runtimeClassName the cluster never heard of is a pod that waits
+		// Pending with no explanation, so a typo stops here.
+		{"an unknown runtime is refused", "gvisor", "runsc", keeps, "", "is not one we run"},
+		{"case matters — the apiserver's does", "gvisor", "gVisor", keeps, "", "is not one we run"},
+	} {
+		t.Run(c.name, func(t *testing.T) {
+			r := &runtime{runtimeClass: c.deploy}
+			got, err := r.runtimeFor(c.m, c.ask)
+			if c.wantErr != "" {
+				if err == nil {
+					t.Fatalf("runtimeFor(%+v, %q) = %q, want a refusal", c.m, c.ask, got)
+				}
+				if !strings.Contains(err.Error(), c.wantErr) {
+					t.Fatalf("refusal %q does not say %q", err, c.wantErr)
+				}
+				return
+			}
+			if err != nil {
+				t.Fatalf("runtimeFor(%+v, %q): %v", c.m, c.ask, err)
+			}
+			if got != c.want {
+				t.Fatalf("runtimeFor(%+v, %q) = %q, want %q", c.m, c.ask, got, c.want)
+			}
+		})
+	}
+}
+
+// THE INVARIANT, stated once and independent of the table above: whatever
+// runtimeFor answers for a sandbox that mounts a volume, that runtime can hold
+// one. Every case that reaches a pod goes through here, so this is the property
+// that makes the silent-tmpfs failure unreachable rather than merely untested.
+// It holds across the TRUST axis too, and that is the point of running both
+// orgs through it: a boundary chosen for who owns the code still has to be one
+// that can hold what the sandbox keeps. Two facts, one answer, no order in
+// which one of them gets forgotten.
+func TestRuntimeForNeverPutsAVolumeOnARuntimeThatCannotHoldIt(t *testing.T) {
+	for _, org := range []string{"acme", authz.AdminOrg} {
+		m := Sandbox{ID: "m_1", Org: org, Class: "dev", Project: "p", Volume: "m-acme-p-abc"}
+		for _, deploy := range append([]string{""}, sorted()...) {
+			for _, want := range append([]string{""}, sorted()...) {
+				for _, contained := range []bool{false, true} {
+					r := &runtime{runtimeClass: deploy}
+					if contained {
+						r.bare = bare()
+					}
+					got, err := r.runtimeFor(m, want)
+					if err != nil {
+						continue // refused, which is the other acceptable answer
+					}
+					if got == "" {
+						// The node's own runtime shares a filesystem like any
+						// ordinary pod, so a volume is safe there — but only the
+						// deployment may ask for it. Deriving it from under a
+						// deployment that NAMED a boundary would be the silent
+						// downgrade this test exists to make unreachable.
+						if deploy != "" {
+							t.Fatalf("org=%q deploy=%q want=%q answered the node default, which the deployment did not ask for",
+								org, deploy, want)
+						}
+						continue
+					}
+					if !runtimes[got].shares {
+						t.Fatalf("org=%q deploy=%q want=%q = %q, which has no shared filesystem — the volume would be a tmpfs",
+							org, deploy, want, got)
+					}
+				}
+			}
+		}
+	}
+}
+
+// THE REFUSAL REACHES THE DOOR, not just the derivation.
+//
+// Lease is where a forced runtime would do its damage, and it refuses before it
+// writes a row, creates a PVC or asks the cluster for anything — so a request
+// that cannot be honoured leaves nothing behind to clean up. No cluster needed
+// to prove it, which is the point: the refusal happens before the first call to
+// one.
+func TestLeaseRefusesAForcedRuntimeBeforeItBuildsAnything(t *testing.T) {
+	s, err := New(cloud.Deps{Logger: luxlog.New("test"), DataDir: t.TempDir()})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	ctx := context.Background()
+	const org = "acme"
+
+	_, err = Lease(s, ctx, org, false, Spec{Class: "dev", Project: "p", Runtime: "kata-fc"})
+	if err == nil {
+		t.Fatal("Lease accepted kata-fc for a sandbox that mounts a volume")
+	}
+	he, ok := err.(*zip.HTTPError)
+	if !ok || he.Status != http.StatusBadRequest {
+		t.Fatalf("refusal is %v, want a 400 — the request is wrong, not the cluster", err)
+	}
+	if !strings.Contains(err.Error(), "cannot mount project volume") {
+		t.Fatalf("refusal %q does not say what is wrong with the request", err)
+	}
+
+	// NOTHING WAS BUILT. A refusal that still wrote the row would leave an
+	// operator reading a sandbox that never existed, and would hold the
+	// one-live-sandbox-per-project slot against a caller who asked correctly.
+	out, err := List(s, ctx, org, "", "")
+	if err != nil {
+		t.Fatalf("List: %v", err)
+	}
+	if len(out) != 0 {
+		t.Fatalf("refused lease left %d row(s) behind: %+v", len(out), out)
+	}
+}
+
+// THE REFUSAL REACHES THE WIRE. The test above proves the domain refuses; this
+// one proves a CLIENT cannot get around it, which is a different claim and the
+// one that matters now that `runtime` is a field on the request body.
+//
+// A browser lets somebody pick a runtime, and a browser can be made to send
+// anything. So the question is not whether the picker offers a safe set — it is
+// whether the door does. It asks for the combination that loses data (a project
+// volume under a runtime with no shared filesystem) the way a crafted client
+// would, straight at the route, and requires a 400 carrying cloud's own sentence
+// rather than a 201 carrying a substituted runtime.
+//
+// A SILENT SUBSTITUTION IS THE FAILURE BEING TESTED FOR, not merely a lost
+// refusal. Handing back gvisor to someone who asked for kata-fc reads as success
+// everywhere: the sandbox starts, the commands run, the files persist — and the
+// person measuring the two runtimes writes down Firecracker's name beside
+// gVisor's numbers. That is why the assertion is on the status AND on the
+// absence of a row, and why the granted runtime is reported at all.
+func TestTheDoorRefusesARuntimeAClientCraftedForItself(t *testing.T) {
+	app := zip.New(zip.Config{Logger: luxlog.New("test")})
+	app.Use(cloud.Bridge())
+	s, err := New(cloud.Deps{Logger: luxlog.New("test"), DataDir: t.TempDir()})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	Routes(app, s)
+	const org = "acme"
+
+	code, body := req(t, app, http.MethodPost, "/v1/sandboxes", org,
+		`{"class":"dev","project":"p","runtime":"kata-fc"}`)
+	if code != http.StatusBadRequest {
+		t.Fatalf("POST /v1/sandboxes with a volume + kata-fc = %d %s, want 400 — "+
+			"a client must not be able to obtain a runtime the policy refuses", code, body)
+	}
+	// The reason has to be READABLE, because a person is going to read it. A bare
+	// 400 sends them to the logs of a service they cannot see.
+	if !strings.Contains(string(body), "cannot mount project volume") {
+		t.Fatalf("refusal body %s does not say why the request is wrong", body)
+	}
+
+	// And it refused BEFORE building anything: no row, so no sandbox an operator
+	// has to explain and no project slot held against the next honest request.
+	code, body = req(t, app, http.MethodGet, "/v1/sandboxes", org, "")
+	if code != http.StatusOK {
+		t.Fatalf("GET /v1/sandboxes = %d %s", code, body)
+	}
+	var listed struct {
+		Sandboxes []Sandbox `json:"sandboxes"`
+	}
+	if err := json.Unmarshal(body, &listed); err != nil {
+		t.Fatalf("shape: %v (%s)", err, body)
+	}
+	if len(listed.Sandboxes) != 0 {
+		t.Fatalf("a refused lease left %d row(s) behind: %+v", len(listed.Sandboxes), listed.Sandboxes)
+	}
+
+	// The unknown-runtime refusal is the same shape, and it is the one a typo
+	// produces: without it the pod sits Pending forever with nothing said.
+	if code, body = req(t, app, http.MethodPost, "/v1/sandboxes", org,
+		`{"class":"exec","runtime":"firecracker"}`); code != http.StatusBadRequest ||
+		!strings.Contains(string(body), "is not one we run") {
+		t.Fatalf("POST with an invented runtime = %d %s, want 400 naming the set we run", code, body)
+	}
+}
+
+// THE SANDBOX SAYS WHICH RUNTIME IT GOT, and the point of the field is that it
+// can differ from the one asked for. A caller that can only read back its own
+// request learns nothing; the derivation is allowed to answer something else,
+// and the answer is the only honest label for a measurement.
+//
+// No cluster is needed to prove the reporting, only the lease that fails to
+// reach one: Lease writes the row with the granted runtime BEFORE it calls the
+// cluster, so the row it leaves behind on a 503 carries exactly the value the
+// pod spec would have been built from.
+func TestASandboxReportsTheRuntimeItGotNotTheOneItAskedFor(t *testing.T) {
+	s, err := New(cloud.Deps{Logger: luxlog.New("test"), DataDir: t.TempDir()})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	// A fleet set to the fast runtime. A dev sandbox cannot have it — it mounts a
+	// volume — so the deployment preference is derived down to the shared one,
+	// and that, not "kata-fc", is what the row must say.
+	s.State.rt.runtimeClass = "kata-fc"
+	// No cluster, said once and immediately. A developer machine may well have a
+	// kubeconfig, and then this test spends the full start timeout waiting for a
+	// pod it does not need — the row is written before the cluster is asked, so
+	// the answer is already there.
+	s.State.rt.dyn = nil
+	ctx := context.Background()
+	const org = "acme"
+
+	if _, err = Lease(s, ctx, org, false, Spec{Class: "dev", Project: "p"}); err == nil {
+		t.Fatal("Lease reached a cluster in a unit test")
+	}
+	out, err := List(s, ctx, org, "", "")
+	if err != nil || len(out) != 1 {
+		t.Fatalf("List = %+v, %v — want the one row Lease recorded", out, err)
+	}
+	if out[0].Runtime != shared {
+		t.Fatalf("row says runtime %q, but a volume-bearing sandbox on a kata-fc fleet gets %q",
+			out[0].Runtime, shared)
+	}
+}

apps/sandbox/sandbox.go

@@ -19,6 +19,14 @@
 //	GET    /v1/sandboxes/:id/terminal        ?ticket=&arg=  the terminal, as a page
 //	GET    /v1/sandboxes/:id/terminal/ws     ?ticket=&arg=  the terminal, as a socket
 //
+// A RUN IS WATCHABLE AND IT IS STOPPABLE. Name a session on a run and the
+// command's output is appended to that session's live log as it is produced, so
+// a surface reading GET /v1/agents/sessions/stream watches the work happen
+// instead of a blank pause; `stop_run` interrupts what a sandbox is running and
+// leaves the sandbox leased, because a run that went wrong is one somebody still
+// wants to look at. See work.go — both are the same fact, that a command in
+// flight is addressable.
+//
 // THERE IS EXACTLY ONE WAY INTO A SANDBOX, and it is the Kubernetes exec
 // subresource. fs read/list/write are not a second channel — they are `cat`,
 // `ls` and `tee` over that one channel, which is also how `kubectl cp` has
@@ -96,6 +104,10 @@ type state struct {
 	// to open a terminal. Per service and in memory — see terminal.go for why
 	// the one credential a WebSocket can carry is minted rather than borrowed.
 	tickets *tickets
+	// work is every command in flight, by the sandbox running it, so a caller can
+	// stop one. In memory for the same reason the tickets are: it holds a live
+	// goroutine's cancel, which exists nowhere but here. See work.go.
+	work *work
 }
 
 // storeFor is the ONE way this package reaches a store, through
@@ -131,6 +143,7 @@ func Mount(app cloud.Router, deps cloud.Deps) error {
 		stores:  cloud.NewOrgStore(b, "sandbox", openStore),
 		rt:      newRuntime(),
 		tickets: newTickets(),
+		work:    newWork(),
 	}}
 	Routes(app, s)
 	// The peer half. Registered beside the routes because they are two adapters
@@ -144,9 +157,18 @@ func Mount(app cloud.Router, deps cloud.Deps) error {
 	// created ran forever — a proof pod was found still Running 64 minutes
 	// after its test had finished.
 	go reap(context.Background(), s)
+	// A runtime that cannot serve says so ONCE AND IN FULL, at startup. `cluster:
+	// false` alone is a symptom with the cause stripped off, and the two causes
+	// read nothing alike: a cluster we cannot reach is an outage to page on, a
+	// SANDBOX_RUNTIME_CLASS the table has never heard of is a typo to fix. Both
+	// fail every lease closed; only one of them is anybody's fault.
+	if err := s.State.rt.ready(); err != nil {
+		s.Log.Error("sandbox cannot serve", "why", err)
+	}
 	s.Log.Info("sandbox mounted",
 		"namespace", s.State.rt.ns, "image", s.State.rt.image,
-		"runtimeClass", s.State.rt.runtimeClass, "cluster", s.State.rt.ready() == nil,
+		"runtimeClass", s.State.rt.runtimeClass, "bare", s.State.rt.bare,
+		"cluster", s.State.rt.ready() == nil,
 		"reapEvery", reapEvery, "idleAfter", idleAfter)
 	return nil
 }
@@ -204,6 +226,13 @@ func Routes(app cloud.Router, s *cloud.Service[state]) {
 		zip.Post[plane.WriteIn, plane.Wrote](reg, "/v1/sandboxes/write", planeWrite,
 			zip.WithOperationID("write_sandbox_file"),
 			zip.WithSummary("Write a file into a sandbox you hold"))
+		// STOP ENDS THE WORK; END ENDS THE RESOURCE. They are two verbs because a
+		// run that has gone wrong is one somebody still wants to look at, and an
+		// agent told to "stop" that deleted the pod would take the checkout, the
+		// logs and the half-written file with it.
+		zip.Post[plane.StopIn, plane.Stopped](reg, "/v1/sandboxes/stop", planeStop,
+			zip.WithOperationID("stop_run"),
+			zip.WithSummary("Stop what a sandbox is running, and keep the sandbox"))
 		zip.Post[plane.EndIn, struct{}](reg, "/v1/sandboxes/end", planeEnd,
 			zip.WithOperationID("end_sandbox"),
 			zip.WithSummary("End a sandbox and release it"))
@@ -231,6 +260,7 @@ func New(deps cloud.Deps) (*Service, error) {
 		stores:  cloud.NewOrgStore(b, "sandbox", openStore),
 		rt:      newRuntime(),
 		tickets: newTickets(),
+		work:    newWork(),
 	}}, nil
 }
 
@@ -240,23 +270,40 @@ func New(deps cloud.Deps) (*Service, error) {
 // part that is genuinely about HTTP: where a value comes from on the wire, and
 // which status carries it back.
 
+// createBody is what a request may state about a sandbox. Named rather than
+// anonymous so the fields it does NOT carry are checkable: no org and no
+// runtime, which are the two facts runtimeFor derives from and the two a caller
+// must never be able to hand in. See trust_test.go.
+type createBody struct {
+	Kind    string `json:"kind"`
+	Class   string `json:"class"`
+	Project string `json:"project"`
+	Image   string `json:"image"`
+	// Runtime is the isolation boundary the caller would LIKE. The server still
+	// decides — runtimeFor refuses a choice it cannot honour instead of quietly
+	// substituting one — so this field may be asked for by anyone and obtained
+	// by no one the policy would turn away. The sandbox that comes back carries
+	// the runtime it GOT, which is the field to read.
+	Runtime string `json:"runtime"`
+	TTLSec  int    `json:"ttlSec"`
+}
+
 func create(s *Service, c *zip.Ctx) error {
 	o, ok := orgOf(c)
 	if !ok {
 		return zip.ErrForbidden("X-Org-Id required")
 	}
-	var body struct {
-		Kind    string `json:"kind"`
-		Class   string `json:"class"`
-		Project string `json:"project"`
-		Image   string `json:"image"`
-		TTLSec  int    `json:"ttlSec"`
-	}
+	var body createBody
 	if err := c.Bind(&body); err != nil {
 		return err
 	}
-	m, err := Lease(s, c.Context(), o, Spec{
-		Class: body.Class, Project: body.Project, Image: body.Image, TTLSec: body.TTLSec})
+	// principal.IsSuperAdmin is THE predicate — membership of the reserved `admin`
+	// org, attested by the identity middleware. It is read here and nowhere else in
+	// this package: what it decides is which image a `dev` sandbox runs (imageFor),
+	// and a second caller of it would be a second answer to drift from.
+	m, err := Lease(s, c.Context(), o, principal.IsSuperAdmin(c), Spec{
+		Class: body.Class, Project: body.Project, Image: body.Image,
+		Runtime: body.Runtime, TTLSec: body.TTLSec})
 	if err != nil {
 		return err
 	}
@@ -522,7 +569,8 @@ func init() {
 			"many terminals as a caller has names for. It is 1-64 characters of letters, digits, "+
 			"`-` or `_` and may not begin with `-`; anything else is 400. Without `arg` the shell "+
 			"is unnamed and unmultiplexed.\n\n"+
-			"The shell is `bash -l`, falling back to `sh -l`, and to the plain shell again when "+
-			"the image has no tmux. Whatever else the image carries — the hanzo CLI included — is "+
-			"a command to type, never a requirement to get a prompt.")
+			"The shell is `zsh -l`, falling back to `bash -l` and then to `sh -l`, and to the "+
+			"plain shell again when the image has no tmux. Every step is a preference and none "+
+			"is a requirement: whatever else the image carries — the hanzo CLI included — is a "+
+			"command to type, never a condition for getting a prompt.")
 }

apps/sandbox/store.go

@@ -24,14 +24,29 @@ var errNotFound = errors.New("sandbox: sandbox not found")
 // other than the IAM edge. The predecessor returned a pod IP in every create,
 // get and list response.
 type Sandbox struct {
-	ID         string `json:"id"`
-	Org        string `json:"org"`
-	Kind       string `json:"kind"`
-	Class      string `json:"class"`
-	Project    string `json:"project,omitempty"`
-	Status     string `json:"status"` // pending | running | error
-	Image      string `json:"image"`
-	Pod        string `json:"-"`
+	ID      string `json:"id"`
+	Org     string `json:"org"`
+	Kind    string `json:"kind"`
+	Class   string `json:"class"`
+	Project string `json:"project,omitempty"`
+	Status  string `json:"status"` // pending | running | error
+	Image   string `json:"image"`
+	Pod     string `json:"-"`
+	// Runtime is the isolation boundary this sandbox GOT, which is not always the
+	// one it asked for: a caller states a preference and runtimeFor answers with
+	// what the sandbox can actually have. Reported so a person comparing two
+	// runtimes is comparing the runtimes they got rather than the ones they typed
+	// — the difference between those two is the whole reason to record it.
+	//
+	// Empty means the node's default runtime, which is a real answer and not a
+	// missing one.
+	//
+	// This is not a copy that can go stale. runtimeClassName is IMMUTABLE on a
+	// pod, a sandbox's pod is created once and never recreated (restartPolicy
+	// Never, no pool), and its name is never reused — so for as long as the pod
+	// this row names exists, it is running this runtime. The alternative, asking
+	// the apiserver on every read, buys nothing and costs a round trip per row.
+	Runtime    string `json:"runtime,omitempty"`
 	Volume     string `json:"volume,omitempty"`
 	Error      string `json:"error,omitempty"`
 	CreatedAt  int64  `json:"createdAt"`
@@ -65,6 +80,7 @@ CREATE TABLE IF NOT EXISTS sandbox (
   status       TEXT NOT NULL DEFAULT 'pending',
   image        TEXT NOT NULL DEFAULT '',
   pod          TEXT NOT NULL DEFAULT '',
+  runtime      TEXT NOT NULL DEFAULT '',
   volume       TEXT NOT NULL DEFAULT '',
   error        TEXT NOT NULL DEFAULT '',
   created_at   INTEGER NOT NULL,
@@ -77,6 +93,14 @@ CREATE INDEX IF NOT EXISTS ix_machines_org_status  ON sandbox(org, status);
 	if _, err := s.db.Exec(ddl); err != nil {
 		return fmt.Errorf("migrate: %w", err)
 	}
+	// runtime: added after the initial schema. Fresh DBs get it from the CREATE
+	// above; a DB that predates it gains it here, and its existing rows read as
+	// the node default — which is what they in fact ran. SQLite has no ADD COLUMN
+	// IF NOT EXISTS, so the duplicate-column error is the expected no-op.
+	if _, err := s.db.Exec(`ALTER TABLE sandbox ADD COLUMN runtime TEXT NOT NULL DEFAULT ''`); err != nil &&
+		!strings.Contains(err.Error(), "duplicate column") {
+		return fmt.Errorf("migrate runtime column: %w", err)
+	}
 	return nil
 }
 
@@ -84,12 +108,13 @@ func (s *Store) Close() error { return s.db.Close() }
 
 func (s *Store) Put(ctx context.Context, m Sandbox) error {
 	_, err := s.db.ExecContext(ctx, `
-INSERT INTO sandbox (id,org,kind,class,project,status,image,pod,volume,error,created_at,last_used_at,expires_at)
-VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)
+INSERT INTO sandbox (id,org,kind,class,project,status,image,pod,runtime,volume,error,created_at,last_used_at,expires_at)
+VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)
 ON CONFLICT(id) DO UPDATE SET
-  status=excluded.status, image=excluded.image, pod=excluded.pod, volume=excluded.volume,
+  status=excluded.status, image=excluded.image, pod=excluded.pod, runtime=excluded.runtime,
+  volume=excluded.volume,
   error=excluded.error, last_used_at=excluded.last_used_at, expires_at=excluded.expires_at`,
-		m.ID, m.Org, m.Kind, m.Class, m.Project, m.Status, m.Image, m.Pod, m.Volume,
+		m.ID, m.Org, m.Kind, m.Class, m.Project, m.Status, m.Image, m.Pod, m.Runtime, m.Volume,
 		m.Error, m.CreatedAt, m.LastUsedAt, m.ExpiresAt)
 	return err
 }
@@ -131,7 +156,7 @@ func (s *Store) List(ctx context.Context, org, project, status string) ([]Sandbo
 	for rows.Next() {
 		var m Sandbox
 		if err := rows.Scan(&m.ID, &m.Org, &m.Kind, &m.Class, &m.Project, &m.Status,
-			&m.Image, &m.Pod, &m.Volume, &m.Error, &m.CreatedAt, &m.LastUsedAt, &m.ExpiresAt); err != nil {
+			&m.Image, &m.Pod, &m.Runtime, &m.Volume, &m.Error, &m.CreatedAt, &m.LastUsedAt, &m.ExpiresAt); err != nil {
 			return nil, err
 		}
 		out = append(out, m)
@@ -163,7 +188,7 @@ func (s *Store) Expired(ctx context.Context, org string, now int64) ([]Sandbox,
 	for rows.Next() {
 		var m Sandbox
 		if err := rows.Scan(&m.ID, &m.Org, &m.Kind, &m.Class, &m.Project, &m.Status,
-			&m.Image, &m.Pod, &m.Volume, &m.Error, &m.CreatedAt, &m.LastUsedAt, &m.ExpiresAt); err != nil {
+			&m.Image, &m.Pod, &m.Runtime, &m.Volume, &m.Error, &m.CreatedAt, &m.LastUsedAt, &m.ExpiresAt); err != nil {
 			return nil, err
 		}
 		out = append(out, m)
@@ -215,12 +240,12 @@ func (s *Store) Delete(ctx context.Context, org, id string) error {
 	return err
 }
 
-const selectCols = `SELECT id,org,kind,class,project,status,image,pod,volume,error,created_at,last_used_at,expires_at FROM sandbox`
+const selectCols = `SELECT id,org,kind,class,project,status,image,pod,runtime,volume,error,created_at,last_used_at,expires_at FROM sandbox`
 
 func scanMachine(row *sql.Row) (Sandbox, error) {
 	var m Sandbox
 	err := row.Scan(&m.ID, &m.Org, &m.Kind, &m.Class, &m.Project, &m.Status,
-		&m.Image, &m.Pod, &m.Volume, &m.Error, &m.CreatedAt, &m.LastUsedAt, &m.ExpiresAt)
+		&m.Image, &m.Pod, &m.Runtime, &m.Volume, &m.Error, &m.CreatedAt, &m.LastUsedAt, &m.ExpiresAt)
 	if errors.Is(err, sql.ErrNoRows) {
 		return Sandbox{}, errNotFound
 	}

apps/sandbox/terminal.go

@@ -75,11 +75,16 @@ const ticketTTL = 30 * time.Second
 
 // shell is what a terminal runs, and everything it needs is `/bin/sh`.
 //
-// It asks for bash and settles for sh, because the three sandbox images are not
-// one image and a shell that must exist is a shell that will one day not — the
-// exec class is a stock node image today. Whatever tools the image carries, the
-// hanzo CLI included, are commands the user types; none is a requirement for a
-// prompt.
+// It asks for zsh, settles for bash, and settles again for sh, because the
+// sandbox images are not one image and a shell that must exist is a shell that
+// will one day not — the exec class is a stock node image today and only the
+// admin image carries zsh. Whatever tools the image carries, the hanzo CLI
+// included, are commands the user types; none is a requirement for a prompt.
+//
+// A PREFERENCE AND NOT A REQUIREMENT, all the way down. Asking the image what
+// it has costs nothing when the answer is no — the next `exec` in the chain
+// simply runs — whereas naming one shell would make the terminal a feature of
+// the image rather than of the sandbox.
 //
 // A NAMED session is the same shell under tmux: `new -A` attaches to the session
 // if it is there and creates it if it is not, which is what lets ONE sandbox hold
@@ -95,7 +100,7 @@ func shell(session string) []string {
 		"command -v tmux >/dev/null 2>&1 && exec tmux new -A -s " + shellQuote(session) + "; " + plain}
 }
 
-const plain = "exec bash -l 2>/dev/null || exec sh -l"
+const plain = "exec zsh -l 2>/dev/null || exec bash -l 2>/dev/null || exec sh -l"
 
 // sessionOK is what a session name may be. It is an allowlist and not an escape,
 // because a name reaches a command line: `-` would be read by tmux as a flag and

apps/sandbox/terminal_live_test.go

@@ -64,7 +64,11 @@ func TestLiveTerminalIsARealShell(t *testing.T) {
 	defer cancel()
 
 	t.Logf("starting %s image=%s ns=%s", m.Pod, m.Image, r.ns)
-	if err := r.start(ctx, m, ""); err != nil {
+	// The isolation boundary comes from the SAME derivation production uses, so the
+	// live proof runs under whatever the fleet is set to rather than under the
+	// node default. want is empty, and runtimeFor cannot refuse an empty ask.
+	m.Runtime, _ = r.runtimeFor(m, "")
+	if err := r.start(ctx, m); err != nil {
 		t.Fatalf("start: %v", err)
 	}
 	defer func() {
@@ -169,12 +173,16 @@ func TestLiveTerminalNamedSession(t *testing.T) {
 	}
 	ctx, cancel := context.WithTimeout(context.Background(), 4*time.Minute)
 	defer cancel()
-	if err := r.start(ctx, m, ""); err != nil {
+	// The isolation boundary comes from the SAME derivation production uses, so the
+	// live proof runs under whatever the fleet is set to rather than under the
+	// node default. want is empty, and runtimeFor cannot refuse an empty ask.
+	m.Runtime, _ = r.runtimeFor(m, "")
+	if err := r.start(ctx, m); err != nil {
 		t.Fatalf("start: %v", err)
 	}
 	defer func() { _ = r.stop(context.Background(), m) }()
 
-	has, err := r.exec(ctx, m, []string{"sh", "-c", "command -v tmux >/dev/null 2>&1 && echo yes || echo no"}, nil, 60)
+	has, err := r.exec(ctx, m, []string{"sh", "-c", "command -v tmux >/dev/null 2>&1 && echo yes || echo no"}, nil, 60, nil)
 	if err != nil {
 		t.Fatalf("probe tmux: %v", err)
 	}

apps/sandbox/terminal_test.go

@@ -229,10 +229,10 @@ func TestWindowReportsTheLatestSizeAndThenEnds(t *testing.T) {
 	w.close()
 }
 
-// The shell must not require anything of the image beyond /bin/sh. The three
-// sandbox classes are three different images and the exec one is stock node
-// today, so a command that assumed a tool would be a terminal that opens and
-// immediately dies with a message nobody can read through a closed socket.
+// The shell must not require anything of the image beyond /bin/sh. The sandbox
+// classes are different images and the exec one is stock node today, so a
+// command that assumed a tool would be a terminal that opens and immediately
+// dies with a message nobody can read through a closed socket.
 func TestShellRequiresOnlySh(t *testing.T) {
 	argv := shell("")
 	if len(argv) != 3 || argv[0] != "/bin/sh" || argv[1] != "-lc" {
@@ -248,6 +248,36 @@ func TestShellRequiresOnlySh(t *testing.T) {
 	}
 }
 
+// THE CHAIN IS AN ORDER AND EVERY LINK IS OPTIONAL.
+//
+// zsh is what the admin image ships and what an operator expects to land in;
+// bash is what the toolchain images have; sh is what everything has. Two
+// properties, and the second is the one that is easy to lose: each step must be
+// asked for and not required, so a class that carries no zsh gets bash and a
+// stock node image still gets a prompt.
+//
+// It is checked by ORDER rather than by matching the whole string, because the
+// string is a shell fragment and asserting it verbatim would make every future
+// edit a test edit. What must not change is which shell wins when two are there.
+func TestShellPrefersZshAndFallsAllTheWayDown(t *testing.T) {
+	cmd := shell("")[2]
+	z, b, s := strings.Index(cmd, "exec zsh"), strings.Index(cmd, "exec bash"), strings.Index(cmd, "exec sh")
+	if z < 0 || b < 0 || s < 0 {
+		t.Fatalf("%q does not name all three of zsh, bash and sh", cmd)
+	}
+	if !(z < b && b < s) {
+		t.Errorf("%q does not prefer zsh, then bash, then sh — an operator's terminal "+
+			"lands in whichever shell comes first", cmd)
+	}
+	// Each step SETTLES. `||` is what makes a missing shell cost the next one on
+	// the list rather than the terminal; a chain joined by `&&` or `;` would open
+	// a socket into an image that has no zsh and close it again.
+	if strings.Count(cmd, "||") < 2 {
+		t.Errorf("%q does not fall through: every shell here is a preference, and an "+
+			"image that lacks one must still give a prompt", cmd)
+	}
+}
+
 // A NAMED terminal is the same shell under tmux, and the name is what lets one
 // sandbox hold many. Two properties have to hold together: the session is
 // ATTACHED if it exists (`new -A`, or every reframe starts a fresh empty shell

apps/sandbox/trust_test.go

@@ -0,0 +1,574 @@
+package sandbox
+
+// The TRUST half of the derivation: who owns the code a sandbox runs, and what
+// that buys it.
+//
+// These are not input-hygiene cases. runc IS the node's kernel, so getting this
+// wrong does not lose a volume — it puts a stranger's prompt one kernel bug away
+// from the node every other tenant's sandbox is on. The refusal has to happen in
+// runtimeFor because there is no later moment at which it could: by the time the
+// pod exists the boundary has already been chosen.
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"reflect"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/hanzoai/authz"
+	"github.com/hanzoai/cloud"
+	"github.com/hanzoai/cloud/apps/k8s"
+	"github.com/hanzoai/cloud/plane"
+	luxlog "github.com/luxfi/log"
+	"github.com/zap-proto/zip"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	k8sruntime "k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+	dynamicfake "k8s.io/client-go/dynamic/fake"
+)
+
+// leased is a sandbox as api.go builds one: the org is the CALLER's, the volume
+// comes from the project. Both facts arrive here already decided, which is why
+// runtimeFor asks the Sandbox and never its own arguments.
+func leased(org, volume string) Sandbox {
+	return Sandbox{ID: "m_1", Org: org, Class: "dev", Project: "p", Volume: volume}
+}
+
+// TestRuntimeForAsksWhoOwnsTheCodeAndWhatItKeeps is the two-axis derivation,
+// stated as the cases that reach it. `contained` is whether the cluster keeps
+// our own boundary to nodes of its own — see confine; without it there is no
+// such boundary to take and our code lands where everybody else's does.
+func TestRuntimeForAsksWhoOwnsTheCodeAndWhatItKeeps(t *testing.T) {
+	const other = "acme"
+	for _, c := range []struct {
+		name      string
+		deploy    string // SANDBOX_RUNTIME_CLASS
+		contained bool
+		ask       string // what a caller asked for, empty for none
+		m         Sandbox
+		want      string
+		wantErr   string
+	}{
+		// OURS, on a pool of its own. Both facts are read at once: the volume
+		// does not cost us the fast boundary, because runc shares a filesystem.
+		{"ours, contained, volumeless", "gvisor", true, "", leased(authz.AdminOrg, ""), "runc", ""},
+		{"ours, contained, with a volume", "gvisor", true, "", leased(authz.AdminOrg, "v"), "runc", ""},
+
+		// OURS, with nowhere to put it. Fail to gvisor rather than run a model's
+		// output on a kernel shared with whatever else the scheduler chose.
+		{"ours, uncontained, volumeless", "gvisor", false, "", leased(authz.AdminOrg, ""), "gvisor", ""},
+		{"ours, uncontained, with a volume", "gvisor", false, "", leased(authz.AdminOrg, "v"), "gvisor", ""},
+
+		// EVERYBODY ELSE, whatever the topology. A pool of our own does not make
+		// a stranger's code ours.
+		{"another org, contained", "gvisor", true, "", leased(other, ""), "gvisor", ""},
+		{"another org, contained, with a volume", "gvisor", true, "", leased(other, "v"), "gvisor", ""},
+
+		// THE SETTING CANNOT HAND THE FLEET AWAY. A deployment that names runc
+		// still gets a kernel for everybody who is not us — one env var was the
+		// whole boundary, and now it is not.
+		{"the deployment names runc: ours takes it", "runc", true, "", leased(authz.AdminOrg, ""), "runc", ""},
+		{"the deployment names runc: a tenant does not", "runc", true, "", leased(other, ""), "gvisor", ""},
+		{"the deployment names runc: a tenant with a volume does not", "runc", true, "", leased(other, "v"), "gvisor", ""},
+
+		// A CALLER states a request, so a contradiction is refused rather than
+		// corrected — a runtime nobody asked for is the same silence one level up.
+		{"a tenant asking for runc is refused", "gvisor", true, "runc", leased(other, ""), "",
+			"only for code of ours"},
+		{"a tenant asking for runc with a volume is refused", "gvisor", true, "runc", leased(other, "v"), "",
+			"only for code of ours"},
+		{"we may ask for runc", "gvisor", true, "runc", leased(authz.AdminOrg, ""), "runc", ""},
+		{"we may ask for more than we need", "gvisor", true, "kata-clh", leased(authz.AdminOrg, "v"), "kata-clh", ""},
+		{"even we cannot put a volume on kata-fc", "gvisor", true, "kata-fc", leased(authz.AdminOrg, "v"), "",
+			"cannot mount project volume"},
+
+		// AN UNCONTAINED runc IS NOT ONE WE RUN, FOR US EITHER. The name is in
+		// the table; the topology is not, so neither path selects it — and the
+		// path that lets us NAME it is the one that would otherwise put it on a
+		// pool it does not own.
+		{"asking for runc with nowhere to put it", "gvisor", false, "runc", leased(authz.AdminOrg, ""), "",
+			"does not keep it to a pool"},
+		{"deriving runc with nowhere to put it", "gvisor", false, "", leased(authz.AdminOrg, ""), "gvisor", ""},
+	} {
+		t.Run(c.name, func(t *testing.T) {
+			r := &runtime{runtimeClass: c.deploy}
+			if c.contained {
+				r.bare = bare()
+			}
+			got, err := r.runtimeFor(c.m, c.ask)
+			if c.wantErr != "" {
+				if err == nil {
+					t.Fatalf("runtimeFor(%+v, %q) = %q, want a refusal", c.m, c.ask, got)
+				}
+				if !strings.Contains(err.Error(), c.wantErr) {
+					t.Fatalf("refusal %q does not say %q", err, c.wantErr)
+				}
+				return
+			}
+			if err != nil {
+				t.Fatalf("runtimeFor(%+v, %q): %v", c.m, c.ask, err)
+			}
+			if got != c.want {
+				t.Fatalf("runtimeFor(%+v, %q) = %q, want %q", c.m, c.ask, got, c.want)
+			}
+		})
+	}
+}
+
+// NEGATIVE CONTROL (b): a caller who is not us can never obtain a boundary that
+// shares the node's kernel.
+//
+// Exhaustive over every input runtimeFor has — the deployment's setting, the
+// caller's request, whether the sandbox keeps anything, and whether the cluster
+// contains our boundary — for an org that is not the reserved one. Stated as a
+// property rather than as rows, because the interesting case is the combination
+// nobody thought to write down.
+//
+// The carve-out is honest and it is the one residual: an EMPTY answer is the
+// node's default runtime, which the deployment declined to name at all. That is
+// a deployment fact rather than a derivation fact — production pins
+// SANDBOX_RUNTIME_CLASS=gvisor — and the property below is the exact one the
+// derivation owns: it never NAMES a kernel-sharing boundary for code that is
+// not ours.
+func TestOnlyOurOwnCodeReachesTheNodesKernel(t *testing.T) {
+	orgs := []string{"acme", "hanzo", "zoo", "Admin", "admin ", "admin\n", ""}
+	asks := append([]string{""}, sorted()...)
+	for _, org := range orgs {
+		for _, deploy := range append([]string{""}, sorted()...) {
+			for _, ask := range asks {
+				for _, vol := range []string{"", "m-acme-p-abc"} {
+					for _, contained := range []bool{false, true} {
+						r := &runtime{runtimeClass: deploy}
+						if contained {
+							r.bare = bare()
+						}
+						got, err := r.runtimeFor(leased(org, vol), ask)
+						if err != nil {
+							continue // refused, which is the other acceptable answer
+						}
+						if got == "" {
+							// The node's default, and ONLY when the deployment
+							// named no boundary of its own. Anything else is a
+							// silent downgrade out from under a setting.
+							if deploy != "" {
+								t.Fatalf("org=%q deploy=%q ask=%q answered the node default, which the deployment did not ask for",
+									org, deploy, ask)
+							}
+							continue
+						}
+						if !runtimes[got].kernel {
+							t.Fatalf("org=%q deploy=%q ask=%q vol=%q contained=%v answered %q, "+
+								"which is the node's own kernel", org, deploy, ask, vol, contained, got)
+						}
+					}
+				}
+			}
+		}
+	}
+}
+
+// THE INVARIANT ACROSS BOTH PATHS, and the one that catches what a table of
+// cases cannot: whatever runtimeFor answers, this deployment can actually place
+// it. A boundary with a kernel of its own may be named anywhere; the one without
+// exists only where the cluster keeps it to nodes of its own.
+//
+// It is written as a property because the gap it found was a gap of SYMMETRY —
+// the derived path consulted the topology and the by-name path did not, so the
+// one caller entitled to ask for the node's kernel was the one caller who could
+// get it on a pool it did not own. Two paths, one fact, checked here.
+func TestRuntimeForNeverAnswersABoundaryTheClusterCannotPlace(t *testing.T) {
+	for _, org := range []string{"acme", authz.AdminOrg} {
+		for _, deploy := range append([]string{""}, sorted()...) {
+			for _, ask := range append([]string{""}, sorted()...) {
+				for _, vol := range []string{"", "m-acme-p-abc"} {
+					for _, contained := range []bool{false, true} {
+						r := &runtime{runtimeClass: deploy}
+						if contained {
+							r.bare = bare()
+						}
+						got, err := r.runtimeFor(leased(org, vol), ask)
+						if err != nil || got == "" {
+							continue
+						}
+						if !runtimes[got].kernel && got != r.bare {
+							t.Fatalf("org=%q deploy=%q ask=%q contained=%v answered %q, which this "+
+								"cluster does not keep to a pool of its own", org, deploy, ask, contained, got)
+						}
+					}
+				}
+			}
+		}
+	}
+}
+
+// NEGATIVE CONTROL (c), first half: there is nothing on a request that could
+// name an org or a runtime.
+//
+// A reflective test rather than a prose promise, because the field that gets
+// added later is exactly the one nobody re-reads this comment for. Spec.ID and
+// Spec.RuntimeClass are what a Go caller in this process may state; the WIRE
+// types carry neither an org nor a runtime, so no request can reach either
+// axis of the derivation.
+// IDENTITY IS NOT A REQUEST, and that is the whole of this rule.
+//
+// An org on the body would BE the escalation. There is no second source of
+// truth to check it against — the value the derivation reads would be the value
+// the caller typed — so accepting the field is accepting the claim, and no code
+// downstream can undo it. Hence: never on the wire, enforced by shape rather
+// than by remembering.
+//
+// A RUNTIME IS A DIFFERENT AXIS and it is deliberately not in this list, though
+// it was. The server holds the entire policy: runtimeFor derives `kernel` from
+// m.Org — the validated principal, never a field — and `shares` from the volume,
+// and answers a request it cannot honour with a refusal rather than with a
+// substitution. So a caller may ASK for runc and cannot GET it, which is not the
+// same shape of hazard at all.
+//
+// Barring the field outright had a cost and no benefit. The benefit was
+// imaginary: `want` still flows into runtimeFor from the deployment, and every
+// refusal branch under `want != ""` — the two runc ones directly above — became
+// unreachable by any real caller, which is to say untested in production
+// forever. The cost was real: comparing two boundaries on one task needed a
+// redeploy, so nobody compared them.
+//
+// What replaces it is the test below, which asserts the thing actually worth
+// asserting — that asking does not get.
+func TestNoWireFieldCanNameAnOrg(t *testing.T) {
+	for _, in := range []any{plane.LeaseIn{}, createBody{}} {
+		ty := reflect.TypeOf(in)
+		for i := 0; i < ty.NumField(); i++ {
+			switch n := strings.ToLower(ty.Field(i).Name); {
+			case strings.Contains(n, "org"), strings.Contains(n, "owner"),
+				strings.Contains(n, "tenant"):
+				t.Fatalf("%s.%s would let a caller state its own %s", ty.Name(), ty.Field(i).Name, n)
+			}
+		}
+	}
+}
+
+// ASKING IS NOT GETTING — the trust axis, held at the DOOR.
+//
+// runc is the node's own kernel. The derivation refuses it to anyone outside the
+// reserved org, and this proves the refusal survives the trip through a request
+// body: a stranger POSTing `{"runtime":"runc"}` is answered 400 with the reason,
+// not 201 with a pod beside every other tenant's.
+//
+// The controls are what make it a proof rather than a coincidence. The SAME body
+// from the SAME org without `runtime` reaches the cluster (503, there is none
+// here) — so the 400 is a refusal of the ASK and not of the route or the org. And
+// the reserved org is refused too, because this cluster keeps no pool of its own
+// (`bare` is empty), which is the second half of the derivation: even ours only
+// gets that boundary where the topology holds.
+func TestAskingForTheNodesKernelOverHTTPDoesNotGetIt(t *testing.T) {
+	app := door(t)
+	for _, c := range []struct {
+		name, org, body string
+		want            int
+	}{
+		{"a stranger asks for the node's kernel", "acme", `{"class":"exec","runtime":"runc"}`, http.StatusBadRequest},
+		{"ours asks, on a cluster that keeps no pool", authz.AdminOrg, `{"class":"exec","runtime":"runc"}`, http.StatusBadRequest},
+		{"an invented runtime", "acme", `{"class":"exec","runtime":"firecracker"}`, http.StatusBadRequest},
+		{"a volume under a boundary that shares nothing", "acme", `{"class":"dev","project":"p","runtime":"kata-fc"}`, http.StatusBadRequest},
+
+		// THE CONTROLS. Both reach the cluster, which this test does not have —
+		// so 503 is "the request was fine", and it is what makes every 400 above
+		// a statement about the runtime rather than about the request at large.
+		{"the control: no runtime named", "acme", `{"class":"exec"}`, http.StatusServiceUnavailable},
+		{"the control: a boundary anyone may take", "acme", `{"class":"exec","runtime":"gvisor"}`, http.StatusServiceUnavailable},
+	} {
+		t.Run(c.name, func(t *testing.T) {
+			if code := ask(t, app, c.org, "u-"+c.org, c.body); code != c.want {
+				t.Fatalf("POST /v1/sandboxes org=%q body=%s = %d, want %d", c.org, c.body, code, c.want)
+			}
+		})
+	}
+}
+
+// NEGATIVE CONTROL (c), second half: naming the reserved org on the wire does
+// not put a caller in it.
+//
+// The org rides a VALIDATED principal. A client that sends X-Org-Id: admin with
+// nothing to back it is anonymous, and an anonymous caller has no org at all —
+// so the request is refused before a sandbox exists to have a boundary. This is
+// the whole reason runtimeFor reads m.Org: the value it reads has already been
+// through here.
+func TestClaimingTheReservedOrgWithoutAPrincipalIsRefused(t *testing.T) {
+	app := door(t)
+	for _, c := range []struct {
+		name, org, user string
+		want            int
+	}{
+		{"the reserved org, unvalidated", authz.AdminOrg, "", http.StatusForbidden},
+		{"any org, unvalidated", "acme", "", http.StatusForbidden},
+		// The CONTROL. The same request with a principal is accepted as far as
+		// the cluster — 503, because these tests have none — which is what makes
+		// the 403 above a refusal of the CLAIM and not of the route.
+		{"the reserved org, validated", authz.AdminOrg, "u-admin", http.StatusServiceUnavailable},
+		{"another org, validated", "acme", "u-acme", http.StatusServiceUnavailable},
+	} {
+		t.Run(c.name, func(t *testing.T) {
+			if code := ask(t, app, c.org, c.user, `{"class":"exec"}`); code != c.want {
+				t.Fatalf("POST /v1/sandboxes org=%q user=%q = %d, want %d", c.org, c.user, code, c.want)
+			}
+		})
+	}
+}
+
+// The RESUME path returns a row before the derivation runs, so a sandbox
+// created under one boundary outlives a deployment that changed its mind. That
+// is correct for the pod — a running pod's runtime cannot be changed by
+// answering differently — and it cannot cross the trust axis, which is what
+// this proves: the store is opened PER ORG, so the only rows a caller can name
+// are ones its own org created, and a row created for us is not in another
+// org's store to be resumed at all.
+func TestResumeCannotCrossTheTrustBoundary(t *testing.T) {
+	away(t)
+	s, err := New(cloud.Deps{Logger: luxlog.New("test"), DataDir: t.TempDir()})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	ctx := context.Background()
+
+	// A running sandbox of OURS, on the boundary only we may take.
+	store, err := storeFor(s, authz.AdminOrg)
+	if err != nil {
+		t.Fatalf("storeFor: %v", err)
+	}
+	m := Sandbox{ID: "m_ours", Org: authz.AdminOrg, Kind: KindSandbox, Class: "exec",
+		Status: "running", Pod: podName("m_ours"), CreatedAt: time.Now().Unix()}
+	if err := store.Put(ctx, m); err != nil {
+		t.Fatalf("Put: %v", err)
+	}
+
+	// The CONTROL: we can resume it, so the id is real and the row is live.
+	//
+	// `super` is false even though the org IS the reserved one, because these are
+	// two different facts and this test is about the first. The org names WHOSE
+	// store a row lives in; the SuperAdmin attestation says who the caller is. A
+	// test that conflated them would pass for the wrong reason.
+	got, err := Lease(s, ctx, authz.AdminOrg, false, Spec{ID: m.ID})
+	if err != nil || got.ID != m.ID {
+		t.Fatalf("Lease(admin, resume) = %+v, %v — the control did not resume", got, err)
+	}
+
+	// ANOTHER ORG NAMING THE SAME ID gets a sandbox of its own, never ours. It
+	// fails at the cluster (there is none here), which is already past the point
+	// where a resume would have handed over a running pod.
+	if _, err = Lease(s, ctx, "acme", false, Spec{ID: m.ID}); err == nil {
+		t.Fatal("Lease(acme) resumed a sandbox belonging to the reserved org")
+	}
+	if !strings.Contains(err.Error(), "start sandbox") {
+		t.Fatalf("Lease(acme) failed with %v, want the failure of a NEW sandbox's start", err)
+	}
+	out, err := List(s, ctx, "acme", "", "")
+	if err != nil {
+		t.Fatalf("List: %v", err)
+	}
+	for _, row := range out {
+		if row.ID == m.ID {
+			t.Fatalf("acme's store holds %s, which belongs to %s", row.ID, authz.AdminOrg)
+		}
+	}
+}
+
+// A SANDBOX_RUNTIME_CLASS the table has never heard of stops at STARTUP.
+//
+// It used to reach a pod spec: the old derivation short-circuited on a sandbox
+// with no volume and returned the setting unread, so the pod named a class the
+// apiserver did not have and sat Pending with no reason given. Checked here, the
+// answer arrives once, at boot, with the name of the thing that is wrong.
+func TestAnUnknownRuntimeClassStopsAtStartup(t *testing.T) {
+	away(t)
+	for _, c := range []struct {
+		set     string
+		wantErr string
+	}{
+		{"gvisor", ""},
+		{"kata-clh", ""},
+		{"", ""},
+		{"runsc", "is not one we run"},   // the HANDLER's name, not the class's
+		{"gVisor", "is not one we run"},  // case matters — the apiserver's does
+		{"gvisor ", ""},                  // trimmed on the way in, as before
+		{"default", "is not one we run"}, // a plausible guess, and wrong
+	} {
+		t.Setenv("SANDBOX_RUNTIME_CLASS", c.set)
+		r := newRuntime()
+		if c.wantErr == "" {
+			if r.initErr != "" && strings.Contains(r.initErr, "not one we run") {
+				t.Fatalf("SANDBOX_RUNTIME_CLASS=%q refused: %s", c.set, r.initErr)
+			}
+			continue
+		}
+		if !strings.Contains(r.initErr, c.wantErr) {
+			t.Fatalf("SANDBOX_RUNTIME_CLASS=%q gave initErr %q, want %q", c.set, r.initErr, c.wantErr)
+		}
+		// And it fails CLOSED: every call through ready() carries the reason,
+		// rather than the value arriving at a pod spec.
+		if err := r.ready(); err == nil || !strings.Contains(err.Error(), c.wantErr) {
+			t.Fatalf("SANDBOX_RUNTIME_CLASS=%q: ready() = %v, want the reason", c.set, err)
+		}
+	}
+}
+
+// confine is the containment requirement, and it is the reason runc is a table
+// entry rather than a switch. Each case is a topology that has been mistaken for
+// containment.
+func TestConfineRequiresAPoolOfItsOwn(t *testing.T) {
+	pool := func(k, v string) map[string]any { return map[string]any{k: v} }
+	taint := []any{map[string]any{"key": "dedicated", "operator": "Equal",
+		"value": "sandbox", "effect": "NoSchedule"}}
+
+	for _, c := range []struct {
+		name    string
+		classes []*unstructured.Unstructured
+		want    bool
+	}{
+		{"no such class", nil, false},
+		{"a class with no scheduling at all",
+			[]*unstructured.Unstructured{class("runc", nil, nil)}, false},
+		{"a pool with no taint — anything may join it",
+			[]*unstructured.Unstructured{class("runc", pool("pool", "sandbox"), nil)}, false},
+		{"a taint with no pool — it may still land anywhere",
+			[]*unstructured.Unstructured{class("runc", nil, taint)}, false},
+		{"a pool of its own",
+			[]*unstructured.Unstructured{class("runc", pool("pool", "sandbox"), taint)}, true},
+		{"a pool it shares with a boundary other tenants take",
+			[]*unstructured.Unstructured{
+				class("runc", pool("pool", "code-exec"), taint),
+				class("gvisor", pool("pool", "code-exec"), taint),
+			}, false},
+		{"its own pool, beside a boundary on another",
+			[]*unstructured.Unstructured{
+				class("runc", pool("pool", "sandbox"), taint),
+				class("gvisor", pool("pool", "code-exec"), taint),
+			}, true},
+		{"its own pool, beside a boundary pinned nowhere",
+			[]*unstructured.Unstructured{
+				class("runc", pool("pool", "sandbox"), taint),
+				class("kata-clh", nil, nil),
+			}, true},
+	} {
+		t.Run(c.name, func(t *testing.T) {
+			r := &runtime{dyn: fakeClasses(c.classes...)}
+			if got := r.confine(context.Background(), "runc"); got != c.want {
+				t.Fatalf("confine = %v, want %v", got, c.want)
+			}
+		})
+	}
+
+	// NO ANSWER IS A NO. A cluster we cannot read — no client, or an RBAC grant
+	// we do not have — leaves our boundary unoffered rather than assumed.
+	if (&runtime{}).confine(context.Background(), "runc") {
+		t.Fatal("confine said yes with no cluster client")
+	}
+	if (&runtime{dyn: fakeClasses()}).confine(context.Background(), "") {
+		t.Fatal("confine said yes for a boundary with no name")
+	}
+}
+
+// THE CONTAINMENT PREDICATE, asked of the cluster it will actually be asked
+// about. The cases above are fakes and prove the RULE; this proves the rule is
+// reading the same objects an operator does.
+//
+//	SANDBOX_LIVE=1 go test ./apps/sandbox/ -run TestLiveConfine -v
+//
+// gVisor is the CONTROL, and it is what makes a `false` for runc mean something:
+// on hanzo-k8s the gvisor RuntimeClass pins to code-exec-pool with a taint, so
+// it must read true. Every boundary reading false would be a broken read, not an
+// uncontained fleet, and the two are indistinguishable without a control.
+func TestLiveConfineReadsTheRealTopology(t *testing.T) {
+	if os.Getenv("SANDBOX_LIVE") != "1" {
+		t.Skip("set SANDBOX_LIVE=1 to run against a real cluster")
+	}
+	r := newRuntime()
+	if err := r.ready(); err != nil {
+		t.Fatalf("no cluster: %v", err)
+	}
+	ctx := context.Background()
+	for _, name := range sorted() {
+		t.Logf("confine(%-9q) = %v", name, r.confine(ctx, name))
+	}
+	if !r.confine(ctx, shared) {
+		t.Fatalf("confine(%q) is false on a cluster that pins it — the read is broken, "+
+			"so every other answer here means nothing", shared)
+	}
+	t.Logf("our boundary %q is %s", bare(),
+		map[bool]string{true: "contained", false: "NOT contained — nothing will select it"}[r.bare != ""])
+}
+
+// class builds a RuntimeClass as the apiserver stores it.
+func class(name string, sel map[string]any, tol []any) *unstructured.Unstructured {
+	u := &unstructured.Unstructured{Object: map[string]any{
+		"apiVersion": "node.k8s.io/v1",
+		"kind":       "RuntimeClass",
+		"metadata":   map[string]any{"name": name},
+		"handler":    name,
+	}}
+	s := map[string]any{}
+	if len(sel) > 0 {
+		s["nodeSelector"] = sel
+	}
+	if len(tol) > 0 {
+		s["tolerations"] = tol
+	}
+	if len(s) > 0 {
+		u.Object["scheduling"] = s
+	}
+	return u
+}
+
+func fakeClasses(objs ...*unstructured.Unstructured) *dynamicfake.FakeDynamicClient {
+	list := map[schema.GroupVersionResource]string{k8s.RuntimeClasses: "RuntimeClassList"}
+	os := make([]k8sruntime.Object, 0, len(objs))
+	for _, o := range objs {
+		os = append(os, o)
+	}
+	return dynamicfake.NewSimpleDynamicClientWithCustomListKinds(k8sruntime.NewScheme(), list, os...)
+}
+
+// away puts the cluster out of REACH for the length of a test, and it is not
+// tidiness. Every case in this file is decided before the first call to one, so
+// a developer's own kubeconfig makes that claim untestable — and it does worse
+// than that: the first run of these tests created real pods in the real sandbox
+// namespace and sat two minutes each waiting for them. A test that proves a
+// refusal must not be able to succeed.
+func away(t *testing.T) {
+	t.Helper()
+	t.Setenv("KUBECONFIG", filepath.Join(t.TempDir(), "none"))
+}
+
+// door mounts the HTTP surface with no cluster behind it, which is all these
+// need: every case is decided before the first call to one.
+func door(t *testing.T) *zip.App {
+	t.Helper()
+	away(t)
+	app := zip.New(zip.Config{Logger: luxlog.New("test")})
+	app.Use(cloud.Bridge())
+	if err := Mount(app, cloud.Deps{Logger: luxlog.New("test"), DataDir: t.TempDir()}); err != nil {
+		t.Fatalf("Mount: %v", err)
+	}
+	return app
+}
+
+func ask(t *testing.T, app *zip.App, org, user, body string) int {
+	t.Helper()
+	r := httptest.NewRequest(http.MethodPost, "/v1/sandboxes", strings.NewReader(body))
+	r.Header.Set("Content-Type", "application/json")
+	if org != "" {
+		r.Header.Set("X-Org-Id", org)
+	}
+	if user != "" {
+		r.Header.Set("X-User-Id", user)
+	}
+	resp, err := app.Test(r, zip.TestConfig{Timeout: 30 * time.Second})
+	if err != nil {
+		t.Fatalf("POST /v1/sandboxes: %v", err)
+	}
+	defer func() { _ = resp.Body.Close() }()
+	return resp.StatusCode
+}

apps/sandbox/volume_live_test.go

@@ -0,0 +1,317 @@
+package sandbox
+
+// THE DATA-LOSS PROOF, both ways, on a real cluster.
+//
+// kata-fc has no shared filesystem, and the way that fails is the reason this
+// file exists: the write SUCCEEDS. Kubernetes reports a Bound PVC, the mount
+// appears at the right path, `cat` returns what was just written — and the
+// bytes are in a tmpfs inside the VM, which is gone the moment the lease ends.
+// Nothing anywhere returns an error. A unit test cannot see that, because the
+// lie is told by the kernel and not by the code.
+//
+// So the assertion is the only one that catches it: write a file, END THE
+// LEASE, lease the same project again, and read it back. A tmpfs cannot survive
+// that. Run it under each runtime — the runtime is one env var, so the same
+// test is the proof and the negative control:
+//
+//	# the shared-filesystem way: the file MUST come back
+//	SANDBOX_LIVE=1 SANDBOX_RUNTIME_CLASS=gvisor \
+//	  go test ./apps/sandbox/ -run TestLiveLease -v
+//
+//	# the fast way: exec gets Firecracker, and no volume is anywhere near it
+//	SANDBOX_LIVE=1 SANDBOX_RUNTIME_CLASS=kata-fc \
+//	  go test ./apps/sandbox/ -run TestLiveLease -v
+//
+// It drives the PLANE routes — lease_sandbox, run_in_sandbox, write, read,
+// end_sandbox — because those are the addresses an agent actually calls, and a
+// proof that skips them proves the runtime rather than the product.
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/hanzoai/cloud/apps/k8s"
+	"github.com/hanzoai/cloud/plane"
+	"github.com/zap-proto/zip"
+
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+)
+
+// nodes is read for ONE field — the node's kernel version, so the guest kernel
+// below is compared against the machine actually underneath the pod. It lives
+// here rather than in apps/k8s because nothing in production reads a node, and a
+// test is a poor reason to widen the surface the service can reach.
+var nodes = schema.GroupVersionResource{Version: "v1", Resource: "nodes"}
+
+// post drives one plane route and decodes its answer, timing the round trip.
+// The duration is as much the point as the body: these are the numbers the lease
+// path actually costs, measured where a caller pays them rather than on a
+// synthetic pod.
+func post[T any](t *testing.T, app *zip.App, path, org string, in any) (T, time.Duration) {
+	t.Helper()
+	var out T
+	body, err := json.Marshal(in)
+	if err != nil {
+		t.Fatalf("marshal %s: %v", path, err)
+	}
+	start := time.Now()
+	code, b := req(t, app, http.MethodPost, path, org, string(body))
+	d := time.Since(start)
+	if code != http.StatusOK && code != http.StatusCreated {
+		t.Fatalf("POST %s: %d %s", path, code, b)
+	}
+	if len(b) > 0 {
+		_ = json.Unmarshal(b, &out)
+	}
+	return out, d
+}
+
+// release ends a lease on the way out and does not care whether it was already
+// ended. The test ends one lease DELIBERATELY — that is the whole experiment —
+// so a deferred cleanup that insisted on 200 would fail every successful run.
+func release(app *zip.App, org, id string, purge bool) {
+	body, _ := json.Marshal(plane.EndIn{ID: id, Purge: purge})
+	r := httptest.NewRequest(http.MethodPost, "/v1/sandboxes/end", strings.NewReader(string(body)))
+	r.Header.Set("Content-Type", "application/json")
+	r.Header.Set("X-Org-Id", org)
+	r.Header.Set("X-User-Id", "u-"+org)
+	if resp, err := app.Test(r, zip.TestConfig{Timeout: 120 * time.Second}); err == nil {
+		_ = resp.Body.Close()
+	}
+}
+
+func TestLiveLeaseKeepsWhatItPromisesToKeep(t *testing.T) {
+	if os.Getenv("SANDBOX_LIVE") != "1" {
+		t.Skip("set SANDBOX_LIVE=1 to run against a real cluster")
+	}
+	app := mountHTTP(t)
+	rt := newRuntime()
+	if err := rt.ready(); err != nil {
+		t.Fatalf("no cluster: %v", err)
+	}
+	const org = "hanzo"
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
+	defer cancel()
+
+	project := fmt.Sprintf("fcproof%d", time.Now().Unix())
+	want := fmt.Sprintf("survived-%d", time.Now().UnixNano())
+	t.Logf("deployment runtime = %q (SANDBOX_RUNTIME_CLASS)", rt.runtimeClass)
+
+	// ---- THE VOLUME-BEARING WAY --------------------------------------------
+	// A project gives the sandbox a volume, and the volume is what decides the
+	// runtime. This half must survive the lease.
+	first, dLease := post[plane.Leased](t, app, "/v1/sandboxes/lease", org,
+		plane.LeaseIn{Class: "dev", Project: project, TTLSec: 900})
+	if first.ID == "" {
+		t.Fatal("lease returned no id")
+	}
+	t.Logf("LEASE dev/%s -> %s in %v", project, first.ID, dLease.Round(time.Millisecond))
+	// purge: this project exists only for the test, so its disk goes too.
+	defer release(app, org, first.ID, true)
+
+	// The runtime it LANDED on, read off the pod rather than inferred from the
+	// env — the pod is what the kubelet obeyed.
+	got := runtimeOfPod(t, ctx, rt, first.ID)
+	if !runtimes[got].shares {
+		t.Fatalf("a sandbox mounting a volume landed on %q, which has no shared filesystem — "+
+			"this is the silent-tmpfs case the derivation exists to prevent", got)
+	}
+	t.Logf("  runtimeClassName=%q  (shares a filesystem: yes)", got)
+
+	// The PVC has to be real and Bound, or "it survived" would only mean the
+	// pod never restarted.
+	if phase := volumePhase(t, ctx, rt, volumeName(org, project)); phase != "Bound" {
+		t.Fatalf("project volume %s is %q, want Bound", volumeName(org, project), phase)
+	}
+
+	_, dWrite := post[plane.Wrote](t, app, "/v1/sandboxes/write", org,
+		plane.WriteIn{ID: first.ID, Path: "keep.txt", Data: []byte(want)})
+	// Written THROUGH the sandbox's own filesystem, so the check below is not
+	// reading back the same buffer it just sent.
+	ran, dRun := post[plane.Ran](t, app, "/v1/sandboxes/run", org,
+		plane.RunIn{ID: first.ID, Command: "sync; cat keep.txt; df -T " + first.Workdir + " | tail -1"})
+	t.Logf("  write %v · run %v", dWrite.Round(time.Millisecond), dRun.Round(time.Millisecond))
+	t.Logf("  in-sandbox view: %s", oneLine(ran.Stdout))
+
+	// THE MOMENT THAT MATTERS. End the lease: the pod goes, the volume stays.
+	_, dEnd := post[struct{}](t, app, "/v1/sandboxes/end", org, plane.EndIn{ID: first.ID})
+	t.Logf("END %s in %v", first.ID, dEnd.Round(time.Millisecond))
+
+	// Same project, new lease. If the write went into a tmpfs, this read fails.
+	second, dRelease := post[plane.Leased](t, app, "/v1/sandboxes/lease", org,
+		plane.LeaseIn{Class: "dev", Project: project, TTLSec: 900})
+	t.Logf("RE-LEASE %s -> %s in %v", project, second.ID, dRelease.Round(time.Millisecond))
+	defer release(app, org, second.ID, true)
+	if second.ID == first.ID {
+		t.Fatalf("re-lease returned the SAME sandbox %s — the lease never ended, so nothing was proven", second.ID)
+	}
+	blob, dRead := post[plane.Blob](t, app, "/v1/sandboxes/read", org,
+		plane.PathIn{ID: second.ID, Path: "keep.txt"})
+	if string(blob.Data) != want {
+		t.Fatalf("DATA LOSS: wrote %q before the lease ended, read %q after it — "+
+			"the volume did not survive, which means the sandbox ran on a runtime "+
+			"that cannot share a filesystem", want, blob.Data)
+	}
+	t.Logf("  read %v — SURVIVED: %q", dRead.Round(time.Millisecond), blob.Data)
+
+	// ---- THE VOLUMELESS WAY ------------------------------------------------
+	// No project, so no volume, so nothing to lose — and therefore free to take
+	// the fast runtime.
+	ex, dExLease := post[plane.Leased](t, app, "/v1/sandboxes/lease", org,
+		plane.LeaseIn{Class: "exec", TTLSec: 600})
+	t.Logf("LEASE exec (no project) -> %s in %v", ex.ID, dExLease.Round(time.Millisecond))
+	defer release(app, org, ex.ID, false)
+
+	// THERE MUST BE NO PVC AT ALL. Not an empty one, not an unbound one — none,
+	// so there is nothing for a runtime without a shared filesystem to drop.
+	if phase := volumePhase(t, ctx, rt, volumeName(org, "")); phase != "" {
+		t.Fatalf("a volumeless sandbox has a PVC (%q) — it should have none", phase)
+	}
+	exRC := runtimeOfPod(t, ctx, rt, ex.ID)
+	t.Logf("  runtimeClassName=%q", exRC)
+	if exRC != rt.runtimeClass {
+		t.Fatalf("volumeless exec landed on %q, but the deployment states %q — "+
+			"a sandbox with nothing to lose should take the deployment's runtime unchanged",
+			exRC, rt.runtimeClass)
+	}
+
+	// THE KERNEL IS THE CONTROL. A runtimeClassName is a label; a different
+	// kernel version from the node's is the VM actually existing. Compared
+	// against the node this pod is on, read from the node object.
+	kern, dKern := post[plane.Ran](t, app, "/v1/sandboxes/run", org,
+		plane.RunIn{ID: ex.ID, Command: "uname -r"})
+	guest, host := oneLine(kern.Stdout), hostKernel(t, ctx, rt, ex.ID)
+	t.Logf("  guest kernel %s vs host %s (run %v)", guest, host, dKern.Round(time.Millisecond))
+	// The table's `kernel` column, checked against the only thing that can
+	// settle it. A boundary that claims a kernel of its own and reports the
+	// node's has not got one — the runtimeClassName was accepted and nothing
+	// behind it was installed. And a boundary that claims none must report the
+	// node's, or the table is describing a runtime we are not running.
+	if b, ok := runtimes[exRC]; ok {
+		if b.kernel && guest == host {
+			t.Fatalf("runtimeClassName=%q says it has a kernel of its own, and the guest kernel %s "+
+				"equals the host's — the pod did not get one, so the label is not the boundary it ran on",
+				exRC, guest)
+		}
+		if !b.kernel && guest != host {
+			t.Fatalf("runtimeClassName=%q says it IS the node's kernel, and the guest reports %s "+
+				"against the host's %s", exRC, guest, host)
+		}
+	}
+}
+
+// runtimeOfPod reads runtimeClassName off the running pod. The env said what we
+// asked for; this says what the kubelet did.
+func runtimeOfPod(t *testing.T, ctx context.Context, r *runtime, id string) string {
+	t.Helper()
+	u, err := r.pods().Get(ctx, podName(id), metav1.GetOptions{})
+	if err != nil {
+		t.Fatalf("get pod for %s: %v", id, err)
+	}
+	rc, _, _ := unstructured.NestedString(u.Object, "spec", "runtimeClassName")
+	return rc
+}
+
+// hostKernel reads the kernel of the NODE the sandbox landed on, so the guest
+// comparison is against the machine underneath it and not a fleet average.
+func hostKernel(t *testing.T, ctx context.Context, r *runtime, id string) string {
+	t.Helper()
+	u, err := r.pods().Get(ctx, podName(id), metav1.GetOptions{})
+	if err != nil {
+		t.Fatalf("get pod: %v", err)
+	}
+	node, _, _ := unstructured.NestedString(u.Object, "spec", "nodeName")
+	n, err := r.dyn.Resource(nodes).Get(ctx, node, metav1.GetOptions{})
+	if err != nil {
+		t.Fatalf("get node %s: %v", node, err)
+	}
+	k, _, _ := unstructured.NestedString(n.Object, "status", "nodeInfo", "kernelVersion")
+	return k
+}
+
+// volumePhase answers "" when the claim does not exist, which is the assertion a
+// volumeless sandbox needs — absence, not an empty value.
+func volumePhase(t *testing.T, ctx context.Context, r *runtime, name string) string {
+	t.Helper()
+	u, err := r.dyn.Resource(k8s.Volumes).Namespace(r.ns).Get(ctx, name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		return ""
+	}
+	if err != nil {
+		t.Fatalf("get pvc %s: %v", name, err)
+	}
+	p, _, _ := unstructured.NestedString(u.Object, "status", "phase")
+	return p
+}
+
+// THE COST OF THE LEASE PATH, per runtime, measured where a caller pays it.
+//
+// The microbenchmarks that made Firecracker look free measured the wrong thing.
+// They timed filesystem work INSIDE an already-running sandbox — git status,
+// copy a tree — and on that axis a guest kernel owning a block device beats
+// proxying every lstat to the host by an order of magnitude. All true, and all
+// irrelevant to a caller who pays for the pod to exist first.
+//
+// So this measures both halves separately, because they point in opposite
+// directions: how long a lease takes to answer, and how fast the sandbox is once
+// it does. Run it under each runtime and read the two columns together.
+//
+//	SANDBOX_LIVE=1 SANDBOX_RUNTIME_CLASS=kata-fc \
+//	  go test ./apps/sandbox/ -run TestLiveLeaseCost -v -timeout 20m
+func TestLiveLeaseCost(t *testing.T) {
+	if os.Getenv("SANDBOX_LIVE") != "1" {
+		t.Skip("set SANDBOX_LIVE=1 to run against a real cluster")
+	}
+	app := mountHTTP(t)
+	rt := newRuntime()
+	if err := rt.ready(); err != nil {
+		t.Fatalf("no cluster: %v", err)
+	}
+	const org, rounds = "hanzo", 3
+
+	// The workload is timed INSIDE the sandbox, so the number is the
+	// filesystem's and not the exec channel's. git status over a few hundred
+	// files is the shape of work a coding sandbox actually does.
+	const work = `cd /mnt/data && rm -rf b && mkdir b && cd b && ` +
+		`i=0; while [ $i -lt 300 ]; do echo x > f$i; i=$((i+1)); done && ` +
+		`git init -q . && git add -A && ` +
+		`s=$(date +%s%N) && git status --porcelain >/dev/null && e=$(date +%s%N) && ` +
+		`echo "git_status_ms=$(( (e-s)/1000000 ))"`
+
+	t.Logf("runtime=%q  rounds=%d", rt.runtimeClass, rounds)
+	for i := 0; i < rounds; i++ {
+		m, dLease := post[plane.Leased](t, app, "/v1/sandboxes/lease", org,
+			plane.LeaseIn{Class: "exec", TTLSec: 600})
+		ran, dRun := post[plane.Ran](t, app, "/v1/sandboxes/run", org,
+			plane.RunIn{ID: m.ID, Command: work, TimeoutSec: 300})
+		_, dEnd := post[struct{}](t, app, "/v1/sandboxes/end", org, plane.EndIn{ID: m.ID})
+		t.Logf("  round %d on %s: lease %v · run %v · end %v · %s",
+			i, runtimeOfPodOrGone(ctx0(), rt, m.ID),
+			dLease.Round(time.Millisecond), dRun.Round(time.Millisecond),
+			dEnd.Round(time.Millisecond), oneLine(ran.Stdout))
+	}
+}
+
+func ctx0() context.Context { return context.Background() }
+
+// runtimeOfPodOrGone reports the runtime the pod ran on, or why it cannot say.
+// The pod is deleted by the time the round is logged, so this is best effort —
+// the per-round runtime is confirmed by the test above, not by this line.
+func runtimeOfPodOrGone(ctx context.Context, r *runtime, id string) string {
+	u, err := r.pods().Get(ctx, podName(id), metav1.GetOptions{})
+	if err != nil {
+		return r.runtimeClass + "(gone)"
+	}
+	rc, _, _ := unstructured.NestedString(u.Object, "spec", "runtimeClassName")
+	return rc
+}

apps/sandbox/volume_test.go

@@ -0,0 +1,186 @@
+package sandbox
+
+// The lifetime of a project disk.
+//
+// A disk is the one thing this service makes that OUTLIVES everything that made
+// it: the pod is gone in an hour, the row is gone with the lease, and the 20Gi
+// stays until somebody deletes it on purpose. So the two properties below are
+// not "does the code work" — they are the two halves of the only decision an
+// operator staring at a namespace full of disks has to get right, and getting it
+// wrong in either direction is expensive:
+//
+//	KEEP  a disk still in use is a tenant's checkout and their uncommitted work.
+//	      Deleting one is unrecoverable, and no lease ending may ever imply it.
+//	DROP  a disk nobody will ever ask for again bills forever. It cannot be
+//	      identified — and therefore cannot be dropped — unless it SAYS what it
+//	      is for and when it was last wanted.
+//
+// Measured 2026-08-07 on the live namespace: 15 disks, 300GiB, and the project
+// each one belonged to was recoverable ONLY by brute-forcing sha256 over guessed
+// names, because the name is a hash and the object carried nothing else. That is
+// what makes reclaim undecidable in practice — not the absence of a policy, the
+// absence of the fact a policy would read.
+
+import (
+	"context"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/hanzoai/cloud/apps/k8s"
+
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	k8sruntime "k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+	dynamicfake "k8s.io/client-go/dynamic/fake"
+)
+
+// fakeRuntime is a runtime whose apiserver is in memory. Only the disk paths are
+// exercised, so nothing here needs a pod, a stream, or a cluster.
+func fakeRuntime(objs ...k8sruntime.Object) *runtime {
+	return &runtime{
+		ns: "hanzo-sandboxes",
+		dyn: dynamicfake.NewSimpleDynamicClientWithCustomListKinds(
+			k8sruntime.NewScheme(),
+			map[schema.GroupVersionResource]string{
+				k8s.Volumes: "PersistentVolumeClaimList",
+				k8s.Pods:    "PodList",
+			}, objs...),
+		bound: Bound{Namespace: "hanzo-sandboxes", Selector: labSandbox},
+	}
+}
+
+// disks is every PVC the fake cluster holds, by name.
+func disks(t *testing.T, r *runtime) map[string]*unstructured.Unstructured {
+	t.Helper()
+	list, err := r.dyn.Resource(k8s.Volumes).Namespace(r.ns).List(context.Background(), metav1.ListOptions{})
+	if err != nil {
+		t.Fatalf("list disks: %v", err)
+	}
+	out := map[string]*unstructured.Unstructured{}
+	for i := range list.Items {
+		out[list.Items[i].GetName()] = &list.Items[i]
+	}
+	return out
+}
+
+// A LEASE ENDING MUST NEVER COST A TENANT THEIR DISK, and a second lease for the
+// same project must find the FIRST disk rather than mint another.
+//
+// Both directions are asserted together because they are the same property seen
+// from either side: the name is a pure function of (org, project), so reuse and
+// survival are the same fact. If this ever fails as "two disks", the service is
+// leaking 20Gi per lease; if it fails as "no disk", it has just destroyed a
+// checkout. Neither is visible in a request — both are only visible here.
+func TestProjectDiskOutlivesItsLeaseAndIsReusedNotRemade(t *testing.T) {
+	r := fakeRuntime()
+	ctx := context.Background()
+	const org, project = "acme", "checkout"
+
+	lease := func(id string) Sandbox {
+		m := Sandbox{ID: id, Org: org, Project: project, Class: "dev", Pod: podName(id)}
+		m.Volume = volumeName(org, project)
+		if err := r.ensureVolume(ctx, m); err != nil {
+			t.Fatalf("ensureVolume(%s): %v", id, err)
+		}
+		return m
+	}
+
+	first := lease("m_1")
+	if got := disks(t, r); len(got) != 1 || got[first.Volume] == nil {
+		t.Fatalf("first lease made %d disk(s) %v, want exactly %q", len(got), keys(got), first.Volume)
+	}
+
+	// The lease ends the way every lease ends — the reaper's `end`, which stops the
+	// pod and keeps the disk. `stop` is what it calls; purge is a different verb the
+	// caller has to ask for.
+	if err := r.stop(ctx, first); err != nil {
+		t.Fatalf("stop: %v", err)
+	}
+	if got := disks(t, r); got[first.Volume] == nil {
+		t.Fatalf("ending a lease destroyed the project disk %q — that is a tenant's uncommitted work", first.Volume)
+	}
+
+	// A SECOND lease, a new sandbox id, the same project. One disk, still.
+	second := lease("m_2")
+	if second.Volume != first.Volume {
+		t.Fatalf("second lease addressed %q, want the same disk %q", second.Volume, first.Volume)
+	}
+	if got := disks(t, r); len(got) != 1 {
+		t.Fatalf("two leases on one project left %d disks %v, want 1 — this is the 20Gi-per-lease leak",
+			len(got), keys(got))
+	}
+
+	// And purge, which is the ONLY thing that may take it, still does.
+	if err := r.purge(ctx, second); err != nil {
+		t.Fatalf("purge: %v", err)
+	}
+	if got := disks(t, r); len(got) != 0 {
+		t.Fatalf("purge left %v", keys(got))
+	}
+}
+
+// A DISK SAYS WHAT IT IS FOR AND WHEN IT WAS LAST WANTED.
+//
+// Without this a disk is an opaque 20Gi charge whose owner is a hash: the org
+// label narrows it to a tenant and nothing narrows it further, so the only
+// honest answer to "can this be deleted" is "no". Reclaim is not blocked by the
+// absence of a policy — it is blocked by the absence of the two facts any policy
+// would have to read. They are written where the disk is already being ensured,
+// so a disk cannot come into existence unlabelled.
+func TestProjectDiskSaysWhoseItIsAndWhenItWasLastLeased(t *testing.T) {
+	r := fakeRuntime()
+	ctx := context.Background()
+	const org, project = "acme", "Deep Research"
+
+	m := Sandbox{ID: "m_1", Org: org, Project: slug(project), Class: "dev"}
+	m.Volume = volumeName(org, slug(project))
+	if err := r.ensureVolume(ctx, m); err != nil {
+		t.Fatalf("ensureVolume: %v", err)
+	}
+
+	d := disks(t, r)[m.Volume]
+	if d == nil {
+		t.Fatalf("no disk %q", m.Volume)
+	}
+	if got := d.GetLabels()[labOrg]; got != "acme" {
+		t.Fatalf("%s = %q, want %q", labOrg, got, "acme")
+	}
+	// The project, PLAINLY. The name carries it only as a sha256 tail, so an
+	// operator holding a namespace of disks cannot get back to a project without
+	// guessing the string that made it.
+	if got := d.GetLabels()[labProject]; got != "deep-research" {
+		t.Fatalf("%s = %q, want %q — the project is otherwise recoverable only by brute force",
+			labProject, got, "deep-research")
+	}
+	today := time.Now().UTC().Format(time.DateOnly)
+	if got := d.GetAnnotations()[annLeased]; got != today {
+		t.Fatalf("%s = %q, want %q", annLeased, got, today)
+	}
+
+	// A LATER LEASE REFRESHES IT. A stamp written once at creation dates the disk
+	// and not its use, which reads a project worked on daily for a year as a year
+	// old — exactly backwards, and it is the reading a reclaim would act on.
+	stale := d.DeepCopy()
+	stale.SetAnnotations(map[string]string{annLeased: "2026-01-01"})
+	if _, err := r.dyn.Resource(k8s.Volumes).Namespace(r.ns).
+		Update(ctx, stale, metav1.UpdateOptions{}); err != nil {
+		t.Fatalf("age the disk: %v", err)
+	}
+	if err := r.ensureVolume(ctx, Sandbox{ID: "m_2", Org: org, Project: slug(project), Volume: m.Volume}); err != nil {
+		t.Fatalf("second ensureVolume: %v", err)
+	}
+	if got := disks(t, r)[m.Volume].GetAnnotations()[annLeased]; got != today {
+		t.Fatalf("%s = %q after a second lease, want %q — a disk in daily use must not look abandoned",
+			annLeased, got, today)
+	}
+}
+
+func keys(m map[string]*unstructured.Unstructured) string {
+	out := make([]string, 0, len(m))
+	for k := range m {
+		out = append(out, k)
+	}
+	return strings.Join(out, ", ")
+}

apps/sandbox/work.go

@@ -0,0 +1,249 @@
+package sandbox
+
+// work.go — what a sandbox is DOING, while it is still doing it.
+//
+// A command used to be a box with two ends: post an argv, and some minutes later
+// receive everything the program had said. For a twenty-five minute agentic run
+// that is a blank screen with a verdict at the end, and nothing anywhere could
+// interrupt it. Two things fix that, and they are one fact seen twice — a command
+// in flight is ADDRESSABLE:
+//
+//	tell   its output is appended to the session watching it AS IT IS PRODUCED,
+//	       so every surface reading that session watches the work happen
+//	work   its cancel is held under the sandbox's id, so a caller can stop it
+//
+// NARRATION GOES WHERE EVERY OTHER RUN'S NARRATION ALREADY GOES. apps/agents owns
+// the fleet's live run feed — one durable ordered event log per session, fanned
+// out to GET /v1/agents/sessions/stream — and that is the ONE place a surface
+// watches a run. This appends to it over the internal plane. It invents no second
+// feed, and it does not touch POST /v1/event, which warehouses product events for
+// the webhooks engine and has no live tail to read at all.
+//
+// THE ORG IS THE ONE THE PLANE PROVED. A caller names a session and never a
+// tenant, so a command can only ever narrate into its own org's session: a
+// session id belonging to somebody else is simply not present in the org this
+// call acts for, and the append is refused on that side.
+
+import (
+	"context"
+	"encoding/json"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/hanzoai/cloud/plane"
+)
+
+// tellEvery is the shortest gap between two appends.
+//
+// A session's log is a durable ordered record, not a byte pipe, and a build
+// prints thousands of lines: without a floor, one `npm install` would write a row
+// per line and every watcher would spend its whole budget rendering a scroll
+// nobody reads. Coalesced, a reader sees the latest work about once a second,
+// which is as often as a person can take it in.
+const tellEvery = time.Second
+
+// tellCap bounds ONE appended chunk, and keeps the TAIL rather than the head. The
+// far side refuses a payload over 64 KiB whole, so a burst has to be cut here —
+// and what a program said most recently is what says why it is stuck.
+const tellCap = 8 << 10
+
+// tellPatience bounds one append. This call sits ON the path the pod's output
+// takes, so a wedged reader must cost the command a moment and never its life.
+const tellPatience = 5 * time.Second
+
+// line is one narration event's body, in the vocabulary a session's log already
+// speaks: `step` names a phase, `message` carries what it said. It is exactly
+// what apps/coding mirrors for a coding run, so a reader that renders one renders
+// this with no new case to learn.
+type line struct {
+	Step    string `json:"step,omitempty"`
+	Message string `json:"message,omitempty"`
+}
+
+// tell appends a command's output to the session watching it.
+//
+// It is an io.Writer so the exec channel writes THROUGH it: the same bytes that
+// fill the result buffer pass here on the way, and nothing is read twice. Both of
+// a command's streams share one tell, which is why it locks — the exec channel
+// writes stdout and stderr from different goroutines.
+type tell struct {
+	org, session string
+
+	mu   sync.Mutex
+	buf  []byte
+	at   time.Time
+	dead bool
+}
+
+// newTell returns the sink for one command, or nil when no session was named. A
+// nil *tell's methods are no-ops, so no caller has to branch on being watched.
+func newTell(org, session string) *tell {
+	if strings.TrimSpace(org) == "" || strings.TrimSpace(session) == "" {
+		return nil
+	}
+	return &tell{org: org, session: session}
+}
+
+func (t *tell) Write(p []byte) (int, error) {
+	if t == nil {
+		return len(p), nil
+	}
+	t.mu.Lock()
+	t.buf = append(t.buf, p...)
+	chunk := t.take(time.Now(), false)
+	t.mu.Unlock()
+	if chunk != "" {
+		t.say("log", line{Message: chunk})
+	}
+	// The bytes are always CONSUMED. A short write aborts the exec stream, and a
+	// stalled narration must never be able to kill the command it is narrating.
+	return len(p), nil
+}
+
+// done flushes what is left and says how the command ended.
+//
+// It is the last thing a watcher hears, and it is said even when the command was
+// stopped — a run that vanishes mid-sentence is the failure this file exists to
+// remove. The exit event carries NO status, deliberately: `done` and `error` are
+// the two words that end a watcher's progress line, and a command exiting is not
+// the run ending.
+func (t *tell) done(code int, err error) {
+	if t == nil {
+		return
+	}
+	t.mu.Lock()
+	rest := t.take(time.Now(), true)
+	t.mu.Unlock()
+	if rest != "" {
+		t.say("log", line{Message: rest})
+	}
+	t.say("tool-call", line{Step: "exit", Message: ending(code, err)})
+}
+
+// take answers the bytes to send now, or "" when it is not yet their turn. force
+// sends whatever is left whatever the clock says — the end of a command must not
+// lose its last words to a rate limit. Called under the lock.
+func (t *tell) take(now time.Time, force bool) string {
+	if t.dead || len(t.buf) == 0 {
+		return ""
+	}
+	if !force && now.Sub(t.at) < tellEvery {
+		return ""
+	}
+	b := t.buf
+	if len(b) > tellCap {
+		b = b[len(b)-tellCap:]
+	}
+	s := string(b)
+	t.buf, t.at = t.buf[:0], now
+	return s
+}
+
+// say appends one event to the session, best-effort.
+//
+// A narration failure must never fail the command it narrates — the work is real
+// and the commentary is not — so a failed append RETIRES this tell and the
+// command runs on in silence rather than paying the timeout again per chunk.
+func (t *tell) say(kind string, l line) {
+	t.mu.Lock()
+	dead := t.dead
+	t.mu.Unlock()
+	if dead {
+		return
+	}
+	payload, err := json.Marshal(l)
+	if err != nil {
+		return
+	}
+	// A DETACHED, TENANT-STATED context. The command's own may already be
+	// cancelled — a stop is exactly that case — and the last thing a stopped run
+	// says is the part a watcher most needs. plane.For supplies the org where
+	// there is no request behind the context, which a fresh Background is.
+	ctx, cancel := context.WithTimeout(plane.For(context.Background(), t.org), tellPatience)
+	defer cancel()
+	if _, err := plane.Ask[plane.SessionEventIn, plane.CodingAck](ctx, "agents", plane.AgentsSessionEvent,
+		&plane.SessionEventIn{Org: t.org, SessionID: t.session, Kind: kind, Payload: payload}); err != nil {
+		t.mu.Lock()
+		t.dead = true
+		t.mu.Unlock()
+	}
+}
+
+// ending is how a command's last line reads. A transport failure is named as
+// itself rather than folded into an exit code, because "the program returned 1"
+// and "we lost the channel" are different facts and a watcher acts on them
+// differently.
+func ending(code int, err error) string {
+	if err != nil {
+		return "ended: " + err.Error()
+	}
+	return "exit " + strconv.Itoa(code)
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// The interrupt
+// ─────────────────────────────────────────────────────────────────────────────
+
+// work is every command this process has in flight, held by the sandbox it runs
+// in, so a caller can reach one and stop it.
+//
+// IT IS PROCESS-LOCAL, and that is a bound with a stated cost rather than an
+// oversight. A cancel belongs to a live goroutine, so it exists only where the
+// exec stream does; cloud-api runs one replica (universe:
+// charts/app/values/hanzo/cloud.yaml, replicas: 1), so today a stop always
+// reaches the process holding the command. The day that number changes, a stop
+// that lands elsewhere honestly reports interrupting nothing — it never claims a
+// kill it did not perform, which is the property worth having when an assumption
+// stops holding.
+type work struct {
+	mu   sync.Mutex
+	next int
+	in   map[string]map[int]context.CancelFunc
+}
+
+func newWork() *work { return &work{in: map[string]map[int]context.CancelFunc{}} }
+
+// start records one command's cancel under its sandbox and answers the func that
+// forgets it. Every start is paired with that func or the set grows a dead cancel
+// per command — which is why the caller defers it beside the cancel itself.
+func (w *work) start(sandbox string, stop context.CancelFunc) func() {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	w.next++
+	id := w.next
+	if w.in[sandbox] == nil {
+		w.in[sandbox] = map[int]context.CancelFunc{}
+	}
+	w.in[sandbox][id] = stop
+	return func() {
+		w.mu.Lock()
+		defer w.mu.Unlock()
+		delete(w.in[sandbox], id)
+		if len(w.in[sandbox]) == 0 {
+			delete(w.in, sandbox)
+		}
+	}
+}
+
+// stop cancels every command running in one sandbox and answers how many it
+// interrupted. Cancelling ends the exec stream, and the kubelet kills the process
+// on the far side of a closed channel — the same thing that happens when a person
+// interrupts `kubectl exec`.
+//
+// The cancels run OUTSIDE the lock: each unwinds a command, whose own deferred
+// work reaches back here to forget it, and a stop holding the lock while that
+// happened would wait on itself.
+func (w *work) stop(sandbox string) int {
+	w.mu.Lock()
+	stops := make([]context.CancelFunc, 0, len(w.in[sandbox]))
+	for _, c := range w.in[sandbox] {
+		stops = append(stops, c)
+	}
+	w.mu.Unlock()
+	for _, c := range stops {
+		c()
+	}
+	return len(stops)
+}

apps/sandbox/work_test.go

@@ -0,0 +1,407 @@
+package sandbox
+
+// The two properties a live run has to hold, measured rather than argued for:
+// its output leaves the sandbox WHILE the command is still running, and a stop
+// reaches exactly one tenant's command and no other's.
+//
+// None of it needs a cluster. The exec channel is an interface for exactly this
+// reason (runtime.go: "the real one opens an SPDY stream to the apiserver, and a
+// test has no apiserver"), so a fake streamer can hold a command open for as long
+// as a test wants to look at it.
+
+import (
+	"context"
+	"errors"
+	"io"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/hanzoai/cloud"
+	luxlog "github.com/luxfi/log"
+	"k8s.io/client-go/dynamic"
+	"k8s.io/client-go/rest"
+	"k8s.io/client-go/tools/remotecommand"
+)
+
+// held is a command that has started and will not finish until the test says so.
+// It writes what it was given, announces that it is running, and then waits.
+type held struct {
+	out     string
+	running chan struct{} // closed once the bytes are written
+	release chan struct{} // the test closes this to let the command end
+	err     error         // what the command ends with
+}
+
+func (h *held) stream(ctx context.Context, ns, pod string, argv []string, stdin io.Reader, stdout, stderr io.Writer) error {
+	if h.out != "" {
+		_, _ = io.WriteString(stdout, h.out)
+	}
+	close(h.running)
+	select {
+	case <-h.release:
+		return h.err
+	case <-ctx.Done():
+		// A CANCELLED command is how a stop looks from in here: the channel to the
+		// pod closes and the far side is killed with it, exactly as it is when a
+		// person interrupts `kubectl exec`.
+		return ctx.Err()
+	}
+}
+
+func (h *held) tty(ctx context.Context, ns, pod string, argv []string, stdin io.Reader, stdout io.Writer, size remotecommand.TerminalSizeQueue) error {
+	return errors.New("not a terminal test")
+}
+
+// service builds the subsystem over a real per-test store, with a runtime whose
+// only cluster-facing part is the fake exec channel. Lease is not reachable this
+// way and does not need to be — every property below is about a sandbox that is
+// already running.
+func service(t *testing.T, str streamer) *Service {
+	t.Helper()
+	s, err := New(cloud.Deps{Logger: luxlog.NewNoOpLogger(), DataDir: t.TempDir()})
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	s.State.rt.str = str
+	s.State.rt.dyn = nil // never reached: exec goes through str, which is fake
+	s.State.rt.initErr = ""
+	s.State.rt.execTimeout = 30 * time.Second
+	return s
+}
+
+// seed writes a running sandbox into one org's store, the way a lease would have.
+func seed(t *testing.T, s *Service, org, id string) Sandbox {
+	t.Helper()
+	store, err := storeFor(s, org)
+	if err != nil {
+		t.Fatalf("storeFor(%s): %v", org, err)
+	}
+	m := Sandbox{ID: id, Org: org, Kind: KindSandbox, Class: "exec", Status: "running",
+		Pod: podName(id), CreatedAt: 1, LastUsedAt: 1, ExpiresAt: time.Now().Unix() + 3600}
+	if err := store.Put(context.Background(), m); err != nil {
+		t.Fatalf("seed: %v", err)
+	}
+	return m
+}
+
+// ready satisfies the runtime's "is there a client at all" check without a
+// cluster. The client is BUILT and never dialled — exec's bytes go through the
+// fake exec channel above, and a dynamic client constructed from a config
+// connects to nothing until somebody asks it for an object.
+func ready(t *testing.T, s *Service) {
+	t.Helper()
+	dyn, err := dynamic.NewForConfig(&rest.Config{Host: "http://127.0.0.1:1"})
+	if err != nil {
+		t.Fatalf("dynamic client: %v", err)
+	}
+	s.State.rt.dyn = dyn
+}
+
+// TestOutputLeavesTheSandboxWhileTheCommandIsStillRunning is the whole point.
+//
+// Before this, a command was a box with two ends: post an argv and, up to
+// twenty-five minutes later, receive everything the program had said. What a
+// person watching a coding run saw was "running the task", then nothing, then a
+// verdict — and there was no way to tell a working agent from a wedged one.
+//
+// The measurement is the timing and not the content: the bytes have to be
+// readable from OUTSIDE the call while the call has not returned.
+func TestOutputLeavesTheSandboxWhileTheCommandIsStillRunning(t *testing.T) {
+	h := &held{out: "installing dependencies…\n", running: make(chan struct{}), release: make(chan struct{})}
+	s := service(t, h)
+	ready(t, s)
+	seed(t, s, "acme", "m_1")
+
+	// The chunk is HELD IN THE BUFFER rather than sent, by stating that an append
+	// just happened. That measures the plumbing — do the command's bytes reach the
+	// narration at all, mid-command — with no peer to send them to.
+	say := newTell("acme", "sess_1")
+	say.at = time.Now()
+
+	done := make(chan ExecResult, 1)
+	go func() {
+		r, _ := s.State.rt.exec(context.Background(), Sandbox{ID: "m_1", Status: "running", Pod: "m-1"},
+			[]string{"npm", "install"}, nil, 60, say)
+		done <- r
+	}()
+
+	<-h.running
+	// The command has NOT returned. Whatever the tell holds now, it learned while
+	// the program was still running.
+	deadline := time.After(2 * time.Second)
+	for {
+		say.mu.Lock()
+		got := string(say.buf)
+		say.mu.Unlock()
+		if strings.Contains(got, "installing dependencies") {
+			break
+		}
+		select {
+		case r := <-done:
+			t.Fatalf("the command finished before its output was readable (%q) — output that "+
+				"only arrives with the result is not a live run", r.Stdout)
+		case <-deadline:
+			t.Fatalf("the command has been running for two seconds and nothing has left the "+
+				"sandbox; a watcher would be looking at a blank screen")
+		case <-time.After(5 * time.Millisecond):
+		}
+	}
+
+	close(h.release)
+	r := <-done
+	// And the tap is a PASS-THROUGH: the result still carries everything, so
+	// narrating a command did not change what running it answers.
+	if !strings.Contains(r.Stdout, "installing dependencies") {
+		t.Errorf("stdout = %q; the tap must not consume the bytes it forwards", r.Stdout)
+	}
+}
+
+// TestStopEndsTheCommandAndKeepsTheSandbox. Stop and End are two verbs because a
+// run that has gone wrong is one somebody still wants to look at: the checkout,
+// the logs and the half-written file are all in the sandbox, and a stop that
+// deleted the pod would take the evidence with it.
+func TestStopEndsTheCommandAndKeepsTheSandbox(t *testing.T) {
+	h := &held{running: make(chan struct{}), release: make(chan struct{})}
+	s := service(t, h)
+	ready(t, s)
+	seed(t, s, "acme", "m_1")
+
+	ran := make(chan error, 1)
+	go func() {
+		_, err := Run(s, context.Background(), "acme", "m_1", Cmd{Argv: []string{"sleep", "600"}})
+		ran <- err
+	}()
+	<-h.running
+
+	n, err := Stop(s, context.Background(), "acme", "m_1")
+	if err != nil {
+		t.Fatalf("Stop: %v", err)
+	}
+	if n != 1 {
+		t.Fatalf("Stop interrupted %d commands, want 1 — a stop that reports a kill it did "+
+			"not perform is worse than one that reports none", n)
+	}
+	select {
+	case err := <-ran:
+		if err == nil {
+			t.Fatal("the stopped command answered success; a run cancelled mid-flight is not one " +
+				"that finished")
+		}
+	case <-time.After(2 * time.Second):
+		t.Fatal("the command outlived its stop — cancelling has to reach the exec channel, or " +
+			"`stop_run` is a button that does nothing")
+	}
+
+	// THE SANDBOX SURVIVES. Stop ends the work; End ends the resource.
+	if _, err := Get(s, context.Background(), "acme", "m_1"); err != nil {
+		t.Fatalf("the sandbox is gone after a stop (%v) — stop must not take the evidence with it", err)
+	}
+}
+
+// TestStopReachesOnlyItsOwnTenant is the one that must not be wrong.
+//
+// A stop is a WRITE against somebody's running work, so a caller able to reach
+// across the org boundary could halt another tenant's run by guessing an id. The
+// refusal is the ordinary org lookup every other operation here walks through,
+// applied BEFORE the in-flight set is consulted at all — and it answers 404 and
+// not 403, because a 403 would confirm the sandbox exists and whether a given
+// sandbox exists is itself a cross-tenant fact.
+func TestStopReachesOnlyItsOwnTenant(t *testing.T) {
+	h := &held{running: make(chan struct{}), release: make(chan struct{})}
+	s := service(t, h)
+	ready(t, s)
+	seed(t, s, "acme", "m_1")
+
+	ran := make(chan error, 1)
+	go func() {
+		_, err := Run(s, context.Background(), "acme", "m_1", Cmd{Argv: []string{"sleep", "600"}})
+		ran <- err
+	}()
+	<-h.running
+
+	// The neighbour knows the id — ids are not secrets — and asks for it by name.
+	n, err := Stop(s, context.Background(), "evil", "m_1")
+	if err == nil {
+		t.Fatalf("another org stopped acme's command (%d interrupted); an id is not an "+
+			"authorization", n)
+	}
+	if n != 0 {
+		t.Errorf("a refused stop reported interrupting %d commands, want 0", n)
+	}
+	if !strings.Contains(strings.ToLower(err.Error()), "not found") {
+		t.Errorf("a cross-tenant stop answered %q; it must be indistinguishable from an id "+
+			"that does not exist, or the refusal itself confirms the sandbox", err)
+	}
+	// And acme's command is STILL RUNNING. The neighbour changed nothing.
+	select {
+	case err := <-ran:
+		t.Fatalf("acme's command ended (%v) after another org asked for it to stop", err)
+	case <-time.After(100 * time.Millisecond):
+	}
+
+	// The owner can, on the same id, through the same call.
+	if n, err := Stop(s, context.Background(), "acme", "m_1"); err != nil || n != 1 {
+		t.Fatalf("the owner's stop = (%d, %v), want (1, nil) — the gate must be the tenant "+
+			"and not the operation", n, err)
+	}
+	<-ran
+
+	// An empty org is refused too, rather than defaulting to one. A stop that
+	// arrived with no caller must halt nothing.
+	if _, err := Stop(s, context.Background(), "", "m_1"); err == nil {
+		t.Error("a stop with no tenant was allowed; it has to fail, not pick an org")
+	}
+}
+
+// TestStopReportsNothingWhenThereIsNothingToStop. Zero is an answer: a command
+// that finished a moment ago is one there is nothing left to interrupt, and a
+// caller cannot win that race. What it does need to tell apart is "already over"
+// from "not yours", which is the 404 above.
+func TestStopReportsNothingWhenThereIsNothingToStop(t *testing.T) {
+	s := service(t, &held{running: make(chan struct{}), release: make(chan struct{})})
+	ready(t, s)
+	seed(t, s, "acme", "m_idle")
+
+	n, err := Stop(s, context.Background(), "acme", "m_idle")
+	if err != nil || n != 0 {
+		t.Fatalf("Stop on an idle sandbox = (%d, %v), want (0, nil)", n, err)
+	}
+}
+
+// TestTheInFlightSetDoesNotGrow. Every start is paired with the func that forgets
+// it; without that pairing the set keeps a dead cancel per command, which for a
+// long-lived exec sandbox is a leak measured in commands and not in bytes.
+func TestTheInFlightSetDoesNotGrow(t *testing.T) {
+	w := newWork()
+	for range 100 {
+		_, stop := context.WithCancel(context.Background())
+		w.start("m_1", stop)()
+		stop()
+	}
+	w.mu.Lock()
+	held := len(w.in)
+	w.mu.Unlock()
+	if held != 0 {
+		t.Fatalf("%d sandboxes still hold cancels after every command ended, want 0", held)
+	}
+}
+
+// TestStopEndsEveryCommandInTheSandbox. A sandbox runs more than one thing — a
+// coding run clones and then works — so a stop that ended only the first would
+// leave the run going.
+func TestStopEndsEveryCommandInTheSandbox(t *testing.T) {
+	w := newWork()
+	var ctxs []context.Context
+	for range 3 {
+		ctx, stop := context.WithCancel(context.Background())
+		ctxs = append(ctxs, ctx)
+		w.start("m_1", stop)
+	}
+	other, stopOther := context.WithCancel(context.Background())
+	defer stopOther()
+	w.start("m_2", stopOther)
+
+	if n := w.stop("m_1"); n != 3 {
+		t.Fatalf("stopped %d, want 3", n)
+	}
+	for i, ctx := range ctxs {
+		if ctx.Err() == nil {
+			t.Errorf("command %d was not cancelled", i)
+		}
+	}
+	if other.Err() != nil {
+		t.Error("stopping one sandbox cancelled another's command")
+	}
+}
+
+// ---- what a watcher is told -------------------------------------------------
+
+// TestNarrationIsCoalesced. A build prints thousands of lines and a session's log
+// is a durable ordered record, not a byte pipe: without a floor, one `npm
+// install` writes a row per line and a watcher spends its whole budget rendering
+// a scroll nobody reads.
+func TestNarrationIsCoalesced(t *testing.T) {
+	x := &tell{org: "acme", session: "sess_1"}
+	now := time.Now()
+
+	x.buf = append(x.buf, "first\n"...)
+	if got := x.take(now, false); got != "first\n" {
+		t.Fatalf("the first line was withheld (%q); nothing has been said yet, so there is "+
+			"nothing to coalesce with", got)
+	}
+	x.buf = append(x.buf, "second\n"...)
+	if got := x.take(now.Add(tellEvery/2), false); got != "" {
+		t.Errorf("a line arriving within the floor was sent immediately (%q)", got)
+	}
+	x.buf = append(x.buf, "third\n"...)
+	if got := x.take(now.Add(tellEvery), false); got != "second\nthird\n" {
+		t.Errorf("take = %q, want both withheld lines together — coalescing must delay, "+
+			"never drop", got)
+	}
+}
+
+// TestTheLastWordsAreAlwaysSaid. force ignores the floor, because the end of a
+// command must not lose what it said last to a rate limit — and the end is
+// exactly when a stopped run has something a watcher needs.
+func TestTheLastWordsAreAlwaysSaid(t *testing.T) {
+	x := &tell{org: "acme", session: "sess_1"}
+	now := time.Now()
+	x.take(now, false) // nothing yet
+	x.buf = append(x.buf, "the reason it failed\n"...)
+	if got := x.take(now, true); got != "the reason it failed\n" {
+		t.Fatalf("take(force) = %q; a command's last line was rate-limited away", got)
+	}
+}
+
+// TestABurstKeepsItsTail. The far side refuses a payload over its ceiling whole,
+// so a burst has to be cut here — and what a program said MOST RECENTLY is what
+// says why it is stuck. Cutting the head would keep the banner and drop the
+// stack trace.
+func TestABurstKeepsItsTail(t *testing.T) {
+	x := &tell{org: "acme", session: "sess_1"}
+	x.buf = append(x.buf, strings.Repeat("x", tellCap*2)...)
+	x.buf = append(x.buf, "THE ERROR"...)
+	got := x.take(time.Now(), true)
+	if len(got) > tellCap {
+		t.Fatalf("chunk is %d bytes, over the %d cap — the append would be refused whole", len(got), tellCap)
+	}
+	if !strings.HasSuffix(got, "THE ERROR") {
+		t.Fatal("the cut kept the head and dropped the end; the end is the part that says why")
+	}
+}
+
+// TestNothingIsSaidWhenNobodyIsWatching. A command with no session named is the
+// ordinary case — a bare exec, a file read — and it must cost nothing at all: no
+// buffer, no peer call, no branch at the call site.
+func TestNothingIsSaidWhenNobodyIsWatching(t *testing.T) {
+	for _, tc := range []struct{ org, session string }{{"acme", ""}, {"acme", "   "}, {"", "sess_1"}} {
+		if x := newTell(tc.org, tc.session); x != nil {
+			t.Errorf("newTell(%q, %q) produced a sink; there is nowhere for it to send", tc.org, tc.session)
+		}
+	}
+	// A nil tell is safely written to and safely finished, so no caller branches.
+	var x *tell
+	if n, err := x.Write([]byte("output")); n != 6 || err != nil {
+		t.Fatalf("nil tell Write = (%d, %v), want (6, nil)", n, err)
+	}
+	x.done(0, nil)
+}
+
+// TestNarrationCannotNameATenant is the tenancy property of the live feed, stated
+// where it is enforced: a tell is built from the org the CALLER PROVED and a
+// session the caller named, and there is no third place an org could come from.
+// Cmd carries no org field, so a request cannot supply one — which is why a run
+// can only ever narrate into its own tenant's session.
+func TestNarrationCannotNameATenant(t *testing.T) {
+	x := newTell("acme", "sess_1")
+	if x.org != "acme" {
+		t.Fatalf("tell org = %q, want the proven caller's", x.org)
+	}
+	// If Cmd ever grows an org, this fails to compile — which is the point.
+	var c Cmd
+	c.Session = "sess_1"
+	if got := newTell("acme", c.Session).org; got != "acme" {
+		t.Errorf("the org came from somewhere other than the caller: %q", got)
+	}
+}

apps/sites/sites.go

@@ -717,6 +717,10 @@ func (s *Server) streamObject(c *zip.Ctx, site Site, obj *s3.Object, size int64,
 	for _, h := range crossOriginIsolation(site.CrossOriginIsolation, contentType) {
 		c.SetHeader(h[0], h[1])
 	}
+	// Who may READ a subresource, decided the same way, from the same content type.
+	for _, h := range cors(site.CrossOriginIsolation, contentType) {
+		c.SetHeader(h[0], h[1])
+	}
 	c.Status(status)
 	return c.Fiber().SendStream(obj, int(size))
 }
@@ -871,6 +875,33 @@ func crossOriginIsolation(enabled bool, contentType string) [][2]string {
 	return [][2]string{{"Cross-Origin-Resource-Policy", "same-origin"}}
 }
 
+// cors is the ONE cross-origin READ policy for a site's subresources, keyed off
+// the same content type as the two policies above.
+//
+// It grants nothing that was not already public. These bytes are served on the
+// unauthenticated edge with no credentials, so anyone can fetch them today; a
+// CORS header decides whether a script may READ the response it already
+// received, and for public files that distinction protects nobody.
+//
+// What it buys is the builder. hanzo.app previews a project inside a frame
+// sandboxed WITHOUT allow-same-origin — deliberately, so untrusted generated
+// HTML cannot reach the IAM tokens — which makes the frame an opaque origin. A
+// Vite build's entry is `<script type="module" crossorigin>`, and a module
+// ALWAYS fetches in CORS mode, so with no Access-Control-Allow-Origin the bundle
+// is refused, nothing mounts into `<div id="root">`, and a perfectly healthy
+// deployed site previews as a blank white page.
+//
+//   - a DOCUMENT is navigated to, not read cross-origin, and the preview brings
+//     its own; it gets nothing.
+//   - a site that opted into cross-origin ISOLATION asked for same-origin
+//     subresources, and this must not quietly widen that back out.
+func cors(isolated bool, contentType string) [][2]string {
+	if isolated || strings.HasPrefix(contentType, "text/html") {
+		return nil
+	}
+	return [][2]string{{"Access-Control-Allow-Origin", "*"}}
+}
+
 // CacheControlFor is the ONE canonical cache policy by asset class, used both when
 // WRITING an object at deploy (projects/blob.go) and when SERVING one here, so a
 // site's TTL is identical on the site-server path and the direct-S3 path.

apps/sites/sites_test.go

@@ -822,3 +822,56 @@ func (f *fakeResolver) reset() {
 	defer f.mu.Unlock()
 	f.calls, f.orgCalls = nil, nil
 }
+
+// TestCors pins the header that decides whether the builder can preview a
+// deployed site at all.
+//
+// Measured in a real sandboxed frame (sandbox="allow-scripts", the preview's own
+// attributes) against megashop.hanzo.app: the bundle loads as an absolute CLASSIC
+// script and is BLOCKED as a module. Vite emits a module, so a site with no
+// Access-Control-Allow-Origin previews as a blank white page.
+func TestCors(t *testing.T) {
+	// A subresource is readable cross-origin. This is the whole fix.
+	for _, ct := range []string{
+		"text/javascript; charset=utf-8",
+		"text/css; charset=utf-8",
+		"application/wasm",
+		"image/png",
+		"application/json",
+		"font/woff2",
+	} {
+		if got := headerMap(cors(false, ct))["Access-Control-Allow-Origin"]; got != "*" {
+			t.Errorf("cors(false, %q) ACAO = %q, want *", ct, got)
+		}
+	}
+
+	// A DOCUMENT is navigated to, never read cross-origin — and the preview
+	// supplies its own. Granting it nothing keeps the surface as small as the
+	// problem.
+	for _, ct := range []string{"text/html; charset=utf-8", "text/html"} {
+		if h := cors(false, ct); h != nil {
+			t.Errorf("cors(false, %q) = %v, want nil", ct, h)
+		}
+	}
+
+	// A site that opted into cross-origin isolation asked for same-origin
+	// subresources. ACAO must not quietly widen that back out — if this ever
+	// returns a header, isolation became decorative.
+	for _, ct := range []string{"text/javascript; charset=utf-8", "application/wasm", "text/html; charset=utf-8"} {
+		if h := cors(true, ct); h != nil {
+			t.Errorf("cors(true, %q) = %v, want nil — isolation must win", ct, h)
+		}
+	}
+
+	// The two policies stay orthogonal: neither emits the other's headers.
+	a := headerMap(cors(false, "application/wasm"))
+	for _, k := range []string{"Cross-Origin-Resource-Policy", "Cross-Origin-Opener-Policy", "Cross-Origin-Embedder-Policy"} {
+		if _, ok := a[k]; ok {
+			t.Errorf("cors must not emit %s, got %v", k, a)
+		}
+	}
+	i := headerMap(crossOriginIsolation(true, "application/wasm"))
+	if _, ok := i["Access-Control-Allow-Origin"]; ok {
+		t.Errorf("crossOriginIsolation must not emit ACAO, got %v", i)
+	}
+}

apps/websearch/brave.go

@@ -0,0 +1,182 @@
+package websearch
+
+// brave.go — the one PAID engine, and the only one that costs us money.
+//
+// Every other engine here scrapes a public result page for free and is rate
+// limited for it: measured from the cluster, DuckDuckGo answers about seven
+// requests and then serves HTTP 202 with a challenge page for minutes. Brave
+// sells an API with no such wall, and it answers better — for "rust tokio" it
+// returns tokio.rs, github.com/tokio-rs/tokio and docs.rs, where the scraped
+// engines have returned a fireworks retailer for "firecracker".
+//
+// SO IT IS OPT-IN, AND IT IS METERED. It is not in the default engine set: an
+// engine that silently spends money the moment it is compiled in is a bill
+// nobody agreed to. Naming it in WEBSEARCH_ENGINES is the agreement, and a
+// missing key means it contributes nothing rather than erroring — the same rule
+// every other engine follows.
+//
+// WHY THE PRICE IS NOT DECLARED AT THE EDGE. price.go's Consumes() makes GET
+// free by construction — "a read spends nothing, so a read costs nothing" — and
+// a search IS a read, so /v1/websearch/search can never carry a per-request edge
+// price. price.go states the answer for exactly this case: "such a surface
+// meters its own units downstream and declares Metered."
+//
+// THE CUSTOMER PAYS FOR THE ANSWER, NOT FOR OUR UPSTREAM CALL. The debit is one
+// per search served with this engine enabled, taken in metaSearch — including
+// when the cache answered and no Brave request was made. That is a deliberate
+// pricing decision and not a cost pass-through: the price is what the answer is
+// worth to the caller, and what we save by not re-asking is margin. Stating it
+// here because a reader who assumed cost-recovery would "fix" the cache path
+// into a discount and quietly change the product.
+
+import (
+	"context"
+	"encoding/json"
+	"io"
+	"net/http"
+	"net/url"
+	"os"
+	"strconv"
+	"strings"
+)
+
+// braveKey is the subscription token. KMS-sourced, synced onto the cloud env as
+// WEBSEARCH_BRAVE_KEY like every other secret this process reads — never a
+// literal, and never a value in a manifest.
+func braveKey() string { return strings.TrimSpace(os.Getenv("WEBSEARCH_BRAVE_KEY")) }
+
+func braveURL() string {
+	return envOr("WEBSEARCH_BRAVE_URL", "https://api.search.brave.com/res/v1/web/search")
+}
+
+// bravePriceMillicents is what ONE search costs the CALLER, in thousandths of a
+// cent. It matches Brave's own list price to us — $5.00 per 1,000 = $0.005 =
+// 0.5 cents = 500 millicents — so the product is sold at cost on the upstream
+// call and earns on every answer the cache serves.
+//
+// Millicents because a cent cannot express a half-cent. Rounding up to 1c
+// doubles the price; rounding down to 0c makes the surface free, which is how a
+// metered product silently stops billing. WEBSEARCH_BRAVE_PRICE overrides.
+const bravePriceMillicents int64 = 500
+
+func bravePrice() int64 {
+	if v := strings.TrimSpace(os.Getenv("WEBSEARCH_BRAVE_PRICE")); v != "" {
+		if n, err := strconv.ParseInt(v, 10, 64); err == nil && n >= 0 {
+			return n
+		}
+	}
+	return bravePriceMillicents
+}
+
+// braveQuery is the request, in ONE place, because build() and the fetch must
+// ask the identical question — a cache keyed on a URL the fetch does not send is
+// a cache that answers for a different query.
+func braveQuery(query, lang string) url.Values {
+	v := url.Values{}
+	v.Set("q", query)
+	v.Set("count", "20")
+	if lang != "" {
+		v.Set("search_lang", lang)
+	}
+	return v
+}
+
+// collapse squeezes whitespace and bounds a snippet: an engine snippet is a few
+// lines, and an unbounded one becomes an enormous "result".
+func collapse(s string) string {
+	out := strings.Join(strings.Fields(s), " ")
+	const maxSnippet = 400
+	if len(out) > maxSnippet {
+		out = strings.TrimSpace(out[:maxSnippet]) + "…"
+	}
+	return out
+}
+
+// braveResponse is the subset of Brave's envelope we read. Its other fields
+// (mixed, query, infobox, discussions) describe a page WE do not render, so
+// reading them would be inventing a contract we do not serve.
+type braveResponse struct {
+	Web struct {
+		Results []struct {
+			URL         string `json:"url"`
+			Title       string `json:"title"`
+			Description string `json:"description"`
+		} `json:"results"`
+	} `json:"web"`
+}
+
+// braveEngine is JSON, not HTML — which is why it carries its own fetch instead
+// of a `parse`. The engine struct's parse takes an *html.Node, and pretending a
+// JSON API is a page in order to fit that shape would be the adapter this
+// codebase keeps deleting. `fetch` is the seam: an engine either parses HTML or
+// fetches for itself, never both.
+var braveEngine = engine{
+	name: braveName,
+	// The full request URL, query included — so the cache keys on the question
+	// exactly as it does for every other engine, with no second key shape.
+	build: func(query, lang string) string { return braveURL() + "?" + braveQuery(query, lang).Encode() },
+	fetch: braveFetch,
+}
+
+func braveFetch(ctx context.Context, query, lang string) ([]webResult, error) {
+	key := braveKey()
+	if key == "" {
+		// No key is not an error. It is this engine being unavailable, which is
+		// the same state a challenged engine reaches, and the request is still
+		// answered by whatever else is enabled.
+		return nil, nil
+	}
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, braveURL()+"?"+braveQuery(query, lang).Encode(), nil)
+	if err != nil {
+		return nil, err
+	}
+	req.Header.Set("Accept", "application/json")
+	req.Header.Set("X-Subscription-Token", key)
+
+	resp, err := searchClient.Do(req)
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return nil, errStatus(braveName, resp.StatusCode)
+	}
+
+	var out braveResponse
+	if err := json.NewDecoder(io.LimitReader(resp.Body, 4<<20)).Decode(&out); err != nil {
+		return nil, err
+	}
+
+	res := make([]webResult, 0, len(out.Web.Results))
+	for _, w := range out.Web.Results {
+		if strings.TrimSpace(w.URL) == "" {
+			continue
+		}
+		res = append(res, webResult{
+			URL:     w.URL,
+			Title:   strings.TrimSpace(w.Title),
+			Content: collapse(stripTags(w.Description)),
+			Engine:  braveName,
+		})
+	}
+	return res, nil
+}
+
+// stripTags removes the <strong> emphasis Brave wraps query terms in. The
+// snippet is text for a person or a model to read, and markup in it would be
+// rendered literally by every consumer of the SearXNG envelope.
+func stripTags(s string) string {
+	var b strings.Builder
+	depth := 0
+	for _, r := range s {
+		switch {
+		case r == '<':
+			depth++
+		case r == '>' && depth > 0:
+			depth--
+		case depth == 0:
+			b.WriteRune(r)
+		}
+	}
+	return b.String()
+}

apps/websearch/cache.go

@@ -0,0 +1,143 @@
+package websearch
+
+// cache.go — the same query does not scrape the same engine twice.
+//
+// WHY THIS IS CORRECTNESS AND NOT SPEED. Every engine here is a public search
+// page fetched over HTTP, and every one of them rate-limits. When lite.duckduckgo
+// is served a challenge instead of results, parseDDG finds nothing and
+// fetchEngine returns ZERO RESULTS WITHOUT AN ERROR — by design, so one unhappy
+// engine cannot fail a request that another engine can answer. The cost of that
+// design is that a rate-limited engine is indistinguishable from a disabled one:
+// both are silently absent.
+//
+// Measured before this existed, four queries run back to back through the real
+// engines (bing then ddg, ~2.4s total):
+//
+//	post quantum cryptography lattice   bing 10   ddg 10
+//	firecracker microvm kvm setup       bing 10   ddg  0   <- challenged
+//	gvisor runsc syscall interception   bing 10   ddg  0   <- challenged
+//	rust tokio select cancellation      bing 10   ddg 10
+//
+// DDG answered the first request and then stopped answering. Nothing was broken;
+// it was simply asked four times in three seconds. A cache removes the repeat ask
+// entirely, which is the only fix that does not involve asking someone else's
+// server more nicely and hoping.
+//
+// SO: A HIT IS NEVER STORED WHEN IT IS EMPTY. Caching a challenge page's zero
+// results would pin the failure for the whole TTL and make the engine look
+// permanently dead — the exact defect this file exists to end. Only a non-empty
+// answer is worth remembering.
+//
+// IN-PROCESS, BOUNDED, NO DEPENDENCY. Not Redis and not a datastore: a search
+// result is derived, public, and cheap to re-fetch, so the correct home for it is
+// the memory of the process that asked. Bounded by count with the oldest entry
+// evicted, so a long-running host cannot grow one query at a time.
+
+import (
+	"os"
+	"strings"
+	"sync"
+	"time"
+)
+
+// cacheTTL is how long an engine's answer for a query stands. Long enough that a
+// person refining a question ("...lattice" then "...lattice kyber") does not
+// re-ask the parts that overlap, short enough that the web is allowed to change
+// within a session. WEBSEARCH_CACHE_TTL overrides; 0 disables the cache.
+func cacheTTL() time.Duration {
+	if v := strings.TrimSpace(os.Getenv("WEBSEARCH_CACHE_TTL")); v != "" {
+		if d, err := time.ParseDuration(v); err == nil {
+			return d
+		}
+	}
+	return 15 * time.Minute
+}
+
+// cacheMax bounds the entry count. Each entry is one engine's page of results
+// for one query, so a few thousand is small; the bound exists so an adversarial
+// query stream cannot grow the process without limit.
+const cacheMax = 2048
+
+type cacheEntry struct {
+	results []webResult
+	stored  time.Time
+}
+
+var (
+	cacheMu sync.Mutex
+	cached  = map[string]cacheEntry{}
+)
+
+// cacheKey names one engine's answer to one question, and the name is the
+// REQUEST URL — the endpoint plus the encoded query — not the engine's label.
+//
+// Keyed on the label instead, the cache answers for a request it never made.
+// Every engine endpoint here is an env override (WEBSEARCH_BING_URL,
+// WEBSEARCH_DDG_URL), so "bing" is not one address; it is whichever address is
+// configured right now. Three tests proved it before this was written: pointed at
+// a stub server, they got the PREVIOUS caller's real results and reported that
+// the engine had not been reached, that a failing engine had returned rows, and
+// that a page with no sources had sources.
+//
+// The request URL is the honest identity of a question. A different endpoint is a
+// different question, in production exactly as in a test.
+func cacheKey(requestURL string) string { return requestURL }
+
+// cacheGet returns a stored non-empty answer that has not expired.
+func cacheGet(requestURL string) ([]webResult, bool) {
+	ttl := cacheTTL()
+	if ttl <= 0 {
+		return nil, false
+	}
+	cacheMu.Lock()
+	defer cacheMu.Unlock()
+	e, ok := cached[cacheKey(requestURL)]
+	if !ok || time.Since(e.stored) > ttl {
+		return nil, false
+	}
+	return e.results, true
+}
+
+// cachePut remembers a non-empty answer. An EMPTY answer is never stored — see
+// the file comment: an engine that was challenged must be allowed to answer the
+// next time it is asked.
+func cachePut(requestURL string, results []webResult) {
+	if len(results) == 0 || cacheTTL() <= 0 {
+		return
+	}
+	cacheMu.Lock()
+	defer cacheMu.Unlock()
+	if len(cached) >= cacheMax {
+		evictOldest()
+	}
+	cached[cacheKey(requestURL)] = cacheEntry{results: results, stored: time.Now()}
+}
+
+// evictOldest drops the least recently stored entry. Called with cacheMu held.
+// A full scan is right at this size and has no bookkeeping to go wrong; if the
+// bound ever grows by an order of magnitude this becomes a heap, not a rewrite.
+func evictOldest() {
+	var oldestKey string
+	var oldest time.Time
+	for k, v := range cached {
+		if oldestKey == "" || v.stored.Before(oldest) {
+			oldestKey, oldest = k, v.stored
+		}
+	}
+	delete(cached, oldestKey)
+}
+
+// cacheSize is for tests and for the one log line that says whether the cache is
+// doing anything.
+func cacheSize() int {
+	cacheMu.Lock()
+	defer cacheMu.Unlock()
+	return len(cached)
+}
+
+// cacheReset empties the cache. Tests only — production has no reason to forget.
+func cacheReset() {
+	cacheMu.Lock()
+	defer cacheMu.Unlock()
+	cached = map[string]cacheEntry{}
+}

apps/websearch/outcome.go

@@ -0,0 +1,172 @@
+package websearch
+
+// outcome.go — an engine that returns nothing is either empty or BLIND, and
+// telling those apart is the whole reliability of a metasearch.
+//
+// Until this file, it could not. fetchEngine returned ([]webResult, error) and a
+// bot-challenge page came back as (nil, nil) — the same value as a query nobody
+// on the web has written about. So an engine could stop working entirely and the
+// only symptom was a slightly shorter page. That is how Brave was dropped rather
+// than fixed, and how DuckDuckGo sat in the default set for weeks contributing
+// zero: both failed SOFT, and soft failure is indistinguishable from calm.
+//
+// Measured from cluster egress, which is what made the three states obvious:
+//
+//	lite.duckduckgo.com  static  25,672 bytes, 0 results — "Unfortunately, bots
+//	                             use DuckDuckGo too. Select all squares
+//	                             containing a duck." A 200. A real page. No results.
+//	lite.duckduckgo.com  browser 24,410 bytes, 10 results — the same URL, rendered.
+//
+// The static fetch and the render disagree about the same page at the same
+// second. A design in which "0 results" is an ANSWER cannot represent that
+// disagreement, so it silently keeps the wrong half.
+//
+// THE THREE STATES
+//
+//	answered  the parser found results.
+//	blind     the fetch succeeded and the parser found NOTHING. Either their
+//	          markup moved (selector rot) or they served a challenge. From here
+//	          those look identical, and the operator's next move is the same for
+//	          both: go look at the page.
+//	failed    the engine was never reached — transport error, non-200. The parser
+//	          never ran, so this says nothing about the parser.
+//
+// WHY ZERO IS BLIND AND NOT EMPTY. There is no reliable "no results" marker to
+// test for, and the measurement says so: Bing NEVER returns zero. Asked three
+// distinct nonsense strings it returned ten results each time — Edmonton property
+// tax, Bastille Day, Microsoft support — and for a fourth, pornography. Mojeek and
+// DDG do return zero, but they also return zero when they serve a captcha, which
+// is the case this file exists to catch. Trusting an engine to self-report
+// emptiness means trusting the engine that is currently lying to us.
+//
+// So zero is blind, per engine, always — and the genuine-empty case is recovered
+// where the evidence for it actually lives: ACROSS engines. If one engine went
+// blind while another answered the same query, the query demonstrably has
+// results and that engine is broken NOW. That is the line worth waking someone
+// for, and it needs no magic strings.
+//
+// TWO INSTRUMENTS, DELIBERATELY DIFFERENT WIDTHS:
+//
+//   - the COUNTER records every outcome, always. An operator reads the ratio:
+//     ddg 95% blind is an outage, ddg 3% blind is a handful of obscure queries.
+//     A rate is the honest shape for this and needs no per-query judgement.
+//   - the LOG line fires only on a CONFIRMED fault — blind while a sibling
+//     answered. Narrow on purpose, so it stays worth reading.
+
+import (
+	"context"
+	"sync"
+
+	luxlog "github.com/luxfi/log"
+	"go.opentelemetry.io/otel"
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/metric"
+)
+
+// outcome is how one engine's turn ended. A string rather than an int because it
+// is published — in the answer's `engines` array and as a metric attribute — and
+// a number there would need a decoder ring on both sides.
+type outcome string
+
+const (
+	answered outcome = "answered"
+	blind    outcome = "blind"
+	failed   outcome = "failed"
+)
+
+// answer is one engine's reply to one query: what it returned, how that turn
+// ended, and whether the browser had already been asked.
+//
+// browsed matters to the reader of a blind: browsed=false means the static fetch
+// came back unreadable and escalation was off or unavailable, which is a
+// CONFIGURATION fact. browsed=true means we rendered the page in a real browser
+// and still could not read it, which is a PARSER fact. Same state, different
+// person to wake.
+type answer struct {
+	engine  string
+	results []webResult
+	outcome outcome
+	browsed bool
+	err     error
+}
+
+// meterName is this package's OTel scope, spelled the way apps/analytics and
+// apps/cron spell theirs.
+const meterName = "github.com/hanzoai/cloud/apps/websearch"
+
+var (
+	turnsOnce sync.Once
+	turns     metric.Int64Counter
+)
+
+// countTurn records one engine's outcome.
+//
+// CARDINALITY is bounded by construction and not by hope: `engine` ranges over
+// the engine registry (three names, chosen by us, never by a caller) and
+// `outcome` over the three constants above. Nine series, whatever the query
+// stream does. The query itself is deliberately NOT an attribute — it is
+// caller-supplied and unbounded, which is how a metric becomes an outage.
+func countTurn(ctx context.Context, a answer) {
+	turnsOnce.Do(func() {
+		turns, _ = otel.Meter(meterName).Int64Counter("hanzo_websearch_engine_total",
+			metric.WithDescription("Engine turns by outcome. A rising `blind` rate is selector rot or a bot challenge: the engine returned a page and we could not read a single result out of it."))
+	})
+	if turns == nil {
+		return
+	}
+	turns.Add(ctx, 1, metric.WithAttributes(
+		attribute.String("engine", a.engine),
+		attribute.String("outcome", string(a.outcome)),
+	))
+}
+
+// logger is set once by Mount. It stays nil for the in-process library callers
+// (compose.go) and in tests, so every use goes through warn. Guarded because
+// metaSearch runs its engines concurrently and a test may set it while another
+// search is in flight.
+var (
+	loggerMu sync.RWMutex
+	logger   luxlog.Logger
+)
+
+func setLogger(l luxlog.Logger) {
+	loggerMu.Lock()
+	defer loggerMu.Unlock()
+	logger = l
+}
+
+func warn(msg string, kv ...any) {
+	loggerMu.RLock()
+	l := logger
+	loggerMu.RUnlock()
+	if l != nil {
+		l.Warn(msg, kv...)
+	}
+}
+
+// report records every engine's outcome and says something ONLY about a
+// confirmed fault: an engine that went blind on a query another engine answered.
+//
+// The condition is the point. "ddg returned nothing" is not evidence of anything
+// on its own — the query may have no answers. "ddg returned nothing while bing
+// returned ten" is proof the query has answers and ddg cannot see them. The
+// first is noise and the second is the incident, and only the second is logged.
+func report(ctx context.Context, query string, answers []answer) {
+	anyAnswered := false
+	for _, a := range answers {
+		countTurn(ctx, a)
+		if a.outcome == answered {
+			anyAnswered = true
+		}
+	}
+	if !anyAnswered {
+		return
+	}
+	for _, a := range answers {
+		if a.outcome != blind {
+			continue
+		}
+		warn("websearch engine returned a page with no results while another engine answered — selector rot or a bot challenge",
+			"engine", a.engine, "query", query, "browsed", a.browsed)
+	}
+}

apps/websearch/outcome_test.go

@@ -0,0 +1,293 @@
+package websearch
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	luxlog "github.com/luxfi/log"
+)
+
+// rotted is a search-results page that RENDERED PERFECTLY and that our parser
+// cannot read: real markup, real anchors, real snippets, one class name changed.
+// This is what selector rot looks like from the inside — it is exactly why Brave
+// was excluded rather than parsed, its classes being Svelte build hashes that
+// change on every deploy.
+//
+// The bytes matter. A challenge page and a rotted page are both "a 200 with no
+// results we can see", and neither is short: DDG's measured challenge page is
+// 25,672 bytes of real HTML that says "Select all squares containing a duck".
+const rotted = `<html><head><title>Results</title></head><body>
+<div class="results">
+  <table>
+    <tr><td><a rel="nofollow" href="https://example.com/one" class="result-link-v2">First Real Result</a></td></tr>
+    <tr><td class="result-snippet-v2">A snippet that a person reading this page would see.</td></tr>
+    <tr><td><a rel="nofollow" href="https://example.com/two" class="result-link-v2">Second Real Result</a></td></tr>
+    <tr><td class="result-snippet-v2">Another snippet, also plainly visible to a person.</td></tr>
+  </table>
+</div></body></html>`
+
+// engineServing replies to every request with body, and points the named engine
+// at it.
+func engineServing(t *testing.T, env, body string) *httptest.Server {
+	t.Helper()
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.Header().Set("Content-Type", "text/html")
+		_, _ = w.Write([]byte(body))
+	}))
+	t.Cleanup(srv.Close)
+	t.Setenv(env, srv.URL)
+	return srv
+}
+
+// browserServing stands in for Hanzo Crawl, answering /crawl with body as the
+// rendered HTML.
+func browserServing(t *testing.T, body string) {
+	t.Helper()
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_ = json.NewEncoder(w).Encode(map[string]any{
+			"results": []map[string]any{{"html": body, "success": true}},
+		})
+	}))
+	t.Cleanup(srv.Close)
+	t.Setenv("CRAWL_URL", srv.URL)
+	t.Setenv("WEBSEARCH_RENDER", "on")
+}
+
+// capture points the package logger at a buffer and returns it.
+func capture(t *testing.T) *bytes.Buffer {
+	t.Helper()
+	buf := &bytes.Buffer{}
+	setLogger(luxlog.New("test").Output(buf))
+	t.Cleanup(func() { setLogger(nil) })
+	return buf
+}
+
+// THE TEST THIS FILE EXISTS FOR.
+//
+// The page rendered — in a real browser, at that — and the parser read nothing
+// out of it. That is a FAULT, and before the outcome existed it was reported as
+// zero results with no error: the same value as a query the web has no answer
+// for. An engine could rot away completely and the only symptom was a shorter
+// page.
+//
+// browsed must be true. We did not merely fail to fetch; we drew the page and
+// still could not read it, which rules out the network and points at the parser.
+func TestRenderedButUnparsedPageIsBlindNotEmpty(t *testing.T) {
+	cacheReset()
+	engineServing(t, "WEBSEARCH_DDG_URL", rotted)
+	browserServing(t, rotted)
+	t.Setenv("WEBSEARCH_ENGINES", "ddg")
+
+	got := metaSearch(context.Background(), "anything", "")
+
+	if len(got.Engines) != 1 {
+		t.Fatalf("engines = %+v, want one entry for the one engine asked", got.Engines)
+	}
+	e := got.Engines[0]
+	if e.Outcome != string(blind) {
+		t.Fatalf("outcome = %q, want %q — a page that rendered and parsed to nothing is a fault, not an empty answer", e.Outcome, blind)
+	}
+	if e.Name != ddgName {
+		t.Fatalf("engine name = %q, want %q", e.Name, ddgName)
+	}
+	// And the answer still stands: a blind engine must not fail the request.
+	if got.Results == nil {
+		t.Fatal("results must be a non-nil array even when every engine went blind")
+	}
+}
+
+// The browser having RUN is part of the report, because it decides who is woken.
+// browsed=false is a configuration fault (escalation off, or crawl unreachable);
+// browsed=true is a parser fault. Collapsing them buries the loudest signal this
+// package has under "not switched on".
+func TestBlindSaysWhetherTheBrowserHadAlreadyRun(t *testing.T) {
+	cacheReset()
+	engineServing(t, "WEBSEARCH_DDG_URL", rotted)
+	t.Setenv("WEBSEARCH_ENGINES", "ddg")
+
+	// Escalation off: the browser never ran.
+	t.Setenv("WEBSEARCH_RENDER", "")
+	off := fetchEngine(context.Background(), ddgEngine, "q", "")
+	if off.outcome != blind || off.browsed {
+		t.Fatalf("render off: outcome=%q browsed=%v, want blind and browsed=false", off.outcome, off.browsed)
+	}
+
+	// Escalation on and the render still unreadable: the browser ran.
+	cacheReset()
+	browserServing(t, rotted)
+	on := fetchEngine(context.Background(), ddgEngine, "q", "")
+	if on.outcome != blind || !on.browsed {
+		t.Fatalf("render on: outcome=%q browsed=%v, want blind and browsed=true", on.outcome, on.browsed)
+	}
+}
+
+// A bot challenge is the case the browser exists for, and it is measured, not
+// imagined: lite.duckduckgo.com served this cluster 25,672 bytes reading
+// "Unfortunately, bots use DuckDuckGo too. Select all squares containing a duck"
+// over static HTTP, and 10 real results through the browser at the same second.
+func TestChallengedEngineRecoversThroughTheBrowser(t *testing.T) {
+	cacheReset()
+	const challenge = `<html><body><h1>Unfortunately, bots use DuckDuckGo too.</h1>
+	<p>Please complete the following challenge to confirm this search was made by a human.</p>
+	<form><input name="duck"><input type="submit"></form></body></html>`
+	const real = `<html><body><table>
+	<tr><td><a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fgithub.com%2Ffirecracker-microvm%2Ffirecracker" class="result-link">firecracker-microvm/firecracker</a></td></tr>
+	<tr><td class="result-snippet">Secure and fast microVMs.</td></tr></table></body></html>`
+
+	engineServing(t, "WEBSEARCH_DDG_URL", challenge)
+	browserServing(t, real)
+	t.Setenv("WEBSEARCH_ENGINES", "ddg")
+
+	got := metaSearch(context.Background(), "firecracker microvm", "")
+	if len(got.Results) != 1 {
+		t.Fatalf("results = %+v, want the browser's one hit to survive the challenge", got.Results)
+	}
+	if got.Results[0].URL != "https://github.com/firecracker-microvm/firecracker" {
+		t.Fatalf("url = %q, want the uddg target unwrapped", got.Results[0].URL)
+	}
+	if got.Engines[0].Outcome != string(answered) {
+		t.Fatalf("outcome = %q, want %q — the browser read it", got.Engines[0].Outcome, answered)
+	}
+}
+
+// THE 202. DuckDuckGo serves its bot challenge under a 2xx — measured three
+// times from cluster egress, HTTP 202 with 14,180 bytes of "Select all squares
+// containing a duck" — and the fetch used to accept only 200. So the challenge
+// became a transport error, the transport error short-circuited past the
+// escalation, and the one engine the browser was deployed to rescue was the one
+// engine that could never reach it. DDG read `failed` on every live query.
+func TestChallengeUnderA202StillReachesTheBrowser(t *testing.T) {
+	cacheReset()
+	const challenge = `<html><body><h1>Unfortunately, bots use DuckDuckGo too.</h1></body></html>`
+	const real = `<html><body><table>
+	<tr><td><a href="https://example.com/rescued" class="result-link">Rescued</a></td></tr>
+	<tr><td class="result-snippet">via the browser</td></tr></table></body></html>`
+
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusAccepted) // 202, exactly as DDG serves it
+		_, _ = w.Write([]byte(challenge))
+	}))
+	defer srv.Close()
+	t.Setenv("WEBSEARCH_DDG_URL", srv.URL)
+	browserServing(t, real)
+	t.Setenv("WEBSEARCH_ENGINES", "ddg")
+
+	got := metaSearch(context.Background(), "q", "")
+	if got.Engines[0].Outcome != string(answered) {
+		t.Fatalf("outcome = %q, want %q — a 202 carries a body worth parsing and, when it holds no results, worth rendering", got.Engines[0].Outcome, answered)
+	}
+	if len(got.Results) != 1 || got.Results[0].URL != "https://example.com/rescued" {
+		t.Fatalf("results = %+v, want the browser's hit", got.Results)
+	}
+}
+
+// A 2xx that ALREADY carries results is parsed and kept, with no render — the
+// status widening must not turn a good answer into an escalation.
+func TestNon200SuccessIsParsedWithoutRendering(t *testing.T) {
+	cacheReset()
+	const good = `<html><body><li class="b_algo"><h2><a href="https://example.com/hit">Hit</a></h2><p>s</p></li></body></html>`
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusNonAuthoritativeInfo) // 203
+		_, _ = w.Write([]byte(good))
+	}))
+	defer srv.Close()
+	t.Setenv("WEBSEARCH_BING_URL", srv.URL)
+	t.Setenv("WEBSEARCH_ENGINES", "bing")
+	// No CRAWL_URL and no render: reaching for the browser here would fail the test.
+	t.Setenv("WEBSEARCH_RENDER", "")
+
+	got := fetchEngine(context.Background(), bingEngine, "q", "")
+	if got.outcome != answered || len(got.results) != 1 {
+		t.Fatalf("answer = %+v, want the 203's results kept", got)
+	}
+	if got.browsed {
+		t.Fatal("browsed = true, want no render for a status that already carried results")
+	}
+}
+
+// An engine that was never REACHED says nothing about the parser, so it must not
+// be reported as blind. Mixing the two would make every network blip look like
+// selector rot and make the blind rate useless as a signal.
+func TestUnreachableEngineIsFailedNotBlind(t *testing.T) {
+	cacheReset()
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.WriteHeader(http.StatusTooManyRequests)
+	}))
+	defer srv.Close()
+	t.Setenv("WEBSEARCH_BING_URL", srv.URL)
+	t.Setenv("WEBSEARCH_ENGINES", "bing")
+
+	got := metaSearch(context.Background(), "q", "")
+	if got.Engines[0].Outcome != string(failed) {
+		t.Fatalf("outcome = %q, want %q for a non-200", got.Engines[0].Outcome, failed)
+	}
+}
+
+// THE LOG LINE IS NARROW ON PURPOSE. An engine that returned nothing proves
+// nothing on its own — the query may have no answers. An engine that returned
+// nothing while a SIBLING answered the same query is proof the query has answers
+// and that engine cannot see them. Only the second is worth reading.
+func TestBlindIsLoggedOnlyWhenAnotherEngineAnswered(t *testing.T) {
+	const good = `<html><body><li class="b_algo"><h2><a href="https://example.com/hit">Hit</a></h2><p>snippet</p></li></body></html>`
+
+	t.Run("sibling answered — logged", func(t *testing.T) {
+		cacheReset()
+		buf := capture(t)
+		engineServing(t, "WEBSEARCH_BING_URL", good)
+		engineServing(t, "WEBSEARCH_DDG_URL", rotted)
+		t.Setenv("WEBSEARCH_ENGINES", "bing,ddg")
+		t.Setenv("WEBSEARCH_RENDER", "")
+
+		metaSearch(context.Background(), "a query with real answers", "")
+		if !strings.Contains(buf.String(), "ddg") {
+			t.Fatalf("log = %q, want the blind engine named — bing answered, so ddg is provably broken", buf.String())
+		}
+	})
+
+	t.Run("nothing answered — silent", func(t *testing.T) {
+		cacheReset()
+		buf := capture(t)
+		engineServing(t, "WEBSEARCH_BING_URL", rotted)
+		engineServing(t, "WEBSEARCH_DDG_URL", rotted)
+		t.Setenv("WEBSEARCH_ENGINES", "bing,ddg")
+		t.Setenv("WEBSEARCH_RENDER", "")
+
+		metaSearch(context.Background(), "a query nobody has answered", "")
+		if strings.Contains(buf.String(), "selector rot") {
+			t.Fatalf("log = %q, want silence — with no engine answering there is no evidence the query HAS results", buf.String())
+		}
+	})
+}
+
+// The blend carries its own explanation. Three results with an engine blind is a
+// different fact from three results with every engine answered, and a caller
+// that cannot tell them apart is the caller that shipped a metasearch quietly
+// running on one index.
+func TestAnswerReportsEveryEngineAsked(t *testing.T) {
+	cacheReset()
+	const good = `<html><body><li class="b_algo"><h2><a href="https://example.com/hit">Hit</a></h2><p>s</p></li></body></html>`
+	engineServing(t, "WEBSEARCH_BING_URL", good)
+	engineServing(t, "WEBSEARCH_DDG_URL", rotted)
+	t.Setenv("WEBSEARCH_ENGINES", "bing,ddg")
+	t.Setenv("WEBSEARCH_RENDER", "")
+
+	got := metaSearch(context.Background(), "q", "")
+	if len(got.Engines) != 2 {
+		t.Fatalf("engines = %+v, want one entry per engine asked", got.Engines)
+	}
+	by := map[string]webEngine{}
+	for _, e := range got.Engines {
+		by[e.Name] = e
+	}
+	if by[bingName].Outcome != string(answered) || by[bingName].Results != 1 {
+		t.Fatalf("bing = %+v, want answered with 1 result", by[bingName])
+	}
+	if by[ddgName].Outcome != string(blind) || by[ddgName].Results != 0 {
+		t.Fatalf("ddg = %+v, want blind with 0 results", by[ddgName])
+	}
+}

apps/websearch/rank.go

@@ -0,0 +1,114 @@
+package websearch
+
+// rank.go — the merged page is ordered by what the engines AGREE on, not by
+// which engine was named first.
+//
+// The merge used to preserve engine order: every hit from the first engine, then
+// every new hit from the second. That is a decision about configuration order
+// masquerading as a decision about relevance, and it is measurably wrong.
+// Measured against the real engines:
+//
+//	query: "post quantum cryptography lattice"
+//	bing  post.ca.gov/Training · post.ca.gov/post-profile · usps.com
+//	ddg   blog.cloudflare.com/lattice-crypto-primer · ssh.com · redhat.com
+//
+// Bing matched the word "post" and returned the California Peace Officer
+// Standards and Training board. DDG answered the actual question. With engine
+// order preserved and bing named first, the user's page opened with three
+// irrelevant results — the better engine's answer pushed below the fold by a
+// comma in an env var.
+//
+// TWO SIGNALS, AND NEITHER IS THE ENGINE'S NAME:
+//
+//   - AGREEMENT. A URL more than one engine returned is more likely to be the
+//     answer than one only a single engine found. This is the whole reason to run
+//     several engines rather than the best one, and the merge was throwing it
+//     away by deduping agreement into a single first-seen hit.
+//   - RANK. Within one engine, position carries that engine's own judgement.
+//     Averaging the positions a URL held preserves it without letting one engine's
+//     ordering dominate the page.
+//
+// Ties break on the best single rank any engine gave the URL, then on the URL
+// itself so the order is TOTAL and the same inputs always produce the same page.
+// A non-deterministic search result is a search result nobody can debug.
+
+import "sort"
+
+// scored is one URL's evidence across every engine that returned it.
+type scored struct {
+	result webResult
+	// engines is how many distinct engines returned this URL.
+	engines int
+	// sumRank is the sum of its zero-based positions, best is the smallest.
+	sumRank int
+	best    int
+	// first is the merge order it was discovered in — the last tiebreak, so the
+	// result is stable rather than map-ordered.
+	first int
+}
+
+// rankMerged orders the per-engine result lists into one page.
+//
+// perEngine is indexed the same way enabledEngines() is, and a nil entry (an
+// engine that failed or was challenged) simply contributes nothing — the same
+// rule the rest of this package follows.
+func rankMerged(perEngine [][]webResult, limit int) []webResult {
+	byURL := map[string]*scored{}
+	order := 0
+	for _, rs := range perEngine {
+		for pos, r := range rs {
+			key := normalizeURL(r.URL)
+			if key == "" {
+				continue
+			}
+			s, ok := byURL[key]
+			if !ok {
+				s = &scored{result: r, best: pos, first: order}
+				order++
+				byURL[key] = s
+			}
+			s.engines++
+			s.sumRank += pos
+			if pos < s.best {
+				s.best = pos
+			}
+			// Keep the richest copy: an engine that returned a snippet says more
+			// than one that returned a bare link, whichever found it first.
+			if len(r.Content) > len(s.result.Content) {
+				s.result = r
+			}
+		}
+	}
+
+	all := make([]*scored, 0, len(byURL))
+	for _, s := range byURL {
+		all = append(all, s)
+	}
+	sort.Slice(all, func(i, j int) bool {
+		a, b := all[i], all[j]
+		// More engines agreeing wins outright.
+		if a.engines != b.engines {
+			return a.engines > b.engines
+		}
+		// Then the better average position across the engines that had it.
+		ai, bi := a.sumRank*b.engines, b.sumRank*a.engines // compare means without floats
+		if ai != bi {
+			return ai < bi
+		}
+		// Then the best single position any engine gave it.
+		if a.best != b.best {
+			return a.best < b.best
+		}
+		// Then discovery order, so the sort is total and deterministic.
+		return a.first < b.first
+	})
+
+	out := make([]webResult, 0, limit)
+	for _, s := range all {
+		if len(out) >= limit {
+			break
+		}
+		out = append(out, s.result)
+	}
+	return out
+}

apps/websearch/rank_test.go

@@ -0,0 +1,155 @@
+package websearch
+
+import (
+	"testing"
+	"time"
+)
+
+// r is a result at a URL, with an optional snippet.
+func r(url, content string) webResult {
+	return webResult{URL: url, Title: url, Content: content}
+}
+
+// TestAgreementOutranksEngineOrder is the defect this ranking exists to fix, in
+// the shape it actually occurred: the first-named engine returned three
+// irrelevant hits for "post quantum cryptography lattice" (it matched the word
+// "post") and the second returned the right ones. Under engine-order merging the
+// user's page opened with the wrong three.
+func TestAgreementOutranksEngineOrder(t *testing.T) {
+	bing := []webResult{r("https://post.ca.gov/Training", ""), r("https://post.ca.gov/post-profile", ""), r("https://blog.cloudflare.com/lattice-crypto-primer/", "")}
+	ddg := []webResult{r("https://blog.cloudflare.com/lattice-crypto-primer/", "lattice primer"), r("https://www.redhat.com/pqc", "")}
+
+	got := rankMerged([][]webResult{bing, ddg}, 10)
+	if len(got) == 0 {
+		t.Fatal("no results")
+	}
+	// Both engines returned the cloudflare URL; nothing else was agreed on. It
+	// leads, even though it was bing's THIRD hit and bing was named first.
+	if got[0].URL != "https://blog.cloudflare.com/lattice-crypto-primer/" {
+		t.Fatalf("agreed-on result did not lead: %s", got[0].URL)
+	}
+	// And the richer copy survived the dedupe — the engine with a snippet wins
+	// the row, whichever engine found the URL first.
+	if got[0].Content != "lattice primer" {
+		t.Fatalf("the richer copy was dropped: %q", got[0].Content)
+	}
+}
+
+// TestRankIsTotalAndStable pins determinism. A search page that reorders itself
+// between identical requests cannot be debugged, and map iteration is random.
+func TestRankIsTotalAndStable(t *testing.T) {
+	a := []webResult{r("https://a.example/1", ""), r("https://b.example/2", ""), r("https://c.example/3", "")}
+	b := []webResult{r("https://c.example/3", ""), r("https://d.example/4", "")}
+
+	first := rankMerged([][]webResult{a, b}, 10)
+	for i := 0; i < 25; i++ {
+		again := rankMerged([][]webResult{a, b}, 10)
+		if len(again) != len(first) {
+			t.Fatalf("length moved: %d then %d", len(first), len(again))
+		}
+		for j := range first {
+			if first[j].URL != again[j].URL {
+				t.Fatalf("order moved at %d: %s then %s", j, first[j].URL, again[j].URL)
+			}
+		}
+	}
+}
+
+// TestChallengedEngineContributesNothing — a nil entry is an engine that failed
+// or was served a challenge. It must not shift the others or produce empty rows.
+func TestChallengedEngineContributesNothing(t *testing.T) {
+	live := []webResult{r("https://a.example/1", ""), r("https://b.example/2", "")}
+	got := rankMerged([][]webResult{nil, live, nil}, 10)
+	if len(got) != 2 || got[0].URL != "https://a.example/1" {
+		t.Fatalf("a challenged engine changed the answer: %+v", got)
+	}
+}
+
+// TestRankHonoursTheCap — the merge cap is the page size, not a suggestion.
+func TestRankHonoursTheCap(t *testing.T) {
+	var many []webResult
+	for i := 0; i < 50; i++ {
+		many = append(many, r("https://e.example/"+string(rune('a'+i%26))+string(rune('0'+i/26)), ""))
+	}
+	if got := rankMerged([][]webResult{many}, 20); len(got) != 20 {
+		t.Fatalf("cap not honoured: %d", len(got))
+	}
+}
+
+// ── the cache ────────────────────────────────────────────────────────────────
+
+// TestCacheNeverStoresAnEmptyAnswer is the whole reason the cache is safe to put
+// in front of a rate-limited engine. Caching a challenge page's zero results
+// would pin the failure for the TTL and make the engine look permanently dead.
+func TestCacheNeverStoresAnEmptyAnswer(t *testing.T) {
+	cacheReset()
+	cachePut("https://ddg.test/?q=q", nil)
+	if _, ok := cacheGet("https://ddg.test/?q=q"); ok {
+		t.Fatal("an empty answer was cached — a challenged engine would stay dead for the whole TTL")
+	}
+	cachePut("https://ddg.test/?q=q", []webResult{r("https://x.example/1", "")})
+	if _, ok := cacheGet("https://ddg.test/?q=q"); !ok {
+		t.Fatal("a real answer was not cached")
+	}
+}
+
+// TestCacheKeyedByRequestURL is the defect that broke three existing tests before
+// this key was corrected. Keyed on the engine LABEL, the cache answered for a
+// request it never made: every endpoint here is an env override, so "bing" is not
+// one address but whichever address is configured right now. Pointed at a stub,
+// TestSearchValidatedPrincipalBypassesKey reported the engine was never reached,
+// TestMetaSearchDegradesOnEngineFailure got rows from a failing engine, and
+// answer's TestSSEEmptySourcesStillTerminates found sources it had removed.
+func TestCacheKeyedByRequestURL(t *testing.T) {
+	cacheReset()
+	cachePut("https://bing.test/search?q=clojure", []webResult{r("https://bing.example/", "")})
+
+	// Same engine, same words, DIFFERENT endpoint — a different question.
+	if _, ok := cacheGet("https://stub.test/search?q=clojure"); ok {
+		t.Fatal("a stubbed endpoint read the real endpoint's answer")
+	}
+	// Same endpoint, different query.
+	if _, ok := cacheGet("https://bing.test/search?q=rust"); ok {
+		t.Fatal("one query read another query's answer")
+	}
+	// The identical request hits.
+	if _, ok := cacheGet("https://bing.test/search?q=clojure"); !ok {
+		t.Fatal("the same request did not hit")
+	}
+}
+
+// TestCacheExpires — the web is allowed to change.
+func TestCacheExpires(t *testing.T) {
+	cacheReset()
+	t.Setenv("WEBSEARCH_CACHE_TTL", "1ns")
+	cachePut("https://bing.test/?q=q", []webResult{r("https://x.example/1", "")})
+	time.Sleep(2 * time.Millisecond)
+	if _, ok := cacheGet("https://bing.test/?q=q"); ok {
+		t.Fatal("an expired entry was served")
+	}
+}
+
+// TestCacheDisabled — TTL 0 means no cache at all, in both directions.
+func TestCacheDisabled(t *testing.T) {
+	cacheReset()
+	t.Setenv("WEBSEARCH_CACHE_TTL", "0s")
+	cachePut("https://bing.test/?q=q", []webResult{r("https://x.example/1", "")})
+	if cacheSize() != 0 {
+		t.Fatal("an entry was stored with the cache disabled")
+	}
+	if _, ok := cacheGet("https://bing.test/?q=q"); ok {
+		t.Fatal("a lookup succeeded with the cache disabled")
+	}
+}
+
+// TestCacheIsBounded — an adversarial query stream must not grow the process one
+// query at a time.
+func TestCacheIsBounded(t *testing.T) {
+	cacheReset()
+	for i := 0; i < cacheMax+64; i++ {
+		cachePut("https://bing.test/?q="+string(rune(i%1000))+"-"+time.Now().Format("150405.000000000"), []webResult{r("https://x.example/1", "")})
+	}
+	if n := cacheSize(); n > cacheMax {
+		t.Fatalf("cache grew past its bound: %d > %d", n, cacheMax)
+	}
+}

apps/websearch/render.go

@@ -0,0 +1,192 @@
+package websearch
+
+// render.go — when an engine answers with nothing, ask it again through a real
+// browser.
+//
+// A search engine served a bot challenge returns 200 with a page that parses to
+// zero results. That is not an error and must not become one: another engine may
+// have answered, and a request that fails because one engine was unhappy is worse
+// than a request with fewer results. But zero is also not an ANSWER, and the
+// static fetch has no way to tell the two apart.
+//
+// A real browser can. It runs the JavaScript the challenge relies on, carries a
+// real fingerprint, and gets the page a person would get. We already run one:
+// Hanzo Crawl (headless Chromium, ghcr.io/hanzoai/crawl) is deployed in-cluster
+// and apps/crawl already escalates to it for client-rendered pages. This is the
+// same escalation, for the same reason, against the same service — the search
+// path simply never used it.
+//
+// ESCALATION IS ONE-WAY AND BEST-EFFORT, exactly as in apps/crawl/browser.go: it
+// is attempted only when the static fetch produced NOTHING, and if the browser is
+// absent, slow or unhappy the zero stands. So this can add results and can never
+// remove them, and a deployment without the browser behaves exactly as before.
+//
+// The RENDERED HTML IS PARSED BY THE ENGINE'S OWN PARSER. There is no second
+// extraction path and no second idea of what a result is — parseBing and parseDDG
+// are the only readers of an engine's markup, whichever fetch produced it.
+//
+// WHAT THE BROWSER FIXES, MEASURED. lite.duckduckgo.com, one URL, one second
+// apart from the same cluster:
+//
+//	static   25,672 bytes, 0 results — "Unfortunately, bots use DuckDuckGo too.
+//	                                   Select all squares containing a duck."
+//	browser  24,410 bytes, 10 results — github.com/firecracker-microvm/firecracker,
+//	                                   fly.io/learn/firecracker-vm, wikipedia.
+//
+// WHAT IT DOES NOT FIX: GOOGLE. Do not add a Google engine here; it was tried
+// and it is not viable from this network. Every path returns the /sorry/
+// interstitial — ~6KB, 19 captcha markers, "unusual traffic", zero results:
+//
+//	crawl, headless, stealth on          /sorry/  0 results
+//	crawl, headless, &udm=14 and &gbv=1  /sorry/  0 results
+//	bot-browser, HEADFUL Chrome 150 on a real X display, second egress IP
+//	                                     /sorry/  0 results
+//
+// The control rules out our technique: that same headful browser reads DDG's ten
+// results, and google.com's HOMEPAGE loads normally through it (268KB, title
+// "Google", no captcha). Only /search is refused, from two different egress IPs,
+// headless and headful alike. That is reputation attached to datacenter
+// addresses, not headless detection, so no browser flag reaches it — the fix
+// would be residential egress, which is a different decision than a parser.
+// Shipping it anyway would mean an engine permanently blind and permanently
+// warning: noise where outcome.go is trying to keep a signal.
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+
+	"golang.org/x/net/html"
+)
+
+// crawlURL is the in-cluster Hanzo Crawl service. Same default as
+// apps/crawl/browser.go, and the same env, because it is the SAME service — two
+// homes for one address is how one of them goes stale.
+func crawlURL() string {
+	if v := strings.TrimSpace(os.Getenv("CRAWL_URL")); v != "" {
+		return strings.TrimRight(v, "/")
+	}
+	return "http://crawl.hanzo.svc:11235"
+}
+
+// renderTimeout bounds the escalation. A browser render costs seconds where a
+// static fetch costs hundreds of milliseconds, and this runs while the caller
+// waits, so it is short: past this the zero result stands and the answer is
+// whatever the other engines returned.
+func renderTimeout() time.Duration {
+	if v := strings.TrimSpace(os.Getenv("WEBSEARCH_RENDER_TIMEOUT")); v != "" {
+		if d, err := time.ParseDuration(v); err == nil {
+			return d
+		}
+	}
+	return 12 * time.Second
+}
+
+// renderEnabled reports whether escalation may be attempted. NAMED, never
+// assumed — WEBSEARCH_RENDER=on turns it on, exactly as WEBSEARCH_ENGINES names
+// the engines.
+//
+// It defaults OFF for a reason found by a test rather than argued from taste.
+// apps/answer READS pages through the same Hanzo Crawl service, so search
+// escalation and answer's reading share one CRAWL_URL. On by default, a caller
+// who stubs crawl for its own reading silently feeds this too:
+// TestSSEEmptySourcesStillTerminates points Bing at an empty page to assert the
+// no-results path, and search then escalated into that test's crawl stub and
+// produced sources the test had gone to trouble to remove.
+//
+// Nothing there is wrong with the escalation; what is wrong is one process-wide
+// endpoint changing a second subsystem's answers without anyone naming it. So it
+// is named. Production names it in universe beside WEBSEARCH_ENGINES.
+func renderEnabled() bool {
+	return strings.EqualFold(strings.TrimSpace(os.Getenv("WEBSEARCH_RENDER")), "on")
+}
+
+// renderedResults asks the browser for the engine's page and parses it with that
+// engine's own parser. Any failure returns nil, which the caller treats as "the
+// static zero stands".
+//
+// The second return says whether the browser actually RENDERED the page — not
+// whether it found anything. The caller stamps it onto a blind answer, and it is
+// the difference between two faults that need different people: browsed=false
+// means escalation was off or the service was unreachable (configuration);
+// browsed=true means a real browser drew the page and our parser still read
+// nothing out of it (selector rot). Collapsing them would put the loudest signal
+// this package has back into the same bucket as "not switched on".
+func renderedResults(ctx context.Context, e engine, query, lang string) ([]webResult, bool) {
+	if !renderEnabled() {
+		return nil, false
+	}
+	ctx, cancel := context.WithTimeout(ctx, renderTimeout())
+	defer cancel()
+
+	body, err := renderPage(ctx, e.build(query, lang))
+	if err != nil || strings.TrimSpace(body) == "" {
+		return nil, false
+	}
+	root, err := html.Parse(strings.NewReader(body))
+	if err != nil {
+		return nil, false
+	}
+	return e.parse(root), true
+}
+
+// renderPage POSTs one URL to Hanzo Crawl and returns the rendered HTML.
+//
+// The request and response shapes are Crawl's, not ours, and they are read
+// defensively: the service is a separate process on its own release cadence, so a
+// field it stops sending must degrade to "no render" rather than to a panic.
+func renderPage(ctx context.Context, target string) (string, error) {
+	payload, err := json.Marshal(map[string]any{
+		"urls":         []string{target},
+		"cache_mode":   "bypass",
+		"screenshot":   false,
+		"only_text":    false,
+		"page_timeout": int(renderTimeout() / time.Millisecond),
+	})
+	if err != nil {
+		return "", err
+	}
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost, crawlURL()+"/crawl", strings.NewReader(string(payload)))
+	if err != nil {
+		return "", err
+	}
+	req.Header.Set("Content-Type", "application/json")
+	// Required by the service, and the same KMS-sourced token apps/crawl sends —
+	// one service, one credential, read from one env.
+	if tok := strings.TrimSpace(os.Getenv("CRAWL_API_TOKEN")); tok != "" {
+		req.Header.Set("Authorization", "Bearer "+tok)
+	}
+
+	resp, err := searchClient.Do(req)
+	if err != nil {
+		return "", err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return "", fmt.Errorf("crawl: http %d", resp.StatusCode)
+	}
+	var out struct {
+		Results []struct {
+			HTML        string `json:"html"`
+			CleanedHTML string `json:"cleaned_html"`
+			Success     bool   `json:"success"`
+		} `json:"results"`
+	}
+	if err := json.NewDecoder(io.LimitReader(resp.Body, 8<<20)).Decode(&out); err != nil {
+		return "", err
+	}
+	for _, r := range out.Results {
+		if r.HTML != "" {
+			return r.HTML, nil
+		}
+		if r.CleanedHTML != "" {
+			return r.CleanedHTML, nil
+		}
+	}
+	return "", nil
+}

apps/websearch/search.go

@@ -14,10 +14,11 @@
 //
 // Composable by construction: an engine is {name, build(query)→URL, parse(HTML)
 // →results}. metaSearch runs the ENABLED engines concurrently and merges+dedupes
-// by normalized URL. WEBSEARCH_ENGINES selects them (default "bing", verified
-// datacenter-tolerant from the cluster egress); adding one is a registry entry,
-// not new plumbing. Any engine that fails or gets bot-challenged contributes zero
-// and never fails the request — search degrades to fewer results, never to a 5xx.
+// by normalized URL. WEBSEARCH_ENGINES selects them; unset means defaultEngines,
+// which is every engine measured to survive the cluster egress (bing + mojeek).
+// Adding one is a registry entry, not new plumbing. Any engine that fails or is
+// bot-challenged contributes zero and never fails the request — search degrades
+// to fewer results, never to a 5xx.
 
 package websearch
 
@@ -40,14 +41,22 @@ const (
 	// A realistic desktop UA. Keyless engines serve datacenter IPs a bot-challenge
 	// page (not results) when the UA looks automated; this one gets real results.
 	browserUA = "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"
-	// maxResults caps the merged set — parity with a typical SearXNG first page and
-	// plenty for the chat web_search agent tool.
-	maxResults = 20
+	// maxResults caps the merged set. Bing's first page is 10 and Mojeek is asked
+	// for 20, so the cap is what the two of them can actually reach rather than a
+	// number one engine could never fill.
+	maxResults = 30
 
 	// Engine names as constants: the parsers stamp result.Engine with them, so
 	// they must NOT read <engine>.name (that closes an init cycle engine→parse→engine).
-	bingName = "bing"
-	ddgName  = "ddg"
+	bingName   = "bing"
+	ddgName    = "ddg"
+	mojeekName = "mojeek"
+	braveName  = "brave"
+
+	// mojeekCount is how many hits Mojeek is asked for. It honours `t` exactly
+	// (measured: t=20 → 20 results, t=30 → 30), so this is the one knob that
+	// raises the answer's size without adding an engine.
+	mojeekCount = "20"
 )
 
 // searchClient is dedicated to engine fetches: a tight timeout so one slow engine
@@ -78,6 +87,26 @@ type webResult struct {
 	Engine string `json:"engine,omitempty"`
 }
 
+// webEngine is what ONE engine did on this query: what it is called, how its
+// turn ended, and how many hits it contributed before the merge.
+//
+// It is published so a thin answer carries its own explanation. Without it, an
+// engine that has stopped working shows up only as fewer results, and the caller
+// cannot tell "the web is quiet on this" from "half our indexes are blind" — the
+// exact ambiguity that let DuckDuckGo sit in the default set contributing zero.
+// Three results with `ddg blind` is a different fact from three results with
+// every engine answered, and the caller deserves to see which one it got.
+type webEngine struct {
+	// Name is the engine, matching the `engine` stamped on each result.
+	Name string `json:"name"`
+	// Outcome is "answered", "blind" or "failed" — see outcome.go. "blind" means
+	// the page came back and no results could be read out of it.
+	Outcome string `json:"outcome"`
+	// Results is how many hits this engine contributed, before the merge
+	// deduplicated them against the others.
+	Results int `json:"results"`
+}
+
 // webSearchResults is the SearXNG /search?format=json envelope. `results` is always
 // a non-nil array so the client never decodes null.
 type webSearchResults struct {
@@ -87,23 +116,42 @@ type webSearchResults struct {
 	// estimate of what the web holds.
 	NumberOfResults int `json:"number_of_results"`
 	// Results are the merged hits, deduplicated by normalised URL and capped at
-	// 20. Always an array and never null: no hits is an ANSWER, not a fault.
+	// 30. Always an array and never null: no hits is an ANSWER, not a fault.
 	Results []webResult `json:"results"`
+	// Engines is one entry per engine asked, in the order they were asked. It is
+	// ADDITIVE to the SearXNG contract, which the LibreChat client ignores as an
+	// unknown field exactly as it ignores `engine` on a result.
+	Engines []webEngine `json:"engines,omitempty"`
 }
 
 // engine is one keyless public web-search backend: build a request URL for a
 // query, parse the returned HTML into results. Pure functions — unit-testable
 // against fixture HTML with no network.
+//
+// `fetch` is the one variation, and it is a seam rather than an adapter: an
+// engine that is a JSON API instead of a page answers for itself. Brave sells
+// one, and reshaping JSON into an *html.Node so it could reach `parse` is
+// exactly the shim this package keeps deleting. When fetch is set, parse is
+// unused; build still names the full request so the cache keys on the question
+// like every other engine.
 type engine struct {
 	name  string
 	build func(query, lang string) string
 	parse func(root *html.Node) []webResult
+	fetch func(ctx context.Context, query, lang string) ([]webResult, error)
+}
+
+// errStatus is the one shape an engine reports a refusing endpoint with, so a
+// 202 challenge and a 429 quota read the same to whatever counts them.
+func errStatus(engine string, code int) error {
+	return fmt.Errorf("%s: http %d", engine, code)
 }
 
 // ── engine endpoints (functions, not vars, so tests override via env) ────────
 
-func bingURL() string { return envOr("WEBSEARCH_BING_URL", "https://www.bing.com/search") }
-func ddgURL() string  { return envOr("WEBSEARCH_DDG_URL", "https://lite.duckduckgo.com/lite/") }
+func bingURL() string   { return envOr("WEBSEARCH_BING_URL", "https://www.bing.com/search") }
+func ddgURL() string    { return envOr("WEBSEARCH_DDG_URL", "https://lite.duckduckgo.com/lite/") }
+func mojeekURL() string { return envOr("WEBSEARCH_MOJEEK_URL", "https://www.mojeek.com/search") }
 
 func envOr(key, def string) string {
 	if v := strings.TrimSpace(os.Getenv(key)); v != "" {
@@ -138,19 +186,67 @@ var ddgEngine = engine{
 	parse: parseDDG,
 }
 
-var engineByName = map[string]engine{
-	bingEngine.name: bingEngine,
-	ddgEngine.name:  ddgEngine,
+// mojeek is an INDEPENDENT crawler rather than a front end onto someone else's
+// index, so its hits are genuinely additive to Bing's instead of the same ten
+// pages in a different order. Two properties earned it the default slot, both
+// measured from cluster egress rather than assumed:
+//
+//   - it serves a datacenter IP real results, where DuckDuckGo serves the
+//     anomaly page on both of its endpoints;
+//   - it honours `site:`, and Bing does not. A site:x.com query answers 10 on
+//     Mojeek and 0 on Bing, which makes this the engine that carries every
+//     scoped search — X, GitHub, Reddit — and not merely a second opinion.
+var mojeekEngine = engine{
+	name: mojeekName,
+	build: func(query, lang string) string {
+		v := url.Values{}
+		v.Set("q", query)
+		v.Set("t", mojeekCount)
+		if lang != "" {
+			v.Set("lb", lang)
+		}
+		return mojeekURL() + "?" + v.Encode()
+	},
+	parse: parseMojeek,
 }
 
+var engineByName = map[string]engine{
+	bingEngine.name:   bingEngine,
+	ddgEngine.name:    ddgEngine,
+	mojeekEngine.name: mojeekEngine,
+	braveEngine.name:  braveEngine,
+}
+
+// defaultEngines is what a deployment that configures nothing searches: every
+// engine measured to answer from datacenter egress. A LIST rather than one name
+// because production once ran with WEBSEARCH_ENGINES unset, which made this
+// default the whole engine set, and it was Bing alone — one index, ten results,
+// no `site:`.
+//
+// DDG is here even though it is served a captcha over static HTTP, because
+// render.go escalates and the browser reads it (measured: 0 static, 10 rendered,
+// same URL, same second). It earns the slot on the rendered number. If the
+// escalation is off, DDG is BLIND rather than quietly absent — outcome.go says
+// so in the answer and in the metric — which is the property that makes putting
+// a browser-dependent engine in the default set honest rather than optimistic.
+//
+// Bing is the weakest of the three and stays because it is the broadest. It has
+// no zero state at all: asked three distinct nonsense strings it returned ten
+// results each time (Edmonton property tax, Bastille Day, Microsoft support),
+// and for a fourth, pornography. It never abstains, so its hits carry no
+// evidence of relevance on their own. That is precisely what rank.go's agreement
+// scoring is for — a Bing hit no other index found ranks below one two of them
+// agree on — and it is why removing Bing is not obviously wrong, only untested.
+var defaultEngines = []engine{bingEngine, ddgEngine, mojeekEngine}
+
 // enabledEngines resolves WEBSEARCH_ENGINES (comma list) to the engine set,
-// defaulting to bing. Unknown names are ignored; an empty result falls back to
-// bing so search is never engine-less. DDG is opt-in (bot-challenged from some
-// datacenter egress) but ships coded so `bing,ddg` needs no new plumbing.
+// defaulting to [defaultEngines]. Unknown names are ignored, and a spec that
+// names none of the known engines falls back to the same default so search is
+// never engine-less.
 func enabledEngines() []engine {
 	spec := strings.TrimSpace(os.Getenv("WEBSEARCH_ENGINES"))
 	if spec == "" {
-		spec = bingEngine.name
+		return defaultEngines
 	}
 	var out []engine
 	for _, name := range strings.Split(spec, ",") {
@@ -159,54 +255,137 @@ func enabledEngines() []engine {
 		}
 	}
 	if len(out) == 0 {
-		out = append(out, bingEngine)
+		return defaultEngines
 	}
 	return out
 }
 
 // metaSearch runs the enabled engines CONCURRENTLY and merges their results,
-// deduped by normalized URL, capped at maxResults. Engine order is preserved
-// (deterministic ranking: first engine's hits lead). A failing/challenged engine
+// deduped by normalized URL, capped at maxResults. A failing or challenged engine
 // contributes nothing — the request never fails on its account.
+//
+// The merge is ranked by AGREEMENT (rank.go), not by the order engines were
+// named. It used to be the latter, and that made WEBSEARCH_ENGINES an accidental
+// relevance knob: with `bing,ddg`, bing's three irrelevant hits for "post quantum
+// cryptography lattice" opened the page while ddg's correct ones were pushed
+// below them. Which engine is listed first is a configuration fact and was never
+// evidence about a result.
 func metaSearch(ctx context.Context, query, lang string) webSearchResults {
 	engs := enabledEngines()
-	perEngine := make([][]webResult, len(engs))
+	answers := make([]answer, len(engs))
 
 	var wg sync.WaitGroup
 	for i := range engs {
 		wg.Add(1)
 		go func(i int) {
 			defer wg.Done()
-			r, err := fetchEngine(ctx, engs[i], query, lang)
-			if err == nil {
-				perEngine[i] = r
-			}
+			answers[i] = fetchEngine(ctx, engs[i], query, lang)
 		}(i)
 	}
 	wg.Wait()
 
-	seen := make(map[string]bool)
-	merged := make([]webResult, 0, maxResults)
-	for _, rs := range perEngine {
-		for _, s := range rs {
-			key := normalizeURL(s.URL)
-			if key == "" || seen[key] {
-				continue
-			}
-			seen[key] = true
-			merged = append(merged, s)
-			if len(merged) >= maxResults {
-				return webSearchResults{Query: query, NumberOfResults: len(merged), Results: merged}
-			}
-		}
+	// Say what happened before saying what was found. An engine that went blind
+	// is a fact about this answer, and reporting it here — once, where every
+	// caller of metaSearch passes — is what stops a broken engine from being
+	// visible only as a slightly shorter page. See outcome.go.
+	report(ctx, query, answers)
+
+	perEngine := make([][]webResult, len(answers))
+	engines := make([]webEngine, len(answers))
+	for i, a := range answers {
+		perEngine[i] = a.results
+		engines[i] = webEngine{Name: a.engine, Outcome: string(a.outcome), Results: len(a.results)}
+	}
+	merged := rankMerged(perEngine, maxResults)
+	return webSearchResults{
+		Query:           query,
+		NumberOfResults: len(merged),
+		Results:         merged,
+		Engines:         engines,
 	}
-	return webSearchResults{Query: query, NumberOfResults: len(merged), Results: merged}
 }
 
-// fetchEngine GETs one engine's result page with a realistic UA and parses it.
-// Returns an error on transport/HTTP failure; a bot-challenge page parses to
-// zero results (not an error), so it simply contributes nothing.
-func fetchEngine(ctx context.Context, e engine, query, lang string) ([]webResult, error) {
+// fetchEngine asks one engine and says how the asking went.
+//
+// It answers from the cache when it can, fetches statically when it cannot, and
+// escalates to a real browser when the static fetch parsed to NOTHING — which is
+// what a bot challenge looks like here, since a challenge is a 200 whose markup
+// holds no results. The three live in that order because that is their cost
+// order: remembered, then one GET, then a browser render. See cache.go and
+// render.go for why each is correctness rather than speed.
+//
+// It returns an `answer` rather than ([]webResult, error) because zero results
+// is NOT the same fact as "nothing to report", and the pair could not tell them
+// apart: a challenged engine and a query with no matches were both (nil, nil).
+// outcome.go has the measurements that make the difference concrete.
+//
+// Note what `blind` means AFTER the escalation ran: we rendered the page in a
+// real browser and still read zero results out of it. That is the strongest
+// evidence of selector rot the system can produce, and it is exactly the signal
+// that was missing when Brave was dropped for having unreadable markup.
+func fetchEngine(ctx context.Context, e engine, query, lang string) answer {
+	url := e.build(query, lang)
+	if hit, ok := cacheGet(url); ok {
+		return answer{engine: e.name, results: hit, outcome: answered}
+	}
+
+	// A JSON engine answers for itself. It is cached like the others — a PAID
+	// API is the one we least want to ask twice for the same question — and it
+	// reports the same outcomes, so a quota refusal reads as `blind` rather than
+	// as an engine that had nothing to say.
+	if e.fetch != nil {
+		out, err := e.fetch(ctx, query, lang)
+		if len(out) > 0 {
+			cachePut(url, out)
+			return answer{engine: e.name, results: out, outcome: answered}
+		}
+		if err != nil {
+			return answer{engine: e.name, outcome: failed}
+		}
+		return answer{engine: e.name, outcome: blind}
+	}
+
+	out, err := fetchEngineStatic(ctx, e, query, lang)
+	if len(out) > 0 {
+		cachePut(url, out)
+		return answer{engine: e.name, results: out, outcome: answered}
+	}
+
+	// NOTHING READABLE CAME BACK, and there is ONE remedy for that whichever way
+	// it happened: render the page in a real browser and read it again.
+	//
+	// This used to be two rules, and the seam between them cost us DuckDuckGo
+	// entirely. Escalation ran only when a 200 parsed to zero, so a bad status
+	// short-circuited to "failed" and never reached the browser. DDG's challenge
+	// is served as HTTP 202 — measured, three times over, 14,180 bytes of "Select
+	// all squares containing a duck" under a 2xx — so the one engine the browser
+	// was deployed to rescue was the one engine that could never reach it.
+	//
+	// A status is a fact about the fetch, not about whether a browser can read
+	// the page. Keeping it as a separate rule was a distinction the remedy does
+	// not have.
+	//
+	// A REFUSAL is worth a render for a reason that is not obvious: the browser
+	// runs in a different pod on a different node, so it leaves the cluster from a
+	// different address than this process does. An engine rate-limiting one of our
+	// egress IPs has not necessarily rate-limited the other. What it will NOT
+	// rescue is a refusal aimed at the browser's own address — asked to render a
+	// URL that had just answered it 403, Crawl returned 0 bytes. So this can
+	// recover an engine and can also spend ~1.1s learning nothing, which is
+	// affordable because the engines run concurrently and renderTimeout bounds it.
+	rendered, browsed := renderedResults(ctx, e, query, lang)
+	if len(rendered) > 0 {
+		cachePut(url, rendered)
+		return answer{engine: e.name, results: rendered, outcome: answered, browsed: browsed}
+	}
+	if err != nil {
+		// Never got a readable page at all, so this says nothing about the parser.
+		return answer{engine: e.name, outcome: failed, browsed: browsed, err: err}
+	}
+	return answer{engine: e.name, outcome: blind, browsed: browsed}
+}
+
+func fetchEngineStatic(ctx context.Context, e engine, query, lang string) ([]webResult, error) {
 	req, err := http.NewRequestWithContext(ctx, http.MethodGet, e.build(query, lang), nil)
 	if err != nil {
 		return nil, err
@@ -220,7 +399,11 @@ func fetchEngine(ctx context.Context, e engine, query, lang string) ([]webResult
 		return nil, err
 	}
 	defer resp.Body.Close()
-	if resp.StatusCode != http.StatusOK {
+	// ANY 2xx is an answer worth parsing, not just 200. DuckDuckGo serves its bot
+	// challenge as 202, and an engine is free to answer 203 or 206 as well;
+	// singling out 200 discarded bodies we had already paid to fetch and turned a
+	// readable page into a transport error.
+	if resp.StatusCode < 200 || resp.StatusCode > 299 {
 		return nil, fmt.Errorf("%s: http %d", e.name, resp.StatusCode)
 	}
 	root, err := html.Parse(io.LimitReader(resp.Body, 4<<20)) // 4 MiB cap
@@ -371,6 +554,37 @@ func ddgRealURL(href string) string {
 	return ""
 }
 
+// ── Mojeek: one <li> per hit, <a class="title"> + <p class="s"> ──────────────
+// Mojeek links straight at the destination — there is no click-tracking redirect
+// to unwrap, which is why this parser has no counterpart to bingRealURL. The
+// result classes are semantic (title, s) rather than build-hashed, so they
+// survive a redeploy; that is what makes this engine parseable at all where
+// Brave, whose classes are Svelte hashes like `svelte-1rq4ngz`, is not.
+
+func parseMojeek(root *html.Node) []webResult {
+	var out []webResult
+	forEach(root, func(n *html.Node) {
+		if n.Type != html.ElementNode || n.Data != "li" {
+			return
+		}
+		a := findFirst(n, func(x *html.Node) bool {
+			return x.Type == html.ElementNode && x.Data == "a" && hasClass(x, "title")
+		})
+		if a == nil {
+			return
+		}
+		u, title := attr(a, "href"), textContent(a)
+		if title == "" || !strings.HasPrefix(u, "http") {
+			return
+		}
+		snip := findFirst(n, func(x *html.Node) bool {
+			return x.Type == html.ElementNode && x.Data == "p" && hasClass(x, "s")
+		})
+		out = append(out, webResult{URL: u, Title: title, Content: textContent(snip), Engine: mojeekName})
+	})
+	return out
+}
+
 // normalizeURL is the dedupe key: lowercased host + path (trailing slash and
 // fragment dropped, query kept — distinct queries are distinct results).
 func normalizeURL(raw string) string {

apps/websearch/websearch.go

@@ -183,7 +183,7 @@ type webSearchQuery struct {
 // a third-party search API and never a search key. The enabled engines run
 // concurrently and their hits are merged, deduplicated by normalised URL (host
 // and path, trailing slash and fragment dropped, query kept, so distinct queries
-// stay distinct results) and capped at 20. Ranking is deterministic rather than
+// stay distinct results) and capped at 30. Ranking is deterministic rather than
 // scored: the first configured engine's hits lead.
 //
 // It fails SOFT on the engines. One that errors, times out or is served a
@@ -297,6 +297,13 @@ func Mount(app cloud.Router, deps cloud.Deps) error {
 		return fmt.Errorf("websearch.Mount: nil deps.Logger")
 	}
 	logger = logger.New("subsystem", "websearch")
+	// The package logs one thing and only one thing: an engine that went blind
+	// on a query another engine answered (outcome.go). Held here rather than
+	// threaded through metaSearch because the three doors into search — this
+	// subsystem's two handlers and compose.go's in-process caller — do not all
+	// have a logger to pass, and a search that must not run without one would be
+	// a worse trade than a warning that stays quiet in a library caller.
+	setLogger(logger)
 
 	// /v1/websearch/search admits a caller two ONE-WAY-equivalent ways, checked at
 	// the zip layer so the same request either reaches native meta-search or is
@@ -399,7 +406,7 @@ func init() {
 				"third-party search API and never a search key. The enabled engines run "+
 				"concurrently and their hits are merged, deduplicated by normalised URL (host and "+
 				"path, trailing slash and fragment dropped, query kept — distinct queries are "+
-				"distinct results) and capped at 20. Ranking is deterministic rather than scored: "+
+				"distinct results) and capped at 30. Ranking is deterministic rather than scored: "+
 				"the first configured engine's hits lead.\n\n"+
 
 				"TWO WAYS IN, one-way equivalent, and no third: a validated principal — the same "+

apps/websearch/websearch_test.go

@@ -6,6 +6,7 @@ import (
 	"io"
 	"net/http"
 	"net/http/httptest"
+	"slices"
 	"strings"
 	"testing"
 	"time"
@@ -309,3 +310,65 @@ func TestScrapeWrongKeyRejected(t *testing.T) {
 		t.Fatalf("status = %d, want 401", rec.Code)
 	}
 }
+
+// mojeekFixture is a Mojeek result page as the CLUSTER receives it: an <li> per
+// hit carrying <a class="title"> (the destination, verbatim — Mojeek uses no
+// click-tracking redirect) and <p class="s"> (the snippet).
+const mojeekFixture = `<html><body><ul class="results-standard">
+<li class="r1"><a title="https://example.com/one" href="https://example.com/one" class="ob"><p class="i"><span class="url">https://example.com</span></p></a><h2><a class="title" href="https://example.com/one">First Title</a></h2><p class="s">The <strong>first</strong> snippet.</p></li>
+<li class="r2 clu-result"><a title="https://example.com/two" href="https://example.com/two" class="ob"></a><h2><a class="title" href="https://example.com/two">Second Title</a></h2><p class="s">The second snippet.</p></li>
+</ul></body></html>`
+
+// Mojeek is the engine that carries the `site:` operator, which is how an X /
+// GitHub / Reddit scoped search reaches results at all — Bing answers a
+// site:x.com query with nothing. Measured from cluster egress 2026-08-07:
+// bing site:x.com → 0 results, mojeek site:x.com → 10.
+func TestParseMojeekReadsTitleURLAndSnippet(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/html")
+		_, _ = io.WriteString(w, mojeekFixture)
+	}))
+	t.Cleanup(srv.Close)
+	t.Setenv("WEBSEARCH_ENGINES", "mojeek")
+	t.Setenv("WEBSEARCH_MOJEEK_URL", srv.URL)
+
+	got := metaSearch(context.Background(), "anything", "")
+	if len(got.Results) != 2 {
+		t.Fatalf("results = %d, want 2 — parseMojeek must read every <li> hit: %+v", len(got.Results), got.Results)
+	}
+	r := got.Results[0]
+	if r.URL != "https://example.com/one" || r.Title != "First Title" || r.Engine != "mojeek" {
+		t.Fatalf("parsed = %+v, want the fixture's url/title stamped engine=mojeek", r)
+	}
+	if !strings.Contains(r.Content, "first") {
+		t.Fatalf("content = %q, want the <p class=\"s\"> snippet", r.Content)
+	}
+}
+
+// A deployment that configures NOTHING must still search more than one engine.
+// WEBSEARCH_ENGINES was unset in production, so the default WAS the whole engine
+// set, and it was bing alone — one engine, 10 results, and no `site:` support.
+func TestDefaultEnginesAreEveryEngineThatSurvivesDatacenterEgress(t *testing.T) {
+	t.Setenv("WEBSEARCH_ENGINES", "")
+	var names []string
+	for _, e := range enabledEngines() {
+		names = append(names, e.name)
+	}
+	if len(names) < 2 {
+		t.Fatalf("default engines = %v, want more than one — a single engine is a single point of failure and a single index", names)
+	}
+	// Every engine measured to ANSWER from cluster egress, by whichever fetch it
+	// takes. DDG belongs here on the rendered number, not the static one: served
+	// a captcha over plain HTTP ("Select all squares containing a duck", 0
+	// results) and 10 real results through the browser at the same URL, the same
+	// second. render.go escalates, so the engine answers.
+	//
+	// It is safe to default an engine that DEPENDS on the browser only because a
+	// browser-less deployment now reports it BLIND rather than dropping it
+	// quietly — see outcome.go. Without that this line would be optimism.
+	for _, want := range []string{bingName, ddgName, mojeekName} {
+		if !slices.Contains(names, want) {
+			t.Fatalf("default engines = %v, want %q among them", names, want)
+		}
+	}
+}

apps/websearch/zipdoc_gen.go

@@ -8,17 +8,21 @@ import (
 
 func init() {
 	zip.Describe("POST /v1/websearch", zip.Doc{
-		Description: "Searches the live web and answers with ranked results.\n\nThis is the fleet's path to what is happening RIGHT NOW — today's weather, an\noutage, a release that postdates any model's training. `q` is the query and\n`language` narrows it to a locale. The answer is `{query, number_of_results,\nresults:[{url, title, content, engine}]}`, where `content` is the ENGINE's\nsnippet and not the page: read a page with POST /v1/crawl.\n\nIt is served in-process by a Go meta-search over keyless public engines — never\na third-party search API and never a search key. The enabled engines run\nconcurrently and their hits are merged, deduplicated by normalised URL (host\nand path, trailing slash and fragment dropped, query kept, so distinct queries\nstay distinct results) and capped at 20. Ranking is deterministic rather than\nscored: the first configured engine's hits lead.\n\nIt fails SOFT on the engines. One that errors, times out or is served a\nbot-challenge page contributes zero results and never fails the call, so an\nempty `results` is a real answer — nothing was found — and not an outage. The\narray is always present, never null.\n\nA VALIDATED PRINCIPAL IS REQUIRED, and there is no tenant beyond that: the\nresults are public web pages, identical for every caller, so nothing here is\nscoped and nothing here can leak across orgs. A typed op is also an MCP tool\nand a CLI command, and tools/call invokes it with no route and therefore no\nmiddleware — so the gate is in the handler, where every door reaches it, rather\nthan in a middleware only one door passes through.",
+		Description: "Searches the live web and answers with ranked results.\n\nThis is the fleet's path to what is happening RIGHT NOW — today's weather, an\noutage, a release that postdates any model's training. `q` is the query and\n`language` narrows it to a locale. The answer is `{query, number_of_results,\nresults:[{url, title, content, engine}]}`, where `content` is the ENGINE's\nsnippet and not the page: read a page with POST /v1/crawl.\n\nIt is served in-process by a Go meta-search over keyless public engines — never\na third-party search API and never a search key. The enabled engines run\nconcurrently and their hits are merged, deduplicated by normalised URL (host\nand path, trailing slash and fragment dropped, query kept, so distinct queries\nstay distinct results) and capped at 30. Ranking is deterministic rather than\nscored: the first configured engine's hits lead.\n\nIt fails SOFT on the engines. One that errors, times out or is served a\nbot-challenge page contributes zero results and never fails the call, so an\nempty `results` is a real answer — nothing was found — and not an outage. The\narray is always present, never null.\n\nA VALIDATED PRINCIPAL IS REQUIRED, and there is no tenant beyond that: the\nresults are public web pages, identical for every caller, so nothing here is\nscoped and nothing here can leak across orgs. A typed op is also an MCP tool\nand a CLI command, and tools/call invokes it with no route and therefore no\nmiddleware — so the gate is in the handler, where every door reaches it, rather\nthan in a middleware only one door passes through.",
 		Fields: map[string]string{
+			"webEngine.name":                     "Name is the engine, matching the `engine` stamped on each result.",
+			"webEngine.outcome":                  "Outcome is \"answered\", \"blind\" or \"failed\" — see outcome.go. \"blind\" means\nthe page came back and no results could be read out of it.",
+			"webEngine.results":                  "Results is how many hits this engine contributed, before the merge\ndeduplicated them against the others.",
 			"webResult.content":                  "Content is the ENGINE's snippet — the few lines shown under the title, not\nthe page's text. Read the page itself with POST /v1/crawl.",
 			"webResult.engine":                   "Engine names the backend that found this hit, so one engine's view of a\nquery can be told from another's.",
 			"webResult.title":                    "Title is the page's title.",
 			"webResult.url":                      "URL is the page's address, as the engine reported it.",
 			"webSearchQuery.language":            "Language narrows the engines to a locale, BCP-47-ish (\"en\", \"ja\", \"de\").\nEmpty means no narrowing.",
 			"webSearchQuery.q":                   "Q is the query. Required — an empty one is refused rather than answered\nwith the whole web.",
+			"webSearchResults.engines":           "Engines is one entry per engine asked, in the order they were asked. It is\nADDITIVE to the SearXNG contract, which the LibreChat client ignores as an\nunknown field exactly as it ignores `engine` on a result.",
 			"webSearchResults.number_of_results": "NumberOfResults is len(results) — what this answer carries, never an\nestimate of what the web holds.",
 			"webSearchResults.query":             "Query is the query that ran, echoed back.",
-			"webSearchResults.results":           "Results are the merged hits, deduplicated by normalised URL and capped at\n20. Always an array and never null: no hits is an ANSWER, not a fault.",
+			"webSearchResults.results":           "Results are the merged hits, deduplicated by normalised URL and capped at\n30. Always an array and never null: no hits is an ANSWER, not a fault.",
 		},
 	})
 }

audit/redact.go

@@ -25,6 +25,7 @@ package audit
 
 import (
 	"encoding/json"
+	"regexp"
 	"strings"
 )
 
@@ -74,6 +75,68 @@ var secretKeyParts = []string{
 	"encryption_key",
 }
 
+// urlCredential matches the userinfo of a URL — the "user:password@" between the
+// scheme and the host.
+//
+// This is the one credential the key denylist above cannot see, because it is in
+// the VALUE and the key naming it is innocent: a clone URL arrives as
+// {"cloneUrl": "https://x-access-token:<token>@github.com/org/repo"} (the shape
+// apps/coding builds to check a repo out), and "cloneUrl" matches nothing in
+// secretKeyParts. Every one of those characters would otherwise be recorded
+// verbatim.
+//
+// Stripping on the value alone is safe HERE and nowhere else in this file,
+// because userinfo in a URL is a credential BY CONSTRUCTION (RFC 3986 3.2.1) —
+// a structural fact, not a guess that a string "looks secret". The deliberate
+// choice this file already documents, to match key names rather than sniff
+// values, is unchanged: this adds one structural rule, not a heuristic.
+var urlCredential = regexp.MustCompile(`([a-zA-Z][a-zA-Z0-9+.\-]*://)([^/?#\s@]+)@`)
+
+// stripURLCredential replaces the secret half of any URL userinfo in s.
+//
+// The USER is kept and only the password is replaced, because the user names how
+// the call authenticated ("x-access-token", "oauth2") and is worth reading in a
+// trace, while the part after the colon is the credential. Userinfo with no colon
+// IS the token — nothing there is a name — so it goes whole.
+func stripURLCredential(s string) string {
+	if !strings.Contains(s, "@") {
+		return s // no userinfo is possible; skip the scan
+	}
+	return urlCredential.ReplaceAllStringFunc(s, func(m string) string {
+		g := urlCredential.FindStringSubmatch(m)
+		scheme, userinfo := g[1], g[2]
+		if user, _, found := strings.Cut(userinfo, ":"); found {
+			return scheme + user + ":" + redactedMarker + "@"
+		}
+		return scheme + redactedMarker + "@"
+	})
+}
+
+// RedactText returns s with credentials removed, whether or not s is JSON.
+//
+// It exists because not everything worth recording is a structured diff. A tool
+// call's ARGUMENTS are a JSON object and get the full treatment — the secret-key
+// denylist plus the URL strip. A tool's RESULT is whatever the tool returned,
+// commonly prose or a log tail, and running that through Redact would hand back
+// the fail-closed marker for the whole thing: correct for a mutation diff that
+// must parse, useless for text that was never meant to.
+//
+// So the shape decides the pass. JSON gets both; text gets the structural URL
+// strip alone, which is the only rule that can be applied to a value with no keys
+// to judge. Text is therefore redacted LESS than JSON, and that is a real limit
+// rather than an oversight: a credential written into prose by the tool that
+// returned it ("your token is X") is not something key-name matching can see.
+// Callers that can supply structure should.
+func RedactText(s string) string {
+	if s == "" {
+		return s
+	}
+	if json.Valid([]byte(s)) {
+		return string(Redact(json.RawMessage(s)))
+	}
+	return stripURLCredential(s)
+}
+
 // isSecretKey reports whether a JSON key names a credential-bearing field.
 func isSecretKey(key string) bool {
 	k := strings.ToLower(key)
@@ -134,6 +197,9 @@ func redactValue(key string, v any) any {
 			out[i] = redactValue("", val) // array elements inherit no key
 		}
 		return out
+	case string:
+		// The one credential a key name cannot reveal (see urlCredential).
+		return stripURLCredential(t)
 	default:
 		return v
 	}

audit/redact_url_test.go

@@ -0,0 +1,58 @@
+package audit
+
+import (
+	"encoding/json"
+	"strings"
+	"testing"
+)
+
+// TestRedact_StripsCredentialInAURL covers the credential the key denylist cannot
+// see: one embedded in a URL VALUE, under a key whose name says nothing.
+//
+// This is the shape apps/coding builds to clone a repo, and it is the exact
+// argument an agent's tool call carries into a trace. Recording it verbatim would
+// put a live token in the span store, where it outlives the sandbox that used it.
+func TestRedact_StripsCredentialInAURL(t *testing.T) {
+	in := json.RawMessage(`{
+      "cloneUrl": "https://x-access-token:ghp_LIVE_TOKEN_VALUE@github.com/acme/repo",
+      "mirror":   "https://ghp_BARE_TOKEN@github.com/acme/repo",
+      "endpoint": "https://api.example.com/v1/things",
+      "note":     "cloned from https://user:hunter2@git.example.com/x",
+      "nested":   {"repo": {"url": "git+ssh://deploy:s3cr3t@git.example.com/y"}},
+      "list":     ["https://u:p4ss@h.example.com/z"]
+    }`)
+	out := Redact(in)
+	s := string(out)
+
+	for _, leak := range []string{"ghp_LIVE_TOKEN_VALUE", "ghp_BARE_TOKEN", "hunter2", "s3cr3t", "p4ss"} {
+		if strings.Contains(s, leak) {
+			t.Fatalf("credential %q survived redaction: %s", leak, s)
+		}
+	}
+	// The USER half is kept — it names how the call authenticated and is not the
+	// secret — and so is every URL that carried no credential at all.
+	for _, keep := range []string{"x-access-token", "github.com/acme/repo", "https://api.example.com/v1/things", "deploy"} {
+		if !strings.Contains(s, keep) {
+			t.Fatalf("redaction dropped non-secret detail %q: %s", keep, s)
+		}
+	}
+	if !strings.Contains(s, redactedMarker) {
+		t.Fatalf("no redaction marker in output: %s", s)
+	}
+}
+
+// TestStripURLCredential_LeavesOrdinaryTextAlone: the rule is structural, so text
+// that merely contains an "@" is not a credential and must survive unchanged.
+func TestStripURLCredential_LeavesOrdinaryTextAlone(t *testing.T) {
+	for _, s := range []string{
+		"mail z@hanzo.ai about it",
+		"https://api.example.com/v1/things",
+		"@hanzo what is the status",
+		"",
+		"user@host without a scheme",
+	} {
+		if got := stripURLCredential(s); got != s {
+			t.Fatalf("stripURLCredential(%q) = %q, want it unchanged", s, got)
+		}
+	}
+}

clients/aihttp.go

@@ -248,6 +248,22 @@ func (a *httpAI) ChatCompletion(ctx context.Context, req *types.ChatRequest) (*t
 		return nil, fmt.Errorf("cloud: chat completion (model %q): upstream returned no choices: %w", model, types.ErrUpstreamBusy)
 	}
 	choice := resp.Choices[0]
+	// WHY the model stopped, which is the difference between an answer and a
+	// sentence that ran out of room. It was read off the wire and returned to the
+	// caller from the first day and recorded nowhere, so a truncated reply
+	// ("length") and a finished one ("stop") were the same span, and a run that
+	// ended mid-tool-call was indistinguishable from one that chose to stop. The
+	// convention spells it as a list because a multi-choice response has one per
+	// choice; this client reads choice 0, so the list has one element and says so
+	// rather than flattening to a scalar the reader would have to special-case.
+	if fr := string(choice.FinishReason); fr != "" {
+		span.SetAttributes(attribute.StringSlice("gen_ai.response.finish_reasons", []string{fr}))
+	}
+	if marker := unparsed(choice.Message.Content); marker != "" {
+		span.SetAttributes(attribute.String("hanzo.ai.unparsed", marker))
+		span.SetStatus(codes.Error, "unparsed tool call")
+		return nil, fmt.Errorf("cloud: chat completion (model %q): upstream returned a tool call it had not parsed: %w", model, types.ErrUpstreamBusy)
+	}
 	return &types.ChatResponse{
 		Content:          choice.Message.Content,
 		PromptTokens:     resp.Usage.PromptTokens,
@@ -317,6 +333,41 @@ func wireTools(defs []types.ToolDef) []openai.Tool {
 	return out
 }
 
+// unparsed names the tool-call markup a completion's CONTENT carries, or "" for
+// content that carries none.
+//
+// These strings are SPECIAL TOKENS. A model emits them so the stack serving it
+// can turn them into structured tool_calls; arriving here as prose means that
+// stack did not, and what we hold is serialization internals rather than an
+// answer. The caller refuses the completion, so the one thing that must never
+// happen — pasting a model's own wire format to the person waiting on a reply —
+// cannot.
+//
+// It RECOGNISES and does not read. Lifting the call out of the text would make
+// this a second tool-call parser, divergent from every real one by construction
+// and stale the first time a family changed its syntax. The parse belongs to the
+// stack that owns the model; this is only the net under it, and a net that
+// started interpreting would quietly become the thing it is catching.
+//
+// The set is the families this gateway routes to, each token taken from that
+// family's own chat template.
+func unparsed(content string) string {
+	for _, marker := range []string{
+		"<｜DSML｜tool_calls>",   // U+FF5C — deepseek's agentic markup
+		"<｜tool▁call▁begin｜>",  // U+FF5C, U+2581 — deepseek's fenced form
+		"<｜tool▁calls▁begin｜>", //   …and its plural opener
+		"<tool_call>",          // qwen / hermes
+		"[TOOL_CALLS]",         // mistral
+		"<|python_tag|>",       // llama
+		"<|tool_call>",         // gemma
+	} {
+		if strings.Contains(content, marker) {
+			return marker
+		}
+	}
+	return ""
+}
+
 // readToolCalls lifts the model's tool calls off a choice. Only function calls
 // are carried: they are the only kind this gateway serves, and a call of some
 // other type has no arguments this side could dispatch.
@@ -378,6 +429,13 @@ func (a *httpAI) ChatStream(ctx context.Context, req *types.ChatRequest, emit fu
 
 	var content strings.Builder
 	out := &types.ChatResponse{}
+	// What the STREAM says about itself, accumulated as it arrives. Both facts
+	// come in frames the delta loop below otherwise skips — the model on any
+	// frame, the finish reason on a terminal one that carries no text — so they
+	// are read before the "no content, move on" shortcuts rather than after.
+	// Falling back to the requested model is honest: a gateway that never names
+	// one has not told us it served something else.
+	streamModel := model
 	for {
 		frame, err := stream.Recv()
 		if errors.Is(err, io.EOF) {
@@ -399,9 +457,18 @@ func (a *httpAI) ChatStream(ctx context.Context, req *types.ChatRequest, emit fu
 		if u := frame.Usage; u != nil {
 			out.PromptTokens, out.CompletionTokens, out.TotalTokens = u.PromptTokens, u.CompletionTokens, u.TotalTokens
 		}
+		if m := strings.TrimSpace(frame.Model); m != "" {
+			streamModel = m
+		}
 		if len(frame.Choices) == 0 {
 			continue // usage-only terminal frame
 		}
+		// The stop reason rides a frame whose delta is empty, which the shortcut
+		// below skips — so it is read here or not at all. Without it a stream that
+		// was CUT at the token ceiling returned exactly like one that finished.
+		if fr := strings.TrimSpace(string(frame.Choices[0].FinishReason)); fr != "" {
+			out.FinishReason = fr
+		}
 		delta := frame.Choices[0].Delta.Content
 		if delta == "" {
 			continue
@@ -421,10 +488,30 @@ func (a *httpAI) ChatStream(ctx context.Context, req *types.ChatRequest, emit fu
 		span.SetStatus(codes.Error, "no content")
 		return nil, fmt.Errorf("cloud: chat stream (model %q): upstream returned no content: %w", model, types.ErrUpstreamBusy)
 	}
+	// The same refusal as ChatCompletion, at the other exit of the same boundary:
+	// streaming is a delivery property, so a completion that is internals rather
+	// than an answer is one here too. The deltas have already been emitted, which
+	// is the honest limit of a net placed after the fact — returning the error
+	// still stops this text from being stored and replayed as the run's answer.
+	if marker := unparsed(out.Content); marker != "" {
+		span.SetAttributes(attribute.String("hanzo.ai.unparsed", marker))
+		span.SetStatus(codes.Error, "unparsed tool call")
+		return nil, fmt.Errorf("cloud: chat stream (model %q): upstream returned a tool call it had not parsed: %w", model, types.ErrUpstreamBusy)
+	}
 	span.SetAttributes(
 		attribute.Int("gen_ai.usage.input_tokens", out.PromptTokens),
 		attribute.Int("gen_ai.usage.output_tokens", out.CompletionTokens),
+		// The model that ANSWERED, on the streaming path too. The buffered path has
+		// always recorded it; a stream did not, so the one shape a reader uses to
+		// tell "the model I asked for" from "the model that served me" was missing
+		// from exactly the calls a user watches arrive. streamModel falls back to
+		// the requested name when the frames never name one, which is honest: a
+		// gateway that does not say is not a gateway that served something else.
+		attribute.String("gen_ai.response.model", streamModel),
 	)
+	if out.FinishReason != "" {
+		span.SetAttributes(attribute.StringSlice("gen_ai.response.finish_reasons", []string{out.FinishReason}))
+	}
 	return out, nil
 }
 

clients/aihttp_test.go

@@ -236,6 +236,94 @@ func TestAIHTTP_EmptyChoices(t *testing.T) {
 	}
 }
 
+// TestAIHTTP_UnparsedCallRefused pins the SAFETY NET, not the fix: when an
+// upstream fails to turn a model's native tool-call tokens into structured
+// tool_calls and hands them back as prose, that text is serialization internals
+// and must never reach the person waiting on the answer.
+//
+// The bodies here are verbatim shapes from the four families this gateway routes
+// to. Each carries a real tool call the upstream did not parse, so tool_calls is
+// empty and the loop would otherwise return the markup as the assistant's reply —
+// which is exactly how a Slack turn came to print one.
+//
+// Two things are asserted, and the second matters as much as the first: the
+// completion is REFUSED, and the refusal itself carries none of the markup. An
+// error that quoted the bytes would leak the same internals by a shorter path.
+// The refusal is tagged ErrUpstreamBusy because an upstream that drops its own
+// parse is a fault a retry or a failover can actually clear.
+func TestAIHTTP_UnparsedCallRefused(t *testing.T) {
+	for _, tc := range []struct {
+		family  string
+		content string
+	}{
+		{"deepseek dsml", "<｜DSML｜tool_calls>\n<｜DSML｜invoke name=\"websearch\">\n" +
+			"<｜DSML｜parameter name=\"op\" string=\"true\">search_web</｜DSML｜parameter>\n" +
+			"</｜DSML｜invoke>\n</｜DSML｜tool_calls>"},
+		{"deepseek native", "<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>websearch\n" +
+			"```json\n{\"op\":\"search_web\"}\n```<｜tool▁call▁end｜>"},
+		{"qwen hermes", "<tool_call>\n{\"name\":\"websearch\",\"arguments\":{\"op\":\"search_web\"}}\n</tool_call>"},
+		{"mistral", "[TOOL_CALLS][{\"name\":\"websearch\",\"arguments\":{\"op\":\"search_web\"}}]"},
+		{"llama", "<|python_tag|>{\"name\":\"websearch\",\"parameters\":{\"op\":\"search_web\"}}"},
+	} {
+		t.Run(tc.family, func(t *testing.T) {
+			srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+				w.Header().Set("Content-Type", "application/json")
+				_ = json.NewEncoder(w).Encode(map[string]any{
+					"id": "chatcmpl-x", "object": "chat.completion", "model": "enso",
+					"choices": []map[string]any{{
+						"index":         0,
+						"message":       map[string]string{"role": "assistant", "content": tc.content},
+						"finish_reason": "stop",
+					}},
+				})
+			}))
+			defer srv.Close()
+
+			got, err := AIHTTPAt(srv.URL, "sk-test", "enso").
+				ChatCompletion(context.Background(), &types.ChatRequest{Prompt: "latest on x.com?"})
+			if err == nil {
+				t.Fatalf("an unparsed tool call was returned as an answer: %q", got.Content)
+			}
+			if !errors.Is(err, types.ErrUpstreamBusy) {
+				t.Errorf("want ErrUpstreamBusy so the runner retries/fails over, got: %v", err)
+			}
+			for _, leak := range []string{"DSML", "｜", "tool▁", "<tool_call>", "[TOOL_CALLS]", "<|python_tag|>"} {
+				if strings.Contains(err.Error(), leak) {
+					t.Errorf("the refusal quotes the markup it exists to withhold (%q): %v", leak, err)
+				}
+			}
+		})
+	}
+}
+
+// TestAIHTTP_ToolCallsKept is the control for the refusal above: a completion the
+// upstream DID parse is untouched. Structured tool_calls are read off the choice,
+// and prose that merely mentions a tool is prose — the net recognises markup, and
+// recognising it is all it does.
+func TestAIHTTP_ToolCallsKept(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		w.Header().Set("Content-Type", "application/json")
+		_, _ = w.Write([]byte(`{"id":"x","object":"chat.completion","choices":[{"index":0,
+			"message":{"role":"assistant","content":"Let me look that up with the websearch tool.",
+			"tool_calls":[{"id":"call_1","type":"function",
+			"function":{"name":"websearch","arguments":"{\"op\":\"search_web\"}"}}]},
+			"finish_reason":"tool_calls"}]}`))
+	}))
+	defer srv.Close()
+
+	got, err := AIHTTPAt(srv.URL, "sk-test", "enso").
+		ChatCompletion(context.Background(), &types.ChatRequest{Prompt: "latest on x.com?"})
+	if err != nil {
+		t.Fatalf("a parsed tool call was refused: %v", err)
+	}
+	if len(got.ToolCalls) != 1 || got.ToolCalls[0].Name != "websearch" {
+		t.Fatalf("tool calls lost: %+v", got.ToolCalls)
+	}
+	if got.FinishReason != "tool_calls" {
+		t.Errorf("finish reason: got %q want tool_calls", got.FinishReason)
+	}
+}
+
 // TestAIHTTP_Models asserts httpAI implements types.ModelLister: it GETs the
 // gateway's OpenAI-compatible /models list (Bearer-authenticated) and returns the
 // served model ids — the catalog the agents subsystem validates a model against.