rescue: in-flight lane work, saved when the session limit killed every agent

Committed verbatim, unreviewed and unverified. Original branch: feat/plugin-solo-builds.
Saved because uncommitted work on no ref is one checkout from gone — that
happened once already today to 463 lines of buildkitd cutover.

.dockerignore

@@ -11,8 +11,22 @@
 LICENSE
 SECURITY.md
 
-# Already-built binary at repo root (matches .gitignore)
+# Already-built binaries at repo root (matches .gitignore). These are local
+# host-arch builds; the image builds its own amd64 ones in the build stage, so
+# none of them can reach the final image. They still cost: the context is
+# uploaded to BuildKit on every build, and `COPY . .` cache-keys on all of it, so
+# touching any one of them invalidates the layer and forces a full re-link of all
+# 121 plugins. Measured 2026-08-06: 1084 MB of context, ~888 MB of it these.
 /cloud
+/hanzo
+/o11y
+/host
+/boxd
+/sandboxes
+
+# Agent worktrees and caches — 189 MB, and it holds its own copies of the
+# binaries above (the /bin/ rule below is root-anchored and misses them).
+.claude/
 
 # Environment files (never bake secrets into images)
 .env

LLM.md

@@ -788,6 +788,211 @@ embedded underneath.
   point: on a defer-encryption-to-checkpoint backend it MUST route through the
   driver's re-encrypting Checkpoint so ship-before-ack reads FRESH ciphertext (P5).
 
+## The rollout, measured — and the four things the map got wrong
+
+Measured 2026-08-06 against the live deployment (`ghcr.io/hanzoai/cloud:sha-827f3e669243`,
+digest `sha256:639e45a8…aada`). Numbers here are measurements, not estimates; re-measure
+rather than trust them after any of the three levers below moves.
+
+### Where the deploy outage actually goes
+
+A values-only roll — same image digest, so `IfNotPresent` pulls nothing — costs
+**27 seconds** of edge 503 (first 503 12:13:39, first 200 12:14:06; new pod Ready 22s
+after container start). A deploy that moves the digest costs ~2 minutes, and the
+difference is almost entirely the **~88s pull of a 1.6 GB image**.
+
+So the pull is roughly three quarters of a deploy, and everything else — terminate,
+boot, bind — is the 27s floor. Shrinking the image is the whole game; probes and
+workload kind cannot touch that 88s.
+
+### The image is 121 binaries, and it is already stripped
+
+| layer | compressed | what made it |
+|---|---|---|
+| 1–6 | 10.7 MB | alpine 3.22.5 rootfs, apk (ca-certs, tzdata, sqlcipher-libs, tini), certs, zoneinfo, passwd/group |
+| 7 | 5.41 MB | `COPY /cloud` — the router |
+| 8 | 2.54 MB | `COPY /smoke` |
+| **9** | **1524.87 MB** | **`COPY /plugins/ /` — 121 CGO plugin binaries** |
+
+**1.618 GB compressed, ~4.09 GiB on disk. Layer 9 is 98.8% of it.** 121 binaries,
+mean 33.59 MB, each linking its own private copy of a largely identical dependency
+graph (600–2200 packages each). No sharing between them. That is the direct cost of
+the plugin-per-binary topology: the mega-link went away and 121 medium links replaced
+it with no dedup.
+
+The obvious levers are already pulled and are not worth re-examining:
+`-ldflags="-s -w"` is applied uniformly (Dockerfile:259), so there is no strip win;
+the base + OS is 22.8 MB, so there is no distroless win; `/cloud` itself is 16.1 MB.
+
+**The trap for anyone about to split layer 9 per plugin.** `GO_LDFLAGS` carries
+`-X …revision=${REVISION}` and Dockerfile:336 applies it to every plugin (asserted at
+:361 against `/plugins/base`). Every plugin binary therefore changes on every commit,
+so 121 separate layers would all change on every build and nothing would be saved.
+Splitting the layer only pays **after** the revision stamp leaves the plugins — stamp
+`/cloud` alone and let children read it from the environment the router already stamps
+(`stampAndScrub`), which also makes revision one runtime value instead of 122
+compile-time constants. Even then it pays only for app-local changes: a commit to the
+root package relinks all 121 regardless. Group into ~16 layers, not 121 — overlayfs
+caps lower dirs near 128 and 121 + 8 exceeds it.
+
+### The build context was 13x larger than the source
+
+`.dockerignore` excluded `/cloud` and nothing else, so the context carried
+`hanzo` 585.6M, `o11y` 125.5M, `sandboxes` 49.8M, `host` 25.6M, `boxd` 13.9M and
+`.claude/` 189M — all host-arch (aarch64) builds that cannot reach an amd64 image.
+None of it changed the image; all of it cache-keyed `COPY . .`, so touching any one
+forced a full relink of all 121 plugins. Excluding them: **1417 MB → 109 MB**.
+
+`sandboxes` is **tracked in git** — it ships to everyone who clones. Removing it from
+the index is still owed.
+
+### The DataDir is mostly a git forge, and the root databases are dead
+
+30.5 GB, and the shape is not what the file names suggest:
+
+| | size | |
+|---|---|---|
+| `git/` | **22.2 GB (73%)** | 365 bare repos backing git.hanzo.ai (`apps/git/git.go:214`), 94% packfiles |
+| `orgs/` | 5.4 GB | 642 dbs, 342 orgs — the live tree |
+| `tasks/` | 1.1 GB | dominated by `tasks/hanzo/gpu-jobs.db` at 729 MB |
+| root `*.db` | 1.33 GB | 167 files — **cold** |
+
+**The root-level databases are pre-migration leftovers.** `orgdb.go:54-56` routes the
+`system` namespace to `{DataDir}/orgs/_platform/{subsystem}.db`, not to the root. The
+files confirm it: root `audit.db` (1013 MB) last written **Jul 30 14:03** and root
+`audit-tasks.db` (234 MB) **Aug 2 04:42**, while `orgs/_platform/audit-tasks.db`
+(300 MB) was written **Aug 6 18:58** — minutes before the sample.
+
+This inverts the received picture of the multi-writer blocker. "Audit is 78% of shared
+cross-org state" describes a **dead file**. The live audit chain lives inside `orgs/`,
+where `_platform` is just another org slug — which is the tree the per-org HA path
+above already governs. Once audit is set aside, what remains at root is **7.4 MB across
+50 files**, the largest being `agents.db` at 2.23 MB (and `apps/agents/legacy.go:41`
+marks it legacy). The residual is a rounding error, and it is cold on top of that.
+
+The real co-tenancy problem is `git/`: 22.2 GB of bare repos sharing the API writer's
+RWO volume. It is 73% of the volume, it is not SQLite, and it drags an unrelated
+failure domain into every HA decision cloud makes. Separating the forge onto its own
+claim takes the writer's volume from 30.5 GB to ~8 GB.
+
+### Four corrections to the HA model
+
+- **The writer lease is armed, not empty.** `internal/writerlease`'s doc comment
+  describes the 2026-08-04 repair as having left "an interlock that reads as armed and
+  holds nothing" — that is history. `cmd/cloud/main.go:169` now takes it at the router
+  root before it spawns anything, and `serve.go:117` makes the same call and Inherits
+  under a router. Present on `forge/main` and `inc2/main`; **`origin/main` (github
+  hanzoai/cloud) lacks the package entirely** — that lineage is stale, do not read the
+  design from it.
+- **The k8s Lease elector is written but not linked.** `writerpin.Resolve` can only
+  return `LeasePin` if something calls `UseElector`, and nothing outside
+  `writerpin_test.go` ever does. So `CLOUD_WRITER_LEASE=1` today arms the **flock only**
+  and silently reports `registered no elector`. One env var drives two unrelated
+  mechanisms; that braid is worth cutting before either is turned on.
+- **A PDB already exists** (`pdb.enabled: true, maxUnavailable: 1`). With one replica it
+  permits the eviction rather than preventing it, which is the only honest setting — a
+  single writer cannot be made drain-proof by a budget.
+- **There is one schedulable node.** `nodeSelector: worker-xl` and that pool holds
+  exactly one node, so "spread the replicas" is not available and the node-partition
+  case a StatefulSet defends against cannot currently occur.
+
+### Deployment → StatefulSet: correct, and three traps in the way
+
+`role/role.go` says the writer *is* a StatefulSet and it is deployed as a Deployment.
+Converting strengthens at-most-one: a StatefulSet refuses to create the replacement
+until the old pod is confirmed gone, where a Deployment on an unreachable node can
+briefly run two. In `charts/app` it is one values key — `workload.kind: StatefulSet`,
+no chart edit. Three things ride along, and two of them break the workload:
+
+1. **`strategy: Recreate` becomes `updateStrategy: OnDelete`** (`workload.yaml:40-41`),
+   so the pod never rolls on an image change and CD reports Synced while running the old
+   image. Set `strategy: RollingUpdate` instead. For a StatefulSet the chart emits **no**
+   `rollingUpdate` block at all, so there is no surge to express: at `replicas: 1` it is
+   strictly terminate-then-create. StatefulSet RollingUpdate and Deployment RollingUpdate
+   share a word and are not the same thing — the maxSurge trap `writerlease.go` warns
+   about is the Deployment one.
+2. **A phantom claim, and a name collision.** Under StatefulSet the chart always emits
+   `volumeClaimTemplates`, synthesizing one from `persistence.*` defaults when none is
+   set — named `data`, which is the name `cloud.yaml` already uses for the volume that
+   carries `cloud-api-data`. Duplicate volume names are rejected, so the pods would not
+   start. `volumeClaimTemplates` is immutable afterwards. Fix in the chart: emit the
+   block only when `persistence.enabled` or `volumeClaimTemplates` is set — verified
+   zero blast radius, since all seven existing StatefulSets set one or the other.
+3. **`prune: false` means the Deployment survives the cutover.** CD creates the
+   StatefulSet and leaves the Deployment running; both match the Service selector, both
+   land on the one eligible node, and RWO is per-node — so **both mount the volume and
+   both open the stores.** That is the corruption case, reached by a routine sync. The
+   cutover must be ordered by hand: delete the Deployment, confirm zero pods, then let
+   CD create the StatefulSet. It is not a drive-by values change.
+
+### A reader is the front door, not a second copy of the data
+
+Say this plainly, because "HA reads" oversells it. `CLOUD_ROLE=reader` opens **no
+stores** and reverse-proxies **every** request to the writer (`reader_proxy.go`). It is
+not a read replica and does not intend to be: the KMS ZapDB store cannot be opened
+read-only while the writer is live (`clients/kms.TestConcurrentOpen_LiveWriterStore-
+IsNotROShareable` — Badger's RO open replays the live memtable WAL and refuses to
+truncate), so rather than braid a partial local replica that must route KMS and every
+audited verb back to the writer anyway, the reader forwards everything.
+
+What that buys is real but narrow: readers roll `RollingUpdate` with no PVC, so the edge
+always has a ready endpoint, and the retry transport re-sends **dial failures only**
+(the request never reached the writer, so a non-idempotent POST cannot double-execute)
+across the writer's handoff. Clients see latency instead of a refused connection. What
+it does not buy: any read scale-out, any data redundancy, any survival of writer loss.
+Every byte still comes from the one writer.
+
+**And it delivers nothing at today's numbers.** `CLOUD_READER_RETRY_BUDGET` defaults to
+**25s** (`config.go:331`) and the writer's digest-changing roll is ~120s. The budget is
+blown four times over and the reader returns 502 — the same outage with an extra hop.
+Raising the budget to 120s is not the fix either: that holds every client connection for
+two minutes and trades 503s for timeouts and a connection pile-up. **Tier 1's image
+shrink is a hard prerequisite for readers to be worth deploying at all.** Get the writer
+gap under ~25s first, then readers convert it to latency.
+
+One more thing readers do not absorb: a writer that is up but has not finished mounting
+returns a real 503, and the retry transport does not retry 503s — only dial failures. So
+the mount window passes straight through the proxy.
+
+### HA writes: the blocker moved, and the mechanism already exists
+
+The per-org path is already HA and already proven — `internal/org` + `hanzoai/ha` (HRW
+election + monotone fencing round) + `vfs` FencedStore, with live K8s membership
+(`apps/membership/k8s.go`, `CLOUD_PEER_SELECTOR`) rather than the static `CLOUD_PEERS`
+list. `TestRollingUpgradeZeroDowntime` rolls 3 pods over 8 orgs with continuous writes
+and asserts zero lost acked writes and zero split-brain; it passes in 10.6s.
+
+Because `system` resolves to `orgs/_platform/`, **the live audit chain is already inside
+that tree**. `_platform` is an org slug like any other, so HRW pins it to exactly one
+owner pod and the "single serializing writer owning the chain head" property holds by
+construction — which is the property the whole design exists to protect. The audit store
+is separately proven shareable by a live read-only second opener
+(`audit/shareability_probe_test.go`), and a fresh RW open recovers the head from disk
+with no gap and no fork, so ownership transfer is safe.
+
+So the three options are not equally weighted, and the recommendation is not a menu:
+
+- **Per-pod audit segments merged on read** — rejected. It destroys the one expensive
+  property: a single gapless total order. N segments is N orders, and reconstructing
+  "which came first" across them is itself a consensus problem, now embedded in the
+  compliance control.
+- **ConsensusPin** — not needed for this, and `writerpin.go` already records why not to
+  hand-roll it (a Raft-style vote over the same peer set took four rounds of real safety
+  bugs and still produced two simultaneous holders). It is the answer for a pin that must
+  span clusters, which is a different question from this one.
+- **One owner, everyone else asks — RECOMMENDED.** This is not a new mechanism; it is the
+  law already written down under "After the split: a store has one owner, and everyone
+  else asks", carried over `plane/` typed ops. Audit is simply a store that has not been
+  split yet. It preserves the chain head exactly, and it preserves the fail-closed
+  semantics unchanged: an append error fails the request closed today, and an unreachable
+  audit owner fails it closed identically. Five subsystems have already been through this
+  transition; the three rules that fell out of it apply here unmodified.
+
+The honest residual, since it has now been enumerated: **7.4 MB across 50 root-level
+databases, all cold since Aug 2.** They are not a blocker; they are cleanup. The work
+that actually remains for multi-writer is (a) splitting the 22.2 GB git forge off the
+writer's volume, and (b) publishing audit's append from its owner over the plane.
+
 ## The route table has three projections, and the router is the source
 
 `serve.go` composes ONE route table and projects it three ways, all after