mod/LLM.md

mod — working notes

The Go module proxy. Read README.md first for what it is and how it is configured; this file holds the decisions behind it.

What it is for, after the OSS decision

This was first built to keep a GitHub credential out of builds that needed private modules. The estate is going fully open source, so that problem is ending rather than being solved, and the justification narrows to the two reasons that never depended on privacy:

• Authority — the forge answers for our module names, so a build does not depend on github.com being up, willing, or still holding the repository.
• Retention — one cache, nothing prunes it, so a module stays reachable because we decided it does.

github.com/hanzoai/s3-go is the whole argument in one line: 404 on GitHub, alive on the forge, and our builds only kept working because a third-party cache happened to still hold it.

Read anything below as serving those two. Nothing here is a privacy mechanism.

Why a proxy rather than the alternatives

1. A proxy we own — this. The module path is answered by us and read from the forge. One change in the client and module identity is untouched.
2. Migrate the module paths to a git.hanzo.ai host. Cleanest in principle, and the blast radius is why it is not the answer: 190 module definitions, 124 dependent go.mod files, ~21,500 importing .go files in the hanzo tree alone, plus 33 required paths not even checked out locally.
3. A machine credential at GitHub. Keeps the dependency, only changes whose name is on it. It does nothing for authority and nothing for retention.

The forge already had half of (1): routers/api/packages/goproxy in hanzoai/git is a read-through cache of the public ecosystem and it is live. It cannot be the build-time proxy, and the reason is worth keeping: it is a registry of what has been PUBLISHED to it, with no way to resolve a module from the repository whose source it is already holding. Different jobs.

The 404 rule — the load-bearing one

In the GOPROXY protocol 404 means "ask the next proxy". The next proxy resolves github.com/hanzoai/x the way its name reads: from github.com. So a 404 inside our namespaces does not fail — it silently returns resolution to the host we exist so we need not depend on, and the build goes green having used it. A false success is worse than a failure because nobody looks at it.

TestOurFailuresNeverFallThrough holds this. Do not "fix" it to return 404. It was written for a privacy reason that has since evaporated; the reason above is the one that survives, and it is stronger.

Checksums

Everything is checked against the public transparency log by default, ours included, and that check is worth more for our modules than for anyone else's: this proxy is the one place that reads the forge while the log remembers what was published under the same name. A forge copy whose bytes disagree with the published tag is caught here or not at all.

UNLOGGED is the exception list and it is EMPTY by default. A module the log has never seen cannot be checked against it and asking fails the fetch, so an entry says "the log has nothing to say about this name" — not "do not verify it". The caller's go.sum still pins it either way. Entries leave the list as their repositories are published; when it is empty for good, delete the variable.

GOPRIVATE is the wrong knob and reaching for it is the original bug. It means "bypass the proxy AND the log", and bypassing the proxy is what sends the toolchain to github.com. A test asserts the server never sets it.

The forge credential is structural, and going open source does not change it

The forge asks every reader to sign in, public repositories included:

curl -o /dev/null -w '%{http_code}\n' \
  https://git.hanzo.ai/hanzoai/ci.git/info/refs?service=git-upload-pack   # 401

So FORGE_TOKEN is required, and it stays required however much of the estate is published. Publishing changes who may read a repository on GitHub; it does not change that this forge authenticates.

An earlier note here claimed the opposite, from a git -c credential.helper= check that quietly kept reading the system and global helpers. Verify anonymity with GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null, or with plain curl against info/refs, which has no helper to inherit.

The code still tolerates an empty token — it degrades to a plain rewrite rather than embedding an empty credential in a URL — because the rewrite is what makes our names resolve from the forge at all, and dropping it would send them back to github.com. That is worth keeping as behaviour. It is not a deployment mode.

In the cluster the value comes from KMS at hanzo:/deploy/FORGE_TOKEN, synced by the git-hanzo-ai-token-kms-sync KMSSecret into git-hanzo-ai-token/token. It carries write scope and only needs read.

Who actually needs this, measured

Of the 167 module versions our namespaces contribute across 22 Go repositories, 164 are served by the public proxy and verified by the public log. Three are not, and they are the entire reason any repository cannot build without this service:

module	not on the public proxy	needed by
github.com/hanzoai/thinking v0.1.1	404	zen, zen-gateway
github.com/hanzoai/voice (pseudo-version)	404	ai
github.com/hanzoai/zrok/v2 v2.0.1	404	mcp-gateway

All three resolve from the forge, and all three are in UNLOGGED because the checksum log has never seen them. Verified against the running service with no GitHub credential present: each answers OK through the proxy and fails with GOPROXY=direct.

So four repositories need the proxy — ai, mcp-gateway, zen, zen-gateway — and eighteen carry a GitHub credential for a module graph that is already entirely public: adnexus, base, cloud, commerce, gateway, git, iam, ingress, kms, notify, o11y, playground, s3-csi, superbase, team-go, visor, vm, world.

Those eighteen do not need this service at all. Their whole graph is on the public proxy, so the credential simply comes out and the checksum log starts verifying what was being fetched around it. Six are done — iam, s3-csi, playground, notify, world, team-go — and carry zero credential-bearing lines; the remaining twelve are the same edit.

Preflight each one before pushing, with the command the build actually runs:

GOFLAGS=-mod=mod GOWORK=off GOPROXY='https://proxy.golang.org,direct' \
GOPRIVATE=none.invalid GONOPROXY=none.invalid GONOSUMDB=none.invalid \
GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null GIT_TERMINAL_PROMPT=0 \
  go mod download

Every part of that is load-bearing. GOPRIVATE= EMPTY does not clear it — Go reads an empty environment variable as unset and falls back to go env's stored value, which on a developer machine already lists our namespaces, so the preflight sends our modules direct and "fails" for a reason that exists only on that laptop. It needs a non-matching value. GIT_CONFIG_GLOBAL=/dev/null is the same trap as the anonymity check above: -c credential.helper= leaves the system helper in place. And use go mod download, not go list -m all — the latter walks the unpruned graph and reports modules a build never loads.

CI passing is NOT this evidence. The shared pipeline still injects a GitHub credential for every job, so a converted repo can go green on the credential it was supposed to stop needing.

Three traps in measuring this again, all of which produced a wrong answer first:

• /vN paths. A pattern that runs <path> <version> skips github.com/hanzoai/zrok/v2 v2.0.1 entirely. Thirteen modules were invisible this way and one of them was the only blocker for mcp-gateway.
• exclude is not require. exclude github.com/luxfi/genesis v1.5.21 appears in four go.mod files for a version that exists nowhere; reading it as a dependency invents four repositories that need a proxy.
• Substring repository names. zen-gateway answers for gateway.

github.com/hanzoai/s3-go deserves its own line: the public proxy DOES still serve it, so nothing is failing today. Its GitHub repository is gone, so what holds our builds up is a cache we do not own and cannot refill.

Reachability: test gates yes, image builds no

Go modules are fetched at two moments and only one of them can reach this.

• Test gates run on the git-runner in the hanzo namespace. Proven: a pod there resolves through the proxy with no GitHub credential present.
• Image builds run in buildkitd in hanzo-build, and that namespace is labelled hanzo.ai/component: build-isolated. Traffic to hanzo times out to the Service AND to the pod IP, so it is neither DNS nor the Service.

The namespace policy admits 10.124.0.0/16 and 10.125.0.0/16 and every pod involved sits in 10.125.0.0/16, which looks like an allow and is not: Cilium resolves an ipBlock against entities outside the cluster, and pod-to-pod traffic carries an identity rather than an address. Naming the namespace is the right shape and is what mod's own policy now does — but it is NOT sufficient here. allow-build-to-s3 names hanzo-build the same way and hanzo-build cannot reach s3:9000 either, so the isolation is enforced below these policies. Reaching it from a build is an owner decision about that boundary, not a rule to add.

Retention has a limit, and it is stated in the README

For our namespaces the guarantee is total, because the forge is the source rather than a copy of one. For third-party modules it covers what has already been served and nothing never fetched. Do not let it be written as more than that.

Two modules are beyond saving by any proxy: hanzoai/gh (required by docdb/tools) and hanzoai/xfail (required by docdb/integration) are absent from github.com, from the public proxy, and from the forge. They need a source before anything can serve them; the likely upstreams are FerretDB/gh and FerretDB/xfail, both public.

Build

go.mod says go 1.26.5 and the Dockerfile pins golang:1.26.5-bookworm. This is a binary — nothing imports it — so the directive constrains only its own build, and naming the patch states the toolchain it is actually built and tested with instead of a floor it happens to clear.

Separately: gover in hanzoai/ci read a two-part directive like go 1.26 as a demand for the newest patch of that minor, so it refused golang:1.26.5 — an image that builds it. Sixteen repos write a two-part directive, mcp-gateway among them. Fixed on ci main (6b1ea4f) with three cases covering the floor; that fix reaches the fleet when v1 next moves, which is blocked on ci main's site gate (four failures from 1177811, unrelated to this).

The caller pins @v1 and must keep doing so. This forge leaves GITHUB_WORKFLOW_REF unset, so build.yml clones its own bin/ tools from v1 no matter which ref the caller named — pinning an immutable tag here gets that build.yml against v1's tools, which is skew, not a canary.