Compare commits

...
6 Commits
Author SHA1 Message Date
hanzo-dev cb56c16d5e platform: stop offering a github token to a host that refuses it
The scheduler handed every build `--secret id=GIT_AUTH_TOKEN`, on the reasoning
written above it: private contexts need it, "public repos ignore it". The second
half is false. A host that does not accept the token does not shrug it off -- it
refuses it, and git falls through to a prompt that does not exist inside a build.

MEASURED against our own forge: with the secret present, fetching
http://hanzo-git.hanzo.svc.cluster.local/hanzoai/cloud.git dies at step #1 on
`could not read Username ... terminal prompts disabled`. Without it, the identical
fetch succeeds anonymously in 3.3s. So the flag that exists to make private
fetches possible was the single thing making our own forge impossible.

The credential now goes out under two ids because it answers two questions:
GO_MOD_TOKEN is ours and is what the Dockerfile mounts for the private Go modules
on github.com; GIT_AUTH_TOKEN is BuildKit's reserved name for the CONTEXT host,
and is offered only when the context is somewhere a github credential could
plausibly be right -- which our own forge, by construction, is not.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-05 01:49:23 -07:00
hanzo-dev 3f2705ba97 build: GIT_AUTH_TOKEN is buildkit's name, not ours
The Dockerfile mounted its github credential as `id=GIT_AUTH_TOKEN`. That id is
RESERVED: buildkit's git source reads a secret by that exact name and offers it
to whatever host the build CONTEXT lives on. So one name carried two meanings --
"the credential for the private Go modules on github.com" to this RUN, and "the
credential for the context host" to buildkit -- and the two agreed only for as
long as both happened to be github.

They stop agreeing the moment the context is our own forge. MEASURED: with
--secret id=GIT_AUTH_TOKEN present, fetching the cloud repo from
http://hanzo-git.hanzo.svc.cluster.local dies at step #1 on `could not read
Username for '...': terminal prompts disabled` -- buildkit offered a github token
to the forge, the forge refused it, and git fell through to a prompt that does
not exist in a build. The identical fetch with no such secret succeeds in 3.3s,
anonymously, because the forge serves these repos anonymously.

So the fix is not to find a token that satisfies both hosts -- there isn't one,
and looking for it is what the shared name makes you do. It is to stop sharing
the name. GO_MOD_TOKEN says which credential it is and which host it is for;
buildkit finds no GIT_AUTH_TOKEN and fetches the context as itself.

github.com/hanzoai/dashscopego is the only genuinely private module of the 52
(ai, commerce, orm, iam and base all answer an anonymous ls-remote), so this one
secret is the whole reason a cloud build needs github at all.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-05 01:45:17 -07:00
hanzo-dev 7abb99e214 build: bump the module cache id the Dockerfile told us to bump
The comment above this mount already wrote the runbook: "a module resolved while
its tag did not yet exist is remembered as 'unknown revision' forever, so
`go mod download` keeps failing on a tag that now exists and resolves fine from a
clean cache. BUMP THE SUFFIX (-v4 -> -v5) to force a cold module cache the next
time a phantom pin poisons it."

It is poisoned. MEASURED: `go mod download` fails on
github.com/hanzoai/dashscopego@v0.6.0 with "could not read Username for
'https://github.com': terminal prompts disabled", out of the VCS cache at
/go/pkg/mod/cache/vcs/2cb27ec8..., and the entry is remembered per node, which is
why the same commit builds on one runner and dies on another.

dashscopego is the one genuinely PRIVATE module of the hanzoai set -- ai,
commerce, orm, iam and base all answer an anonymous ls-remote -- so it is the
single dependency that keeps this build tied to a GitHub credential, and the
single entry a failed tokenless attempt can poison for everyone after it.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-05 01:40:13 -07:00
hanzo-dev 60242b7ed4 platform: "use the native forge" was unreachable, not misconfigured
The host allowlist learned to TRUST our own forge (dc84b46d). It still could not
FETCH from it. git.hanzo.ai resolves to this cluster's own LoadBalancer, DO's LB
does not hairpin, and so the fetch did not fail cleanly -- it hung. MEASURED from
ns hanzo-build: the public name times out at 25.0s, and a real build sat on it
for 134678 ms before giving up, while the in-cluster Service answers 303 in
0.014s. The object-store lane hit the identical wall at 133755 ms and
artifact-publish-egress.yaml already wrote down the shape of the answer: reach
the internal endpoint, keep publishing the public URL.

So the public URL stays the API contract -- callers POST /v1/runner with
https://git.hanzo.ai/... and validation still judges THAT host -- and only the
fetch is redirected. The two concerns stay apart: hostAllowed decides what we are
willing to build, internalizeForgeURL decides how to go and get it. A test pins
that setting an internal endpoint does not widen what the API accepts.

isSelfGit is one predicate because it answers one question asked twice: trust
(our own forge is always a legitimate source) and routing (our own forge is
reachable in-cluster and must not be dialled by its public name). Two copies
could drift into trusting a host we cannot fetch.

Empty by default, so this changes nothing until an operator says where the forge
lives: CLOUD_PLATFORM_FORGE_INTERNAL_URL=http://hanzo-git.hanzo.svc.cluster.local.
Which endpoint is reachable is a deployment fact, not a code opinion.

The context string was assembled identically at two call sites and the rewrite
has to apply to both, so there is now one gitContext.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-05 01:33:42 -07:00
hanzo-dev 62527d876f build: 120 plugins linked one at a time on an eight-core pod
The plugin loop is almost pure LINK time, and Go links one binary on one core.
MEASURED: 120 plugins, 119 inter-plugin deltas, mean 2.40s, median 1.80s, 285.1s
total -- 22% of a 21.8m build spent using one of the pod's eight cores while
seven idled. The slowest are commerce 12.1s, ai 12.0s, base 7.4s, zen 7.4s.

-P 4, not more, because the ceiling here is RAM and not CPU: the pod's memory
limit is 16Gi and concurrent links of the heavy binaries are what peak it. Go's
build cache is concurrency-safe, so there is no correctness question, only a
resource one.

busybox xargs was VERIFIED in this exact builder image rather than assumed --
Alpine's xargs is a busybox applet and -P is frequently compiled out. Four 2s
jobs at -P 4 finished in 2s elapsed, not 8s, and a failing child returned 123.
123 is non-zero, so the `set -eu` above still fails the build, and the FATAL
guard still names the plugin it tripped on.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-05 01:33:42 -07:00
hanzo-dev 3f34212f1d platform: the build cache exported 265s of layers nothing can hit
mode=max exports the intermediate stages on the argument that the Go compiles
live there, so exporting only the final stage would make the expensive steps
rerun anyway. They rerun anyway REGARDLESS. Every expensive step in the cloud
image sits below `COPY . .`; the source is what changed; those layers are
invalidated on entry and no exported copy of them is reachable. The export was
paying, every build, to write layers that cannot be read.

MEASURED on three consecutive cloud builds -- exporting cache to registry cost
265.0s, 263.0s and 275.6s, ~20% of a 21.8m build, and bought SEVEN cached steps
out of ~45: an apk add, an addgroup, a symlink, and three small COPYs. Four of
the seven are in the final stage and survive mode=min. The three that do not are
worth ~10-30s. Net ~240s per build, on every repo that builds this way, plus
~1.9GB of dead layers no longer pushed to the registry each time.

One constant rather than two literals: the registry and the object store were
asking the same question and could drift to different answers. It is named
cacheExportMode so the thing to re-check is findable -- and what to re-check is
the DOCKERFILE, not the backend: this flips back the day expensive work moves
above `COPY . .`.

The two tests asserted mode=max and its rationale verbatim, so they are updated
to assert the measurement rather than the assumption.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-05 01:33:42 -07:00
6 changed files with 284 additions and 38 deletions
+52 -21
View File
@@ -131,6 +131,25 @@ ENV CGO_CFLAGS="-DSQLITE_HAS_CODEC -DSQLITE_USE_URI=1 -I/usr/include/sqlcipher"
GOPROXY=https://proxy.golang.org,direct \
GOFLAGS=-mod=readonly
COPY go.mod go.sum ./
# The secret is GO_MOD_TOKEN and not GIT_AUTH_TOKEN, because GIT_AUTH_TOKEN is
# NOT a name we own. BuildKit reserves it: the git source reads a secret by that
# exact id and applies it to the CONTEXT fetch. So one name meant two things --
# "the credential for the private Go modules on github.com" here, and "the
# credential for wherever the context lives" to buildkit -- and they only agreed
# while both were github.
#
# They stop agreeing the moment the context is our own forge. MEASURED: with
# --secret id=GIT_AUTH_TOKEN present, fetching
# http://hanzo-git.hanzo.svc.cluster.local/hanzoai/cloud.git dies on
# `could not read Username ... terminal prompts disabled` -- buildkit offered the
# github token to the forge, the forge refused it, and git fell through to a
# prompt that is not there. The identical fetch with NO such secret succeeds in
# 3.3s, anonymously, because the forge serves these repos anonymously.
#
# Renaming it here gives each credential one meaning: buildkit finds no
# GIT_AUTH_TOKEN and fetches the context as itself, and this RUN still gets its
# github token under a name that says what it is for. That is what makes a
# forge-sourced build of this image possible at all.
# The cache mounts carry an EXPLICIT id so they can be busted. Without one,
# BuildKit keys the cache by target path alone, and a poisoned entry is immortal:
# a module resolved while its tag did not yet exist is remembered as "unknown
@@ -138,10 +157,10 @@ COPY go.mod go.sum ./
# and resolves fine from a clean cache. That is exactly what wedged the release
# on otel-collector v0.144.10. BUMP THE SUFFIX (-v4 -> -v5) to force a cold
# module cache the next time a phantom pin poisons it.
RUN --mount=type=secret,id=GIT_AUTH_TOKEN \
--mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
if [ -s /run/secrets/GIT_AUTH_TOKEN ]; then \
git config --global url."https://x-access-token:$(cat /run/secrets/GIT_AUTH_TOKEN)@github.com/".insteadOf "https://github.com/"; \
RUN --mount=type=secret,id=GO_MOD_TOKEN \
--mount=type=cache,id=cloud-gomod-v5,target=/go/pkg/mod,sharing=locked \
if [ -s /run/secrets/GO_MOD_TOKEN ]; then \
git config --global url."https://x-access-token:$(cat /run/secrets/GO_MOD_TOKEN)@github.com/".insteadOf "https://github.com/"; \
fi && \
go mod download
COPY . .
@@ -180,8 +199,8 @@ RUN set -eu; \
# the host and the per-app graphs (./cmd/... ./plugin/...) is the same package set
# it linked, so listing them together is the equivalent guard — one modernc import
# in ANY app fails here.
RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
--mount=type=cache,id=cloud-gobuild-v4,target=/root/.cache/go-build,sharing=locked \
RUN --mount=type=cache,id=cloud-gomod-v5,target=/go/pkg/mod,sharing=locked \
--mount=type=cache,id=cloud-gobuild-v5,target=/root/.cache/go-build,sharing=locked \
MODERNC="$(CGO_ENABLED=1 go list -tags "libsqlite3 sqlite_fts5 sqlite_math_functions" -deps ./cmd/... ./plugin/... 2>/dev/null | grep -c 'modernc.org/sqlite' || true)"; \
[ "$MODERNC" = "0" ] || { echo "SQLITE-GATE FAIL: a per-app binary links modernc.org/sqlite ($MODERNC pkgs) under CGO=1 — double-registers \"sqlite\" with hanzoai/sqlite(mattn) and panics at init. Find it: CGO_ENABLED=1 go list -tags 'libsqlite3 sqlite_fts5 sqlite_math_functions' -deps ./plugin/<app> | grep modernc"; exit 1; }
# RED gate — ENCRYPTION PROOF + the cek.go GOLDEN-VECTOR KAT, under the SAME CGO +
@@ -190,8 +209,8 @@ RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
# image). TestUnwrapGoldenFixture asserts a FROZEN pre-luxfi-swap 61-byte DEK
# sidecar still decrypts under the shipped luxfi/crypto-AEAD code — existing
# encrypted stores stay readable, or NO image.
RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
--mount=type=cache,id=cloud-gobuild-v4,target=/root/.cache/go-build,sharing=locked \
RUN --mount=type=cache,id=cloud-gomod-v5,target=/go/pkg/mod,sharing=locked \
--mount=type=cache,id=cloud-gobuild-v5,target=/root/.cache/go-build,sharing=locked \
SQLITE_REQUIRE_CODEC=1 CGO_ENABLED=1 go test -count=1 -tags "libsqlite3 sqlite_fts5 sqlite_math_functions" \
-run 'TestEncryptionProof|TestUnwrapGoldenFixture|TestWrapUnwrapRoundTripPinsLayout' \
github.com/hanzoai/sqlite
@@ -205,8 +224,8 @@ RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
# api.hanzo.ai/v1/openapi.json serves 1441 operations with ZERO descriptions, which
# is exactly the binary mk/plugin.mk warns about. The SDK repos and the CLI read that
# document, so the prose never reached any of them either.
RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
--mount=type=cache,id=cloud-gobuild-v4,target=/root/.cache/go-build,sharing=locked \
RUN --mount=type=cache,id=cloud-gomod-v5,target=/go/pkg/mod,sharing=locked \
--mount=type=cache,id=cloud-gobuild-v5,target=/root/.cache/go-build,sharing=locked \
go generate -run zipdoc ./...
# THE LIGHT HOST (cmd/cloud) — ~400 packages, pure Go, no codec and no subsystem
# (it links zip + the manifest + the light webui console embed, and nothing else).
@@ -214,15 +233,15 @@ RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
# answers, and loads each app as its OWN process (a plugin) on the first request
# that reaches it. There is no fused binary anymore: the fleet never links
# together, so no build in this image is the mega link that once dominated it.
RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
--mount=type=cache,id=cloud-gobuild-v4,target=/root/.cache/go-build,sharing=locked \
RUN --mount=type=cache,id=cloud-gomod-v5,target=/go/pkg/mod,sharing=locked \
--mount=type=cache,id=cloud-gobuild-v5,target=/root/.cache/go-build,sharing=locked \
CGO_ENABLED=0 go build \
-ldflags="-s -w -X github.com/hanzoai/cloud.Version=${VERSION}" -o /cloud ./cmd/cloud
# The functional smoke prober (plugin/smoke) — a stdlib-only static binary shipped
# alongside the host so the release gate can `docker exec` it against the freshly-
# built image (and any deployment can be smoked via `docker run --entrypoint /smoke`).
RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
--mount=type=cache,id=cloud-gobuild-v4,target=/root/.cache/go-build,sharing=locked \
RUN --mount=type=cache,id=cloud-gomod-v5,target=/go/pkg/mod,sharing=locked \
--mount=type=cache,id=cloud-gobuild-v5,target=/root/.cache/go-build,sharing=locked \
CGO_ENABLED=0 go build -ldflags="-s -w" -o /smoke ./plugin/smoke
# EVERY subsystem, each as its OWN binary in /plugins beside the host. The host
# fork/execs a sibling <dir>/<name> (manifest.App.Plugin) on the first request that
@@ -259,16 +278,28 @@ RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
# no-ops PRAGMA key), so they are built uniformly — one contract for all, the
# non-sqlite apps merely carrying a libc dep they do not use. The modernc gate above
# already proved none of them double-registers "sqlite" under this tag.
RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
--mount=type=cache,id=cloud-gobuild-v4,target=/root/.cache/go-build,sharing=locked \
#
# -P 4 because this step is almost pure LINK time and Go links one binary on one
# core. MEASURED: 120 plugins, 119 inter-plugin deltas, mean 2.40s, median 1.80s,
# 285.1s total — 22% of a 21.8m build spent using one of the pod's eight cores.
# Four workers, not more: the pod's memory LIMIT is 16Gi and a link of the heavy
# ones (commerce 12.1s, ai 12.0s, base 7.4s, zen 7.4s) is what peaks it, so the
# ceiling here is RAM, not CPU. Go's build cache is concurrency-safe.
#
# BusyBox xargs, verified in this exact image rather than assumed — Alpine's xargs
# is a busybox applet and -P is frequently absent: four 2s jobs at -P 4 finished in
# 2s elapsed, and a failing child returned 123. 123 is non-zero, so the `set -eu`
# above still fails the build, and the FATAL guard keeps naming the plugin it
# tripped on.
RUN --mount=type=cache,id=cloud-gomod-v5,target=/go/pkg/mod,sharing=locked \
--mount=type=cache,id=cloud-gobuild-v5,target=/root/.cache/go-build,sharing=locked \
set -eu; mkdir -p /plugins; \
names="$(sed -n 's/.*{Name: "\([^"]*\)".*/\1/p' manifest/apps.go)"; \
[ -n "$names" ] || { echo "FATAL: no apps parsed from manifest/apps.go — the derivation broke, not the app list"; exit 1; }; \
for p in $names; do \
[ -d "./plugin/$p" ] || { echo "FATAL: manifest app '$p' has no plugin/$p — run 'make generate' and commit"; exit 1; }; \
echo "building plugin $p"; \
CGO_ENABLED=1 go build -tags "libsqlite3 sqlite_fts5 sqlite_math_functions" -ldflags="-s -w" -o "/plugins/$p" "./plugin/$p"; \
done
printf '%s\n' $names | xargs -P 4 -I{} sh -c '\
[ -d "./plugin/{}" ] || { echo "FATAL: manifest app \"{}\" has no plugin/{} — run \"make generate\" and commit"; exit 1; }; \
echo "building plugin {}"; \
CGO_ENABLED=1 go build -tags "libsqlite3 sqlite_fts5 sqlite_math_functions" -ldflags="-s -w" -o "/plugins/{}" "./plugin/{}"'
# Prove a SHIPPED sqlite-backed plugin binds sqlite3_* to libsqlcipher, not a
# plaintext libsqlite3. /plugins/base opens per-org stores under the SAME CGO=1 +
# libsqlite3 build every plugin above got, so it is a real witness for the set.
+6 -2
View File
@@ -90,8 +90,12 @@ func TestTheCacheBackendFollowsWhatIsReachable(t *testing.T) {
"use_path_style=true",
// Keyed per repository inside the shared bucket.
"name=hanzoai-cloud",
// The intermediate stages are where the expensive steps live.
"mode=max",
// min, not max. The intermediate stages ARE where the expensive steps
// live, and that is exactly why exporting them is wasted: they all sit
// below `COPY . .`, so the source that changed invalidates them on entry
// and no exported copy can be hit. MEASURED at mode=max: 265s per build
// to buy 7 trivial cached steps out of ~45. See cacheExportMode.
"mode=min",
} {
if !strings.Contains(got, want) {
t.Errorf("s3 cache missing %q: %s", want, got)
+3 -1
View File
@@ -59,7 +59,9 @@ func TestBuildFrontendCmdCarriesASharedCache(t *testing.T) {
got := join(buildFrontendCmd("ctx", "Dockerfile", "ghcr.io/hanzoai/studio:v1.2.3"))
for _, want := range []string{
"--import-cache type=registry,ref=ghcr.io/hanzoai/studio:buildcache",
"--export-cache type=registry,ref=ghcr.io/hanzoai/studio:buildcache,mode=max",
// Export is min: what mode=max would add is unreachable by construction,
// because every expensive step is below `COPY . .`. See cacheExportMode.
"--export-cache type=registry,ref=ghcr.io/hanzoai/studio:buildcache,mode=min",
} {
if !strings.Contains(got, want) {
t.Errorf("build command missing %q\ngot:%s", want, got)
+99 -13
View File
@@ -22,6 +22,7 @@ import (
"encoding/json"
"errors"
"fmt"
"net/url"
"strings"
"time"
@@ -827,7 +828,7 @@ func (k *k8sClient) launchBuildJob(ctx context.Context, org string, a Applicatio
jobName := truncate("pf-build-"+namespace.Sanitize(org)+"-"+a.Slug+"-"+jobIDSuffix(buildID), 63)
// buildctl-daemonless git frontend context: https://<repo>.git#<ref>.
buildCtx := strings.TrimSuffix(cleanURL, ".git") + ".git#" + cleanRef
buildCtx := gitContext(cleanURL, cleanRef)
// (CRIT-1) Emit buildctl as EXEC-FORM argv — NEVER a `sh -c` string. Each
// validated value is a single argv element handed to execve, so no shell ever
@@ -863,11 +864,22 @@ const platformBuildOrg = "platform"
// builds with zero config. The output image is FORCED here as its own argv
// element, so a caller can never redirect --output to another repo.
//
// The GIT_AUTH_TOKEN build secret (sourced from the Job env, which buildJobSpec
// wires from the gitTokenSecret Secret when present) is what lets the git
// context fetch PRIVATE repos — BuildKit's gitsource presents it as the HTTPS
// credential. Public repos ignore it; without the Secret the env is empty and
// fetches are anonymous, exactly as before.
// The github credential is offered to a build under TWO ids, because it answers
// two different questions and only one of the names is ours.
//
// GO_MOD_TOKEN is the Dockerfile's: the credential for the private Go modules on
// github.com. GIT_AUTH_TOKEN is BuildKit's, reserved — its gitsource reads a
// secret by that exact id and presents it as the HTTPS credential to whatever
// host the CONTEXT lives on. That is what lets a private repo be fetched.
//
// "Public repos ignore it" was the standing claim here and it is FALSE. A host
// that does not accept the token does not shrug it off; it refuses it and git
// falls through to a prompt that does not exist in a build. MEASURED against our
// own forge: with the secret present the fetch dies at step #1 on `could not read
// Username ... terminal prompts disabled`; with it absent the same fetch succeeds
// anonymously in 3.3s. So the secret is offered to the context ONLY when the
// context is somewhere the github credential could plausibly be right — which our
// own forge, by construction, is not.
func buildFrontendCmd(buildCtx, dockerfile, image string) []any {
return buildFrontendCmdRev(buildCtx, dockerfile, image, "")
}
@@ -907,6 +919,59 @@ func isCommitSHA(ref string) bool {
// release that lost had already been pinned. A version is only a receipt if the
// image can name its own commit, so the arg is passed here and the label is true
// no matter which builder ran.
// gitContext turns a validated clone URL and ref into the buildctl git-frontend
// context (`<url>.git#<ref>`), and points the cloud's OWN forge at the endpoint a
// build can actually reach. Two call sites assembled this string identically; the
// rewrite has to apply to both, so there is now one of it.
func gitContext(cleanURL, cleanRef string) string {
return strings.TrimSuffix(internalizeForgeURL(cleanURL), ".git") + ".git#" + cleanRef
}
// internalizeForgeURL rewrites a clone URL for the cloud's own forge to the
// in-cluster endpoint, and leaves every other host alone.
//
// A build could not fetch from the company's own forge. git.hanzo.ai resolves to
// this cluster's own LoadBalancer, and DO's LB does not hairpin, so a build pod
// dialling it does not fail — it HANGS. MEASURED from ns hanzo-build: the public
// name times out (25.0s, and 134.7s on a real build before it gave up) while the
// in-cluster Service answers 303 in 0.014s. The object-store lane hit the
// identical wall at 133.8s and resolved it the same way, which is the shape to
// copy: reach the internal endpoint, keep publishing the public URL.
//
// So the PUBLIC url stays the API contract — callers POST /v1/runner with
// https://git.hanzo.ai/... and validation still judges that host — and only the
// fetch is redirected. The two concerns stay separate: hostAllowed decides what
// we are willing to build, this decides how to go and get it.
//
// Empty by default, which rewrites nothing and preserves today's behaviour
// exactly. Which endpoint is reachable is a deployment fact, not a code opinion:
// set CLOUD_PLATFORM_FORGE_INTERNAL_URL (e.g. http://hanzo-git.hanzo.svc.cluster.local)
// and native git works; leave it unset and nothing moves.
// contextIsSelfForge reports whether a build context resolves to our own forge —
// i.e. whether internalizeForgeURL redirected it. It is asked so the github
// credential is not offered to a host that will refuse it.
func contextIsSelfForge(buildCtx string) bool {
base := strings.TrimSpace(getenv("CLOUD_PLATFORM_FORGE_INTERNAL_URL", ""))
return base != "" && strings.HasPrefix(buildCtx, strings.TrimSuffix(base, "/")+"/")
}
func internalizeForgeURL(raw string) string {
base := strings.TrimSpace(getenv("CLOUD_PLATFORM_FORGE_INTERNAL_URL", ""))
if base == "" {
return raw
}
u, err := url.Parse(raw)
if err != nil || !isSelfGit(strings.ToLower(u.Hostname())) {
return raw
}
b, err := url.Parse(strings.TrimSuffix(base, "/"))
if err != nil || b.Host == "" {
return raw
}
u.Scheme, u.Host = b.Scheme, b.Host
return u.String()
}
func buildFrontendCmdRev(buildCtx, dockerfile, image, revision string) []any {
cmd := []any{"buildctl-daemonless.sh", "build"}
if strings.TrimSpace(dockerfile) != "" {
@@ -922,7 +987,10 @@ func buildFrontendCmdRev(buildCtx, dockerfile, image, revision string) []any {
"--opt", "context="+buildCtx,
)
}
cmd = append(cmd, "--secret", "id=GIT_AUTH_TOKEN,env=GIT_AUTH_TOKEN")
cmd = append(cmd, "--secret", "id=GO_MOD_TOKEN,env=GIT_AUTH_TOKEN")
if !contextIsSelfForge(buildCtx) {
cmd = append(cmd, "--secret", "id=GIT_AUTH_TOKEN,env=GIT_AUTH_TOKEN")
}
// Hand the image's own tag to the build as VERSION, so a binary can stamp the
// version it was published under (cloud's Dockerfile links it into
// cloud.Version, the X-Api-Version header) instead of shipping the "dev"
@@ -994,18 +1062,36 @@ func cacheArgs(repo string) []string {
",region=" + getenv("S3_REGION", "us-east-1") +
",endpoint_url=" + s3CacheEndpoint(ep) +
",use_path_style=true,name=" + cacheKey(repo)
// mode=max exports the intermediate stages too, which is where the Go
// compiles live; without it a multi-stage build caches only its final
// layers and the expensive steps rerun anyway.
return []string{"--import-cache", common, "--export-cache", common + ",mode=max"}
return []string{"--import-cache", common, "--export-cache", common + cacheExportMode}
}
ref := repo + ":buildcache"
return []string{
"--import-cache", "type=registry,ref=" + ref,
"--export-cache", "type=registry,ref=" + ref + ",mode=max",
"--export-cache", "type=registry,ref=" + ref + cacheExportMode,
}
}
// cacheExportMode is min, not max, and that is a fact about the Dockerfile rather
// than an opinion about caching.
//
// mode=max exports the intermediate stages too. The argument for it was that the
// Go compiles live there, so exporting only the final stage would make the
// expensive steps rerun anyway. They rerun anyway REGARDLESS: every expensive
// step sits below `COPY . .`, and the source is what changed, so those layers are
// invalidated on entry and no exported copy of them can ever be hit. mode=max was
// paying to export layers that are unreachable by construction.
//
// MEASURED on three consecutive cloud builds: exporting cache cost 265.0s, 263.0s
// and 275.6s -- ~20% of a 21.8m build -- to buy SEVEN cached steps out of ~45, all
// of them trivial setup (apk add, addgroup, a symlink, three small COPYs). Four of
// the seven are in the final stage and survive mode=min. The three that do not are
// a build-stage apk add, worth ~10-30s. Net saving ~240s per build, on every repo.
//
// This flips back the day the Dockerfile puts expensive work ABOVE `COPY . .` --
// that, not the cache backend, is the thing to check. It is one constant because
// the question it answers is the same for the registry and the object store.
const cacheExportMode = ",mode=min"
// cacheKey names one repository's cache inside the shared bucket, so a new repo
// needs no provisioning and two repos never read each other's layers.
func cacheKey(repo string) string {
@@ -1285,7 +1371,7 @@ func (k *k8sClient) launchDirectBuild(ctx context.Context, repoURL, ref, image,
return "", errTooManyBuilds
}
jobName := truncate("pf-runner-"+jobIDSuffix(buildID), 63)
buildCtx := strings.TrimSuffix(cleanURL, ".git") + ".git#" + cleanRef
buildCtx := gitContext(cleanURL, cleanRef)
command := buildFrontendCmdRev(buildCtx, cleanDockerfile, image, cleanRef)
pushSecret, err := buildPushSecret(image)
if err != nil {
+11 -1
View File
@@ -220,8 +220,18 @@ func apexOf(raw string) string {
return apex
}
// isSelfGit reports whether host is the cloud's OWN git — its apex or anything
// under it. It is one predicate because it answers one question, asked twice for
// different reasons: hostAllowed asks it to decide TRUST (our own forge is always
// a legitimate build source), and internalizeForgeURL asks it to decide ROUTING
// (our own forge is reachable in-cluster and must not be dialled by its public
// name). Two copies of this rule could drift into trusting a host we cannot fetch.
func isSelfGit(host string) bool {
return selfGitHost != "" && (host == selfGitHost || strings.HasSuffix(host, "."+selfGitHost))
}
func hostAllowed(host string) bool {
if selfGitHost != "" && (host == selfGitHost || strings.HasSuffix(host, "."+selfGitHost)) {
if isSelfGit(host) {
return true
}
for _, allowed := range gitProviderHosts {
+113
View File
@@ -0,0 +1,113 @@
package platform
import (
"strings"
"testing"
)
// The github credential must not be offered to our own forge. "Public repos
// ignore it" was the standing assumption and it is false: a host that does not
// accept the token refuses it, and git falls through to a prompt that does not
// exist in a build. MEASURED — with id=GIT_AUTH_TOKEN present the forge fetch
// dies at step #1 on "could not read Username ... terminal prompts disabled";
// without it the same fetch succeeds anonymously in 3.3s.
func TestTheGithubTokenIsNotOfferedToOurOwnForge(t *testing.T) {
saved := selfGitHost
defer func() { selfGitHost = saved }()
selfGitHost = apexOf("api.hanzo.ai")
t.Setenv("CLOUD_PLATFORM_FORGE_INTERNAL_URL", "http://hanzo-git.hanzo.svc.cluster.local")
join := func(cmd []any) string {
out := ""
for _, a := range cmd {
out += " " + a.(string)
}
return out
}
// Our own forge: the Dockerfile still gets its token, the CONTEXT does not.
own := join(buildFrontendCmd(gitContext("https://git.hanzo.ai/hanzoai/cloud", "main"), "Dockerfile", "ghcr.io/hanzoai/cloud:t"))
if !strings.Contains(own, "id=GO_MOD_TOKEN,env=GIT_AUTH_TOKEN") {
t.Error("the Dockerfile lost its go-module credential")
}
if strings.Contains(own, "id=GIT_AUTH_TOKEN,env=") {
t.Errorf("the github token was offered to our own forge, which refuses it:%s", own)
}
// An external context still gets it — that is what fetches a PRIVATE repo.
ext := join(buildFrontendCmd(gitContext("https://github.com/hanzo-inc/cloud", "main"), "Dockerfile", "ghcr.io/hanzoai/cloud:t"))
if !strings.Contains(ext, "id=GIT_AUTH_TOKEN,env=GIT_AUTH_TOKEN") {
t.Errorf("a private github context lost its credential:%s", ext)
}
}
// The defect this pins: a build could not fetch from the company's own forge.
// git.hanzo.ai resolves to the cluster's own LoadBalancer, DO's LB does not
// hairpin, and so the fetch did not fail — it hung, MEASURED at 25.0s from ns
// hanzo-build and 134.7s on a real build, while the in-cluster Service answered
// 303 in 0.014s. "Use native git" was unreachable by construction, not
// misconfigured.
func TestTheOwnForgeIsFetchedInCluster(t *testing.T) {
saved := selfGitHost
defer func() { selfGitHost = saved }()
selfGitHost = apexOf("api.hanzo.ai") // hanzo.ai
t.Setenv("CLOUD_PLATFORM_FORGE_INTERNAL_URL", "http://hanzo-git.hanzo.svc.cluster.local")
// Our own forge is redirected to the endpoint that answers, and the path,
// the ref and the .git suffix all survive the rewrite.
got := gitContext("https://git.hanzo.ai/hanzoai/cloud", "refs/heads/main")
want := "http://hanzo-git.hanzo.svc.cluster.local/hanzoai/cloud.git#refs/heads/main"
if got != want {
t.Errorf("own forge not internalized:\n got %q\nwant %q", got, want)
}
// A URL that already ends in .git must not end up as ".git.git".
if got := gitContext("https://git.hanzo.ai/hanzoai/cloud.git", "abc"); got != "http://hanzo-git.hanzo.svc.cluster.local/hanzoai/cloud.git#abc" {
t.Errorf("double .git suffix: %q", got)
}
// An EXTERNAL provider is left completely alone. Rewriting github.com to our
// own forge would silently build someone else's source, or nothing at all.
for _, ext := range []string{
"https://github.com/hanzo-inc/cloud",
"https://gitlab.com/x/y",
} {
if got := gitContext(ext, "main"); got != ext+".git#main" {
t.Errorf("external host was rewritten: %q -> %q", ext, got)
}
}
}
// Unset is the default, and the default must change nothing: which endpoint is
// reachable is a deployment fact. A cluster that has not been told where its
// forge lives internally keeps dialling the public name, exactly as before.
func TestUnsetInternalURLRewritesNothing(t *testing.T) {
saved := selfGitHost
defer func() { selfGitHost = saved }()
selfGitHost = apexOf("api.hanzo.ai")
t.Setenv("CLOUD_PLATFORM_FORGE_INTERNAL_URL", "")
const pub = "https://git.hanzo.ai/hanzoai/cloud"
if got := gitContext(pub, "main"); got != pub+".git#main" {
t.Errorf("unset must be a no-op, got %q", got)
}
}
// The public URL stays the API contract. Callers POST /v1/runner with the public
// host and validation judges THAT host; only the fetch is redirected. If the
// rewrite ever leaked into validation, an operator setting an internal endpoint
// would quietly change what the API accepts.
func TestInternalizingDoesNotWidenWhatIsAccepted(t *testing.T) {
saved := selfGitHost
defer func() { selfGitHost = saved }()
selfGitHost = apexOf("api.hanzo.ai")
t.Setenv("CLOUD_PLATFORM_FORGE_INTERNAL_URL", "http://hanzo-git.hanzo.svc.cluster.local")
if hostAllowed("git.evil.com") {
t.Error("an internal endpoint must not make a foreign host allowed")
}
if !hostAllowed("git.hanzo.ai") {
t.Error("our own forge must still be allowed")
}
}