cloud purges the tag site-<org>-<slug> when a site publishes and Cloudflare
answers it, "cloudflare purge ok" for site-hanzo-hanzo-ai on every hanzo.ai
publish, but no page served from s3://hanzo-sites/<org>/<slug> carried a tag,
so the purge flushed nothing and the edge kept each page for its s-maxage
after the store had the new one: 8 to 38 s on hanzo.ai. A page of such a root
now names the tag; an asset, whose name changes with its bytes, does not.
HTML was no-cache, which also forbids Cloudflare from holding it, so every
page view crossed the country to the origin. It is now public, max-age=0,
s-maxage=60: a browser still revalidates every time, and a cache told to keep
pages holds one for a minute. Sites publish in place and nothing purges the
edge, so a minute is the longest a publish waits there. Cloudflare holds no
page unless a cache rule names the host.
Co-authored-by: Hanzo Dev <[email protected]>
- Reads into the cache run at most storeConns (64, the client's kept-alive
pool) at once. A refresh with no free slot holds no bytes and its request
streams, as for an object over the size limit.
- A failed or refused read no longer marks the entry unchecked, so one
request costs one HEAD and a stream, not a second refresh.
- An encoding lands on whatever entry still holds the bytes it encoded, so a
revalidation of the same version no longer throws it away.
- A read fills a buffer of the size the HEAD reported and refuses one longer.
The deployed gateway answers a matching If-Match with 200 and a stale one
with 412 (hanzo-sites/hanzo/hanzo-ai/index.html, from a pod in ns hanzo).
Co-authored-by: Hanzo Dev <[email protected]>
Review of the first cut:
- Brotli ran on the request path, unbounded: a 2 MiB object took 128 ms and
45 MiB, and 256 concurrent first requests took the heap to 3.1 GiB. The
first request is now answered as identity and the twin is made behind it,
at most GOMAXPROCS/2 at once, skipped rather than queued when no slot is
free, with the window sized to the input: 2 MiB is 65 ms and 25 MiB.
- Absences shared the objects' bound, so a flood of random paths emptied
every site. They have their own 4 MiB.
- The read now carries the HEAD's ETag as If-Match, so a copy one write
behind fails the read instead of filing build A under build B's tag.
- An object whose read failed once was never held again while its ETag
stood; it is read again.
- A 412 on the brotli representation promised a Content-Length it never
sent. The label goes on at the status line, on a 200 only, and that
representation offers no ranges.
- A late writer (the outage fallback, the encoder) replaces an entry only
while it is still current. "br;Q=0" refuses br.
Co-authored-by: Hanzo Dev <[email protected]>
Every request to an object-store root cost a HEAD and a GET to S3, 80 ms at
the median, before a byte left the ingress. One process-wide cache, bounded at
256 MiB and shared by every site, now holds objects up to 2 MiB with their
absence: an entry checked within the last second is served from memory, an
older one costs one HEAD, and the bytes are read again only when the ETag
moved. Sites publish in place, so a publish is served within a second.
Concurrent misses ask the store once; a store that stops answering leaves held
objects serving.
Text is held brotli-encoded beside its bytes and sent that way to a client
that accepts br, as its own representation (ETag "<tag>-br", Vary). Cloudflare
was pulling 557 KB of docs HTML uncompressed across the country per miss.
Everything under /_next/static/ is immutable whatever its extension. Every
middleware shares one S3 client and its connection pool.
Co-authored-by: Hanzo Dev <[email protected]>
1.27 is the released toolchain and the tree already compiles clean on it, so
the go directive names it. Nothing here depends on a 1.27 language feature
yet; this is the floor moving, which is what makes generic methods, the wider
type inference and promoted-field struct literals available to the code that
wants them next.
Co-authored-by: Hanzo Dev <[email protected]>
Most of it was stale, not cosmetic: the docs taught `TRAEFIK_*` env,
`traefik.http.*` labels, `traefik.ingress.kubernetes.io/` annotations,
`.traefik.yml` plugins and `traefik_*` metrics. This binary answers to
`INGRESS_*`, `ingress.http.*`, `ingress.kubernetes.io/`, `.ingress.yml` and
`ingress.*` — a reader following the old text configured nothing.
Two were worse than stale. Gateway API examples gave `group: traefik.io` for an
IngressService backendRef, which kubernetes.go:1265 refuses against hanzo.ai.
The contributing guide built with `github.com/traefik/ingress` ldflags against
a module that is `github.com/hanzoai/ingress/v3`.
script/code-gen.sh globs `hanzo.ai_*.yaml` and matched nothing — the per-CRD
dumps had been deleted and never regenerated, so `cat` failed the script under
`set -e`. controller-gen v0.19.0 restores all ten; they concatenate
byte-for-byte into the committed definition and fixture.
CHANGELOG.md is untouched: 6,789 lines of imported release history, where a
rewrite would falsify the record rather than rebrand it. `@traefiklabs/faency`,
`traefik/whoami`, k3s's `--disable=traefik`, the `traefik.io` CRD group and the
MIT attribution stay — upstream's addresses, not our brand.
Co-authored-by: Hanzo Dev <[email protected]>
query-string 9 exports a single object, so the pagination hook reads stringify
off it, the way the return-to hook already does.
react-router 7 is one package. Every import names it, so the app and its tests
share one router instance and one context; react-router-dom and its v5 types
are gone.
Vite 8 resolves tsconfig paths on its own, so vite-tsconfig-paths is gone and
resolve.tsconfigPaths takes its place.
faency carries @types/react 18 as a dependency. A pnpm override holds the whole
tree at the React 19 types the app declares, so faency's components type as JSX
components again.
The rest is what React 19 and TypeScript 7 ask for: useRef takes an initial
value, the error fallback narrows the unknown it is handed, sha512's output is
narrowed once where it meets @noble/ed25519, and the test setup replaces fetch
through vi.stubGlobal.
pnpm build, pnpm tsc and pnpm test are green; eslint still waits on
typescript-eslint support for TypeScript 7.
Co-authored-by: Hanzo Dev <[email protected]>
The static plane sets nosniff, which turns an unknown Content-Type from a
guess into a refusal. Go's built-in table stops at the web's first dozen
extensions and takes the rest from the system mime.types, which the
distroless runtime does not ship — so .md and .woff2 resolve on a
developer's box and answer empty in production. hanzoskills.com publishes
a markdown twin of every document, and a browser opening one was offered a
download instead of a page.
Co-authored-by: Hanzo Dev <[email protected]>
One toolchain across the estate. The tree carried 1.26.0, .1, .3, .4 and .5
at once, so which compiler built a binary depended on which repo it came
from.
Co-authored-by: Hanzo Dev <[email protected]>
The forge reads .github/workflows natively and github.com reads nothing else,
so one directory serves both planes instead of two conventions that disagree
about which one is live. The reusable in hanzoai/ci already lives there, so a
caller naming that path now resolves to a file that exists.
Co-authored-by: Hanzo Dev <[email protected]>
hanzo-build-linux-amd64, lux-build-linux-amd64, zoo-build-linux-amd64 and
every other org-prefixed spelling all meant the same machine. Which org a
job belongs to does not change which host can build it, so the org bought
one pool name per org per arch and each could go stale on its own -- the
evo host alone answered to fourteen names for one label.
One vocabulary now: linux-amd64, linux-arm64, macos-arm64, win-amd64. The
generic labels every plain workflow uses (ubuntu-latest, self-hosted) are
untouched, so naive CI keeps working.
Co-authored-by: Hanzo Dev <[email protected]>
hanzoai/ci keeps the reusable at .github/workflows/build.yml — the one path
both planes resolve, since github.com reads a reusable only from there and the
forge reads it too. The v1 tag moved with it, so a caller still naming
.hanzo/workflows/build.yml asks for a blob that is not in the tree:
InsertRun: read hanzoai/ci@v1:.hanzo/workflows/build.yml: object does not exist
The caller itself stays at .hanzo/workflows/cicd.yml on purpose: github.com has
no runner for these labels, so a caller placed there could never be scheduled.
Co-authored-by: Hanzo Dev <[email protected]>
OWASP Coraza with the Core Rule Set v4 embedded, declared as `waf` on
every provider: file, labels and the Middleware CR. Rules load Core Rule
Set, then files, then inline directives.
The engine state is written after every rule. The Core Rule Set's own
recommended configuration opens with SecRuleEngine DetectionOnly, so a
state written first is overridden by the rules it governs and the
firewall loads all of CRS and refuses nothing; detectionOnly on the
middleware is the one place the mode is decided.
geoBlock rode along: it was declared in dynamic config only, absent from
MiddlewareSpec, both deep copies and the CRD schema, so a Middleware CR
carrying it produced a middleware without it. The +groupName marker now
matches the registered group, which is what makes controller-gen's output
land where code-gen.sh reads it; the stale traefik.io_*.yaml dumps go.
CI asks for the arm64 runner git.hanzo.ai actually has and builds both
platforms.
hanzo-ingress is CGO_ENABLED=0 and statically linked. Alpine supplied three
things: ca-certificates and tzdata, which are data the binary reads; an account,
which is two files the kernel reads to name a uid it already enforces; and
libcap-utils, which existed for one line.
That line, setcap cap_net_bind_service=+ep, is redundant and it is measured
rather than assumed: the live deployment runs with securityContext capabilities
add NET_BIND_SERVICE, drop ALL, so the orchestrator grants the capability to the
process and a capability baked into the file grants nothing it does not already
have. The image's own ports are unprivileged regardless — 8080 and 8443, mapped
by the Service, which is what the line under the EXPOSE has always said.
The one posture this changes is docker run as non-root binding :80 with no
--cap-add. That is not how this is deployed, and the remedy there is
--cap-add=NET_BIND_SERVICE rather than a capability in the artifact.
zip 1.1.0, which this had been pinned to, is far behind what the rest of the
estate runs; http, log, zap and the kv client were likewise stale.
metric stays at v1.9.0 deliberately. v1.10.0 deleted NewOTelZAPExporter, the
adapter that gives a ZAP exporter the OpenTelemetry sdkmetric.Exporter shape,
and the surviving NewZAPExporter implements neither Temporality, Aggregation
nor ForceFlush. Taking the newer line means re-implementing here what upstream
removed, so the pin holds until metric carries the adapter again -- and that
pin is what caps zip at v1.24.3 rather than v1.36.
Two thousand and twenty-one packages carried more than one spec across this
estate — eslint alone in a hundred and four forms — and a package resolved twice
is two copies at run time, which is how react-hook-form came to report "two
different types with this name" from files nobody had edited. None do now.
The number lives in the catalog: a manifest names the package, the workspace
names the version. Where a manifest cannot reach a catalog it still carries the
one version, because the table is the source either way.
A version written into a manifest is written into every manifest that wants it,
which is how eslint came to exist in a hundred and four forms across this estate
and how a package came to resolve twice at run time. The catalog moves the number
out of the manifest: a manifest names the package, the workspace names the
version.
Four are held below their latest because the newer release cannot be used, and
the reason is recorded rather than the version alone: @noble/hashes 2 renamed
every subpath the web3 stack imports, @splinetool/runtime 2 depends on a package
that was never published, @dynamic-labs 5.4.1 the same, and react-resizable-panels
3 points its types condition at the bundle.
The social card asked for Inter first. That is not a face this estate
licenses, ships or ever drew, so a renderer with Inter installed painted
the wrong face and one without it silently fell through to system-ui.
Zen leads now; the same tail follows it.
help is the canonical default verb, and bare `make` here ran `generate
binary` — go generate plus a webui build — which is a heavy surprise for
someone typing `make` to see what a repo offers.
Safe because nothing invokes a bare make: the Dockerfile has none,
ci.yml calls `go test` directly rather than a make target, and no script
or doc references one. `make default` still builds exactly as before.
One line, deliberately. The #? help style is upstream's and it works, so
it stays; rewriting it to match another repo's would be churn, not a fix.
Co-authored-by: Hanzo Dev <[email protected]>
build aliases binary — the target that already produces this repo's
artifact (./dist/$GOOS/$GOARCH/hanzo-ingress).
The default is UNCHANGED: bare `make` still means `generate binary`.
Changing what it does is a behaviour change with no upside, and the repo's
own help target is kept as it is.
clean already removed the binary and dist/. cover.out is test-unit's
-coverprofile output — generated, gitignored, and left behind by every run.
NOT webui/static: that is what clean-webui is for, and wiping it costs a
full docker webui image rebuild.
Co-authored-by: Hanzo Dev <[email protected]>
webui/src/components/icons/Logo.tsx transcribed the block-H — five body paths plus the two shade slivers — into the component. It maps MARK_BLOCKS and MARK_SHADE now, so the geometry has one home and this file cannot drift from the mark every other Hanzo surface draws. Same fill and the same 0.7 sliver opacity; nothing about the rendering moves.
An adopted plaintext keeps the same digest as the pre-seal snapshot, so while
the process kept adopting, a rollback that replanted that snapshot would be
re-admitted — the same bytes adoption had just taken. Seal.Persisted, called by
both stores once a sealed write reaches disk or the Secret, ends adoption: the
store is sealed now, so a later unsealed store is a different store and is
refused. Reads of the sealed store are unaffected.
It fires on the write that LANDS, not the one attempted — a conflicted
SharedStore write does not reach it, so the retry re-read of a still-unsealed
object is unchanged. The signal is a method on the Seal contract rather than a
runtime type check: the store tells the seal a sealed write persisted, and what
a seal does with that is the seal's own concern; the identity seal does nothing.
The two things a human creates are the sealing key and the credential that
reads it; numbering them said less than naming them.
Adoption's entry is corrected to what the code does: one store per process, it
writes, and it adopts again after a restart while the setting is left on.
Adds what the declared state means today — universe sets no INGRESS_KMS_* and
no ingress-kms Secret for hanzo, so the new image there takes the identity seal
and warns. The image is safe to roll before the pre-flight; it just will not
seal until universe carries the env.
edgeSecrets ran on every boot, so an edge that orders no certificates still had
to reach KMS before it served anything — including routes whose certificates
come from a Kubernetes Secret, and plain-HTTP routes. It now returns the
identity seal when no resolver orders certificates, and reaches KMS only when
there is ACME state to seal.
Adoption covers ONE store. It admits the first unsealed document and the same
document again — a store is read more than once before the write that seals it
lands, and a replica that cannot write reads it every poll — but not a
different one. A seal left adopting was taking whatever plaintext it was handed
for as long as the process ran. The setting's comment now says what it does:
it writes, it covers one store, and it adopts again after a restart, which is
why it is removed once it has done its work.
One spelling for the KMS credential. The bare IAM_CLIENT_ID fallback undid the
reason the names are prefixed — a pod carrying another subsystem's IAM client
had it picked up here.
k8s/hanzo/deployment.yaml and service.yaml are deleted. They described a
hostNetwork DaemonSet named hanzo-ingress binding hostPort 80 and 443, while
what runs in that cluster is a Deployment named ingress declared in universe.
`kubectl apply -f k8s/hanzo/` appears in this README and in the sibling
gateway's Makefile, so the file was one command from taking :80 and :443 on
every node beside the workload that already had them. Universe is the one place
that says what runs; the cluster-level objects stay here.
The OTLP endpoint paths in the docs and the vendored Knative fixture are
restored: /v1/metrics is the OpenTelemetry receiver path, not a route of ours.
next reserved a number and recorded it at the same time, so a write that was
encoded and then refused — a resourceVersion conflict, a filesystem error —
left the store's own view of the document ahead of the document. The re-read of
those same bytes then looked like a step back, and every write after it was
refused: the writer keeps ordering certificates and stops persisting them, and
a restart reads a stale store and orders the estate again.
Reserving and recording are now two things. next returns the write about to be
made and records nothing; wrote records one that landed. One rule for both
stores.
The shared store's tests run under a real seal, which is what the deployment
is. Under the identity seal every document reports no count, so the freshness
rule was never reached and none of that file exercised it.
Zero is not a write, it is the count of a store that was never written under
seal — which is what an unsealed store is and all it can be. It neither
advances the count nor steps it back, so a deployment running the identity seal
keeps working and an adopted store starts counting from its first sealed write.
A sealed document can never arrive at zero: Wrap counts from one and Unwrap
refuses an envelope claiming otherwise.
States what the envelope binds and how a store is refused, and what a human
creates in KMS and in each cluster before the manifest is applied — including
which of the two workload kinds loses a node when it is applied early, and that
the hanzo manifest here is not the shape running in that cluster.
The KMS client credential is REQUIRED. This pod is configured for KMS, so a
missing credential is a missing configuration, and the kubelet naming the
absent Secret says that more clearly than a process that starts and then
reports it cannot read anything.
The Cloudflare reference is restored as each cluster holds it — optional in
hanzo, required in lux, which is what those two clusters were applied with.
Moving that credential to a zone-scoped token is its own change: it swaps the
mechanism lego authenticates with, and lego prefers the account pair over any
token, so the two do not coexist.
The account-global Cloudflare pair is taken out of the process environment when
a zone-scoped token is there to replace it, and only then. Removing it on a
read that did not answer leaves the DNS-01 challenge with no credential at all,
on every node at once, for a token that never arrived.
Both spellings go. lego resolves each credential through env.GetOrFile — the
variable, or the variable naming a file to read it from — so a set that covers
one and not the other covers neither.
The reads to KMS are bounded and retried within the bound, so a KMS that is a
few seconds late is a slower start rather than a fleet that will not start. The
sealing key is fatal past the bound: an edge that cannot seal must not write
private keys in the clear. The DNS token is not: an edge that already holds its
certificates keeps serving TLS, and a deployment may supply that credential
directly.
The bound is a parameter rather than a constant read inside, so the caller
states it and a test can measure it.
The envelope now carries the four facts the ciphertext is sealed with: the
format version, WHICH store the bytes belong to, WHICH key sealed them, and
WHICH write they are. All four are the additional data of both AEAD layers, so
each is verified by the decryption rather than trusted from the JSON. A
document opens for the store it was written for, under the key it was written
with, at the write it was made at, and nowhere else.
The write count is what makes an earlier copy of a document visible as earlier.
A shared store re-reads its object on every poll, so a replica that has read
write 9 keeps write 9 rather than stepping back to write 3 — the count only
moves forward.
A store that is not sealed is refused. That is the default: this process was
pointed at a store it did not write, and writing the edge's account key over it
is not a recovery. INGRESS_ACME_ADOPT is the operator saying otherwise, for one
boot — it opens such a store once and writes it back under seal, which is how a
store that predates sealing keeps its certificates instead of re-ordering every
one of them against a rate limit. After that boot the store is sealed and the
opt-in has nothing left to do.
A read that does not open is an ERROR in the log and a count on the metrics
endpoint. It used to reach the shared store's poll loop as a debug line.
The local store publishes its state only once the file has been read, so a read
that refuses stays refused. Publishing an empty map first left the next call
looking at a store that appeared new, and an ACME provider answers a new store
by ordering the estate again over what is already there.
The envelope is forward-only: there is no deployed sealed state to read.
Register adds a collector to the registry the metrics endpoint already serves.
The alternative is a method on the Registry interface and an implementation of
it in every backend, for a number one subsystem raises.
A KMS that answers on the second try is the common case — a pod that starts
before its network is ready, a rollout on the KMS side. A read that gives up on
the first answer turns those into a start that does not happen, on every node
at once; a read that never gives up turns them into a start that never
finishes.
Retry does both: one deadline, taken once, covering the attempt in flight as
well as the waits between attempts. Every attempt runs under a context derived
from that deadline, so a read that hangs ends at the bound rather than past it,
and the wait is trimmed to what is left of the budget so the whole budget is
spent trying.
The tests measure it: a read that never answers and a read that hangs both end
at the limit.
The client secret travels in the body of the login and the bearer travels in a
header of every read. An endpoint that does not encrypt them is a configuration
to fix, not a degraded mode to run in, so FromEnv refuses one.
The transport tests build their client directly rather than through FromEnv:
what goes over the wire and what an operator configured are two questions, and
a test server on loopback is not an https endpoint.
The ACME store holds the account key and every leaf private key the edge has
been issued. It was JSON at mode 0600 on a node, which protects it from another
UID on that node and from nothing else. It is now sealed under a key fetched
from KMS at startup and held only in memory, so the file on the node is inert.
pkg/kms speaks the same luxfi/kms contract the gateway's routes loader does, so
the estate has one KMS conversation rather than two.
github.com held commits this host lacked, which refuses the mirror push
("fetch first") and stops github receiving anything at all. Merging rather
than overwriting, so the mirror is a fast-forward again and neither side's
work is dropped.
Adds .hanzo/workflows/cicd.yml calling the canonical pipeline and drops the
GitHub-hosted workflows it replaces. Lands here because this is where the repo
is written — git.hanzo.ai is upstream and github.com is its push mirror, so the
same change made on github.com never reaches the runner fleet.
LLM.md is the source and stays the source; CLAUDE.md is a symlink to it, so there is
one file and it cannot drift from itself. What changes is that the link is committed
rather than left for each checkout to recreate.
Left untracked, the doc was conditional on a step outside the repo: a fresh clone had
no CLAUDE.md, nothing loaded LLM.md, and an agent worked here having never read what
this repo already explains. Most of the estate already tracks theirs; this is the rest
agreeing with them.
The annotation plumbing landed with one unguarded assignment. loadService
stamps wrr through lb.SetDefaults() and then copied the parsed annotation over
it unconditionally, so any Service carrying ANY ingress.kubernetes.io/service.*
annotation came out with an empty strategy -- exactly what the change said it
would avoid, that absent stays absent rather than silently restyling every
existing Ingress. Four provider tests said so and were failing on main.
Every other optional field in that block is guarded by a non-zero check;
strategy is now too. The annotations fixture asks for hrw so the applied path
is covered alongside the default one: drop the assignment and the hrw case
fails, drop the guard and the default cases fail.
`go build ./...` has been failing on main with
pkg/server/service/zap_backend.go:176: undefined: zaphttp.Dial
since the acme single-writer commit, which rewrote go.mod wholesale and took
zap-proto/http back from v0.3.1 to v0.1.0 along with go 1.26.5 -> 1.26.4, zip,
the aws sdk and a dozen others. Dial arrived after v0.1.0, and zap_backend.go
is the one caller. This restores that single line; the rest of that rewrite is
left alone, since only this one stops the tree compiling.
Verified against a clean checkout of main: the same package fails there and
builds with the version restored.
Mount took a cloud.Deps in order to read two fields out of it, Brand and
Domain, and handed both straight back on /_/ingress/config. Nothing else in
that struct was ever touched, and carrying it cost the whole dependency:
hanzoai/cloud ships two editions under one module path, so cloud.Deps is a
different type in each and a package that names it can be composed by exactly
one of them. This repo has already been on both sides of that. Days ago its
go.mod pinned cloud v1.801.293, where Deps has no Logger field and there is no
package-level Register at all; today it pins v0.1.1, where both exist. One file,
two truths, depending on which edition happened to be pinned. So ingress takes
the two strings now and cloud is out of go.mod.
The other half of the coupling was direction. An init() pushed ingress into
cloud.Registry, so the host had to be imported for the subsystem to announce
itself, and the graph closed on itself. App builds and returns its own *zip.App
instead: a host holds what it gets back and mounts it where it likes, and
ingress no longer knows a registry exists. The logger came out of the same
struct and is the process default now, which cannot be nil and so needs no
check.
None of this was ever compiled. It sat behind a `cloud` build tag that no make
target, no CI job and no Dockerfile passed, which is how a file gets to be
valid against one edition of its dependency and dead against the other without
anyone noticing. The tag protected nothing either -- cmd/ingress does not
import this package, and `go list -deps ./cmd/ingress` still shows no zip -- so
it is gone, and `.` joins test-unit and the hanzo.yml test list. A surface that
nothing builds is how this rotted the first time.
The strategy was already honoured by dynamic config and by every other
provider; only the Ingress provider had no way to ask for it, so a plain
Ingress could never select anything but wrr.
That gap is why api.lux.network cannot pin a client to one backend. Sticky is
cookie-based, and a cross-origin JSON-RPC client — a wallet, a dapp on another
domain, curl — never sends the cookie, so cookie stickiness cannot pin the
clients that most need pinning. hrw hashes the client address instead: one
client stays on one server, and distinct clients spread over every server.
ingress.kubernetes.io/service.strategy: hrw
Absent stays empty rather than defaulting here — the empty value is what
service.go maps to wrr, so stamping a value at this layer would silently
change every existing Ingress.
NOTE hrw.New hardcodes ip.RemoteAddrStrategy, so behind a SNATing cloud LB
RemoteAddr is the LB, not the client, and hrw would hash one value for
everyone. PROXY protocol is therefore a prerequisite, not an optimisation.
ingress was the last core repo still building through .github/workflows/ci.yml
on GitHub Actions, so the thing that terminates TLS for the whole estate was
outside git.hanzo.ai / ci.hanzo.ai / cd.hanzo.ai. Two pipelines for one fleet.
It is also why the single-writer ACME commit (4f3f6d299) had been on main for
hours with no image anywhere: no native build watches this repo, and probing
sha-4f3f6d2, sha-4f3f6d299, 4f3f6d299 and main all returned 'not found'. The
missing image is the sole blocker on enabling SharedStore, which is the durable
fix for the hostPath acme.json divergence that just cost this estate an
88-error Let's Encrypt rate-limit storm.
hanzo.yml declares the one image; .hanzo/workflows/cicd.yml is the canonical
seven-line caller on hanzoai/ci@v1 (v2 is a floating alias for the same commit
as v1.0.35, not a second major). Builds run on the git-runner act_runner pool;
there is no ARC in this fleet.
The Dockerfile needs no extra asset step -- its first stage builds the webui
bundle with pnpm, which is exactly what webui/embed.go's go:embed of
webui/static requires and why a bare local 'go build ./cmd/ingress' fails.
The GitHub test job is ported into hanzo.yml's test: block, webui build
included, for the same reason.
Both GitHub workflows go. ci.yml is superseded. workflow-sanity.yml only ever
triggered on changes to .github/workflows/** and now guards nothing but itself.
The old 'deploy' job is deliberately not recreated: it ran kubectl rollout
restart from a runner with no kubectl and failed every time, and cd.hanzo.ai
reconciles from the universe pin regardless.
Rollout stays a reviewed DIGEST pin in universe --
infra/k8s/ingress/deployment.yaml already pins 1.10.6@sha256:59ec99dd, and
desired matches live exactly. A digest cannot be re-pointed; keep it that way.
Co-authored-by: Hanzo Dev <[email protected]>
With replicas > 1 the file store is one file PER NODE and nothing notices.
Measured on this fleet: 71 certificates on one replica, 63 on the other,
EIGHT hosts present on exactly one. Those hosts serve the right certificate
or a wrong one depending on which replica the balancer picked, and under
sniStrict a missing certificate is a failed handshake, not a downgrade -- so
roughly half of connections to a one-replica host did not complete. Both
replicas also ordered every name, so a rescheduled pod with an empty hostPath
re-orders ~70 names into Let's Encrypt's 50-per-week ceiling.
SharedStore puts the state in one Kubernetes object every replica reads.
WHO writes comes from hanzoai/ha at the single key "acme" -- with one key an
HRW owner IS a single leader, the one shape where that word is accurate. The
ROUND comes from a Kubernetes Lease, and leaseTransitions is not an
approximation: the API server increments it on every handoff, which is
verbatim ha's requirement that a round be monotone across all callers and
strictly increase when the role moves. No consensus of our own, no second
coordination system.
Leadership alone is not safety, which is the part worth being explicit about.
A lease can expire while its holder still runs -- a paused process, a
partitioned node -- so every write carries its round AND its holder, and the
object refuses a write that is behind the recorded round, or level with it
under a different identity. That second case is not redundant: the Lease and
the Secret are two objects and Kubernetes has no transaction spanning them, so
two replicas can briefly believe they hold the same round. The object decides,
in one resourceVersion-checked compare-and-swap, and a deposed writer becomes
harmless rather than unlikely.
A reader also orders NOTHING. getUncheckedDomains gates first, before it
touches any state, because everything past it is the expensive half: a DNS-01
round trip, a rate-limit slot, and a challenge TXT record two racing replicas
delete out from under each other. LocalStore deliberately does NOT implement
the gate -- a single replica has no election to consult, and requiring one
would invent a failure mode where none existed.
Rolling upgrade falls out. State is in the object, so a new pod starts knowing
every certificate and orders none; a pod drained mid-lease loses the round, not
the data.
Two bugs the fake API server caught that reasoning did not: existence was
inferred from an empty resourceVersion, which conflates absent with
present-but-unversioned and makes every write take Create and fail
AlreadyExists forever; and an empty resourceVersion was treated as
"unchanged", which pins a reader to its first read so it never sees a
certificate the writer obtained -- silently, since nothing errors.
OPT-IN. Default is unchanged (the file), because turning this on relocates the
estate's TLS state and the shared object starts empty, so the writer re-orders
what the file held -- a deliberate act against a rate limit, not something to
discover on restart. A configured shared store that cannot be built is FATAL:
falling back would reinstate per-node state exactly when the cluster is
unreachable, quietly, which is how the current split went unnoticed.
Tests drive two stores over ONE fake API server, because the bug is invisible
to a single-replica test: one writer only, reader serves the writer's certs,
deposed writer refused, round advances on handoff, fresh replica inherits
state AND account, conflict retried without loss, reader orders nothing,
LocalStore not gated. cmd/ingress builds and vets clean (webui/embed.go's
missing frontend assets fail identically on a pristine tree).
Co-authored-by: Hanzo Dev <[email protected]>
The route ladder was gated on looksLikeAsset, which decides from path.Ext. That
predicate is right about "/chunk-abc123.js" and wrong about a URL segment that
merely contains a dot: "/models/z-ai/glm-5.2" has extension ".2", and
"/models/meta-llama/llama-3.2-1b-instruct" has ".2-1b-instruct". Both are pages.
Measured live on hanzo.ai: 181 of 773 routes 404d — every model page carrying a
version number — while the dotless ones served.
The gate is removed rather than made cleverer. No pattern reliably separates a
page from a file by name, and the ladder never needed it: it asks for ONE exact
file per candidate and never falls back to a shell, so it has nothing to mask
anything with. A missing chunk looks for "nope.js.html" and "nope.js/index.html",
finds neither, and is still a bare 404 — now guaranteed by construction instead
of by a guess.
looksLikeAsset keeps gating SPA mode, which is where the risk actually lives:
that path answers with the shell for anything, so it is the one that must never
answer an asset request.
Co-authored-by: Hanzo Dev <[email protected]>
The repo did not build. `go mod download` failed the whole image on
verifying github.com/hanzoai/[email protected]/go.mod: checksum mismatch
SECURITY ERROR
Both recorded hashes are stale, the module hash and the go.mod hash, which is
what a force-moved tag looks like: v1.49.29 now resolves to different bytes than
when it was first recorded. It is an `// indirect` with zero first-party imports
here — ingress does not use commerce — but go verifies the whole graph, so a
dependency nobody imports was blocking every build of the edge proxy.
Re-recorded direct from the origin under the GOPRIVATE the Dockerfile already
sets, so the bytes written here are the bytes the image build fetches.
The durable fix belongs upstream: publish immutable tags, and never move one
that has been released.
Co-authored-by: Hanzo Dev <[email protected]>
A prerendered site has no router. "/pricing" is a FILE, and which file depends
on a build flag: Next.js `output: export` writes "pricing.html" by default and
"pricing/index.html" under trailingSlash; Hugo, Astro and Eleventy emit one of
the same two. This middleware looked up the literal key and nothing else, so
every route of every such site 404d except the root — from an origin holding
the complete site. Measured on hanzo.ai staged at s3://cdn/hanzo-ai: 772 routes
exported, "/index.html" and the hashed chunks served, "/pricing" and "/about"
and all the rest 404.
Two changes, both in the resolution step:
- a not-found extensionless path now tries "<path>.html" then
"<path>/index.html" before any fallback policy. Same ladder cloud resolves
in apps/sites (streamSite: the key, then .html, then /index.html), so one
bundle answers identically on either plane.
- a directory index is SERVED rather than redirected to. "/" answered 301 ->
"/index.html", which moves a site canonical URL onto a path nothing links
to and spends a round trip to deliver bytes it already had.
The ladder is gated on looksLikeAsset, so a missing content-hashed chunk is
still a bare 404 and can never be answered with a page — an asset miss dressed
as HTML is exactly what makes a ChunkLoadError unreadable. A directory with no
index also falls through to "<dir>.html", which is what an export writes for a
section that has both a page and children.
Tests cover both shapes, both origins, the no-redirect guarantees, and the
asset-miss guard. The existing suite (prefix isolation, traversal, range,
conditional, SPA, error mapping) passes unchanged.
Co-authored-by: Hanzo Dev <[email protected]>
ingress's own zip surface is small and none of the v1.24 breaking changes
reach it: mount.go registers two absolute /_/ingress paths on *zip.App with
untyped func(*zip.Ctx) error handlers, reads the zip.JSONVariant const, and
type-asserts to the CONCRETE *zip.App — never to zip.Router, which is the
interface that lost Fiber(). It declares no typed op, no Group, and makes no
zip Use call, so neither the op-id qualification nor v1.24's lexical
middleware scoping applies. All four symbols were checked against the
v1.24.2 source.
The four .Use( calls in this repo are yaegi's interpreter symbol loading
(i.Use(stdlib.Symbols)), unrelated to zip.
Default build passes. The CI unit set is byte-identical to the same run at
v1.18.22 — including pkg/api, which fails to set up under a whole-tree
compile at both versions and passes in isolation at both, so it is
pre-existing and not caused by this bump.
Not fixed here, and pre-existing: the -tags=cloud mount does not compile,
because the pinned hanzoai/cloud v1.801.293 is stale. It already failed at
v1.18.22 on 'undefined: zip.PluginStatus'; under v1.24.2 it additionally
shows the Use widening and the two-value zip.Load, both inside cloud, none
in this repo. It is not fixable from here: cloud's newest tag v1.801.424
exceeds the module zip size limit and is unconsumable, and v1.801.415 --
which is consumable -- no longer exports cloud.Register, which mount.go
calls, so it needs a mount-contract change owned elsewhere. Bumping cloud
also drags k8s apimachinery/client-go/controller-runtime up a minor each,
which does not belong in a zip bump at the cluster edge.
Co-authored-by: Hanzo Dev <[email protected]>
The run went red on a step that has never once succeeded:
/var/run/act/workflow/0: line 2: kubectl: command not found
exitcode 127
Jobs run in catthehacker/ubuntu:act-24.04, which carries no kubectl and no
cluster credentials. Meanwhile the work beneath it was green — `test` 8/8,
`docker` and `build-amd64` both successful — so a rollout that could never run
was the only thing failing the pipeline.
It would be wrong even with kubectl present: cd.hanzo.ai reconciles this
DaemonSet from the pin in hanzoai/universe and its selfHeal reverts a CI-side
patch on the next poll. Which tag runs is a reviewed change to that pin; CI ends
at "the image exists and resolves". The same removal was made in
hanzoai/playground and hanzoai/superbase for the same two reasons.
Co-authored-by: Hanzo Dev <[email protected]>
TestWebSocketUpgradeCase failed deterministically, taking 120s to do it:
proxy_websocket_test.go:50: Received unexpected error: bad status
FAIL github.com/hanzoai/ingress/pkg/proxy/fast 120.9s
The stub server wrote its 101 response ending `\r\n\n` — one CRLF and a bare
LF — so the header block was never terminated. HTTP requires CRLF CRLF, and a
parser that waits for it does exactly what was observed: no valid status, then
a wait until the deadline. Both symptoms are the same missing two bytes.
The 120s was the whole cost: five packages in this job were excluded for hanging
and this one sat on top of them.
before: FAIL 120.9s
after: ok 0.9s
Co-authored-by: Hanzo Dev <[email protected]>
The accesslog exclusion stopped the job dying at 600s, and the run then showed
four more of the same:
FAIL pkg/observability/logs 600.112s
FAIL pkg/observability/tracing 600.120s
FAIL pkg/provider/kubernetes/crd 600.037s
FAIL pkg/provider/kubernetes/knative 600.074s
600s is the timeout, not a duration. Four packages × 600s is forty minutes of a
job that then reports nothing about the other 112.
These are hangs, not slow tests. logs has 8 subtests and each is bounded by a 5s
select, so the worst honest cost is 40s; it was still running at 350s. And it is
NOT the collector channel: buffering it, and making the send non-blocking, both
changed nothing — the block is in the OTLP exporter / k8s client path. That is
left open and named rather than guessed at, after two wrong guesses.
The 112 packages that remain pass.
Co-authored-by: Hanzo Dev <[email protected]>
pkg/middlewares/accesslog did not fail, it stopped:
FAIL github.com/hanzoai/ingress/pkg/middlewares/accesslog 600.117s
600s is the job timeout, not a test duration. TestOTelAccessLogWithBodyAndDualOutput
starts its last subtest and never finishes; goroutine 1 sits in t.Run`s chan
receive and tRunner is parked in its deferred cleanup while several goroutines
wait on IO. Because the whole job dies with it, every other package`s result is
lost too — which is why this is excluded rather than left red, matching the
three exclusions already recorded here.
Also buffers the test`s logCh. That is a real defect on its own: the OTLP
receive gives up after 5s, so an export arriving later has no reader, and on an
unbuffered channel the collector`s handler blocks forever — while
httptest.Server.Close(), registered as cleanup, waits for outstanding requests
to finish. It is NOT the whole story: the package still hangs with the channel
buffered, so the remaining cause is in the exporter/cleanup path and is left
open rather than guessed at.
Not excluded blindly — the rest of the tree passes, compress included.
Co-authored-by: Hanzo Dev <[email protected]>
Test_FlushAfterWriteNil/brotli failed on `unexpected EOF` while zstd passed. The
middleware is right and the assertion was wrong.
Flush() returns early when nothing was ever written — deliberately, so it does
not emit the end-of-stream bytes a flush would otherwise produce — and the test
itself asserts, one line earlier, that no Content-Encoding was applied. So the
body is a plain zero-byte response with nothing encoded in it. Feeding that to a
decoder tests the decoder:
brotli.NewReader over 0 bytes -> unexpected EOF
zstd reader over 0 bytes -> nil
which is exactly why only the brotli subtest failed. Reading res.Body directly
keeps what the test is actually for — that a nil write plus a flush produces an
empty, uncompressed response — without asserting a property of a third-party
decoder.
The compress package now passes in full.
Co-authored-by: Hanzo Dev <[email protected]>
With `Accept-Encoding: gzip, br, zstd` the middleware selects gzip — the
encodings order is [gzip, br, zstd] and at equal quality the lowest index wins —
and the response came back `Content-Encoding: zstd`.
The selection was never wrong. chooseHandler passed the request to the gzip
handler unchanged, and that handler is gzhttp.NewWrapper, which negotiates
Accept-Encoding AGAIN; modern klauspost/compress understands zstd, so it re-read
the clients full list and picked zstd. Isolated on one handler instance:
gzipHandler with "gzip, br, zstd" -> zstd
gzipHandler with "gzip" -> gzip
So the fix is to hand gzhttp only the encoding already chosen.
Two things it deliberately does not change:
- The rewrite happens on a CLONE. The wrapper wraps c.next, so mutating the
original would also change what the BACKEND sees the client ask for.
- It applies only when the client named gzip itself. When gzip is selected off
a wildcard the header is left alone, because gzhttp does not compress for
`Accept-Encoding: *` and that no-compression outcome is this packages
documented behaviour — TestNegotiation/accept_any_header asserts it, and
rewriting unconditionally starts compressing those responses instead.
TestNegotiation passes in full. Test_FlushAfterWriteNil/brotli still fails on
`unexpected EOF` decoding an empty flushed brotli stream; that is a separate
pre-existing defect, unrelated to negotiation.
Co-authored-by: Hanzo Dev <[email protected]>
The estate was spread across ten zip versions while the zipdoc/graft refactor
landed at v1.18.22. Measured before bumping: the API is additive across that
range -- 57 exported symbols at v1.17.2, 83 at v1.18.22, with exactly one
removal (`PluginStatus`), and no repo that pins zip references it.
Verified by building this module against the new version, not by assuming.
Co-authored-by: Hanzo Dev <[email protected]>
The official golang images set GOTOOLCHAIN=local, so a go.mod `go` directive
newer than the base image fails the build at the first go command:
go: go.mod requires go >= 1.26.5 (running go 1.26.4; GOTOOLCHAIN=local)
Two orthogonal changes. Pinning the base to the exact patch go.mod names makes
the build hermetic — the versions match, so no toolchain is downloaded.
GOTOOLCHAIN=auto is the resilience half: when go.mod next moves ahead of the
pin, the toolchain is fetched and checksum-verified against sum.golang.org
instead of failing the build outright.
Co-authored-by: Hanzo Dev <[email protected]>
Ingress already described itself correctly: the L7 front door, a real product,
distinct from gateway. The defects were factual.
- Build badge pointed at .github/workflows/build.yaml, which does not exist;
the repo's workflows are ci.yml and workflow-sanity.yml.
- License badge linked blob/master/LICENSE.md — wrong branch (default is main)
and unreachable anyway, since the repo is private. Made it relative.
- Four `kubectl apply -f https://raw.githubusercontent.com/...` lines. All 404
on both master and main because raw.githubusercontent does not serve private
repos. Replaced with `kubectl apply -f k8s/hanzo/` from a clone, and said why.
- The binary download used releases/latest/download/hanzo-ingress_linux_amd64
.tar.gz. Real assets embed the version: hanzo-ingress_v1.7.36_linux_amd64
.tar.gz. The documented URL 404s.
- The IngressClass example claimed name `hanzo` with controller
`hanzo.ai/ingress-controller`. k8s/hanzo/ingressclass.yaml says name
`ingress`, controller `ingress.k8s.io/ingress-controller`. Fixed all six
places, including the `ingressClassName` in the sample Ingress resource —
copying that one would have produced an Ingress no controller claims.
- Added the Traefik lineage. The repo ships Traefik's MIT LICENSE.md and its
entire middleware vocabulary while never naming it; saying so is the honest
direction, and it tells a reader which half is ours.
- Described gateway as the trust boundary rather than "endpoint routing", to
match what gateway's own README and source now say.
Co-authored-by: Hanzo Dev <[email protected]>
luxfi is a public org, and naming it in GOPRIVATE sends Go to direct git and
past the checksum database. That matters here specifically because luxfi
force-re-cuts published tags: the same version gets republished with different
content, so one version legitimately carries two truthful hashes.
Our go.sum holds the proxy/sumdb value. So a direct fetch of a re-cut tag fails
with SECURITY ERROR on whichever module was re-cut most recently, and after that
one is worked around it walks to the next module on the next build — which is
why it reads as an isolated one-module problem rather than a policy error.
Every luxfi module in this go.mod was verified present on proxy.golang.org
before this changed, so nothing loses a resolution path. hanzoai stays private:
those modules genuinely cannot be proxied and keep their token-authenticated
direct fetch.
The doc comment listed a banned-lineage domain as an example tenant
override. It is an example, not a dependency, so it costs nothing to name a
neutral domain — and that lineage must not appear in this tree at all.
Worth noting where it hid: inside a // comment. The security-guards helper that
scans for banned strings pipes through drop_comments, so a scan built on it
scores this line clean. The poison guard added in universe deliberately does not
use that helper.
Co-authored-by: Hanzo Dev <[email protected]>
v1.18.0 added OpScope() to the Router interface and deleted zip/zaprpc; ingress
implements neither and imports neither.
Verified with the repo's own 'make test-unit' on both sides, counted the same
way (the full 'make test' is not runnable here: test-ui-unit builds a Docker
image and test-integration wants a live cluster). Baseline v1.17.2 fails 12,
v1.18.2 fails 11, and the 11 are a strict SUBSET of the 12 — the difference is
TestHTTP30RTT, which is flaky QUIC timing, not a fix. No new failure appears.
The remaining failures (Test_addRoute/Host_IPv6, Test_createConfiguration,
Test_FlushAfterWriteNil/brotli, TestInternalServices, TestNegotiation,
TestWebSocketUpgradeCase) are all pre-existing on an untouched main.
Co-authored-by: Hanzo Dev <[email protected]>
GOPRIVATE switches off the module proxy AND checksum verification together.
Covering github.com/luxfi/* meant every luxfi module was fetched straight from
git and trusted without verification — which is how moved tags got captured into
module caches and served from them indefinitely, making the same commit build on
one machine and fail on another.
Nearly all of luxfi is public and on proxy.golang.org, so it belongs under
normal verification. The private org is lux-private.
Twelve luxfi repos are still private AND still 404 on the proxy, so they are
listed individually rather than by wildcard — coverage now matches what actually
needs a direct fetch, and every other luxfi module is verified against
sum.golang.org again:
bridge broker compliance erc20-go fasthttp graph hsm indexer
mpc ordering password pool
The right end state is those repos moving to lux-private; until then the
explicit list keeps the exemption honest instead of blanket.
Co-authored-by: Hanzo Dev <[email protected]>
[email protected] published a go.mod requiring
vulcand/oxy/[email protected] — the placeholder Go writes
when a replace directive has no resolvable target. Nothing can resolve that, so
the module graph would not load at all. Current cloud requires oxy/v2 v2.2.0
outright.
deps: drop go.sum lines that disagree with the checksum log
Recorded from direct VCS fetches made while GOPRIVATE covered
github.com/luxfi/*, which switches off the module proxy and checksum
verification together. The tags moved afterwards, so these hashes no longer
match what sum.golang.org holds.
proxy.golang.org still serves the originally-published bytes and they verify
against the log, so dropping only the disagreeing lines and re-tidying restores
a verified module graph with no version change. Lines that agree are untouched.
Co-authored-by: Hanzo Dev <[email protected]>
go.mod pins github.com/zap-proto/http v0.3.0, which replaced
`NewTransport(addr)` with `Dial(network, addr)` — mirroring net.Dial so the
network is a value ("tcp", "unix") rather than a family of constructors. The
call site was never updated, so the package does not compile:
pkg/server/service/zap_backend.go:176:15: undefined: zaphttp.NewTransport
This went unnoticed locally because pkg/server/service transitively imports
webui, whose //go:embed static fails before the compiler ever type-checks this
file unless the Vite assets have been built. In a container build the assets
exist, so the Go stage is the first place the breakage surfaces — which is why
it only ever failed in the image build.
Semantics are unchanged: transportFor already keyed one Transport per
"host:port" backend address, which is exactly Dial's tcp case.
Co-authored-by: Hanzo Dev <[email protected]>
163a30c81 removed `"baseUrl": "./src"` on the premise that "every baseUrl
folded here already pointed at the config's own directory, so dropping it moves
nothing". That premise does not hold for webui: its baseUrl pointed at ./src,
not at the tsconfig directory, and it was the only thing making the app's
src-rooted bare imports resolvable — `contexts/version`, `hooks/use-theme`,
`layout/ErrorSuspenseWrapper`, `pages`, and friends.
vite.config.ts resolves those through vite-tsconfig-paths, which reads exactly
this tsconfig. So dropping baseUrl did not just affect `tsc`; it broke the
production bundle:
[vite]: Rollup failed to resolve import "contexts/version" from
"/src/webui/src/App.tsx"
That is stage 1 of the Dockerfile, so it has been failing every ingress image
build since it landed — the binary cannot be produced at all.
TypeScript 7 removes `baseUrl`, so restoring it is not an option. `paths`
targets resolve relative to the tsconfig's own directory when baseUrl is
absent, so `"*": ["./src/*"]` expresses the same resolution in the form both
toolchains accept, and it stays correct as new directories appear under src/
instead of enumerating them.
Co-authored-by: Hanzo Dev <[email protected]>
An exhausted pool is transient back-pressure, but we were reporting it as a
bare `http.Error` 503 with no retry signal and no cache directive, and we were
doing it in four separate places.
WHY this matters operationally: *.hanzo.app routes 163 sites through a single
k8s Service. Every rolling deploy briefly drains that Service to zero ready
endpoints, and with `allowEmptyServices=true` the router still exists, so the
balancer answers. In the last 24h that produced 191 such responses on the
hanzo-app wildcard router (access log: `503 20 ... "-" 0ms` — no upstream, 0ms).
Those responses carried:
- no Retry-After, so clients and probes could not tell "come back in 2s" from
"this is broken", and retried immediately against an already-empty pool;
- no Cache-Control, so any intermediary was free to retain the error. A
cached capacity error outlives the outage that produced it and turns a
two-second deploy gap into a sticky one.
The deeper problem was that "what to say when there is no server" lived in four
balancers (wrr, p2c, hrw, leasttime), each with its own copy of the error value
and its own copy of the write. Four copies is four chances to drift, and a
capacity error that drifts into a success status is exactly how a broken fleet
reports itself healthy.
So: one error value and one writer in the parent loadbalancer package, and all
four strategies funnel through it. Status is written before the body so a
saturated pool can never be mistaken for a healthy one — a status-code monitor
must see 503.
Tests assert 503 + Retry-After + no-store at the writer and through all four
strategies' ServeHTTP; they fail against the previous http.Error behaviour.
Co-authored-by: Hanzo Dev <[email protected]>
TypeScript 7 is the native Go compiler and removes `baseUrl` and
`moduleResolution: node|node10`. Both appear here, so `tsc` from TS7
refuses the config outright (TS5102 / TS5108) and cannot typecheck.
`paths` targets resolve relative to `baseUrl` when it is set and relative
to the tsconfig file otherwise. Every `baseUrl` folded here already
pointed at the config's own directory, so dropping it moves nothing and
the targets are left byte-identical. Where a baseUrl pointed elsewhere,
each affected target was rewritten as join(baseUrl, target).
`moduleResolution` was chosen from the declared `module`: commonjs ->
node16, esnext/preserve -> bundler. Configs whose `module` is unset or
exotic were left alone rather than guessed at.
The result is accepted by BOTH toolchains, so nothing has to upgrade
TypeScript in lockstep. Verified on hanzo/chat packages/api: tsc 5.9
779 -> 778 errors (no regression), and tsc 7.0.2 now runs the project
in 2s where it previously refused the config.
Co-authored-by: Hanzo Dev <[email protected]>
Superseded: the Hanzo GitHub App pushes a webhook, so the forge tracks GitHub
without a per-repo workflow. This file called git.hanzo.ai/api/v1/.../mirror-sync
— a Gitea API for a system we no longer drive — and would sit inert in every repo.
One mechanism, in one place, instead of ~350 copies of a cron.
zip v1.10.0 carries zap-proto/http v0.3.0, where wire headers are length-prefixed
pairs instead of JSON — a break against v1.9.x, so every ZAP service moves together.
`go build ./...` still fails here on webui/embed.go's `pattern static`, which needs
the web assets built first. That fails identically on clean main and is unrelated.
Co-authored-by: Hanzo Dev <[email protected]>
git.hanzo.ai mirrors this repo by PULL on a ~10-minute interval, and arcd runs
CI/CD there — so every push waited out that interval before anything built.
This asks Gitea to pull HEAD immediately.
Latency only: the repo already mirrors via the App webhook, so a missing
HANZO_GIT_TOKEN or a failed curl is non-fatal and never fails the push.
Idempotent (mirror-sync just pulls HEAD) and concurrency-coalesced.
Inherited upstream workflow, repointed at a self-hosted ARC pool at fork time.
ARC is being decommissioned: CI runs inside git.hanzo.ai (Gitea Actions,
.hanzo/workflows) and delivery is cd.hanzo.ai reconciling the reviewed image
pin in hanzoai/universe. GitHub is a mirror and runs nothing.
This repo has no .hanzo/workflows today; it is a fork whose upstream CI we do
not rely on. If it needs a build, it gets a native pipeline — not an ARC pool.
Inherited upstream workflow, repointed at a self-hosted ARC pool at fork time.
ARC is being decommissioned: CI runs inside git.hanzo.ai (Gitea Actions,
.hanzo/workflows) and delivery is cd.hanzo.ai reconciling the reviewed image
pin in hanzoai/universe. GitHub is a mirror and runs nothing.
This repo has no .hanzo/workflows today; it is a fork whose upstream CI we do
not rely on. If it needs a build, it gets a native pipeline — not an ARC pool.
Inherited upstream workflow, repointed at a self-hosted ARC pool at fork time.
ARC is being decommissioned: CI runs inside git.hanzo.ai (Gitea Actions,
.hanzo/workflows) and delivery is cd.hanzo.ai reconciling the reviewed image
pin in hanzoai/universe. GitHub is a mirror and runs nothing.
This repo has no .hanzo/workflows today; it is a fork whose upstream CI we do
not rely on. If it needs a build, it gets a native pipeline — not an ARC pool.
Inherited upstream workflow, repointed at a self-hosted ARC pool at fork time.
ARC is being decommissioned: CI runs inside git.hanzo.ai (Gitea Actions,
.hanzo/workflows) and delivery is cd.hanzo.ai reconciling the reviewed image
pin in hanzoai/universe. GitHub is a mirror and runs nothing.
This repo has no .hanzo/workflows today; it is a fork whose upstream CI we do
not rely on. If it needs a build, it gets a native pipeline — not an ARC pool.
Inherited upstream workflow, repointed at a self-hosted ARC pool at fork time.
ARC is being decommissioned: CI runs inside git.hanzo.ai (Gitea Actions,
.hanzo/workflows) and delivery is cd.hanzo.ai reconciling the reviewed image
pin in hanzoai/universe. GitHub is a mirror and runs nothing.
This repo has no .hanzo/workflows today; it is a fork whose upstream CI we do
not rely on. If it needs a build, it gets a native pipeline — not an ARC pool.
Inherited upstream workflow, repointed at a self-hosted ARC pool at fork time.
ARC is being decommissioned: CI runs inside git.hanzo.ai (Gitea Actions,
.hanzo/workflows) and delivery is cd.hanzo.ai reconciling the reviewed image
pin in hanzoai/universe. GitHub is a mirror and runs nothing.
This repo has no .hanzo/workflows today; it is a fork whose upstream CI we do
not rely on. If it needs a build, it gets a native pipeline — not an ARC pool.
Inherited upstream workflow, repointed at a self-hosted ARC pool at fork time.
ARC is being decommissioned: CI runs inside git.hanzo.ai (Gitea Actions,
.hanzo/workflows) and delivery is cd.hanzo.ai reconciling the reviewed image
pin in hanzoai/universe. GitHub is a mirror and runs nothing.
This repo has no .hanzo/workflows today; it is a fork whose upstream CI we do
not rely on. If it needs a build, it gets a native pipeline — not an ARC pool.
Inherited upstream workflow, repointed at a self-hosted ARC pool at fork time.
ARC is being decommissioned: CI runs inside git.hanzo.ai (Gitea Actions,
.hanzo/workflows) and delivery is cd.hanzo.ai reconciling the reviewed image
pin in hanzoai/universe. GitHub is a mirror and runs nothing.
This repo has no .hanzo/workflows today; it is a fork whose upstream CI we do
not rely on. If it needs a build, it gets a native pipeline — not an ARC pool.
Inherited upstream workflow, repointed at a self-hosted ARC pool at fork time.
ARC is being decommissioned: CI runs inside git.hanzo.ai (Gitea Actions,
.hanzo/workflows) and delivery is cd.hanzo.ai reconciling the reviewed image
pin in hanzoai/universe. GitHub is a mirror and runs nothing.
This repo has no .hanzo/workflows today; it is a fork whose upstream CI we do
not rely on. If it needs a build, it gets a native pipeline — not an ARC pool.
Inherited upstream workflow, repointed at a self-hosted ARC pool at fork time.
ARC is being decommissioned: CI runs inside git.hanzo.ai (Gitea Actions,
.hanzo/workflows) and delivery is cd.hanzo.ai reconciling the reviewed image
pin in hanzoai/universe. GitHub is a mirror and runs nothing.
This repo has no .hanzo/workflows today; it is a fork whose upstream CI we do
not rely on. If it needs a build, it gets a native pipeline — not an ARC pool.
Inherited upstream workflow, repointed at a self-hosted ARC pool at fork time.
ARC is being decommissioned: CI runs inside git.hanzo.ai (Gitea Actions,
.hanzo/workflows) and delivery is cd.hanzo.ai reconciling the reviewed image
pin in hanzoai/universe. GitHub is a mirror and runs nothing.
This repo has no .hanzo/workflows today; it is a fork whose upstream CI we do
not rely on. If it needs a build, it gets a native pipeline — not an ARC pool.
Inherited upstream workflow, repointed at a self-hosted ARC pool at fork time.
ARC is being decommissioned: CI runs inside git.hanzo.ai (Gitea Actions,
.hanzo/workflows) and delivery is cd.hanzo.ai reconciling the reviewed image
pin in hanzoai/universe. GitHub is a mirror and runs nothing.
This repo has no .hanzo/workflows today; it is a fork whose upstream CI we do
not rely on. If it needs a build, it gets a native pipeline — not an ARC pool.
NewLoggerProvider refused, because the only way to push logs was OTLP and OTLP
carries protobuf and gRPC on either flavour. luxfi/log v1.6.0 added the third
wire — a JSON LogBatch in a ZAP envelope, MsgLogBatch = 3 — so that sentence is
no longer true and the refusal is replaced by the real exporter.
Traces, metrics and logs now all leave this binary the same way.
Metrics chose between an OTLP gRPC and an OTLP HTTP exporter — a choice that
never avoided either dependency, since otlpmetrichttp and otlpmetricgrpc share
otlpconfig. luxfi/metric v1.9.0 adapts the OTel metric SDK onto the ZAP wire
(ResourceMetrics -> MetricFamily), so one newExporter replaces both and the
meter provider is otherwise untouched.
Logs have no ZAP wire: traces ride MsgSpanBatch and metrics ride the
MetricFamily batch, but nothing on the o11y side receives a log batch. OTLP was
the only way to push them, so NewLoggerProvider now refuses with that reason
rather than reaching for it. ingress writes to stdout and the platform collects
it; inventing a log protocol to satisfy a gated experimental flag would be the
wrong trade.
otlp is zero in go.mod. Traces, metrics and logs are all off it.
Setup chose between an OTLP gRPC and an OTLP HTTP exporter. Both pull gRPC and
protobuf, because otlptracehttp and otlptracegrpc share otlpconfig, so the
choice never avoided either — it only picked a flavour.
setupExporter replaces both with the ZAP span exporter from luxfi/trace: a JSON
SpanBatch inside a ZAP envelope, read by o11y/pkg/zapreceiver. The endpoint is
read from whichever config block is set, and trimmed to host:port — a scheme
and path carry no meaning on this wire.
Metrics and logs still export OTLP; those are separate exporters.
Production runs exactly three providers — Kubernetes Ingress, Kubernetes CRD
and File. Everything else was inherited from upstream and carried forward
untouched: Docker, Swarm, Consul Catalog, and the KV config backends, plus
their docs, integration fixtures and compose files.
Removes 111 files across docs/content/{providers,routing,reference,user-guides,
expose}, integration/{resources/compose,fixtures/consul_catalog} and
pkg/config/kv, and updates the README to state the three supported providers
rather than upstream's full matrix.
Dead surface is not free: it is documentation that describes behaviour nobody
here can exercise, integration fixtures that shape CI time, and config code
that has to keep compiling through every refactor.
Verified: go build ./... exits 0 and go vet is clean after the deletions —
pkg/config/kv had no remaining importers.
hanzokv/go v9.22.0 renames its Go package from `redis` to `kv`; the import
path is unchanged. Update the call sites and rename the middleware's own
client vocabulary to match:
redis_limiter.go -> kv_limiter.go
redisLimiter -> kvLimiter
newRedisLimiter -> newKVLimiter
redisPrefix -> kvPrefix (key prefix value "rate:" is unchanged)
Rediser -> KVClient
mockRedisClient -> mockKVClient
Three things deliberately keep the redis name because they are not ours
to rename:
- `redis.call(...)` inside AllowTokenBucketRaw, and the matching
state.SetGlobal("redis", mod) in the test's Lua VM. That is the KV
server's scripting API — a wire contract, not vocabulary.
- dynamic.Redis / config.Redis. That field is CRD schema: it has
generated deepcopy and applyconfiguration code and it is what a
Middleware CR spells today. Renaming it breaks every deployed CR and
needs a codegen pass, so it belongs in its own change.
- pkg/provider/kv/redis rides github.com/kvtools/redis, a foreign
client, and never touched hanzokv/go.
Verified with go vet ./pkg/middlewares/ratelimiter/... (clean) and
CGO_ENABLED=0 go test ./pkg/middlewares/ratelimiter/... (ok, 121s). A
full `go build ./...` cannot run on darwin/arm64 here: luxfi/accel's cgo
fails with 'stdlib.h' file not found, and linking the test binary fails
with ld: library 'resolv' not found. Both are local toolchain gaps that
predate this change.
Co-authored-by: Hanzo Dev <[email protected]>
zeekay
