cloud purges the tag site-<org>-<slug> when a site publishes and Cloudflare
answers it, "cloudflare purge ok" for site-hanzo-hanzo-ai on every hanzo.ai
publish, but no page served from s3://hanzo-sites/<org>/<slug> carried a tag,
so the purge flushed nothing and the edge kept each page for its s-maxage
after the store had the new one: 8 to 38 s on hanzo.ai. A page of such a root
now names the tag; an asset, whose name changes with its bytes, does not.
When the middleware decides not to compress a response (its type is
excluded or not included, it is already encoded, or its type does not
parse), Write sent the held status but left headersSent false, so close
sent it again. net/http logs that second call as "superfluous
response.WriteHeader call", once per such response: under a config that
compresses only text bodies, once per streamed completion.
The four branches now go through pass, which sends the status, marks it
sent, and writes the body through.
Co-authored-by: Hanzo Dev <[email protected]>
HTML was no-cache, which also forbids Cloudflare from holding it, so every
page view crossed the country to the origin. It is now public, max-age=0,
s-maxage=60: a browser still revalidates every time, and a cache told to keep
pages holds one for a minute. Sites publish in place and nothing purges the
edge, so a minute is the longest a publish waits there. Cloudflare holds no
page unless a cache rule names the host.
Co-authored-by: Hanzo Dev <[email protected]>
- Reads into the cache run at most storeConns (64, the client's kept-alive
pool) at once. A refresh with no free slot holds no bytes and its request
streams, as for an object over the size limit.
- A failed or refused read no longer marks the entry unchecked, so one
request costs one HEAD and a stream, not a second refresh.
- An encoding lands on whatever entry still holds the bytes it encoded, so a
revalidation of the same version no longer throws it away.
- A read fills a buffer of the size the HEAD reported and refuses one longer.
The deployed gateway answers a matching If-Match with 200 and a stale one
with 412 (hanzo-sites/hanzo/hanzo-ai/index.html, from a pod in ns hanzo).
Co-authored-by: Hanzo Dev <[email protected]>
Review of the first cut:
- Brotli ran on the request path, unbounded: a 2 MiB object took 128 ms and
45 MiB, and 256 concurrent first requests took the heap to 3.1 GiB. The
first request is now answered as identity and the twin is made behind it,
at most GOMAXPROCS/2 at once, skipped rather than queued when no slot is
free, with the window sized to the input: 2 MiB is 65 ms and 25 MiB.
- Absences shared the objects' bound, so a flood of random paths emptied
every site. They have their own 4 MiB.
- The read now carries the HEAD's ETag as If-Match, so a copy one write
behind fails the read instead of filing build A under build B's tag.
- An object whose read failed once was never held again while its ETag
stood; it is read again.
- A 412 on the brotli representation promised a Content-Length it never
sent. The label goes on at the status line, on a 200 only, and that
representation offers no ranges.
- A late writer (the outage fallback, the encoder) replaces an entry only
while it is still current. "br;Q=0" refuses br.
Co-authored-by: Hanzo Dev <[email protected]>
Every request to an object-store root cost a HEAD and a GET to S3, 80 ms at
the median, before a byte left the ingress. One process-wide cache, bounded at
256 MiB and shared by every site, now holds objects up to 2 MiB with their
absence: an entry checked within the last second is served from memory, an
older one costs one HEAD, and the bytes are read again only when the ETag
moved. Sites publish in place, so a publish is served within a second.
Concurrent misses ask the store once; a store that stops answering leaves held
objects serving.
Text is held brotli-encoded beside its bytes and sent that way to a client
that accepts br, as its own representation (ETag "<tag>-br", Vary). Cloudflare
was pulling 557 KB of docs HTML uncompressed across the country per miss.
Everything under /_next/static/ is immutable whatever its extension. Every
middleware shares one S3 client and its connection pool.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / cicd (push) Blocked by required conditions
Hanzo CI/CD / test (linux-arm64) (push) Waiting to run
CI/CD / cicd (push) Waiting to run
Hanzo CI/CD / test (linux-amd64) (push) Successful in 12m24s
Client-IP (read as HTTP_CLIENT_IP by CGI and PHP), Proxy-Client-IP,
WL-Proxy-Client-IP, X-ProxyUser-IP, X-Originating-IP, X-Remote-IP,
X-Remote-Addr and X-Envoy-External-Address are what IP-resolution
libraries fall back to. From an untrusted peer they go with the rest.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / cicd (push) Blocked by required conditions
Hanzo CI/CD / test (linux-arm64) (push) Waiting to run
CI/CD / cicd (push) Waiting to run
Hanzo CI/CD / test (linux-amd64) (push) Successful in 7m1s
forwardedHeaders stripped only the X-Forwarded set and X-Real-Ip from an
untrusted peer. Every other header a proxy writes about the client passed
through: CF-Connecting-IP, CF-IPCountry and the rest of Cloudflare's
namespace, True-Client-IP, RFC 7239 Forwarded, X-AppEngine-Country and the
address aliases IP-resolution libraries consult. A backend that keys a rate
limit on CF-Connecting-IP or a country rule on CF-IPCountry was keyed on
whatever a peer outside the trusted set chose to send.
From an untrusted peer the entrypoint now drops every such claim, matched
case-folded with '_' read as '-' (CGI and WSGI read CF_Connecting_IP and
CF-Connecting-IP as one variable). From a trusted peer nothing changes.
Co-authored-by: Hanzo Dev <[email protected]>
1.27 is the released toolchain and the tree already compiles clean on it, so
the go directive names it. Nothing here depends on a 1.27 language feature
yet; this is the floor moving, which is what makes generic methods, the wider
type inference and promoted-field struct literals available to the code that
wants them next.
Co-authored-by: Hanzo Dev <[email protected]>
Most of it was stale, not cosmetic: the docs taught `TRAEFIK_*` env,
`traefik.http.*` labels, `traefik.ingress.kubernetes.io/` annotations,
`.traefik.yml` plugins and `traefik_*` metrics. This binary answers to
`INGRESS_*`, `ingress.http.*`, `ingress.kubernetes.io/`, `.ingress.yml` and
`ingress.*` — a reader following the old text configured nothing.
Two were worse than stale. Gateway API examples gave `group: traefik.io` for an
IngressService backendRef, which kubernetes.go:1265 refuses against hanzo.ai.
The contributing guide built with `github.com/traefik/ingress` ldflags against
a module that is `github.com/hanzoai/ingress/v3`.
script/code-gen.sh globs `hanzo.ai_*.yaml` and matched nothing — the per-CRD
dumps had been deleted and never regenerated, so `cat` failed the script under
`set -e`. controller-gen v0.19.0 restores all ten; they concatenate
byte-for-byte into the committed definition and fixture.
CHANGELOG.md is untouched: 6,789 lines of imported release history, where a
rewrite would falsify the record rather than rebrand it. `@traefiklabs/faency`,
`traefik/whoami`, k3s's `--disable=traefik`, the `traefik.io` CRD group and the
MIT attribution stay — upstream's addresses, not our brand.
Co-authored-by: Hanzo Dev <[email protected]>
query-string 9 exports a single object, so the pagination hook reads stringify
off it, the way the return-to hook already does.
react-router 7 is one package. Every import names it, so the app and its tests
share one router instance and one context; react-router-dom and its v5 types
are gone.
Vite 8 resolves tsconfig paths on its own, so vite-tsconfig-paths is gone and
resolve.tsconfigPaths takes its place.
faency carries @types/react 18 as a dependency. A pnpm override holds the whole
tree at the React 19 types the app declares, so faency's components type as JSX
components again.
The rest is what React 19 and TypeScript 7 ask for: useRef takes an initial
value, the error fallback narrows the unknown it is handed, sha512's output is
narrowed once where it meets @noble/ed25519, and the test setup replaces fetch
through vi.stubGlobal.
pnpm build, pnpm tsc and pnpm test are green; eslint still waits on
typescript-eslint support for TypeScript 7.
Co-authored-by: Hanzo Dev <[email protected]>
staticFiles had one answer for every miss and the wrong status for the one it
was configured to give: with errorPage404 set it served that page with HTTP 200,
for pages and assets alike, which is why the Sites plane declares it for no host
— 30KB of markup answering a request for a chunk, and a soft 404 on every dead
address.
It now answers the way cloud's own site plane does (apps/sites notFound): a page
miss gets the site's 404 page with status 404, JSON gets a JSON 404, and every
other file a bare 404. What decides is the media type the path names, read from
the same table that types a served file — so /models/glm-5.2 is a page, not a
file with the extension ".2", and that one predicate now gates the SPA shell too
(a dotted client route used to 404).
Registers .map and .webmanifest beside the fonts: the runtime image ships no
system mime.types, so an unregistered extension reads as "no type", and this
decision is made from the type.
Co-authored-by: Hanzo Dev <[email protected]>
The static plane sets nosniff, which turns an unknown Content-Type from a
guess into a refusal. Go's built-in table stops at the web's first dozen
extensions and takes the rest from the system mime.types, which the
distroless runtime does not ship — so .md and .woff2 resolve on a
developer's box and answer empty in production. hanzoskills.com publishes
a markdown twin of every document, and a browser opening one was offered a
download instead of a page.
Co-authored-by: Hanzo Dev <[email protected]>
One toolchain across the estate. The tree carried 1.26.0, .1, .3, .4 and .5
at once, so which compiler built a binary depended on which repo it came
from.
Co-authored-by: Hanzo Dev <[email protected]>
The forge reads .github/workflows natively and github.com reads nothing else,
so one directory serves both planes instead of two conventions that disagree
about which one is live. The reusable in hanzoai/ci already lives there, so a
caller naming that path now resolves to a file that exists.
Co-authored-by: Hanzo Dev <[email protected]>
hanzo-build-linux-amd64, lux-build-linux-amd64, zoo-build-linux-amd64 and
every other org-prefixed spelling all meant the same machine. Which org a
job belongs to does not change which host can build it, so the org bought
one pool name per org per arch and each could go stale on its own -- the
evo host alone answered to fourteen names for one label.
One vocabulary now: linux-amd64, linux-arm64, macos-arm64, win-amd64. The
generic labels every plain workflow uses (ubuntu-latest, self-hosted) are
untouched, so naive CI keeps working.
Co-authored-by: Hanzo Dev <[email protected]>
hanzoai/ci keeps the reusable at .github/workflows/build.yml — the one path
both planes resolve, since github.com reads a reusable only from there and the
forge reads it too. The v1 tag moved with it, so a caller still naming
.hanzo/workflows/build.yml asks for a blob that is not in the tree:
InsertRun: read hanzoai/ci@v1:.hanzo/workflows/build.yml: object does not exist
The caller itself stays at .hanzo/workflows/cicd.yml on purpose: github.com has
no runner for these labels, so a caller placed there could never be scheduled.
Co-authored-by: Hanzo Dev <[email protected]>
OWASP Coraza with the Core Rule Set v4 embedded, declared as `waf` on
every provider: file, labels and the Middleware CR. Rules load Core Rule
Set, then files, then inline directives.
The engine state is written after every rule. The Core Rule Set's own
recommended configuration opens with SecRuleEngine DetectionOnly, so a
state written first is overridden by the rules it governs and the
firewall loads all of CRS and refuses nothing; detectionOnly on the
middleware is the one place the mode is decided.
geoBlock rode along: it was declared in dynamic config only, absent from
MiddlewareSpec, both deep copies and the CRD schema, so a Middleware CR
carrying it produced a middleware without it. The +groupName marker now
matches the registered group, which is what makes controller-gen's output
land where code-gen.sh reads it; the stale traefik.io_*.yaml dumps go.
CI asks for the arm64 runner git.hanzo.ai actually has and builds both
platforms.
hanzo-ingress is CGO_ENABLED=0 and statically linked. Alpine supplied three
things: ca-certificates and tzdata, which are data the binary reads; an account,
which is two files the kernel reads to name a uid it already enforces; and
libcap-utils, which existed for one line.
That line, setcap cap_net_bind_service=+ep, is redundant and it is measured
rather than assumed: the live deployment runs with securityContext capabilities
add NET_BIND_SERVICE, drop ALL, so the orchestrator grants the capability to the
process and a capability baked into the file grants nothing it does not already
have. The image's own ports are unprivileged regardless — 8080 and 8443, mapped
by the Service, which is what the line under the EXPOSE has always said.
The one posture this changes is docker run as non-root binding :80 with no
--cap-add. That is not how this is deployed, and the remedy there is
--cap-add=NET_BIND_SERVICE rather than a capability in the artifact.
zip 1.1.0, which this had been pinned to, is far behind what the rest of the
estate runs; http, log, zap and the kv client were likewise stale.
metric stays at v1.9.0 deliberately. v1.10.0 deleted NewOTelZAPExporter, the
adapter that gives a ZAP exporter the OpenTelemetry sdkmetric.Exporter shape,
and the surviving NewZAPExporter implements neither Temporality, Aggregation
nor ForceFlush. Taking the newer line means re-implementing here what upstream
removed, so the pin holds until metric carries the adapter again -- and that
pin is what caps zip at v1.24.3 rather than v1.36.
Two thousand and twenty-one packages carried more than one spec across this
estate — eslint alone in a hundred and four forms — and a package resolved twice
is two copies at run time, which is how react-hook-form came to report "two
different types with this name" from files nobody had edited. None do now.
The number lives in the catalog: a manifest names the package, the workspace
names the version. Where a manifest cannot reach a catalog it still carries the
one version, because the table is the source either way.
A version written into a manifest is written into every manifest that wants it,
which is how eslint came to exist in a hundred and four forms across this estate
and how a package came to resolve twice at run time. The catalog moves the number
out of the manifest: a manifest names the package, the workspace names the
version.
Four are held below their latest because the newer release cannot be used, and
the reason is recorded rather than the version alone: @noble/hashes 2 renamed
every subpath the web3 stack imports, @splinetool/runtime 2 depends on a package
that was never published, @dynamic-labs 5.4.1 the same, and react-resizable-panels
3 points its types condition at the bundle.
The social card asked for Inter first. That is not a face this estate
licenses, ships or ever drew, so a renderer with Inter installed painted
the wrong face and one without it silently fell through to system-ui.
Zen leads now; the same tail follows it.
help is the canonical default verb, and bare `make` here ran `generate
binary` — go generate plus a webui build — which is a heavy surprise for
someone typing `make` to see what a repo offers.
Safe because nothing invokes a bare make: the Dockerfile has none,
ci.yml calls `go test` directly rather than a make target, and no script
or doc references one. `make default` still builds exactly as before.
One line, deliberately. The #? help style is upstream's and it works, so
it stays; rewriting it to match another repo's would be churn, not a fix.
Co-authored-by: Hanzo Dev <[email protected]>
build aliases binary — the target that already produces this repo's
artifact (./dist/$GOOS/$GOARCH/hanzo-ingress).
The default is UNCHANGED: bare `make` still means `generate binary`.
Changing what it does is a behaviour change with no upside, and the repo's
own help target is kept as it is.
clean already removed the binary and dist/. cover.out is test-unit's
-coverprofile output — generated, gitignored, and left behind by every run.
NOT webui/static: that is what clean-webui is for, and wiping it costs a
full docker webui image rebuild.
Co-authored-by: Hanzo Dev <[email protected]>
webui/src/components/icons/Logo.tsx transcribed the block-H — five body paths plus the two shade slivers — into the component. It maps MARK_BLOCKS and MARK_SHADE now, so the geometry has one home and this file cannot drift from the mark every other Hanzo surface draws. Same fill and the same 0.7 sliver opacity; nothing about the rendering moves.
An adopted plaintext keeps the same digest as the pre-seal snapshot, so while
the process kept adopting, a rollback that replanted that snapshot would be
re-admitted — the same bytes adoption had just taken. Seal.Persisted, called by
both stores once a sealed write reaches disk or the Secret, ends adoption: the
store is sealed now, so a later unsealed store is a different store and is
refused. Reads of the sealed store are unaffected.
It fires on the write that LANDS, not the one attempted — a conflicted
SharedStore write does not reach it, so the retry re-read of a still-unsealed
object is unchanged. The signal is a method on the Seal contract rather than a
runtime type check: the store tells the seal a sealed write persisted, and what
a seal does with that is the seal's own concern; the identity seal does nothing.
The two things a human creates are the sealing key and the credential that
reads it; numbering them said less than naming them.
Adoption's entry is corrected to what the code does: one store per process, it
writes, and it adopts again after a restart while the setting is left on.
Adds what the declared state means today — universe sets no INGRESS_KMS_* and
no ingress-kms Secret for hanzo, so the new image there takes the identity seal
and warns. The image is safe to roll before the pre-flight; it just will not
seal until universe carries the env.
edgeSecrets ran on every boot, so an edge that orders no certificates still had
to reach KMS before it served anything — including routes whose certificates
come from a Kubernetes Secret, and plain-HTTP routes. It now returns the
identity seal when no resolver orders certificates, and reaches KMS only when
there is ACME state to seal.
Adoption covers ONE store. It admits the first unsealed document and the same
document again — a store is read more than once before the write that seals it
lands, and a replica that cannot write reads it every poll — but not a
different one. A seal left adopting was taking whatever plaintext it was handed
for as long as the process ran. The setting's comment now says what it does:
it writes, it covers one store, and it adopts again after a restart, which is
why it is removed once it has done its work.
One spelling for the KMS credential. The bare IAM_CLIENT_ID fallback undid the
reason the names are prefixed — a pod carrying another subsystem's IAM client
had it picked up here.
k8s/hanzo/deployment.yaml and service.yaml are deleted. They described a
hostNetwork DaemonSet named hanzo-ingress binding hostPort 80 and 443, while
what runs in that cluster is a Deployment named ingress declared in universe.
`kubectl apply -f k8s/hanzo/` appears in this README and in the sibling
gateway's Makefile, so the file was one command from taking :80 and :443 on
every node beside the workload that already had them. Universe is the one place
that says what runs; the cluster-level objects stay here.
The OTLP endpoint paths in the docs and the vendored Knative fixture are
restored: /v1/metrics is the OpenTelemetry receiver path, not a route of ours.
next reserved a number and recorded it at the same time, so a write that was
encoded and then refused — a resourceVersion conflict, a filesystem error —
left the store's own view of the document ahead of the document. The re-read of
those same bytes then looked like a step back, and every write after it was
refused: the writer keeps ordering certificates and stops persisting them, and
a restart reads a stale store and orders the estate again.
Reserving and recording are now two things. next returns the write about to be
made and records nothing; wrote records one that landed. One rule for both
stores.
The shared store's tests run under a real seal, which is what the deployment
is. Under the identity seal every document reports no count, so the freshness
rule was never reached and none of that file exercised it.
Zero is not a write, it is the count of a store that was never written under
seal — which is what an unsealed store is and all it can be. It neither
advances the count nor steps it back, so a deployment running the identity seal
keeps working and an adopted store starts counting from its first sealed write.
A sealed document can never arrive at zero: Wrap counts from one and Unwrap
refuses an envelope claiming otherwise.
States what the envelope binds and how a store is refused, and what a human
creates in KMS and in each cluster before the manifest is applied — including
which of the two workload kinds loses a node when it is applied early, and that
the hanzo manifest here is not the shape running in that cluster.
The KMS client credential is REQUIRED. This pod is configured for KMS, so a
missing credential is a missing configuration, and the kubelet naming the
absent Secret says that more clearly than a process that starts and then
reports it cannot read anything.
The Cloudflare reference is restored as each cluster holds it — optional in
hanzo, required in lux, which is what those two clusters were applied with.
Moving that credential to a zone-scoped token is its own change: it swaps the
mechanism lego authenticates with, and lego prefers the account pair over any
token, so the two do not coexist.
The account-global Cloudflare pair is taken out of the process environment when
a zone-scoped token is there to replace it, and only then. Removing it on a
read that did not answer leaves the DNS-01 challenge with no credential at all,
on every node at once, for a token that never arrived.
Both spellings go. lego resolves each credential through env.GetOrFile — the
variable, or the variable naming a file to read it from — so a set that covers
one and not the other covers neither.
The reads to KMS are bounded and retried within the bound, so a KMS that is a
few seconds late is a slower start rather than a fleet that will not start. The
sealing key is fatal past the bound: an edge that cannot seal must not write
private keys in the clear. The DNS token is not: an edge that already holds its
certificates keeps serving TLS, and a deployment may supply that credential
directly.
The bound is a parameter rather than a constant read inside, so the caller
states it and a test can measure it.
The envelope now carries the four facts the ciphertext is sealed with: the
format version, WHICH store the bytes belong to, WHICH key sealed them, and
WHICH write they are. All four are the additional data of both AEAD layers, so
each is verified by the decryption rather than trusted from the JSON. A
document opens for the store it was written for, under the key it was written
with, at the write it was made at, and nowhere else.
The write count is what makes an earlier copy of a document visible as earlier.
A shared store re-reads its object on every poll, so a replica that has read
write 9 keeps write 9 rather than stepping back to write 3 — the count only
moves forward.
A store that is not sealed is refused. That is the default: this process was
pointed at a store it did not write, and writing the edge's account key over it
is not a recovery. INGRESS_ACME_ADOPT is the operator saying otherwise, for one
boot — it opens such a store once and writes it back under seal, which is how a
store that predates sealing keeps its certificates instead of re-ordering every
one of them against a rate limit. After that boot the store is sealed and the
opt-in has nothing left to do.
A read that does not open is an ERROR in the log and a count on the metrics
endpoint. It used to reach the shared store's poll loop as a debug line.
The local store publishes its state only once the file has been read, so a read
that refuses stays refused. Publishing an empty map first left the next call
looking at a store that appeared new, and an ACME provider answers a new store
by ordering the estate again over what is already there.
The envelope is forward-only: there is no deployed sealed state to read.
Register adds a collector to the registry the metrics endpoint already serves.
The alternative is a method on the Registry interface and an implementation of
it in every backend, for a number one subsystem raises.
zeekay
