hanzo.yml stops publishing the o11y-site Sites project: o11y.hanzo.ai,
obs.hanzo.ai and dashboards.hanzo.ai redirect to platform.hanzo.ai/o11y.
Sign-out goes to IAM's end-session endpoint instead of a forward-auth's
/__guard/logout, and the "local" authz comment states the boundary as it is:
only cloud reaches a standalone runtime, with headers it minted from a
validated IAM bearer.
Co-authored-by: Hanzo Dev <[email protected]>
The {{$variable}} rewrite turned the Prometheus spelling into {{index .Labels "labels.service"}}, which no series carries, so every rule written that way rendered the label empty (" has failed its health probe"). Both $-forms now strip the labels. prefix, and {{ $x }} with spaces reads its name from the match rather than from trimming the braces.
Co-authored-by: Hanzo Dev <[email protected]>
The event plane folded every occurrence into event.fact, and the Sentry
module still read and wrote event.error. Its reads now select the error
signal of event.fact, mapping issue, env, origin and the severity band.
Errors enter through /v1/event only. The Sentry envelope and store
routes, DSN keys and their rotation leave the module; the consumer of
event.error files each fact through Ingest(org, product), which keeps
the issue lifecycle and returns the issues a batch created.
The test's organizations table now answers another org for the first row,
so it fails on any code that addresses alerts to the first org rather than
the rule's own.
Co-authored-by: Hanzo Dev <[email protected]>
SendAlerts addressed every alert to `SELECT id FROM organizations LIMIT 1`,
which is whichever org sorts first, not the rule's. That org's
alertmanager has none of the rule's channels or route policies, so the
alert sat on its default receiver and reached nobody, and that org's
members could see it. Alerts now carry the rule's own org id.
Co-authored-by: Hanzo Dev <[email protected]>
GetRecentLogs selected timestamp, resources_string and resource, which
event.log does not have, so every call answered 500 "Unknown expression
identifier timestamp". It now selects time, service and the k8s labels
from the attributes map, and binds org, the window and the limit as
arguments. The route answers 403 when the request names no tenant.
Co-authored-by: Hanzo Dev <[email protected]>
The v5 logs and traces statement builders read event.log and event.span,
where every row carries its org. Each read now binds `org = ?` to the
tenant: the validated X-Org-Id for a request (set by the Hanzo IAM
identity), or the rule's org name for a scheduled rule that reads logs or
traces. A read with no tenant is refused as forbidden instead of reading
every org. This covers list, time series, scalar and trace requests and
both span reads of the trace operator.
Co-authored-by: Hanzo Dev <[email protected]>
Every release since v1.5.72 died in this gate on three tests pinned to answers
the toolchain no longer gives, so no image or site was published for any of them:
- binding: from Go 1.27 encoding/json names a field from the value whose
UnmarshalJSON failed, so a nested custom type reports 'a', not 's.a'.
- publish: zip 1.37 answers a miss or a refused method with an RFC 9457 problem
document, and /mcp with plain application/json, protocol 2026-07-28 and the
jsonrpc member first.
- wiring: the console catch-all is registered through app.Raw(zip.MethodAll,
"/*", …) now, so the order check finds it by its "/*" path, not by the verb.
Co-authored-by: Hanzo Dev <[email protected]>
The guard once asserted a name where it now asserts the IAM subject, so a row
seated then is keyed by uuid5 of the name. The subject misses that row, the
create collides on the email, and ensureUser answered the collision with
success, leaving GET /users/me and every row keyed to the person on an id no
session asserts.
On that collision ensureUser now calls user.Setter.RekeyUser: in one
transaction it finds the org's row by email and moves it, its user_role rows and
every other row that references users.id onto the subject. SQLite checks the
foreign keys at COMMIT (defer_foreign_keys), so a reference left behind refuses
the whole move. The role grants follow the rows once they commit.
Co-authored-by: Hanzo Dev <[email protected]>
The derived-fields guide they pointed at lived on o11y.io, which
resolves to no server and is registered to another company. The
waterfall span's calculated-fields note and the ten CalculatedFields
descriptions now say what each field is and link nothing. That note
is what docs.hanzo.ai prints for the waterfall API's db_name, through
cloud's openapi.yaml.
Co-authored-by: Hanzo Dev <[email protected]>
An op learns its address from zip.OpOf and zip.AddressOf, so the claim table,
the five declaring verbs and the address bookkeeping are gone. A host that
declares one of these addresses itself now gets a Build conflict from zip.
Co-authored-by: Hanzo Dev <[email protected]>
The five claim verbs need the address an operation will answer at, which is a
root joined with a path. They used to ask the target for its prefix through a
struct written to answer with one — and that struct existed for a second reason
that has since gone: an operation with no declared id took a shape-derived one
qualified by the prefix of the occurrence it answered under, so a plain group
would have renamed every undeclared id. All 319 operations state their own id
now, so a declared id survives composition verbatim and the group is just a
group.
The roots are this package's own constants, so each group carries the one it was
made with and nothing can disagree about the address. The escape hatches name
their whole address and register through Raw, which says which method each one
answers.
The test that measured the gap reached zero and said so; it now holds the
property from the other side, that no operation goes undeclared.
Co-authored-by: Hanzo Dev <[email protected]>
o11y carried a second identity system: email/password sign-in, Google,
OIDC and SAML callbacks, self-registration, invites, reset-password
tokens, roles-on-users, SSO auth domains, and a pair of tokenizers that
minted and stored its own sessions. Hanzo IAM answers all of it, once,
at the edge. Two credential stores is one too many, and the one nobody
signs into is the one that rots.
What is left is Identity: WHO a request is, resolved from what the edge
asserted or from a service-account key, never issued here. The routes are
not merely unrouted — the handlers, stores, tables and templates behind
them are gone, and anonymous_test.go names each dead address so that
re-exempting one fails even though no route can come back.
The census moves with it: 366 addresses to 329, 353 typed ops to 319, ten
escape hatches to seven. zipdoc_gen.go loses the 34 descriptions whose
addresses left, because zip keys its doc registry by method and path, so
a description outliving its declaration attaches this module's prose to
whatever a host serves at that address. zipdoc_test.go pins that.
Cherry-picked from delete/o11y-identity 89854a68ae.
HIP-0132 unified o11y_logs, o11y_traces and o11y_metrics into one `event`
database and dropped them. The commit that restored the reads repointed
three files and said so; it was thirty. The rest went on naming databases
that no longer exist, so trace funnels, the Kafka and queue panels, the
v3/v4 logs and traces builders and the PromQL adapter each answered
Code: 81 UNKNOWN_DATABASE, and the golden tests agreed with them because
they pinned the same dead SQL.
pkg/telemetryplane is the one address for the name, imports nothing, and
is reachable from every depth that needs it. Its test walks the tree and
fails the build on a literal that names a dropped database, tests
included — a golden encoding broken SQL is a second vote for the bug, not
coverage.
Two service views turned a failed read into an offer to instrument an
already-instrumented application. A failure now shows the failure, and
onboarding is offered on a successful empty read.
Cherry-picked from salvage/blue-wt-o11y-readers 83ecbec03c.
A version written into a manifest is written into every manifest that wants it,
which is how eslint came to exist in a hundred and four forms across this estate
and how a package came to resolve twice at run time. The catalog moves the number
out of the manifest: a manifest names the package, the workspace names the
version.
Four are held below their latest because the newer release cannot be used, and
the reason is recorded rather than the version alone: @noble/hashes 2 renamed
every subpath the web3 stack imports, @splinetool/runtime 2 depends on a package
that was never published, @dynamic-labs 5.4.1 the same, and react-resizable-panels
3 points its types condition at the bundle.
publish reached the declaration app through
zip.AdaptNetHTTP(adaptor.FiberApp(d.Fiber())) — out through net/http and back
into a router that is native at both ends. Per request that is: the live
fasthttp request rendered into an http.Request, then a second pooled
fasthttp.Request taking the method, URI, host and every header, the body copied
twice, r.RemoteAddr through net.ResolveTCPAddr, a second pooled
fasthttp.RequestCtx, and every response header copied back one at a time.
dispatch hands the request already in hand to that router, which is what a
listener does with one.
Composing instead is refused rather than chosen: the declaration names the same
367 addresses the service registers, and Build will not take a program where two
definitions claim one address. TestTheDeclarationCannotBeComposedIn measures the
refusal instead of asserting it in prose.
The probe group loses its per-request adapter. probe called zip.AdaptNetHTTP
INSIDE its own handler, so every liveness poll allocated one and re-wrote zip's
terminal marker into a process-wide map to answer a request whose shape has not
changed since boot. It and hatch now share the package's one bridge, and the
installed health handler is read once — where the answer is chosen — so a
SetHealth landing mid-request can no longer make the branch and the call
disagree. runtime.go's answers() had no callers and is gone with it.
Both wires were pinned before the change and pass after.
TestPublishServesTheDeclarationVerbatim asserts each control path's status,
content type, Allow, the absence of X-Content-Type-Options and the body —
literal where short, byte-equal to a declaration app's own answer where long —
and the miss a dot segment, a bare "." segment and a doubled slash each produce,
none of which this router cleans. TestProbesAnswerVerbatim asserts every probe's
whole answer in each of the four states a host can leave the module in, refusal
included: text/plain and nosniff, which is what says an answer came from the
net/http side. The five single-fact health tests it subsumes are gone.
The sweep left these: comments calling an address that accepts requests a door,
and one saying a writer is refused "at the door" where it means the edge.
Test fixture values ("seam-door", "sub_door") and the English idiom "opens the
door for other business opportunities" stay — neither is the metaphor.
go build ./... exits 0 for both. o11y's `go vet` reports unreachable code in
generated parsers, which it reports identically on unmodified HEAD.
One word stood for several: the MCP server, one address that accepts requests,
the single way in, and the boundary a credential is validated at. Doc comments
here are lifted into OpenAPI descriptions and republished on docs.hanzo.ai, so
the ambiguity was reaching customers.
Each site now names its own sense — endpoint, entry point, the MCP server, the
edge, address — or takes the word the surrounding code already uses.
Comments and prose strings only. Identifiers keep their names, including the doc
comments that must open with the identifier they document. go build ./... passes.
Two addresses were still spelled at a root that is not this capability's, and
HIP-0139 §3 gives every route one home: /v1/<name>. Both close by fold, and
neither leaves an alias behind.
/v1/sentinel/* -> /v1/o11y/sentinel/*. The error-tracking face is twelve
concrete paths the runtime registers literally, plus the declaration table's
group over the same root, so one constant and the registrations under it move
together. It is one capability's second face, not a second capability: same
store, same runtime, same tenant rule. The ingest door is unaffected — a minted
DSN names /v1/event and always did.
/ws/query_progress merges onto /v1/o11y/query_progress. It was never a second
read: one handler answered both mount points, upgrading when the caller asked
and long-polling when it did not. The Upgrade is a property of the request, not
of the address, so two paths bought a second spelling and nothing else. /ws was
also the last address this service answered outside /v1, and the console's
api-prefix list drops it with the route.
The hatch census goes eleven to ten and the surface 367 to 366 — the numbers are
tests here, not prose, so they move in the same commit as the route.
One wire had two public addresses. The Sentry-wire hatch was registered under
sentinelRoot, so /v1/sentinel/{project}/envelope|store answered alongside the
real door — and a face is not a door. A face is read by a signed-in person and
answers about what is already stored; a door is knocked on by a keyed beacon
that holds no session and carries something in. Different callers, different
credentials, so they no longer share a root.
eventRoot ("/v1/event") is that door, and it is the SAME string mintDSN spells,
so the address a client is told about and the address this service opens are one
value. IngestWire matches on it, and still by METHOD + PREFIX + SUFFIX rather
than a bare prefix, so nothing else on either root is reachable through the
exemption — the faces stay gated, and so does /v1/event itself, the product
event door beside this one.
The /api/ segment stays where it is. A stock SDK appends its own fixed
/api/<project>/envelope/ to whatever DSN path it is given, so that spelling is
the protocol's and not ours to rename.
/v1/sentry named our product after somebody else's. The face is /v1/sentinel now
— issues, projects, stats, logs, traces, discover: the fifteen addresses a
signed-in person reads. One const carries it (relay.go's sentinelRoot), which is
why the whole surface moved from one line; the groups in telemetry.go and
sentryerrors.go derive from it and nothing restates a path.
The WIRE is untouched and stays untouched. sentry_key, X-Sentry-Auth and
application/x-sentry-envelope are a third party's protocol that we RECEIVE, not
a name we chose, so renaming them would break every SDK in the field and buy
nothing. Sentinel is what we call the product; the envelope is what arrives.
mintDSN now addresses /v1/event. A DSN is the INGEST door and the face is not —
@hanzo/event 0.3.29 derives its ingest URL from the DSN's own path, so this line
decides where every reporting client posts. It said /v1/sentry, which after the
rename would have pointed clients at a read surface. Ingest and face are
different nouns and no longer share a root.
zipdoc_gen.go's fifteen route keys were moved by hand, and that is not a licence
to hand-edit generated files: `go generate -run zipdoc ./...` FAILS on this tree
before any change of mine — claim.go:146's generic opGet wrapper makes the path
non-constant, so zipdoc refuses the root package. Verified against a stashed
pristine tree. The keys are literal strings and this is what a working generator
would emit; the generator is the real repair and is owed.
Every count the repo pins still holds: 353 typed ops, 353 published operations,
353 MCP tools, 367 routes. 155 packages green.
BREAKING CHANGE: the error-tracking read surface moves from /v1/sentry to
/v1/sentinel, and minted DSNs address /v1/event.
Verified against ghcr.io/hanzoai/datastore:26.6.1.1, the image these files run:
config /etc/datastore-server, data /var/lib/datastore, config root <datastore>,
client /usr/bin/datastore-client, and an entrypoint that reads DATASTORE_* and
nothing else. The mounts pointed at /etc/clickhouse-server and
/var/lib/clickhouse, which the image does not have, and passed CLICKHOUSE_* the
entrypoint ignores.
.devenv bound ${PWD}/../../../deploy/common/clickhouse/custom-function.xml; that
directory is deploy/common/datastore.
Service, hostname, volume, anchor and swarm-config names follow, and the cluster
XML replicas match the compose hostnames again.
The eight built-in integrations credited [email protected] and o11y.io, which
does not resolve; ours is o11y.hanzo.ai. The datastore integration is titled
Hanzo Datastore. Its id stays — an installed integration references it.
data-collected.json and the dashboards keep ClickHouseProfileEvents_* and
ClickHouseAsyncMetrics_*: those are the names the server binary emits, checked
in the running pod.
All three ZAP receivers narrowed Listen through a port parse into
NodeConfig.Port, which threw the host away and made a socket path
unrepresentable — the receiver could only ever bind TCP. zap already reads an
address's form to choose its network (a leading /, ./, ../ or @ is unix), and
BOTH emit sides already passed the address through untouched, so the receiver
was the only half that could not say "a path". Nothing failed loudly when it
could not: it quietly bound a TCP port instead.
Listen is now passed whole. A host:port still binds TCP, so every off-pod sender
is unaffected — the otel agent on 28 nodes, the gateway, and another cluster
through the otlz door all keep working, and the port parse still refuses a
malformed one. The socket is ADDITIVE; anyone planning to move :4317 onto a
socket instead would take the fleet's telemetry down.
What it buys is the in-pod hop. About 110 plugin processes and the o11y process
share a pod and a run directory, and their telemetry crosses loopback TCP today
for no reason — on an ear bound to 0.0.0.0 that any pod in the cluster can write
to. Over a socket the filesystem's own permissions decide who may write, which
is the discipline the rest of the plane already keeps (0700 directory, 0600
socket, SO_PEERCRED).
uds_test.go pins the property rather than the reading: a path produces a file
that is really a socket AND answers a dial, a port still binds TCP, and a
malformed port is still refused.
Every logs query answered 500 "failed to get logs keys". The metadata step ahead
of each one reads SHOW CREATE TABLE through hanzo-ds/sqlparser, and event.log
carries `el Tuple(label String, ...)` — a NAMED tuple, which only parsed in its
unnamed form, so the parse stopped at the first element's type and the whole
query failed. event.span carries the same column, so traces read back now too.
sqlparser v0.4.17 routes the named form to the column-list grammar its
parseNestedType already read. The test here is over this plane's own schema
shape, because that is the invariant worth holding: a table this service defines
must be one it can read.
ResolveIngest was rewritten to attribute ingest by the org's publishable key —
the same key /v1/event takes — instead of a per-project DSN watermark. Two
things did not come along.
The project's own status stopped being consulted. Disabling a project is how a
tenant stops ingest without rotating an org-wide key, and ProjectStatus says as
much where it is declared, so an existing row that is not active now refuses.
Only an EXISTING row can refuse: a project first seen on this ingest is
provisioned active a line later, and a store that errors must not be read as
permission.
TestResolveIngest_FailsClosed still built per-project versioned keys and
rotated watermarks, none of which the code has any more — that was the red.
It now covers the gate as written: no resolver installed, a key the resolver
rejects, an empty key, and a resolver that answers ok while naming no org, all
closed; a good key resolves to the org it names and provisions the project.
TestResolveIngest_DisabledProjectFailsClosed was GREEN and proved nothing —
with no resolver installed every call returns false, so its assertion held for
the wrong reason. It installs one now, and checked both ways: it fails without
the status check and passes with it.
Full suite green (`go build ./...`, `go test -count=1 -skip '^TestEmailRejected$' ./...`).
Co-authored-by: Hanzo Dev <[email protected]>
The metrics path costs INSERTS, not bytes, and nothing in it was shaped
for that. Every insert into the samples table fans out through the 5m and
30m rollup views — one insert becomes three parts — and the ZAP wire
carries one batch per RESOURCE, so insert volume tracks the number of
objects a fleet observes rather than the amount of data it produces.
Measured on hanzo-k8s 2026-08-08: ~518 writes a minute against a merge
pool already pinned 32/32 on a node at load average 43. event.metric_30m
is AggregatingMergeTree on 30-minute buckets, so merges re-aggregate the
same rows endlessly and its parts reached merge level 21,310 while
metric/series sat near 50. It crossed the part ceiling and ClickHouse
answered code 252 to EVERY metric write for an hour — app telemetry that
had been landing for months, not just the new collectors. A k8s_cluster
receiver, which models each object as its own resource, would have been
~16,000 inserts a minute.
Rows are cheap to concatenate; statements are not. WriteMetricsMany
builds each batch exactly as WriteMetrics would — same resource
attributes, same fingerprints — and lands the lot in ONE time_series and
ONE samples statement. A nil or empty batch is skipped rather than
refused, so one dead sender cannot discard a window of everyone else's
metrics.
Additive: nothing calls it yet. The caller change is cloud buffering what
arrives in a window instead of writing per batch.
buildRowsMany is split out so the contract is testable without a
datastore connection, and TestBuildRowsManyPreservesPerBatchIdentity is
mutation-proven — reuse one batch's resource for all and it fails on the
env leaking staging -> prod, which is the exact hazard of coalescing.
Every metric read through PromQL failed with "Database o11y_metrics does not
exist": dashboards, service health, ingest, memory, plane throughput and alert
delivery all returned an error the UI renders as an empty chart. 96 failures in
30 minutes on one pod, every one of them this.
Metrics moved into the one event plane and telemetrymetrics moved with them --
DBName is "event", the tables are series and metric. This package did not,
because it kept its OWN copy of those names, and a copy has no reason to change
when the original does. It went on addressing signoz-era names in a database
that the rename left behind.
So it no longer keeps a copy. databaseName and samplesTableName alias the
telemetrymetrics constants, and the rollup choice is telemetrymetrics'
WhichTSTableToUse -- the same function every other metrics builder calls -- so
one window resolves to one table service-wide instead of two selectors drifting
apart. That also gains the 1-week tier this package never had: a 30-day PromQL
range read the 1-day rollup because the duplicate stopped at 1 day.
The tests assert the rendered SQL. Nothing did before, which is exactly why a
dead database name survived a rename and shipped.
The Sentry envelope plane authenticated against a per-project DSN watermark (its
own secret, per-project keys, provisioned out of band) — a SECOND attribution
model braided beside the org publishable key /v1/event already accepts, and the
reason the plane was dark (every DSN 401'd). Decomplect: delete the watermark
branch, resolve the presented pk- key to its org through the host-injected
resolver (the SAME seam /v1/event uses), and auto-provision the DSN's project
under that org on first ingest (the zero-onboarding move o11y makes for a tenant).
One key, one endpoint: the key that feeds analytics + insights feeds errors too.
- iamidentn: export OrgUUID (org slug->UUID mapping the keyed path needs).
- implsentry: SetIngestKeyResolver seam (mirrors o11y.SetHandler), rewritten
ResolveIngest, ensureProject (idempotent, fail-soft). Reads/projects API unchanged.
Co-authored-by: Hanzo Dev <[email protected]>
o11y's surface was stated twice: once as the implementation (367 routes through
routing.Router) and once as the module's own declaration of it (353 typed ops +
11 hatches + 3 probes, the same 367 addresses with an input, an output and their
prose). The declaration had to REACH the implementation, and the only way it
could was to speak HTTP to the WHOLE service — build a request, hand it to one
http.Handler that was the entire router, and let that router match the path a
second time to find the handler the declaration had already named. An in-process
round trip, across a net/http<->fasthttp bridge, through a recorder, to answer a
question the caller had the answer to.
A registration is a map from address to handler. routing.Table.Handler reads it
as one and o11y.SetRuntime hands the whole table over, so the second match is
gone and with it the failure class the reachability census exists for: a request
cannot lose its route to a router that is never consulted.
Two things could not be said before and now can:
- "this address is not implemented." A declared address the runtime does not
serve came back as the runtime's own 404, indistinguishable from a caller's
typo. It is a nil at the seam now. The two sides HAD drifted: three addresses
the declaration names {traceId} were registered {traceID}, invisible because
a router matches by POSITION and a segment's name never reaches it.
- "the runtime is right here." The standalone server IS the runtime and could
not install itself as that one handler without an op relaying into the router
already serving it, so it installed nothing — and every op on its /mcp
surface and its by-name call plane answered 503 in the binary holding all 367
handlers. It installs its own table now.
The address is stated ONCE, at the registration, and rides the call: claim.go's
verbs wrap every op in addressed, which puts the address into the context, and
relay reads it there. relay takes no method and no path — which door was knocked
on is not something an op's body decides. That retires 706 spellings of 367
addresses; the 119 hand-built concatenations are rendered by zip.Address from the
same input zip bound the segments into, so the round trip is exact by
construction rather than by eye.
Reached BY NAME there is no router in front of the handler, so the two chain
members that need the route — Audit keys its record on the template, Resource
resolves the declared resources — would read an empty one, silently. Table.Handler
hands back the chain carrying the template it was registered at, written through
coretypes.SetRoute: the same representation bind writes for a matched request,
one read on the other side. TestByNameCarriesTheSameRouteAsMatching drives both
doors against one registration and compares them to each other.
Addresses did not move: the census — 367 fiber routes, 367 declared, 353
operation ids — is byte-identical to the parent. 155 packages ok, 0 fail, the
same set by name as the parent.
wantQueryServiceRoutes closes the equality argument the sibling address test
makes. It rested on four facts and 134 was a sentence, so a route deleted in
pkg/query-service/app left both address tests green while the declaration went on
naming an address nothing serves — which costs more now that it is a 503 rather
than a 404.
Co-authored-by: Hanzo Dev <[email protected]>
App.Test calls prepare, which installs the deferred projections — /mcp, the
OpenAPI document, the op-call plane, the plugin route. Reaching through Fiber()
skips that, so a test written against the escape hatch cannot see a surface
production exposes, and the closer it gets to the real program the more
confidently it reports the wrong thing.
12 call sites across 7 files.
Full suite green on this host. The `unreachable code` vet notices in
pkg/parser/filterquery/grammar are pre-existing and in generated parser output.
Co-authored-by: zeekay <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
Four changes zip v1.23 forced. Only the first is a compile error; the other
three are silent, and two of them would have shipped a broken service.
1. Prepare() -> Build() error. publish already returned an error, so the verdict
propagates: a composition this server cannot serve now fails at startup
naming both claimants, where before it could only be found by the routes
going missing.
2. routing.Router read its prefix back off zip (OpScope().Prefix). A group is
now an App INCLUDED at a prefix rather than a router carrying one, so
OpScope().Prefix is empty for every group — and reading it STILL COMPILES.
The Table would have silently recorded every nested route at its leaf path.
The prefix is composed at Group now, by the same join Handle records with.
3. THE OPERATION IDS. zip qualifies an op's id by the prefix of the occurrence
it answers under, so every id declared under app.Group(o11yRoot) became
"v1.o11y.CreateRole". That is 353 renamed MCP tools, OpenAPI operationIds,
CLI commands and generated SDK methods — in a patch release, with no opt-out
(an explicit WithOperationID is qualified too). The rule is right for a
composition; o11yRoot is not one. It is this service's own address, and an
occurrence at the ROOT prefix is unqualified. So the ops register on the app
with OpScope.Prefix doing what it documents — prepend to the op's path. Same
method, same full path, same middleware, and the id the declaration wrote.
One type (relay.go `under`), 18 call sites, no route literal touched. It is
also what mount.go's PATH UNTOUCHED already claimed and what mountHealth
already did by hand.
4. ONE ADDRESS, ONE DEFINITION — and the standalone server declared 367 twice.
The service's own handlers SERVE those addresses and the published table
DESCRIBES them; zip has no way to spell that difference ("the route and the
op are the same fact recorded once"), so Build refused the program and the
community server could not boot. The two facts now live on two apps: the
declaration builds on an app of its own and only its CONTROL routes — the
document, /docs, /mcp, the call plane, the plugin declaration — are served
from the listener. zip keeps control routes off the walk, and none of the
five is one of this service's 367, so nothing can shadow or be shadowed.
This also makes ORDER IS THE CONTRACT structural instead of positional: the
declaration is not on the listener at all, so there is no order to get wrong,
and livetail, the long-poll and the chunked export still stream unbuffered.
TestPublishServesTheDocument now reads the tool count from a tools/list over
the served door rather than len(app.MCPTools()) — an in-process accessor on the
app the test just built is the shape of proof that file exists to reject.
Measured: go build ./... clean; go vet ./... clean apart from pre-existing
"unreachable code" in the generated parsers; go test ./... 155 ok, 0 failed.
The escape QuoteIdent applies is not ours to choose — the datastore driver
already spells it, as colEscape/colUnEscape in hanzo-ds/go
lib/column/column.go. The test now asserts QuoteIdent writes exactly what the
driver would and that the driver's own inverse recovers the name, so if that
dialect ever moves this fails instead of quietly emitting an identifier the
server reads differently than we wrote it.
The comment claimed "order matters" between the two replacements. It does not,
and the claim was the dangerous kind of wrong: it invites someone to unroll the
Replacer into two sequential ReplaceAll calls and put them in the "right" order.
A Replacer is SINGLE PASS over non-overlapping patterns, so every input byte is
rewritten exactly once and order is irrelevant. Sequential ReplaceAll would be
the broken form — escaping the backtick first and the backslash second doubles
the backslashes the first call just introduced. Said plainly now, with the
warning attached to the thing it protects.
No behaviour change.
Co-authored-by: Hanzo Dev <[email protected]>
The v3/v4 builders are live — querier.go wires tracesV4.PrepareTracesQuery — and
they carry the same bug as the v5 statement builders, one layer down and one
position wider. Three caller values reached ClickHouse raw:
- the ORDER BY DIRECTION. Direction is a bare string off the request body and
every builder interpolated it beside the column, so
"asc,(select count() from system.tables)" was an entire extra ordering term.
It needed no quote and no backtick to get there — the cheapest of the three.
- the ORDER BY / GROUP BY / alias COLUMN, in backticks it can close.
- the ATTRIBUTE KEY, in the single quotes of the map and JSON accessors. That
one lands in the WHERE clause, so it is not an ordering oracle but a
predicate the caller writes: mapContains(attributes_string, 'a'),(select 1))
or ('1'='1') parses and runs.
The asymmetry is stark here. On the very lines that build a filter, the VALUE
goes through utils.QuoteEscapedString and the KEY beside it goes in bare. And
the allowlist that appears to gate the column — tagLookup — is built out of the
caller's own GroupBy, so a name repeated in both fields validates itself.
utils.AddBackTickToFormatTag is the pass-through underneath, and it declined to
quote in two ways: a name with no "." and no "-" came back BARE, and a name
already wrapped in backticks came back AS IS, which let a caller supply its own
quoting. Neither branch escaped a backtick. It now returns bare only for what it
was assuming — a plain identifier — and quotes and escapes everything else.
Direction.Keyword() is the guard for the direction, on the value rather than at
each of the sixteen sites that interpolate it. It folds case to compare because
callers send both, and returns the ORIGINAL on a match, which is safe precisely
because the match succeeded: a string whose lowercase is "asc" or "desc" is
those letters and nothing else. Anything else renders as empty, which is what an
absent direction already rendered as.
Identifiers and literals go through the same querybuilder.QuoteIdent and
EscapeLiteral the v5 builders now use. A real key takes the path it always did,
so the golden SQL in every one of these packages is unchanged.
Co-authored-by: Hanzo Dev <[email protected]>
A telemetry field key name arrives as JSON on /api/v5/query_range and is
replayed out of stored dashboards and alert rules. It lands in two syntactic
positions and a bound parameter can occupy NEITHER:
- a backtick-quoted IDENTIFIER — the SELECT alias, the GROUP BY term, the
ORDER BY term;
- a single-quoted LITERAL — the accessors that read one attribute out of a
column, JSONExtractString(labels, '<name>') and friends.
Nothing quoted it. The rendering was written inline about forty times, always
as "`"+name+"`" or '%s', and the one call that sat next to a name —
sqlbuilder.Escape — replaces "$" with "$$" for the builder's own placeholder
syntax and is not SQL escaping. Measured against the real builder, a group-by
name of
le`,(select/**/grouparray(name)/**/from/**/system.tables)/**/as/**/`x
emitted that subquery as an ADDITIONAL SELECT COLUMN, whose rows come back in
the response body; "/**/" stands in for the spaces the builder strips and the
payload is all-lowercase so nothing folds it. A name carrying "'" breaks the
labels accessor the same way, and one ending in "\" eats whichever delimiter
closes it.
The request validator is not where this can be fixed. The rule evaluator builds
a QueryRangeRequest and calls the querier directly, so a stored rule's name
never meets it; and validateOrderBy, which does gate the DIRECTION strictly to
asc|desc, gates the sibling key NAME against an allowlist it builds out of the
caller's own GroupBy — so a name repeated in both fields validates itself.
Metrics settles it: a metric label is free-form, an unrecognised name falls back
to the labels column BY DESIGN, and there is no key allowlist to fail against.
So the guard goes where the name becomes text. QuoteIdent and EscapeLiteral are
the one renderer, in querybuilder, which every signal package already imports;
the ~40 inline spellings now call them. Both escape the backslash BEFORE the
delimiter, or a trailing backslash would consume the delimiter being added. A
name with none of those characters — every real attribute key, which is a dotted
identifier — renders byte for byte as it did, which is why the golden-SQL suites
in all six signal packages are unchanged.
The test asserts on emitted SQL and structurally rather than textually: the
payload MAY appear, quoted, as the inert name of a label nobody has, but it must
never appear in the executable skeleton — the text left once every quoted region
is removed. That is the property an injection breaks, so it fails if any layer
underneath stops escaping.
Co-authored-by: Hanzo Dev <[email protected]>
impersonationidentn.Test() matches every request and GetIdentity resolves every
caller to the ROOT user. That is the mode's contract — a single-user local
deployment with no authentication — and nothing at request time constrains it:
the resolver tries providers in order and this one matches unconditionally, so
if it is in the list at all, anything that fails to match IAM falls through to
root.
The only thing standing between that and a production host is Config.Validate
refusing the combination at boot, which pkg/o11y's validateConfig runs over every
config before the process serves. That control had no test. A control with no
test is one that can be relaxed by accident, and relaxing this one would not look
like a security change in review.
Pinned per forbidden pairing (IAM, tokenizer, api key), plus the two facts that
keep the guard honest: impersonation alone still validates, so the mode it exists
for is still runnable and no pressure builds to weaken the check; and it is off
in the shipped default.
Deliberately NOT changed: Test() still matches unconditionally. Gating it on an
opt-in header would add no factor — whoever can set Impersonation.Enabled can set
a header just as easily — while breaking the one mode it exists to serve. The
boot-time exclusion is the control that actually costs an attacker something.
Co-authored-by: Hanzo Dev <[email protected]>
publish_test.go proves what publish does — 353 operations, 353 MCP tools — and
every assertion it makes calls publish directly. That is the shape of the defect
it was written for: 353 typed ops once shipped in files nothing called, and an
uncalled package-level func is legal Go, so the package built and the running
server published nothing. A test that calls the function cannot see a missing
CALL.
Deleting `publish(app)` from createPublicServer leaves the whole o11y suite
green. Measured, not assumed. The only evidence the link existed was `go tool nm`
on ./cmd/community, run by hand, in no CI.
State the invariant where it can fail. createPublicServer is the composition
root for this surface, and two facts about it are load-bearing: it calls publish,
and it calls it BEFORE the console catch-all, since the router matches in
registration order and a terminal route registered first would swallow every
published path. Both are read from the source — constructing a real Server would
need the whole module graph, and the source is where the defect lives.
Fails on the exact deletion, and fails again with the call merely moved below
app.All("/*", …).
Co-authored-by: Hanzo Dev <[email protected]>
The tuple set is the whole of the authorization state and it is in-process. A
grant writes BOTH the tuple and a durable user_role row — user.Setter calls
authz.Grant and createUserRoleEntries in one operation — but only the row
survives a restart.
The founding grant rides iamidentn's first sight of a principal, and ensureUser
returns early once the durable user row exists. So after any restart every
previously-seated principal held a durable role and NO tuple: exact-match
membership is the entire check, so ViewAccess, EditAccess and AdminAccess all
answered 403 and o11y was unreachable for everyone until each user was
re-provisioned by hand. This package's doc asserted the set was "re-populated on
the first request after a restart"; nothing did that. o11y runs one replica, so
this was a latent outage rather than a live one — and the same hole is what makes
a second replica answer differently from the first.
Rehydrate from the durable rows at Start. That makes the rows the single source
of truth and the map a cache of them, so a restart and a scale-out both converge
on what is stored. Reading exactly what is stored is also what keeps this a
rehydration rather than a re-grant: re-issuing the founding admin grant would
silently restore a role that had been revoked. Service accounts are covered too —
they are checked by the same path and are the callers least able to notice a
lockout.
A failure to load is reported, not fatal: returning from Start tears the whole
process down, and in the embed that process also serves iam, commerce, ai and
the gateway. Carrying on leaves the set as whatever loaded, which denies rather
than grants.
Co-authored-by: Hanzo Dev <[email protected]>
relay.go has said since the collapse that wrote it: "One function, one place:
every typed op in this package reaches the o11y runtime through relay and
through nothing else." That was aspirational. Four more bodies sat in the op
files — infraRelay, metricsRelay, rulesAlertsRelay, integrationsRelay — each a
copy of relay differing only in concatenating o11yRoot itself.
So when a handler turned out to be a SERVER seam and needed a request-target,
the fix landed in one of the five. The other four kept handing the runtime a
request with RequestURI empty; adaptor.FiberApp copies that field into fasthttp
verbatim, the path was erased, fasthttp normalized it to "/", and no API route
matched. 94 of 367 routes answered 404 at the unified door — all of infra,
metrics, rules/alerts, integrations, cloud_integrations, downtime_schedules,
route_policies, metric_reduction_rules and gateway ingestion_keys — while the
standalone served the identical paths 200. The commit that believed it had
closed this had closed a fifth of it.
Collapse the four into relay at all 108 call sites, which now pass the whole
public path exactly as the other faces already did.
The reason a copy could survive the repair is that nothing measured the surface:
routes_test.go counts what Mount REGISTERS, and every one of those 94 was
registered. reach_test.go adds the census that was missing — every registered
route driven through the production embed shape (adaptor.FiberApp over a runtime
that routes) against a runtime serving all of them, where a 404 means the request
never arrived. It fails at 94 before this change and at 293 with the surviving
request-target removed. Its structural half fails on the existence of a second
body, so a sixth relay cannot be added quietly.
Also fix what the census made reachable. PlannedMaintenance.Schedule is a
pointer that IsActive, IsUpcoming and IsRecurring all dereference unguarded, and
MarshalJSON has a value receiver and calls them — so an answer carrying no
"data" decodes to the zero value and panics inside encoding/json. Those routes
were 404 until now, which is why it had never fired; in the embed a handler
panic takes iam, commerce and ai down with o11y. A window with no schedule is
not active, not upcoming and not recurring.
Co-authored-by: Hanzo Dev <[email protected]>
schema/event-fact carried the write side of the event plane: the DDL for
event_fact and event_sample, the rollup plane that folds occurrences into
measurements, and the tenant column that makes both readable by exactly one
org. It was the only branch in this tree holding hanzo-authored work that
origin/main did not already contain in substance.
Merged whole — no hunk dropped. Build green, pkg/modules/sentry/implsentry,
pkg/o11y and the root slice suite all pass on the merged tree.
Co-authored-by: Hanzo Dev <[email protected]>
`event.error.org` held ONE tenant under TWO spellings: 86 rows as
cd35a51b-f7e7-5412-b67f-58b8703e5219 and 2 as `hanzo`, measured on the live
warehouse 2026-08-02, newest uuid row 02:08 — so the writer was doing it then.
They are the same tenant. That uuid is uuid5(NameSpaceURL,
"hanzo:o11y:org:hanzo"), which iamidentn mints from the slug. A reader binding
either spelling saw a fraction of its own errors, and `product` was split the
same way: a Sentry project uuid where every other writer says `docs`.
The slug wins, because the uuid is DERIVED FROM it. A projection can always be
recomputed from the original; the original cannot be recovered from the
projection. `org` is also LowCardinality and leads every sort key in the
namespace, every other writer and reader on the plane already says slug, and the
plane's own DDL says so in as many words.
So the two vocabularies now meet in ONE place. `Scope` (plane.go) resolves the
ids this module works in to the names the plane stores, reading each from the
record that owns it — the org row and the project row — and memoizing, because
neither name has an update path. It is a constructor argument rather than an
option: a store that could be built without one could write a row under a
spelling no reader binds. It fails CLOSED — an id with no name refuses the
operation rather than writing under a placeholder, since a bucket nobody owns is
a bucket several tenants share.
`row` is extracted as a pure function so the written row can be asserted
directly: the regression gate reads `org` and `product` off the result, rather
than trusting the call that produced it.
And the tenant contract stops being a comment. 0002 asserted it with two bare
SELECTs at the end of the file — a check that printed a table and was believed by
whoever read it, and which would have FAILED on 0002's own backfill. It is now a
CONSTRAINT on event.fact, evaluated on every insert forever, plus a precondition
over the four source tables that refuses to start the backfill rather than
aborting it halfway. Verified against the live engine (26.2.3): a uuid org and a
uuid product are each rejected with VIOLATED_CONSTRAINT and the table is
unchanged; the 36-character slug `maxpower-team-alpha-beta-gamma-delta` is
admitted — which is why the predicate is the uuid grammar and not
`toUUIDOrNull(x) IS NULL`, a spelling that reads better and would refuse a real
tenant's telemetry. The precondition fires against the live sources today.
Repairing the 88 already-written rows is the operator's, because the uuid -> slug
map is this deployment's; the derivation that proves each pair is in the note.
Co-authored-by: Hanzo Dev <[email protected]>
The selection comment said 'iam' is what standalone means and 'local' what the
embed means. That stopped being true the moment a standalone host was put behind
the same gateway hop: o11y.hanzo.ai is fronted by admin-guard verify-authn, so
its boundary IS the embed's boundary.
Also states what the default cannot: 'iam' requires IAM to serve add-policy /
remove-policy / get-policies / batch-enforce, and IAM v1.34 (clean-room zip v2)
does not. Selecting it today fails the founding grant with authz_unavailable on
every sign-in — measured, not inferred.
Co-authored-by: Hanzo Dev <[email protected]>
zeekayandHanzo Dev