openVerified copied every plugin binary into the instance's private
directory before exec, digest or not. The copy is what makes "what was
checked" and "what runs" one object, and it stays for a plugin with a
Sum. Without one nothing is checked, so the copy kept nothing equal:
the bytes at the path when it was taken are the bytes that ran. It cost
a full write of the binary on every lazy start (Hanzo's per-app
plugins are 24-145 MB), on whatever disk Dir sits on, and gave every
instance its own page cache.
A plugin with no Sum is now opened read-only and executed through that
descriptor. An embedded Bin is written once, under ForkLock (place),
and executed from there; before, it was written and then copied.
Benchmark_OpenBinary (the file handling alone, Apple M4 Max, APFS,
go1.27.1, -count=6, median):
size before after
16 MB 3.94 ms 0.026 ms
48 MB 12.11 ms 0.026 ms
144 MB 36.66 ms 0.026 ms
Benchmark_ColdStart (Load lazy, first request through the host, test
plugin padded to size, same host). On macOS the delta is far larger
than the copy itself: exec of a just-written file costs it more than
the bytes do. Expect the Linux gain to be the copy above, not this:
binary before after
15.8 MB 156.8 ms 7.8 ms
47.8 MB 294.8 ms 7.1 ms
143.8 MB 767.2 ms 7.3 ms
Co-authored-by: Hanzo Dev <[email protected]>
Every child writes to the same host stream, and the tag and the line went out
as two writes under only that child's lock, so another child's line could land
between them: it carried the wrong plugin's tag and the next line carried none.
The line is assembled and written once, under a lock all children share; a last
line without a newline is flushed when the child exits.
Co-authored-by: Hanzo Dev <[email protected]>
A plugin host is where a service becomes an arbitrary-code-execution vector:
whatever it starts runs with the host's authority — verifying blocks, signing,
serving. Sum existed but only gated URL, so the case that looked dangerous was
checked and the case that is was not. A binary read off disk is not more
trustworthy than a downloaded one; it is the same bytes with the download
already done by somebody else.
Now every source takes the same path, and the check is worth something because
what runs is what was checked:
- The binary is COPIED into a directory only the host can write, and the copy
is hashed and executed. Verifying in place and then running leaves whoever
can write that file free to change it afterwards, and starting a plugin is
an observable event, so they get to choose when.
- Holding a descriptor to the original would not have closed that, which the
test proves rather than assumes: a rename swaps the inode, but a WRITE keeps
it, and a write is visible through an already-open descriptor. Only the copy
puts the bytes out of reach.
- The copy is executed through its own descriptor (/proc/self/fd on Linux), so
no path is resolved a second time. It is reopened read-only first — Linux
refuses to execute a file anyone holds open for writing.
- A mismatch refuses. A host that logs and carries on has performed the check
and kept the risk.
What this cannot decide is where a trustworthy Sum comes from. A digest in the
same config an attacker just edited proves nothing, so the anchor has to be
something they do not control. For a network that already agrees on things by
construction, that is consensus.
Running=false means "deployed but down" and Disabled separates a maintenance stop
from a crash. Neither can say WHY, and for a LAZY plugin that was the only trace
there was.
An eager plugin that will not boot fails at Load, in front of the caller. A lazy
one cannot: its mount succeeds with no process behind it — that is what Lazy means
— so the start happens on the first request, inside the child, long after anyone
was watching. The host logged one line and kept nothing, and the route answered
503 "no instance running" for that request and every later one.
So Running=false could not be read: a lazy plugin nobody has asked for yet reports
exactly what one whose every start dies reports. A host could answer
{"status":"ok"} with nine of its surfaces permanently 503, which is what
api.hanzo.ai did.
Status.Error is that missing fact. It is DERIVED against cur rather than cleared
on success — an instance becomes current in three places, and a clear that must be
remembered at all three is a clear that will be forgotten at one. A plugin that has
never been asked still reports nothing, because cold is not an error and must not
read as one.
Co-authored-by: Hanzo Dev <[email protected]>
sun_path is a fixed field holding a terminated string, so a unix address has
103 usable bytes on Darwin and 107 on Linux. Over that the kernel refuses the
bind with EINVAL, which surfaces as "invalid argument" and names neither the
length nor the path — the plugin child exits 1 and the host reports "exited
before listening", which reads like a timing bug and is not.
The limit now lives in internal/sock beside the path shape, since both are
properties of the same address: minting one is not enough, it also has to be
bindable. start() checks the longest address in play — the child's .sock.http
upgrade listener — and refuses to launch rather than watching it die.
103 is measured, not read off a header: on Darwin a 103-byte path binds and a
104-byte one does not. The first cut of this used 104 and let through a path
that was exactly one byte too long.
The private instance directory also stops repeating the plugin's name. The
binary and the socket inside it are already named for the plugin, so the copy
in the directory spent len(name)+1 of a 103-byte budget saying it twice, and
that was what pushed a long name over.
Four idle tests and the status test built their Plugin.Dir from t.TempDir,
which spends 89 bytes on the test's own name; sockDir has existed for exactly
this reason. shortDir was a third copy of sockDir in the same test package,
written from a comment that blamed sockDir for t.TempDir's length.
Co-authored-by: Hanzo Dev <[email protected]>
startOnDemand held p.mu, checked that no instance was current, started a child
and published it with a plain Store. p.mu does not exclude the supervisor: a
restart swaps its own instance in without taking that lock, holding it only long
enough to read closed and spec. So between the check and the Store the current
instance can become non-nil, and the Store overwrites a LIVE child.
The overwritten one is then referenced by nothing — no cur, no supervise loop
that can reach it, nothing to stop it on Unload or Shutdown. It keeps running,
keeps its socket, and keeps writing whatever per-process state it owns.
Measured in production before this change: 15 live `ai` children in one pod,
accumulated over two hours, every one appending to the same audit chain and
racing for its sequence. The audit gate fails closed on that, correctly, so
audited POSTs were refused fleet-wide — chat completions, alert delivery and
inbound webhooks among them.
The restart path already published by compare-and-swap and stopped the loser.
This does the same, so the two agree about who is current. Losing is not an
error: the winner is a started, supervised instance of this same plugin, so the
loser stops the child it opened and serves from the winner.
Co-authored-by: Hanzo Dev <[email protected]>
Warm was an argument to Evict, so the count was TRIMMED once a minute rather than
held. That is a bound only while starts arrive slower than the sweep, and the
thing that starts fastest is exactly the thing that needs bounding: a host whose
fan-out door asks every subsystem at once started ~100 children inside 90
seconds, held all of them until the next sweep, stopped answering its own
liveness probe, and was killed. The ceiling appeared only in the logs of a
container that was already gone.
So Warm is Config now and makeRoom applies it in startOnDemand, before another
process exists: if the host is at its ceiling, the least recently used evictable
plugin is stopped to make room. Age keeps its own sweep — steady state and burst
are different bounds and both belong to the same policy.
Adoption is what makes it possible. A plugin's `app` is the definition that
Load'ed it, which for a composed service is a sub-app holding one plugin and no
ceiling, so asking it "how many are running" answers 1 and "what is the budget"
answers 0 — measured, and the reason a first attempt at this was inert. The root
of a composition is known in exactly one place, where the generation computes its
hosts, so that is where every plugin learns its owner.
Both halves are proven necessary by mutation: dropping the start-path call, or
dropping the adoption, each turns the burst test from peak 4 into peak 12 against
a ceiling of 4. Race-clean over two runs; full suite green.
Co-authored-by: Hanzo Dev <[email protected]>
IdleAfter bounds the steady state and cannot bound the burst: it only reclaims a
plugin that has already been quiet for its whole window, so in the minutes after
a cold start — when every prefix that gets a request starts a child and none is
old enough to be idle — it reclaims nothing. Measured on a host of this shape,
37 children at ~152MiB each in steady state and an OOM kill six minutes after
boot, well before anything was 15 minutes idle.
Evict now takes that bound: age first, then, while more than warm plugins are
still running, the least recently used. The plugin that has gone longest without
a request is the one whose restart is least likely to be paid for by a caller
waiting. Zero is unbounded, which is what every existing host passes.
warm is an argument rather than a field because the number of processes a host
may hold is a fact about its memory budget, which the host knows and a library
cannot. Reap takes it for the same reason it takes its cadence.
A plugin with IdleAfter unset is exempt from both bounds, and still COUNTS
toward this one — it holds the same memory, and counting only the evictable ones
would authorise the budget twice. When the exempt alone exceed the bound this
reclaims what it can and stays over it, because a host is better over budget
than deprived of the identity service every other call goes through.
EvictIdle and ReapIdle are gone rather than kept beside these: two names for one
sweep is how the two policies come to disagree about how a child is taken down.
Co-authored-by: Hanzo Dev <[email protected]>
Lazy defers the first start and nothing has ever stopped a plugin, so a host
pays for every service it has EVER served, forever. Resident memory tracks the
size of the catalog rather than the traffic. Measured on a host running two
dozen subsystems: 24 children, ~150MiB of live heap each, ~4.4GiB resident,
while three of them sat at 12-20MiB. The catalog was the bill, and it is what
stops that host adding the other seventy services it wants.
IdleAfter stops a lazy plugin that has not served for that long; the next
request starts it again through startOnDemand, which already exists. Zero means
never, which is both the historical behaviour and the right setting for an
identity or config plugin every other call goes through — nobody should pay a
cold start for auth.
Safe against the supervisor by construction: it restarts a child whose process
exits and detects a deliberate stop by CAS-ing cur from the instance it watches
to nil. Eviction clears cur BEFORE the child dies, so that CAS always fails and
no restart storm follows — the same contract Reload and Unload use, minus the
disabled flag, because coming back is the point.
Two defects the checks caught before this shipped, both worth naming:
- stop() from ReapIdle signalled and returned, so a sweep could still be
running when the host began shutting down. That is a real data race
(evictIfIdle reads App state shutdown writes) and the detector found it.
stop() now waits for an in-flight sweep.
- TestIdle_EagerIsNeverEvicted passed for the wrong reason: with no request
served, lastUse stays zero and evictIfIdle returns on that check before it
ever consults Lazy. Deleting the eager guard did not fail the suite. The
test now serves a request first, and that mutation fails it.
Verified: 0 races in 10 -race stress runs; the eager, zero-means-never and
comes-back-with-a-new-pid guards each fail when their rule is removed.
(TestServePathAllocsAreChainDepthInvariant fails identically on this commit's
parent — pre-existing, unrelated, 34 allocs on the serve path.)
Co-authored-by: Hanzo Dev <[email protected]>
walk(app, visit func(n node, ctx scope, site callsite) error). The callback's
parameters WERE the payload, so the struct wrapping them was a noun invented to
carry what three arguments already carry — and retaining the slice made every
backend a consumer of a materialised intermediate rather than a reducer over a
traversal. The English word survives in prose because the definition/occurrence
DISTINCTION is load-bearing; the type does not.
Validation stays its own first pass (verify), so cycles and conflicts complete
with trails before any backend sees a visit. A generation retains what was
REDUCED — router, ops, host set — never the traversal.
Also: #4 verified, not changed. Own-op tool names already ride the operation-id
rule because they read Registry(); a plugin catalogue tool is invoked by NAME at
the plugin's own door and never consults a prefix, so ONE definition is ONE tool
and namespacing it would invent a callable that does not exist. Tests pin both.
#2: Drop and Include share one transact path, so the guarantee is identical, and
the lock discipline that makes it deadlock-free is now written down with the two
invariants it rests on. Documented honestly: commitment across servers, not
simultaneity.
Godoc: every bracket reference to a deleted symbol rewritten. LLM.md carries the
canonical three-document spec.
Co-authored-by: Hanzo Dev <[email protected]>
Decisions, all one principle — the framework may refuse, never be silently wrong:
- A projection of a program that does not compose PANICS, carrying the joined
error. Registry and Declaration return values, not (value, error), so there is
no honest answer; a sentinel loses because a sentinel can be ignored, and a
caller ignoring it was the failure. Build validates BEFORE prepare, so the one
function whose job is to return the error still returns it. project() writes
nothing and fails, so a pipeline can no longer ship an empty API document.
- Include on a non-root WORKS. It rebuilds every live server the edit reaches,
builds them all before swapping any, and swaps only if all succeed. The freeze
panic recommends Include; a recommendation that no-ops is a bug in one of the
two, and this keeps the recommendation true.
- A definition with middleware and no routes ANYWHERE BENEATH IT is refused at
seal. Inert is correct under lexical anchoring, so the scoping rule is not
bent; what was unacceptable is that it was silent and reads exactly like
working code. A group whose routes come from its children is fine.
The MCP catalogue joins Plugins/Reload/Unload/Start on the walk: a plugin is a
definition now, so its tools live with it and the host aggregates. Duplicate
tool names and two open plugins are refused at build, where two definitions can
finally see each other.
And appendEntry takes buildMu. Making the flag atomic was not enough — a
transaction rewrites the entry slice while holding the lock.
Co-authored-by: Hanzo Dev <[email protected]>
The verb deletion is done and the library builds: Graft, Add, Service, the
Mount VERB and the claims ledger are gone; Load and Mount return definitions;
Plugins/Reload/Unload now find their plugin by walking.
NOT DONE — the same entanglement, one layer over: a plugin's MCP catalogue
(pluginTools/toolOwners/open) is still read off the host that loaded it, but a
plugin is now a definition, so the host reads an empty set and serves no door.
Four mcp_compose/mcp_open tests fail on exactly that. The fix is the same one
that worked for Plugins(): aggregate across App.hosts() instead of reading a
private map, and move the duplicate-tool-name refusal to the walk so two
plugins can actually see each other.
Co-authored-by: Hanzo Dev <[email protected]>
Everything composes through Use. A unit of functionality is an *App — whether
it is linked in, loaded from a binary, or another process — so Load and Mount
become definition CONSTRUCTORS returning (*App, error) instead of verbs.
App.claim/release and the claims ledger go too: the walk's conflict accumulator
already answers 'who else wants this address', and answers it better — every
collision at once, with both claimants and their call sites named. Two
mechanisms for one question are two answers waiting to disagree.
Co-authored-by: Hanzo Dev <[email protected]>
A typed op is a value known at build time, so zip projects it once and serves the
array as bytes; that memcpy is what makes tools/list affordable for a host
composing 112 lazy children. A tenant's own capabilities are ROWS — an org's
connected connectors, its authored skills, the tools on the external MCP servers
it registered — and no build-time projection can hold them. They were therefore
reachable only THROUGH a tool, never as tools, which is a second registry wearing
a different hat.
Two additions, one idea: some tools depend on WHO is asking.
MCPConfig.Source the leaf: an app contributes the caller's own tools to its
own door. Called with the context the request is served on —
the same value a typed op gets — so an implementation reads
its caller from there and never from the arguments, which the
caller wrote.
Plugin.Open the host: this plugin's catalogue is INCOMPLETE by
construction, so ask it per caller and hand it a tools/call
no catalogue claimed. At most one may be open, for the same
reason two may not own one tool name.
The invariant that made the door affordable is kept, not traded: the build-time
array is still bytes and still byte-identical to the artifact, the per-caller half
is APPENDED rather than merged, and it runs only on a request that NAMES a caller.
A per-caller answer needs a caller, so an anonymous tools/list still costs a
memcpy and starts nothing. A projected name can never be shadowed: composeTools
now returns its name set with its bytes, so the dedup is a map lookup rather than
a re-parse of hundreds of schemas per request.
mcpAnswer is lifted out of mcpCall so a typed op and a Source render one shape —
a handler error is isError content either way, which is what lets a model read a
refusal instead of a transport failure.
Co-authored-by: Hanzo Dev <[email protected]>
The fourth projection was already free per app — every typed op is a tool, at
/mcp, over every transport the app listens on — and completely unreachable in a
fleet, because a HOST has no ops of its own and MCPTools() is in-process: it
cannot ask a plugin that is not running, and asking would defeat Lazy. tools/list
is the method an MCP client calls constantly, so a host that woke 112 children to
answer it would trade the one invariant that makes 112 services affordable for a
question whose answer is fixed at build time.
So Plugin.Tools carries that answer: the JSON array the plugin's own MCPTools()
projected, captured when it was built. load indexes it name→plugin (refusing a
duplicate name at compose time, the analogue of a duplicate prefix claim — a name
is dispatch, so two owners make it unroutable), installMCP renders own ops ++
every catalogue ONCE as bytes, and tools/list is a memcpy of a constant. Zero
processes.
tools/call is the only trigger, and it starts exactly one: p.target(), the same
single-flighted lazy path a prefix request takes, then the SAME message forwarded
to the owner's own /mcp. The child's registry answers, so the host can only NAME
a tool, never invoke one the child did not declare — a stale catalogue yields the
child's -32602, not a wrong call. mountVia's proxy body is extracted rather than
copied: one forwarder, differing only by a rewritten path.
mcpTools() also sorts now, because this projection is SERIALIZED — an artifact
ordered by registration churns on an edit that changed nothing a client can see.
Co-authored-by: Hanzo Dev <[email protected]>
HIP-0106's plugin contract, in the framework rather than in 116 repositories.
Declaration projects the LIVE ROUTER — every method+pattern, complete by
construction, so a route a plugin serves and does not publish is still
declared. That is the analytics-ingestion outage class: the published paths
were listed, the four ingestion doors were not, and every product beacon in
the fleet answered 405.
Declaration/Route/(*App).Declaration what a host discovers
/.well-known/zip/plugin.json the same value off a running child
(*App).Described `<bin> openapi <f>` / `<bin> declare <f>`
Tenant the tenancy read, with the real rule
Delegate a continuation for another org
Ask dial a peer by name, invoke one op
(*App).Peer peer ops on their own app, own socket
(*App).Ops /healthz + /readyz + /metrics
Config.Eager the one fact a router cannot show
App.claim a second claim of a prefix is an ERROR
zip/runtime -> zip/js a language runtime is an adaptor
Four things the reference plugin falsified while being built:
- Describe was already taken. zip.Describe(methodPath, Doc) is the doc
registry generated code calls. The projection writer is (*App).Described,
one call, because argv is read in one place and a main cannot forget to
forward it.
- zip's control plane is NOT all under /.well-known/. /docs and the MCP path
are not, so a prefix filter let a child declare them — paths outside /v1/
that every plugin serves and whose first claimant takes the HOST's document
and agent door. App.control now registers and records in one call, so
leaking one is impossible rather than unlikely.
- OPS_PORT appeared in no Go file in the fleet. HIP-0119 named the paths and
nobody implemented them, so conformance meant 116 repos hand-rolling three
routes and a listener. It is one sibling app now, brought up by Listen.
- There is deliberately NO default ops port. A default made every second app
in a process die on "address already in use" — a plugin has two apps in it
and a test process has many. The deployment states OPS_PORT; a host states
none on a child, so a child correctly owns none.
Load's duplicate claim used to return nil while the first registration kept
answering: the losing plugin's whole surface served another plugin's 404 and
nothing said so. It is now an error naming both claimants, all-or-nothing
across a plugin's prefixes, taken before anything is spawned.
a.ops was the op registry and is now a.registry — the ONE registry the
OpenAPI document, the MCP tool, the CLI command and the plane op are all
projected from. a.ops is the ops sibling.
Co-authored-by: Hanzo Dev <[email protected]>
Lazy is what makes many plugins affordable, and a lazy plugin starts on
exactly one trigger: a request reaching one of its prefixes. A host that
reaches its plugins another way has no way in at all — Hanzo's fleet calls
app-to-app over each app's own unix socket, which never touches the router,
so nothing ever started the callee and the socket it would have bound was
never there. Correct by design, inert in practice.
Start is that second door. It goes through target() — the SAME
single-flighted path a prefix request takes — rather than a parallel one, so
a burst of first callers arriving by either door still produces exactly one
child. It is what Reload is not: idempotent. Reload on a running plugin
starts a second child and retires the first, which is right for a version
swap and wrong for 'make sure this exists'. An Unload'ed plugin stays down;
a deliberate stop the next caller silently undoes is not a stop.
shortDir in the test: a plugin's socket lives inside Plugin.Dir and the
kernel caps a unix path at 108 bytes, so t.TempDir() plus a descriptive test
name overflows it and the child reports 'exited before listening'.
Co-authored-by: Hanzo Dev <[email protected]>
The typed-op registry gets its fourth projection. An aggregator that reaches
another app today either imports its package — dragging in that app's whole
dependency graph, which is what turns a fleet of small services into one
mega-binary — or hand-writes a client against its URLs, which is the schema
copied into a second place to drift. Neither is necessary: the op already has
a stable name, and now that name is callable.
c, err := zip.DialApp("flags") // $ZIP_RUNTIME_DIR/flags.sock
out, err := zip.Call[BoolIn, BoolOut](ctx, c, "flags_bool", &in)
The plane is an ordinary route (/.well-known/zip/op/:op), so it rides every
transport Listen was given and inherits the app's middleware, error handler,
validator and Authorizer — exposed exactly as much as the REST routes and no
more. It resolves through opByName + op.invoke, the same pair tools/call runs,
without MCP's agent-facing envelope. Errors cross whole: the wire form is what
errorHandler already writes, so errors.As sees the *HTTPError the callee
returned, status intact. A void op yields (nil, nil).
Generic, not generated, deliberately: a generated per-op Go client would put a
second copy of the schema in the caller's repo on its own regeneration
schedule — the drift this plane removes, one directory over — and would be a
fourth client generator racing openapi-generator, which already builds the
published SDKs from the OpenAPI projection.
ONE socket-path scheme, used by both halves so they cannot drift:
SocketPath(name) = RuntimeDir() + "/" + name + ".sock"
RuntimeDir() = $ZIP_RUNTIME_DIR | $XDG_RUNTIME_DIR/zip | /run/zip
A service serves at zip.Addr(zip.SocketPath("flags")), a caller reaches it with
zip.DialApp("flags"). socketIn states the shape once; the plugin loader names a
child's private socket with the same function. Listen creates the directory
0700 when missing, so the canonical path works on a fresh host.
Identity is never forged, because the caller never states it. WHO the request
is for is the gateway's assertion, forwarded by Ctx.Forward and read by
CallerOf; WHAT is calling is SO_PEERCRED, read by PeerOf — the peer does not
send it, the kernel answers on its behalf. nil means "cannot attest", so a
check fails closed rather than reading a zero Peer as root.
MCPTools() exposes the tool surface the same way OpenAPISpec() exposes the
document, so a host can compose plugins' tools in-process without a round trip.
openapi.go now reads operationId from opName, the one place that rule lives,
instead of re-deriving it inline — all four projections address an op by the
same token.
Cost: Benchmark_TypedRoute/typed 28 -> 29 allocs/op (the one context value that
makes CallerOf and PeerOf possible). The untyped path is unchanged at 27 and
still pinned at one heap value per request.
Proven end to end: a real plugin on a unix socket, a client dialing it, a typed
round trip with no HTTP in the path — the reply frame is decoded with zaphttp's
own codec, HTTP/1.1 text gets no HTTP answer, and the peer credential the
handler observes is proof the transport was unix. Full suite green with
CGO_ENABLED=0 and under -race.
Co-authored-by: Hanzo Dev <[email protected]>
The cache path was Name+"-"+Sum while the comment said the name IS the digest.
It was not, and the gap is expensive: one multi-call binary mounted under 108
names downloaded 195MB 108 times and kept 108 identical copies — ~21GB to
negate the whole point of shipping one artifact.
Keyed on Sum, the first plugin to start warms every other.
Co-authored-by: Hanzo Dev <[email protected]>
ReloadTo was Reload wearing a second name. A restart is a reload to what is
already there, and every source a plugin can have is a field of Plugin, so
widening the parameter subsumes the byte form: Reload(name, Plugin{Bin: b}).
One operation, one name, one path through the three invariants.
PluginStatus -> Status: zip.PluginStatus says plugin twice.
Co-authored-by: Hanzo Dev <[email protected]>
Three things a control plane needs that Reload alone could not express.
ReloadTo(name, Plugin) swaps a plugin to a DIFFERENT artifact — another
URL+Sum — where Reload only restarts the one it was loaded with. Rolling a
version back is that call naming an earlier digest, and since the digest is
the cache key the bits are already on disk: the rollback touches no network.
Reload is now that call with a Bin set, so there is still ONE reload path and
the three invariants hold for both by construction.
A new artifact replaces the old one whole. Leaving the previous Bin in place
would shadow the URL just asked for, since start() prefers Bin — the swap
would report success having restarted the version it was already running, and
PluginStatus.Version would name bits that are not the ones serving.
Unload now pins a LAZY plugin down. target() starts a lazy plugin on the
request that reaches it, so the next request undid the Unload: disable did not
stick for exactly the plugins a many-service host runs, which is nearly all of
them. PluginStatus.Disabled tells that deliberate stop from a crash — both
answer 503, so without it an operator pages for a maintenance window.
TestUnload_LazyStaysDown fails against the previous behaviour with status 200.
Co-authored-by: Hanzo Dev <[email protected]>
Routes register at Load either way, so the served surface is identical; only
the process is deferred. That is what makes many plugins affordable. A host
composing 69 services eagerly pays 69 processes, 69 resident sets and 69
startup times at boot for a set that is mostly idle. Lazily it pays for the
ones traffic actually reaches, and a CLI that touches one service starts one.
zip.Load(zip.Plugin{Name: "reports", Bin: bin, Lazy: true}, "/v1/reports")
The first caller through the mount starts it, single-flighted, so a burst of
concurrent first requests produces ONE child rather than one per request. A
failed start is not cached — the next request tries again, because the usual
cause is a dependency that has not come up yet. Steady state is one atomic load,
the same as an eager plugin.
Opt-in rather than default: the cost moves to the first request, so a
latency-critical prefix should stay eager. Status reports it honestly —
registered but Running=false with no PID until something asks for it.
Co-authored-by: Hanzo Dev <[email protected]>
A plugin that panicked was invisible and permanent. Nothing watched the child,
so when it died cur still pointed at the corpse: every request became a
connection error dressed as a 502, Plugins() kept reporting Running, and only a
manual Reload recovered. A plugin is a separate process precisely so a panic
cannot take the host down — which is only true if the host notices.
It notices now. The supervisor sees the exit, stops serving from the dead
instance (503 "deployed but down", the truth, rather than 502 blaming an
upstream that no longer exists), and restarts with capped backoff. An instance
retired by Reload or Unload also exits; a compare-and-swap tells the two apart,
so only a death that was still current counts as a crash. Restarts is reported
separately from Reloads — deliberate swaps and crashes should never be one
number, and a climbing Restarts is a crash loop.
Proven, not argued: TestPlugins_SurvivesPanic panics the plugin on a goroutine,
where no recover() can catch it, and asserts the host survives, the pid changes,
Restarts increments, and the same routes serve again with no re-registration.
Child exit is now a closed channel rather than a single-value one, because the
supervisor, a drain and Shutdown all need to observe the same exit and the first
reader would otherwise starve the rest.
Also adds per-plugin resource accounting, read from /proc:
Usage{CPU, RSS, Threads, FDs} // measured, not self-reported
This is the observability a monolith cannot offer at any price. One process has
one RSS and one CPU clock, and nothing inside it can be attributed to a
subsystem — "which app is eating the memory" is an argument. Running a service
as its own process makes it a question with an exact answer. Measured on a live
plugin: rss=12.7MB threads=5 fds=7.
And plugin output is tagged with the plugin name. An untagged line in a merged
stream is worse than useless: you can see something is wrong and not what.
Co-authored-by: Hanzo Dev <[email protected]>
Load took a single prefix, so a service owning several route subtrees silently
404'd every one it could not name. Real services own several: o11y answers both
/v1/o11y and /v1/sentry, and extracting it as a plugin dropped /v1/sentry/*
entirely — the host started, reported healthy, and served nothing there. That is
the worst failure mode this API can have.
zip.Load(zip.Plugin{Name: "o11y", Bin: bin}, "/v1/o11y", "/v1/sentry")
Prefixes are variadic and last, so one call declares everything the plugin owns
rather than a prefix field competing with an argument. Zero prefixes is refused
rather than mounting a plugin nowhere.
The test plugin now echoes the path it received, because a host that never
registered a prefix and a plugin that has no such route are indistinguishable
from outside otherwise — both are a 404. The echo is what tells them apart, and
the multi-prefix test asserts on it.
Found by the o11y extraction, which hit this immediately.
Co-authored-by: Hanzo Dev <[email protected]>
Graceful Shutdown already stops every child. A host that is SIGKILLed,
OOM-killed, or crashes never runs its hooks — and a plugin is a separate
process, so it kept running, kept holding its unix socket, and kept serving
requests for a host that no longer existed. Orphaned plugins on stale sockets
are the one way a subprocess plugin is genuinely more dangerous than a
linked-in one, so it is worth the platform-specific code.
Children are now started with Pdeathsig on Linux: the kernel signals them when
their parent dies, whether or not the parent got to say so. Non-Linux builds
get a documented no-op — Shutdown still covers every ordinary exit path there.
Found by noticing empty /tmp/zip-* directories left behind by hosts that had
been killed rather than shut down.
Co-authored-by: Hanzo Dev <[email protected]>
A fleet view needs one thing this package did not expose: the ability to ask a
live process what it loaded, at which version, and whether it is still up.
Deployment config answers what was INTENDED; only the process knows what is
TRUE — which is exactly the difference that matters mid-rollout, when the two
disagree by design.
for _, p := range app.Plugins() {
// Name, Prefix, Source, Version, Addr, PID, Running, Since, Reloads
}
Version is the artifact's SHA-256 for a URL-installed plugin — the only version
identifier that cannot drift from the bits running, because it IS the bits.
Since resets on Reload, so it reports the age of what is running rather than of
the mount, and Reloads climbing on one host and not its peers is the signal that
a rollout is uneven. An Unloaded plugin stays listed with Running=false, because
"deployed but down" and "not deployed" are different states and its routes are
still registered answering 503.
Remote (Addr) mounts are recorded too. They took an early return and were
invisible before; a fleet view that silently omits part of the surface is worse
than none.
Two fixes found by having the tests exercise this:
Shutdown now waits for instances a Reload or Unload is still draining. Those
retire asynchronously so the caller is not blocked for the drain window, which
let a child outlive the host that spawned it — and since children inherit the
host's stdout, the process appeared to hang after main returned. The suite
showed it as "Test I/O incomplete 1m0s after exiting"; it now finishes in 22s
instead of 74s.
Tests no longer hardcode ports. A fixed 19653 meant an unrelated process on the
box made TestListen_ZAP fail with 404 rather than a connection error, because
the port answers — it is just answering for someone else. That reads as a
routing bug and is not one.
Co-authored-by: Hanzo Dev <[email protected]>
A host can now name a release artifact instead of carrying the binary:
app.Add(zip.Load("/v1/billing", zip.Plugin{
Name: "billing",
URL: "https://github.com/hanzoai/billing/releases/download/v1.2.3/billing-linux-arm64",
Sum: "9f2c…",
}))
This is what makes the two build cycles independent. The host links zip and a
transport, relinks in 0.65s, and never compiles a plugin at all; a plugin ships
from its own repository on its own cadence. Embedded (Bin), on-disk (Path),
installed (URL) and remote (Addr) are the same call and the same type, so which
one a deployment uses is configuration rather than code.
Sum is REQUIRED with URL. Fetching code over a network and executing it is the
one place a plugin host turns into an arbitrary-code-execution vector, so an
unverified download is refused rather than trusted. The digest is checked before
the file is made executable or given its final name, so a truncated or
substituted download is never runnable — the failure mode is a missing plugin,
not a wrong one.
Sum doubles as the cache key: the cached file IS named by its digest, so a
restart costs no download, a rollback to a previously run version is free and
offline, and a concurrent host either sees nothing or sees a verified binary.
Tests serve a real built binary over httptest and prove all three: it installs
and answers, a wrong Sum is refused before execution, and a second load at the
same digest does not hit the network.
Co-authored-by: Hanzo Dev <[email protected]>
Two things, both found by real consumers breaking.
Fronting a net/http handler is Group's job, not Mount's. Mount delegates a
prefix to a REMOTE address over a transport, so it takes a string; a local
handler is composed, not delegated. The taught form is now:
app.Group("/legacy").All("/*", zip.AdaptNetHTTP(h))
and for a net/http middleware, which IS middleware:
app.Group("/legacy").Use(zip.AdaptNetHTTPMiddleware(mw))
Both keep the wildcard an ordinary route, so a route registered on the group
afterwards still beats it for its exact path. Repurposing Mount's old
(prefix, http.Handler) signature broke hanzoai/kms and lux-private/sovereign;
this is the form both were fixed to.
Plugin.Dir: Load extracted an embedded binary via os.MkdirTemp("") — the system
temp dir, which on many hosts is a tmpfs, i.e. RAM. A plugin binary is tens to
hundreds of megabytes, so that spends real memory and fails outright when the
tmpfs fills. Found the hard way: the reload test died with "no space left on
device" while other work shared /tmp. Callers can now put it on disk.
Co-authored-by: Hanzo Dev <[email protected]>
A plugin is an ordinary zip app. It needs no SDK and no schema: it serves
routes, and a host mounts them over ZAP on a unix socket.
//go:embed bin/billing
var billingBin []byte
app.Add(
billing.New(deps), // linked in
zip.Load("/v1/billing", zip.Plugin{Name: "billing", // its own binary
Bin: billingBin}),
zip.Load("/v1/ml", zip.Plugin{Name: "ml", // already running
Addr: os.Getenv("ML_ADDR")}),
)
One type covers all three, so where a service runs is a deployment decision
rather than a code change. Service is func(*App) error — a constructor taking
dependencies and returning one is the same thing curried, which is what lets a
composition root see only Service.
The host links zip and a transport, never a plugin's dependency graph. Plugins
build in parallel, one changing rebuilds only itself, and go:embed keeps the
deployment a single artifact.
Reload swaps a running plugin for a new build without dropping a request. This
is why a plugin is a process: Go can map a shared object in but never unmap
one, so an in-process plugin's code, globals and goroutines are unreclaimable
for the life of the host. A child releases every byte when it exits. Three
invariants keep repeated reloads flat in memory:
- routes register ONCE at Load and resolve their target per request, so the
route table never grows
- the replacement must be listening before any request moves to it, so a bad
build cannot take the route down
- the old process is killed, reaped exactly once, its pooled connections
closed and its directory removed
Unload leaves the routes answering 503 rather than vanishing, so load/unload
cycles are flat too and a Reload restores the same paths.
Tests build two genuinely different binaries and prove which one answered
across four swaps, a failed swap, and an unload/restore. Race detector clean.
Co-authored-by: Hanzo Dev <[email protected]>
zeekay