Rust answered four queries in five against a warm root. The fifth came back
`rpc -32801: content modified`, which the daemon passed up as an error and the
door turned into a 502.
-32801 is the protocol's one retryable answer: the server computed against state
it has since replaced, so the result is stale rather than wrong, and the spec
has the client ask again. rust-analyzer says it for anything asked while a
workspace is still loading, which is exactly when a cold root is queried.
Retried in Call, so every op gets it without each caller learning the code, and
bounded twice over: a few attempts, and the caller's context underneath that. A
server that never settles still ends as an error rather than a request that
never returns.
Co-authored-by: Hanzo Dev <[email protected]>
TypeScript, JavaScript and Python answered exactly one question per root and
then went silent. PHP managed two. The daemon reported `server stopped`, and
0.1.10 finally attached the missing half of that sentence: `[exit status 1]`.
Not a signal, so nothing killed them — they LEFT.
processId is a watchdog. The spec says a server whose client process has died
should exit, and every server built on vscode-languageserver implements it by
polling kill(pid, 0) every few seconds. This daemon sent its own pid, which
means something only inside its own pid namespace — and the jail gives each
server a NEW one, where that number belongs to nothing at all. So each of them
looked for its client, correctly found nothing, and exited. They were obeying
the protocol; we were the ones lying to them.
null is the spec's way to say nobody is watching, and the jail is what makes it
free: the server is PID 1 of its namespace, so Close kills it and everything it
forked, and a daemon that dies takes the whole namespace with it. The lifetime
was never the pid's to enforce.
gopls, clangd, jdtls and ruby-lsp do not poll, which is why Go, C, C++, Java and
Ruby scored 5/5 the whole time and the fault read as TypeScript's. It was never
about package.json, or a lockfile, or address space.
Measured, not assumed: the acceptance probe now fires five sequential queries at
one WARM root per language. Asking once proves a server can resolve and proves
nothing about whether it is still there for the second question — which is the
state every caller is in after the first, and the entire reason the pool exists.
Co-authored-by: Hanzo Dev <[email protected]>
0.1.9 taught stopped() to carry the server's last words and then reported none,
because it read them too early. attach's handshake path already says why: Close
kills the process AND WAITS, which is what drains the copy of its stderr, and
reading the tail before that races the words away. The reader's own exit path
had exactly that bug — it closed done first, so every mid-session death was
published with nothing attached.
done now means the reader has stopped AND the process is reaped, so anything
read after it is final. Both endings go through one reap, guarded by its own
Once: two concurrent Waits on one process is a race with no useful outcome.
And the exit STATUS, which is the part that matters here. A process killed by
the kernel — a seccomp verdict, the OOM killer, an rlimit — says nothing on
stderr at all, so the status is the only account of it that exists, and it was
being discarded. It now lands in the same tail as the words, so "server stopped"
can never again be the whole story.
This is diagnosis, not a cure. TypeScript answers its first query and then dies,
on EVERY tree shape — pure source, package.json, lockfile, .js/.mjs/.cjs alike
— which rules out the lockfile as the trigger and rules out my own earlier
reading of it. What is left is a silent death, and this is what makes it speak.
Co-authored-by: Hanzo Dev <[email protected]>
Two defects, found from one report: a TypeScript tree with a package.json and no
lockfile answered the first query and then 502'd every one after it.
FIRST, the fetch. `npm ci` installs exactly what a lockfile pins and refuses
outright without one, so naming package.json as the marker bought a fetch that
could only ever fail — on every library that commits no lock, which is most of
them. Roots locates the FETCH and nothing else reads it, so the marker has to be
the file the fetch REQUIRES: package-lock.json or npm-shrinkwrap.json, and
Cargo.lock for `cargo fetch --locked`, which had the same defect and was only
quieter about it. A locked tree fetches exactly as before; a lockless one is
served from source, which is the trade every other fetchless language already
makes.
SECOND, and the reason the first took guesswork to investigate: a server that
died MID-SESSION took its reason with it. attach already keeps the last 4 KiB of
its stderr, but only spent it on a handshake failure — so a server that started
cleanly and died on the third query reported `write |1: broken pipe`, which
names the symptom and nothing about the cause. That is the blindness `exit
status 126` was for Probe, and it is fixed the same way: stopped() carries the
server's last words, and a failed write waits briefly for the reader to reach
the same conclusion rather than racing it and reporting the pipe.
The tree left behind by the failed fetch is CLEAN — npm refuses before it writes
anything — and package.json, a read-only tree and an unreachable registry all
fail to reproduce the death outside the jail. So the crash itself is not yet
explained, and this is what makes explaining it possible: the next occurrence
arrives with the words the server died saying.
Co-authored-by: Hanzo Dev <[email protected]>
0.1.6 served eight languages and logged `spawn EPERM` for TypeScript and
JavaScript once per root. 0.1.7 serves all nine, verified on the gVisor node.
Keeping the failure in the record rather than only the result: it is the
clearest evidence the graceful-degrade design works, and the next language to
need a jail change will look exactly like it.
Co-authored-by: Hanzo Dev <[email protected]>
TypeScript and JavaScript were the two languages the new image could not serve.
The daemon said why and kept going: `language server did not start … spawn
EPERM`, once per root, while the other eight answered.
socketpair was denied beside socket, on the reading that a served process must
be able to obtain no socket fd at all. But libuv gives a child its stdio through
a socketpair, so what that denied was not a network — it was Node starting a
process, and typescript-language-server exists to run tsserver.
A socketpair reaches nothing: both ends come back already connected to each
other, in AF_UNIX, inside this jail's namespaces. It cannot be connect()ed
elsewhere, and a peer that could pass a network fd over it would have to be
outside the jail, where nothing is. socket() is still denied, so there is still
no way to obtain a fd to a network.
The filter now reads args[0] and admits AF_UNIX alone. Allowing it outright
would have been equivalent today — the kernel implements socketpair for no other
family — but that is a fact about the kernel, not a property of this program,
and the claim in the header is worth keeping true by construction.
The arithmetic is the risk: those two jump edges are hand-computed offsets over
a list whose length changes with the phase, and an off-by-one reads as ALLOW.
TestSocketpairIsDecidedByItsFamily interprets the program for AF_UNIX and for
five families that must fail, in both phases, and pins that the carve-out did
not widen to socket().
Co-authored-by: Hanzo Dev <[email protected]>
They are how a Node package says which module system a file is in, and tsserver
reads both. Left out, a package whose entry point is .mjs looked to Speaks()
like a repository with no JavaScript in it — no server started, and the daemon
answered that it had nothing for the file. .mts and .cts are the same omission
on the TypeScript side.
ID() has to move with the list or a .mjs is opened as TypeScript and gets type
errors it was never written to satisfy.
Co-authored-by: Hanzo Dev <[email protected]>
The table always carried the other entries; the image carried one toolchain, so
Available() answered false and the daemon served Go alone. Install the rest —
node with typescript-language-server, pyright and intelephense; the Rust
component tarballs with rust-src; clangd's own release; a JDK with jdtls; ruby
with ruby-lsp; composer for PHP — every version pinned, and add the java, php
and ruby entries beside them.
The three new entries keep the scripts-off bargain rather than bending it. PHP
fetches with composer's own --no-scripts --no-plugins. Java has two dependency
resolvers and both ARE build tools, so it has no fetch and answers from source
and the JDK. Ruby's bundle install compiles native extensions, so it is marked
Executes and refused, as python's is.
Three things that would otherwise be silent:
* typescript-language-server prefers the TREE's node_modules/typescript —
dependency-authored JavaScript, loaded and run in the serve phase, undoing
what --ignore-scripts bought one phase earlier. Pin tsserver to the image's.
* ruby-lsp creates .ruby-lsp/ in the project and shells out to bundle install
unless BUNDLE_GEMFILE is set at all. On a read-only tree that is EACCES
before the server starts.
* Equinox takes its writable configuration area from the passwd entry, not
$HOME, so jdtls would write to /home/lsp, which the jail does not bind.
Name the area under the jail's own /tmp.
RLIMIT_NPROC is per UID and every jail shares it, so 256 became a way for a
JVM's forty threads to stop another language forking at all.
Co-authored-by: Hanzo Dev <[email protected]>
gopls answered 'no package metadata' / 500 'language server did not answer' for
cloud, because `go mod download all` failed: cloud's go.mod requires go>=1.26.5
and the image shipped 1.26.4 with GOTOOLCHAIN=local (never auto-downloads). The
symbols op worked (no dep resolution); locate/hover/definition did not. 1.26.5 is
the latest 1.26.x and what cloud names — bump the pin, per the daemon's own design.
Co-authored-by: Hanzo Dev <[email protected]>
Run 36829 for 9b338d0 died with a 34-byte log — no output at all. The fleet was
failing about half of every job at the time (107 failed to 81 succeeded in 90
minutes, all with 42-235 byte logs, across universe, mirrors, id and auto),
which is the runner-death signature, not a gate. The same gate — go build, go
vet, gopls install, go test ./... — was run by hand on the runner pool at that
commit and is green.
Co-authored-by: Hanzo Dev <[email protected]>
gopls started and died at initialize. /root returned langs:[] and the daemon
said only:
language server did not start lang=go err="lsp: initialize: lsp: server stopped"
That is the whole diagnosis, and it is the same blindness Probe had with `exit
status 126`: conn.go set cmd.Stderr = io.Discard, so the one place the server
explains itself was thrown away. With stderr visible, on the node, it said:
failed to start telemetry sidecar: os.Executable: readlink /proc/self/exe:
no such file or directory
gopls cannot access its persistent index: can't hash gopls executable:
readlink /proc/self/exe: no such file or directory
THE JAIL HAD NO /proc. os.Executable reads /proc/self/exe and x/telemetry calls
it on init, so no Go program can start. GOTELEMETRY=off does not help —
x/telemetry reads its mode file, not the environment.
The jail now brings a PRIVATE procfs, mounted inside its own PID namespace, so
it shows the jailed process and its children and nothing of the host: measured
on the node, 4 pids, no kcore, read-only, nosuid+nodev+noexec. This is a
tightening rather than a loosening — it also removes the reason GOROOT had to be
pinned by hand, since `go` can find itself again.
Fixing that exposed the next one: no /dev either, so gopls could not open
/dev/null. A user namespace cannot mknod a device, so null, zero, full, random
and urandom are BOUND from the host, writable, in both phases — none of them
holds state, and read-only would make them useless (Go's os/exec opens
/dev/null for every subprocess with no stdin, so `go list` could not run).
They are `must`: a jail missing /dev/null fails obscurely later, which is the
same shape as a mount that hides another.
stderr is now kept as the LAST 4 KiB and surfaced on ONE path — the error from
attach, built after Close, which kills and waits so the tail is complete.
Nothing is streamed while the server is healthy: it is chatty and can echo
tenant source.
And the acceptance test could never have caught any of this, because it has
never run jailed. A file:// GOPROXY is not on the jail's bind allow-list, so
`go mod download` reported a module that does not exist; on Linux the test
skipped via needJail and on a Mac it passed unjailed. proxyDirs binds it. That
is a production shape too — an air-gapped mirror on a PVC was a configuration
this daemon accepted and could never read.
Verified on the gVisor node (code-exec-pool, runsc), everything jailed:
go test ./... → internal/jail ok, internal/lsp ok
TestDefinitionCrossesIntoTheDependency PASS, including locate definition,
hover, second root is warm, warm is accepted
serve phase: tree still read-only, /dev/null writable, 4 pids visible
Before this commit the same test on the same node failed at
"/root served no Go server: map[cold:true langs:[] ready:true]" — the
production symptom.
Co-authored-by: Hanzo Dev <[email protected]>
hanzoai is the public OSS org; a repo without a licence cannot go public,
because "no licence" means all rights reserved and nobody may use it.
Apache-2.0 matches the Go infrastructure already published under this org
(hanzoai/cloud, hanzoai/go-sdk) rather than introducing a third convention.
Co-authored-by: Hanzo Dev <[email protected]>
The shipped go is -trimpath (no compiled-in GOROOT), and the jail binds a host
allow-list with no /proc, so os.Executable fails and go/gopls die with 'cannot
find GOROOT directory' — every query answered null. Set GOROOT=/usr/local/go
(present via the /usr/local bind) and GOTELEMETRY=off (its counter child also
readlinks the absent /proc/self/exe). /v1/code/lsp answers Go queries after this.
Bump 0.1.2 -> 0.1.3.
Co-authored-by: Hanzo Dev <[email protected]>
The previous note said code-exec 'has the same bug'. It has the same ORDERING —
binds first, /tmp after — but binds its session tree at a fixed /work rather than
at the tree's own absolute path, so nothing of its lands under the tmpfs. It
fails the moment a bind path does. Naming which is which is the difference
between a warning someone can act on and one they learn to ignore.
Co-authored-by: Hanzo Dev <[email protected]>
The boot self-test failed on every host it ran on. The pod reported
JAIL SELF-TEST FAILED — refusing every request
err="jail: serve canary did not run: exit status 126"
and refused every request, correctly. 126 is not execve's — it is this jail's
own setupFailed. The child's stderr said what actually happened, and Probe threw
it away:
jail: chdir /tmp/jail: no such file or directory
ROOT CAUSE, and it is an ordering one. run() bound every allowed path at its own
absolute path and THEN mounted the writable tmpfs at <root>/tmp. A mount whose
path is an ancestor of an earlier one covers it, and nothing fails when it
happens: the bind is still there and simply cannot be seen. Every bind under
/tmp was placed and erased — and /tmp is where the staging directory lives, so
the self-test masked its own working tree and died on chdir with ENOENT.
The filesystem is now a value: plan() returns the mounts ordered shallowest path
first, one mount per path, so an ancestor-after-descendant order is not
representable and nobody has to remember which directories are safe to hand a
jail. The jail's own /tmp claims that path first, so a caller naming /tmp cannot
swap the host's directory in for the private tmpfs that dies with the process.
The working tree is the one path that is required: missing, it names itself
instead of leaving a chdir to report the symptom.
The canary is now EXEC'D — `lsp canary`, this same binary, reached through argv
because the child strips every control variable before exec. It used to report
from inside the process that built the jail, so the exec a language server
depends on was the one step the self-test never took.
Probe loses its second argument. It took a staging root AND a scratch working
directory, and shipped with the two in a relationship that cannot work; there is
now nothing to get wrong. It also distinguishes ErrHost — this host cannot
create namespaces — from every other failure, which is the jail's own.
Tests, all of which fail on the previous code:
TestThePlanHidesNothing no kernel, no privileges: nothing may cover an
earlier mount. Fails with the shipped order.
TestThePlanIsThePhase serve writes nothing, fetch writes the tree and
the cache, the tree is required.
TestTheJailKeepsItsOwnTmp an argument cannot replace the jail's /tmp.
TestProbeProves… now FAILS rather than skips when the jail builds
and misbehaves; skips only on ErrHost.
TestMain (both packages) a jail re-execs `self`, which under `go test` is
the test binary — without this the child re-ran
the suite and the canary compared "PASS" against
"socket=denied".
Measured on the gVisor node (code-exec-pool, runsc), work under /tmp:
serve reports `socket=denied operation not permitted`, fetch `socket=open`.
Probe's rlimits are generous on purpose — RLIMIT_NPROC is per-UID and shared
with the daemon's own threads, and at 32 a Go program cannot start a third.
hanzoai/code-exec's jail_linux.go, which this was copied from, orders its mounts
the same way and has the same bug wherever it eventually deploys.
Co-authored-by: Hanzo Dev <[email protected]>
hanzoai/ci derives an image version from the repo's own manifest — package.json,
Cargo.toml, VERSION, pyproject.toml. A Go module is none of those, so with
nothing declared and nothing yet in the registry `imgver` refuses the build:
imgver: no version for ghcr.io/hanzoai/lsp. Declare one — a
package.json/Cargo.toml/VERSION/pyproject.toml under '.', or IMGVER_VERSION.
We don't ship shas.
Which is the right refusal — restarting a series at 0, or publishing only a sha,
is how an image ends up with no deployable tag — but it means this repo has to
say the number itself, and hanzo.yml is the one place it can.
0.1.0 is a floor, not the output: ci publishes a declared number verbatim only
while it is above everything in the registry, and otherwise the next patch above
what is published. So this cannot mint a tag that already covers other bytes.
Co-authored-by: Hanzo Dev <[email protected]>
The entry described a Deployment somebody would have to write. It is written —
hanzoai/universe charts/app/values/hanzo/lsp.yaml, synced by cd.hanzo.ai, with
LSP_KEY coming from hanzo/lsp/LSP_KEY@prod through a KMSSecret and read by cloud
from the SAME path, so the proxy's X-API-Key and the daemon's key are one value
rather than a pair to keep in step.
What replaces the aspiration is the honest list of what is still in the way, and
none of it is code in this repository: gVisor is installed on no node (the
installer named a pool that does not exist and has been 0/0/0 for a month; the
fix is committed and unapplied), the pool it now names cannot hold an 8Gi pod,
and the first image is still building.
The last paragraph is the one worth keeping: none of that is dangerous, because
the boot-time probe refuses readiness when the jail does not work here. A wrong
node produces no endpoints, not an unjailed language server.
Co-authored-by: Hanzo Dev <[email protected]>
`useradd -r` allocated whatever system id was free in debian:bookworm-slim, and
`USER lsp` named the result. Both halves break the pod this image exists to run:
* The kubelet refuses `runAsNonRoot: true` against a non-numeric USER — it
will not resolve /etc/passwd inside the image to find out, so it fails the
container outright:
container has runAsNonRoot and image has non-numeric user (lsp),
cannot verify user is non-root
That is a CreateContainerConfigError on every start, and it would have hit
the first time this image met the hardened Deployment rather than in CI.
* `fsGroup` on the PVC has to name the same group. An id the base image
chooses is an id the values file has to guess, and a wrong guess is a
root-owned volume and an EACCES on the first `go mod download`.
65532 is the conventional distroless non-root id, so this agrees with the rest
of the fleet instead of inventing a number.
Co-authored-by: Hanzo Dev <[email protected]>
Two files, and neither is new policy — this is the fleet's one pipeline,
imported the way ~20 other repos import it. hanzo.yml says what to gate and
what to publish; .hanzo/workflows/cicd.yml is the ~10-line caller.
The one thing worth reading twice is the go-test gate. It installs gopls before
running the suite, at the version READ OUT OF THE DOCKERFILE, because
internal/lsp/gopls_test.go skips itself when gopls is not on PATH — and that
test is the acceptance test: it stands up a directory laid out as a Go module
proxy, points the fetch phase at it with GOPROXY=file:// so the real
`go mod download` runs with no network, and requires `locate definition` on a
dependency symbol to come back external at the module coordinate. Without the
install this repo would have shipped a green CI that never once ran the thing it
exists to do.
Reading the version from the Dockerfile rather than repeating it keeps ONE pin.
Two copies are right twice and then wrong once, and the wrong once is CI
certifying a language server the image does not run.
No `deploy:` block. Which build runs is a reviewed hanzoai/universe change —
tag plus digest — synced by cd.hanzo.ai. This daemon runs third-party language
servers over tenant source behind a jail; "which binary is behind that jail" is
the last thing that should move as a CI side effect.
Co-authored-by: Hanzo Dev <[email protected]>
A static symbol index knows what a repository says about itself. It cannot say
where dep.Greet is DEFINED, because the definition is in a module the repository
only names — and finding it means downloading that module and type-checking both
halves, which means running a third-party binary over untrusted bytes. That does
not belong in the API process. It belongs here, behind a jail.
The invariant is the separation: the network and untrusted bytes are never in the
same process. FETCH runs `go mod download all` with a socket, first-party
toolchain only, writing the tree and the dependency cache. SERVE runs gopls with
seccomp denying socket/socketpair/io_uring and every bind read-only. The phase is
not a field anyone sets — it is jail.Fetch or jail.Serve, so there is no value to
get wrong, and the only difference between their filters is two syscall numbers.
Roots are (org, repo, resolved-sha). Refusing a branch name at the door is the
entire cache-invalidation design: a sha names one tree for all time, a push is a
different key, and nothing decides when anything is stale. The daemon holds no
git credential and makes no git request; the caller sends the tree, and /ask
answers 409 {"need":"tree"} when it has not.
Fail closed. On Linux the jail is required: at boot a canary is built in each
phase and asked the kernel for a socket — serving must be refused one, fetching
must get one — and if that does not hold, /readyz is 503 and every request that
would run a language server is refused. There is no environment variable that
turns the jail off, so there is none that can be set by accident.
REUSED, not rewritten:
internal/lsp/conn.go, langs.go — ported from hanzoai/cloud branch `lsp`
(apps/lsp, 67975187) with their tests, including the two regressions that
build pinned: Close hanging on a wedged server, and a frame reader that
would allocate whatever Content-Length it was told.
internal/jail/ — a copy of hanzoai/code-exec's jailer (d2794ce), generalized
to two phases. It SHOULD be github.com/hanzoai/jail imported by both; a
boundary that exists twice drifts. Noted in LLM.md, not decided here.
Three bugs found and fixed on the way to green, all of which made the service
answer null with no error anywhere:
- the jail child now locks its OS thread before installing seccomp. The filter
and no_new_privs are per-THREAD and syscall.Exec execs the calling thread; a
goroutine migration between them would leave the exec'd program unfiltered.
This is a fix over the code-exec original and belongs upstream there too.
- clean() returns the root's own spelling, not the symlink-resolved one. The
proof of containment still resolves, but a server rooted at /var/x and handed
a document at /private/var/x does not consider it to be in its workspace.
- didOpen is sent once per document. It is not idempotent in LSP, and a root is
immutable, so once is also all that is ever needed.
- the fetch is `go mod download all`: the bare form records only each
dependency's go.mod hash, so the read-only serve phase then failed on a
missing go.sum entry.
Phase 1 serves Go. The table carries five complete languages and Available()
decides, so adding one is installing its server in the Dockerfile.
Acceptance is hermetic: a module is published into a directory laid out as a Go
proxy and fetched with GOPROXY=file://, so the real `go mod download` runs with
no network, and locate/definition on dep.Greet must return external:true at the
module coordinate. Passes in 0.08s once warm.
Co-authored-by: Hanzo Dev <[email protected]>
zeekayandHanzo Dev