Rust answered four queries in five against a warm root. The fifth came back
`rpc -32801: content modified`, which the daemon passed up as an error and the
door turned into a 502.
-32801 is the protocol's one retryable answer: the server computed against state
it has since replaced, so the result is stale rather than wrong, and the spec
has the client ask again. rust-analyzer says it for anything asked while a
workspace is still loading, which is exactly when a cold root is queried.
Retried in Call, so every op gets it without each caller learning the code, and
bounded twice over: a few attempts, and the caller's context underneath that. A
server that never settles still ends as an error rather than a request that
never returns.
Co-authored-by: Hanzo Dev <[email protected]>
TypeScript, JavaScript and Python answered exactly one question per root and
then went silent. PHP managed two. The daemon reported `server stopped`, and
0.1.10 finally attached the missing half of that sentence: `[exit status 1]`.
Not a signal, so nothing killed them — they LEFT.
processId is a watchdog. The spec says a server whose client process has died
should exit, and every server built on vscode-languageserver implements it by
polling kill(pid, 0) every few seconds. This daemon sent its own pid, which
means something only inside its own pid namespace — and the jail gives each
server a NEW one, where that number belongs to nothing at all. So each of them
looked for its client, correctly found nothing, and exited. They were obeying
the protocol; we were the ones lying to them.
null is the spec's way to say nobody is watching, and the jail is what makes it
free: the server is PID 1 of its namespace, so Close kills it and everything it
forked, and a daemon that dies takes the whole namespace with it. The lifetime
was never the pid's to enforce.
gopls, clangd, jdtls and ruby-lsp do not poll, which is why Go, C, C++, Java and
Ruby scored 5/5 the whole time and the fault read as TypeScript's. It was never
about package.json, or a lockfile, or address space.
Measured, not assumed: the acceptance probe now fires five sequential queries at
one WARM root per language. Asking once proves a server can resolve and proves
nothing about whether it is still there for the second question — which is the
state every caller is in after the first, and the entire reason the pool exists.
Co-authored-by: Hanzo Dev <[email protected]>
0.1.9 taught stopped() to carry the server's last words and then reported none,
because it read them too early. attach's handshake path already says why: Close
kills the process AND WAITS, which is what drains the copy of its stderr, and
reading the tail before that races the words away. The reader's own exit path
had exactly that bug — it closed done first, so every mid-session death was
published with nothing attached.
done now means the reader has stopped AND the process is reaped, so anything
read after it is final. Both endings go through one reap, guarded by its own
Once: two concurrent Waits on one process is a race with no useful outcome.
And the exit STATUS, which is the part that matters here. A process killed by
the kernel — a seccomp verdict, the OOM killer, an rlimit — says nothing on
stderr at all, so the status is the only account of it that exists, and it was
being discarded. It now lands in the same tail as the words, so "server stopped"
can never again be the whole story.
This is diagnosis, not a cure. TypeScript answers its first query and then dies,
on EVERY tree shape — pure source, package.json, lockfile, .js/.mjs/.cjs alike
— which rules out the lockfile as the trigger and rules out my own earlier
reading of it. What is left is a silent death, and this is what makes it speak.
Co-authored-by: Hanzo Dev <[email protected]>
Two defects, found from one report: a TypeScript tree with a package.json and no
lockfile answered the first query and then 502'd every one after it.
FIRST, the fetch. `npm ci` installs exactly what a lockfile pins and refuses
outright without one, so naming package.json as the marker bought a fetch that
could only ever fail — on every library that commits no lock, which is most of
them. Roots locates the FETCH and nothing else reads it, so the marker has to be
the file the fetch REQUIRES: package-lock.json or npm-shrinkwrap.json, and
Cargo.lock for `cargo fetch --locked`, which had the same defect and was only
quieter about it. A locked tree fetches exactly as before; a lockless one is
served from source, which is the trade every other fetchless language already
makes.
SECOND, and the reason the first took guesswork to investigate: a server that
died MID-SESSION took its reason with it. attach already keeps the last 4 KiB of
its stderr, but only spent it on a handshake failure — so a server that started
cleanly and died on the third query reported `write |1: broken pipe`, which
names the symptom and nothing about the cause. That is the blindness `exit
status 126` was for Probe, and it is fixed the same way: stopped() carries the
server's last words, and a failed write waits briefly for the reader to reach
the same conclusion rather than racing it and reporting the pipe.
The tree left behind by the failed fetch is CLEAN — npm refuses before it writes
anything — and package.json, a read-only tree and an unreachable registry all
fail to reproduce the death outside the jail. So the crash itself is not yet
explained, and this is what makes explaining it possible: the next occurrence
arrives with the words the server died saying.
Co-authored-by: Hanzo Dev <[email protected]>
0.1.6 served eight languages and logged `spawn EPERM` for TypeScript and
JavaScript once per root. 0.1.7 serves all nine, verified on the gVisor node.
Keeping the failure in the record rather than only the result: it is the
clearest evidence the graceful-degrade design works, and the next language to
need a jail change will look exactly like it.
Co-authored-by: Hanzo Dev <[email protected]>
TypeScript and JavaScript were the two languages the new image could not serve.
The daemon said why and kept going: `language server did not start … spawn
EPERM`, once per root, while the other eight answered.
socketpair was denied beside socket, on the reading that a served process must
be able to obtain no socket fd at all. But libuv gives a child its stdio through
a socketpair, so what that denied was not a network — it was Node starting a
process, and typescript-language-server exists to run tsserver.
A socketpair reaches nothing: both ends come back already connected to each
other, in AF_UNIX, inside this jail's namespaces. It cannot be connect()ed
elsewhere, and a peer that could pass a network fd over it would have to be
outside the jail, where nothing is. socket() is still denied, so there is still
no way to obtain a fd to a network.
The filter now reads args[0] and admits AF_UNIX alone. Allowing it outright
would have been equivalent today — the kernel implements socketpair for no other
family — but that is a fact about the kernel, not a property of this program,
and the claim in the header is worth keeping true by construction.
The arithmetic is the risk: those two jump edges are hand-computed offsets over
a list whose length changes with the phase, and an off-by-one reads as ALLOW.
TestSocketpairIsDecidedByItsFamily interprets the program for AF_UNIX and for
five families that must fail, in both phases, and pins that the carve-out did
not widen to socket().
Co-authored-by: Hanzo Dev <[email protected]>
They are how a Node package says which module system a file is in, and tsserver
reads both. Left out, a package whose entry point is .mjs looked to Speaks()
like a repository with no JavaScript in it — no server started, and the daemon
answered that it had nothing for the file. .mts and .cts are the same omission
on the TypeScript side.
ID() has to move with the list or a .mjs is opened as TypeScript and gets type
errors it was never written to satisfy.
Co-authored-by: Hanzo Dev <[email protected]>
The table always carried the other entries; the image carried one toolchain, so
Available() answered false and the daemon served Go alone. Install the rest —
node with typescript-language-server, pyright and intelephense; the Rust
component tarballs with rust-src; clangd's own release; a JDK with jdtls; ruby
with ruby-lsp; composer for PHP — every version pinned, and add the java, php
and ruby entries beside them.
The three new entries keep the scripts-off bargain rather than bending it. PHP
fetches with composer's own --no-scripts --no-plugins. Java has two dependency
resolvers and both ARE build tools, so it has no fetch and answers from source
and the JDK. Ruby's bundle install compiles native extensions, so it is marked
Executes and refused, as python's is.
Three things that would otherwise be silent:
* typescript-language-server prefers the TREE's node_modules/typescript —
dependency-authored JavaScript, loaded and run in the serve phase, undoing
what --ignore-scripts bought one phase earlier. Pin tsserver to the image's.
* ruby-lsp creates .ruby-lsp/ in the project and shells out to bundle install
unless BUNDLE_GEMFILE is set at all. On a read-only tree that is EACCES
before the server starts.
* Equinox takes its writable configuration area from the passwd entry, not
$HOME, so jdtls would write to /home/lsp, which the jail does not bind.
Name the area under the jail's own /tmp.
RLIMIT_NPROC is per UID and every jail shares it, so 256 became a way for a
JVM's forty threads to stop another language forking at all.
Co-authored-by: Hanzo Dev <[email protected]>
gopls answered 'no package metadata' / 500 'language server did not answer' for
cloud, because `go mod download all` failed: cloud's go.mod requires go>=1.26.5
and the image shipped 1.26.4 with GOTOOLCHAIN=local (never auto-downloads). The
symbols op worked (no dep resolution); locate/hover/definition did not. 1.26.5 is
the latest 1.26.x and what cloud names — bump the pin, per the daemon's own design.
Co-authored-by: Hanzo Dev <[email protected]>
The shipped go is -trimpath (no compiled-in GOROOT), and the jail binds a host
allow-list with no /proc, so os.Executable fails and go/gopls die with 'cannot
find GOROOT directory' — every query answered null. Set GOROOT=/usr/local/go
(present via the /usr/local bind) and GOTELEMETRY=off (its counter child also
readlinks the absent /proc/self/exe). /v1/code/lsp answers Go queries after this.
Bump 0.1.2 -> 0.1.3.
Co-authored-by: Hanzo Dev <[email protected]>
A static symbol index knows what a repository says about itself. It cannot say
where dep.Greet is DEFINED, because the definition is in a module the repository
only names — and finding it means downloading that module and type-checking both
halves, which means running a third-party binary over untrusted bytes. That does
not belong in the API process. It belongs here, behind a jail.
The invariant is the separation: the network and untrusted bytes are never in the
same process. FETCH runs `go mod download all` with a socket, first-party
toolchain only, writing the tree and the dependency cache. SERVE runs gopls with
seccomp denying socket/socketpair/io_uring and every bind read-only. The phase is
not a field anyone sets — it is jail.Fetch or jail.Serve, so there is no value to
get wrong, and the only difference between their filters is two syscall numbers.
Roots are (org, repo, resolved-sha). Refusing a branch name at the door is the
entire cache-invalidation design: a sha names one tree for all time, a push is a
different key, and nothing decides when anything is stale. The daemon holds no
git credential and makes no git request; the caller sends the tree, and /ask
answers 409 {"need":"tree"} when it has not.
Fail closed. On Linux the jail is required: at boot a canary is built in each
phase and asked the kernel for a socket — serving must be refused one, fetching
must get one — and if that does not hold, /readyz is 503 and every request that
would run a language server is refused. There is no environment variable that
turns the jail off, so there is none that can be set by accident.
REUSED, not rewritten:
internal/lsp/conn.go, langs.go — ported from hanzoai/cloud branch `lsp`
(apps/lsp, 67975187) with their tests, including the two regressions that
build pinned: Close hanging on a wedged server, and a frame reader that
would allocate whatever Content-Length it was told.
internal/jail/ — a copy of hanzoai/code-exec's jailer (d2794ce), generalized
to two phases. It SHOULD be github.com/hanzoai/jail imported by both; a
boundary that exists twice drifts. Noted in LLM.md, not decided here.
Three bugs found and fixed on the way to green, all of which made the service
answer null with no error anywhere:
- the jail child now locks its OS thread before installing seccomp. The filter
and no_new_privs are per-THREAD and syscall.Exec execs the calling thread; a
goroutine migration between them would leave the exec'd program unfiltered.
This is a fix over the code-exec original and belongs upstream there too.
- clean() returns the root's own spelling, not the symlink-resolved one. The
proof of containment still resolves, but a server rooted at /var/x and handed
a document at /private/var/x does not consider it to be in its workspace.
- didOpen is sent once per document. It is not idempotent in LSP, and a root is
immutable, so once is also all that is ever needed.
- the fetch is `go mod download all`: the bare form records only each
dependency's go.mod hash, so the read-only serve phase then failed on a
missing go.sum entry.
Phase 1 serves Go. The table carries five complete languages and Available()
decides, so adding one is installing its server in the Dockerfile.
Acceptance is hermetic: a module is published into a directory laid out as a Go
proxy and fetched with GOPROXY=file://, so the real `go mod download` runs with
no network, and locate/definition on dep.Greet must return external:true at the
module coordinate. Passes in 0.08s once warm.
Co-authored-by: Hanzo Dev <[email protected]>
