hanzoai/lsp
2
0
Fork 0
mirror of https://github.com/hanzoai/lsp.git synced 2026-09-30 22:52:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

jail: give it a private /proc and the device nodes, and keep the server's last words

Browse Source 
gopls started and died at initialize. /root returned langs:[] and the daemon
said only:

  language server did not start  lang=go  err="lsp: initialize: lsp: server stopped"

That is the whole diagnosis, and it is the same blindness Probe had with `exit
status 126`: conn.go set cmd.Stderr = io.Discard, so the one place the server
explains itself was thrown away. With stderr visible, on the node, it said:

  failed to start telemetry sidecar: os.Executable: readlink /proc/self/exe:
    no such file or directory
  gopls cannot access its persistent index: can't hash gopls executable:
    readlink /proc/self/exe: no such file or directory

THE JAIL HAD NO /proc. os.Executable reads /proc/self/exe and x/telemetry calls
it on init, so no Go program can start. GOTELEMETRY=off does not help —
x/telemetry reads its mode file, not the environment.

The jail now brings a PRIVATE procfs, mounted inside its own PID namespace, so
it shows the jailed process and its children and nothing of the host: measured
on the node, 4 pids, no kcore, read-only, nosuid+nodev+noexec. This is a
tightening rather than a loosening — it also removes the reason GOROOT had to be
pinned by hand, since `go` can find itself again.

Fixing that exposed the next one: no /dev either, so gopls could not open
/dev/null. A user namespace cannot mknod a device, so null, zero, full, random
and urandom are BOUND from the host, writable, in both phases — none of them
holds state, and read-only would make them useless (Go's os/exec opens
/dev/null for every subprocess with no stdin, so `go list` could not run).
They are `must`: a jail missing /dev/null fails obscurely later, which is the
same shape as a mount that hides another.

stderr is now kept as the LAST 4 KiB and surfaced on ONE path — the error from
attach, built after Close, which kills and waits so the tail is complete.
Nothing is streamed while the server is healthy: it is chatty and can echo
tenant source.

And the acceptance test could never have caught any of this, because it has
never run jailed. A file:// GOPROXY is not on the jail's bind allow-list, so
`go mod download` reported a module that does not exist; on Linux the test
skipped via needJail and on a Mac it passed unjailed. proxyDirs binds it. That
is a production shape too — an air-gapped mirror on a PVC was a configuration
this daemon accepted and could never read.

Verified on the gVisor node (code-exec-pool, runsc), everything jailed:
  go test ./...  →  internal/jail ok, internal/lsp ok
  TestDefinitionCrossesIntoTheDependency PASS, including locate definition,
  hover, second root is warm, warm is accepted
  serve phase: tree still read-only, /dev/null writable, 4 pids visible
Before this commit the same test on the same node failed at
"/root served no Go server: map[cold:true langs:[] ready:true]" — the
production symptom.

Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
zeekay
2026-08-06 12:59:20 -07:00
co-authored by Hanzo Dev
parent e07a3b4a45
commit 9b338d0333
6 changed files with 245 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
LLM.md
+30
View File
@@ -71,6 +71,27 @@ order that hid part of it.
Sorting by depth makes an ancestor-after-descendant order unrepresentable, and
`TestThePlanHidesNothing` asserts it with no kernel and no privileges.
### What the jail brings with it
Three things are the jail's own, present in BOTH phases, and no argument can
replace them (the first claim on a path wins):
| | what | why |
|---|---|---|
| `/tmp` | fresh tmpfs, writable, nosuid+nodev | the only thing a served process can write; dies with the process |
| `/proc` | fresh procfs **in the jail's own PID namespace**, read-only, nosuid+nodev+noexec | shows the jailed process and its children, nothing of the host |
| `/dev/{null,zero,full,random,urandom}` | bound from the host, writable | a user namespace cannot `mknod`; none of them holds state |
`/proc` is not a hole — it is a private one, and without it no Go program starts.
`os.Executable` reads `/proc/self/exe`, `x/telemetry` calls it on init, and gopls
died at startup with `readlink /proc/self/exe: no such file or directory` while
the daemon reported only `server stopped`. `GOTELEMETRY=off` does not prevent it:
x/telemetry reads its mode file, not the environment.
`/dev/null` is required for the same class of reason — Go's `os/exec` opens it
for every subprocess with no stdin, so without it `go list` cannot run. It is
bound writable because a read-only mount makes `open(O_WRONLY)` return EROFS.
**hanzoai/code-exec carries the same ordering, and it is one change away from
the same bug.** This jailer was copied from its `internal/codeexec/jail_linux.go`,
which also binds first and mounts `/tmp` after. It is not hit TODAY only because
@@ -281,6 +302,15 @@ network, and then requires `locate definition` on `dep.Greet` to come back
`external: true` at the module coordinate. It skips where `gopls` is not
installed.
It could not run JAILED until a `file://` GOPROXY was bound into the fetch jail
(`proxyDirs` in `internal/lsp/root.go`). The jail binds an allow-list of host
paths, the proxy was not on it, and `go mod download` reported a module that does
not exist — so on Linux the test skipped via `needJail` and on a Mac it passed
unjailed. That is a real deployment shape too, not only a test's: an air-gapped
mirror on a PVC was a configuration the daemon accepted and could never read.
Verified end to end on the gVisor node: fetch, gopls start, `locate definition`
and `hover` all pass inside the jail.
The jail tests interpret the cBPF program the way the kernel would, so the jump
arithmetic — where an off-by-one silently ALLOWS a denied syscall — is checked
on any Linux with no privileges. `TestThePlanHidesNothing`, `TestThePlanIsThePhase`
hanzo.yml
+1 -1
View File
@@ -76,7 +76,7 @@ test:
# a number at or below what is published publishes the next patch above the
# registry. So this line is the floor, not the output, and it can never mint a
# tag that already covers different bytes.
version: 0.1.3
version: 0.1.4
images:
- name: lsp
internal/jail/jail_linux.go
+74 -16
View File
@@ -173,20 +173,26 @@ func run(p phase) error {
// ── what the jail's filesystem is ────────────────────────────────────────────
// mount is one line of that filesystem, as a value: what appears at path inside
// the jail, and whether the program may write it. An empty src means a fresh
// tmpfs of size MiB rather than a bind of a host path.
// the jail, and whether the program may write it. fs names a filesystem to
// create fresh; empty fs means a bind of src.
//
// path is ALWAYS the source's own absolute path. Same-path is not a convenience:
// a language server reports every answer as a file: URI of a path it was given,
// so a jail that renamed /roots/x to /work would hand the caller paths that name
// nothing — or would need a translation table, which is a second place for the
// two sides to disagree. There is nothing to translate here.
//
// write is the ONE truth about read-only-ness, for a bind and a fresh filesystem
// alike: MS_RDONLY is derived from it rather than carried beside it, so the two
// cannot disagree.
type mount struct {
src string // the host path to bind; empty ⇒ a fresh tmpfs
path string // absolute, and the same inside the jail as outside
size uint64 // MiB, for a tmpfs
write bool // may the jailed program write it
must bool // a missing source is an error rather than a skip
src string // the host path to bind; empty when fs is set
fs string // "tmpfs" | "proc": a fresh filesystem; empty ⇒ a bind of src
opt string // mount options for fs
flag uintptr // mount flags for fs, minus MS_RDONLY — see write
path string // absolute, and the same inside the jail as outside
write bool // may the jailed program write it
must bool // a missing source is an error rather than a skip
}
// plan is the whole filesystem the jail will have, ORDERED SHALLOWEST PATH
@@ -205,10 +211,59 @@ type mount struct {
// the rule is enforced rather than remembered. TestThePlanHidesNothing asserts
// it directly, needs no privileges, and would have caught this before it shipped.
func plan(p phase, work string, read, write []string, tmp uint64) []mount {
// The jail's own writable /tmp: the ONLY thing a served process can write,
// and it dies with the process. Build caches live here for exactly that
// reason. Shallowest of all, so everything binds ON it.
ms := []mount{{path: "/tmp", size: tmp, write: true}}
// The two filesystems the jail brings with it, shallowest of all so
// everything binds ON them.
ms := []mount{
// The jail's own writable /tmp: the ONLY thing a served process can
// write, and it dies with the process. Build caches live here for
// exactly that reason. nosuid/nodev because a scratch directory has no
// business holding a setuid binary or a device node; NOT noexec, since
// the go command compiles and runs out of $TMPDIR.
{fs: "tmpfs", path: "/tmp", write: true,
opt: fmt.Sprintf("mode=1777,size=%dm", tmp),
flag: syscall.MS_NOSUID | syscall.MS_NODEV},
// A PRIVATE procfs, and it is not a hole in the jail — it is this jail's
// own, mounted inside this jail's PID namespace, so it shows the jailed
// process and its children and NOTHING of the host. Read-only, nosuid,
// nodev, noexec.
//
// It is here because a Go program cannot start without it. os.Executable
// reads /proc/self/exe, and x/telemetry calls it on init, so gopls died
// at startup with
//
// failed to start telemetry sidecar: os.Executable: readlink
// /proc/self/exe: no such file or directory
// gopls cannot access its persistent index: can't hash gopls executable
//
// and the daemon reported only "server stopped". GOTELEMETRY=off does not
// prevent it: x/telemetry reads its mode file, not the environment. Every
// language server this daemon runs is a Go, Node or Python binary that
// expects a /proc, so giving the jail its own is the general answer and
// removes the reason GOROOT had to be pinned by hand.
{fs: "proc", path: "/proc",
flag: syscall.MS_NOSUID | syscall.MS_NODEV | syscall.MS_NOEXEC},
}
// The device nodes every toolchain assumes exist. They are BOUND from the
// host rather than created, because a user namespace cannot mknod a device —
// that is the one thing CAP_MKNOD inside a userns does not buy.
//
// They are writable in BOTH phases, and that is not a hole in the serve
// phase: none of them holds state. Writes to null and zero are discarded,
// full returns ENOSPC, and a write to random or urandom cannot credit
// entropy without CAP_SYS_ADMIN in the initial namespace. Read-only would
// break them for their actual purpose — `open("/dev/null", O_WRONLY)` on a
// read-only mount is EROFS, and Go's os/exec opens exactly that for every
// subprocess with no stdin, so `go list` could not run at all.
//
// REQUIRED, not skipped when absent: a jail without /dev/null fails later and
// obscurely, which is the same shape of bug as a mount that hides another.
for _, dev := range []string{
"/dev/null", "/dev/zero", "/dev/full", "/dev/random", "/dev/urandom",
} {
ms = append(ms, mount{src: dev, path: dev, write: true, must: true})
}
// Serve binds EVERYTHING read-only, including the working tree: a root is
// immutable and a language server has no business writing to one. Fetch binds
@@ -256,13 +311,16 @@ func depth(p string) int { return strings.Count(p, string(filepath.Separator)) }
// make performs one line of the plan, inside the staging root.
func (m mount) make(root string) error {
dst := filepath.Join(root, m.path)
if m.src == "" {
if err := os.MkdirAll(dst, 0o1777); err != nil {
if m.fs != "" {
if err := os.MkdirAll(dst, 0o755); err != nil {
return fmt.Errorf("mkdir %s: %w", m.path, err)
}
if err := syscall.Mount("tmpfs", dst, "tmpfs", 0,
fmt.Sprintf("mode=1777,size=%dm", m.size)); err != nil {
return fmt.Errorf("mount %s: %w", m.path, err)
flag := m.flag
if !m.write {
flag |= syscall.MS_RDONLY
}
if err := syscall.Mount(m.fs, dst, m.fs, flag, m.opt); err != nil {
return fmt.Errorf("mount %s %s: %w", m.fs, m.path, err)
}
return nil
}
internal/jail/jail_linux_test.go
+72 -9
View File
@@ -7,6 +7,7 @@ import (
"errors"
"os"
"strings"
"syscall"
"testing"
"time"
)
@@ -139,10 +140,17 @@ func TestThePlanIsThePhase(t *testing.T) {
}{{serve, false}, {fetch, true}} {
var sawWork, sawTmp bool
for _, m := range plan(c.p, work, []string{tool}, []string{cache}, 16) {
// The jail's OWN entries — its filesystems and its device nodes — are
// the same in both phases and are pinned by
// TestTheJailBringsItsOwnFilesystems. This test is about the paths a
// CALLER supplied.
if m.fs != "" && m.path != "/tmp" || strings.HasPrefix(m.path, "/dev/") {
continue
}
switch m.path {
case "/tmp":
// The jail's own scratch: writable in BOTH phases, and it is the
// only thing a served process may write.
// only thing a served process may write to that survives a syscall.
sawTmp = true
if !m.write || m.src != "" {
t.Fatalf("%s: /tmp must be a writable tmpfs, got %+v", c.p, m)
@@ -170,14 +178,69 @@ func TestThePlanIsThePhase(t *testing.T) {
}
}
// TestTheJailKeepsItsOwnTmp pins a boundary that must not be reachable from an
// argument. The jail's /tmp is a private tmpfs that dies with the process and is
// the only thing a served program may write; a caller naming /tmp as a tree or a
// cache must not be able to swap the host's directory in for it.
func TestTheJailKeepsItsOwnTmp(t *testing.T) {
for _, m := range plan(fetch, "/tmp", []string{"/tmp"}, []string{"/tmp"}, 16) {
if m.path == "/tmp" && m.src != "" {
t.Fatalf("a caller replaced the jail's private /tmp with a bind of %s", m.src)
// TestTheJailBringsItsOwnFilesystems pins the two filesystems the jail creates
// rather than borrows, and the fact that no argument can reach them.
//
// /tmp is a private tmpfs that dies with the process and is the only thing a
// served program may write. /proc is a private procfs mounted inside the jail's
// own PID namespace — it shows the jailed process and its children and nothing
// of the host, and it is read-only. A caller naming either as a tree or a cache
// must not be able to swap a host directory in for it: a boundary that an
// argument can replace is not a boundary.
func TestTheJailBringsItsOwnFilesystems(t *testing.T) {
for _, p := range []phase{serve, fetch} {
var tmp, proc bool
for _, m := range plan(p, "/tmp", []string{"/proc", "/tmp"}, []string{"/proc"}, 16) {
switch m.path {
case "/tmp":
tmp = true
if m.fs != "tmpfs" || m.src != "" {
t.Fatalf("%s: a caller replaced the jail's /tmp with %+v", p, m)
}
if !m.write {
t.Fatalf("%s: the jail's /tmp must be writable", p)
}
case "/proc":
proc = true
if m.fs != "proc" || m.src != "" {
t.Fatalf("%s: a caller replaced the jail's /proc with %+v", p, m)
}
// Read-only, and never a place to execute from. os.Executable
// needs to READ /proc/self/exe; nothing needs to write /proc.
if m.write {
t.Fatalf("%s: /proc must be read-only", p)
}
if m.flag&syscall.MS_NOEXEC == 0 || m.flag&syscall.MS_NOSUID == 0 ||
m.flag&syscall.MS_NODEV == 0 {
t.Fatalf("%s: /proc must be nosuid+nodev+noexec, flags=%#x", p, m.flag)
}
}
}
if !tmp || !proc {
t.Fatalf("%s: the jail must bring its own /tmp and /proc", p)
}
// The device nodes, in BOTH phases. Bound (a userns cannot mknod) and
// writable, because Go's os/exec opens /dev/null for every subprocess
// with no stdin and a read-only mount makes that EROFS. Required, so a
// host without them fails at the cause rather than inside a toolchain.
want := map[string]bool{
"/dev/null": false, "/dev/zero": false, "/dev/full": false,
"/dev/random": false, "/dev/urandom": false,
}
for _, m := range plan(p, "/roots/x", nil, nil, 16) {
if _, ok := want[m.path]; !ok {
continue
}
want[m.path] = true
if m.src != m.path || !m.write || !m.must {
t.Fatalf("%s: %s must be a required writable bind of itself, got %+v", p, m.path, m)
}
}
for dev, saw := range want {
if !saw {
t.Fatalf("%s: the jail must bring %s — Go opens it for every subprocess", p, dev)
}
}
}
}
internal/lsp/conn.go
+47 -3
View File
@@ -100,8 +100,16 @@ func (e *rpcError) Error() string { return fmt.Sprintf("lsp: rpc %d: %s", e.Code
// is what ends it.
func attach(ctx context.Context, cmd *exec.Cmd, l Lang, root string) (*Conn, error) {
// The server's stderr is its own log, not ours to relay: it is chatty and it
// echoes tenant source. Dropping it keeps both out of our logs.
cmd.Stderr = io.Discard
// can echo tenant source, so it is never streamed anywhere. But a server that
// DIES writes its reason there and nowhere else, and discarding it outright
// left "lsp: server stopped" as the entire account of a startup crash —
// exactly the blindness Probe had with `exit status 126`.
//
// So: keep the LAST 4 KiB and surface it on ONE path, the error below. That
// error is built after Close, which kills the process and waits, so what the
// server managed to say is all there. Nothing is logged while it is healthy.
last := &tail{n: 4 << 10}
cmd.Stderr = last
in, err := cmd.StdinPipe()
if err != nil {
@@ -124,12 +132,48 @@ func attach(ctx context.Context, cmd *exec.Cmd, l Lang, root string) (*Conn, err
_ = cmd.Wait()
})
if err := c.handshake(ctx, l, root); err != nil {
// Close first: it kills the process AND waits for it, which drains the
// stderr copy. Reading the tail before that races the last words away.
c.Close()
return nil, err
return nil, fmt.Errorf("%w%s", err, last.why())
}
return c, nil
}
// tail is a writer that remembers only the last n bytes given to it. A language
// server is a subprocess parsing a hostile tree and its log is unbounded; this
// is the bound, so a server that fails while printing forever cannot become the
// pod's memory problem.
type tail struct {
mu sync.Mutex
buf []byte
n int
}
func (t *tail) Write(p []byte) (int, error) {
t.mu.Lock()
defer t.mu.Unlock()
t.buf = append(t.buf, p...)
if len(t.buf) > t.n {
t.buf = t.buf[len(t.buf)-t.n:]
}
return len(p), nil
}
// why renders the tail for an error message, and empty when there is nothing to
// say — so a failure with a silent server does not grow a dangling separator.
func (t *tail) why() string {
if t == nil {
return ""
}
t.mu.Lock()
defer t.mu.Unlock()
if s := strings.TrimSpace(string(t.buf)); s != "" {
return ": " + s
}
return ""
}
// newConn puts a Conn on an already-open transport and starts its reader. attach
// uses it for a real process; a test uses it for a pipe, and both then drive the
// identical client.
internal/lsp/root.go
+21 -1
View File
@@ -233,7 +233,7 @@ func (s *Service) fetch(ctx context.Context, l Lang, dir, mod string) error {
Root: s.cfg.Stage,
Work: dir,
Dir: filepath.Join(dir, filepath.FromSlash(mod)),
Read: toolchainRead,
Read: append(append([]string{}, toolchainRead...), proxyDirs(s.cfg.Proxy)...),
Write: []string{cache},
Argv: argv,
Env: s.env(l, l.FetchEnv),
@@ -312,6 +312,26 @@ func (s *Service) env(l Lang, phase []string) []string {
return out
}
// proxyDirs are the directories a file:// GOPROXY names, to be bound read-only
// into the fetch jail.
//
// Without them a file:// proxy is a configuration the daemon ACCEPTS and can
// never read: the jail binds an allow-list of host paths, the proxy is not on
// it, and `go mod download` reports a module that does not exist. That is a real
// deployment — an air-gapped mirror on a PVC — and it is also what the hermetic
// acceptance test uses, which is why that test could only ever pass unjailed.
//
// GOPROXY is a comma-separated list and may name several.
func proxyDirs(goproxy string) []string {
var out []string
for _, p := range strings.Split(goproxy, ",") {
if rest, ok := strings.CutPrefix(strings.TrimSpace(p), "file://"); ok && rest != "" {
out = append(out, filepath.FromSlash(rest))
}
}
return out
}
// absolute resolves argv[0] on PATH here, in the parent, where PATH means
// something. Inside the jail there is no shell and no lookup: execve is given a
// path or it fails.