Fix a panic when writing to cgroup v2 control files from outside the sandbox
(e.g., via Cgroups.WriteControlFiles / WriteControl), where writes occur
without an open in-sandbox VFS file descriptor (fd == nil). cgroup.procs now
safely falls back to context credentials with unconfined namespace migration,
and checkNSDelegateWrite skips namespace delegation checks. Add an integration
test for runsc cgroup v2 control file writes.
Also:
- Export cgroup2fs.InitHierarchy to initialize sandbox-wide cgroup2 state
and root controller enablement.
- Plumb CgroupNamespace through sentry lifecycle and task context to derive
initial container cgroup2 placements.
PiperOrigin-RevId: 978087817
Scope the signal mutex to just the tg.tty clear in CreateSession, matching how
CreateProcessGroup, JoinProcessGroup, and the exit path already call
handleOrphan without holding a signal mutex.
CreateSession held tg.signalHandlers.mu across the entire process group
manipulation, but it only needs that mutex to clear tg.tty. When setsid(2)
orphans the caller's old process group, decRefWithParent calls handleOrphan,
which locks the signal mutex of every thread group remaining in the group. If
the caller itself, or a sibling thread group that shares the caller's
SignalHandlers (created via clone(CLONE_VM|CLONE_SIGHAND) without CLONE_THREAD),
is still a member, handleOrphan re-locks the already-held, non-reentrant mutex
and the sentry deadlocks while holding TaskSet.mu, freezing the whole sandbox.
Regression test hangs prior to the fix.
A tty inode was kept alive by a single ownership chain: the stdio file
description's reference on the import-time dentry. Everything else that used
the inode referred to it uncounted: a file description opened via /dev/tty
pins only the devtmpfs dentry, and the controlling-terminal association
(ThreadGroup.tty) is a bare pointer.
PiperOrigin-RevId: 962479941
When --mount-cgroup-v2 is enabled, a container whose spec includes a
cgroup mount is now given its own cgroup "/<cid>" in the cgroup2
hierarchy, with its init process placed there.
Cgroup namespaces from the spec are now honored, mirroring PID
namespace policy: no entry means the root namespace, an entry with a
path joins the container that declared the same path, and an entry
without a path gets a new namespace rooted at the container's cgroup.
Since a fresh cgroup2 mount is rooted at the mounting namespace's
root, a private-namespace container's /sys/fs/cgroup shows only its
own subtree, as on Linux, with no need for bind mounts.
At boot the sentry now enables nsdelegate and delegates controllers
to per-container cgroups via the root cgroup.subtree_control. Exec
sessions join the container's cgroup and cgroup namespace, like
runc exec. Per-container "runsc events" and read-control queries
now resolve for any container with a cgroup mount.
Note two differences from the existing v1 support. In v1 we bind mount:
the per-container dir "/$cid" of each controller hierarchy appears as
/sys/fs/cgroup/<controller>. With v2 no bind mounts are needed: a fresh
cgroup2 mount is rooted at the mounting namespace's root, so honoring
the spec's cgroupns request yields the scoped view naturally. (Binding
a subtree a la v1 was not an option anyway: systemd explicitly rejects
a sub-hierarchy mounted without a matching cgroup namespace.) Second,
on container destroy we now make a best-effort, recursive attempt to
remove the container's cgroup subtree; v1 leaves its per-container
dirs behind.
The root container is not special-cased. As with v1, where even the
root container gets container-id-stamped cgroup dirs and the stats
queries they enable, it gets its own "/$root_cid", and its cgroup
namespace comes from the spec like everyone else's. Docker and
Kubernetes both request a private cgroup namespace for every container
on cgroup2 hosts, so in practice the root container sees its cgroup as
a pleasing "/" rather than "/$root_cid"; a spec with no cgroupns entry
(e.g. docker run --cgroupns=host) instead sees the full sandbox
hierarchy, as on Linux. One consequence of the private default: in a
multi-container sandbox the root container cannot see the other
containers' cgroups (namespace roots are not hierarchical). I don't
expect that to be problematic, and a spec without a cgroupns entry
regains the full view if ever needed.
PiperOrigin-RevId: 960153510
Adds initial scaffolding for eBPF in gVisor. It implements the bpf(2) syscall
with support for BPF_PROG_LOAD, BPF_PROG_QUERY, and BPF_PROG_ATTACH.
Necessary for the cgroups v2 flag to be enabled by default, as this fixes
Docker-in-gVisor with cgroups v2.
This mount option turns cgroup namespaces into delegation boundaries,
which results in the following two behaviors:
First, resource control. A cgroup's resource-control files carve out
the parent's resources, so they belong to the delegater, not to the
subtree's occupants. Accordingly, tasks in a non-init cgroup namespace
may no longer write to interface files of their own namespace root.
The delegatable files (cgroup.procs, cgroup.subtree_control) are exempt.
Subtree tasks cannot change what share of resources the subtree receives,
but remain free to organize processes and distribute that share within it.
Second, containment. Migration (including via CLONE_INTO_CGROUP) now
requires that both the source and destination cgroups be descendants of
the cgroupns root of the task performing the migration, failing with
ENOENT otherwise (actually it is the cgroupns of the task that opened the
fd for the migration file: this detail forces me to snapshot the cgroupns
against the fd, increasing the size of this patch).
PiperOrigin-RevId: 954892289
cgroup namespaces are non hierarchical and they are defined primarily by a
"root" cgroup node. The init cgroup namespaces's root node is the true root of
the (only) cgroup2fs tree. The root node of other cgroup namespaces is the
cgroup node of the task at the time of namespace creation.
Residing in a cgroup namespace affects tasks this way:
1) The views into /proc/[pid]/cgroup and the "root" field under
/proc/[pid]/mountinfo are virtualized: they are rendered relative to the
reading task's cgroupns.
2) mount()ing a cgroup2 fs furnishes only a subtree of the full cgroup2fs
tree, rooted at the "root" cgroup of the hosting cgroupns.
Fixes#11989
PiperOrigin-RevId: 948882249
The cgroup v2 fs is initialized early in the boot process, so that every task
in the sandbox resides in at least the root v2 cgroup. A new runsc flag
--mount-cgroup-v2 has been added: this controls whether runsc mounts a
cgroupv2 filesystem at /sys/fs/cgroup automatically (currently it always
mounts a v1 hierarchy at this path). This flag also gates whether userspace
can mount a cgroup2 fs. When this flag is false, which is the default, all
tasks in the sandbox reside forever in the root cgroup.
Like in Linux, the v2 hierarchy is deemed the default owner of a controller,
that is, until the corresponding v1 hierarchy is mounted: then, the v1 tree
steals ownership of that controller and it remains this way till as long as
a v1 mount exists. In other words, hybrid mode is supported, even if such
workloads are rare these days.
The pids controller is the only controller implemented in full: it serves
as validation for the unified hierarchy. The memory controller can be used
to learn the usage stats of a subtree: but like the current v1 implementation,
and unlike Linux, it doesn't achieve this through charging up the hierarchy
during allocation, instead it queries pgalloc for every active controller's
id. And it does no enforcing. The cpu and cpuset controllers are mere stubs,
like in the current v1 implementation.
The following are supported:
* General matters of topology: cgroup.subtree_control, cgroup.controllers,
cgroup.procs etc
* CLONE_INTO_CGROUP
* cgroup.kill
* cgroup.events
* A working pids controller and its events files
* A minimal memory controller that just reports mem usage
* Stubbed cpu and cpuset controllers
The following are NOT supported:
* threaded cgroups
* cgroup namespaces
* BPF_PROG_TYPE_CGROUP_DEVICE
* cgroup.freeze
Updates #13007
PiperOrigin-RevId: 943519748
User namespace entries under `/proc/[pid]/ns` currently render as fake
namespace symlinks. They look like the other namespace files, but opening
them does not produce an `nsfs` file that `setns(2)` can use. Rootless
container tools such as `buildah` and `podman` rely on that file when they
re-enter the pause process user namespace, so the second lifecycle command
fails with `EINVAL`.
Make `UserNamespace` implement `vfs.Namespace` and give each user namespace
an `nsfs` inode when it is created. `/proc/[pid]/ns/user` now uses the
regular namespace symlink path, so opening it returns a joinable namespace
file instead of a fake link target.
`Setns` now accepts `CLONE_NEWUSER` from both `nsfd`s and `pidfd`s. It
follows the Linux restrictions for user namespace joins by rejecting the
caller's current user namespace, requiring `CAP_SYS_ADMIN` in the target
user namespace, rejecting multithreaded callers, and rejecting callers with
`fs` state shared outside the thread group. The capability checks for any
other namespaces in the same `setns` call use the credentials the caller
would have after joining the user namespace.
Add a syscall regression test that creates a child user namespace, opens
`/proc/<pid>/ns/user`, and verifies that `setns(CLONE_NEWUSER)` succeeds.
Fixes https://github.com/google/gvisor/issues/13314
COPYBARA_INTEGRATE_REVIEW=https://github.com/google/gvisor/pull/13323 from shayonj:issue-13314-userns-setns 8060b5f91ba1f2e9c7b6746da233d644728be2e9
PiperOrigin-RevId: 926855389
Previously we'd create a pidfd after NewTask() returns. If say there
was an error in copying the new pidfd to userspace, then we were voiding
the new task via failCloneAfterNewTaskCreation(), which would short circuit
the new task goroutine to a speedy death. But NewTask() had already presented
the new task to the rest of the system: allowing another thread to wait on
the task, as the data race discovered by syzkaller has revealed.
This change moves clone3 pidfd creation into NewTask() and before making the
new task visible.
I also changed how it creates the pidfd: previously we used the new task's tid
to look it up, which is wasteful and looks problematic, for looking up via tid
makes it seem like there is a threat to the atomic delivery of pidfds in
clone3(CLONE_PIDFD): but there actually was no such threat because the new
task cannot exit from under us to surrender its tid, but why make it a hard
read for no gain. So now the pidfd is created directly from the task itself.
Reported-by: [email protected]
PiperOrigin-RevId: 911099062
There are two issues being fixed: 1) checkAndPreventSharingOutsideTG() was
accessing t.fsContext without grabbing t.mu and thus causing a data race with
a concurrent unshare and 2) the concurrent unshare might present an incomplete
view of the shared fscontext and cause execve to incorrectly conclude that it
was shared outside the thread group.
Reported-by: [email protected]
Reported-by: [email protected]
PiperOrigin-RevId: 800327461
There is already some support for privileged executables before this change:
file capabilities were taken into account when creating new processes inside
the sandbox from external instigation (but not during execve(2)). set-user-id
and set-group-id bits were never taken into account, and the Sentry behaved
as if NO_NEW_PRIVS is always true. After this change, execve(2) and the
external Exec entry point both take the set-user(group)-id bits and file
capabilities into account.
Because NO_NEW_PRIVS can no longer be assumed to be always set, a couple of
extra checks to prevent privilege elevation during execve(2) are needed:
the presence of a ptracer and the use of a shared fsContext. Moreover,
execve had to be serialized with seccomp(2) when it is syncing filters to
all threads in a group. Serializing execve with PTRACE_ATTACH and seccomp
required some Task runState gymnastics.
Task dumpability is reset when privilege elevation takes place and privilege
elevation is denied for binaries living on nosuid mounts. To fix an old bug
where some aux vectors were populated using data from the old pre-execve
creds, the computation of the (file-privilege-augmented) creds during execve
now takes place before the stack for the new program is set up.
gvisor users who thus far have exec'd into the sandbox with UID as (sandbox)
root will lose the caps an all-powerful namespace root'd have if: 1) the binary
is a suid/guid binary 2) the owner of the binary is host root and there is
no mapping for host root in the host user namespace of the sandbox. To avoid
this surprise, the suid/sgid bits are ignored by default. Users can opt-in
via the --allow-suid flag.
One result of this change is that sudo now works inside the sandbox.
PiperOrigin-RevId: 794414101
Otherwise tasks can be created via the control server between when
Kernel.WaitExited() returns and when the control server is stopped, resulting
in task goroutines running when Kernel.Release() is called.
PiperOrigin-RevId: 658891833
Processes that are exec'ed into a container cannot be properly
restored because the caller is no longer present. This change
tracks processes that are exec'ed and kill them upon restore.
Updates #1956
PiperOrigin-RevId: 623644184
This removes the compare-and-branch involved in casting
`ptraceTracer.Load()` to `*Task`.
This is on the hot syscall path, and in the overwhelmingly-likely case that
the task is not being traced, we should be as fast as possible.
Shaves off about 20 nanoseconds from the syscall hot path:
```
│ before │ after │
│ sec/op │ sec/op vs base │
Syscallbench/syscall.getpidopt-8 1.355µ ± 2% 1.334µ ± 2% -1.59% (p=0.030 n=64)
```
PiperOrigin-RevId: 611277514
This adds a per-task cache of seccomp actions to take for syscall numbers
where the filters return an action without depending on anything other than
the syscall number and the architecture code of the seccomp program input.
This avoids evaluating seccomp-bpf programs in the syscall hot path, for
programs that use seccomp *within* gVisor (aka on themselves).
Benchmarks show that this removes about 50ns from the syscall hot path
for a trivial filter like the one in the benchmark.
Real-world filters are much longer, and the benefit is magnified the more
complex the filter is.
```
│ not_cached │ cached │
│ sec/op │ sec/op vs base │
SyscallUnderSeccomp 1.282µ ± 3% 1.230µ ± 1% -4.06% (p=0.002 n=6)
```
PiperOrigin-RevId: 586522068
- Added docstrings for exported functions.
- Exported kernel.userCounters since it is used by another package (testutil)
and is an exported field of an exported type `kernel.TaskConfig`.
PiperOrigin-RevId: 580615070
It is an idea of running codespell as part of our presubmit checks.
Before enabling it for new changes, let's fix what it has found.
Signed-off-by: Andrei Vagin <[email protected]>
The intention of this change is to cover a sufficient surface to accommodate
the use of running Docker within gVisor, rather than a full implementation.
This implements the following features:
- Keys as a first-class concept in the kernel.
- Tracking keys in user namespaces.
- Task session keyrings: possession, inheritance.
- Key permission enforcement.
- The following `keyctl(2)` operations:
- `KEYCTL_GET_KEYRING_ID`
- `KEYCTL_DESCRIBE`
- `KEYCTL_JOIN_SESSION_KEYRING`
- `KEYCTL_SETPERM`
Notably, this does not implement:
- The ability to actually add any keys other than the session keyring
(which does not hold any cryptographic key data).
- Other special keyrings (thread keyring, process keyring, user session
keyring, etc.).
- Lots of `keyctl(2)` operations.
- Key expiration.
- Key garbage collection. Keys live until their user namespace is destroyed.
However, each user namespace is limited to 200 keys, so memory growth is
bounded.
- `add_key(2)`
- `request_key(2)`
... However, this makes design choices that seem odd given the limited scope
of this change, but make sense when taking into account the desire to
eventually accommodate them in the future. For example, there are many
`switch` statements with only one option for session keyrings, which would get
more options when adding support for other special keyrings. Similarly, the
signature of `PossessedKeys` takes in all 3 special "possessed" keyrings, but
currently only ever gets the session keyring as non-nil.
PiperOrigin-RevId: 567047896
task.netns is always changed from a task goroutine under task.mu.
It means that we can access it without any locks from a task goroutine
we don't need to increment a reference counter in such cases.
In all other cases, we need to take task.mu.
PiperOrigin-RevId: 552913323
This change introduces the nsfs file system. Each new namespace allocates
a new nsfs inode.
Here are reasons why we need these inodes:
* each namespace has to have an unique id.
* proc/pid/ns/ contains one entry for each namespace. Bind mounting one of
the files in this directory to somewhere else in the filesystem keeps the
corresponding namespace alive even if all processes currently in
the namespace terminate.
* setns() allows the calling process to join an existing namespace specified
by a file descriptor.
PiperOrigin-RevId: 550694515
Currently there's no implementation to enable the container to be
initialized to it's cgroups and hence the `EnterInitialCgroups` would
inherit all the root's cgroups when the container would start.
With this implementation we make sure that when the container starts
it enter's into the cgroups that is passed down it to from the sandbox.
PiperOrigin-RevId: 540650868
We can't DecRef a cgroup with TaskSet.mu held as it leads to circular
locking. Restructure task creation to drop cgroup refs outside the
TaskSet.mu critical section.
Reported-by: [email protected]
Reported-by: [email protected]
PiperOrigin-RevId: 492589548
When charging a pids cgroup during thread creation, it is possible for
the hierachy containing the pids controller to be destroyed and
recreated. If thread creation fails and the charge has to be rolled
back, an intervening hierachy change previously caused a charge
underflow during the rollback.
If the hierachy changes between the charge and uncharge, the uncharge
is uncessary.
Reported-by: [email protected]
PiperOrigin-RevId: 471112484
This avoids requiring a lock in `ThreadGroup.ID`, which in turn breaks the
following lock cycle:
`kernel.taskSetRWMutex` -> `kernel.taskMutex` -> `mm.metadataMutex`
-> `mm.mappingRWMutex` -> `kernel.taskSetRWMutex`
(Also, less locking within `createVMALocked` is probably for the better in
general.)
PiperOrigin-RevId: 449588573
All atomic 64 bit ints are changed to atomicbitops.(Ui|I)nt64. A nogo checker
enforces that sync/atomic 64 bit functions are not called.
For reviewers: the interesting changes are in the atomicbitops and checkaligned
packages.
Why do this?
- It is very easy to accidentally use atomic values without sync/atomic funcs.
- We have checkatomics, but this is optional and is forgotten in several places.
- Using a type+checker to enforce this seems less error prone and simpler.
- We get NoCopy protection.
- Use of 64 bit atomics can break 32 bit builds. We have types to handle this
without any runtime cost, so we might as well use them.
PiperOrigin-RevId: 440473398
