Add an efaproxy driver plug-in for AWS Elastic Fabric Adapter (EFA) devices
bound to the host `efa` driver. EFA's driver-private uverbs ABI carries no
userspace buffer pointers on CQ/QP CREATE: queue memory is kernel-allocated
and handed back via mmap keys in the CREATE response, which the core already
forwards to the host FD verbatim. PrepareCreateDMA is therefore a no-op; the
plug-in exists to register the "efa" driver name and to model EFA's MR_QUERY
driver-namespace method (used by libfabric to fetch RDMA read/write
interconnect IDs).
Model the standard AH object's destroy method (UVERBS_OBJECT_AH): EFA (SRD)
creates address handles via the legacy write path and destroys them through
the modern object, which the core must allow.
Extend the virtual RDMA sysfs for libfabric/hwloc topology discovery:
- device/driver symlink and /sys/bus/pci/drivers/<driver> tree, which
libfabric's EFA provider resolves during device discovery.
- Mirror each PCI node's raw config space ("config" file). hwloc (used by
aws-ofi-nccl to build the NCCL topology) reads it to recover the PCI
bridge bus-number registers; without it hwloc cannot reconstruct the
bridge hierarchy and aws-ofi-nccl aborts its topology write.
- Serve the per-port lid_mask_count attribute alongside the other port
attributes libibverbs reads from sysfs; ib_core creates it on every
port.
The sysfs integration tests cover the new driver tree and config mirroring.
Advertise a >= 5.12 kernel release in RDMA-enabled sandboxes so aws-ofi-nccl
enables dmabuf-based GPU memory registration (ibv_reg_dmabuf_mr), which the
proxy handles via the existing REG_DMABUF_MR path. Sandboxes without RDMA
keep the current 4.19 release to limit the bump's blast radius; a later
change can make the newer release the default.
Co-authored-by: Alessio Ricci Toniolo <[email protected]>
Containers with /dev/infiniband/uverbs* devices need a sysfs surface for
device discovery: libibverbs enumerates /sys/class/infiniband_verbs, and
NCCL builds its hardware topology by resolving /sys/class symlinks with
realpath() and walking the PCI hierarchy.
This adds a --rdmaproxy flag which will gate all RDMA support. For now it
only wires up the sysfs topology: runsc snapshots the host sysfs closure
derived from the spec's uverbs devices (ibdevs, netdevs, PCI ancestor
chains, GPU functions, NUMA nodes) before pivot_root, and the sentry
constructs a path-identical virtual tree: one canonical /sys/devices/pci...
subtree per device, with /sys/class/* and /sys/bus/pci/devices as symlink
farms matching the kernel layout. Per-port dynamic state (RoCE GID tables,
link state, counters) is served live through targeted read-only bind mounts
of the per-ibdev ports subtrees; identity and PCI attributes are a static
snapshot, since the collecting process and the constructed sandbox tree do
not observe the same sysfs mount (the RDMA netdevs move into the sandbox
network namespace only after the sandbox is created).
Each PCI function directory also gets a subsystem symlink into a
synthesized /sys/bus/pci, and every node's bus gets a pci_bus entry: NCCL
classifies a directory as a PCI device by following its subsystem symlink,
and aborts topology discovery without it.
The NUMA topology is not mirrored: the sentry advertises a single node
holding every sandbox CPU, consistent with its single-node memory model
(set_mempolicy's maxNodes == 1) and its single-socket /proc/cpuinfo view.
Host node cpumasks name CPUs the sandbox does not have, which breaks
libnuma/hwloc consumers whenever the sandbox CPU count is below the
host's (numactl node binding, which NUMA-aware stacks exec in front of
worker subprocesses, kills the child). The PCI numa_node, local_cpus and
local_cpulist attributes are rewritten to match the single-node view.
The verbs device proxy that makes RDMA usable end-to-end is not part of
this change; --rdmaproxy is a work in progress and RDMA is not yet
functional.
Co-authored-by: Alessio Ricci Toniolo <[email protected]>
gVisor does not expose real block/char devices, the directories remain
empty, but their existence is required for compatibility.
Tools like lsblk expect /sys/dev/block/ to exist in order to enumerate
block devices. Currently, gVisor's sysfs creates /sys/dev/ as an empty
directory without the standard block/ and char/ children, causing lsblk
to fail with ENOENT.
Signed-off-by: Tan Yifeng <[email protected]>
linkat(AT_EMPTY_PATH) needs to check for CAP_DAC_READ_SEARCH in the userns in
which the fd was opened, not the userns of the task running linkat().
To get at that userns, vfs.FileDescription now saves a pointer to the creds
at the time of its creation: plumbing these creds in accounts for much of the
diff. This is similar to how `struct file` stores creds in `f_cred` in Linux.
Moreover, since the Linux patch "vfs: relax linkat() AT_EMPTY_PATH..." [1], the
need for this cap is excused if the creds of the calling task are literally
the same as the one stored in the fd.
[1] https://github.com/torvalds/linux/commit/42bd2af5950456d
PiperOrigin-RevId: 853374474
Ubuntu TPU images do not have the vfio-dev directories that COS images do,
so we need a more robust way of setting up the sandbox chroot to handle this
case. This change implements a way to get devices and minor numbers into the
sandbox with minimal support from the host filesystem and cleans up a few
methods to reflect their current usage.
Addresses #10795
PiperOrigin-RevId: 674363342
This is required for Open MPI => hwloc to detect the number of CPUs and set the
number of Open MPI slots accordingly.
Before this CL, on amd64:
```
root@0496c77e84e9:/# ls /sys/devices/system/cpu/ | grep cpu | wc -l
96
root@0496c77e84e9:/# mpirun --allow-run-as-root -np 96 /bin/true
[hwloc/linux] failed to find sysfs cpu topology directory, aborting linux discovery.
[0496c77e84e9:00667] OPAL ERROR: Not supported in file ../../../../../opal/mca/hwloc/base/hwloc_base_util.c at line 418
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):
topology discovery failed
--> Returned value Not supported (-8) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
```
Before this CL, on arm64:
```
root@bfad6b71e996:/# ls /sys/devices/system/cpu/ | grep cpu | wc -l
4
root@bfad6b71e996:/# mpirun --allow-run-as-root -np 4 /bin/true
[hwloc/linux] failed to find sysfs cpu topology directory, aborting linux discovery.
[hwloc/linux] failed to find sysfs cpu topology directory, aborting linux discovery.
--------------------------------------------------------------------------
All nodes which are allocated for this job are already filled.
--------------------------------------------------------------------------
```
After this CL, on amd64:
```
root@0512ce557005:/# ls /sys/devices/system/cpu/ | grep cpu | wc -l
96
root@0512ce557005:/# mpirun --allow-run-as-root -np 96 /bin/true
root@0512ce557005:/# mpirun --allow-run-as-root -np 97 /bin/true
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 97
slots that were requested by the application:
...
```
After this CL, on arm64:
```
root@faf6ff491bc5:/# ls /sys/devices/system/cpu/ | grep cpu | wc -l
4
root@faf6ff491bc5:/# mpirun --allow-run-as-root -np 4 /bin/true
root@faf6ff491bc5:/# mpirun --allow-run-as-root -np 5 /bin/true
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 5
slots that were requested by the application:
...
```
Per #10484, this also lets `nvidia-smi topo` make more progress:
```
root@d5a696bb7e2c:/# nvidia-smi topo -m
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity GPU NUMA ID
Failed to run topology matrix
```
PiperOrigin-RevId: 663395602
V5 support was broken in a few different ways:
- tpu device files (/dev/vfio/X) were created with incorrect minor device nums.
- PCI paths on bus' other than 0000:00 were not supported.
- VFIO unmap was broken and not properly added to the seccomp allowlist.
- The VFIO main device file (/dev/vfio/vfio) did not account for overlapping
device address ranges that correspond to different VFIO container groups.
Previously TPU support was tested on machines with single TPUs, which masked
most of these issues.
All these issues should be fixed by this change. Tested manually on GKE.
PiperOrigin-RevId: 656037853
The symlinks mirrors the relationship between TPU devices and its respective
IOMMU group and on the host.
Before the change, iommu_group has been treated as a normal file, which doesn't
create such symlink for the devices.
PiperOrigin-RevId: 597939650
- Added `fsutil.ForEachDirent()` (used by directfs).
- Added `fsutil.DirentNames()`, which uses `ForEachDirent()`. Used by sysfs
and in the future will be used by nvproxy/tpuproxy.
Separately, fixed some bugs in sysfs:
- We were leaking FDs in `hostDirEntries()`.
- We were leaking FD in `hostFile.Generate()` on error path.
- Got rid of `hostFileBufSize` users. They expected contents to be of 4096
bytes only. Instead made the functions agnostic of file size.
PiperOrigin-RevId: 580688250
The TPU userspace driver needs access to specific PCI device information
located in Linux sysfs. We mirror the sysfs paths the driver reads on the host
in the Sentry sysfs. This way we can ensure we only expose the host device
information that's strictly necessary for TPU to run.
PiperOrigin-RevId: 550005271
These inodes can never be part of a filesystem tree. They are nameless and
never have a parent.
This allows us to avoid taking a lock in kernfs.InotifyWithParent for such
anonymous inodes.
PiperOrigin-RevId: 538823227
This catches up the interface to the `EmitUnimplementedEvent` method signature
on `kernel.Kernel`.
Also add build-time test to verify that `kernel.Kernel` implements this
interface, in order to catch such breakages at build time in the future.
PiperOrigin-RevId: 519000411
Some implementations handle more flags than others, so it doesn't
make sense to have one set of rules for all.
This change should functionally be a no-op.
PiperOrigin-RevId: 502712415
This is consistent with Linux. Some applications recursively add
files to inotify FD. They fail if they run into kernfs files
because inotify_add_watch(2) returns EPERM in that case.
PiperOrigin-RevId: 462642204
Since refsvfs2 are used in netstack, we should use atomicbitops to avoid
breaking 32-bit builds.
On 64-bit builds there is no performance difference.
PiperOrigin-RevId: 439687980
Some applications use product_name to find out if they are running inside
cloud providers and behave differently, like changing network settings or
expecting services to be available to the VM.
More details here: https://docs.google.com/document/d/1xQXE27x9wTvwPsgiX9Hn0o8mcq5z3SKi-1jwscQsCAk
This is also needed for: go/auth-library-noping
PiperOrigin-RevId: 435454155
Create the /sys/fs/cgroup directory when cgroups are available. This
creates the empty directory to serve as the mountpoint, actually
mounting cgroups is left to the launcher/userspace. This is consistent
with Linux behaviour.
Without this mountpoint, getdents(2) on /sys/fs indicates an empty
directory even if the launcher mounts cgroupfs at /sys/fs/cgroup. The
launcher can't create the mountpoint directory since sysfs doesn't
support mkdir.
PiperOrigin-RevId: 398596698
Removes package syserror and moves still relevant code to either linuxerr
or to syserr (to be later removed).
Internal errors are converted from random types to *errors.Error types used
in linuxerr. Internal errors are in linuxerr/internal.go.
PiperOrigin-RevId: 390724202
Update all instances of the above errors to the faster linuxerr implementation.
With the temporary linuxerr.Equals(), no logical changes are made.
PiperOrigin-RevId: 382306655
Remove three syserror entries duplicated in linuxerr. Because of the
linuxerr.Equals method, this is a mere change of return values from
syserror to linuxerr definitions.
Done with only these three errnos as CLs removing all grow to a significantly
large size.
PiperOrigin-RevId: 382173835
Add a coverage-report flag that will cause the sandbox to generate a coverage
report (with suffix .cov) in the debug log directory upon exiting. For the
report to be generated, runsc must have been built with the following Bazel
flags: `--collect_code_coverage --instrumentation_filter=...`.
With coverage reports, we should be able to aggregate results across all tests
to surface code coverage statistics for the project as a whole.
The report is simply a text file with each line representing a covered block
as `file:start_line.start_col,end_line.end_col`. Note that this is similar to
the format of coverage reports generated with `go test -coverprofile`,
although we omit the count and number of statements, which are not useful for
us.
Some simple ways of getting coverage reports:
bazel test <some_test> --collect_code_coverage \
--instrumentation_filter=//pkg/...
bazel build //runsc --collect_code_coverage \
--instrumentation_filter=//pkg/...
runsc -coverage-report=dir/ <other_flags> do ...
PiperOrigin-RevId: 368952911
This lets us avoid treating a value of 0 as one reference. All references
using the refsvfs2 template must call InitRefs() before the reference is
incremented/decremented, or else a panic will occur. Therefore, it should be
pretty easy to identify missing InitRef calls during testing.
Updates #1486.
PiperOrigin-RevId: 341411151
Much like the VFS2 gofer client, kernfs too now caches dentries. The size of the
LRU cache is configurable via mount options.
Have adopted the same reference semantics from gofer client dentry.
Only sysfs and procfs use this LRU cache. The rest of the kernfs users (devpts,
fusefs, host, pipefs, sockfs) still use the no cache approach.
PiperOrigin-RevId: 339139835
Added the following fields in kernfs.InodeAttr:
- blockSize
- atime
- mtime
- ctime
Also resolved all TODOs for #1193.
Fixes#1193
PiperOrigin-RevId: 338714527
Our current reference leak checker uses finalizers to verify whether an object
has reached zero references before it is garbage collected. There are multiple
problems with this mechanism, so a rewrite is in order.
With finalizers, there is no way to guarantee that a finalizer will run before
the program exits. When an unreachable object with a finalizer is garbage
collected, its finalizer will be added to a queue and run asynchronously. The
best we can do is run garbage collection upon sandbox exit to make sure that
all finalizers are enqueued.
Furthermore, if there is a chain of finalized objects, e.g. A points to B
points to C, garbage collection needs to run multiple times before all of the
finalizers are enqueued. The first GC run will register the finalizer for A but
not free it. It takes another GC run to free A, at which point B's finalizer
can be registered. As a result, we need to run GC as many times as the length
of the longest such chain to have a somewhat reliable leak checker.
Finally, a cyclical chain of structs pointing to one another will never be
garbage collected if a finalizer is set. This is a well-known issue with Go
finalizers (https://github.com/golang/go/issues/7358). Using leak checking on
filesystem objects that produce cycles will not work and even result in memory
leaks.
The new leak checker stores reference counted objects in a global map when
leak check is enabled and removes them once they are destroyed. At sandbox
exit, any remaining objects in the map are considered as leaked. This provides
a deterministic way of detecting leaks without relying on the complexities of
finalizers and garbage collection.
This approach has several benefits over the former, including:
- Always detects leaks of objects that should be destroyed very close to
sandbox exit. The old checker very rarely detected these leaks, because it
relied on garbage collection to be run in a short window of time.
- Panics if we forgot to enable leak check on a ref-counted object (we will try
to remove it from the map when it is destroyed, but it will never have been
added).
- Can store extra logging information in the map values without adding to the
size of the ref count struct itself. With the size of just an int64, the ref
count object remains compact, meaning frequent operations like IncRef/DecRef
are more cache-efficient.
- Can aggregate leak results in a single report after the sandbox exits.
Instead of having warnings littered in the log, which were
non-deterministically triggered by garbage collection, we can print all
warning messages at once. Note that this could also be a limitation--the
sandbox must exit properly for leaks to be detected.
Some basic benchmarking indicates that this change does not significantly
affect performance when leak checking is enabled, which is understandable
since registering/unregistering is only done once for each filesystem object.
Updates #1486.
PiperOrigin-RevId: 338685972
