mirror of
https://github.com/hanzoai/visor.git
synced 2026-09-24 17:04:14 +00:00
Ayush RanjanandAlessio Ricci Toniolo
1f926a2ab7
Containers with /dev/infiniband/uverbs* devices need a sysfs surface for device discovery: libibverbs enumerates /sys/class/infiniband_verbs, and NCCL builds its hardware topology by resolving /sys/class symlinks with realpath() and walking the PCI hierarchy. This adds a --rdmaproxy flag which will gate all RDMA support. For now it only wires up the sysfs topology: runsc snapshots the host sysfs closure derived from the spec's uverbs devices (ibdevs, netdevs, PCI ancestor chains, GPU functions, NUMA nodes) before pivot_root, and the sentry constructs a path-identical virtual tree: one canonical /sys/devices/pci... subtree per device, with /sys/class/* and /sys/bus/pci/devices as symlink farms matching the kernel layout. Per-port dynamic state (RoCE GID tables, link state, counters) is served live through targeted read-only bind mounts of the per-ibdev ports subtrees; identity and PCI attributes are a static snapshot, since the collecting process and the constructed sandbox tree do not observe the same sysfs mount (the RDMA netdevs move into the sandbox network namespace only after the sandbox is created). Each PCI function directory also gets a subsystem symlink into a synthesized /sys/bus/pci, and every node's bus gets a pci_bus entry: NCCL classifies a directory as a PCI device by following its subsystem symlink, and aborts topology discovery without it. The NUMA topology is not mirrored: the sentry advertises a single node holding every sandbox CPU, consistent with its single-node memory model (set_mempolicy's maxNodes == 1) and its single-socket /proc/cpuinfo view. Host node cpumasks name CPUs the sandbox does not have, which breaks libnuma/hwloc consumers whenever the sandbox CPU count is below the host's (numactl node binding, which NUMA-aware stacks exec in front of worker subprocesses, kills the child). The PCI numa_node, local_cpus and local_cpulist attributes are rewritten to match the single-node view. The verbs device proxy that makes RDMA usable end-to-end is not part of this change; --rdmaproxy is a work in progress and RDMA is not yet functional. Co-authored-by: Alessio Ricci Toniolo <[email protected]>
64 lines
1.7 KiB
Go
64 lines
1.7 KiB
Go
// Copyright 2026 The gVisor Authors.
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
package specutils
|
|
|
|
import (
|
|
"path"
|
|
"strings"
|
|
|
|
specs "github.com/opencontainers/runtime-spec/specs-go"
|
|
"gvisor.dev/gvisor/pkg/rdma"
|
|
"gvisor.dev/gvisor/runsc/config"
|
|
)
|
|
|
|
// UverbsDevicesInSpec returns the /dev/infiniband/uverbs* devices listed in
|
|
// the OCI spec, in spec order.
|
|
func UverbsDevicesInSpec(spec *specs.Spec) []rdma.UverbsSpec {
|
|
if spec.Linux == nil {
|
|
return nil
|
|
}
|
|
var out []rdma.UverbsSpec
|
|
for _, dev := range spec.Linux.Devices {
|
|
if !strings.HasPrefix(dev.Path, "/dev/infiniband/uverbs") {
|
|
continue
|
|
}
|
|
out = append(out, rdma.UverbsSpec{
|
|
Name: path.Base(dev.Path),
|
|
Major: dev.Major,
|
|
Minor: dev.Minor,
|
|
})
|
|
}
|
|
return out
|
|
}
|
|
|
|
// RDMAEnabled returns true if the sandbox should set up RDMA support.
|
|
func RDMAEnabled(spec *specs.Spec, conf *config.Config) bool {
|
|
return conf.RDMAProxy && specHasUverbsDevice(spec)
|
|
}
|
|
|
|
// specHasUverbsDevice reports whether the spec lists any
|
|
// /dev/infiniband/uverbs* device.
|
|
func specHasUverbsDevice(spec *specs.Spec) bool {
|
|
if spec.Linux == nil {
|
|
return false
|
|
}
|
|
for _, dev := range spec.Linux.Devices {
|
|
if strings.HasPrefix(dev.Path, "/dev/infiniband/uverbs") {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|