- Add /proc/devices.
- Provide /proc/driver/nvidia/params when nvproxy is enabled. CUDA-using
applications already try to open this file, but silently tolerate its
non-existence. However, both the IMEX daemon and IMEX-using applications are
believed to require this file for ImexChannelCount. (libnvidia-container
additionally provides /proc/driver/nvidia/{registry,version}, but we have yet
to see any application use these.)
- Add new capability nvconf.CapFabricIMEXManagement. When enabled, provide
/proc/driver/nvidia/capabilities/fabric-imex-mgmt and the corresponding
/dev/nvidia-caps/nvidia-cap#.
- Add support for /dev/nvidia-caps-imex-channels/ (irrespective of the new
capability, since IMEX-using applications need this directory as well).
- Allow devutil.GoferClient.OpenAt() to take a relative path (i.e. containing
more than one path component) rather than only a basename, since this is
needed to open /dev/nvidia-caps/*.
- Add nvproxy support for all known-required controls, and all allocation
classes to which those controls correspond.
This CL is specifically for nvidia-imex, the "NVIDIA IMEX Service for NVLink
Networks" (or "IMEX daemon"); applications using IMEX haven't been tested yet,
and may require additional allocation classes / control commands, or some
existing ones marked as requiring CapFabricIMEXManagement as being enabled more
broadly.
PiperOrigin-RevId: 859286209
Tested on a T4 GPU with driver version 525.60.13:
```
$ docker run --runtime=runsc --gpus=all --rm -it nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubi8
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
```
Also tested this on GKE with the same vectoradd workload. Checked that the
device gofer connection is actually being closed when the container is deleted.
Something to note is that the gofer logs for the GPU-container sometimes end
abruptly (the "All lisafs servers exited." line does not print). This is
because runsc/container/container.go:stop() SIGKILLs the gofer before it can
cleanup naturally. The device gofer connection is only closed at the end of
Loader.destroySubcontainer(), which gives little time before the gofer is
SIGKILL-ed.
PiperOrigin-RevId: 581365665
Adds a gofer connection for /dev directory on the gofer when GPU functionality
is requested. This gofer connection is currently unused. The gofer client is
owned by the kernel, which injects the connection into the context. The gofer
connection is closed on container exit. S/R should be supported with this.
PiperOrigin-RevId: 581298536
