Exclusive GPU tests(tests that are failed) now launch a dedicated container
instead of using one from the pre-warmed pool.
PiperOrigin-RevId: 915689394
This change moves the Server interface, DockerServer implementation, ResponseMetrics, APIResponse, and MakeAPIResponse functions from the individual LLM test packages (ollama, sglang, triton, vllm) into a new shared package llmutil. This reduces code duplication across the different LLM test clients.
PiperOrigin-RevId: 907987791
The answer might come in multiple tokens. In that case, we add a space in
between. Let's add that case as a valid answer for the test as a fix.
PiperOrigin-RevId: 855360375
The `conjugateGradientMultiDeviceCG` CUDA sample is now passing after
62023143e7 ("nvproxy: Add support for UVM_ENABLE_READ_DUPLICATION and
UVM_SET_ACCESSED_BY.") and no longer needs to be marked as broken in gVisor.
This fixes BuildKite failures.
PiperOrigin-RevId: 853365121
Docker's `restore` functionality is broken starting from v28 due to
https://github.com/moby/moby/issues/50750. This change adds a version check to
`dockerutil` and skips tests that rely on `docker restore` if the Docker
version is v28 or greater. TCP checkpoint/restore tests are also refactored to
centralize common setup and version checks.
PiperOrigin-RevId: 818827864
CUDA tests run in reasonable time with this flag enabled as well.
Remove this flag (hence enabling it be default) so that we catch
any test compatibility issues on pre-submits.
PiperOrigin-RevId: 796618710
Add all CUDA test suites except for a few
lingering failing tests. Most of the failures
are build problems where make fails for some
reason.
Also, have the `MaxSupportedCUDAVersion` method
take an image so we choose the CUDA image per test.
PiperOrigin-RevId: 762043438
Add two more suites of tests to the CUDA 12.8 tests
after testing on two types of GPUs. Also add
a test definition in the release pipeline.
PiperOrigin-RevId: 758393605
Simply running nvidia-smi can expose unsupported IOCTLs
quickly and efficiently. It is also portable. During one
qualification, this picked up some control IOCTLs that the
other workloads did not.
PiperOrigin-RevId: 744866697
This can be used by all test users. Avoids duplicated code. We can handle all
known issues in one place.
There is a Docker bug which causes restore to fail sporadically. See
https://github.com/moby/moby/issues/42900. This has been broken at least since
Docker v19.03.12 (when the issue was reported) and was fixed in v25.0.4. Added
the handling for this issue.
Also got rid of the testutil.Poll() around restore. That can hide gVisor
restore flakiness issues. That was added in 1ec5baeb31 ("Make
checkpoint/restore e2e test less flaky"). The original sleep has been restored.
PiperOrigin-RevId: 734303878
Add image for ARM workloads for cuda-tests and mark tests that work on ARM.
Most tests don't work due to cross-compiling between sbma and aarch64.
However, a few do. Add an image to support them.
While this test does use the P2P capability, it also has fallback code to
do regular memcopies when P2P is not available. So it does not require the
P2P capability.
PiperOrigin-RevId: 714344567
COS can take a day or two to release a new version. While this is happening
versions can appear in gcloud projects, but not on the public site. In this
case, pass the tests so that we don't have failures while release is happening.
PiperOrigin-RevId: 713388040
This test was broken by 8e3b1bb7d4 ("Update CUDA test compatibility to keep
up with added gVisor support.") which requires all
images/gpu/cuda-tests/run_sample.go users to specify "all" driver capabilities.
PiperOrigin-RevId: 713170590
These CUDA tests were initially broken in gVisor but now appear to pass.
The test now also verifies that all capabilities are enabled when running.
PiperOrigin-RevId: 713094806
This refreshes the set of models built into the image to a more diverse
set of models while keeping the same categories covered.
It also adds support for embedding generation and benchmark metrics for
embedding-type models.
The image is also (slightly) smaller which helps make benchmarks not take
forever.
PiperOrigin-RevId: 706594537
Our current check of COS drivers often lags behind COS releases.
This is due to needing to preload GPU docker images onto the
images that run in our CI pipelines.
In addition, COS can be a bit more complex than originally thought
releasing driver versions both across GPU types and release branches.
Thus, this test searches the latest COS images on each family for
new drivers. It does this by looking at COS's published release notes
which include a proto of LATEST/DEFAULT drivers selected for each device.
This will flag new versions faster with more coverage than our
CI pipeline currently. Due to this not actually needing a GPU
to run, this can run on any VM.
PiperOrigin-RevId: 693736100
The presubmit pipeline does not exercise these, so commit
ab5ca5f42d broke them.
This change disables the sniffer on all the non-smoke tests to unbreak
the release pipeline. I will then send another change to re-enable them
on tests where the sniffer works fine after manual testing.
Updates #10885
PiperOrigin-RevId: 673555026
This wraps all GPU tests' command line with the nvproxy ioctl sniffer.
This has multiple functions:
- Verifying that the application does not call ioctls unsupported by
nvproxy. This is controlled by a `AllowIncompatibleIoctl` option, which
is initially set to `true` in all tests to mirror current behavior, but
should be flipped as we verify that they do not call unsupported ioctls.
- Verifying that the sniffer itself works transparently for a wide range
of applications.
- Later down the line, enforcing that the application only calls ioctls
that are part of GPU capabilities that it has a need for. This is
controlled by a capability string which is currently only used to set
the `NVIDIA_DRIVER_CAPABILITIES` environment variable.
Updates issue #10856
PiperOrigin-RevId: 672714520
