Files
+10 f68857e9ff [release-1.5] Cherry-pick commits for v1.5.1 (#4214)
* [Bug] Sidecar mode shouldn't restart head pod when head pod is deleted (#4141)

* [Bug] Sidecar mode shouldn't restart head pod when head pod is deleted

Signed-off-by: 400Ping <[email protected]>

* [Fix] Fix e2e error

Signed-off-by: 400Ping <[email protected]>

* [Fix] fix according to rueian's comment

Signed-off-by: 400Ping <[email protected]>

* [Chore] fix ci error

Signed-off-by: 400Ping <[email protected]>

* Update ray-operator/controllers/ray/raycluster_controller.go

Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>
Signed-off-by: Ping <[email protected]>

* Update ray-operator/controllers/ray/rayjob_controller.go

Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>
Signed-off-by: Ping <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* Trigger CI

Signed-off-by: Future-Outlier <[email protected]>

---------

Signed-off-by: 400Ping <[email protected]>
Signed-off-by: Ping <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>

* fix: dashboard build for kuberay 1.5.0 (#4161)

Signed-off-by: Future-Outlier <[email protected]>

* [Feature Enhancement] Set ordered replica index label to support multi-slice (#4163)

* [Feature Enhancement] Set ordered replica index label to support multi-slice

Signed-off-by: Ryan O'Leary <[email protected]>

* rename replica-id -> replica-name

Signed-off-by: Ryan O'Leary <[email protected]>

* Separate replica index feature gate logic

Signed-off-by: Ryan O'Leary <[email protected]>

* remove index arg in createWorkerPod

Signed-off-by: Ryan O'Leary <[email protected]>

---------

Signed-off-by: Ryan O'Leary <[email protected]>

* update stale feature gate comments (#4174)

Signed-off-by: Andrew Sy Kim <[email protected]>

* [RayCluster] Add more context why we don't recreate head Pod for RayJob (#4175)

Signed-off-by: Kai-Hsun Chen <[email protected]>

* feature: Remove empty resource list initialization. (#4168)

Fixes #4142.

* [Dockerfile] [KubeRay Dashboard]: Fix Dockerfile warnings (ENV format, CMD JSON args) (#4167)

* [#4166] improvement: Fix Dockerfile warnings (ENV format, CMD JSON args)

* extract the hostname from CMD

Signed-off-by: Neo Chien <[email protected]>

---------

Signed-off-by: Neo Chien <[email protected]>
Co-authored-by: cchung100m <[email protected]>

* [Fix] Resolve int32 overflow by having the calculation in int64 and c… (#4158)

* [Fix] Resolve int32 overflow by having the calculation in int64 and cap it if the count is over math.MaxInt32

Signed-off-by: justinyeh1995 <[email protected]>

* [Test] Add unit tests for CalculateReadyReplicas

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] Add a nosec comment to pass the Lint (pre-commit) test

Signed-off-by: justinyeh1995 <[email protected]>

* [Refactor] Add CapInt64ToInt32 to replace #nosec directives

Signed-off-by: justinyeh1995 <[email protected]>

* [Refactor] Rename function to SafeInt64ToInt32 and add a underflowing prevention (it also help pass the lint test)

Signed-off-by: justinyeh1995 <[email protected]>

* [Refactor] Remove the early return as SafeInt64ToInt32 handles the int32 overflow and underflow checking.

Signed-off-by: justinyeh1995 <[email protected]>

---------

Signed-off-by: justinyeh1995 <[email protected]>

* Add RayService incremental upgrade sample for guide (#4164)

Signed-off-by: Ryan O'Leary <[email protected]>

* Edit RayCluster example config for label selectors (#4151)

Signed-off-by: Ryan O'Leary <[email protected]>

* [RayJob] update light weight submitter image from quay.io (#4181)

Signed-off-by: Future-Outlier <[email protected]>

* [flaky] RayJob fails when head Pod is deleted when job is running (#4182)

Signed-off-by: Future-Outlier <[email protected]>

* [CI] Pin Docker api version to avoid API version mismatch (#4188)

Signed-off-by: win5923 <[email protected]>

* Make replicas configurable for kuberay-operator #4180 (#4195)

* Make replicas configurable for kuberay-operator #4180

* Make replicas configurable for kuberay-operator #4180

* [Fix] rayjob update raycluster status (#4192)

* feat: check if raycluster status update in rayjob

* test: e2e test to check the rayjob raycluster status update

* fix: dashboard http client tests discovered and passing (#4173)

Signed-off-by: alimaazamat <[email protected]>

* [RayJob] Lift cluster status while initializing (#4191)

Signed-off-by: Spencer Peterson <[email protected]>

* [RayJob] Remove updateJobStatus call (#4198)

Fast follow to #4191

Signed-off-by: Spencer Peterson <[email protected]>

* Add support for Ray token auth (#4179)

* Add support for Ray token auth

Signed-off-by: Andrew Sy Kim <[email protected]>

* add e2e test for Ray cluster auth

Signed-off-by: Andrew Sy Kim <[email protected]>

* address nits from Ruiean

Signed-off-by: Andrew Sy Kim <[email protected]>

* update RAY_auth_mode -> RAY_AUTH_MODE

Signed-off-by: Andrew Sy Kim <[email protected]>

* configure auth for Ray autoscaler

Signed-off-by: Andrew Sy Kim <[email protected]>

---------

Signed-off-by: Andrew Sy Kim <[email protected]>

* Bump js-yaml from 4.1.0 to 4.1.1 in /dashboard (#4194)

Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 4.1.0 to 4.1.1.
- [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodeca/js-yaml/compare/4.1.0...4.1.1)

---
updated-dependencies:
- dependency-name: js-yaml
  dependency-version: 4.1.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update minimum Ray version required for token authentication to 2.52.0 (#4201)

* update minimum Ray version required for token authentication to 2.52.0

Signed-off-by: Andrew Sy Kim <[email protected]>

* update RayCluster auth e2e test to use Ray v2.52

Signed-off-by: Andrew Sy Kim <[email protected]>

---------

Signed-off-by: Andrew Sy Kim <[email protected]>

* add samples for RayCluster token auth (#4200)

Signed-off-by: Andrew Sy Kim <[email protected]>

* update (#4208)

Signed-off-by: Future-Outlier <[email protected]>

* [RayJob] Add token authentication support for All mode (#4210)

* dashboard client authentication support

Signed-off-by: Future-Outlier <[email protected]>

* support rayjob

Signed-off-by: Future-Outlier <[email protected]>

* update to fix api serverr err

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* updarte

Signed-off-by: Future-Outlier <[email protected]>

* Rayjob sidecar mode auth token mode support

Signed-off-by: Future-Outlier <[email protected]>

* RayJob support k8s job mode

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* Address Andrew's advice

Signed-off-by: Future-Outlier <[email protected]>

* add todo x-ray-authorization comments

Signed-off-by: Future-Outlier <[email protected]>

---------

Signed-off-by: Future-Outlier <[email protected]>

* [RayCluster] Enable Secret informer watch/list and remove unused RBAC verbs (#4202)

* Add authentication secret reconciliation support

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* fix flaky test

Signed-off-by: Future-Outlier <[email protected]>

* remove test fix

Signed-off-by: Rueian <[email protected]>

---------

Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Rueian <[email protected]>
Co-authored-by: Rueian <[email protected]>

* [APIServer][Docs] Add user guide for retry behavior & configuration (#4144)

* [Docs] Add the draft description about feature intro, configurations, and usecases

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] Update the retry walk-through

Signed-off-by: justinyeh1995 <[email protected]>

* [Doc] rewrite the first 2 sections

Signed-off-by: justinyeh1995 <[email protected]>

* [Doc] Revise documentation wording and add Observing Retry Behavior section

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] fix linting issue by running pre-commit run berfore commiting

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] fix linting errors in the Markdown linting

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] Clean up the math equation

Signed-off-by: justinyeh1995 <[email protected]>

* Update the math formula of Backoff calculation.

Co-authored-by: Nary Yeh <[email protected]>
Signed-off-by: JustinYeh <[email protected]>

* [Fix] Explicitly mentioned exponential backoff and removed the customization parts

Signed-off-by: justinyeh1995 <[email protected]>

* [Docs] Clarify naming by replacing “APIServer” with “KubeRay APIServer”

Co-authored-by: Cheng-Yeh Chung <[email protected]>
Signed-off-by: JustinYeh <[email protected]>

* [Docs] Rename retry-configuration.md to retry-behavior.md for accuracy

Signed-off-by: justinyeh1995 <[email protected]>

* Update Title to KubeRay APIServer Retry Behavior

Co-authored-by: Cheng-Yeh Chung <[email protected]>
Signed-off-by: JustinYeh <[email protected]>

* [Docs] Add a note about the limitation of retry configuration

Signed-off-by: justinyeh1995 <[email protected]>

---------

Signed-off-by: justinyeh1995 <[email protected]>
Signed-off-by: JustinYeh <[email protected]>
Co-authored-by: Nary Yeh <[email protected]>
Co-authored-by: Cheng-Yeh Chung <[email protected]>

* Support X-Ray-Authorization fallback header for accepting auth token via proxy (#4213)

* Support X-Ray-Authorization fallback header for accepting auth token in dashboard

Signed-off-by: Future-Outlier <[email protected]>

* remove todo comment

Signed-off-by: Future-Outlier <[email protected]>

---------

Signed-off-by: Future-Outlier <[email protected]>

* [RayCluster] make auth token secret name consistency (#4216)

Signed-off-by: fscnick <[email protected]>

* [RayCluster] Status includes head containter status message (#4196)

* [RayCluster] Status includes head containter status message

Signed-off-by: Spencer Peterson <[email protected]>

* lint

Signed-off-by: Spencer Peterson <[email protected]>

* [RayCluster] Containers not ready status reflects structured reason

Signed-off-by: Spencer Peterson <[email protected]>

* nit

Signed-off-by: Spencer Peterson <[email protected]>

---------

Signed-off-by: Spencer Peterson <[email protected]>

* Remove erroneous  call in applyServeTargetCapacity (#4212)

Signed-off-by: Ryan O'Leary <[email protected]>

* [RayJob] Add token authentication support for light weight job submitter (#4215)

* [RayJob] light weight job submitter auth token support

Signed-off-by: Future-Outlier <[email protected]>

* X-Ray-Authorization

Signed-off-by: Rueian <[email protected]>

---------

Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Rueian <[email protected]>
Co-authored-by: Rueian <[email protected]>

* feat: kubectl ray get token command (#4218)

* feat: kubectl ray get token command

Signed-off-by: Rueian <[email protected]>

* Update kubectl-plugin/pkg/cmd/get/get_token_test.go

Co-authored-by: Copilot <[email protected]>
Signed-off-by: Rueian <[email protected]>

* Update kubectl-plugin/pkg/cmd/get/get_token.go

Co-authored-by: Copilot <[email protected]>
Signed-off-by: Rueian <[email protected]>

* make sure the raycluster exists before getting the secret

Signed-off-by: Rueian <[email protected]>

* better ux

Signed-off-by: Rueian <[email protected]>

* Update kubectl-plugin/pkg/cmd/get/get_token.go

Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>
Signed-off-by: Rueian <[email protected]>

---------

Signed-off-by: Rueian <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>

---------

Signed-off-by: 400Ping <[email protected]>
Signed-off-by: Ping <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Ryan O'Leary <[email protected]>
Signed-off-by: Andrew Sy Kim <[email protected]>
Signed-off-by: Kai-Hsun Chen <[email protected]>
Signed-off-by: Neo Chien <[email protected]>
Signed-off-by: justinyeh1995 <[email protected]>
Signed-off-by: win5923 <[email protected]>
Signed-off-by: alimaazamat <[email protected]>
Signed-off-by: Spencer Peterson <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Rueian <[email protected]>
Signed-off-by: JustinYeh <[email protected]>
Signed-off-by: fscnick <[email protected]>
Co-authored-by: Ping <[email protected]>
Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>
Co-authored-by: Ryan O'Leary <[email protected]>
Co-authored-by: Kai-Hsun Chen <[email protected]>
Co-authored-by: Kavish <[email protected]>
Co-authored-by: Neo Chien <[email protected]>
Co-authored-by: cchung100m <[email protected]>
Co-authored-by: JustinYeh <[email protected]>
Co-authored-by: Jun-Hao Wan <[email protected]>
Co-authored-by: Divyam Raj <[email protected]>
Co-authored-by: Nary Yeh <[email protected]>
Co-authored-by: Alima Azamat <[email protected]>
Co-authored-by: Spencer Peterson <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Rueian <[email protected]>
Co-authored-by: Cheng-Yeh Chung <[email protected]>
Co-authored-by: fscnick <[email protected]>
Co-authored-by: Copilot <[email protected]>
2025-11-21 10:18:05 -05:00

1885 lines
53 KiB
Go

package utils
import (
"fmt"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/utils/ptr"
rayv1 "github.com/ray-project/kuberay/ray-operator/apis/ray/v1"
"github.com/ray-project/kuberay/ray-operator/pkg/features"
)
func TestValidateRayClusterStatus(t *testing.T) {
tests := []struct {
name string
conditions []metav1.Condition
expectError bool
}{
{
name: "Both suspending and suspended are true",
conditions: []metav1.Condition{
{
Type: string(rayv1.RayClusterSuspending),
Status: metav1.ConditionTrue,
},
{
Type: string(rayv1.RayClusterSuspended),
Status: metav1.ConditionTrue,
},
},
expectError: true,
},
{
name: "Only suspending is true",
conditions: []metav1.Condition{
{
Type: string(rayv1.RayClusterSuspending),
Status: metav1.ConditionTrue,
},
{
Type: string(rayv1.RayClusterSuspended),
Status: metav1.ConditionFalse,
},
},
expectError: false,
},
{
name: "Only suspended is true",
conditions: []metav1.Condition{
{
Type: string(rayv1.RayClusterSuspending),
Status: metav1.ConditionFalse,
},
{
Type: string(rayv1.RayClusterSuspended),
Status: metav1.ConditionTrue,
},
},
expectError: false,
},
{
name: "Both suspending and suspended are false",
conditions: []metav1.Condition{
{
Type: string(rayv1.RayClusterSuspending),
Status: metav1.ConditionFalse,
},
{
Type: string(rayv1.RayClusterSuspended),
Status: metav1.ConditionFalse,
},
},
expectError: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
instance := &rayv1.RayCluster{
Status: rayv1.RayClusterStatus{
Conditions: tt.conditions,
},
}
err := ValidateRayClusterStatus(instance)
if (err != nil) != tt.expectError {
t.Errorf("ValidateRayClusterStatus() error = %v, wantErr %v", err, tt.expectError)
}
})
}
}
func TestValidateRayClusterSpecGcsFaultToleranceOptions(t *testing.T) {
errorMessageBothSet := fmt.Sprintf("%s annotation and GcsFaultToleranceOptions are both set. "+
"Please use only GcsFaultToleranceOptions to configure GCS fault tolerance", RayFTEnabledAnnotationKey)
errorMessageRedisAddressSet := fmt.Sprintf("%s is set which implicitly enables GCS fault tolerance, "+
"but GcsFaultToleranceOptions is not set. Please set GcsFaultToleranceOptions "+
"to enable GCS fault tolerance", RAY_REDIS_ADDRESS)
errorMessageRedisAddressConflict := fmt.Sprintf("cannot set `%s` env var in head Pod when "+
"GcsFaultToleranceOptions is enabled - use GcsFaultToleranceOptions.RedisAddress instead", RAY_REDIS_ADDRESS)
errorMessageExternalStorageNamespaceConflict := fmt.Sprintf("cannot set `%s` annotation when "+
"GcsFaultToleranceOptions is enabled - use GcsFaultToleranceOptions.ExternalStorageNamespace instead", RayExternalStorageNSAnnotationKey)
tests := []struct {
rayStartParams map[string]string
gcsFaultToleranceOptions *rayv1.GcsFaultToleranceOptions
annotations map[string]string
name string
errorMessage string
envVars []corev1.EnvVar
expectError bool
}{
// GcsFaultToleranceOptions and ray.io/ft-enabled should not be both set.
{
name: "ray.io/ft-enabled is set to false and GcsFaultToleranceOptions is set",
annotations: map[string]string{
RayFTEnabledAnnotationKey: "false",
},
gcsFaultToleranceOptions: &rayv1.GcsFaultToleranceOptions{},
expectError: true,
errorMessage: errorMessageBothSet,
},
{
name: "ray.io/ft-enabled is set to true and GcsFaultToleranceOptions is set",
annotations: map[string]string{
RayFTEnabledAnnotationKey: "true",
},
gcsFaultToleranceOptions: &rayv1.GcsFaultToleranceOptions{},
expectError: true,
errorMessage: errorMessageBothSet,
},
{
name: "ray.io/ft-enabled is not set and GcsFaultToleranceOptions is set",
gcsFaultToleranceOptions: &rayv1.GcsFaultToleranceOptions{},
expectError: false,
},
{
name: "ray.io/ft-enabled is not set and GcsFaultToleranceOptions is not set",
gcsFaultToleranceOptions: nil,
expectError: false,
},
// RAY_REDIS_ADDRESS should not be set if KubeRay is not aware that GCS fault tolerance is enabled.
{
name: "ray.io/ft-enabled is set to false and RAY_REDIS_ADDRESS is set",
annotations: map[string]string{
RayFTEnabledAnnotationKey: "false",
},
envVars: []corev1.EnvVar{
{
Name: RAY_REDIS_ADDRESS,
Value: "redis:6379",
},
},
expectError: true,
errorMessage: errorMessageRedisAddressSet,
},
{
name: "gcsFaultToleranceOptions is set and RAY_REDIS_ADDRESS is set",
envVars: []corev1.EnvVar{
{
Name: RAY_REDIS_ADDRESS,
Value: "redis:6379",
},
},
gcsFaultToleranceOptions: &rayv1.GcsFaultToleranceOptions{},
expectError: true,
errorMessage: errorMessageRedisAddressConflict,
},
{
name: "FT is disabled and RAY_REDIS_ADDRESS is set",
envVars: []corev1.EnvVar{
{
Name: RAY_REDIS_ADDRESS,
Value: "redis:6379",
},
},
expectError: true,
errorMessage: errorMessageRedisAddressSet,
},
{
name: "ray.io/ft-enabled is set to true and RAY_REDIS_ADDRESS is set",
annotations: map[string]string{
RayFTEnabledAnnotationKey: "true",
},
envVars: []corev1.EnvVar{
{
Name: RAY_REDIS_ADDRESS,
Value: "redis:6379",
},
},
expectError: false,
},
{
name: "gcsFaultToleranceOptions is set and ray.io/external-storage-namespace is set",
annotations: map[string]string{
RayExternalStorageNSAnnotationKey: "myns",
},
gcsFaultToleranceOptions: &rayv1.GcsFaultToleranceOptions{},
expectError: true,
errorMessage: errorMessageExternalStorageNamespaceConflict,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateRayClusterSpec(&rayv1.RayClusterSpec{
GcsFaultToleranceOptions: tt.gcsFaultToleranceOptions,
HeadGroupSpec: rayv1.HeadGroupSpec{
RayStartParams: tt.rayStartParams,
Template: podTemplateSpec(tt.envVars, nil),
},
}, tt.annotations)
if tt.expectError {
require.Error(t, err)
assert.EqualError(t, err, tt.errorMessage)
} else {
require.NoError(t, err)
}
})
}
}
func TestValidateRayClusterSpecRedisPassword(t *testing.T) {
tests := []struct {
gcsFaultToleranceOptions *rayv1.GcsFaultToleranceOptions
name string
rayStartParams map[string]string
envVars []corev1.EnvVar
expectError bool
}{
{
name: "GcsFaultToleranceOptions is set and `redis-password` is also set in rayStartParams",
gcsFaultToleranceOptions: &rayv1.GcsFaultToleranceOptions{},
rayStartParams: map[string]string{
"redis-password": "password",
},
expectError: true,
},
{
name: "GcsFaultToleranceOptions is set and `REDIS_PASSWORD` env var is also set in the head Pod",
gcsFaultToleranceOptions: &rayv1.GcsFaultToleranceOptions{},
envVars: []corev1.EnvVar{
{
Name: REDIS_PASSWORD,
Value: "password",
},
},
expectError: true,
},
{
name: "GcsFaultToleranceOptions.RedisPassword is set",
gcsFaultToleranceOptions: &rayv1.GcsFaultToleranceOptions{
RedisPassword: &rayv1.RedisCredential{
Value: "password",
},
},
expectError: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
rayCluster := &rayv1.RayCluster{
Spec: rayv1.RayClusterSpec{
GcsFaultToleranceOptions: tt.gcsFaultToleranceOptions,
HeadGroupSpec: rayv1.HeadGroupSpec{
RayStartParams: tt.rayStartParams,
Template: podTemplateSpec(tt.envVars, nil),
},
},
}
err := ValidateRayClusterSpec(&rayCluster.Spec, rayCluster.Annotations)
if tt.expectError {
require.Error(t, err)
} else {
require.NoError(t, err)
}
})
}
}
func TestValidateRayClusterSpecRedisUsername(t *testing.T) {
errorMessageRedisUsername := "cannot set redis username in rayStartParams or environment variables - use GcsFaultToleranceOptions.RedisUsername instead"
tests := []struct {
gcsFaultToleranceOptions *rayv1.GcsFaultToleranceOptions
name string
errorMessage string
rayStartParams map[string]string
envVars []corev1.EnvVar
expectError bool
}{
{
name: "`redis-username` is set in rayStartParams of the Head Pod",
rayStartParams: map[string]string{
"redis-username": "username",
},
expectError: true,
errorMessage: errorMessageRedisUsername,
},
{
name: "`REDIS_USERNAME` env var is set in the Head Pod",
envVars: []corev1.EnvVar{
{
Name: REDIS_USERNAME,
Value: "username",
},
},
expectError: true,
errorMessage: errorMessageRedisUsername,
},
{
name: "GcsFaultToleranceOptions.RedisUsername is set",
gcsFaultToleranceOptions: &rayv1.GcsFaultToleranceOptions{
RedisUsername: &rayv1.RedisCredential{
Value: "username",
},
},
expectError: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
rayCluster := &rayv1.RayCluster{
Spec: rayv1.RayClusterSpec{
GcsFaultToleranceOptions: tt.gcsFaultToleranceOptions,
HeadGroupSpec: rayv1.HeadGroupSpec{
RayStartParams: tt.rayStartParams,
Template: podTemplateSpec(tt.envVars, nil),
},
},
}
err := ValidateRayClusterSpec(&rayCluster.Spec, rayCluster.Annotations)
if tt.expectError {
require.Error(t, err)
assert.EqualError(t, err, tt.errorMessage)
} else {
require.NoError(t, err)
}
})
}
}
func TestValidateRayClusterSpecNames(t *testing.T) {
tests := []struct {
name string
errorMessage string
metadata metav1.ObjectMeta
expectError bool
}{
{
name: "RayCluster name is too long (> MaxRayClusterNameLength characters)",
metadata: metav1.ObjectMeta{
Name: strings.Repeat("a", MaxRayClusterNameLength+1),
},
expectError: true,
errorMessage: fmt.Sprintf("RayCluster name should be no more than %d characters", MaxRayClusterNameLength),
},
{
name: "RayCluster name is ok (== MaxRayClusterNameLength)",
metadata: metav1.ObjectMeta{
Name: strings.Repeat("a", MaxRayClusterNameLength),
},
expectError: false,
},
{
name: "RayCluster name is not a DNS1035 label",
metadata: metav1.ObjectMeta{
Name: strings.Repeat("1", MaxRayClusterNameLength),
},
expectError: true,
errorMessage: "RayCluster name should be a valid DNS1035 label: [a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?')]",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateRayClusterMetadata(tt.metadata)
if tt.expectError {
require.Error(t, err)
assert.EqualError(t, err, tt.errorMessage)
} else {
require.NoError(t, err)
}
})
}
}
func TestValidateRayClusterSpecEmptyContainers(t *testing.T) {
headGroupSpecWithOneContainer := rayv1.HeadGroupSpec{
Template: podTemplateSpec(nil, nil),
}
workerGroupSpecWithOneContainer := rayv1.WorkerGroupSpec{
Template: podTemplateSpec(nil, nil),
}
headGroupSpecWithNoContainers := *headGroupSpecWithOneContainer.DeepCopy()
headGroupSpecWithNoContainers.Template.Spec.Containers = []corev1.Container{}
workerGroupSpecWithNoContainers := *workerGroupSpecWithOneContainer.DeepCopy()
workerGroupSpecWithNoContainers.Template.Spec.Containers = []corev1.Container{}
tests := []struct {
rayCluster *rayv1.RayCluster
name string
errorMessage string
expectError bool
}{
{
name: "headGroupSpec has no containers",
rayCluster: &rayv1.RayCluster{
Spec: rayv1.RayClusterSpec{
HeadGroupSpec: headGroupSpecWithNoContainers,
},
},
expectError: true,
errorMessage: "headGroupSpec should have at least one container",
},
{
name: "workerGroupSpec has no containers",
rayCluster: &rayv1.RayCluster{
Spec: rayv1.RayClusterSpec{
HeadGroupSpec: headGroupSpecWithOneContainer,
WorkerGroupSpecs: []rayv1.WorkerGroupSpec{workerGroupSpecWithNoContainers},
},
},
expectError: true,
errorMessage: "workerGroupSpec should have at least one container",
},
{
name: "valid cluster with containers in both head and worker groups",
rayCluster: &rayv1.RayCluster{
Spec: rayv1.RayClusterSpec{
HeadGroupSpec: headGroupSpecWithOneContainer,
WorkerGroupSpecs: []rayv1.WorkerGroupSpec{workerGroupSpecWithOneContainer},
},
},
expectError: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateRayClusterSpec(&tt.rayCluster.Spec, tt.rayCluster.Annotations)
if tt.expectError {
require.Error(t, err)
assert.EqualError(t, err, tt.errorMessage)
} else {
require.NoError(t, err)
}
})
}
}
func TestValidateRayClusterSpecSuspendingWorkerGroup(t *testing.T) {
headGroupSpec := rayv1.HeadGroupSpec{
Template: podTemplateSpec(nil, nil),
}
workerGroupSpecSuspended := rayv1.WorkerGroupSpec{
GroupName: "worker-group-1",
Template: podTemplateSpec(nil, nil),
}
workerGroupSpecSuspended.Suspend = ptr.To(true)
tests := []struct {
rayCluster *rayv1.RayCluster
name string
errorMessage string
expectError bool
featureGate bool
}{
{
name: "suspend without autoscaler and the feature gate",
rayCluster: &rayv1.RayCluster{
Spec: rayv1.RayClusterSpec{
HeadGroupSpec: headGroupSpec,
WorkerGroupSpecs: []rayv1.WorkerGroupSpec{workerGroupSpecSuspended},
},
},
featureGate: false,
expectError: true,
errorMessage: fmt.Sprintf("worker group %s can be suspended only when the RayJobDeletionPolicy feature gate is enabled", workerGroupSpecSuspended.GroupName),
},
{
name: "suspend without autoscaler",
rayCluster: &rayv1.RayCluster{
Spec: rayv1.RayClusterSpec{
HeadGroupSpec: headGroupSpec,
WorkerGroupSpecs: []rayv1.WorkerGroupSpec{workerGroupSpecSuspended},
},
},
featureGate: true,
expectError: false,
},
{
// TODO (rueian): This can be supported in future Ray. We should check the RayVersion once we know the version.
name: "suspend with autoscaler",
rayCluster: &rayv1.RayCluster{
Spec: rayv1.RayClusterSpec{
HeadGroupSpec: headGroupSpec,
WorkerGroupSpecs: []rayv1.WorkerGroupSpec{workerGroupSpecSuspended},
EnableInTreeAutoscaling: ptr.To(true),
},
},
featureGate: true,
expectError: true,
errorMessage: fmt.Sprintf("worker group %s cannot be suspended with Autoscaler enabled", workerGroupSpecSuspended.GroupName),
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
features.SetFeatureGateDuringTest(t, features.RayJobDeletionPolicy, tt.featureGate)
err := ValidateRayClusterSpec(&tt.rayCluster.Spec, tt.rayCluster.Annotations)
if tt.expectError {
require.Error(t, err)
assert.EqualError(t, err, tt.errorMessage)
} else {
require.NoError(t, err)
}
})
}
}
func podTemplateSpec(envVars []corev1.EnvVar, restartPolicy *corev1.RestartPolicy) corev1.PodTemplateSpec {
spec := corev1.PodTemplateSpec{
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{
Env: envVars,
},
},
},
}
if restartPolicy != nil {
spec.Spec.RestartPolicy = *restartPolicy
}
return spec
}
func TestValidateRayClusterSpecAutoscaler(t *testing.T) {
tests := map[string]struct {
expectedErr string
spec rayv1.RayClusterSpec
}{
"should return error if autoscaler is enabled and any worker group is suspended": {
spec: rayv1.RayClusterSpec{
EnableInTreeAutoscaling: ptr.To(true),
HeadGroupSpec: rayv1.HeadGroupSpec{
Template: podTemplateSpec(nil, nil),
},
WorkerGroupSpecs: []rayv1.WorkerGroupSpec{
{
GroupName: "worker-group-1",
Template: podTemplateSpec(nil, nil),
Suspend: ptr.To(true),
},
},
},
expectedErr: "worker group worker-group-1 cannot be suspended with Autoscaler enabled",
},
fmt.Sprintf("should return error if %s env var is set to '1' when autoscaler is disabled", RAY_ENABLE_AUTOSCALER_V2): {
spec: rayv1.RayClusterSpec{
EnableInTreeAutoscaling: ptr.To(false),
HeadGroupSpec: rayv1.HeadGroupSpec{
Template: podTemplateSpec([]corev1.EnvVar{
{
Name: RAY_ENABLE_AUTOSCALER_V2,
Value: "1",
},
}, nil),
},
},
expectedErr: fmt.Sprintf("environment variable %s cannot be set to '1' when enableInTreeAutoscaling is false. Please set enableInTreeAutoscaling: true to use autoscaler v2", RAY_ENABLE_AUTOSCALER_V2),
},
fmt.Sprintf("should return error if %s env var is set to 'true' when autoscaler is disabled", RAY_ENABLE_AUTOSCALER_V2): {
spec: rayv1.RayClusterSpec{
EnableInTreeAutoscaling: ptr.To(false),
HeadGroupSpec: rayv1.HeadGroupSpec{
Template: podTemplateSpec([]corev1.EnvVar{
{
Name: RAY_ENABLE_AUTOSCALER_V2,
Value: "true",
},
}, nil),
},
},
expectedErr: fmt.Sprintf("environment variable %s cannot be set to 'true' when enableInTreeAutoscaling is false. Please set enableInTreeAutoscaling: true to use autoscaler v2", RAY_ENABLE_AUTOSCALER_V2),
},
fmt.Sprintf("should return error if autoscaler v2 is enabled and head Pod has env var %s", RAY_ENABLE_AUTOSCALER_V2): {
spec: rayv1.RayClusterSpec{
EnableInTreeAutoscaling: ptr.To(true),
AutoscalerOptions: &rayv1.AutoscalerOptions{
Version: ptr.To(rayv1.AutoscalerVersionV2),
},
HeadGroupSpec: rayv1.HeadGroupSpec{
Template: podTemplateSpec([]corev1.EnvVar{
{
Name: RAY_ENABLE_AUTOSCALER_V2,
Value: "true",
},
}, nil),
},
},
expectedErr: fmt.Sprintf("both .spec.autoscalerOptions.version and head Pod env var %s are set, please only use the former", RAY_ENABLE_AUTOSCALER_V2),
},
"should return error if autoscaler v2 is enabled and head Pod has a restartPolicy other than Never or unset": {
spec: rayv1.RayClusterSpec{
EnableInTreeAutoscaling: ptr.To(true),
AutoscalerOptions: &rayv1.AutoscalerOptions{
Version: ptr.To(rayv1.AutoscalerVersionV2),
},
HeadGroupSpec: rayv1.HeadGroupSpec{
Template: podTemplateSpec(nil, ptr.To(corev1.RestartPolicyAlways)),
},
},
expectedErr: "restartPolicy for head Pod should be Never or unset when using autoscaler V2",
},
"should return error if autoscaler v2 is enabled and a worker group has a restartPolicy other than Never or unset": {
spec: rayv1.RayClusterSpec{
EnableInTreeAutoscaling: ptr.To(true),
AutoscalerOptions: &rayv1.AutoscalerOptions{
Version: ptr.To(rayv1.AutoscalerVersionV2),
},
HeadGroupSpec: rayv1.HeadGroupSpec{
Template: podTemplateSpec(nil, ptr.To(corev1.RestartPolicyNever)),
},
WorkerGroupSpecs: []rayv1.WorkerGroupSpec{
{
GroupName: "worker-group-1",
Template: podTemplateSpec(nil, ptr.To(corev1.RestartPolicyNever)),
},
{
GroupName: "worker-group-2",
Template: podTemplateSpec(nil, ptr.To(corev1.RestartPolicyAlways)),
},
},
},
expectedErr: "restartPolicy for worker group worker-group-2 should be Never or unset when using autoscaler V2",
},
"should not return error if autoscaler configs are valid": {
spec: rayv1.RayClusterSpec{
EnableInTreeAutoscaling: ptr.To(true),
AutoscalerOptions: &rayv1.AutoscalerOptions{
Version: ptr.To(rayv1.AutoscalerVersionV2),
},
HeadGroupSpec: rayv1.HeadGroupSpec{
Template: podTemplateSpec(nil, nil),
},
WorkerGroupSpecs: []rayv1.WorkerGroupSpec{
{
GroupName: "worker-group-1",
Template: podTemplateSpec(nil, nil),
},
{
GroupName: "worker-group-2",
Template: podTemplateSpec(nil, ptr.To(corev1.RestartPolicyNever)),
},
},
},
},
}
features.SetFeatureGateDuringTest(t, features.RayJobDeletionPolicy, true)
defer func() {
features.SetFeatureGateDuringTest(t, features.RayJobDeletionPolicy, false)
}()
for name, tc := range tests {
t.Run(name, func(t *testing.T) {
err := ValidateRayClusterSpec(&tc.spec, map[string]string{})
if tc.expectedErr != "" {
require.Error(t, err)
require.EqualError(t, err, tc.expectedErr)
} else {
require.NoError(t, err)
}
})
}
}
func TestValidateRayClusterSpec_Resources(t *testing.T) {
// Util function to create a RayCluster spec.
createSpec := func() rayv1.RayClusterSpec {
return rayv1.RayClusterSpec{
HeadGroupSpec: rayv1.HeadGroupSpec{
Template: podTemplateSpec(nil, nil),
},
WorkerGroupSpecs: []rayv1.WorkerGroupSpec{
{
GroupName: "worker-group",
Template: podTemplateSpec(nil, nil),
},
},
}
}
tests := []struct {
name string
errorMessage string
spec rayv1.RayClusterSpec
expectError bool
}{
{
name: "Invalid: Head group has resources in both rayStartParams and top-level Resources",
spec: func() rayv1.RayClusterSpec {
s := createSpec()
s.HeadGroupSpec.RayStartParams = map[string]string{"num-cpus": "1"}
s.HeadGroupSpec.Resources = map[string]string{"CPU": "1"}
return s
}(),
expectError: true,
errorMessage: "resource fields should not be set in both rayStartParams and Resources for Head group; please use only one",
},
{
name: "Invalid: Worker group has resources in both rayStartParams and .Resources",
spec: func() rayv1.RayClusterSpec {
s := createSpec()
s.WorkerGroupSpecs[0].RayStartParams = map[string]string{"num-gpus": "1"}
s.WorkerGroupSpecs[0].Resources = map[string]string{"GPU": "1"}
return s
}(),
expectError: true,
errorMessage: "resource fields should not be set in both rayStartParams and Resources for worker-group group; please use only one",
},
{
name: "Valid: Only rayStartParams resources are set for head",
spec: func() rayv1.RayClusterSpec {
s := createSpec()
s.HeadGroupSpec.RayStartParams = map[string]string{
"num-cpus": "2",
"memory": "4G",
"resources": "{\"TPU\": \"8\"}",
}
return s
}(),
expectError: false,
},
{
name: "Valid: Only .Resources field is set for worker",
spec: func() rayv1.RayClusterSpec {
s := createSpec()
s.WorkerGroupSpecs[0].Resources = map[string]string{"CPU": "2", "memory": "4G", "TPU": "8"}
return s
}(),
expectError: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateRayClusterSpec(&tt.spec, nil)
if tt.expectError {
require.Error(t, err)
assert.EqualError(t, err, tt.errorMessage)
} else {
require.NoError(t, err)
}
})
}
}
func TestValidateRayClusterSpec_Labels(t *testing.T) {
// Util function to create a RayCluster spec.
createSpec := func() rayv1.RayClusterSpec {
return rayv1.RayClusterSpec{
HeadGroupSpec: rayv1.HeadGroupSpec{
Template: podTemplateSpec(nil, nil),
},
WorkerGroupSpecs: []rayv1.WorkerGroupSpec{
{
GroupName: "worker-group",
Template: podTemplateSpec(nil, nil),
},
},
}
}
tests := []struct {
name string
errorMessage string
spec rayv1.RayClusterSpec
expectError bool
}{
{
name: "Invalid: Head group has 'labels' in rayStartParams",
spec: func() rayv1.RayClusterSpec {
s := createSpec()
s.HeadGroupSpec.RayStartParams = map[string]string{"labels": "ray.io/node-group=worker-group-1"}
return s
}(),
expectError: true,
errorMessage: "rayStartParams['labels'] is not supported for Head group; please use the top-level Labels field instead",
},
{
name: "Invalid: Worker group has 'labels' in rayStartParams",
spec: func() rayv1.RayClusterSpec {
s := createSpec()
s.WorkerGroupSpecs[0].RayStartParams = map[string]string{"labels": "ray.io/node-group=worker-group-1"}
return s
}(),
expectError: true,
errorMessage: "rayStartParams['labels'] is not supported for worker-group group; please use the top-level Labels field instead",
},
{
name: "Valid: Only .Labels field is set",
spec: func() rayv1.RayClusterSpec {
s := createSpec()
s.HeadGroupSpec.Labels = map[string]string{"ray.io/market-type": "on-demand"}
s.WorkerGroupSpecs[0].Labels = map[string]string{"ray.io/accelerator-type": "TPU-V6E"}
return s
}(),
expectError: false,
},
{
name: "Invalid: Label key does not follow Kubernetes syntax",
spec: func() rayv1.RayClusterSpec {
s := createSpec()
s.WorkerGroupSpecs[0].Labels = map[string]string{"invalid_key!": "value"}
return s
}(),
expectError: true,
errorMessage: "invalid label key for worker-group group: 'invalid_key!', error: name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character",
},
{
name: "Invalid: Label value does not follow Kubernetes syntax",
spec: func() rayv1.RayClusterSpec {
s := createSpec()
s.HeadGroupSpec.Labels = map[string]string{"valid-key": "invalid/value"}
return s
}(),
expectError: true,
errorMessage: "invalid label value for key 'valid-key' in Head group: 'invalid/value', error: a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateRayClusterSpec(&tt.spec, nil)
if tt.expectError {
require.Error(t, err)
assert.Contains(t, err.Error(), tt.errorMessage)
} else {
require.NoError(t, err)
}
})
}
}
func TestValidateRayClusterSpecRayVersionForAuth(t *testing.T) {
tests := []struct {
name string
rayVersion string
errorMessage string
expectError bool
}{
{
name: "Valid Ray version 2.52.0",
rayVersion: "2.52.0",
expectError: false,
},
{
name: "Valid Ray version 3.0.0",
rayVersion: "3.0.0",
expectError: false,
},
{
name: "Invalid Ray version 2.50.0",
rayVersion: "2.50.0",
expectError: true,
errorMessage: "authOptions.mode is 'token' but minimum Ray version is 2.52.0, got 2.50.0",
},
{
name: "Invalid Ray version format",
rayVersion: "invalid-version",
expectError: true,
errorMessage: "authOptions.mode is 'token' but RayVersion format is invalid: invalid-version, could not parse \"invalid-version\" as version",
},
{
name: "Empty Ray version",
rayVersion: "",
expectError: true,
errorMessage: "authOptions.mode is 'token' but RayVersion was not specified. Ray version 2.52.0 or later is required",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
spec := &rayv1.RayClusterSpec{
RayVersion: tt.rayVersion,
HeadGroupSpec: rayv1.HeadGroupSpec{
Template: podTemplateSpec(nil, nil),
},
AuthOptions: &rayv1.AuthOptions{
Mode: rayv1.AuthModeToken,
},
}
err := ValidateRayClusterSpec(spec, nil)
if tt.expectError {
require.Error(t, err)
assert.Contains(t, err.Error(), tt.errorMessage)
} else {
require.NoError(t, err)
}
})
}
}
func TestValidateRayJobStatus(t *testing.T) {
tests := []struct {
name string
jobSpec rayv1.RayJobSpec
jobStatus rayv1.RayJobStatus
expectError bool
}{
{
name: "JobDeploymentStatus is Waiting and SubmissionMode is not InteractiveMode",
jobStatus: rayv1.RayJobStatus{
JobDeploymentStatus: rayv1.JobDeploymentStatusWaiting,
},
jobSpec: rayv1.RayJobSpec{
SubmissionMode: rayv1.K8sJobMode,
},
expectError: true,
},
{
name: "JobDeploymentStatus is Waiting and SubmissionMode is InteractiveMode",
jobStatus: rayv1.RayJobStatus{
JobDeploymentStatus: rayv1.JobDeploymentStatusWaiting,
},
jobSpec: rayv1.RayJobSpec{
SubmissionMode: rayv1.InteractiveMode,
},
expectError: false,
},
{
name: "JobDeploymentStatus is not Waiting and SubmissionMode is not InteractiveMode",
jobStatus: rayv1.RayJobStatus{
JobDeploymentStatus: rayv1.JobDeploymentStatusRunning,
},
jobSpec: rayv1.RayJobSpec{
SubmissionMode: rayv1.HTTPMode,
},
expectError: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
rayJob := &rayv1.RayJob{
Status: tt.jobStatus,
Spec: tt.jobSpec,
}
err := ValidateRayJobStatus(rayJob)
if (err != nil) != tt.expectError {
t.Errorf("ValidateRayJobStatus() error = %v, wantErr %v", err, tt.expectError)
}
})
}
}
func TestValidateRayJobSpec(t *testing.T) {
tests := []struct {
name string
spec rayv1.RayJobSpec
expectError bool
}{
{
name: "one of RayClusterSpec or ClusterSelector must be set",
spec: rayv1.RayJobSpec{},
expectError: true,
},
{
name: "a RayJob with shutdownAfterJobFinishes set to false is not allowed to be suspended",
spec: rayv1.RayJobSpec{
Suspend: true,
ShutdownAfterJobFinishes: false,
},
expectError: true,
},
{
name: "valid RayJob",
spec: rayv1.RayJobSpec{
Suspend: true,
ShutdownAfterJobFinishes: true,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: false,
},
{
name: "the ClusterSelector mode doesn't support the suspend operation",
spec: rayv1.RayJobSpec{
Suspend: true,
ShutdownAfterJobFinishes: true,
ClusterSelector: map[string]string{
"key": "value",
},
},
expectError: true,
},
{
name: "failed to unmarshal RuntimeEnvYAML",
spec: rayv1.RayJobSpec{
RuntimeEnvYAML: "invalid_yaml_str",
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "backoffLimit must be a positive integer",
spec: rayv1.RayJobSpec{
BackoffLimit: ptr.To[int32](-1),
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "RayJobDeletionPolicy feature gate must be enabled to use the DeletionStrategy feature",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
OnSuccess: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteCluster),
},
OnFailure: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteCluster),
},
},
ShutdownAfterJobFinishes: true,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "BackoffLimit is incompatible with InteractiveMode",
spec: rayv1.RayJobSpec{
BackoffLimit: ptr.To[int32](1),
SubmissionMode: rayv1.InteractiveMode,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "BackoffLimit is 0 and SubmissionMode is InteractiveMode",
spec: rayv1.RayJobSpec{
BackoffLimit: ptr.To[int32](0),
SubmissionMode: rayv1.InteractiveMode,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: false,
},
{
name: "BackoffLimit is nil and SubmissionMode is InteractiveMode",
spec: rayv1.RayJobSpec{
BackoffLimit: nil,
SubmissionMode: rayv1.InteractiveMode,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: false,
},
{
name: "ShutdownAfterJobFinishes is false and TTLSecondsAfterFinished is not zero",
spec: rayv1.RayJobSpec{
ShutdownAfterJobFinishes: false,
TTLSecondsAfterFinished: 5,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "ShutdownAfterJobFinishes is true and TTLSecondsAfterFinished is not zero",
spec: rayv1.RayJobSpec{
ShutdownAfterJobFinishes: true,
TTLSecondsAfterFinished: 5,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: false,
},
{
name: "ShutdownAfterJobFinishes is true and TTLSecondsAfterFinished is negative",
spec: rayv1.RayJobSpec{
ShutdownAfterJobFinishes: true,
TTLSecondsAfterFinished: -5,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "SidecarMode",
spec: rayv1.RayJobSpec{
SubmissionMode: rayv1.SidecarMode,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: false,
},
{
name: "SidecarMode doesn't support SubmitterPodTemplate",
spec: rayv1.RayJobSpec{
SubmissionMode: rayv1.SidecarMode,
SubmitterPodTemplate: &corev1.PodTemplateSpec{
Spec: corev1.PodSpec{},
},
},
expectError: true,
},
{
name: "SidecarMode doesn't support SubmitterConfig",
spec: rayv1.RayJobSpec{
SubmissionMode: rayv1.SidecarMode,
SubmitterConfig: &rayv1.SubmitterConfig{
BackoffLimit: ptr.To[int32](1),
},
},
expectError: true,
},
{
name: "SidecarMode RayCluster head pod should only be Never or unset",
spec: rayv1.RayJobSpec{
SubmissionMode: rayv1.SidecarMode,
RayClusterSpec: &rayv1.RayClusterSpec{
HeadGroupSpec: rayv1.HeadGroupSpec{
Template: podTemplateSpec(nil, ptr.To(corev1.RestartPolicyAlways)),
},
},
},
expectError: true,
},
{
name: "SidecarMode doesn't support ClusterSelector",
spec: rayv1.RayJobSpec{
SubmissionMode: rayv1.SidecarMode,
ClusterSelector: map[string]string{"ray.io/cluster": "ray-cluster"},
},
expectError: true,
},
{
name: "failed to get cluster name in ClusterSelector map",
spec: rayv1.RayJobSpec{
ClusterSelector: map[string]string{},
},
expectError: true,
},
{
name: "cluster name in ClusterSelector is empty",
spec: rayv1.RayJobSpec{
ClusterSelector: map[string]string{"ray.io/cluster": ""},
},
expectError: true,
},
{
name: "cluster name in ClusterSelector is not empty",
spec: rayv1.RayJobSpec{
ClusterSelector: map[string]string{"ray.io/cluster": "ray-cluster"},
},
expectError: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateRayJobSpec(&rayv1.RayJob{
Spec: tt.spec,
})
if tt.expectError {
require.Error(t, err, tt.name)
} else {
require.NoError(t, err, tt.name)
}
})
}
}
func TestValidateRayJobSpecWithFeatureGate(t *testing.T) {
headGroupSpecWithOneContainer := rayv1.HeadGroupSpec{
Template: podTemplateSpec(nil, nil),
}
tests := []struct {
name string
spec rayv1.RayJobSpec
expectError bool
}{
// Legacy DeletionStrategy tests
{
name: "the ClusterSelector mode doesn't support DeletionStrategy=DeleteCluster",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
OnSuccess: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteCluster),
},
OnFailure: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteCluster),
},
}, ClusterSelector: map[string]string{"key": "value"},
},
expectError: true,
},
{
name: "the ClusterSelector mode doesn't support DeletionStrategy=DeleteWorkers",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
OnSuccess: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteWorkers),
},
OnFailure: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteWorkers),
},
}, ClusterSelector: map[string]string{"key": "value"},
},
expectError: true,
},
{
name: "DeletionStrategy=DeleteWorkers currently does not support RayCluster with autoscaling enabled",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
OnSuccess: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteWorkers),
},
OnFailure: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteWorkers),
},
}, RayClusterSpec: &rayv1.RayClusterSpec{
EnableInTreeAutoscaling: ptr.To(true),
HeadGroupSpec: headGroupSpecWithOneContainer,
},
},
expectError: true,
},
{
name: "valid RayJob with DeletionStrategy=DeleteCluster",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
OnSuccess: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteCluster),
},
OnFailure: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteCluster),
},
}, ShutdownAfterJobFinishes: true,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: false,
},
{
name: "valid RayJob without DeletionStrategy",
spec: rayv1.RayJobSpec{
DeletionStrategy: nil,
ShutdownAfterJobFinishes: true,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: false,
},
{
name: "shutdownAfterJobFinshes is set to 'true' while deletion policy is 'DeleteNone'",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
OnSuccess: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteNone),
},
OnFailure: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteNone),
},
}, ShutdownAfterJobFinishes: true,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "OnSuccess unset",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
OnFailure: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteNone),
},
}, ShutdownAfterJobFinishes: true,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "OnSuccess.DeletionPolicyType unset",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
OnFailure: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteNone),
},
}, ShutdownAfterJobFinishes: true,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "OnFailure unset",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
OnSuccess: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteNone),
},
}, ShutdownAfterJobFinishes: true,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "OnFailure.DeletionPolicyType unset",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
OnSuccess: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteNone),
},
OnFailure: &rayv1.DeletionPolicy{},
}, ShutdownAfterJobFinishes: true,
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "headGroupSpec should have at least one container",
spec: rayv1.RayJobSpec{
RayClusterSpec: &rayv1.RayClusterSpec{
HeadGroupSpec: rayv1.HeadGroupSpec{},
},
},
expectError: true,
},
// New Deletion Rules tests
{
name: "valid deletionRules",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
DeletionRules: []rayv1.DeletionRule{
{
Policy: rayv1.DeleteSelf,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: 10,
},
},
},
},
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: false,
},
{
name: "deletionRules and ShutdownAfterJobFinishes both set",
spec: rayv1.RayJobSpec{
ShutdownAfterJobFinishes: true,
DeletionStrategy: &rayv1.DeletionStrategy{
DeletionRules: []rayv1.DeletionRule{
{
Policy: rayv1.DeleteSelf,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: 10,
},
},
},
},
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "deletionRules and legacy onSuccess both set",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
OnSuccess: &rayv1.DeletionPolicy{
Policy: ptr.To(rayv1.DeleteCluster),
},
DeletionRules: []rayv1.DeletionRule{
{
Policy: rayv1.DeleteSelf,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: 10,
},
},
},
},
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "nil DeletionStrategy",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{},
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "empty DeletionStrategy",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
DeletionRules: []rayv1.DeletionRule{},
},
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "duplicate rule in deletionRules",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
DeletionRules: []rayv1.DeletionRule{
{
Policy: rayv1.DeleteSelf,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: 10,
},
},
{
Policy: rayv1.DeleteSelf,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: 20,
},
},
},
},
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "negative TTLSeconds in deletionRules",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
DeletionRules: []rayv1.DeletionRule{
{
Policy: rayv1.DeleteSelf,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: -10,
},
},
},
},
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "deletionRules with ClusterSelector and DeleteWorkers policy",
spec: rayv1.RayJobSpec{
ClusterSelector: map[string]string{"key": "value"},
DeletionStrategy: &rayv1.DeletionStrategy{
DeletionRules: []rayv1.DeletionRule{
{
Policy: rayv1.DeleteWorkers,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: 10,
},
},
},
},
},
expectError: true,
},
{
name: "deletionRules with ClusterSelector and DeleteCluster policy",
spec: rayv1.RayJobSpec{
ClusterSelector: map[string]string{"key": "value"},
DeletionStrategy: &rayv1.DeletionStrategy{
DeletionRules: []rayv1.DeletionRule{
{
Policy: rayv1.DeleteCluster,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: 10,
},
},
},
},
},
expectError: true,
},
{
name: "deletionRules with autoscaling and DeleteWorkers policy",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
DeletionRules: []rayv1.DeletionRule{
{
Policy: rayv1.DeleteWorkers,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: 10,
},
},
},
},
RayClusterSpec: &rayv1.RayClusterSpec{
EnableInTreeAutoscaling: ptr.To(true),
HeadGroupSpec: headGroupSpecWithOneContainer,
},
},
expectError: true,
},
{
name: "inconsistent TTLs in deletionRules (DeleteCluster < DeleteWorkers)",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
DeletionRules: []rayv1.DeletionRule{
{
Policy: rayv1.DeleteWorkers,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: 20,
},
},
{
Policy: rayv1.DeleteCluster,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: 10,
},
},
},
},
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "inconsistent TTLs in deletionRules (DeleteSelf < DeleteCluster)",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
DeletionRules: []rayv1.DeletionRule{
{
Policy: rayv1.DeleteCluster,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: 20,
},
},
{
Policy: rayv1.DeleteSelf,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: 10,
},
},
},
},
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "valid complex deletionRules",
spec: rayv1.RayJobSpec{
DeletionStrategy: &rayv1.DeletionStrategy{
DeletionRules: []rayv1.DeletionRule{
{
Policy: rayv1.DeleteWorkers,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: 10,
},
},
{
Policy: rayv1.DeleteCluster,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: 20,
},
},
{
Policy: rayv1.DeleteSelf,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusSucceeded,
TTLSeconds: 30,
},
},
{
Policy: rayv1.DeleteSelf,
Condition: rayv1.DeletionCondition{
JobStatus: rayv1.JobStatusFailed,
TTLSeconds: 0,
},
},
},
},
RayClusterSpec: createBasicRayClusterSpec(),
},
expectError: false,
},
}
features.SetFeatureGateDuringTest(t, features.RayJobDeletionPolicy, true)
defer func() {
features.SetFeatureGateDuringTest(t, features.RayJobDeletionPolicy, false)
}()
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateRayJobSpec(&rayv1.RayJob{
Spec: tt.spec,
})
if tt.expectError {
require.Error(t, err, tt.name)
} else {
require.NoError(t, err, tt.name)
}
})
}
}
func TestValidateRayJobMetadata(t *testing.T) {
err := ValidateRayJobMetadata(metav1.ObjectMeta{
Name: strings.Repeat("j", MaxRayJobNameLength+1),
})
require.ErrorContains(t, err, fmt.Sprintf("RayJob name should be no more than %d characters", MaxRayJobNameLength))
err = ValidateRayJobMetadata(metav1.ObjectMeta{
Name: strings.Repeat("1", MaxRayJobNameLength),
})
require.ErrorContains(t, err, "RayJob name should be a valid DNS1035 label: [a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?')]")
err = ValidateRayJobMetadata(metav1.ObjectMeta{
Name: strings.Repeat("j", MaxRayJobNameLength),
})
require.NoError(t, err)
}
func TestValidateRayServiceSpec(t *testing.T) {
upgradeStrat := rayv1.RayServiceUpgradeType("invalidStrategy")
tests := []struct {
name string
unexpectedMessage string
spec rayv1.RayServiceSpec
expectError bool
}{
{
name: "spec.rayClusterConfig.headGroupSpec.headService.metadata.name should not be set",
spec: rayv1.RayServiceSpec{
RayClusterSpec: rayv1.RayClusterSpec{
HeadGroupSpec: rayv1.HeadGroupSpec{
HeadService: &corev1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: "my-head-service",
},
},
},
},
},
expectError: true,
},
{
name: "The RayService spec is valid.",
spec: rayv1.RayServiceSpec{
RayClusterSpec: *createBasicRayClusterSpec(),
},
expectError: false,
},
{
name: "spec.UpgradeSpec.Type is invalid",
spec: rayv1.RayServiceSpec{
UpgradeStrategy: &rayv1.RayServiceUpgradeStrategy{
Type: &upgradeStrat,
},
RayClusterSpec: *createBasicRayClusterSpec(),
},
expectError: true,
},
{
name: "headGroupSpec should have at least one container",
spec: rayv1.RayServiceSpec{},
expectError: true,
},
{
name: "Spec.RayClusterDeletionDelaySeconds is negative",
spec: rayv1.RayServiceSpec{
RayClusterSpec: *createBasicRayClusterSpec(),
RayClusterDeletionDelaySeconds: ptr.To[int32](-1),
},
expectError: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateRayServiceSpec(&rayv1.RayService{
Spec: tt.spec,
})
if tt.expectError {
require.Error(t, err, tt.name)
} else {
require.NoError(t, err, tt.name)
}
})
}
}
func TestValidateRayServiceMetadata(t *testing.T) {
err := ValidateRayServiceMetadata(metav1.ObjectMeta{
Name: strings.Repeat("j", MaxRayServiceNameLength+1),
})
require.ErrorContains(t, err, fmt.Sprintf("RayService name should be no more than %d characters", MaxRayServiceNameLength))
err = ValidateRayServiceMetadata(metav1.ObjectMeta{
Name: strings.Repeat("1", MaxRayServiceNameLength),
})
require.ErrorContains(t, err, "RayService name should be a valid DNS1035 label: [a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?')]")
err = ValidateRayServiceMetadata(metav1.ObjectMeta{
Name: strings.Repeat("j", MaxRayServiceNameLength),
})
require.NoError(t, err)
// Test valid initializing timeout annotations
err = ValidateRayServiceMetadata(metav1.ObjectMeta{
Name: "valid-service",
Annotations: map[string]string{
RayServiceInitializingTimeoutAnnotation: "5m",
},
})
require.NoError(t, err)
err = ValidateRayServiceMetadata(metav1.ObjectMeta{
Name: "valid-service",
Annotations: map[string]string{
RayServiceInitializingTimeoutAnnotation: "300",
},
})
require.NoError(t, err)
// Test invalid initializing timeout annotations
err = ValidateRayServiceMetadata(metav1.ObjectMeta{
Name: "invalid-service",
Annotations: map[string]string{
RayServiceInitializingTimeoutAnnotation: "0",
},
})
require.ErrorContains(t, err, "must be a positive")
err = ValidateRayServiceMetadata(metav1.ObjectMeta{
Name: "invalid-service",
Annotations: map[string]string{
RayServiceInitializingTimeoutAnnotation: "-100",
},
})
require.ErrorContains(t, err, "must be a positive")
err = ValidateRayServiceMetadata(metav1.ObjectMeta{
Name: "invalid-service",
Annotations: map[string]string{
RayServiceInitializingTimeoutAnnotation: "-5m",
},
})
require.ErrorContains(t, err, "must be a positive duration")
err = ValidateRayServiceMetadata(metav1.ObjectMeta{
Name: "invalid-service",
Annotations: map[string]string{
RayServiceInitializingTimeoutAnnotation: "invalid",
},
})
require.ErrorContains(t, err, "invalid format")
}
func createBasicRayClusterSpec() *rayv1.RayClusterSpec {
return &rayv1.RayClusterSpec{
HeadGroupSpec: rayv1.HeadGroupSpec{
Template: podTemplateSpec(nil, nil),
},
}
}
func TestValidateClusterUpgradeOptions(t *testing.T) {
tests := []struct {
maxSurgePercent *int32
stepSizePercent *int32
intervalSeconds *int32
name string
gatewayClassName string
spec rayv1.RayServiceSpec
enableAutoscaling bool
expectError bool
}{
{
name: "valid config",
maxSurgePercent: ptr.To(int32(50)),
stepSizePercent: ptr.To(int32(50)),
intervalSeconds: ptr.To(int32(10)),
gatewayClassName: "istio",
enableAutoscaling: true,
expectError: false,
},
{
name: "missing autoscaler",
maxSurgePercent: ptr.To(int32(50)),
stepSizePercent: ptr.To(int32(50)),
intervalSeconds: ptr.To(int32(10)),
gatewayClassName: "istio",
enableAutoscaling: false,
expectError: true,
},
{
name: "missing options",
enableAutoscaling: true,
expectError: true,
},
{
name: "invalid MaxSurgePercent",
maxSurgePercent: ptr.To(int32(200)),
stepSizePercent: ptr.To(int32(50)),
intervalSeconds: ptr.To(int32(10)),
gatewayClassName: "istio",
enableAutoscaling: true,
expectError: true,
},
{
name: "missing StepSizePercent",
maxSurgePercent: ptr.To(int32(50)),
intervalSeconds: ptr.To(int32(10)),
gatewayClassName: "istio",
enableAutoscaling: true,
expectError: true,
},
{
name: "invalid IntervalSeconds",
maxSurgePercent: ptr.To(int32(50)),
stepSizePercent: ptr.To(int32(50)),
intervalSeconds: ptr.To(int32(0)),
gatewayClassName: "istio",
enableAutoscaling: true,
expectError: true,
},
{
name: "missing GatewayClassName",
maxSurgePercent: ptr.To(int32(50)),
stepSizePercent: ptr.To(int32(50)),
intervalSeconds: ptr.To(int32(10)),
enableAutoscaling: true,
expectError: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var upgradeStrategy *rayv1.RayServiceUpgradeStrategy
if tt.maxSurgePercent != nil || tt.stepSizePercent != nil || tt.intervalSeconds != nil || tt.gatewayClassName != "" {
upgradeStrategy = &rayv1.RayServiceUpgradeStrategy{
Type: ptr.To(rayv1.NewClusterWithIncrementalUpgrade),
ClusterUpgradeOptions: &rayv1.ClusterUpgradeOptions{
MaxSurgePercent: tt.maxSurgePercent,
StepSizePercent: tt.stepSizePercent,
IntervalSeconds: tt.intervalSeconds,
GatewayClassName: tt.gatewayClassName,
},
}
} else if tt.expectError {
upgradeStrategy = &rayv1.RayServiceUpgradeStrategy{
Type: ptr.To(rayv1.NewClusterWithIncrementalUpgrade),
}
}
rayClusterSpec := *createBasicRayClusterSpec()
rayClusterSpec.EnableInTreeAutoscaling = ptr.To(tt.enableAutoscaling)
rayService := &rayv1.RayService{
Spec: rayv1.RayServiceSpec{
RayClusterSpec: rayClusterSpec,
UpgradeStrategy: upgradeStrategy,
},
}
err := ValidateClusterUpgradeOptions(rayService)
if tt.expectError {
require.Error(t, err, tt.name)
} else {
require.NoError(t, err, tt.name)
}
})
}
}