Files
+10 f68857e9ff [release-1.5] Cherry-pick commits for v1.5.1 (#4214)
* [Bug] Sidecar mode shouldn't restart head pod when head pod is deleted (#4141)

* [Bug] Sidecar mode shouldn't restart head pod when head pod is deleted

Signed-off-by: 400Ping <[email protected]>

* [Fix] Fix e2e error

Signed-off-by: 400Ping <[email protected]>

* [Fix] fix according to rueian's comment

Signed-off-by: 400Ping <[email protected]>

* [Chore] fix ci error

Signed-off-by: 400Ping <[email protected]>

* Update ray-operator/controllers/ray/raycluster_controller.go

Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>
Signed-off-by: Ping <[email protected]>

* Update ray-operator/controllers/ray/rayjob_controller.go

Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>
Signed-off-by: Ping <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* Trigger CI

Signed-off-by: Future-Outlier <[email protected]>

---------

Signed-off-by: 400Ping <[email protected]>
Signed-off-by: Ping <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>

* fix: dashboard build for kuberay 1.5.0 (#4161)

Signed-off-by: Future-Outlier <[email protected]>

* [Feature Enhancement] Set ordered replica index label to support multi-slice (#4163)

* [Feature Enhancement] Set ordered replica index label to support multi-slice

Signed-off-by: Ryan O'Leary <[email protected]>

* rename replica-id -> replica-name

Signed-off-by: Ryan O'Leary <[email protected]>

* Separate replica index feature gate logic

Signed-off-by: Ryan O'Leary <[email protected]>

* remove index arg in createWorkerPod

Signed-off-by: Ryan O'Leary <[email protected]>

---------

Signed-off-by: Ryan O'Leary <[email protected]>

* update stale feature gate comments (#4174)

Signed-off-by: Andrew Sy Kim <[email protected]>

* [RayCluster] Add more context why we don't recreate head Pod for RayJob (#4175)

Signed-off-by: Kai-Hsun Chen <[email protected]>

* feature: Remove empty resource list initialization. (#4168)

Fixes #4142.

* [Dockerfile] [KubeRay Dashboard]: Fix Dockerfile warnings (ENV format, CMD JSON args) (#4167)

* [#4166] improvement: Fix Dockerfile warnings (ENV format, CMD JSON args)

* extract the hostname from CMD

Signed-off-by: Neo Chien <[email protected]>

---------

Signed-off-by: Neo Chien <[email protected]>
Co-authored-by: cchung100m <[email protected]>

* [Fix] Resolve int32 overflow by having the calculation in int64 and c… (#4158)

* [Fix] Resolve int32 overflow by having the calculation in int64 and cap it if the count is over math.MaxInt32

Signed-off-by: justinyeh1995 <[email protected]>

* [Test] Add unit tests for CalculateReadyReplicas

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] Add a nosec comment to pass the Lint (pre-commit) test

Signed-off-by: justinyeh1995 <[email protected]>

* [Refactor] Add CapInt64ToInt32 to replace #nosec directives

Signed-off-by: justinyeh1995 <[email protected]>

* [Refactor] Rename function to SafeInt64ToInt32 and add a underflowing prevention (it also help pass the lint test)

Signed-off-by: justinyeh1995 <[email protected]>

* [Refactor] Remove the early return as SafeInt64ToInt32 handles the int32 overflow and underflow checking.

Signed-off-by: justinyeh1995 <[email protected]>

---------

Signed-off-by: justinyeh1995 <[email protected]>

* Add RayService incremental upgrade sample for guide (#4164)

Signed-off-by: Ryan O'Leary <[email protected]>

* Edit RayCluster example config for label selectors (#4151)

Signed-off-by: Ryan O'Leary <[email protected]>

* [RayJob] update light weight submitter image from quay.io (#4181)

Signed-off-by: Future-Outlier <[email protected]>

* [flaky] RayJob fails when head Pod is deleted when job is running (#4182)

Signed-off-by: Future-Outlier <[email protected]>

* [CI] Pin Docker api version to avoid API version mismatch (#4188)

Signed-off-by: win5923 <[email protected]>

* Make replicas configurable for kuberay-operator #4180 (#4195)

* Make replicas configurable for kuberay-operator #4180

* Make replicas configurable for kuberay-operator #4180

* [Fix] rayjob update raycluster status (#4192)

* feat: check if raycluster status update in rayjob

* test: e2e test to check the rayjob raycluster status update

* fix: dashboard http client tests discovered and passing (#4173)

Signed-off-by: alimaazamat <[email protected]>

* [RayJob] Lift cluster status while initializing (#4191)

Signed-off-by: Spencer Peterson <[email protected]>

* [RayJob] Remove updateJobStatus call (#4198)

Fast follow to #4191

Signed-off-by: Spencer Peterson <[email protected]>

* Add support for Ray token auth (#4179)

* Add support for Ray token auth

Signed-off-by: Andrew Sy Kim <[email protected]>

* add e2e test for Ray cluster auth

Signed-off-by: Andrew Sy Kim <[email protected]>

* address nits from Ruiean

Signed-off-by: Andrew Sy Kim <[email protected]>

* update RAY_auth_mode -> RAY_AUTH_MODE

Signed-off-by: Andrew Sy Kim <[email protected]>

* configure auth for Ray autoscaler

Signed-off-by: Andrew Sy Kim <[email protected]>

---------

Signed-off-by: Andrew Sy Kim <[email protected]>

* Bump js-yaml from 4.1.0 to 4.1.1 in /dashboard (#4194)

Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 4.1.0 to 4.1.1.
- [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodeca/js-yaml/compare/4.1.0...4.1.1)

---
updated-dependencies:
- dependency-name: js-yaml
  dependency-version: 4.1.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update minimum Ray version required for token authentication to 2.52.0 (#4201)

* update minimum Ray version required for token authentication to 2.52.0

Signed-off-by: Andrew Sy Kim <[email protected]>

* update RayCluster auth e2e test to use Ray v2.52

Signed-off-by: Andrew Sy Kim <[email protected]>

---------

Signed-off-by: Andrew Sy Kim <[email protected]>

* add samples for RayCluster token auth (#4200)

Signed-off-by: Andrew Sy Kim <[email protected]>

* update (#4208)

Signed-off-by: Future-Outlier <[email protected]>

* [RayJob] Add token authentication support for All mode (#4210)

* dashboard client authentication support

Signed-off-by: Future-Outlier <[email protected]>

* support rayjob

Signed-off-by: Future-Outlier <[email protected]>

* update to fix api serverr err

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* updarte

Signed-off-by: Future-Outlier <[email protected]>

* Rayjob sidecar mode auth token mode support

Signed-off-by: Future-Outlier <[email protected]>

* RayJob support k8s job mode

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* Address Andrew's advice

Signed-off-by: Future-Outlier <[email protected]>

* add todo x-ray-authorization comments

Signed-off-by: Future-Outlier <[email protected]>

---------

Signed-off-by: Future-Outlier <[email protected]>

* [RayCluster] Enable Secret informer watch/list and remove unused RBAC verbs (#4202)

* Add authentication secret reconciliation support

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* fix flaky test

Signed-off-by: Future-Outlier <[email protected]>

* remove test fix

Signed-off-by: Rueian <[email protected]>

---------

Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Rueian <[email protected]>
Co-authored-by: Rueian <[email protected]>

* [APIServer][Docs] Add user guide for retry behavior & configuration (#4144)

* [Docs] Add the draft description about feature intro, configurations, and usecases

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] Update the retry walk-through

Signed-off-by: justinyeh1995 <[email protected]>

* [Doc] rewrite the first 2 sections

Signed-off-by: justinyeh1995 <[email protected]>

* [Doc] Revise documentation wording and add Observing Retry Behavior section

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] fix linting issue by running pre-commit run berfore commiting

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] fix linting errors in the Markdown linting

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] Clean up the math equation

Signed-off-by: justinyeh1995 <[email protected]>

* Update the math formula of Backoff calculation.

Co-authored-by: Nary Yeh <[email protected]>
Signed-off-by: JustinYeh <[email protected]>

* [Fix] Explicitly mentioned exponential backoff and removed the customization parts

Signed-off-by: justinyeh1995 <[email protected]>

* [Docs] Clarify naming by replacing “APIServer” with “KubeRay APIServer”

Co-authored-by: Cheng-Yeh Chung <[email protected]>
Signed-off-by: JustinYeh <[email protected]>

* [Docs] Rename retry-configuration.md to retry-behavior.md for accuracy

Signed-off-by: justinyeh1995 <[email protected]>

* Update Title to KubeRay APIServer Retry Behavior

Co-authored-by: Cheng-Yeh Chung <[email protected]>
Signed-off-by: JustinYeh <[email protected]>

* [Docs] Add a note about the limitation of retry configuration

Signed-off-by: justinyeh1995 <[email protected]>

---------

Signed-off-by: justinyeh1995 <[email protected]>
Signed-off-by: JustinYeh <[email protected]>
Co-authored-by: Nary Yeh <[email protected]>
Co-authored-by: Cheng-Yeh Chung <[email protected]>

* Support X-Ray-Authorization fallback header for accepting auth token via proxy (#4213)

* Support X-Ray-Authorization fallback header for accepting auth token in dashboard

Signed-off-by: Future-Outlier <[email protected]>

* remove todo comment

Signed-off-by: Future-Outlier <[email protected]>

---------

Signed-off-by: Future-Outlier <[email protected]>

* [RayCluster] make auth token secret name consistency (#4216)

Signed-off-by: fscnick <[email protected]>

* [RayCluster] Status includes head containter status message (#4196)

* [RayCluster] Status includes head containter status message

Signed-off-by: Spencer Peterson <[email protected]>

* lint

Signed-off-by: Spencer Peterson <[email protected]>

* [RayCluster] Containers not ready status reflects structured reason

Signed-off-by: Spencer Peterson <[email protected]>

* nit

Signed-off-by: Spencer Peterson <[email protected]>

---------

Signed-off-by: Spencer Peterson <[email protected]>

* Remove erroneous  call in applyServeTargetCapacity (#4212)

Signed-off-by: Ryan O'Leary <[email protected]>

* [RayJob] Add token authentication support for light weight job submitter (#4215)

* [RayJob] light weight job submitter auth token support

Signed-off-by: Future-Outlier <[email protected]>

* X-Ray-Authorization

Signed-off-by: Rueian <[email protected]>

---------

Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Rueian <[email protected]>
Co-authored-by: Rueian <[email protected]>

* feat: kubectl ray get token command (#4218)

* feat: kubectl ray get token command

Signed-off-by: Rueian <[email protected]>

* Update kubectl-plugin/pkg/cmd/get/get_token_test.go

Co-authored-by: Copilot <[email protected]>
Signed-off-by: Rueian <[email protected]>

* Update kubectl-plugin/pkg/cmd/get/get_token.go

Co-authored-by: Copilot <[email protected]>
Signed-off-by: Rueian <[email protected]>

* make sure the raycluster exists before getting the secret

Signed-off-by: Rueian <[email protected]>

* better ux

Signed-off-by: Rueian <[email protected]>

* Update kubectl-plugin/pkg/cmd/get/get_token.go

Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>
Signed-off-by: Rueian <[email protected]>

---------

Signed-off-by: Rueian <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>

---------

Signed-off-by: 400Ping <[email protected]>
Signed-off-by: Ping <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Ryan O'Leary <[email protected]>
Signed-off-by: Andrew Sy Kim <[email protected]>
Signed-off-by: Kai-Hsun Chen <[email protected]>
Signed-off-by: Neo Chien <[email protected]>
Signed-off-by: justinyeh1995 <[email protected]>
Signed-off-by: win5923 <[email protected]>
Signed-off-by: alimaazamat <[email protected]>
Signed-off-by: Spencer Peterson <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Rueian <[email protected]>
Signed-off-by: JustinYeh <[email protected]>
Signed-off-by: fscnick <[email protected]>
Co-authored-by: Ping <[email protected]>
Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>
Co-authored-by: Ryan O'Leary <[email protected]>
Co-authored-by: Kai-Hsun Chen <[email protected]>
Co-authored-by: Kavish <[email protected]>
Co-authored-by: Neo Chien <[email protected]>
Co-authored-by: cchung100m <[email protected]>
Co-authored-by: JustinYeh <[email protected]>
Co-authored-by: Jun-Hao Wan <[email protected]>
Co-authored-by: Divyam Raj <[email protected]>
Co-authored-by: Nary Yeh <[email protected]>
Co-authored-by: Alima Azamat <[email protected]>
Co-authored-by: Spencer Peterson <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Rueian <[email protected]>
Co-authored-by: Cheng-Yeh Chung <[email protected]>
Co-authored-by: fscnick <[email protected]>
Co-authored-by: Copilot <[email protected]>
2025-11-21 10:18:05 -05:00

1795 lines
49 KiB
Go

package utils
import (
"context"
"errors"
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/utils/ptr"
gwv1 "sigs.k8s.io/gateway-api/apis/v1"
rayv1 "github.com/ray-project/kuberay/ray-operator/apis/ray/v1"
"github.com/ray-project/kuberay/ray-operator/controllers/ray/utils/dashboardclient"
"github.com/ray-project/kuberay/ray-operator/pkg/features"
)
func TestGetClusterDomainName(t *testing.T) {
tests := []struct {
name string
env string
want string
}{{
name: "all good from env",
env: "abc.com",
want: "abc.com",
}, {
name: "No env set",
env: "",
want: DefaultDomainName,
}}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if len(tt.env) > 0 {
t.Setenv(ClusterDomainEnvKey, tt.env)
}
got := GetClusterDomainName()
if got != tt.want {
t.Errorf("Test %s failed expected: %s but got: %s", tt.name, tt.want, got)
}
})
}
}
func TestStatus(t *testing.T) {
pod := createSomePod()
pod.Status.Phase = corev1.PodPending
if !IsCreated(pod) {
t.Fail()
}
}
func TestCheckAllPodsRunning(t *testing.T) {
tests := []struct {
name string
pods corev1.PodList
expected bool
}{
{
name: "should return true if all Pods are running",
pods: corev1.PodList{
Items: []corev1.Pod{
*createSomePodWithPhase(corev1.PodRunning),
*createSomePodWithPhase(corev1.PodRunning),
},
},
expected: true,
},
{
name: "should return false if there are no Pods",
pods: corev1.PodList{
Items: []corev1.Pod{},
},
expected: false,
},
{
name: "should return false if any Pods don't have .status.phase Running",
pods: corev1.PodList{
Items: []corev1.Pod{
*createSomePodWithPhase(corev1.PodPending),
*createSomePodWithPhase(corev1.PodRunning),
},
},
expected: false,
},
{
name: "should return false if any Pods have a .status.condition of type: Ready that's not status: True",
pods: corev1.PodList{
Items: []corev1.Pod{
*createSomePodWithPhase(corev1.PodRunning),
*createSomePodWithCondition(corev1.PodReady, corev1.ConditionFalse),
},
},
expected: false,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
assert.Equal(t, tc.expected, CheckAllPodsRunning(context.Background(), tc.pods))
})
}
}
func TestWorkerPodName(t *testing.T) {
tests := []struct {
name string
prefix string
expected string
}{
{
name: "short cluster name, worker pod",
prefix: "ray-cluster-group-name-01",
expected: "ray-cluster-group-name-01-worker-",
},
{
name: "long cluster name, worker pod",
prefix: "ray-cluster-0000000000000000000000011111111122222233333333333333-group-name",
expected: "ray-cluster-00000000000000000000000111111111222222-worker-",
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
str := PodName(test.prefix, rayv1.WorkerNode, true)
if str != test.expected {
t.Logf("expected: %q", test.expected)
t.Logf("actual: %q", str)
t.Error("PodName returned an unexpected string")
}
// 63 (max pod name length) - 5 random hexadecimal characters from generateName
if len(str) > 58 {
t.Error("Generated pod name is too long")
}
})
}
}
func TestHeadPodName(t *testing.T) {
defer os.Unsetenv(ENABLE_DETERMINISTIC_HEAD_POD_NAME)
tests := []struct {
name string
prefix string
enableDeterministicHeadPod string
expected string
}{
{
name: "short cluster name, deterministic head pod name",
prefix: "ray-cluster-01",
enableDeterministicHeadPod: "true",
expected: "ray-cluster-01-head",
},
{
name: "short cluster name, non-deterministic head pod name",
prefix: "ray-cluster-01",
enableDeterministicHeadPod: "false",
expected: "ray-cluster-01-head-",
},
{
name: "short cluster name, feature flag not set",
prefix: "ray-cluster-01",
enableDeterministicHeadPod: "unset",
expected: "ray-cluster-01-head-",
},
{
name: "long cluster name, deterministic head pod name",
prefix: "ray-cluster-0000000000000000000000011111111122222233333333333333",
enableDeterministicHeadPod: "true",
expected: "ray-cluster-00000000000000000000000111111111222222-head",
},
{
name: "long cluster name, non-deterministic head pod name",
prefix: "ray-cluster-0000000000000000000000011111111122222233333333333333",
enableDeterministicHeadPod: "false",
expected: "ray-cluster-00000000000000000000000111111111222222-head-",
},
{
name: "long cluster name, feature flag not set",
prefix: "ray-cluster-0000000000000000000000011111111122222233333333333333",
enableDeterministicHeadPod: "unset",
expected: "ray-cluster-00000000000000000000000111111111222222-head-",
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
if test.enableDeterministicHeadPod == "unset" {
os.Unsetenv(ENABLE_DETERMINISTIC_HEAD_POD_NAME)
} else {
os.Setenv(ENABLE_DETERMINISTIC_HEAD_POD_NAME, test.enableDeterministicHeadPod)
}
str := PodName(test.prefix, rayv1.HeadNode, !IsDeterministicHeadPodNameEnabled())
if str != test.expected {
t.Logf("expected: %q", test.expected)
t.Logf("actual: %q", str)
t.Error("PodName returned an unexpected string")
}
// 63 (max pod name length) - 5 random hexadecimal characters from generateName
if len(str) > 58 {
t.Error("Generated pod name is too long")
}
})
}
}
func TestCheckName(t *testing.T) {
tests := []struct {
name string
input string
expected string
}{
{
name: "shorten long string starting with numeric character",
input: "72fbcc7e-a661-4b18e-ca41-e903-fc3ae634b18e-lazer090scholar-director-s",
expected: "rca41-e903-fc3ae634b18e-lazer090scholar-director-s",
},
{
name: "shorten long string starting with special character",
input: "--------566666--------444433-----------222222----------4444",
expected: "r6666--------444433-----------222222----------4444",
},
{
name: "unchanged",
input: "acceptable-name-head-12345",
expected: "acceptable-name-head-12345",
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
str := CheckName(test.input)
if str != test.expected {
t.Logf("expected: %q", test.expected)
t.Logf("actual: %q", str)
t.Error("CheckName returned an unexpected string")
}
})
}
}
func TestCheckRouteName(t *testing.T) {
tests := []struct {
name string
routeName string
namespace string
want string
}{{
name: "long route name truncated",
routeName: "cv-traffic-training-202402090958",
namespace: "development-namespace",
want: "cv-traffic-training-2024020909",
}, {
name: "long route name w/number start truncated and number replaced",
routeName: "2-step-cv-training-network-revisited",
namespace: "development-namespace",
want: "r-step-cv-training-network-rev",
}, {
name: "well-formatted and well-sized route name unaffected",
routeName: "acceptable-name-head-12345",
namespace: "development-namespace",
want: "acceptable-name-head-12345",
}}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
name := CheckRouteName(context.Background(), tc.routeName, tc.namespace)
assert.Equal(t, tc.want, name)
})
}
}
func createSomePod() (pod *corev1.Pod) {
return &corev1.Pod{
TypeMeta: metav1.TypeMeta{
APIVersion: "v1",
Kind: "Pod",
},
ObjectMeta: metav1.ObjectMeta{
Name: "raycluster-sample-small-group-worker-0",
Namespace: "default",
},
}
}
func createSomePodWithPhase(phase corev1.PodPhase) (pod *corev1.Pod) {
return &corev1.Pod{
TypeMeta: metav1.TypeMeta{
APIVersion: "v1",
Kind: "Pod",
},
ObjectMeta: metav1.ObjectMeta{
Name: "raycluster-sample-small-group-worker-0",
Namespace: "default",
},
Status: corev1.PodStatus{
Phase: phase,
},
}
}
func createSomePodWithCondition(typ corev1.PodConditionType, status corev1.ConditionStatus) (pod *corev1.Pod) {
return &corev1.Pod{
TypeMeta: metav1.TypeMeta{
APIVersion: "v1",
Kind: "Pod",
},
ObjectMeta: metav1.ObjectMeta{
Name: "raycluster-sample-small-group-worker-0",
Namespace: "default",
},
Status: corev1.PodStatus{
Conditions: []corev1.PodCondition{
{
Type: typ,
Status: status,
},
},
},
}
}
func createRayHeadPodWithPhaseAndCondition(phase corev1.PodPhase, status corev1.ConditionStatus) (pod *corev1.Pod) {
return &corev1.Pod{
TypeMeta: metav1.TypeMeta{
APIVersion: "v1",
Kind: "Pod",
},
ObjectMeta: metav1.ObjectMeta{
Name: "raycluster-sample-head",
Namespace: "default",
Labels: map[string]string{
"ray.io/node-type": string(rayv1.HeadNode),
},
},
Status: corev1.PodStatus{
Phase: phase,
Conditions: []corev1.PodCondition{
{
Type: corev1.PodReady,
Status: status,
Reason: ContainersNotReady,
},
},
},
}
}
func TestGetHeadGroupServiceAccountName(t *testing.T) {
tests := []struct {
name string
input *rayv1.RayCluster
want string
}{
{
name: "Ray cluster with head group service account",
input: &rayv1.RayCluster{
ObjectMeta: metav1.ObjectMeta{
Name: "raycluster-sample",
Namespace: "default",
},
Spec: rayv1.RayClusterSpec{
HeadGroupSpec: rayv1.HeadGroupSpec{
Template: corev1.PodTemplateSpec{
Spec: corev1.PodSpec{
ServiceAccountName: "my-service-account",
},
},
},
},
},
want: "my-service-account",
},
{
name: "Ray cluster without head group service account",
input: &rayv1.RayCluster{
ObjectMeta: metav1.ObjectMeta{
Name: "raycluster-sample",
Namespace: "default",
},
Spec: rayv1.RayClusterSpec{
HeadGroupSpec: rayv1.HeadGroupSpec{
Template: corev1.PodTemplateSpec{
Spec: corev1.PodSpec{},
},
},
},
},
want: "raycluster-sample",
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
assert.Equal(t, tc.want, GetHeadGroupServiceAccountName(tc.input))
})
}
}
func TestCalculateAvailableReplicas(t *testing.T) {
podList := corev1.PodList{
Items: []corev1.Pod{
{
ObjectMeta: metav1.ObjectMeta{
Name: "pod1",
Labels: map[string]string{
"ray.io/node-type": string(rayv1.HeadNode),
},
},
Status: corev1.PodStatus{
Phase: corev1.PodRunning,
},
},
{
ObjectMeta: metav1.ObjectMeta{
Name: "pod2",
Labels: map[string]string{
"ray.io/node-type": string(rayv1.WorkerNode),
},
},
Status: corev1.PodStatus{
Phase: corev1.PodRunning,
Conditions: []corev1.PodCondition{
{
Type: corev1.PodReady,
Status: corev1.ConditionTrue,
},
},
},
},
{
ObjectMeta: metav1.ObjectMeta{
Name: "pod2",
Labels: map[string]string{
"ray.io/node-type": string(rayv1.WorkerNode),
},
},
Status: corev1.PodStatus{
Phase: corev1.PodPending,
Conditions: []corev1.PodCondition{
{
Type: corev1.PodReady,
Status: corev1.ConditionFalse,
},
},
},
},
{
ObjectMeta: metav1.ObjectMeta{
Name: "pod2",
Labels: map[string]string{
"ray.io/node-type": string(rayv1.WorkerNode),
},
},
Status: corev1.PodStatus{
Phase: corev1.PodFailed,
},
},
},
}
availableCount := CalculateAvailableReplicas(podList)
assert.Equal(t, int32(1), availableCount, "expect 1 available replica")
readyCount := CalculateReadyReplicas(podList)
assert.Equal(t, int32(1), readyCount, "expect 1 ready replica")
}
func TestFindContainerPort(t *testing.T) {
container := corev1.Container{
Name: "ray-head",
Ports: []corev1.ContainerPort{
{
Name: "port1",
ContainerPort: 10001,
},
{
Name: "port2",
ContainerPort: 10002,
},
},
}
port := FindContainerPort(&container, "port1", -1)
assert.NotEqual(t, port, int32(-1), "expect port1 found")
port = FindContainerPort(&container, "port2", -1)
assert.NotEqual(t, port, int32(-1), "expect port2 found")
port = FindContainerPort(&container, "port3", -1)
assert.Equal(t, port, int32(-1), "expect port3 not found")
}
func TestGenerateHeadServiceName(t *testing.T) {
// GenerateHeadServiceName generates a Ray head service name. Note that there are two types of head services:
//
// (1) For RayCluster: If `HeadService.Name` in the cluster spec is not empty, it will be used as the head service name.
// Otherwise, the name is generated based on the RayCluster CR's name.
// (2) For RayService: It's important to note that the RayService CR not only possesses a head service owned by its RayCluster CR
// but also maintains a separate head service for itself to facilitate zero-downtime upgrades. The name of the head service owned
// by the RayService CR is generated based on the RayService CR's name.
// [RayCluster]
// Test 1: `HeadService.Name` is empty.
headSvcName, err := GenerateHeadServiceName(RayClusterCRD, rayv1.RayClusterSpec{}, "raycluster-sample")
expectedGeneratedSvcName := "raycluster-sample-head-svc"
require.NoError(t, err)
assert.Equal(t, expectedGeneratedSvcName, headSvcName)
// Test 2: `HeadService.Name` is not empty.
clusterSpecWithHeadService := rayv1.RayClusterSpec{
HeadGroupSpec: rayv1.HeadGroupSpec{
HeadService: &corev1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: "my-head-svc",
},
},
},
}
headSvcName, err = GenerateHeadServiceName(RayClusterCRD, *clusterSpecWithHeadService.DeepCopy(), "raycluster-sample")
require.NoError(t, err)
assert.Equal(t, "my-head-svc", headSvcName)
// [RayService]
// Test 3: `HeadService.Name` is empty.
headSvcName, err = GenerateHeadServiceName(RayServiceCRD, rayv1.RayClusterSpec{}, "rayservice-sample")
expectedGeneratedSvcName = "rayservice-sample-head-svc"
require.NoError(t, err)
assert.Equal(t, expectedGeneratedSvcName, headSvcName)
// Test 4: `HeadService.Name` is not empty.
headSvcName, err = GenerateHeadServiceName(RayServiceCRD, *clusterSpecWithHeadService.DeepCopy(), "rayservice-sample")
require.NoError(t, err)
assert.Equal(t, expectedGeneratedSvcName, headSvcName)
// [RayJob]
// Test 5: `HeadService.Name` is empty.
headSvcName, err = GenerateHeadServiceName(RayJobCRD, rayv1.RayClusterSpec{}, "rayjob-sample")
expectedGeneratedSvcName = "rayjob-sample-head-svc"
require.NoError(t, err)
assert.Equal(t, expectedGeneratedSvcName, headSvcName)
// Test 6: `HeadService.Name` is not empty.
headSvcName, err = GenerateHeadServiceName(RayJobCRD, *clusterSpecWithHeadService.DeepCopy(), "rayjob-sample")
require.NoError(t, err)
assert.Equal(t, expectedGeneratedSvcName, headSvcName)
}
func TestGetWorkerGroupDesiredReplicas(t *testing.T) {
ctx := context.Background()
// Test 1: `WorkerGroupSpec.Replicas` is nil.
// `Replicas` is impossible to be nil in a real RayCluster CR as it has a default value assigned in the CRD.
numOfHosts := int32(1)
minReplicas := int32(1)
maxReplicas := int32(5)
workerGroupSpec := rayv1.WorkerGroupSpec{
NumOfHosts: numOfHosts,
MinReplicas: &minReplicas,
MaxReplicas: &maxReplicas,
}
assert.Equal(t, GetWorkerGroupDesiredReplicas(ctx, workerGroupSpec), minReplicas)
// Test 2: `WorkerGroupSpec.Replicas` is not nil and is within the range.
replicas := int32(3)
workerGroupSpec.Replicas = &replicas
assert.Equal(t, GetWorkerGroupDesiredReplicas(ctx, workerGroupSpec), replicas)
// Test 3: `WorkerGroupSpec.Replicas` is not nil but is more than maxReplicas.
replicas = int32(6)
workerGroupSpec.Replicas = &replicas
assert.Equal(t, GetWorkerGroupDesiredReplicas(ctx, workerGroupSpec), maxReplicas)
// Test 4: `WorkerGroupSpec.Replicas` is not nil but is less than minReplicas.
replicas = int32(0)
workerGroupSpec.Replicas = &replicas
assert.Equal(t, GetWorkerGroupDesiredReplicas(ctx, workerGroupSpec), minReplicas)
// Test 5: `WorkerGroupSpec.Replicas` is nil and minReplicas is less than maxReplicas.
workerGroupSpec.Replicas = nil
workerGroupSpec.MinReplicas = &maxReplicas
workerGroupSpec.MaxReplicas = &minReplicas
assert.Equal(t, GetWorkerGroupDesiredReplicas(ctx, workerGroupSpec), *workerGroupSpec.MaxReplicas)
// Test 6: `WorkerGroupSpec.Suspend` is true.
suspend := true
workerGroupSpec.MinReplicas = &maxReplicas
workerGroupSpec.MaxReplicas = &minReplicas
workerGroupSpec.Suspend = &suspend
assert.Zero(t, GetWorkerGroupDesiredReplicas(ctx, workerGroupSpec))
// Test 7: `WorkerGroupSpec.NumOfHosts` is 4.
numOfHosts = int32(4)
replicas = int32(5)
suspend = false
workerGroupSpec.NumOfHosts = numOfHosts
workerGroupSpec.Replicas = &replicas
workerGroupSpec.Suspend = &suspend
workerGroupSpec.MinReplicas = &minReplicas
workerGroupSpec.MaxReplicas = &maxReplicas
assert.Equal(t, GetWorkerGroupDesiredReplicas(ctx, workerGroupSpec), replicas*numOfHosts)
}
func TestCalculateMinAndMaxReplicas(t *testing.T) {
suspend := true
tests := []struct {
name string
specs []rayv1.WorkerGroupSpec
expected struct {
minReplicas int32
maxReplicas int32
}
}{
{
name: "Single group with one host",
specs: []rayv1.WorkerGroupSpec{
{
NumOfHosts: 1,
MinReplicas: ptr.To[int32](2),
MaxReplicas: ptr.To[int32](3),
},
},
expected: struct {
minReplicas int32
maxReplicas int32
}{
minReplicas: 2,
maxReplicas: 3,
},
},
{
name: "Single group with four hosts",
specs: []rayv1.WorkerGroupSpec{
{
NumOfHosts: 4,
MinReplicas: ptr.To[int32](2),
MaxReplicas: ptr.To[int32](3),
},
},
expected: struct {
minReplicas int32
maxReplicas int32
}{
minReplicas: 8,
maxReplicas: 12,
},
},
{
name: "Two worker groups: one with a single host, one with two hosts",
specs: []rayv1.WorkerGroupSpec{
{
NumOfHosts: 1,
MinReplicas: ptr.To[int32](4),
MaxReplicas: ptr.To[int32](4),
},
{
NumOfHosts: 2,
MinReplicas: ptr.To[int32](3),
MaxReplicas: ptr.To[int32](3),
},
},
expected: struct {
minReplicas int32
maxReplicas int32
}{
minReplicas: 10,
maxReplicas: 10,
},
},
{
name: "Two groups with suspended",
specs: []rayv1.WorkerGroupSpec{
{
NumOfHosts: 1,
MinReplicas: ptr.To[int32](3),
MaxReplicas: ptr.To[int32](3),
Suspend: &suspend,
},
{
NumOfHosts: 1,
MinReplicas: ptr.To[int32](1),
MaxReplicas: ptr.To[int32](1),
Suspend: &suspend,
},
},
expected: struct {
minReplicas int32
maxReplicas int32
}{
minReplicas: 0,
maxReplicas: 0,
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cluster := &rayv1.RayCluster{
Spec: rayv1.RayClusterSpec{
WorkerGroupSpecs: tt.specs,
},
}
// Check min replicas
assert.Equal(t, tt.expected.minReplicas, CalculateMinReplicas(cluster))
// Check max replicas
assert.Equal(t, tt.expected.maxReplicas, CalculateMaxReplicas(cluster))
})
}
}
func TestCalculateDesiredReplicas(t *testing.T) {
tests := []struct {
group1Replicas *int32
group1MinReplicas *int32
group1MaxReplicas *int32
group2Replicas *int32
group2MinReplicas *int32
group2MaxReplicas *int32
name string
group1NumOfHosts int32
group2NumOfHosts int32
answer int32
}{
{
group1Replicas: nil,
group1NumOfHosts: 1,
group1MinReplicas: ptr.To[int32](1),
group1MaxReplicas: ptr.To[int32](5),
group2Replicas: nil,
group2NumOfHosts: 1,
group2MinReplicas: ptr.To[int32](2),
group2MaxReplicas: ptr.To[int32](5),
name: "Both groups' Replicas are nil",
answer: 3,
},
{
group1Replicas: ptr.To[int32](0),
group1NumOfHosts: 1,
group1MinReplicas: ptr.To[int32](2),
group1MaxReplicas: ptr.To[int32](5),
group2Replicas: ptr.To[int32](6),
group2NumOfHosts: 1,
group2MinReplicas: ptr.To[int32](2),
group2MaxReplicas: ptr.To[int32](5),
name: "Group1's Replicas is smaller than MinReplicas, and Group2's Replicas is more than MaxReplicas.",
answer: 7,
},
{
group1Replicas: ptr.To[int32](6),
group1NumOfHosts: 1,
group1MinReplicas: ptr.To[int32](2),
group1MaxReplicas: ptr.To[int32](5),
group2Replicas: ptr.To[int32](3),
group2NumOfHosts: 1,
group2MinReplicas: ptr.To[int32](2),
group2MaxReplicas: ptr.To[int32](5),
name: "Group1's Replicas is more than MaxReplicas.",
answer: 8,
},
{
group1Replicas: ptr.To[int32](3),
group1NumOfHosts: 4,
group1MinReplicas: ptr.To[int32](1),
group1MaxReplicas: ptr.To[int32](6),
group2Replicas: ptr.To[int32](3),
group2NumOfHosts: 1,
group2MinReplicas: ptr.To[int32](2),
group2MaxReplicas: ptr.To[int32](5),
name: "Group1's NumOfHosts is 4, and Group2's Replicas is 1.",
answer: 15,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
cluster := rayv1.RayCluster{
Spec: rayv1.RayClusterSpec{
WorkerGroupSpecs: []rayv1.WorkerGroupSpec{
{
GroupName: "group1",
Replicas: tc.group1Replicas,
NumOfHosts: tc.group1NumOfHosts,
MinReplicas: tc.group1MinReplicas,
MaxReplicas: tc.group1MaxReplicas,
},
{
GroupName: "group2",
Replicas: tc.group2Replicas,
NumOfHosts: tc.group2NumOfHosts,
MinReplicas: tc.group2MinReplicas,
MaxReplicas: tc.group2MaxReplicas,
},
},
},
}
assert.Equal(t, CalculateDesiredReplicas(context.Background(), &cluster), tc.answer)
})
}
}
func TestCalculateMaxReplicasOverflow(t *testing.T) {
tests := []struct {
name string
specs []rayv1.WorkerGroupSpec
expected int32
}{
{
name: "Bug reproduction: issue report with replicas=1, minReplicas=3, numOfHosts=4",
specs: []rayv1.WorkerGroupSpec{
{
GroupName: "workergroup",
Replicas: ptr.To[int32](1),
MinReplicas: ptr.To[int32](3),
MaxReplicas: ptr.To[int32](2147483647), // Default max int32
NumOfHosts: 4,
},
},
expected: 2147483647, // Was -4 before fix, should be capped at max int32
},
{
name: "Single group overflow with default maxReplicas and numOfHosts=4",
specs: []rayv1.WorkerGroupSpec{
{
NumOfHosts: 4,
MinReplicas: ptr.To[int32](3),
MaxReplicas: ptr.To[int32](2147483647),
},
},
expected: 2147483647, // Should be capped at max int32
},
{
name: "Single group overflow with large values",
specs: []rayv1.WorkerGroupSpec{
{
NumOfHosts: 1000,
MinReplicas: ptr.To[int32](1),
MaxReplicas: ptr.To[int32](2147483647),
},
},
expected: 2147483647, // Should be capped
},
{
name: "Multiple groups causing overflow when summed",
specs: []rayv1.WorkerGroupSpec{
{
NumOfHosts: 2,
MinReplicas: ptr.To[int32](1),
MaxReplicas: ptr.To[int32](1500000000),
},
{
NumOfHosts: 1,
MinReplicas: ptr.To[int32](1),
MaxReplicas: ptr.To[int32](1000000000),
},
},
expected: 2147483647, // 3B + 1B > max int32, should be capped
},
{
name: "No overflow with reasonable values",
specs: []rayv1.WorkerGroupSpec{
{
NumOfHosts: 4,
MinReplicas: ptr.To[int32](2),
MaxReplicas: ptr.To[int32](100),
},
},
expected: 400, // 100 * 4 = 400, no overflow
},
{
name: "Edge case: exactly at max int32",
specs: []rayv1.WorkerGroupSpec{
{
NumOfHosts: 1,
MinReplicas: ptr.To[int32](1),
MaxReplicas: ptr.To[int32](2147483647),
},
},
expected: 2147483647, // Exactly at limit
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
cluster := &rayv1.RayCluster{
Spec: rayv1.RayClusterSpec{
WorkerGroupSpecs: tc.specs,
},
}
result := CalculateMaxReplicas(cluster)
assert.Equal(t, tc.expected, result)
})
}
}
func TestUnmarshalRuntimeEnv(t *testing.T) {
tests := []struct {
name string
runtimeEnvYAML string
isErrorNil bool
}{
{
name: "Empty runtimeEnvYAML",
runtimeEnvYAML: "",
isErrorNil: true,
},
{
name: "Valid runtimeEnvYAML",
runtimeEnvYAML: `
env_vars:
counter_name: test_counter
`,
isErrorNil: true,
},
{
name: "Invalid runtimeEnvYAML",
runtimeEnvYAML: `invalid_yaml_str`,
isErrorNil: false,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
_, err := dashboardclient.UnmarshalRuntimeEnvYAML(tc.runtimeEnvYAML)
if tc.isErrorNil {
require.NoError(t, err)
} else {
require.Error(t, err)
}
})
}
}
func TestFindHeadPodReadyCondition(t *testing.T) {
tests := []struct {
name string
pod *corev1.Pod
expected metav1.Condition
}{
{
name: "condition true if Ray head pod is running and ready",
pod: createRayHeadPodWithPhaseAndCondition(corev1.PodRunning, corev1.ConditionTrue),
expected: metav1.Condition{
Type: string(rayv1.HeadPodReady),
Status: metav1.ConditionTrue,
},
},
{
name: "condition false if Ray head pod is not running",
pod: createRayHeadPodWithPhaseAndCondition(corev1.PodPending, corev1.ConditionFalse),
expected: metav1.Condition{
Type: string(rayv1.HeadPodReady),
Status: metav1.ConditionFalse,
},
},
{
name: "condition false if Ray head pod is not ready",
pod: createRayHeadPodWithPhaseAndCondition(corev1.PodRunning, corev1.ConditionFalse),
expected: metav1.Condition{
Type: string(rayv1.HeadPodReady),
Status: metav1.ConditionFalse,
},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
headPodReadyCondition := FindHeadPodReadyCondition(tc.pod)
assert.Equal(t, tc.expected.Status, headPodReadyCondition.Status)
})
}
}
func TestFindHeadPodReadyMessage(t *testing.T) {
tests := []struct {
name string
message string
wantMessage string
wantReason string
status []corev1.ContainerStatus
}{{
name: "no message no status want original reason",
wantReason: ContainersNotReady,
}, {
name: "no container status want original reason",
message: "TooEarlyInTheMorning",
wantMessage: "TooEarlyInTheMorning",
wantReason: ContainersNotReady,
}, {
name: "one reason one status",
message: "containers not ready",
status: []corev1.ContainerStatus{{
Name: "ray",
State: corev1.ContainerState{
Waiting: &corev1.ContainerStateWaiting{
Reason: "ImagePullBackOff",
Message: `Back-off pulling image royproject/roy:latest: ErrImagePull: rpc error: code = NotFound`,
},
},
}},
wantReason: "ImagePullBackOff",
wantMessage: `containers not ready; ray: Back-off pulling image royproject/roy:latest: ErrImagePull: rpc error: code = NotFound`,
}, {
name: "one reason two statuses only copy first",
message: "aesthetic problems",
status: []corev1.ContainerStatus{{
Name: "indigo",
State: corev1.ContainerState{
Waiting: &corev1.ContainerStateWaiting{
Reason: "BadColor",
Message: "too blue",
},
},
}, {
Name: "circle",
State: corev1.ContainerState{
Terminated: &corev1.ContainerStateTerminated{
Reason: "BadGeometry",
Message: "too round",
},
},
}},
wantReason: "BadColor",
wantMessage: "aesthetic problems; indigo: too blue",
}, {
name: "no reason one status",
status: []corev1.ContainerStatus{{
Name: "my-image",
State: corev1.ContainerState{
Terminated: &corev1.ContainerStateTerminated{
Reason: "Crashed",
Message: "bash not found",
},
},
}},
wantReason: "Crashed",
wantMessage: "my-image: bash not found",
}}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
pod := createRayHeadPodWithPhaseAndCondition(corev1.PodPending, corev1.ConditionFalse)
pod.Status.Conditions[0].Message = tc.message
pod.Status.ContainerStatuses = tc.status
cond := FindHeadPodReadyCondition(pod)
if cond.Message != tc.wantMessage {
t.Errorf("FindHeadPodReadyCondition(...) returned condition with message %q, but wanted %q", cond.Message, tc.wantMessage)
}
if cond.Reason != tc.wantReason {
t.Errorf("FindHeadPodReadyCondition(...) returned condition with reason %q, but wanted %q", cond.Reason, tc.wantReason)
}
})
}
}
func TestErrRayClusterReplicaFailureReason(t *testing.T) {
assert.Equal(t, "FailedDeleteAllPods", RayClusterReplicaFailureReason(ErrFailedDeleteAllPods))
assert.Equal(t, "FailedDeleteHeadPod", RayClusterReplicaFailureReason(ErrFailedDeleteHeadPod))
assert.Equal(t, "FailedCreateHeadPod", RayClusterReplicaFailureReason(ErrFailedCreateHeadPod))
assert.Equal(t, "FailedDeleteWorkerPod", RayClusterReplicaFailureReason(ErrFailedDeleteWorkerPod))
assert.Equal(t, "FailedCreateWorkerPod", RayClusterReplicaFailureReason(ErrFailedCreateWorkerPod))
assert.Equal(t, "FailedDeleteAllPods", RayClusterReplicaFailureReason(errors.Join(ErrFailedDeleteAllPods, errors.New("other error"))))
assert.Equal(t, "FailedDeleteHeadPod", RayClusterReplicaFailureReason(errors.Join(ErrFailedDeleteHeadPod, errors.New("other error"))))
assert.Equal(t, "FailedCreateHeadPod", RayClusterReplicaFailureReason(errors.Join(ErrFailedCreateHeadPod, errors.New("other error"))))
assert.Equal(t, "FailedDeleteWorkerPod", RayClusterReplicaFailureReason(errors.Join(ErrFailedDeleteWorkerPod, errors.New("other error"))))
assert.Equal(t, "FailedCreateWorkerPod", RayClusterReplicaFailureReason(errors.Join(ErrFailedCreateWorkerPod, errors.New("other error"))))
assert.Empty(t, RayClusterReplicaFailureReason(errors.New("other error")))
}
func TestIsAutoscalingEnabled(t *testing.T) {
tests := map[string]struct {
spec *rayv1.RayClusterSpec
expected bool
}{
"should be false when spec is nil": {
spec: nil,
expected: false,
},
"should be false when enableInTreeAutoscaling is nil": {
spec: &rayv1.RayClusterSpec{
EnableInTreeAutoscaling: nil,
},
expected: false,
},
"should be false when enableInTreeAutoscaling is false": {
spec: &rayv1.RayClusterSpec{
EnableInTreeAutoscaling: ptr.To(false),
},
expected: false,
},
"should be true when enableInTreeAutoscaling is true": {
spec: &rayv1.RayClusterSpec{
EnableInTreeAutoscaling: ptr.To(true),
},
expected: true,
},
}
for name, tc := range tests {
t.Run(name, func(t *testing.T) {
assert.Equal(t, tc.expected, IsAutoscalingEnabled(tc.spec))
})
}
}
func TestIsAutoscalingV2Enabled(t *testing.T) {
tests := map[string]struct {
spec *rayv1.RayClusterSpec
expected bool
}{
"should be false when spec is nil": {
spec: nil,
expected: false,
},
"should be false when autoscaler options is nil": {
spec: &rayv1.RayClusterSpec{
AutoscalerOptions: nil,
},
expected: false,
},
"should be false when autoscaler options is not v2": {
spec: &rayv1.RayClusterSpec{
AutoscalerOptions: &rayv1.AutoscalerOptions{Version: ptr.To(rayv1.AutoscalerVersionV1)},
},
expected: false,
},
"should be true when autoscaler options is v2": {
spec: &rayv1.RayClusterSpec{
AutoscalerOptions: &rayv1.AutoscalerOptions{Version: ptr.To(rayv1.AutoscalerVersionV2)},
},
expected: true,
},
}
for name, tc := range tests {
t.Run(name, func(t *testing.T) {
assert.Equal(t, tc.expected, IsAutoscalingV2Enabled(tc.spec))
})
}
}
func TestIsGCSFaultToleranceEnabled(t *testing.T) {
tests := []struct {
name string
instance rayv1.RayCluster
expected bool
}{
{
name: "ray.io/ft-enabled is true",
instance: rayv1.RayCluster{
ObjectMeta: metav1.ObjectMeta{
Annotations: map[string]string{
RayFTEnabledAnnotationKey: "true",
},
},
},
expected: true,
},
{
name: "ray.io/ft-enabled is not set and GcsFaultToleranceOptions is set",
instance: rayv1.RayCluster{
Spec: rayv1.RayClusterSpec{
GcsFaultToleranceOptions: &rayv1.GcsFaultToleranceOptions{},
},
},
expected: true,
},
{
name: "ray.io/ft-enabled is false",
instance: rayv1.RayCluster{
ObjectMeta: metav1.ObjectMeta{
Annotations: map[string]string{
RayFTEnabledAnnotationKey: "false",
},
},
},
expected: false,
},
{
name: "ray.io/ft-enabled is not set and GcsFaultToleranceOptions is not set",
instance: rayv1.RayCluster{
ObjectMeta: metav1.ObjectMeta{
Annotations: map[string]string{},
},
},
expected: false,
},
{
name: "ray.io/ft-enabled is using uppercase true",
instance: rayv1.RayCluster{
ObjectMeta: metav1.ObjectMeta{
Annotations: map[string]string{
RayFTEnabledAnnotationKey: "TRUE",
},
},
},
expected: true,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
result := IsGCSFaultToleranceEnabled(&test.instance.Spec, test.instance.Annotations)
assert.Equal(t, test.expected, result)
})
}
}
func createPodSpec(cpu, memory string) corev1.PodSpec {
return corev1.PodSpec{
Containers: []corev1.Container{
{
Resources: corev1.ResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse(cpu),
corev1.ResourceMemory: resource.MustParse(memory),
},
},
},
},
}
}
func createRayClusterTemplate(
head struct {
cpu string
memory string
},
workers []struct {
replicas *int32
minReplicas *int32
suspend *bool
cpu string
memory string
numOfHosts int32
},
) *rayv1.RayCluster {
cluster := &rayv1.RayCluster{
Spec: rayv1.RayClusterSpec{
HeadGroupSpec: rayv1.HeadGroupSpec{
Template: corev1.PodTemplateSpec{
Spec: createPodSpec(head.cpu, head.memory),
},
},
},
}
for _, w := range workers {
cluster.Spec.WorkerGroupSpecs = append(cluster.Spec.WorkerGroupSpecs, rayv1.WorkerGroupSpec{
NumOfHosts: w.numOfHosts,
Replicas: w.replicas,
MinReplicas: w.minReplicas,
Suspend: w.suspend,
Template: corev1.PodTemplateSpec{
Spec: createPodSpec(w.cpu, w.memory),
},
})
}
return cluster
}
func TestCalculateResources(t *testing.T) {
headStruct := struct {
cpu string
memory string
}{
cpu: "1",
memory: "100Mi",
}
tests := []struct {
expected struct {
desiredResources corev1.ResourceList
minResources corev1.ResourceList
}
cluster *rayv1.RayCluster
name string
}{
{
name: "Single head pod with no worker groups",
cluster: createRayClusterTemplate(headStruct, nil),
expected: struct {
desiredResources corev1.ResourceList
minResources corev1.ResourceList
}{
desiredResources: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse("1"),
corev1.ResourceMemory: resource.MustParse("100Mi"),
},
minResources: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse("1"),
corev1.ResourceMemory: resource.MustParse("100Mi"),
},
},
},
{
name: "Head pod with one worker group",
cluster: createRayClusterTemplate(headStruct, []struct {
replicas *int32
minReplicas *int32
suspend *bool
cpu string
memory string
numOfHosts int32
}{
{
numOfHosts: 2,
replicas: ptr.To[int32](4),
minReplicas: ptr.To[int32](3),
cpu: "1",
memory: "200Mi",
suspend: nil,
},
}),
expected: struct {
desiredResources corev1.ResourceList
minResources corev1.ResourceList
}{
desiredResources: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse("9"),
corev1.ResourceMemory: resource.MustParse("1700Mi"),
},
minResources: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse("7"),
corev1.ResourceMemory: resource.MustParse("1300Mi"),
},
},
},
{
name: "Head pod with two worker group with different resources",
cluster: createRayClusterTemplate(headStruct, []struct {
replicas *int32
minReplicas *int32
suspend *bool
cpu string
memory string
numOfHosts int32
}{
{
numOfHosts: 2,
replicas: ptr.To[int32](2),
minReplicas: ptr.To[int32](1),
cpu: "2",
memory: "100Mi",
suspend: nil,
},
{
numOfHosts: 1,
replicas: ptr.To[int32](3),
minReplicas: ptr.To[int32](0),
cpu: "1",
memory: "200Mi",
suspend: nil,
},
}),
expected: struct {
desiredResources corev1.ResourceList
minResources corev1.ResourceList
}{
desiredResources: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse("12"),
corev1.ResourceMemory: resource.MustParse("1100Mi"),
},
minResources: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse("5"),
corev1.ResourceMemory: resource.MustParse("300Mi"),
},
},
},
{
name: "Head pod with suspended worker group",
cluster: createRayClusterTemplate(headStruct, []struct {
replicas *int32
minReplicas *int32
suspend *bool
cpu string
memory string
numOfHosts int32
}{
{
numOfHosts: 2,
replicas: ptr.To[int32](3),
minReplicas: ptr.To[int32](0),
cpu: "2",
memory: "200Mi",
suspend: ptr.To(true),
},
}),
expected: struct {
desiredResources corev1.ResourceList
minResources corev1.ResourceList
}{
desiredResources: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse("1"),
corev1.ResourceMemory: resource.MustParse("100Mi"),
},
minResources: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse("1"),
corev1.ResourceMemory: resource.MustParse("100Mi"),
},
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
deepCopyCluster := tt.cluster.DeepCopy()
desiredResource := CalculateDesiredResources(tt.cluster)
assert.Equal(t, tt.expected.desiredResources.Cpu().String(), desiredResource.Cpu().String())
assert.Equal(t, tt.expected.desiredResources.Memory().String(), desiredResource.Memory().String())
assert.Equal(t, deepCopyCluster, tt.cluster)
minResource := CalculateMinResources(tt.cluster)
assert.Equal(t, tt.expected.minResources.Cpu().String(), minResource.Cpu().String())
assert.Equal(t, tt.expected.minResources.Memory().String(), minResource.Memory().String())
assert.Equal(t, deepCopyCluster, tt.cluster)
})
}
}
// helper function to return a Gateway object with GatewayStatus Conditions for testing.
func makeGatewayWithCondition(accepted bool, programmed bool) *gwv1.Gateway {
var conditions []metav1.Condition
if accepted {
conditions = append(conditions, metav1.Condition{
Type: string(gwv1.GatewayConditionAccepted),
Status: metav1.ConditionTrue,
})
}
if programmed {
conditions = append(conditions, metav1.Condition{
Type: string(gwv1.GatewayConditionProgrammed),
Status: metav1.ConditionTrue,
})
}
return &gwv1.Gateway{
Status: gwv1.GatewayStatus{
Conditions: conditions,
},
}
}
func TestIsGatewayReady(t *testing.T) {
tests := []struct {
gateway *gwv1.Gateway
name string
expected bool
}{
{
name: "missing Gateway instance",
gateway: nil,
expected: false,
},
{
name: "Gateway created with Programmed condition only",
gateway: makeGatewayWithCondition(false, true),
expected: false,
},
{
name: "Gateway created with Accepted condition only",
gateway: makeGatewayWithCondition(true, false),
expected: false,
},
{
name: "Gateway created with both Accepted and Programmed conditions",
gateway: makeGatewayWithCondition(true, true),
expected: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.expected, IsGatewayReady(tt.gateway))
})
}
}
// helper function to return a HTTPRoute with HTTPRouteStatus for testing
func makeHTTPRouteWithParentRef(
parentRefName string,
namespace string,
accepted bool,
resolvedRefs bool,
) *gwv1.HTTPRoute {
var acceptedStatus, resolvedRefsStatus metav1.ConditionStatus
if accepted {
acceptedStatus = metav1.ConditionTrue
} else {
acceptedStatus = metav1.ConditionFalse
}
if resolvedRefs {
resolvedRefsStatus = metav1.ConditionTrue
} else {
resolvedRefsStatus = metav1.ConditionFalse
}
return &gwv1.HTTPRoute{
Status: gwv1.HTTPRouteStatus{
RouteStatus: gwv1.RouteStatus{
Parents: []gwv1.RouteParentStatus{
{
ParentRef: gwv1.ParentReference{
Name: gwv1.ObjectName(parentRefName),
Namespace: ptr.To(gwv1.Namespace(namespace)),
},
Conditions: []metav1.Condition{
{
Type: string(gwv1.RouteConditionAccepted),
Status: acceptedStatus,
},
{
Type: string(gwv1.RouteConditionResolvedRefs),
Status: resolvedRefsStatus,
},
},
},
},
},
},
}
}
func TestIsHTTPRouteReady(t *testing.T) {
gateway := &gwv1.Gateway{
ObjectMeta: metav1.ObjectMeta{Name: "test-gateway", Namespace: "test-ns"},
}
tests := []struct {
httpRoute *gwv1.HTTPRoute
name string
expected bool
}{
{
name: "missing HTTPRoute",
httpRoute: nil,
expected: false,
},
{
name: "ParentRef does not match",
httpRoute: makeHTTPRouteWithParentRef("not-a-match", "other-test-ns", true, true),
expected: false,
},
{
name: "matching ParentRef with Accepted condition but without ResolvedRefs",
httpRoute: makeHTTPRouteWithParentRef("test-gateway", "test-ns", true, false),
expected: false,
},
{
name: "matching ParentRef with ResolvedRefs but without Accepted",
httpRoute: makeHTTPRouteWithParentRef("test-gateway", "test-ns", false, true),
expected: false,
},
{
name: "ready HTTPRoute with all required conditions",
httpRoute: makeHTTPRouteWithParentRef("test-gateway", "test-ns", true, true),
expected: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.expected, IsHTTPRouteReady(gateway, tt.httpRoute))
})
}
}
func TestIsIncrementalUpgradeEnabled(t *testing.T) {
tests := []struct {
spec *rayv1.RayServiceSpec
name string
featureEnabled bool
expected bool
}{
{
name: "missing UpgradeStrategy Type",
spec: &rayv1.RayServiceSpec{},
featureEnabled: true,
expected: false,
},
{
name: "UpgradeStrategy Type is NewClusterWithIncrementalUpgrade but feature disabled",
spec: &rayv1.RayServiceSpec{
UpgradeStrategy: &rayv1.RayServiceUpgradeStrategy{
Type: ptr.To(rayv1.NewClusterWithIncrementalUpgrade),
},
},
featureEnabled: false,
expected: false,
},
{
name: "UpgradeStrategy Type is NewClusterWithIncrementalUpgrade and feature enabled",
spec: &rayv1.RayServiceSpec{
UpgradeStrategy: &rayv1.RayServiceUpgradeStrategy{
Type: ptr.To(rayv1.NewClusterWithIncrementalUpgrade),
},
},
featureEnabled: true,
expected: true,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
features.SetFeatureGateDuringTest(t, features.RayServiceIncrementalUpgrade, tc.featureEnabled)
assert.Equal(t, tc.expected, IsIncrementalUpgradeEnabled(tc.spec))
})
}
}
func TestGetRayServiceClusterUpgradeOptions(t *testing.T) {
upgradeOptions := &rayv1.ClusterUpgradeOptions{GatewayClassName: "gateway-class"}
tests := []struct {
rayServiceSpec *rayv1.RayServiceSpec
expectedOptions *rayv1.ClusterUpgradeOptions
name string
}{
{
name: "RayServiceSpec is nil, return nil ClusterUpgradeOptions",
rayServiceSpec: nil,
expectedOptions: nil,
},
{
name: "UpgradeStrategy is nil, return nil ClusterUpgradeOptions",
rayServiceSpec: &rayv1.RayServiceSpec{},
expectedOptions: nil,
},
{
name: "Valid ClusterUpgradeOptions",
rayServiceSpec: &rayv1.RayServiceSpec{
UpgradeStrategy: &rayv1.RayServiceUpgradeStrategy{
ClusterUpgradeOptions: upgradeOptions,
},
},
expectedOptions: upgradeOptions,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
actualOptions := GetRayServiceClusterUpgradeOptions(tt.rayServiceSpec)
assert.Equal(t, tt.expectedOptions, actualOptions)
})
}
}
func TestGetContainerCommand(t *testing.T) {
tests := []struct {
name string
additionalOptions []string
expected []string
enableLoginShell bool
}{
{
name: "enable login shell is false",
enableLoginShell: false,
additionalOptions: []string{},
expected: []string{"/bin/bash", "-c", "--"},
},
{
name: "enable login shell is true",
enableLoginShell: true,
additionalOptions: []string{},
expected: []string{"/bin/bash", "-cl", "--"},
},
{
name: "enable login shell is false and additional options is not empty",
enableLoginShell: false,
additionalOptions: []string{"e"},
expected: []string{"/bin/bash", "-ce", "--"},
},
{
name: "enable login shell is true and additional options is not empty",
enableLoginShell: true,
additionalOptions: []string{"e"},
expected: []string{"/bin/bash", "-cel", "--"},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
if test.enableLoginShell {
os.Setenv("ENABLE_LOGIN_SHELL", "true")
defer os.Unsetenv("ENABLE_LOGIN_SHELL")
}
assert.Equal(t, test.expected, GetContainerCommand(test.additionalOptions))
})
}
}
func TestGetWeightsFromHTTPRoute(t *testing.T) {
activeClusterName := "rayservice-active"
pendingClusterName := "rayservice-pending"
// Helper to create a RayService with specified cluster names in its status.
makeRayService := func(activeName, pendingName string) *rayv1.RayService {
return &rayv1.RayService{
Status: rayv1.RayServiceStatuses{
ActiveServiceStatus: rayv1.RayServiceStatus{RayClusterName: activeName},
PendingServiceStatus: rayv1.RayServiceStatus{RayClusterName: pendingName},
},
}
}
// Helper to create an HTTPRoute with specified backend weights.
makeHTTPRoute := func(activeWeight, pendingWeight *int32) *gwv1.HTTPRoute {
backends := []gwv1.HTTPBackendRef{}
if activeWeight != nil {
backends = append(backends, gwv1.HTTPBackendRef{
BackendRef: gwv1.BackendRef{
BackendObjectReference: gwv1.BackendObjectReference{Name: gwv1.ObjectName(GenerateServeServiceName(activeClusterName))},
Weight: activeWeight,
},
})
}
if pendingWeight != nil {
backends = append(backends, gwv1.HTTPBackendRef{
BackendRef: gwv1.BackendRef{
BackendObjectReference: gwv1.BackendObjectReference{Name: gwv1.ObjectName(GenerateServeServiceName(pendingClusterName))},
Weight: pendingWeight,
},
})
}
return &gwv1.HTTPRoute{
Spec: gwv1.HTTPRouteSpec{
Rules: []gwv1.HTTPRouteRule{{BackendRefs: backends}},
},
}
}
tests := []struct {
httpRoute *gwv1.HTTPRoute
rayService *rayv1.RayService
name string
expectedActive int32
expectedPending int32
}{
{
name: "No HTTPRoute, return defaults for both weights",
httpRoute: nil,
rayService: makeRayService(activeClusterName, ""),
expectedActive: -1,
expectedPending: -1,
},
{
name: "HTTPRoute with missing backends, return defaults for both weights",
httpRoute: &gwv1.HTTPRoute{Spec: gwv1.HTTPRouteSpec{Rules: []gwv1.HTTPRouteRule{{}}}},
rayService: makeRayService(activeClusterName, pendingClusterName),
expectedActive: -1,
expectedPending: -1,
},
{
name: "Valid weights returned for both active and pending clusters",
httpRoute: makeHTTPRoute(ptr.To(int32(80)), ptr.To(int32(20))),
rayService: makeRayService(activeClusterName, pendingClusterName),
expectedActive: 80,
expectedPending: 20,
},
{
name: "Valid HTTPRoute with only active cluster backend",
httpRoute: makeHTTPRoute(ptr.To(int32(100)), nil),
rayService: makeRayService(activeClusterName, ""),
expectedActive: 100,
expectedPending: -1,
},
{
name: "Valid HTTPRoute with only pending cluster backend",
httpRoute: makeHTTPRoute(nil, ptr.To(int32(100))),
rayService: makeRayService("", pendingClusterName),
expectedActive: -1,
expectedPending: 100,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
active, pending := GetWeightsFromHTTPRoute(tt.httpRoute, tt.rayService)
assert.Equal(t, tt.expectedActive, active, "Active weight mismatch")
assert.Equal(t, tt.expectedPending, pending, "Pending weight mismatch")
})
}
}