Files
520159ca30 [RayService][Kueue] Support top-level Spec.Suspend for zero-downtime upgrade (#4841)
* [RayService] Apply Suspend to RayCluster only at creation time

Propagate RayService.Spec.RayClusterSpec.Suspend onto the RayCluster only
when the RayCluster is first created. After the RayCluster exists, its
Suspend is delegated to Kueue: hash comparisons ignore Suspend, and
modifyRayCluster preserves the existing cluster's Suspend instead of
overwriting it with the RayService spec.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Add top-level Spec.Suspend to tear down owned resources

When Spec.Suspend is true, the RayService controller deletes every
Kubernetes resource it owns (RayClusters, head/serve Services, and
Gateway/HTTPRoute when the RayServiceIncrementalUpgrade gate is on) and
reports the lifecycle through two new conditions, Suspending and
Suspended. The transition is atomic: the first reconcile commits
Suspending=True together with the reset of ActiveServiceStatus,
PendingServiceStatus, NumServeEndpoints, and ServiceStatus in a single
Status update; deletion runs on the next reconcile once that commit is
durable, so an errored or interrupted attempt is always resumed.
Flipping Spec.Suspend back to false removes the Suspended condition and
the regular reconcile recreates the resources.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Remove suspend handler unit tests

The suspend behavior is exercised end-to-end on a kind cluster, so the
handler-level unit tests are removed for now.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Sync top-level Spec.Suspend into Helm chart CRD

Run `make helm` so the helm-chart/kuberay-operator/crds copy of the
RayService CRD matches config/crd/bases after adding Spec.Suspend.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Drop create-only Suspend propagation unit tests

These tests reached into private helpers and duplicated coverage already
exercised by the zero-downtime upgrade + Suspend e2e walkthrough; drop
them to reduce coupling to internal structure.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Add e2e tests for Spec.Suspend lifecycle

Covers four scenarios:
- Suspend a Running service then resume it.
- Atomic suspend: deletion completes even if Spec.Suspend is flipped
  back to false mid-suspend; service then exits Suspended and serves
  traffic again.
- Service created with Spec.Suspend=true: never spins up resources,
  reaches Suspended directly, comes up normally on resume.
- Suspend during a zero-downtime upgrade: both active and pending
  clusters are deleted; resuming applies the upgraded spec.

The atomic case surfaced a bug where the controller would stay in
Suspended forever if Spec.Suspend had been flipped to false before the
transition landed: after persisting Suspended=True we returned
ctrl.Result{} with no requeue, and the status-only update did not
re-trigger the watch predicate. Fix by requeuing after the transition so
the next reconcile observes Spec.Suspend and exits.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Fix lint: trailing newline and import grouping

- Remove trailing blank line at end of rayservice_controller_unit_test.go.
- Group corev1 with the other k8s.io imports in
  rayservice_suspend_test.go so goimports leaves it alone.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Use ExecPodCmdWithError directly in suspend e2e

Drop the curlRayServiceFruitWithError wrapper; it duplicated what
ExecPodCmdWithError already does. Build the curl command inline at the
single call site.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Harden TestRayServiceSuspendAtomic against transient state

Stop asserting that Suspended=True is observed during the atomic flow.
That condition is only persisted for ~2s (one requeue interval) after
deletion completes if Spec.Suspend has already been flipped to false,
which made the test vulnerable to scheduling jitter under load.

Instead record the original RayCluster name before suspending and
assert (1) that cluster is deleted and (2) the eventually-Ready
RayService is backed by a different RayCluster. This proves atomic
completion more directly: the underlying cluster was actually torn down
and recreated.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Mirror RayJob's atomic-suspend test in TestRayServiceSuspendAtomic

Pin the underlying RayCluster with a synthetic finalizer so deletion
cannot complete, then flip Spec.Suspend back and forth while asserting
via Consistently that Suspending stays True. This matches the
"RayJob suspend operation shoud be atomic" pattern in
rayjob_controller_suspended_test.go and exercises the atomicity
property directly instead of inferring it from cluster recreation.

After removing the finalizer the test still verifies that the original
RayCluster is deleted and a different RayCluster eventually backs the
Ready RayService.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Drop redundant comment above handleSuspend call

The function name and its own doc comment already convey what the call
site does; the inline comment was duplicative.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Drop redundant create-time Suspend comment

The comment block above `clusterSpec := rayService.Spec.RayClusterSpec.DeepCopy()`
in constructRayClusterForRayService restated what is already documented on
rayClusterSpecForHashing and modifyRayCluster.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Split suspend e2e into its own target + fix curl hang

The full ./test/e2erayservice suite was nearly exhausting the 30m
Go-test timeout, and TestRayServiceSuspendResume was failing because a
single curl attempt hung on TCP retransmits past TestTimeoutShort —
Eventually couldn't retry. Two fixes:

- Split the suspend tests into their own Make target
  (test-e2e-rayservice-suspend) and Buildkite job; the existing
  test-e2e-rayservice / rayservice job runs with `-skip Suspend`. Each
  job gets its own 30m budget.
- Add --connect-timeout 3 --max-time 5 to the resumed-service curl so
  each attempt fails fast and Eventually can actually iterate.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Drop Gateway/HTTPRoute deletion from suspend handler

Suspend tear-down should only touch the resources that exist outside
the incremental-upgrade feature. Remove the gated Gateway/HTTPRoute
branch in deleteRayServiceOwnedResources, the new FailedToDeleteGateway
and FailedToDeleteHTTPRoute event types, and drop Gateway/HTTPRoute
mentions from condition messages and the Spec.Suspend doc.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* sync

Signed-off-by: Kai-Hsun Chen <[email protected]>

* [RayService] Retain suspend conditions and switch to suspendIsOperative

Two related polish-ups to handleSuspend:

1. Replace meta.RemoveStatusCondition calls for RayServiceSuspending and
   RayServiceSuspended with setCondition(False, ...). Per the K8s API
   convention, absent conditions are interpreted as Unknown — but the
   controller does know these are False at each former-removal site
   (reaching Suspended, exiting Suspended). Setting False preserves
   lastTransitionTime, reason, and message, and keeps
   kube_*_status_condition timeseries from gapping. Add a new
   RayServiceResumed reason constant for the Suspended=False transition
   when Spec.Suspend is flipped back to false.

2. Drop the handled bool return from handleSuspend. Reconcile now
   short-circuits via a new suspendIsOperative helper that reads the
   Suspending / Suspended conditions handleSuspend just staged. Those
   two conditions are already the source of truth for the suspend state
   machine, so the bool was a redundant cache. Reduces the signature
   from (bool, ctrl.Result, error) to (ctrl.Result, error).

Add two e2e assertions in TestRayServiceSuspendResume that catch the
condition-removal regression directly: after suspend completes,
Suspending must remain as False/SuspendComplete; after resume, Suspended
must remain as False/RayServiceResumed. The existing
IsRayServiceSuspended-based assertions cannot distinguish absent from
False, so this is the regression guard.

Verified end-to-end against kind-kueue-rayservice with all four suspend
e2e tests (SuspendResume, SuspendAtomic, CreatedSuspended,
SuspendDuringUpgrade) passing.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Retain Upgrade/Rollback conditions through suspend lifecycle

Extend the prior commit's "retain conditions as False instead of
removing" treatment to UpgradeInProgress and RollbackInProgress, which
were still meta.RemoveStatusCondition'd when entering Suspending.

Two changes in handleSuspend:

1. When entering Suspending, set Upgrade/RollbackInProgress to False
   with reason SuspendInProgress and a tense-neutral message ("No upgrade
   in progress.") instead of removing them. Neutral wording stays
   accurate at the Suspended terminal state too.

2. When transitioning to Suspended, re-stamp Upgrade/RollbackInProgress
   with reason SuspendComplete to match the existing re-stamp of Ready
   at that point. After this, every condition in the Suspended terminal
   state consistently shows reason SuspendComplete, which reads cleanly
   in kubectl describe.

Extend the existing Suspended-state e2e assertion in
TestRayServiceSuspendResume to loop over Suspending / UpgradeInProgress
/ RollbackInProgress so the new contract is regression-guarded by the
same mechanism that already guards Suspending.

Verified end-to-end against kind-kueue-rayservice; all four suspend
e2e tests pass, including TestRayServiceSuspendDuringUpgrade which
exercises the non-trivial case where UpgradeInProgress was actually
True before suspend was triggered.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

---------

Signed-off-by: Kai-Hsun Chen <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-27 17:13:57 -07:00

317 lines
10 KiB
Go

package support
import (
"errors"
"strings"
. "github.com/onsi/gomega"
"github.com/onsi/gomega/format"
"github.com/onsi/gomega/types"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/meta"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
gwv1 "sigs.k8s.io/gateway-api/apis/v1"
rayv1 "github.com/ray-project/kuberay/ray-operator/apis/ray/v1"
"github.com/ray-project/kuberay/ray-operator/controllers/ray/common"
"github.com/ray-project/kuberay/ray-operator/controllers/ray/utils"
)
func RayCronJob(t Test, namespace, name string) func() (*rayv1.RayCronJob, error) {
return func() (*rayv1.RayCronJob, error) {
return GetRayCronJob(t, namespace, name)
}
}
func GetRayCronJob(t Test, namespace, name string) (*rayv1.RayCronJob, error) {
return t.Client().Ray().RayV1().RayCronJobs(namespace).Get(t.Ctx(), name, metav1.GetOptions{})
}
func RayJob(t Test, namespace, name string) func() (*rayv1.RayJob, error) {
return func() (*rayv1.RayJob, error) {
return GetRayJob(t, namespace, name)
}
}
func GetRayJob(t Test, namespace, name string) (*rayv1.RayJob, error) {
return t.Client().Ray().RayV1().RayJobs(namespace).Get(t.Ctx(), name, metav1.GetOptions{})
}
func RayJobStatus(job *rayv1.RayJob) rayv1.JobStatus {
return job.Status.JobStatus
}
func RayJobDeploymentStatus(job *rayv1.RayJob) rayv1.JobDeploymentStatus {
return job.Status.JobDeploymentStatus
}
func RayJobManagedBy(job *rayv1.RayJob) *string {
return job.Spec.ManagedBy
}
func RayJobReason(job *rayv1.RayJob) rayv1.JobFailedReason {
return job.Status.Reason
}
func RayJobFailed(job *rayv1.RayJob) int32 {
if job.Status.Failed == nil {
return 0
}
return *job.Status.Failed
}
func RayJobSucceeded(job *rayv1.RayJob) int32 {
if job.Status.Succeeded == nil {
return 0
}
return *job.Status.Succeeded
}
func RayJobClusterName(job *rayv1.RayJob) string {
return job.Status.RayClusterName
}
func RayCluster(t Test, namespace, name string) func() (*rayv1.RayCluster, error) {
return func() (*rayv1.RayCluster, error) {
return GetRayCluster(t, namespace, name)
}
}
func GetRayCluster(t Test, namespace, name string) (*rayv1.RayCluster, error) {
return t.Client().Ray().RayV1().RayClusters(namespace).Get(t.Ctx(), name, metav1.GetOptions{})
}
func RayClusterState(cluster *rayv1.RayCluster) rayv1.ClusterState {
return cluster.Status.State
}
func StatusCondition(condType rayv1.RayClusterConditionType) func(*rayv1.RayCluster) metav1.Condition {
return func(cluster *rayv1.RayCluster) metav1.Condition {
if cluster != nil {
for _, cond := range cluster.Status.Conditions {
if cond.Type == string(condType) {
return cond
}
}
}
return metav1.Condition{}
}
}
type ConditionMatcher struct {
expected metav1.Condition
}
func (c *ConditionMatcher) Match(actual any) (success bool, err error) {
if actual == nil {
return false, errors.New("<actual> should be a metav1.Condition but it is nil")
}
a, ok := actual.(metav1.Condition)
if !ok {
return false, errors.New("<actual> should be a metav1.Condition")
}
messageMatch := true
if c.expected.Message != "" {
messageMatch = strings.Contains(a.Message, c.expected.Message)
}
return a.Reason == c.expected.Reason && a.Status == c.expected.Status && messageMatch, nil
}
func (c *ConditionMatcher) FailureMessage(actual any) (message string) {
a := actual.(metav1.Condition)
return format.Message(a, "to equal", c.expected)
}
func (c *ConditionMatcher) NegatedFailureMessage(actual any) (message string) {
a := actual.(metav1.Condition)
return format.Message(a, "not to equal", c.expected)
}
func MatchCondition(status metav1.ConditionStatus, reason string) types.GomegaMatcher {
return &ConditionMatcher{expected: metav1.Condition{Status: status, Reason: reason}}
}
func MatchConditionContainsMessage(status metav1.ConditionStatus, reason string, message string) types.GomegaMatcher {
return &ConditionMatcher{expected: metav1.Condition{Status: status, Reason: reason, Message: message}}
}
func RayClusterDesiredWorkerReplicas(cluster *rayv1.RayCluster) int32 {
return cluster.Status.DesiredWorkerReplicas
}
func HeadPod(t Test, rayCluster *rayv1.RayCluster) func() (*corev1.Pod, error) {
return func() (*corev1.Pod, error) {
return GetHeadPod(t, rayCluster)
}
}
func GetHeadPod(t Test, rayCluster *rayv1.RayCluster) (*corev1.Pod, error) {
pods, err := t.Client().Core().CoreV1().Pods(rayCluster.Namespace).List(
t.Ctx(),
common.RayClusterHeadPodsAssociationOptions(rayCluster).ToMetaV1ListOptions(),
)
if err != nil {
return nil, err
}
if len(pods.Items) != 1 {
return nil, errors.New("number of head pods is not 1")
}
return &pods.Items[0], nil
}
func WorkerPods(t Test, rayCluster *rayv1.RayCluster) func() ([]corev1.Pod, error) {
return func() ([]corev1.Pod, error) {
return GetWorkerPods(t, rayCluster)
}
}
func GetWorkerPods(t Test, rayCluster *rayv1.RayCluster) ([]corev1.Pod, error) {
pods, err := t.Client().Core().CoreV1().Pods(rayCluster.Namespace).List(
t.Ctx(),
common.RayClusterWorkerPodsAssociationOptions(rayCluster).ToMetaV1ListOptions(),
)
if pods == nil {
return nil, err
}
return pods.Items, err
}
func GetAllPods(t Test, rayCluster *rayv1.RayCluster) ([]corev1.Pod, error) {
pods, err := t.Client().Core().CoreV1().Pods(rayCluster.Namespace).List(
t.Ctx(),
common.RayClusterAllPodsAssociationOptions(rayCluster).ToMetaV1ListOptions(),
)
if pods == nil {
return nil, err
}
return pods.Items, err
}
func GetGroupPods(t Test, rayCluster *rayv1.RayCluster, group string) ([]corev1.Pod, error) {
t.T().Helper()
pods, err := t.Client().Core().CoreV1().Pods(rayCluster.Namespace).List(
t.Ctx(),
common.RayClusterGroupPodsAssociationOptions(rayCluster, group).ToMetaV1ListOptions(),
)
return pods.Items, err
}
func GroupPods(t Test, rayCluster *rayv1.RayCluster, group string) func() ([]corev1.Pod, error) {
return func() ([]corev1.Pod, error) {
return GetGroupPods(t, rayCluster, group)
}
}
func RayClusterManagedBy(rayCluster *rayv1.RayCluster) *string {
return rayCluster.Spec.ManagedBy
}
func RayServiceManagedBy(rayService *rayv1.RayService) *string {
return rayService.Spec.ManagedBy
}
func GetRayService(t Test, namespace, name string) (*rayv1.RayService, error) {
return t.Client().Ray().RayV1().RayServices(namespace).Get(t.Ctx(), name, metav1.GetOptions{})
}
func RayService(t Test, namespace, name string) func() (*rayv1.RayService, error) {
return func() (*rayv1.RayService, error) {
return GetRayService(t, namespace, name)
}
}
func RayServiceStatus(service *rayv1.RayService) rayv1.ServiceStatus {
return service.Status.ServiceStatus
}
func IsRayServiceReady(service *rayv1.RayService) bool {
return meta.IsStatusConditionTrue(service.Status.Conditions, string(rayv1.RayServiceReady))
}
func IsRayServiceUpgrading(service *rayv1.RayService) bool {
return meta.IsStatusConditionTrue(service.Status.Conditions, string(rayv1.UpgradeInProgress))
}
func IsRayServiceRollingBack(service *rayv1.RayService) bool {
return meta.IsStatusConditionTrue(service.Status.Conditions, string(rayv1.RollbackInProgress))
}
func IsRayServiceSuspending(service *rayv1.RayService) bool {
return meta.IsStatusConditionTrue(service.Status.Conditions, string(rayv1.RayServiceSuspending))
}
func IsRayServiceSuspended(service *rayv1.RayService) bool {
return meta.IsStatusConditionTrue(service.Status.Conditions, string(rayv1.RayServiceSuspended))
}
func RayServicesNumEndPoints(service *rayv1.RayService) int32 {
return service.Status.NumServeEndpoints
}
func GetRayClusterWorkerGroupReplicaSum(cluster *rayv1.RayCluster) int32 {
var replicas int32
for _, workerGroup := range cluster.Spec.WorkerGroupSpecs {
replicas += *workerGroup.Replicas
}
return replicas
}
func GetHTTPRoute(t Test, namespace, name string) (*gwv1.HTTPRoute, error) {
return t.Client().Gateway().GatewayV1().HTTPRoutes(namespace).Get(t.Ctx(), name, metav1.GetOptions{})
}
func HTTPRoute(t Test, namespace, name string) func() (*gwv1.HTTPRoute, error) {
return func() (*gwv1.HTTPRoute, error) {
return GetHTTPRoute(t, namespace, name)
}
}
func GetGateway(t Test, namespace, name string) (*gwv1.Gateway, error) {
return t.Client().Gateway().GatewayV1().Gateways(namespace).Get(t.Ctx(), name, metav1.GetOptions{})
}
func Gateway(t Test, namespace, name string) func() (*gwv1.Gateway, error) {
return func() (*gwv1.Gateway, error) {
return GetGateway(t, namespace, name)
}
}
// VerifyContainerAuthTokenEnvVars verifies that the specified container has the correct auth token environment variables.
// This is a common helper function used across all auth-related E2E tests.
func VerifyContainerAuthTokenEnvVars(t Test, rayCluster *rayv1.RayCluster, container *corev1.Container) {
t.T().Helper()
g := NewWithT(t.T())
// Verify RAY_AUTH_MODE environment variable
var rayAuthModeEnvVar *corev1.EnvVar
for _, envVar := range container.Env {
if envVar.Name == utils.RAY_AUTH_MODE_ENV_VAR {
rayAuthModeEnvVar = &envVar
break
}
}
g.Expect(rayAuthModeEnvVar).NotTo(BeNil(),
"RAY_AUTH_MODE environment variable should be set in container %s", container.Name)
g.Expect(rayAuthModeEnvVar.Value).To(Equal(string(rayv1.AuthModeToken)),
"RAY_AUTH_MODE should be %s in container %s", rayv1.AuthModeToken, container.Name)
// Verify RAY_AUTH_TOKEN environment variable
var rayAuthTokenEnvVar *corev1.EnvVar
for _, envVar := range container.Env {
if envVar.Name == utils.RAY_AUTH_TOKEN_ENV_VAR {
rayAuthTokenEnvVar = &envVar
break
}
}
g.Expect(rayAuthTokenEnvVar).NotTo(BeNil(),
"RAY_AUTH_TOKEN environment variable should be set for AuthModeToken in container %s", container.Name)
g.Expect(rayAuthTokenEnvVar.ValueFrom).NotTo(BeNil(),
"RAY_AUTH_TOKEN should be populated from a secret in container %s", container.Name)
g.Expect(rayAuthTokenEnvVar.ValueFrom.SecretKeyRef).NotTo(BeNil(),
"RAY_AUTH_TOKEN should be populated from a secret key ref in container %s", container.Name)
g.Expect(rayAuthTokenEnvVar.ValueFrom.SecretKeyRef.Name).To(ContainSubstring(rayCluster.Name),
"Secret name should contain RayCluster name in container %s", container.Name)
g.Expect(rayAuthTokenEnvVar.ValueFrom.SecretKeyRef.Key).To(Equal(utils.RAY_AUTH_TOKEN_SECRET_KEY),
"Secret key should be %s in container %s", utils.RAY_AUTH_TOKEN_SECRET_KEY, container.Name)
}