mirror of
https://github.com/hanzo-ml/kuberay.git
synced 2026-10-30 19:48:47 +00:00
* [RayService] Apply Suspend to RayCluster only at creation time Propagate RayService.Spec.RayClusterSpec.Suspend onto the RayCluster only when the RayCluster is first created. After the RayCluster exists, its Suspend is delegated to Kueue: hash comparisons ignore Suspend, and modifyRayCluster preserves the existing cluster's Suspend instead of overwriting it with the RayService spec. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Add top-level Spec.Suspend to tear down owned resources When Spec.Suspend is true, the RayService controller deletes every Kubernetes resource it owns (RayClusters, head/serve Services, and Gateway/HTTPRoute when the RayServiceIncrementalUpgrade gate is on) and reports the lifecycle through two new conditions, Suspending and Suspended. The transition is atomic: the first reconcile commits Suspending=True together with the reset of ActiveServiceStatus, PendingServiceStatus, NumServeEndpoints, and ServiceStatus in a single Status update; deletion runs on the next reconcile once that commit is durable, so an errored or interrupted attempt is always resumed. Flipping Spec.Suspend back to false removes the Suspended condition and the regular reconcile recreates the resources. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Remove suspend handler unit tests The suspend behavior is exercised end-to-end on a kind cluster, so the handler-level unit tests are removed for now. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Sync top-level Spec.Suspend into Helm chart CRD Run `make helm` so the helm-chart/kuberay-operator/crds copy of the RayService CRD matches config/crd/bases after adding Spec.Suspend. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Drop create-only Suspend propagation unit tests These tests reached into private helpers and duplicated coverage already exercised by the zero-downtime upgrade + Suspend e2e walkthrough; drop them to reduce coupling to internal structure. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Add e2e tests for Spec.Suspend lifecycle Covers four scenarios: - Suspend a Running service then resume it. - Atomic suspend: deletion completes even if Spec.Suspend is flipped back to false mid-suspend; service then exits Suspended and serves traffic again. - Service created with Spec.Suspend=true: never spins up resources, reaches Suspended directly, comes up normally on resume. - Suspend during a zero-downtime upgrade: both active and pending clusters are deleted; resuming applies the upgraded spec. The atomic case surfaced a bug where the controller would stay in Suspended forever if Spec.Suspend had been flipped to false before the transition landed: after persisting Suspended=True we returned ctrl.Result{} with no requeue, and the status-only update did not re-trigger the watch predicate. Fix by requeuing after the transition so the next reconcile observes Spec.Suspend and exits. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Fix lint: trailing newline and import grouping - Remove trailing blank line at end of rayservice_controller_unit_test.go. - Group corev1 with the other k8s.io imports in rayservice_suspend_test.go so goimports leaves it alone. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Use ExecPodCmdWithError directly in suspend e2e Drop the curlRayServiceFruitWithError wrapper; it duplicated what ExecPodCmdWithError already does. Build the curl command inline at the single call site. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Harden TestRayServiceSuspendAtomic against transient state Stop asserting that Suspended=True is observed during the atomic flow. That condition is only persisted for ~2s (one requeue interval) after deletion completes if Spec.Suspend has already been flipped to false, which made the test vulnerable to scheduling jitter under load. Instead record the original RayCluster name before suspending and assert (1) that cluster is deleted and (2) the eventually-Ready RayService is backed by a different RayCluster. This proves atomic completion more directly: the underlying cluster was actually torn down and recreated. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Mirror RayJob's atomic-suspend test in TestRayServiceSuspendAtomic Pin the underlying RayCluster with a synthetic finalizer so deletion cannot complete, then flip Spec.Suspend back and forth while asserting via Consistently that Suspending stays True. This matches the "RayJob suspend operation shoud be atomic" pattern in rayjob_controller_suspended_test.go and exercises the atomicity property directly instead of inferring it from cluster recreation. After removing the finalizer the test still verifies that the original RayCluster is deleted and a different RayCluster eventually backs the Ready RayService. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Drop redundant comment above handleSuspend call The function name and its own doc comment already convey what the call site does; the inline comment was duplicative. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Drop redundant create-time Suspend comment The comment block above `clusterSpec := rayService.Spec.RayClusterSpec.DeepCopy()` in constructRayClusterForRayService restated what is already documented on rayClusterSpecForHashing and modifyRayCluster. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Split suspend e2e into its own target + fix curl hang The full ./test/e2erayservice suite was nearly exhausting the 30m Go-test timeout, and TestRayServiceSuspendResume was failing because a single curl attempt hung on TCP retransmits past TestTimeoutShort — Eventually couldn't retry. Two fixes: - Split the suspend tests into their own Make target (test-e2e-rayservice-suspend) and Buildkite job; the existing test-e2e-rayservice / rayservice job runs with `-skip Suspend`. Each job gets its own 30m budget. - Add --connect-timeout 3 --max-time 5 to the resumed-service curl so each attempt fails fast and Eventually can actually iterate. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Drop Gateway/HTTPRoute deletion from suspend handler Suspend tear-down should only touch the resources that exist outside the incremental-upgrade feature. Remove the gated Gateway/HTTPRoute branch in deleteRayServiceOwnedResources, the new FailedToDeleteGateway and FailedToDeleteHTTPRoute event types, and drop Gateway/HTTPRoute mentions from condition messages and the Spec.Suspend doc. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * sync Signed-off-by: Kai-Hsun Chen <[email protected]> * [RayService] Retain suspend conditions and switch to suspendIsOperative Two related polish-ups to handleSuspend: 1. Replace meta.RemoveStatusCondition calls for RayServiceSuspending and RayServiceSuspended with setCondition(False, ...). Per the K8s API convention, absent conditions are interpreted as Unknown — but the controller does know these are False at each former-removal site (reaching Suspended, exiting Suspended). Setting False preserves lastTransitionTime, reason, and message, and keeps kube_*_status_condition timeseries from gapping. Add a new RayServiceResumed reason constant for the Suspended=False transition when Spec.Suspend is flipped back to false. 2. Drop the handled bool return from handleSuspend. Reconcile now short-circuits via a new suspendIsOperative helper that reads the Suspending / Suspended conditions handleSuspend just staged. Those two conditions are already the source of truth for the suspend state machine, so the bool was a redundant cache. Reduces the signature from (bool, ctrl.Result, error) to (ctrl.Result, error). Add two e2e assertions in TestRayServiceSuspendResume that catch the condition-removal regression directly: after suspend completes, Suspending must remain as False/SuspendComplete; after resume, Suspended must remain as False/RayServiceResumed. The existing IsRayServiceSuspended-based assertions cannot distinguish absent from False, so this is the regression guard. Verified end-to-end against kind-kueue-rayservice with all four suspend e2e tests (SuspendResume, SuspendAtomic, CreatedSuspended, SuspendDuringUpgrade) passing. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Retain Upgrade/Rollback conditions through suspend lifecycle Extend the prior commit's "retain conditions as False instead of removing" treatment to UpgradeInProgress and RollbackInProgress, which were still meta.RemoveStatusCondition'd when entering Suspending. Two changes in handleSuspend: 1. When entering Suspending, set Upgrade/RollbackInProgress to False with reason SuspendInProgress and a tense-neutral message ("No upgrade in progress.") instead of removing them. Neutral wording stays accurate at the Suspended terminal state too. 2. When transitioning to Suspended, re-stamp Upgrade/RollbackInProgress with reason SuspendComplete to match the existing re-stamp of Ready at that point. After this, every condition in the Suspended terminal state consistently shows reason SuspendComplete, which reads cleanly in kubectl describe. Extend the existing Suspended-state e2e assertion in TestRayServiceSuspendResume to loop over Suspending / UpgradeInProgress / RollbackInProgress so the new contract is regression-guarded by the same mechanism that already guards Suspending. Verified end-to-end against kind-kueue-rayservice; all four suspend e2e tests pass, including TestRayServiceSuspendDuringUpgrade which exercises the non-trivial case where UpgradeInProgress was actually True before suspend was triggered. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Signed-off-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
317 lines
10 KiB
Go
317 lines
10 KiB
Go
package support
|
|
|
|
import (
|
|
"errors"
|
|
"strings"
|
|
|
|
. "github.com/onsi/gomega"
|
|
"github.com/onsi/gomega/format"
|
|
"github.com/onsi/gomega/types"
|
|
corev1 "k8s.io/api/core/v1"
|
|
"k8s.io/apimachinery/pkg/api/meta"
|
|
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
|
gwv1 "sigs.k8s.io/gateway-api/apis/v1"
|
|
|
|
rayv1 "github.com/ray-project/kuberay/ray-operator/apis/ray/v1"
|
|
"github.com/ray-project/kuberay/ray-operator/controllers/ray/common"
|
|
"github.com/ray-project/kuberay/ray-operator/controllers/ray/utils"
|
|
)
|
|
|
|
func RayCronJob(t Test, namespace, name string) func() (*rayv1.RayCronJob, error) {
|
|
return func() (*rayv1.RayCronJob, error) {
|
|
return GetRayCronJob(t, namespace, name)
|
|
}
|
|
}
|
|
|
|
func GetRayCronJob(t Test, namespace, name string) (*rayv1.RayCronJob, error) {
|
|
return t.Client().Ray().RayV1().RayCronJobs(namespace).Get(t.Ctx(), name, metav1.GetOptions{})
|
|
}
|
|
|
|
func RayJob(t Test, namespace, name string) func() (*rayv1.RayJob, error) {
|
|
return func() (*rayv1.RayJob, error) {
|
|
return GetRayJob(t, namespace, name)
|
|
}
|
|
}
|
|
|
|
func GetRayJob(t Test, namespace, name string) (*rayv1.RayJob, error) {
|
|
return t.Client().Ray().RayV1().RayJobs(namespace).Get(t.Ctx(), name, metav1.GetOptions{})
|
|
}
|
|
|
|
func RayJobStatus(job *rayv1.RayJob) rayv1.JobStatus {
|
|
return job.Status.JobStatus
|
|
}
|
|
|
|
func RayJobDeploymentStatus(job *rayv1.RayJob) rayv1.JobDeploymentStatus {
|
|
return job.Status.JobDeploymentStatus
|
|
}
|
|
|
|
func RayJobManagedBy(job *rayv1.RayJob) *string {
|
|
return job.Spec.ManagedBy
|
|
}
|
|
|
|
func RayJobReason(job *rayv1.RayJob) rayv1.JobFailedReason {
|
|
return job.Status.Reason
|
|
}
|
|
|
|
func RayJobFailed(job *rayv1.RayJob) int32 {
|
|
if job.Status.Failed == nil {
|
|
return 0
|
|
}
|
|
return *job.Status.Failed
|
|
}
|
|
|
|
func RayJobSucceeded(job *rayv1.RayJob) int32 {
|
|
if job.Status.Succeeded == nil {
|
|
return 0
|
|
}
|
|
return *job.Status.Succeeded
|
|
}
|
|
|
|
func RayJobClusterName(job *rayv1.RayJob) string {
|
|
return job.Status.RayClusterName
|
|
}
|
|
|
|
func RayCluster(t Test, namespace, name string) func() (*rayv1.RayCluster, error) {
|
|
return func() (*rayv1.RayCluster, error) {
|
|
return GetRayCluster(t, namespace, name)
|
|
}
|
|
}
|
|
|
|
func GetRayCluster(t Test, namespace, name string) (*rayv1.RayCluster, error) {
|
|
return t.Client().Ray().RayV1().RayClusters(namespace).Get(t.Ctx(), name, metav1.GetOptions{})
|
|
}
|
|
|
|
func RayClusterState(cluster *rayv1.RayCluster) rayv1.ClusterState {
|
|
return cluster.Status.State
|
|
}
|
|
|
|
func StatusCondition(condType rayv1.RayClusterConditionType) func(*rayv1.RayCluster) metav1.Condition {
|
|
return func(cluster *rayv1.RayCluster) metav1.Condition {
|
|
if cluster != nil {
|
|
for _, cond := range cluster.Status.Conditions {
|
|
if cond.Type == string(condType) {
|
|
return cond
|
|
}
|
|
}
|
|
}
|
|
return metav1.Condition{}
|
|
}
|
|
}
|
|
|
|
type ConditionMatcher struct {
|
|
expected metav1.Condition
|
|
}
|
|
|
|
func (c *ConditionMatcher) Match(actual any) (success bool, err error) {
|
|
if actual == nil {
|
|
return false, errors.New("<actual> should be a metav1.Condition but it is nil")
|
|
}
|
|
a, ok := actual.(metav1.Condition)
|
|
if !ok {
|
|
return false, errors.New("<actual> should be a metav1.Condition")
|
|
}
|
|
messageMatch := true
|
|
if c.expected.Message != "" {
|
|
messageMatch = strings.Contains(a.Message, c.expected.Message)
|
|
}
|
|
return a.Reason == c.expected.Reason && a.Status == c.expected.Status && messageMatch, nil
|
|
}
|
|
|
|
func (c *ConditionMatcher) FailureMessage(actual any) (message string) {
|
|
a := actual.(metav1.Condition)
|
|
return format.Message(a, "to equal", c.expected)
|
|
}
|
|
|
|
func (c *ConditionMatcher) NegatedFailureMessage(actual any) (message string) {
|
|
a := actual.(metav1.Condition)
|
|
return format.Message(a, "not to equal", c.expected)
|
|
}
|
|
|
|
func MatchCondition(status metav1.ConditionStatus, reason string) types.GomegaMatcher {
|
|
return &ConditionMatcher{expected: metav1.Condition{Status: status, Reason: reason}}
|
|
}
|
|
|
|
func MatchConditionContainsMessage(status metav1.ConditionStatus, reason string, message string) types.GomegaMatcher {
|
|
return &ConditionMatcher{expected: metav1.Condition{Status: status, Reason: reason, Message: message}}
|
|
}
|
|
|
|
func RayClusterDesiredWorkerReplicas(cluster *rayv1.RayCluster) int32 {
|
|
return cluster.Status.DesiredWorkerReplicas
|
|
}
|
|
|
|
func HeadPod(t Test, rayCluster *rayv1.RayCluster) func() (*corev1.Pod, error) {
|
|
return func() (*corev1.Pod, error) {
|
|
return GetHeadPod(t, rayCluster)
|
|
}
|
|
}
|
|
|
|
func GetHeadPod(t Test, rayCluster *rayv1.RayCluster) (*corev1.Pod, error) {
|
|
pods, err := t.Client().Core().CoreV1().Pods(rayCluster.Namespace).List(
|
|
t.Ctx(),
|
|
common.RayClusterHeadPodsAssociationOptions(rayCluster).ToMetaV1ListOptions(),
|
|
)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if len(pods.Items) != 1 {
|
|
return nil, errors.New("number of head pods is not 1")
|
|
}
|
|
return &pods.Items[0], nil
|
|
}
|
|
|
|
func WorkerPods(t Test, rayCluster *rayv1.RayCluster) func() ([]corev1.Pod, error) {
|
|
return func() ([]corev1.Pod, error) {
|
|
return GetWorkerPods(t, rayCluster)
|
|
}
|
|
}
|
|
|
|
func GetWorkerPods(t Test, rayCluster *rayv1.RayCluster) ([]corev1.Pod, error) {
|
|
pods, err := t.Client().Core().CoreV1().Pods(rayCluster.Namespace).List(
|
|
t.Ctx(),
|
|
common.RayClusterWorkerPodsAssociationOptions(rayCluster).ToMetaV1ListOptions(),
|
|
)
|
|
if pods == nil {
|
|
return nil, err
|
|
}
|
|
return pods.Items, err
|
|
}
|
|
|
|
func GetAllPods(t Test, rayCluster *rayv1.RayCluster) ([]corev1.Pod, error) {
|
|
pods, err := t.Client().Core().CoreV1().Pods(rayCluster.Namespace).List(
|
|
t.Ctx(),
|
|
common.RayClusterAllPodsAssociationOptions(rayCluster).ToMetaV1ListOptions(),
|
|
)
|
|
if pods == nil {
|
|
return nil, err
|
|
}
|
|
return pods.Items, err
|
|
}
|
|
|
|
func GetGroupPods(t Test, rayCluster *rayv1.RayCluster, group string) ([]corev1.Pod, error) {
|
|
t.T().Helper()
|
|
pods, err := t.Client().Core().CoreV1().Pods(rayCluster.Namespace).List(
|
|
t.Ctx(),
|
|
common.RayClusterGroupPodsAssociationOptions(rayCluster, group).ToMetaV1ListOptions(),
|
|
)
|
|
return pods.Items, err
|
|
}
|
|
|
|
func GroupPods(t Test, rayCluster *rayv1.RayCluster, group string) func() ([]corev1.Pod, error) {
|
|
return func() ([]corev1.Pod, error) {
|
|
return GetGroupPods(t, rayCluster, group)
|
|
}
|
|
}
|
|
|
|
func RayClusterManagedBy(rayCluster *rayv1.RayCluster) *string {
|
|
return rayCluster.Spec.ManagedBy
|
|
}
|
|
|
|
func RayServiceManagedBy(rayService *rayv1.RayService) *string {
|
|
return rayService.Spec.ManagedBy
|
|
}
|
|
|
|
func GetRayService(t Test, namespace, name string) (*rayv1.RayService, error) {
|
|
return t.Client().Ray().RayV1().RayServices(namespace).Get(t.Ctx(), name, metav1.GetOptions{})
|
|
}
|
|
|
|
func RayService(t Test, namespace, name string) func() (*rayv1.RayService, error) {
|
|
return func() (*rayv1.RayService, error) {
|
|
return GetRayService(t, namespace, name)
|
|
}
|
|
}
|
|
|
|
func RayServiceStatus(service *rayv1.RayService) rayv1.ServiceStatus {
|
|
return service.Status.ServiceStatus
|
|
}
|
|
|
|
func IsRayServiceReady(service *rayv1.RayService) bool {
|
|
return meta.IsStatusConditionTrue(service.Status.Conditions, string(rayv1.RayServiceReady))
|
|
}
|
|
|
|
func IsRayServiceUpgrading(service *rayv1.RayService) bool {
|
|
return meta.IsStatusConditionTrue(service.Status.Conditions, string(rayv1.UpgradeInProgress))
|
|
}
|
|
|
|
func IsRayServiceRollingBack(service *rayv1.RayService) bool {
|
|
return meta.IsStatusConditionTrue(service.Status.Conditions, string(rayv1.RollbackInProgress))
|
|
}
|
|
|
|
func IsRayServiceSuspending(service *rayv1.RayService) bool {
|
|
return meta.IsStatusConditionTrue(service.Status.Conditions, string(rayv1.RayServiceSuspending))
|
|
}
|
|
|
|
func IsRayServiceSuspended(service *rayv1.RayService) bool {
|
|
return meta.IsStatusConditionTrue(service.Status.Conditions, string(rayv1.RayServiceSuspended))
|
|
}
|
|
|
|
func RayServicesNumEndPoints(service *rayv1.RayService) int32 {
|
|
return service.Status.NumServeEndpoints
|
|
}
|
|
|
|
func GetRayClusterWorkerGroupReplicaSum(cluster *rayv1.RayCluster) int32 {
|
|
var replicas int32
|
|
for _, workerGroup := range cluster.Spec.WorkerGroupSpecs {
|
|
replicas += *workerGroup.Replicas
|
|
}
|
|
return replicas
|
|
}
|
|
|
|
func GetHTTPRoute(t Test, namespace, name string) (*gwv1.HTTPRoute, error) {
|
|
return t.Client().Gateway().GatewayV1().HTTPRoutes(namespace).Get(t.Ctx(), name, metav1.GetOptions{})
|
|
}
|
|
|
|
func HTTPRoute(t Test, namespace, name string) func() (*gwv1.HTTPRoute, error) {
|
|
return func() (*gwv1.HTTPRoute, error) {
|
|
return GetHTTPRoute(t, namespace, name)
|
|
}
|
|
}
|
|
|
|
func GetGateway(t Test, namespace, name string) (*gwv1.Gateway, error) {
|
|
return t.Client().Gateway().GatewayV1().Gateways(namespace).Get(t.Ctx(), name, metav1.GetOptions{})
|
|
}
|
|
|
|
func Gateway(t Test, namespace, name string) func() (*gwv1.Gateway, error) {
|
|
return func() (*gwv1.Gateway, error) {
|
|
return GetGateway(t, namespace, name)
|
|
}
|
|
}
|
|
|
|
// VerifyContainerAuthTokenEnvVars verifies that the specified container has the correct auth token environment variables.
|
|
// This is a common helper function used across all auth-related E2E tests.
|
|
func VerifyContainerAuthTokenEnvVars(t Test, rayCluster *rayv1.RayCluster, container *corev1.Container) {
|
|
t.T().Helper()
|
|
g := NewWithT(t.T())
|
|
|
|
// Verify RAY_AUTH_MODE environment variable
|
|
var rayAuthModeEnvVar *corev1.EnvVar
|
|
for _, envVar := range container.Env {
|
|
if envVar.Name == utils.RAY_AUTH_MODE_ENV_VAR {
|
|
rayAuthModeEnvVar = &envVar
|
|
break
|
|
}
|
|
}
|
|
g.Expect(rayAuthModeEnvVar).NotTo(BeNil(),
|
|
"RAY_AUTH_MODE environment variable should be set in container %s", container.Name)
|
|
g.Expect(rayAuthModeEnvVar.Value).To(Equal(string(rayv1.AuthModeToken)),
|
|
"RAY_AUTH_MODE should be %s in container %s", rayv1.AuthModeToken, container.Name)
|
|
|
|
// Verify RAY_AUTH_TOKEN environment variable
|
|
var rayAuthTokenEnvVar *corev1.EnvVar
|
|
for _, envVar := range container.Env {
|
|
if envVar.Name == utils.RAY_AUTH_TOKEN_ENV_VAR {
|
|
rayAuthTokenEnvVar = &envVar
|
|
break
|
|
}
|
|
}
|
|
g.Expect(rayAuthTokenEnvVar).NotTo(BeNil(),
|
|
"RAY_AUTH_TOKEN environment variable should be set for AuthModeToken in container %s", container.Name)
|
|
g.Expect(rayAuthTokenEnvVar.ValueFrom).NotTo(BeNil(),
|
|
"RAY_AUTH_TOKEN should be populated from a secret in container %s", container.Name)
|
|
g.Expect(rayAuthTokenEnvVar.ValueFrom.SecretKeyRef).NotTo(BeNil(),
|
|
"RAY_AUTH_TOKEN should be populated from a secret key ref in container %s", container.Name)
|
|
g.Expect(rayAuthTokenEnvVar.ValueFrom.SecretKeyRef.Name).To(ContainSubstring(rayCluster.Name),
|
|
"Secret name should contain RayCluster name in container %s", container.Name)
|
|
g.Expect(rayAuthTokenEnvVar.ValueFrom.SecretKeyRef.Key).To(Equal(utils.RAY_AUTH_TOKEN_SECRET_KEY),
|
|
"Secret key should be %s in container %s", utils.RAY_AUTH_TOKEN_SECRET_KEY, container.Name)
|
|
}
|