mirror of
https://github.com/hanzo-ml/kuberay.git
synced 2026-10-30 19:48:47 +00:00
+10









Andrew Sy Kim
GitHub
Ping
Han-Ju Chen
Ryan O'Leary
Kai-Hsun Chen
Kavish
Neo Chien
cchung100m
JustinYeh
Jun-Hao Wan
Divyam Raj
Nary Yeh
Alima Azamat
Spencer Peterson
dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Rueian
Cheng-Yeh Chung
fscnick
Copilot
f68857e9ff
* [Bug] Sidecar mode shouldn't restart head pod when head pod is deleted (#4141) * [Bug] Sidecar mode shouldn't restart head pod when head pod is deleted Signed-off-by: 400Ping <[email protected]> * [Fix] Fix e2e error Signed-off-by: 400Ping <[email protected]> * [Fix] fix according to rueian's comment Signed-off-by: 400Ping <[email protected]> * [Chore] fix ci error Signed-off-by: 400Ping <[email protected]> * Update ray-operator/controllers/ray/raycluster_controller.go Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]> Signed-off-by: Ping <[email protected]> * Update ray-operator/controllers/ray/rayjob_controller.go Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]> Signed-off-by: Ping <[email protected]> * update Signed-off-by: Future-Outlier <[email protected]> * update Signed-off-by: Future-Outlier <[email protected]> * Trigger CI Signed-off-by: Future-Outlier <[email protected]> --------- Signed-off-by: 400Ping <[email protected]> Signed-off-by: Ping <[email protected]> Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]> * fix: dashboard build for kuberay 1.5.0 (#4161) Signed-off-by: Future-Outlier <[email protected]> * [Feature Enhancement] Set ordered replica index label to support multi-slice (#4163) * [Feature Enhancement] Set ordered replica index label to support multi-slice Signed-off-by: Ryan O'Leary <[email protected]> * rename replica-id -> replica-name Signed-off-by: Ryan O'Leary <[email protected]> * Separate replica index feature gate logic Signed-off-by: Ryan O'Leary <[email protected]> * remove index arg in createWorkerPod Signed-off-by: Ryan O'Leary <[email protected]> --------- Signed-off-by: Ryan O'Leary <[email protected]> * update stale feature gate comments (#4174) Signed-off-by: Andrew Sy Kim <[email protected]> * [RayCluster] Add more context why we don't recreate head Pod for RayJob (#4175) Signed-off-by: Kai-Hsun Chen <[email protected]> * feature: Remove empty resource list initialization. (#4168) Fixes #4142. * [Dockerfile] [KubeRay Dashboard]: Fix Dockerfile warnings (ENV format, CMD JSON args) (#4167) * [#4166] improvement: Fix Dockerfile warnings (ENV format, CMD JSON args) * extract the hostname from CMD Signed-off-by: Neo Chien <[email protected]> --------- Signed-off-by: Neo Chien <[email protected]> Co-authored-by: cchung100m <[email protected]> * [Fix] Resolve int32 overflow by having the calculation in int64 and c… (#4158) * [Fix] Resolve int32 overflow by having the calculation in int64 and cap it if the count is over math.MaxInt32 Signed-off-by: justinyeh1995 <[email protected]> * [Test] Add unit tests for CalculateReadyReplicas Signed-off-by: justinyeh1995 <[email protected]> * [Fix] Add a nosec comment to pass the Lint (pre-commit) test Signed-off-by: justinyeh1995 <[email protected]> * [Refactor] Add CapInt64ToInt32 to replace #nosec directives Signed-off-by: justinyeh1995 <[email protected]> * [Refactor] Rename function to SafeInt64ToInt32 and add a underflowing prevention (it also help pass the lint test) Signed-off-by: justinyeh1995 <[email protected]> * [Refactor] Remove the early return as SafeInt64ToInt32 handles the int32 overflow and underflow checking. Signed-off-by: justinyeh1995 <[email protected]> --------- Signed-off-by: justinyeh1995 <[email protected]> * Add RayService incremental upgrade sample for guide (#4164) Signed-off-by: Ryan O'Leary <[email protected]> * Edit RayCluster example config for label selectors (#4151) Signed-off-by: Ryan O'Leary <[email protected]> * [RayJob] update light weight submitter image from quay.io (#4181) Signed-off-by: Future-Outlier <[email protected]> * [flaky] RayJob fails when head Pod is deleted when job is running (#4182) Signed-off-by: Future-Outlier <[email protected]> * [CI] Pin Docker api version to avoid API version mismatch (#4188) Signed-off-by: win5923 <[email protected]> * Make replicas configurable for kuberay-operator #4180 (#4195) * Make replicas configurable for kuberay-operator #4180 * Make replicas configurable for kuberay-operator #4180 * [Fix] rayjob update raycluster status (#4192) * feat: check if raycluster status update in rayjob * test: e2e test to check the rayjob raycluster status update * fix: dashboard http client tests discovered and passing (#4173) Signed-off-by: alimaazamat <[email protected]> * [RayJob] Lift cluster status while initializing (#4191) Signed-off-by: Spencer Peterson <[email protected]> * [RayJob] Remove updateJobStatus call (#4198) Fast follow to #4191 Signed-off-by: Spencer Peterson <[email protected]> * Add support for Ray token auth (#4179) * Add support for Ray token auth Signed-off-by: Andrew Sy Kim <[email protected]> * add e2e test for Ray cluster auth Signed-off-by: Andrew Sy Kim <[email protected]> * address nits from Ruiean Signed-off-by: Andrew Sy Kim <[email protected]> * update RAY_auth_mode -> RAY_AUTH_MODE Signed-off-by: Andrew Sy Kim <[email protected]> * configure auth for Ray autoscaler Signed-off-by: Andrew Sy Kim <[email protected]> --------- Signed-off-by: Andrew Sy Kim <[email protected]> * Bump js-yaml from 4.1.0 to 4.1.1 in /dashboard (#4194) Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 4.1.0 to 4.1.1. - [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md) - [Commits](https://github.com/nodeca/js-yaml/compare/4.1.0...4.1.1) --- updated-dependencies: - dependency-name: js-yaml dependency-version: 4.1.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * update minimum Ray version required for token authentication to 2.52.0 (#4201) * update minimum Ray version required for token authentication to 2.52.0 Signed-off-by: Andrew Sy Kim <[email protected]> * update RayCluster auth e2e test to use Ray v2.52 Signed-off-by: Andrew Sy Kim <[email protected]> --------- Signed-off-by: Andrew Sy Kim <[email protected]> * add samples for RayCluster token auth (#4200) Signed-off-by: Andrew Sy Kim <[email protected]> * update (#4208) Signed-off-by: Future-Outlier <[email protected]> * [RayJob] Add token authentication support for All mode (#4210) * dashboard client authentication support Signed-off-by: Future-Outlier <[email protected]> * support rayjob Signed-off-by: Future-Outlier <[email protected]> * update to fix api serverr err Signed-off-by: Future-Outlier <[email protected]> * update Signed-off-by: Future-Outlier <[email protected]> * updarte Signed-off-by: Future-Outlier <[email protected]> * Rayjob sidecar mode auth token mode support Signed-off-by: Future-Outlier <[email protected]> * RayJob support k8s job mode Signed-off-by: Future-Outlier <[email protected]> * update Signed-off-by: Future-Outlier <[email protected]> * update Signed-off-by: Future-Outlier <[email protected]> * update Signed-off-by: Future-Outlier <[email protected]> * Address Andrew's advice Signed-off-by: Future-Outlier <[email protected]> * add todo x-ray-authorization comments Signed-off-by: Future-Outlier <[email protected]> --------- Signed-off-by: Future-Outlier <[email protected]> * [RayCluster] Enable Secret informer watch/list and remove unused RBAC verbs (#4202) * Add authentication secret reconciliation support Signed-off-by: Future-Outlier <[email protected]> * update Signed-off-by: Future-Outlier <[email protected]> * update Signed-off-by: Future-Outlier <[email protected]> * fix flaky test Signed-off-by: Future-Outlier <[email protected]> * remove test fix Signed-off-by: Rueian <[email protected]> --------- Signed-off-by: Future-Outlier <[email protected]> Signed-off-by: Rueian <[email protected]> Co-authored-by: Rueian <[email protected]> * [APIServer][Docs] Add user guide for retry behavior & configuration (#4144) * [Docs] Add the draft description about feature intro, configurations, and usecases Signed-off-by: justinyeh1995 <[email protected]> * [Fix] Update the retry walk-through Signed-off-by: justinyeh1995 <[email protected]> * [Doc] rewrite the first 2 sections Signed-off-by: justinyeh1995 <[email protected]> * [Doc] Revise documentation wording and add Observing Retry Behavior section Signed-off-by: justinyeh1995 <[email protected]> * [Fix] fix linting issue by running pre-commit run berfore commiting Signed-off-by: justinyeh1995 <[email protected]> * [Fix] fix linting errors in the Markdown linting Signed-off-by: justinyeh1995 <[email protected]> * [Fix] Clean up the math equation Signed-off-by: justinyeh1995 <[email protected]> * Update the math formula of Backoff calculation. Co-authored-by: Nary Yeh <[email protected]> Signed-off-by: JustinYeh <[email protected]> * [Fix] Explicitly mentioned exponential backoff and removed the customization parts Signed-off-by: justinyeh1995 <[email protected]> * [Docs] Clarify naming by replacing “APIServer” with “KubeRay APIServer” Co-authored-by: Cheng-Yeh Chung <[email protected]> Signed-off-by: JustinYeh <[email protected]> * [Docs] Rename retry-configuration.md to retry-behavior.md for accuracy Signed-off-by: justinyeh1995 <[email protected]> * Update Title to KubeRay APIServer Retry Behavior Co-authored-by: Cheng-Yeh Chung <[email protected]> Signed-off-by: JustinYeh <[email protected]> * [Docs] Add a note about the limitation of retry configuration Signed-off-by: justinyeh1995 <[email protected]> --------- Signed-off-by: justinyeh1995 <[email protected]> Signed-off-by: JustinYeh <[email protected]> Co-authored-by: Nary Yeh <[email protected]> Co-authored-by: Cheng-Yeh Chung <[email protected]> * Support X-Ray-Authorization fallback header for accepting auth token via proxy (#4213) * Support X-Ray-Authorization fallback header for accepting auth token in dashboard Signed-off-by: Future-Outlier <[email protected]> * remove todo comment Signed-off-by: Future-Outlier <[email protected]> --------- Signed-off-by: Future-Outlier <[email protected]> * [RayCluster] make auth token secret name consistency (#4216) Signed-off-by: fscnick <[email protected]> * [RayCluster] Status includes head containter status message (#4196) * [RayCluster] Status includes head containter status message Signed-off-by: Spencer Peterson <[email protected]> * lint Signed-off-by: Spencer Peterson <[email protected]> * [RayCluster] Containers not ready status reflects structured reason Signed-off-by: Spencer Peterson <[email protected]> * nit Signed-off-by: Spencer Peterson <[email protected]> --------- Signed-off-by: Spencer Peterson <[email protected]> * Remove erroneous call in applyServeTargetCapacity (#4212) Signed-off-by: Ryan O'Leary <[email protected]> * [RayJob] Add token authentication support for light weight job submitter (#4215) * [RayJob] light weight job submitter auth token support Signed-off-by: Future-Outlier <[email protected]> * X-Ray-Authorization Signed-off-by: Rueian <[email protected]> --------- Signed-off-by: Future-Outlier <[email protected]> Signed-off-by: Rueian <[email protected]> Co-authored-by: Rueian <[email protected]> * feat: kubectl ray get token command (#4218) * feat: kubectl ray get token command Signed-off-by: Rueian <[email protected]> * Update kubectl-plugin/pkg/cmd/get/get_token_test.go Co-authored-by: Copilot <[email protected]> Signed-off-by: Rueian <[email protected]> * Update kubectl-plugin/pkg/cmd/get/get_token.go Co-authored-by: Copilot <[email protected]> Signed-off-by: Rueian <[email protected]> * make sure the raycluster exists before getting the secret Signed-off-by: Rueian <[email protected]> * better ux Signed-off-by: Rueian <[email protected]> * Update kubectl-plugin/pkg/cmd/get/get_token.go Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]> Signed-off-by: Rueian <[email protected]> --------- Signed-off-by: Rueian <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]> --------- Signed-off-by: 400Ping <[email protected]> Signed-off-by: Ping <[email protected]> Signed-off-by: Future-Outlier <[email protected]> Signed-off-by: Ryan O'Leary <[email protected]> Signed-off-by: Andrew Sy Kim <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Neo Chien <[email protected]> Signed-off-by: justinyeh1995 <[email protected]> Signed-off-by: win5923 <[email protected]> Signed-off-by: alimaazamat <[email protected]> Signed-off-by: Spencer Peterson <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Rueian <[email protected]> Signed-off-by: JustinYeh <[email protected]> Signed-off-by: fscnick <[email protected]> Co-authored-by: Ping <[email protected]> Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]> Co-authored-by: Ryan O'Leary <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Kavish <[email protected]> Co-authored-by: Neo Chien <[email protected]> Co-authored-by: cchung100m <[email protected]> Co-authored-by: JustinYeh <[email protected]> Co-authored-by: Jun-Hao Wan <[email protected]> Co-authored-by: Divyam Raj <[email protected]> Co-authored-by: Nary Yeh <[email protected]> Co-authored-by: Alima Azamat <[email protected]> Co-authored-by: Spencer Peterson <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Rueian <[email protected]> Co-authored-by: Cheng-Yeh Chung <[email protected]> Co-authored-by: fscnick <[email protected]> Co-authored-by: Copilot <[email protected]>
597 lines
25 KiB
Go
597 lines
25 KiB
Go
package utils
|
|
|
|
import (
|
|
errstd "errors"
|
|
"fmt"
|
|
"strconv"
|
|
"strings"
|
|
"time"
|
|
|
|
corev1 "k8s.io/api/core/v1"
|
|
"k8s.io/apimachinery/pkg/api/meta"
|
|
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
|
"k8s.io/apimachinery/pkg/util/validation"
|
|
"k8s.io/apimachinery/pkg/util/version"
|
|
|
|
rayv1 "github.com/ray-project/kuberay/ray-operator/apis/ray/v1"
|
|
"github.com/ray-project/kuberay/ray-operator/controllers/ray/utils/dashboardclient"
|
|
"github.com/ray-project/kuberay/ray-operator/pkg/features"
|
|
)
|
|
|
|
func ValidateRayClusterStatus(instance *rayv1.RayCluster) error {
|
|
suspending := meta.IsStatusConditionTrue(instance.Status.Conditions, string(rayv1.RayClusterSuspending))
|
|
suspended := meta.IsStatusConditionTrue(instance.Status.Conditions, string(rayv1.RayClusterSuspended))
|
|
if suspending && suspended {
|
|
return errstd.New("invalid RayCluster State: rayv1.RayClusterSuspending and rayv1.RayClusterSuspended conditions should not be both true")
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func ValidateRayClusterMetadata(metadata metav1.ObjectMeta) error {
|
|
if len(metadata.Name) > MaxRayClusterNameLength {
|
|
return fmt.Errorf("RayCluster name should be no more than %d characters", MaxRayClusterNameLength)
|
|
}
|
|
if errs := validation.IsDNS1035Label(metadata.Name); len(errs) > 0 {
|
|
return fmt.Errorf("RayCluster name should be a valid DNS1035 label: %v", errs)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// validateRayGroupResources checks for conflicting resource definitions.
|
|
func validateRayGroupResources(groupName string, rayStartParams, resources map[string]string) error {
|
|
hasRayStartResources := rayStartParams["num-cpus"] != "" ||
|
|
rayStartParams["num-gpus"] != "" ||
|
|
rayStartParams["memory"] != "" ||
|
|
rayStartParams["resources"] != ""
|
|
if hasRayStartResources && len(resources) > 0 {
|
|
return fmt.Errorf("resource fields should not be set in both rayStartParams and Resources for %s group; please use only one", groupName)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// validateRayGroupLabels checks for invalid label definitions and correct label syntax.
|
|
func validateRayGroupLabels(groupName string, rayStartParams, labels map[string]string) error {
|
|
if _, ok := rayStartParams["labels"]; ok {
|
|
return fmt.Errorf("rayStartParams['labels'] is not supported for %s group; please use the top-level Labels field instead", groupName)
|
|
}
|
|
|
|
// Validate that labels conforms to Kubernetes label syntax.
|
|
var allErrs []string
|
|
for key, val := range labels {
|
|
// Validate the label key.
|
|
if errs := validation.IsQualifiedName(key); len(errs) > 0 {
|
|
for _, err := range errs {
|
|
allErrs = append(allErrs, fmt.Sprintf("invalid label key for %s group: '%s', error: %s", groupName, key, err))
|
|
}
|
|
}
|
|
|
|
// Validate the label value.
|
|
if errs := validation.IsValidLabelValue(val); len(errs) > 0 {
|
|
for _, err := range errs {
|
|
allErrs = append(allErrs, fmt.Sprintf("invalid label value for key '%s' in %s group: '%s', error: %s", key, groupName, val, err))
|
|
}
|
|
}
|
|
}
|
|
|
|
if len(allErrs) > 0 {
|
|
return fmt.Errorf("%s", strings.Join(allErrs, "; "))
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
// Validation for invalid Ray Cluster configurations.
|
|
func ValidateRayClusterSpec(spec *rayv1.RayClusterSpec, annotations map[string]string) error {
|
|
if len(spec.HeadGroupSpec.Template.Spec.Containers) == 0 {
|
|
return fmt.Errorf("headGroupSpec should have at least one container")
|
|
}
|
|
|
|
if err := validateRayGroupResources("Head", spec.HeadGroupSpec.RayStartParams, spec.HeadGroupSpec.Resources); err != nil {
|
|
return err
|
|
}
|
|
if err := validateRayGroupLabels("Head", spec.HeadGroupSpec.RayStartParams, spec.HeadGroupSpec.Labels); err != nil {
|
|
return err
|
|
}
|
|
|
|
for _, workerGroup := range spec.WorkerGroupSpecs {
|
|
if len(workerGroup.Template.Spec.Containers) == 0 {
|
|
return fmt.Errorf("workerGroupSpec should have at least one container")
|
|
}
|
|
|
|
if err := validateRayGroupResources(workerGroup.GroupName, workerGroup.RayStartParams, workerGroup.Resources); err != nil {
|
|
return err
|
|
}
|
|
if err := validateRayGroupLabels(workerGroup.GroupName, workerGroup.RayStartParams, workerGroup.Labels); err != nil {
|
|
return err
|
|
}
|
|
}
|
|
|
|
if annotations[RayFTEnabledAnnotationKey] != "" && spec.GcsFaultToleranceOptions != nil {
|
|
return fmt.Errorf("%s annotation and GcsFaultToleranceOptions are both set. "+
|
|
"Please use only GcsFaultToleranceOptions to configure GCS fault tolerance", RayFTEnabledAnnotationKey)
|
|
}
|
|
|
|
if !IsGCSFaultToleranceEnabled(spec, annotations) {
|
|
if EnvVarExists(RAY_REDIS_ADDRESS, spec.HeadGroupSpec.Template.Spec.Containers[RayContainerIndex].Env) {
|
|
return fmt.Errorf("%s is set which implicitly enables GCS fault tolerance, "+
|
|
"but GcsFaultToleranceOptions is not set. Please set GcsFaultToleranceOptions "+
|
|
"to enable GCS fault tolerance", RAY_REDIS_ADDRESS)
|
|
}
|
|
}
|
|
|
|
headContainer := spec.HeadGroupSpec.Template.Spec.Containers[RayContainerIndex]
|
|
if spec.GcsFaultToleranceOptions != nil {
|
|
if redisPassword := spec.HeadGroupSpec.RayStartParams["redis-password"]; redisPassword != "" {
|
|
return fmt.Errorf("cannot set `redis-password` in rayStartParams when " +
|
|
"GcsFaultToleranceOptions is enabled - use GcsFaultToleranceOptions.RedisPassword instead")
|
|
}
|
|
|
|
if EnvVarExists(REDIS_PASSWORD, headContainer.Env) {
|
|
return fmt.Errorf("cannot set `REDIS_PASSWORD` env var in head Pod when " +
|
|
"GcsFaultToleranceOptions is enabled - use GcsFaultToleranceOptions.RedisPassword instead")
|
|
}
|
|
|
|
if EnvVarExists(RAY_REDIS_ADDRESS, headContainer.Env) {
|
|
return fmt.Errorf("cannot set `RAY_REDIS_ADDRESS` env var in head Pod when " +
|
|
"GcsFaultToleranceOptions is enabled - use GcsFaultToleranceOptions.RedisAddress instead")
|
|
}
|
|
|
|
if annotations[RayExternalStorageNSAnnotationKey] != "" {
|
|
return fmt.Errorf("cannot set `ray.io/external-storage-namespace` annotation when " +
|
|
"GcsFaultToleranceOptions is enabled - use GcsFaultToleranceOptions.ExternalStorageNamespace instead")
|
|
}
|
|
}
|
|
if spec.HeadGroupSpec.RayStartParams["redis-username"] != "" || EnvVarExists(REDIS_USERNAME, headContainer.Env) {
|
|
return fmt.Errorf("cannot set redis username in rayStartParams or environment variables" +
|
|
" - use GcsFaultToleranceOptions.RedisUsername instead")
|
|
}
|
|
|
|
if !features.Enabled(features.RayJobDeletionPolicy) {
|
|
for _, workerGroup := range spec.WorkerGroupSpecs {
|
|
if workerGroup.Suspend != nil && *workerGroup.Suspend {
|
|
return fmt.Errorf("worker group %s can be suspended only when the RayJobDeletionPolicy feature gate is enabled", workerGroup.GroupName)
|
|
}
|
|
}
|
|
}
|
|
|
|
// Check if autoscaling is enabled once to avoid repeated calls
|
|
isAutoscalingEnabled := IsAutoscalingEnabled(spec)
|
|
|
|
// Validate that RAY_enable_autoscaler_v2 environment variable is not set to "1" or "true" when autoscaler is disabled
|
|
if !isAutoscalingEnabled {
|
|
if envVar, exists := EnvVarByName(RAY_ENABLE_AUTOSCALER_V2, spec.HeadGroupSpec.Template.Spec.Containers[RayContainerIndex].Env); exists {
|
|
if envVar.Value == "1" || envVar.Value == "true" {
|
|
return fmt.Errorf("environment variable %s cannot be set to '%s' when enableInTreeAutoscaling is false. Please set enableInTreeAutoscaling: true to use autoscaler v2", RAY_ENABLE_AUTOSCALER_V2, envVar.Value)
|
|
}
|
|
}
|
|
}
|
|
|
|
if isAutoscalingEnabled {
|
|
for _, workerGroup := range spec.WorkerGroupSpecs {
|
|
if workerGroup.Suspend != nil && *workerGroup.Suspend {
|
|
// TODO (rueian): This can be supported in future Ray. We should check the RayVersion once we know the version.
|
|
return fmt.Errorf("worker group %s cannot be suspended with Autoscaler enabled", workerGroup.GroupName)
|
|
}
|
|
}
|
|
|
|
if spec.AutoscalerOptions != nil && spec.AutoscalerOptions.Version != nil && EnvVarExists(RAY_ENABLE_AUTOSCALER_V2, spec.HeadGroupSpec.Template.Spec.Containers[RayContainerIndex].Env) {
|
|
return fmt.Errorf("both .spec.autoscalerOptions.version and head Pod env var %s are set, please only use the former", RAY_ENABLE_AUTOSCALER_V2)
|
|
}
|
|
|
|
if IsAutoscalingV2Enabled(spec) {
|
|
if spec.HeadGroupSpec.Template.Spec.RestartPolicy != "" && spec.HeadGroupSpec.Template.Spec.RestartPolicy != corev1.RestartPolicyNever {
|
|
return fmt.Errorf("restartPolicy for head Pod should be Never or unset when using autoscaler V2")
|
|
}
|
|
|
|
for _, workerGroup := range spec.WorkerGroupSpecs {
|
|
if workerGroup.Template.Spec.RestartPolicy != "" && workerGroup.Template.Spec.RestartPolicy != corev1.RestartPolicyNever {
|
|
return fmt.Errorf("restartPolicy for worker group %s should be Never or unset when using autoscaler V2", workerGroup.GroupName)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
if IsAuthEnabled(spec) {
|
|
if spec.RayVersion == "" {
|
|
return fmt.Errorf("authOptions.mode is 'token' but RayVersion was not specified. Ray version 2.52.0 or later is required")
|
|
}
|
|
|
|
rayVersion, err := version.ParseGeneric(spec.RayVersion)
|
|
if err != nil {
|
|
return fmt.Errorf("authOptions.mode is 'token' but RayVersion format is invalid: %s, %w", spec.RayVersion, err)
|
|
}
|
|
|
|
// Require minimum Ray version 2.52.0
|
|
minVersion := version.MustParseGeneric("2.52.0")
|
|
if rayVersion.LessThan(minVersion) {
|
|
return fmt.Errorf("authOptions.mode is 'token' but minimum Ray version is 2.52.0, got %s", spec.RayVersion)
|
|
}
|
|
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
func ValidateRayJobStatus(rayJob *rayv1.RayJob) error {
|
|
if rayJob.Status.JobDeploymentStatus == rayv1.JobDeploymentStatusWaiting && rayJob.Spec.SubmissionMode != rayv1.InteractiveMode {
|
|
return fmt.Errorf("The RayJob status is invalid: JobDeploymentStatus cannot be `Waiting` when SubmissionMode is not InteractiveMode")
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func ValidateRayJobMetadata(metadata metav1.ObjectMeta) error {
|
|
if len(metadata.Name) > MaxRayJobNameLength {
|
|
return fmt.Errorf("The RayJob metadata is invalid: RayJob name should be no more than %d characters", MaxRayJobNameLength)
|
|
}
|
|
if errs := validation.IsDNS1035Label(metadata.Name); len(errs) > 0 {
|
|
return fmt.Errorf("The RayJob metadata is invalid: RayJob name should be a valid DNS1035 label: %v", errs)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func ValidateRayJobSpec(rayJob *rayv1.RayJob) error {
|
|
// KubeRay has some limitations for the suspend operation. The limitations are a subset of the limitations of
|
|
// Kueue (https://kueue.sigs.k8s.io/docs/tasks/run_rayjobs/#c-limitations). For example, KubeRay allows users
|
|
// to suspend a RayJob with autoscaling enabled, but Kueue doesn't.
|
|
if rayJob.Spec.Suspend && !rayJob.Spec.ShutdownAfterJobFinishes {
|
|
return fmt.Errorf("The RayJob spec is invalid: a RayJob with shutdownAfterJobFinishes set to false is not allowed to be suspended")
|
|
}
|
|
|
|
if rayJob.Spec.TTLSecondsAfterFinished < 0 {
|
|
return fmt.Errorf("The RayJob spec is invalid: TTLSecondsAfterFinished must be a non-negative integer")
|
|
}
|
|
|
|
// Validate TTL and deletion strategy together
|
|
if err := validateDeletionConfiguration(rayJob); err != nil {
|
|
return err
|
|
}
|
|
|
|
isClusterSelectorMode := len(rayJob.Spec.ClusterSelector) != 0
|
|
if rayJob.Spec.Suspend && isClusterSelectorMode {
|
|
return fmt.Errorf("The RayJob spec is invalid: the ClusterSelector mode doesn't support the suspend operation")
|
|
}
|
|
if rayJob.Spec.RayClusterSpec == nil && !isClusterSelectorMode {
|
|
return fmt.Errorf("The RayJob spec is invalid: one of RayClusterSpec or ClusterSelector must be set")
|
|
}
|
|
if isClusterSelectorMode {
|
|
clusterName := rayJob.Spec.ClusterSelector[RayJobClusterSelectorKey]
|
|
if len(clusterName) == 0 {
|
|
return fmt.Errorf("cluster name in ClusterSelector should not be empty")
|
|
}
|
|
if rayJob.Spec.SubmissionMode == rayv1.SidecarMode {
|
|
return fmt.Errorf("ClusterSelector is not supported in SidecarMode")
|
|
}
|
|
}
|
|
|
|
// InteractiveMode does not support backoffLimit > 1.
|
|
// When a RayJob fails (e.g., due to a missing script) and retries,
|
|
// spec.JobId remains set, causing the new job to incorrectly transition
|
|
// to Running instead of Waiting or Failed.
|
|
// After discussion, we decided to disallow retries in InteractiveMode
|
|
// to avoid ambiguous state handling and unintended behavior.
|
|
// https://github.com/ray-project/kuberay/issues/3525
|
|
if rayJob.Spec.SubmissionMode == rayv1.InteractiveMode && rayJob.Spec.BackoffLimit != nil && *rayJob.Spec.BackoffLimit > 0 {
|
|
return fmt.Errorf("The RayJob spec is invalid: BackoffLimit is incompatible with InteractiveMode")
|
|
}
|
|
|
|
if rayJob.Spec.SubmissionMode == rayv1.SidecarMode {
|
|
if rayJob.Spec.SubmitterPodTemplate != nil {
|
|
return fmt.Errorf("Currently, SidecarMode doesn't support SubmitterPodTemplate")
|
|
}
|
|
|
|
if rayJob.Spec.SubmitterConfig != nil {
|
|
return fmt.Errorf("Currently, SidecarMode doesn't support SubmitterConfig")
|
|
}
|
|
|
|
if rayJob.Spec.RayClusterSpec.HeadGroupSpec.Template.Spec.RestartPolicy != "" && rayJob.Spec.RayClusterSpec.HeadGroupSpec.Template.Spec.RestartPolicy != corev1.RestartPolicyNever {
|
|
return fmt.Errorf("restartPolicy for head Pod should be Never or unset when using SidecarMode")
|
|
}
|
|
}
|
|
|
|
if rayJob.Spec.RayClusterSpec != nil {
|
|
if err := ValidateRayClusterSpec(rayJob.Spec.RayClusterSpec, rayJob.Annotations); err != nil {
|
|
return fmt.Errorf("The RayJob spec is invalid: %w", err)
|
|
}
|
|
}
|
|
|
|
// Validate whether RuntimeEnvYAML is a valid YAML string. Note that this only checks its validity
|
|
// as a YAML string, not its adherence to the runtime environment schema.
|
|
if _, err := dashboardclient.UnmarshalRuntimeEnvYAML(rayJob.Spec.RuntimeEnvYAML); err != nil {
|
|
return err
|
|
}
|
|
if rayJob.Spec.ActiveDeadlineSeconds != nil && *rayJob.Spec.ActiveDeadlineSeconds <= 0 {
|
|
return fmt.Errorf("The RayJob spec is invalid: activeDeadlineSeconds must be a positive integer")
|
|
}
|
|
if rayJob.Spec.BackoffLimit != nil && *rayJob.Spec.BackoffLimit < 0 {
|
|
return fmt.Errorf("The RayJob spec is invalid: backoffLimit must be a positive integer")
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
func ValidateRayServiceMetadata(metadata metav1.ObjectMeta) error {
|
|
if len(metadata.Name) > MaxRayServiceNameLength {
|
|
return fmt.Errorf("RayService name should be no more than %d characters", MaxRayServiceNameLength)
|
|
}
|
|
if errs := validation.IsDNS1035Label(metadata.Name); len(errs) > 0 {
|
|
return fmt.Errorf("RayService name should be a valid DNS1035 label: %v", errs)
|
|
}
|
|
|
|
// Validate initializing timeout annotation if present
|
|
if err := validateInitializingTimeout(metadata.Annotations); err != nil {
|
|
return fmt.Errorf("RayService annotations is invalid: %w", err)
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
// validateInitializingTimeout validates the ray.io/initializing-timeout annotation if present.
|
|
// Accepts Go duration format (e.g., "5m", "1h") or integer seconds.
|
|
// Returns an error if the annotation is present but invalid.
|
|
func validateInitializingTimeout(annotations map[string]string) error {
|
|
if annotations == nil {
|
|
return nil
|
|
}
|
|
|
|
timeoutStr, exists := annotations[RayServiceInitializingTimeoutAnnotation]
|
|
if !exists || timeoutStr == "" {
|
|
return nil
|
|
}
|
|
|
|
// Try parsing as Go duration first (e.g., "30m", "1h")
|
|
if timeout, err := time.ParseDuration(timeoutStr); err == nil {
|
|
if timeout <= 0 {
|
|
return fmt.Errorf("annotation %s must be a positive duration, got: %s", RayServiceInitializingTimeoutAnnotation, timeoutStr)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// Try parsing as integer seconds
|
|
if seconds, err := strconv.Atoi(timeoutStr); err == nil {
|
|
if seconds <= 0 {
|
|
return fmt.Errorf("annotation %s must be a positive integer (seconds), got: %s", RayServiceInitializingTimeoutAnnotation, timeoutStr)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
return fmt.Errorf("annotation %s has invalid format: %s. Expected Go duration format (e.g., '5m', '1h') or positive integer seconds", RayServiceInitializingTimeoutAnnotation, timeoutStr)
|
|
}
|
|
|
|
func ValidateRayServiceSpec(rayService *rayv1.RayService) error {
|
|
if err := ValidateRayClusterSpec(&rayService.Spec.RayClusterSpec, rayService.Annotations); err != nil {
|
|
return err
|
|
}
|
|
|
|
if headSvc := rayService.Spec.RayClusterSpec.HeadGroupSpec.HeadService; headSvc != nil && headSvc.Name != "" {
|
|
return fmt.Errorf("spec.rayClusterConfig.headGroupSpec.headService.metadata.name should not be set")
|
|
}
|
|
|
|
// only NewClusterWithIncrementalUpgrade, NewCluster, and None are valid upgradeType
|
|
if rayService.Spec.UpgradeStrategy != nil &&
|
|
rayService.Spec.UpgradeStrategy.Type != nil &&
|
|
*rayService.Spec.UpgradeStrategy.Type != rayv1.None &&
|
|
*rayService.Spec.UpgradeStrategy.Type != rayv1.NewCluster &&
|
|
*rayService.Spec.UpgradeStrategy.Type != rayv1.NewClusterWithIncrementalUpgrade {
|
|
return fmt.Errorf("Spec.UpgradeStrategy.Type value %s is invalid, valid options are %s, %s, or %s", *rayService.Spec.UpgradeStrategy.Type, rayv1.NewClusterWithIncrementalUpgrade, rayv1.NewCluster, rayv1.None)
|
|
}
|
|
|
|
if rayService.Spec.RayClusterDeletionDelaySeconds != nil &&
|
|
*rayService.Spec.RayClusterDeletionDelaySeconds < 0 {
|
|
return fmt.Errorf("Spec.RayClusterDeletionDelaySeconds should be a non-negative integer, got %d", *rayService.Spec.RayClusterDeletionDelaySeconds)
|
|
}
|
|
|
|
// If type is NewClusterWithIncrementalUpgrade, validate the ClusterUpgradeOptions
|
|
if IsIncrementalUpgradeEnabled(&rayService.Spec) {
|
|
return ValidateClusterUpgradeOptions(rayService)
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
func ValidateClusterUpgradeOptions(rayService *rayv1.RayService) error {
|
|
if !IsAutoscalingEnabled(&rayService.Spec.RayClusterSpec) {
|
|
return fmt.Errorf("Ray Autoscaler is required for NewClusterWithIncrementalUpgrade")
|
|
}
|
|
|
|
options := rayService.Spec.UpgradeStrategy.ClusterUpgradeOptions
|
|
if options == nil {
|
|
return fmt.Errorf("ClusterUpgradeOptions are required for NewClusterWithIncrementalUpgrade")
|
|
}
|
|
|
|
// MaxSurgePercent defaults to 100% if unset.
|
|
if *options.MaxSurgePercent < 0 || *options.MaxSurgePercent > 100 {
|
|
return fmt.Errorf("maxSurgePercent must be between 0 and 100")
|
|
}
|
|
|
|
if options.StepSizePercent == nil || *options.StepSizePercent < 0 || *options.StepSizePercent > 100 {
|
|
return fmt.Errorf("stepSizePercent must be between 0 and 100")
|
|
}
|
|
|
|
if options.IntervalSeconds == nil || *options.IntervalSeconds <= 0 {
|
|
return fmt.Errorf("intervalSeconds must be greater than 0")
|
|
}
|
|
|
|
if options.GatewayClassName == "" {
|
|
return fmt.Errorf("gatewayClassName is required for NewClusterWithIncrementalUpgrade")
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
// validateDeletionConfiguration validates both deletion strategy and TTL configuration
|
|
func validateDeletionConfiguration(rayJob *rayv1.RayJob) error {
|
|
if !rayJob.Spec.ShutdownAfterJobFinishes && rayJob.Spec.TTLSecondsAfterFinished > 0 {
|
|
return fmt.Errorf("The RayJob spec is invalid: a RayJob with shutdownAfterJobFinishes set to false cannot have TTLSecondsAfterFinished")
|
|
}
|
|
|
|
// No strategy block: nothing else to validate.
|
|
if rayJob.Spec.DeletionStrategy == nil {
|
|
return nil
|
|
}
|
|
|
|
// Feature gate must be enabled for any strategy usage.
|
|
if !features.Enabled(features.RayJobDeletionPolicy) {
|
|
return fmt.Errorf("RayJobDeletionPolicy feature gate must be enabled to use DeletionStrategy")
|
|
}
|
|
|
|
legacyConfigured := rayJob.Spec.DeletionStrategy.OnSuccess != nil || rayJob.Spec.DeletionStrategy.OnFailure != nil
|
|
rulesConfigured := len(rayJob.Spec.DeletionStrategy.DeletionRules) > 0
|
|
|
|
// Mutual exclusivity: rules mode forbids shutdown & legacy. (TTL+rules is implicitly invalid because TTL requires shutdown.)
|
|
if rulesConfigured && rayJob.Spec.ShutdownAfterJobFinishes {
|
|
return fmt.Errorf("The RayJob spec is invalid: spec.shutdownAfterJobFinishes and spec.deletionStrategy.deletionRules are mutually exclusive")
|
|
}
|
|
if rulesConfigured && legacyConfigured {
|
|
return fmt.Errorf("The RayJob spec is invalid: Cannot use both legacy onSuccess/onFailure fields and deletionRules simultaneously")
|
|
}
|
|
|
|
// Detailed content validation
|
|
if legacyConfigured {
|
|
if err := validateLegacyDeletionPolicies(rayJob); err != nil {
|
|
return err
|
|
}
|
|
} else if rulesConfigured {
|
|
if err := validateDeletionRules(rayJob); err != nil {
|
|
return err
|
|
}
|
|
} else {
|
|
return fmt.Errorf("The RayJob spec is invalid: DeletionStrategy requires either BOTH onSuccess and onFailure, OR the deletionRules field (cannot be empty)")
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
// validateDeletionRules validates the deletion rules in the RayJob spec.
|
|
// It performs per-rule validations, checks for uniqueness, and ensures logical TTL consistency.
|
|
// Errors are collected and returned as a single aggregated error using errors.Join for better user feedback.
|
|
func validateDeletionRules(rayJob *rayv1.RayJob) error {
|
|
rules := rayJob.Spec.DeletionStrategy.DeletionRules
|
|
isClusterSelectorMode := len(rayJob.Spec.ClusterSelector) != 0
|
|
|
|
// Group TTLs by JobStatus for cross-rule validation and uniqueness checking.
|
|
rulesByStatus := make(map[rayv1.JobStatus]map[rayv1.DeletionPolicyType]int32)
|
|
var errs []error
|
|
|
|
// Single pass: Validate each rule individually and group for later consistency checks.
|
|
for i, rule := range rules {
|
|
// Validate TTL is non-negative.
|
|
if rule.Condition.TTLSeconds < 0 {
|
|
errs = append(errs, fmt.Errorf("deletionRules[%d]: TTLSeconds must be non-negative", i))
|
|
continue
|
|
}
|
|
|
|
// Contextual validations based on spec.
|
|
if isClusterSelectorMode && (rule.Policy == rayv1.DeleteCluster || rule.Policy == rayv1.DeleteWorkers) {
|
|
errs = append(errs, fmt.Errorf("deletionRules[%d]: DeletionPolicyType '%s' not supported when ClusterSelector is set", i, rule.Policy))
|
|
continue
|
|
}
|
|
if IsAutoscalingEnabled(rayJob.Spec.RayClusterSpec) && rule.Policy == rayv1.DeleteWorkers {
|
|
// TODO (rueian): Support in future Ray versions by checking RayVersion.
|
|
errs = append(errs, fmt.Errorf("deletionRules[%d]: DeletionPolicyType 'DeleteWorkers' not supported with autoscaling enabled", i))
|
|
continue
|
|
}
|
|
|
|
// Group valid rule for consistency check.
|
|
policyTTLs, ok := rulesByStatus[rule.Condition.JobStatus]
|
|
if !ok {
|
|
policyTTLs = make(map[rayv1.DeletionPolicyType]int32)
|
|
rulesByStatus[rule.Condition.JobStatus] = policyTTLs
|
|
}
|
|
|
|
// Check for uniqueness of (JobStatus, DeletionPolicyType) pair.
|
|
if _, exists := policyTTLs[rule.Policy]; exists {
|
|
errs = append(errs, fmt.Errorf("deletionRules[%d]: duplicate rule for DeletionPolicyType '%s' and JobStatus '%s'", i, rule.Policy, rule.Condition.JobStatus))
|
|
continue
|
|
}
|
|
|
|
policyTTLs[rule.Policy] = rule.Condition.TTLSeconds
|
|
}
|
|
|
|
// Second pass: Validate TTL consistency per JobStatus.
|
|
for status, policyTTLs := range rulesByStatus {
|
|
if err := validateTTLConsistency(policyTTLs, status); err != nil {
|
|
errs = append(errs, err)
|
|
}
|
|
}
|
|
|
|
return errstd.Join(errs...)
|
|
}
|
|
|
|
// validateTTLConsistency ensures TTLs follow the deletion hierarchy: Workers <= Cluster <= Self.
|
|
// (Lower TTL means deletes earlier.)
|
|
func validateTTLConsistency(policyTTLs map[rayv1.DeletionPolicyType]int32, status rayv1.JobStatus) error {
|
|
// Define the required deletion order. TTLs must be non-decreasing along this sequence.
|
|
deletionOrder := []rayv1.DeletionPolicyType{
|
|
rayv1.DeleteWorkers,
|
|
rayv1.DeleteCluster,
|
|
rayv1.DeleteSelf,
|
|
}
|
|
|
|
var prevPolicy rayv1.DeletionPolicyType
|
|
var prevTTL int32
|
|
var hasPrev bool
|
|
|
|
var errs []error
|
|
|
|
for _, policy := range deletionOrder {
|
|
ttl, exists := policyTTLs[policy]
|
|
if !exists {
|
|
continue
|
|
}
|
|
|
|
if hasPrev && ttl < prevTTL {
|
|
errs = append(errs, fmt.Errorf(
|
|
"for JobStatus '%s': %s TTL (%d) must be >= %s TTL (%d)",
|
|
status, policy, ttl, prevPolicy, prevTTL,
|
|
))
|
|
}
|
|
|
|
prevPolicy = policy
|
|
prevTTL = ttl
|
|
hasPrev = true
|
|
}
|
|
|
|
return errstd.Join(errs...)
|
|
}
|
|
|
|
// validateLegacyDeletionPolicies handles validation for the old `onSuccess` and `onFailure` fields.
|
|
func validateLegacyDeletionPolicies(rayJob *rayv1.RayJob) error {
|
|
isClusterSelectorMode := len(rayJob.Spec.ClusterSelector) != 0
|
|
|
|
// Both policies must be set if using the legacy API.
|
|
if rayJob.Spec.DeletionStrategy.OnSuccess == nil || rayJob.Spec.DeletionStrategy.OnFailure == nil {
|
|
return fmt.Errorf("both DeletionStrategy.OnSuccess and DeletionStrategy.OnFailure must be set when using the legacy deletion policy fields of DeletionStrategy")
|
|
}
|
|
|
|
// Validate that the Policy field is set within each policy.
|
|
onSuccessPolicy := rayJob.Spec.DeletionStrategy.OnSuccess
|
|
onFailurePolicy := rayJob.Spec.DeletionStrategy.OnFailure
|
|
|
|
if onSuccessPolicy.Policy == nil {
|
|
return fmt.Errorf("the DeletionPolicyType field of DeletionStrategy.OnSuccess cannot be unset when DeletionStrategy is enabled")
|
|
}
|
|
if onFailurePolicy.Policy == nil {
|
|
return fmt.Errorf("the DeletionPolicyType field of DeletionStrategy.OnFailure cannot be unset when DeletionStrategy is enabled")
|
|
}
|
|
|
|
if isClusterSelectorMode {
|
|
if *onSuccessPolicy.Policy == rayv1.DeleteCluster || *onSuccessPolicy.Policy == rayv1.DeleteWorkers {
|
|
return fmt.Errorf("the ClusterSelector mode doesn't support DeletionStrategy=%s on success", *onSuccessPolicy.Policy)
|
|
}
|
|
if *onFailurePolicy.Policy == rayv1.DeleteCluster || *onFailurePolicy.Policy == rayv1.DeleteWorkers {
|
|
return fmt.Errorf("the ClusterSelector mode doesn't support DeletionStrategy=%s on failure", *onFailurePolicy.Policy)
|
|
}
|
|
}
|
|
|
|
if (*onSuccessPolicy.Policy == rayv1.DeleteWorkers || *onFailurePolicy.Policy == rayv1.DeleteWorkers) && IsAutoscalingEnabled(rayJob.Spec.RayClusterSpec) {
|
|
// TODO (rueian): This can be supported in a future Ray version. We should check the RayVersion once we know it.
|
|
return fmt.Errorf("DeletionStrategy=DeleteWorkers currently does not support RayCluster with autoscaling enabled")
|
|
}
|
|
|
|
if rayJob.Spec.ShutdownAfterJobFinishes && (*onSuccessPolicy.Policy == rayv1.DeleteNone || *onFailurePolicy.Policy == rayv1.DeleteNone) {
|
|
return fmt.Errorf("The RayJob spec is invalid: shutdownAfterJobFinshes is set to 'true' while deletion policy is 'DeleteNone'")
|
|
}
|
|
|
|
return nil
|
|
}
|