Files
+10 f68857e9ff [release-1.5] Cherry-pick commits for v1.5.1 (#4214)
* [Bug] Sidecar mode shouldn't restart head pod when head pod is deleted (#4141)

* [Bug] Sidecar mode shouldn't restart head pod when head pod is deleted

Signed-off-by: 400Ping <[email protected]>

* [Fix] Fix e2e error

Signed-off-by: 400Ping <[email protected]>

* [Fix] fix according to rueian's comment

Signed-off-by: 400Ping <[email protected]>

* [Chore] fix ci error

Signed-off-by: 400Ping <[email protected]>

* Update ray-operator/controllers/ray/raycluster_controller.go

Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>
Signed-off-by: Ping <[email protected]>

* Update ray-operator/controllers/ray/rayjob_controller.go

Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>
Signed-off-by: Ping <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* Trigger CI

Signed-off-by: Future-Outlier <[email protected]>

---------

Signed-off-by: 400Ping <[email protected]>
Signed-off-by: Ping <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>

* fix: dashboard build for kuberay 1.5.0 (#4161)

Signed-off-by: Future-Outlier <[email protected]>

* [Feature Enhancement] Set ordered replica index label to support multi-slice (#4163)

* [Feature Enhancement] Set ordered replica index label to support multi-slice

Signed-off-by: Ryan O'Leary <[email protected]>

* rename replica-id -> replica-name

Signed-off-by: Ryan O'Leary <[email protected]>

* Separate replica index feature gate logic

Signed-off-by: Ryan O'Leary <[email protected]>

* remove index arg in createWorkerPod

Signed-off-by: Ryan O'Leary <[email protected]>

---------

Signed-off-by: Ryan O'Leary <[email protected]>

* update stale feature gate comments (#4174)

Signed-off-by: Andrew Sy Kim <[email protected]>

* [RayCluster] Add more context why we don't recreate head Pod for RayJob (#4175)

Signed-off-by: Kai-Hsun Chen <[email protected]>

* feature: Remove empty resource list initialization. (#4168)

Fixes #4142.

* [Dockerfile] [KubeRay Dashboard]: Fix Dockerfile warnings (ENV format, CMD JSON args) (#4167)

* [#4166] improvement: Fix Dockerfile warnings (ENV format, CMD JSON args)

* extract the hostname from CMD

Signed-off-by: Neo Chien <[email protected]>

---------

Signed-off-by: Neo Chien <[email protected]>
Co-authored-by: cchung100m <[email protected]>

* [Fix] Resolve int32 overflow by having the calculation in int64 and c… (#4158)

* [Fix] Resolve int32 overflow by having the calculation in int64 and cap it if the count is over math.MaxInt32

Signed-off-by: justinyeh1995 <[email protected]>

* [Test] Add unit tests for CalculateReadyReplicas

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] Add a nosec comment to pass the Lint (pre-commit) test

Signed-off-by: justinyeh1995 <[email protected]>

* [Refactor] Add CapInt64ToInt32 to replace #nosec directives

Signed-off-by: justinyeh1995 <[email protected]>

* [Refactor] Rename function to SafeInt64ToInt32 and add a underflowing prevention (it also help pass the lint test)

Signed-off-by: justinyeh1995 <[email protected]>

* [Refactor] Remove the early return as SafeInt64ToInt32 handles the int32 overflow and underflow checking.

Signed-off-by: justinyeh1995 <[email protected]>

---------

Signed-off-by: justinyeh1995 <[email protected]>

* Add RayService incremental upgrade sample for guide (#4164)

Signed-off-by: Ryan O'Leary <[email protected]>

* Edit RayCluster example config for label selectors (#4151)

Signed-off-by: Ryan O'Leary <[email protected]>

* [RayJob] update light weight submitter image from quay.io (#4181)

Signed-off-by: Future-Outlier <[email protected]>

* [flaky] RayJob fails when head Pod is deleted when job is running (#4182)

Signed-off-by: Future-Outlier <[email protected]>

* [CI] Pin Docker api version to avoid API version mismatch (#4188)

Signed-off-by: win5923 <[email protected]>

* Make replicas configurable for kuberay-operator #4180 (#4195)

* Make replicas configurable for kuberay-operator #4180

* Make replicas configurable for kuberay-operator #4180

* [Fix] rayjob update raycluster status (#4192)

* feat: check if raycluster status update in rayjob

* test: e2e test to check the rayjob raycluster status update

* fix: dashboard http client tests discovered and passing (#4173)

Signed-off-by: alimaazamat <[email protected]>

* [RayJob] Lift cluster status while initializing (#4191)

Signed-off-by: Spencer Peterson <[email protected]>

* [RayJob] Remove updateJobStatus call (#4198)

Fast follow to #4191

Signed-off-by: Spencer Peterson <[email protected]>

* Add support for Ray token auth (#4179)

* Add support for Ray token auth

Signed-off-by: Andrew Sy Kim <[email protected]>

* add e2e test for Ray cluster auth

Signed-off-by: Andrew Sy Kim <[email protected]>

* address nits from Ruiean

Signed-off-by: Andrew Sy Kim <[email protected]>

* update RAY_auth_mode -> RAY_AUTH_MODE

Signed-off-by: Andrew Sy Kim <[email protected]>

* configure auth for Ray autoscaler

Signed-off-by: Andrew Sy Kim <[email protected]>

---------

Signed-off-by: Andrew Sy Kim <[email protected]>

* Bump js-yaml from 4.1.0 to 4.1.1 in /dashboard (#4194)

Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 4.1.0 to 4.1.1.
- [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodeca/js-yaml/compare/4.1.0...4.1.1)

---
updated-dependencies:
- dependency-name: js-yaml
  dependency-version: 4.1.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update minimum Ray version required for token authentication to 2.52.0 (#4201)

* update minimum Ray version required for token authentication to 2.52.0

Signed-off-by: Andrew Sy Kim <[email protected]>

* update RayCluster auth e2e test to use Ray v2.52

Signed-off-by: Andrew Sy Kim <[email protected]>

---------

Signed-off-by: Andrew Sy Kim <[email protected]>

* add samples for RayCluster token auth (#4200)

Signed-off-by: Andrew Sy Kim <[email protected]>

* update (#4208)

Signed-off-by: Future-Outlier <[email protected]>

* [RayJob] Add token authentication support for All mode (#4210)

* dashboard client authentication support

Signed-off-by: Future-Outlier <[email protected]>

* support rayjob

Signed-off-by: Future-Outlier <[email protected]>

* update to fix api serverr err

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* updarte

Signed-off-by: Future-Outlier <[email protected]>

* Rayjob sidecar mode auth token mode support

Signed-off-by: Future-Outlier <[email protected]>

* RayJob support k8s job mode

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* Address Andrew's advice

Signed-off-by: Future-Outlier <[email protected]>

* add todo x-ray-authorization comments

Signed-off-by: Future-Outlier <[email protected]>

---------

Signed-off-by: Future-Outlier <[email protected]>

* [RayCluster] Enable Secret informer watch/list and remove unused RBAC verbs (#4202)

* Add authentication secret reconciliation support

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* fix flaky test

Signed-off-by: Future-Outlier <[email protected]>

* remove test fix

Signed-off-by: Rueian <[email protected]>

---------

Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Rueian <[email protected]>
Co-authored-by: Rueian <[email protected]>

* [APIServer][Docs] Add user guide for retry behavior & configuration (#4144)

* [Docs] Add the draft description about feature intro, configurations, and usecases

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] Update the retry walk-through

Signed-off-by: justinyeh1995 <[email protected]>

* [Doc] rewrite the first 2 sections

Signed-off-by: justinyeh1995 <[email protected]>

* [Doc] Revise documentation wording and add Observing Retry Behavior section

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] fix linting issue by running pre-commit run berfore commiting

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] fix linting errors in the Markdown linting

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] Clean up the math equation

Signed-off-by: justinyeh1995 <[email protected]>

* Update the math formula of Backoff calculation.

Co-authored-by: Nary Yeh <[email protected]>
Signed-off-by: JustinYeh <[email protected]>

* [Fix] Explicitly mentioned exponential backoff and removed the customization parts

Signed-off-by: justinyeh1995 <[email protected]>

* [Docs] Clarify naming by replacing “APIServer” with “KubeRay APIServer”

Co-authored-by: Cheng-Yeh Chung <[email protected]>
Signed-off-by: JustinYeh <[email protected]>

* [Docs] Rename retry-configuration.md to retry-behavior.md for accuracy

Signed-off-by: justinyeh1995 <[email protected]>

* Update Title to KubeRay APIServer Retry Behavior

Co-authored-by: Cheng-Yeh Chung <[email protected]>
Signed-off-by: JustinYeh <[email protected]>

* [Docs] Add a note about the limitation of retry configuration

Signed-off-by: justinyeh1995 <[email protected]>

---------

Signed-off-by: justinyeh1995 <[email protected]>
Signed-off-by: JustinYeh <[email protected]>
Co-authored-by: Nary Yeh <[email protected]>
Co-authored-by: Cheng-Yeh Chung <[email protected]>

* Support X-Ray-Authorization fallback header for accepting auth token via proxy (#4213)

* Support X-Ray-Authorization fallback header for accepting auth token in dashboard

Signed-off-by: Future-Outlier <[email protected]>

* remove todo comment

Signed-off-by: Future-Outlier <[email protected]>

---------

Signed-off-by: Future-Outlier <[email protected]>

* [RayCluster] make auth token secret name consistency (#4216)

Signed-off-by: fscnick <[email protected]>

* [RayCluster] Status includes head containter status message (#4196)

* [RayCluster] Status includes head containter status message

Signed-off-by: Spencer Peterson <[email protected]>

* lint

Signed-off-by: Spencer Peterson <[email protected]>

* [RayCluster] Containers not ready status reflects structured reason

Signed-off-by: Spencer Peterson <[email protected]>

* nit

Signed-off-by: Spencer Peterson <[email protected]>

---------

Signed-off-by: Spencer Peterson <[email protected]>

* Remove erroneous  call in applyServeTargetCapacity (#4212)

Signed-off-by: Ryan O'Leary <[email protected]>

* [RayJob] Add token authentication support for light weight job submitter (#4215)

* [RayJob] light weight job submitter auth token support

Signed-off-by: Future-Outlier <[email protected]>

* X-Ray-Authorization

Signed-off-by: Rueian <[email protected]>

---------

Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Rueian <[email protected]>
Co-authored-by: Rueian <[email protected]>

* feat: kubectl ray get token command (#4218)

* feat: kubectl ray get token command

Signed-off-by: Rueian <[email protected]>

* Update kubectl-plugin/pkg/cmd/get/get_token_test.go

Co-authored-by: Copilot <[email protected]>
Signed-off-by: Rueian <[email protected]>

* Update kubectl-plugin/pkg/cmd/get/get_token.go

Co-authored-by: Copilot <[email protected]>
Signed-off-by: Rueian <[email protected]>

* make sure the raycluster exists before getting the secret

Signed-off-by: Rueian <[email protected]>

* better ux

Signed-off-by: Rueian <[email protected]>

* Update kubectl-plugin/pkg/cmd/get/get_token.go

Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>
Signed-off-by: Rueian <[email protected]>

---------

Signed-off-by: Rueian <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>

---------

Signed-off-by: 400Ping <[email protected]>
Signed-off-by: Ping <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Signed-off-by: Ryan O'Leary <[email protected]>
Signed-off-by: Andrew Sy Kim <[email protected]>
Signed-off-by: Kai-Hsun Chen <[email protected]>
Signed-off-by: Neo Chien <[email protected]>
Signed-off-by: justinyeh1995 <[email protected]>
Signed-off-by: win5923 <[email protected]>
Signed-off-by: alimaazamat <[email protected]>
Signed-off-by: Spencer Peterson <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Rueian <[email protected]>
Signed-off-by: JustinYeh <[email protected]>
Signed-off-by: fscnick <[email protected]>
Co-authored-by: Ping <[email protected]>
Co-authored-by: Han-Ju Chen (Future-Outlier) <[email protected]>
Co-authored-by: Ryan O'Leary <[email protected]>
Co-authored-by: Kai-Hsun Chen <[email protected]>
Co-authored-by: Kavish <[email protected]>
Co-authored-by: Neo Chien <[email protected]>
Co-authored-by: cchung100m <[email protected]>
Co-authored-by: JustinYeh <[email protected]>
Co-authored-by: Jun-Hao Wan <[email protected]>
Co-authored-by: Divyam Raj <[email protected]>
Co-authored-by: Nary Yeh <[email protected]>
Co-authored-by: Alima Azamat <[email protected]>
Co-authored-by: Spencer Peterson <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Rueian <[email protected]>
Co-authored-by: Cheng-Yeh Chung <[email protected]>
Co-authored-by: fscnick <[email protected]>
Co-authored-by: Copilot <[email protected]>
2025-11-21 10:18:05 -05:00

36 KiB
Raw Permalink Blame History

API Reference

Packages

ray.io/v1

Package v1 contains API Schema definitions for the ray v1 API group

Resource Types

AuthMode

Underlying type: string

AuthMode describes the authentication mode for the Ray cluster.

Appears in:

AuthOptions

AuthOptions defines the authentication options for a RayCluster.

Appears in:

Field Description Default Validation
mode AuthMode Mode specifies the authentication mode.
Supported values are "disabled" and "token".
Defaults to "token".
Enum: [disabled token]

AutoscalerOptions

AutoscalerOptions specifies optional configuration for the Ray autoscaler.

Appears in:

Field Description Default Validation
resources ResourceRequirements Resources specifies optional resource request and limit overrides for the autoscaler container.
Default values: 500m CPU request and limit. 512Mi memory request and limit.
image string Image optionally overrides the autoscaler's container image. This override is provided for autoscaler testing and development.
imagePullPolicy PullPolicy ImagePullPolicy optionally overrides the autoscaler container's image pull policy. This override is provided for autoscaler testing and development.
securityContext SecurityContext SecurityContext defines the security options the container should be run with.
If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext.
More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
idleTimeoutSeconds integer IdleTimeoutSeconds is the number of seconds to wait before scaling down a worker pod which is not using Ray resources.
Defaults to 60 (one minute). It is not read by the KubeRay operator but by the Ray autoscaler.
upscalingMode UpscalingMode UpscalingMode is "Conservative", "Default", or "Aggressive."
Conservative: Upscaling is rate-limited; the number of pending worker pods is at most the size of the Ray cluster.
Default: Upscaling is not rate-limited.
Aggressive: An alias for Default; upscaling is not rate-limited.
It is not read by the KubeRay operator but by the Ray autoscaler.
Enum: [Default Aggressive Conservative]
version AutoscalerVersion Version is the version of the Ray autoscaler.
Setting this to v1 will explicitly use autoscaler v1.
Setting this to v2 will explicitly use autoscaler v2.
If this isn't set, the Ray version determines the autoscaler version.
In Ray 2.47.0 and later, the default autoscaler version is v2. It's v1 before that.
Enum: [v1 v2]
env EnvVar array Optional list of environment variables to set in the autoscaler container.
envFrom EnvFromSource array Optional list of sources to populate environment variables in the autoscaler container.
volumeMounts VolumeMount array Optional list of volumeMounts. This is needed for enabling TLS for the autoscaler container.

AutoscalerVersion

Underlying type: string

Validation:

  • Enum: [v1 v2]

Appears in:

ClusterUpgradeOptions

These options are currently only supported for the IncrementalUpgrade type.

Appears in:

Field Description Default Validation
maxSurgePercent integer The capacity of serve requests the upgraded cluster should scale to handle each interval.
Defaults to 100%.
100
stepSizePercent integer The percentage of traffic to switch to the upgraded RayCluster at a set interval after scaling by MaxSurgePercent.
intervalSeconds integer The interval in seconds between transferring StepSize traffic from the old to new RayCluster.
gatewayClassName string The name of the Gateway Class installed by the Kubernetes Cluster admin.

DeletionCondition

DeletionCondition specifies the trigger conditions for a deletion action.

Appears in:

Field Description Default Validation
ttlSeconds integer TTLSeconds is the time in seconds from when the JobStatus
reaches the specified terminal state to when this deletion action should be triggered.
The value must be a non-negative integer.
0 Minimum: 0

DeletionPolicy

DeletionPolicy is the legacy single-stage deletion policy. Deprecated: This struct is part of the legacy API. Use DeletionRule for new configurations.

Appears in:

Field Description Default Validation
policy DeletionPolicyType Policy is the action to take when the condition is met.
This field is logically required when using the legacy OnSuccess/OnFailure policies.
It is marked as '+optional' at the API level to allow the 'deletionRules' field to be used instead.
Enum: [DeleteCluster DeleteWorkers DeleteSelf DeleteNone]

DeletionPolicyType

Underlying type: string

Appears in:

DeletionRule

DeletionRule defines a single deletion action and its trigger condition. This is the new, recommended way to define deletion behavior.

Appears in:

Field Description Default Validation
policy DeletionPolicyType Policy is the action to take when the condition is met. This field is required. Enum: [DeleteCluster DeleteWorkers DeleteSelf DeleteNone]
condition DeletionCondition The condition under which this deletion rule is triggered. This field is required.

DeletionStrategy

DeletionStrategy configures automated cleanup after the RayJob reaches a terminal state. Two mutually exclusive styles are supported:

Legacy: provide both onSuccess and onFailure (deprecated; removal planned for 1.6.0). May be combined with shutdownAfterJobFinishes and (optionally) global TTLSecondsAfterFinished.
Rules: provide deletionRules (non-empty list). Rules mode is incompatible with shutdownAfterJobFinishes, legacy fields, and the global TTLSecondsAfterFinished (use per‑rule condition.ttlSeconds instead).

Semantics:

  • A non-empty deletionRules selects rules mode; empty lists are treated as unset.
  • Legacy requires both onSuccess and onFailure; specifying only one is invalid.
  • Global TTLSecondsAfterFinished > 0 requires shutdownAfterJobFinishes=true; therefore it cannot be used with rules mode or with legacy alone (no shutdown).
  • Feature gate RayJobDeletionPolicy must be enabled when this block is present.

Validation:

  • CRD XValidations prevent mixing legacy fields with deletionRules and enforce legacy completeness.
  • Controller logic enforces rules vs shutdown exclusivity and TTL constraints.
  • onSuccess/onFailure are deprecated; migration to deletionRules is encouraged.

Appears in:

Field Description Default Validation
onSuccess DeletionPolicy OnSuccess is the deletion policy for a successful RayJob.
Deprecated: Use deletionRules instead for more flexible, multi-stage deletion strategies.
This field will be removed in release 1.6.0.
onFailure DeletionPolicy OnFailure is the deletion policy for a failed RayJob.
Deprecated: Use deletionRules instead for more flexible, multi-stage deletion strategies.
This field will be removed in release 1.6.0.
deletionRules DeletionRule array DeletionRules is a list of deletion rules, processed based on their trigger conditions.
While the rules can be used to define a sequence, if multiple rules are overdue (e.g., due to controller downtime),
the most impactful rule (e.g., DeleteSelf) will be executed first to prioritize resource cleanup.
MinItems: 1

GcsFaultToleranceOptions

GcsFaultToleranceOptions contains configs for GCS FT

Appears in:

Field Description Default Validation
redisUsername RedisCredential
redisPassword RedisCredential
externalStorageNamespace string
redisAddress string

HeadGroupSpec

HeadGroupSpec are the spec for the head pod

Appears in:

Field Description Default Validation
template PodTemplateSpec Template is the exact pod template used in K8s deployments, statefulsets, etc.
headService Service HeadService is the Kubernetes service of the head pod.
enableIngress boolean EnableIngress indicates whether operator should create ingress object for head service or not.
resources object (keys:string, values:string) Resources specifies the resource quantities for the head group.
These values override the resources passed to rayStartParams for the group, but
have no effect on the resources set at the K8s Pod container level.
labels object (keys:string, values:string) Labels specifies the Ray node labels for the head group.
These labels will also be added to the Pods of this head group and override the --labels
argument passed to rayStartParams.
rayStartParams object (keys:string, values:string) RayStartParams are the params of the start command: node-manager-port, object-store-memory, ...
serviceType ServiceType ServiceType is Kubernetes service type of the head service. it will be used by the workers to connect to the head pod

JobSubmissionMode

Underlying type: string

Appears in:

RayCluster

RayCluster is the Schema for the RayClusters API

Field Description Default Validation
apiVersion string ray.io/v1
kind string RayCluster
metadata ObjectMeta Refer to Kubernetes API documentation for fields of metadata.
spec RayClusterSpec Specification of the desired behavior of the RayCluster.

RayClusterSpec

RayClusterSpec defines the desired state of RayCluster

Appears in:

Field Description Default Validation
authOptions AuthOptions AuthOptions specifies the authentication options for the RayCluster.
suspend boolean Suspend indicates whether a RayCluster should be suspended.
A suspended RayCluster will have head pods and worker pods deleted.
managedBy string ManagedBy is an optional configuration for the controller or entity that manages a RayCluster.
The value must be either 'ray.io/kuberay-operator' or 'kueue.x-k8s.io/multikueue'.
The kuberay-operator reconciles a RayCluster which doesn't have this field at all or
the field value is the reserved string 'ray.io/kuberay-operator',
but delegates reconciling the RayCluster with 'kueue.x-k8s.io/multikueue' to the Kueue.
The field is immutable.
autoscalerOptions AutoscalerOptions AutoscalerOptions specifies optional configuration for the Ray autoscaler.
headServiceAnnotations object (keys:string, values:string)
enableInTreeAutoscaling boolean EnableInTreeAutoscaling indicates whether operator should create in tree autoscaling configs
gcsFaultToleranceOptions GcsFaultToleranceOptions GcsFaultToleranceOptions for enabling GCS FT
headGroupSpec HeadGroupSpec HeadGroupSpec is the spec for the head pod
rayVersion string RayVersion is used to determine the command for the Kubernetes Job managed by RayJob
workerGroupSpecs WorkerGroupSpec array WorkerGroupSpecs are the specs for the worker pods

RayJob

RayJob is the Schema for the rayjobs API

Field Description Default Validation
apiVersion string ray.io/v1
kind string RayJob
metadata ObjectMeta Refer to Kubernetes API documentation for fields of metadata.
spec RayJobSpec

RayJobSpec

RayJobSpec defines the desired state of RayJob

Appears in:

Field Description Default Validation
activeDeadlineSeconds integer ActiveDeadlineSeconds is the duration in seconds that the RayJob may be active before
KubeRay actively tries to terminate the RayJob; value must be positive integer.
backoffLimit integer Specifies the number of retries before marking this job failed.
Each retry creates a new RayCluster.
0
rayClusterSpec RayClusterSpec RayClusterSpec is the cluster template to run the job
submitterPodTemplate PodTemplateSpec SubmitterPodTemplate is the template for the pod that will run ray job submit.
metadata object (keys:string, values:string) Refer to Kubernetes API documentation for fields of metadata.
clusterSelector object (keys:string, values:string) clusterSelector is used to select running rayclusters by labels
submitterConfig SubmitterConfig Configurations of submitter k8s job.
managedBy string ManagedBy is an optional configuration for the controller or entity that manages a RayJob.
The value must be either 'ray.io/kuberay-operator' or 'kueue.x-k8s.io/multikueue'.
The kuberay-operator reconciles a RayJob which doesn't have this field at all or
the field value is the reserved string 'ray.io/kuberay-operator',
but delegates reconciling the RayJob with 'kueue.x-k8s.io/multikueue' to the Kueue.
The field is immutable.
deletionStrategy DeletionStrategy DeletionStrategy automates post-completion cleanup.
Choose one style or omit:
- Legacy: both onSuccess & onFailure (deprecated; may combine with shutdownAfterJobFinishes and TTLSecondsAfterFinished).
- Rules: deletionRules (non-empty) — incompatible with shutdownAfterJobFinishes, legacy fields, and global TTLSecondsAfterFinished (use per-rule condition.ttlSeconds).
Global TTLSecondsAfterFinished > 0 requires shutdownAfterJobFinishes=true.
Feature gate RayJobDeletionPolicy must be enabled when this field is set.
entrypoint string Entrypoint represents the command to start execution.
runtimeEnvYAML string RuntimeEnvYAML represents the runtime environment configuration
provided as a multi-line YAML string.
jobId string If jobId is not set, a new jobId will be auto-generated.
submissionMode JobSubmissionMode SubmissionMode specifies how RayJob submits the Ray job to the RayCluster.
In "K8sJobMode", the KubeRay operator creates a submitter Kubernetes Job to submit the Ray job.
In "HTTPMode", the KubeRay operator sends a request to the RayCluster to create a Ray job.
In "InteractiveMode", the KubeRay operator waits for a user to submit a job to the Ray cluster.
In "SidecarMode", the KubeRay operator injects a container into the Ray head Pod that acts as the job submitter to submit the Ray job.
K8sJobMode
entrypointResources string EntrypointResources specifies the custom resources and quantities to reserve for the
entrypoint command.
entrypointNumCpus float EntrypointNumCpus specifies the number of cpus to reserve for the entrypoint command.
entrypointNumGpus float EntrypointNumGpus specifies the number of gpus to reserve for the entrypoint command.
ttlSecondsAfterFinished integer TTLSecondsAfterFinished is the TTL to clean up RayCluster.
It's only working when ShutdownAfterJobFinishes set to true.
0
shutdownAfterJobFinishes boolean ShutdownAfterJobFinishes will determine whether to delete the ray cluster once rayJob succeed or failed.
suspend boolean suspend specifies whether the RayJob controller should create a RayCluster instance
If a job is applied with the suspend field set to true,
the RayCluster will not be created and will wait for the transition to false.
If the RayCluster is already created, it will be deleted.
In case of transition to false a new RayCluster will be created.

RayService

RayService is the Schema for the rayservices API

Field Description Default Validation
apiVersion string ray.io/v1
kind string RayService
metadata ObjectMeta Refer to Kubernetes API documentation for fields of metadata.
spec RayServiceSpec

RayServiceSpec

RayServiceSpec defines the desired state of RayService

Appears in:

Field Description Default Validation
rayClusterDeletionDelaySeconds integer RayClusterDeletionDelaySeconds specifies the delay, in seconds, before deleting old RayClusters.
The default value is 60 seconds.
Minimum: 0
serviceUnhealthySecondThreshold integer Deprecated: This field is not used anymore. ref: https://github.com/ray-project/kuberay/issues/1685
deploymentUnhealthySecondThreshold integer Deprecated: This field is not used anymore. ref: https://github.com/ray-project/kuberay/issues/1685
serveService Service ServeService is the Kubernetes service for head node and worker nodes who have healthy http proxy to serve traffics.
upgradeStrategy RayServiceUpgradeStrategy UpgradeStrategy defines the scaling policy used when upgrading the RayService.
serveConfigV2 string Important: Run "make" to regenerate code after modifying this file
Defines the applications and deployments to deploy, should be a YAML multi-line scalar string.
rayClusterConfig RayClusterSpec
excludeHeadPodFromServeSvc boolean If the field is set to true, the value of the label ray.io/serve on the head Pod should always be false.
Therefore, the head Pod's endpoint will not be added to the Kubernetes Serve service.

RayServiceUpgradeStrategy

Appears in:

Field Description Default Validation
type RayServiceUpgradeType Type represents the strategy used when upgrading the RayService. Currently supports NewCluster and None.
clusterUpgradeOptions ClusterUpgradeOptions ClusterUpgradeOptions defines the behavior of a NewClusterWithIncrementalUpgrade type.
RayServiceIncrementalUpgrade feature gate must be enabled to set ClusterUpgradeOptions.

RayServiceUpgradeType

Underlying type: string

Appears in:

RedisCredential

RedisCredential is the redis username/password or a reference to the source containing the username/password

Appears in:

Field Description Default Validation
valueFrom EnvVarSource
value string

ScaleStrategy

ScaleStrategy to remove workers

Appears in:

Field Description Default Validation
workersToDelete string array WorkersToDelete workers to be deleted

SubmitterConfig

Appears in:

Field Description Default Validation
backoffLimit integer BackoffLimit of the submitter k8s job.

UpscalingMode

Underlying type: string

Validation:

  • Enum: [Default Aggressive Conservative]

Appears in:

WorkerGroupSpec

WorkerGroupSpec are the specs for the worker pods

Appears in:

Field Description Default Validation
suspend boolean Suspend indicates whether a worker group should be suspended.
A suspended worker group will have all pods deleted.
This is not a user-facing API and is only used by RayJob DeletionStrategy.
groupName string we can have multiple worker groups, we distinguish them by name
replicas integer Replicas is the number of desired Pods for this worker group. See https://github.com/ray-project/kuberay/pull/1443 for more details about the reason for making this field optional. 0
minReplicas integer MinReplicas denotes the minimum number of desired Pods for this worker group. 0
maxReplicas integer MaxReplicas denotes the maximum number of desired Pods for this worker group, and the default value is maxInt32. 2147483647
idleTimeoutSeconds integer IdleTimeoutSeconds denotes the number of seconds to wait before the v2 autoscaler terminates an idle worker pod of this type.
This value is only used with the Ray Autoscaler enabled and defaults to the value set by the AutoscalingConfig if not specified for this worker group.
resources object (keys:string, values:string) Resources specifies the resource quantities for this worker group.
These values override the resources passed to rayStartParams for the group, but
have no effect on the resources set at the K8s Pod container level.
labels object (keys:string, values:string) Labels specifies the Ray node labels for this worker group.
These labels will also be added to the Pods of this worker group and override the --labels
argument passed to rayStartParams.
rayStartParams object (keys:string, values:string) RayStartParams are the params of the start command: address, object-store-memory, ...
template PodTemplateSpec Template is a pod template for the worker
scaleStrategy ScaleStrategy ScaleStrategy defines which pods to remove
numOfHosts integer NumOfHosts denotes the number of hosts to create per replica. The default value is 1. 1

ray.io/v1alpha1

Package v1alpha1 contains API Schema definitions for the ray v1alpha1 API group

Resource Types

AutoscalerOptions

AutoscalerOptions specifies optional configuration for the Ray autoscaler.

Appears in:

Field Description Default Validation
resources ResourceRequirements Resources specifies optional resource request and limit overrides for the autoscaler container.
Default values: 500m CPU request and limit. 512Mi memory request and limit.
image string Image optionally overrides the autoscaler's container image. This override is provided for autoscaler testing and development.
imagePullPolicy PullPolicy ImagePullPolicy optionally overrides the autoscaler container's image pull policy. This override is provided for autoscaler testing and development.
securityContext SecurityContext SecurityContext defines the security options the container should be run with.
If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext.
More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
idleTimeoutSeconds integer IdleTimeoutSeconds is the number of seconds to wait before scaling down a worker pod which is not using Ray resources.
Defaults to 60 (one minute). It is not read by the KubeRay operator but by the Ray autoscaler.
upscalingMode UpscalingMode UpscalingMode is "Conservative", "Default", or "Aggressive."
Conservative: Upscaling is rate-limited; the number of pending worker pods is at most the size of the Ray cluster.
Default: Upscaling is not rate-limited.
Aggressive: An alias for Default; upscaling is not rate-limited.
It is not read by the KubeRay operator but by the Ray autoscaler.
Enum: [Default Aggressive Conservative]
env EnvVar array Optional list of environment variables to set in the autoscaler container.
envFrom EnvFromSource array Optional list of sources to populate environment variables in the autoscaler container.
volumeMounts VolumeMount array Optional list of volumeMounts. This is needed for enabling TLS for the autoscaler container.

HeadGroupSpec

HeadGroupSpec are the spec for the head pod

Appears in:

Field Description Default Validation
template PodTemplateSpec Template is the exact pod template used in K8s deployments, statefulsets, etc.
headService Service HeadService is the Kubernetes service of the head pod.
enableIngress boolean EnableIngress indicates whether operator should create ingress object for head service or not.
rayStartParams object (keys:string, values:string) RayStartParams are the params of the start command: node-manager-port, object-store-memory, ...
serviceType ServiceType ServiceType is Kubernetes service type of the head service. it will be used by the workers to connect to the head pod

RayCluster

RayCluster is the Schema for the RayClusters API

Field Description Default Validation
apiVersion string ray.io/v1alpha1
kind string RayCluster
metadata ObjectMeta Refer to Kubernetes API documentation for fields of metadata.
spec RayClusterSpec Specification of the desired behavior of the RayCluster.

RayClusterSpec

RayClusterSpec defines the desired state of RayCluster

Appears in:

Field Description Default Validation
enableInTreeAutoscaling boolean EnableInTreeAutoscaling indicates whether operator should create in tree autoscaling configs
autoscalerOptions AutoscalerOptions AutoscalerOptions specifies optional configuration for the Ray autoscaler.
suspend boolean Suspend indicates whether a RayCluster should be suspended.
A suspended RayCluster will have head pods and worker pods deleted.
headServiceAnnotations object (keys:string, values:string)
headGroupSpec HeadGroupSpec HeadGroupSpec is the spec for the head pod
rayVersion string RayVersion is used to determine the command for the Kubernetes Job managed by RayJob
workerGroupSpecs WorkerGroupSpec array WorkerGroupSpecs are the specs for the worker pods

RayJob

RayJob is the Schema for the rayjobs API

Field Description Default Validation
apiVersion string ray.io/v1alpha1
kind string RayJob
metadata ObjectMeta Refer to Kubernetes API documentation for fields of metadata.
spec RayJobSpec

RayJobSpec

RayJobSpec defines the desired state of RayJob

Appears in:

Field Description Default Validation
submitterPodTemplate PodTemplateSpec SubmitterPodTemplate is the template for the pod that will run ray job submit.
metadata object (keys:string, values:string) Refer to Kubernetes API documentation for fields of metadata.
rayClusterSpec RayClusterSpec RayClusterSpec is the cluster template to run the job
clusterSelector object (keys:string, values:string) ClusterSelector is used to select running rayclusters by labels
entrypoint string Entrypoint represents the command to start execution.
runtimeEnvYAML string RuntimeEnvYAML represents the runtime environment configuration
provided as a multi-line YAML string.
jobId string If jobId is not set, a new jobId will be auto-generated.
entrypointResources string EntrypointResources specifies the custom resources and quantities to reserve for the
entrypoint command.
ttlSecondsAfterFinished integer TTLSecondsAfterFinished is the TTL to clean up RayCluster.
It's only working when ShutdownAfterJobFinishes set to true.
0
entrypointNumCpus float EntrypointNumCpus specifies the number of cpus to reserve for the entrypoint command.
entrypointNumGpus float EntrypointNumGpus specifies the number of gpus to reserve for the entrypoint command.
shutdownAfterJobFinishes boolean ShutdownAfterJobFinishes will determine whether to delete the ray cluster once rayJob succeed or failed.
suspend boolean Suspend specifies whether the RayJob controller should create a RayCluster instance
If a job is applied with the suspend field set to true,
the RayCluster will not be created and will wait for the transition to false.
If the RayCluster is already created, it will be deleted.
In case of transition to false a new RayCluster will be created.

RayService

RayService is the Schema for the rayservices API

Field Description Default Validation
apiVersion string ray.io/v1alpha1
kind string RayService
metadata ObjectMeta Refer to Kubernetes API documentation for fields of metadata.
spec RayServiceSpec

RayServiceSpec

RayServiceSpec defines the desired state of RayService

Appears in:

Field Description Default Validation
serveService Service ServeService is the Kubernetes service for head node and worker nodes who have healthy http proxy to serve traffics.
serviceUnhealthySecondThreshold integer Deprecated: This field is not used anymore. ref: https://github.com/ray-project/kuberay/issues/1685
deploymentUnhealthySecondThreshold integer Deprecated: This field is not used anymore. ref: https://github.com/ray-project/kuberay/issues/1685
serveConfigV2 string Important: Run "make" to regenerate code after modifying this file
Defines the applications and deployments to deploy, should be a YAML multi-line scalar string.
rayClusterConfig RayClusterSpec

ScaleStrategy

ScaleStrategy to remove workers

Appears in:

Field Description Default Validation
workersToDelete string array WorkersToDelete workers to be deleted

UpscalingMode

Underlying type: string

Validation:

  • Enum: [Default Aggressive Conservative]

Appears in:

WorkerGroupSpec

WorkerGroupSpec are the specs for the worker pods

Appears in:

Field Description Default Validation
groupName string we can have multiple worker groups, we distinguish them by name
replicas integer Replicas is the number of desired Pods for this worker group. See https://github.com/ray-project/kuberay/pull/1443 for more details about the reason for making this field optional. 0
minReplicas integer MinReplicas denotes the minimum number of desired Pods for this worker group. 0
maxReplicas integer MaxReplicas denotes the maximum number of desired Pods for this worker group, and the default value is maxInt32. 2147483647
rayStartParams object (keys:string, values:string) RayStartParams are the params of the start command: address, object-store-memory, ...
template PodTemplateSpec Template is a pod template for the worker
scaleStrategy ScaleStrategy ScaleStrategy defines which pods to remove