* Changes required make a build after update of component-base (#3004) * Update packages required for bump to component-base * Update version of controller-gen required by packages update Migrate to kube_codegen.sh * Sync CRDs post generator update * Docker and go-toolchain update * Update tests to align with package changes A in SetFeatureGateDuringTest, that stopped returning a cleanup func - https://github.com/kubernetes/kubernetes/issues/123677 Required tests update to not use `It()` while it could be `By()` as using `It()` resulted in parallel access to the feature gates. * Change enforced by golangci-lint - fieldalignment * Regenerate client code with new generator * [CI] fix locust versions (#3100) Signed-off-by: Rueian <[email protected]> * [ci] remove the test for modin because its certificate is broken (#3135) Signed-off-by: Rueian <[email protected]> --------- Signed-off-by: Rueian <[email protected]> Co-authored-by: Rueian <[email protected]>
28 KiB
API Reference
Packages
ray.io/v1
Package v1 contains API Schema definitions for the ray v1 API group
Resource Types
AutoscalerOptions
AutoscalerOptions specifies optional configuration for the Ray autoscaler.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
resources ResourceRequirements |
Resources specifies optional resource request and limit overrides for the autoscaler container. Default values: 500m CPU request and limit. 512Mi memory request and limit. |
||
image string |
Image optionally overrides the autoscaler's container image. This override is for provided for autoscaler testing and development. | ||
imagePullPolicy PullPolicy |
ImagePullPolicy optionally overrides the autoscaler container's image pull policy. This override is for provided for autoscaler testing and development. | ||
securityContext SecurityContext |
SecurityContext defines the security options the container should be run with. If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext. More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ |
||
idleTimeoutSeconds integer |
IdleTimeoutSeconds is the number of seconds to wait before scaling down a worker pod which is not using Ray resources. Defaults to 60 (one minute). It is not read by the KubeRay operator but by the Ray autoscaler. |
||
upscalingMode UpscalingMode |
UpscalingMode is "Conservative", "Default", or "Aggressive." Conservative: Upscaling is rate-limited; the number of pending worker pods is at most the size of the Ray cluster. Default: Upscaling is not rate-limited. Aggressive: An alias for Default; upscaling is not rate-limited. It is not read by the KubeRay operator but by the Ray autoscaler. |
Enum: [Default Aggressive Conservative] |
|
env EnvVar array |
Optional list of environment variables to set in the autoscaler container. | ||
envFrom EnvFromSource array |
Optional list of sources to populate environment variables in the autoscaler container. | ||
volumeMounts VolumeMount array |
Optional list of volumeMounts. This is needed for enabling TLS for the autoscaler container. |
DeletionPolicy
Underlying type: string
Appears in:
GcsFaultToleranceOptions
GcsFaultToleranceOptions contains configs for GCS FT
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
redisUsername RedisCredential |
|||
redisPassword RedisCredential |
|||
externalStorageNamespace string |
|||
redisAddress string |
HeadGroupSpec
HeadGroupSpec are the spec for the head pod
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
template PodTemplateSpec |
Template is the exact pod template used in K8s depoyments, statefulsets, etc. | ||
headService Service |
HeadService is the Kubernetes service of the head pod. | ||
enableIngress boolean |
EnableIngress indicates whether operator should create ingress object for head service or not. | ||
rayStartParams object (keys:string, values:string) |
RayStartParams are the params of the start command: node-manager-port, object-store-memory, ... | ||
serviceType ServiceType |
ServiceType is Kubernetes service type of the head service. it will be used by the workers to connect to the head pod |
JobSubmissionMode
Underlying type: string
Appears in:
RayCluster
RayCluster is the Schema for the RayClusters API
| Field | Description | Default | Validation |
|---|---|---|---|
apiVersion string |
ray.io/v1 |
||
kind string |
RayCluster |
||
metadata ObjectMeta |
Refer to Kubernetes API documentation for fields of metadata. |
||
spec RayClusterSpec |
Specification of the desired behavior of the RayCluster. |
RayClusterSpec
RayClusterSpec defines the desired state of RayCluster
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
suspend boolean |
Suspend indicates whether a RayCluster should be suspended. A suspended RayCluster will have head pods and worker pods deleted. |
||
managedBy string |
ManagedBy is an optional configuration for the controller or entity that manages a RayCluster. The value must be either 'ray.io/kuberay-operator' or 'kueue.x-k8s.io/multikueue'. The kuberay-operator reconciles a RayCluster which doesn't have this field at all or the field value is the reserved string 'ray.io/kuberay-operator', but delegates reconciling the RayCluster with 'kueue.x-k8s.io/multikueue' to the Kueue. The field is immutable. |
||
autoscalerOptions AutoscalerOptions |
AutoscalerOptions specifies optional configuration for the Ray autoscaler. | ||
headServiceAnnotations object (keys:string, values:string) |
|||
enableInTreeAutoscaling boolean |
EnableInTreeAutoscaling indicates whether operator should create in tree autoscaling configs | ||
gcsFaultToleranceOptions GcsFaultToleranceOptions |
GcsFaultToleranceOptions for enabling GCS FT | ||
headGroupSpec HeadGroupSpec |
HeadGroupSpec is the spec for the head pod | ||
rayVersion string |
RayVersion is used to determine the command for the Kubernetes Job managed by RayJob | ||
workerGroupSpecs WorkerGroupSpec array |
WorkerGroupSpecs are the specs for the worker pods |
RayJob
RayJob is the Schema for the rayjobs API
| Field | Description | Default | Validation |
|---|---|---|---|
apiVersion string |
ray.io/v1 |
||
kind string |
RayJob |
||
metadata ObjectMeta |
Refer to Kubernetes API documentation for fields of metadata. |
||
spec RayJobSpec |
RayJobSpec
RayJobSpec defines the desired state of RayJob
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
activeDeadlineSeconds integer |
ActiveDeadlineSeconds is the duration in seconds that the RayJob may be active before KubeRay actively tries to terminate the RayJob; value must be positive integer. |
||
backoffLimit integer |
Specifies the number of retries before marking this job failed. Each retry creates a new RayCluster. |
0 | |
rayClusterSpec RayClusterSpec |
RayClusterSpec is the cluster template to run the job | ||
submitterPodTemplate PodTemplateSpec |
SubmitterPodTemplate is the template for the pod that will run ray job submit. |
||
metadata object (keys:string, values:string) |
Refer to Kubernetes API documentation for fields of metadata. |
||
clusterSelector object (keys:string, values:string) |
clusterSelector is used to select running rayclusters by labels | ||
submitterConfig SubmitterConfig |
Configurations of submitter k8s job. | ||
managedBy string |
ManagedBy is an optional configuration for the controller or entity that manages a RayJob. The value must be either 'ray.io/kuberay-operator' or 'kueue.x-k8s.io/multikueue'. The kuberay-operator reconciles a RayJob which doesn't have this field at all or the field value is the reserved string 'ray.io/kuberay-operator', but delegates reconciling the RayJob with 'kueue.x-k8s.io/multikueue' to the Kueue. The field is immutable. |
||
deletionPolicy DeletionPolicy |
DeletionPolicy indicates what resources of the RayJob are deleted upon job completion. Valid values are 'DeleteCluster', 'DeleteWorkers', 'DeleteSelf' or 'DeleteNone'. If unset, deletion policy is based on 'spec.shutdownAfterJobFinishes'. This field requires the RayJobDeletionPolicy feature gate to be enabled. |
||
entrypoint string |
Entrypoint represents the command to start execution. | ||
runtimeEnvYAML string |
RuntimeEnvYAML represents the runtime environment configuration provided as a multi-line YAML string. |
||
jobId string |
If jobId is not set, a new jobId will be auto-generated. | ||
submissionMode JobSubmissionMode |
SubmissionMode specifies how RayJob submits the Ray job to the RayCluster. In "K8sJobMode", the KubeRay operator creates a submitter Kubernetes Job to submit the Ray job. In "HTTPMode", the KubeRay operator sends a request to the RayCluster to create a Ray job. In "InteractiveMode", the KubeRay operator waits for a user to submit a job to the Ray cluster. |
K8sJobMode | |
entrypointResources string |
EntrypointResources specifies the custom resources and quantities to reserve for the entrypoint command. |
||
entrypointNumCpus float |
EntrypointNumCpus specifies the number of cpus to reserve for the entrypoint command. | ||
entrypointNumGpus float |
EntrypointNumGpus specifies the number of gpus to reserve for the entrypoint command. | ||
ttlSecondsAfterFinished integer |
TTLSecondsAfterFinished is the TTL to clean up RayCluster. It's only working when ShutdownAfterJobFinishes set to true. |
0 | |
shutdownAfterJobFinishes boolean |
ShutdownAfterJobFinishes will determine whether to delete the ray cluster once rayJob succeed or failed. | ||
suspend boolean |
suspend specifies whether the RayJob controller should create a RayCluster instance If a job is applied with the suspend field set to true, the RayCluster will not be created and will wait for the transition to false. If the RayCluster is already created, it will be deleted. In case of transition to false a new RayCluster will be created. |
RayService
RayService is the Schema for the rayservices API
| Field | Description | Default | Validation |
|---|---|---|---|
apiVersion string |
ray.io/v1 |
||
kind string |
RayService |
||
metadata ObjectMeta |
Refer to Kubernetes API documentation for fields of metadata. |
||
spec RayServiceSpec |
RayServiceSpec
RayServiceSpec defines the desired state of RayService
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
serviceUnhealthySecondThreshold integer |
Deprecated: This field is not used anymore. ref: https://github.com/ray-project/kuberay/issues/1685 | ||
deploymentUnhealthySecondThreshold integer |
Deprecated: This field is not used anymore. ref: https://github.com/ray-project/kuberay/issues/1685 | ||
serveService Service |
ServeService is the Kubernetes service for head node and worker nodes who have healthy http proxy to serve traffics. | ||
upgradeStrategy RayServiceUpgradeStrategy |
UpgradeStrategy defines the scaling policy used when upgrading the RayService. | ||
serveConfigV2 string |
Important: Run "make" to regenerate code after modifying this file Defines the applications and deployments to deploy, should be a YAML multi-line scalar string. |
||
rayClusterConfig RayClusterSpec |
|||
excludeHeadPodFromServeSvc boolean |
If the field is set to true, the value of the label ray.io/serve on the head Pod should always be false.Therefore, the head Pod's endpoint will not be added to the Kubernetes Serve service. |
RayServiceUpgradeStrategy
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
type RayServiceUpgradeType |
Type represents the strategy used when upgrading the RayService. Currently supports NewCluster and None. |
RayServiceUpgradeType
Underlying type: string
Appears in:
RedisCredential
RedisCredential is the redis username/password or a reference to the source containing the username/password
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
valueFrom EnvVarSource |
|||
value string |
ScaleStrategy
ScaleStrategy to remove workers
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
workersToDelete string array |
WorkersToDelete workers to be deleted |
SubmitterConfig
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
backoffLimit integer |
BackoffLimit of the submitter k8s job. |
UpscalingMode
Underlying type: string
Validation:
- Enum: [Default Aggressive Conservative]
Appears in:
WorkerGroupSpec
WorkerGroupSpec are the specs for the worker pods
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
suspend boolean |
Suspend indicates whether a worker group should be suspended. A suspended worker group will have all pods deleted. This is not a user-facing API and is only used by RayJob DeletionPolicy. |
||
groupName string |
we can have multiple worker groups, we distinguish them by name | ||
replicas integer |
Replicas is the number of desired Pods for this worker group. See https://github.com/ray-project/kuberay/pull/1443 for more details about the reason for making this field optional. | 0 | |
minReplicas integer |
MinReplicas denotes the minimum number of desired Pods for this worker group. | 0 | |
maxReplicas integer |
MaxReplicas denotes the maximum number of desired Pods for this worker group, and the default value is maxInt32. | 2147483647 | |
idleTimeoutSeconds integer |
IdleTimeoutSeconds denotes the number of seconds to wait before the v2 autoscaler terminates an idle worker pod of this type. This value is only used with the Ray Autoscaler enabled and defaults to the value set by the AutoscalingConfig if not specified for this worker group. |
||
rayStartParams object (keys:string, values:string) |
RayStartParams are the params of the start command: address, object-store-memory, ... | ||
template PodTemplateSpec |
Template is a pod template for the worker | ||
scaleStrategy ScaleStrategy |
ScaleStrategy defines which pods to remove | ||
numOfHosts integer |
NumOfHosts denotes the number of hosts to create per replica. The default value is 1. | 1 |
ray.io/v1alpha1
Package v1alpha1 contains API Schema definitions for the ray v1alpha1 API group
Resource Types
AutoscalerOptions
AutoscalerOptions specifies optional configuration for the Ray autoscaler.
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
resources ResourceRequirements |
Resources specifies optional resource request and limit overrides for the autoscaler container. Default values: 500m CPU request and limit. 512Mi memory request and limit. |
||
image string |
Image optionally overrides the autoscaler's container image. This override is for provided for autoscaler testing and development. | ||
imagePullPolicy PullPolicy |
ImagePullPolicy optionally overrides the autoscaler container's image pull policy. This override is for provided for autoscaler testing and development. | ||
securityContext SecurityContext |
SecurityContext defines the security options the container should be run with. If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext. More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ |
||
idleTimeoutSeconds integer |
IdleTimeoutSeconds is the number of seconds to wait before scaling down a worker pod which is not using Ray resources. Defaults to 60 (one minute). It is not read by the KubeRay operator but by the Ray autoscaler. |
||
upscalingMode UpscalingMode |
UpscalingMode is "Conservative", "Default", or "Aggressive." Conservative: Upscaling is rate-limited; the number of pending worker pods is at most the size of the Ray cluster. Default: Upscaling is not rate-limited. Aggressive: An alias for Default; upscaling is not rate-limited. It is not read by the KubeRay operator but by the Ray autoscaler. |
Enum: [Default Aggressive Conservative] |
|
env EnvVar array |
Optional list of environment variables to set in the autoscaler container. | ||
envFrom EnvFromSource array |
Optional list of sources to populate environment variables in the autoscaler container. | ||
volumeMounts VolumeMount array |
Optional list of volumeMounts. This is needed for enabling TLS for the autoscaler container. |
HeadGroupSpec
HeadGroupSpec are the spec for the head pod
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
template PodTemplateSpec |
Template is the exact pod template used in K8s depoyments, statefulsets, etc. | ||
headService Service |
HeadService is the Kubernetes service of the head pod. | ||
enableIngress boolean |
EnableIngress indicates whether operator should create ingress object for head service or not. | ||
rayStartParams object (keys:string, values:string) |
RayStartParams are the params of the start command: node-manager-port, object-store-memory, ... | ||
serviceType ServiceType |
ServiceType is Kubernetes service type of the head service. it will be used by the workers to connect to the head pod |
RayCluster
RayCluster is the Schema for the RayClusters API
| Field | Description | Default | Validation |
|---|---|---|---|
apiVersion string |
ray.io/v1alpha1 |
||
kind string |
RayCluster |
||
metadata ObjectMeta |
Refer to Kubernetes API documentation for fields of metadata. |
||
spec RayClusterSpec |
Specification of the desired behavior of the RayCluster. |
RayClusterSpec
RayClusterSpec defines the desired state of RayCluster
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
enableInTreeAutoscaling boolean |
EnableInTreeAutoscaling indicates whether operator should create in tree autoscaling configs | ||
autoscalerOptions AutoscalerOptions |
AutoscalerOptions specifies optional configuration for the Ray autoscaler. | ||
suspend boolean |
Suspend indicates whether a RayCluster should be suspended. A suspended RayCluster will have head pods and worker pods deleted. |
||
headServiceAnnotations object (keys:string, values:string) |
|||
headGroupSpec HeadGroupSpec |
HeadGroupSpec is the spec for the head pod | ||
rayVersion string |
RayVersion is used to determine the command for the Kubernetes Job managed by RayJob | ||
workerGroupSpecs WorkerGroupSpec array |
WorkerGroupSpecs are the specs for the worker pods |
RayJob
RayJob is the Schema for the rayjobs API
| Field | Description | Default | Validation |
|---|---|---|---|
apiVersion string |
ray.io/v1alpha1 |
||
kind string |
RayJob |
||
metadata ObjectMeta |
Refer to Kubernetes API documentation for fields of metadata. |
||
spec RayJobSpec |
RayJobSpec
RayJobSpec defines the desired state of RayJob
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
submitterPodTemplate PodTemplateSpec |
SubmitterPodTemplate is the template for the pod that will run ray job submit. |
||
metadata object (keys:string, values:string) |
Refer to Kubernetes API documentation for fields of metadata. |
||
rayClusterSpec RayClusterSpec |
RayClusterSpec is the cluster template to run the job | ||
clusterSelector object (keys:string, values:string) |
ClusterSelector is used to select running rayclusters by labels | ||
entrypoint string |
Entrypoint represents the command to start execution. | ||
runtimeEnvYAML string |
RuntimeEnvYAML represents the runtime environment configuration provided as a multi-line YAML string. |
||
jobId string |
If jobId is not set, a new jobId will be auto-generated. | ||
entrypointResources string |
EntrypointResources specifies the custom resources and quantities to reserve for the entrypoint command. |
||
ttlSecondsAfterFinished integer |
TTLSecondsAfterFinished is the TTL to clean up RayCluster. It's only working when ShutdownAfterJobFinishes set to true. |
0 | |
entrypointNumCpus float |
EntrypointNumCpus specifies the number of cpus to reserve for the entrypoint command. | ||
entrypointNumGpus float |
EntrypointNumGpus specifies the number of gpus to reserve for the entrypoint command. | ||
shutdownAfterJobFinishes boolean |
ShutdownAfterJobFinishes will determine whether to delete the ray cluster once rayJob succeed or failed. | ||
suspend boolean |
Suspend specifies whether the RayJob controller should create a RayCluster instance If a job is applied with the suspend field set to true, the RayCluster will not be created and will wait for the transition to false. If the RayCluster is already created, it will be deleted. In case of transition to false a new RayCluster will be created. |
RayService
RayService is the Schema for the rayservices API
| Field | Description | Default | Validation |
|---|---|---|---|
apiVersion string |
ray.io/v1alpha1 |
||
kind string |
RayService |
||
metadata ObjectMeta |
Refer to Kubernetes API documentation for fields of metadata. |
||
spec RayServiceSpec |
RayServiceSpec
RayServiceSpec defines the desired state of RayService
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
serveService Service |
ServeService is the Kubernetes service for head node and worker nodes who have healthy http proxy to serve traffics. | ||
serviceUnhealthySecondThreshold integer |
Deprecated: This field is not used anymore. ref: https://github.com/ray-project/kuberay/issues/1685 | ||
deploymentUnhealthySecondThreshold integer |
Deprecated: This field is not used anymore. ref: https://github.com/ray-project/kuberay/issues/1685 | ||
serveConfigV2 string |
Important: Run "make" to regenerate code after modifying this file Defines the applications and deployments to deploy, should be a YAML multi-line scalar string. |
||
rayClusterConfig RayClusterSpec |
ScaleStrategy
ScaleStrategy to remove workers
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
workersToDelete string array |
WorkersToDelete workers to be deleted |
UpscalingMode
Underlying type: string
Validation:
- Enum: [Default Aggressive Conservative]
Appears in:
WorkerGroupSpec
WorkerGroupSpec are the specs for the worker pods
Appears in:
| Field | Description | Default | Validation |
|---|---|---|---|
groupName string |
we can have multiple worker groups, we distinguish them by name | ||
replicas integer |
Replicas is the number of desired Pods for this worker group. See https://github.com/ray-project/kuberay/pull/1443 for more details about the reason for making this field optional. | 0 | |
minReplicas integer |
MinReplicas denotes the minimum number of desired Pods for this worker group. | 0 | |
maxReplicas integer |
MaxReplicas denotes the maximum number of desired Pods for this worker group, and the default value is maxInt32. | 2147483647 | |
rayStartParams object (keys:string, values:string) |
RayStartParams are the params of the start command: address, object-store-memory, ... | ||
template PodTemplateSpec |
Template is a pod template for the worker | ||
scaleStrategy ScaleStrategy |
ScaleStrategy defines which pods to remove |