Files
Jiaxin ShanandGitHub 40f394372f Update README.md (#52)
* Update README.md

Update root and ray-operator README.md.
Separate DEVELOPMENT.md from original ray-operator README.md

* Remove unrelated folders

* Address feedbacks
2021-10-07 10:07:45 -07:00

6.4 KiB

Ray Kubernetes Operator

KubeRay operator makes deploying and managing Ray clusters on top of Kubernetes painless - clusters are defined as a custom RayCluster resource and managed by a fault-tolerant Ray controller. The Ray Operator is a Kubernetes operator to automate provisioning, management, autoscaling and operations of Ray clusters deployed to Kubernetes.

overview

Some of the main features of the operator are:

  • Management of first-class RayClusters via a custom resource.
  • Support for hetergenous worker types in a single Ray cluster.
  • Built-in monitoring via Prometheus.
  • Use of PodTemplate to create Ray pods
  • Updated status based on the running pods
  • Events added to the RayCluster instance
  • Automatically populate environment variables in the containers
  • Automatically prefix your container command with the ray start command
  • Automatically adding the volumeMount at /dev/shm for shared memory
  • Use of ScaleStartegy to remove specific nodes in specific groups

Overview

When deployed, the ray operator will watch for K8s events (create/delete/update) for the raycluster resources. The ray operator can create a raycluster (head + multipe workers), delete a cluster, or update the cluster by adding or removing worker pods.

Ray cluster creation

Once a raycluster resource is created, the operator will configure and create the ray-head and the ray-workers specified in the raycluster manifest as shown below.

Ray cluster Update

You can update the number of replicas in a worker goup, and specify which exact replica to remove by updated the raycluster resource manifest:

Ray cluster example code

An example ray code is defined in this configmap that is mounted into the ray head-pod. By examining the logs of the head pod, we can see the list of the IP addresses of the nodes that joined the ray cluster:

Deploy the operator

kubectl apply -k "github.com/ray-project/kuberay/ray-operator/config/default"

Check that the controller is running.

$ kubectl get deployments -n ray-system
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
ray-operator   1/1     1            1           40s

$ kubectl get pods -n ray-system
NAME                            READY   STATUS    RESTARTS   AGE
ray-operator-75dbbf8587-5lrvn   1/1     Running   0          31s

Delete the operator

kubectl delete -k "github.com/ray-project/kuberay/ray-operator/config/default"

Running an example cluster

There are three example config files to deploy RayClusters included here:

Sample Description
ray-cluster.mini.yaml Small example consisting of 1 head pod.
ray-cluster.heterogeneous.yaml Example with heterogenous worker types. 1 head pod and 2 worker pods, each of which has a different resource quota.
ray-cluster.complete.yaml Shows all available custom resouce properties.
# Create a configmap with a hello world Ray code.
kubectl create -f config/samples/config-map-ray-code.yaml
configmap/ray-code created
# Create a cluster.
$ kubectl create -f config/samples/ray-cluster.heterogeneous.yaml
raycluster.ray.io/raycluster-heterogeneous created

# List running clusters.
$ kubectl get rayclusters
NAME                AGE
raycluster-heterogeneous   2m48s

# The created cluster should include a head pod, worker pod, and a head service.
$ kubectl get pods
NAME                                                 READY   STATUS    RESTARTS   AGE
raycluster-heterogeneous-head-5r6qr                  1/1     Running   0          14m
raycluster-heterogeneous-worker-medium-group-ljzzt   1/1     Running   0          14m
raycluster-heterogeneous-worker-small-group-76qxb    1/1     Running   0          14m
raycluster-heterogeneous-worker-small-group-dcl4d    1/1     Running   0          14m
$ kubectl get services
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kubernetes                        ClusterIP   10.152.183.1   <none>        443/TCP   35d
raycluster-heterogeneous-my-svc   ClusterIP   None           <none>        80/TCP    15m

The logs of the head pod should show 4 nodes in the Ray cluster

Ray Nodes:  {'10.1.73.139', '10.1.73.138', '10.1.73.140', '10.1.73.141'}
# check the logs of the head pod
$ kubectl logs raycluster-heterogeneous-head-5r6qr
2020-11-18 09:23:32,069 INFO services.py:1092 -- View the Ray dashboard at http://10.1.73.141:8265
2020-11-18 09:23:31,668 INFO scripts.py:467 -- Local node IP: 10.1.73.141
2020-11-18 09:23:32,093 SUCC scripts.py:497 -- --------------------
2020-11-18 09:23:32,093 SUCC scripts.py:498 -- Ray runtime started.
2020-11-18 09:23:32,093 SUCC scripts.py:499 -- --------------------
2020-11-18 09:23:32,093 INFO scripts.py:501 -- Next steps
2020-11-18 09:23:32,093 INFO scripts.py:503 -- To connect to this Ray runtime from another node, run
2020-11-18 09:23:32,093 INFO scripts.py:507 --   ray start --address='10.1.73.141:6379' --redis-password='LetMeInRay'
2020-11-18 09:23:32,093 INFO scripts.py:509 -- Alternatively, use the following Python code:
2020-11-18 09:23:32,093 INFO scripts.py:512 -- import ray
2020-11-18 09:23:32,093 INFO scripts.py:519 -- ray.init(address='auto', _redis_password='LetMeInRay')
2020-11-18 09:23:32,093 INFO scripts.py:522 -- If connection fails, check your firewall settings and network configuration.
2020-11-18 09:23:32,093 INFO scripts.py:526 -- To terminate the Ray runtime, run
2020-11-18 09:23:32,094 INFO scripts.py:527 --   ray stop
2020-11-18 09:23:32,656 INFO worker.py:651 -- Connecting to existing Ray cluster at address: 10.1.73.141:6379
2020-11-18 09:23:32,669 WARNING services.py:202 -- Some processes that the driver needs to connect to have not registered with Redis, so retrying. Have you run 'ray start' on this node?
trying to connect to Ray!
now executing some code with Ray!
Ray Nodes:  {'10.1.73.139', '10.1.73.138', '10.1.73.140', '10.1.73.141'}
Execution time =  6.961702346801758
# Delete the cluster.
$ kubectl delete raycluster raycluster-heterogeneous