* Update helm chart version to 0.3.0 (#461) * helm-charts/ray-cluster: Allow extra workers (#451) * helm-charts/ray-cluster: Allow extra workers We can not set additional workers with the current helm chart. There are some PRs that aim to fix this but they appear to be stale https://github.com/ray-project/kuberay/pull/266 We wanted to keep untouched the default worker that is already included in the helm chart but allow the addition of new workers. The changes are adding the following logic: - Allow to disable the default worker group, if needed, via worker.disabled = true - Allow to set additional workers by using the extraWorkers map The extraWorkers map has a different structure compared to the default worker because we wanted to add additional flexibility. Due to that, some variables in the extraWorkers have different naming (which matches the name of the related CRD attributed) Note: PR was done to support additional workers in the infrastructure of Metrika.co * Sync up helm chart's role (#472) Co-authored-by: chenyu.jiang <[email protected]> * Document the raycluster status (#473) Co-authored-by: chenyu.jiang <[email protected]> Co-authored-by: Christos Kotsis <[email protected]> Co-authored-by: Chenyu Jiang <[email protected]> Co-authored-by: chenyu.jiang <[email protected]>
2.5 KiB
Observability
RayCluster Status
State
In the RayCluster resource definition, we use State to represent the current status of the ray cluster.
For now, there are three types of the status exposed by the resouce status: ready, unhealthy and failed.
| State | Description |
|---|---|
| ready | the ray cluster is ready for use. |
| unhealthy | there is something miss configed in the startParams and the ray cluster may not act correctly |
| failed | there are some severe fatal and result in the head node or worker node start failed. |
If you use apiserver to retrieve the resource, you may find the state in the clusterState field.
curl --request GET '<baseUrl>/apis/v1alpha2/namespaces/<namespace>/clusters/<raycluster-name>'
{
"name": "<raycluster-name>",
"namespace": "<namespace>",
//...
"createdAt": "2022-08-10T10:31:25Z",
"clusterState": "ready",
//...
}
Endpoint
If you use the nodeport as service to expose the raycluster endpoint, like dashboard or redis, there are endpoints field in the status to record the service endpoints.
you can directly use the ports in the endpoints to connect to the related service.
Also, if you use apiserver to retrieve the resource, you can find the endpoints in the serviceEndpoint field.
curl --request GET '<baseUrl>/apis/v1alpha2/namespaces/<namespace>/clusters/<raycluster-name>'
{
"name": "<raycluster-name>",
"namespace": "<namespace>",
//...
"serviceEndpoint": {
"dashboard": "30001",
"head": "30002",
"metrics": "30003",
"redis": "30004"
},
//...
}
Monitor
We have added a parameter --metrics-expose-port=8080 to open the port and expose metrics both for the ray cluster and our control plane. We also leverage the Prometheus Operator to start the whole monitoring system.
You can quickly deploy one by the following on your own kubernetes cluster by using the scripts in install:
./install/prometheus/install.sh
It will set up the prometheus stack and deploy the related service monitor in config/prometheus
Then you can also use the json in config/grafana to generate the dashboards.