Files
kuberay/docs/guidance/observability.md
2759c6d841 Cherry pick master fixes to release-0.3 (#478)
* Update helm chart version to 0.3.0 (#461)

* helm-charts/ray-cluster: Allow extra workers (#451)

* helm-charts/ray-cluster: Allow extra workers

We can not set additional workers with the current helm chart.
There are some PRs that aim to fix this but they appear to be stale
https://github.com/ray-project/kuberay/pull/266

We wanted to keep untouched the default worker that is already
included in the helm chart but allow the addition of new workers.

The changes are adding the following logic:
- Allow to disable the default worker group, if needed, via worker.disabled = true
- Allow to set additional workers by using the extraWorkers map

The extraWorkers map has a different structure compared to the default worker because we
wanted to add additional flexibility. Due to that, some variables in the extraWorkers
have different naming (which matches the name of the related CRD attributed)

Note: PR was done to support additional workers in the infrastructure of Metrika.co

* Sync up helm chart's role (#472)

Co-authored-by: chenyu.jiang <[email protected]>

* Document the raycluster status (#473)

Co-authored-by: chenyu.jiang <[email protected]>

Co-authored-by: Christos Kotsis <[email protected]>
Co-authored-by: Chenyu Jiang <[email protected]>
Co-authored-by: chenyu.jiang <[email protected]>
2022-08-16 11:44:43 -07:00

2.5 KiB

Observability

RayCluster Status

State

In the RayCluster resource definition, we use State to represent the current status of the ray cluster.

For now, there are three types of the status exposed by the resouce status: ready, unhealthy and failed.

State Description
ready the ray cluster is ready for use.
unhealthy there is something miss configed in the startParams and the ray cluster may not act correctly
failed there are some severe fatal and result in the head node or worker node start failed.

If you use apiserver to retrieve the resource, you may find the state in the clusterState field.

curl --request GET '<baseUrl>/apis/v1alpha2/namespaces/<namespace>/clusters/<raycluster-name>'
{
    "name": "<raycluster-name>",
    "namespace": "<namespace>",
    //...
    "createdAt": "2022-08-10T10:31:25Z",
    "clusterState": "ready",
    //...
}

Endpoint

If you use the nodeport as service to expose the raycluster endpoint, like dashboard or redis, there are endpoints field in the status to record the service endpoints.

you can directly use the ports in the endpoints to connect to the related service.

Also, if you use apiserver to retrieve the resource, you can find the endpoints in the serviceEndpoint field.

curl --request GET '<baseUrl>/apis/v1alpha2/namespaces/<namespace>/clusters/<raycluster-name>'
{
    "name": "<raycluster-name>",
    "namespace": "<namespace>",
    //...
    "serviceEndpoint": {
        "dashboard": "30001",
        "head": "30002",
        "metrics": "30003",
        "redis": "30004"
    },
    //...
}

Monitor

We have added a parameter --metrics-expose-port=8080 to open the port and expose metrics both for the ray cluster and our control plane. We also leverage the Prometheus Operator to start the whole monitoring system.

You can quickly deploy one by the following on your own kubernetes cluster by using the scripts in install:

./install/prometheus/install.sh

It will set up the prometheus stack and deploy the related service monitor in config/prometheus

Then you can also use the json in config/grafana to generate the dashboards.