Files
kuberay/apiserver/Monitoring.md

3.2 KiB

Monitoring of the API server and created Ray clusters

In order to ensure a proper functioning of the API server and created Ray clusters, it is typically necessary to monitor the API server itself and created Ray clusters. This document describes how to instrument both API server and created clusters with Prometheus and Grafana

Monitoring of the API server

Current implementation of the API server provides a flag collectMetricsFlag that defines whether to collect and expose Prometheus metrics, which is by default set to true. To disable metrics collection, this flag needs to be set to false and the API server image needs to be rebuild.

If this flag is enabled, an http port at /metrics endpoint provides API server metrics in Prometheus format.

Testing monitoring of the API server

On the kind server Install Kubernetes Prometheus Stack via Helm chart using the following command. The script additionally creates ray-head-monitor and ray-workers-monitor in the prometheus-system namespace, that we do not need. We can delete them using:

kubectl delete servicemonitor ray-head-monitor -n prometheus-system
kubectl delete podmonitor ray-workers-monitor -n prometheus-system

To install API server, please follow the instructions from the README.md on installing API server using Helm

Now we can install a service monitor to scrape Api Server metrics into Prometheus using the following command:

kubectl apply -f deploy/prometheus/api_server_service_monitor.yaml

With this in place, you can use port-forward to expose Prometheus:

kubectl port-forward svc/prometheus-operated -n prometheus-system 9090

Now you can point your browser to http://localhost:9090/ to get a PromQL panel. Start typing apiserver and you will see all of the api server metrics in Prometheus.

Monitoring of the Ray Cluster created by the API server

Ray provides documentation describing how to monitor Ray cluster created using KubeRay operator. Because API server is using KubeRay operator to create the cluster. this documentation can be used directly. Here we will show a slightly simpler approach to monitor cluster. Instead of creating service monitor for scraping head node and pod monitor for scraping worker nodes we suggest we create a single pod monitor that can be installed using the following command:

kubectl apply -f deploy/prometheus/ray_cluster_pod_monitor.yaml

With this in place, you can again use port-forward to expose Prometheus:

kubectl port-forward svc/prometheus-operated -n prometheus-system 9090

Now you can point your browser to http://localhost:9090/ to get a PromQL panel. Start typing ray and you will see all of the Ray cluster metrics in Prometheus.

Also take a look at the Ray documentation for additional monitoring features, including Recording rules, Alerts and Grafana integration.