Files
kuberay/historyserver/docs/set_up_historyserver.md
2026-05-16 21:59:11 -07:00

7.3 KiB

History Server Quick Start Guide

Prerequisites

  • Kind
  • Docker
  • kubectl
  • Go 1.24+

Setup Steps

1. Create Kind Cluster

kind create cluster --image=kindest/node:v1.29.0

2. Build and Run Ray Operator

Build and deploy the KubeRay operator (binary or deployment). For details, please refer to the ray-operator development guide.

3. Deploy & Access MinIO

kubectl apply -f historyserver/config/minio.yaml

Use the following command to port-forward the console and API ports. The API port is required only when running the history server outside the kind cluster.

kubectl --namespace minio-dev port-forward svc/minio-service 9001:9001 9000:9000

Note

Get the correct session directory from MinIO console. Login: minioadmin / minioadmin See: MinIO Setup Guide

4. Build and Load Collector & History Server Images

If you'd like to run the history server outside the Kind cluster, you don't need to build the history server image.

make -C historyserver localimage-build
kind load docker-image historyserver:v0.1.0
kind load docker-image collector:v0.1.0

5. Deploy Ray Cluster

kubectl apply -f historyserver/config/raycluster.yaml

6. Submit Ray Job

kubectl apply -f historyserver/config/rayjob.yaml

7. Delete Ray Cluster (Trigger Log Upload)

kubectl delete -f historyserver/config/raycluster.yaml

8. Create Service Account

kubectl apply -f historyserver/config/service_account.yaml

9. Run and Access History Server

Deploy In-Cluster History Server

kubectl apply -f historyserver/config/historyserver.yaml

# Port-forward to access the history server.
kubectl port-forward svc/historyserver 8080:30080

Run History Server Outside the Kind Cluster

You can also run the history server outside the Kind cluster to accelerate the development iteration and enable debugging in your own IDE. For example, you can set up .vscode/launch.json as follows:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Debug (historyserver)",
            "type": "go",
            "request": "launch",
            "program": "${workspaceFolder}/historyserver/cmd/historyserver/main.go",
            "cwd": "${workspaceFolder}",
            "args": [
                "--runtime-class-name=s3",
                "--ray-root-dir=log"
            ],
            "env": {
                "S3_REGION": "test",
                "S3_ENDPOINT": "localhost:9000",
                "S3_BUCKET": "ray-historyserver",
                "AWS_ACCESS_KEY_ID": "minioadmin",
                "AWS_SECRET_ACCESS_KEY": "minioadmin",
                "AWS_SESSION_TOKEN": "",
                "S3FORCE_PATH_STYLE": "true",
                "S3DISABLE_SSL": "true"
            }
        }
    ]
}

For setting up the args and env fields, please refer to spec.template.spec.containers.command and spec.template.spec.containers.env in historyserver/config/historyserver.yaml.

You can also build and run the history server binary directly from the command line:

# Build the history server binary.
cd historyserver
make buildhistoryserver

# Configure S3 connection via environment variables.
export S3_REGION=test
export S3_ENDPOINT=localhost:9000
export S3_BUCKET=ray-historyserver
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export AWS_SESSION_TOKEN=
export S3FORCE_PATH_STYLE=true
export S3DISABLE_SSL=true

# Run the history server.
./output/bin/historyserver \
  --runtime-class-name=s3 \
  --ray-root-dir=log \
  --use-kubernetes-proxy=true

API Endpoints

Health Check

curl "http://localhost:8080/readz"
curl "http://localhost:8080/livez"

List Clusters

curl "http://localhost:8080/clusters"

Enter a Session (Dead Cluster)

SESSION="session_2026-01-11_19-38-40_146706_1"  # Replace with actual session
curl -c ~/cookies.txt "http://localhost:8080/enter_cluster/default/raycluster-historyserver/$SESSION"

Dead Cluster Endpoints

# All Tasks
curl -b ~/cookies.txt "http://localhost:8080/api/v0/tasks"

# Tasks by job_id
curl -b ~/cookies.txt "http://localhost:8080/api/v0/tasks?filter_keys=job_id&filter_predicates==&filter_values=AgAAAA=="

# Task by task_id
curl -b ~/cookies.txt "http://localhost:8080/api/v0/tasks?filter_keys=task_id&filter_predicates==&filter_values=Z6Loz6WgbbP///////////////8CAAAA"

# All Actors
curl -b ~/cookies.txt "http://localhost:8080/logical/actors"

# Single Actor
curl -b ~/cookies.txt "http://localhost:8080/logical/actors/<ACTOR_ID>"

# Nodes
curl -b ~/cookies.txt "http://localhost:8080/nodes?view=summary"

Enter a Session (Live Cluster)

SESSION="live"
curl -c ~/cookies.txt "http://localhost:8080/enter_cluster/default/raycluster-historyserver/$SESSION"

If the command returns a "RayCluster not found" error, you need to deploy a new, live cluster before connecting:

kubectl apply -f historyserver/config/raycluster.yaml

Then submit a new RayJob:

kubectl apply -f historyserver/config/rayjob.yaml

# If rayjob already exists, please delete it first and re-apply
# kubectl delete -f historyserver/config/rayjob.yaml

Live Cluster Endpoints

Switch to live session first, then:

# All Tasks
curl -b ~/cookies.txt "http://localhost:8080/api/v0/tasks"

# Tasks by job_id
curl -b ~/cookies.txt "http://localhost:8080/api/v0/tasks?filter_keys=job_id&filter_predicates==&filter_values=04000000"

# Task Summarize
curl -b ~/cookies.txt "http://localhost:8080/api/v0/tasks/summarize"

# All Actors
curl -b ~/cookies.txt "http://localhost:8080/logical/actors"

# Single Actor
curl -b ~/cookies.txt "http://localhost:8080/logical/actors/<ACTOR_ID>"

# Nodes Summary
curl -b ~/cookies.txt "http://localhost:8080/nodes?view=summary"

# Jobs
curl -b ~/cookies.txt "http://localhost:8080/api/jobs/"

# Cluster Status
curl -b ~/cookies.txt "http://localhost:8080/api/cluster_status"

Live Cluster with prometheus and grafana

# Install prometheus and grafana. ref: https://docs.ray.io/en/latest/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.html#step-2-install-kubernetes-prometheus-stack-via-helm-chart
./install/prometheus/install.sh --auto-load-dashboard true

# Apply RayCluster with Grafana setting
kubectl apply -f ray-operator/config/samples/ray-cluster.embed-grafana.yaml

# Get live session cookie. (Port-forward is required)
curl -c ~/cookies.txt "http://localhost:8080/enter_cluster/default/raycluster-embed-grafana/live"

# Request to prometheus health endpoint
curl -b ~/cookies.txt http://localhost:8080/api/prometheus_health

# Request to grafana health endpoint
curl -b ~/cookies.txt http://localhost:8080/api/grafana_health

After completing the Prometheus and Grafana testing, you can clean up the associated resources using the following commands:

# 1. Delete the RayCluster
kubectl delete -f ray-operator/config/samples/ray-cluster.embed-grafana.yaml

# 2. Uninstall the kube-prometheus-stack helm chart
helm --namespace prometheus-system uninstall prometheus

# 3. Delete the prometheus-system namespace
kubectl delete namespace prometheus-system