Files
1d52354fbd [Docs] Add History Server local development guide (#4719)
* docs: Add hist server local dev with s3 and ray dashboard

Signed-off-by: JiangJiaWei1103 <[email protected]>

* docs: Update fig url

Signed-off-by: JiangJiaWei1103 <[email protected]>

* docs: Add head-only raycluster for serving dashboard

Signed-off-by: JiangJiaWei1103 <[email protected]>

* docs: Add browser screenshots

Signed-off-by: JiangJiaWei1103 <[email protected]>

* docs: Fix fig link hash

Signed-off-by: JiangJiaWei1103 <[email protected]>

* Remove redundant manifest

Signed-off-by: JiangJiaWei1103 <[email protected]>

* docs: Use local ray only

Signed-off-by: JiangJiaWei1103 <[email protected]>

* docs: Ensure this guide passes by copy-paste all steps

Signed-off-by: JiangJiaWei1103 <[email protected]>

* nit

Signed-off-by: Future-Outlier <[email protected]>

---------

Signed-off-by: JiangJiaWei1103 <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Co-authored-by: Future-Outlier <[email protected]>
2026-04-17 14:26:56 -04:00

7.0 KiB

History Server - Local Development with Ray Dashboard

This guide walks through how to develop the History Server locally with MinIO as the S3-compatible object store, and browse the UI through the Ray Dashboard's middleware (from ray-project/ray#61295).

Table of Contents

Prerequisites

  • Kind, kubectl
  • Go v1.25+
  • Docker
  • GNU Make
  • K9s (optional)

Step 1: Set Up Kind and KubeRay Operator

Make sure you are under the kuberay directory.

# Spin up the kind cluster.
kind create cluster --image=kindest/node:v1.29.0

# Build and load the operator.
IMG=kuberay/operator:latest make -C ray-operator docker-build
kind load docker-image kuberay/operator:latest

# Install via Helm.
helm install kuberay-operator \
  --set image.repository=kuberay/operator \
  --set image.tag=latest \
  ./helm-chart/kuberay-operator

Step 2: Build and Load Images

Build the collector and the history server images, then load both of them into the kind cluster.

# Build images.
make -C historyserver localimage-build

# Load into kind.
kind load docker-image collector:v0.1.0
kind load docker-image historyserver:v0.1.0

# Verify.
docker exec -it kind-control-plane crictl images | grep -E 'collector|historyserver'

Step 3: Deploy MinIO

kubectl apply -f historyserver/config/minio.yaml

Port forward the console and API ports:

kubectl -n minio-dev port-forward svc/minio-service 9001:9001 9000:9000

Step 4: Generate a Dead Session

Important

Finish this step before Step 5: the history server only scans object store on startup and hourly, so a late session won't show up until the next tick (see Troubleshooting).

# 1. Apply the data-generating RayCluster (has the collector sidecar).
kubectl apply -f historyserver/config/raycluster.yaml
kubectl wait pod -l ray.io/cluster=raycluster-historyserver --for=condition=Ready=True --timeout=5m

# 2. Submit the RayJob.
kubectl apply -f historyserver/config/rayjob.yaml
# RayJob uses status.jobStatus (not status.conditions); use a jsonpath wait until the user job succeeds.
kubectl wait rayjob/rayjob --for=jsonpath='{.status.jobStatus}=SUCCEEDED' --timeout=5m

# 3. Delete to trigger a final event/log flush.
kubectl delete -f historyserver/config/rayjob.yaml
kubectl delete -f historyserver/config/raycluster.yaml

Note

Open http://localhost:9001/browser, log in with minioadmin / minioadmin, and confirm the ray-historyserver bucket exists.

Step 5: Deploy History Server

Run the History Server in-cluster so that the dashboard middleware can reach it via http://historyserver:30080.

kubectl apply -f historyserver/config/service_account.yaml
kubectl apply -f historyserver/config/historyserver.yaml

Step 6: Access the Local Ray Dashboard

To access the local Ray Dashboard, you have to port forward the History Server service:

kubectl port-forward svc/historyserver 8080:30080

Install Ray locally. Make sure to use at least Ray v2.55.

pip uninstall -y ray
pip install -U "ray[default]==2.55.0"

Run the ray start command:

ray start --head --num-cpus=1 --proxy-server-url=http://localhost:8080

Note

Use --proxy-server-url parameter to route requests to the port-forwarded History Server.

Step 7: Validate the Dead Path

Open a browser and request the /clusters endpoint to view all clusters (including live and dead):

http://localhost:8265/clusters

The result should look something like the following:

clusters_endpoint

Substitute your real <SELECTED_SESSION_ID>:

http://localhost:8265/enter_cluster/default/raycluster-historyserver/<SELECTED_SESSION_ID>

A successful request produces output like the following:

enter_cluster_dead

Once set up, you can hit any endpoint via the Ray Dashboard. Take http://localhost:8265/#/cluster as an example:

dead_raycluster

Step 8: Validate the Live Path

Re-apply the data-generating cluster and submit a RayJob.

kubectl apply -f historyserver/config/raycluster.yaml
kubectl wait pod -l ray.io/cluster=raycluster-historyserver --for=condition=Ready=True --timeout=5m

kubectl apply -f historyserver/config/rayjob.yaml
kubectl wait rayjob/rayjob --for=jsonpath='{.status.jobStatus}=SUCCEEDED' --timeout=5m

In the browser, switch to the live session by visiting:

http://localhost:8265/enter_cluster/default/raycluster-historyserver/live

The result should look like the following:

enter_cluster_live

Now open http://localhost:8265/#/cluster. Unlike Step 7, the History Server reverse-proxies each API call to the live RayCluster's head dashboard service, so you see real-time state instead of replay:

live_raycluster

Cleanup

kubectl delete -f historyserver/config/rayjob.yaml
kubectl delete -f historyserver/config/raycluster.yaml
kubectl delete -f historyserver/config/historyserver.yaml
kubectl delete -f historyserver/config/service_account.yaml
kubectl delete -f historyserver/config/minio.yaml

# Or delete the whole cluster directly.
# kind delete cluster

Troubleshooting

  • Pages are empty but /clusters and /enter_cluster work, history server logs Task map not found for cluster session: ...: The EventHandler's in-memory maps were not populated for this session.

    To fix this problem, force a re-scan by restarting the history server:

    kubectl rollout restart deploy/historyserver-demo
    kubectl rollout status deploy/historyserver-demo
    
    # The previous port-forward tunnel dies with the old pod; re-run it.
    kubectl port-forward svc/historyserver 8080:30080
    

    Then, hard-refresh the browser, the UI will now render.