* docs: Add hist server local dev with s3 and ray dashboard Signed-off-by: JiangJiaWei1103 <[email protected]> * docs: Update fig url Signed-off-by: JiangJiaWei1103 <[email protected]> * docs: Add head-only raycluster for serving dashboard Signed-off-by: JiangJiaWei1103 <[email protected]> * docs: Add browser screenshots Signed-off-by: JiangJiaWei1103 <[email protected]> * docs: Fix fig link hash Signed-off-by: JiangJiaWei1103 <[email protected]> * Remove redundant manifest Signed-off-by: JiangJiaWei1103 <[email protected]> * docs: Use local ray only Signed-off-by: JiangJiaWei1103 <[email protected]> * docs: Ensure this guide passes by copy-paste all steps Signed-off-by: JiangJiaWei1103 <[email protected]> * nit Signed-off-by: Future-Outlier <[email protected]> --------- Signed-off-by: JiangJiaWei1103 <[email protected]> Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: Future-Outlier <[email protected]>
7.0 KiB
History Server - Local Development with Ray Dashboard
This guide walks through how to develop the History Server locally with MinIO as the S3-compatible object store, and browse the UI through the Ray Dashboard's middleware (from ray-project/ray#61295).
Table of Contents
- Prerequisites
- Step 1: Set Up Kind and KubeRay Operator
- Step 2: Build and Load Images
- Step 3: Deploy MinIO
- Step 4: Generate a Dead Session
- Step 5: Deploy History Server
- Step 6: Access the Local Ray Dashboard
- Step 7: Validate the Dead Path
- Step 8: Validate the Live Path
- Cleanup
- Troubleshooting
Prerequisites
- Kind, kubectl
- Go v1.25+
- Docker
- GNU Make
- K9s (optional)
Step 1: Set Up Kind and KubeRay Operator
Make sure you are under the kuberay directory.
# Spin up the kind cluster.
kind create cluster --image=kindest/node:v1.29.0
# Build and load the operator.
IMG=kuberay/operator:latest make -C ray-operator docker-build
kind load docker-image kuberay/operator:latest
# Install via Helm.
helm install kuberay-operator \
--set image.repository=kuberay/operator \
--set image.tag=latest \
./helm-chart/kuberay-operator
Step 2: Build and Load Images
Build the collector and the history server images, then load both of them into the kind cluster.
# Build images.
make -C historyserver localimage-build
# Load into kind.
kind load docker-image collector:v0.1.0
kind load docker-image historyserver:v0.1.0
# Verify.
docker exec -it kind-control-plane crictl images | grep -E 'collector|historyserver'
Step 3: Deploy MinIO
kubectl apply -f historyserver/config/minio.yaml
Port forward the console and API ports:
kubectl -n minio-dev port-forward svc/minio-service 9001:9001 9000:9000
Step 4: Generate a Dead Session
Important
Finish this step before Step 5: the history server only scans object store on startup and hourly, so a late session won't show up until the next tick (see Troubleshooting).
# 1. Apply the data-generating RayCluster (has the collector sidecar).
kubectl apply -f historyserver/config/raycluster.yaml
kubectl wait pod -l ray.io/cluster=raycluster-historyserver --for=condition=Ready=True --timeout=5m
# 2. Submit the RayJob.
kubectl apply -f historyserver/config/rayjob.yaml
# RayJob uses status.jobStatus (not status.conditions); use a jsonpath wait until the user job succeeds.
kubectl wait rayjob/rayjob --for=jsonpath='{.status.jobStatus}=SUCCEEDED' --timeout=5m
# 3. Delete to trigger a final event/log flush.
kubectl delete -f historyserver/config/rayjob.yaml
kubectl delete -f historyserver/config/raycluster.yaml
Note
Open
http://localhost:9001/browser, log in withminioadmin/minioadmin, and confirm theray-historyserverbucket exists.
Step 5: Deploy History Server
Run the History Server in-cluster so that the dashboard middleware can reach it via http://historyserver:30080.
kubectl apply -f historyserver/config/service_account.yaml
kubectl apply -f historyserver/config/historyserver.yaml
Step 6: Access the Local Ray Dashboard
To access the local Ray Dashboard, you have to port forward the History Server service:
kubectl port-forward svc/historyserver 8080:30080
Install Ray locally. Make sure to use at least Ray v2.55.
pip uninstall -y ray
pip install -U "ray[default]==2.55.0"
Run the ray start command:
ray start --head --num-cpus=1 --proxy-server-url=http://localhost:8080
Note
Use
--proxy-server-urlparameter to route requests to the port-forwarded History Server.
Step 7: Validate the Dead Path
Open a browser and request the /clusters endpoint to view all clusters (including live and dead):
http://localhost:8265/clusters
The result should look something like the following:
Substitute your real <SELECTED_SESSION_ID>:
http://localhost:8265/enter_cluster/default/raycluster-historyserver/<SELECTED_SESSION_ID>
A successful request produces output like the following:
Once set up, you can hit any endpoint via the Ray Dashboard. Take http://localhost:8265/#/cluster as an example:
Step 8: Validate the Live Path
Re-apply the data-generating cluster and submit a RayJob.
kubectl apply -f historyserver/config/raycluster.yaml
kubectl wait pod -l ray.io/cluster=raycluster-historyserver --for=condition=Ready=True --timeout=5m
kubectl apply -f historyserver/config/rayjob.yaml
kubectl wait rayjob/rayjob --for=jsonpath='{.status.jobStatus}=SUCCEEDED' --timeout=5m
In the browser, switch to the live session by visiting:
http://localhost:8265/enter_cluster/default/raycluster-historyserver/live
The result should look like the following:
Now open http://localhost:8265/#/cluster. Unlike Step 7, the History Server reverse-proxies each API
call to the live RayCluster's head dashboard service, so you see real-time state instead of replay:
Cleanup
kubectl delete -f historyserver/config/rayjob.yaml
kubectl delete -f historyserver/config/raycluster.yaml
kubectl delete -f historyserver/config/historyserver.yaml
kubectl delete -f historyserver/config/service_account.yaml
kubectl delete -f historyserver/config/minio.yaml
# Or delete the whole cluster directly.
# kind delete cluster
Troubleshooting
-
Pages are empty but
/clustersand/enter_clusterwork, history server logsTask map not found for cluster session: ...: The EventHandler's in-memory maps were not populated for this session.To fix this problem, force a re-scan by restarting the history server:
kubectl rollout restart deploy/historyserver-demo kubectl rollout status deploy/historyserver-demo # The previous port-forward tunnel dies with the old pod; re-run it. kubectl port-forward svc/historyserver 8080:30080Then, hard-refresh the browser, the UI will now render.




