* [History server] fix mismatch func signature Signed-off-by: fscnick <[email protected]> * [History server] fix func description Signed-off-by: fscnick <[email protected]> --------- Signed-off-by: fscnick <[email protected]>
KubeRay History Server
This project is under active development. See #ray-history-server channel to provide feedback.
Ray History Server is a service for collecting, storing, and viewing historical logs and metadata from Ray clusters. It provides a web interface to explore the history of Ray jobs, tasks, actors, and other cluster activities.
Components
The History Server consists of two main components:
- Collector: Runs as a sidecar container in Ray clusters to collect logs and metadata
- History Server: Serves a Ray Dashboard compatible HTTP API and ingests cluster sessions' events on demand
Building
Prerequisites
- Go 1.19 or higher
- Docker (for building container images)
- Make
Building Binaries
To build the binaries locally:
make build
This will generate two binaries in the output/bin/ directory:
collector: The collector service that runs alongside Ray nodeshistoryserver: The main history server service
You can also build individual components:
make buildcollector # Build only the collector
make buildhistoryserver # Build only the history server
Building Docker Images
To build a Docker image:
make localimage
This creates a Docker image named historyserver:laster with both binaries and necessary assets.
For multi-platform builds, you can use:
docker buildx build -t <image-name>:<tag> --platform linux/amd64,linux/arm64 . --push
Configuration
History Server Configuration
The history server can be configured using command-line flags:
--runtime-class-name: Storage backend type (e.g., "s3", "aliyunoss", "localtest")--ray-root-dir: Root directory for Ray logs--kubeconfigs: Path to kubeconfig file(s) for accessing Kubernetes clusters--dashboard-dir: Directory containing dashboard assets (default: "/dashboard")--runtime-class-config-path: Path to runtime class configuration file
Collector Configuration
The collector can be configured using command-line flags:
--role: Node role ("Head" or "Worker")--runtime-class-name: Storage backend type (e.g., "s3", "aliyunoss")--ray-cluster-name: Name of the Ray cluster--ray-cluster-namespace: Namespace of the Ray cluster--ray-root-dir: Root directory for Ray logs--log-batching: Number of log entries to batch before writing--events-port: Port for the events server--push-interval: Interval between pushes to storage--runtime-class-config-path: Path to runtime class configuration file
Supported Storage Backends
History Server supports multiple storage backends:
- S3/MinIO: For AWS S3 or MinIO compatible storage
- Aliyun OSS: For Alibaba Cloud Object Storage Service
- Local Test: For local testing and development
Each backend requires specific configuration parameters passed through environment variables or configuration files.
Running
Running the History Server
./output/bin/historyserver \
--runtime-class-name=s3 \
--ray-root-dir=/path/to/logs
Running the Collector
./output/bin/collector \
--role=Head \
--runtime-class-name=s3 \
--ray-cluster-name=my-cluster \
--ray-root-dir=/path/to/logs
Development
Code Structure
cmd/: Main applications (collector and historyserver)pkg/: Core logic for storage backends and collectionpkg/collector/: Collector-specific codepkg/storage/: Storage backend implementationsdashboard/: Web UI files
Testing
To run tests:
make test
Linting
To run lint checks:
make alllint
Smoke Tests
1. Deploy History Server
Apply MinIO and the History Server manifests:
kubectl apply -f historyserver/config/minio.yaml
kubectl apply -f historyserver/config/service_account.yaml
kubectl apply -f historyserver/config/historyserver.yaml
Port-forward the HS service:
kubectl port-forward svc/historyserver 8080:30080
2. Generate a Dead Session
Deploy the sample RayCluster, run a deterministic workload, then delete the CR:
kubectl apply -f historyserver/config/raycluster.yaml
kubectl wait pod -l ray.io/node-type=head --for=condition=Ready --timeout=180s
# Run a workload so events are written to MinIO
kubectl exec $(kubectl get pod -l ray.io/node-type=head -o name) \
-c ray-head -- python -c "
import ray
ray.init(address='auto')
@ray.remote
def add(x, y): return x + y
print('tasks:', ray.get([add.remote(i, i) for i in range(5)]))
"
# Delete the cluster — produces a 'dead' session
kubectl delete -f historyserver/config/raycluster.yaml
# Discover the session name. /clusters lists both live and dead sessions;
# dead sessions carry the `session_*` name you'll feed into /enter_cluster.
curl -sS http://localhost:8080/clusters
3. Cold Path (first visit)
Trigger the lazy load synchronously. Replace <session> with the session name from §2:
time curl -s -o /dev/null \
http://localhost:8080/enter_cluster/default/raycluster-historyserver/<session>
Note
Cold path runs synchronously: K8s probe + event parse. Expect this to take seconds. Subsequent endpoint calls (
/api/v0/jobs,/api/v0/tasks/...) read from in-memory state populated by this call.
4. Warm Path (subsequent visits on same replica)
Re-enter the same cluster:
time curl -s -o /dev/null \
http://localhost:8080/enter_cluster/default/raycluster-historyserver/<session>
Note
Warm path returns immediately — SessionLoader's loaded-session set fast-paths the request without re-parsing. If the HS process restarts, the next visit returns to cold-path latency.
5. Tail Logs
kubectl logs -f -l app=historyserver --tail=50
You should see one current eventFileList for cluster ... line (and the per-file Reading event file: ... entries)
per first-time cold-path call. Warm-path calls produce no parse log lines.
Deployment
History Server can be deployed in Kubernetes using the manifests in the config/samples/ directory.
Examples are provided for different storage backends including MinIO and Aliyun OSS.