* refactor: Standardize main.go style - Add package doc - Move flag declarations into a var() block - Add description strings to each flag - Add section dividers for readability - Group imports into stdlib / third-party / project - Remove dead initialization of runtimeClassConfigPath (overwritten by flag default) No behavior change. Signed-off-by: JiangJiaWei1103 <[email protected]> * fix: Surface configuration errors clearly - Use logrus.Fatalf(...) for consistent log formatting - Add required-flag check on --runtime-class-name for fast fail Signed-off-by: JiangJiaWei1103 <[email protected]> * Refactor: Switch to context-based shutdown via signal.NotifyContext - Replace manual sigChan + signal.Notify with signal.NotifyContext - Add bridge goroutine that closes the legacy stop ch when serverCtx fires No behavior change. Signed-off-by: JiangJiaWei1103 <[email protected]> * feat: Add ProcessSingleSession for synchronous per-session loading Signed-off-by: JiangJiaWei1103 <[email protected]> * feat: Lazy session processing via Supervisor with isDead probe - Add a Supervisor as /enter_cluster broker - Use in-process loaded set to record processed sessions - Add a Pipeline that wraps the parse steps - Remove eager processing on startup and ticker Signed-off-by: JiangJiaWei1103 <[email protected]> * docs: Add smoke tests Signed-off-by: JiangJiaWei1103 <[email protected]> * test: Cover Supervisor error propagation and no-retry contracts Signed-off-by: JiangJiaWei1103 <[email protected]> * docs: Add missing sa Signed-off-by: JiangJiaWei1103 <[email protected]> * fix: Surface storage outage error in ProcessSingleSession Signed-off-by: JiangJiaWei1103 <[email protected]> * fix: Handle isDead for RayCluster recreation with the same namespacedname Signed-off-by: JiangJiaWei1103 <[email protected]> * fix: Avoid setting cookie before a successful load or a live-skip Signed-off-by: JiangJiaWei1103 <[email protected]> * fix: Use live sentinel to avoid querying empty in-mem state Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Skip singleflight for already-loaded sessions Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Reuse k8s client Signed-off-by: JiangJiaWei1103 <[email protected]> * fix: Surface errors of two subsystems Signed-off-by: JiangJiaWei1103 <[email protected]> * docs: Remove redundant comments Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Rename to SessionLoader and LoadSession Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Rename to SessionProcessor for clarity Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Address PR review nits for cleanness Signed-off-by: JiangJiaWei1103 <[email protected]> * docs: Clean up logs and comments Signed-off-by: JiangJiaWei1103 <[email protected]> * docs: Clean up docstrings Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Rename to SessionStatusClusterStateUnknown to indicate status can not be determined Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Push ctx into ProcessSingleSession to avoid dup ctx err polling Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Consolidate ProcessSingleSession test into eventserver_test.go Signed-off-by: JiangJiaWei1103 <[email protected]> * docs: Clean up more comments Signed-off-by: JiangJiaWei1103 <[email protected]> * docs: Address more nits for clarity Signed-off-by: JiangJiaWei1103 <[email protected]> * fix: Fix assertion string Signed-off-by: JiangJiaWei1103 <[email protected]> * fix: Add session input validation and resolve Nary's nits Signed-off-by: JiangJiaWei1103 <[email protected]> * fix: Add timeout for a single cold-load to avoid resource leakage Signed-off-by: JiangJiaWei1103 <[email protected]> * feat: Add HTTPLiveSessionResolver for determining session name of live cluster Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Remove client-side redundant timeout setup Signed-off-by: JiangJiaWei1103 <[email protected]> * fix: Enable cross-ns dns lookup Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Fix nits Signed-off-by: JiangJiaWei1103 <[email protected]> * docs: Clarify comments Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Improve error msg, comments and fix nits Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Session key as explicit param of storeEvent without mutating event obj Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Rename process session timeout and make it configurable Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Remove all dead code Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Remove HTTPLiveSessionResolver to simplify live-session detection Signed-off-by: JiangJiaWei1103 <[email protected]> * fix: Add zero-value session status as a defensive guard Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Fix nits Signed-off-by: JiangJiaWei1103 <[email protected]> * refactor: Reuse sess key gen fn Signed-off-by: JiangJiaWei1103 <[email protected]> * docs: Clarify known limitation Signed-off-by: JiangJiaWei1103 <[email protected]> --------- Signed-off-by: JiangJiaWei1103 <[email protected]>
6.1 KiB
KubeRay History Server
This project is under active development. See #ray-history-server channel to provide feedback.
Ray History Server is a service for collecting, storing, and viewing historical logs and metadata from Ray clusters. It provides a web interface to explore the history of Ray jobs, tasks, actors, and other cluster activities.
Components
The History Server consists of two main components:
- Collector: Runs as a sidecar container in Ray clusters to collect logs and metadata
- History Server: Serves a Ray Dashboard compatible HTTP API and ingests cluster sessions' events on demand
Building
Prerequisites
- Go 1.19 or higher
- Docker (for building container images)
- Make
Building Binaries
To build the binaries locally:
make build
This will generate two binaries in the output/bin/ directory:
collector: The collector service that runs alongside Ray nodeshistoryserver: The main history server service
You can also build individual components:
make buildcollector # Build only the collector
make buildhistoryserver # Build only the history server
Building Docker Images
To build a Docker image:
make localimage
This creates a Docker image named historyserver:laster with both binaries and necessary assets.
For multi-platform builds, you can use:
docker buildx build -t <image-name>:<tag> --platform linux/amd64,linux/arm64 . --push
Configuration
History Server Configuration
The history server can be configured using command-line flags:
--runtime-class-name: Storage backend type (e.g., "s3", "aliyunoss", "localtest")--ray-root-dir: Root directory for Ray logs--kubeconfigs: Path to kubeconfig file(s) for accessing Kubernetes clusters--dashboard-dir: Directory containing dashboard assets (default: "/dashboard")--runtime-class-config-path: Path to runtime class configuration file
Collector Configuration
The collector can be configured using command-line flags:
--role: Node role ("Head" or "Worker")--runtime-class-name: Storage backend type (e.g., "s3", "aliyunoss")--ray-cluster-name: Name of the Ray cluster--ray-cluster-namespace: Namespace of the Ray cluster--ray-root-dir: Root directory for Ray logs--log-batching: Number of log entries to batch before writing--events-port: Port for the events server--push-interval: Interval between pushes to storage--runtime-class-config-path: Path to runtime class configuration file
Supported Storage Backends
History Server supports multiple storage backends:
- S3/MinIO: For AWS S3 or MinIO compatible storage
- Aliyun OSS: For Alibaba Cloud Object Storage Service
- Local Test: For local testing and development
Each backend requires specific configuration parameters passed through environment variables or configuration files.
Running
Running the History Server
./output/bin/historyserver \
--runtime-class-name=s3 \
--ray-root-dir=/path/to/logs
Running the Collector
./output/bin/collector \
--role=Head \
--runtime-class-name=s3 \
--ray-cluster-name=my-cluster \
--ray-root-dir=/path/to/logs
Development
Code Structure
cmd/: Main applications (collector and historyserver)pkg/: Core logic for storage backends and collectionpkg/collector/: Collector-specific codepkg/storage/: Storage backend implementationsdashboard/: Web UI files
Testing
To run tests:
make test
Linting
To run lint checks:
make alllint
Smoke Tests
1. Deploy History Server
Apply MinIO and the History Server manifests:
kubectl apply -f historyserver/config/minio.yaml
kubectl apply -f historyserver/config/service_account.yaml
kubectl apply -f historyserver/config/historyserver.yaml
Port-forward the HS service:
kubectl port-forward svc/historyserver 8080:30080
2. Generate a Dead Session
Deploy the sample RayCluster, run a deterministic workload, then delete the CR:
kubectl apply -f historyserver/config/raycluster.yaml
kubectl wait pod -l ray.io/node-type=head --for=condition=Ready --timeout=180s
# Run a workload so events are written to MinIO
kubectl exec $(kubectl get pod -l ray.io/node-type=head -o name) \
-c ray-head -- python -c "
import ray
ray.init(address='auto')
@ray.remote
def add(x, y): return x + y
print('tasks:', ray.get([add.remote(i, i) for i in range(5)]))
"
# Delete the cluster โ produces a 'dead' session
kubectl delete -f historyserver/config/raycluster.yaml
# Discover the session name. /clusters lists both live and dead sessions;
# dead sessions carry the `session_*` name you'll feed into /enter_cluster.
curl -sS http://localhost:8080/clusters
3. Cold Path (first visit)
Trigger the lazy load synchronously. Replace <session> with the session name from ยง2:
time curl -s -o /dev/null \
http://localhost:8080/enter_cluster/default/raycluster-historyserver/<session>
Note
Cold path runs synchronously: K8s probe + event parse. Expect this to take seconds. Subsequent endpoint calls (
/api/v0/jobs,/api/v0/tasks/...) read from in-memory state populated by this call.
4. Warm Path (subsequent visits on same replica)
Re-enter the same cluster:
time curl -s -o /dev/null \
http://localhost:8080/enter_cluster/default/raycluster-historyserver/<session>
Note
Warm path returns immediately โ SessionLoader's loaded-session set fast-paths the request without re-parsing. If the HS process restarts, the next visit returns to cold-path latency.
5. Tail Logs
kubectl logs -f -l app=historyserver --tail=50
You should see one current eventFileList for cluster ... line (and the per-file Reading event file: ... entries)
per first-time cold-path call. Warm-path calls produce no parse log lines.
Deployment
History Server can be deployed in Kubernetes using the manifests in the config/samples/ directory.
Examples are provided for different storage backends including MinIO and Aliyun OSS.