Files
Jia-Wei JiangandGitHub 2444257068 [1/N] [History Server Beta] [Feat] Lazy loading (#4795)
* refactor: Standardize main.go style

- Add package doc
- Move flag declarations into a var() block
- Add description strings to each flag
- Add section dividers for readability
- Group imports into stdlib / third-party / project
- Remove dead initialization of runtimeClassConfigPath (overwritten by flag default)

No behavior change.

Signed-off-by: JiangJiaWei1103 <[email protected]>

* fix: Surface configuration errors clearly

- Use logrus.Fatalf(...) for consistent log formatting
- Add required-flag check on --runtime-class-name for fast fail

Signed-off-by: JiangJiaWei1103 <[email protected]>

* Refactor: Switch to context-based shutdown via signal.NotifyContext

- Replace manual sigChan + signal.Notify with signal.NotifyContext
- Add bridge goroutine that closes the legacy stop ch when serverCtx fires

No behavior change.

Signed-off-by: JiangJiaWei1103 <[email protected]>

* feat: Add ProcessSingleSession for synchronous per-session loading

Signed-off-by: JiangJiaWei1103 <[email protected]>

* feat: Lazy session processing via Supervisor with isDead probe

- Add a Supervisor as /enter_cluster broker
  - Use in-process loaded set to record processed sessions
- Add a Pipeline that wraps the parse steps
- Remove eager processing on startup and ticker

Signed-off-by: JiangJiaWei1103 <[email protected]>

* docs: Add smoke tests

Signed-off-by: JiangJiaWei1103 <[email protected]>

* test: Cover Supervisor error propagation and no-retry contracts

Signed-off-by: JiangJiaWei1103 <[email protected]>

* docs: Add missing sa

Signed-off-by: JiangJiaWei1103 <[email protected]>

* fix: Surface storage outage error in ProcessSingleSession

Signed-off-by: JiangJiaWei1103 <[email protected]>

* fix: Handle isDead for RayCluster recreation with the same namespacedname

Signed-off-by: JiangJiaWei1103 <[email protected]>

* fix: Avoid setting cookie before a successful load or a live-skip

Signed-off-by: JiangJiaWei1103 <[email protected]>

* fix: Use live sentinel to avoid querying empty in-mem state

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Skip singleflight for already-loaded sessions

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Reuse k8s client

Signed-off-by: JiangJiaWei1103 <[email protected]>

* fix: Surface errors of two subsystems

Signed-off-by: JiangJiaWei1103 <[email protected]>

* docs: Remove redundant comments

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Rename to SessionLoader and LoadSession

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Rename to SessionProcessor for clarity

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Address PR review nits for cleanness

Signed-off-by: JiangJiaWei1103 <[email protected]>

* docs: Clean up logs and comments

Signed-off-by: JiangJiaWei1103 <[email protected]>

* docs: Clean up docstrings

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Rename to SessionStatusClusterStateUnknown to indicate status can not be determined

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Push ctx into ProcessSingleSession to avoid dup ctx err polling

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Consolidate ProcessSingleSession test into eventserver_test.go

Signed-off-by: JiangJiaWei1103 <[email protected]>

* docs: Clean up more comments

Signed-off-by: JiangJiaWei1103 <[email protected]>

* docs: Address more nits for clarity

Signed-off-by: JiangJiaWei1103 <[email protected]>

* fix: Fix assertion string

Signed-off-by: JiangJiaWei1103 <[email protected]>

* fix: Add session input validation and resolve Nary's nits

Signed-off-by: JiangJiaWei1103 <[email protected]>

* fix: Add timeout for a single cold-load to avoid resource leakage

Signed-off-by: JiangJiaWei1103 <[email protected]>

* feat: Add HTTPLiveSessionResolver for determining session name of live cluster

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Remove client-side redundant timeout setup

Signed-off-by: JiangJiaWei1103 <[email protected]>

* fix: Enable cross-ns dns lookup

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Fix nits

Signed-off-by: JiangJiaWei1103 <[email protected]>

* docs: Clarify comments

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Improve error msg, comments and fix nits

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Session key as explicit param of storeEvent without mutating event obj

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Rename process session timeout and make it configurable

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Remove all dead code

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Remove HTTPLiveSessionResolver to simplify live-session detection

Signed-off-by: JiangJiaWei1103 <[email protected]>

* fix: Add zero-value session status as a defensive guard

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Fix nits

Signed-off-by: JiangJiaWei1103 <[email protected]>

* refactor: Reuse sess key gen fn

Signed-off-by: JiangJiaWei1103 <[email protected]>

* docs: Clarify known limitation

Signed-off-by: JiangJiaWei1103 <[email protected]>

---------

Signed-off-by: JiangJiaWei1103 <[email protected]>
2026-05-22 10:16:12 -04:00

6.1 KiB

KubeRay History Server

This project is under active development. See #ray-history-server channel to provide feedback.

Ray History Server is a service for collecting, storing, and viewing historical logs and metadata from Ray clusters. It provides a web interface to explore the history of Ray jobs, tasks, actors, and other cluster activities.

Components

The History Server consists of two main components:

  1. Collector: Runs as a sidecar container in Ray clusters to collect logs and metadata
  2. History Server: Serves a Ray Dashboard compatible HTTP API and ingests cluster sessions' events on demand

Building

Prerequisites

  • Go 1.19 or higher
  • Docker (for building container images)
  • Make

Building Binaries

To build the binaries locally:

make build

This will generate two binaries in the output/bin/ directory:

  • collector: The collector service that runs alongside Ray nodes
  • historyserver: The main history server service

You can also build individual components:

make buildcollector      # Build only the collector
make buildhistoryserver  # Build only the history server

Building Docker Images

To build a Docker image:

make localimage

This creates a Docker image named historyserver:laster with both binaries and necessary assets.

For multi-platform builds, you can use:

docker buildx build -t <image-name>:<tag> --platform linux/amd64,linux/arm64 . --push

Configuration

History Server Configuration

The history server can be configured using command-line flags:

  • --runtime-class-name: Storage backend type (e.g., "s3", "aliyunoss", "localtest")
  • --ray-root-dir: Root directory for Ray logs
  • --kubeconfigs: Path to kubeconfig file(s) for accessing Kubernetes clusters
  • --dashboard-dir: Directory containing dashboard assets (default: "/dashboard")
  • --runtime-class-config-path: Path to runtime class configuration file

Collector Configuration

The collector can be configured using command-line flags:

  • --role: Node role ("Head" or "Worker")
  • --runtime-class-name: Storage backend type (e.g., "s3", "aliyunoss")
  • --ray-cluster-name: Name of the Ray cluster
  • --ray-cluster-namespace: Namespace of the Ray cluster
  • --ray-root-dir: Root directory for Ray logs
  • --log-batching: Number of log entries to batch before writing
  • --events-port: Port for the events server
  • --push-interval: Interval between pushes to storage
  • --runtime-class-config-path: Path to runtime class configuration file

Supported Storage Backends

History Server supports multiple storage backends:

  1. S3/MinIO: For AWS S3 or MinIO compatible storage
  2. Aliyun OSS: For Alibaba Cloud Object Storage Service
  3. Local Test: For local testing and development

Each backend requires specific configuration parameters passed through environment variables or configuration files.

Running

Running the History Server

./output/bin/historyserver \
  --runtime-class-name=s3 \
  --ray-root-dir=/path/to/logs

Running the Collector

./output/bin/collector \
  --role=Head \
  --runtime-class-name=s3 \
  --ray-cluster-name=my-cluster \
  --ray-root-dir=/path/to/logs

Development

Code Structure

  • cmd/: Main applications (collector and historyserver)
  • pkg/: Core logic for storage backends and collection
  • pkg/collector/: Collector-specific code
  • pkg/storage/: Storage backend implementations
  • dashboard/: Web UI files

Testing

To run tests:

make test

Linting

To run lint checks:

make alllint

Smoke Tests

1. Deploy History Server

Apply MinIO and the History Server manifests:

kubectl apply -f historyserver/config/minio.yaml
kubectl apply -f historyserver/config/service_account.yaml
kubectl apply -f historyserver/config/historyserver.yaml

Port-forward the HS service:

kubectl port-forward svc/historyserver 8080:30080

2. Generate a Dead Session

Deploy the sample RayCluster, run a deterministic workload, then delete the CR:

kubectl apply -f historyserver/config/raycluster.yaml
kubectl wait pod -l ray.io/node-type=head --for=condition=Ready --timeout=180s

# Run a workload so events are written to MinIO
kubectl exec $(kubectl get pod -l ray.io/node-type=head -o name) \
  -c ray-head -- python -c "
import ray
ray.init(address='auto')

@ray.remote
def add(x, y): return x + y

print('tasks:', ray.get([add.remote(i, i) for i in range(5)]))
"

# Delete the cluster โ€” produces a 'dead' session
kubectl delete -f historyserver/config/raycluster.yaml

# Discover the session name. /clusters lists both live and dead sessions;
# dead sessions carry the `session_*` name you'll feed into /enter_cluster.
curl -sS http://localhost:8080/clusters

3. Cold Path (first visit)

Trigger the lazy load synchronously. Replace <session> with the session name from ยง2:

time curl -s -o /dev/null \
  http://localhost:8080/enter_cluster/default/raycluster-historyserver/<session>

Note

Cold path runs synchronously: K8s probe + event parse. Expect this to take seconds. Subsequent endpoint calls (/api/v0/jobs, /api/v0/tasks/...) read from in-memory state populated by this call.

4. Warm Path (subsequent visits on same replica)

Re-enter the same cluster:

time curl -s -o /dev/null \
  http://localhost:8080/enter_cluster/default/raycluster-historyserver/<session>

Note

Warm path returns immediately โ€” SessionLoader's loaded-session set fast-paths the request without re-parsing. If the HS process restarts, the next visit returns to cold-path latency.

5. Tail Logs

kubectl logs -f -l app=historyserver --tail=50

You should see one current eventFileList for cluster ... line (and the per-file Reading event file: ... entries) per first-time cold-path call. Warm-path calls produce no parse log lines.

Deployment

History Server can be deployed in Kubernetes using the manifests in the config/samples/ directory. Examples are provided for different storage backends including MinIO and Aliyun OSS.