Files
7c7264ab6d [history server][e2e] Add Azure Blob Storage E2E test for History Server (#4460)
* [history server][config] Add Azure Blob Storage deployment manifest

* [history server][e2e] Add Azure History Server support functions

* [history server][e2e] Add Azure Blob Storage E2E test for History Server

* [history server][docs] Add historyserver-azureblob.yaml to config files table

* [history server][e2e] Remove duplicate test support functions

Refactor ApplyHistoryServer to accept an optional manifest path parameter,
eliminating the need for a separate ApplyAzureHistoryServer function.
Remove PrepareAzureHistoryServerTestEnv as it was identical to
PrepareAzureBlobTestEnv.

* fix merge conflict

Signed-off-by: Future-Outlier <[email protected]>

* [history server][e2e] Add dead cluster tests for Azure Blob Storage

Add e2e tests for Azure Blob Storage that verify history server
functionality after cluster deletion:
- testAzureDeadClusters: verifies all endpoints work for dead clusters
- testAzureLogFileEndpointLiveCluster: verifies log file access for live
- testAzureLogFileEndpointDeadCluster: verifies log file access for dead

Extract shared test helpers to avoid duplication between S3 and Azure:
- VerifyLogFileEndpointReturnsContent
- VerifyLogFileEndpointRejectsPathTraversal
- DeleteRayClusterAndWait

* [history server][e2e] Fix Azure dead cluster tests by injecting ray-cluster-id

The Azure dead cluster tests were failing because the collector uploaded
logs to a path based on the hardcoded 'default' namespace, but the history
server searched for clusters in the test namespace.

Add --ray-cluster-id injection to ApplyAzureRayClusterWithCollector,
matching the behavior of the S3 test helper, so logs are uploaded to the
correct namespace-prefixed path.

* [history server][e2e] Fix Azure collector tests to use dynamic namespace

Update Azure collector tests to use namespace.Name instead of hardcoded
RayClusterID constant, matching the S3 collector test pattern. This
ensures the expected blob path matches the actual upload path when
--ray-cluster-id is injected dynamically.

* update

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

---------

Signed-off-by: Future-Outlier <[email protected]>
Co-authored-by: Future-Outlier <[email protected]>
2026-02-10 11:52:34 -05:00

3.3 KiB

KubeRay History Server documentation

Note

The KubeRay History Server is under active development. This directory contains guides to set up and run the History Server components locally for development and testing.

Overview

The KubeRay History Server collects, stores, and displays historical logs and metadata from Ray clusters. It has two parts:

  1. Collector: A sidecar container that runs alongside Ray nodes to collect logs and metadata, then uploads them to blob storage.
  2. History Server: A central service that reads data from blob storage and provides a web interface to explore the history of Ray jobs, tasks, actors, and other cluster activities.

Guides

Guide Description
Collector setup How to set up the Collector on a Kind cluster with MinIO or Azure storage
History Server setup Quick start guide for deploying and using the History Server with API examples

Supported storage backends

The History Server supports multiple storage backends:

Backend Description Configuration
S3/MinIO AWS S3 or MinIO-compatible storage Use --runtime-class-name=s3
Azure Blob Storage Microsoft Azure Blob Storage Use --runtime-class-name=azureblob
Aliyun OSS Alibaba Cloud Object Storage Service Use --runtime-class-name=aliyunoss
Local test For local testing and development Use --runtime-class-name=localtest

Running locally

Prerequisites

  • Go v1.24+
  • Docker
  • Kind
  • kubectl
  • GNU Make

Quick start

  1. Set up the environment: Follow the Collector setup guide to spin up a Kind cluster, deploy the KubeRay operator, and configure storage.

  2. Build the components:

    # Build both collector and history server images
    make -C historyserver localimage-build
    
    # Or build individually
    make -C historyserver localimage-collector
    make -C historyserver localimage-historyserver
    
  3. Load images into Kind:

    kind load docker-image collector:v0.1.0
    kind load docker-image historyserver:v0.1.0
    
  4. Deploy and test: Follow the History Server setup guide to deploy the Ray cluster, submit jobs, and access the History Server API.

Configuration files

Sample configs are in the config/ directory:

File Description
minio.yaml MinIO deployment for S3-compatible storage
azurite.yaml Azurite deployment for Azure Blob Storage emulation
raycluster.yaml Ray cluster with collector sidecar (S3/MinIO)
raycluster-azureblob.yaml Ray cluster with collector sidecar (Azure Blob)
rayjob.yaml Sample Ray job for testing
historyserver.yaml History Server deployment (S3/MinIO)
historyserver-azureblob.yaml History Server deployment (Azure Blob)
service_account.yaml Service account for History Server

Additional resources