mirror of
https://github.com/hanzo-ml/kuberay.git
synced 2026-10-30 19:48:47 +00:00
+6








a5adbbe18b
* [RayService] More envtests that follow the most common scenario in the RayService code path (#2880) Signed-off-by: Rueian <[email protected]> * [RayService] Remove outdated env tests (#2886) Signed-off-by: kaihsun <[email protected]> * [RayService] Refactor envtests (#2888) * [docs][ray-operator] fix typo in Golang version (#2893) Project uses Golang 1.22 not 1.20. Signed-off-by: David Xia <[email protected]> * [Fix][kubectl-plugin] Fix no context nil error SIGSEGV in tests (#2892) Signed-off-by: Chi-Sheng Liu <[email protected]> * [CI] Enable testifylint rule (#2896) Signed-off-by: Chi-Sheng Liu <[email protected]> * [CI] Enable `testifylint` `error-nil` rule (#2907) * [kubectl-plugin][feat] support specifying number of head GPUs (#2895) when creating a RayCluster with `kubectl ray create cluster NAME --head-gpu N`. Similar to the `--worker-gpu` switch. Signed-off-by: David Xia <[email protected]> * Use webhook.CustomValidator instead of deprecated webhook.Validator. (#2803) * Use webhook.CustomValidator instead of deprecated webhook.Validator. * Move RayClusterWebhook to pkg/webhook/v1. * [kubectl-plugin] update context error messages (#2891) to tell user they can use `--context` to set the K8s context. Add tests Signed-off-by: David Xia <[email protected]> Co-authored-by: Chi-Sheng Liu <[email protected]> * [Fix][kubectl-plugin] make tests use a temporary kube config (#2894) file that's not at the default path. Right now tests fail if there's a K8s current context set with a command like `kubectl config use-context my-context`. This change allows tests to pass regardless of current context. Signed-off-by: David Xia <[email protected]> * [CI] Enable testifylint formatter rule (#2915) Signed-off-by: 400Ping <[email protected]> * [CI] Enable `testifylint` `empty` rule (#2908) * [CI] Enable testifylint bool-compare rule (#2911) Signed-off-by: 400Ping <[email protected]> * [CI] Auto download golang tools in pre-commit (#2917) Signed-off-by: Chi-Sheng Liu <[email protected]> * [RayService] a safeguard for preventing overriding the pending cluster during a upgrade (#2887) Signed-off-by: Rueian <[email protected]> * [RayService] Refactor unit tests for ShouldPrepareNewCluster (#2928) Signed-off-by: kaihsun <[email protected]> * [chore][kubectl-plugin] use consistent capitalization (#2922) in comments and console messages. * capitalize "Ray" when used as a proper noun * capitalize Ray K8s CRDs like "RayJob" and "RayCluster" * capitalize acronyms like "YAML" * fix some minor typos No functional changes. Signed-off-by: David Xia <[email protected]> * [CI] Enable `testifylint` `require-error` rule (#2909) * [kubectl-plugin] support general `kubectl` switches like `--context` (#2883) * [CI] Fix lint error (require-error) (#2931) Signed-off-by: Chi-Sheng Liu <[email protected]> * [CI] Enable `testifylint` `float-compare` rule (#2910) * [docs] move pre-commit instructions to main dev docs (#2921) * [CI] Enable `testifylint` `expected-actual` rule (#2914) * [CI] Generate CRD json schema separately in pre-commit (#2930) * [release][1/N] Update YAMLs from Ray 2.9 to Ray 2.41 (#2934) * Delete `[raycluster|rayjob|rayservice]_types_test.go` unnecessary tests (#2935) * [release][2/N] Update RayCluster Helm chart from Ray 2.9 to Ray 2.41 (#2936) Signed-off-by: kaihsun <[email protected]> * [release][3/N] Update RayService e2e tests YAML files from Ray 2.9 to Ray 2.41 (#2937) * [release][4/N] Update Ray images / versions in kubectl plugin (#2938) Signed-off-by: kaihsun <[email protected]> * [release][5/N] Update some RayJob YAMLs from Ray 2.9 to Ray 2.41 (#2941) Signed-off-by: kaihsun <[email protected]> * [release][6/N] Remove unnecessary YAMLs (#2946) Signed-off-by: kaihsun <[email protected]> * [docs][kubectl-plugin] add dev docs (#2912) Signed-off-by: David Xia <[email protected]> Co-authored-by: Chi-Sheng Liu <[email protected]> * [CI] Add shellcheck and fix error of it (#2933) * [chore][kubectl-plugin] use better test assertions (#2955) Fix two places. * use `assert.EqualError()` instead of `assert.Error()` to check the error string is what we expect * use testify assertion instead of string checking for better error output Signed-off-by: David Xia <[email protected]> * [chore] add Markdown linting pre-commit hook (#2953) Ignore most rules we violate for now. Fix these two violations. ``` pre-commit run markdownlint --all-files --show-diff-on-failure ... clients/python-client/README.md:106 MD001/heading-increment Heading levels should only increment by one level at a time [Expected: h3; Actual: h4] CHANGELOG.md:420:1 MD030/list-marker-space Spaces after list markers [Expected: 1; Actual: 2] ``` Signed-off-by: David Xia <[email protected]> * [chore][kubectl-plugin] use consistent capitalization (#2950) in comments and console messages. capitalize "Ray" when used as a proper noun. Similar to #2922. Signed-off-by: David Xia <[email protected]> * [Fix][RayJob] Invalid quote for RayJob submitter (#2949) Closes: ray-project/kuberay#2943 Signed-off-by: Chi-Sheng Liu <[email protected]> * [release][7/N] Update RayService YAMLs (#2956) * [CI] Enable testifylint len rule (#2945) Signed-off-by: LeoLiao123 <[email protected]> * [docs][kubectl-plugin] improve help messages (#2952) Signed-off-by: David Xia <[email protected]> * [kubectl-plugin] Fix panic when GPU resource is not set (#2954) Signed-off-by: win5923 <[email protected]> * [release][8/N] Upgrade Stable Diffusion RayService to Ray 2.41 (#2960) * [docs][kubectl-plugin] fix incorrect example commands (#2951) Signed-off-by: David Xia <[email protected]> Co-authored-by: Chi-Sheng Liu <[email protected]> * [fix][kubectl-plugin] set worker group CPU limit (#2958) when creating a new worker group with `kubectl ray create workergroup`. Write a unit test. I noticed we are setting resource limits equal to resource requests everywhere else but in this command. I have a K8s [LimitRange] that prevented the creation of these worker Pods that had CPU limit defaulting to less than their CPU requests. Describing the RayCluster showed this warning event. `Failed to create worker Pod hyperkube/, Pod "dxia-test-other-group-worker-pm2sh" is invalid: spec.containers[0].resources.requests: Invalid value: "2": must be less than or equal to cpu limit of 250m` Signed-off-by: David Xia <[email protected]> [LimitRange]: https://kubernetes.io/docs/concepts/policy/limit-range/ * [RayJob] Deflaky RayJob e2e tests (#2963) Signed-off-by: kaihsun <[email protected]> * [RayService] Deflaky RayService envtest (#2962) Signed-off-by: kaihsun <[email protected]> * [release][9/N] Update text summarizer RayService to Ray 2.41 (#2961) * [RayService] adapter vllm 0.6.1.post2 (#2823) * adapter vllm 0.6.1.post2 * fix var define * Unify the cpu Settings in serve.py in service.yaml, all set to 1 * Maintain the configuration with vllm0.5x * [Release] Upgrade ray-job.batch-inference.yaml image to 2.41 (#2971) Signed-off-by: Chi-Sheng Liu <[email protected]> * [chore][docs] enable Markdownlint rule MD010 (#2975) * [CI] Change Pre-commit-shellcheck-to-shellcheck-py (#2974) Signed-off-by: owenowenisme <[email protected]> * [release] Update Yunikorn YAML file to Ray 2.41 (#2969) Signed-off-by: Cheng-Yeh Chung <[email protected]> * [release] Update YuniKorn YAML files to Ray 2.41 (#2976) Signed-off-by: win5923 <[email protected]> * [chore][docs] enable Markdownlint rule MD004 (#2973) [Unordered list style][1] [1]: https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md004---unordered-list-style * [Test] Use GcsFaultToleranceOptions in test and backward compatibility (#2972) * Update samples to use Ray 2.41.0 images (#2964) Signed-off-by: Andrew Sy Kim <[email protected]> * Update TPU Ray CR manifests to use Ray 2.41.0 (#2965) * [Refactor] Use constants for image tag, image repo, and versions in golang to avoid hard-coded strings (#2978) --------- Signed-off-by: Rueian <[email protected]> Signed-off-by: kaihsun <[email protected]> Signed-off-by: David Xia <[email protected]> Signed-off-by: Chi-Sheng Liu <[email protected]> Signed-off-by: 400Ping <[email protected]> Signed-off-by: LeoLiao123 <[email protected]> Signed-off-by: win5923 <[email protected]> Signed-off-by: owenowenisme <[email protected]> Signed-off-by: Cheng-Yeh Chung <[email protected]> Signed-off-by: Andrew Sy Kim <[email protected]> Co-authored-by: Rueian <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: David Xia <[email protected]> Co-authored-by: Chi-Sheng Liu <[email protected]> Co-authored-by: David Xia <[email protected]> Co-authored-by: Mykhailo Bobrovskyi <[email protected]> Co-authored-by: Ping <[email protected]> Co-authored-by: Owen Lin <[email protected]> Co-authored-by: Leo Liao <[email protected]> Co-authored-by: Blocka <[email protected]> Co-authored-by: zrant <[email protected]> Co-authored-by: kenchung285 <[email protected]> Co-authored-by: fscnick <[email protected]> Co-authored-by: ryanaoleary <[email protected]>
515 lines
22 KiB
Go
515 lines
22 KiB
Go
/*
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License");
|
|
you may not use this file except in compliance with the License.
|
|
You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License.
|
|
*/
|
|
|
|
package ray
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"math/rand/v2"
|
|
"strconv"
|
|
"time"
|
|
|
|
"github.com/ray-project/kuberay/ray-operator/controllers/ray/utils"
|
|
"github.com/ray-project/kuberay/ray-operator/test/support"
|
|
|
|
. "github.com/onsi/ginkgo/v2"
|
|
. "github.com/onsi/gomega"
|
|
|
|
rayv1 "github.com/ray-project/kuberay/ray-operator/apis/ray/v1"
|
|
|
|
corev1 "k8s.io/api/core/v1"
|
|
"k8s.io/client-go/util/retry"
|
|
"k8s.io/utils/ptr"
|
|
|
|
"k8s.io/apimachinery/pkg/api/meta"
|
|
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
|
"sigs.k8s.io/controller-runtime/pkg/client"
|
|
// +kubebuilder:scaffold:imports
|
|
)
|
|
|
|
func serveConfigV2Template(serveAppName string) string {
|
|
return fmt.Sprintf(`
|
|
applications:
|
|
- name: %s
|
|
import_path: fruit.deployment_graph
|
|
route_prefix: /fruit
|
|
runtime_env:
|
|
working_dir: "https://github.com/ray-project/test_dag/archive/41d09119cbdf8450599f993f51318e9e27c59098.zip"
|
|
deployments:
|
|
- name: MangoStand
|
|
num_replicas: 1
|
|
user_config:
|
|
price: 3
|
|
ray_actor_options:
|
|
num_cpus: 0.1
|
|
- name: OrangeStand
|
|
num_replicas: 1
|
|
user_config:
|
|
price: 2
|
|
ray_actor_options:
|
|
num_cpus: 0.1
|
|
- name: PearStand
|
|
num_replicas: 1
|
|
user_config:
|
|
price: 1
|
|
ray_actor_options:
|
|
num_cpus: 0.1`, serveAppName)
|
|
}
|
|
|
|
func rayServiceTemplate(name string, namespace string, serveAppName string) *rayv1.RayService {
|
|
serveConfigV2 := serveConfigV2Template(serveAppName)
|
|
return &rayv1.RayService{
|
|
ObjectMeta: metav1.ObjectMeta{
|
|
Name: name,
|
|
Namespace: namespace,
|
|
},
|
|
Spec: rayv1.RayServiceSpec{
|
|
ServeConfigV2: serveConfigV2,
|
|
UpgradeStrategy: &rayv1.RayServiceUpgradeStrategy{
|
|
Type: ptr.To(rayv1.NewCluster),
|
|
},
|
|
RayClusterSpec: rayv1.RayClusterSpec{
|
|
RayVersion: support.GetRayVersion(),
|
|
HeadGroupSpec: rayv1.HeadGroupSpec{
|
|
RayStartParams: map[string]string{},
|
|
Template: corev1.PodTemplateSpec{
|
|
Spec: corev1.PodSpec{
|
|
Containers: []corev1.Container{
|
|
{
|
|
Name: "ray-head",
|
|
Image: support.GetRayImage(),
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
WorkerGroupSpecs: []rayv1.WorkerGroupSpec{
|
|
{
|
|
Replicas: ptr.To[int32](3),
|
|
MinReplicas: ptr.To[int32](0),
|
|
MaxReplicas: ptr.To[int32](10000),
|
|
GroupName: "small-group",
|
|
RayStartParams: map[string]string{},
|
|
Template: corev1.PodTemplateSpec{
|
|
Spec: corev1.PodSpec{
|
|
Containers: []corev1.Container{
|
|
{
|
|
Name: "ray-worker",
|
|
Image: support.GetRayImage(),
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
},
|
|
}
|
|
}
|
|
|
|
func endpointsTemplate(name string, namespace string) *corev1.Endpoints {
|
|
return &corev1.Endpoints{
|
|
ObjectMeta: metav1.ObjectMeta{
|
|
Name: name,
|
|
Namespace: namespace,
|
|
},
|
|
Subsets: []corev1.EndpointSubset{
|
|
{
|
|
Addresses: []corev1.EndpointAddress{
|
|
{
|
|
IP: "10.9.8.7",
|
|
},
|
|
},
|
|
},
|
|
},
|
|
}
|
|
}
|
|
|
|
func appendNewWorkerGroup(workerGroupSpecs []rayv1.WorkerGroupSpec, newGroupName string) []rayv1.WorkerGroupSpec {
|
|
newWorkerGroupSpec := workerGroupSpecs[0].DeepCopy()
|
|
newWorkerGroupSpec.GroupName = newGroupName
|
|
return append(workerGroupSpecs, *newWorkerGroupSpec)
|
|
}
|
|
|
|
var _ = Context("RayService env tests", func() {
|
|
Describe("Autoscaler updates RayCluster should not trigger zero downtime upgrade", Ordered, func() {
|
|
// If Autoscaler scales up the pending or active RayCluster, zero downtime upgrade should not be triggered.
|
|
ctx := context.Background()
|
|
namespace := "default"
|
|
serveAppName := "app1"
|
|
rayService := rayServiceTemplate("test-autoscaler", namespace, serveAppName)
|
|
rayCluster := &rayv1.RayCluster{}
|
|
|
|
It("Create a RayService custom resource", func() {
|
|
err := k8sClient.Create(ctx, rayService)
|
|
Expect(err).NotTo(HaveOccurred(), "failed to create RayService resource")
|
|
Eventually(
|
|
getResourceFunc(ctx, client.ObjectKey{Name: rayService.Name, Namespace: namespace}, rayService),
|
|
time.Second*3, time.Millisecond*500).Should(BeNil(), "RayService: %v", rayService.Name)
|
|
})
|
|
|
|
It("Should create a pending RayCluster", func() {
|
|
Eventually(
|
|
getPreparingRayClusterNameFunc(ctx, rayService),
|
|
time.Second*15, time.Millisecond*500).Should(Not(BeEmpty()), "Pending RayCluster name: %v", rayService.Status.PendingServiceStatus.RayClusterName)
|
|
})
|
|
|
|
It("Autoscaler updates the pending RayCluster and should not switch to a new RayCluster", func() {
|
|
// Simulate autoscaler by updating the pending RayCluster directly. Note that the autoscaler
|
|
// will not update the RayService directly.
|
|
clusterName, _ := getPreparingRayClusterNameFunc(ctx, rayService)()
|
|
err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
|
|
Eventually(
|
|
getResourceFunc(ctx, client.ObjectKey{Name: clusterName, Namespace: namespace}, rayCluster),
|
|
time.Second*3, time.Millisecond*500).Should(BeNil(), "Pending RayCluster: %v", rayCluster.Name)
|
|
*rayCluster.Spec.WorkerGroupSpecs[0].Replicas++
|
|
return k8sClient.Update(ctx, rayCluster)
|
|
})
|
|
Expect(err).NotTo(HaveOccurred(), "Failed to update the pending RayCluster.")
|
|
|
|
// Confirm not switch to a new RayCluster
|
|
Consistently(
|
|
getPreparingRayClusterNameFunc(ctx, rayService),
|
|
time.Second*5, time.Millisecond*500).Should(Equal(clusterName), "Pending RayCluster: %v", rayService.Status.PendingServiceStatus.RayClusterName)
|
|
})
|
|
|
|
It("Promote the pending RayCluster to the active RayCluster", func() {
|
|
// Update the status of the head Pod to Running. Note that the default fake dashboard client
|
|
// will return a healthy serve application status.
|
|
pendingRayClusterName := rayService.Status.PendingServiceStatus.RayClusterName
|
|
updateHeadPodToRunningAndReady(ctx, pendingRayClusterName, namespace)
|
|
|
|
// Make sure the pending RayCluster becomes the active RayCluster.
|
|
Eventually(
|
|
getRayClusterNameFunc(ctx, rayService),
|
|
time.Second*15, time.Millisecond*500).Should(Equal(pendingRayClusterName), "Active RayCluster name: %v", rayService.Status.ActiveServiceStatus.RayClusterName)
|
|
|
|
// Initialize RayCluster for the following tests.
|
|
Eventually(
|
|
getResourceFunc(ctx, client.ObjectKey{Name: rayService.Status.ActiveServiceStatus.RayClusterName, Namespace: namespace}, rayCluster),
|
|
time.Second*3, time.Millisecond*500).Should(BeNil(), "RayCluster: %v", rayCluster.Name)
|
|
})
|
|
|
|
It("Autoscaler updates the active RayCluster and should not switch to a new RayCluster", func() {
|
|
// Simulate autoscaler by updating the active RayCluster directly. Note that the autoscaler
|
|
// will not update the RayService directly.
|
|
clusterName, _ := getRayClusterNameFunc(ctx, rayService)()
|
|
err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
|
|
Eventually(
|
|
getResourceFunc(ctx, client.ObjectKey{Name: clusterName, Namespace: namespace}, rayCluster),
|
|
time.Second*3, time.Millisecond*500).Should(BeNil(), "Active RayCluster: %v", rayCluster.Name)
|
|
*rayCluster.Spec.WorkerGroupSpecs[0].Replicas++
|
|
return k8sClient.Update(ctx, rayCluster)
|
|
})
|
|
Expect(err).NotTo(HaveOccurred(), "Failed to update the active RayCluster.")
|
|
|
|
// Confirm not switch to a new RayCluster
|
|
Consistently(
|
|
getRayClusterNameFunc(ctx, rayService),
|
|
time.Second*5, time.Millisecond*500).Should(Equal(clusterName), "Active RayCluster: %v", rayService.Status.ActiveServiceStatus.RayClusterName)
|
|
})
|
|
})
|
|
|
|
Describe("After a RayService is running", func() {
|
|
ctx := context.Background()
|
|
var rayService *rayv1.RayService
|
|
var rayCluster *rayv1.RayCluster
|
|
var endpoints *corev1.Endpoints
|
|
serveAppName := "app1"
|
|
namespace := "default"
|
|
|
|
BeforeEach(OncePerOrdered, func() {
|
|
// This simulates the most common scenario in the RayService code path:
|
|
// (1) Create a RayService custom resource
|
|
// (2) The RayService controller creates a pending RayCluster
|
|
// (3) The serve application becomes ready on the pending RayCluster
|
|
// (4) The Kubernetes head and serve services are created
|
|
// (5) The pending RayCluster transitions to become the active RayCluster
|
|
rayService = rayServiceTemplate("test-base-path-"+strconv.Itoa(rand.IntN(1000)), namespace, serveAppName) //nolint:gosec // no need for cryptographically secure random number
|
|
rayCluster = &rayv1.RayCluster{}
|
|
|
|
By("Create a RayService custom resource")
|
|
err := k8sClient.Create(ctx, rayService)
|
|
Expect(err).NotTo(HaveOccurred(), "failed to create RayService resource")
|
|
Eventually(
|
|
getResourceFunc(ctx, client.ObjectKey{Name: rayService.Name, Namespace: namespace}, rayService),
|
|
time.Second*3, time.Millisecond*500).Should(BeNil(), "RayService: %v", rayService.Name)
|
|
|
|
By("Conditions should be initialized correctly")
|
|
Eventually(
|
|
func() bool {
|
|
return meta.IsStatusConditionTrue(rayService.Status.Conditions, string(rayv1.UpgradeInProgress))
|
|
},
|
|
time.Second*3, time.Millisecond*500).Should(BeFalse(), "UpgradeInProgress condition: %v", rayService.Status.Conditions)
|
|
Eventually(
|
|
func() bool {
|
|
return meta.IsStatusConditionTrue(rayService.Status.Conditions, string(rayv1.RayServiceReady))
|
|
},
|
|
time.Second*3, time.Millisecond*500).Should(BeFalse(), "RayServiceReady condition: %v", rayService.Status.Conditions)
|
|
|
|
By("Should create a pending RayCluster")
|
|
Eventually(
|
|
getPreparingRayClusterNameFunc(ctx, rayService),
|
|
time.Second*15, time.Millisecond*500).Should(Not(BeEmpty()), "Pending RayCluster name: %v", rayService.Status.PendingServiceStatus.RayClusterName)
|
|
|
|
By("Promote the pending RayCluster to the active RayCluster")
|
|
// Update the status of the head Pod to Running. Note that the default fake dashboard client
|
|
// will return a healthy serve application status.
|
|
pendingRayClusterName := rayService.Status.PendingServiceStatus.RayClusterName
|
|
updateHeadPodToRunningAndReady(ctx, pendingRayClusterName, namespace)
|
|
|
|
// Make sure the pending RayCluster becomes the active RayCluster.
|
|
Eventually(
|
|
getRayClusterNameFunc(ctx, rayService),
|
|
time.Second*15, time.Millisecond*500).Should(Equal(pendingRayClusterName), "Active RayCluster name: %v", rayService.Status.ActiveServiceStatus.RayClusterName)
|
|
|
|
// Initialize RayCluster for the following tests.
|
|
Eventually(
|
|
getResourceFunc(ctx, client.ObjectKey{Name: rayService.Status.ActiveServiceStatus.RayClusterName, Namespace: namespace}, rayCluster),
|
|
time.Second*3, time.Millisecond*500).Should(BeNil(), "RayCluster: %v", rayCluster.Name)
|
|
|
|
By("Check the serve application status in the RayService status")
|
|
// Check the serve application status in the RayService status.
|
|
// The serve application should be healthy.
|
|
Eventually(
|
|
checkServiceHealth(ctx, rayService),
|
|
time.Second*3, time.Millisecond*500).Should(BeTrue(), "RayService status: %v", rayService.Status)
|
|
|
|
By("Should create a new head service resource")
|
|
svc := &corev1.Service{}
|
|
headSvcName, err := utils.GenerateHeadServiceName(utils.RayServiceCRD, rayService.Spec.RayClusterSpec, rayService.Name)
|
|
Expect(err).ToNot(HaveOccurred())
|
|
Eventually(
|
|
getResourceFunc(ctx, client.ObjectKey{Name: headSvcName, Namespace: namespace}, svc),
|
|
time.Second*3, time.Millisecond*500).Should(BeNil(), "Head service: %v", svc)
|
|
// TODO: Verify the head service by checking labels and annotations.
|
|
|
|
By("Should create a new serve service resource")
|
|
svc = &corev1.Service{}
|
|
Eventually(
|
|
getResourceFunc(ctx, client.ObjectKey{Name: utils.GenerateServeServiceName(rayService.Name), Namespace: namespace}, svc),
|
|
time.Second*3, time.Millisecond*500).Should(BeNil(), "Serve service: %v", svc)
|
|
// TODO: Verify the serve service by checking labels and annotations.
|
|
|
|
By("The RayServiceReady condition should be true when the number of endpoints is greater than 0")
|
|
endpoints = endpointsTemplate(utils.GenerateServeServiceName(rayService.Name), namespace)
|
|
err = k8sClient.Create(ctx, endpoints)
|
|
Expect(err).NotTo(HaveOccurred(), "failed to create Endpoints resource")
|
|
Eventually(func() int32 {
|
|
if err := k8sClient.Get(ctx, client.ObjectKey{Name: rayService.Name, Namespace: namespace}, rayService); err != nil {
|
|
return 0
|
|
}
|
|
return rayService.Status.NumServeEndpoints
|
|
}, time.Second*3, time.Millisecond*500).Should(BeNumerically(">", 0), "RayService status: %v", rayService.Status)
|
|
Expect(meta.IsStatusConditionTrue(rayService.Status.Conditions, string(rayv1.RayServiceReady))).Should(BeTrue())
|
|
})
|
|
|
|
AfterEach(OncePerOrdered, func() {
|
|
By(fmt.Sprintf("Delete the RayService custom resource %v", rayService.Name))
|
|
err := k8sClient.Delete(ctx, rayService)
|
|
Expect(err).NotTo(HaveOccurred(), "failed to delete the test RayService resource")
|
|
By(fmt.Sprintf("Delete the Endpoints %v", endpoints.Name))
|
|
err = k8sClient.Delete(ctx, endpoints)
|
|
Expect(err).NotTo(HaveOccurred(), "failed to delete the test Endpoints resource")
|
|
})
|
|
|
|
When("Testing in-place update: updating the serveConfigV2", Ordered, func() {
|
|
// Update serveConfigV2 to trigger an in-place update. The new serve application should appear in the RayService status,
|
|
// and a zero-downtime upgrade should not be triggered.
|
|
var newConfigV2 string
|
|
newServeAppName := "newAppName"
|
|
|
|
BeforeAll(func() {
|
|
newConfigV2 = serveConfigV2Template(newServeAppName)
|
|
|
|
// Update the serveConfigV2 to trigger an in-place update.
|
|
err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
|
|
Eventually(
|
|
getResourceFunc(ctx, client.ObjectKey{Name: rayService.Name, Namespace: namespace}, rayService),
|
|
time.Second*3, time.Millisecond*500).Should(BeNil(), "RayService: %v", rayService.Name)
|
|
rayService.Spec.ServeConfigV2 = newConfigV2
|
|
return k8sClient.Update(ctx, rayService)
|
|
})
|
|
|
|
Expect(err).NotTo(HaveOccurred(), "failed to update RayService resource")
|
|
|
|
// Update the fake Ray dashboard client to return serve application statuses with the new serve application.
|
|
healthyStatus := generateServeStatus(rayv1.DeploymentStatusEnum.HEALTHY, rayv1.ApplicationStatusEnum.RUNNING)
|
|
fakeRayDashboardClient.SetMultiApplicationStatuses(map[string]*utils.ServeApplicationStatus{newServeAppName: &healthyStatus})
|
|
})
|
|
|
|
It("New serve application should be shown in the RayService status", func() {
|
|
Eventually(checkServeApplicationExists(ctx, rayService, "newAppName"), time.Second*10, time.Millisecond*500).Should(BeTrue())
|
|
})
|
|
|
|
It("Should create an UpdatedServeApplications event", func() {
|
|
var eventList corev1.EventList
|
|
listOpts := []client.ListOption{
|
|
client.InNamespace(rayService.Namespace),
|
|
client.MatchingFields{
|
|
"involvedObject.uid": string(rayService.UID),
|
|
"reason": string(utils.UpdatedServeApplications),
|
|
},
|
|
}
|
|
err := k8sClient.List(ctx, &eventList, listOpts...)
|
|
Expect(err).NotTo(HaveOccurred(), "failed to list events")
|
|
Expect(eventList.Items).To(HaveLen(1))
|
|
})
|
|
|
|
It("Refresh RayService", func() {
|
|
err := k8sClient.Get(ctx, client.ObjectKey{Name: rayService.Name, Namespace: rayService.Namespace}, rayService)
|
|
Expect(err).NotTo(HaveOccurred(), "failed to get RayService resource")
|
|
})
|
|
|
|
It("Should not create a new pending cluster", func() {
|
|
Expect(rayService.Status.PendingServiceStatus.RayClusterName).To(BeEmpty())
|
|
})
|
|
|
|
It("Should not switch to a new active cluster", func() {
|
|
Expect(rayService.Status.ActiveServiceStatus.RayClusterName).To(Equal(rayCluster.Name))
|
|
})
|
|
})
|
|
|
|
When("Testing appending a new worker group to the active RayCluster", Ordered, func() {
|
|
// When a RayService has only an active RayCluster and no pending RayCluster, appending a new worker group
|
|
// to the RayService spec will update the active RayCluster CR instead of triggering a zero-downtime upgrade.
|
|
BeforeAll(func() {
|
|
// Verify RayService has only one worker group.
|
|
Expect(rayService.Spec.RayClusterSpec.WorkerGroupSpecs).To(HaveLen(1))
|
|
|
|
// Append a new worker group to the active RayCluster.
|
|
workerGroupSpecs := appendNewWorkerGroup(rayService.Spec.RayClusterSpec.WorkerGroupSpecs, "worker-group-to-active-cluster")
|
|
err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
|
|
Eventually(
|
|
getResourceFunc(ctx, client.ObjectKey{Name: rayService.Name, Namespace: namespace}, rayService),
|
|
time.Second*3, time.Millisecond*500).Should(BeNil(), "RayService: %v", rayService.Name)
|
|
rayService.Spec.RayClusterSpec.WorkerGroupSpecs = workerGroupSpecs
|
|
return k8sClient.Update(ctx, rayService)
|
|
})
|
|
Expect(err).NotTo(HaveOccurred(), "failed to update test RayService resource")
|
|
})
|
|
|
|
It("Should reflect the changes in the active cluster's WorkerGroupSpecs", func() {
|
|
Eventually(
|
|
getActiveRayClusterWorkerGroupSpecsFunc(ctx, rayService),
|
|
time.Second*10, time.Millisecond*500).Should(HaveLen(2))
|
|
})
|
|
|
|
It("Should not create a new pending cluster", func() {
|
|
Expect(rayService.Status.PendingServiceStatus.RayClusterName).To(BeEmpty())
|
|
})
|
|
|
|
It("Should not switch to a new active cluster", func() {
|
|
clusterName := rayCluster.Name
|
|
Consistently(
|
|
getRayClusterNameFunc(ctx, rayService),
|
|
time.Second*3, time.Millisecond*500).Should(Equal(clusterName), "Active RayCluster: %v", rayService.Status.ActiveServiceStatus.RayClusterName)
|
|
})
|
|
})
|
|
|
|
When("During the zero-downtime upgrade process", func() {
|
|
var pendingClusterName string
|
|
|
|
BeforeEach(OncePerOrdered, func() {
|
|
mockRayVersion := "mock-ray-version-1"
|
|
err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
|
|
Eventually(
|
|
getResourceFunc(ctx, client.ObjectKey{Name: rayService.Name, Namespace: namespace}, rayService),
|
|
time.Second*3, time.Millisecond*500).Should(BeNil(), "RayService: %v", rayService.Name)
|
|
Expect(rayService.Spec.RayClusterSpec.RayVersion).NotTo(Equal(mockRayVersion))
|
|
rayService.Spec.RayClusterSpec.RayVersion = mockRayVersion
|
|
return k8sClient.Update(ctx, rayService)
|
|
})
|
|
Expect(err).NotTo(HaveOccurred(), "failed to update test RayService resource")
|
|
|
|
Eventually(
|
|
getPreparingRayClusterNameFunc(ctx, rayService),
|
|
time.Second*15, time.Millisecond*500).Should(Not(BeEmpty()), "Pending RayCluster name: %v", rayService.Status.PendingServiceStatus.RayClusterName)
|
|
pendingClusterName = rayService.Status.PendingServiceStatus.RayClusterName
|
|
|
|
// Wait until the pending RayCluster is created. If we update the RayService again before the cluster is created,
|
|
// the RayCluster will be constructed based on the new RayService spec and zero-downtime upgrade will not be triggered.
|
|
By(fmt.Sprintf("Waiting until the pending RayCluster %v is created", pendingClusterName))
|
|
pendingCluster := &rayv1.RayCluster{}
|
|
Eventually(
|
|
getResourceFunc(ctx, client.ObjectKey{Name: pendingClusterName, Namespace: namespace}, pendingCluster),
|
|
time.Second*15, time.Millisecond*500).Should(BeNil(), "Pending RayCluster: %v", pendingCluster.Name)
|
|
})
|
|
|
|
When("Promote the pending RayCluster to the active RayCluster", Ordered, func() {
|
|
It("Update the head pod to Running and Ready", func() {
|
|
updateHeadPodToRunningAndReady(ctx, pendingClusterName, namespace)
|
|
Eventually(
|
|
getRayClusterNameFunc(ctx, rayService),
|
|
time.Second*60, time.Millisecond*500).Should(Equal(pendingClusterName), "Active RayCluster: %v", rayService.Status.ActiveServiceStatus.RayClusterName)
|
|
})
|
|
})
|
|
|
|
When("Updating the RayVersion again to prepare a new pending cluster", Ordered, func() {
|
|
BeforeAll(func() {
|
|
mockRayVersion := "mock-ray-version-2"
|
|
err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
|
|
Eventually(
|
|
getResourceFunc(ctx, client.ObjectKey{Name: rayService.Name, Namespace: namespace}, rayService),
|
|
time.Second*3, time.Millisecond*500).Should(BeNil(), "RayService: %v", rayService.Name)
|
|
Expect(rayService.Spec.RayClusterSpec.RayVersion).NotTo(Equal(mockRayVersion))
|
|
rayService.Spec.RayClusterSpec.RayVersion = mockRayVersion
|
|
return k8sClient.Update(ctx, rayService)
|
|
})
|
|
Expect(err).NotTo(HaveOccurred(), "failed to update RayService resource")
|
|
})
|
|
|
|
It("Should prepare a new pending cluster", func() {
|
|
Eventually(
|
|
getPreparingRayClusterNameFunc(ctx, rayService),
|
|
time.Second*10, time.Millisecond*500).ShouldNot(Equal(pendingClusterName), "Pending RayCluster name = %v", rayService.Status.PendingServiceStatus.RayClusterName)
|
|
})
|
|
})
|
|
|
|
When("Testing appending a new worker group to the pending RayCluster", Ordered, func() {
|
|
BeforeAll(func() {
|
|
// Verify RayService has only one worker group.
|
|
Expect(rayService.Spec.RayClusterSpec.WorkerGroupSpecs).To(HaveLen(1))
|
|
|
|
// Append a new worker group to the pending RayCluster.
|
|
workerGroupSpecs := appendNewWorkerGroup(rayService.Spec.RayClusterSpec.WorkerGroupSpecs, "worker-group-to-active-cluster")
|
|
err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
|
|
Eventually(
|
|
getResourceFunc(ctx, client.ObjectKey{Name: rayService.Name, Namespace: namespace}, rayService),
|
|
time.Second*3, time.Millisecond*500).Should(BeNil(), "RayService: %v", rayService.Name)
|
|
rayService.Spec.RayClusterSpec.WorkerGroupSpecs = workerGroupSpecs
|
|
return k8sClient.Update(ctx, rayService)
|
|
})
|
|
Expect(err).NotTo(HaveOccurred(), "failed to update test RayService resource")
|
|
})
|
|
|
|
It("Should reflect the changes in the pending cluster's WorkerGroupSpecs", func() {
|
|
Eventually(
|
|
getPendingRayClusterWorkerGroupSpecsFunc(ctx, rayService),
|
|
time.Second*15, time.Millisecond*500).Should(HaveLen(2))
|
|
})
|
|
|
|
It("Should not prepare a new pending cluster", func() {
|
|
pendingClusterName := rayService.Status.PendingServiceStatus.RayClusterName
|
|
Consistently(
|
|
getPreparingRayClusterNameFunc(ctx, rayService),
|
|
time.Second*3, time.Millisecond*500).Should(Equal(pendingClusterName), "Pending RayCluster: %v", rayService.Status.PendingServiceStatus.RayClusterName)
|
|
})
|
|
})
|
|
})
|
|
})
|
|
})
|