mirror of
https://github.com/hanzo-ml/kuberay.git
synced 2026-10-30 19:48:47 +00:00
* [RayService] Support Spec.Suspend for incremental upgrade Re-introduce Gateway and HTTPRoute deletion in deleteRayServiceOwnedResources, gated on the RayServiceIncrementalUpgrade feature gate. Per-cluster serve Services are owned by their RayCluster, so they are garbage-collected as part of the existing RayCluster deletion above. Tolerate meta.IsNoMatchError so a manager that started without gateway-api CRDs installed (but later had the gate flipped on) doesn't fail the suspend transition. Update the Suspending condition messages and the Spec.Suspend godoc to mention Gateway and HTTPRoute. Restore the FailedToDeleteGateway / FailedToDeleteHTTPRoute event constants used by the new deletion paths. docs/reference/api.md and the generated applyconfiguration are refreshed via make sync. Add TestRayServiceSuspendDuringIncrementalUpgrade in test/e2eincrementalupgrade: bootstrap an incremental RayService (BlueGreen), trigger an upgrade so both active and pending clusters exist, set Spec.Suspend=true and assert that Gateway / HTTPRoute / RayClusters / Services are all torn down; then resume and verify the Gateway is recreated and serves traffic again from a fresh external IP. Lives in e2eincrementalupgrade because that suite already enables the feature gate and brings up MetalLB + gateway-api CRDs + Istio in buildkite, so the existing 'Test RayService Incremental Upgrade E2E' job picks it up without YAML changes. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Grant delete on gateways/httproutes; fix suspend e2e assertion Two follow-ups surfaced by running the new suspend-during-incremental-upgrade e2e test against a real cluster. 1. RBAC: the kubebuilder marker on the RayService reconciler granted only get;list;watch;create;update on gateways and httproutes, so the new Spec.Suspend tear-down failed with "forbidden: cannot delete resource 'gateways' / 'httproutes'" and looped on FailedToDeleteGateway / FailedToDeleteHTTPRoute warning events. Add delete to both markers and regenerate config/rbac/role.yaml. Without this the previous commit would not actually clean up Gateway/HTTPRoute in real deployments. 2. Test assertion: incrementalUpgrade() bumps MangoStand price 3->4 via the API before Spec.Suspend is flipped, so when the resumed RayService serves traffic it uses the upgraded spec — (MANGO, 2) -> 8, not 6. Update the post-resume curl assertion accordingly. This also doubles as a guard that resume applied the upgraded spec rather than reviving the pre-upgrade state. Verified end-to-end against a local kind cluster (MetalLB + gateway-api v1.4.1 + Istio 1.28.3 + RayServiceIncrementalUpgrade feature gate on): TestRayServiceSuspendDuringIncrementalUpgrade passes in ~53s. Gateway, HTTPRoute, RayClusters, and Services are all deleted on suspend; on resume the Gateway is recreated (new external IP from MetalLB), HTTPRoute reaches Ready, and curl through the new IP returns 8 from the upgraded MangoStand. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Trim Spec.Suspend godoc Drop the parenthetical resource enumeration on Spec.Suspend. "All Kubernetes resources owned by the RayService controller" already conveys the contract; listing RayClusters / Services / Gateway / HTTPRoute and gating part of the list on a feature gate name made the field comment harder to read. docs/reference/api.md and the applyconfiguration are regenerated via make sync. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [Helm] Sync gateways/httproutes RBAC delete verb into chart The previous commit added the delete verb to the kubebuilder marker on the RayService reconciler so config/rbac/role.yaml grants delete on gateways and httproutes, but the helm chart keeps a hand-maintained copy of the ClusterRole rules in templates/_helpers.tpl that wasn't updated. Without this, a helm-installed operator would still 403 on Spec.Suspend tear-down. Add delete to the gateway.networking.k8s.io rule so the chart matches the generated role.yaml exactly. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Emit success events on Service/Gateway/HTTPRoute delete during suspend deleteRayServiceOwnedResources was asymmetric: RayCluster delete fired both DeletedRayCluster and FailedToDeleteRayCluster, but Service / Gateway / HTTPRoute delete only fired the Failed-side event. Add the missing success events so operators reading kubectl describe / event streams see the same shape across all four resource kinds. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Group Deleted{Gateway,HTTPRoute,Service} constants together Cosmetic: move the three success-side Deleted* constants added in the prior commit next to each other at the end of the RayService event list instead of interleaving each one after its FailedTo counterpart. Easier to scan. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Trim Suspending condition messages Match the trimmed Spec.Suspend godoc: drop the resource enumeration from the two Suspending condition messages and just say "all Kubernetes resources owned by this RayService" instead of "RayClusters, Services, Gateway, and HTTPRoute owned by this RayService". The set of owned resources will grow as the controller does and this phrasing won't drift. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Drop comment above suspend Gateway/HTTPRoute branch The feature-gate guard and the meta.IsNoMatchError tolerance speak for themselves at the call site; the explanatory block above the branch wasn't adding information a reader couldn't get from the code. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Send suspend e2e curls through cluster DNS, not LB IP Switch the two curl targets in TestRayServiceSuspendDuringIncrementalUpgrade from the MetalLB-assigned external IP to the backing Istio Service's cluster DNS handle <gateway-name>-istio.<ns>.svc.cluster.local. Two improvements: - The DNS handle is stable across Gateway delete + recreate (same Service name in the same namespace), so the post-resume re-resolve of the Gateway IP can go away — the test no longer cares whether MetalLB hands back the same address or a fresh one. - The Eventually loop that verifies the upgraded spec after resume can hit envoy's xDS warmup window where Programmed=True is set but HTTPRoute config hasn't propagated yet. CurlRayServiceGateway wraps ExecPodCmd which require.NoError's on non-zero curl exits and would break out of the retry loop. Switch to ExecPodCmdWithError + short --connect-timeout / --max-time so each attempt fails fast and Eventually can iterate. Bootstrap is unchanged — it still reads Gateway.Status.Addresses for the non-empty assertion (so the test still needs MetalLB at setup time), but the test's correctness path no longer depends on that external IP. Removing the bootstrap helper's external-IP dependency is a separate refactor: it would also unhook MetalLB from the buildkite job, but it touches every other incremental upgrade test, which is out of scope here. Verified locally on kind+MetalLB+Istio: ~57s, post-resume retry rides through the xDS warmup as expected. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Trim suspend e2e comments Drop the test-function doc block and the inline rationale above the post-resume Eventually loop. The function name says what the test does, and the rationale for ExecPodCmdWithError + short timeouts belongs in git history (4c88b5b1) rather than at the call site. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Slow suspend e2e upgrade + assert active/pending coexistence Addresses PR feedback on the suspend incremental upgrade e2e: - Switch upgrade params from BlueGreen (100/1/100) to 25/20/30 so there is a deterministic window in which active and pending RayClusters coexist rather than racing the suspend transition against an instantaneous cutover. - Strengthen the pre-suspend wait: instead of only checking that the pending cluster name is non-empty, assert that the API lists exactly two RayClusters owned by this RayService (and that both ActiveServiceStatus and PendingServiceStatus carry a name). This directly verifies the "both clusters exist at the same time" precondition the test is meant to exercise. Verified locally on kind+MetalLB+Istio: 54.97s, coexistence assertion satisfied within ~2s of the upgrade trigger. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Drop meta.IsNoMatchError guard from suspend Gateway/HTTPRoute Get The guard only matters in a misconfigured state: RayServiceIncrementalUpgrade gate is on but the gateway-api CRDs are not installed in the cluster. In that state the operator's normal reconcile (reconcileGateway / reconcileHTTPRoute) fails the same way at Create time, so the Gateway/HTTPRoute never exist and the suspend tear-down has nothing to delete anyway. Defending against it here adds code complexity for a configuration that is not operational end-to-end. Trust the feature gate and the gateway-api CRDs being installed together as documented. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * [RayService] Gate suspend Gateway/HTTPRoute deletion on per-spec incremental upgrade Switch the outer gate in deleteRayServiceOwnedResources from features.Enabled(features.RayServiceIncrementalUpgrade) (global) to utils.IsIncrementalUpgradeEnabled(&rayServiceInstance.Spec) (global AND spec.UpgradeStrategy.Type == NewClusterWithIncrementalUpgrade). This fixes the regression where the e2erayservice suspend buildkite job was failing: the test overlay enables the feature gate globally, but the job does not install gateway-api CRDs (those are only installed in the incremental upgrade job). With the old global gate, r.Get for the Gateway returned meta.NoMatchError because the GVK had no REST mapping, which propagated as a hard error and stalled the suspend transition. The previous fix relied on meta.IsNoMatchError; this commit makes that check unnecessary by only entering the branch when the RayService actually uses incremental upgrade — which implies CRDs are installed (otherwise the controller would have already failed to create the Gateway during normal reconcile). This also mirrors the gating used by the create path in Reconcile, which calls reconcileGateway / reconcileHTTPRoute under the same per-spec condition. The two paths are now symmetric. Verified locally on kind+gateway-api+Istio: - TestRayServiceSuspendDuringIncrementalUpgrade (incremental spec) ~57s, PASS - TestRayServiceSuspend{Resume,Atomic,CreatedSuspended,SuspendDuringUpgrade} (non-incremental spec) ~189s, PASS Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>