Files
51c9f734a1 [RayService][Kueue] Support top-level Spec.Suspend for incremental upgrade (#4881)
* [RayService] Support Spec.Suspend for incremental upgrade

Re-introduce Gateway and HTTPRoute deletion in deleteRayServiceOwnedResources,
gated on the RayServiceIncrementalUpgrade feature gate. Per-cluster serve
Services are owned by their RayCluster, so they are garbage-collected as part
of the existing RayCluster deletion above. Tolerate meta.IsNoMatchError so a
manager that started without gateway-api CRDs installed (but later had the
gate flipped on) doesn't fail the suspend transition.

Update the Suspending condition messages and the Spec.Suspend godoc to mention
Gateway and HTTPRoute. Restore the FailedToDeleteGateway / FailedToDeleteHTTPRoute
event constants used by the new deletion paths. docs/reference/api.md and the
generated applyconfiguration are refreshed via make sync.

Add TestRayServiceSuspendDuringIncrementalUpgrade in test/e2eincrementalupgrade:
bootstrap an incremental RayService (BlueGreen), trigger an upgrade so both
active and pending clusters exist, set Spec.Suspend=true and assert that
Gateway / HTTPRoute / RayClusters / Services are all torn down; then resume
and verify the Gateway is recreated and serves traffic again from a fresh
external IP. Lives in e2eincrementalupgrade because that suite already enables
the feature gate and brings up MetalLB + gateway-api CRDs + Istio in
buildkite, so the existing 'Test RayService Incremental Upgrade E2E' job
picks it up without YAML changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Grant delete on gateways/httproutes; fix suspend e2e assertion

Two follow-ups surfaced by running the new suspend-during-incremental-upgrade
e2e test against a real cluster.

1. RBAC: the kubebuilder marker on the RayService reconciler granted only
   get;list;watch;create;update on gateways and httproutes, so the new
   Spec.Suspend tear-down failed with "forbidden: cannot delete resource
   'gateways' / 'httproutes'" and looped on FailedToDeleteGateway /
   FailedToDeleteHTTPRoute warning events. Add delete to both markers and
   regenerate config/rbac/role.yaml. Without this the previous commit
   would not actually clean up Gateway/HTTPRoute in real deployments.

2. Test assertion: incrementalUpgrade() bumps MangoStand price 3->4 via
   the API before Spec.Suspend is flipped, so when the resumed RayService
   serves traffic it uses the upgraded spec — (MANGO, 2) -> 8, not 6.
   Update the post-resume curl assertion accordingly. This also doubles
   as a guard that resume applied the upgraded spec rather than reviving
   the pre-upgrade state.

Verified end-to-end against a local kind cluster
(MetalLB + gateway-api v1.4.1 + Istio 1.28.3 + RayServiceIncrementalUpgrade
feature gate on): TestRayServiceSuspendDuringIncrementalUpgrade passes in
~53s. Gateway, HTTPRoute, RayClusters, and Services are all deleted on
suspend; on resume the Gateway is recreated (new external IP from MetalLB),
HTTPRoute reaches Ready, and curl through the new IP returns 8 from the
upgraded MangoStand.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Trim Spec.Suspend godoc

Drop the parenthetical resource enumeration on Spec.Suspend. "All Kubernetes
resources owned by the RayService controller" already conveys the contract;
listing RayClusters / Services / Gateway / HTTPRoute and gating part of the
list on a feature gate name made the field comment harder to read.

docs/reference/api.md and the applyconfiguration are regenerated via make sync.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [Helm] Sync gateways/httproutes RBAC delete verb into chart

The previous commit added the delete verb to the kubebuilder marker on the
RayService reconciler so config/rbac/role.yaml grants delete on gateways
and httproutes, but the helm chart keeps a hand-maintained copy of the
ClusterRole rules in templates/_helpers.tpl that wasn't updated. Without
this, a helm-installed operator would still 403 on Spec.Suspend tear-down.
Add delete to the gateway.networking.k8s.io rule so the chart matches the
generated role.yaml exactly.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Emit success events on Service/Gateway/HTTPRoute delete during suspend

deleteRayServiceOwnedResources was asymmetric: RayCluster delete fired both
DeletedRayCluster and FailedToDeleteRayCluster, but Service / Gateway /
HTTPRoute delete only fired the Failed-side event. Add the missing success
events so operators reading kubectl describe / event streams see the same
shape across all four resource kinds.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Group Deleted{Gateway,HTTPRoute,Service} constants together

Cosmetic: move the three success-side Deleted* constants added in the prior
commit next to each other at the end of the RayService event list instead of
interleaving each one after its FailedTo counterpart. Easier to scan.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Trim Suspending condition messages

Match the trimmed Spec.Suspend godoc: drop the resource enumeration from the
two Suspending condition messages and just say "all Kubernetes resources
owned by this RayService" instead of "RayClusters, Services, Gateway, and
HTTPRoute owned by this RayService". The set of owned resources will grow
as the controller does and this phrasing won't drift.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Drop comment above suspend Gateway/HTTPRoute branch

The feature-gate guard and the meta.IsNoMatchError tolerance speak for
themselves at the call site; the explanatory block above the branch wasn't
adding information a reader couldn't get from the code.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Send suspend e2e curls through cluster DNS, not LB IP

Switch the two curl targets in TestRayServiceSuspendDuringIncrementalUpgrade
from the MetalLB-assigned external IP to the backing Istio Service's cluster
DNS handle <gateway-name>-istio.<ns>.svc.cluster.local. Two improvements:

- The DNS handle is stable across Gateway delete + recreate (same Service
  name in the same namespace), so the post-resume re-resolve of the Gateway
  IP can go away — the test no longer cares whether MetalLB hands back the
  same address or a fresh one.

- The Eventually loop that verifies the upgraded spec after resume can hit
  envoy's xDS warmup window where Programmed=True is set but HTTPRoute
  config hasn't propagated yet. CurlRayServiceGateway wraps ExecPodCmd
  which require.NoError's on non-zero curl exits and would break out of
  the retry loop. Switch to ExecPodCmdWithError + short --connect-timeout
  / --max-time so each attempt fails fast and Eventually can iterate.

Bootstrap is unchanged — it still reads Gateway.Status.Addresses for the
non-empty assertion (so the test still needs MetalLB at setup time), but
the test's correctness path no longer depends on that external IP.
Removing the bootstrap helper's external-IP dependency is a separate
refactor: it would also unhook MetalLB from the buildkite job, but it
touches every other incremental upgrade test, which is out of scope here.

Verified locally on kind+MetalLB+Istio: ~57s, post-resume retry rides
through the xDS warmup as expected.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Trim suspend e2e comments

Drop the test-function doc block and the inline rationale above the
post-resume Eventually loop. The function name says what the test does,
and the rationale for ExecPodCmdWithError + short timeouts belongs in
git history (4c88b5b1) rather than at the call site.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Slow suspend e2e upgrade + assert active/pending coexistence

Addresses PR feedback on the suspend incremental upgrade e2e:

- Switch upgrade params from BlueGreen (100/1/100) to 25/20/30 so there is
  a deterministic window in which active and pending RayClusters coexist
  rather than racing the suspend transition against an instantaneous
  cutover.

- Strengthen the pre-suspend wait: instead of only checking that the
  pending cluster name is non-empty, assert that the API lists exactly
  two RayClusters owned by this RayService (and that both ActiveServiceStatus
  and PendingServiceStatus carry a name). This directly verifies the
  "both clusters exist at the same time" precondition the test is meant
  to exercise.

Verified locally on kind+MetalLB+Istio: 54.97s, coexistence assertion
satisfied within ~2s of the upgrade trigger.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Drop meta.IsNoMatchError guard from suspend Gateway/HTTPRoute Get

The guard only matters in a misconfigured state: RayServiceIncrementalUpgrade
gate is on but the gateway-api CRDs are not installed in the cluster. In
that state the operator's normal reconcile (reconcileGateway / reconcileHTTPRoute)
fails the same way at Create time, so the Gateway/HTTPRoute never exist and
the suspend tear-down has nothing to delete anyway. Defending against it
here adds code complexity for a configuration that is not operational
end-to-end. Trust the feature gate and the gateway-api CRDs being installed
together as documented.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* [RayService] Gate suspend Gateway/HTTPRoute deletion on per-spec incremental upgrade

Switch the outer gate in deleteRayServiceOwnedResources from
features.Enabled(features.RayServiceIncrementalUpgrade) (global) to
utils.IsIncrementalUpgradeEnabled(&rayServiceInstance.Spec) (global AND
spec.UpgradeStrategy.Type == NewClusterWithIncrementalUpgrade).

This fixes the regression where the e2erayservice suspend buildkite job
was failing: the test overlay enables the feature gate globally, but the
job does not install gateway-api CRDs (those are only installed in the
incremental upgrade job). With the old global gate, r.Get for the Gateway
returned meta.NoMatchError because the GVK had no REST mapping, which
propagated as a hard error and stalled the suspend transition. The
previous fix relied on meta.IsNoMatchError; this commit makes that check
unnecessary by only entering the branch when the RayService actually uses
incremental upgrade — which implies CRDs are installed (otherwise the
controller would have already failed to create the Gateway during normal
reconcile).

This also mirrors the gating used by the create path in Reconcile, which
calls reconcileGateway / reconcileHTTPRoute under the same per-spec
condition. The two paths are now symmetric.

Verified locally on kind+gateway-api+Istio:
- TestRayServiceSuspendDuringIncrementalUpgrade (incremental spec) ~57s, PASS
- TestRayServiceSuspend{Resume,Atomic,CreatedSuspended,SuspendDuringUpgrade}
  (non-incremental spec) ~189s, PASS

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
2026-05-29 13:43:53 -07:00
..