Files
a284b829f8 [Fix][RayJob SidecarMode] Prevent premature job termination during transient head node spikes (#4399)
* [Fix][1/N] defer sidecar submitter failure until dashboard check

Signed-off-by: justinyeh1995 <[email protected]>

* [Feature][SidecarMode] Add SidecarSubmitterRestart feature gate for per-container restart

Signed-off-by: justinyeh1995 <[email protected]>

* [Feature][SidecarMode] Add SidecarSubmitterRestart feature gate unit test

Signed-off-by: justinyeh1995 <[email protected]>

* [Feature][SidecarMode] Add K8s version check if SidecarSubmitterRestart feature gate is enabled at operator startup

Signed-off-by: justinyeh1995 <[email protected]>

* [Chore] better explainations

Signed-off-by: justinyeh1995 <[email protected]>

* [Chore] remove incorrect nil check

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] SidecarMode should not fail job based on container exit status

Signed-off-by: justinyeh1995 <[email protected]>

* Trigger CI

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix][SidecarMode] add e2e tests

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix][SidecarMode] clarify RestartPolicyRules exit code rationale

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix][Sidecar Mode] Use ParseGeneric + GitVersion, and add patch version for more robust k8s version comparison

Signed-off-by: justinyeh1995 <[email protected]>

* Apply suggestions

Co-authored-by: Jia-Wei Jiang <[email protected]>
Signed-off-by: JustinYeh <[email protected]>

* [Fix] use ContainerRestartPolicyOnFailure to express the intention better; update the comment to explain the intention better

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] bump version check to v1.35 and update sidecar mode test to check onFailure restart policy

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] check the existence of the feature gate instead of relying on the derivate result

Signed-off-by: justinyeh1995 <[email protected]>

* improve the why for using K8s v1.35

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] Revert checkSidecarContainerStatus removal; scope PR to K8s v1.35+ use case

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] use correct health command constant and argument order in the feature gate test.

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] remove leftover changes

Signed-off-by: justinyeh1995 <[email protected]>

* [Fix] prevent panicing due to failing to fetch head pod

Signed-off-by: justinyeh1995 <[email protected]>

* add warning about the risk of having kubelet version skew

Signed-off-by: justinyeh1995 <[email protected]>

* document ray v2.54.0+ is needed

Co-authored-by: Jun-Hao Wan <[email protected]>
Signed-off-by: JustinYeh <[email protected]>

* add max restart count annotaion, envVar, and default max restart count

Signed-off-by: justinyeh1995 <[email protected]>

* do not fail the job if the submitter container exited successfully

Signed-off-by: justinyeh1995 <[email protected]>

* fix BasePythonHealthCommand argument

Signed-off-by: justinyeh1995 <[email protected]>

* trigger ci

Signed-off-by: justinyeh1995 <[email protected]>

* address suggestion part 1

Signed-off-by: justinyeh1995 <[email protected]>

* k8s 1.35 version

Signed-off-by: Future-Outlier <[email protected]>

* add override

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* upd for helm and kustomization

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* fix typo

Signed-off-by: justinyeh1995 <[email protected]>

* address comment, assert no resubmission

Signed-off-by: justinyeh1995 <[email protected]>

* bump kind version to the latest version which supports v1.35.0 by default

Signed-off-by: justinyeh1995 <[email protected]>

* incremental upgrade use kind-config-buildkite.yml (v1.35)

Signed-off-by: justinyeh1995 <[email protected]>

* replace hardcoded value to use rayJob.Spec.SubmitterConfig.BackoffLimit

Signed-off-by: justinyeh1995 <[email protected]>

---------

Signed-off-by: justinyeh1995 <[email protected]>
Signed-off-by: JustinYeh <[email protected]>
Signed-off-by: Future-Outlier <[email protected]>
Co-authored-by: Jia-Wei Jiang <[email protected]>
Co-authored-by: Jun-Hao Wan <[email protected]>
Co-authored-by: Future-Outlier <[email protected]>
2026-06-01 21:20:44 -07:00
..