* fix(runtimes): set file modes on MPI SSH secret volume
Set per-item file modes on the MPI SSH auth Secret volume to prevent
the private key from being world-readable inside training pods.
Private key uses 0o640 (not 0o600) because the builtin
deepspeed-distributed and mlx-distributed runtimes run as
runAsUser: 1000 while Secret volumes mount as root:root.
A volume-level DefaultMode of 0o400 is added as defense-in-depth
for any future items added without an explicit per-item mode.
Fixes#3262
Signed-off-by: Dmytro Hryshchenko <[email protected]>
* docs: update pkg/constants/constants.go
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Dmytro Hryshchenko <[email protected]>
---------
Signed-off-by: Dmytro Hryshchenko <[email protected]>
Signed-off-by: Dmytro Hryshchenko <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
Adds terminationGracePeriodSeconds field to PodSpecPatch so users can
configure pod termination grace period per TrainJob via RuntimePatches.
This is needed for distributed training with PyTorch Elastic (torchrun)
where large models (70B+ parameters) require more than the default 30s
to complete JIT checkpointing before SIGKILL on node drain or TrainJob
pause.
No changes to merge logic in trainingruntime.go are required since the
existing StrategicMergePatch applied at batchv1.JobTemplateSpec level
already handles this field automatically.
Closes#3285
Signed-off-by: krishdef7 <[email protected]>
* feat(api): add ActiveDeadlineSeconds and TTLSecondsAfterFinished to Trainer APIs
- Add ActiveDeadlineSeconds *int64 to TrainJobSpec (immutable, min=1)
with reason constant TrainJobDeadlineExceededReason="DeadlineExceeded"
- Add TTLSecondsAfterFinished *int32 to TrainingRuntimeSpec (min=0)
with cross-field CEL rules blocking conflicting lifecycle fields
in the JobSet/Job template
- Add TrainJobDeadlineExceededMessage to constants
- Regenerate zz_generated.deepcopy.go
Part of KEP-2899
Signed-off-by: XploY04 <[email protected]>
* feat(controller): add TTL-based auto-deletion for finished TrainJobs
- Add early-exit guard in Reconcile before prevTrainJob DeepCopy and
removeFailedCondition, mirroring the Kubernetes Job controller pattern.
Once a TrainJob is terminal (Complete or Failed), only TTL cleanup runs.
- Add fetchTTLFromRuntime helper to retrieve TTLSecondsAfterFinished from
the referenced TrainingRuntime or ClusterTrainingRuntime.
- Add reconcileTTL helper to delete finished TrainJobs after their TTL
expires, or requeue at the exact expiry time.
Part of KEP-2899
Signed-off-by: XploY04 <[email protected]>
* feat(controller): enforce ActiveDeadlineSeconds on TrainJobs
- Add reconcileDeadline helper to terminate running TrainJobs that
exceed their ActiveDeadlineSeconds.
- Handle job suspension: StartTime is determined by the LastTransitionTime
of the Suspended=False condition (if it exists) to ensure the deadline
timer resets correctly upon resume.
- Jobs exceeding the deadline are moved to a Failed state with
reason DeadlineExceeded.
Part of KEP-2899
Signed-off-by: XploY04 <[email protected]>
* feat(webhooks): warn on short ttlSecondsAfterFinished
- Add an admission warning to both TrainingRuntime and
ClusterTrainingRuntime webhooks if the TTLSecondsAfterFinished
is configured to less than 60 seconds, as this may cause
unexpected data/log loss for data scientists due to overly
aggressive cleanup.
Part of KEP-2899
Signed-off-by: XploY04 <[email protected]>
* test: add integration tests for TrainJob TTL and deadline lifecycle
- Add integration tests for DeadlineExceeded behavior in TrainJob controller
- Add integration tests for TTL validation and warnings in webhooks
Part of KEP-2899
Signed-off-by: XploY04 <[email protected]>
* cleanup
Signed-off-by: XploY04 <[email protected]>
* fix: ensure JobSet is suspended on expiration and fix webhook test regress
Signed-off-by: XploY04 <[email protected]>
* fix(controller): return errors from fetchTTLFromRuntime for proper requeue
Signed-off-by: XploY04 <[email protected]>
* test: remove unused wantWarning from webhook tests
Signed-off-by: XploY04 <[email protected]>
* fix
Signed-off-by: XploY04 <[email protected]>
* feat: Remove from TrainingRuntime and ClusterTrainingRuntime specifications and associated validation.
Signed-off-by: XploY04 <[email protected]>
* fix(api): restore `activeDeadlineSeconds` validation rule on TrainingRuntime
Signed-off-by: XploY04 <[email protected]>
* refactor: remove unnecessary blank lines and improve logging in ClusterTrainingRuntime validation
Signed-off-by: XploY04 <[email protected]>
* feat: add activeDeadlineSeconds field to TrainJobSpec and update related tests
Signed-off-by: XploY04 <[email protected]>
* Remove unnecessary newline in test file
Signed-off-by: Yash Agarwal <[email protected]>
* Remove unnecessary newline in trainingruntime_webhook_test.go
Signed-off-by: Yash Agarwal <[email protected]>
* refactor: remove unnecessary blank lines in TrainJob reconciliation and deadline tests
Signed-off-by: XploY04 <[email protected]>
* feat: change ActiveDeadlineSeconds from pointer to value type and update related code
Signed-off-by: XploY04 <[email protected]>
* feat: add condition check for TrainJob deadline exceeded before suspending
Signed-off-by: XploY04 <[email protected]>
* feat: enhance TrainJob deadline handling and suspend logic
Signed-off-by: XploY04 <[email protected]>
* feat: enhance TrainJob deadline handling and suspend logic
Signed-off-by: XploY04 <[email protected]>
* fix
Signed-off-by: XploY04 <[email protected]>
* feat: change ActiveDeadlineSeconds to pointer type and update related logic
Signed-off-by: XploY04 <[email protected]>
* feat: ensure requeueAfter is at least 1 second when deadline is exceeded
Signed-off-by: XploY04 <[email protected]>
* feat: change ActiveDeadlineSeconds from pointer to value type and update related logic
Signed-off-by: XploY04 <[email protected]>
* fix: remove early return for finished TrainJobs to prevent status sync issues
The early return was preventing the controller from syncing JobSet
status updates for finished TrainJobs, causing e2e tests to timeout
waiting for conditions that never got updated.
This fix restores the upstream behavior where finished TrainJobs still
go through status reconciliation (though reconcileObjects is skipped).
Signed-off-by: XploY04 <[email protected]>
* fix
Signed-off-by: XploY04 <[email protected]>
* fix: remove x-kubernetes-validations for activeDeadlineSeconds in TrainingRuntime CRDs
Signed-off-by: XploY04 <[email protected]>
* fix: update validation message for ActiveDeadlineSeconds to be more generic
Signed-off-by: XploY04 <[email protected]>
* replacing suspend with delete
Signed-off-by: XploY04 <[email protected]>
* fix: remove SpecLabel and SpecAnnotation methods from TrainJobWrapper
Signed-off-by: XploY04 <[email protected]>
* fix: remove annotations field from TrainJobSpec and CRD definitions
Signed-off-by: XploY04 <[email protected]>
* ensure JobSet deletion in e2e tests
Signed-off-by: XploY04 <[email protected]>
* fix: add JobSet APIs to the scheme in e2e tests
Signed-off-by: XploY04 <[email protected]>
* fix: simplify error handling in Reconcile method of TrainJobReconciler
Signed-off-by: XploY04 <[email protected]>
---------
Signed-off-by: XploY04 <[email protected]>
Signed-off-by: Yash Agarwal <[email protected]>
* Add XGBoost runtime plugin scaffold and register in framework
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
* regenerated assets
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
* Add XGBoost plugin unit tests and update framework test registry
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
* added :Runtime image , unit tests for :(Resources are not set, Resources are set in TrainJob) ,using the ContainerTrainerPort instead of default port
Signed-off-by: krishna-kg732 <[email protected]>
* minor changes
Signed-off-by: krishna-kg732 <[email protected]>
* added plainML fallback(JAX+XGBOOST)
Signed-off-by: krishna-kg732 <[email protected]>
* test: add XGBoost runtime resource resolution unit tests
- Add test case for resources set in Runtime only
- Add test case for resources set in both Runtime and TrainJob
- Add xgboost_distributed.yaml ClusterTrainingRuntime manifest
- Update kustomization.yaml with xgboost-runtime image
Addresses feedback from PR #3200 for complete resource resolution test coverage.
Signed-off-by: krishna-kg732 <[email protected]>
* added autogenerat3ed files
Signed-off-by: krishna-kg732 <[email protected]>
* CI fixes
Signed-off-by: krishna-kg732 <[email protected]>
* chore: fix ci lint errors
Signed-off-by: krishna-kg732 <[email protected]>
* Add XGBoost integration and E2E tests, fix Validate guard clause and nil safety
Signed-off-by: krishna-kg732 <[email protected]>
* trigger ci
Signed-off-by: krishna-kg732 <[email protected]>
* fix: Add XGBoost support to test wrapper, CEL validation, and CI publish workflow
- Add XGBoostPolicy() method to MLPolicySourceWrapper for integration tests
- Add has(self.xgboost) to MLPolicy CEL mutual-exclusivity validation rule
- Regenerate CRD manifests with updated CEL rule
- Add xgboost-runtime to build-and-push-images CI workflow
- Add XGBoost image build step to E2E setup script (commented out due to Kind bug)
Signed-off-by: krishna-kg732 <[email protected]>
* docs: Add XGBoost distributed training example notebook
Add example notebook demonstrating distributed XGBoost training using
the Kubeflow Trainer SDK and the xgboost-distributed ClusterTrainingRuntime.
The notebook uses the modern XGBoost Collective API with RabitTracker
for distributed coordination and synthetic classification data.
Signed-off-by: krishna-kg732 <[email protected]>
* test: Skip XGBoost E2E test until runtime image is published
Signed-off-by: krishna-kg732 <[email protected]>
* chore : address review comments
Signed-off-by: krishna-kg732 <[email protected]>
* chore:resolve merge conflicts
Signed-off-by: krishna-kg732 <[email protected]>
---------
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
