mirror of
https://github.com/hanzo-ml/trainer.git
synced 2026-09-24 00:31:34 +00:00
Yash AgarwalandGitHub
587010f54d
* feat(api): add ActiveDeadlineSeconds and TTLSecondsAfterFinished to Trainer APIs - Add ActiveDeadlineSeconds *int64 to TrainJobSpec (immutable, min=1) with reason constant TrainJobDeadlineExceededReason="DeadlineExceeded" - Add TTLSecondsAfterFinished *int32 to TrainingRuntimeSpec (min=0) with cross-field CEL rules blocking conflicting lifecycle fields in the JobSet/Job template - Add TrainJobDeadlineExceededMessage to constants - Regenerate zz_generated.deepcopy.go Part of KEP-2899 Signed-off-by: XploY04 <[email protected]> * feat(controller): add TTL-based auto-deletion for finished TrainJobs - Add early-exit guard in Reconcile before prevTrainJob DeepCopy and removeFailedCondition, mirroring the Kubernetes Job controller pattern. Once a TrainJob is terminal (Complete or Failed), only TTL cleanup runs. - Add fetchTTLFromRuntime helper to retrieve TTLSecondsAfterFinished from the referenced TrainingRuntime or ClusterTrainingRuntime. - Add reconcileTTL helper to delete finished TrainJobs after their TTL expires, or requeue at the exact expiry time. Part of KEP-2899 Signed-off-by: XploY04 <[email protected]> * feat(controller): enforce ActiveDeadlineSeconds on TrainJobs - Add reconcileDeadline helper to terminate running TrainJobs that exceed their ActiveDeadlineSeconds. - Handle job suspension: StartTime is determined by the LastTransitionTime of the Suspended=False condition (if it exists) to ensure the deadline timer resets correctly upon resume. - Jobs exceeding the deadline are moved to a Failed state with reason DeadlineExceeded. Part of KEP-2899 Signed-off-by: XploY04 <[email protected]> * feat(webhooks): warn on short ttlSecondsAfterFinished - Add an admission warning to both TrainingRuntime and ClusterTrainingRuntime webhooks if the TTLSecondsAfterFinished is configured to less than 60 seconds, as this may cause unexpected data/log loss for data scientists due to overly aggressive cleanup. Part of KEP-2899 Signed-off-by: XploY04 <[email protected]> * test: add integration tests for TrainJob TTL and deadline lifecycle - Add integration tests for DeadlineExceeded behavior in TrainJob controller - Add integration tests for TTL validation and warnings in webhooks Part of KEP-2899 Signed-off-by: XploY04 <[email protected]> * cleanup Signed-off-by: XploY04 <[email protected]> * fix: ensure JobSet is suspended on expiration and fix webhook test regress Signed-off-by: XploY04 <[email protected]> * fix(controller): return errors from fetchTTLFromRuntime for proper requeue Signed-off-by: XploY04 <[email protected]> * test: remove unused wantWarning from webhook tests Signed-off-by: XploY04 <[email protected]> * fix Signed-off-by: XploY04 <[email protected]> * feat: Remove from TrainingRuntime and ClusterTrainingRuntime specifications and associated validation. Signed-off-by: XploY04 <[email protected]> * fix(api): restore `activeDeadlineSeconds` validation rule on TrainingRuntime Signed-off-by: XploY04 <[email protected]> * refactor: remove unnecessary blank lines and improve logging in ClusterTrainingRuntime validation Signed-off-by: XploY04 <[email protected]> * feat: add activeDeadlineSeconds field to TrainJobSpec and update related tests Signed-off-by: XploY04 <[email protected]> * Remove unnecessary newline in test file Signed-off-by: Yash Agarwal <[email protected]> * Remove unnecessary newline in trainingruntime_webhook_test.go Signed-off-by: Yash Agarwal <[email protected]> * refactor: remove unnecessary blank lines in TrainJob reconciliation and deadline tests Signed-off-by: XploY04 <[email protected]> * feat: change ActiveDeadlineSeconds from pointer to value type and update related code Signed-off-by: XploY04 <[email protected]> * feat: add condition check for TrainJob deadline exceeded before suspending Signed-off-by: XploY04 <[email protected]> * feat: enhance TrainJob deadline handling and suspend logic Signed-off-by: XploY04 <[email protected]> * feat: enhance TrainJob deadline handling and suspend logic Signed-off-by: XploY04 <[email protected]> * fix Signed-off-by: XploY04 <[email protected]> * feat: change ActiveDeadlineSeconds to pointer type and update related logic Signed-off-by: XploY04 <[email protected]> * feat: ensure requeueAfter is at least 1 second when deadline is exceeded Signed-off-by: XploY04 <[email protected]> * feat: change ActiveDeadlineSeconds from pointer to value type and update related logic Signed-off-by: XploY04 <[email protected]> * fix: remove early return for finished TrainJobs to prevent status sync issues The early return was preventing the controller from syncing JobSet status updates for finished TrainJobs, causing e2e tests to timeout waiting for conditions that never got updated. This fix restores the upstream behavior where finished TrainJobs still go through status reconciliation (though reconcileObjects is skipped). Signed-off-by: XploY04 <[email protected]> * fix Signed-off-by: XploY04 <[email protected]> * fix: remove x-kubernetes-validations for activeDeadlineSeconds in TrainingRuntime CRDs Signed-off-by: XploY04 <[email protected]> * fix: update validation message for ActiveDeadlineSeconds to be more generic Signed-off-by: XploY04 <[email protected]> * replacing suspend with delete Signed-off-by: XploY04 <[email protected]> * fix: remove SpecLabel and SpecAnnotation methods from TrainJobWrapper Signed-off-by: XploY04 <[email protected]> * fix: remove annotations field from TrainJobSpec and CRD definitions Signed-off-by: XploY04 <[email protected]> * ensure JobSet deletion in e2e tests Signed-off-by: XploY04 <[email protected]> * fix: add JobSet APIs to the scheme in e2e tests Signed-off-by: XploY04 <[email protected]> * fix: simplify error handling in Reconcile method of TrainJobReconciler Signed-off-by: XploY04 <[email protected]> --------- Signed-off-by: XploY04 <[email protected]> Signed-off-by: Yash Agarwal <[email protected]>
Kubeflow Trainer API
Python package containing Kubeflow Trainer API models.