* test(torch): add unit tests for getModelFromRuntimeRef
Cover the runtime-ref name parsing used to resolve the TorchTune
pretrained model: normalization of valid names and the empty-string
fallback for malformed refs (wrong number of parts, empty input).
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): add unit tests for isLoraConfigEnabled
Cover LoRA detection from trainer args: the model.lora_attn_modules
marker present, absent, and the empty-args case.
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): add unit tests for isUseQLoraFinetune
Cover QLoRA detection from trainer args, including the DoRA
short-circuit that disables QLoRA even when quantize_base is set.
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): add unit tests for getRecipeAndConfig
Cover the recipe/config selection matrix: single-device, single-node
multi-GPU, and multi-node, each across full / LoRA / QLoRA fine-tuning.
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): add unit tests for extractOverridesFromRuntime
Cover extraction of immutable runtime config overrides from the JobSet
template: trainer-ancestor node container returns only immutable configs,
while non-trainer ancestors, non-node containers, and an absent template
yield no overrides.
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): use year-less copyright header
Align the new test file with the repo boilerplate (hack/boilerplate/boilerplate.go.txt) and CNCF/LF guidance, which drop the year from the copyright notice.
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): cover integer numProcPerNode in getRecipeAndConfig
Exercise integer numProcPerNode counts alongside the existing "auto" cases: numProcPerNode=1 forces the single-device recipe even with multiple GPUs, while a count >1 takes the distributed path.
Signed-off-by: yassinnouh21 <[email protected]>
* fix(torch): respect boolean value when detecting QLoRA args
isUseQLoraFinetune matched quantize_base/use_dora by key only, so model.quantize_base=False enabled QLoRA and model.use_dora=False short-circuited it off. Parse the key=value pair and only treat a truthy value as enabled. Add =False regression cases.
Signed-off-by: yassinnouh21 <[email protected]>
---------
Signed-off-by: yassinnouh21 <[email protected]>
* fix(runtimes): set file modes on MPI SSH secret volume
Set per-item file modes on the MPI SSH auth Secret volume to prevent
the private key from being world-readable inside training pods.
Private key uses 0o640 (not 0o600) because the builtin
deepspeed-distributed and mlx-distributed runtimes run as
runAsUser: 1000 while Secret volumes mount as root:root.
A volume-level DefaultMode of 0o400 is added as defense-in-depth
for any future items added without an explicit per-item mode.
Fixes#3262
Signed-off-by: Dmytro Hryshchenko <[email protected]>
* docs: update pkg/constants/constants.go
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Dmytro Hryshchenko <[email protected]>
---------
Signed-off-by: Dmytro Hryshchenko <[email protected]>
Signed-off-by: Dmytro Hryshchenko <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
* feat: inject PET envs into init containers via envInjection config
- Add EnvInjection and EnvInjectionTarget types to TorchMLPolicySource
- Add FindContainerByPodSetName helper in runtime package
- Update torch plugin to inject PET envs into additional containers
- Update jobset plugin to sync init containers
- Add 5 comprehensive unit tests for envInjection scenarios
- Add Python API models for envInjection types
Signed-off-by: Peter Pan <[email protected]>
* fix: add missing +optional markers for kubeapilinter validation
Signed-off-by: Peter Pan <[email protected]>
* fix: change JobName to optional field for kubeapilinter
Change JobName from required to optional with omitempty tag to satisfy
kubeapilinter validation. Add default marker to CRD schema for listMapKey
requirement.
Signed-off-by: Peter Pan <[email protected]>
* fix test
Signed-off-by: Peter Pan <[email protected]>
* fix: address envInjection review feedback
Reject missing envInjection targets instead of silently skipping them, align the API schema with required fields, and harden init container propagation so misconfigured runtimes fail predictably.
Signed-off-by: Peter Pan <[email protected]>
* fix: sync envInjection generated files
Add the generated boilerplate and Python model updates expected by make generate so CI does not report stale generated assets.
Signed-off-by: Peter Pan <[email protected]>
* fix: satisfy kube API lint for envInjection fields
Keep the envInjection fields marked as required while preserving omitempty tags required by kubeapilinter.
Signed-off-by: Peter Pan <[email protected]>
* fix: update envInjection OpenAPI defaults
Sync generated OpenAPI and Python model output after restoring omitempty on required envInjection fields.
Signed-off-by: Peter Pan <[email protected]>
* fix: address latest review comments
Signed-off-by: Peter Pan <[email protected]>
* chore: remove test plan file per review feedback
The KEP README already has a Test Plan section; a separate file with
local environment details does not belong in the upstream PR.
Signed-off-by: Peter Pan <[email protected]>
---------
Signed-off-by: Peter Pan <[email protected]>
The `cache` dataset initializer compared the exception instance to the
`ConflictError` class using `is`, which is always False:
except ApiException as e:
if e is ConflictError: # always False
`core_v1.create_namespaced_service()` raises a plain `ApiException` with
`status=409`, not a `ConflictError`, so this branch was unreachable. When
the Service already existed (e.g., re-reconcile after a crash between
Service creation and LWS readiness), the code re-raised, fell into the
outer cleanup, and deleted the ServiceAccount — leaving the cache
cluster in a broken state.
Match the pattern used a few lines above for the ServiceAccount and
check `e.status == 409` instead.
Add a unit test that mocks `create_namespaced_service` to raise 409 and
asserts `download_dataset` completes without calling
`delete_namespaced_service_account`.
Signed-off-by: 1fanwang <[email protected]>
Adds terminationGracePeriodSeconds field to PodSpecPatch so users can
configure pod termination grace period per TrainJob via RuntimePatches.
This is needed for distributed training with PyTorch Elastic (torchrun)
where large models (70B+ parameters) require more than the default 30s
to complete JIT checkpointing before SIGKILL on node drain or TrainJob
pause.
No changes to merge logic in trainingruntime.go are required since the
existing StrategicMergePatch applied at batchv1.JobTemplateSpec level
already handles this field automatically.
Closes#3285
Signed-off-by: krishdef7 <[email protected]>
* fix(torchtune): add validation for LoRA multi-node and immutable args
Add two missing validations to validateTorchTune:
1. Reject LoRA fine-tuning when numNodes > 1. getRecipeAndConfig
silently falls through to full_finetune_distributed in this case,
discarding LoRA args without any user-visible error.
2. Reject immutable runtime configs (output_dir, tokenizer.path,
checkpointer.checkpoint_dir, tokenizer.merges_file) in
spec.trainer.args. These are injected by the runtime via
extractOverridesFromRuntime and must not be overridden by the user.
Resolves the TODO added by @Electronic-Waste in torchtune.go.
Signed-off-by: Krish Garg <[email protected]>
Signed-off-by: krishdef7 <[email protected]>
* fix(runtimes): add validation for LoRA/QLoRA multi-node and immutable trainer args
Signed-off-by: krishdef7 <[email protected]>
---------
Signed-off-by: Krish Garg <[email protected]>
Signed-off-by: krishdef7 <[email protected]>
* test(util): add unit tests for trainingruntime and trainjob utilities
Fixes#3152
pkg/util/trainingruntime (0 percent to 100 percent coverage):
- New trainingruntime_test.go with 8 cases for IsSupportDeprecated
covering nil, empty, absent key, wrong value, empty value,
correct deprecated, multiple labels, and partial match edge case.
pkg/util/trainjob (extended):
- TestRuntimeRefIsClusterTrainingRuntime: 4 new edge cases added
covering wrong APIGroup, nil APIGroup, nil Kind, both nil.
- TestIsTrainJobFinished: 6 new cases covering all condition states.
- Added metav1 import required for Condition status constants.
Signed-off-by: Yugendhar S <[email protected]>
* chore(test): address review feedback from andreyvelich
- Remove year from Copyright headers in both test files
- Add trailing newline at end of both files
- Fix error message format in TestRuntimeRefIsTrainingRuntime
to use 'FunctionName(%v) = %v, want %v' pattern for consistency
- Add nil edge cases to TestRuntimeRefIsTrainingRuntime for symmetry
with TestRuntimeRefIsClusterTrainingRuntime (nil APIGroup, nil Kind,
both nil, wrong APIGroup)
Signed-off-by: Yugendhar S <[email protected]>
* chore(test): fix line endings and formatting for CI
Signed-off-by: Yugendhar S <[email protected]>
---------
Signed-off-by: Yugendhar S <[email protected]>
* test: add unit tests for jobset builder
Add comprehensive unit tests for the Builder type methods:
- TestBuilderInitializer: dataset/model initializer env and secret injection
- TestBuilderTrainer: trainer image/command/args/resources application
- TestBuilderPodLabels, TestBuilderPodAnnotations, TestBuilderSuspend
Uses map[string]struct test pattern and go-cmp for assertions,
following established codebase conventions.
Built on top of @upsaurav12's initial work in #3176, with all review
feedback addressed.
Ref #3152
Signed-off-by: Sridhar1030 <[email protected]>
* test: add env merge coverage for initializer and trainer builder tests
Address Copilot review feedback on PR #3490:
- Add test case for Dataset.Env merge/upsert in TestBuilderInitializer
- Add test case for Trainer.Env merge/upsert in TestBuilderTrainer
Ref #3152
Signed-off-by: Sridhar1030 <[email protected]>
* test: address maintainer review feedback on builder tests
- Consolidate makeJobSetWithInitializer/makeJobSetWithTrainer into single makeJobSet helper
- Add nil Initializer/Trainer spec edge cases (replicas=1, no panic)
- Add MPI runLauncherAsNode + Trainer.Env injection test
- Add multi-replicated-job cases for PodLabels/PodAnnotations
- Add suspend=nil case (unset vs explicit false)
Ref #3152
Signed-off-by: Sridhar1030 <[email protected]>
---------
Signed-off-by: Sridhar1030 <[email protected]>
Co-authored-by: upsaurav12 <[email protected]>
* fix(webhook): allow atomic RuntimePatches update on unsuspend
Change checkRuntimePatchesImmutability to check the *old* TrainJob's
suspend state instead of the new one. This lets external controllers
(notably Kueue, see kubernetes-sigs/kueue#8296) modify spec.runtimePatches
and set spec.suspend=false in a single API request, eliminating the
two-step update workaround and the race window between the calls.
Mirrors the Kubernetes core Job validation pattern:
https://github.com/kubernetes/kubernetes/blob/86b66f6f333a/pkg/registry/batch/job/strategy.go#L191
The downstream JobSet activity check (no active replicatedJobs) still
runs, so updates while pods are running remain forbidden.
Refs: #3043
Signed-off-by: Sabari Narayana <[email protected]>
* test(jobset): cover atomic suspend->unsuspend RuntimePatches update
Add a unit test that exercises the new atomic-update path: a single
TrainJob update that both clears spec.suspend and modifies
spec.runtimePatches must succeed when the previous state was suspended.
Refs: #3043
Signed-off-by: Sabari Narayana <[email protected]>
* test(integration): cover atomic unsuspend + RuntimePatches update
Extend the trainjob webhook update table with an entry that creates a
suspended TrainJob with RuntimePatches and then, in a single Update,
both unsuspends it and modifies the runtimePatches NodeSelector. This
exercises the relaxed validation against a real apiserver and protects
the suspended -> unsuspended transition path used by Kueue.
Refs: #3043
Signed-off-by: Sabari Narayana <[email protected]>
* fix(api): block RuntimePatches change only when both states are unsuspended
Per @mimowo's review, the previous condition (!oldSuspended) over-blocked
the (oldSuspended=false, newSuspended=true) transition — i.e. suspending
a running TrainJob while modifying spec.runtimePatches in the same call,
which is safe because the JobSet activity check still guards against
concurrent pod runs.
Change the predicate to !oldSuspended && !newSuspended so the webhook
only forbids RuntimePatches mutations when the TrainJob is staying fully
unsuspended. Reword the error message accordingly so callers see
explicitly that suspension at either side of the update is acceptable.
Refs: #3043, kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>
* test: cover unsuspended->suspended atomic RuntimePatches transition
Per @mimowo's request, complete the transition table coverage:
- (false, false) -> blocked (existing test)
- (true, true) -> allowed (existing test)
- (true, false) -> allowed (added previously)
- (false, true) -> allowed (added here)
Mirrors the existing atomic-unsuspend cases — one unit case in
jobset_test.go and one webhook integration entry in trainjob_test.go.
Refs: #3043, kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>
* docs(api): shorten RuntimePatches validation comment
Trim the comment block above checkRuntimePatchesImmutability to a 4-line
summary, per @kaisoz's review feedback (the comment was longer than the
change itself). Drops the kueue#8296 backreference and the k/k Job
validation link — both are already in the PR description and commit
history.
Refs: kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>
* fix(api): use stdlib for suspend pointer checks in jobset webhook
Per @tenzen-y's review nit, swap the ptr.Equal(x, ptr.To(true)) idiom
for an inline nil-check + dereference. Equivalent semantics, no extra
generic-comparison hop, and stays in the stdlib for these two lines.
Refs: kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>
* test(jobset): cover (false,true) transition with active jobSet
Per @tenzen-y's request, add a unit case where the old TrainJob is
unsuspended, the new one is being suspended with a runtimePatches
change, and the underlying JobSet still has an active replicatedJob.
The relaxed suspend predicate alone would let this through; the
JobSet activity check should still reject it.
Refs: kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>
* test(integration): drop atomic transition cases duplicated by UTs
Per @tenzen-y's review, the two atomic-transition ginkgo entries
(unsuspend + RuntimePatches, suspend + RuntimePatches) duplicate the
unit cases in jobset_test.go that already exercise the same validator
path. The pre-existing "Should succeed to update runtimePatches when
suspend is true" entry stays as the integration smoke test for the
webhook chain.
Refs: kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>
* fix(api): use new(true) for suspend pointer comparisons
@tenzen-y is right that the value-form `new()` builtin is valid Go 1.26,
which the repo already targets (`go 1.26.0` in go.mod). Apply his
original suggestion verbatim — keeps `ptr.Equal` for the two-pointer
comparison and replaces only `ptr.To(true)`, which is the smaller and
more idiomatic delta vs the inline nil-check + deref pattern I tried
in the previous commit.
Refs: kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>
---------
Signed-off-by: Sabari Narayana <[email protected]>
* feat: customize R generation to include GPUs
Flux detection of GPUs depends on hwloc plugins, and a newer
version. To get around any edge cases where the dependencies are missing,
we can easily generate the R from the expected resource spec (cores
and gpus). We can also add a shared memory mount to ensure the job
MPI gets all available shared memory of the host. The current
default is 64M (automatic from container runtime) and it can have
implications for MPI performance.
Signed-off-by: vsoch <[email protected]>
* review: use generateRange function for gpus
Also update volume to be added to container. This still
is pending testing on AWS!
Signed-off-by: vsoch <[email protected]>
* review and add gpu trainer example
Signed-off-by: vsoch <[email protected]>
---------
Signed-off-by: vsoch <[email protected]>
Co-authored-by: vsoch <[email protected]>
* feat(api): add progress types to api
Signed-off-by: Rob Bell <[email protected]>
* feat: progress server stub
Signed-off-by: Rob Bell <[email protected]>
* feat: setup tls for server
Signed-off-by: Rob Bell <[email protected]>
* refactor: pull out middleware
Signed-off-by: Rob Bell <[email protected]>
* refactor: remove logger message noise from tests
Signed-off-by: Rob Bell <[email protected]>
* fix: send server error logs to main logger
Signed-off-by: Rob Bell <[email protected]>
* feat: update the trainer status
Signed-off-by: Rob Bell <[email protected]>
* feat: use separate client to avoid impacting main reconciler client-side rate limits
Signed-off-by: Rob Bell <[email protected]>
* feat: configure progress server in helm chart and configmap
Signed-off-by: Rob Bell <[email protected]>
* fix: prevent race condition
Signed-off-by: Rob Bell <[email protected]>
* feat: inject progress runtime config
Signed-off-by: Rob Bell <[email protected]>
* fix: progress client config
Signed-off-by: Rob Bell <[email protected]>
* fix: update generated config
Signed-off-by: Rob Bell <[email protected]>
* feat: validate auth tokens in endpoint
Signed-off-by: Rob Bell <[email protected]>
* test: validate auth tokens in endpoint
Signed-off-by: Rob Bell <[email protected]>
* refactor: tidy tests to use same format as other tests
Signed-off-by: Rob Bell <[email protected]>
* fix: look up the cluster issuer from the service account token
Different clusters can use different issuers so look up the issuer from the trusted in-cluster service account token.
Signed-off-by: Rob Bell <[email protected]>
* refactor: rename authn middleware
Signed-off-by: Rob Bell <[email protected]>
* feat: add authn to status update endpoint
Signed-off-by: Rob Bell <[email protected]>
* refactor: tidy up reading in-cluster service account token
Signed-off-by: Rob Bell <[email protected]>
* fix: rbac permissions
Signed-off-by: Rob Bell <[email protected]>
* fix: better error response
Signed-off-by: Rob Bell <[email protected]>
* refactor: move kubebuilder annotation higher up
Signed-off-by: Rob Bell <[email protected]>
* fix: make lastUpdateTime required
Signed-off-by: Rob Bell <[email protected]>
* chore: fix config tests after rebase
Signed-off-by: Rob Bell <[email protected]>
* test: update config test with new progress server config
Signed-off-by: Rob Bell <[email protected]>
* test: add progress tracking e2e test
Signed-off-by: Rob Bell <[email protected]>
* fix: allow feature gates to be configurable from command line argument
So they can be overridden in e2e tests
Signed-off-by: Rob Bell <[email protected]>
* refactor: reuse shared progress StatusUrl
Signed-off-by: Rob Bell <[email protected]>
* test: tidy up tests
Signed-off-by: Rob Bell <[email protected]>
* refactor: allow plugins to use config
Signed-off-by: Rob Bell <[email protected]>
* refactor: centralise shared constants
Signed-off-by: Rob Bell <[email protected]>
* fix: disable http2 in progress server tls opts if disabled by config
Signed-off-by: Rob Bell <[email protected]>
* fix: minor improvements suggested by copilot
Signed-off-by: Rob Bell <[email protected]>
* chore: rename progress server => status server
Signed-off-by: Rob Bell <[email protected]>
* chore: update type doc
Signed-off-by: Rob Bell <[email protected]>
* chore: update name of feature gate
Signed-off-by: Rob Bell <[email protected]>
* fix: after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: fix typo
Signed-off-by: Rob Bell <[email protected]>
* chore: rename handler
Signed-off-by: Rob Bell <[email protected]>
* fix: run status server on all replicas
Signed-off-by: Rob Bell <[email protected]>
* fix: deep copy the rest config
Signed-off-by: Rob Bell <[email protected]>
* refactor: use ssa
Signed-off-by: Rob Bell <[email protected]>
* refactor: separate token audience per train job
Signed-off-by: Rob Bell <[email protected]>
* chore: rename status server
Signed-off-by: Rob Bell <[email protected]>
* fix: revert adding status pkg to manifest gen paths
Signed-off-by: Rob Bell <[email protected]>
* chore: update log messages
Signed-off-by: Rob Bell <[email protected]>
* chore: update api docs
Signed-off-by: Rob Bell <[email protected]>
* chore: update format
Signed-off-by: Rob Bell <[email protected]>
* fix: revert inadvertent change
Signed-off-by: Rob Bell <[email protected]>
* chore: update api name and type docs
Signed-off-by: Rob Bell <[email protected]>
* refactor: initialise oidc provider in manager startup
Signed-off-by: Rob Bell <[email protected]>
* fix: allow empty update request
Signed-off-by: Rob Bell <[email protected]>
* fix: wait for server shut down
Signed-off-by: Rob Bell <[email protected]>
* fix: handle missing train job
Signed-off-by: Rob Bell <[email protected]>
* refactor: update env var names
Signed-off-by: Rob Bell <[email protected]>
* chore: rename packages, plugins, feature gate
Signed-off-by: Rob Bell <[email protected]>
* refactor: update env var names
Signed-off-by: Rob Bell <[email protected]>
* chore: regenerate after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: fixes after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: minor doc updates
Signed-off-by: Rob Bell <[email protected]>
* chore: rename status server config
Signed-off-by: Rob Bell <[email protected]>
* chore: replace placeholder config with nil
Signed-off-by: Rob Bell <[email protected]>
* chore: revert separate runtime utils package
Signed-off-by: Rob Bell <[email protected]>
* chore: remove unused config from tests
Signed-off-by: Rob Bell <[email protected]>
---------
Signed-off-by: Rob Bell <[email protected]>
* feat(api): add ActiveDeadlineSeconds and TTLSecondsAfterFinished to Trainer APIs
- Add ActiveDeadlineSeconds *int64 to TrainJobSpec (immutable, min=1)
with reason constant TrainJobDeadlineExceededReason="DeadlineExceeded"
- Add TTLSecondsAfterFinished *int32 to TrainingRuntimeSpec (min=0)
with cross-field CEL rules blocking conflicting lifecycle fields
in the JobSet/Job template
- Add TrainJobDeadlineExceededMessage to constants
- Regenerate zz_generated.deepcopy.go
Part of KEP-2899
Signed-off-by: XploY04 <[email protected]>
* feat(controller): add TTL-based auto-deletion for finished TrainJobs
- Add early-exit guard in Reconcile before prevTrainJob DeepCopy and
removeFailedCondition, mirroring the Kubernetes Job controller pattern.
Once a TrainJob is terminal (Complete or Failed), only TTL cleanup runs.
- Add fetchTTLFromRuntime helper to retrieve TTLSecondsAfterFinished from
the referenced TrainingRuntime or ClusterTrainingRuntime.
- Add reconcileTTL helper to delete finished TrainJobs after their TTL
expires, or requeue at the exact expiry time.
Part of KEP-2899
Signed-off-by: XploY04 <[email protected]>
* feat(controller): enforce ActiveDeadlineSeconds on TrainJobs
- Add reconcileDeadline helper to terminate running TrainJobs that
exceed their ActiveDeadlineSeconds.
- Handle job suspension: StartTime is determined by the LastTransitionTime
of the Suspended=False condition (if it exists) to ensure the deadline
timer resets correctly upon resume.
- Jobs exceeding the deadline are moved to a Failed state with
reason DeadlineExceeded.
Part of KEP-2899
Signed-off-by: XploY04 <[email protected]>
* feat(webhooks): warn on short ttlSecondsAfterFinished
- Add an admission warning to both TrainingRuntime and
ClusterTrainingRuntime webhooks if the TTLSecondsAfterFinished
is configured to less than 60 seconds, as this may cause
unexpected data/log loss for data scientists due to overly
aggressive cleanup.
Part of KEP-2899
Signed-off-by: XploY04 <[email protected]>
* test: add integration tests for TrainJob TTL and deadline lifecycle
- Add integration tests for DeadlineExceeded behavior in TrainJob controller
- Add integration tests for TTL validation and warnings in webhooks
Part of KEP-2899
Signed-off-by: XploY04 <[email protected]>
* cleanup
Signed-off-by: XploY04 <[email protected]>
* fix: ensure JobSet is suspended on expiration and fix webhook test regress
Signed-off-by: XploY04 <[email protected]>
* fix(controller): return errors from fetchTTLFromRuntime for proper requeue
Signed-off-by: XploY04 <[email protected]>
* test: remove unused wantWarning from webhook tests
Signed-off-by: XploY04 <[email protected]>
* fix
Signed-off-by: XploY04 <[email protected]>
* feat: Remove from TrainingRuntime and ClusterTrainingRuntime specifications and associated validation.
Signed-off-by: XploY04 <[email protected]>
* fix(api): restore `activeDeadlineSeconds` validation rule on TrainingRuntime
Signed-off-by: XploY04 <[email protected]>
* refactor: remove unnecessary blank lines and improve logging in ClusterTrainingRuntime validation
Signed-off-by: XploY04 <[email protected]>
* feat: add activeDeadlineSeconds field to TrainJobSpec and update related tests
Signed-off-by: XploY04 <[email protected]>
* Remove unnecessary newline in test file
Signed-off-by: Yash Agarwal <[email protected]>
* Remove unnecessary newline in trainingruntime_webhook_test.go
Signed-off-by: Yash Agarwal <[email protected]>
* refactor: remove unnecessary blank lines in TrainJob reconciliation and deadline tests
Signed-off-by: XploY04 <[email protected]>
* feat: change ActiveDeadlineSeconds from pointer to value type and update related code
Signed-off-by: XploY04 <[email protected]>
* feat: add condition check for TrainJob deadline exceeded before suspending
Signed-off-by: XploY04 <[email protected]>
* feat: enhance TrainJob deadline handling and suspend logic
Signed-off-by: XploY04 <[email protected]>
* feat: enhance TrainJob deadline handling and suspend logic
Signed-off-by: XploY04 <[email protected]>
* fix
Signed-off-by: XploY04 <[email protected]>
* feat: change ActiveDeadlineSeconds to pointer type and update related logic
Signed-off-by: XploY04 <[email protected]>
* feat: ensure requeueAfter is at least 1 second when deadline is exceeded
Signed-off-by: XploY04 <[email protected]>
* feat: change ActiveDeadlineSeconds from pointer to value type and update related logic
Signed-off-by: XploY04 <[email protected]>
* fix: remove early return for finished TrainJobs to prevent status sync issues
The early return was preventing the controller from syncing JobSet
status updates for finished TrainJobs, causing e2e tests to timeout
waiting for conditions that never got updated.
This fix restores the upstream behavior where finished TrainJobs still
go through status reconciliation (though reconcileObjects is skipped).
Signed-off-by: XploY04 <[email protected]>
* fix
Signed-off-by: XploY04 <[email protected]>
* fix: remove x-kubernetes-validations for activeDeadlineSeconds in TrainingRuntime CRDs
Signed-off-by: XploY04 <[email protected]>
* fix: update validation message for ActiveDeadlineSeconds to be more generic
Signed-off-by: XploY04 <[email protected]>
* replacing suspend with delete
Signed-off-by: XploY04 <[email protected]>
* fix: remove SpecLabel and SpecAnnotation methods from TrainJobWrapper
Signed-off-by: XploY04 <[email protected]>
* fix: remove annotations field from TrainJobSpec and CRD definitions
Signed-off-by: XploY04 <[email protected]>
* ensure JobSet deletion in e2e tests
Signed-off-by: XploY04 <[email protected]>
* fix: add JobSet APIs to the scheme in e2e tests
Signed-off-by: XploY04 <[email protected]>
* fix: simplify error handling in Reconcile method of TrainJobReconciler
Signed-off-by: XploY04 <[email protected]>
---------
Signed-off-by: XploY04 <[email protected]>
Signed-off-by: Yash Agarwal <[email protected]>
