Commit Graph
90 Commits
Author SHA1 Message Date
Immanuel TikhonovandGitHub 4d9f287c3f fix(operator): guard against nil trainer PodSet in JAX EnforceMLPolicy (#3563)
Signed-off-by: Immanuel Tikhonov <[email protected]>
Signed-off-by: immanuwell <[email protected]>
2026-06-11 20:46:10 +00:00
Federico BongiovanniandGitHub af6bac58d1 fix(operator): support multi-replica endpoint generation in IdentifyP… (#3539)
* fix(operator): support multi-replica endpoint generation in IdentifyPodNetwork

Signed-off-by: Federico Bongiovanni <[email protected]>

* style(operator): address Copilot review comments on IdentifyPodNetwork

Signed-off-by: Federico Bongiovanni <[email protected]>

---------

Signed-off-by: Federico Bongiovanni <[email protected]>
2026-06-02 13:07:36 +00:00
Tariq HasanandGitHub 9183dfd24f chore(scripts): add copyright boilerplate verification (#3359)
* chore(scripts): add copyright boilerplate verification system

Signed-off-by: tariq-hasan <[email protected]>

* chore(ci): add boilerplate verification to CI pipeline

Signed-off-by: tariq-hasan <[email protected]>

* chore: regenerate files with year-less copyright headers

Signed-off-by: tariq-hasan <[email protected]>

* refactor(boilerplate): simplify verification logic

Signed-off-by: tariq-hasan <[email protected]>

* chore(boilerplate): add missing copyright headers

Signed-off-by: tariq-hasan <[email protected]>

* ci(boilerplate): move verifier to code-quality workflow and run unit tests

Signed-off-by: tariq-hasan <[email protected]>

* feat(boilerplate): accept license headers with any year and remove year validation logic

Signed-off-by: tariq-hasan <[email protected]>

* feat(boilerplate): add copyright headers to generated Python API

Signed-off-by: tariq-hasan <[email protected]>

* refactor(boilerplate): use block-comment header for generated Go files

Signed-off-by: tariq-hasan <[email protected]>

* feat(boilerplate): verify copyright headers in Rust files

Signed-off-by: tariq-hasan <[email protected]>

---------

Signed-off-by: tariq-hasan <[email protected]>
2026-05-29 10:50:56 +00:00
Krish GargandGitHub 4f8920c902 feat(api): Add terminationGracePeriodSeconds to PodSpecPatch in TrainJob (#3324)
Adds terminationGracePeriodSeconds field to PodSpecPatch so users can
configure pod termination grace period per TrainJob via RuntimePatches.

This is needed for distributed training with PyTorch Elastic (torchrun)
where large models (70B+ parameters) require more than the default 30s
to complete JIT checkpointing before SIGKILL on node drain or TrainJob
pause.

No changes to merge logic in trainingruntime.go are required since the
existing StrategicMergePatch applied at batchv1.JobTemplateSpec level
already handles this field automatically.

Closes #3285

Signed-off-by: krishdef7 <[email protected]>
2026-05-26 17:36:09 +00:00
krishdef7andGitHub cdff07ff48 fix(runtimes): add validation for LoRA multi-node and immutable trainer args (#3302)
* fix(torchtune): add validation for LoRA multi-node and immutable args

Add two missing validations to validateTorchTune:

1. Reject LoRA fine-tuning when numNodes > 1. getRecipeAndConfig
   silently falls through to full_finetune_distributed in this case,
   discarding LoRA args without any user-visible error.

2. Reject immutable runtime configs (output_dir, tokenizer.path,
   checkpointer.checkpoint_dir, tokenizer.merges_file) in
   spec.trainer.args. These are injected by the runtime via
   extractOverridesFromRuntime and must not be overridden by the user.

Resolves the TODO added by @Electronic-Waste in torchtune.go.

Signed-off-by: Krish Garg <[email protected]>
Signed-off-by: krishdef7 <[email protected]>

* fix(runtimes): add validation for LoRA/QLoRA multi-node and immutable trainer args

Signed-off-by: krishdef7 <[email protected]>

---------

Signed-off-by: Krish Garg <[email protected]>
Signed-off-by: krishdef7 <[email protected]>
2026-05-20 14:15:20 +00:00
6da5d948b2 feat(runtimes): add validation for reserved MPI environment variables (#3491)
Signed-off-by: Vishal Painjane <[email protected]>
Co-authored-by: Vishal Painjane <[email protected]>
2026-05-09 17:39:35 +00:00
ec4c5c0fd5 chore(test): add unit tests for jobset builder (#3490)
* test: add unit tests for jobset builder

Add comprehensive unit tests for the Builder type methods:
- TestBuilderInitializer: dataset/model initializer env and secret injection
- TestBuilderTrainer: trainer image/command/args/resources application
- TestBuilderPodLabels, TestBuilderPodAnnotations, TestBuilderSuspend

Uses map[string]struct test pattern and go-cmp for assertions,
following established codebase conventions.

Built on top of @upsaurav12's initial work in #3176, with all review
feedback addressed.

Ref #3152

Signed-off-by: Sridhar1030 <[email protected]>

* test: add env merge coverage for initializer and trainer builder tests
Address Copilot review feedback on PR #3490:
- Add test case for Dataset.Env merge/upsert in TestBuilderInitializer
- Add test case for Trainer.Env merge/upsert in TestBuilderTrainer
Ref #3152

Signed-off-by: Sridhar1030 <[email protected]>

* test: address maintainer review feedback on builder tests

- Consolidate makeJobSetWithInitializer/makeJobSetWithTrainer into single makeJobSet helper
- Add nil Initializer/Trainer spec edge cases (replicas=1, no panic)
- Add MPI runLauncherAsNode + Trainer.Env injection test
- Add multi-replicated-job cases for PodLabels/PodAnnotations
- Add suspend=nil case (unset vs explicit false)

Ref #3152

Signed-off-by: Sridhar1030 <[email protected]>

---------

Signed-off-by: Sridhar1030 <[email protected]>
Co-authored-by: upsaurav12 <[email protected]>
2026-05-08 20:30:29 +00:00
Kevin HannonandGitHub 33f01679cc feat: update k8s to 0.36 (#3470)
Signed-off-by: Kevin Hannon <[email protected]>
2026-05-08 12:38:28 +00:00
Sameer_YadavandGitHub 3135ce6d89 chore: unit tests for runtime registry and RuntimeInfo error handling (#3168)
* Add unit tests for runtime registry and RuntimeInfo error handling

Signed-off-by: Sameer_yadav <[email protected]>

* chore: apply formatting fixes

Signed-off-by: Sameer_yadav <[email protected]>

* chore: fix tests for runtime registry and RuntimeInfo error handling

Signed-off-by: Sameer_yadav <[email protected]>

* chore: format code with gofmt

Signed-off-by: Sameer_yadav <[email protected]>

* test: improve registry test robustness

Signed-off-by: Sameer_yadav <[email protected]>

* removed year and changed the test function name

Signed-off-by: Sameer_yadav <[email protected]>

---------

Signed-off-by: Sameer_yadav <[email protected]>
2026-05-06 11:41:38 +00:00
Kevin HannonandGitHub 52d7fd69c4 chore: update golangci-linter and kube-api-linter (#3471)
Signed-off-by: Kevin Hannon <[email protected]>
2026-05-05 04:54:27 +00:00
Sabari NarayanaandGitHub 844656a01d fix(api): allow atomic RuntimePatches update on unsuspend (#3469)
* fix(webhook): allow atomic RuntimePatches update on unsuspend

Change checkRuntimePatchesImmutability to check the *old* TrainJob's
suspend state instead of the new one. This lets external controllers
(notably Kueue, see kubernetes-sigs/kueue#8296) modify spec.runtimePatches
and set spec.suspend=false in a single API request, eliminating the
two-step update workaround and the race window between the calls.

Mirrors the Kubernetes core Job validation pattern:
https://github.com/kubernetes/kubernetes/blob/86b66f6f333a/pkg/registry/batch/job/strategy.go#L191

The downstream JobSet activity check (no active replicatedJobs) still
runs, so updates while pods are running remain forbidden.

Refs: #3043
Signed-off-by: Sabari Narayana <[email protected]>

* test(jobset): cover atomic suspend->unsuspend RuntimePatches update

Add a unit test that exercises the new atomic-update path: a single
TrainJob update that both clears spec.suspend and modifies
spec.runtimePatches must succeed when the previous state was suspended.

Refs: #3043
Signed-off-by: Sabari Narayana <[email protected]>

* test(integration): cover atomic unsuspend + RuntimePatches update

Extend the trainjob webhook update table with an entry that creates a
suspended TrainJob with RuntimePatches and then, in a single Update,
both unsuspends it and modifies the runtimePatches NodeSelector. This
exercises the relaxed validation against a real apiserver and protects
the suspended -> unsuspended transition path used by Kueue.

Refs: #3043
Signed-off-by: Sabari Narayana <[email protected]>

* fix(api): block RuntimePatches change only when both states are unsuspended

Per @mimowo's review, the previous condition (!oldSuspended) over-blocked
the (oldSuspended=false, newSuspended=true) transition — i.e. suspending
a running TrainJob while modifying spec.runtimePatches in the same call,
which is safe because the JobSet activity check still guards against
concurrent pod runs.

Change the predicate to !oldSuspended && !newSuspended so the webhook
only forbids RuntimePatches mutations when the TrainJob is staying fully
unsuspended. Reword the error message accordingly so callers see
explicitly that suspension at either side of the update is acceptable.

Refs: #3043, kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>

* test: cover unsuspended->suspended atomic RuntimePatches transition

Per @mimowo's request, complete the transition table coverage:

- (false, false) -> blocked   (existing test)
- (true,  true)  -> allowed   (existing test)
- (true,  false) -> allowed   (added previously)
- (false, true)  -> allowed   (added here)

Mirrors the existing atomic-unsuspend cases — one unit case in
jobset_test.go and one webhook integration entry in trainjob_test.go.

Refs: #3043, kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>

* docs(api): shorten RuntimePatches validation comment

Trim the comment block above checkRuntimePatchesImmutability to a 4-line
summary, per @kaisoz's review feedback (the comment was longer than the
change itself). Drops the kueue#8296 backreference and the k/k Job
validation link — both are already in the PR description and commit
history.

Refs: kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>

* fix(api): use stdlib for suspend pointer checks in jobset webhook

Per @tenzen-y's review nit, swap the ptr.Equal(x, ptr.To(true)) idiom
for an inline nil-check + dereference. Equivalent semantics, no extra
generic-comparison hop, and stays in the stdlib for these two lines.

Refs: kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>

* test(jobset): cover (false,true) transition with active jobSet

Per @tenzen-y's request, add a unit case where the old TrainJob is
unsuspended, the new one is being suspended with a runtimePatches
change, and the underlying JobSet still has an active replicatedJob.
The relaxed suspend predicate alone would let this through; the
JobSet activity check should still reject it.

Refs: kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>

* test(integration): drop atomic transition cases duplicated by UTs

Per @tenzen-y's review, the two atomic-transition ginkgo entries
(unsuspend + RuntimePatches, suspend + RuntimePatches) duplicate the
unit cases in jobset_test.go that already exercise the same validator
path. The pre-existing "Should succeed to update runtimePatches when
suspend is true" entry stays as the integration smoke test for the
webhook chain.

Refs: kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>

* fix(api): use new(true) for suspend pointer comparisons

@tenzen-y is right that the value-form `new()` builtin is valid Go 1.26,
which the repo already targets (`go 1.26.0` in go.mod). Apply his
original suggestion verbatim — keeps `ptr.Equal` for the two-pointer
comparison and replaces only `ptr.To(true)`, which is the smaller and
more idiomatic delta vs the inline nil-check + deref pattern I tried
in the previous commit.

Refs: kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>

---------

Signed-off-by: Sabari Narayana <[email protected]>
2026-05-05 04:52:26 +00:00
1510b1c0b1 feat: customize R generation to include GPUs (#3325)
* feat: customize R generation to include GPUs

Flux detection of GPUs depends on hwloc plugins, and a newer
version. To get around any edge cases where the dependencies are missing,
we can easily generate the R from the expected resource spec (cores
and gpus). We can also add a shared memory mount to ensure the job
MPI gets all available shared memory of the host. The current
default is 64M (automatic from container runtime) and it can have
implications for MPI performance.

Signed-off-by: vsoch <[email protected]>

* review: use generateRange function for gpus

Also update volume to be added to container. This still
is pending testing on AWS!

Signed-off-by: vsoch <[email protected]>

* review and add gpu trainer example

Signed-off-by: vsoch <[email protected]>

---------

Signed-off-by: vsoch <[email protected]>
Co-authored-by: vsoch <[email protected]>
2026-03-23 15:07:25 +00:00
Tomas TormoandGitHub f47c01b1c1 chore(runtimes): persist runtimes map and expose Runtimes function (#3360)
Signed-off-by: Tomas Tormo <[email protected]>
2026-03-19 10:37:26 +00:00
Rob BellandGitHub 78b4a00929 feat: add support for tracking TrainJob progress and training metrics (#3227)
* feat(api): add progress types to api

Signed-off-by: Rob Bell <[email protected]>

* feat: progress server stub

Signed-off-by: Rob Bell <[email protected]>

* feat: setup tls for server

Signed-off-by: Rob Bell <[email protected]>

* refactor: pull out middleware

Signed-off-by: Rob Bell <[email protected]>

* refactor: remove logger message noise from tests

Signed-off-by: Rob Bell <[email protected]>

* fix: send server error logs to main logger

Signed-off-by: Rob Bell <[email protected]>

* feat: update the trainer status

Signed-off-by: Rob Bell <[email protected]>

* feat: use separate client to avoid impacting main reconciler client-side rate limits

Signed-off-by: Rob Bell <[email protected]>

* feat: configure progress server in helm chart and configmap

Signed-off-by: Rob Bell <[email protected]>

* fix: prevent race condition

Signed-off-by: Rob Bell <[email protected]>

* feat: inject progress runtime config

Signed-off-by: Rob Bell <[email protected]>

* fix: progress client config

Signed-off-by: Rob Bell <[email protected]>

* fix: update generated config

Signed-off-by: Rob Bell <[email protected]>

* feat: validate auth tokens in endpoint

Signed-off-by: Rob Bell <[email protected]>

* test: validate auth tokens in endpoint

Signed-off-by: Rob Bell <[email protected]>

* refactor: tidy tests to use same format as other tests

Signed-off-by: Rob Bell <[email protected]>

* fix: look up the cluster issuer from the service account token

Different clusters can use different issuers so look up the issuer from the trusted in-cluster service account token.

Signed-off-by: Rob Bell <[email protected]>

* refactor: rename authn middleware

Signed-off-by: Rob Bell <[email protected]>

* feat: add authn to status update endpoint

Signed-off-by: Rob Bell <[email protected]>

* refactor: tidy up reading in-cluster service account token

Signed-off-by: Rob Bell <[email protected]>

* fix: rbac permissions

Signed-off-by: Rob Bell <[email protected]>

* fix: better error response

Signed-off-by: Rob Bell <[email protected]>

* refactor: move kubebuilder annotation higher up

Signed-off-by: Rob Bell <[email protected]>

* fix: make lastUpdateTime required

Signed-off-by: Rob Bell <[email protected]>

* chore: fix config tests after rebase

Signed-off-by: Rob Bell <[email protected]>

* test: update config test with new progress server config

Signed-off-by: Rob Bell <[email protected]>

* test: add progress tracking e2e test

Signed-off-by: Rob Bell <[email protected]>

* fix: allow feature gates to be configurable from command line argument

So they can be overridden in e2e tests

Signed-off-by: Rob Bell <[email protected]>

* refactor: reuse shared progress StatusUrl

Signed-off-by: Rob Bell <[email protected]>

* test: tidy up tests

Signed-off-by: Rob Bell <[email protected]>

* refactor: allow plugins to use config

Signed-off-by: Rob Bell <[email protected]>

* refactor: centralise shared constants

Signed-off-by: Rob Bell <[email protected]>

* fix: disable http2 in progress server tls opts if disabled by config

Signed-off-by: Rob Bell <[email protected]>

* fix: minor improvements suggested by copilot

Signed-off-by: Rob Bell <[email protected]>

* chore: rename progress server => status server

Signed-off-by: Rob Bell <[email protected]>

* chore: update type doc

Signed-off-by: Rob Bell <[email protected]>

* chore: update name of feature gate

Signed-off-by: Rob Bell <[email protected]>

* fix: after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: fix typo

Signed-off-by: Rob Bell <[email protected]>

* chore: rename handler

Signed-off-by: Rob Bell <[email protected]>

* fix: run status server on all replicas

Signed-off-by: Rob Bell <[email protected]>

* fix: deep copy the rest config

Signed-off-by: Rob Bell <[email protected]>

* refactor: use ssa

Signed-off-by: Rob Bell <[email protected]>

* refactor: separate token audience per train job

Signed-off-by: Rob Bell <[email protected]>

* chore: rename status server

Signed-off-by: Rob Bell <[email protected]>

* fix: revert adding status pkg to manifest gen paths

Signed-off-by: Rob Bell <[email protected]>

* chore: update log messages

Signed-off-by: Rob Bell <[email protected]>

* chore: update api docs

Signed-off-by: Rob Bell <[email protected]>

* chore: update format

Signed-off-by: Rob Bell <[email protected]>

* fix: revert inadvertent change

Signed-off-by: Rob Bell <[email protected]>

* chore: update api name and type docs

Signed-off-by: Rob Bell <[email protected]>

* refactor: initialise oidc provider in manager startup

Signed-off-by: Rob Bell <[email protected]>

* fix: allow empty update request

Signed-off-by: Rob Bell <[email protected]>

* fix: wait for server shut down

Signed-off-by: Rob Bell <[email protected]>

* fix: handle missing train job

Signed-off-by: Rob Bell <[email protected]>

* refactor: update env var names

Signed-off-by: Rob Bell <[email protected]>

* chore: rename packages, plugins, feature gate

Signed-off-by: Rob Bell <[email protected]>

* refactor: update env var names

Signed-off-by: Rob Bell <[email protected]>

* chore: regenerate after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: fixes after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: minor doc updates

Signed-off-by: Rob Bell <[email protected]>

* chore: rename status server config

Signed-off-by: Rob Bell <[email protected]>

* chore: replace placeholder config with nil

Signed-off-by: Rob Bell <[email protected]>

* chore: revert separate runtime utils package

Signed-off-by: Rob Bell <[email protected]>

* chore: remove unused config from tests

Signed-off-by: Rob Bell <[email protected]>

---------

Signed-off-by: Rob Bell <[email protected]>
2026-03-13 20:01:08 +00:00
Yash AgarwalandGitHub 587010f54d feat: add activeDeadlineSeconds (#3258)
* feat(api): add ActiveDeadlineSeconds and TTLSecondsAfterFinished to Trainer APIs

- Add ActiveDeadlineSeconds *int64 to TrainJobSpec (immutable, min=1)
  with reason constant TrainJobDeadlineExceededReason="DeadlineExceeded"
- Add TTLSecondsAfterFinished *int32 to TrainingRuntimeSpec (min=0)
  with cross-field CEL rules blocking conflicting lifecycle fields
  in the JobSet/Job template
- Add TrainJobDeadlineExceededMessage to constants
- Regenerate zz_generated.deepcopy.go

Part of KEP-2899

Signed-off-by: XploY04 <[email protected]>

* feat(controller): add TTL-based auto-deletion for finished TrainJobs

- Add early-exit guard in Reconcile before prevTrainJob DeepCopy and
  removeFailedCondition, mirroring the Kubernetes Job controller pattern.
  Once a TrainJob is terminal (Complete or Failed), only TTL cleanup runs.
- Add fetchTTLFromRuntime helper to retrieve TTLSecondsAfterFinished from
  the referenced TrainingRuntime or ClusterTrainingRuntime.
- Add reconcileTTL helper to delete finished TrainJobs after their TTL
  expires, or requeue at the exact expiry time.

Part of KEP-2899

Signed-off-by: XploY04 <[email protected]>

* feat(controller): enforce ActiveDeadlineSeconds on TrainJobs

- Add reconcileDeadline helper to terminate running TrainJobs that
  exceed their ActiveDeadlineSeconds.
- Handle job suspension: StartTime is determined by the LastTransitionTime
  of the Suspended=False condition (if it exists) to ensure the deadline
  timer resets correctly upon resume.
- Jobs exceeding the deadline are moved to a Failed state with
  reason DeadlineExceeded.

Part of KEP-2899

Signed-off-by: XploY04 <[email protected]>

* feat(webhooks): warn on short ttlSecondsAfterFinished

- Add an admission warning to both TrainingRuntime and
  ClusterTrainingRuntime webhooks if the TTLSecondsAfterFinished
  is configured to less than 60 seconds, as this may cause
  unexpected data/log loss for data scientists due to overly
  aggressive cleanup.

Part of KEP-2899

Signed-off-by: XploY04 <[email protected]>

* test: add integration tests for TrainJob TTL and deadline lifecycle

- Add integration tests for DeadlineExceeded behavior in TrainJob controller
- Add integration tests for TTL validation and warnings in webhooks

Part of KEP-2899

Signed-off-by: XploY04 <[email protected]>

* cleanup

Signed-off-by: XploY04 <[email protected]>

* fix: ensure JobSet is suspended on expiration and fix webhook test regress

Signed-off-by: XploY04 <[email protected]>

* fix(controller): return errors from fetchTTLFromRuntime for proper requeue

Signed-off-by: XploY04 <[email protected]>

* test: remove unused wantWarning from webhook tests

Signed-off-by: XploY04 <[email protected]>

* fix

Signed-off-by: XploY04 <[email protected]>

* feat: Remove  from TrainingRuntime and ClusterTrainingRuntime specifications and associated validation.

Signed-off-by: XploY04 <[email protected]>

* fix(api): restore `activeDeadlineSeconds` validation rule on TrainingRuntime

Signed-off-by: XploY04 <[email protected]>

* refactor: remove unnecessary blank lines and improve logging in ClusterTrainingRuntime validation

Signed-off-by: XploY04 <[email protected]>

* feat: add activeDeadlineSeconds field to TrainJobSpec and update related tests

Signed-off-by: XploY04 <[email protected]>

* Remove unnecessary newline in test file

Signed-off-by: Yash Agarwal <[email protected]>

* Remove unnecessary newline in trainingruntime_webhook_test.go

Signed-off-by: Yash Agarwal <[email protected]>

* refactor: remove unnecessary blank lines in TrainJob reconciliation and deadline tests

Signed-off-by: XploY04 <[email protected]>

* feat: change ActiveDeadlineSeconds from pointer to value type and update related code

Signed-off-by: XploY04 <[email protected]>

* feat: add condition check for TrainJob deadline exceeded before suspending

Signed-off-by: XploY04 <[email protected]>

* feat: enhance TrainJob deadline handling and suspend logic

Signed-off-by: XploY04 <[email protected]>

* feat: enhance TrainJob deadline handling and suspend logic

Signed-off-by: XploY04 <[email protected]>

* fix

Signed-off-by: XploY04 <[email protected]>

* feat: change ActiveDeadlineSeconds to pointer type and update related logic

Signed-off-by: XploY04 <[email protected]>

* feat: ensure requeueAfter is at least 1 second when deadline is exceeded

Signed-off-by: XploY04 <[email protected]>

* feat: change ActiveDeadlineSeconds from pointer to value type and update related logic

Signed-off-by: XploY04 <[email protected]>

* fix: remove early return for finished TrainJobs to prevent status sync issues

The early return was preventing the controller from syncing JobSet
status updates for finished TrainJobs, causing e2e tests to timeout
waiting for conditions that never got updated.

This fix restores the upstream behavior where finished TrainJobs still
go through status reconciliation (though reconcileObjects is skipped).

Signed-off-by: XploY04 <[email protected]>

* fix

Signed-off-by: XploY04 <[email protected]>

* fix: remove x-kubernetes-validations for activeDeadlineSeconds in TrainingRuntime CRDs

Signed-off-by: XploY04 <[email protected]>

* fix: update validation message for ActiveDeadlineSeconds to be more generic

Signed-off-by: XploY04 <[email protected]>

* replacing suspend with delete

Signed-off-by: XploY04 <[email protected]>

* fix: remove SpecLabel and SpecAnnotation methods from TrainJobWrapper

Signed-off-by: XploY04 <[email protected]>

* fix: remove annotations field from TrainJobSpec and CRD definitions

Signed-off-by: XploY04 <[email protected]>

* ensure JobSet deletion in e2e tests

Signed-off-by: XploY04 <[email protected]>

* fix: add JobSet APIs to the scheme in e2e tests

Signed-off-by: XploY04 <[email protected]>

* fix: simplify error handling in Reconcile method of TrainJobReconciler

Signed-off-by: XploY04 <[email protected]>

---------

Signed-off-by: XploY04 <[email protected]>
Signed-off-by: Yash Agarwal <[email protected]>
2026-03-13 18:37:31 +00:00
Amir IbrahimandGitHub 926b01ab02 fix: rename JAX and Torch runtime plugin tests to descriptive names (#3283)
Signed-off-by: Amir380-A <[email protected]>
2026-03-13 01:52:14 +00:00
37c8a07a8a feat(api): Replace PodTemplateOverrides with RuntimePatches API (#3309)
* feat(api): Replace PodTemplateOverrides with RuntimePatches API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update goal with RuntimePatches API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add diagram for RuntimePatches

Signed-off-by: Andrey Velichkevich <[email protected]>

* feat(api): Replace PodTemplateOverrides with RuntimePatches API

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix kube lint errors

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update Python models

Signed-off-by: Antonin Stefanutti <[email protected]>

* Enable maxlength Kube-API-Linter rule

Signed-off-by: Antonin Stefanutti <[email protected]>

* Restore manger field IsDomainPrefixedPath validation

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix integration tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Apply StrategicMergePatch to batchv1.JobTemplateSpec

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Antonin Stefanutti <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
2026-03-12 16:29:44 +00:00
Krishna GuptaandGitHub ccaf20bc20 feat(runtimes): Add XGBoost runtime(KEP-2598) (#3200)
* Add XGBoost runtime plugin scaffold and register in framework

Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>

* regenerated assets

Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>

* Add XGBoost plugin unit tests and update framework test registry

Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>

* added :Runtime image , unit tests for :(Resources are not set, Resources are set in TrainJob) ,using the ContainerTrainerPort instead of default port

Signed-off-by: krishna-kg732 <[email protected]>

* minor changes

Signed-off-by: krishna-kg732 <[email protected]>

* added plainML fallback(JAX+XGBOOST)

Signed-off-by: krishna-kg732 <[email protected]>

* test: add XGBoost runtime resource resolution unit tests

- Add test case for resources set in Runtime only
- Add test case for resources set in both Runtime and TrainJob
- Add xgboost_distributed.yaml ClusterTrainingRuntime manifest
- Update kustomization.yaml with xgboost-runtime image

Addresses feedback from PR #3200 for complete resource resolution test coverage.

Signed-off-by: krishna-kg732 <[email protected]>

* added autogenerat3ed files

Signed-off-by: krishna-kg732 <[email protected]>

* CI fixes

Signed-off-by: krishna-kg732 <[email protected]>

* chore: fix ci lint errors

Signed-off-by: krishna-kg732 <[email protected]>

* Add XGBoost integration and E2E tests, fix Validate guard clause and nil safety

Signed-off-by: krishna-kg732 <[email protected]>

* trigger ci

Signed-off-by: krishna-kg732 <[email protected]>

* fix: Add XGBoost support to test wrapper, CEL validation, and CI publish workflow

- Add XGBoostPolicy() method to MLPolicySourceWrapper for integration tests
- Add has(self.xgboost) to MLPolicy CEL mutual-exclusivity validation rule
- Regenerate CRD manifests with updated CEL rule
- Add xgboost-runtime to build-and-push-images CI workflow
- Add XGBoost image build step to E2E setup script (commented out due to Kind bug)

Signed-off-by: krishna-kg732 <[email protected]>

* docs: Add XGBoost distributed training example notebook

Add example notebook demonstrating distributed XGBoost training using
the Kubeflow Trainer SDK and the xgboost-distributed ClusterTrainingRuntime.

The notebook uses the modern XGBoost Collective API with RabitTracker
for distributed coordination and synthetic classification data.

Signed-off-by: krishna-kg732 <[email protected]>

* test: Skip XGBoost E2E test until runtime image is published

Signed-off-by: krishna-kg732 <[email protected]>

* chore : address review comments

Signed-off-by: krishna-kg732 <[email protected]>

* chore:resolve merge conflicts

Signed-off-by: krishna-kg732 <[email protected]>

---------

Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
2026-03-10 10:04:42 +00:00
88a92a0483 feat: support for flux framework as hpc manager (#3188)
* feat: support for flux framework as hpc manager
Flux supports the majority of MPI flavors/variants, and can be used
to bootstrap MPI as a plugin. It adds other features for scheduling
and topology that can be used for simulations and ai/ml jobs.
This changeset adds the plugin implementation, including the
plugin module, tests, and an example with a small README to
serve as documentation for the time being.

Signed-off-by: vsoch <[email protected]>

* review: slim down use of annotations

We still need to put the original command in an annotation
to retieve later, but others can be re-derived from the
environment

Signed-off-by: vsoch <[email protected]>

* review: clean up tasks and remove extra FluxPolicySource

Signed-off-by: vsoch <[email protected]>

* review: ensure to unset trainer args

Signed-off-by: vsoch <[email protected]>

* review: better support for nproc per node and nodes

Signed-off-by: vsoch <[email protected]>

* review: add constants types for flux

Signed-off-by: vsoch <[email protected]>

* review: add gpu specification from resources

Signed-off-by: vsoch <[email protected]>

* Fix TrainJob NumProcPerNode

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove elasticPolicy from kubebuilder:validation

Signed-off-by: Andrey Velichkevich <[email protected]>

* chore: use go embed for long string templates flux plugin

Signed-off-by: vsoch <[email protected]>

* cleanup: removed un-used annotation

Signed-off-by: vsoch <[email protected]>

---------

Signed-off-by: vsoch <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>
Co-authored-by: vsoch <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
2026-03-10 03:19:05 +00:00
Andrey VelichkevichandGitHub 63d928468c feat(api): BREAKING CHANGE: Remove numProcPerNode from Torch API (#3239)
* feat(api): BREAKING CHANGE: Remove numProcPerNode from Torch API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove trainer_v1alpha1_torch_elastic_policy.py from Py models

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix numNodes validation in torchtune

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2026-03-01 06:53:03 +00:00
Andrey VelichkevichandGitHub 60004df3d3 feat(api): BREAKING CHANGE – Remove ElasticPolicy API (#3235)
Signed-off-by: Andrey Velichkevich <[email protected]>
2026-02-24 20:58:47 +00:00
Krishna GuptaandGitHub a671d3d82c fix: Enforce single ML policy constraint with CEL validation for Torch, MPI, and JAX (#3225)
* fix: enforce single ML policy constraint with CEL validation

Signed-off-by: krishna-kg732 <[email protected]>

* added plainML fallback test case

Signed-off-by: krishna-kg732 <[email protected]>

* added autogenerated files

Signed-off-by: krishna-kg732 <[email protected]>

* added integration tests

Signed-off-by: krishna-kg732 <[email protected]>

* bumped the version in charts to fix ci

Signed-off-by: krishna-kg732 <[email protected]>

* added autogenerated file

Signed-off-by: krishna-kg732 <[email protected]>

* chore: bump version

Signed-off-by: krishna-kg732 <[email protected]>

---------

Signed-off-by: krishna-kg732 <[email protected]>
2026-02-24 18:30:48 +00:00
Tomas TormoandGitHub bb945b03dd feat: Add the manager field to the podTemplateOverride object (#3020)
Signed-off-by: Tomas Tormo <[email protected]>
2026-02-16 23:54:28 +00:00
Tomas TormoandGitHub 62812f9ce3 feat(runtimes): Add JAX training runtime (#3151)
Signed-off-by: mahdikhashan <[email protected]>
Signed-off-by: Tomas Tormo <[email protected]>
2026-02-06 18:15:22 +00:00
Shashank SinghandGitHub db4330d386 fix: fix resourcePerNode override not applied with Volcano scheduler (#2982)
* volcano integeration for resourcePerNode override in trainjob

Signed-off-by: sksingh2005 <[email protected]>

* Fixed the issue where PodGroup resources weren't scaling with spec.trainer.resourcesPerNode when specified on a TrainJob

Signed-off-by: sksingh2005 <[email protected]>

* limits removed

Signed-off-by: sksingh2005 <[email protected]>

* maintains consistency across all MLPolicy plugins

Signed-off-by: sksingh2005 <[email protected]>

* Fix pre-commit formatting

Signed-off-by: sksingh2005 <[email protected]>

* Unit test updates for mpi and torch pluggins

Signed-off-by: sksingh2005 <[email protected]>

* framework unit test prblm solved

Signed-off-by: sksingh2005 <[email protected]>

* fix:All MLPolicy Plugins update SinglePodRequests from TrainJob.spec.trainer.resourcesPerNode

Signed-off-by: sksingh2005 <[email protected]>

* Refactor ResourcesPerNode handling to use PodRequest helper for correct scaling with init/sidecar containers

Signed-off-by: sksingh2005 <[email protected]>

* refactor: simplify ResourcesPerNode by modifying jobSetTemplateSpec directly

Signed-off-by: sksingh2005 <[email protected]>

* test: modify TrainJob resources in coscheduling test to verify node container override

Signed-off-by: sksingh2005 <[email protected]>

* Refactor: remove duplicate resource logic from plugins

Signed-off-by: sksingh2005 <[email protected]>

---------

Signed-off-by: sksingh2005 <[email protected]>
2026-01-18 21:12:22 +00:00
Sanskar GurdasaniandGitHub b5532e5e33 feat(api): Add securityContext support to PodTemplateSpecOverride in TrainJob (#3066)
Signed-off-by: Sanskarzz <[email protected]>
2026-01-15 15:30:18 +00:00
Antonin StefanuttiandGitHub 4c5fb81f42 fix(operator): Prevent JobSet recreation when its TTL has expired (#3013)
Signed-off-by: Antonin Stefanutti <[email protected]>
2026-01-13 17:52:52 +00:00
Antonin StefanuttiandGitHub 9519b9960f fix(ci): Fix kube-api-linter install (#3023)
Signed-off-by: Antonin Stefanutti <[email protected]>
2025-12-09 01:46:39 +00:00
Antonin StefanuttiandGitHub 0f9825e89e chore(operator): Use SSA throughout runtime framework (#2877)
* chore(operator): Use SSA throughout runtime framework

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix lint error

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update go.mod file

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
2025-10-31 11:43:12 +00:00
576e0896b7 feat(runtimes): add support for launcher resource allocation in MPI jobs (#2653)
* feat(runtime): add support for launcher resource allocation in MPI jobs

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add unit tests

Signed-off-by: Andrey Velichkevich <[email protected]>

* Set numProcPerNode for MPI plugin

Signed-off-by: Andrey Velichkevich <[email protected]>

* Move util func to runtime package

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix torchtune plugin

Signed-off-by: Andrey Velichkevich <[email protected]>

* Inline if for GPU check

Signed-off-by: Andrey Velichkevich <[email protected]>

* Assign container resources once

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add todo for test wrappers

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
2025-10-20 13:26:03 +00:00
Antonin StefanuttiandGitHub 5de58d1d9e fix(api): Keep mpiImplementation field a pointer (#2897)
* fix(api): Keep mpiImplementation field a pointer

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix integration test

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
2025-10-17 16:08:00 +00:00
Antonin StefanuttiandGitHub 7755220575 chore(ci): Enable Kubernetes API Linter (#2858)
* chore(ci): Enable Kubernetes API Linter

Signed-off-by: Antonin Stefanutti <[email protected]>

* lint files

Signed-off-by: Antonin Stefanutti <[email protected]>

* Generate

Signed-off-by: Antonin Stefanutti <[email protected]>

* Review

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix golangci-lint target

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix commands max length

Signed-off-by: Antonin Stefanutti <[email protected]>

* Re: Fix golangci-lint target

Signed-off-by: Antonin Stefanutti <[email protected]>

* Enable the integers linter

Signed-off-by: Antonin Stefanutti <[email protected]>

* Disable maxlength linter

Signed-off-by: Antonin Stefanutti <[email protected]>

* make generate

Signed-off-by: Antonin Stefanutti <[email protected]>

* Make fields optional

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix Suspended field in JobStatus

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
2025-10-17 14:40:59 +00:00
ec48455076 feat(runtimes): implement clusterTrainingRuntime deprecation process (#2791)
* constants: add deprecation label constants for runtimes

Signed-off-by: Tarun Duhan <[email protected]>

* webhooks: warn on creation of deprecated ClusterTrainingRuntime

Signed-off-by: Tarun Duhan <[email protected]>

* runtime: warn when TrainJob references deprecated ClusterTrainingRuntime

Signed-off-by: Tarun Duhan <[email protected]>

* tests: add unit test coverage for deprecation warnings

Signed-off-by: Tarun Duhan <[email protected]>

* webhooks/runtime: update deprecation warning wording and tests

Signed-off-by: Tarun Duhan <[email protected]>

* constants/webhooks/runtime: centralize runtime policy URL and reference constant

Signed-off-by: Tarun Duhan <[email protected]>

* tests(integration/webhooks): add create case for deprecated ClusterTrainingRuntime

Create ClusterTrainingRuntime with deprecated label; assert creation succeeds (warning is non-blocking)

Signed-off-by: Tarun Duhan <[email protected]>

* tests(integration/webhooks): add TrainJob create with deprecated ClusterTrainingRuntime

Mark ClusterTrainingRuntime deprecated; create TrainJob referencing it; assert creation succeeds (warning is non-blocking)

Signed-off-by: Tarun Duhan <[email protected]>

* tests/runtime: cover deprecation warning in ClusterTrainingRuntime.ValidateObjects

Signed-off-by: Tarun Duhan <[email protected]>

* tests(integration): remove deprecation-warning cases for CTR/TrainJob webhooks

Signed-off-by: Tarun Duhan <[email protected]>

* feat(runtimes): switch to support=deprecated and centralize deprecation check

Signed-off-by: Tarun Duhan <[email protected]>

* tests(webhooks): focus UTs on entrypoints and clean up tests/test data

Signed-off-by: Tarun Duhan <[email protected]>

* Rename package from labels to trainingruntime

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Tarun Duhan <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
2025-10-16 10:50:53 +00:00
615e338bf4 feat(runtimes): Add LoRA/QLoRA/DoRA support in LLM Trainer V2 (#2832)
* feat(runtimes): Add lora recipe and config in torch.plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* chore: add --trainer-use-lora flag and related logics.

Signed-off-by: Electronic-Waste <[email protected]>

* feat(lora): Add support for QLoRA.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lora): fix extra rdzv backend in single device mode.

Signed-off-by: Electronic-Waste <[email protected]>

* test(lora): Add UTs for lora/qlora in trainingruntime.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lora): remove extra quote symbol in lora attn module.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtime): check default GPU resources allocated in CTRs.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(runtime): add getNumProcPerNode.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtime): remove specific vendor for GPU.

Signed-off-by: Electronic-Waste <[email protected]>

* test(runtime): add UTs for lora validation.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): remove extra flags for torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* Update pkg/runtime/framework/plugins/torch/torch.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Shao Wang <[email protected]>

* chore(runtime): add extractGPUCountFromRuntime.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): remove filtering args logics.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): use fallbackNumProcPerNode if GPU is set in runtimes.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(runtimes): refactor getNumProcPerNode logics.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): call getNumProcPerNode only once.

Signed-off-by: Electronic-Waste <[email protected]>

* test(runtime): fix UTs for torch.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): add limits to calculation.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(runtimes): reorg nppNode related logics.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): fix lints.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): update condition that use nppNode in Limits.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(runtimes): add torchtune.go

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(torch): only reset nppNode when using CPU.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(runtimes): remove getNumProcPerNodeFromCPU.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
Signed-off-by: Shao Wang <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
2025-10-15 12:44:09 +00:00
Xigang WangandGitHub 1d01c440b7 feat(api): Add PodTemplateOverrides API into TrainJob (#2785)
Signed-off-by: xigang <[email protected]>
2025-10-09 10:08:14 +00:00
Antonin StefanuttiandGitHub e9586f9c78 feat(api): Sync TrainJob JobsStatus from JobSet ReplicatedJobsStatus (#2802)
* feat(api): Sync TrainJob JobsStatus from JobSet ReplicatedJobsStatus

Signed-off-by: Antonin Stefanutti <[email protected]>

* Add integration tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update e2e tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Remove extra check

Signed-off-by: Antonin Stefanutti <[email protected]>

* Sort JobsStatus in e2e tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix e2e test for MPI job

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fail-fast when multiple terminal condition and JobsStatus plugins exist

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fold TerminalCondition and JobsStatus plugins

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
2025-10-07 13:19:12 +00:00
Tomas TormoandGitHub b1f1a4ab6f fix: Allow multiple podSpec overrides to target the same TargetJob (#2880)
Signed-off-by: Tomas Tormo <[email protected]>
2025-10-07 12:55:13 +00:00
9802f954ab feat: KEP-2437 - PodGroup Creation for Volcano Scheduler (#2729)
* feat: api for volcano scheduling plugin

Signed-off-by: Xinmin Du <[email protected]>

* feat: init volcano-plugin

Signed-off-by: Xinmin Du <[email protected]>

* feat: init test file

Signed-off-by: Xinmin Du <[email protected]>

* feat: register volcano plugin

Signed-off-by: Xinmin Du <[email protected]>

* feat: deal with minTaskMember, minMember, NetworkTopo

Signed-off-by: Xinmin Du <[email protected]>

* fix: calculate of minResource

Signed-off-by: Xinmin Du <[email protected]>

* test: build PodGroup test

Signed-off-by: Xinmin Du <[email protected]>

* refactor: separate to 2 prs(build&handler)

Signed-off-by: Xinmin Du <[email protected]>

* test: add test for new&reconcile_builder

Signed-off-by: Xinmin Du <[email protected]>

* fix: typo

Signed-off-by: Xinmin Du <[email protected]>

* fix: trainer/v2 import

Signed-off-by: Xinmin Du <[email protected]>

* fix: networktopo type

Signed-off-by: Xinmin Du <[email protected]>

* fix: OpenAPI validation errors

Signed-off-by: Xinmin Du <[email protected]>

* fix: remove minTaskMembers

Signed-off-by: Xinmin Du <[email protected]>

* test: test coverage 100%

Signed-off-by: Xinmin Du <[email protected]>

* feat: update apis

Signed-off-by: Xinmin Du <[email protected]>

* feat: replace testify

Signed-off-by: Xinmin Du <[email protected]>

* fix: registry Volcano CRDs to the scheme

Signed-off-by: Xinmin Du <[email protected]>

* fix: add volcano to scheme

Signed-off-by: Xinmin Du <[email protected]>

* fix: fix networktopo schema

Signed-off-by: Xinmin Du <[email protected]>

* fix: add networktopo spec in trainer

Signed-off-by: Xinmin Du <[email protected]>

* fix: unit test

Signed-off-by: Xinmin Du <[email protected]>

* feat: import networkTopo directly

Signed-off-by: Xinmin Du <[email protected]>

* fix: make generate

Signed-off-by: Xinmin Du <[email protected]>

* fix: make generate

Signed-off-by: Xinmin Du <[email protected]>

* fix: golangci-lint

Signed-off-by: Xinmin Du <[email protected]>

* fix: golangci-lint

Signed-off-by: Xinmin Du <[email protected]>

* feat: add volcano installation in integration test

Signed-off-by: Xinmin Du <[email protected]>

* fix: filter volcano api

Signed-off-by: Xinmin Du <[email protected]>

* fix: get volcano.podgroup  with local version

Signed-off-by: Xinmin Du <[email protected]>

* fix: init test env with volcano podgroup installed

Signed-off-by: Xinmin Du <[email protected]>

* fix: check plugin in enforcePodgroupPolicy

Signed-off-by: Xinmin Du <[email protected]>

* fix: group-name label in unit test

Signed-off-by: Xinmin Du <[email protected]>

* fix: ReconcilerBuilders

Signed-off-by: Xinmin Du <[email protected]>

* feat: add PodGroupHandler

Signed-off-by: Xinmin Du <[email protected]>

* feat: unit test for handlers

Signed-off-by: Xinmin Du <[email protected]>

* fix: group name annotation

Signed-off-by: Xinmin Du <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: no need to delete RBAC

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/volcano/indexer.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: nil checking for trainjob

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/volcano/volcano.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: make generate

Signed-off-by: Xinmin Du <[email protected]>

* fix: index conflict

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/coscheduling/coscheduling.go

Co-authored-by: Shao Wang <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: update volcano to v1.12.2

Signed-off-by: Xinmin Du <[email protected]>

* feat: re-use indexer

Signed-off-by: Xinmin Du <[email protected]>

* feat: add validate

Signed-off-by: Xinmin Du <[email protected]>

* fix: no scheduler when coscheduling is nil

Signed-off-by: Xinmin Du <[email protected]>

* fix: put group-name in annotations

Signed-off-by: Xinmin Du <[email protected]>

* feat: validate if priorityClass installed

Signed-off-by: Xinmin Du <[email protected]>

* feat: propagate annotations to pod

Signed-off-by: Xinmin Du <[email protected]>

* feat: integration test for volcano

Signed-off-by: Xinmin Du <[email protected]>

* fix: golangci-lint check

Signed-off-by: Xinmin Du <[email protected]>

* feat: use shared indexer

Signed-off-by: Xinmin Du <[email protected]>

* feat: remove indexer to runtime/

Signed-off-by: Xinmin Du <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: append owner reference & missing import

Signed-off-by: Xinmin Du <[email protected]>

* fix: rewrite volcano UT

Signed-off-by: Xinmin Du <[email protected]>

* feat: add copyright

Signed-off-by: Xinmin Du <[email protected]>

* fix: sync RBAC to Helm charts

Signed-off-by: Xinmin Du <[email protected]>

* fix: refactor UTs

Signed-off-by: Xinmin Du <[email protected]>

* fix: test validation separately

Signed-off-by: Xinmin Du <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: refactor TestVolcano

Signed-off-by: Xinmin Du <[email protected]>

* fix: refactor TestValidate

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/volcano/volcano_test.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

---------

Signed-off-by: Xinmin Du <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Shao Wang <[email protected]>
Co-authored-by: Yuki Iwai <[email protected]>
2025-10-06 05:51:10 +00:00
Tomas TormoandGitHub 71ac41935d feat: Add a public function to create runtime info objects (#2837)
Signed-off-by: Tomas Tormo <[email protected]>
2025-09-22 15:47:02 +00:00
AsklvandGitHub 124e3dbcd8 chore(test): add uts for coscheduling plugin. (#2582)
* feat: add uts for coscheduling plugin.

Signed-off-by: Asklv <[email protected]>

* fix: add suspend, multiple podset, min resource case for coscheduling test.

Signed-off-by: Asklv <[email protected]>

* fix: update ut comments for coscheduling.

Signed-off-by: Asklv <[email protected]>

---------

Signed-off-by: Asklv <[email protected]>
2025-09-16 16:46:34 +00:00
Tsubasa NagasawaandGitHub 451c7bf65f chore: merge test cases using PodSpecOverrides into a single case (#2822)
* chore: merge test cases using PodSpecOverrides into a single case

Signed-off-by: Tsubasa Nagasawa <[email protected]>

* chore: merge test cases for container env with PodSpecOverride

Signed-off-by: Tsubasa Nagasawa <[email protected]>

* chore: remove existing test case for container with PodSpecOverride

Signed-off-by: Tsubasa Nagasawa <[email protected]>

---------

Signed-off-by: Tsubasa Nagasawa <[email protected]>
2025-09-09 12:22:37 +00:00
Andrey VelichkevichandGitHub ca158302e7 chore(operator): Bump JobSet to v0.9.0 version (#2821)
* chore(operator): Bump JobSet to v0.9.0 version

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix Go tests

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2025-09-09 06:55:36 +00:00
39e4b08a03 feat: support imagePullSecrets in TrainJob pod spec overrides (#2806)
Signed-off-by: Tsubasa Nagasawa <[email protected]>
Co-authored-by: Antonin Stefanutti <[email protected]>
2025-09-08 12:43:36 +00:00
Tsubasa NagasawaandGitHub 187a5de400 feat: support affinity in TrainJob pod spec overrides (#2796)
Signed-off-by: Tsubasa Nagasawa <[email protected]>
2025-08-28 15:52:22 +00:00
Harshvir PotposeandGitHub bf4c11567e chore: Add unit tests for pkg/apply (#2479)
* add unit tests

Signed-off-by: Harshvir Potpose <[email protected]>

* refactor tests

Signed-off-by: Harshvir Potpose <[email protected]>

* add FromTypedObjWithFields tests

Signed-off-by: Harshvir Potpose <[email protected]>

* keep lines short

Signed-off-by: Harshvir Potpose <[email protected]>

* go mod tidy

Signed-off-by: Harshvir Potpose <[email protected]>

* consolidate UpsertEnvVars func

Signed-off-by: Harshvir Potpose <[email protected]>

* consolidate wantError

Signed-off-by: Harshvir Potpose <[email protected]>

* rm unecessary returns

Signed-off-by: Harshvir Potpose <[email protected]>

* use UpsertEnvVar

Signed-off-by: Harshvir Potpose <[email protected]>

* standerdize interface

Signed-off-by: Harshvir Potpose <[email protected]>

---------

Signed-off-by: Harshvir Potpose <[email protected]>
2025-08-18 10:10:03 +00:00
Yuki IwaiandGitHub 87dbc8390e fix(manifests): Add missing permissions for the RuntimeClass and LimitRange (#2787)
Signed-off-by: Yuki Iwai <[email protected]>
2025-08-13 17:55:12 +00:00
Andrey VelichkevichandGitHub 8655798f02 fix(module): Change Go module name to v2 (#2707)
* fix(module): Change Go module name to v2

Signed-off-by: Andrey Velichkevich <[email protected]>

* Bump x/net to v0.38.0

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2025-07-03 17:46:20 +00:00
Antonin StefanuttiandGitHub 8dc6334ccb feat: Add schedulingGates to PodSpecOverrides (#2700)
* feat: Add schedulingGates to PodSpecOverrides

Signed-off-by: Antonin Stefanutti <[email protected]>

* Change desired job to target job in PodSpecOverrides comments

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
2025-07-03 11:03:20 +00:00
Antonin StefanuttiandGitHub cdc702bf25 feat: Mutable PodSpecOverrides for suspended TrainJob (#2683)
* feat: Mutable PodSpecOverrides for suspended TrainJob

Signed-off-by: Antonin Stefanutti <[email protected]>

* Include @tenzen-y review

Signed-off-by: Antonin Stefanutti <[email protected]>

* Add unit tests

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
2025-07-01 13:57:18 +00:00
Yuki IwaiandGitHub 54b23d2e04 chore: Replace the deprecated intstr.FromInt with intstr.FromInt32 (#2695)
Signed-off-by: Yuki Iwai <[email protected]>
2025-06-30 16:41:21 +00:00