hanzo-ml/trainer
1
0
Fork 0
mirror of https://github.com/hanzo-ml/trainer.git synced 2026-09-24 00:31:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,565 Commits 2 Branches 1 Tag
37c8a07a8a839ff4a2987ef045cc11be2e87f2bb
Commit Graph
64 Commits
Author SHA1 Message Date
Antonin StefanuttiGitHubAndrey Velichkevich
 		37c8a07a8a feat(api): Replace PodTemplateOverrides with RuntimePatches API (#3309) 
* feat(api): Replace PodTemplateOverrides with RuntimePatches API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update goal with RuntimePatches API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add diagram for RuntimePatches

Signed-off-by: Andrey Velichkevich <[email protected]>

* feat(api): Replace PodTemplateOverrides with RuntimePatches API

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix kube lint errors

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update Python models

Signed-off-by: Antonin Stefanutti <[email protected]>

* Enable maxlength Kube-API-Linter rule

Signed-off-by: Antonin Stefanutti <[email protected]>

* Restore manger field IsDomainPrefixedPath validation

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix integration tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Apply StrategicMergePatch to batchv1.JobTemplateSpec

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Antonin Stefanutti <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
 2026-03-12 16:29:44 +00:00
Krishna GuptaandGitHub ccaf20bc20 feat(runtimes): Add XGBoost runtime(KEP-2598) (#3200) 
* Add XGBoost runtime plugin scaffold and register in framework

Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>

* regenerated assets

Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>

* Add XGBoost plugin unit tests and update framework test registry

Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>

* added :Runtime image , unit tests for :(Resources are not set, Resources are set in TrainJob) ,using the ContainerTrainerPort instead of default port

Signed-off-by: krishna-kg732 <[email protected]>

* minor changes

Signed-off-by: krishna-kg732 <[email protected]>

* added plainML fallback(JAX+XGBOOST)

Signed-off-by: krishna-kg732 <[email protected]>

* test: add XGBoost runtime resource resolution unit tests

- Add test case for resources set in Runtime only
- Add test case for resources set in both Runtime and TrainJob
- Add xgboost_distributed.yaml ClusterTrainingRuntime manifest
- Update kustomization.yaml with xgboost-runtime image

Addresses feedback from PR #3200 for complete resource resolution test coverage.

Signed-off-by: krishna-kg732 <[email protected]>

* added autogenerat3ed files

Signed-off-by: krishna-kg732 <[email protected]>

* CI fixes

Signed-off-by: krishna-kg732 <[email protected]>

* chore: fix ci lint errors

Signed-off-by: krishna-kg732 <[email protected]>

* Add XGBoost integration and E2E tests, fix Validate guard clause and nil safety

Signed-off-by: krishna-kg732 <[email protected]>

* trigger ci

Signed-off-by: krishna-kg732 <[email protected]>

* fix: Add XGBoost support to test wrapper, CEL validation, and CI publish workflow

- Add XGBoostPolicy() method to MLPolicySourceWrapper for integration tests
- Add has(self.xgboost) to MLPolicy CEL mutual-exclusivity validation rule
- Regenerate CRD manifests with updated CEL rule
- Add xgboost-runtime to build-and-push-images CI workflow
- Add XGBoost image build step to E2E setup script (commented out due to Kind bug)

Signed-off-by: krishna-kg732 <[email protected]>

* docs: Add XGBoost distributed training example notebook

Add example notebook demonstrating distributed XGBoost training using
the Kubeflow Trainer SDK and the xgboost-distributed ClusterTrainingRuntime.

The notebook uses the modern XGBoost Collective API with RabitTracker
for distributed coordination and synthetic classification data.

Signed-off-by: krishna-kg732 <[email protected]>

* test: Skip XGBoost E2E test until runtime image is published

Signed-off-by: krishna-kg732 <[email protected]>

* chore : address review comments

Signed-off-by: krishna-kg732 <[email protected]>

* chore:resolve merge conflicts

Signed-off-by: krishna-kg732 <[email protected]>

---------

Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
 2026-03-10 10:04:42 +00:00
Vanessa SochatGitHubvsochAndrey Velichkevich
 		88a92a0483 feat: support for flux framework as hpc manager (#3188) 
* feat: support for flux framework as hpc manager
Flux supports the majority of MPI flavors/variants, and can be used
to bootstrap MPI as a plugin. It adds other features for scheduling
and topology that can be used for simulations and ai/ml jobs.
This changeset adds the plugin implementation, including the
plugin module, tests, and an example with a small README to
serve as documentation for the time being.

Signed-off-by: vsoch <[email protected]>

* review: slim down use of annotations

We still need to put the original command in an annotation
to retieve later, but others can be re-derived from the
environment

Signed-off-by: vsoch <[email protected]>

* review: clean up tasks and remove extra FluxPolicySource

Signed-off-by: vsoch <[email protected]>

* review: ensure to unset trainer args

Signed-off-by: vsoch <[email protected]>

* review: better support for nproc per node and nodes

Signed-off-by: vsoch <[email protected]>

* review: add constants types for flux

Signed-off-by: vsoch <[email protected]>

* review: add gpu specification from resources

Signed-off-by: vsoch <[email protected]>

* Fix TrainJob NumProcPerNode

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove elasticPolicy from kubebuilder:validation

Signed-off-by: Andrey Velichkevich <[email protected]>

* chore: use go embed for long string templates flux plugin

Signed-off-by: vsoch <[email protected]>

* cleanup: removed un-used annotation

Signed-off-by: vsoch <[email protected]>

---------

Signed-off-by: vsoch <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>
Co-authored-by: vsoch <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
 2026-03-10 03:19:05 +00:00
Andrey VelichkevichandGitHub 63d928468c feat(api): BREAKING CHANGE: Remove numProcPerNode from Torch API (#3239) 
* feat(api): BREAKING CHANGE: Remove numProcPerNode from Torch API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove trainer_v1alpha1_torch_elastic_policy.py from Py models

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix numNodes validation in torchtune

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2026-03-01 06:53:03 +00:00
Andrey VelichkevichandGitHub 60004df3d3 feat(api): BREAKING CHANGE – Remove ElasticPolicy API (#3235) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2026-02-24 20:58:47 +00:00
Krishna GuptaandGitHub a671d3d82c fix: Enforce single ML policy constraint with CEL validation for Torch, MPI, and JAX (#3225) 
* fix: enforce single ML policy constraint with CEL validation

Signed-off-by: krishna-kg732 <[email protected]>

* added plainML fallback test case

Signed-off-by: krishna-kg732 <[email protected]>

* added autogenerated files

Signed-off-by: krishna-kg732 <[email protected]>

* added integration tests

Signed-off-by: krishna-kg732 <[email protected]>

* bumped the version in charts to fix ci

Signed-off-by: krishna-kg732 <[email protected]>

* added autogenerated file

Signed-off-by: krishna-kg732 <[email protected]>

* chore: bump version

Signed-off-by: krishna-kg732 <[email protected]>

---------

Signed-off-by: krishna-kg732 <[email protected]>
 2026-02-24 18:30:48 +00:00
Tomas TormoandGitHub bb945b03dd feat: Add the manager field to the podTemplateOverride object (#3020) 
Signed-off-by: Tomas Tormo <[email protected]>
 2026-02-16 23:54:28 +00:00
Tomas TormoandGitHub 62812f9ce3 feat(runtimes): Add JAX training runtime (#3151) 
Signed-off-by: mahdikhashan <[email protected]>
Signed-off-by: Tomas Tormo <[email protected]>
 2026-02-06 18:15:22 +00:00
Shashank SinghandGitHub db4330d386 fix: fix resourcePerNode override not applied with Volcano scheduler (#2982) 
* volcano integeration for resourcePerNode override in trainjob

Signed-off-by: sksingh2005 <[email protected]>

* Fixed the issue where PodGroup resources weren't scaling with spec.trainer.resourcesPerNode when specified on a TrainJob

Signed-off-by: sksingh2005 <[email protected]>

* limits removed

Signed-off-by: sksingh2005 <[email protected]>

* maintains consistency across all MLPolicy plugins

Signed-off-by: sksingh2005 <[email protected]>

* Fix pre-commit formatting

Signed-off-by: sksingh2005 <[email protected]>

* Unit test updates for mpi and torch pluggins

Signed-off-by: sksingh2005 <[email protected]>

* framework unit test prblm solved

Signed-off-by: sksingh2005 <[email protected]>

* fix:All MLPolicy Plugins update SinglePodRequests from TrainJob.spec.trainer.resourcesPerNode

Signed-off-by: sksingh2005 <[email protected]>

* Refactor ResourcesPerNode handling to use PodRequest helper for correct scaling with init/sidecar containers

Signed-off-by: sksingh2005 <[email protected]>

* refactor: simplify ResourcesPerNode by modifying jobSetTemplateSpec directly

Signed-off-by: sksingh2005 <[email protected]>

* test: modify TrainJob resources in coscheduling test to verify node container override

Signed-off-by: sksingh2005 <[email protected]>

* Refactor: remove duplicate resource logic from plugins

Signed-off-by: sksingh2005 <[email protected]>

---------

Signed-off-by: sksingh2005 <[email protected]>
 2026-01-18 21:12:22 +00:00
Antonin StefanuttiandGitHub 4c5fb81f42 fix(operator): Prevent JobSet recreation when its TTL has expired (#3013) 
Signed-off-by: Antonin Stefanutti <[email protected]>
 2026-01-13 17:52:52 +00:00
Antonin StefanuttiandGitHub 9519b9960f fix(ci): Fix kube-api-linter install (#3023) 
Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-12-09 01:46:39 +00:00
Antonin StefanuttiandGitHub 0f9825e89e chore(operator): Use SSA throughout runtime framework (#2877) 
* chore(operator): Use SSA throughout runtime framework

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix lint error

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update go.mod file

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-10-31 11:43:12 +00:00
Krishnaswamy SubramanianGitHubAndrey Velichkevich
 		576e0896b7 feat(runtimes): add support for launcher resource allocation in MPI jobs (#2653) 
* feat(runtime): add support for launcher resource allocation in MPI jobs

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add unit tests

Signed-off-by: Andrey Velichkevich <[email protected]>

* Set numProcPerNode for MPI plugin

Signed-off-by: Andrey Velichkevich <[email protected]>

* Move util func to runtime package

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix torchtune plugin

Signed-off-by: Andrey Velichkevich <[email protected]>

* Inline if for GPU check

Signed-off-by: Andrey Velichkevich <[email protected]>

* Assign container resources once

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add todo for test wrappers

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
 2025-10-20 13:26:03 +00:00
Antonin StefanuttiandGitHub 5de58d1d9e fix(api): Keep mpiImplementation field a pointer (#2897) 
* fix(api): Keep mpiImplementation field a pointer

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix integration test

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-10-17 16:08:00 +00:00
Antonin StefanuttiandGitHub 7755220575 chore(ci): Enable Kubernetes API Linter (#2858) 
* chore(ci): Enable Kubernetes API Linter

Signed-off-by: Antonin Stefanutti <[email protected]>

* lint files

Signed-off-by: Antonin Stefanutti <[email protected]>

* Generate

Signed-off-by: Antonin Stefanutti <[email protected]>

* Review

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix golangci-lint target

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix commands max length

Signed-off-by: Antonin Stefanutti <[email protected]>

* Re: Fix golangci-lint target

Signed-off-by: Antonin Stefanutti <[email protected]>

* Enable the integers linter

Signed-off-by: Antonin Stefanutti <[email protected]>

* Disable maxlength linter

Signed-off-by: Antonin Stefanutti <[email protected]>

* make generate

Signed-off-by: Antonin Stefanutti <[email protected]>

* Make fields optional

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix Suspended field in JobStatus

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-10-17 14:40:59 +00:00
Shao WangGitHubAndrey Velichkevich
 		615e338bf4 feat(runtimes): Add LoRA/QLoRA/DoRA support in LLM Trainer V2 (#2832) 
* feat(runtimes): Add lora recipe and config in torch.plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* chore: add --trainer-use-lora flag and related logics.

Signed-off-by: Electronic-Waste <[email protected]>

* feat(lora): Add support for QLoRA.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lora): fix extra rdzv backend in single device mode.

Signed-off-by: Electronic-Waste <[email protected]>

* test(lora): Add UTs for lora/qlora in trainingruntime.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lora): remove extra quote symbol in lora attn module.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtime): check default GPU resources allocated in CTRs.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(runtime): add getNumProcPerNode.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtime): remove specific vendor for GPU.

Signed-off-by: Electronic-Waste <[email protected]>

* test(runtime): add UTs for lora validation.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): remove extra flags for torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* Update pkg/runtime/framework/plugins/torch/torch.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Shao Wang <[email protected]>

* chore(runtime): add extractGPUCountFromRuntime.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): remove filtering args logics.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): use fallbackNumProcPerNode if GPU is set in runtimes.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(runtimes): refactor getNumProcPerNode logics.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): call getNumProcPerNode only once.

Signed-off-by: Electronic-Waste <[email protected]>

* test(runtime): fix UTs for torch.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): add limits to calculation.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(runtimes): reorg nppNode related logics.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): fix lints.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): update condition that use nppNode in Limits.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(runtimes): add torchtune.go

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(torch): only reset nppNode when using CPU.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(runtimes): remove getNumProcPerNodeFromCPU.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
Signed-off-by: Shao Wang <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
 2025-10-15 12:44:09 +00:00
Xigang WangandGitHub 1d01c440b7 feat(api): Add PodTemplateOverrides API into TrainJob (#2785) 
Signed-off-by: xigang <[email protected]>
 2025-10-09 10:08:14 +00:00
Antonin StefanuttiandGitHub e9586f9c78 feat(api): Sync TrainJob JobsStatus from JobSet ReplicatedJobsStatus (#2802) 
* feat(api): Sync TrainJob JobsStatus from JobSet ReplicatedJobsStatus

Signed-off-by: Antonin Stefanutti <[email protected]>

* Add integration tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update e2e tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Remove extra check

Signed-off-by: Antonin Stefanutti <[email protected]>

* Sort JobsStatus in e2e tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix e2e test for MPI job

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fail-fast when multiple terminal condition and JobsStatus plugins exist

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fold TerminalCondition and JobsStatus plugins

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-10-07 13:19:12 +00:00
Tomas TormoandGitHub b1f1a4ab6f fix: Allow multiple podSpec overrides to target the same TargetJob (#2880) 
Signed-off-by: Tomas Tormo <[email protected]>
 2025-10-07 12:55:13 +00:00
Du XinminGitHubAndrey VelichkevichShao WangYuki Iwai
 		9802f954ab feat: KEP-2437 - PodGroup Creation for Volcano Scheduler (#2729) 
* feat: api for volcano scheduling plugin

Signed-off-by: Xinmin Du <[email protected]>

* feat: init volcano-plugin

Signed-off-by: Xinmin Du <[email protected]>

* feat: init test file

Signed-off-by: Xinmin Du <[email protected]>

* feat: register volcano plugin

Signed-off-by: Xinmin Du <[email protected]>

* feat: deal with minTaskMember, minMember, NetworkTopo

Signed-off-by: Xinmin Du <[email protected]>

* fix: calculate of minResource

Signed-off-by: Xinmin Du <[email protected]>

* test: build PodGroup test

Signed-off-by: Xinmin Du <[email protected]>

* refactor: separate to 2 prs（build&handler）

Signed-off-by: Xinmin Du <[email protected]>

* test: add test for new&reconcile_builder

Signed-off-by: Xinmin Du <[email protected]>

* fix: typo

Signed-off-by: Xinmin Du <[email protected]>

* fix: trainer/v2 import

Signed-off-by: Xinmin Du <[email protected]>

* fix: networktopo type

Signed-off-by: Xinmin Du <[email protected]>

* fix: OpenAPI validation errors

Signed-off-by: Xinmin Du <[email protected]>

* fix: remove minTaskMembers

Signed-off-by: Xinmin Du <[email protected]>

* test: test coverage 100%

Signed-off-by: Xinmin Du <[email protected]>

* feat: update apis

Signed-off-by: Xinmin Du <[email protected]>

* feat: replace testify

Signed-off-by: Xinmin Du <[email protected]>

* fix: registry Volcano CRDs to the scheme

Signed-off-by: Xinmin Du <[email protected]>

* fix: add volcano to scheme

Signed-off-by: Xinmin Du <[email protected]>

* fix: fix networktopo schema

Signed-off-by: Xinmin Du <[email protected]>

* fix: add networktopo spec in trainer

Signed-off-by: Xinmin Du <[email protected]>

* fix: unit test

Signed-off-by: Xinmin Du <[email protected]>

* feat: import networkTopo directly

Signed-off-by: Xinmin Du <[email protected]>

* fix: make generate

Signed-off-by: Xinmin Du <[email protected]>

* fix: make generate

Signed-off-by: Xinmin Du <[email protected]>

* fix: golangci-lint

Signed-off-by: Xinmin Du <[email protected]>

* fix: golangci-lint

Signed-off-by: Xinmin Du <[email protected]>

* feat: add volcano installation in integration test

Signed-off-by: Xinmin Du <[email protected]>

* fix: filter volcano api

Signed-off-by: Xinmin Du <[email protected]>

* fix: get volcano.podgroup  with local version

Signed-off-by: Xinmin Du <[email protected]>

* fix: init test env with volcano podgroup installed

Signed-off-by: Xinmin Du <[email protected]>

* fix: check plugin in enforcePodgroupPolicy

Signed-off-by: Xinmin Du <[email protected]>

* fix: group-name label in unit test

Signed-off-by: Xinmin Du <[email protected]>

* fix: ReconcilerBuilders

Signed-off-by: Xinmin Du <[email protected]>

* feat: add PodGroupHandler

Signed-off-by: Xinmin Du <[email protected]>

* feat: unit test for handlers

Signed-off-by: Xinmin Du <[email protected]>

* fix: group name annotation

Signed-off-by: Xinmin Du <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: no need to delete RBAC

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/volcano/indexer.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: nil checking for trainjob

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/volcano/volcano.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: make generate

Signed-off-by: Xinmin Du <[email protected]>

* fix: index conflict

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/coscheduling/coscheduling.go

Co-authored-by: Shao Wang <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: update volcano to v1.12.2

Signed-off-by: Xinmin Du <[email protected]>

* feat: re-use indexer

Signed-off-by: Xinmin Du <[email protected]>

* feat: add validate

Signed-off-by: Xinmin Du <[email protected]>

* fix: no scheduler when coscheduling is nil

Signed-off-by: Xinmin Du <[email protected]>

* fix: put group-name in annotations

Signed-off-by: Xinmin Du <[email protected]>

* feat: validate if priorityClass installed

Signed-off-by: Xinmin Du <[email protected]>

* feat: propagate annotations to pod

Signed-off-by: Xinmin Du <[email protected]>

* feat: integration test for volcano

Signed-off-by: Xinmin Du <[email protected]>

* fix: golangci-lint check

Signed-off-by: Xinmin Du <[email protected]>

* feat: use shared indexer

Signed-off-by: Xinmin Du <[email protected]>

* feat: remove indexer to runtime/

Signed-off-by: Xinmin Du <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: append owner reference & missing import

Signed-off-by: Xinmin Du <[email protected]>

* fix: rewrite volcano UT

Signed-off-by: Xinmin Du <[email protected]>

* feat: add copyright

Signed-off-by: Xinmin Du <[email protected]>

* fix: sync RBAC to Helm charts

Signed-off-by: Xinmin Du <[email protected]>

* fix: refactor UTs

Signed-off-by: Xinmin Du <[email protected]>

* fix: test validation separately

Signed-off-by: Xinmin Du <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: refactor TestVolcano

Signed-off-by: Xinmin Du <[email protected]>

* fix: refactor TestValidate

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/volcano/volcano_test.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

---------

Signed-off-by: Xinmin Du <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Shao Wang <[email protected]>
Co-authored-by: Yuki Iwai <[email protected]>
 2025-10-06 05:51:10 +00:00
AsklvandGitHub 124e3dbcd8 chore(test): add uts for coscheduling plugin. (#2582) 
* feat: add uts for coscheduling plugin.

Signed-off-by: Asklv <[email protected]>

* fix: add suspend, multiple podset, min resource case for coscheduling test.

Signed-off-by: Asklv <[email protected]>

* fix: update ut comments for coscheduling.

Signed-off-by: Asklv <[email protected]>

---------

Signed-off-by: Asklv <[email protected]>
 2025-09-16 16:46:34 +00:00
Andrey VelichkevichandGitHub ca158302e7 chore(operator): Bump JobSet to v0.9.0 version (#2821) 
* chore(operator): Bump JobSet to v0.9.0 version

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix Go tests

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2025-09-09 06:55:36 +00:00
Harshvir PotposeandGitHub bf4c11567e chore: Add unit tests for pkg/apply (#2479) 
* add unit tests

Signed-off-by: Harshvir Potpose <[email protected]>

* refactor tests

Signed-off-by: Harshvir Potpose <[email protected]>

* add FromTypedObjWithFields tests

Signed-off-by: Harshvir Potpose <[email protected]>

* keep lines short

Signed-off-by: Harshvir Potpose <[email protected]>

* go mod tidy

Signed-off-by: Harshvir Potpose <[email protected]>

* consolidate UpsertEnvVars func

Signed-off-by: Harshvir Potpose <[email protected]>

* consolidate wantError

Signed-off-by: Harshvir Potpose <[email protected]>

* rm unecessary returns

Signed-off-by: Harshvir Potpose <[email protected]>

* use UpsertEnvVar

Signed-off-by: Harshvir Potpose <[email protected]>

* standerdize interface

Signed-off-by: Harshvir Potpose <[email protected]>

---------

Signed-off-by: Harshvir Potpose <[email protected]>
 2025-08-18 10:10:03 +00:00
Yuki IwaiandGitHub 87dbc8390e fix(manifests): Add missing permissions for the RuntimeClass and LimitRange (#2787) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-08-13 17:55:12 +00:00
Andrey VelichkevichandGitHub 8655798f02 fix(module): Change Go module name to v2 (#2707) 
* fix(module): Change Go module name to v2

Signed-off-by: Andrey Velichkevich <[email protected]>

* Bump x/net to v0.38.0

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2025-07-03 17:46:20 +00:00
Antonin StefanuttiandGitHub cdc702bf25 feat: Mutable PodSpecOverrides for suspended TrainJob (#2683) 
* feat: Mutable PodSpecOverrides for suspended TrainJob

Signed-off-by: Antonin Stefanutti <[email protected]>

* Include @tenzen-y review

Signed-off-by: Antonin Stefanutti <[email protected]>

* Add unit tests

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-07-01 13:57:18 +00:00
Yuki IwaiandGitHub 54b23d2e04 chore: Replace the deprecated intstr.FromInt with intstr.FromInt32 (#2695) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-06-30 16:41:21 +00:00
Yuki IwaiandGitHub f99f29e732 chore: Remove the vendor specific parameters (#2691) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-06-30 14:59:21 +00:00
Shao WangandGitHub c6d3ee01fc fix(plugins): Fix some errors in torchtune mutation process. (#2675) 
* fix(plugins): fix errors in trainer command mutation of torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugins): remove config file format suffix.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): update UTs.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(initializer): Update the workspace of dataset/model initializer.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugins): parse nproc_per_node from GPU resource.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(torchtune): Add bitsandbytes dependency in requirements.txt

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(torchtune): Remove unnecessary num_proc_per_node calculation.

Signed-off-by: Electronic-Waste <[email protected]>

* test(torch): Update invalid parameters.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
 2025-06-30 14:56:21 +00:00
Shao WangandGitHub 3c3bf09c00 KEP-2401: Support loading local LLMs (#2644) 
* feat(torch): Support loading local models.

Signed-off-by: Electronic-Waste <[email protected]>

* test(torch): Update UTs for loading local model.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(manifest): Update loading local LLMs command.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(manifest): Update pvc & model dir in manifests.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): update UTs.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(torchtune): Update checkpoint dir and tokenzier path in torch plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* chore: tidy up code.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(torchtune): Remove CheckpointDir.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(torchtune): remove DefaultTorchTuneModel.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(torchtune): add extractOverridesFromRuntime func.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(test): update UTs in pkg/runtime/core.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(manifest): update output dir.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
 2025-06-06 18:12:16 +00:00
Andrey VelichkevichGitHubShao WangAntonin StefanuttiAntonin Stefanutti
 		434d953004 feat(controller): Implement PodSpecOverride API (#2614) 
* feat(controller): Implement PodSpecOverride API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update API comments

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix InitContainers override

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix generate

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update pkg/apis/trainer/v1alpha1/trainjob_types.go

Co-authored-by: Shao Wang <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>

* Change API to targetJob and add validation

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add unit test for the JobSet builder

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add integration tests

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update pkg/apis/trainer/v1alpha1/trainjob_types.go

Co-authored-by: Shao Wang <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>

* Update pkg/apis/trainer/v1alpha1/trainjob_types.go

Co-authored-by: Shao Wang <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>

* Update pkg/apis/trainer/v1alpha1/trainjob_types.go

Co-authored-by: Antonin Stefanutti <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>

* Merge PodSpecOverrides before the build phase

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix make generate

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix PodSpec SMT patch unmarshalling into JobSet template

Signed-off-by: Antonin Stefanutti <[email protected]>

* Add UT for volumes PodSpecOverrides

Signed-off-by: Antonin Stefanutti <[email protected]>

* Add UT for tolerations PodSpecOverrides

Signed-off-by: Antonin Stefanutti <[email protected]>

* Add UT for node selector PodSpecOverrides

Signed-off-by: Antonin Stefanutti <[email protected]>

* Revert back to using TargetJobs []PodSpecOverrideTargetJob

Signed-off-by: Antonin Stefanutti <[email protected]>

* Run make fmt

Signed-off-by: Antonin Stefanutti <[email protected]>

* Add target jobs unicity validation for PodSpecOverrides

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update with code review from @tenzen-y

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Antonin Stefanutti <[email protected]>
Co-authored-by: Shao Wang <[email protected]>
Co-authored-by: Antonin Stefanutti <[email protected]>
Co-authored-by: Antonin Stefanutti <[email protected]>
 2025-06-06 17:04:18 +00:00
Yuki IwaiandGitHub f5a2d0f250 Implement trainer.kubeflow.org/resource-in-use finalizer mechanism to TrainingRuntime (#2608) 
* Implement trainer.kubeflow.org/resource-in-use finalizer mechanism to TrainingRuntime

Signed-off-by: Yuki Iwai <[email protected]>

# Conflicts:
#	pkg/constants/constants.go

* Reduce permissions

Signed-off-by: Yuki Iwai <[email protected]>

* Dispatch reconciliations even when TrainJob is updated

Signed-off-by: Yuki Iwai <[email protected]>

---------

Signed-off-by: Yuki Iwai <[email protected]>
 2025-05-01 20:17:04 +00:00
Shao WangandGitHub 4493eb8770 KEP-2401: Complement torch plugin to support torchtune config mutation (#2587) 
* chore(plugin): Add torchtune-related constants & update current torch plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(plugin): Add EnforceMLPolicy for torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(plugin): Add UTs in torch plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): fix error in torch plugin UTs.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(plugin): Choose recipe according to numNodes & numProcPerNode & Args.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(sdk): Add PretrainedModel enum type.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(plugin): Add torchtune config arg.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(test): add UT for single-device full fine-tuning with torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(test): Add test for multi-nodes full fine-tuning with torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(test): Update torch validate UTs.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(sdk): remove pretrained model enum type in sdk.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugin): retrieve model name from runtimeRef.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix typo.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugin): make some adjustments according to the review.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(sdk): remove runtime in get_trainer_crd_from_builtin_trainer.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugin): pass PET_ env variables in torch plugin for torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugin): add env validation for torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugin): update comments.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugins): fix the implementation according to the review.

Signed-off-by: Electronic-Waste <[email protected]>

* test(plugins): fix UT error in torch plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* fix: fix UT and e2e tests error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix: remove debug info.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): add args in UTs related to torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): update torchtune related args.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): Add a UT for multi-node mode check in torch plugin.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
 2025-04-29 13:21:03 +00:00
Harshal MalaniandGitHub e75a4a7e33 test(coscheduling) : add test coverage for coscheduling indexer (#2604) 
* test(coscheduling) add test coverage for coscheduling indexer

Signed-off-by: Harshal292004 <[email protected]>

* update import order

Signed-off-by: Harshal292004 <[email protected]>

* updated according to review

Signed-off-by: Harshal292004 <[email protected]>

* changed DataIntailizer and ModelIntailizer with containerd

Signed-off-by: Harshal292004 <[email protected]>

---------

Signed-off-by: Harshal292004 <[email protected]>
 2025-04-26 13:35:59 +00:00
Harshal MalaniandGitHub c5788c38ca test(runtime): add UT for jobset runtime valid function. (#2562) 
* test(runtime): add UT for jobset runtime valid function.

Signed-off-by: Harshal292004 <[email protected]>

* Update the test cases

Signed-off-by: Harshal292004 <[email protected]>

* update the test cases

Signed-off-by: Harshal292004 <[email protected]>

* format imports

Signed-off-by: Harshal292004 <[email protected]>

* Updated as suggested

Signed-off-by: Harshal292004 <[email protected]>

---------

Signed-off-by: Harshal292004 <[email protected]>
 2025-04-15 14:10:23 +00:00
Harshal MalaniGitHubYuki Iwai
 		46bfd7bf91 [feature]:add validations for MPIRuntime with RunLauncherAsNode (#2551) 
* [feature]:add validatioons for MPIRuntime with RunLauncherAsNode

Signed-off-by: Harshal292004 <[email protected]>

* fix:changed "launcher" and "node" with constants

Signed-off-by: Harshal292004 <[email protected]>

* fix:removed the constants with string values

Signed-off-by: Harshal292004 <[email protected]>

* fix: add nil checks for before derefrencing pointers

Signed-off-by: Harshal292004 <[email protected]>

* fix:add check for trainjob being nil

Signed-off-by: Harshal292004 <[email protected]>

* updae with suggestions

Signed-off-by: Harshal292004 <[email protected]>

* update with suggestions

Signed-off-by: Harshal292004 <[email protected]>

* Update pkg/runtime/framework/plugins/mpi/mpi.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Harshal Malani <[email protected]>

* Update pkg/runtime/framework/plugins/mpi/mpi.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Harshal Malani <[email protected]>

* fix merge conflicts

Signed-off-by: Harshal292004 <[email protected]>

* test(runtime): add UT for mpi runtime validate function

Signed-off-by: Harshal292004 <[email protected]>

* Update mpi and mpi_test according to suggestions

Signed-off-by: Harshal292004 <[email protected]>

* Update mpi_test

Signed-off-by: Harshal292004 <[email protected]>

---------

Signed-off-by: Harshal292004 <[email protected]>
Signed-off-by: Harshal Malani <[email protected]>
Co-authored-by: Yuki Iwai <[email protected]>
 2025-03-28 15:46:29 +00:00
Yuki IwaiandGitHub 311e72b410 Fix MPI Test runnable errors (#2570) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-27 18:05:23 +00:00
Yuki IwaiandGitHub 7d9593a2e0 Implement CustomValidation UT for MPI plugin (#2555) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-27 12:11:23 +00:00
AsklvandGitHub ccbd58e4ee test(runtime): add UT for torch runtime valid function. (#2560) 
Signed-off-by: Asklv <[email protected]>
 2025-03-26 07:47:04 +00:00
Akshay ChitneniGitHubAkshay Chitneni
 		d6a2b00e0c Fix issue with fetching clustertrainingruntime for validations (#2564) 
Signed-off-by: Akshay Chitneni <[email protected]>
Signed-off-by: Akshay Chitneni <[email protected]>
Co-authored-by: Akshay Chitneni <[email protected]>
 2025-03-24 15:14:06 +00:00
AsklvandGitHub 07ce28219c feat(runtime): remove needless Launcher chainer. (#2558) 
Signed-off-by: Asklv <[email protected]>
 2025-03-24 11:14:05 +00:00
Yuki IwaiandGitHub 3dd0fb0b82 Retrieve JobSetSpec from runtime.Info in CustomValidations (#2557) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-24 11:05:06 +00:00
Andrey VelichkevichandGitHub eb1ee9d0f4 feat(sdk): Support MPI-based TrainJobs (#2545) 
* feat(sdk): Support MPI-based TrainJobs

Signed-off-by: Andrey Velichkevich <[email protected]>

* Refactor list_runtimes

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix example

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add Runtime Trainer object

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update for new Runtime object

Signed-off-by: Andrey Velichkevich <[email protected]>

* Implement get_runtime API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix Torch example

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove un-unsed consts

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update func args

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update SDK constants

Signed-off-by: Andrey Velichkevich <[email protected]>

* Change to 16Gi

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix container name for MPI

Signed-off-by: Andrey Velichkevich <[email protected]>

* Keep launcher container for MPI

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2025-03-20 18:08:02 +00:00
Yuki IwaiandGitHub 50d6be8324 Construct Trainer based on trainer.kubeflow.org/trainjob-ancestor-step label (#2548) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-20 14:01:57 +00:00
Yuki IwaiandGitHub 64404f6565 Store the TrainingRuntime numNodes as runtime.Info.PodSet.Count (#2539) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-18 15:55:44 +00:00
Yuki IwaiandGitHub ed1b434f99 chore: Enable GCI for golangci-lint (#2540) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-18 14:47:21 +00:00
Andrey VelichkevichandGitHub 239b27d311 feat(controller): Refactor the Initializer APIs of TrainJob (#2523) 
* feat(controller): Refactor the Initializer APIs of TrainJob

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix go unit test

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix integration test

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2025-03-17 07:15:51 +00:00
Yuki IwaiandGitHub 38e16d718d Migrate InfoOptions.podSpecReplias and info.Scheduler.TotalRequests to info.TemplateSpec.PodSet (#2524) 
Signed-off-by: Yuki Iwai <[email protected]>

# Conflicts:
#	pkg/runtime/core/trainingruntime.go
#	pkg/runtime/runtime.go
 2025-03-16 17:09:49 +00:00
Akshay ChitneniGitHubAkshay Chitneni
 		a589903c24 KEP-2170: Adding validation webhook for v2 trainjob (#2307) 
Signed-off-by: Akshay Chitneni <[email protected]>
Co-authored-by: Akshay Chitneni <[email protected]>
 2025-03-16 03:41:51 +00:00
Yuki IwaiandGitHub f1a0d6178e Migrate Info.Trainer to Info.TemplateSpec.PodSet (#2520) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-15 03:34:11 +00:00