hanzo-ml/trainer
1
0
Fork 0
mirror of https://github.com/hanzo-ml/trainer.git synced 2026-09-24 00:31:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,765 Commits 2 Branches 1 Tag
master
Commit Graph
51 Commits
Author SHA1 Message Date
Dmytro HryshchenkoGitHubHanzo Dev
 		a6e862ee0c fix(runtimes): set defaultMode on MPI SSH Secret volume (#3649) 
* fix(runtimes): set file modes on MPI SSH secret volume

Set per-item file modes on the MPI SSH auth Secret volume to prevent
the private key from being world-readable inside training pods.

Private key uses 0o640 (not 0o600) because the builtin
deepspeed-distributed and mlx-distributed runtimes run as
runAsUser: 1000 while Secret volumes mount as root:root.

A volume-level DefaultMode of 0o400 is added as defense-in-depth
for any future items added without an explicit per-item mode.

Fixes #3262

Signed-off-by: Dmytro Hryshchenko <[email protected]>

* docs: update pkg/constants/constants.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Dmytro Hryshchenko <[email protected]>

---------

Signed-off-by: Dmytro Hryshchenko <[email protected]>
Signed-off-by: Dmytro Hryshchenko <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>

Co-authored-by: Hanzo Dev <[email protected]>
 2026-06-24 10:40:35 +00:00
Tariq HasanandGitHub 9183dfd24f chore(scripts): add copyright boilerplate verification (#3359) 
* chore(scripts): add copyright boilerplate verification system

Signed-off-by: tariq-hasan <[email protected]>

* chore(ci): add boilerplate verification to CI pipeline

Signed-off-by: tariq-hasan <[email protected]>

* chore: regenerate files with year-less copyright headers

Signed-off-by: tariq-hasan <[email protected]>

* refactor(boilerplate): simplify verification logic

Signed-off-by: tariq-hasan <[email protected]>

* chore(boilerplate): add missing copyright headers

Signed-off-by: tariq-hasan <[email protected]>

* ci(boilerplate): move verifier to code-quality workflow and run unit tests

Signed-off-by: tariq-hasan <[email protected]>

* feat(boilerplate): accept license headers with any year and remove year validation logic

Signed-off-by: tariq-hasan <[email protected]>

* feat(boilerplate): add copyright headers to generated Python API

Signed-off-by: tariq-hasan <[email protected]>

* refactor(boilerplate): use block-comment header for generated Go files

Signed-off-by: tariq-hasan <[email protected]>

* feat(boilerplate): verify copyright headers in Rust files

Signed-off-by: tariq-hasan <[email protected]>

---------

Signed-off-by: tariq-hasan <[email protected]>
 2026-05-29 10:50:56 +00:00
Krish GargandGitHub 4f8920c902 feat(api): Add terminationGracePeriodSeconds to PodSpecPatch in TrainJob (#3324) 
Adds terminationGracePeriodSeconds field to PodSpecPatch so users can
configure pod termination grace period per TrainJob via RuntimePatches.

This is needed for distributed training with PyTorch Elastic (torchrun)
where large models (70B+ parameters) require more than the default 30s
to complete JIT checkpointing before SIGKILL on node drain or TrainJob
pause.

No changes to merge logic in trainingruntime.go are required since the
existing StrategicMergePatch applied at batchv1.JobTemplateSpec level
already handles this field automatically.

Closes #3285

Signed-off-by: krishdef7 <[email protected]>
 2026-05-26 17:36:09 +00:00
Sameer_YadavandGitHub 3135ce6d89 chore: unit tests for runtime registry and RuntimeInfo error handling (#3168) 
* Add unit tests for runtime registry and RuntimeInfo error handling

Signed-off-by: Sameer_yadav <[email protected]>

* chore: apply formatting fixes

Signed-off-by: Sameer_yadav <[email protected]>

* chore: fix tests for runtime registry and RuntimeInfo error handling

Signed-off-by: Sameer_yadav <[email protected]>

* chore: format code with gofmt

Signed-off-by: Sameer_yadav <[email protected]>

* test: improve registry test robustness

Signed-off-by: Sameer_yadav <[email protected]>

* removed year and changed the test function name

Signed-off-by: Sameer_yadav <[email protected]>

---------

Signed-off-by: Sameer_yadav <[email protected]>
 2026-05-06 11:41:38 +00:00
Tomas TormoandGitHub f47c01b1c1 chore(runtimes): persist runtimes map and expose Runtimes function (#3360) 
Signed-off-by: Tomas Tormo <[email protected]>
 2026-03-19 10:37:26 +00:00
Rob BellandGitHub 78b4a00929 feat: add support for tracking TrainJob progress and training metrics (#3227) 
* feat(api): add progress types to api

Signed-off-by: Rob Bell <[email protected]>

* feat: progress server stub

Signed-off-by: Rob Bell <[email protected]>

* feat: setup tls for server

Signed-off-by: Rob Bell <[email protected]>

* refactor: pull out middleware

Signed-off-by: Rob Bell <[email protected]>

* refactor: remove logger message noise from tests

Signed-off-by: Rob Bell <[email protected]>

* fix: send server error logs to main logger

Signed-off-by: Rob Bell <[email protected]>

* feat: update the trainer status

Signed-off-by: Rob Bell <[email protected]>

* feat: use separate client to avoid impacting main reconciler client-side rate limits

Signed-off-by: Rob Bell <[email protected]>

* feat: configure progress server in helm chart and configmap

Signed-off-by: Rob Bell <[email protected]>

* fix: prevent race condition

Signed-off-by: Rob Bell <[email protected]>

* feat: inject progress runtime config

Signed-off-by: Rob Bell <[email protected]>

* fix: progress client config

Signed-off-by: Rob Bell <[email protected]>

* fix: update generated config

Signed-off-by: Rob Bell <[email protected]>

* feat: validate auth tokens in endpoint

Signed-off-by: Rob Bell <[email protected]>

* test: validate auth tokens in endpoint

Signed-off-by: Rob Bell <[email protected]>

* refactor: tidy tests to use same format as other tests

Signed-off-by: Rob Bell <[email protected]>

* fix: look up the cluster issuer from the service account token

Different clusters can use different issuers so look up the issuer from the trusted in-cluster service account token.

Signed-off-by: Rob Bell <[email protected]>

* refactor: rename authn middleware

Signed-off-by: Rob Bell <[email protected]>

* feat: add authn to status update endpoint

Signed-off-by: Rob Bell <[email protected]>

* refactor: tidy up reading in-cluster service account token

Signed-off-by: Rob Bell <[email protected]>

* fix: rbac permissions

Signed-off-by: Rob Bell <[email protected]>

* fix: better error response

Signed-off-by: Rob Bell <[email protected]>

* refactor: move kubebuilder annotation higher up

Signed-off-by: Rob Bell <[email protected]>

* fix: make lastUpdateTime required

Signed-off-by: Rob Bell <[email protected]>

* chore: fix config tests after rebase

Signed-off-by: Rob Bell <[email protected]>

* test: update config test with new progress server config

Signed-off-by: Rob Bell <[email protected]>

* test: add progress tracking e2e test

Signed-off-by: Rob Bell <[email protected]>

* fix: allow feature gates to be configurable from command line argument

So they can be overridden in e2e tests

Signed-off-by: Rob Bell <[email protected]>

* refactor: reuse shared progress StatusUrl

Signed-off-by: Rob Bell <[email protected]>

* test: tidy up tests

Signed-off-by: Rob Bell <[email protected]>

* refactor: allow plugins to use config

Signed-off-by: Rob Bell <[email protected]>

* refactor: centralise shared constants

Signed-off-by: Rob Bell <[email protected]>

* fix: disable http2 in progress server tls opts if disabled by config

Signed-off-by: Rob Bell <[email protected]>

* fix: minor improvements suggested by copilot

Signed-off-by: Rob Bell <[email protected]>

* chore: rename progress server => status server

Signed-off-by: Rob Bell <[email protected]>

* chore: update type doc

Signed-off-by: Rob Bell <[email protected]>

* chore: update name of feature gate

Signed-off-by: Rob Bell <[email protected]>

* fix: after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: fix typo

Signed-off-by: Rob Bell <[email protected]>

* chore: rename handler

Signed-off-by: Rob Bell <[email protected]>

* fix: run status server on all replicas

Signed-off-by: Rob Bell <[email protected]>

* fix: deep copy the rest config

Signed-off-by: Rob Bell <[email protected]>

* refactor: use ssa

Signed-off-by: Rob Bell <[email protected]>

* refactor: separate token audience per train job

Signed-off-by: Rob Bell <[email protected]>

* chore: rename status server

Signed-off-by: Rob Bell <[email protected]>

* fix: revert adding status pkg to manifest gen paths

Signed-off-by: Rob Bell <[email protected]>

* chore: update log messages

Signed-off-by: Rob Bell <[email protected]>

* chore: update api docs

Signed-off-by: Rob Bell <[email protected]>

* chore: update format

Signed-off-by: Rob Bell <[email protected]>

* fix: revert inadvertent change

Signed-off-by: Rob Bell <[email protected]>

* chore: update api name and type docs

Signed-off-by: Rob Bell <[email protected]>

* refactor: initialise oidc provider in manager startup

Signed-off-by: Rob Bell <[email protected]>

* fix: allow empty update request

Signed-off-by: Rob Bell <[email protected]>

* fix: wait for server shut down

Signed-off-by: Rob Bell <[email protected]>

* fix: handle missing train job

Signed-off-by: Rob Bell <[email protected]>

* refactor: update env var names

Signed-off-by: Rob Bell <[email protected]>

* chore: rename packages, plugins, feature gate

Signed-off-by: Rob Bell <[email protected]>

* refactor: update env var names

Signed-off-by: Rob Bell <[email protected]>

* chore: regenerate after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: fixes after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: minor doc updates

Signed-off-by: Rob Bell <[email protected]>

* chore: rename status server config

Signed-off-by: Rob Bell <[email protected]>

* chore: replace placeholder config with nil

Signed-off-by: Rob Bell <[email protected]>

* chore: revert separate runtime utils package

Signed-off-by: Rob Bell <[email protected]>

* chore: remove unused config from tests

Signed-off-by: Rob Bell <[email protected]>

---------

Signed-off-by: Rob Bell <[email protected]>
 2026-03-13 20:01:08 +00:00
Antonin StefanuttiGitHubAndrey Velichkevich
 		37c8a07a8a feat(api): Replace PodTemplateOverrides with RuntimePatches API (#3309) 
* feat(api): Replace PodTemplateOverrides with RuntimePatches API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update goal with RuntimePatches API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add diagram for RuntimePatches

Signed-off-by: Andrey Velichkevich <[email protected]>

* feat(api): Replace PodTemplateOverrides with RuntimePatches API

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix kube lint errors

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update Python models

Signed-off-by: Antonin Stefanutti <[email protected]>

* Enable maxlength Kube-API-Linter rule

Signed-off-by: Antonin Stefanutti <[email protected]>

* Restore manger field IsDomainPrefixedPath validation

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix integration tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Apply StrategicMergePatch to batchv1.JobTemplateSpec

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Antonin Stefanutti <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
 2026-03-12 16:29:44 +00:00
Andrey VelichkevichandGitHub 63d928468c feat(api): BREAKING CHANGE: Remove numProcPerNode from Torch API (#3239) 
* feat(api): BREAKING CHANGE: Remove numProcPerNode from Torch API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove trainer_v1alpha1_torch_elastic_policy.py from Py models

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix numNodes validation in torchtune

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2026-03-01 06:53:03 +00:00
Andrey VelichkevichandGitHub 60004df3d3 feat(api): BREAKING CHANGE – Remove ElasticPolicy API (#3235) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2026-02-24 20:58:47 +00:00
Tomas TormoandGitHub bb945b03dd feat: Add the manager field to the podTemplateOverride object (#3020) 
Signed-off-by: Tomas Tormo <[email protected]>
 2026-02-16 23:54:28 +00:00
Shashank SinghandGitHub db4330d386 fix: fix resourcePerNode override not applied with Volcano scheduler (#2982) 
* volcano integeration for resourcePerNode override in trainjob

Signed-off-by: sksingh2005 <[email protected]>

* Fixed the issue where PodGroup resources weren't scaling with spec.trainer.resourcesPerNode when specified on a TrainJob

Signed-off-by: sksingh2005 <[email protected]>

* limits removed

Signed-off-by: sksingh2005 <[email protected]>

* maintains consistency across all MLPolicy plugins

Signed-off-by: sksingh2005 <[email protected]>

* Fix pre-commit formatting

Signed-off-by: sksingh2005 <[email protected]>

* Unit test updates for mpi and torch pluggins

Signed-off-by: sksingh2005 <[email protected]>

* framework unit test prblm solved

Signed-off-by: sksingh2005 <[email protected]>

* fix:All MLPolicy Plugins update SinglePodRequests from TrainJob.spec.trainer.resourcesPerNode

Signed-off-by: sksingh2005 <[email protected]>

* Refactor ResourcesPerNode handling to use PodRequest helper for correct scaling with init/sidecar containers

Signed-off-by: sksingh2005 <[email protected]>

* refactor: simplify ResourcesPerNode by modifying jobSetTemplateSpec directly

Signed-off-by: sksingh2005 <[email protected]>

* test: modify TrainJob resources in coscheduling test to verify node container override

Signed-off-by: sksingh2005 <[email protected]>

* Refactor: remove duplicate resource logic from plugins

Signed-off-by: sksingh2005 <[email protected]>

---------

Signed-off-by: sksingh2005 <[email protected]>
 2026-01-18 21:12:22 +00:00
Sanskar GurdasaniandGitHub b5532e5e33 feat(api): Add securityContext support to PodTemplateSpecOverride in TrainJob (#3066) 
Signed-off-by: Sanskarzz <[email protected]>
 2026-01-15 15:30:18 +00:00
Antonin StefanuttiandGitHub 0f9825e89e chore(operator): Use SSA throughout runtime framework (#2877) 
* chore(operator): Use SSA throughout runtime framework

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix lint error

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update go.mod file

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-10-31 11:43:12 +00:00
Antonin StefanuttiandGitHub 7755220575 chore(ci): Enable Kubernetes API Linter (#2858) 
* chore(ci): Enable Kubernetes API Linter

Signed-off-by: Antonin Stefanutti <[email protected]>

* lint files

Signed-off-by: Antonin Stefanutti <[email protected]>

* Generate

Signed-off-by: Antonin Stefanutti <[email protected]>

* Review

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix golangci-lint target

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix commands max length

Signed-off-by: Antonin Stefanutti <[email protected]>

* Re: Fix golangci-lint target

Signed-off-by: Antonin Stefanutti <[email protected]>

* Enable the integers linter

Signed-off-by: Antonin Stefanutti <[email protected]>

* Disable maxlength linter

Signed-off-by: Antonin Stefanutti <[email protected]>

* make generate

Signed-off-by: Antonin Stefanutti <[email protected]>

* Make fields optional

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix Suspended field in JobStatus

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-10-17 14:40:59 +00:00
Tarun DuhanGitHubAndrey Velichkevich
 		ec48455076 feat(runtimes): implement clusterTrainingRuntime deprecation process (#2791) 
* constants: add deprecation label constants for runtimes

Signed-off-by: Tarun Duhan <[email protected]>

* webhooks: warn on creation of deprecated ClusterTrainingRuntime

Signed-off-by: Tarun Duhan <[email protected]>

* runtime: warn when TrainJob references deprecated ClusterTrainingRuntime

Signed-off-by: Tarun Duhan <[email protected]>

* tests: add unit test coverage for deprecation warnings

Signed-off-by: Tarun Duhan <[email protected]>

* webhooks/runtime: update deprecation warning wording and tests

Signed-off-by: Tarun Duhan <[email protected]>

* constants/webhooks/runtime: centralize runtime policy URL and reference constant

Signed-off-by: Tarun Duhan <[email protected]>

* tests(integration/webhooks): add create case for deprecated ClusterTrainingRuntime

Create ClusterTrainingRuntime with deprecated label; assert creation succeeds (warning is non-blocking)

Signed-off-by: Tarun Duhan <[email protected]>

* tests(integration/webhooks): add TrainJob create with deprecated ClusterTrainingRuntime

Mark ClusterTrainingRuntime deprecated; create TrainJob referencing it; assert creation succeeds (warning is non-blocking)

Signed-off-by: Tarun Duhan <[email protected]>

* tests/runtime: cover deprecation warning in ClusterTrainingRuntime.ValidateObjects

Signed-off-by: Tarun Duhan <[email protected]>

* tests(integration): remove deprecation-warning cases for CTR/TrainJob webhooks

Signed-off-by: Tarun Duhan <[email protected]>

* feat(runtimes): switch to support=deprecated and centralize deprecation check

Signed-off-by: Tarun Duhan <[email protected]>

* tests(webhooks): focus UTs on entrypoints and clean up tests/test data

Signed-off-by: Tarun Duhan <[email protected]>

* Rename package from labels to trainingruntime

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Tarun Duhan <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
 2025-10-16 10:50:53 +00:00
Shao WangGitHubAndrey Velichkevich
 		615e338bf4 feat(runtimes): Add LoRA/QLoRA/DoRA support in LLM Trainer V2 (#2832) 
* feat(runtimes): Add lora recipe and config in torch.plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* chore: add --trainer-use-lora flag and related logics.

Signed-off-by: Electronic-Waste <[email protected]>

* feat(lora): Add support for QLoRA.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lora): fix extra rdzv backend in single device mode.

Signed-off-by: Electronic-Waste <[email protected]>

* test(lora): Add UTs for lora/qlora in trainingruntime.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lora): remove extra quote symbol in lora attn module.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtime): check default GPU resources allocated in CTRs.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(runtime): add getNumProcPerNode.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtime): remove specific vendor for GPU.

Signed-off-by: Electronic-Waste <[email protected]>

* test(runtime): add UTs for lora validation.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): remove extra flags for torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* Update pkg/runtime/framework/plugins/torch/torch.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Shao Wang <[email protected]>

* chore(runtime): add extractGPUCountFromRuntime.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): remove filtering args logics.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): use fallbackNumProcPerNode if GPU is set in runtimes.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(runtimes): refactor getNumProcPerNode logics.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): call getNumProcPerNode only once.

Signed-off-by: Electronic-Waste <[email protected]>

* test(runtime): fix UTs for torch.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): add limits to calculation.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(runtimes): reorg nppNode related logics.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): fix lints.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): update condition that use nppNode in Limits.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(runtimes): add torchtune.go

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(torch): only reset nppNode when using CPU.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(runtimes): remove getNumProcPerNodeFromCPU.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
Signed-off-by: Shao Wang <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
 2025-10-15 12:44:09 +00:00
Xigang WangandGitHub 1d01c440b7 feat(api): Add PodTemplateOverrides API into TrainJob (#2785) 
Signed-off-by: xigang <[email protected]>
 2025-10-09 10:08:14 +00:00
Antonin StefanuttiandGitHub e9586f9c78 feat(api): Sync TrainJob JobsStatus from JobSet ReplicatedJobsStatus (#2802) 
* feat(api): Sync TrainJob JobsStatus from JobSet ReplicatedJobsStatus

Signed-off-by: Antonin Stefanutti <[email protected]>

* Add integration tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update e2e tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Remove extra check

Signed-off-by: Antonin Stefanutti <[email protected]>

* Sort JobsStatus in e2e tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix e2e test for MPI job

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fail-fast when multiple terminal condition and JobsStatus plugins exist

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fold TerminalCondition and JobsStatus plugins

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-10-07 13:19:12 +00:00
Tomas TormoandGitHub b1f1a4ab6f fix: Allow multiple podSpec overrides to target the same TargetJob (#2880) 
Signed-off-by: Tomas Tormo <[email protected]>
 2025-10-07 12:55:13 +00:00
Tomas TormoandGitHub 71ac41935d feat: Add a public function to create runtime info objects (#2837) 
Signed-off-by: Tomas Tormo <[email protected]>
 2025-09-22 15:47:02 +00:00
Tsubasa NagasawaandGitHub 451c7bf65f chore: merge test cases using PodSpecOverrides into a single case (#2822) 
* chore: merge test cases using PodSpecOverrides into a single case

Signed-off-by: Tsubasa Nagasawa <[email protected]>

* chore: merge test cases for container env with PodSpecOverride

Signed-off-by: Tsubasa Nagasawa <[email protected]>

* chore: remove existing test case for container with PodSpecOverride

Signed-off-by: Tsubasa Nagasawa <[email protected]>

---------

Signed-off-by: Tsubasa Nagasawa <[email protected]>
 2025-09-09 12:22:37 +00:00
Tsubasa NagasawaGitHubAntonin Stefanutti
 		39e4b08a03 feat: support imagePullSecrets in TrainJob pod spec overrides (#2806) 
Signed-off-by: Tsubasa Nagasawa <[email protected]>
Co-authored-by: Antonin Stefanutti <[email protected]>
 2025-09-08 12:43:36 +00:00
Tsubasa NagasawaandGitHub 187a5de400 feat: support affinity in TrainJob pod spec overrides (#2796) 
Signed-off-by: Tsubasa Nagasawa <[email protected]>
 2025-08-28 15:52:22 +00:00
Harshvir PotposeandGitHub bf4c11567e chore: Add unit tests for pkg/apply (#2479) 
* add unit tests

Signed-off-by: Harshvir Potpose <[email protected]>

* refactor tests

Signed-off-by: Harshvir Potpose <[email protected]>

* add FromTypedObjWithFields tests

Signed-off-by: Harshvir Potpose <[email protected]>

* keep lines short

Signed-off-by: Harshvir Potpose <[email protected]>

* go mod tidy

Signed-off-by: Harshvir Potpose <[email protected]>

* consolidate UpsertEnvVars func

Signed-off-by: Harshvir Potpose <[email protected]>

* consolidate wantError

Signed-off-by: Harshvir Potpose <[email protected]>

* rm unecessary returns

Signed-off-by: Harshvir Potpose <[email protected]>

* use UpsertEnvVar

Signed-off-by: Harshvir Potpose <[email protected]>

* standerdize interface

Signed-off-by: Harshvir Potpose <[email protected]>

---------

Signed-off-by: Harshvir Potpose <[email protected]>
 2025-08-18 10:10:03 +00:00
Andrey VelichkevichandGitHub 8655798f02 fix(module): Change Go module name to v2 (#2707) 
* fix(module): Change Go module name to v2

Signed-off-by: Andrey Velichkevich <[email protected]>

* Bump x/net to v0.38.0

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2025-07-03 17:46:20 +00:00
Antonin StefanuttiandGitHub 8dc6334ccb feat: Add schedulingGates to PodSpecOverrides (#2700) 
* feat: Add schedulingGates to PodSpecOverrides

Signed-off-by: Antonin Stefanutti <[email protected]>

* Change desired job to target job in PodSpecOverrides comments

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-07-03 11:03:20 +00:00
Antonin StefanuttiandGitHub cdc702bf25 feat: Mutable PodSpecOverrides for suspended TrainJob (#2683) 
* feat: Mutable PodSpecOverrides for suspended TrainJob

Signed-off-by: Antonin Stefanutti <[email protected]>

* Include @tenzen-y review

Signed-off-by: Antonin Stefanutti <[email protected]>

* Add unit tests

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-07-01 13:57:18 +00:00
Yuki IwaiandGitHub f99f29e732 chore: Remove the vendor specific parameters (#2691) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-06-30 14:59:21 +00:00
Shao WangandGitHub c6d3ee01fc fix(plugins): Fix some errors in torchtune mutation process. (#2675) 
* fix(plugins): fix errors in trainer command mutation of torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugins): remove config file format suffix.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): update UTs.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(initializer): Update the workspace of dataset/model initializer.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugins): parse nproc_per_node from GPU resource.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(torchtune): Add bitsandbytes dependency in requirements.txt

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(torchtune): Remove unnecessary num_proc_per_node calculation.

Signed-off-by: Electronic-Waste <[email protected]>

* test(torch): Update invalid parameters.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
 2025-06-30 14:56:21 +00:00
Shao WangandGitHub 3c3bf09c00 KEP-2401: Support loading local LLMs (#2644) 
* feat(torch): Support loading local models.

Signed-off-by: Electronic-Waste <[email protected]>

* test(torch): Update UTs for loading local model.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(manifest): Update loading local LLMs command.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(manifest): Update pvc & model dir in manifests.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): update UTs.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(torchtune): Update checkpoint dir and tokenzier path in torch plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* chore: tidy up code.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(torchtune): Remove CheckpointDir.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(torchtune): remove DefaultTorchTuneModel.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(torchtune): add extractOverridesFromRuntime func.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(test): update UTs in pkg/runtime/core.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(manifest): update output dir.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
 2025-06-06 18:12:16 +00:00
Andrey VelichkevichGitHubShao WangAntonin StefanuttiAntonin Stefanutti
 		434d953004 feat(controller): Implement PodSpecOverride API (#2614) 
* feat(controller): Implement PodSpecOverride API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update API comments

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix InitContainers override

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix generate

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update pkg/apis/trainer/v1alpha1/trainjob_types.go

Co-authored-by: Shao Wang <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>

* Change API to targetJob and add validation

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add unit test for the JobSet builder

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add integration tests

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update pkg/apis/trainer/v1alpha1/trainjob_types.go

Co-authored-by: Shao Wang <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>

* Update pkg/apis/trainer/v1alpha1/trainjob_types.go

Co-authored-by: Shao Wang <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>

* Update pkg/apis/trainer/v1alpha1/trainjob_types.go

Co-authored-by: Antonin Stefanutti <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>

* Merge PodSpecOverrides before the build phase

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix make generate

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix PodSpec SMT patch unmarshalling into JobSet template

Signed-off-by: Antonin Stefanutti <[email protected]>

* Add UT for volumes PodSpecOverrides

Signed-off-by: Antonin Stefanutti <[email protected]>

* Add UT for tolerations PodSpecOverrides

Signed-off-by: Antonin Stefanutti <[email protected]>

* Add UT for node selector PodSpecOverrides

Signed-off-by: Antonin Stefanutti <[email protected]>

* Revert back to using TargetJobs []PodSpecOverrideTargetJob

Signed-off-by: Antonin Stefanutti <[email protected]>

* Run make fmt

Signed-off-by: Antonin Stefanutti <[email protected]>

* Add target jobs unicity validation for PodSpecOverrides

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update with code review from @tenzen-y

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Antonin Stefanutti <[email protected]>
Co-authored-by: Shao Wang <[email protected]>
Co-authored-by: Antonin Stefanutti <[email protected]>
Co-authored-by: Antonin Stefanutti <[email protected]>
 2025-06-06 17:04:18 +00:00
Shao WangandGitHub 4493eb8770 KEP-2401: Complement torch plugin to support torchtune config mutation (#2587) 
* chore(plugin): Add torchtune-related constants & update current torch plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(plugin): Add EnforceMLPolicy for torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(plugin): Add UTs in torch plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): fix error in torch plugin UTs.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(plugin): Choose recipe according to numNodes & numProcPerNode & Args.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(sdk): Add PretrainedModel enum type.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(plugin): Add torchtune config arg.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(test): add UT for single-device full fine-tuning with torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(test): Add test for multi-nodes full fine-tuning with torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(test): Update torch validate UTs.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(sdk): remove pretrained model enum type in sdk.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugin): retrieve model name from runtimeRef.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix typo.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugin): make some adjustments according to the review.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(sdk): remove runtime in get_trainer_crd_from_builtin_trainer.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugin): pass PET_ env variables in torch plugin for torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugin): add env validation for torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugin): update comments.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugins): fix the implementation according to the review.

Signed-off-by: Electronic-Waste <[email protected]>

* test(plugins): fix UT error in torch plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* fix: fix UT and e2e tests error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix: remove debug info.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): add args in UTs related to torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): update torchtune related args.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): Add a UT for multi-node mode check in torch plugin.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
 2025-04-29 13:21:03 +00:00
AsklvandGitHub ccbd58e4ee test(runtime): add UT for torch runtime valid function. (#2560) 
Signed-off-by: Asklv <[email protected]>
 2025-03-26 07:47:04 +00:00
Akshay ChitneniGitHubAkshay Chitneni
 		d6a2b00e0c Fix issue with fetching clustertrainingruntime for validations (#2564) 
Signed-off-by: Akshay Chitneni <[email protected]>
Signed-off-by: Akshay Chitneni <[email protected]>
Co-authored-by: Akshay Chitneni <[email protected]>
 2025-03-24 15:14:06 +00:00
AsklvandGitHub 07ce28219c feat(runtime): remove needless Launcher chainer. (#2558) 
Signed-off-by: Asklv <[email protected]>
 2025-03-24 11:14:05 +00:00
Yuki IwaiandGitHub 3dd0fb0b82 Retrieve JobSetSpec from runtime.Info in CustomValidations (#2557) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-24 11:05:06 +00:00
Andrey VelichkevichandGitHub eb1ee9d0f4 feat(sdk): Support MPI-based TrainJobs (#2545) 
* feat(sdk): Support MPI-based TrainJobs

Signed-off-by: Andrey Velichkevich <[email protected]>

* Refactor list_runtimes

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix example

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add Runtime Trainer object

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update for new Runtime object

Signed-off-by: Andrey Velichkevich <[email protected]>

* Implement get_runtime API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix Torch example

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove un-unsed consts

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update func args

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update SDK constants

Signed-off-by: Andrey Velichkevich <[email protected]>

* Change to 16Gi

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix container name for MPI

Signed-off-by: Andrey Velichkevich <[email protected]>

* Keep launcher container for MPI

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2025-03-20 18:08:02 +00:00
Yuki IwaiandGitHub 50d6be8324 Construct Trainer based on trainer.kubeflow.org/trainjob-ancestor-step label (#2548) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-20 14:01:57 +00:00
Yuki IwaiandGitHub 64404f6565 Store the TrainingRuntime numNodes as runtime.Info.PodSet.Count (#2539) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-18 15:55:44 +00:00
Andrey VelichkevichandGitHub 239b27d311 feat(controller): Refactor the Initializer APIs of TrainJob (#2523) 
* feat(controller): Refactor the Initializer APIs of TrainJob

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix go unit test

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix integration test

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2025-03-17 07:15:51 +00:00
Yuki IwaiandGitHub 38e16d718d Migrate InfoOptions.podSpecReplias and info.Scheduler.TotalRequests to info.TemplateSpec.PodSet (#2524) 
Signed-off-by: Yuki Iwai <[email protected]>

# Conflicts:
#	pkg/runtime/core/trainingruntime.go
#	pkg/runtime/runtime.go
 2025-03-16 17:09:49 +00:00
Akshay ChitneniGitHubAkshay Chitneni
 		a589903c24 KEP-2170: Adding validation webhook for v2 trainjob (#2307) 
Signed-off-by: Akshay Chitneni <[email protected]>
Co-authored-by: Akshay Chitneni <[email protected]>
 2025-03-16 03:41:51 +00:00
DiaskerGitHubYuki Iwai
 		f98108aa6b Fix #2407: Cap nproc_per_node based on CPU resources for PyTorch TrainJob (#2492) 
* Fix #2407: Cap nproc_per_node based on CPU resources for PyTorch TrainJob

Signed-off-by: Diasker <[email protected]>

* Refactor the fix using resource.MustParse and RoundUp

Signed-off-by: Diasker <[email protected]>

* remove unecessary log message

Signed-off-by: Diasker <[email protected]>

* update test case to expect nproc_per_node=1 without GPU

Signed-off-by: Diasker <[email protected]>

* Implement numProcPerNode=cpu option and refactor tests.

Signed-off-by: Diasker <[email protected]>

* refactor: improve torch test cases

Signed-off-by: Diasker <[email protected]>

* remove the redundant logic

Signed-off-by: Diasker <[email protected]>

* Update pkg/runtime/framework/plugins/torch/torch.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Diasker <[email protected]>

* Update pkg/runtime/framework/plugins/torch/torch_test.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Diasker <[email protected]>

* simply code and remove redundant comments

Signed-off-by: Diasker <[email protected]>

* Update pkg/runtime/framework/plugins/torch/torch.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Diasker <[email protected]>

* Update pkg/runtime/framework/plugins/torch/torch_test.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Diasker <[email protected]>

* Update pkg/runtime/framework/plugins/torch/torch.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Diasker <[email protected]>

* remove get_num_proc_per_node for sdk and simply torch.go

Signed-off-by: Diasker <[email protected]>

* Update expected nproc_per_node value in trainjob_controller_test.go

Signed-off-by: Diasker <[email protected]>

* Update expected nproc_per_node value in trainjob_controller_test.go and remove redundant code of sdk

Signed-off-by: Diasker <[email protected]>

* update trainjob_controller_test.go

Signed-off-by: Diasker <[email protected]>

* update torch.go

Signed-off-by: Diasker <[email protected]>

---------

Signed-off-by: Diasker <[email protected]>
Signed-off-by: Diasker <[email protected]>
Co-authored-by: Yuki Iwai <[email protected]>
 2025-03-13 19:29:37 +00:00
Yuki IwaiandGitHub e8d6727da6 Implemenet MPI Plugin for OpenMPI (#2493) 
* Implemenet MPI Plugin for OpenMPI

Signed-off-by: Yuki Iwai <[email protected]>

* Directory pass the JobSetApplyconfiguration to RuntimeInfo

Signed-off-by: Yuki Iwai <[email protected]>

* Make repeated string as constants

Signed-off-by: Yuki Iwai <[email protected]>

* Use numNodes=1 as default mpi_distributed ClusterTrainingRuntime

Signed-off-by: Yuki Iwai <[email protected]>

* Remove unused errors

Signed-off-by: Yuki Iwai <[email protected]>

* Rename runLauncherAsWorker with runLauncherAsNode

Signed-off-by: Yuki Iwai <[email protected]>

* Fix unintended constants usage for ModelMountPath

Signed-off-by: Yuki Iwai <[email protected]>

* Rename SecretDataComparer with MPISecretDataComparer

Signed-off-by: Yuki Iwai <[email protected]>

* Add TODO STAEEMENT to deprecated env wrappers.

Signed-off-by: Yuki Iwai <[email protected]>

---------

Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-13 05:00:56 +00:00
Andrey VelichkevichGitHubYuki Iwai
 		0cae7e1b0b feat(controller): Integrate DependsOn API (#2484) 
* feat(controller): Integrate DependsOn API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Use go for unit test

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update Makefile

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>

* Update Makefile

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix integration test

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix e2e

Signed-off-by: Andrey Velichkevich <[email protected]>

* Exit 1 if e2e fails

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Yuki Iwai <[email protected]>
 2025-03-07 01:11:34 +00:00
Yuki IwaiandGitHub c5222095c2 Add dependencies to RuntimeRegistrar (#2476) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-06 02:13:13 +00:00
Yuki IwaiandGitHub a3a1324f41 Implement UTs for PlainML plugin (#2469) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-05 04:40:06 +00:00
Antonin StefanuttiandGitHub 4ad02aca05 KEP-2170: Use SSA to reconcile TrainJob components (#2431) 
* KEP-2170: Use SSA to reconcile TrainJob components

Signed-off-by: Antonin Stefanutti <[email protected]>

* Enable Unstructured caching in controller manager config

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix PodGroup apply configuration

Signed-off-by: Antonin Stefanutti <[email protected]>

* API to apply config conversion util functions

Signed-off-by: Antonin Stefanutti <[email protected]>

* Only add namespace to TrainingRuntime object key

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix EnvVar upsert

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update unit tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix JobSet resource requirements

Signed-off-by: Antonin Stefanutti <[email protected]>

* Resolve build issues with launcher job

Signed-off-by: Antonin Stefanutti <[email protected]>

* Use apply config for MPI ConfigMap and Secret

Signed-off-by: Antonin Stefanutti <[email protected]>

* ComponentBuilderPlugin now returns an array

Signed-off-by: Antonin Stefanutti <[email protected]>

* Use plain apply configurations instead of unstructured

Signed-off-by: Antonin Stefanutti <[email protected]>

* Use apply config in EnforceMLPolicy plugins

Signed-off-by: Antonin Stefanutti <[email protected]>

* Do not update JobSets that are not suspended

Signed-off-by: Antonin Stefanutti <[email protected]>

* Address review feedback

Signed-off-by: Antonin Stefanutti <[email protected]>

* Do not update PodGroup if TrainJob is not suspended

Signed-off-by: Antonin Stefanutti <[email protected]>

* Remove obsolete TODO

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-02-26 08:39:09 +00:00
Antonin StefanuttiandGitHub d954ed3272 KEP-2170: Add validation to Torch numProcPerNode field (#2409) 
Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-02-14 20:57:37 +00:00
Akshay ChitneniGitHubAkshay Chitneni
 		a41677611e KEP: 2170: Adding cel validations on TrainingRuntime/ClusterTrainingRuntime CRDs (#2313) 
Signed-off-by: Akshay Chitneni <[email protected]>
Co-authored-by: Akshay Chitneni <[email protected]>
 2025-02-14 19:40:37 +00:00