hanzo-ml/trainer
1
0
Fork 0
mirror of https://github.com/hanzo-ml/trainer.git synced 2026-09-24 00:31:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,730 Commits 2 Branches 1 Tag
316064e6985885094de71463172fcafa81a93ed5
Commit Graph
33 Commits
Author SHA1 Message Date
krishdef7andGitHub cdff07ff48 fix(runtimes): add validation for LoRA multi-node and immutable trainer args (#3302) 
* fix(torchtune): add validation for LoRA multi-node and immutable args

Add two missing validations to validateTorchTune:

1. Reject LoRA fine-tuning when numNodes > 1. getRecipeAndConfig
   silently falls through to full_finetune_distributed in this case,
   discarding LoRA args without any user-visible error.

2. Reject immutable runtime configs (output_dir, tokenizer.path,
   checkpointer.checkpoint_dir, tokenizer.merges_file) in
   spec.trainer.args. These are injected by the runtime via
   extractOverridesFromRuntime and must not be overridden by the user.

Resolves the TODO added by @Electronic-Waste in torchtune.go.

Signed-off-by: Krish Garg <[email protected]>
Signed-off-by: krishdef7 <[email protected]>

* fix(runtimes): add validation for LoRA/QLoRA multi-node and immutable trainer args

Signed-off-by: krishdef7 <[email protected]>

---------

Signed-off-by: Krish Garg <[email protected]>
Signed-off-by: krishdef7 <[email protected]>
 2026-05-20 14:15:20 +00:00
Rob BellandGitHub 78b4a00929 feat: add support for tracking TrainJob progress and training metrics (#3227) 
* feat(api): add progress types to api

Signed-off-by: Rob Bell <[email protected]>

* feat: progress server stub

Signed-off-by: Rob Bell <[email protected]>

* feat: setup tls for server

Signed-off-by: Rob Bell <[email protected]>

* refactor: pull out middleware

Signed-off-by: Rob Bell <[email protected]>

* refactor: remove logger message noise from tests

Signed-off-by: Rob Bell <[email protected]>

* fix: send server error logs to main logger

Signed-off-by: Rob Bell <[email protected]>

* feat: update the trainer status

Signed-off-by: Rob Bell <[email protected]>

* feat: use separate client to avoid impacting main reconciler client-side rate limits

Signed-off-by: Rob Bell <[email protected]>

* feat: configure progress server in helm chart and configmap

Signed-off-by: Rob Bell <[email protected]>

* fix: prevent race condition

Signed-off-by: Rob Bell <[email protected]>

* feat: inject progress runtime config

Signed-off-by: Rob Bell <[email protected]>

* fix: progress client config

Signed-off-by: Rob Bell <[email protected]>

* fix: update generated config

Signed-off-by: Rob Bell <[email protected]>

* feat: validate auth tokens in endpoint

Signed-off-by: Rob Bell <[email protected]>

* test: validate auth tokens in endpoint

Signed-off-by: Rob Bell <[email protected]>

* refactor: tidy tests to use same format as other tests

Signed-off-by: Rob Bell <[email protected]>

* fix: look up the cluster issuer from the service account token

Different clusters can use different issuers so look up the issuer from the trusted in-cluster service account token.

Signed-off-by: Rob Bell <[email protected]>

* refactor: rename authn middleware

Signed-off-by: Rob Bell <[email protected]>

* feat: add authn to status update endpoint

Signed-off-by: Rob Bell <[email protected]>

* refactor: tidy up reading in-cluster service account token

Signed-off-by: Rob Bell <[email protected]>

* fix: rbac permissions

Signed-off-by: Rob Bell <[email protected]>

* fix: better error response

Signed-off-by: Rob Bell <[email protected]>

* refactor: move kubebuilder annotation higher up

Signed-off-by: Rob Bell <[email protected]>

* fix: make lastUpdateTime required

Signed-off-by: Rob Bell <[email protected]>

* chore: fix config tests after rebase

Signed-off-by: Rob Bell <[email protected]>

* test: update config test with new progress server config

Signed-off-by: Rob Bell <[email protected]>

* test: add progress tracking e2e test

Signed-off-by: Rob Bell <[email protected]>

* fix: allow feature gates to be configurable from command line argument

So they can be overridden in e2e tests

Signed-off-by: Rob Bell <[email protected]>

* refactor: reuse shared progress StatusUrl

Signed-off-by: Rob Bell <[email protected]>

* test: tidy up tests

Signed-off-by: Rob Bell <[email protected]>

* refactor: allow plugins to use config

Signed-off-by: Rob Bell <[email protected]>

* refactor: centralise shared constants

Signed-off-by: Rob Bell <[email protected]>

* fix: disable http2 in progress server tls opts if disabled by config

Signed-off-by: Rob Bell <[email protected]>

* fix: minor improvements suggested by copilot

Signed-off-by: Rob Bell <[email protected]>

* chore: rename progress server => status server

Signed-off-by: Rob Bell <[email protected]>

* chore: update type doc

Signed-off-by: Rob Bell <[email protected]>

* chore: update name of feature gate

Signed-off-by: Rob Bell <[email protected]>

* fix: after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: fix typo

Signed-off-by: Rob Bell <[email protected]>

* chore: rename handler

Signed-off-by: Rob Bell <[email protected]>

* fix: run status server on all replicas

Signed-off-by: Rob Bell <[email protected]>

* fix: deep copy the rest config

Signed-off-by: Rob Bell <[email protected]>

* refactor: use ssa

Signed-off-by: Rob Bell <[email protected]>

* refactor: separate token audience per train job

Signed-off-by: Rob Bell <[email protected]>

* chore: rename status server

Signed-off-by: Rob Bell <[email protected]>

* fix: revert adding status pkg to manifest gen paths

Signed-off-by: Rob Bell <[email protected]>

* chore: update log messages

Signed-off-by: Rob Bell <[email protected]>

* chore: update api docs

Signed-off-by: Rob Bell <[email protected]>

* chore: update format

Signed-off-by: Rob Bell <[email protected]>

* fix: revert inadvertent change

Signed-off-by: Rob Bell <[email protected]>

* chore: update api name and type docs

Signed-off-by: Rob Bell <[email protected]>

* refactor: initialise oidc provider in manager startup

Signed-off-by: Rob Bell <[email protected]>

* fix: allow empty update request

Signed-off-by: Rob Bell <[email protected]>

* fix: wait for server shut down

Signed-off-by: Rob Bell <[email protected]>

* fix: handle missing train job

Signed-off-by: Rob Bell <[email protected]>

* refactor: update env var names

Signed-off-by: Rob Bell <[email protected]>

* chore: rename packages, plugins, feature gate

Signed-off-by: Rob Bell <[email protected]>

* refactor: update env var names

Signed-off-by: Rob Bell <[email protected]>

* chore: regenerate after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: fixes after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: minor doc updates

Signed-off-by: Rob Bell <[email protected]>

* chore: rename status server config

Signed-off-by: Rob Bell <[email protected]>

* chore: replace placeholder config with nil

Signed-off-by: Rob Bell <[email protected]>

* chore: revert separate runtime utils package

Signed-off-by: Rob Bell <[email protected]>

* chore: remove unused config from tests

Signed-off-by: Rob Bell <[email protected]>

---------

Signed-off-by: Rob Bell <[email protected]>
 2026-03-13 20:01:08 +00:00
Amir IbrahimandGitHub 926b01ab02 fix: rename JAX and Torch runtime plugin tests to descriptive names (#3283) 
Signed-off-by: Amir380-A <[email protected]>
 2026-03-13 01:52:14 +00:00
Andrey VelichkevichandGitHub 63d928468c feat(api): BREAKING CHANGE: Remove numProcPerNode from Torch API (#3239) 
* feat(api): BREAKING CHANGE: Remove numProcPerNode from Torch API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove trainer_v1alpha1_torch_elastic_policy.py from Py models

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix numNodes validation in torchtune

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2026-03-01 06:53:03 +00:00
Andrey VelichkevichandGitHub 60004df3d3 feat(api): BREAKING CHANGE – Remove ElasticPolicy API (#3235) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2026-02-24 20:58:47 +00:00
Antonin StefanuttiandGitHub 0f9825e89e chore(operator): Use SSA throughout runtime framework (#2877) 
* chore(operator): Use SSA throughout runtime framework

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix lint error

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update go.mod file

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-10-31 11:43:12 +00:00
Krishnaswamy SubramanianGitHubAndrey Velichkevich
 		576e0896b7 feat(runtimes): add support for launcher resource allocation in MPI jobs (#2653) 
* feat(runtime): add support for launcher resource allocation in MPI jobs

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add unit tests

Signed-off-by: Andrey Velichkevich <[email protected]>

* Set numProcPerNode for MPI plugin

Signed-off-by: Andrey Velichkevich <[email protected]>

* Move util func to runtime package

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix torchtune plugin

Signed-off-by: Andrey Velichkevich <[email protected]>

* Inline if for GPU check

Signed-off-by: Andrey Velichkevich <[email protected]>

* Assign container resources once

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add todo for test wrappers

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
 2025-10-20 13:26:03 +00:00
Shao WangGitHubAndrey Velichkevich
 		615e338bf4 feat(runtimes): Add LoRA/QLoRA/DoRA support in LLM Trainer V2 (#2832) 
* feat(runtimes): Add lora recipe and config in torch.plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* chore: add --trainer-use-lora flag and related logics.

Signed-off-by: Electronic-Waste <[email protected]>

* feat(lora): Add support for QLoRA.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lora): fix extra rdzv backend in single device mode.

Signed-off-by: Electronic-Waste <[email protected]>

* test(lora): Add UTs for lora/qlora in trainingruntime.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lora): remove extra quote symbol in lora attn module.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtime): check default GPU resources allocated in CTRs.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(runtime): add getNumProcPerNode.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtime): remove specific vendor for GPU.

Signed-off-by: Electronic-Waste <[email protected]>

* test(runtime): add UTs for lora validation.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): remove extra flags for torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* Update pkg/runtime/framework/plugins/torch/torch.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Shao Wang <[email protected]>

* chore(runtime): add extractGPUCountFromRuntime.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): remove filtering args logics.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): use fallbackNumProcPerNode if GPU is set in runtimes.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(runtimes): refactor getNumProcPerNode logics.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): call getNumProcPerNode only once.

Signed-off-by: Electronic-Waste <[email protected]>

* test(runtime): fix UTs for torch.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): add limits to calculation.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(runtimes): reorg nppNode related logics.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): fix lints.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(runtimes): update condition that use nppNode in Limits.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(runtimes): add torchtune.go

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* refactor(torch): only reset nppNode when using CPU.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(runtimes): remove getNumProcPerNodeFromCPU.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
Signed-off-by: Shao Wang <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
 2025-10-15 12:44:09 +00:00
Harshvir PotposeandGitHub bf4c11567e chore: Add unit tests for pkg/apply (#2479) 
* add unit tests

Signed-off-by: Harshvir Potpose <[email protected]>

* refactor tests

Signed-off-by: Harshvir Potpose <[email protected]>

* add FromTypedObjWithFields tests

Signed-off-by: Harshvir Potpose <[email protected]>

* keep lines short

Signed-off-by: Harshvir Potpose <[email protected]>

* go mod tidy

Signed-off-by: Harshvir Potpose <[email protected]>

* consolidate UpsertEnvVars func

Signed-off-by: Harshvir Potpose <[email protected]>

* consolidate wantError

Signed-off-by: Harshvir Potpose <[email protected]>

* rm unecessary returns

Signed-off-by: Harshvir Potpose <[email protected]>

* use UpsertEnvVar

Signed-off-by: Harshvir Potpose <[email protected]>

* standerdize interface

Signed-off-by: Harshvir Potpose <[email protected]>

---------

Signed-off-by: Harshvir Potpose <[email protected]>
 2025-08-18 10:10:03 +00:00
Andrey VelichkevichandGitHub 8655798f02 fix(module): Change Go module name to v2 (#2707) 
* fix(module): Change Go module name to v2

Signed-off-by: Andrey Velichkevich <[email protected]>

* Bump x/net to v0.38.0

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2025-07-03 17:46:20 +00:00
Antonin StefanuttiandGitHub cdc702bf25 feat: Mutable PodSpecOverrides for suspended TrainJob (#2683) 
* feat: Mutable PodSpecOverrides for suspended TrainJob

Signed-off-by: Antonin Stefanutti <[email protected]>

* Include @tenzen-y review

Signed-off-by: Antonin Stefanutti <[email protected]>

* Add unit tests

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-07-01 13:57:18 +00:00
Yuki IwaiandGitHub 54b23d2e04 chore: Replace the deprecated intstr.FromInt with intstr.FromInt32 (#2695) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-06-30 16:41:21 +00:00
Yuki IwaiandGitHub f99f29e732 chore: Remove the vendor specific parameters (#2691) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-06-30 14:59:21 +00:00
Shao WangandGitHub c6d3ee01fc fix(plugins): Fix some errors in torchtune mutation process. (#2675) 
* fix(plugins): fix errors in trainer command mutation of torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugins): remove config file format suffix.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): update UTs.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(initializer): Update the workspace of dataset/model initializer.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugins): parse nproc_per_node from GPU resource.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(torchtune): Add bitsandbytes dependency in requirements.txt

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(torchtune): Remove unnecessary num_proc_per_node calculation.

Signed-off-by: Electronic-Waste <[email protected]>

* test(torch): Update invalid parameters.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
 2025-06-30 14:56:21 +00:00
Shao WangandGitHub 3c3bf09c00 KEP-2401: Support loading local LLMs (#2644) 
* feat(torch): Support loading local models.

Signed-off-by: Electronic-Waste <[email protected]>

* test(torch): Update UTs for loading local model.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(manifest): Update loading local LLMs command.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(manifest): Update pvc & model dir in manifests.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): update UTs.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(torchtune): Update checkpoint dir and tokenzier path in torch plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* chore: tidy up code.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(torchtune): Remove CheckpointDir.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(torchtune): remove DefaultTorchTuneModel.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(torchtune): add extractOverridesFromRuntime func.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(test): update UTs in pkg/runtime/core.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(manifest): update output dir.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
 2025-06-06 18:12:16 +00:00
Shao WangandGitHub 4493eb8770 KEP-2401: Complement torch plugin to support torchtune config mutation (#2587) 
* chore(plugin): Add torchtune-related constants & update current torch plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(plugin): Add EnforceMLPolicy for torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(plugin): Add UTs in torch plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): fix error in torch plugin UTs.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(plugin): Choose recipe according to numNodes & numProcPerNode & Args.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(sdk): Add PretrainedModel enum type.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(plugin): Add torchtune config arg.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(test): add UT for single-device full fine-tuning with torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(test): Add test for multi-nodes full fine-tuning with torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(test): Update torch validate UTs.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(sdk): remove pretrained model enum type in sdk.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugin): retrieve model name from runtimeRef.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(lint): fix typo.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugin): make some adjustments according to the review.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(sdk): remove runtime in get_trainer_crd_from_builtin_trainer.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugin): pass PET_ env variables in torch plugin for torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugin): add env validation for torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugin): update comments.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(plugins): fix the implementation according to the review.

Signed-off-by: Electronic-Waste <[email protected]>

* test(plugins): fix UT error in torch plugin.

Signed-off-by: Electronic-Waste <[email protected]>

* fix: fix UT and e2e tests error.

Signed-off-by: Electronic-Waste <[email protected]>

* fix: remove debug info.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): add args in UTs related to torchtune.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): update torchtune related args.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(test): Add a UT for multi-node mode check in torch plugin.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
 2025-04-29 13:21:03 +00:00
AsklvandGitHub ccbd58e4ee test(runtime): add UT for torch runtime valid function. (#2560) 
Signed-off-by: Asklv <[email protected]>
 2025-03-26 07:47:04 +00:00
Akshay ChitneniGitHubAkshay Chitneni
 		d6a2b00e0c Fix issue with fetching clustertrainingruntime for validations (#2564) 
Signed-off-by: Akshay Chitneni <[email protected]>
Signed-off-by: Akshay Chitneni <[email protected]>
Co-authored-by: Akshay Chitneni <[email protected]>
 2025-03-24 15:14:06 +00:00
Yuki IwaiandGitHub 3dd0fb0b82 Retrieve JobSetSpec from runtime.Info in CustomValidations (#2557) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-24 11:05:06 +00:00
Yuki IwaiandGitHub 50d6be8324 Construct Trainer based on trainer.kubeflow.org/trainjob-ancestor-step label (#2548) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-20 14:01:57 +00:00
Yuki IwaiandGitHub 64404f6565 Store the TrainingRuntime numNodes as runtime.Info.PodSet.Count (#2539) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-18 15:55:44 +00:00
Yuki IwaiandGitHub ed1b434f99 chore: Enable GCI for golangci-lint (#2540) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-18 14:47:21 +00:00
Yuki IwaiandGitHub 38e16d718d Migrate InfoOptions.podSpecReplias and info.Scheduler.TotalRequests to info.TemplateSpec.PodSet (#2524) 
Signed-off-by: Yuki Iwai <[email protected]>

# Conflicts:
#	pkg/runtime/core/trainingruntime.go
#	pkg/runtime/runtime.go
 2025-03-16 17:09:49 +00:00
Akshay ChitneniGitHubAkshay Chitneni
 		a589903c24 KEP-2170: Adding validation webhook for v2 trainjob (#2307) 
Signed-off-by: Akshay Chitneni <[email protected]>
Co-authored-by: Akshay Chitneni <[email protected]>
 2025-03-16 03:41:51 +00:00
Yuki IwaiandGitHub f1a0d6178e Migrate Info.Trainer to Info.TemplateSpec.PodSet (#2520) 
Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-15 03:34:11 +00:00
DiaskerandGitHub 5e00e1dc5a fix: remove unused parameter name in default case of shouldUseCPU function (#2521) 
Signed-off-by: Diasker <[email protected]>
 2025-03-14 17:21:31 +00:00
DiaskerGitHubYuki Iwai
 		f98108aa6b Fix #2407: Cap nproc_per_node based on CPU resources for PyTorch TrainJob (#2492) 
* Fix #2407: Cap nproc_per_node based on CPU resources for PyTorch TrainJob

Signed-off-by: Diasker <[email protected]>

* Refactor the fix using resource.MustParse and RoundUp

Signed-off-by: Diasker <[email protected]>

* remove unecessary log message

Signed-off-by: Diasker <[email protected]>

* update test case to expect nproc_per_node=1 without GPU

Signed-off-by: Diasker <[email protected]>

* Implement numProcPerNode=cpu option and refactor tests.

Signed-off-by: Diasker <[email protected]>

* refactor: improve torch test cases

Signed-off-by: Diasker <[email protected]>

* remove the redundant logic

Signed-off-by: Diasker <[email protected]>

* Update pkg/runtime/framework/plugins/torch/torch.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Diasker <[email protected]>

* Update pkg/runtime/framework/plugins/torch/torch_test.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Diasker <[email protected]>

* simply code and remove redundant comments

Signed-off-by: Diasker <[email protected]>

* Update pkg/runtime/framework/plugins/torch/torch.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Diasker <[email protected]>

* Update pkg/runtime/framework/plugins/torch/torch_test.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Diasker <[email protected]>

* Update pkg/runtime/framework/plugins/torch/torch.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Diasker <[email protected]>

* remove get_num_proc_per_node for sdk and simply torch.go

Signed-off-by: Diasker <[email protected]>

* Update expected nproc_per_node value in trainjob_controller_test.go

Signed-off-by: Diasker <[email protected]>

* Update expected nproc_per_node value in trainjob_controller_test.go and remove redundant code of sdk

Signed-off-by: Diasker <[email protected]>

* update trainjob_controller_test.go

Signed-off-by: Diasker <[email protected]>

* update torch.go

Signed-off-by: Diasker <[email protected]>

---------

Signed-off-by: Diasker <[email protected]>
Signed-off-by: Diasker <[email protected]>
Co-authored-by: Yuki Iwai <[email protected]>
 2025-03-13 19:29:37 +00:00
Yuki IwaiandGitHub e8d6727da6 Implemenet MPI Plugin for OpenMPI (#2493) 
* Implemenet MPI Plugin for OpenMPI

Signed-off-by: Yuki Iwai <[email protected]>

* Directory pass the JobSetApplyconfiguration to RuntimeInfo

Signed-off-by: Yuki Iwai <[email protected]>

* Make repeated string as constants

Signed-off-by: Yuki Iwai <[email protected]>

* Use numNodes=1 as default mpi_distributed ClusterTrainingRuntime

Signed-off-by: Yuki Iwai <[email protected]>

* Remove unused errors

Signed-off-by: Yuki Iwai <[email protected]>

* Rename runLauncherAsWorker with runLauncherAsNode

Signed-off-by: Yuki Iwai <[email protected]>

* Fix unintended constants usage for ModelMountPath

Signed-off-by: Yuki Iwai <[email protected]>

* Rename SecretDataComparer with MPISecretDataComparer

Signed-off-by: Yuki Iwai <[email protected]>

* Add TODO STAEEMENT to deprecated env wrappers.

Signed-off-by: Yuki Iwai <[email protected]>

---------

Signed-off-by: Yuki Iwai <[email protected]>
 2025-03-13 05:00:56 +00:00
Antonin StefanuttiandGitHub 4ad02aca05 KEP-2170: Use SSA to reconcile TrainJob components (#2431) 
* KEP-2170: Use SSA to reconcile TrainJob components

Signed-off-by: Antonin Stefanutti <[email protected]>

* Enable Unstructured caching in controller manager config

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix PodGroup apply configuration

Signed-off-by: Antonin Stefanutti <[email protected]>

* API to apply config conversion util functions

Signed-off-by: Antonin Stefanutti <[email protected]>

* Only add namespace to TrainingRuntime object key

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix EnvVar upsert

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update unit tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix JobSet resource requirements

Signed-off-by: Antonin Stefanutti <[email protected]>

* Resolve build issues with launcher job

Signed-off-by: Antonin Stefanutti <[email protected]>

* Use apply config for MPI ConfigMap and Secret

Signed-off-by: Antonin Stefanutti <[email protected]>

* ComponentBuilderPlugin now returns an array

Signed-off-by: Antonin Stefanutti <[email protected]>

* Use plain apply configurations instead of unstructured

Signed-off-by: Antonin Stefanutti <[email protected]>

* Use apply config in EnforceMLPolicy plugins

Signed-off-by: Antonin Stefanutti <[email protected]>

* Do not update JobSets that are not suspended

Signed-off-by: Antonin Stefanutti <[email protected]>

* Address review feedback

Signed-off-by: Antonin Stefanutti <[email protected]>

* Do not update PodGroup if TrainJob is not suspended

Signed-off-by: Antonin Stefanutti <[email protected]>

* Remove obsolete TODO

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-02-26 08:39:09 +00:00
Antonin StefanuttiandGitHub d954ed3272 KEP-2170: Add validation to Torch numProcPerNode field (#2409) 
Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-02-14 20:57:37 +00:00
Akshay ChitneniGitHubAkshay Chitneni
 		a41677611e KEP: 2170: Adding cel validations on TrainingRuntime/ClusterTrainingRuntime CRDs (#2313) 
Signed-off-by: Akshay Chitneni <[email protected]>
Co-authored-by: Akshay Chitneni <[email protected]>
 2025-02-14 19:40:37 +00:00
Andrey VelichkevichandGitHub fa214522ce KEP-2170: Implement MPI Plugin for Kubeflow Trainer (#2394) 
* Implement MPI Plugin for Kubeflow Trainer

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update RBAC

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove old manifests

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix unit test

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix comments

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2025-02-14 17:55:38 +00:00
Andrey VelichkevichandGitHub 1ff1d8ad28 Update the naming conventions for Kubeflow Trainer (#2415) 
* Update the naming conventions for Kubeflow Trainer

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix webhooks

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix paths for webhooks

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update go test cmd

Signed-off-by: Andrey Velichkevich <[email protected]>

* Rename kubeflowv1 to trainer pkg

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2025-02-06 13:48:30 +00:00