19 Commits
Author SHA1 Message Date
Rob BellandGitHub 78b4a00929 feat: add support for tracking TrainJob progress and training metrics (#3227)
* feat(api): add progress types to api

Signed-off-by: Rob Bell <[email protected]>

* feat: progress server stub

Signed-off-by: Rob Bell <[email protected]>

* feat: setup tls for server

Signed-off-by: Rob Bell <[email protected]>

* refactor: pull out middleware

Signed-off-by: Rob Bell <[email protected]>

* refactor: remove logger message noise from tests

Signed-off-by: Rob Bell <[email protected]>

* fix: send server error logs to main logger

Signed-off-by: Rob Bell <[email protected]>

* feat: update the trainer status

Signed-off-by: Rob Bell <[email protected]>

* feat: use separate client to avoid impacting main reconciler client-side rate limits

Signed-off-by: Rob Bell <[email protected]>

* feat: configure progress server in helm chart and configmap

Signed-off-by: Rob Bell <[email protected]>

* fix: prevent race condition

Signed-off-by: Rob Bell <[email protected]>

* feat: inject progress runtime config

Signed-off-by: Rob Bell <[email protected]>

* fix: progress client config

Signed-off-by: Rob Bell <[email protected]>

* fix: update generated config

Signed-off-by: Rob Bell <[email protected]>

* feat: validate auth tokens in endpoint

Signed-off-by: Rob Bell <[email protected]>

* test: validate auth tokens in endpoint

Signed-off-by: Rob Bell <[email protected]>

* refactor: tidy tests to use same format as other tests

Signed-off-by: Rob Bell <[email protected]>

* fix: look up the cluster issuer from the service account token

Different clusters can use different issuers so look up the issuer from the trusted in-cluster service account token.

Signed-off-by: Rob Bell <[email protected]>

* refactor: rename authn middleware

Signed-off-by: Rob Bell <[email protected]>

* feat: add authn to status update endpoint

Signed-off-by: Rob Bell <[email protected]>

* refactor: tidy up reading in-cluster service account token

Signed-off-by: Rob Bell <[email protected]>

* fix: rbac permissions

Signed-off-by: Rob Bell <[email protected]>

* fix: better error response

Signed-off-by: Rob Bell <[email protected]>

* refactor: move kubebuilder annotation higher up

Signed-off-by: Rob Bell <[email protected]>

* fix: make lastUpdateTime required

Signed-off-by: Rob Bell <[email protected]>

* chore: fix config tests after rebase

Signed-off-by: Rob Bell <[email protected]>

* test: update config test with new progress server config

Signed-off-by: Rob Bell <[email protected]>

* test: add progress tracking e2e test

Signed-off-by: Rob Bell <[email protected]>

* fix: allow feature gates to be configurable from command line argument

So they can be overridden in e2e tests

Signed-off-by: Rob Bell <[email protected]>

* refactor: reuse shared progress StatusUrl

Signed-off-by: Rob Bell <[email protected]>

* test: tidy up tests

Signed-off-by: Rob Bell <[email protected]>

* refactor: allow plugins to use config

Signed-off-by: Rob Bell <[email protected]>

* refactor: centralise shared constants

Signed-off-by: Rob Bell <[email protected]>

* fix: disable http2 in progress server tls opts if disabled by config

Signed-off-by: Rob Bell <[email protected]>

* fix: minor improvements suggested by copilot

Signed-off-by: Rob Bell <[email protected]>

* chore: rename progress server => status server

Signed-off-by: Rob Bell <[email protected]>

* chore: update type doc

Signed-off-by: Rob Bell <[email protected]>

* chore: update name of feature gate

Signed-off-by: Rob Bell <[email protected]>

* fix: after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: fix typo

Signed-off-by: Rob Bell <[email protected]>

* chore: rename handler

Signed-off-by: Rob Bell <[email protected]>

* fix: run status server on all replicas

Signed-off-by: Rob Bell <[email protected]>

* fix: deep copy the rest config

Signed-off-by: Rob Bell <[email protected]>

* refactor: use ssa

Signed-off-by: Rob Bell <[email protected]>

* refactor: separate token audience per train job

Signed-off-by: Rob Bell <[email protected]>

* chore: rename status server

Signed-off-by: Rob Bell <[email protected]>

* fix: revert adding status pkg to manifest gen paths

Signed-off-by: Rob Bell <[email protected]>

* chore: update log messages

Signed-off-by: Rob Bell <[email protected]>

* chore: update api docs

Signed-off-by: Rob Bell <[email protected]>

* chore: update format

Signed-off-by: Rob Bell <[email protected]>

* fix: revert inadvertent change

Signed-off-by: Rob Bell <[email protected]>

* chore: update api name and type docs

Signed-off-by: Rob Bell <[email protected]>

* refactor: initialise oidc provider in manager startup

Signed-off-by: Rob Bell <[email protected]>

* fix: allow empty update request

Signed-off-by: Rob Bell <[email protected]>

* fix: wait for server shut down

Signed-off-by: Rob Bell <[email protected]>

* fix: handle missing train job

Signed-off-by: Rob Bell <[email protected]>

* refactor: update env var names

Signed-off-by: Rob Bell <[email protected]>

* chore: rename packages, plugins, feature gate

Signed-off-by: Rob Bell <[email protected]>

* refactor: update env var names

Signed-off-by: Rob Bell <[email protected]>

* chore: regenerate after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: fixes after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: minor doc updates

Signed-off-by: Rob Bell <[email protected]>

* chore: rename status server config

Signed-off-by: Rob Bell <[email protected]>

* chore: replace placeholder config with nil

Signed-off-by: Rob Bell <[email protected]>

* chore: revert separate runtime utils package

Signed-off-by: Rob Bell <[email protected]>

* chore: remove unused config from tests

Signed-off-by: Rob Bell <[email protected]>

---------

Signed-off-by: Rob Bell <[email protected]>
2026-03-13 20:01:08 +00:00
Andrey VelichkevichandGitHub 63d928468c feat(api): BREAKING CHANGE: Remove numProcPerNode from Torch API (#3239)
* feat(api): BREAKING CHANGE: Remove numProcPerNode from Torch API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove trainer_v1alpha1_torch_elastic_policy.py from Py models

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix numNodes validation in torchtune

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2026-03-01 06:53:03 +00:00
Andrey VelichkevichandGitHub 60004df3d3 feat(api): BREAKING CHANGE – Remove ElasticPolicy API (#3235)
Signed-off-by: Andrey Velichkevich <[email protected]>
2026-02-24 20:58:47 +00:00
Krishna GuptaandGitHub a671d3d82c fix: Enforce single ML policy constraint with CEL validation for Torch, MPI, and JAX (#3225)
* fix: enforce single ML policy constraint with CEL validation

Signed-off-by: krishna-kg732 <[email protected]>

* added plainML fallback test case

Signed-off-by: krishna-kg732 <[email protected]>

* added autogenerated files

Signed-off-by: krishna-kg732 <[email protected]>

* added integration tests

Signed-off-by: krishna-kg732 <[email protected]>

* bumped the version in charts to fix ci

Signed-off-by: krishna-kg732 <[email protected]>

* added autogenerated file

Signed-off-by: krishna-kg732 <[email protected]>

* chore: bump version

Signed-off-by: krishna-kg732 <[email protected]>

---------

Signed-off-by: krishna-kg732 <[email protected]>
2026-02-24 18:30:48 +00:00
Antonin StefanuttiandGitHub 0f9825e89e chore(operator): Use SSA throughout runtime framework (#2877)
* chore(operator): Use SSA throughout runtime framework

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix lint error

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update go.mod file

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
2025-10-31 11:43:12 +00:00
Antonin StefanuttiandGitHub 7755220575 chore(ci): Enable Kubernetes API Linter (#2858)
* chore(ci): Enable Kubernetes API Linter

Signed-off-by: Antonin Stefanutti <[email protected]>

* lint files

Signed-off-by: Antonin Stefanutti <[email protected]>

* Generate

Signed-off-by: Antonin Stefanutti <[email protected]>

* Review

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix golangci-lint target

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix commands max length

Signed-off-by: Antonin Stefanutti <[email protected]>

* Re: Fix golangci-lint target

Signed-off-by: Antonin Stefanutti <[email protected]>

* Enable the integers linter

Signed-off-by: Antonin Stefanutti <[email protected]>

* Disable maxlength linter

Signed-off-by: Antonin Stefanutti <[email protected]>

* make generate

Signed-off-by: Antonin Stefanutti <[email protected]>

* Make fields optional

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix Suspended field in JobStatus

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
2025-10-17 14:40:59 +00:00
Andrey VelichkevichandGitHub 8655798f02 fix(module): Change Go module name to v2 (#2707)
* fix(module): Change Go module name to v2

Signed-off-by: Andrey Velichkevich <[email protected]>

* Bump x/net to v0.38.0

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2025-07-03 17:46:20 +00:00
AsklvandGitHub ccbd58e4ee test(runtime): add UT for torch runtime valid function. (#2560)
Signed-off-by: Asklv <[email protected]>
2025-03-26 07:47:04 +00:00
Yuki IwaiandGitHub 50d6be8324 Construct Trainer based on trainer.kubeflow.org/trainjob-ancestor-step label (#2548)
Signed-off-by: Yuki Iwai <[email protected]>
2025-03-20 14:01:57 +00:00
Yuki IwaiandGitHub 64404f6565 Store the TrainingRuntime numNodes as runtime.Info.PodSet.Count (#2539)
Signed-off-by: Yuki Iwai <[email protected]>
2025-03-18 15:55:44 +00:00
Andrey VelichkevichandGitHub 239b27d311 feat(controller): Refactor the Initializer APIs of TrainJob (#2523)
* feat(controller): Refactor the Initializer APIs of TrainJob

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix go unit test

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix integration test

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2025-03-17 07:15:51 +00:00
Yuki IwaiandGitHub 38e16d718d Migrate InfoOptions.podSpecReplias and info.Scheduler.TotalRequests to info.TemplateSpec.PodSet (#2524)
Signed-off-by: Yuki Iwai <[email protected]>

# Conflicts:
#	pkg/runtime/core/trainingruntime.go
#	pkg/runtime/runtime.go
2025-03-16 17:09:49 +00:00
Yuki IwaiandGitHub f1a0d6178e Migrate Info.Trainer to Info.TemplateSpec.PodSet (#2520)
Signed-off-by: Yuki Iwai <[email protected]>
2025-03-15 03:34:11 +00:00
Yuki IwaiandGitHub e8d6727da6 Implemenet MPI Plugin for OpenMPI (#2493)
* Implemenet MPI Plugin for OpenMPI

Signed-off-by: Yuki Iwai <[email protected]>

* Directory pass the JobSetApplyconfiguration to RuntimeInfo

Signed-off-by: Yuki Iwai <[email protected]>

* Make repeated string as constants

Signed-off-by: Yuki Iwai <[email protected]>

* Use numNodes=1 as default mpi_distributed ClusterTrainingRuntime

Signed-off-by: Yuki Iwai <[email protected]>

* Remove unused errors

Signed-off-by: Yuki Iwai <[email protected]>

* Rename runLauncherAsWorker with runLauncherAsNode

Signed-off-by: Yuki Iwai <[email protected]>

* Fix unintended constants usage for ModelMountPath

Signed-off-by: Yuki Iwai <[email protected]>

* Rename SecretDataComparer with MPISecretDataComparer

Signed-off-by: Yuki Iwai <[email protected]>

* Add TODO STAEEMENT to deprecated env wrappers.

Signed-off-by: Yuki Iwai <[email protected]>

---------

Signed-off-by: Yuki Iwai <[email protected]>
2025-03-13 05:00:56 +00:00
Yuki IwaiandGitHub 0faab34bfa Add MPIMLPolicySource CRD defaulters (#2474)
Signed-off-by: Yuki Iwai <[email protected]>
2025-03-05 22:00:42 +00:00
Yuki IwaiandGitHub 6abdcabe9d Make MPIMLPolicySource optional fields as a pointer (#2472)
Signed-off-by: Yuki Iwai <[email protected]>
2025-03-05 12:13:07 +00:00
Yuki IwaiandGitHub a3a1324f41 Implement UTs for PlainML plugin (#2469)
Signed-off-by: Yuki Iwai <[email protected]>
2025-03-05 04:40:06 +00:00
Antonin StefanuttiandGitHub 4ad02aca05 KEP-2170: Use SSA to reconcile TrainJob components (#2431)
* KEP-2170: Use SSA to reconcile TrainJob components

Signed-off-by: Antonin Stefanutti <[email protected]>

* Enable Unstructured caching in controller manager config

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix PodGroup apply configuration

Signed-off-by: Antonin Stefanutti <[email protected]>

* API to apply config conversion util functions

Signed-off-by: Antonin Stefanutti <[email protected]>

* Only add namespace to TrainingRuntime object key

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix EnvVar upsert

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update unit tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix JobSet resource requirements

Signed-off-by: Antonin Stefanutti <[email protected]>

* Resolve build issues with launcher job

Signed-off-by: Antonin Stefanutti <[email protected]>

* Use apply config for MPI ConfigMap and Secret

Signed-off-by: Antonin Stefanutti <[email protected]>

* ComponentBuilderPlugin now returns an array

Signed-off-by: Antonin Stefanutti <[email protected]>

* Use plain apply configurations instead of unstructured

Signed-off-by: Antonin Stefanutti <[email protected]>

* Use apply config in EnforceMLPolicy plugins

Signed-off-by: Antonin Stefanutti <[email protected]>

* Do not update JobSets that are not suspended

Signed-off-by: Antonin Stefanutti <[email protected]>

* Address review feedback

Signed-off-by: Antonin Stefanutti <[email protected]>

* Do not update PodGroup if TrainJob is not suspended

Signed-off-by: Antonin Stefanutti <[email protected]>

* Remove obsolete TODO

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
2025-02-26 08:39:09 +00:00
Andrey VelichkevichandGitHub 1ff1d8ad28 Update the naming conventions for Kubeflow Trainer (#2415)
* Update the naming conventions for Kubeflow Trainer

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix webhooks

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix paths for webhooks

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update go test cmd

Signed-off-by: Andrey Velichkevich <[email protected]>

* Rename kubeflowv1 to trainer pkg

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2025-02-06 13:48:30 +00:00