15 Commits
Author SHA1 Message Date
Rob BellandGitHub 78b4a00929 feat: add support for tracking TrainJob progress and training metrics (#3227)
* feat(api): add progress types to api

Signed-off-by: Rob Bell <[email protected]>

* feat: progress server stub

Signed-off-by: Rob Bell <[email protected]>

* feat: setup tls for server

Signed-off-by: Rob Bell <[email protected]>

* refactor: pull out middleware

Signed-off-by: Rob Bell <[email protected]>

* refactor: remove logger message noise from tests

Signed-off-by: Rob Bell <[email protected]>

* fix: send server error logs to main logger

Signed-off-by: Rob Bell <[email protected]>

* feat: update the trainer status

Signed-off-by: Rob Bell <[email protected]>

* feat: use separate client to avoid impacting main reconciler client-side rate limits

Signed-off-by: Rob Bell <[email protected]>

* feat: configure progress server in helm chart and configmap

Signed-off-by: Rob Bell <[email protected]>

* fix: prevent race condition

Signed-off-by: Rob Bell <[email protected]>

* feat: inject progress runtime config

Signed-off-by: Rob Bell <[email protected]>

* fix: progress client config

Signed-off-by: Rob Bell <[email protected]>

* fix: update generated config

Signed-off-by: Rob Bell <[email protected]>

* feat: validate auth tokens in endpoint

Signed-off-by: Rob Bell <[email protected]>

* test: validate auth tokens in endpoint

Signed-off-by: Rob Bell <[email protected]>

* refactor: tidy tests to use same format as other tests

Signed-off-by: Rob Bell <[email protected]>

* fix: look up the cluster issuer from the service account token

Different clusters can use different issuers so look up the issuer from the trusted in-cluster service account token.

Signed-off-by: Rob Bell <[email protected]>

* refactor: rename authn middleware

Signed-off-by: Rob Bell <[email protected]>

* feat: add authn to status update endpoint

Signed-off-by: Rob Bell <[email protected]>

* refactor: tidy up reading in-cluster service account token

Signed-off-by: Rob Bell <[email protected]>

* fix: rbac permissions

Signed-off-by: Rob Bell <[email protected]>

* fix: better error response

Signed-off-by: Rob Bell <[email protected]>

* refactor: move kubebuilder annotation higher up

Signed-off-by: Rob Bell <[email protected]>

* fix: make lastUpdateTime required

Signed-off-by: Rob Bell <[email protected]>

* chore: fix config tests after rebase

Signed-off-by: Rob Bell <[email protected]>

* test: update config test with new progress server config

Signed-off-by: Rob Bell <[email protected]>

* test: add progress tracking e2e test

Signed-off-by: Rob Bell <[email protected]>

* fix: allow feature gates to be configurable from command line argument

So they can be overridden in e2e tests

Signed-off-by: Rob Bell <[email protected]>

* refactor: reuse shared progress StatusUrl

Signed-off-by: Rob Bell <[email protected]>

* test: tidy up tests

Signed-off-by: Rob Bell <[email protected]>

* refactor: allow plugins to use config

Signed-off-by: Rob Bell <[email protected]>

* refactor: centralise shared constants

Signed-off-by: Rob Bell <[email protected]>

* fix: disable http2 in progress server tls opts if disabled by config

Signed-off-by: Rob Bell <[email protected]>

* fix: minor improvements suggested by copilot

Signed-off-by: Rob Bell <[email protected]>

* chore: rename progress server => status server

Signed-off-by: Rob Bell <[email protected]>

* chore: update type doc

Signed-off-by: Rob Bell <[email protected]>

* chore: update name of feature gate

Signed-off-by: Rob Bell <[email protected]>

* fix: after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: fix typo

Signed-off-by: Rob Bell <[email protected]>

* chore: rename handler

Signed-off-by: Rob Bell <[email protected]>

* fix: run status server on all replicas

Signed-off-by: Rob Bell <[email protected]>

* fix: deep copy the rest config

Signed-off-by: Rob Bell <[email protected]>

* refactor: use ssa

Signed-off-by: Rob Bell <[email protected]>

* refactor: separate token audience per train job

Signed-off-by: Rob Bell <[email protected]>

* chore: rename status server

Signed-off-by: Rob Bell <[email protected]>

* fix: revert adding status pkg to manifest gen paths

Signed-off-by: Rob Bell <[email protected]>

* chore: update log messages

Signed-off-by: Rob Bell <[email protected]>

* chore: update api docs

Signed-off-by: Rob Bell <[email protected]>

* chore: update format

Signed-off-by: Rob Bell <[email protected]>

* fix: revert inadvertent change

Signed-off-by: Rob Bell <[email protected]>

* chore: update api name and type docs

Signed-off-by: Rob Bell <[email protected]>

* refactor: initialise oidc provider in manager startup

Signed-off-by: Rob Bell <[email protected]>

* fix: allow empty update request

Signed-off-by: Rob Bell <[email protected]>

* fix: wait for server shut down

Signed-off-by: Rob Bell <[email protected]>

* fix: handle missing train job

Signed-off-by: Rob Bell <[email protected]>

* refactor: update env var names

Signed-off-by: Rob Bell <[email protected]>

* chore: rename packages, plugins, feature gate

Signed-off-by: Rob Bell <[email protected]>

* refactor: update env var names

Signed-off-by: Rob Bell <[email protected]>

* chore: regenerate after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: fixes after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: minor doc updates

Signed-off-by: Rob Bell <[email protected]>

* chore: rename status server config

Signed-off-by: Rob Bell <[email protected]>

* chore: replace placeholder config with nil

Signed-off-by: Rob Bell <[email protected]>

* chore: revert separate runtime utils package

Signed-off-by: Rob Bell <[email protected]>

* chore: remove unused config from tests

Signed-off-by: Rob Bell <[email protected]>

---------

Signed-off-by: Rob Bell <[email protected]>
2026-03-13 20:01:08 +00:00
9802f954ab feat: KEP-2437 - PodGroup Creation for Volcano Scheduler (#2729)
* feat: api for volcano scheduling plugin

Signed-off-by: Xinmin Du <[email protected]>

* feat: init volcano-plugin

Signed-off-by: Xinmin Du <[email protected]>

* feat: init test file

Signed-off-by: Xinmin Du <[email protected]>

* feat: register volcano plugin

Signed-off-by: Xinmin Du <[email protected]>

* feat: deal with minTaskMember, minMember, NetworkTopo

Signed-off-by: Xinmin Du <[email protected]>

* fix: calculate of minResource

Signed-off-by: Xinmin Du <[email protected]>

* test: build PodGroup test

Signed-off-by: Xinmin Du <[email protected]>

* refactor: separate to 2 prs(build&handler)

Signed-off-by: Xinmin Du <[email protected]>

* test: add test for new&reconcile_builder

Signed-off-by: Xinmin Du <[email protected]>

* fix: typo

Signed-off-by: Xinmin Du <[email protected]>

* fix: trainer/v2 import

Signed-off-by: Xinmin Du <[email protected]>

* fix: networktopo type

Signed-off-by: Xinmin Du <[email protected]>

* fix: OpenAPI validation errors

Signed-off-by: Xinmin Du <[email protected]>

* fix: remove minTaskMembers

Signed-off-by: Xinmin Du <[email protected]>

* test: test coverage 100%

Signed-off-by: Xinmin Du <[email protected]>

* feat: update apis

Signed-off-by: Xinmin Du <[email protected]>

* feat: replace testify

Signed-off-by: Xinmin Du <[email protected]>

* fix: registry Volcano CRDs to the scheme

Signed-off-by: Xinmin Du <[email protected]>

* fix: add volcano to scheme

Signed-off-by: Xinmin Du <[email protected]>

* fix: fix networktopo schema

Signed-off-by: Xinmin Du <[email protected]>

* fix: add networktopo spec in trainer

Signed-off-by: Xinmin Du <[email protected]>

* fix: unit test

Signed-off-by: Xinmin Du <[email protected]>

* feat: import networkTopo directly

Signed-off-by: Xinmin Du <[email protected]>

* fix: make generate

Signed-off-by: Xinmin Du <[email protected]>

* fix: make generate

Signed-off-by: Xinmin Du <[email protected]>

* fix: golangci-lint

Signed-off-by: Xinmin Du <[email protected]>

* fix: golangci-lint

Signed-off-by: Xinmin Du <[email protected]>

* feat: add volcano installation in integration test

Signed-off-by: Xinmin Du <[email protected]>

* fix: filter volcano api

Signed-off-by: Xinmin Du <[email protected]>

* fix: get volcano.podgroup  with local version

Signed-off-by: Xinmin Du <[email protected]>

* fix: init test env with volcano podgroup installed

Signed-off-by: Xinmin Du <[email protected]>

* fix: check plugin in enforcePodgroupPolicy

Signed-off-by: Xinmin Du <[email protected]>

* fix: group-name label in unit test

Signed-off-by: Xinmin Du <[email protected]>

* fix: ReconcilerBuilders

Signed-off-by: Xinmin Du <[email protected]>

* feat: add PodGroupHandler

Signed-off-by: Xinmin Du <[email protected]>

* feat: unit test for handlers

Signed-off-by: Xinmin Du <[email protected]>

* fix: group name annotation

Signed-off-by: Xinmin Du <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: no need to delete RBAC

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/volcano/indexer.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: nil checking for trainjob

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/volcano/volcano.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: make generate

Signed-off-by: Xinmin Du <[email protected]>

* fix: index conflict

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/coscheduling/coscheduling.go

Co-authored-by: Shao Wang <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: update volcano to v1.12.2

Signed-off-by: Xinmin Du <[email protected]>

* feat: re-use indexer

Signed-off-by: Xinmin Du <[email protected]>

* feat: add validate

Signed-off-by: Xinmin Du <[email protected]>

* fix: no scheduler when coscheduling is nil

Signed-off-by: Xinmin Du <[email protected]>

* fix: put group-name in annotations

Signed-off-by: Xinmin Du <[email protected]>

* feat: validate if priorityClass installed

Signed-off-by: Xinmin Du <[email protected]>

* feat: propagate annotations to pod

Signed-off-by: Xinmin Du <[email protected]>

* feat: integration test for volcano

Signed-off-by: Xinmin Du <[email protected]>

* fix: golangci-lint check

Signed-off-by: Xinmin Du <[email protected]>

* feat: use shared indexer

Signed-off-by: Xinmin Du <[email protected]>

* feat: remove indexer to runtime/

Signed-off-by: Xinmin Du <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: append owner reference & missing import

Signed-off-by: Xinmin Du <[email protected]>

* fix: rewrite volcano UT

Signed-off-by: Xinmin Du <[email protected]>

* feat: add copyright

Signed-off-by: Xinmin Du <[email protected]>

* fix: sync RBAC to Helm charts

Signed-off-by: Xinmin Du <[email protected]>

* fix: refactor UTs

Signed-off-by: Xinmin Du <[email protected]>

* fix: test validation separately

Signed-off-by: Xinmin Du <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: refactor TestVolcano

Signed-off-by: Xinmin Du <[email protected]>

* fix: refactor TestValidate

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/volcano/volcano_test.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

---------

Signed-off-by: Xinmin Du <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Shao Wang <[email protected]>
Co-authored-by: Yuki Iwai <[email protected]>
2025-10-06 05:51:10 +00:00
Andrey VelichkevichandGitHub 8655798f02 fix(module): Change Go module name to v2 (#2707)
* fix(module): Change Go module name to v2

Signed-off-by: Andrey Velichkevich <[email protected]>

* Bump x/net to v0.38.0

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2025-07-03 17:46:20 +00:00
Andrey VelichkevichandGitHub 239b27d311 feat(controller): Refactor the Initializer APIs of TrainJob (#2523)
* feat(controller): Refactor the Initializer APIs of TrainJob

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix go unit test

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix integration test

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2025-03-17 07:15:51 +00:00
a589903c24 KEP-2170: Adding validation webhook for v2 trainjob (#2307)
Signed-off-by: Akshay Chitneni <[email protected]>
Co-authored-by: Akshay Chitneni <[email protected]>
2025-03-16 03:41:51 +00:00
Andrey VelichkevichandGitHub 1ff1d8ad28 Update the naming conventions for Kubeflow Trainer (#2415)
* Update the naming conventions for Kubeflow Trainer

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix webhooks

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix paths for webhooks

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update go test cmd

Signed-off-by: Andrey Velichkevich <[email protected]>

* Rename kubeflowv1 to trainer pkg

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2025-02-06 13:48:30 +00:00
Shao WangandGitHub 7777f61acb KEP-2170: Change API Group Name to trainer.kubeflow.org (#2413)
* fix(apis): change the group of API to trainer.kubeflow.org.

Signed-off-by: Electronic-Waste <[email protected]>

* chore(manifests): update crds in manifests using make manifests.

Signed-off-by: Electronic-Waste <[email protected]>

* chore: change the apia dir name to trainer.kubeflow.org and update reference.

Signed-off-by: Electronic-Waste <[email protected]>

* chore: execute make generate.

Signed-off-by: Electronic-Waste <[email protected]>

* fix: remove remaining kubeflow.org dirs.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(sdk): remove outdated docs & update models reference.

Signed-off-by: Electronic-Waste <[email protected]>

* fix: rename apis dir to ttrainer.

Signed-off-by: Electronic-Waste <[email protected]>

* chore: execute make generate.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(sdk): remove outdated docs & update models reference.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(sdk): update model reference in code.

Signed-off-by: Electronic-Waste <[email protected]>

* fix(doc): update api grou p in KEP-2170.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
2025-02-05 13:34:37 +00:00
Andrey VelichkevichandGitHub 812c308cc2 Remove the Training Operator V1 Source Code (#2389)
* Remove Training Operator V1 files

Signed-off-by: Andrey Velichkevich <[email protected]>

* Refactor README and Makefile

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove v1 GitHub workflows

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove finish action from Go

Signed-off-by: Andrey Velichkevich <[email protected]>

* Rename Kubeflow Training to Kubeflow Trainer

Signed-off-by: Andrey Velichkevich <[email protected]>

* Rename comment

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add steps for free-up space

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update Python tests

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove integration workflow

Signed-off-by: Andrey Velichkevich <[email protected]>

* Install SDK in Python integration tests

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove generate file

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add Kubeflow Trainer logo

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add tech diagram

Signed-off-by: Andrey Velichkevich <[email protected]>

* Exclude images from pre-commit

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix git diff in CI

Signed-off-by: Andrey Velichkevich <[email protected]>

* Run go mod tidy

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update diagram

Signed-off-by: Andrey Velichkevich <[email protected]>

* Cleanup examples

Signed-off-by: Andrey Velichkevich <[email protected]>

* Always install Go bins

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update diagram

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update hack
Remove manifests for JobSet
Rename to tools.go

Signed-off-by: Andrey Velichkevich <[email protected]>

* Create tools.go

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2025-02-04 15:25:38 +00:00
Antonin StefanuttiandGitHub 7b68a4458c Upgrade Kubernetes to v1.31.3 (#2330)
* Upgrade Kubernetes to v1.31.3

Signed-off-by: Antonin Stefanutti <[email protected]>

* Skip controller names uniqueness check for tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Move cache argument from Registry to ReconcilerBuilder API

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
2024-12-09 18:17:04 +00:00
Yuki IwaiandGitHub 39925d84d6 KEP-2170: Add TrainJob conditions (#2322)
* KEP-2170: Implement TrainJob conditions

Signed-off-by: Yuki Iwai <[email protected]>

* Fix API comments

Signed-off-by: Yuki Iwai <[email protected]>

* Make condition message constants

Signed-off-by: Yuki Iwai <[email protected]>

* Stop connecting condition type and reason in JobSet plugin

Signed-off-by: Yuki Iwai <[email protected]>

---------

Signed-off-by: Yuki Iwai <[email protected]>
2024-11-09 18:14:15 +00:00
Yuki IwaiandGitHub 4af75710ee KEP-2170: Implement TrainJob Reconciler to manage objects (#2295)
* KEP-2170: Implement TrainJob Reconciler to manage objects

Signed-off-by: Yuki Iwai <[email protected]>

* Mode dep-crds to manifests/external-crds

Signed-off-by: Yuki Iwai <[email protected]>

* Rename run with runtime

Signed-off-by: Yuki Iwai <[email protected]>

---------

Signed-off-by: Yuki Iwai <[email protected]>
2024-10-23 02:11:29 +00:00
Andrey VelichkevichandGitHub dd66c2cfa8 KEP-2170: Add manifests for Kubeflow Training V2 (#2289)
* KEP-2170: Add manifests for Kubeflow Training V2

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix invalid name for webhook config in cert

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix integration tests

Signed-off-by: Andrey Velichkevich <[email protected]>

* Move kubebuilder markers to runtime framework

Signed-off-by: Andrey Velichkevich <[email protected]>

* Use Kubernetes recommended labels

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2024-10-21 19:33:28 +00:00
Yuki IwaiandGitHub 8d84c14962 KEP-2170: Implement runtime framework (#2248)
* KEP-2170: Implement runtime framework interfaces

Signed-off-by: Yuki Iwai <[email protected]>

* Remove grep dependency

Signed-off-by: Yuki Iwai <[email protected]>

* KEP-2170: Implement ValidateObjects interface to the runtime framework

Signed-off-by: Yuki Iwai <[email protected]>

* KEP-2170: Expose the TrainingRuntime and ClusterTrainingRuntime Kind

Signed-off-by: Yuki Iwai <[email protected]>

* KEP-2170: Remove unneeded scheme field from the internal TrainingRuntime

Signed-off-by: Yuki Iwai <[email protected]>

* Rephrase the error message

Signed-off-by: Yuki Iwai <[email protected]>

* Distinguish TrainingRuntime and ClusterTrainingRuntime when creating indexes for the TrainJobs

Signed-off-by: Yuki Iwai <[email protected]>

* Propagate the TrainJob labels and annotations to the JobSet

Signed-off-by: Yuki Iwai <[email protected]>

* Remove PodAnnotations from the runtime info

Signed-off-by: Yuki Iwai <[email protected]>

* Implement TrainingRuntime ReplicatedJob validation

Signed-off-by: Yuki Iwai <[email protected]>

* Add TODO comments

Signed-off-by: Yuki Iwai <[email protected]>

* Replace queueSuspendedTrainJob with queueSuspendedTrainJobs

Signed-off-by: Yuki Iwai <[email protected]>

---------

Signed-off-by: Yuki Iwai <[email protected]>
2024-10-17 14:14:19 +00:00
Yuki IwaiandGitHub 2fc0a796a0 KEP-2170: Implement skeleton webhook servers (#2251)
Signed-off-by: Yuki Iwai <[email protected]>
2024-09-09 20:12:19 +00:00
Yuki IwaiandGitHub 5c4259f5c1 KEP-2170: Initial Implementations for v2 Manager (#2236)
Signed-off-by: Yuki Iwai <[email protected]>
2024-09-06 14:10:16 +00:00