Commit Graph
15 Commits
Author SHA1 Message Date
Rob BellandGitHub 78b4a00929 feat: add support for tracking TrainJob progress and training metrics (#3227)
* feat(api): add progress types to api

Signed-off-by: Rob Bell <[email protected]>

* feat: progress server stub

Signed-off-by: Rob Bell <[email protected]>

* feat: setup tls for server

Signed-off-by: Rob Bell <[email protected]>

* refactor: pull out middleware

Signed-off-by: Rob Bell <[email protected]>

* refactor: remove logger message noise from tests

Signed-off-by: Rob Bell <[email protected]>

* fix: send server error logs to main logger

Signed-off-by: Rob Bell <[email protected]>

* feat: update the trainer status

Signed-off-by: Rob Bell <[email protected]>

* feat: use separate client to avoid impacting main reconciler client-side rate limits

Signed-off-by: Rob Bell <[email protected]>

* feat: configure progress server in helm chart and configmap

Signed-off-by: Rob Bell <[email protected]>

* fix: prevent race condition

Signed-off-by: Rob Bell <[email protected]>

* feat: inject progress runtime config

Signed-off-by: Rob Bell <[email protected]>

* fix: progress client config

Signed-off-by: Rob Bell <[email protected]>

* fix: update generated config

Signed-off-by: Rob Bell <[email protected]>

* feat: validate auth tokens in endpoint

Signed-off-by: Rob Bell <[email protected]>

* test: validate auth tokens in endpoint

Signed-off-by: Rob Bell <[email protected]>

* refactor: tidy tests to use same format as other tests

Signed-off-by: Rob Bell <[email protected]>

* fix: look up the cluster issuer from the service account token

Different clusters can use different issuers so look up the issuer from the trusted in-cluster service account token.

Signed-off-by: Rob Bell <[email protected]>

* refactor: rename authn middleware

Signed-off-by: Rob Bell <[email protected]>

* feat: add authn to status update endpoint

Signed-off-by: Rob Bell <[email protected]>

* refactor: tidy up reading in-cluster service account token

Signed-off-by: Rob Bell <[email protected]>

* fix: rbac permissions

Signed-off-by: Rob Bell <[email protected]>

* fix: better error response

Signed-off-by: Rob Bell <[email protected]>

* refactor: move kubebuilder annotation higher up

Signed-off-by: Rob Bell <[email protected]>

* fix: make lastUpdateTime required

Signed-off-by: Rob Bell <[email protected]>

* chore: fix config tests after rebase

Signed-off-by: Rob Bell <[email protected]>

* test: update config test with new progress server config

Signed-off-by: Rob Bell <[email protected]>

* test: add progress tracking e2e test

Signed-off-by: Rob Bell <[email protected]>

* fix: allow feature gates to be configurable from command line argument

So they can be overridden in e2e tests

Signed-off-by: Rob Bell <[email protected]>

* refactor: reuse shared progress StatusUrl

Signed-off-by: Rob Bell <[email protected]>

* test: tidy up tests

Signed-off-by: Rob Bell <[email protected]>

* refactor: allow plugins to use config

Signed-off-by: Rob Bell <[email protected]>

* refactor: centralise shared constants

Signed-off-by: Rob Bell <[email protected]>

* fix: disable http2 in progress server tls opts if disabled by config

Signed-off-by: Rob Bell <[email protected]>

* fix: minor improvements suggested by copilot

Signed-off-by: Rob Bell <[email protected]>

* chore: rename progress server => status server

Signed-off-by: Rob Bell <[email protected]>

* chore: update type doc

Signed-off-by: Rob Bell <[email protected]>

* chore: update name of feature gate

Signed-off-by: Rob Bell <[email protected]>

* fix: after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: fix typo

Signed-off-by: Rob Bell <[email protected]>

* chore: rename handler

Signed-off-by: Rob Bell <[email protected]>

* fix: run status server on all replicas

Signed-off-by: Rob Bell <[email protected]>

* fix: deep copy the rest config

Signed-off-by: Rob Bell <[email protected]>

* refactor: use ssa

Signed-off-by: Rob Bell <[email protected]>

* refactor: separate token audience per train job

Signed-off-by: Rob Bell <[email protected]>

* chore: rename status server

Signed-off-by: Rob Bell <[email protected]>

* fix: revert adding status pkg to manifest gen paths

Signed-off-by: Rob Bell <[email protected]>

* chore: update log messages

Signed-off-by: Rob Bell <[email protected]>

* chore: update api docs

Signed-off-by: Rob Bell <[email protected]>

* chore: update format

Signed-off-by: Rob Bell <[email protected]>

* fix: revert inadvertent change

Signed-off-by: Rob Bell <[email protected]>

* chore: update api name and type docs

Signed-off-by: Rob Bell <[email protected]>

* refactor: initialise oidc provider in manager startup

Signed-off-by: Rob Bell <[email protected]>

* fix: allow empty update request

Signed-off-by: Rob Bell <[email protected]>

* fix: wait for server shut down

Signed-off-by: Rob Bell <[email protected]>

* fix: handle missing train job

Signed-off-by: Rob Bell <[email protected]>

* refactor: update env var names

Signed-off-by: Rob Bell <[email protected]>

* chore: rename packages, plugins, feature gate

Signed-off-by: Rob Bell <[email protected]>

* refactor: update env var names

Signed-off-by: Rob Bell <[email protected]>

* chore: regenerate after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: fixes after rebase

Signed-off-by: Rob Bell <[email protected]>

* chore: minor doc updates

Signed-off-by: Rob Bell <[email protected]>

* chore: rename status server config

Signed-off-by: Rob Bell <[email protected]>

* chore: replace placeholder config with nil

Signed-off-by: Rob Bell <[email protected]>

* chore: revert separate runtime utils package

Signed-off-by: Rob Bell <[email protected]>

* chore: remove unused config from tests

Signed-off-by: Rob Bell <[email protected]>

---------

Signed-off-by: Rob Bell <[email protected]>
2026-03-13 20:01:08 +00:00
Antonin StefanuttiandGitHub 9519b9960f fix(ci): Fix kube-api-linter install (#3023)
Signed-off-by: Antonin Stefanutti <[email protected]>
2025-12-09 01:46:39 +00:00
Antonin StefanuttiandGitHub 0f9825e89e chore(operator): Use SSA throughout runtime framework (#2877)
* chore(operator): Use SSA throughout runtime framework

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix lint error

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update go.mod file

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
2025-10-31 11:43:12 +00:00
9802f954ab feat: KEP-2437 - PodGroup Creation for Volcano Scheduler (#2729)
* feat: api for volcano scheduling plugin

Signed-off-by: Xinmin Du <[email protected]>

* feat: init volcano-plugin

Signed-off-by: Xinmin Du <[email protected]>

* feat: init test file

Signed-off-by: Xinmin Du <[email protected]>

* feat: register volcano plugin

Signed-off-by: Xinmin Du <[email protected]>

* feat: deal with minTaskMember, minMember, NetworkTopo

Signed-off-by: Xinmin Du <[email protected]>

* fix: calculate of minResource

Signed-off-by: Xinmin Du <[email protected]>

* test: build PodGroup test

Signed-off-by: Xinmin Du <[email protected]>

* refactor: separate to 2 prs(build&handler)

Signed-off-by: Xinmin Du <[email protected]>

* test: add test for new&reconcile_builder

Signed-off-by: Xinmin Du <[email protected]>

* fix: typo

Signed-off-by: Xinmin Du <[email protected]>

* fix: trainer/v2 import

Signed-off-by: Xinmin Du <[email protected]>

* fix: networktopo type

Signed-off-by: Xinmin Du <[email protected]>

* fix: OpenAPI validation errors

Signed-off-by: Xinmin Du <[email protected]>

* fix: remove minTaskMembers

Signed-off-by: Xinmin Du <[email protected]>

* test: test coverage 100%

Signed-off-by: Xinmin Du <[email protected]>

* feat: update apis

Signed-off-by: Xinmin Du <[email protected]>

* feat: replace testify

Signed-off-by: Xinmin Du <[email protected]>

* fix: registry Volcano CRDs to the scheme

Signed-off-by: Xinmin Du <[email protected]>

* fix: add volcano to scheme

Signed-off-by: Xinmin Du <[email protected]>

* fix: fix networktopo schema

Signed-off-by: Xinmin Du <[email protected]>

* fix: add networktopo spec in trainer

Signed-off-by: Xinmin Du <[email protected]>

* fix: unit test

Signed-off-by: Xinmin Du <[email protected]>

* feat: import networkTopo directly

Signed-off-by: Xinmin Du <[email protected]>

* fix: make generate

Signed-off-by: Xinmin Du <[email protected]>

* fix: make generate

Signed-off-by: Xinmin Du <[email protected]>

* fix: golangci-lint

Signed-off-by: Xinmin Du <[email protected]>

* fix: golangci-lint

Signed-off-by: Xinmin Du <[email protected]>

* feat: add volcano installation in integration test

Signed-off-by: Xinmin Du <[email protected]>

* fix: filter volcano api

Signed-off-by: Xinmin Du <[email protected]>

* fix: get volcano.podgroup  with local version

Signed-off-by: Xinmin Du <[email protected]>

* fix: init test env with volcano podgroup installed

Signed-off-by: Xinmin Du <[email protected]>

* fix: check plugin in enforcePodgroupPolicy

Signed-off-by: Xinmin Du <[email protected]>

* fix: group-name label in unit test

Signed-off-by: Xinmin Du <[email protected]>

* fix: ReconcilerBuilders

Signed-off-by: Xinmin Du <[email protected]>

* feat: add PodGroupHandler

Signed-off-by: Xinmin Du <[email protected]>

* feat: unit test for handlers

Signed-off-by: Xinmin Du <[email protected]>

* fix: group name annotation

Signed-off-by: Xinmin Du <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: no need to delete RBAC

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/volcano/indexer.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: nil checking for trainjob

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/volcano/volcano.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: make generate

Signed-off-by: Xinmin Du <[email protected]>

* fix: index conflict

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/coscheduling/coscheduling.go

Co-authored-by: Shao Wang <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: update volcano to v1.12.2

Signed-off-by: Xinmin Du <[email protected]>

* feat: re-use indexer

Signed-off-by: Xinmin Du <[email protected]>

* feat: add validate

Signed-off-by: Xinmin Du <[email protected]>

* fix: no scheduler when coscheduling is nil

Signed-off-by: Xinmin Du <[email protected]>

* fix: put group-name in annotations

Signed-off-by: Xinmin Du <[email protected]>

* feat: validate if priorityClass installed

Signed-off-by: Xinmin Du <[email protected]>

* feat: propagate annotations to pod

Signed-off-by: Xinmin Du <[email protected]>

* feat: integration test for volcano

Signed-off-by: Xinmin Du <[email protected]>

* fix: golangci-lint check

Signed-off-by: Xinmin Du <[email protected]>

* feat: use shared indexer

Signed-off-by: Xinmin Du <[email protected]>

* feat: remove indexer to runtime/

Signed-off-by: Xinmin Du <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: append owner reference & missing import

Signed-off-by: Xinmin Du <[email protected]>

* fix: rewrite volcano UT

Signed-off-by: Xinmin Du <[email protected]>

* feat: add copyright

Signed-off-by: Xinmin Du <[email protected]>

* fix: sync RBAC to Helm charts

Signed-off-by: Xinmin Du <[email protected]>

* fix: refactor UTs

Signed-off-by: Xinmin Du <[email protected]>

* fix: test validation separately

Signed-off-by: Xinmin Du <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: refactor TestVolcano

Signed-off-by: Xinmin Du <[email protected]>

* fix: refactor TestValidate

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/volcano/volcano_test.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

---------

Signed-off-by: Xinmin Du <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Shao Wang <[email protected]>
Co-authored-by: Yuki Iwai <[email protected]>
2025-10-06 05:51:10 +00:00
AsklvandGitHub 124e3dbcd8 chore(test): add uts for coscheduling plugin. (#2582)
* feat: add uts for coscheduling plugin.

Signed-off-by: Asklv <[email protected]>

* fix: add suspend, multiple podset, min resource case for coscheduling test.

Signed-off-by: Asklv <[email protected]>

* fix: update ut comments for coscheduling.

Signed-off-by: Asklv <[email protected]>

---------

Signed-off-by: Asklv <[email protected]>
2025-09-16 16:46:34 +00:00
Yuki IwaiandGitHub 87dbc8390e fix(manifests): Add missing permissions for the RuntimeClass and LimitRange (#2787)
Signed-off-by: Yuki Iwai <[email protected]>
2025-08-13 17:55:12 +00:00
Andrey VelichkevichandGitHub 8655798f02 fix(module): Change Go module name to v2 (#2707)
* fix(module): Change Go module name to v2

Signed-off-by: Andrey Velichkevich <[email protected]>

* Bump x/net to v0.38.0

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2025-07-03 17:46:20 +00:00
Yuki IwaiandGitHub f5a2d0f250 Implement trainer.kubeflow.org/resource-in-use finalizer mechanism to TrainingRuntime (#2608)
* Implement trainer.kubeflow.org/resource-in-use finalizer mechanism to TrainingRuntime

Signed-off-by: Yuki Iwai <[email protected]>

# Conflicts:
#	pkg/constants/constants.go

* Reduce permissions

Signed-off-by: Yuki Iwai <[email protected]>

* Dispatch reconciliations even when TrainJob is updated

Signed-off-by: Yuki Iwai <[email protected]>

---------

Signed-off-by: Yuki Iwai <[email protected]>
2025-05-01 20:17:04 +00:00
Harshal MalaniandGitHub e75a4a7e33 test(coscheduling) : add test coverage for coscheduling indexer (#2604)
* test(coscheduling) add test coverage for coscheduling indexer

Signed-off-by: Harshal292004 <[email protected]>

* update import order

Signed-off-by: Harshal292004 <[email protected]>

* updated according to review

Signed-off-by: Harshal292004 <[email protected]>

* changed DataIntailizer and ModelIntailizer with containerd

Signed-off-by: Harshal292004 <[email protected]>

---------

Signed-off-by: Harshal292004 <[email protected]>
2025-04-26 13:35:59 +00:00
Yuki IwaiandGitHub 64404f6565 Store the TrainingRuntime numNodes as runtime.Info.PodSet.Count (#2539)
Signed-off-by: Yuki Iwai <[email protected]>
2025-03-18 15:55:44 +00:00
Yuki IwaiandGitHub 38e16d718d Migrate InfoOptions.podSpecReplias and info.Scheduler.TotalRequests to info.TemplateSpec.PodSet (#2524)
Signed-off-by: Yuki Iwai <[email protected]>

# Conflicts:
#	pkg/runtime/core/trainingruntime.go
#	pkg/runtime/runtime.go
2025-03-16 17:09:49 +00:00
Andrey VelichkevichandGitHub 5401e5fec5 Bump JobSet to v0.8.0 (#2463)
Signed-off-by: Andrey Velichkevich <[email protected]>
2025-03-01 02:46:38 +00:00
Antonin StefanuttiandGitHub 4ad02aca05 KEP-2170: Use SSA to reconcile TrainJob components (#2431)
* KEP-2170: Use SSA to reconcile TrainJob components

Signed-off-by: Antonin Stefanutti <[email protected]>

* Enable Unstructured caching in controller manager config

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix PodGroup apply configuration

Signed-off-by: Antonin Stefanutti <[email protected]>

* API to apply config conversion util functions

Signed-off-by: Antonin Stefanutti <[email protected]>

* Only add namespace to TrainingRuntime object key

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix EnvVar upsert

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update unit tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix JobSet resource requirements

Signed-off-by: Antonin Stefanutti <[email protected]>

* Resolve build issues with launcher job

Signed-off-by: Antonin Stefanutti <[email protected]>

* Use apply config for MPI ConfigMap and Secret

Signed-off-by: Antonin Stefanutti <[email protected]>

* ComponentBuilderPlugin now returns an array

Signed-off-by: Antonin Stefanutti <[email protected]>

* Use plain apply configurations instead of unstructured

Signed-off-by: Antonin Stefanutti <[email protected]>

* Use apply config in EnforceMLPolicy plugins

Signed-off-by: Antonin Stefanutti <[email protected]>

* Do not update JobSets that are not suspended

Signed-off-by: Antonin Stefanutti <[email protected]>

* Address review feedback

Signed-off-by: Antonin Stefanutti <[email protected]>

* Do not update PodGroup if TrainJob is not suspended

Signed-off-by: Antonin Stefanutti <[email protected]>

* Remove obsolete TODO

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
2025-02-26 08:39:09 +00:00
Andrey VelichkevichandGitHub fa214522ce KEP-2170: Implement MPI Plugin for Kubeflow Trainer (#2394)
* Implement MPI Plugin for Kubeflow Trainer

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update RBAC

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove old manifests

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix unit test

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix comments

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2025-02-14 17:55:38 +00:00
Andrey VelichkevichandGitHub 1ff1d8ad28 Update the naming conventions for Kubeflow Trainer (#2415)
* Update the naming conventions for Kubeflow Trainer

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix webhooks

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix paths for webhooks

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update go test cmd

Signed-off-by: Andrey Velichkevich <[email protected]>

* Rename kubeflowv1 to trainer pkg

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
2025-02-06 13:48:30 +00:00