hanzo-ml/trainer
1
0
Fork 0
mirror of https://github.com/hanzo-ml/trainer.git synced 2026-09-24 00:31:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,765 Commits 2 Branches 1 Tag
master
Commit Graph
26 Commits
Author SHA1 Message Date
Antonin StefanuttiandGitHub 9519b9960f fix(ci): Fix kube-api-linter install (#3023) 
Signed-off-by: Antonin Stefanutti <[email protected]>
 2025-12-09 01:46:39 +00:00
Du XinminGitHubAndrey VelichkevichShao WangYuki Iwai
 		9802f954ab feat: KEP-2437 - PodGroup Creation for Volcano Scheduler (#2729) 
* feat: api for volcano scheduling plugin

Signed-off-by: Xinmin Du <[email protected]>

* feat: init volcano-plugin

Signed-off-by: Xinmin Du <[email protected]>

* feat: init test file

Signed-off-by: Xinmin Du <[email protected]>

* feat: register volcano plugin

Signed-off-by: Xinmin Du <[email protected]>

* feat: deal with minTaskMember, minMember, NetworkTopo

Signed-off-by: Xinmin Du <[email protected]>

* fix: calculate of minResource

Signed-off-by: Xinmin Du <[email protected]>

* test: build PodGroup test

Signed-off-by: Xinmin Du <[email protected]>

* refactor: separate to 2 prs（build&handler）

Signed-off-by: Xinmin Du <[email protected]>

* test: add test for new&reconcile_builder

Signed-off-by: Xinmin Du <[email protected]>

* fix: typo

Signed-off-by: Xinmin Du <[email protected]>

* fix: trainer/v2 import

Signed-off-by: Xinmin Du <[email protected]>

* fix: networktopo type

Signed-off-by: Xinmin Du <[email protected]>

* fix: OpenAPI validation errors

Signed-off-by: Xinmin Du <[email protected]>

* fix: remove minTaskMembers

Signed-off-by: Xinmin Du <[email protected]>

* test: test coverage 100%

Signed-off-by: Xinmin Du <[email protected]>

* feat: update apis

Signed-off-by: Xinmin Du <[email protected]>

* feat: replace testify

Signed-off-by: Xinmin Du <[email protected]>

* fix: registry Volcano CRDs to the scheme

Signed-off-by: Xinmin Du <[email protected]>

* fix: add volcano to scheme

Signed-off-by: Xinmin Du <[email protected]>

* fix: fix networktopo schema

Signed-off-by: Xinmin Du <[email protected]>

* fix: add networktopo spec in trainer

Signed-off-by: Xinmin Du <[email protected]>

* fix: unit test

Signed-off-by: Xinmin Du <[email protected]>

* feat: import networkTopo directly

Signed-off-by: Xinmin Du <[email protected]>

* fix: make generate

Signed-off-by: Xinmin Du <[email protected]>

* fix: make generate

Signed-off-by: Xinmin Du <[email protected]>

* fix: golangci-lint

Signed-off-by: Xinmin Du <[email protected]>

* fix: golangci-lint

Signed-off-by: Xinmin Du <[email protected]>

* feat: add volcano installation in integration test

Signed-off-by: Xinmin Du <[email protected]>

* fix: filter volcano api

Signed-off-by: Xinmin Du <[email protected]>

* fix: get volcano.podgroup  with local version

Signed-off-by: Xinmin Du <[email protected]>

* fix: init test env with volcano podgroup installed

Signed-off-by: Xinmin Du <[email protected]>

* fix: check plugin in enforcePodgroupPolicy

Signed-off-by: Xinmin Du <[email protected]>

* fix: group-name label in unit test

Signed-off-by: Xinmin Du <[email protected]>

* fix: ReconcilerBuilders

Signed-off-by: Xinmin Du <[email protected]>

* feat: add PodGroupHandler

Signed-off-by: Xinmin Du <[email protected]>

* feat: unit test for handlers

Signed-off-by: Xinmin Du <[email protected]>

* fix: group name annotation

Signed-off-by: Xinmin Du <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: no need to delete RBAC

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/volcano/indexer.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: nil checking for trainjob

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/volcano/volcano.go

Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: make generate

Signed-off-by: Xinmin Du <[email protected]>

* fix: index conflict

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/coscheduling/coscheduling.go

Co-authored-by: Shao Wang <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: update volcano to v1.12.2

Signed-off-by: Xinmin Du <[email protected]>

* feat: re-use indexer

Signed-off-by: Xinmin Du <[email protected]>

* feat: add validate

Signed-off-by: Xinmin Du <[email protected]>

* fix: no scheduler when coscheduling is nil

Signed-off-by: Xinmin Du <[email protected]>

* fix: put group-name in annotations

Signed-off-by: Xinmin Du <[email protected]>

* feat: validate if priorityClass installed

Signed-off-by: Xinmin Du <[email protected]>

* feat: propagate annotations to pod

Signed-off-by: Xinmin Du <[email protected]>

* feat: integration test for volcano

Signed-off-by: Xinmin Du <[email protected]>

* fix: golangci-lint check

Signed-off-by: Xinmin Du <[email protected]>

* feat: use shared indexer

Signed-off-by: Xinmin Du <[email protected]>

* feat: remove indexer to runtime/

Signed-off-by: Xinmin Du <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: append owner reference & missing import

Signed-off-by: Xinmin Du <[email protected]>

* fix: rewrite volcano UT

Signed-off-by: Xinmin Du <[email protected]>

* feat: add copyright

Signed-off-by: Xinmin Du <[email protected]>

* fix: sync RBAC to Helm charts

Signed-off-by: Xinmin Du <[email protected]>

* fix: refactor UTs

Signed-off-by: Xinmin Du <[email protected]>

* fix: test validation separately

Signed-off-by: Xinmin Du <[email protected]>

* Update hack/swagger/main.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

* fix: refactor TestVolcano

Signed-off-by: Xinmin Du <[email protected]>

* fix: refactor TestValidate

Signed-off-by: Xinmin Du <[email protected]>

* Update pkg/runtime/framework/plugins/volcano/volcano_test.go

Co-authored-by: Yuki Iwai <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>

---------

Signed-off-by: Xinmin Du <[email protected]>
Signed-off-by: Du Xinmin <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Shao Wang <[email protected]>
Co-authored-by: Yuki Iwai <[email protected]>
 2025-10-06 05:51:10 +00:00
Andrey VelichkevichandGitHub 8655798f02 fix(module): Change Go module name to v2 (#2707) 
* fix(module): Change Go module name to v2

Signed-off-by: Andrey Velichkevich <[email protected]>

* Bump x/net to v0.38.0

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2025-07-03 17:46:20 +00:00
Andrey VelichkevichandGitHub 59d2d31a7f feat(sdk): Migrate to OpenAPI V3 (#2490) 
* feat(sdk): Migrate to OpenAPI V3

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update SDK to support OpenAPI V3

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove metadata check

Signed-off-by: Andrey Velichkevich <[email protected]>

* Assign nil for empty default value

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2025-03-10 13:45:02 +00:00
Andrey VelichkevichandGitHub 0b5d3be8f1 feat(sdk): Generate external Kubernetes and JobSet models (#2466) 
* feat(sdk): Generate external Kubernetes and JobSet models

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update JobSet commit in Swagger

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix formats for resources and num proc

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add custom Kind config

Signed-off-by: Andrey Velichkevich <[email protected]>

* Use ginkgo to run e2e

Signed-off-by: Andrey Velichkevich <[email protected]>

* Revert Kind config change

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix ginkgo binary

Signed-off-by: Andrey Velichkevich <[email protected]>

* Print controller logs in case of failure

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2025-03-05 19:08:21 +00:00
Andrey VelichkevichandGitHub 1ff1d8ad28 Update the naming conventions for Kubeflow Trainer (#2415) 
* Update the naming conventions for Kubeflow Trainer

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix webhooks

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix paths for webhooks

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update go test cmd

Signed-off-by: Andrey Velichkevich <[email protected]>

* Rename kubeflowv1 to trainer pkg

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2025-02-06 13:48:30 +00:00
Andrey VelichkevichandGitHub 812c308cc2 Remove the Training Operator V1 Source Code (#2389) 
* Remove Training Operator V1 files

Signed-off-by: Andrey Velichkevich <[email protected]>

* Refactor README and Makefile

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove v1 GitHub workflows

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove finish action from Go

Signed-off-by: Andrey Velichkevich <[email protected]>

* Rename Kubeflow Training to Kubeflow Trainer

Signed-off-by: Andrey Velichkevich <[email protected]>

* Rename comment

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add steps for free-up space

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update Python tests

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove integration workflow

Signed-off-by: Andrey Velichkevich <[email protected]>

* Install SDK in Python integration tests

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove generate file

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add Kubeflow Trainer logo

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add tech diagram

Signed-off-by: Andrey Velichkevich <[email protected]>

* Exclude images from pre-commit

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix git diff in CI

Signed-off-by: Andrey Velichkevich <[email protected]>

* Run go mod tidy

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update diagram

Signed-off-by: Andrey Velichkevich <[email protected]>

* Cleanup examples

Signed-off-by: Andrey Velichkevich <[email protected]>

* Always install Go bins

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update diagram

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update hack
Remove manifests for JobSet
Rename to tools.go

Signed-off-by: Andrey Velichkevich <[email protected]>

* Create tools.go

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2025-02-04 15:25:38 +00:00
Yuki IwaiandGitHub d5330e89cb Upgrade Go version to v1.23 (#2302) 
Signed-off-by: Yuki Iwai <[email protected]>
 2024-12-13 19:22:06 +00:00
Antonin StefanuttiandGitHub 7b68a4458c Upgrade Kubernetes to v1.31.3 (#2330) 
* Upgrade Kubernetes to v1.31.3

Signed-off-by: Antonin Stefanutti <[email protected]>

* Skip controller names uniqueness check for tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Move cache argument from Registry to ReconcilerBuilder API

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2024-12-09 18:17:04 +00:00
Antonin StefanuttiandGitHub 3801b090d6 Upgrade Kubernetes to v1.30.7 (#2332) 
* Upgrade Kubernetes to v1.30.7

Signed-off-by: Antonin Stefanutti <[email protected]>

* Use typed event handlers and predicates in job controllers

Signed-off-by: Antonin Stefanutti <[email protected]>

* Re-organize pkg/common/util/reconciler.go

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update installation instructions in README

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2024-11-27 15:05:54 +00:00
Yuki IwaiandGitHub 8d84c14962 KEP-2170: Implement runtime framework (#2248) 
* KEP-2170: Implement runtime framework interfaces

Signed-off-by: Yuki Iwai <[email protected]>

* Remove grep dependency

Signed-off-by: Yuki Iwai <[email protected]>

* KEP-2170: Implement ValidateObjects interface to the runtime framework

Signed-off-by: Yuki Iwai <[email protected]>

* KEP-2170: Expose the TrainingRuntime and ClusterTrainingRuntime Kind

Signed-off-by: Yuki Iwai <[email protected]>

* KEP-2170: Remove unneeded scheme field from the internal TrainingRuntime

Signed-off-by: Yuki Iwai <[email protected]>

* Rephrase the error message

Signed-off-by: Yuki Iwai <[email protected]>

* Distinguish TrainingRuntime and ClusterTrainingRuntime when creating indexes for the TrainJobs

Signed-off-by: Yuki Iwai <[email protected]>

* Propagate the TrainJob labels and annotations to the JobSet

Signed-off-by: Yuki Iwai <[email protected]>

* Remove PodAnnotations from the runtime info

Signed-off-by: Yuki Iwai <[email protected]>

* Implement TrainingRuntime ReplicatedJob validation

Signed-off-by: Yuki Iwai <[email protected]>

* Add TODO comments

Signed-off-by: Yuki Iwai <[email protected]>

* Replace queueSuspendedTrainJob with queueSuspendedTrainJobs

Signed-off-by: Yuki Iwai <[email protected]>

---------

Signed-off-by: Yuki Iwai <[email protected]>
 2024-10-17 14:14:19 +00:00
Yuki IwaiandGitHub ba71ab0a0c Clean up Go modules (#2238) 
Signed-off-by: Yuki Iwai <[email protected]>
 2024-08-28 02:30:02 +00:00
Yuki IwaiandGitHub 6190352b06 Upgrade scheduler-plugins to v0.28.9 (#2065) 
Signed-off-by: Yuki Iwai <[email protected]>
 2024-04-18 06:52:39 +00:00
Yuki IwaiandGitHub fb721d3a4b Upgrade Go version to v1.22 (#2046) 
Signed-off-by: Yuki Iwai <[email protected]>
 2024-04-07 00:56:26 +00:00
Yuki IwaiandGitHub 7b09fd0449 Support K8s v1.29 and Drop K8s v1.26 (#2039) 
Signed-off-by: Yuki Iwai <[email protected]>
 2024-03-28 21:02:18 +00:00
Yuki IwaiandGitHub 5779f6c384 Support K8s v1.28 and Drop K8s v1.25 (#2038) 
Signed-off-by: Yuki Iwai <[email protected]>
 2024-03-28 19:59:18 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 		93878c7664 Bump google.golang.org/protobuf from 1.30.0 to 1.33.0 in /hack/swagger (#2030) 
Bumps google.golang.org/protobuf from 1.30.0 to 1.33.0.

---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-25 15:40:18 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 		d6576c80cc Bump golang.org/x/net from 0.13.0 to 0.17.0 in /hack/swagger (#1933) 
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.13.0 to 0.17.0.
- [Commits](https://github.com/golang/net/compare/v0.13.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-10-12 18:36:21 +00:00
Andreas FritzlerandGitHub 6f40f317e0 Bump k8s.io/* deps to 1.28 (#1920) 
* Bump k8s.io/* deps to 1.28

- Bump k8s.io/* deps to 1.28
- Fix metrics bind address assignment in manager setup
- Rename metrics-port flag to webhook-server-port as it was wrongly used

* Revert envtest 1.27 and use generate-groups.sh

- Revert envtest to 1.27
- Use generate-groups.sh instead of kube_codegen.sh

* Revert monitoring port flag
 2023-09-29 15:24:42 +00:00
Yuki IwaiandGitHub e8db129ba0 Remove klog v1 (#1886) 
Signed-off-by: Yuki Iwai <[email protected]>
 2023-08-14 06:14:26 +00:00
Yuki IwaiandGitHub a37451b513 Upgrade Go version to v1.20 (#1873) 
Signed-off-by: Yuki Iwai <[email protected]>
 2023-08-01 15:01:07 +00:00
Johnu GeorgeGitHubYuki Iwaiyu lin
 		c320716a1a Merge kubeflow/common to training-operator (#1813) 
* Support for k8s v1.25 in CI

* Support for k8s v1.25 in CI

* Change k8s api to v1.25

* Upgrade golangci-lint version

* Add changelog

* Update CHANGELOG.md

* Update Changelog

* Merge common repo

* Avoid to depend on local env when installing the code-generators (#1810)

Signed-off-by: Yuki Iwai <[email protected]>

* Make scheduler-plugins the default gang scheduler. (#1747)

Signed-off-by: Syulin7 <[email protected]>

* Fix tests

* Fix merge conflicts

* Fix CI issues

* Fix CI issues

* Fix review comments

* Add contributors in Readme file

* Fix review comments

* Fix review comments

---------

Signed-off-by: Yuki Iwai <[email protected]>
Signed-off-by: Syulin7 <[email protected]>
Co-authored-by: Yuki Iwai <[email protected]>
Co-authored-by: yu lin <[email protected]>
 2023-05-31 11:04:04 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 		8c4e56db09 Bump golang.org/x/net from 0.0.0-20220607020251-c690dde0001d to 0.7.0 in /hack/swagger (#1766) 
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.0.0-20220607020251-c690dde0001d to 0.7.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](https://github.com/golang/net/commits/v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-03-27 05:28:24 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 		bf5e63d407 Bump github.com/emicklei/go-restful/v3 from 3.7.5-0.20220308211933-7c971ca4d0fd to 3.8.0 in /hack/swagger (#1770) 
Bumps [github.com/emicklei/go-restful/v3](https://github.com/emicklei/go-restful) from 3.7.5-0.20220308211933-7c971ca4d0fd to 3.8.0.
- [Release notes](https://github.com/emicklei/go-restful/releases)
- [Changelog](https://github.com/emicklei/go-restful/blob/v3/CHANGES.md)
- [Commits](https://github.com/emicklei/go-restful/commits/v3.8.0)

---
updated-dependencies:
- dependency-name: github.com/emicklei/go-restful/v3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-03-27 04:34:24 +00:00
Yuki IwaiandGitHub 2b6dbf86d9 Upgrade Go version to v1.19 (#1663) 
Signed-off-by: tenzen-y <[email protected]>

Signed-off-by: tenzen-y <[email protected]>
 2022-09-21 21:54:38 +00:00
Yuki IwaiandGitHub 0166d87791 Add clientset for MPIJob, PytorchJob, MXJob, and XGBoostJob (#1610) 2022-06-14 04:20:31 +00:00