hanzo-ml/trainer
1
0
Fork 0
mirror of https://github.com/hanzo-ml/trainer.git synced 2026-09-24 00:31:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,101 Commits 2 Branches 1 Tag
dc299c59cd6ed6bdda8ea0587e7bbf6f9cc67406
Commit Graph
116 Commits
Author SHA1 Message Date
Antonin StefanuttiandGitHub 3801b090d6 Upgrade Kubernetes to v1.30.7 (#2332) 
* Upgrade Kubernetes to v1.30.7

Signed-off-by: Antonin Stefanutti <[email protected]>

* Use typed event handlers and predicates in job controllers

Signed-off-by: Antonin Stefanutti <[email protected]>

* Re-organize pkg/common/util/reconciler.go

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update installation instructions in README

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Antonin Stefanutti <[email protected]>
 2024-11-27 15:05:54 +00:00
Wei-Cheng LaiandGitHub 8480fd427b [SDK] test: add unit test for get_job_logs method of the training_client (#2275) 
Signed-off-by: wei-chenglai <[email protected]>
 2024-10-12 02:25:10 +00:00
Andrey VelichkevichandGitHub 5951386612 [SDK] Use torchrun to create PyTorchJob from function (#2276) 
* [SDK] Use torchrun to create PyTorchJob from function

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update PyTorchJob SDK example

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add consts for entrypoint

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add check for num procs per worker

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-10-11 18:52:11 +00:00
Wei-Cheng LaiandGitHub dd50cc8046 [SDK] test: add unit test for list_jobs method of the training_client (#2267) 
Signed-off-by: wei-chenglai <[email protected]>
 2024-10-03 15:29:03 +00:00
Mark CampbellandGitHub 8b2e410267 [SDK] Training Client Conditions related unit tests (#2253) 
* test: add unit test for get_job_conditions function of training client

Signed-off-by: Bobbins228 <[email protected]>

* test: add unit test for is_job_created function of training client

Signed-off-by: Bobbins228 <[email protected]>

* test: add unit test for is_job_running function of training client

Signed-off-by: Bobbins228 <[email protected]>

* test: add unit test for is_job_restarting function of training client

Signed-off-by: Bobbins228 <[email protected]>

* test: add unit test for is_job_failed function of training client

Signed-off-by: Bobbins228 <[email protected]>

* test: add unit test for is_job_succeded function of training client

Signed-off-by: Bobbins228 <[email protected]>

* test: improve job condition unit tests efficiency

Signed-off-by: Bobbins228 <[email protected]>

---------

Signed-off-by: Bobbins228 <[email protected]>
 2024-09-30 21:41:39 +00:00
saileshd1402andGitHub f08234af9e [SDK] Minor fix in wait_for_job_conditions with job_kind python training API (#2265) 
* Bug fix regarding job kind in wait_for_job_conditions python sdk

Signed-off-by: sailesh duddupudi <[email protected]>

* Linting update

Signed-off-by: sailesh duddupudi <[email protected]>

---------

Signed-off-by: sailesh duddupudi <[email protected]>
 2024-09-23 19:51:33 +00:00
VarshaandGitHub 529fa5b12d [SDK] move env var to constants.py (#2268) 
Signed-off-by: Varsha Prasad Narsing <[email protected]>
 2024-09-23 16:08:33 +00:00
Sandipan PandaandGitHub 9422bd3baa Update JAX image to use image published by Kubeflow (#2264) 
* Use JAX image published by Kubeflow

Signed-off-by: Sandipan Panda <[email protected]>

* Update README to include JAX API Definition

Signed-off-by: Sandipan Panda <[email protected]>

---------

Signed-off-by: Sandipan Panda <[email protected]>
 2024-09-20 17:33:29 +00:00
Sandipan PandaandGitHub 4ac84cd4bc Add JAX controller (#2194) 
* Add JAX controller

Add JAX controller, controller tests, webhook validations, examples,
e2e tests for JAXJob

Extend the Training Operator Python SDK to simplify the creation and
management of JAXJob resources.

Signed-off-by: Sandipan Panda <[email protected]>

* Add envvar tests

Signed-off-by: Sandipan Panda <[email protected]>

---------

Signed-off-by: Sandipan Panda <[email protected]>
 2024-09-20 14:54:30 +00:00
VarshaandGitHub e18b1be01c [SDK] Allow customising base trainer and storage images in Train API (#2261) 
Allow customizing base storage_initializer and trainer images through
Env vars.

Signed-off-by: Varsha Prasad Narsing <[email protected]>
 2024-09-19 08:48:28 +00:00
Michał SzadkowskiandGitHub 26c9112c07 [Feature] Support managed by external controller (#2203) 
* Introduce ManagedBy field to RunPolicy that is used by each Kubeflow Job Spec

Signed-off-by: Michal Szadkowski <[email protected]>

* Update Kubeflow JOb manifests

Signed-off-by: Michal Szadkowski <[email protected]>

* Update Kubeflow Jobs Reconcile to use ManagedBy field to decide if skip the process

Signed-off-by: Michal Szadkowski <[email protected]>

* job controller test

Signed-off-by: Michal Szadkowski <[email protected]>

* spec validation webhook

Signed-off-by: Michal Szadkowski <[email protected]>

* add manageBy maxLenght const

Signed-off-by: Michal Szadkowski <[email protected]>

* generate new manifest

Signed-off-by: Michal Szadkowski <[email protected]>

* revert webhook formatting

Signed-off-by: Michal Szadkowski <[email protected]>

* Move allowed controllers constants in one place

Signed-off-by: Michal Szadkowski <[email protected]>

* Make validatation for allowed managedBy values

Signed-off-by: Michal Szadkowski <[email protected]>

* Update after controllers constants move

Signed-off-by: Michal Szadkowski <[email protected]>

* Update jobs controller tests

Signed-off-by: Michal Szadkowski <[email protected]>

* Update validateManagedBy webhook

Signed-off-by: Michal Szadkowski <[email protected]>

* Remove validation for the length of ManagedBy field

Signed-off-by: Michal Szadkowski <[email protected]>

* Update after code review

Signed-off-by: Michal Szadkowski <[email protected]>

* Update ManagedBy comment

Signed-off-by: Michal Szadkowski <[email protected]>

* E2E tests for managedBy

Signed-off-by: Michal Szadkowski <[email protected]>

* Update generated files and manifests

Signed-off-by: Michal Szadkowski <[email protected]>

* Rework after code review

Signed-off-by: Michal Szadkowski <[email protected]>

* Revert kustomization change

Signed-off-by: Michal Szadkowski <[email protected]>

* Update job_test and logging

Signed-off-by: Michal Szadkowski <[email protected]>

* Provide immutability check for ManagedBy

Signed-off-by: Michal Szadkowski <[email protected]>

* Avoid making copy of runPolicy

Signed-off-by: Michal Szadkowski <[email protected]>

* Split RunPolicy validators to Update and Create

Signed-off-by: Michal Szadkowski <[email protected]>

* Fix the naming and call validate always

Signed-off-by: Michal Szadkowski <[email protected]>

* Update tests

Signed-off-by: Michal Szadkowski <[email protected]>

---------

Signed-off-by: Michal Szadkowski <[email protected]>
 2024-09-18 18:09:27 +00:00
Andrey VelichkevichandGitHub 93977ad4f3 Bump Training Python SDK to 1.8.1 version (#2257) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-09-10 13:00:20 +00:00
Andrey VelichkevichandGitHub 3af18dbdbb [SDK] Read namespace from the current context (#2255) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-09-10 11:59:20 +00:00
Hezhi (Helen) XieandGitHub 3a1cb82f09 [SDK] Fix typo of "get_pvc_spec" (#2250) 
* fix typo of 'get_pvc_spec'

Signed-off-by: helenxie-bit <[email protected]>

* rerun tests

Signed-off-by: helenxie-bit <[email protected]>

* rerun tests

Signed-off-by: helenxie-bit <[email protected]>

---------

Signed-off-by: helenxie-bit <[email protected]>
 2024-09-05 16:21:15 +00:00
Mark CampbellandGitHub d5cc2f36d8 [SDK] test: add unit test for get_job method of the training_client (#2205) 
Signed-off-by: Bobbins228 <[email protected]>
 2024-09-04 12:15:15 +00:00
Mark CampbellandGitHub 1b2030938b [SDK] test: add unit tests for delete_job() method (#2232) 
Signed-off-by: Bobbins228 <[email protected]>
 2024-09-02 22:18:13 +00:00
Shao WangandGitHub f6521d6e53 [SDK] Add UTs for wait_for_job_conditions (#2196) 
* test(sdk): add unit test for wait_for_job_conditions.

Signed-off-by: Electronic-Waste <[email protected]>

* test(sdk): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* test(sdk): add patch for load_kube_config.

Signed-off-by: Electronic-Waste <[email protected]>

* test(trial): fix lint error with black.

Signed-off-by: Electronic-Waste <[email protected]>

* test(trial): add package dependency.

Signed-off-by: Electronic-Waste <[email protected]>

* test(sdk): reuse exisiting fixture.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
 2024-08-30 13:40:05 +00:00
Hezhi (Helen) XieandGitHub 8c5212ca01 [SDK] Fix trainer error: Update the version of base image and add "num_labels" for downloading pretrained models (#2230) 
* fix trainer error

Signed-off-by: helenxie-bit <[email protected]>

* rerun tests

Signed-off-by: helenxie-bit <[email protected]>

* update the process of num_labels in trainer

Signed-off-by: helenxie-bit <[email protected]>

* rerun tests

Signed-off-by: helenxie-bit <[email protected]>

* adjust the  default value of 'num_labels'

Signed-off-by: helenxie-bit <[email protected]>

---------

Signed-off-by: helenxie-bit <[email protected]>
 2024-08-28 16:24:04 +00:00
Ignas BaranauskasandGitHub 156507c7d9 Change isort profile to black for full compatibility (#2234) 
* Change isort profile to black for full compatibility

Signed-off-by: Ignas Baranauskas <[email protected]>

* Fix isort formatting

Signed-off-by: Ignas Baranauskas <[email protected]>

---------

Signed-off-by: Ignas Baranauskas <[email protected]>
 2024-08-27 01:55:02 +00:00
YosiEliasGitHubyelias
 		def23aa652 [SDK] Unit tests for TrainingClient APIs - get_job_pod_names and update_job (#2192) 
* [SDK] Add more unit tests for TrainingClient APIs - get_job_pod_names and update_job

Signed-off-by: yelias <[email protected]>

* fix isort

Signed-off-by: yelias <[email protected]>

* Remove dict_to_object and SimpleNamespace, and add option of return pod obj

Signed-off-by: yelias <[email protected]>

* Fix typo in docstring

Signed-off-by: yelias <[email protected]>

* Change str to constants

Signed-off-by: yelias <[email protected]>

---------

Signed-off-by: yelias <[email protected]>
Co-authored-by: yelias <[email protected]>
 2024-08-22 16:34:57 +00:00
Ignas BaranauskasandGitHub 9f25ccd77b Enhance pre-commit hooks with flake8 linting (#2195) 
* Add pre-commit linter

Signed-off-by: Ignas Baranauskas <[email protected]>

* Fix flake8 linting issues

Signed-off-by: Ignas Baranauskas <[email protected]>

---------

Signed-off-by: Ignas Baranauskas <[email protected]>
 2024-08-12 14:31:41 +00:00
Hezhi XieandGitHub 52a24b911e Update trainer to ensure type consistency for train_args and lora_config (#2181) 
* update-trainer

Signed-off-by: helenxie-bit <[email protected]>

* fix typo

Signed-off-by: helenxie-bit <[email protected]>

* reformat with black

Signed-off-by: helenxie-bit <[email protected]>

---------

Signed-off-by: helenxie-bit <[email protected]>
 2024-08-12 12:27:40 +00:00
Hezhi XieandGitHub ccf8ca572f Update the name of PVC in train API (#2187) 
* update the name of pvc

Signed-off-by: helenxie-bit <[email protected]>

* reformat with black

Signed-off-by: helenxie-bit <[email protected]>

* update 'storage_initializer_volume'

Signed-off-by: helenxie-bit <[email protected]>

---------

Signed-off-by: helenxie-bit <[email protected]>
 2024-08-07 12:48:40 +00:00
AlexandGitHub 7c3307701e Implement pre-commit hooks (#2184) 
* Implement pre-commit hooks

Signed-off-by: droctothorpe <[email protected]>

* Fix broken file links and update reqs

Signed-off-by: droctothorpe <[email protected]>

* Add examples folder

Signed-off-by: droctothorpe <[email protected]>

* Remove black instructions from developer guide

Signed-off-by: droctothorpe <[email protected]>

---------

Signed-off-by: droctothorpe <[email protected]>
 2024-08-05 17:03:39 +00:00
Hezhi XieandGitHub 0f6aa199dd Update huggingface_hub Version in the storage initializer to fix ImportError (#2180) 
Signed-off-by: helenxie-bit <[email protected]>
 2024-08-05 12:44:38 +00:00
Sandipan PandaandGitHub 08fb25f8a0 Add JAX API (#2163) 
Support JAXJob

Signed-off-by: Sandipan Panda <[email protected]>
 2024-08-02 03:58:07 +00:00
Tariq HasanandGitHub a2812b5fb4 Remove support for MXJob (#2150) 
Signed-off-by: tariq-hasan <[email protected]>
 2024-07-28 10:37:56 +00:00
YosiEliasGitHubyelias
 		bba48d4143 [SDK] Add more unit tests for TrainingClient APIs - get_job_pods (#2175) 
* [SDK] Add more unit tests for TrainingClient APIs - get_job_pods

Signed-off-by: yelias <[email protected]>

* Fixed after review

Signed-off-by: yelias <[email protected]>

---------

Signed-off-by: yelias <[email protected]>
Co-authored-by: yelias <[email protected]>
 2024-07-22 23:09:01 +00:00
Andrey VelichkevichandGitHub 0615ea1fca Bump Training Python SDK to 1.8.0 version (#2172) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-07-18 21:46:58 +00:00
Andrey VelichkevichandGitHub 796c0f3df9 [SDK] Fix Failed condition in wait Job API (#2160) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-07-11 17:51:21 +00:00
Andrey VelichkevichandGitHub 5b9d22af13 [SDK] Sync Transformers version for train API (#2146) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-06-14 14:27:10 +00:00
Andrey VelichkevichandGitHub 53ac5ebd8c [SDK] Explain Python version support cycle (#2144) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-06-13 16:38:08 +00:00
Andrey VelichkevichandGitHub 31ba928cf6 Update Slack Invitation (#2142) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-06-11 12:09:14 +00:00
Andrey VelichkevichandGitHub dc8e4e0380 [SDK] Fix Incorrect Events in get_job_logs API (#2122) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-05-21 16:28:25 +00:00
Christophe BrownGitHubChristophe Brown
 		af29559386 changed package name to flake8 to fix pytests pip install (#2109) 
Signed-off-by: Christophe Brown <[email protected]>
Co-authored-by: Christophe Brown <[email protected]>
 2024-05-10 14:31:45 +00:00
Yuki IwaiandGitHub f310db59a8 Support Python 3.11 and Drop Python 3.7 (#2105) 
Signed-off-by: Yuki Iwai <[email protected]>
 2024-05-09 20:58:45 +00:00
Yuki IwaiandGitHub 096a7bc6ab E2E: Replace outdated images with latest ones (#2083) 
Signed-off-by: Yuki Iwai <[email protected]>
 2024-04-25 21:54:46 +00:00
Andrey VelichkevichandGitHub 3ee6efcb10 [SDK] Add docstring for Train API (#2075) 
* [SDK] Add docstring for Train API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update a few args

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-04-24 18:24:19 +00:00
Yuki IwaiandGitHub 3d741b0995 SDK: Upgrade the minimum required Kubernetes version to v1.27.2 (#2066) 
Signed-off-by: Yuki Iwai <[email protected]>
 2024-04-15 12:59:51 +00:00
deepanker13andGitHub 42a897c96f adding fine tune example with s3 as the dataset store (#2006) 
* s3 as dataset source code review changes

Signed-off-by: deepanker13 <[email protected]>

* fixing python black test

Signed-off-by: deepanker13 <[email protected]>

* removing conflicts in example file

Signed-off-by: deepanker13 <[email protected]>

* retriggering CI

Signed-off-by: deepanker13 <[email protected]>

* removing dummy keys

Signed-off-by: deepanker13 <[email protected]>

* code review change for adding s3 keys block

Signed-off-by: deepanker13 <[email protected]>

---------

Signed-off-by: deepanker13 <[email protected]>
 2024-04-12 19:54:48 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 		90f027853d Bump transformers from 4.37.2 to 4.38.0 in /sdk/python/kubeflow/storage_initializer (#2056) 
Bumps [transformers](https://github.com/huggingface/transformers) from 4.37.2 to 4.38.0.
- [Release notes](https://github.com/huggingface/transformers/releases)
- [Commits](https://github.com/huggingface/transformers/compare/v4.37.2...v4.38.0)

---
updated-dependencies:
- dependency-name: transformers
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-04-11 07:24:47 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 		b39271e152 Bump transformers from 4.37.2 to 4.38.0 in /sdk/python/kubeflow/trainer (#2055) 
Bumps [transformers](https://github.com/huggingface/transformers) from 4.37.2 to 4.38.0.
- [Release notes](https://github.com/huggingface/transformers/releases)
- [Commits](https://github.com/huggingface/transformers/compare/v4.37.2...v4.38.0)

---
updated-dependencies:
- dependency-name: transformers
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-04-11 07:18:47 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 		b24d26c474 Bump black from 21.12b0 to 24.3.0 in /sdk/python (#2033) 
Bumps [black](https://github.com/psf/black) from 21.12b0 to 24.3.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/commits/24.3.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-25 14:01:18 +00:00
Andrey VelichkevichandGitHub 8634427e4d Modify LLM Trainer to support BERT and Tiny LLaMA (#2031) 
* Modify LLM Trainer to support BERT and Tiny LLaMA

Signed-off-by: Andrey Velichkevich <[email protected]>

* Access PVC access modes to storage config

Signed-off-by: Andrey Velichkevich <[email protected]>

* Format Python files

Signed-off-by: Andrey Velichkevich <[email protected]>

* Distribute datasets

Fix access modes in storage config

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update example to fine tune BERT with Train API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove dataset download twice

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-03-15 19:28:30 +00:00
Andrey VelichkevichandGitHub 386392ded2 Add Fine-Tune BERT LLM Example (#2021) 
* Add Fine-Tune BERT LLM Example

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add 3 GPUs in Notebook requirements

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-03-12 11:35:10 +00:00
Gary MiguelandGitHub d365c12b0e Fix URL in python SDK setup.py (#2011) 
Signed-off-by: Gary Miguel <[email protected]>
 2024-03-01 23:29:05 +00:00
Andrey VelichkevichandGitHub a44b9d0b93 [SDK] Add resources per worker for Create Job API (#1990) 
* [SDK] Add resources for create Job API

* Fix unbound var

* Assign values in get pod template

* Add torchrun issue

* Test to create PyTorchJob from Image

* Fix e2e to create from image

* Fix condition

* Modify check test conditions
 2024-01-18 16:38:59 +00:00
Andrey VelichkevichandGitHub 99a719d266 [SDK] Fix Worker and Master templates for PyTorchJob (#1988) 2024-01-16 19:09:19 +00:00
deepanker13andGitHub 93ff8e54f3 publish trainer hugging face image (#1985) 
* publish trainer hugging face

* renaming container image names
 2024-01-12 14:29:14 +00:00
deepanker13andGitHub 46a80709a2 Adding Training image needed for train api (#1963) 
* adding training image creationcode

* reformatting code using black

* code review changes

* fixes

* running workflow on push

* correcting context

* correcting context

* fixing trainer code

* code review changes

* resolving merge conflict

* code review changes

* downgrading pytorch version, removing changes for running things locally, adding jupyter notebook

* ci fix and removing unused parameter and adding check if pvc exists already

* gpu training example fix
 2024-01-11 21:06:09 +00:00