hanzo-ml/trainer
1
0
Fork 0
mirror of https://github.com/hanzo-ml/trainer.git synced 2026-09-24 00:31:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,066 Commits 2 Branches 1 Tag
e18b1be01c19ff28d36a4626fcdf2163b02ad5fd
Commit Graph
107 Commits
Author SHA1 Message Date
VarshaandGitHub e18b1be01c [SDK] Allow customising base trainer and storage images in Train API (#2261) 
Allow customizing base storage_initializer and trainer images through
Env vars.

Signed-off-by: Varsha Prasad Narsing <[email protected]>
 2024-09-19 08:48:28 +00:00
Michał SzadkowskiandGitHub 26c9112c07 [Feature] Support managed by external controller (#2203) 
* Introduce ManagedBy field to RunPolicy that is used by each Kubeflow Job Spec

Signed-off-by: Michal Szadkowski <[email protected]>

* Update Kubeflow JOb manifests

Signed-off-by: Michal Szadkowski <[email protected]>

* Update Kubeflow Jobs Reconcile to use ManagedBy field to decide if skip the process

Signed-off-by: Michal Szadkowski <[email protected]>

* job controller test

Signed-off-by: Michal Szadkowski <[email protected]>

* spec validation webhook

Signed-off-by: Michal Szadkowski <[email protected]>

* add manageBy maxLenght const

Signed-off-by: Michal Szadkowski <[email protected]>

* generate new manifest

Signed-off-by: Michal Szadkowski <[email protected]>

* revert webhook formatting

Signed-off-by: Michal Szadkowski <[email protected]>

* Move allowed controllers constants in one place

Signed-off-by: Michal Szadkowski <[email protected]>

* Make validatation for allowed managedBy values

Signed-off-by: Michal Szadkowski <[email protected]>

* Update after controllers constants move

Signed-off-by: Michal Szadkowski <[email protected]>

* Update jobs controller tests

Signed-off-by: Michal Szadkowski <[email protected]>

* Update validateManagedBy webhook

Signed-off-by: Michal Szadkowski <[email protected]>

* Remove validation for the length of ManagedBy field

Signed-off-by: Michal Szadkowski <[email protected]>

* Update after code review

Signed-off-by: Michal Szadkowski <[email protected]>

* Update ManagedBy comment

Signed-off-by: Michal Szadkowski <[email protected]>

* E2E tests for managedBy

Signed-off-by: Michal Szadkowski <[email protected]>

* Update generated files and manifests

Signed-off-by: Michal Szadkowski <[email protected]>

* Rework after code review

Signed-off-by: Michal Szadkowski <[email protected]>

* Revert kustomization change

Signed-off-by: Michal Szadkowski <[email protected]>

* Update job_test and logging

Signed-off-by: Michal Szadkowski <[email protected]>

* Provide immutability check for ManagedBy

Signed-off-by: Michal Szadkowski <[email protected]>

* Avoid making copy of runPolicy

Signed-off-by: Michal Szadkowski <[email protected]>

* Split RunPolicy validators to Update and Create

Signed-off-by: Michal Szadkowski <[email protected]>

* Fix the naming and call validate always

Signed-off-by: Michal Szadkowski <[email protected]>

* Update tests

Signed-off-by: Michal Szadkowski <[email protected]>

---------

Signed-off-by: Michal Szadkowski <[email protected]>
 2024-09-18 18:09:27 +00:00
Andrey VelichkevichandGitHub 93977ad4f3 Bump Training Python SDK to 1.8.1 version (#2257) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-09-10 13:00:20 +00:00
Andrey VelichkevichandGitHub 3af18dbdbb [SDK] Read namespace from the current context (#2255) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-09-10 11:59:20 +00:00
Hezhi (Helen) XieandGitHub 3a1cb82f09 [SDK] Fix typo of "get_pvc_spec" (#2250) 
* fix typo of 'get_pvc_spec'

Signed-off-by: helenxie-bit <[email protected]>

* rerun tests

Signed-off-by: helenxie-bit <[email protected]>

* rerun tests

Signed-off-by: helenxie-bit <[email protected]>

---------

Signed-off-by: helenxie-bit <[email protected]>
 2024-09-05 16:21:15 +00:00
Mark CampbellandGitHub d5cc2f36d8 [SDK] test: add unit test for get_job method of the training_client (#2205) 
Signed-off-by: Bobbins228 <[email protected]>
 2024-09-04 12:15:15 +00:00
Mark CampbellandGitHub 1b2030938b [SDK] test: add unit tests for delete_job() method (#2232) 
Signed-off-by: Bobbins228 <[email protected]>
 2024-09-02 22:18:13 +00:00
Shao WangandGitHub f6521d6e53 [SDK] Add UTs for wait_for_job_conditions (#2196) 
* test(sdk): add unit test for wait_for_job_conditions.

Signed-off-by: Electronic-Waste <[email protected]>

* test(sdk): fix lint error.

Signed-off-by: Electronic-Waste <[email protected]>

* test(sdk): add patch for load_kube_config.

Signed-off-by: Electronic-Waste <[email protected]>

* test(trial): fix lint error with black.

Signed-off-by: Electronic-Waste <[email protected]>

* test(trial): add package dependency.

Signed-off-by: Electronic-Waste <[email protected]>

* test(sdk): reuse exisiting fixture.

Signed-off-by: Electronic-Waste <[email protected]>

---------

Signed-off-by: Electronic-Waste <[email protected]>
 2024-08-30 13:40:05 +00:00
Hezhi (Helen) XieandGitHub 8c5212ca01 [SDK] Fix trainer error: Update the version of base image and add "num_labels" for downloading pretrained models (#2230) 
* fix trainer error

Signed-off-by: helenxie-bit <[email protected]>

* rerun tests

Signed-off-by: helenxie-bit <[email protected]>

* update the process of num_labels in trainer

Signed-off-by: helenxie-bit <[email protected]>

* rerun tests

Signed-off-by: helenxie-bit <[email protected]>

* adjust the  default value of 'num_labels'

Signed-off-by: helenxie-bit <[email protected]>

---------

Signed-off-by: helenxie-bit <[email protected]>
 2024-08-28 16:24:04 +00:00
Ignas BaranauskasandGitHub 156507c7d9 Change isort profile to black for full compatibility (#2234) 
* Change isort profile to black for full compatibility

Signed-off-by: Ignas Baranauskas <[email protected]>

* Fix isort formatting

Signed-off-by: Ignas Baranauskas <[email protected]>

---------

Signed-off-by: Ignas Baranauskas <[email protected]>
 2024-08-27 01:55:02 +00:00
YosiEliasGitHubyelias
 		def23aa652 [SDK] Unit tests for TrainingClient APIs - get_job_pod_names and update_job (#2192) 
* [SDK] Add more unit tests for TrainingClient APIs - get_job_pod_names and update_job

Signed-off-by: yelias <[email protected]>

* fix isort

Signed-off-by: yelias <[email protected]>

* Remove dict_to_object and SimpleNamespace, and add option of return pod obj

Signed-off-by: yelias <[email protected]>

* Fix typo in docstring

Signed-off-by: yelias <[email protected]>

* Change str to constants

Signed-off-by: yelias <[email protected]>

---------

Signed-off-by: yelias <[email protected]>
Co-authored-by: yelias <[email protected]>
 2024-08-22 16:34:57 +00:00
Ignas BaranauskasandGitHub 9f25ccd77b Enhance pre-commit hooks with flake8 linting (#2195) 
* Add pre-commit linter

Signed-off-by: Ignas Baranauskas <[email protected]>

* Fix flake8 linting issues

Signed-off-by: Ignas Baranauskas <[email protected]>

---------

Signed-off-by: Ignas Baranauskas <[email protected]>
 2024-08-12 14:31:41 +00:00
Hezhi XieandGitHub 52a24b911e Update trainer to ensure type consistency for train_args and lora_config (#2181) 
* update-trainer

Signed-off-by: helenxie-bit <[email protected]>

* fix typo

Signed-off-by: helenxie-bit <[email protected]>

* reformat with black

Signed-off-by: helenxie-bit <[email protected]>

---------

Signed-off-by: helenxie-bit <[email protected]>
 2024-08-12 12:27:40 +00:00
Hezhi XieandGitHub ccf8ca572f Update the name of PVC in train API (#2187) 
* update the name of pvc

Signed-off-by: helenxie-bit <[email protected]>

* reformat with black

Signed-off-by: helenxie-bit <[email protected]>

* update 'storage_initializer_volume'

Signed-off-by: helenxie-bit <[email protected]>

---------

Signed-off-by: helenxie-bit <[email protected]>
 2024-08-07 12:48:40 +00:00
AlexandGitHub 7c3307701e Implement pre-commit hooks (#2184) 
* Implement pre-commit hooks

Signed-off-by: droctothorpe <[email protected]>

* Fix broken file links and update reqs

Signed-off-by: droctothorpe <[email protected]>

* Add examples folder

Signed-off-by: droctothorpe <[email protected]>

* Remove black instructions from developer guide

Signed-off-by: droctothorpe <[email protected]>

---------

Signed-off-by: droctothorpe <[email protected]>
 2024-08-05 17:03:39 +00:00
Hezhi XieandGitHub 0f6aa199dd Update huggingface_hub Version in the storage initializer to fix ImportError (#2180) 
Signed-off-by: helenxie-bit <[email protected]>
 2024-08-05 12:44:38 +00:00
Sandipan PandaandGitHub 08fb25f8a0 Add JAX API (#2163) 
Support JAXJob

Signed-off-by: Sandipan Panda <[email protected]>
 2024-08-02 03:58:07 +00:00
Tariq HasanandGitHub a2812b5fb4 Remove support for MXJob (#2150) 
Signed-off-by: tariq-hasan <[email protected]>
 2024-07-28 10:37:56 +00:00
YosiEliasGitHubyelias
 		bba48d4143 [SDK] Add more unit tests for TrainingClient APIs - get_job_pods (#2175) 
* [SDK] Add more unit tests for TrainingClient APIs - get_job_pods

Signed-off-by: yelias <[email protected]>

* Fixed after review

Signed-off-by: yelias <[email protected]>

---------

Signed-off-by: yelias <[email protected]>
Co-authored-by: yelias <[email protected]>
 2024-07-22 23:09:01 +00:00
Andrey VelichkevichandGitHub 0615ea1fca Bump Training Python SDK to 1.8.0 version (#2172) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-07-18 21:46:58 +00:00
Andrey VelichkevichandGitHub 796c0f3df9 [SDK] Fix Failed condition in wait Job API (#2160) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-07-11 17:51:21 +00:00
Andrey VelichkevichandGitHub 5b9d22af13 [SDK] Sync Transformers version for train API (#2146) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-06-14 14:27:10 +00:00
Andrey VelichkevichandGitHub 53ac5ebd8c [SDK] Explain Python version support cycle (#2144) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-06-13 16:38:08 +00:00
Andrey VelichkevichandGitHub 31ba928cf6 Update Slack Invitation (#2142) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-06-11 12:09:14 +00:00
Andrey VelichkevichandGitHub dc8e4e0380 [SDK] Fix Incorrect Events in get_job_logs API (#2122) 
Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-05-21 16:28:25 +00:00
Christophe BrownGitHubChristophe Brown
 		af29559386 changed package name to flake8 to fix pytests pip install (#2109) 
Signed-off-by: Christophe Brown <[email protected]>
Co-authored-by: Christophe Brown <[email protected]>
 2024-05-10 14:31:45 +00:00
Yuki IwaiandGitHub f310db59a8 Support Python 3.11 and Drop Python 3.7 (#2105) 
Signed-off-by: Yuki Iwai <[email protected]>
 2024-05-09 20:58:45 +00:00
Yuki IwaiandGitHub 096a7bc6ab E2E: Replace outdated images with latest ones (#2083) 
Signed-off-by: Yuki Iwai <[email protected]>
 2024-04-25 21:54:46 +00:00
Andrey VelichkevichandGitHub 3ee6efcb10 [SDK] Add docstring for Train API (#2075) 
* [SDK] Add docstring for Train API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update a few args

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-04-24 18:24:19 +00:00
Yuki IwaiandGitHub 3d741b0995 SDK: Upgrade the minimum required Kubernetes version to v1.27.2 (#2066) 
Signed-off-by: Yuki Iwai <[email protected]>
 2024-04-15 12:59:51 +00:00
deepanker13andGitHub 42a897c96f adding fine tune example with s3 as the dataset store (#2006) 
* s3 as dataset source code review changes

Signed-off-by: deepanker13 <[email protected]>

* fixing python black test

Signed-off-by: deepanker13 <[email protected]>

* removing conflicts in example file

Signed-off-by: deepanker13 <[email protected]>

* retriggering CI

Signed-off-by: deepanker13 <[email protected]>

* removing dummy keys

Signed-off-by: deepanker13 <[email protected]>

* code review change for adding s3 keys block

Signed-off-by: deepanker13 <[email protected]>

---------

Signed-off-by: deepanker13 <[email protected]>
 2024-04-12 19:54:48 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 		90f027853d Bump transformers from 4.37.2 to 4.38.0 in /sdk/python/kubeflow/storage_initializer (#2056) 
Bumps [transformers](https://github.com/huggingface/transformers) from 4.37.2 to 4.38.0.
- [Release notes](https://github.com/huggingface/transformers/releases)
- [Commits](https://github.com/huggingface/transformers/compare/v4.37.2...v4.38.0)

---
updated-dependencies:
- dependency-name: transformers
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-04-11 07:24:47 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 		b39271e152 Bump transformers from 4.37.2 to 4.38.0 in /sdk/python/kubeflow/trainer (#2055) 
Bumps [transformers](https://github.com/huggingface/transformers) from 4.37.2 to 4.38.0.
- [Release notes](https://github.com/huggingface/transformers/releases)
- [Commits](https://github.com/huggingface/transformers/compare/v4.37.2...v4.38.0)

---
updated-dependencies:
- dependency-name: transformers
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-04-11 07:18:47 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 		b24d26c474 Bump black from 21.12b0 to 24.3.0 in /sdk/python (#2033) 
Bumps [black](https://github.com/psf/black) from 21.12b0 to 24.3.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/commits/24.3.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-25 14:01:18 +00:00
Andrey VelichkevichandGitHub 8634427e4d Modify LLM Trainer to support BERT and Tiny LLaMA (#2031) 
* Modify LLM Trainer to support BERT and Tiny LLaMA

Signed-off-by: Andrey Velichkevich <[email protected]>

* Access PVC access modes to storage config

Signed-off-by: Andrey Velichkevich <[email protected]>

* Format Python files

Signed-off-by: Andrey Velichkevich <[email protected]>

* Distribute datasets

Fix access modes in storage config

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update example to fine tune BERT with Train API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Remove dataset download twice

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-03-15 19:28:30 +00:00
Andrey VelichkevichandGitHub 386392ded2 Add Fine-Tune BERT LLM Example (#2021) 
* Add Fine-Tune BERT LLM Example

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add 3 GPUs in Notebook requirements

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
 2024-03-12 11:35:10 +00:00
Gary MiguelandGitHub d365c12b0e Fix URL in python SDK setup.py (#2011) 
Signed-off-by: Gary Miguel <[email protected]>
 2024-03-01 23:29:05 +00:00
Andrey VelichkevichandGitHub a44b9d0b93 [SDK] Add resources per worker for Create Job API (#1990) 
* [SDK] Add resources for create Job API

* Fix unbound var

* Assign values in get pod template

* Add torchrun issue

* Test to create PyTorchJob from Image

* Fix e2e to create from image

* Fix condition

* Modify check test conditions
 2024-01-18 16:38:59 +00:00
Andrey VelichkevichandGitHub 99a719d266 [SDK] Fix Worker and Master templates for PyTorchJob (#1988) 2024-01-16 19:09:19 +00:00
deepanker13andGitHub 93ff8e54f3 publish trainer hugging face image (#1985) 
* publish trainer hugging face

* renaming container image names
 2024-01-12 14:29:14 +00:00
deepanker13andGitHub 46a80709a2 Adding Training image needed for train api (#1963) 
* adding training image creationcode

* reformatting code using black

* code review changes

* fixes

* running workflow on push

* correcting context

* correcting context

* fixing trainer code

* code review changes

* resolving merge conflict

* code review changes

* downgrading pytorch version, removing changes for running things locally, adding jupyter notebook

* ci fix and removing unused parameter and adding check if pvc exists already

* gpu training example fix
 2024-01-11 21:06:09 +00:00
Andrey VelichkevichandGitHub 16cb71035c Add test to create PyTorchJob from func (#1979) 
* Add test to create PyTorchJob from func

* Use unique Job name for each test

* Fix job name in test

* Describe PyTorchJob when test fails

* Skip test for gang scheduling
 2024-01-11 16:51:12 +00:00
Andrey VelichkevichandGitHub 8d8cb96ada [SDK] Get Kubernetes Events for Job (#1975) 
* [SDK] Get Kubernetes Events for Job

* Move class at the top

* Add object kind to event message

* Add example for events
 2024-01-11 16:30:12 +00:00
deepanker13andGitHub 6a34d14f04 [SDK] Train API (#1962) 
* adding constant for init container name

* github workflow fixes

* removing constants file changes from this pr

* code review changes

* initial skeleton of train api

* train api updated

* fixes

* code review changes

* code review changes

* code review changes

* code review changes

* fixing python library requirements

* adding hugging face dataset download class

* code review changes

* fixing github workflow

* code review comments

* import fixes

* integration test fix for python3.7

* torch version fix for python3.7

* removing unused variable

* fixing library versions for python3.7

* removing alpine distribution

* removing torch ad dependency

* removing literal usage as python 3.7 doesn't support it

* adding types.py

* ci fix

* storage init container changes, fixing imports

* adding extra requires in setup.py, fixinf ci

* adding commit to retrigger go test

* renaming folder to storage initalizer

* bug fix

* removing extra gpu check as discussed with johnu

* retriggering ci
 2024-01-10 17:06:08 +00:00
Andrey VelichkevichGitHubAlex
 		7d0df036ad [SDK] Add information about TrainingClient logging (#1973) 
* [SDK] Use logging to print Job logs

* Update func description

* Add info to configure logging in debug mode

* Update sdk/python/kubeflow/training/api/training_client.py

Co-authored-by: Alex <[email protected]>

* Return logs if follow is True

---------

Co-authored-by: Alex <[email protected]>
 2024-01-05 15:46:09 +00:00
deepanker13andGitHub 8fd0527327 Train api dataset download changes (#1959) 
* adding constant for init container name

* adding master pod template spec and chief replica spec

* renaming vars and adding master spec

* github workflow fixes

* removing constants file changes from this pr

* code review changes

* adding s3/nutanix object store as  dataset provider

* adding validations
 2023-12-14 10:42:53 +00:00
deepanker13andGitHub 138ad9c26c Train api init container creation (#1958) 
* adding constant for init container name

* adding master pod template spec and chief replica spec

* renaming vars and adding master spec

* github workflow fixes

* removing constants file changes from this pr

* code review changes

* adding new folder for creating storage init container images

* changing variable name

* adding validation for model uri

* adding peft_config field

* updating python version
 2023-12-12 14:02:24 +00:00
deepanker13andGitHub 3748eaa5ef utils changes needed to add train api (#1954) 
* utils changes needed to add train api

* adding constant for init container name

* adding master pod template spec and chief replica spec

* renaming vars and adding master spec

* github workflow fixes

* removing constants file changes from this pr

* code review changes

* bug fix

* adding get_container_spec in get_pod_template_spec
 2023-12-08 11:34:27 +00:00
deepanker13andGitHub 240e547c41 Training operator SDK unit test (#1938) 
* adding bug fix in gen-sdk.sh and unit test for create_job function in training client

* code reformatted using black

* move unittest to a different folder

* code review comments changes

* adding pytest github action

* removing 3rd party github pytest action
 2023-11-21 09:25:10 +00:00
AlexandGitHub 25da25b1c4 Avoid modifying log level globally (#1944) 
* Avoid modifying log level globally

* Address get_job_logs

* Fix integration tests
 2023-11-13 19:07:42 +00:00