mirror of
https://github.com/hanzo-ml/trainer.git
synced 2026-09-24 00:31:34 +00:00
master
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Amir IbrahimandGitHub
|
49f4eaadd2 |
feat: add Flux integration and e2e tests (#3561)
* test: add Flux framework integration and e2e tests Signed-off-by: Amir380-A <[email protected]> * fix: fix integration and e2e Signed-off-by: Amir380-A <[email protected]> * fix: EOF linting issue Signed-off-by: Amir380-A <[email protected]> * chore: add logs for E2E ClusterTrainingRuntimes in Github Actions Signed-off-by: Amir380-A <[email protected]> * feat: add Flux runtime to Helm chart Signed-off-by: Amir380-A <[email protected]> * fix: fix EOF linting issue Signed-off-by: Amir380-A <[email protected]> * fix: install Flux runtime only for E2E tests Signed-off-by: Amir380-A <[email protected]> * chore: regenerate Helm chart docs and generated assets Signed-off-by: Amir380-A <[email protected]> * fix: retry failed deepspeed CI dependency Signed-off-by: Amir380-A <[email protected]> * revert back to the previous commit Signed-off-by: Amir380-A <[email protected]> * fix(e2e): install Flux runtime only during E2E setup Signed-off-by: Amir380-A <[email protected]> * fix: fix linting issue Signed-off-by: Amir380-A <[email protected]> * fix: fix runtimeRef for CI Signed-off-by: Amir380-A <[email protected]> * fix: fix flux examples runtimeref Signed-off-by: Amir380-A <[email protected]> --------- Signed-off-by: Amir380-A <[email protected]> Signed-off-by: Amir Ibrahim <[email protected]> |
||
|
RobertandGitHub
|
422c2297bb |
chore(test): refactor TestFlux to use wantInfo to match MPI plugin (#3601)
* test(flux): refactor TestFlux to use wantInfo to match MPI plugin. Signed-off-by: roberthluo <[email protected]> * Address comments in PR. Signed-off-by: roberthluo <[email protected]> * Fix tc.wantInfo comparison. Signed-off-by: roberthluo <[email protected]> --------- Signed-off-by: roberthluo <[email protected]> |
||
|
Tariq HasanandGitHub
|
9183dfd24f |
chore(scripts): add copyright boilerplate verification (#3359)
* chore(scripts): add copyright boilerplate verification system Signed-off-by: tariq-hasan <[email protected]> * chore(ci): add boilerplate verification to CI pipeline Signed-off-by: tariq-hasan <[email protected]> * chore: regenerate files with year-less copyright headers Signed-off-by: tariq-hasan <[email protected]> * refactor(boilerplate): simplify verification logic Signed-off-by: tariq-hasan <[email protected]> * chore(boilerplate): add missing copyright headers Signed-off-by: tariq-hasan <[email protected]> * ci(boilerplate): move verifier to code-quality workflow and run unit tests Signed-off-by: tariq-hasan <[email protected]> * feat(boilerplate): accept license headers with any year and remove year validation logic Signed-off-by: tariq-hasan <[email protected]> * feat(boilerplate): add copyright headers to generated Python API Signed-off-by: tariq-hasan <[email protected]> * refactor(boilerplate): use block-comment header for generated Go files Signed-off-by: tariq-hasan <[email protected]> * feat(boilerplate): verify copyright headers in Rust files Signed-off-by: tariq-hasan <[email protected]> --------- Signed-off-by: tariq-hasan <[email protected]> |
||
|
|
1510b1c0b1 |
feat: customize R generation to include GPUs (#3325)
* feat: customize R generation to include GPUs Flux detection of GPUs depends on hwloc plugins, and a newer version. To get around any edge cases where the dependencies are missing, we can easily generate the R from the expected resource spec (cores and gpus). We can also add a shared memory mount to ensure the job MPI gets all available shared memory of the host. The current default is 64M (automatic from container runtime) and it can have implications for MPI performance. Signed-off-by: vsoch <[email protected]> * review: use generateRange function for gpus Also update volume to be added to container. This still is pending testing on AWS! Signed-off-by: vsoch <[email protected]> * review and add gpu trainer example Signed-off-by: vsoch <[email protected]> --------- Signed-off-by: vsoch <[email protected]> Co-authored-by: vsoch <[email protected]> |
||
|
Rob BellandGitHub
|
78b4a00929 |
feat: add support for tracking TrainJob progress and training metrics (#3227)
* feat(api): add progress types to api Signed-off-by: Rob Bell <[email protected]> * feat: progress server stub Signed-off-by: Rob Bell <[email protected]> * feat: setup tls for server Signed-off-by: Rob Bell <[email protected]> * refactor: pull out middleware Signed-off-by: Rob Bell <[email protected]> * refactor: remove logger message noise from tests Signed-off-by: Rob Bell <[email protected]> * fix: send server error logs to main logger Signed-off-by: Rob Bell <[email protected]> * feat: update the trainer status Signed-off-by: Rob Bell <[email protected]> * feat: use separate client to avoid impacting main reconciler client-side rate limits Signed-off-by: Rob Bell <[email protected]> * feat: configure progress server in helm chart and configmap Signed-off-by: Rob Bell <[email protected]> * fix: prevent race condition Signed-off-by: Rob Bell <[email protected]> * feat: inject progress runtime config Signed-off-by: Rob Bell <[email protected]> * fix: progress client config Signed-off-by: Rob Bell <[email protected]> * fix: update generated config Signed-off-by: Rob Bell <[email protected]> * feat: validate auth tokens in endpoint Signed-off-by: Rob Bell <[email protected]> * test: validate auth tokens in endpoint Signed-off-by: Rob Bell <[email protected]> * refactor: tidy tests to use same format as other tests Signed-off-by: Rob Bell <[email protected]> * fix: look up the cluster issuer from the service account token Different clusters can use different issuers so look up the issuer from the trusted in-cluster service account token. Signed-off-by: Rob Bell <[email protected]> * refactor: rename authn middleware Signed-off-by: Rob Bell <[email protected]> * feat: add authn to status update endpoint Signed-off-by: Rob Bell <[email protected]> * refactor: tidy up reading in-cluster service account token Signed-off-by: Rob Bell <[email protected]> * fix: rbac permissions Signed-off-by: Rob Bell <[email protected]> * fix: better error response Signed-off-by: Rob Bell <[email protected]> * refactor: move kubebuilder annotation higher up Signed-off-by: Rob Bell <[email protected]> * fix: make lastUpdateTime required Signed-off-by: Rob Bell <[email protected]> * chore: fix config tests after rebase Signed-off-by: Rob Bell <[email protected]> * test: update config test with new progress server config Signed-off-by: Rob Bell <[email protected]> * test: add progress tracking e2e test Signed-off-by: Rob Bell <[email protected]> * fix: allow feature gates to be configurable from command line argument So they can be overridden in e2e tests Signed-off-by: Rob Bell <[email protected]> * refactor: reuse shared progress StatusUrl Signed-off-by: Rob Bell <[email protected]> * test: tidy up tests Signed-off-by: Rob Bell <[email protected]> * refactor: allow plugins to use config Signed-off-by: Rob Bell <[email protected]> * refactor: centralise shared constants Signed-off-by: Rob Bell <[email protected]> * fix: disable http2 in progress server tls opts if disabled by config Signed-off-by: Rob Bell <[email protected]> * fix: minor improvements suggested by copilot Signed-off-by: Rob Bell <[email protected]> * chore: rename progress server => status server Signed-off-by: Rob Bell <[email protected]> * chore: update type doc Signed-off-by: Rob Bell <[email protected]> * chore: update name of feature gate Signed-off-by: Rob Bell <[email protected]> * fix: after rebase Signed-off-by: Rob Bell <[email protected]> * chore: fix typo Signed-off-by: Rob Bell <[email protected]> * chore: rename handler Signed-off-by: Rob Bell <[email protected]> * fix: run status server on all replicas Signed-off-by: Rob Bell <[email protected]> * fix: deep copy the rest config Signed-off-by: Rob Bell <[email protected]> * refactor: use ssa Signed-off-by: Rob Bell <[email protected]> * refactor: separate token audience per train job Signed-off-by: Rob Bell <[email protected]> * chore: rename status server Signed-off-by: Rob Bell <[email protected]> * fix: revert adding status pkg to manifest gen paths Signed-off-by: Rob Bell <[email protected]> * chore: update log messages Signed-off-by: Rob Bell <[email protected]> * chore: update api docs Signed-off-by: Rob Bell <[email protected]> * chore: update format Signed-off-by: Rob Bell <[email protected]> * fix: revert inadvertent change Signed-off-by: Rob Bell <[email protected]> * chore: update api name and type docs Signed-off-by: Rob Bell <[email protected]> * refactor: initialise oidc provider in manager startup Signed-off-by: Rob Bell <[email protected]> * fix: allow empty update request Signed-off-by: Rob Bell <[email protected]> * fix: wait for server shut down Signed-off-by: Rob Bell <[email protected]> * fix: handle missing train job Signed-off-by: Rob Bell <[email protected]> * refactor: update env var names Signed-off-by: Rob Bell <[email protected]> * chore: rename packages, plugins, feature gate Signed-off-by: Rob Bell <[email protected]> * refactor: update env var names Signed-off-by: Rob Bell <[email protected]> * chore: regenerate after rebase Signed-off-by: Rob Bell <[email protected]> * chore: fixes after rebase Signed-off-by: Rob Bell <[email protected]> * chore: minor doc updates Signed-off-by: Rob Bell <[email protected]> * chore: rename status server config Signed-off-by: Rob Bell <[email protected]> * chore: replace placeholder config with nil Signed-off-by: Rob Bell <[email protected]> * chore: revert separate runtime utils package Signed-off-by: Rob Bell <[email protected]> * chore: remove unused config from tests Signed-off-by: Rob Bell <[email protected]> --------- Signed-off-by: Rob Bell <[email protected]> |
||
|
|
88a92a0483 |
feat: support for flux framework as hpc manager (#3188)
* feat: support for flux framework as hpc manager Flux supports the majority of MPI flavors/variants, and can be used to bootstrap MPI as a plugin. It adds other features for scheduling and topology that can be used for simulations and ai/ml jobs. This changeset adds the plugin implementation, including the plugin module, tests, and an example with a small README to serve as documentation for the time being. Signed-off-by: vsoch <[email protected]> * review: slim down use of annotations We still need to put the original command in an annotation to retieve later, but others can be re-derived from the environment Signed-off-by: vsoch <[email protected]> * review: clean up tasks and remove extra FluxPolicySource Signed-off-by: vsoch <[email protected]> * review: ensure to unset trainer args Signed-off-by: vsoch <[email protected]> * review: better support for nproc per node and nodes Signed-off-by: vsoch <[email protected]> * review: add constants types for flux Signed-off-by: vsoch <[email protected]> * review: add gpu specification from resources Signed-off-by: vsoch <[email protected]> * Fix TrainJob NumProcPerNode Signed-off-by: Andrey Velichkevich <[email protected]> * Remove elasticPolicy from kubebuilder:validation Signed-off-by: Andrey Velichkevich <[email protected]> * chore: use go embed for long string templates flux plugin Signed-off-by: vsoch <[email protected]> * cleanup: removed un-used annotation Signed-off-by: vsoch <[email protected]> --------- Signed-off-by: vsoch <[email protected]> Signed-off-by: Andrey Velichkevich <[email protected]> Co-authored-by: vsoch <[email protected]> Co-authored-by: Andrey Velichkevich <[email protected]> |