* fix(runtimes): set file modes on MPI SSH secret volume
Set per-item file modes on the MPI SSH auth Secret volume to prevent
the private key from being world-readable inside training pods.
Private key uses 0o640 (not 0o600) because the builtin
deepspeed-distributed and mlx-distributed runtimes run as
runAsUser: 1000 while Secret volumes mount as root:root.
A volume-level DefaultMode of 0o400 is added as defense-in-depth
for any future items added without an explicit per-item mode.
Fixes#3262
Signed-off-by: Dmytro Hryshchenko <[email protected]>
* docs: update pkg/constants/constants.go
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Dmytro Hryshchenko <[email protected]>
---------
Signed-off-by: Dmytro Hryshchenko <[email protected]>
Signed-off-by: Dmytro Hryshchenko <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
* feat(api): add progress types to api
Signed-off-by: Rob Bell <[email protected]>
* feat: progress server stub
Signed-off-by: Rob Bell <[email protected]>
* feat: setup tls for server
Signed-off-by: Rob Bell <[email protected]>
* refactor: pull out middleware
Signed-off-by: Rob Bell <[email protected]>
* refactor: remove logger message noise from tests
Signed-off-by: Rob Bell <[email protected]>
* fix: send server error logs to main logger
Signed-off-by: Rob Bell <[email protected]>
* feat: update the trainer status
Signed-off-by: Rob Bell <[email protected]>
* feat: use separate client to avoid impacting main reconciler client-side rate limits
Signed-off-by: Rob Bell <[email protected]>
* feat: configure progress server in helm chart and configmap
Signed-off-by: Rob Bell <[email protected]>
* fix: prevent race condition
Signed-off-by: Rob Bell <[email protected]>
* feat: inject progress runtime config
Signed-off-by: Rob Bell <[email protected]>
* fix: progress client config
Signed-off-by: Rob Bell <[email protected]>
* fix: update generated config
Signed-off-by: Rob Bell <[email protected]>
* feat: validate auth tokens in endpoint
Signed-off-by: Rob Bell <[email protected]>
* test: validate auth tokens in endpoint
Signed-off-by: Rob Bell <[email protected]>
* refactor: tidy tests to use same format as other tests
Signed-off-by: Rob Bell <[email protected]>
* fix: look up the cluster issuer from the service account token
Different clusters can use different issuers so look up the issuer from the trusted in-cluster service account token.
Signed-off-by: Rob Bell <[email protected]>
* refactor: rename authn middleware
Signed-off-by: Rob Bell <[email protected]>
* feat: add authn to status update endpoint
Signed-off-by: Rob Bell <[email protected]>
* refactor: tidy up reading in-cluster service account token
Signed-off-by: Rob Bell <[email protected]>
* fix: rbac permissions
Signed-off-by: Rob Bell <[email protected]>
* fix: better error response
Signed-off-by: Rob Bell <[email protected]>
* refactor: move kubebuilder annotation higher up
Signed-off-by: Rob Bell <[email protected]>
* fix: make lastUpdateTime required
Signed-off-by: Rob Bell <[email protected]>
* chore: fix config tests after rebase
Signed-off-by: Rob Bell <[email protected]>
* test: update config test with new progress server config
Signed-off-by: Rob Bell <[email protected]>
* test: add progress tracking e2e test
Signed-off-by: Rob Bell <[email protected]>
* fix: allow feature gates to be configurable from command line argument
So they can be overridden in e2e tests
Signed-off-by: Rob Bell <[email protected]>
* refactor: reuse shared progress StatusUrl
Signed-off-by: Rob Bell <[email protected]>
* test: tidy up tests
Signed-off-by: Rob Bell <[email protected]>
* refactor: allow plugins to use config
Signed-off-by: Rob Bell <[email protected]>
* refactor: centralise shared constants
Signed-off-by: Rob Bell <[email protected]>
* fix: disable http2 in progress server tls opts if disabled by config
Signed-off-by: Rob Bell <[email protected]>
* fix: minor improvements suggested by copilot
Signed-off-by: Rob Bell <[email protected]>
* chore: rename progress server => status server
Signed-off-by: Rob Bell <[email protected]>
* chore: update type doc
Signed-off-by: Rob Bell <[email protected]>
* chore: update name of feature gate
Signed-off-by: Rob Bell <[email protected]>
* fix: after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: fix typo
Signed-off-by: Rob Bell <[email protected]>
* chore: rename handler
Signed-off-by: Rob Bell <[email protected]>
* fix: run status server on all replicas
Signed-off-by: Rob Bell <[email protected]>
* fix: deep copy the rest config
Signed-off-by: Rob Bell <[email protected]>
* refactor: use ssa
Signed-off-by: Rob Bell <[email protected]>
* refactor: separate token audience per train job
Signed-off-by: Rob Bell <[email protected]>
* chore: rename status server
Signed-off-by: Rob Bell <[email protected]>
* fix: revert adding status pkg to manifest gen paths
Signed-off-by: Rob Bell <[email protected]>
* chore: update log messages
Signed-off-by: Rob Bell <[email protected]>
* chore: update api docs
Signed-off-by: Rob Bell <[email protected]>
* chore: update format
Signed-off-by: Rob Bell <[email protected]>
* fix: revert inadvertent change
Signed-off-by: Rob Bell <[email protected]>
* chore: update api name and type docs
Signed-off-by: Rob Bell <[email protected]>
* refactor: initialise oidc provider in manager startup
Signed-off-by: Rob Bell <[email protected]>
* fix: allow empty update request
Signed-off-by: Rob Bell <[email protected]>
* fix: wait for server shut down
Signed-off-by: Rob Bell <[email protected]>
* fix: handle missing train job
Signed-off-by: Rob Bell <[email protected]>
* refactor: update env var names
Signed-off-by: Rob Bell <[email protected]>
* chore: rename packages, plugins, feature gate
Signed-off-by: Rob Bell <[email protected]>
* refactor: update env var names
Signed-off-by: Rob Bell <[email protected]>
* chore: regenerate after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: fixes after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: minor doc updates
Signed-off-by: Rob Bell <[email protected]>
* chore: rename status server config
Signed-off-by: Rob Bell <[email protected]>
* chore: replace placeholder config with nil
Signed-off-by: Rob Bell <[email protected]>
* chore: revert separate runtime utils package
Signed-off-by: Rob Bell <[email protected]>
* chore: remove unused config from tests
Signed-off-by: Rob Bell <[email protected]>
---------
Signed-off-by: Rob Bell <[email protected]>
* Add XGBoost runtime plugin scaffold and register in framework
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
* regenerated assets
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
* Add XGBoost plugin unit tests and update framework test registry
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
* added :Runtime image , unit tests for :(Resources are not set, Resources are set in TrainJob) ,using the ContainerTrainerPort instead of default port
Signed-off-by: krishna-kg732 <[email protected]>
* minor changes
Signed-off-by: krishna-kg732 <[email protected]>
* added plainML fallback(JAX+XGBOOST)
Signed-off-by: krishna-kg732 <[email protected]>
* test: add XGBoost runtime resource resolution unit tests
- Add test case for resources set in Runtime only
- Add test case for resources set in both Runtime and TrainJob
- Add xgboost_distributed.yaml ClusterTrainingRuntime manifest
- Update kustomization.yaml with xgboost-runtime image
Addresses feedback from PR #3200 for complete resource resolution test coverage.
Signed-off-by: krishna-kg732 <[email protected]>
* added autogenerat3ed files
Signed-off-by: krishna-kg732 <[email protected]>
* CI fixes
Signed-off-by: krishna-kg732 <[email protected]>
* chore: fix ci lint errors
Signed-off-by: krishna-kg732 <[email protected]>
* Add XGBoost integration and E2E tests, fix Validate guard clause and nil safety
Signed-off-by: krishna-kg732 <[email protected]>
* trigger ci
Signed-off-by: krishna-kg732 <[email protected]>
* fix: Add XGBoost support to test wrapper, CEL validation, and CI publish workflow
- Add XGBoostPolicy() method to MLPolicySourceWrapper for integration tests
- Add has(self.xgboost) to MLPolicy CEL mutual-exclusivity validation rule
- Regenerate CRD manifests with updated CEL rule
- Add xgboost-runtime to build-and-push-images CI workflow
- Add XGBoost image build step to E2E setup script (commented out due to Kind bug)
Signed-off-by: krishna-kg732 <[email protected]>
* docs: Add XGBoost distributed training example notebook
Add example notebook demonstrating distributed XGBoost training using
the Kubeflow Trainer SDK and the xgboost-distributed ClusterTrainingRuntime.
The notebook uses the modern XGBoost Collective API with RabitTracker
for distributed coordination and synthetic classification data.
Signed-off-by: krishna-kg732 <[email protected]>
* test: Skip XGBoost E2E test until runtime image is published
Signed-off-by: krishna-kg732 <[email protected]>
* chore : address review comments
Signed-off-by: krishna-kg732 <[email protected]>
* chore:resolve merge conflicts
Signed-off-by: krishna-kg732 <[email protected]>
---------
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
* feat: support for flux framework as hpc manager
Flux supports the majority of MPI flavors/variants, and can be used
to bootstrap MPI as a plugin. It adds other features for scheduling
and topology that can be used for simulations and ai/ml jobs.
This changeset adds the plugin implementation, including the
plugin module, tests, and an example with a small README to
serve as documentation for the time being.
Signed-off-by: vsoch <[email protected]>
* review: slim down use of annotations
We still need to put the original command in an annotation
to retieve later, but others can be re-derived from the
environment
Signed-off-by: vsoch <[email protected]>
* review: clean up tasks and remove extra FluxPolicySource
Signed-off-by: vsoch <[email protected]>
* review: ensure to unset trainer args
Signed-off-by: vsoch <[email protected]>
* review: better support for nproc per node and nodes
Signed-off-by: vsoch <[email protected]>
* review: add constants types for flux
Signed-off-by: vsoch <[email protected]>
* review: add gpu specification from resources
Signed-off-by: vsoch <[email protected]>
* Fix TrainJob NumProcPerNode
Signed-off-by: Andrey Velichkevich <[email protected]>
* Remove elasticPolicy from kubebuilder:validation
Signed-off-by: Andrey Velichkevich <[email protected]>
* chore: use go embed for long string templates flux plugin
Signed-off-by: vsoch <[email protected]>
* cleanup: removed un-used annotation
Signed-off-by: vsoch <[email protected]>
---------
Signed-off-by: vsoch <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>
Co-authored-by: vsoch <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
