mirror of
https://github.com/hanzo-ml/trainer.git
synced 2026-09-24 00:31:34 +00:00
Sabari NarayanaandGitHub
ba2bc4a07a
* added yaml examples for trainer Signed-off-by: narayanasabari <[email protected]> * Examples fixed Signed-off-by: narayanasabari <[email protected]> * fix(examples): address maintainer feedback on YAML examples - Remove hello-world, with-runtime, pytorch-simple examples - Fix multi-node: remove custom image, use PET_* env var prefix - Fix volcano: move podGroupPolicy to ClusterTrainingRuntime - Fix podspec-overrides: move podTemplateOverrides to top-level, use trainer.resourcesPerNode and trainer.env instead of invalid ContainerOverride fields - Trim all READMEs to avoid duplicating kubeflow.org docs Signed-off-by: Sabari Narayana <[email protected]> * feat(examples): migrate YAML examples to RuntimePatches API Address review feedback from @andreyvelich on PR #2925: - Rename 01-podspec-overrides.yaml -> 01-runtime-patches.yaml and rewrite using the new spec.runtimePatches API (PodTemplateOverrides was removed in favor of the manager-keyed RuntimePatches mechanism). - Inline a TrainingRuntime in 04-multi-step.yaml so the dataset-initializer step is actually defined; the default torch-distributed runtime has no initializer replicatedJob and the example would fail validation otherwise. - Replace nonexistent trainer.kubeflow.org/job-name selectors throughout with jobset.sigs.k8s.io/jobset-name (for pods/podgroups/workloads) and trainer.kubeflow.org/trainjob-ancestor-step (to target a specific step). - Bump default PyTorch image to 2.10.0 to match upstream torch-distributed ClusterTrainingRuntime. - Add ManagedBy hint for MultiKueue in the Kueue example. - Trim READMEs to avoid duplicating kubeflow.org documentation. Signed-off-by: Sabari Narayana <[email protected]> * docs(examples): address review feedback on YAML examples Address review feedback from @andreyvelich on PR #2925: - Flatten examples/yaml/: drop the basic/ and advanced/ subfolders since there is only one "basic" example. All YAMLs now live directly under examples/yaml/ and are renumbered 01-05. - Remove namespace: default from all manifests (it is the implicit default and adds noise). - Use torchrun as the entrypoint in 01-multi-node.yaml and 02-runtime-patches.yaml, with torch.distributed API in the script (matches the pattern in the PyTorch user guide). - Restructure examples/README.md: SDK section first, YAML section last; list all framework subfolders (megatron, flux, xgboost, jax, local); refresh SDK snippet from kubeflow/sdk README; link sdk.kubeflow.org. Signed-off-by: Sabari Narayana <[email protected]> --------- Signed-off-by: narayanasabari <[email protected]> Signed-off-by: Sabari Narayana <[email protected]>
Kubeflow Trainer YAML Examples
Standalone manifests that can be applied directly with kubectl. For an end-to-end conceptual walkthrough, see the Kubeflow Trainer documentation.
Prerequisites
-
Kubernetes cluster with Kubeflow Trainer installed
-
kubectlconfigured against the cluster -
The default ClusterTrainingRuntimes installed (shipped with Kubeflow Trainer):
kubectl get clustertrainingruntimes
Examples
| File | Description |
|---|---|
01-multi-node.yaml |
Multi-node distributed PyTorch training launched with torchrun |
02-runtime-patches.yaml |
Pod customization with the runtimePatches API (nodeSelector, tolerations, serviceAccountName, labels, annotations) |
03-kueue-integration.yaml |
Queue-based scheduling with Kueue |
04-volcano-integration.yaml |
Gang scheduling with Volcano |
05-multi-step.yaml |
Dataset-initializer step running before the trainer |
Quick start
kubectl apply -f 01-multi-node.yaml
kubectl get trainjob multi-node-example
kubectl get pods -l jobset.sigs.k8s.io/jobset-name=multi-node-example
kubectl logs -l jobset.sigs.k8s.io/jobset-name=multi-node-example
kubectl delete trainjob multi-node-example