mirror of
https://github.com/hanzo-ml/trainer.git
synced 2026-09-24 00:31:34 +00:00
feat(examples): Add kubectl-friendly YAML examples for TrainJob and TrainingRuntime (#2925)
* added yaml examples for trainer Signed-off-by: narayanasabari <[email protected]> * Examples fixed Signed-off-by: narayanasabari <[email protected]> * fix(examples): address maintainer feedback on YAML examples - Remove hello-world, with-runtime, pytorch-simple examples - Fix multi-node: remove custom image, use PET_* env var prefix - Fix volcano: move podGroupPolicy to ClusterTrainingRuntime - Fix podspec-overrides: move podTemplateOverrides to top-level, use trainer.resourcesPerNode and trainer.env instead of invalid ContainerOverride fields - Trim all READMEs to avoid duplicating kubeflow.org docs Signed-off-by: Sabari Narayana <[email protected]> * feat(examples): migrate YAML examples to RuntimePatches API Address review feedback from @andreyvelich on PR #2925: - Rename 01-podspec-overrides.yaml -> 01-runtime-patches.yaml and rewrite using the new spec.runtimePatches API (PodTemplateOverrides was removed in favor of the manager-keyed RuntimePatches mechanism). - Inline a TrainingRuntime in 04-multi-step.yaml so the dataset-initializer step is actually defined; the default torch-distributed runtime has no initializer replicatedJob and the example would fail validation otherwise. - Replace nonexistent trainer.kubeflow.org/job-name selectors throughout with jobset.sigs.k8s.io/jobset-name (for pods/podgroups/workloads) and trainer.kubeflow.org/trainjob-ancestor-step (to target a specific step). - Bump default PyTorch image to 2.10.0 to match upstream torch-distributed ClusterTrainingRuntime. - Add ManagedBy hint for MultiKueue in the Kueue example. - Trim READMEs to avoid duplicating kubeflow.org documentation. Signed-off-by: Sabari Narayana <[email protected]> * docs(examples): address review feedback on YAML examples Address review feedback from @andreyvelich on PR #2925: - Flatten examples/yaml/: drop the basic/ and advanced/ subfolders since there is only one "basic" example. All YAMLs now live directly under examples/yaml/ and are renumbered 01-05. - Remove namespace: default from all manifests (it is the implicit default and adds noise). - Use torchrun as the entrypoint in 01-multi-node.yaml and 02-runtime-patches.yaml, with torch.distributed API in the script (matches the pattern in the PyTorch user guide). - Restructure examples/README.md: SDK section first, YAML section last; list all framework subfolders (megatron, flux, xgboost, jax, local); refresh SDK snippet from kubeflow/sdk README; link sdk.kubeflow.org. Signed-off-by: Sabari Narayana <[email protected]> --------- Signed-off-by: narayanasabari <[email protected]> Signed-off-by: Sabari Narayana <[email protected]>
This commit is contained in:
Sabari Narayana
+84
-2
@@ -1,5 +1,87 @@
|
||||
# Kubeflow Trainer Examples
|
||||
|
||||
Welcome to Kubeflow Trainer examples!
|
||||
This directory contains examples for using Kubeflow Trainer with different interfaces and frameworks.
|
||||
|
||||
The Kubeflow Trainer documentation is available on [kubeflow.org](https://www.kubeflow.org/docs/components/trainer/).
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
examples/
|
||||
├── pytorch/ # PyTorch SDK examples (AI Practitioners)
|
||||
├── deepspeed/ # DeepSpeed framework examples
|
||||
├── mlx/ # MLX framework examples
|
||||
├── torchtune/ # TorchTune fine-tuning examples
|
||||
├── megatron/ # Megatron-LM examples
|
||||
├── flux/ # Flux workload manager examples
|
||||
├── xgboost/ # XGBoost examples
|
||||
├── jax/ # JAX framework examples
|
||||
├── local/ # Local backend examples
|
||||
└── yaml/ # YAML examples for kubectl users (Platform Admins)
|
||||
```
|
||||
|
||||
## For AI Practitioners (Python SDK)
|
||||
|
||||
Use the [Kubeflow Python SDK](https://sdk.kubeflow.org/en/latest/) for a code-first experience. Install with `pip install -U kubeflow`, then submit a TrainJob:
|
||||
|
||||
```python
|
||||
from kubeflow.trainer import TrainerClient, CustomTrainer, TrainJobTemplate
|
||||
|
||||
def get_torch_dist(learning_rate: str, num_epochs: str):
|
||||
import os
|
||||
import torch
|
||||
import torch.distributed as dist
|
||||
|
||||
dist.init_process_group(backend="gloo")
|
||||
print("PyTorch Distributed Environment")
|
||||
print(f"WORLD_SIZE: {dist.get_world_size()}")
|
||||
print(f"RANK: {dist.get_rank()}")
|
||||
print(f"LOCAL_RANK: {os.environ['LOCAL_RANK']}")
|
||||
|
||||
lr = float(learning_rate)
|
||||
epochs = int(num_epochs)
|
||||
loss = 1.0 - (lr * 2) - (epochs * 0.01)
|
||||
if dist.get_rank() == 0:
|
||||
print(f"loss={loss}")
|
||||
|
||||
template = TrainJobTemplate(
|
||||
runtime="torch-distributed",
|
||||
trainer=CustomTrainer(
|
||||
func=get_torch_dist,
|
||||
func_args={"learning_rate": "0.01", "num_epochs": "5"},
|
||||
num_nodes=3,
|
||||
resources_per_node={"cpu": 2},
|
||||
),
|
||||
)
|
||||
|
||||
job_id = TrainerClient().train(**template)
|
||||
TrainerClient().wait_for_job_status(job_id)
|
||||
print("\n".join(TrainerClient().get_job_logs(name=job_id)))
|
||||
```
|
||||
|
||||
**[Browse Python SDK Examples](./pytorch/)**
|
||||
|
||||
## For Platform Administrators (YAML + kubectl)
|
||||
|
||||
Ready-to-use YAML manifests that can be applied directly with `kubectl`:
|
||||
|
||||
```bash
|
||||
# Multi-node distributed training
|
||||
kubectl apply -f yaml/01-multi-node.yaml
|
||||
|
||||
# Pod customization with the runtimePatches API
|
||||
kubectl apply -f yaml/02-runtime-patches.yaml
|
||||
```
|
||||
|
||||
**[Browse YAML Examples](./yaml/)**
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Kubeflow Trainer Documentation](https://www.kubeflow.org/docs/components/trainer/)
|
||||
- [Getting Started Guide](https://www.kubeflow.org/docs/components/trainer/getting-started/)
|
||||
- [Runtime Guide](https://www.kubeflow.org/docs/components/trainer/operator-guides/runtime/)
|
||||
- [Kubeflow SDK Documentation](https://sdk.kubeflow.org/en/latest/)
|
||||
|
||||
## Contributing
|
||||
|
||||
Found a bug or have a feature request? Please [open an issue](https://github.com/kubeflow/trainer/issues/new)!
|
||||
|
||||
Want to contribute an example? Check out our [contributing guidelines](../CONTRIBUTING.md).
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# Example 1: Multi-Node Distributed Training
|
||||
#
|
||||
# Multi-node distributed PyTorch training on the default torch-distributed
|
||||
# ClusterTrainingRuntime. The trainer is launched with torchrun, which reads
|
||||
# the rendezvous info from the PET_* env vars injected by Trainer
|
||||
# (PET_NNODES, PET_NPROC_PER_NODE, PET_NODE_RANK, PET_MASTER_ADDR,
|
||||
# PET_MASTER_PORT) and exposes WORLD_SIZE / RANK / LOCAL_RANK to each worker.
|
||||
#
|
||||
# See https://www.kubeflow.org/docs/components/trainer/user-guides/pytorch/#pytorch-distributed-environment
|
||||
#
|
||||
# Apply: kubectl apply -f 01-multi-node.yaml
|
||||
# Check: kubectl get trainjob multi-node-example
|
||||
# Logs: kubectl logs -l jobset.sigs.k8s.io/jobset-name=multi-node-example
|
||||
# Clean: kubectl delete trainjob multi-node-example
|
||||
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: TrainJob
|
||||
metadata:
|
||||
name: multi-node-example
|
||||
spec:
|
||||
runtimeRef:
|
||||
apiGroup: trainer.kubeflow.org
|
||||
kind: ClusterTrainingRuntime
|
||||
name: torch-distributed
|
||||
|
||||
trainer:
|
||||
numNodes: 3
|
||||
command:
|
||||
- torchrun
|
||||
- --no-python
|
||||
- python3
|
||||
- -c
|
||||
- |
|
||||
import os
|
||||
import torch
|
||||
import torch.distributed as dist
|
||||
|
||||
device, backend = ("cuda", "nccl") if torch.cuda.is_available() else ("cpu", "gloo")
|
||||
dist.init_process_group(backend=backend)
|
||||
|
||||
print("PyTorch Distributed Environment")
|
||||
print(f"Using device: {device}")
|
||||
print(f"WORLD_SIZE: {dist.get_world_size()}")
|
||||
print(f"RANK: {dist.get_rank()}")
|
||||
print(f"LOCAL_RANK: {os.environ['LOCAL_RANK']}")
|
||||
@@ -0,0 +1,92 @@
|
||||
# Example 2: Runtime Patches
|
||||
#
|
||||
# Pod customization with the runtimePatches API. Each patch is keyed by a
|
||||
# "manager" so users, Kueue, and admission webhooks can each own one entry
|
||||
# without conflicting.
|
||||
#
|
||||
# Notes:
|
||||
# - Resources for training nodes are set via spec.trainer.resourcesPerNode.
|
||||
# - Env vars for the "node" container go in spec.trainer.env
|
||||
# (RuntimePatches container env is forbidden for the "node",
|
||||
# "dataset-initializer", and "model-initializer" containers).
|
||||
# - runtimePatches expose JobSet metadata, Job metadata, Pod metadata, and
|
||||
# a restricted set of Pod spec fields (serviceAccountName, volumes,
|
||||
# nodeSelector, affinity, tolerations, etc.).
|
||||
#
|
||||
# Apply: kubectl apply -f 02-runtime-patches.yaml
|
||||
# Check: kubectl describe trainjob runtime-patches-example
|
||||
# Pods: kubectl get pods -l jobset.sigs.k8s.io/jobset-name=runtime-patches-example
|
||||
# Clean: kubectl delete trainjob runtime-patches-example
|
||||
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: TrainJob
|
||||
metadata:
|
||||
name: runtime-patches-example
|
||||
spec:
|
||||
runtimeRef:
|
||||
apiGroup: trainer.kubeflow.org
|
||||
kind: ClusterTrainingRuntime
|
||||
name: torch-distributed
|
||||
|
||||
trainer:
|
||||
numNodes: 2
|
||||
command:
|
||||
- torchrun
|
||||
- --no-python
|
||||
- python3
|
||||
- -c
|
||||
- |
|
||||
import os
|
||||
import torch
|
||||
import torch.distributed as dist
|
||||
|
||||
device, backend = ("cuda", "nccl") if torch.cuda.is_available() else ("cpu", "gloo")
|
||||
dist.init_process_group(backend=backend)
|
||||
|
||||
print("PyTorch Distributed Environment")
|
||||
print(f"Using device: {device}")
|
||||
print(f"WORLD_SIZE: {dist.get_world_size()}")
|
||||
print(f"RANK: {dist.get_rank()}")
|
||||
print(f"LOCAL_RANK: {os.environ['LOCAL_RANK']}")
|
||||
print(f"CUSTOM_VAR: {os.getenv('CUSTOM_VAR')}")
|
||||
|
||||
env:
|
||||
- name: CUSTOM_VAR
|
||||
value: "custom-value"
|
||||
- name: PYTHONUNBUFFERED
|
||||
value: "1"
|
||||
|
||||
resourcesPerNode:
|
||||
requests:
|
||||
cpu: "2"
|
||||
memory: "4Gi"
|
||||
limits:
|
||||
cpu: "4"
|
||||
memory: "8Gi"
|
||||
|
||||
runtimePatches:
|
||||
- manager: example.io/platform-team
|
||||
trainingRuntimeSpec:
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
team: ml-platform
|
||||
cost-center: research
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "8080"
|
||||
spec:
|
||||
replicatedJobs:
|
||||
- name: node
|
||||
template:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
serviceAccountName: training-sa
|
||||
nodeSelector:
|
||||
node-type: training
|
||||
tolerations:
|
||||
- key: training-workload
|
||||
operator: Equal
|
||||
value: "true"
|
||||
effect: NoSchedule
|
||||
@@ -0,0 +1,68 @@
|
||||
# Advanced Example 2: Kueue Integration
|
||||
#
|
||||
# Queue-based scheduling and quota management with Kueue. The TrainJob is
|
||||
# created suspended; Kueue unsuspends it when its admission check passes
|
||||
# and the LocalQueue/ClusterQueue has capacity.
|
||||
#
|
||||
# Prerequisites:
|
||||
# - Kueue installed in the cluster
|
||||
# - A LocalQueue named "training-queue" in the target namespace, backed
|
||||
# by a ClusterQueue with quota
|
||||
#
|
||||
# Apply: kubectl apply -f 03-kueue-integration.yaml
|
||||
# Check: kubectl get trainjob kueue-example
|
||||
# Queue: kubectl get workload -l kueue.x-k8s.io/job-name=kueue-example
|
||||
# Pods: kubectl get pods -l jobset.sigs.k8s.io/jobset-name=kueue-example
|
||||
# Clean: kubectl delete trainjob kueue-example
|
||||
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: TrainJob
|
||||
metadata:
|
||||
name: kueue-example
|
||||
labels:
|
||||
# Tells Kueue which LocalQueue to admit this TrainJob into.
|
||||
kueue.x-k8s.io/queue-name: training-queue
|
||||
spec:
|
||||
runtimeRef:
|
||||
apiGroup: trainer.kubeflow.org
|
||||
kind: ClusterTrainingRuntime
|
||||
name: torch-distributed
|
||||
|
||||
# Required: Kueue admits the workload by flipping suspend to false.
|
||||
suspend: true
|
||||
|
||||
# Optional: set to "kueue.x-k8s.io/multikueue" to delegate to MultiKueue
|
||||
# for multi-cluster scheduling. Defaults to the built-in trainjob controller.
|
||||
# managedBy: kueue.x-k8s.io/multikueue
|
||||
|
||||
trainer:
|
||||
numNodes: 3
|
||||
image: python:3.11-slim
|
||||
command:
|
||||
- python3
|
||||
- -c
|
||||
- |
|
||||
import time
|
||||
print("Training started via Kueue")
|
||||
for i in range(10):
|
||||
print(f"Step {i + 1}/10")
|
||||
time.sleep(2)
|
||||
print("Training complete")
|
||||
|
||||
resourcesPerNode:
|
||||
requests:
|
||||
cpu: "2"
|
||||
memory: "4Gi"
|
||||
limits:
|
||||
cpu: "4"
|
||||
memory: "8Gi"
|
||||
|
||||
---
|
||||
# Reference LocalQueue (apply separately if needed):
|
||||
#
|
||||
# apiVersion: kueue.x-k8s.io/v1beta1
|
||||
# kind: LocalQueue
|
||||
# metadata:
|
||||
# name: training-queue
|
||||
# spec:
|
||||
# clusterQueue: cluster-queue
|
||||
@@ -0,0 +1,82 @@
|
||||
# Advanced Example 3: Volcano Gang Scheduling
|
||||
#
|
||||
# Gang scheduling guarantees that all training pods start together (or none
|
||||
# do), avoiding the deadlocks that can occur when a distributed job is
|
||||
# partially admitted.
|
||||
#
|
||||
# podGroupPolicy is a field on the TrainingRuntime, not the TrainJob, so this
|
||||
# example creates a Volcano-enabled ClusterTrainingRuntime and a TrainJob
|
||||
# that references it.
|
||||
#
|
||||
# Prerequisites:
|
||||
# - Volcano scheduler installed in the cluster
|
||||
#
|
||||
# Apply: kubectl apply -f 04-volcano-integration.yaml
|
||||
# Check: kubectl get trainjob volcano-example
|
||||
# Pods: kubectl get pods -l jobset.sigs.k8s.io/jobset-name=volcano-example
|
||||
# Group: kubectl get podgroup -l jobset.sigs.k8s.io/jobset-name=volcano-example
|
||||
# Clean: kubectl delete trainjob volcano-example && \
|
||||
# kubectl delete clustertrainingruntime torch-distributed-volcano
|
||||
#
|
||||
# Reference: https://www.kubeflow.org/docs/components/trainer/operator-guides/job-scheduling/volcano/
|
||||
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: ClusterTrainingRuntime
|
||||
metadata:
|
||||
name: torch-distributed-volcano
|
||||
labels:
|
||||
trainer.kubeflow.org/framework: torch
|
||||
spec:
|
||||
mlPolicy:
|
||||
numNodes: 1
|
||||
torch: {}
|
||||
podGroupPolicy:
|
||||
volcano: {}
|
||||
template:
|
||||
spec:
|
||||
replicatedJobs:
|
||||
- name: node
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: trainer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: node
|
||||
image: pytorch/pytorch:2.10.0-cuda12.8-cudnn9-runtime
|
||||
|
||||
---
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: TrainJob
|
||||
metadata:
|
||||
name: volcano-example
|
||||
spec:
|
||||
runtimeRef:
|
||||
apiGroup: trainer.kubeflow.org
|
||||
kind: ClusterTrainingRuntime
|
||||
name: torch-distributed-volcano
|
||||
|
||||
trainer:
|
||||
numNodes: 4
|
||||
command:
|
||||
- python3
|
||||
- -c
|
||||
- |
|
||||
import os
|
||||
import time
|
||||
node_rank = os.getenv("PET_NODE_RANK", "0")
|
||||
nnodes = os.getenv("PET_NNODES", "1")
|
||||
print(f"Node {node_rank}/{nnodes} started via Volcano gang scheduling")
|
||||
print(f"All {nnodes} nodes are guaranteed to start together")
|
||||
time.sleep(5)
|
||||
print(f"Node {node_rank} training complete")
|
||||
|
||||
resourcesPerNode:
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "2Gi"
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: "4Gi"
|
||||
@@ -0,0 +1,111 @@
|
||||
# Advanced Example 4: Multi-Step Training with Dataset Initialization
|
||||
#
|
||||
# Runs a dataset-initializer Job that downloads a HuggingFace dataset, then
|
||||
# a trainer Job that consumes it from a shared volume. The default
|
||||
# torch-distributed runtime has no dataset-initializer step, so this example
|
||||
# inlines a TrainingRuntime that adds one.
|
||||
#
|
||||
# spec.initializer.dataset.storageUri populates STORAGE_URI on the
|
||||
# dataset-initializer container. The dataset is mounted at /workspace/dataset
|
||||
# (constants.DatasetMountPath) on both the initializer and trainer pods.
|
||||
#
|
||||
# Apply: kubectl apply -f 05-multi-step.yaml
|
||||
# Check: kubectl get trainjob multi-step-example
|
||||
# Init logs: kubectl logs -l trainer.kubeflow.org/trainjob-ancestor-step=dataset-initializer -f
|
||||
# Train logs: kubectl logs -l trainer.kubeflow.org/trainjob-ancestor-step=trainer -f
|
||||
# Clean: kubectl delete trainjob multi-step-example && \
|
||||
# kubectl delete trainingruntime torch-with-dataset-init
|
||||
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: TrainingRuntime
|
||||
metadata:
|
||||
name: torch-with-dataset-init
|
||||
labels:
|
||||
trainer.kubeflow.org/framework: torch
|
||||
spec:
|
||||
mlPolicy:
|
||||
numNodes: 1
|
||||
torch: {}
|
||||
template:
|
||||
spec:
|
||||
replicatedJobs:
|
||||
- name: dataset-initializer
|
||||
replicas: 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: dataset-initializer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: dataset-initializer
|
||||
image: ghcr.io/kubeflow/trainer/dataset-initializer
|
||||
- name: node
|
||||
dependsOn:
|
||||
- name: dataset-initializer
|
||||
status: Complete
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: trainer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: node
|
||||
image: pytorch/pytorch:2.10.0-cuda12.8-cudnn9-runtime
|
||||
|
||||
---
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: TrainJob
|
||||
metadata:
|
||||
name: multi-step-example
|
||||
spec:
|
||||
runtimeRef:
|
||||
apiGroup: trainer.kubeflow.org
|
||||
kind: TrainingRuntime
|
||||
name: torch-with-dataset-init
|
||||
|
||||
initializer:
|
||||
dataset:
|
||||
# Scheme determines the provider: hf:// for HuggingFace, s3:// for S3.
|
||||
storageUri: hf://tatsu-lab/alpaca
|
||||
# Uncomment to authenticate against private datasets:
|
||||
# secretRef:
|
||||
# name: hf-secret # must contain HF_TOKEN
|
||||
|
||||
trainer:
|
||||
numNodes: 2
|
||||
command:
|
||||
- python3
|
||||
- -c
|
||||
- |
|
||||
import os
|
||||
path = "/workspace/dataset"
|
||||
if os.path.exists(path):
|
||||
print(f"Dataset ready at {path} ({len(os.listdir(path))} entries)")
|
||||
else:
|
||||
print(f"Dataset path {path} missing — initializer did not run")
|
||||
for epoch in range(3):
|
||||
print(f"Epoch {epoch + 1}/3")
|
||||
print("Training complete")
|
||||
|
||||
resourcesPerNode:
|
||||
requests:
|
||||
cpu: "2"
|
||||
memory: "4Gi"
|
||||
limits:
|
||||
cpu: "4"
|
||||
memory: "8Gi"
|
||||
|
||||
---
|
||||
# Reference Secret for private HuggingFace datasets (apply separately):
|
||||
#
|
||||
# apiVersion: v1
|
||||
# kind: Secret
|
||||
# metadata:
|
||||
# name: hf-secret
|
||||
# type: Opaque
|
||||
# stringData:
|
||||
# HF_TOKEN: "your_huggingface_token_here"
|
||||
@@ -0,0 +1,39 @@
|
||||
# Kubeflow Trainer YAML Examples
|
||||
|
||||
Standalone manifests that can be applied directly with `kubectl`. For an end-to-end conceptual walkthrough, see the [Kubeflow Trainer documentation](https://www.kubeflow.org/docs/components/trainer/).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Kubernetes cluster with Kubeflow Trainer installed
|
||||
- `kubectl` configured against the cluster
|
||||
- The default ClusterTrainingRuntimes installed (shipped with Kubeflow Trainer):
|
||||
|
||||
```bash
|
||||
kubectl get clustertrainingruntimes
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| [`01-multi-node.yaml`](01-multi-node.yaml) | Multi-node distributed PyTorch training launched with `torchrun` |
|
||||
| [`02-runtime-patches.yaml`](02-runtime-patches.yaml) | Pod customization with the `runtimePatches` API (nodeSelector, tolerations, serviceAccountName, labels, annotations) |
|
||||
| [`03-kueue-integration.yaml`](03-kueue-integration.yaml) | Queue-based scheduling with [Kueue](https://kueue.sigs.k8s.io/) |
|
||||
| [`04-volcano-integration.yaml`](04-volcano-integration.yaml) | Gang scheduling with [Volcano](https://volcano.sh/) |
|
||||
| [`05-multi-step.yaml`](05-multi-step.yaml) | Dataset-initializer step running before the trainer |
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
kubectl apply -f 01-multi-node.yaml
|
||||
kubectl get trainjob multi-node-example
|
||||
kubectl get pods -l jobset.sigs.k8s.io/jobset-name=multi-node-example
|
||||
kubectl logs -l jobset.sigs.k8s.io/jobset-name=multi-node-example
|
||||
kubectl delete trainjob multi-node-example
|
||||
```
|
||||
|
||||
## See also
|
||||
|
||||
- [Runtime guide](https://www.kubeflow.org/docs/components/trainer/operator-guides/runtime/)
|
||||
- [Job scheduling guide](https://www.kubeflow.org/docs/components/trainer/operator-guides/job-scheduling/)
|
||||
- [Python SDK examples](../pytorch/)
|
||||
Reference in New Issue
Block a user