Adds one transformers + PEFT + TRL ClusterTrainingRuntime per method that
fine-tunes ANY HuggingFace causal LM. The base model + dataset come from the
TrainJob's model/dataset initializers (hf:// or s3://, secretRef for private/gated
repos); every hyperparameter is read from the trainer container env that the Hanzo
Cloud broker sets. Adapters are merged into the base on completion, so one serving
path (KServe HuggingFace runtime) works for lora/qlora/full alike; output is
written to OUTPUT_DIR (the org's S3 Space when s3://).
- manifests/base/runtimes/hanzo_finetune.yaml 3 ClusterTrainingRuntimes (+kustomization)
- cmd/trainers/hanzo/finetune.py env-driven entrypoint (LoRA/QLoRA/full)
- cmd/trainers/hanzo/finetune_test.py pytest for the pure config/format helpers
- cmd/trainers/hanzo/{Dockerfile,requirements.txt,README.md}
- examples/yaml/hanzo-finetune-qlora.yaml example TrainJob (the broker's output)
kustomize build clean · ruff clean · 14 pytest pass. Image built in CI
(ghcr.io/hanzoai/finetune-runtime), not locally.
Co-authored-by: Hanzo Dev <[email protected]>
* added yaml examples for trainer
Signed-off-by: narayanasabari <[email protected]>
* Examples fixed
Signed-off-by: narayanasabari <[email protected]>
* fix(examples): address maintainer feedback on YAML examples
- Remove hello-world, with-runtime, pytorch-simple examples
- Fix multi-node: remove custom image, use PET_* env var prefix
- Fix volcano: move podGroupPolicy to ClusterTrainingRuntime
- Fix podspec-overrides: move podTemplateOverrides to top-level,
use trainer.resourcesPerNode and trainer.env instead of invalid
ContainerOverride fields
- Trim all READMEs to avoid duplicating kubeflow.org docs
Signed-off-by: Sabari Narayana <[email protected]>
* feat(examples): migrate YAML examples to RuntimePatches API
Address review feedback from @andreyvelich on PR #2925:
- Rename 01-podspec-overrides.yaml -> 01-runtime-patches.yaml and rewrite
using the new spec.runtimePatches API (PodTemplateOverrides was removed
in favor of the manager-keyed RuntimePatches mechanism).
- Inline a TrainingRuntime in 04-multi-step.yaml so the dataset-initializer
step is actually defined; the default torch-distributed runtime has no
initializer replicatedJob and the example would fail validation otherwise.
- Replace nonexistent trainer.kubeflow.org/job-name selectors throughout
with jobset.sigs.k8s.io/jobset-name (for pods/podgroups/workloads) and
trainer.kubeflow.org/trainjob-ancestor-step (to target a specific step).
- Bump default PyTorch image to 2.10.0 to match upstream torch-distributed
ClusterTrainingRuntime.
- Add ManagedBy hint for MultiKueue in the Kueue example.
- Trim READMEs to avoid duplicating kubeflow.org documentation.
Signed-off-by: Sabari Narayana <[email protected]>
* docs(examples): address review feedback on YAML examples
Address review feedback from @andreyvelich on PR #2925:
- Flatten examples/yaml/: drop the basic/ and advanced/ subfolders since
there is only one "basic" example. All YAMLs now live directly under
examples/yaml/ and are renumbered 01-05.
- Remove namespace: default from all manifests (it is the implicit default
and adds noise).
- Use torchrun as the entrypoint in 01-multi-node.yaml and 02-runtime-patches.yaml,
with torch.distributed API in the script (matches the pattern in the
PyTorch user guide).
- Restructure examples/README.md: SDK section first, YAML section last;
list all framework subfolders (megatron, flux, xgboost, jax, local);
refresh SDK snippet from kubeflow/sdk README; link sdk.kubeflow.org.
Signed-off-by: Sabari Narayana <[email protected]>
---------
Signed-off-by: narayanasabari <[email protected]>
Signed-off-by: Sabari Narayana <[email protected]>
zeekay
