The lora runtime's node container is ghcr.io/hanzoai/gym:0.1.1-amd64 —
the Rust trainer, driven by the same env contract. qlora and full stay on
finetune-runtime until gym serves them.
Co-authored-by: Hanzo Dev <[email protected]>
ns kubeflow runs ghcr.io/kubeflow/* images — upstream, and on :latest — even
though this fork exists and carries our commits. It runs them because nothing
here ever published an alternative: the repo has upstream .github/workflows
(which do not run on our estate) and no .hanzo lane at all, so there was no
hanzo image to point at.
Builds exactly the images the cluster consumes, no more. Building artifacts
nothing deploys is how a pipeline rots unnoticed.
⚠️ The hanzo-ml ORG HAS NO ACTIONS SECRETS — hanzoai carries GHCR_USER,
GHCR_TOKEN, GH_PAT, OCI_*, KMS_*; hanzo-ml carries none. So this BUILDS
unconditionally (which proves the fork still compiles — the thing most worth
knowing) and PUSHES only when a token is present, emitting a ::warning:: when
it is not. Until the org gets credentials the run is honestly green with "built
but not published" in the log, rather than the cluster silently serving upstream
forever.
To finish: give hanzo-ml GHCR_USER/GHCR_TOKEN (KMS-sourced, per policy), then
repoint the ns kubeflow deployments at ghcr.io/hanzoai/<image>:sha-<sha7>.
Also note the repo is a PULL MIRROR on the forge, so Actions were off entirely
(has_actions=false); enabled now, and the sync interval cut 8h -> 10m so a push
is picked up in minutes.
Co-authored-by: Hanzo Dev <[email protected]>
Adds one transformers + PEFT + TRL ClusterTrainingRuntime per method that
fine-tunes ANY HuggingFace causal LM. The base model + dataset come from the
TrainJob's model/dataset initializers (hf:// or s3://, secretRef for private/gated
repos); every hyperparameter is read from the trainer container env that the Hanzo
Cloud broker sets. Adapters are merged into the base on completion, so one serving
path (KServe HuggingFace runtime) works for lora/qlora/full alike; output is
written to OUTPUT_DIR (the org's S3 Space when s3://).
- manifests/base/runtimes/hanzo_finetune.yaml 3 ClusterTrainingRuntimes (+kustomization)
- cmd/trainers/hanzo/finetune.py env-driven entrypoint (LoRA/QLoRA/full)
- cmd/trainers/hanzo/finetune_test.py pytest for the pure config/format helpers
- cmd/trainers/hanzo/{Dockerfile,requirements.txt,README.md}
- examples/yaml/hanzo-finetune-qlora.yaml example TrainJob (the broker's output)
kustomize build clean · ruff clean · 14 pytest pass. Image built in CI
(ghcr.io/hanzoai/finetune-runtime), not locally.
Co-authored-by: Hanzo Dev <[email protected]>
* test(torch): add unit tests for getModelFromRuntimeRef
Cover the runtime-ref name parsing used to resolve the TorchTune
pretrained model: normalization of valid names and the empty-string
fallback for malformed refs (wrong number of parts, empty input).
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): add unit tests for isLoraConfigEnabled
Cover LoRA detection from trainer args: the model.lora_attn_modules
marker present, absent, and the empty-args case.
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): add unit tests for isUseQLoraFinetune
Cover QLoRA detection from trainer args, including the DoRA
short-circuit that disables QLoRA even when quantize_base is set.
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): add unit tests for getRecipeAndConfig
Cover the recipe/config selection matrix: single-device, single-node
multi-GPU, and multi-node, each across full / LoRA / QLoRA fine-tuning.
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): add unit tests for extractOverridesFromRuntime
Cover extraction of immutable runtime config overrides from the JobSet
template: trainer-ancestor node container returns only immutable configs,
while non-trainer ancestors, non-node containers, and an absent template
yield no overrides.
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): use year-less copyright header
Align the new test file with the repo boilerplate (hack/boilerplate/boilerplate.go.txt) and CNCF/LF guidance, which drop the year from the copyright notice.
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): cover integer numProcPerNode in getRecipeAndConfig
Exercise integer numProcPerNode counts alongside the existing "auto" cases: numProcPerNode=1 forces the single-device recipe even with multiple GPUs, while a count >1 takes the distributed path.
Signed-off-by: yassinnouh21 <[email protected]>
* fix(torch): respect boolean value when detecting QLoRA args
isUseQLoraFinetune matched quantize_base/use_dora by key only, so model.quantize_base=False enabled QLoRA and model.use_dora=False short-circuited it off. Parse the key=value pair and only treat a truthy value as enabled. Add =False regression cases.
Signed-off-by: yassinnouh21 <[email protected]>
---------
Signed-off-by: yassinnouh21 <[email protected]>
* chore(test): add helm unit tests for torchtune llama3.2-1b runtime
Signed-off-by: yassinnouh21 <[email protected]>
* chore(test): address review feedback on torchtune helm tests
- Assert isKind and metadata.name in the runtimes.defaultEnabled case
for parity with the per-runtime enabled=true case.
- Add a test covering the node job's dependsOn ordering, so the
guarantee that both initializers must Complete before training starts
is protected against silent regression.
- Assert the dataset-initializer STORAGE_URI (hf://tatsu-lab/alpaca)
alongside the model-initializer for symmetric coverage.
Signed-off-by: yassinnouh21 <[email protected]>
* chore(test): add helm unit tests for torchtune llama3.2-3b and qwen2.5-1.5b runtimes
Mirror the llama3.2-1b suite for the remaining two TorchTune
ClusterTrainingRuntimes so all three have equivalent helm-unittest
coverage: render gating (per-runtime enabled + defaultEnabled),
kind/name, framework label, mlPolicy, the three replicatedJobs, the
node job's dependsOn ordering, dataset and model STORAGE_URI, recipe
config, GPU limit, and custom image tag override.
Signed-off-by: yassinnouh21 <[email protected]>
---------
Signed-off-by: yassinnouh21 <[email protected]>
* fix: move ads to footer, light top nav, disable link previews
- Restyle top navigation bar to use light background matching
Kubeflow SDK site (theme-aware CSS variables for light/dark mode)
- Move EthicalAds placement from sidebar to page footer via
Furo template overrides
- Force-disable RTD link preview tooltips via CSS
Signed-off-by: Sridhar1030 <[email protected]>
* fix: use horizontal text ad at page bottom
Switch ad type from vertical image card to horizontal text layout
and push it to the very bottom of the footer.
Signed-off-by: Sridhar1030 <[email protected]>
---------
Signed-off-by: Sridhar1030 <[email protected]>
* fix(runtimes): set file modes on MPI SSH secret volume
Set per-item file modes on the MPI SSH auth Secret volume to prevent
the private key from being world-readable inside training pods.
Private key uses 0o640 (not 0o600) because the builtin
deepspeed-distributed and mlx-distributed runtimes run as
runAsUser: 1000 while Secret volumes mount as root:root.
A volume-level DefaultMode of 0o400 is added as defense-in-depth
for any future items added without an explicit per-item mode.
Fixes#3262
Signed-off-by: Dmytro Hryshchenko <[email protected]>
* docs: update pkg/constants/constants.go
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Dmytro Hryshchenko <[email protected]>
---------
Signed-off-by: Dmytro Hryshchenko <[email protected]>
Signed-off-by: Dmytro Hryshchenko <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
Replaces every mutable version tag (e.g. @v6) with its corresponding
commit SHA1SUM, retaining the tag as a comment for readability.
Pinning prevents supply-chain attacks where a tag is silently moved to
malicious code after review.
Dependabot will keep the pins and the comments for them updates as
upstream actions release new versions.
Signed-off-by: Christian Heusel <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
The webhook Secret template declared empty ca.crt, ca.key, tls.crt and
tls.key keys. Under Helm v4 (which defaults to server-side apply) this
makes Helm the field manager of those keys, conflicting with the
cert-controller rotator that owns and writes them at runtime, breaking
`helm upgrade`. The rotator only updates the Secret (never creates it),
so the empty Secret object is kept while the data block is removed.
Signed-off-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
* feat(ci): add govulncheck CVE detection workflow for PR checks
Signed-off-by: Shilpa Chugh <[email protected]>
* fix(ci): address review feedback for govulncheck workflow
- Fix jq filter bug: use $f.osv instead of .finding.osv after group_by
(context is the group array, not an individual finding)
- Fix script injection: pass new_cves via env vars instead of direct
${{ }} interpolation in JS template literals and shell heredocs
- Fix workflow_dispatch: add fallbacks for concurrency group and base
branch ref, gate PR comment steps on pull_request event
- Add fork gate to skip workflow on fork PRs
- Add set -eo pipefail to compare step for robust error handling
- Assert non-empty scan output before diffing to catch truncated scans
- Paginate listComments calls to handle PRs with many comments
- Add fallback warning when security comment deletion fails
Signed-off-by: Shilpa Chugh <[email protected]>
---------
Signed-off-by: Shilpa Chugh <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
* feat(docs): add initial documentation structure and configuration for Kubeflow Trainer
- Introduced .readthedocs.yaml for ReadTheDocs configuration.
- Created Makefile for building and serving documentation.
- Added Sphinx configuration in docs/conf.py.
- Established index.rst as the main entry point for documentation.
- Developed user guides for various frameworks including PyTorch, JAX, and DeepSpeed.
- Implemented custom CSS for documentation styling.
- Included a distributed data cache guide and a DeepSpeed integration guide.
This commit sets up the foundational documentation for the Kubeflow Trainer, enhancing accessibility and usability for users.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): update user guides and restructure documentation for Kubeflow Trainer
- Renamed and reorganized user guides to better serve different audiences: AI practitioners, cluster operators, and contributors.
- Added new sections for documentation on the Kubeflow Training Operator v1 and legacy guides.
- Enhanced descriptions to clarify the purpose and content of each guide, improving overall accessibility and usability.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): add comprehensive operator guides for Kubeflow Trainer
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): enhance operator guides for Kubeflow Trainer
Signed-off-by: Sridhar1030 <[email protected]>
* fixed user-guides
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): add initial API reference and contributor guides for Kubeflow Trainer
- Introduced API reference documentation for Python SDK and Kubernetes CRD types, including TrainJob, TrainingRuntime, and ClusterTrainingRuntime.
- Created contributor guides covering architecture, community, and contributing processes.
- Added legacy v1 documentation structure with sections for installation and user guides for various frameworks.
- Enhanced local execution documentation with updated examples.
This commit establishes a comprehensive documentation framework to support users and contributors of the Kubeflow Trainer.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): update user guides for distributed training frameworks
- Revised user guides for distributed training with Kubeflow Trainer, including JAX, PyTorch, MLX, DeepSpeed, and the data cache feature.
- Enhanced clarity and structure of documentation to improve user experience and accessibility.
- Added detailed instructions for using TrainJob with various frameworks, emphasizing configuration-driven training and runtime packages.
- Removed outdated content and streamlined sections for better readability.
This commit enhances the documentation framework, making it easier for users to implement distributed training solutions.
Signed-off-by: Sridhar1030 <[email protected]>
* refactor(docs): update index.rst and custom.css for improved documentation layout
- Replaced the old container structure with HTML for doc cards in index.rst, enhancing the visual presentation of quick links.
- Updated CSS styles for doc cards to improve interactivity and appearance, including hover effects and text decoration.
- Added an API Reference section to the documentation layout for better accessibility to technical details.
These changes aim to enhance the user experience and accessibility of the Kubeflow Trainer documentation.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): enhance documentation with new sidebar toggle and external link handling
- Added JavaScript functionality for a collapsible sidebar to improve navigation on larger screens.
- Implemented external link handling to open links in a new tab for better user experience.
- Updated CSS to accommodate new sidebar features and reduce whitespace in the content area.
- Included Sphinx documentation build artifacts in .gitignore for cleaner repository management.
These changes aim to improve the usability and accessibility of the Kubeflow Trainer documentation.
Signed-off-by: Sridhar1030 <[email protected]>
* added leagcy training docs
Signed-off-by: Sridhar1030 <[email protected]>
* refactor(docs): remove API reference section and update documentation layout
- Deleted the API reference section and related files to streamline the documentation.
- Updated index.rst to remove references to the API documentation.
- Enhanced CSS styles for doc cards to improve visual consistency and interactivity.
These changes aim to simplify the documentation structure and improve user navigation within the Kubeflow Trainer documentation.
Signed-off-by: Sridhar1030 <[email protected]>
* refactor(docs): remove CRD API reference generation and update Sphinx configuration
- Eliminated the CRD API reference generation from the Makefile and .readthedocs.yaml to simplify the build process.
- Updated Sphinx configuration to disable fail_on_warning, allowing builds to proceed despite warnings.
- Removed AutoAPI configuration from conf.py and related dependencies from requirements.txt to streamline documentation setup.
These changes aim to enhance the documentation build process and reduce complexity in the project structure.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): expand JAX and MPI user guides with detailed instructions
- Updated the JAX user guide to include comprehensive instructions for creating and monitoring JAX training jobs using the JAXJob custom resource.
- Enhanced the MPI user guide with additional metrics and Docker image building instructions, improving clarity and usability.
- Introduced a new section in the multi-cluster guide detailing the `MultiKueue` feature for efficient management of MPI jobs across clusters.
These changes aim to provide users with clearer guidance and enhance the overall documentation for distributed training frameworks in Kubeflow.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): enhance user guides with grid layout and sidebar improvements
- Introduced a grid layout for the BuiltinTrainer and local execution user guides, improving visual organization and accessibility.
- Added new sections for overview and backend options in the local execution guide, detailing how to run TrainJobs with different backends.
- Updated CSS to make the sidebar sticky during scrolling, enhancing navigation on larger screens.
These changes aim to improve the overall user experience and clarity of the documentation for Kubeflow Trainer.
Signed-off-by: Sridhar1030 <[email protected]>
* fix\(docs\):\ update\ dataset\ download\ instructions\ and\ improve\ links\ in\ user\ guides$'\n'$'\n'-\ Clarified\ that\ each\ rank\ in\ a\ multi-node\ TrainJob\ must\ download\ the\ dataset\ independently.$'\n'-\ Updated\ links\ in\ the\ local\ execution\ and\ Podman\ user\ guides\ to\ point\ directly\ to\ the\ relevant\ sections\ in\ the\ overview.$'\n'-\ Corrected\ the\ path\ for\ accessing\ the\ fine-tuned\ model\ in\ the\ BuiltinTrainer\ guide.$'\n'$'\n'These\ changes\ enhance\ the\ clarity\ and\ usability\ of\ the\ documentation\ for\ Kubeflow\ Trainer. (resolved the copliot reviews)
Signed-off-by: Sridhar1030 <[email protected]>
* docs: update extension framework guide and remove pod template documentation
- Added architecture diagrams to the Kubeflow Trainer Extension Framework guide for better visualization.
- Updated terminology in the index to reflect the transition from Pod Templates to Runtime Patches.
- Removed the outdated PodTemplate documentation to streamline the guides and focus on current practices.
These changes aim to enhance clarity and usability in the documentation for the Kubeflow Trainer.
Signed-off-by: Sridhar1030 <[email protected]>
* docs: update overview and add diagrams
Signed-off-by: Sridhar1030 <[email protected]>
* docs: update Sphinx configuration and improve link references
- Enabled fail_on_warning in Sphinx configuration to ensure builds fail on warnings, enhancing documentation quality.
- Updated links in user guides to point to the correct paths, improving navigation and clarity for users.
These changes aim to enhance the reliability and usability of the documentation for Kubeflow Trainer.
Signed-off-by: Sridhar1030 <[email protected]>
* fixed the MNIST link , updated local process backend to match the main documentation website
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): add missing newline at end of file for pre-commit compliance
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): remove trailing whitespace in podman guide
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): format conf.py for black/flake8 compliance
Signed-off-by: Sridhar1030 <[email protected]>
Made-with: Cursor
* docs: update theme configuration and enhance CSS for improved navigation
- Updated the Furo theme options in `conf.py` to refine color variables for light and dark modes, enhancing visual consistency.
- Introduced a top navigation bar in the CSS with links to examples, GitHub, Slack, and the blog, improving accessibility and user experience.
- Expanded the CSS file to include a comprehensive color palette and responsive design adjustments for better usability across devices.
These changes aim to enhance the overall look and feel of the Kubeflow Trainer documentation, making it more user-friendly and visually appealing.
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): wrap long lines in conf.py to satisfy E501 lint
Break announcement HTML string into concatenated short lines
to keep all lines under the 100-character limit.
Ref #3308
Signed-off-by: Sridhar1030 <[email protected]>
* docs: enhance theme configuration and update local execution guides
- Updated color variables in `conf.py` for improved light and dark mode support.
- Refined CSS styles to utilize new color variables, enhancing visual consistency across the documentation.
- Added architecture diagrams for Docker and Podman backends in their respective guides, clarifying the orchestration process.
- Improved the structure and clarity of the local execution documentation, including error handling and configuration examples.
These changes aim to enhance the overall user experience and visual appeal of the Kubeflow Trainer documentation.
Signed-off-by: Sridhar1030 <[email protected]>
* added homepage
Signed-off-by: Sridhar1030 <[email protected]>
* fixed routing
Signed-off-by: Sridhar1030 <[email protected]>
* Added remaining pages
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): trim trailing whitespace for pre-commit
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): add Kubeflow boilerplate header to conf.py
Required by the code-quality verify-boilerplate check added upstream.
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): address PR review feedback
- Consolidate landing page: remove home.rst, migrate links to index.rst
- Vendor overview and data-cache images into docs/images/
- Fix apiVersion v2alpha1 → v1alpha1 in runtime-patches and megatron
- Add missing runtimes to supported-runtimes table
- Expand overview ecosystem list, remove phantom SDK bullet and stray emoji
- Replace hard-coded GitHub stats with ADOPTERS.md link
- Delete kueue.md stub, use external toctree link
- Fix torchtune overview link and wildcard import
- Fix xgboost typos and add distributed training example
- Replace vendored contributing/community with link-out stubs
- Add data-cache architecture diagrams
- Move top nav to Sphinx extension (docs/_ext/kubeflow_topnav/)
- Fix brokeGitHub line anchors in legacy fine-tuning reference
- Add missing import os in legacy getting-started
Signed-off-by: Sridhar1030 <[email protected]>
* docs(operator-guides): add torch envInjection section to ml-policy
Add documentation for torch.envInjection.targets in MLPolicy, allowing
PET_* env vars to be injected into init/sidecar containers.
Ref: kubeflow/website#4399, kubeflow/trainer#3417
Signed-off-by: Sridhar1030 <[email protected]>
---------
Signed-off-by: Sridhar1030 <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
* feat: inject PET envs into init containers via envInjection config
- Add EnvInjection and EnvInjectionTarget types to TorchMLPolicySource
- Add FindContainerByPodSetName helper in runtime package
- Update torch plugin to inject PET envs into additional containers
- Update jobset plugin to sync init containers
- Add 5 comprehensive unit tests for envInjection scenarios
- Add Python API models for envInjection types
Signed-off-by: Peter Pan <[email protected]>
* fix: add missing +optional markers for kubeapilinter validation
Signed-off-by: Peter Pan <[email protected]>
* fix: change JobName to optional field for kubeapilinter
Change JobName from required to optional with omitempty tag to satisfy
kubeapilinter validation. Add default marker to CRD schema for listMapKey
requirement.
Signed-off-by: Peter Pan <[email protected]>
* fix test
Signed-off-by: Peter Pan <[email protected]>
* fix: address envInjection review feedback
Reject missing envInjection targets instead of silently skipping them, align the API schema with required fields, and harden init container propagation so misconfigured runtimes fail predictably.
Signed-off-by: Peter Pan <[email protected]>
* fix: sync envInjection generated files
Add the generated boilerplate and Python model updates expected by make generate so CI does not report stale generated assets.
Signed-off-by: Peter Pan <[email protected]>
* fix: satisfy kube API lint for envInjection fields
Keep the envInjection fields marked as required while preserving omitempty tags required by kubeapilinter.
Signed-off-by: Peter Pan <[email protected]>
* fix: update envInjection OpenAPI defaults
Sync generated OpenAPI and Python model output after restoring omitempty on required envInjection fields.
Signed-off-by: Peter Pan <[email protected]>
* fix: address latest review comments
Signed-off-by: Peter Pan <[email protected]>
* chore: remove test plan file per review feedback
The KEP README already has a Test Plan section; a separate file with
local environment details does not belong in the upstream PR.
Signed-off-by: Peter Pan <[email protected]>
---------
Signed-off-by: Peter Pan <[email protected]>
The `cache` dataset initializer compared the exception instance to the
`ConflictError` class using `is`, which is always False:
except ApiException as e:
if e is ConflictError: # always False
`core_v1.create_namespaced_service()` raises a plain `ApiException` with
`status=409`, not a `ConflictError`, so this branch was unreachable. When
the Service already existed (e.g., re-reconcile after a crash between
Service creation and LWS readiness), the code re-raised, fell into the
outer cleanup, and deleted the ServiceAccount — leaving the cache
cluster in a broken state.
Match the pattern used a few lines above for the ServiceAccount and
check `e.status == 409` instead.
Add a unit test that mocks `create_namespaced_service` to raise 409 and
asserts `download_dataset` completes without calling
`delete_namespaced_service_account`.
Signed-off-by: 1fanwang <[email protected]>
* docs: Add CODE_OF_CONDUCT.md for CNCF graduation recommendation
Part of #976
Adds a CODE_OF_CONDUCT.md file to improve discoverability of the Kubeflow Code of Conduct and surface the Code of Conduct tab in the repository.
Signed-off-by: Lucien <[email protected]>
* Update title and add reporting instructions
Signed-off-by: Lucien <[email protected]>
---------
Signed-off-by: Lucien <[email protected]>
zeekay
