* fix: move ads to footer, light top nav, disable link previews
- Restyle top navigation bar to use light background matching
Kubeflow SDK site (theme-aware CSS variables for light/dark mode)
- Move EthicalAds placement from sidebar to page footer via
Furo template overrides
- Force-disable RTD link preview tooltips via CSS
Signed-off-by: Sridhar1030 <[email protected]>
* fix: use horizontal text ad at page bottom
Switch ad type from vertical image card to horizontal text layout
and push it to the very bottom of the footer.
Signed-off-by: Sridhar1030 <[email protected]>
---------
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): add initial documentation structure and configuration for Kubeflow Trainer
- Introduced .readthedocs.yaml for ReadTheDocs configuration.
- Created Makefile for building and serving documentation.
- Added Sphinx configuration in docs/conf.py.
- Established index.rst as the main entry point for documentation.
- Developed user guides for various frameworks including PyTorch, JAX, and DeepSpeed.
- Implemented custom CSS for documentation styling.
- Included a distributed data cache guide and a DeepSpeed integration guide.
This commit sets up the foundational documentation for the Kubeflow Trainer, enhancing accessibility and usability for users.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): update user guides and restructure documentation for Kubeflow Trainer
- Renamed and reorganized user guides to better serve different audiences: AI practitioners, cluster operators, and contributors.
- Added new sections for documentation on the Kubeflow Training Operator v1 and legacy guides.
- Enhanced descriptions to clarify the purpose and content of each guide, improving overall accessibility and usability.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): add comprehensive operator guides for Kubeflow Trainer
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): enhance operator guides for Kubeflow Trainer
Signed-off-by: Sridhar1030 <[email protected]>
* fixed user-guides
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): add initial API reference and contributor guides for Kubeflow Trainer
- Introduced API reference documentation for Python SDK and Kubernetes CRD types, including TrainJob, TrainingRuntime, and ClusterTrainingRuntime.
- Created contributor guides covering architecture, community, and contributing processes.
- Added legacy v1 documentation structure with sections for installation and user guides for various frameworks.
- Enhanced local execution documentation with updated examples.
This commit establishes a comprehensive documentation framework to support users and contributors of the Kubeflow Trainer.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): update user guides for distributed training frameworks
- Revised user guides for distributed training with Kubeflow Trainer, including JAX, PyTorch, MLX, DeepSpeed, and the data cache feature.
- Enhanced clarity and structure of documentation to improve user experience and accessibility.
- Added detailed instructions for using TrainJob with various frameworks, emphasizing configuration-driven training and runtime packages.
- Removed outdated content and streamlined sections for better readability.
This commit enhances the documentation framework, making it easier for users to implement distributed training solutions.
Signed-off-by: Sridhar1030 <[email protected]>
* refactor(docs): update index.rst and custom.css for improved documentation layout
- Replaced the old container structure with HTML for doc cards in index.rst, enhancing the visual presentation of quick links.
- Updated CSS styles for doc cards to improve interactivity and appearance, including hover effects and text decoration.
- Added an API Reference section to the documentation layout for better accessibility to technical details.
These changes aim to enhance the user experience and accessibility of the Kubeflow Trainer documentation.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): enhance documentation with new sidebar toggle and external link handling
- Added JavaScript functionality for a collapsible sidebar to improve navigation on larger screens.
- Implemented external link handling to open links in a new tab for better user experience.
- Updated CSS to accommodate new sidebar features and reduce whitespace in the content area.
- Included Sphinx documentation build artifacts in .gitignore for cleaner repository management.
These changes aim to improve the usability and accessibility of the Kubeflow Trainer documentation.
Signed-off-by: Sridhar1030 <[email protected]>
* added leagcy training docs
Signed-off-by: Sridhar1030 <[email protected]>
* refactor(docs): remove API reference section and update documentation layout
- Deleted the API reference section and related files to streamline the documentation.
- Updated index.rst to remove references to the API documentation.
- Enhanced CSS styles for doc cards to improve visual consistency and interactivity.
These changes aim to simplify the documentation structure and improve user navigation within the Kubeflow Trainer documentation.
Signed-off-by: Sridhar1030 <[email protected]>
* refactor(docs): remove CRD API reference generation and update Sphinx configuration
- Eliminated the CRD API reference generation from the Makefile and .readthedocs.yaml to simplify the build process.
- Updated Sphinx configuration to disable fail_on_warning, allowing builds to proceed despite warnings.
- Removed AutoAPI configuration from conf.py and related dependencies from requirements.txt to streamline documentation setup.
These changes aim to enhance the documentation build process and reduce complexity in the project structure.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): expand JAX and MPI user guides with detailed instructions
- Updated the JAX user guide to include comprehensive instructions for creating and monitoring JAX training jobs using the JAXJob custom resource.
- Enhanced the MPI user guide with additional metrics and Docker image building instructions, improving clarity and usability.
- Introduced a new section in the multi-cluster guide detailing the `MultiKueue` feature for efficient management of MPI jobs across clusters.
These changes aim to provide users with clearer guidance and enhance the overall documentation for distributed training frameworks in Kubeflow.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): enhance user guides with grid layout and sidebar improvements
- Introduced a grid layout for the BuiltinTrainer and local execution user guides, improving visual organization and accessibility.
- Added new sections for overview and backend options in the local execution guide, detailing how to run TrainJobs with different backends.
- Updated CSS to make the sidebar sticky during scrolling, enhancing navigation on larger screens.
These changes aim to improve the overall user experience and clarity of the documentation for Kubeflow Trainer.
Signed-off-by: Sridhar1030 <[email protected]>
* fix\(docs\):\ update\ dataset\ download\ instructions\ and\ improve\ links\ in\ user\ guides$'\n'$'\n'-\ Clarified\ that\ each\ rank\ in\ a\ multi-node\ TrainJob\ must\ download\ the\ dataset\ independently.$'\n'-\ Updated\ links\ in\ the\ local\ execution\ and\ Podman\ user\ guides\ to\ point\ directly\ to\ the\ relevant\ sections\ in\ the\ overview.$'\n'-\ Corrected\ the\ path\ for\ accessing\ the\ fine-tuned\ model\ in\ the\ BuiltinTrainer\ guide.$'\n'$'\n'These\ changes\ enhance\ the\ clarity\ and\ usability\ of\ the\ documentation\ for\ Kubeflow\ Trainer. (resolved the copliot reviews)
Signed-off-by: Sridhar1030 <[email protected]>
* docs: update extension framework guide and remove pod template documentation
- Added architecture diagrams to the Kubeflow Trainer Extension Framework guide for better visualization.
- Updated terminology in the index to reflect the transition from Pod Templates to Runtime Patches.
- Removed the outdated PodTemplate documentation to streamline the guides and focus on current practices.
These changes aim to enhance clarity and usability in the documentation for the Kubeflow Trainer.
Signed-off-by: Sridhar1030 <[email protected]>
* docs: update overview and add diagrams
Signed-off-by: Sridhar1030 <[email protected]>
* docs: update Sphinx configuration and improve link references
- Enabled fail_on_warning in Sphinx configuration to ensure builds fail on warnings, enhancing documentation quality.
- Updated links in user guides to point to the correct paths, improving navigation and clarity for users.
These changes aim to enhance the reliability and usability of the documentation for Kubeflow Trainer.
Signed-off-by: Sridhar1030 <[email protected]>
* fixed the MNIST link , updated local process backend to match the main documentation website
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): add missing newline at end of file for pre-commit compliance
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): remove trailing whitespace in podman guide
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): format conf.py for black/flake8 compliance
Signed-off-by: Sridhar1030 <[email protected]>
Made-with: Cursor
* docs: update theme configuration and enhance CSS for improved navigation
- Updated the Furo theme options in `conf.py` to refine color variables for light and dark modes, enhancing visual consistency.
- Introduced a top navigation bar in the CSS with links to examples, GitHub, Slack, and the blog, improving accessibility and user experience.
- Expanded the CSS file to include a comprehensive color palette and responsive design adjustments for better usability across devices.
These changes aim to enhance the overall look and feel of the Kubeflow Trainer documentation, making it more user-friendly and visually appealing.
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): wrap long lines in conf.py to satisfy E501 lint
Break announcement HTML string into concatenated short lines
to keep all lines under the 100-character limit.
Ref #3308
Signed-off-by: Sridhar1030 <[email protected]>
* docs: enhance theme configuration and update local execution guides
- Updated color variables in `conf.py` for improved light and dark mode support.
- Refined CSS styles to utilize new color variables, enhancing visual consistency across the documentation.
- Added architecture diagrams for Docker and Podman backends in their respective guides, clarifying the orchestration process.
- Improved the structure and clarity of the local execution documentation, including error handling and configuration examples.
These changes aim to enhance the overall user experience and visual appeal of the Kubeflow Trainer documentation.
Signed-off-by: Sridhar1030 <[email protected]>
* added homepage
Signed-off-by: Sridhar1030 <[email protected]>
* fixed routing
Signed-off-by: Sridhar1030 <[email protected]>
* Added remaining pages
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): trim trailing whitespace for pre-commit
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): add Kubeflow boilerplate header to conf.py
Required by the code-quality verify-boilerplate check added upstream.
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): address PR review feedback
- Consolidate landing page: remove home.rst, migrate links to index.rst
- Vendor overview and data-cache images into docs/images/
- Fix apiVersion v2alpha1 → v1alpha1 in runtime-patches and megatron
- Add missing runtimes to supported-runtimes table
- Expand overview ecosystem list, remove phantom SDK bullet and stray emoji
- Replace hard-coded GitHub stats with ADOPTERS.md link
- Delete kueue.md stub, use external toctree link
- Fix torchtune overview link and wildcard import
- Fix xgboost typos and add distributed training example
- Replace vendored contributing/community with link-out stubs
- Add data-cache architecture diagrams
- Move top nav to Sphinx extension (docs/_ext/kubeflow_topnav/)
- Fix brokeGitHub line anchors in legacy fine-tuning reference
- Add missing import os in legacy getting-started
Signed-off-by: Sridhar1030 <[email protected]>
* docs(operator-guides): add torch envInjection section to ml-policy
Add documentation for torch.envInjection.targets in MLPolicy, allowing
PET_* env vars to be injected into init/sidecar containers.
Ref: kubeflow/website#4399, kubeflow/trainer#3417
Signed-off-by: Sridhar1030 <[email protected]>
---------
Signed-off-by: Sridhar1030 <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
* feat(docs): add KEP-2599 for mutable runtimes
Proposes allowing TrainingRuntimes and ClusterTrainingRuntimes to be
mutable by introducing TrainingRuntimeSnapshot resources. TrainJobs
snapshot their referenced runtime configuration on first reconciliation,
decoupling job execution from runtime changes.
Key changes:
- New TrainingRuntimeSnapshot CRD to store point-in-time runtime config
- Remove finalizers from runtimes (no longer needed)
- Remove TrainJobWatcher interface and boilerplate
- Automatic migration for existing TrainJobs on upgrade
Addresses issue #2599
Signed-off-by: Rob Bell <[email protected]>
* feat(docs): improve KEP-2599 clarity and completeness
Improvements to the mutable runtimes KEP:
- Clarify operational problems with finalizers (orphaned finalizers,
namespace deletion issues)
- Better explain why runtime updates are risky (fetched on every
reconciliation, no design-level guarantees)
- Add concrete examples of runtime proliferation (pytorch-2.0, pytorch-2.1)
- Add RBAC section showing required permissions
- Improve summary to clearly state API change (new CRD)
- Clarify test cases for migration scenarios
Signed-off-by: Rob Bell <[email protected]>
* docs: address Copilot review comments on KEP-2599
- Fix relative link to include README.md in Trainer v2 design reference
- Remove extra space in Goals heading
- Change 'Training Job' to 'TrainJobs' for consistency
- Fix 'lookup' to 'look up' (correct verb form)
- Add 'create' verb to RBAC permissions for TrainingRuntimeSnapshot
Signed-off-by: Rob Bell <[email protected]>
* docs: clarify KEP-2599 divergence from original design
Signed-off-by: Rob Bell <[email protected]>
* docs: add source runtime annotation to KEP-2599
Signed-off-by: Rob Bell <[email protected]>
* fix: update the KEP based on feedback
Signed-off-by: Rob Bell <[email protected]>
* fix: apply suggestion
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Rob Bell <[email protected]>
* fix: apply suggestion
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Rob Bell <[email protected]>
* fix: apply suggestion
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Rob Bell <[email protected]>
* fix: remove annotation for snapshot
Signed-off-by: Rob Bell <[email protected]>
---------
Signed-off-by: Rob Bell <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
* docs(proposals): add draft KEP for inject PET envs into trainer init containers
Signed-off-by: Peter Pan <[email protected]>
* fix comments
Signed-off-by: Peter Pan <[email protected]>
* address comments
Signed-off-by: Peter Pan <[email protected]>
* Add OPT-In annotation
Polish more
Signed-off-by: Peter Pan <[email protected]>
* add more detailed user story
Signed-off-by: Peter Pan <[email protected]>
* change annotation name to more generic
Signed-off-by: Peter Pan <[email protected]>
* docs(proposals): clarify plugin env injection mode in KEP-3416
Signed-off-by: Peter Pan <[email protected]>
* docs(proposals): switch from annotation to TorchMLPolicy API field for env injection
Signed-off-by: Peter Pan <[email protected]>
* docs(proposals): clarify envInjection default state semantics
Signed-off-by: Peter Pan <[email protected]>
* docs(proposals): refine envInjection API to use initContainers name list
Replace containerTypes enum with explicit initContainers []string for
precise targeting. The main container (node) always receives PET_* envs,
eliminating the need for container type distinction and extra validation.
Signed-off-by: Peter Pan <[email protected]>
* docs(proposals): adopt targets/jobName/containerNames API per review consensus
Use explicit targets list with jobName and containerNames for declarative
API. Main container (node) always injected automatically. jobName chosen
over replicatedJobName for future CRD compatibility (LWS, Grove, Slurm).
Signed-off-by: Peter Pan <[email protected]>
* docs(proposals): fix stale text to align with targets API
Proposal description, Goal, and Non-Goals previously referenced
trainer init containers only. With the targets/jobName/containerNames
API, injection targets any container in any replicated job.
Signed-off-by: Peter Pan <[email protected]>
* docs(proposals): finalize API per offline sync with Andre/tenzen-y
Keep jobName + containerNames (plural) design. Add container existence
validation. Rename TorchEnvInjection to EnvInjection for policy reuse.
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Peter Pan <[email protected]>
* Update docs/proposals/3416-pet-env-init-containers/README.md
Co-authored-by: Rob Bell <[email protected]>
Signed-off-by: Peter Pan <[email protected]>
* chore: fix trailing whitespace in KEP README
Signed-off-by: Peter Pan <[email protected]>
---------
Signed-off-by: Peter Pan <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Rob Bell <[email protected]>
* feat(api): add progress types to api
Signed-off-by: Rob Bell <[email protected]>
* feat: progress server stub
Signed-off-by: Rob Bell <[email protected]>
* feat: setup tls for server
Signed-off-by: Rob Bell <[email protected]>
* refactor: pull out middleware
Signed-off-by: Rob Bell <[email protected]>
* refactor: remove logger message noise from tests
Signed-off-by: Rob Bell <[email protected]>
* fix: send server error logs to main logger
Signed-off-by: Rob Bell <[email protected]>
* feat: update the trainer status
Signed-off-by: Rob Bell <[email protected]>
* feat: use separate client to avoid impacting main reconciler client-side rate limits
Signed-off-by: Rob Bell <[email protected]>
* feat: configure progress server in helm chart and configmap
Signed-off-by: Rob Bell <[email protected]>
* fix: prevent race condition
Signed-off-by: Rob Bell <[email protected]>
* feat: inject progress runtime config
Signed-off-by: Rob Bell <[email protected]>
* fix: progress client config
Signed-off-by: Rob Bell <[email protected]>
* fix: update generated config
Signed-off-by: Rob Bell <[email protected]>
* feat: validate auth tokens in endpoint
Signed-off-by: Rob Bell <[email protected]>
* test: validate auth tokens in endpoint
Signed-off-by: Rob Bell <[email protected]>
* refactor: tidy tests to use same format as other tests
Signed-off-by: Rob Bell <[email protected]>
* fix: look up the cluster issuer from the service account token
Different clusters can use different issuers so look up the issuer from the trusted in-cluster service account token.
Signed-off-by: Rob Bell <[email protected]>
* refactor: rename authn middleware
Signed-off-by: Rob Bell <[email protected]>
* feat: add authn to status update endpoint
Signed-off-by: Rob Bell <[email protected]>
* refactor: tidy up reading in-cluster service account token
Signed-off-by: Rob Bell <[email protected]>
* fix: rbac permissions
Signed-off-by: Rob Bell <[email protected]>
* fix: better error response
Signed-off-by: Rob Bell <[email protected]>
* refactor: move kubebuilder annotation higher up
Signed-off-by: Rob Bell <[email protected]>
* fix: make lastUpdateTime required
Signed-off-by: Rob Bell <[email protected]>
* chore: fix config tests after rebase
Signed-off-by: Rob Bell <[email protected]>
* test: update config test with new progress server config
Signed-off-by: Rob Bell <[email protected]>
* test: add progress tracking e2e test
Signed-off-by: Rob Bell <[email protected]>
* fix: allow feature gates to be configurable from command line argument
So they can be overridden in e2e tests
Signed-off-by: Rob Bell <[email protected]>
* refactor: reuse shared progress StatusUrl
Signed-off-by: Rob Bell <[email protected]>
* test: tidy up tests
Signed-off-by: Rob Bell <[email protected]>
* refactor: allow plugins to use config
Signed-off-by: Rob Bell <[email protected]>
* refactor: centralise shared constants
Signed-off-by: Rob Bell <[email protected]>
* fix: disable http2 in progress server tls opts if disabled by config
Signed-off-by: Rob Bell <[email protected]>
* fix: minor improvements suggested by copilot
Signed-off-by: Rob Bell <[email protected]>
* chore: rename progress server => status server
Signed-off-by: Rob Bell <[email protected]>
* chore: update type doc
Signed-off-by: Rob Bell <[email protected]>
* chore: update name of feature gate
Signed-off-by: Rob Bell <[email protected]>
* fix: after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: fix typo
Signed-off-by: Rob Bell <[email protected]>
* chore: rename handler
Signed-off-by: Rob Bell <[email protected]>
* fix: run status server on all replicas
Signed-off-by: Rob Bell <[email protected]>
* fix: deep copy the rest config
Signed-off-by: Rob Bell <[email protected]>
* refactor: use ssa
Signed-off-by: Rob Bell <[email protected]>
* refactor: separate token audience per train job
Signed-off-by: Rob Bell <[email protected]>
* chore: rename status server
Signed-off-by: Rob Bell <[email protected]>
* fix: revert adding status pkg to manifest gen paths
Signed-off-by: Rob Bell <[email protected]>
* chore: update log messages
Signed-off-by: Rob Bell <[email protected]>
* chore: update api docs
Signed-off-by: Rob Bell <[email protected]>
* chore: update format
Signed-off-by: Rob Bell <[email protected]>
* fix: revert inadvertent change
Signed-off-by: Rob Bell <[email protected]>
* chore: update api name and type docs
Signed-off-by: Rob Bell <[email protected]>
* refactor: initialise oidc provider in manager startup
Signed-off-by: Rob Bell <[email protected]>
* fix: allow empty update request
Signed-off-by: Rob Bell <[email protected]>
* fix: wait for server shut down
Signed-off-by: Rob Bell <[email protected]>
* fix: handle missing train job
Signed-off-by: Rob Bell <[email protected]>
* refactor: update env var names
Signed-off-by: Rob Bell <[email protected]>
* chore: rename packages, plugins, feature gate
Signed-off-by: Rob Bell <[email protected]>
* refactor: update env var names
Signed-off-by: Rob Bell <[email protected]>
* chore: regenerate after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: fixes after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: minor doc updates
Signed-off-by: Rob Bell <[email protected]>
* chore: rename status server config
Signed-off-by: Rob Bell <[email protected]>
* chore: replace placeholder config with nil
Signed-off-by: Rob Bell <[email protected]>
* chore: revert separate runtime utils package
Signed-off-by: Rob Bell <[email protected]>
* chore: remove unused config from tests
Signed-off-by: Rob Bell <[email protected]>
---------
Signed-off-by: Rob Bell <[email protected]>
* feat(docs): KEP-2779 Track TrainJob progress and expose training metrics
Signed-off-by: Rob Bell <[email protected]>
* feat(docs): update progress tracking KEP to use a push-based HTTP mechanism
Signed-off-by: Rob Bell <[email protected]>
* feat(docs): update progress tracking KEP to add an alternative pull-based approach
Signed-off-by: Rob Bell <[email protected]>
* feat(docs): update progress tracking KEP pull-based approach
Add a simpler, more reliable approach to collect the final status.
Signed-off-by: Rob Bell <[email protected]>
* feat(docs): update TrainerStatus api and minor formatting/text updates
Signed-off-by: Rob Bell <[email protected]>
* feat(docs): update to use the push-based approach
Signed-off-by: Rob Bell <[email protected]>
* feat(docs): remove the framework-specific custom trainers.
Signed-off-by: Rob Bell <[email protected]>
* feat(docs): update the endpoint spec and add details of the feature gate
Signed-off-by: Rob Bell <[email protected]>
* feat(docs): update the endpoint spec and update feature gate spec
Signed-off-by: Rob Bell <[email protected]>
* feat(docs): update spec for sidecar
Signed-off-by: Rob Bell <[email protected]>
---------
Signed-off-by: Rob Bell <[email protected]>
* feat: kep for flux hpc (2841)
This KEP proposes adding an hpcPolicy to support Flux
Framework and (in the future) other workload managers
that provide more traditional HPC features.
Signed-off-by: vsoch <[email protected]>
* review: see updates below.
Changed crd examples to reflect documentation
removed tasks from definition - can go in settings
removed mentions of minicluster out of context
specified train image instead of custom logic
added user stories
Signed-off-by: vsoch <[email protected]>
* feat: flux policy
Update the KEP to define a FluxMLPolicySource that
exposes attributes specific to Flux.
Signed-off-by: vsoch <[email protected]>
* review: add details of cm and init container
Signed-off-by: vsoch <[email protected]>
---------
Signed-off-by: vsoch <[email protected]>
Co-authored-by: vsoch <[email protected]>
