* feat(docs): add initial documentation structure and configuration for Kubeflow Trainer
- Introduced .readthedocs.yaml for ReadTheDocs configuration.
- Created Makefile for building and serving documentation.
- Added Sphinx configuration in docs/conf.py.
- Established index.rst as the main entry point for documentation.
- Developed user guides for various frameworks including PyTorch, JAX, and DeepSpeed.
- Implemented custom CSS for documentation styling.
- Included a distributed data cache guide and a DeepSpeed integration guide.
This commit sets up the foundational documentation for the Kubeflow Trainer, enhancing accessibility and usability for users.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): update user guides and restructure documentation for Kubeflow Trainer
- Renamed and reorganized user guides to better serve different audiences: AI practitioners, cluster operators, and contributors.
- Added new sections for documentation on the Kubeflow Training Operator v1 and legacy guides.
- Enhanced descriptions to clarify the purpose and content of each guide, improving overall accessibility and usability.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): add comprehensive operator guides for Kubeflow Trainer
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): enhance operator guides for Kubeflow Trainer
Signed-off-by: Sridhar1030 <[email protected]>
* fixed user-guides
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): add initial API reference and contributor guides for Kubeflow Trainer
- Introduced API reference documentation for Python SDK and Kubernetes CRD types, including TrainJob, TrainingRuntime, and ClusterTrainingRuntime.
- Created contributor guides covering architecture, community, and contributing processes.
- Added legacy v1 documentation structure with sections for installation and user guides for various frameworks.
- Enhanced local execution documentation with updated examples.
This commit establishes a comprehensive documentation framework to support users and contributors of the Kubeflow Trainer.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): update user guides for distributed training frameworks
- Revised user guides for distributed training with Kubeflow Trainer, including JAX, PyTorch, MLX, DeepSpeed, and the data cache feature.
- Enhanced clarity and structure of documentation to improve user experience and accessibility.
- Added detailed instructions for using TrainJob with various frameworks, emphasizing configuration-driven training and runtime packages.
- Removed outdated content and streamlined sections for better readability.
This commit enhances the documentation framework, making it easier for users to implement distributed training solutions.
Signed-off-by: Sridhar1030 <[email protected]>
* refactor(docs): update index.rst and custom.css for improved documentation layout
- Replaced the old container structure with HTML for doc cards in index.rst, enhancing the visual presentation of quick links.
- Updated CSS styles for doc cards to improve interactivity and appearance, including hover effects and text decoration.
- Added an API Reference section to the documentation layout for better accessibility to technical details.
These changes aim to enhance the user experience and accessibility of the Kubeflow Trainer documentation.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): enhance documentation with new sidebar toggle and external link handling
- Added JavaScript functionality for a collapsible sidebar to improve navigation on larger screens.
- Implemented external link handling to open links in a new tab for better user experience.
- Updated CSS to accommodate new sidebar features and reduce whitespace in the content area.
- Included Sphinx documentation build artifacts in .gitignore for cleaner repository management.
These changes aim to improve the usability and accessibility of the Kubeflow Trainer documentation.
Signed-off-by: Sridhar1030 <[email protected]>
* added leagcy training docs
Signed-off-by: Sridhar1030 <[email protected]>
* refactor(docs): remove API reference section and update documentation layout
- Deleted the API reference section and related files to streamline the documentation.
- Updated index.rst to remove references to the API documentation.
- Enhanced CSS styles for doc cards to improve visual consistency and interactivity.
These changes aim to simplify the documentation structure and improve user navigation within the Kubeflow Trainer documentation.
Signed-off-by: Sridhar1030 <[email protected]>
* refactor(docs): remove CRD API reference generation and update Sphinx configuration
- Eliminated the CRD API reference generation from the Makefile and .readthedocs.yaml to simplify the build process.
- Updated Sphinx configuration to disable fail_on_warning, allowing builds to proceed despite warnings.
- Removed AutoAPI configuration from conf.py and related dependencies from requirements.txt to streamline documentation setup.
These changes aim to enhance the documentation build process and reduce complexity in the project structure.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): expand JAX and MPI user guides with detailed instructions
- Updated the JAX user guide to include comprehensive instructions for creating and monitoring JAX training jobs using the JAXJob custom resource.
- Enhanced the MPI user guide with additional metrics and Docker image building instructions, improving clarity and usability.
- Introduced a new section in the multi-cluster guide detailing the `MultiKueue` feature for efficient management of MPI jobs across clusters.
These changes aim to provide users with clearer guidance and enhance the overall documentation for distributed training frameworks in Kubeflow.
Signed-off-by: Sridhar1030 <[email protected]>
* feat(docs): enhance user guides with grid layout and sidebar improvements
- Introduced a grid layout for the BuiltinTrainer and local execution user guides, improving visual organization and accessibility.
- Added new sections for overview and backend options in the local execution guide, detailing how to run TrainJobs with different backends.
- Updated CSS to make the sidebar sticky during scrolling, enhancing navigation on larger screens.
These changes aim to improve the overall user experience and clarity of the documentation for Kubeflow Trainer.
Signed-off-by: Sridhar1030 <[email protected]>
* fix\(docs\):\ update\ dataset\ download\ instructions\ and\ improve\ links\ in\ user\ guides$'\n'$'\n'-\ Clarified\ that\ each\ rank\ in\ a\ multi-node\ TrainJob\ must\ download\ the\ dataset\ independently.$'\n'-\ Updated\ links\ in\ the\ local\ execution\ and\ Podman\ user\ guides\ to\ point\ directly\ to\ the\ relevant\ sections\ in\ the\ overview.$'\n'-\ Corrected\ the\ path\ for\ accessing\ the\ fine-tuned\ model\ in\ the\ BuiltinTrainer\ guide.$'\n'$'\n'These\ changes\ enhance\ the\ clarity\ and\ usability\ of\ the\ documentation\ for\ Kubeflow\ Trainer. (resolved the copliot reviews)
Signed-off-by: Sridhar1030 <[email protected]>
* docs: update extension framework guide and remove pod template documentation
- Added architecture diagrams to the Kubeflow Trainer Extension Framework guide for better visualization.
- Updated terminology in the index to reflect the transition from Pod Templates to Runtime Patches.
- Removed the outdated PodTemplate documentation to streamline the guides and focus on current practices.
These changes aim to enhance clarity and usability in the documentation for the Kubeflow Trainer.
Signed-off-by: Sridhar1030 <[email protected]>
* docs: update overview and add diagrams
Signed-off-by: Sridhar1030 <[email protected]>
* docs: update Sphinx configuration and improve link references
- Enabled fail_on_warning in Sphinx configuration to ensure builds fail on warnings, enhancing documentation quality.
- Updated links in user guides to point to the correct paths, improving navigation and clarity for users.
These changes aim to enhance the reliability and usability of the documentation for Kubeflow Trainer.
Signed-off-by: Sridhar1030 <[email protected]>
* fixed the MNIST link , updated local process backend to match the main documentation website
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): add missing newline at end of file for pre-commit compliance
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): remove trailing whitespace in podman guide
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): format conf.py for black/flake8 compliance
Signed-off-by: Sridhar1030 <[email protected]>
Made-with: Cursor
* docs: update theme configuration and enhance CSS for improved navigation
- Updated the Furo theme options in `conf.py` to refine color variables for light and dark modes, enhancing visual consistency.
- Introduced a top navigation bar in the CSS with links to examples, GitHub, Slack, and the blog, improving accessibility and user experience.
- Expanded the CSS file to include a comprehensive color palette and responsive design adjustments for better usability across devices.
These changes aim to enhance the overall look and feel of the Kubeflow Trainer documentation, making it more user-friendly and visually appealing.
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): wrap long lines in conf.py to satisfy E501 lint
Break announcement HTML string into concatenated short lines
to keep all lines under the 100-character limit.
Ref #3308
Signed-off-by: Sridhar1030 <[email protected]>
* docs: enhance theme configuration and update local execution guides
- Updated color variables in `conf.py` for improved light and dark mode support.
- Refined CSS styles to utilize new color variables, enhancing visual consistency across the documentation.
- Added architecture diagrams for Docker and Podman backends in their respective guides, clarifying the orchestration process.
- Improved the structure and clarity of the local execution documentation, including error handling and configuration examples.
These changes aim to enhance the overall user experience and visual appeal of the Kubeflow Trainer documentation.
Signed-off-by: Sridhar1030 <[email protected]>
* added homepage
Signed-off-by: Sridhar1030 <[email protected]>
* fixed routing
Signed-off-by: Sridhar1030 <[email protected]>
* Added remaining pages
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): trim trailing whitespace for pre-commit
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): add Kubeflow boilerplate header to conf.py
Required by the code-quality verify-boilerplate check added upstream.
Signed-off-by: Sridhar1030 <[email protected]>
* fix(docs): address PR review feedback
- Consolidate landing page: remove home.rst, migrate links to index.rst
- Vendor overview and data-cache images into docs/images/
- Fix apiVersion v2alpha1 → v1alpha1 in runtime-patches and megatron
- Add missing runtimes to supported-runtimes table
- Expand overview ecosystem list, remove phantom SDK bullet and stray emoji
- Replace hard-coded GitHub stats with ADOPTERS.md link
- Delete kueue.md stub, use external toctree link
- Fix torchtune overview link and wildcard import
- Fix xgboost typos and add distributed training example
- Replace vendored contributing/community with link-out stubs
- Add data-cache architecture diagrams
- Move top nav to Sphinx extension (docs/_ext/kubeflow_topnav/)
- Fix brokeGitHub line anchors in legacy fine-tuning reference
- Add missing import os in legacy getting-started
Signed-off-by: Sridhar1030 <[email protected]>
* docs(operator-guides): add torch envInjection section to ml-policy
Add documentation for torch.envInjection.targets in MLPolicy, allowing
PET_* env vars to be injected into init/sidecar containers.
Ref: kubeflow/website#4399, kubeflow/trainer#3417
Signed-off-by: Sridhar1030 <[email protected]>
---------
Signed-off-by: Sridhar1030 <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
* Refactor and merge setup-gpu-cluster into single script based on flag
Signed-off-by: aniket2405 <[email protected]>
* Update gpu script to call the older script
Signed-off-by: aniket2405 <[email protected]>
* merge CPU and GPU E2E workflows into a single file
Signed-off-by: aniket2405 <[email protected]>
* Remove separate test-e2e-gpu.yaml file
Signed-off-by: aniket2405 <[email protected]>
* Extra space added back
Signed-off-by: aniket2405 <[email protected]>
* merge the GPU cluster setup logic into the main e2e script
Signed-off-by: aniket2405 <[email protected]>
* Delete gpu script
Signed-off-by: aniket2405 <[email protected]>
* Remove old GPU YAML file
Signed-off-by: aniket2405 <[email protected]>
* extract repetitive setup steps into reusable composite action
Signed-off-by: aniket2405 <[email protected]>
* fix: final pipeline configuration and custom template setup
Signed-off-by: aniket2405 <[email protected]>
* Update python version
Signed-off-by: aniket2405 <[email protected]>
* fix: restore required checkout step for local actions
Signed-off-by: aniket2405 <[email protected]>
* style: apply end-of-file pre-commit fixes
Signed-off-by: aniket2405 <[email protected]>
* Add new line at end of file
Signed-off-by: aniket2405 <[email protected]>
* Load image structure together
Signed-off-by: aniket2405 <[email protected]>
* Minor refactoring
Signed-off-by: aniket2405 <[email protected]>
* Uncomment JAX image pre-load to prevent E2E test timeout
Signed-off-by: aniket2405 <[email protected]>
* Refactor and update helm test with Oracle runner
Signed-off-by: aniket2405 <[email protected]>
* Add new line at the end for CI pipeline
Signed-off-by: aniket2405 <[email protected]>
---------
Signed-off-by: aniket2405 <[email protected]>
- Update Go from v1.23 to v1.24
- Update golangci-lint from v1.61.0 to v1.64.8
- Fix formatting in CONTRIBUTING.md
Signed-off-by: Vinodh Zamboulingame <[email protected]>
* Add Helm chart for kubeflow trainer
Signed-off-by: Yi Chen <[email protected]>
* Remove Role and RoleBinding resources
Signed-off-by: Yi Chen <[email protected]>
* Update RBAC rules for ConfigMap and Secret
Signed-off-by: Yi Chen <[email protected]>
* Remove RBAC rules for LimitRange
Signed-off-by: Yi Chen <[email protected]>
* Remove RBAC rules for RuntimeClass
Signed-off-by: Yi Chen <[email protected]>
* Update the year for header license comment
Signed-off-by: Yi Chen <[email protected]>
* Update header license comments
Signed-off-by: Yi Chen <[email protected]>
* Update Helm unit tests for manager deployment and service
Signed-off-by: Yi Chen <[email protected]>
* Add more Helm chart unit tests
Signed-off-by: Yi Chen <[email protected]>
* Remove manager role and roleBinding template helpers
Signed-off-by: Yi Chen <[email protected]>
* Remove trailing whitespace
Signed-off-by: Yi Chen <[email protected]>
* Update charts/kubeflow-trainer/values.yaml
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Yi Chen <[email protected]>
* set image.tag to latest
Signed-off-by: Yi Chen <[email protected]>
* Update chart README
Signed-off-by: Yi Chen <[email protected]>
---------
Signed-off-by: Yi Chen <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
