* feat(ci): add Python dependency scanning to OSV-Scanner workflow
Add lockfiles and scanning for Python dependencies in the initializers
and python_api. This extends the nightly OSV-Scanner workflow to cover
Python in addition to Go.
- Generate requirements-lock.txt files via pip-compile for:
- cmd/initializers/dataset/
- cmd/initializers/model/
- api/python_api/
- Add osv-scan-python job with SARIF scanning and auto-fix
- Add validate-lockfile.yaml to ensure lockfiles stay in sync on PRs
- Handle both direct and transitive dependency upgrades
Closes#3528
Signed-off-by: Fiona-Waters <[email protected]>
* fix(ci): address review feedback on Python OSV-Scanner workflow
- Fix validate-lockfile false positives by stripping header comments
before diffing (pip-compile embeds output filename in header)
- Regenerate lockfiles in clean Python 3.12 venv with --strip-extras
to include all transitive deps and silence pip-tools deprecation warning
- Fix PR body table formatting: use Markdown table rows instead of
bullet items in both Go and Python auto-fix jobs
- Escape dots in package names for grep/sed regex safety
Signed-off-by: Fiona-Waters <[email protected]>
* fix(ci): address review feedback on OSV-Scanner Python workflow
- Replace shell grep/sed with Python helper script using
packaging.requirements.Requirement for exact name matching,
fixing prefix collision (requests vs requests-oauthlib),
constraint dropping, and extras stripping
- Skip entry when package not found in lockfile after pip-compile
instead of recording a false version
- Bump actions/setup-python from v5 to v6 for consistency
Signed-off-by: Fiona-Waters <[email protected]>
* fix(ci): pin pip-tools version in CI workflows
Pin pip-tools to 7.4.1 in osv-scanner and validate-lockfile workflows
to mitigate supply-chain risk in workflows with elevated permissions.
Signed-off-by: Fiona-Waters <[email protected]>
* fix(deps): update lockfiles for certifi 2026.5.20
Signed-off-by: Fiona-Waters <[email protected]>
---------
Signed-off-by: Fiona-Waters <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
* Refactor and merge setup-gpu-cluster into single script based on flag
Signed-off-by: aniket2405 <[email protected]>
* Update gpu script to call the older script
Signed-off-by: aniket2405 <[email protected]>
* merge CPU and GPU E2E workflows into a single file
Signed-off-by: aniket2405 <[email protected]>
* Remove separate test-e2e-gpu.yaml file
Signed-off-by: aniket2405 <[email protected]>
* Extra space added back
Signed-off-by: aniket2405 <[email protected]>
* merge the GPU cluster setup logic into the main e2e script
Signed-off-by: aniket2405 <[email protected]>
* Delete gpu script
Signed-off-by: aniket2405 <[email protected]>
* Remove old GPU YAML file
Signed-off-by: aniket2405 <[email protected]>
* extract repetitive setup steps into reusable composite action
Signed-off-by: aniket2405 <[email protected]>
* fix: final pipeline configuration and custom template setup
Signed-off-by: aniket2405 <[email protected]>
* Update python version
Signed-off-by: aniket2405 <[email protected]>
* fix: restore required checkout step for local actions
Signed-off-by: aniket2405 <[email protected]>
* style: apply end-of-file pre-commit fixes
Signed-off-by: aniket2405 <[email protected]>
* Add new line at end of file
Signed-off-by: aniket2405 <[email protected]>
* Load image structure together
Signed-off-by: aniket2405 <[email protected]>
* Minor refactoring
Signed-off-by: aniket2405 <[email protected]>
* Uncomment JAX image pre-load to prevent E2E test timeout
Signed-off-by: aniket2405 <[email protected]>
* Refactor and update helm test with Oracle runner
Signed-off-by: aniket2405 <[email protected]>
* Add new line at the end for CI pipeline
Signed-off-by: aniket2405 <[email protected]>
---------
Signed-off-by: aniket2405 <[email protected]>
* feat: add Megatron-Core GPT Tensor Parallelism example notebook
Signed-off-by: XploY04 <[email protected]>
* added megatron notebook to the e2e gpu test
Signed-off-by: XploY04 <[email protected]>
* feat: Parameterize Megatron core GPT notebook's tensor parallelism and GPU count, and update the e2e test workflow to pass these parameters.
Signed-off-by: XploY04 <[email protected]>
* change the number of gpus to 2
Signed-off-by: XploY04 <[email protected]>
* docs: update Megatron GPT tensor parallelism example notebook.
Signed-off-by: XploY04 <[email protected]>
* fix: correct minor typos and improve code readability in Megatron-Core GPT notebook
Signed-off-by: XploY04 <[email protected]>
* fix: update tensor model parallel size retrieval and improve code clarity in Megatron-Core GPT notebook
Signed-off-by: XploY04 <[email protected]>
* fix: correct wording in training function description for clarity in Megatron-Core GPT notebook
Signed-off-by: XploY04 <[email protected]>
* feat: add verification step for TrainJob completion in Megatron-Core GPT notebook
Signed-off-by: XploY04 <[email protected]>
* fix: use multi-node setup for Megatron TP to work with GPU time-slicing
GPU time-slicing (replicas=2) advertises 2 GPUs to Kubernetes but
exposes only 1 CUDA device inside the container. This caused torchrun
to launch 1 worker (WORLD_SIZE=1), failing Megatron's TP requirement
of world_size >= 2.
Switch from 1 node with 2 GPUs to 2 nodes with 1 GPU each. This
creates 2 pods, each getting 1 time-sliced GPU, giving WORLD_SIZE=2
for tensor parallelism across pods.
Signed-off-by: XploY04 <[email protected]>
* fix: use PyTorch devel image for Megatron compile_helpers() support
Megatron-Core's compile_helpers() requires `make` and `gcc` to build
C dataset helpers. The default runtime image doesn't include build
tools. Switch to the devel image which has them pre-installed.
Signed-off-by: XploY04 <[email protected]>
* fix: install megatron-core from git for compile_helpers() support
The pip package for megatron-core doesn't include the Makefile and
C source files needed by compile_helpers(). Install from the git
repository instead, which includes all source files for compiling
the C++ dataset helpers required by MockGPTDataset.
Signed-off-by: XploY04 <[email protected]>
* fix: install megatron-core from zip archive instead of git
The devel image doesn't have git installed, so pip install from
git+https fails. Use the GitHub zip archive URL instead, which
pip can download and extract directly without needing git.
Signed-off-by: XploY04 <[email protected]>
* fix: download Megatron C++ source files for compile_helpers()
The megatron-core pip package excludes the Makefile and helpers.cpp
needed by compile_helpers(). Download these two files from GitHub
at runtime before compilation. Also add pybind11 as an explicit
dependency since it's needed by the Makefile but only included as
a build-time dependency in the pip package.
Signed-off-by: XploY04 <[email protected]>
* fix: use correct null tokenizer library name for pip megatron-core
The pip release of megatron-core uses "null" as the tokenizer library
name, while the GitHub main branch uses "null-text". Change to "null"
to match the pip version.
Signed-off-by: XploY04 <[email protected]>
* fix: increase wait_for_job_status timeout to 1800s
The megatron training job takes longer than 900s due to pip install,
C++ compilation, and multi-node training setup on time-sliced GPUs.
Increase timeout to 1800s to match PAPERMILL_TIMEOUT.
Signed-off-by: XploY04 <[email protected]>
* fix: use runtime image with apt-get for build tools instead of devel image
The devel image (3.5GB) is too slow to pull, causing PAPERMILL_TIMEOUT.
Switch back to the runtime image and install only make and g++ via
apt-get (~50MB). This provides everything compile_helpers() needs
without the overhead of pulling the full devel image.
Signed-off-by: XploY04 <[email protected]>
* fix: disable NCCL shared memory to avoid /dev/shm limit in Kubernetes
Kubernetes pods get 64MB /dev/shm by default. NCCL needs more for
its communication buffers, causing checkpoint save to fail with
"Error while creating shared memory segment". Set NCCL_SHM_DISABLE=1
to use network sockets instead.
Also reduce wait_for_job_status timeout to 60s since training
completes quickly and the main wait happens in get_job_logs.
Signed-off-by: XploY04 <[email protected]>
* debug: add NCCL_DEBUG=INFO to diagnose /dev/shm failure on node-1
Signed-off-by: XploY04 <[email protected]>
* fix: mount /dev/shm as emptyDir to fix NCCL shared memory exhaustion
NCCL proxy service allocates ~33MB per communicator in /dev/shm. The
default Kubernetes /dev/shm is 64MB (Docker default), which is
insufficient for workloads that create multiple NCCL communicators.
Patch ClusterTrainingRuntimes in GPU e2e setup to mount /dev/shm as
a Memory-backed emptyDir, and remove unnecessary NCCL workaround
env vars from the Megatron notebook.
Signed-off-by: XploY04 <[email protected]>
* fix: remove dist_checkpointing from Megatron notebook
Megatron-Core dist_checkpointing uses multiprocessing.spawn internally
to create a Manager queue for async writes. The Kubeflow SDK generates
training scripts without an if __name__ == '__main__' guard, so the
spawned child re-imports the script and re-executes the training
function, causing a RuntimeError. Remove the checkpoint step since
training (Steps 1-5) is the core TP demonstration. Also remove unused
imports and fix the GPU prerequisites text.
Signed-off-by: XploY04 <[email protected]>
---------
Signed-off-by: XploY04 <[email protected]>
* feat(api): add progress types to api
Signed-off-by: Rob Bell <[email protected]>
* feat: progress server stub
Signed-off-by: Rob Bell <[email protected]>
* feat: setup tls for server
Signed-off-by: Rob Bell <[email protected]>
* refactor: pull out middleware
Signed-off-by: Rob Bell <[email protected]>
* refactor: remove logger message noise from tests
Signed-off-by: Rob Bell <[email protected]>
* fix: send server error logs to main logger
Signed-off-by: Rob Bell <[email protected]>
* feat: update the trainer status
Signed-off-by: Rob Bell <[email protected]>
* feat: use separate client to avoid impacting main reconciler client-side rate limits
Signed-off-by: Rob Bell <[email protected]>
* feat: configure progress server in helm chart and configmap
Signed-off-by: Rob Bell <[email protected]>
* fix: prevent race condition
Signed-off-by: Rob Bell <[email protected]>
* feat: inject progress runtime config
Signed-off-by: Rob Bell <[email protected]>
* fix: progress client config
Signed-off-by: Rob Bell <[email protected]>
* fix: update generated config
Signed-off-by: Rob Bell <[email protected]>
* feat: validate auth tokens in endpoint
Signed-off-by: Rob Bell <[email protected]>
* test: validate auth tokens in endpoint
Signed-off-by: Rob Bell <[email protected]>
* refactor: tidy tests to use same format as other tests
Signed-off-by: Rob Bell <[email protected]>
* fix: look up the cluster issuer from the service account token
Different clusters can use different issuers so look up the issuer from the trusted in-cluster service account token.
Signed-off-by: Rob Bell <[email protected]>
* refactor: rename authn middleware
Signed-off-by: Rob Bell <[email protected]>
* feat: add authn to status update endpoint
Signed-off-by: Rob Bell <[email protected]>
* refactor: tidy up reading in-cluster service account token
Signed-off-by: Rob Bell <[email protected]>
* fix: rbac permissions
Signed-off-by: Rob Bell <[email protected]>
* fix: better error response
Signed-off-by: Rob Bell <[email protected]>
* refactor: move kubebuilder annotation higher up
Signed-off-by: Rob Bell <[email protected]>
* fix: make lastUpdateTime required
Signed-off-by: Rob Bell <[email protected]>
* chore: fix config tests after rebase
Signed-off-by: Rob Bell <[email protected]>
* test: update config test with new progress server config
Signed-off-by: Rob Bell <[email protected]>
* test: add progress tracking e2e test
Signed-off-by: Rob Bell <[email protected]>
* fix: allow feature gates to be configurable from command line argument
So they can be overridden in e2e tests
Signed-off-by: Rob Bell <[email protected]>
* refactor: reuse shared progress StatusUrl
Signed-off-by: Rob Bell <[email protected]>
* test: tidy up tests
Signed-off-by: Rob Bell <[email protected]>
* refactor: allow plugins to use config
Signed-off-by: Rob Bell <[email protected]>
* refactor: centralise shared constants
Signed-off-by: Rob Bell <[email protected]>
* fix: disable http2 in progress server tls opts if disabled by config
Signed-off-by: Rob Bell <[email protected]>
* fix: minor improvements suggested by copilot
Signed-off-by: Rob Bell <[email protected]>
* chore: rename progress server => status server
Signed-off-by: Rob Bell <[email protected]>
* chore: update type doc
Signed-off-by: Rob Bell <[email protected]>
* chore: update name of feature gate
Signed-off-by: Rob Bell <[email protected]>
* fix: after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: fix typo
Signed-off-by: Rob Bell <[email protected]>
* chore: rename handler
Signed-off-by: Rob Bell <[email protected]>
* fix: run status server on all replicas
Signed-off-by: Rob Bell <[email protected]>
* fix: deep copy the rest config
Signed-off-by: Rob Bell <[email protected]>
* refactor: use ssa
Signed-off-by: Rob Bell <[email protected]>
* refactor: separate token audience per train job
Signed-off-by: Rob Bell <[email protected]>
* chore: rename status server
Signed-off-by: Rob Bell <[email protected]>
* fix: revert adding status pkg to manifest gen paths
Signed-off-by: Rob Bell <[email protected]>
* chore: update log messages
Signed-off-by: Rob Bell <[email protected]>
* chore: update api docs
Signed-off-by: Rob Bell <[email protected]>
* chore: update format
Signed-off-by: Rob Bell <[email protected]>
* fix: revert inadvertent change
Signed-off-by: Rob Bell <[email protected]>
* chore: update api name and type docs
Signed-off-by: Rob Bell <[email protected]>
* refactor: initialise oidc provider in manager startup
Signed-off-by: Rob Bell <[email protected]>
* fix: allow empty update request
Signed-off-by: Rob Bell <[email protected]>
* fix: wait for server shut down
Signed-off-by: Rob Bell <[email protected]>
* fix: handle missing train job
Signed-off-by: Rob Bell <[email protected]>
* refactor: update env var names
Signed-off-by: Rob Bell <[email protected]>
* chore: rename packages, plugins, feature gate
Signed-off-by: Rob Bell <[email protected]>
* refactor: update env var names
Signed-off-by: Rob Bell <[email protected]>
* chore: regenerate after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: fixes after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: minor doc updates
Signed-off-by: Rob Bell <[email protected]>
* chore: rename status server config
Signed-off-by: Rob Bell <[email protected]>
* chore: replace placeholder config with nil
Signed-off-by: Rob Bell <[email protected]>
* chore: revert separate runtime utils package
Signed-off-by: Rob Bell <[email protected]>
* chore: remove unused config from tests
Signed-off-by: Rob Bell <[email protected]>
---------
Signed-off-by: Rob Bell <[email protected]>
* Add XGBoost runtime plugin scaffold and register in framework
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
* regenerated assets
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
* Add XGBoost plugin unit tests and update framework test registry
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
* added :Runtime image , unit tests for :(Resources are not set, Resources are set in TrainJob) ,using the ContainerTrainerPort instead of default port
Signed-off-by: krishna-kg732 <[email protected]>
* minor changes
Signed-off-by: krishna-kg732 <[email protected]>
* added plainML fallback(JAX+XGBOOST)
Signed-off-by: krishna-kg732 <[email protected]>
* test: add XGBoost runtime resource resolution unit tests
- Add test case for resources set in Runtime only
- Add test case for resources set in both Runtime and TrainJob
- Add xgboost_distributed.yaml ClusterTrainingRuntime manifest
- Update kustomization.yaml with xgboost-runtime image
Addresses feedback from PR #3200 for complete resource resolution test coverage.
Signed-off-by: krishna-kg732 <[email protected]>
* added autogenerat3ed files
Signed-off-by: krishna-kg732 <[email protected]>
* CI fixes
Signed-off-by: krishna-kg732 <[email protected]>
* chore: fix ci lint errors
Signed-off-by: krishna-kg732 <[email protected]>
* Add XGBoost integration and E2E tests, fix Validate guard clause and nil safety
Signed-off-by: krishna-kg732 <[email protected]>
* trigger ci
Signed-off-by: krishna-kg732 <[email protected]>
* fix: Add XGBoost support to test wrapper, CEL validation, and CI publish workflow
- Add XGBoostPolicy() method to MLPolicySourceWrapper for integration tests
- Add has(self.xgboost) to MLPolicy CEL mutual-exclusivity validation rule
- Regenerate CRD manifests with updated CEL rule
- Add xgboost-runtime to build-and-push-images CI workflow
- Add XGBoost image build step to E2E setup script (commented out due to Kind bug)
Signed-off-by: krishna-kg732 <[email protected]>
* docs: Add XGBoost distributed training example notebook
Add example notebook demonstrating distributed XGBoost training using
the Kubeflow Trainer SDK and the xgboost-distributed ClusterTrainingRuntime.
The notebook uses the modern XGBoost Collective API with RabitTracker
for distributed coordination and synthetic classification data.
Signed-off-by: krishna-kg732 <[email protected]>
* test: Skip XGBoost E2E test until runtime image is published
Signed-off-by: krishna-kg732 <[email protected]>
* chore : address review comments
Signed-off-by: krishna-kg732 <[email protected]>
* chore:resolve merge conflicts
Signed-off-by: krishna-kg732 <[email protected]>
---------
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
* feat: add VERSION file and use it instead of hardcoding
Signed-off-by: milinddethe15 <[email protected]>
* fix: update VERSION file format and adjust API version extraction
Signed-off-by: milinddethe15 <[email protected]>
---------
Signed-off-by: milinddethe15 <[email protected]>
