The `cache` dataset initializer compared the exception instance to the
`ConflictError` class using `is`, which is always False:
except ApiException as e:
if e is ConflictError: # always False
`core_v1.create_namespaced_service()` raises a plain `ApiException` with
`status=409`, not a `ConflictError`, so this branch was unreachable. When
the Service already existed (e.g., re-reconcile after a crash between
Service creation and LWS readiness), the code re-raised, fell into the
outer cleanup, and deleted the ServiceAccount — leaving the cache
cluster in a broken state.
Match the pattern used a few lines above for the ServiceAccount and
check `e.status == 409` instead.
Add a unit test that mocks `create_namespaced_service` to raise 409 and
asserts `download_dataset` completes without calling
`delete_namespaced_service_account`.
Signed-off-by: 1fanwang <[email protected]>
* docs: Add CODE_OF_CONDUCT.md for CNCF graduation recommendation
Part of #976
Adds a CODE_OF_CONDUCT.md file to improve discoverability of the Kubeflow Code of Conduct and surface the Code of Conduct tab in the repository.
Signed-off-by: Lucien <[email protected]>
* Update title and add reporting instructions
Signed-off-by: Lucien <[email protected]>
---------
Signed-off-by: Lucien <[email protected]>
Adds terminationGracePeriodSeconds field to PodSpecPatch so users can
configure pod termination grace period per TrainJob via RuntimePatches.
This is needed for distributed training with PyTorch Elastic (torchrun)
where large models (70B+ parameters) require more than the default 30s
to complete JIT checkpointing before SIGKILL on node drain or TrainJob
pause.
No changes to merge logic in trainingruntime.go are required since the
existing StrategicMergePatch applied at batchv1.JobTemplateSpec level
already handles this field automatically.
Closes#3285
Signed-off-by: krishdef7 <[email protected]>
* fix(torchtune): add validation for LoRA multi-node and immutable args
Add two missing validations to validateTorchTune:
1. Reject LoRA fine-tuning when numNodes > 1. getRecipeAndConfig
silently falls through to full_finetune_distributed in this case,
discarding LoRA args without any user-visible error.
2. Reject immutable runtime configs (output_dir, tokenizer.path,
checkpointer.checkpoint_dir, tokenizer.merges_file) in
spec.trainer.args. These are injected by the runtime via
extractOverridesFromRuntime and must not be overridden by the user.
Resolves the TODO added by @Electronic-Waste in torchtune.go.
Signed-off-by: Krish Garg <[email protected]>
Signed-off-by: krishdef7 <[email protected]>
* fix(runtimes): add validation for LoRA/QLoRA multi-node and immutable trainer args
Signed-off-by: krishdef7 <[email protected]>
---------
Signed-off-by: Krish Garg <[email protected]>
Signed-off-by: krishdef7 <[email protected]>
* feat(ci): add Python dependency scanning to OSV-Scanner workflow
Add lockfiles and scanning for Python dependencies in the initializers
and python_api. This extends the nightly OSV-Scanner workflow to cover
Python in addition to Go.
- Generate requirements-lock.txt files via pip-compile for:
- cmd/initializers/dataset/
- cmd/initializers/model/
- api/python_api/
- Add osv-scan-python job with SARIF scanning and auto-fix
- Add validate-lockfile.yaml to ensure lockfiles stay in sync on PRs
- Handle both direct and transitive dependency upgrades
Closes#3528
Signed-off-by: Fiona-Waters <[email protected]>
* fix(ci): address review feedback on Python OSV-Scanner workflow
- Fix validate-lockfile false positives by stripping header comments
before diffing (pip-compile embeds output filename in header)
- Regenerate lockfiles in clean Python 3.12 venv with --strip-extras
to include all transitive deps and silence pip-tools deprecation warning
- Fix PR body table formatting: use Markdown table rows instead of
bullet items in both Go and Python auto-fix jobs
- Escape dots in package names for grep/sed regex safety
Signed-off-by: Fiona-Waters <[email protected]>
* fix(ci): address review feedback on OSV-Scanner Python workflow
- Replace shell grep/sed with Python helper script using
packaging.requirements.Requirement for exact name matching,
fixing prefix collision (requests vs requests-oauthlib),
constraint dropping, and extras stripping
- Skip entry when package not found in lockfile after pip-compile
instead of recording a false version
- Bump actions/setup-python from v5 to v6 for consistency
Signed-off-by: Fiona-Waters <[email protected]>
* fix(ci): pin pip-tools version in CI workflows
Pin pip-tools to 7.4.1 in osv-scanner and validate-lockfile workflows
to mitigate supply-chain risk in workflows with elevated permissions.
Signed-off-by: Fiona-Waters <[email protected]>
* fix(deps): update lockfiles for certifi 2026.5.20
Signed-off-by: Fiona-Waters <[email protected]>
---------
Signed-off-by: Fiona-Waters <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
* feat: add nightly OSV-Scanner vulnerability scan workflow
Adds a nightly GitHub Actions workflow that scans go.mod for known
vulnerabilities using OSV-Scanner, uploads SARIF results to the
Security tab, and auto-creates a PR with dependency fixes.
Signed-off-by: Fiona-Waters <[email protected]>
* fix: address review feedback on osv-scanner workflow
- Use sort -V for semver-aware version deduplication instead of
jq tonumber which mishandles pre-release suffixes
- Snapshot go.mod/go.sum per iteration so a failed fix only reverts
itself, not previously successful updates
Signed-off-by: Fiona-Waters <[email protected]>
* fix(ci): Address review comments for osv-scanner workflow
- Add set -euo pipefail to catch pipeline and unset variable errors
- Remove 2>/dev/null from jq call so schema drift is a hard failure
- Normalize OSV fix versions with sub("^v"; "") to prevent double v prefix
- Use ::warning:: annotations so skipped fixes surface in Actions UI
- Fix go list -m fallback to record fix version instead of silently skipping
- Remove redundant jq empty pre-check
Signed-off-by: Fiona-Waters <[email protected]>
---------
Signed-off-by: Fiona-Waters <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
* feat(docs): add KEP-2599 for mutable runtimes
Proposes allowing TrainingRuntimes and ClusterTrainingRuntimes to be
mutable by introducing TrainingRuntimeSnapshot resources. TrainJobs
snapshot their referenced runtime configuration on first reconciliation,
decoupling job execution from runtime changes.
Key changes:
- New TrainingRuntimeSnapshot CRD to store point-in-time runtime config
- Remove finalizers from runtimes (no longer needed)
- Remove TrainJobWatcher interface and boilerplate
- Automatic migration for existing TrainJobs on upgrade
Addresses issue #2599
Signed-off-by: Rob Bell <[email protected]>
* feat(docs): improve KEP-2599 clarity and completeness
Improvements to the mutable runtimes KEP:
- Clarify operational problems with finalizers (orphaned finalizers,
namespace deletion issues)
- Better explain why runtime updates are risky (fetched on every
reconciliation, no design-level guarantees)
- Add concrete examples of runtime proliferation (pytorch-2.0, pytorch-2.1)
- Add RBAC section showing required permissions
- Improve summary to clearly state API change (new CRD)
- Clarify test cases for migration scenarios
Signed-off-by: Rob Bell <[email protected]>
* docs: address Copilot review comments on KEP-2599
- Fix relative link to include README.md in Trainer v2 design reference
- Remove extra space in Goals heading
- Change 'Training Job' to 'TrainJobs' for consistency
- Fix 'lookup' to 'look up' (correct verb form)
- Add 'create' verb to RBAC permissions for TrainingRuntimeSnapshot
Signed-off-by: Rob Bell <[email protected]>
* docs: clarify KEP-2599 divergence from original design
Signed-off-by: Rob Bell <[email protected]>
* docs: add source runtime annotation to KEP-2599
Signed-off-by: Rob Bell <[email protected]>
* fix: update the KEP based on feedback
Signed-off-by: Rob Bell <[email protected]>
* fix: apply suggestion
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Rob Bell <[email protected]>
* fix: apply suggestion
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Rob Bell <[email protected]>
* fix: apply suggestion
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Rob Bell <[email protected]>
* fix: remove annotation for snapshot
Signed-off-by: Rob Bell <[email protected]>
---------
Signed-off-by: Rob Bell <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
* docs(proposals): add draft KEP for inject PET envs into trainer init containers
Signed-off-by: Peter Pan <[email protected]>
* fix comments
Signed-off-by: Peter Pan <[email protected]>
* address comments
Signed-off-by: Peter Pan <[email protected]>
* Add OPT-In annotation
Polish more
Signed-off-by: Peter Pan <[email protected]>
* add more detailed user story
Signed-off-by: Peter Pan <[email protected]>
* change annotation name to more generic
Signed-off-by: Peter Pan <[email protected]>
* docs(proposals): clarify plugin env injection mode in KEP-3416
Signed-off-by: Peter Pan <[email protected]>
* docs(proposals): switch from annotation to TorchMLPolicy API field for env injection
Signed-off-by: Peter Pan <[email protected]>
* docs(proposals): clarify envInjection default state semantics
Signed-off-by: Peter Pan <[email protected]>
* docs(proposals): refine envInjection API to use initContainers name list
Replace containerTypes enum with explicit initContainers []string for
precise targeting. The main container (node) always receives PET_* envs,
eliminating the need for container type distinction and extra validation.
Signed-off-by: Peter Pan <[email protected]>
* docs(proposals): adopt targets/jobName/containerNames API per review consensus
Use explicit targets list with jobName and containerNames for declarative
API. Main container (node) always injected automatically. jobName chosen
over replicatedJobName for future CRD compatibility (LWS, Grove, Slurm).
Signed-off-by: Peter Pan <[email protected]>
* docs(proposals): fix stale text to align with targets API
Proposal description, Goal, and Non-Goals previously referenced
trainer init containers only. With the targets/jobName/containerNames
API, injection targets any container in any replicated job.
Signed-off-by: Peter Pan <[email protected]>
* docs(proposals): finalize API per offline sync with Andre/tenzen-y
Keep jobName + containerNames (plural) design. Add container existence
validation. Rename TorchEnvInjection to EnvInjection for policy reuse.
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Peter Pan <[email protected]>
* Update docs/proposals/3416-pet-env-init-containers/README.md
Co-authored-by: Rob Bell <[email protected]>
Signed-off-by: Peter Pan <[email protected]>
* chore: fix trailing whitespace in KEP README
Signed-off-by: Peter Pan <[email protected]>
---------
Signed-off-by: Peter Pan <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Rob Bell <[email protected]>
sshd in the MPI runtime listens on port 2222 (non-privileged),
so CAP_NET_BIND_SERVICE is not needed. Also removes libcap2-bin
which was only installed to provide the setcap binary.
Fixes#3254
Signed-off-by: kapil27 <[email protected]>
Direct contributors to review the Kubeflow AI Policy before using
AI assistant tooling for contributions.
Signed-off-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
