* test(torch): add unit tests for getModelFromRuntimeRef
Cover the runtime-ref name parsing used to resolve the TorchTune
pretrained model: normalization of valid names and the empty-string
fallback for malformed refs (wrong number of parts, empty input).
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): add unit tests for isLoraConfigEnabled
Cover LoRA detection from trainer args: the model.lora_attn_modules
marker present, absent, and the empty-args case.
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): add unit tests for isUseQLoraFinetune
Cover QLoRA detection from trainer args, including the DoRA
short-circuit that disables QLoRA even when quantize_base is set.
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): add unit tests for getRecipeAndConfig
Cover the recipe/config selection matrix: single-device, single-node
multi-GPU, and multi-node, each across full / LoRA / QLoRA fine-tuning.
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): add unit tests for extractOverridesFromRuntime
Cover extraction of immutable runtime config overrides from the JobSet
template: trainer-ancestor node container returns only immutable configs,
while non-trainer ancestors, non-node containers, and an absent template
yield no overrides.
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): use year-less copyright header
Align the new test file with the repo boilerplate (hack/boilerplate/boilerplate.go.txt) and CNCF/LF guidance, which drop the year from the copyright notice.
Signed-off-by: yassinnouh21 <[email protected]>
* test(torch): cover integer numProcPerNode in getRecipeAndConfig
Exercise integer numProcPerNode counts alongside the existing "auto" cases: numProcPerNode=1 forces the single-device recipe even with multiple GPUs, while a count >1 takes the distributed path.
Signed-off-by: yassinnouh21 <[email protected]>
* fix(torch): respect boolean value when detecting QLoRA args
isUseQLoraFinetune matched quantize_base/use_dora by key only, so model.quantize_base=False enabled QLoRA and model.use_dora=False short-circuited it off. Parse the key=value pair and only treat a truthy value as enabled. Add =False regression cases.
Signed-off-by: yassinnouh21 <[email protected]>
---------
Signed-off-by: yassinnouh21 <[email protected]>
* feat: inject PET envs into init containers via envInjection config
- Add EnvInjection and EnvInjectionTarget types to TorchMLPolicySource
- Add FindContainerByPodSetName helper in runtime package
- Update torch plugin to inject PET envs into additional containers
- Update jobset plugin to sync init containers
- Add 5 comprehensive unit tests for envInjection scenarios
- Add Python API models for envInjection types
Signed-off-by: Peter Pan <[email protected]>
* fix: add missing +optional markers for kubeapilinter validation
Signed-off-by: Peter Pan <[email protected]>
* fix: change JobName to optional field for kubeapilinter
Change JobName from required to optional with omitempty tag to satisfy
kubeapilinter validation. Add default marker to CRD schema for listMapKey
requirement.
Signed-off-by: Peter Pan <[email protected]>
* fix test
Signed-off-by: Peter Pan <[email protected]>
* fix: address envInjection review feedback
Reject missing envInjection targets instead of silently skipping them, align the API schema with required fields, and harden init container propagation so misconfigured runtimes fail predictably.
Signed-off-by: Peter Pan <[email protected]>
* fix: sync envInjection generated files
Add the generated boilerplate and Python model updates expected by make generate so CI does not report stale generated assets.
Signed-off-by: Peter Pan <[email protected]>
* fix: satisfy kube API lint for envInjection fields
Keep the envInjection fields marked as required while preserving omitempty tags required by kubeapilinter.
Signed-off-by: Peter Pan <[email protected]>
* fix: update envInjection OpenAPI defaults
Sync generated OpenAPI and Python model output after restoring omitempty on required envInjection fields.
Signed-off-by: Peter Pan <[email protected]>
* fix: address latest review comments
Signed-off-by: Peter Pan <[email protected]>
* chore: remove test plan file per review feedback
The KEP README already has a Test Plan section; a separate file with
local environment details does not belong in the upstream PR.
Signed-off-by: Peter Pan <[email protected]>
---------
Signed-off-by: Peter Pan <[email protected]>
* fix(torchtune): add validation for LoRA multi-node and immutable args
Add two missing validations to validateTorchTune:
1. Reject LoRA fine-tuning when numNodes > 1. getRecipeAndConfig
silently falls through to full_finetune_distributed in this case,
discarding LoRA args without any user-visible error.
2. Reject immutable runtime configs (output_dir, tokenizer.path,
checkpointer.checkpoint_dir, tokenizer.merges_file) in
spec.trainer.args. These are injected by the runtime via
extractOverridesFromRuntime and must not be overridden by the user.
Resolves the TODO added by @Electronic-Waste in torchtune.go.
Signed-off-by: Krish Garg <[email protected]>
Signed-off-by: krishdef7 <[email protected]>
* fix(runtimes): add validation for LoRA/QLoRA multi-node and immutable trainer args
Signed-off-by: krishdef7 <[email protected]>
---------
Signed-off-by: Krish Garg <[email protected]>
Signed-off-by: krishdef7 <[email protected]>
* feat(api): add progress types to api
Signed-off-by: Rob Bell <[email protected]>
* feat: progress server stub
Signed-off-by: Rob Bell <[email protected]>
* feat: setup tls for server
Signed-off-by: Rob Bell <[email protected]>
* refactor: pull out middleware
Signed-off-by: Rob Bell <[email protected]>
* refactor: remove logger message noise from tests
Signed-off-by: Rob Bell <[email protected]>
* fix: send server error logs to main logger
Signed-off-by: Rob Bell <[email protected]>
* feat: update the trainer status
Signed-off-by: Rob Bell <[email protected]>
* feat: use separate client to avoid impacting main reconciler client-side rate limits
Signed-off-by: Rob Bell <[email protected]>
* feat: configure progress server in helm chart and configmap
Signed-off-by: Rob Bell <[email protected]>
* fix: prevent race condition
Signed-off-by: Rob Bell <[email protected]>
* feat: inject progress runtime config
Signed-off-by: Rob Bell <[email protected]>
* fix: progress client config
Signed-off-by: Rob Bell <[email protected]>
* fix: update generated config
Signed-off-by: Rob Bell <[email protected]>
* feat: validate auth tokens in endpoint
Signed-off-by: Rob Bell <[email protected]>
* test: validate auth tokens in endpoint
Signed-off-by: Rob Bell <[email protected]>
* refactor: tidy tests to use same format as other tests
Signed-off-by: Rob Bell <[email protected]>
* fix: look up the cluster issuer from the service account token
Different clusters can use different issuers so look up the issuer from the trusted in-cluster service account token.
Signed-off-by: Rob Bell <[email protected]>
* refactor: rename authn middleware
Signed-off-by: Rob Bell <[email protected]>
* feat: add authn to status update endpoint
Signed-off-by: Rob Bell <[email protected]>
* refactor: tidy up reading in-cluster service account token
Signed-off-by: Rob Bell <[email protected]>
* fix: rbac permissions
Signed-off-by: Rob Bell <[email protected]>
* fix: better error response
Signed-off-by: Rob Bell <[email protected]>
* refactor: move kubebuilder annotation higher up
Signed-off-by: Rob Bell <[email protected]>
* fix: make lastUpdateTime required
Signed-off-by: Rob Bell <[email protected]>
* chore: fix config tests after rebase
Signed-off-by: Rob Bell <[email protected]>
* test: update config test with new progress server config
Signed-off-by: Rob Bell <[email protected]>
* test: add progress tracking e2e test
Signed-off-by: Rob Bell <[email protected]>
* fix: allow feature gates to be configurable from command line argument
So they can be overridden in e2e tests
Signed-off-by: Rob Bell <[email protected]>
* refactor: reuse shared progress StatusUrl
Signed-off-by: Rob Bell <[email protected]>
* test: tidy up tests
Signed-off-by: Rob Bell <[email protected]>
* refactor: allow plugins to use config
Signed-off-by: Rob Bell <[email protected]>
* refactor: centralise shared constants
Signed-off-by: Rob Bell <[email protected]>
* fix: disable http2 in progress server tls opts if disabled by config
Signed-off-by: Rob Bell <[email protected]>
* fix: minor improvements suggested by copilot
Signed-off-by: Rob Bell <[email protected]>
* chore: rename progress server => status server
Signed-off-by: Rob Bell <[email protected]>
* chore: update type doc
Signed-off-by: Rob Bell <[email protected]>
* chore: update name of feature gate
Signed-off-by: Rob Bell <[email protected]>
* fix: after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: fix typo
Signed-off-by: Rob Bell <[email protected]>
* chore: rename handler
Signed-off-by: Rob Bell <[email protected]>
* fix: run status server on all replicas
Signed-off-by: Rob Bell <[email protected]>
* fix: deep copy the rest config
Signed-off-by: Rob Bell <[email protected]>
* refactor: use ssa
Signed-off-by: Rob Bell <[email protected]>
* refactor: separate token audience per train job
Signed-off-by: Rob Bell <[email protected]>
* chore: rename status server
Signed-off-by: Rob Bell <[email protected]>
* fix: revert adding status pkg to manifest gen paths
Signed-off-by: Rob Bell <[email protected]>
* chore: update log messages
Signed-off-by: Rob Bell <[email protected]>
* chore: update api docs
Signed-off-by: Rob Bell <[email protected]>
* chore: update format
Signed-off-by: Rob Bell <[email protected]>
* fix: revert inadvertent change
Signed-off-by: Rob Bell <[email protected]>
* chore: update api name and type docs
Signed-off-by: Rob Bell <[email protected]>
* refactor: initialise oidc provider in manager startup
Signed-off-by: Rob Bell <[email protected]>
* fix: allow empty update request
Signed-off-by: Rob Bell <[email protected]>
* fix: wait for server shut down
Signed-off-by: Rob Bell <[email protected]>
* fix: handle missing train job
Signed-off-by: Rob Bell <[email protected]>
* refactor: update env var names
Signed-off-by: Rob Bell <[email protected]>
* chore: rename packages, plugins, feature gate
Signed-off-by: Rob Bell <[email protected]>
* refactor: update env var names
Signed-off-by: Rob Bell <[email protected]>
* chore: regenerate after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: fixes after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: minor doc updates
Signed-off-by: Rob Bell <[email protected]>
* chore: rename status server config
Signed-off-by: Rob Bell <[email protected]>
* chore: replace placeholder config with nil
Signed-off-by: Rob Bell <[email protected]>
* chore: revert separate runtime utils package
Signed-off-by: Rob Bell <[email protected]>
* chore: remove unused config from tests
Signed-off-by: Rob Bell <[email protected]>
---------
Signed-off-by: Rob Bell <[email protected]>
