The `cache` dataset initializer compared the exception instance to the
`ConflictError` class using `is`, which is always False:
except ApiException as e:
if e is ConflictError: # always False
`core_v1.create_namespaced_service()` raises a plain `ApiException` with
`status=409`, not a `ConflictError`, so this branch was unreachable. When
the Service already existed (e.g., re-reconcile after a crash between
Service creation and LWS readiness), the code re-raised, fell into the
outer cleanup, and deleted the ServiceAccount — leaving the cache
cluster in a broken state.
Match the pattern used a few lines above for the ServiceAccount and
check `e.status == 409` instead.
Add a unit test that mocks `create_namespaced_service` to raise 409 and
asserts `download_dataset` completes without calling
`delete_namespaced_service_account`.
Signed-off-by: 1fanwang <[email protected]>
Adds terminationGracePeriodSeconds field to PodSpecPatch so users can
configure pod termination grace period per TrainJob via RuntimePatches.
This is needed for distributed training with PyTorch Elastic (torchrun)
where large models (70B+ parameters) require more than the default 30s
to complete JIT checkpointing before SIGKILL on node drain or TrainJob
pause.
No changes to merge logic in trainingruntime.go are required since the
existing StrategicMergePatch applied at batchv1.JobTemplateSpec level
already handles this field automatically.
Closes#3285
Signed-off-by: krishdef7 <[email protected]>
* fix(torchtune): add validation for LoRA multi-node and immutable args
Add two missing validations to validateTorchTune:
1. Reject LoRA fine-tuning when numNodes > 1. getRecipeAndConfig
silently falls through to full_finetune_distributed in this case,
discarding LoRA args without any user-visible error.
2. Reject immutable runtime configs (output_dir, tokenizer.path,
checkpointer.checkpoint_dir, tokenizer.merges_file) in
spec.trainer.args. These are injected by the runtime via
extractOverridesFromRuntime and must not be overridden by the user.
Resolves the TODO added by @Electronic-Waste in torchtune.go.
Signed-off-by: Krish Garg <[email protected]>
Signed-off-by: krishdef7 <[email protected]>
* fix(runtimes): add validation for LoRA/QLoRA multi-node and immutable trainer args
Signed-off-by: krishdef7 <[email protected]>
---------
Signed-off-by: Krish Garg <[email protected]>
Signed-off-by: krishdef7 <[email protected]>
* test(util): add unit tests for trainingruntime and trainjob utilities
Fixes#3152
pkg/util/trainingruntime (0 percent to 100 percent coverage):
- New trainingruntime_test.go with 8 cases for IsSupportDeprecated
covering nil, empty, absent key, wrong value, empty value,
correct deprecated, multiple labels, and partial match edge case.
pkg/util/trainjob (extended):
- TestRuntimeRefIsClusterTrainingRuntime: 4 new edge cases added
covering wrong APIGroup, nil APIGroup, nil Kind, both nil.
- TestIsTrainJobFinished: 6 new cases covering all condition states.
- Added metav1 import required for Condition status constants.
Signed-off-by: Yugendhar S <[email protected]>
* chore(test): address review feedback from andreyvelich
- Remove year from Copyright headers in both test files
- Add trailing newline at end of both files
- Fix error message format in TestRuntimeRefIsTrainingRuntime
to use 'FunctionName(%v) = %v, want %v' pattern for consistency
- Add nil edge cases to TestRuntimeRefIsTrainingRuntime for symmetry
with TestRuntimeRefIsClusterTrainingRuntime (nil APIGroup, nil Kind,
both nil, wrong APIGroup)
Signed-off-by: Yugendhar S <[email protected]>
* chore(test): fix line endings and formatting for CI
Signed-off-by: Yugendhar S <[email protected]>
---------
Signed-off-by: Yugendhar S <[email protected]>
* test: add unit tests for jobset builder
Add comprehensive unit tests for the Builder type methods:
- TestBuilderInitializer: dataset/model initializer env and secret injection
- TestBuilderTrainer: trainer image/command/args/resources application
- TestBuilderPodLabels, TestBuilderPodAnnotations, TestBuilderSuspend
Uses map[string]struct test pattern and go-cmp for assertions,
following established codebase conventions.
Built on top of @upsaurav12's initial work in #3176, with all review
feedback addressed.
Ref #3152
Signed-off-by: Sridhar1030 <[email protected]>
* test: add env merge coverage for initializer and trainer builder tests
Address Copilot review feedback on PR #3490:
- Add test case for Dataset.Env merge/upsert in TestBuilderInitializer
- Add test case for Trainer.Env merge/upsert in TestBuilderTrainer
Ref #3152
Signed-off-by: Sridhar1030 <[email protected]>
* test: address maintainer review feedback on builder tests
- Consolidate makeJobSetWithInitializer/makeJobSetWithTrainer into single makeJobSet helper
- Add nil Initializer/Trainer spec edge cases (replicas=1, no panic)
- Add MPI runLauncherAsNode + Trainer.Env injection test
- Add multi-replicated-job cases for PodLabels/PodAnnotations
- Add suspend=nil case (unset vs explicit false)
Ref #3152
Signed-off-by: Sridhar1030 <[email protected]>
---------
Signed-off-by: Sridhar1030 <[email protected]>
Co-authored-by: upsaurav12 <[email protected]>
* fix(webhook): allow atomic RuntimePatches update on unsuspend
Change checkRuntimePatchesImmutability to check the *old* TrainJob's
suspend state instead of the new one. This lets external controllers
(notably Kueue, see kubernetes-sigs/kueue#8296) modify spec.runtimePatches
and set spec.suspend=false in a single API request, eliminating the
two-step update workaround and the race window between the calls.
Mirrors the Kubernetes core Job validation pattern:
https://github.com/kubernetes/kubernetes/blob/86b66f6f333a/pkg/registry/batch/job/strategy.go#L191
The downstream JobSet activity check (no active replicatedJobs) still
runs, so updates while pods are running remain forbidden.
Refs: #3043
Signed-off-by: Sabari Narayana <[email protected]>
* test(jobset): cover atomic suspend->unsuspend RuntimePatches update
Add a unit test that exercises the new atomic-update path: a single
TrainJob update that both clears spec.suspend and modifies
spec.runtimePatches must succeed when the previous state was suspended.
Refs: #3043
Signed-off-by: Sabari Narayana <[email protected]>
* test(integration): cover atomic unsuspend + RuntimePatches update
Extend the trainjob webhook update table with an entry that creates a
suspended TrainJob with RuntimePatches and then, in a single Update,
both unsuspends it and modifies the runtimePatches NodeSelector. This
exercises the relaxed validation against a real apiserver and protects
the suspended -> unsuspended transition path used by Kueue.
Refs: #3043
Signed-off-by: Sabari Narayana <[email protected]>
* fix(api): block RuntimePatches change only when both states are unsuspended
Per @mimowo's review, the previous condition (!oldSuspended) over-blocked
the (oldSuspended=false, newSuspended=true) transition — i.e. suspending
a running TrainJob while modifying spec.runtimePatches in the same call,
which is safe because the JobSet activity check still guards against
concurrent pod runs.
Change the predicate to !oldSuspended && !newSuspended so the webhook
only forbids RuntimePatches mutations when the TrainJob is staying fully
unsuspended. Reword the error message accordingly so callers see
explicitly that suspension at either side of the update is acceptable.
Refs: #3043, kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>
* test: cover unsuspended->suspended atomic RuntimePatches transition
Per @mimowo's request, complete the transition table coverage:
- (false, false) -> blocked (existing test)
- (true, true) -> allowed (existing test)
- (true, false) -> allowed (added previously)
- (false, true) -> allowed (added here)
Mirrors the existing atomic-unsuspend cases — one unit case in
jobset_test.go and one webhook integration entry in trainjob_test.go.
Refs: #3043, kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>
* docs(api): shorten RuntimePatches validation comment
Trim the comment block above checkRuntimePatchesImmutability to a 4-line
summary, per @kaisoz's review feedback (the comment was longer than the
change itself). Drops the kueue#8296 backreference and the k/k Job
validation link — both are already in the PR description and commit
history.
Refs: kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>
* fix(api): use stdlib for suspend pointer checks in jobset webhook
Per @tenzen-y's review nit, swap the ptr.Equal(x, ptr.To(true)) idiom
for an inline nil-check + dereference. Equivalent semantics, no extra
generic-comparison hop, and stays in the stdlib for these two lines.
Refs: kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>
* test(jobset): cover (false,true) transition with active jobSet
Per @tenzen-y's request, add a unit case where the old TrainJob is
unsuspended, the new one is being suspended with a runtimePatches
change, and the underlying JobSet still has an active replicatedJob.
The relaxed suspend predicate alone would let this through; the
JobSet activity check should still reject it.
Refs: kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>
* test(integration): drop atomic transition cases duplicated by UTs
Per @tenzen-y's review, the two atomic-transition ginkgo entries
(unsuspend + RuntimePatches, suspend + RuntimePatches) duplicate the
unit cases in jobset_test.go that already exercise the same validator
path. The pre-existing "Should succeed to update runtimePatches when
suspend is true" entry stays as the integration smoke test for the
webhook chain.
Refs: kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>
* fix(api): use new(true) for suspend pointer comparisons
@tenzen-y is right that the value-form `new()` builtin is valid Go 1.26,
which the repo already targets (`go 1.26.0` in go.mod). Apply his
original suggestion verbatim — keeps `ptr.Equal` for the two-pointer
comparison and replaces only `ptr.To(true)`, which is the smaller and
more idiomatic delta vs the inline nil-check + deref pattern I tried
in the previous commit.
Refs: kubeflow/trainer#3469 (review feedback)
Signed-off-by: Sabari Narayana <[email protected]>
---------
Signed-off-by: Sabari Narayana <[email protected]>
* feat: customize R generation to include GPUs
Flux detection of GPUs depends on hwloc plugins, and a newer
version. To get around any edge cases where the dependencies are missing,
we can easily generate the R from the expected resource spec (cores
and gpus). We can also add a shared memory mount to ensure the job
MPI gets all available shared memory of the host. The current
default is 64M (automatic from container runtime) and it can have
implications for MPI performance.
Signed-off-by: vsoch <[email protected]>
* review: use generateRange function for gpus
Also update volume to be added to container. This still
is pending testing on AWS!
Signed-off-by: vsoch <[email protected]>
* review and add gpu trainer example
Signed-off-by: vsoch <[email protected]>
---------
Signed-off-by: vsoch <[email protected]>
Co-authored-by: vsoch <[email protected]>
* feat(api): add progress types to api
Signed-off-by: Rob Bell <[email protected]>
* feat: progress server stub
Signed-off-by: Rob Bell <[email protected]>
* feat: setup tls for server
Signed-off-by: Rob Bell <[email protected]>
* refactor: pull out middleware
Signed-off-by: Rob Bell <[email protected]>
* refactor: remove logger message noise from tests
Signed-off-by: Rob Bell <[email protected]>
* fix: send server error logs to main logger
Signed-off-by: Rob Bell <[email protected]>
* feat: update the trainer status
Signed-off-by: Rob Bell <[email protected]>
* feat: use separate client to avoid impacting main reconciler client-side rate limits
Signed-off-by: Rob Bell <[email protected]>
* feat: configure progress server in helm chart and configmap
Signed-off-by: Rob Bell <[email protected]>
* fix: prevent race condition
Signed-off-by: Rob Bell <[email protected]>
* feat: inject progress runtime config
Signed-off-by: Rob Bell <[email protected]>
* fix: progress client config
Signed-off-by: Rob Bell <[email protected]>
* fix: update generated config
Signed-off-by: Rob Bell <[email protected]>
* feat: validate auth tokens in endpoint
Signed-off-by: Rob Bell <[email protected]>
* test: validate auth tokens in endpoint
Signed-off-by: Rob Bell <[email protected]>
* refactor: tidy tests to use same format as other tests
Signed-off-by: Rob Bell <[email protected]>
* fix: look up the cluster issuer from the service account token
Different clusters can use different issuers so look up the issuer from the trusted in-cluster service account token.
Signed-off-by: Rob Bell <[email protected]>
* refactor: rename authn middleware
Signed-off-by: Rob Bell <[email protected]>
* feat: add authn to status update endpoint
Signed-off-by: Rob Bell <[email protected]>
* refactor: tidy up reading in-cluster service account token
Signed-off-by: Rob Bell <[email protected]>
* fix: rbac permissions
Signed-off-by: Rob Bell <[email protected]>
* fix: better error response
Signed-off-by: Rob Bell <[email protected]>
* refactor: move kubebuilder annotation higher up
Signed-off-by: Rob Bell <[email protected]>
* fix: make lastUpdateTime required
Signed-off-by: Rob Bell <[email protected]>
* chore: fix config tests after rebase
Signed-off-by: Rob Bell <[email protected]>
* test: update config test with new progress server config
Signed-off-by: Rob Bell <[email protected]>
* test: add progress tracking e2e test
Signed-off-by: Rob Bell <[email protected]>
* fix: allow feature gates to be configurable from command line argument
So they can be overridden in e2e tests
Signed-off-by: Rob Bell <[email protected]>
* refactor: reuse shared progress StatusUrl
Signed-off-by: Rob Bell <[email protected]>
* test: tidy up tests
Signed-off-by: Rob Bell <[email protected]>
* refactor: allow plugins to use config
Signed-off-by: Rob Bell <[email protected]>
* refactor: centralise shared constants
Signed-off-by: Rob Bell <[email protected]>
* fix: disable http2 in progress server tls opts if disabled by config
Signed-off-by: Rob Bell <[email protected]>
* fix: minor improvements suggested by copilot
Signed-off-by: Rob Bell <[email protected]>
* chore: rename progress server => status server
Signed-off-by: Rob Bell <[email protected]>
* chore: update type doc
Signed-off-by: Rob Bell <[email protected]>
* chore: update name of feature gate
Signed-off-by: Rob Bell <[email protected]>
* fix: after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: fix typo
Signed-off-by: Rob Bell <[email protected]>
* chore: rename handler
Signed-off-by: Rob Bell <[email protected]>
* fix: run status server on all replicas
Signed-off-by: Rob Bell <[email protected]>
* fix: deep copy the rest config
Signed-off-by: Rob Bell <[email protected]>
* refactor: use ssa
Signed-off-by: Rob Bell <[email protected]>
* refactor: separate token audience per train job
Signed-off-by: Rob Bell <[email protected]>
* chore: rename status server
Signed-off-by: Rob Bell <[email protected]>
* fix: revert adding status pkg to manifest gen paths
Signed-off-by: Rob Bell <[email protected]>
* chore: update log messages
Signed-off-by: Rob Bell <[email protected]>
* chore: update api docs
Signed-off-by: Rob Bell <[email protected]>
* chore: update format
Signed-off-by: Rob Bell <[email protected]>
* fix: revert inadvertent change
Signed-off-by: Rob Bell <[email protected]>
* chore: update api name and type docs
Signed-off-by: Rob Bell <[email protected]>
* refactor: initialise oidc provider in manager startup
Signed-off-by: Rob Bell <[email protected]>
* fix: allow empty update request
Signed-off-by: Rob Bell <[email protected]>
* fix: wait for server shut down
Signed-off-by: Rob Bell <[email protected]>
* fix: handle missing train job
Signed-off-by: Rob Bell <[email protected]>
* refactor: update env var names
Signed-off-by: Rob Bell <[email protected]>
* chore: rename packages, plugins, feature gate
Signed-off-by: Rob Bell <[email protected]>
* refactor: update env var names
Signed-off-by: Rob Bell <[email protected]>
* chore: regenerate after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: fixes after rebase
Signed-off-by: Rob Bell <[email protected]>
* chore: minor doc updates
Signed-off-by: Rob Bell <[email protected]>
* chore: rename status server config
Signed-off-by: Rob Bell <[email protected]>
* chore: replace placeholder config with nil
Signed-off-by: Rob Bell <[email protected]>
* chore: revert separate runtime utils package
Signed-off-by: Rob Bell <[email protected]>
* chore: remove unused config from tests
Signed-off-by: Rob Bell <[email protected]>
---------
Signed-off-by: Rob Bell <[email protected]>
* feat(api): add ActiveDeadlineSeconds and TTLSecondsAfterFinished to Trainer APIs
- Add ActiveDeadlineSeconds *int64 to TrainJobSpec (immutable, min=1)
with reason constant TrainJobDeadlineExceededReason="DeadlineExceeded"
- Add TTLSecondsAfterFinished *int32 to TrainingRuntimeSpec (min=0)
with cross-field CEL rules blocking conflicting lifecycle fields
in the JobSet/Job template
- Add TrainJobDeadlineExceededMessage to constants
- Regenerate zz_generated.deepcopy.go
Part of KEP-2899
Signed-off-by: XploY04 <[email protected]>
* feat(controller): add TTL-based auto-deletion for finished TrainJobs
- Add early-exit guard in Reconcile before prevTrainJob DeepCopy and
removeFailedCondition, mirroring the Kubernetes Job controller pattern.
Once a TrainJob is terminal (Complete or Failed), only TTL cleanup runs.
- Add fetchTTLFromRuntime helper to retrieve TTLSecondsAfterFinished from
the referenced TrainingRuntime or ClusterTrainingRuntime.
- Add reconcileTTL helper to delete finished TrainJobs after their TTL
expires, or requeue at the exact expiry time.
Part of KEP-2899
Signed-off-by: XploY04 <[email protected]>
* feat(controller): enforce ActiveDeadlineSeconds on TrainJobs
- Add reconcileDeadline helper to terminate running TrainJobs that
exceed their ActiveDeadlineSeconds.
- Handle job suspension: StartTime is determined by the LastTransitionTime
of the Suspended=False condition (if it exists) to ensure the deadline
timer resets correctly upon resume.
- Jobs exceeding the deadline are moved to a Failed state with
reason DeadlineExceeded.
Part of KEP-2899
Signed-off-by: XploY04 <[email protected]>
* feat(webhooks): warn on short ttlSecondsAfterFinished
- Add an admission warning to both TrainingRuntime and
ClusterTrainingRuntime webhooks if the TTLSecondsAfterFinished
is configured to less than 60 seconds, as this may cause
unexpected data/log loss for data scientists due to overly
aggressive cleanup.
Part of KEP-2899
Signed-off-by: XploY04 <[email protected]>
* test: add integration tests for TrainJob TTL and deadline lifecycle
- Add integration tests for DeadlineExceeded behavior in TrainJob controller
- Add integration tests for TTL validation and warnings in webhooks
Part of KEP-2899
Signed-off-by: XploY04 <[email protected]>
* cleanup
Signed-off-by: XploY04 <[email protected]>
* fix: ensure JobSet is suspended on expiration and fix webhook test regress
Signed-off-by: XploY04 <[email protected]>
* fix(controller): return errors from fetchTTLFromRuntime for proper requeue
Signed-off-by: XploY04 <[email protected]>
* test: remove unused wantWarning from webhook tests
Signed-off-by: XploY04 <[email protected]>
* fix
Signed-off-by: XploY04 <[email protected]>
* feat: Remove from TrainingRuntime and ClusterTrainingRuntime specifications and associated validation.
Signed-off-by: XploY04 <[email protected]>
* fix(api): restore `activeDeadlineSeconds` validation rule on TrainingRuntime
Signed-off-by: XploY04 <[email protected]>
* refactor: remove unnecessary blank lines and improve logging in ClusterTrainingRuntime validation
Signed-off-by: XploY04 <[email protected]>
* feat: add activeDeadlineSeconds field to TrainJobSpec and update related tests
Signed-off-by: XploY04 <[email protected]>
* Remove unnecessary newline in test file
Signed-off-by: Yash Agarwal <[email protected]>
* Remove unnecessary newline in trainingruntime_webhook_test.go
Signed-off-by: Yash Agarwal <[email protected]>
* refactor: remove unnecessary blank lines in TrainJob reconciliation and deadline tests
Signed-off-by: XploY04 <[email protected]>
* feat: change ActiveDeadlineSeconds from pointer to value type and update related code
Signed-off-by: XploY04 <[email protected]>
* feat: add condition check for TrainJob deadline exceeded before suspending
Signed-off-by: XploY04 <[email protected]>
* feat: enhance TrainJob deadline handling and suspend logic
Signed-off-by: XploY04 <[email protected]>
* feat: enhance TrainJob deadline handling and suspend logic
Signed-off-by: XploY04 <[email protected]>
* fix
Signed-off-by: XploY04 <[email protected]>
* feat: change ActiveDeadlineSeconds to pointer type and update related logic
Signed-off-by: XploY04 <[email protected]>
* feat: ensure requeueAfter is at least 1 second when deadline is exceeded
Signed-off-by: XploY04 <[email protected]>
* feat: change ActiveDeadlineSeconds from pointer to value type and update related logic
Signed-off-by: XploY04 <[email protected]>
* fix: remove early return for finished TrainJobs to prevent status sync issues
The early return was preventing the controller from syncing JobSet
status updates for finished TrainJobs, causing e2e tests to timeout
waiting for conditions that never got updated.
This fix restores the upstream behavior where finished TrainJobs still
go through status reconciliation (though reconcileObjects is skipped).
Signed-off-by: XploY04 <[email protected]>
* fix
Signed-off-by: XploY04 <[email protected]>
* fix: remove x-kubernetes-validations for activeDeadlineSeconds in TrainingRuntime CRDs
Signed-off-by: XploY04 <[email protected]>
* fix: update validation message for ActiveDeadlineSeconds to be more generic
Signed-off-by: XploY04 <[email protected]>
* replacing suspend with delete
Signed-off-by: XploY04 <[email protected]>
* fix: remove SpecLabel and SpecAnnotation methods from TrainJobWrapper
Signed-off-by: XploY04 <[email protected]>
* fix: remove annotations field from TrainJobSpec and CRD definitions
Signed-off-by: XploY04 <[email protected]>
* ensure JobSet deletion in e2e tests
Signed-off-by: XploY04 <[email protected]>
* fix: add JobSet APIs to the scheme in e2e tests
Signed-off-by: XploY04 <[email protected]>
* fix: simplify error handling in Reconcile method of TrainJobReconciler
Signed-off-by: XploY04 <[email protected]>
---------
Signed-off-by: XploY04 <[email protected]>
Signed-off-by: Yash Agarwal <[email protected]>
* Add XGBoost runtime plugin scaffold and register in framework
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
* regenerated assets
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
* Add XGBoost plugin unit tests and update framework test registry
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
* added :Runtime image , unit tests for :(Resources are not set, Resources are set in TrainJob) ,using the ContainerTrainerPort instead of default port
Signed-off-by: krishna-kg732 <[email protected]>
* minor changes
Signed-off-by: krishna-kg732 <[email protected]>
* added plainML fallback(JAX+XGBOOST)
Signed-off-by: krishna-kg732 <[email protected]>
* test: add XGBoost runtime resource resolution unit tests
- Add test case for resources set in Runtime only
- Add test case for resources set in both Runtime and TrainJob
- Add xgboost_distributed.yaml ClusterTrainingRuntime manifest
- Update kustomization.yaml with xgboost-runtime image
Addresses feedback from PR #3200 for complete resource resolution test coverage.
Signed-off-by: krishna-kg732 <[email protected]>
* added autogenerat3ed files
Signed-off-by: krishna-kg732 <[email protected]>
* CI fixes
Signed-off-by: krishna-kg732 <[email protected]>
* chore: fix ci lint errors
Signed-off-by: krishna-kg732 <[email protected]>
* Add XGBoost integration and E2E tests, fix Validate guard clause and nil safety
Signed-off-by: krishna-kg732 <[email protected]>
* trigger ci
Signed-off-by: krishna-kg732 <[email protected]>
* fix: Add XGBoost support to test wrapper, CEL validation, and CI publish workflow
- Add XGBoostPolicy() method to MLPolicySourceWrapper for integration tests
- Add has(self.xgboost) to MLPolicy CEL mutual-exclusivity validation rule
- Regenerate CRD manifests with updated CEL rule
- Add xgboost-runtime to build-and-push-images CI workflow
- Add XGBoost image build step to E2E setup script (commented out due to Kind bug)
Signed-off-by: krishna-kg732 <[email protected]>
* docs: Add XGBoost distributed training example notebook
Add example notebook demonstrating distributed XGBoost training using
the Kubeflow Trainer SDK and the xgboost-distributed ClusterTrainingRuntime.
The notebook uses the modern XGBoost Collective API with RabitTracker
for distributed coordination and synthetic classification data.
Signed-off-by: krishna-kg732 <[email protected]>
* test: Skip XGBoost E2E test until runtime image is published
Signed-off-by: krishna-kg732 <[email protected]>
* chore : address review comments
Signed-off-by: krishna-kg732 <[email protected]>
* chore:resolve merge conflicts
Signed-off-by: krishna-kg732 <[email protected]>
---------
Signed-off-by: Krishna-kg732 <[email protected]>
Signed-off-by: krishna-kg732 <[email protected]>
