ns kubeflow runs ghcr.io/kubeflow/* images — upstream, and on :latest — even
though this fork exists and carries our commits. It runs them because nothing
here ever published an alternative: the repo has upstream .github/workflows
(which do not run on our estate) and no .hanzo lane at all, so there was no
hanzo image to point at.
Builds exactly the images the cluster consumes, no more. Building artifacts
nothing deploys is how a pipeline rots unnoticed.
⚠️ The hanzo-ml ORG HAS NO ACTIONS SECRETS — hanzoai carries GHCR_USER,
GHCR_TOKEN, GH_PAT, OCI_*, KMS_*; hanzo-ml carries none. So this BUILDS
unconditionally (which proves the fork still compiles — the thing most worth
knowing) and PUSHES only when a token is present, emitting a ::warning:: when
it is not. Until the org gets credentials the run is honestly green with "built
but not published" in the log, rather than the cluster silently serving upstream
forever.
To finish: give hanzo-ml GHCR_USER/GHCR_TOKEN (KMS-sourced, per policy), then
repoint the ns kubeflow deployments at ghcr.io/hanzoai/<image>:sha-<sha7>.
Also note the repo is a PULL MIRROR on the forge, so Actions were off entirely
(has_actions=false); enabled now, and the sync interval cut 8h -> 10m so a push
is picked up in minutes.
Co-authored-by: Hanzo Dev <[email protected]>
The Optuna suggestion service mapped only UNIFORM and LOG_UNIFORM for int
and double parameters. A parameter declaring NORMAL or LOG_NORMAL fell
through every branch in _get_optuna_search_space and was left out of the
search space, so Optuna never sampled it. The hyperparameter was silently
dropped from tuning with no error or log.
Optuna has no native normal or log-normal distribution, so the backend
cannot honor those values. Add an explicit branch for int and double that
raises a ValueError naming the parameter and distribution, turning a
silent correctness bug into an actionable error. The hyperopt suggestion
service already supports these distributions, so the proto values are
valid input that this backend must reject explicitly rather than ignore.
Adds a parametrized regression test covering NORMAL and LOG_NORMAL for
both int and double parameters.
Fixes#2665
Signed-off-by: Saivedant Hava <[email protected]>
5 new root markdown files. UPSTREAM_README.md preserves original.
No upstream code modified. Branding consumed at runtime from a
brand package per BRAND.md (@hanzo/brand, @luxfi/brand,
@zooai/brand, etc.). Multi-tenant via JWT tenant claim per
MULTI_TENANT.md. Apache-2.0 preserved; NOTICE attributes upstream.
5 new root markdown files. UPSTREAM_README.md preserves original.
No upstream code modified. Branding consumed at runtime from a
brand package per BRAND.md (@hanzo/brand, @luxfi/brand,
@zooai/brand, etc.). Multi-tenant via JWT tenant claim per
MULTI_TENANT.md. Apache-2.0 preserved; NOTICE attributes upstream.
- Add optional trial_active_deadline_seconds parameter to tune() function
- Support timeout for both Job and PyTorchJob trial types
- Set active_deadline_seconds on Job spec for Job-based trials
- Set active_deadline_seconds on RunPolicy for PyTorchJob-based trials
- Add comprehensive documentation and usage examples
- Add test cases for both Job and PyTorchJob timeout scenarios
- Maintain backward compatibility with existing code
This feature prevents individual trials from running indefinitely
and consuming cluster resources by allowing users to specify
per-trial timeouts in seconds.
Signed-off-by: wassimbensalem <[email protected]>
* Out-of-the-box support TrainJob
Signed-off-by: Ram Lau <[email protected]>
* Example for Pytorch Distributed
Signed-off-by: Ram Lau <[email protected]>
* Update examples/v1beta1/kubeflow-training-operator/trainjob-pytorch.yaml
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Ram Lau <[email protected]>
* Create folder for Trainer as suggested
Signed-off-by: Ram Lau <[email protected]>
* Movethe exmaple of trainjob to the new folder
Signed-off-by: Ram Lau <[email protected]>
* Ref the primaryContainerName to that of ClusterTrainingRuntime
Signed-off-by: Ram Lau <[email protected]>
* tenzen-y steps down from Katib approver role (#2561)
Signed-off-by: Yuki Iwai <[email protected]>
Signed-off-by: Ram Lau <[email protected]>
* Set Default value for TrainJob Success, Failure Condition and PrimaryPodLabels in the trial Template
Signed-off-by: Ram Lau <[email protected]>
* Enchance Handling for default value of Success, Fail Cond & Pod Label
Signed-off-by: Ram Lau <[email protected]>
* Bug fix for default value condition
Signed-off-by: Ram Lau <[email protected]>
* code format by hack/update-gofmt.sh
Signed-off-by: Ram Lau <[email protected]>
* add TrainJob trial Resources to cert manager config
Signed-off-by: Ram Lau <[email protected]>
* add trainjob to controller rbac
Signed-off-by: Ram Lau <[email protected]>
* Grant JobSet permission to Katib controller
Signed-off-by: Andrey Velichkevich <[email protected]>
* Remove create/delete RBAC for TrainJob
Signed-off-by: Andrey Velichkevich <[email protected]>
* Fix docker build with libpcre2
Signed-off-by: Andrey Velichkevich <[email protected]>
---------
Signed-off-by: Ram Lau <[email protected]>
Signed-off-by: Yuki Iwai <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Yuki Iwai <[email protected]>
* add e2e test for tune api
Signed-off-by: helenxie-bit <[email protected]>
* upgrade training-operator sdk
Signed-off-by: helenxie-bit <[email protected]>
* specify the version of training operator sdk
Signed-off-by: helenxie-bit <[email protected]>
* fix num_labels error and update the version of training operator controller
Signed-off-by: helenxie-bit <[email protected]>
* check the version of training operator
Signed-off-by: helenxie-bit <[email protected]>
* debug
Signed-off-by: helenxie-bit <[email protected]>
* check import path of HuggingFaceModelParams
Signed-off-by: helenxie-bit <[email protected]>
* update the version of training operator sdk
Signed-off-by: helenxie-bit <[email protected]>
* update the name of experiment
Signed-off-by: helenxie-bit <[email protected]>
* add step of checking pod
Signed-off-by: helenxie-bit <[email protected]>
* check the logs of pod
Signed-off-by: helenxie-bit <[email protected]>
* add check
Signed-off-by: helenxie-bit <[email protected]>
* check reason for imagepullbackoff
Signed-off-by: helenxie-bit <[email protected]>
* revert timeout limit
Signed-off-by: helenxie-bit <[email protected]>
* fix format
Signed-off-by: helenxie-bit <[email protected]>
* extend timeout limit
Signed-off-by: helenxie-bit <[email protected]>
* update training operator sdk version
Signed-off-by: helenxie-bit <[email protected]>
* check the logs of pod
Signed-off-by: helenxie-bit <[email protected]>
* rerun tests
Signed-off-by: helenxie-bit <[email protected]>
* update the function of getting logs
Signed-off-by: helenxie-bit <[email protected]>
* add the step of describing pod
Signed-off-by: helenxie-bit <[email protected]>
* check disk space
Signed-off-by: helenxie-bit <[email protected]>
* change work directory
Signed-off-by: helenxie-bit <[email protected]>
* change work directory
Signed-off-by: helenxie-bit <[email protected]>
* increase timeout limit
Signed-off-by: helenxie-bit <[email protected]>
* check the logs of controller and events
Signed-off-by: helenxie-bit <[email protected]>
* change work directory
Signed-off-by: helenxie-bit <[email protected]>
* change work directory
Signed-off-by: helenxie-bit <[email protected]>
* change work directory
Signed-off-by: helenxie-bit <[email protected]>
* check the logs of kubelet
Signed-off-by: helenxie-bit <[email protected]>
* check the logs of kubelet
Signed-off-by: helenxie-bit <[email protected]>
* increase cpu
Signed-off-by: helenxie-bit <[email protected]>
* check the logs of training operator
Signed-off-by: helenxie-bit <[email protected]>
* check the use of resources
Signed-off-by: helenxie-bit <[email protected]>
* check the logs of container 'pytorch' and 'storage_initializer'
Signed-off-by: helenxie-bit <[email protected]>
* fix error of checking use of resources
Signed-off-by: helenxie-bit <[email protected]>
* add other checks to find the error reason
Signed-off-by: helenxie-bit <[email protected]>
* set 'storage_config'
Signed-off-by: helenxie-bit <[email protected]>
* reduce the number of tests
Signed-off-by: helenxie-bit <[email protected]>
* Check container runtime logs
Signed-off-by: helenxie-bit <[email protected]>
* set the driver of minikube as docker
Signed-off-by: helenxie-bit <[email protected]>
* set the driver of minikube to none
Signed-off-by: helenxie-bit <[email protected]>
* check logs of pod
Signed-off-by: helenxie-bit <[email protected]>
* check memory usage
Signed-off-by: helenxie-bit <[email protected]>
* increase 'termination_grace_period_seconds' in podspec
Signed-off-by: helenxie-bit <[email protected]>
* fix annotations error
Signed-off-by: helenxie-bit <[email protected]>
* restart docker
Signed-off-by: helenxie-bit <[email protected]>
* delete restarting docker
Signed-off-by: helenxie-bit <[email protected]>
* use original docker data directory
Signed-off-by: helenxie-bit <[email protected]>
* update installation of Katib SDK with extra requires
Signed-off-by: helenxie-bit <[email protected]>
* test trainer image built with cpu
Signed-off-by: helenxie-bit <[email protected]>
* add action of free up disk space (including move docker data directory)
Signed-off-by: helenxie-bit <[email protected]>
* delete unnecessary checks and update the part of fetching pod description and logs
Signed-off-by: helenxie-bit <[email protected]>
* delete fetching pod logs
Signed-off-by: helenxie-bit <[email protected]>
* add blank line at the end of free-up-disk-space yaml file
Signed-off-by: helenxie-bit <[email protected]>
* update experiment name
Signed-off-by: helenxie-bit <[email protected]>
* update test function name to be consistent with experiment name
Signed-off-by: helenxie-bit <[email protected]>
* move import statements inside the function
Signed-off-by: helenxie-bit <[email protected]>
* apply pprint for the logging output
Signed-off-by: helenxie-bit <[email protected]>
* update experiment names
Signed-off-by: helenxie-bit <[email protected]>
* fix format
Signed-off-by: helenxie-bit <[email protected]>
* fix format
Signed-off-by: helenxie-bit <[email protected]>
* fix the sequence of arguments in 'trial_template'
Signed-off-by: helenxie-bit <[email protected]>
* test example in user guide
Signed-off-by: helenxie-bit <[email protected]>
* fix access token error
Signed-off-by: helenxie-bit <[email protected]>
* fix the error of setup
Signed-off-by: helenxie-bit <[email protected]>
* fix the error of setup
Signed-off-by: helenxie-bit <[email protected]>
* reverse back
Signed-off-by: helenxie-bit <[email protected]>
* fix format
Signed-off-by: helenxie-bit <[email protected]>
* fix format
Signed-off-by: helenxie-bit <[email protected]>
---------
Signed-off-by: helenxie-bit <[email protected]>
* Fix Istio sidecar injection by moving from annotations to labels
Signed-off-by: madmecodes <[email protected]>
* Update Istio sidecar injection from annotations to labels across the codebase
Replace annotations with labels for Istio sidecar injection according to Istio recommendations. Update conformance tests, examples, constants, composers, and utilities to use the new label-based approach consistently.
Signed-off-by: madmecodes <[email protected]>
* fix: Update SuggestionLabels function and composer implementation for Istio label injection
Signed-off-by: madmecodes <[email protected]>
* Fix linting issues in mpi-job-horovod.py
Signed-off-by: madmecodes <[email protected]>
* update: function moved from annotations to labels
Signed-off-by: madmecodes <[email protected]>
---------
Signed-off-by: madmecodes <[email protected]>
zeekayandHanzo Dev