* Out-of-the-box support TrainJob
Signed-off-by: Ram Lau <[email protected]>
* Example for Pytorch Distributed
Signed-off-by: Ram Lau <[email protected]>
* Update examples/v1beta1/kubeflow-training-operator/trainjob-pytorch.yaml
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Ram Lau <[email protected]>
* Create folder for Trainer as suggested
Signed-off-by: Ram Lau <[email protected]>
* Movethe exmaple of trainjob to the new folder
Signed-off-by: Ram Lau <[email protected]>
* Ref the primaryContainerName to that of ClusterTrainingRuntime
Signed-off-by: Ram Lau <[email protected]>
* tenzen-y steps down from Katib approver role (#2561)
Signed-off-by: Yuki Iwai <[email protected]>
Signed-off-by: Ram Lau <[email protected]>
* Set Default value for TrainJob Success, Failure Condition and PrimaryPodLabels in the trial Template
Signed-off-by: Ram Lau <[email protected]>
* Enchance Handling for default value of Success, Fail Cond & Pod Label
Signed-off-by: Ram Lau <[email protected]>
* Bug fix for default value condition
Signed-off-by: Ram Lau <[email protected]>
* code format by hack/update-gofmt.sh
Signed-off-by: Ram Lau <[email protected]>
* add TrainJob trial Resources to cert manager config
Signed-off-by: Ram Lau <[email protected]>
* add trainjob to controller rbac
Signed-off-by: Ram Lau <[email protected]>
* Grant JobSet permission to Katib controller
Signed-off-by: Andrey Velichkevich <[email protected]>
* Remove create/delete RBAC for TrainJob
Signed-off-by: Andrey Velichkevich <[email protected]>
* Fix docker build with libpcre2
Signed-off-by: Andrey Velichkevich <[email protected]>
---------
Signed-off-by: Ram Lau <[email protected]>
Signed-off-by: Yuki Iwai <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Yuki Iwai <[email protected]>
* Consolidate the katib-cert-generator to the katib-controller
Signed-off-by: Yuki Iwai <[email protected]>
* Use deployed secret instead of creating a new secret when the cert-generator saves certs on secret
Signed-off-by: Yuki Iwai <[email protected]>
* Rename secretName with webhookSecretName in the .init.certGenerator
Signed-off-by: Yuki Iwai <[email protected]>
* Fix manifests
Signed-off-by: Yuki Iwai <[email protected]>
* Remove unneeded comments
Signed-off-by: Yuki Iwai <[email protected]>
* Restore unintentionally deleted log
Signed-off-by: Yuki Iwai <[email protected]>
* Rename package cert-generator with certgenerator
Signed-off-by: Yuki Iwai <[email protected]>
* Add test cases to check if the enable is set to true when the webhookServiceName or webhookSecretName is set
Signed-off-by: Yuki Iwai <[email protected]>
* Update the developer guide
Signed-off-by: Yuki Iwai <[email protected]>
* Swap livness probe and readiness probe
Signed-off-by: Yuki Iwai <[email protected]>
* Introduce SSA to the cert-generator
Signed-off-by: Yuki Iwai <[email protected]>
* Use the same member names between CertGenerator and KatibConfig
Signed-off-by: Yuki Iwai <[email protected]>
* Disable leader election on the cert-generator
Signed-off-by: Yuki Iwai <[email protected]>
* Drop unneeded fields from SSA patches
Signed-off-by: Yuki Iwai <[email protected]>
---------
Signed-off-by: Yuki Iwai <[email protected]>
* Update training operator image in CI
* Remove deprecated GRPC var
* Remove deprecated GRPC var
* Remove deprecated GRPC var
* Support for k8s v1.25 in CI
* Revert "Support for k8s v1.25 in CI"
This reverts commit 16e6fe4b16820aea30e266a5c69560a92cfb851c.
* docs: update new algorithm service details
* feat: trial augmentation strategy
* feat: pbt suggestion service
* feat: PbtTemplate and associated test image
* feat: introduce annotation field to trial specifications
* feat: trial assignment changes to support annotations from suggestion
- Add new Annotation types to suggestion_types.go
- Add Annotation object and update Trial parser in trial.py
* feat: update pbt suggestion to use new Annotation api
- Suggestion uses exact match to track spawned trials
- Trials that get transmitted, but not created (or added to experiment) are added back to the respawn pool (population_size consistency)
* chore: gofmt and black run across PBT changes
* feedback: remove tf summary export, change default print unit, reduce range to be percentage compatible.
* feedback: move PBT template to example.
* feedback: changes to inject_webhook and utils.
- Rename mutateVolume to mutateMetricsCollectorVolume
- Add addContainerVolumeMount
- Add getPrimaryContainerIndex
* feedback: change suggestion mutation mount variable name and add to consts
* feedback: Add trial_names to GetSuggestionsReply and change suggestion path to <experiment>/<trial>
* feedback: removed unnecessary checks and moved to async pbt implementation
* feedback: update trial name override location and change annotations override to labels.
* feedback: add pbt to github workflow
* feedback: move labels to ParameterAssignments in GetSuggestionsReply and cleanup pbt.yaml.
* feedback: remove operator changes
* feedback: GHA updates
* feedback: new formatting changes
* feedback: add suggestion-pbt to gh-actions build-load.sh.
* fix: missing pbt->simple-pbt name changes, add simple-pbt to update-images.sh update yaml function (causing failing gha).
* feedback: add pointer to website from main readme for pbt
* include MetricsUnavailable condition to Complete in Trial
It is not easy for users to find why Trial failed when training code output incorrect format logs
since the trial-controller sets Succeeded condition with False to Trial if there are unavailable metrics in Katib DB as described in https://github.com/kubeflow/katib/issues/1343.
So we also include MetricsUnavailable condition to Complete in Trial.
* add gh-actions tasks to verify generated codes
* fix gh-actions workflow
* when the number of Failed Trials reaches maxTrialCount, experiment-controller sets Failed to Experiment status
* fix e2e test
* To avoid being set Failed in Experiment status when and is equal to 0, we need to add condition,
* upgrade kubebuilder version from v2.3.0 to v3.2.0
* fix envtest for experiment-controller
* fix suite test
To avoid the `timeout waiting for process kube-apiserver to stop` error, we must use the `context.WithCancel`.
Ref: https://github.com/kubernetes-sigs/controller-runtime/issues/1571#issuecomment-945535598
* update Go version to v1.17 in kubeflow-katib-presubmit
To avoid the `../../../../pkg/mod/k8s.io/[email protected].2/plugin/pkg/client/auth/exec/metrics.go:21:2: package io/fs is not in GOROOT (/usr/local/go/src/io/fs)` error,
we must use Go v1.16 or later, but as described in https://github.com/kubeflow/training-operator/issues/1581,
we do not have permission to update `public.ecr.aws/j1r0q0g6/kubeflow-testing:latest` so we need to update it in this.
* support JSON format logs in file-metrics-collector
* review: convert fileFormat to type FileSystemFileFormat
* Update cmd/metricscollector/v1beta1/file-metricscollector/main.go
Co-authored-by: Andrey Velichkevich <[email protected]>
* review: remove func (f FileSystemFileFormat) String()
* review: get metricRegList only when the format is TEXT
* review: change var name in a script for e2e
* review: explict specify the cloudml-hypyertune in the Dockerfile
* review: use reflect.DeepEqual instead of go-cmp.Diff
* review: stop using 'JSON' directly in error statements
* review: install specific version cloudml-hypertune
* review: get objType in the updateStopRules function
* review: save optimalObjValue across multiple stopRules
* review: add warning messages to parseTimestamp func
* review: generate test files with go test command
* review: change api for new feature
Co-authored-by: Andrey Velichkevich <[email protected]>
* implement validation for early stopping
* fix some documents
* fix error messages
* implement gRPC API to verify parameters for early stopping
* review: use early_stopping as gRPC API
Co-authored-by: Andrey Velichkevich <[email protected]>
* review: fix error description
Co-authored-by: Andrey Velichkevich <[email protected]>
* review: remove t.Run
* review: remove condition to verify algorithmName for early stopping
* remove description about updating gRPC API docs in kubeflow website
Co-authored-by: Andrey Velichkevich <[email protected]>
* Update boilerplate for clients
* Modify not format boilerplate
* Modify boilerplate script for go files
* Update boilerplate manually for go files
* Modify script
* Generate boilerplate for Go files
* Add script for Python files
* Generate boilerplate for the Py files
* Include shell
* Generate boilerplate for shell scripts
* Add to makefile
* Fix comments
* Change comments
* Update all dependencies to the latest versions
Add cert generator for the webhooks
Add manifests for the webhooks
* Modify Dockerfile for manager
* Remove comments
* Update Dockerfiles for Go images
* Add signerName: kubernetes.io/kube-apiserver-client to csr
Update roles for controller RBAC
Changes after review
* Fix not installed CRD error
* Update scripts
* Revert operator changes
* Describe controller pod in test
* Add log line to test
* Move kubectl version
* Change csr version to v1beta1
* Remove log
* Change signerName to kubernetes.io/kubelet-serving
* Modify common name
Co-authored-by: Yuki Iwai <[email protected]>
* Add env variable to init container
Co-authored-by: Yuki Iwai <[email protected]>
* Get namespace from env
Co-authored-by: Yuki Iwai <[email protected]>
* Remove quotes
* Remove spaces
* Run cert generator script from the Job
* Modify new ui Dockerfile
* Disable Actions on PR
* Modify setup Katib script
* Fix PODNUM
* Remove imagePullPolicy from PyTorch and TFJob examples
* Disable Pytorch examples in e2e
* Add sleep to e2e test
* Activate Actions
* Disable actions
Co-authored-by: Yuki Iwai <[email protected]>
* Init commit
* Experiment API changes
* Add APIs
* Remove old es apis
* Remove logging
* Add Early Stopping implementation
* Show metrics for early stopped trials
* Clean const
* Remove pid file with completed line from metrics collector
* Add unit test
* Add EarlyStopping as unique service
* Add early stopped Trials to completed
* Fix few comments
* Generate clients and SDK
* Fix tests
* Fix goptuna test
* Remove cluster role from test
* Update observation for early stopped Trials
* Fix pv name in e2e test
* Add Katib config for Early Stopping
* Fix comment
* Remove unused gRPC Experiment spec
* Remove legacy test files
* Remove labels and conditions from example
* Modify API to be consistent with Algorithm
* Ignore no such file error
* Add comments to proto
* Add median stop implementation
* Fix few comments
* Fix unit tests
* Add o-type flag to tfevent metrics collector
* Fix es unit test
* Fix hyperband suggestion
* Init commit
* Modify Insert function
Add retry on empty observation
* Fix mutate volume test
* Fix validate experiment test
* Fix invalid experiment
* Don't get deployed job status when trial is completed
* Not send Trial with unavailable metrics to Suggestion
* Refactor requeue
If objective metric value is not reported metrics collector reports unavailable value to the DB
Controller reconciles Trial until DB is empty
* Add condition before change trial status
* Remove prints
* Fix tfevent parser
