* Out-of-the-box support TrainJob
Signed-off-by: Ram Lau <[email protected]>
* Example for Pytorch Distributed
Signed-off-by: Ram Lau <[email protected]>
* Update examples/v1beta1/kubeflow-training-operator/trainjob-pytorch.yaml
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Ram Lau <[email protected]>
* Create folder for Trainer as suggested
Signed-off-by: Ram Lau <[email protected]>
* Movethe exmaple of trainjob to the new folder
Signed-off-by: Ram Lau <[email protected]>
* Ref the primaryContainerName to that of ClusterTrainingRuntime
Signed-off-by: Ram Lau <[email protected]>
* tenzen-y steps down from Katib approver role (#2561)
Signed-off-by: Yuki Iwai <[email protected]>
Signed-off-by: Ram Lau <[email protected]>
* Set Default value for TrainJob Success, Failure Condition and PrimaryPodLabels in the trial Template
Signed-off-by: Ram Lau <[email protected]>
* Enchance Handling for default value of Success, Fail Cond & Pod Label
Signed-off-by: Ram Lau <[email protected]>
* Bug fix for default value condition
Signed-off-by: Ram Lau <[email protected]>
* code format by hack/update-gofmt.sh
Signed-off-by: Ram Lau <[email protected]>
* add TrainJob trial Resources to cert manager config
Signed-off-by: Ram Lau <[email protected]>
* add trainjob to controller rbac
Signed-off-by: Ram Lau <[email protected]>
* Grant JobSet permission to Katib controller
Signed-off-by: Andrey Velichkevich <[email protected]>
* Remove create/delete RBAC for TrainJob
Signed-off-by: Andrey Velichkevich <[email protected]>
* Fix docker build with libpcre2
Signed-off-by: Andrey Velichkevich <[email protected]>
---------
Signed-off-by: Ram Lau <[email protected]>
Signed-off-by: Yuki Iwai <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Yuki Iwai <[email protected]>
* docs: update new algorithm service details
* feat: trial augmentation strategy
* feat: pbt suggestion service
* feat: PbtTemplate and associated test image
* feat: introduce annotation field to trial specifications
* feat: trial assignment changes to support annotations from suggestion
- Add new Annotation types to suggestion_types.go
- Add Annotation object and update Trial parser in trial.py
* feat: update pbt suggestion to use new Annotation api
- Suggestion uses exact match to track spawned trials
- Trials that get transmitted, but not created (or added to experiment) are added back to the respawn pool (population_size consistency)
* chore: gofmt and black run across PBT changes
* feedback: remove tf summary export, change default print unit, reduce range to be percentage compatible.
* feedback: move PBT template to example.
* feedback: changes to inject_webhook and utils.
- Rename mutateVolume to mutateMetricsCollectorVolume
- Add addContainerVolumeMount
- Add getPrimaryContainerIndex
* feedback: change suggestion mutation mount variable name and add to consts
* feedback: Add trial_names to GetSuggestionsReply and change suggestion path to <experiment>/<trial>
* feedback: removed unnecessary checks and moved to async pbt implementation
* feedback: update trial name override location and change annotations override to labels.
* feedback: add pbt to github workflow
* feedback: move labels to ParameterAssignments in GetSuggestionsReply and cleanup pbt.yaml.
* feedback: remove operator changes
* feedback: GHA updates
* feedback: new formatting changes
* feedback: add suggestion-pbt to gh-actions build-load.sh.
* fix: missing pbt->simple-pbt name changes, add simple-pbt to update-images.sh update yaml function (causing failing gha).
* feedback: add pointer to website from main readme for pbt
* include MetricsUnavailable condition to Complete in Trial
It is not easy for users to find why Trial failed when training code output incorrect format logs
since the trial-controller sets Succeeded condition with False to Trial if there are unavailable metrics in Katib DB as described in https://github.com/kubeflow/katib/issues/1343.
So we also include MetricsUnavailable condition to Complete in Trial.
* add gh-actions tasks to verify generated codes
* fix gh-actions workflow
* when the number of Failed Trials reaches maxTrialCount, experiment-controller sets Failed to Experiment status
* fix e2e test
* To avoid being set Failed in Experiment status when and is equal to 0, we need to add condition,
* support JSON format logs in file-metrics-collector
* review: convert fileFormat to type FileSystemFileFormat
* Update cmd/metricscollector/v1beta1/file-metricscollector/main.go
Co-authored-by: Andrey Velichkevich <[email protected]>
* review: remove func (f FileSystemFileFormat) String()
* review: get metricRegList only when the format is TEXT
* review: change var name in a script for e2e
* review: explict specify the cloudml-hypyertune in the Dockerfile
* review: use reflect.DeepEqual instead of go-cmp.Diff
* review: stop using 'JSON' directly in error statements
* review: install specific version cloudml-hypertune
* review: get objType in the updateStopRules function
* review: save optimalObjValue across multiple stopRules
* review: add warning messages to parseTimestamp func
* review: generate test files with go test command
* review: change api for new feature
Co-authored-by: Andrey Velichkevich <[email protected]>
* Update boilerplate for clients
* Modify not format boilerplate
* Modify boilerplate script for go files
* Update boilerplate manually for go files
* Modify script
* Generate boilerplate for Go files
* Add script for Python files
* Generate boilerplate for the Py files
* Include shell
* Generate boilerplate for shell scripts
* Add to makefile
* Fix comments
* Change comments
* Update all dependencies to the latest versions
Add cert generator for the webhooks
Add manifests for the webhooks
* Modify Dockerfile for manager
* Remove comments
* Update Dockerfiles for Go images
* Add signerName: kubernetes.io/kube-apiserver-client to csr
Update roles for controller RBAC
Changes after review
* Fix not installed CRD error
* Update scripts
* Revert operator changes
* Describe controller pod in test
* Add log line to test
* Move kubectl version
* Change csr version to v1beta1
* Remove log
* Change signerName to kubernetes.io/kubelet-serving
* Modify common name
Co-authored-by: Yuki Iwai <[email protected]>
* Add env variable to init container
Co-authored-by: Yuki Iwai <[email protected]>
* Get namespace from env
Co-authored-by: Yuki Iwai <[email protected]>
* Remove quotes
* Remove spaces
* Run cert generator script from the Job
* Modify new ui Dockerfile
* Disable Actions on PR
* Modify setup Katib script
* Fix PODNUM
* Remove imagePullPolicy from PyTorch and TFJob examples
* Disable Pytorch examples in e2e
* Add sleep to e2e test
* Activate Actions
* Disable actions
Co-authored-by: Yuki Iwai <[email protected]>
* Init commit
* Experiment API changes
* Add APIs
* Remove old es apis
* Remove logging
* Add Early Stopping implementation
* Show metrics for early stopped trials
* Clean const
* Remove pid file with completed line from metrics collector
* Add unit test
* Add EarlyStopping as unique service
* Add early stopped Trials to completed
* Fix few comments
* Generate clients and SDK
* Fix tests
* Fix goptuna test
* Remove cluster role from test
* Update observation for early stopped Trials
* Fix pv name in e2e test
* Add Katib config for Early Stopping
* Fix comment
* Remove unused gRPC Experiment spec
* Remove legacy test files
* Remove labels and conditions from example
* Modify API to be consistent with Algorithm
* Ignore no such file error
* Add comments to proto
* Add median stop implementation
* Fix few comments
* Fix unit tests
* Add o-type flag to tfevent metrics collector
* Fix es unit test
* Fix hyperband suggestion
* Init commit
* Modify Insert function
Add retry on empty observation
* Fix mutate volume test
* Fix validate experiment test
* Fix invalid experiment
* Don't get deployed job status when trial is completed
* Not send Trial with unavailable metrics to Suggestion
* Refactor requeue
If objective metric value is not reported metrics collector reports unavailable value to the DB
Controller reconciles Trial until DB is empty
* Add condition before change trial status
* Remove prints
* Fix tfevent parser
* Add unit test for Experiment and Suggestion controller reconcile
* Delete buildTrialMetaForRunSpec from controller
* Modify condition check
* Fix format
* Run mock for new version
* Refactor experiment reconcile test
* Remove comment
* Resume experiment from the PV
* Add comment
* Remove old api comments
* Change reason for Running suggestion
* Fix few comments
* Rename volume name like suggestion deployment
* Add corev1 to const
* Change metric type from float to string
* Check unavailable latest objective metric in isTrialObservationAvailable
* Fix e2e test
* Delete consts.UnavailableMetricValue from e2e
* Add new Trial Template API
* Add TrialSource in TrialTemplate
Remove GoTemplate from Experiment API
* Add init logic for new trial template
* Change examples
Implement new Trial Template to controller
* Modify trial template configMap for new version of Trial Template
Fix experiment defaults
change valid/invalid experiment
* Run gofmt
* Fix hyperband
* Remove num-epochs from grid example
* Add tag to file mc example
* Modify create Trials loop
* Add v1beta1 version
* Swagger for v1alpha3 and v1beta1 versions
Fix format in bash scripts
* Change make build to make buildv1alpha3
* Add folder path to python test
* Fix folder in python test
* Add goptuna and darts suggestions to check-katib-ready
* Disable custom metrics collector e2e in v1beta1
* Delete Suggestion deployment after Experiment is finished
* fix
* update
* update openapi
* fix validator test
* refine
* add debug info
* try to fix
* try to fix
* try to fix
* keep API style consistent
* rebase master
