mirror of
https://github.com/hanzo-ml/katib.git
synced 2026-10-30 14:46:14 +00:00
Ayush GuptaandGitHub
1c19d3a6eb
* Fix Istio sidecar injection by moving from annotations to labels Signed-off-by: madmecodes <[email protected]> * Update Istio sidecar injection from annotations to labels across the codebase Replace annotations with labels for Istio sidecar injection according to Istio recommendations. Update conformance tests, examples, constants, composers, and utilities to use the new label-based approach consistently. Signed-off-by: madmecodes <[email protected]> * fix: Update SuggestionLabels function and composer implementation for Istio label injection Signed-off-by: madmecodes <[email protected]> * Fix linting issues in mpi-job-horovod.py Signed-off-by: madmecodes <[email protected]> * update: function moved from annotations to labels Signed-off-by: madmecodes <[email protected]> --------- Signed-off-by: madmecodes <[email protected]>
22 KiB
22 KiB
In [1]:
# Install required packages (Kubeflow Pipelines and Katib SDK).
!pip install kfp==1.8.12
!pip install kubeflow-katib==0.13.0Requirement already satisfied: kfp==1.8.12 in /opt/conda/lib/python3.8/site-packages (1.8.12) Requirement already satisfied: fire<1,>=0.3.1 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (0.4.0) Requirement already satisfied: absl-py<2,>=0.9 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (0.11.0) Requirement already satisfied: jsonschema<4,>=3.0.1 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (3.2.0) Requirement already satisfied: typer<1.0,>=0.3.2 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (0.4.1) Requirement already satisfied: uritemplate<4,>=3.0.1 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (3.0.1) Requirement already satisfied: typing-extensions<4,>=3.7.4 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (3.10.0.2) Requirement already satisfied: tabulate<1,>=0.8.6 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (0.8.9) Requirement already satisfied: google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (2.7.1) Requirement already satisfied: kfp-pipeline-spec<0.2.0,>=0.1.14 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (0.1.14) Requirement already satisfied: pydantic<2,>=1.8.2 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (1.9.0) Requirement already satisfied: cloudpickle<3,>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (2.0.0) Requirement already satisfied: docstring-parser<1,>=0.7.3 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (0.13) Requirement already satisfied: kfp-server-api<2.0.0,>=1.1.2 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (1.6.0) Requirement already satisfied: google-api-python-client<2,>=1.7.8 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (1.12.10) Requirement already satisfied: google-auth<2,>=1.6.1 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (1.35.0) Requirement already satisfied: strip-hints<1,>=0.1.8 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (0.1.10) Requirement already satisfied: Deprecated<2,>=1.2.7 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (1.2.13) Requirement already satisfied: click<9,>=7.1.2 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (7.1.2) Requirement already satisfied: PyYAML<6,>=5.3 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (5.4.1) Requirement already satisfied: kubernetes<19,>=8.0.0 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (12.0.1) Requirement already satisfied: google-cloud-storage<2,>=1.20.0 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (1.44.0) Requirement already satisfied: protobuf<4,>=3.13.0 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (3.17.3) Requirement already satisfied: requests-toolbelt<1,>=0.8.0 in /opt/conda/lib/python3.8/site-packages (from kfp==1.8.12) (0.9.1) Requirement already satisfied: six in /opt/conda/lib/python3.8/site-packages (from absl-py<2,>=0.9->kfp==1.8.12) (1.16.0) Requirement already satisfied: wrapt<2,>=1.10 in /opt/conda/lib/python3.8/site-packages (from Deprecated<2,>=1.2.7->kfp==1.8.12) (1.13.3) Requirement already satisfied: termcolor in /opt/conda/lib/python3.8/site-packages (from fire<1,>=0.3.1->kfp==1.8.12) (1.1.0) Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.52.0 in /opt/conda/lib/python3.8/site-packages (from google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->kfp==1.8.12) (1.55.0) Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.8/site-packages (from google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->kfp==1.8.12) (2.25.1) Requirement already satisfied: httplib2<1dev,>=0.15.0 in /opt/conda/lib/python3.8/site-packages (from google-api-python-client<2,>=1.7.8->kfp==1.8.12) (0.20.4) Requirement already satisfied: google-auth-httplib2>=0.0.3 in /opt/conda/lib/python3.8/site-packages (from google-api-python-client<2,>=1.7.8->kfp==1.8.12) (0.1.0) Requirement already satisfied: setuptools>=40.3.0 in /opt/conda/lib/python3.8/site-packages (from google-auth<2,>=1.6.1->kfp==1.8.12) (49.6.0.post20210108) Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from google-auth<2,>=1.6.1->kfp==1.8.12) (4.2.4) Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.8/site-packages (from google-auth<2,>=1.6.1->kfp==1.8.12) (4.8) Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.8/site-packages (from google-auth<2,>=1.6.1->kfp==1.8.12) (0.2.8) Requirement already satisfied: google-resumable-media<3.0dev,>=1.3.0 in /opt/conda/lib/python3.8/site-packages (from google-cloud-storage<2,>=1.20.0->kfp==1.8.12) (2.3.2) Requirement already satisfied: google-cloud-core<3.0dev,>=1.6.0 in /opt/conda/lib/python3.8/site-packages (from google-cloud-storage<2,>=1.20.0->kfp==1.8.12) (2.2.3) Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.8/site-packages (from google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage<2,>=1.20.0->kfp==1.8.12) (1.3.0) Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /opt/conda/lib/python3.8/site-packages (from httplib2<1dev,>=0.15.0->google-api-python-client<2,>=1.7.8->kfp==1.8.12) (2.4.7) Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.8/site-packages (from jsonschema<4,>=3.0.1->kfp==1.8.12) (21.2.0) Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.8/site-packages (from jsonschema<4,>=3.0.1->kfp==1.8.12) (0.17.3) Requirement already satisfied: python-dateutil in /opt/conda/lib/python3.8/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp==1.8.12) (2.8.1) Requirement already satisfied: urllib3>=1.15 in /opt/conda/lib/python3.8/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp==1.8.12) (1.26.5) Requirement already satisfied: certifi in /opt/conda/lib/python3.8/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp==1.8.12) (2021.5.30) Requirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.8/site-packages (from kubernetes<19,>=8.0.0->kfp==1.8.12) (1.3.1) Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /opt/conda/lib/python3.8/site-packages (from kubernetes<19,>=8.0.0->kfp==1.8.12) (1.0.1) Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.1->kfp==1.8.12) (0.4.8) Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->kfp==1.8.12) (2.10) Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.8/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->kfp==1.8.12) (4.0.0) Requirement already satisfied: wheel in /opt/conda/lib/python3.8/site-packages (from strip-hints<1,>=0.1.8->kfp==1.8.12) (0.36.2) Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.8/site-packages (from requests-oauthlib->kubernetes<19,>=8.0.0->kfp==1.8.12) (3.2.0) Requirement already satisfied: kubeflow-katib==0.13.0 in /opt/conda/lib/python3.8/site-packages (0.13.0) Requirement already satisfied: kubernetes>=12.0.0 in /opt/conda/lib/python3.8/site-packages (from kubeflow-katib==0.13.0) (12.0.1) Requirement already satisfied: certifi>=14.05.14 in /opt/conda/lib/python3.8/site-packages (from kubeflow-katib==0.13.0) (2021.5.30) Requirement already satisfied: six>=1.10 in /opt/conda/lib/python3.8/site-packages (from kubeflow-katib==0.13.0) (1.16.0) Requirement already satisfied: urllib3>=1.15.1 in /opt/conda/lib/python3.8/site-packages (from kubeflow-katib==0.13.0) (1.26.5) Requirement already satisfied: setuptools>=21.0.0 in /opt/conda/lib/python3.8/site-packages (from kubeflow-katib==0.13.0) (49.6.0.post20210108) Requirement already satisfied: requests in /opt/conda/lib/python3.8/site-packages (from kubernetes>=12.0.0->kubeflow-katib==0.13.0) (2.25.1) Requirement already satisfied: python-dateutil>=2.5.3 in /opt/conda/lib/python3.8/site-packages (from kubernetes>=12.0.0->kubeflow-katib==0.13.0) (2.8.1) Requirement already satisfied: pyyaml>=3.12 in /opt/conda/lib/python3.8/site-packages (from kubernetes>=12.0.0->kubeflow-katib==0.13.0) (5.4.1) Requirement already satisfied: google-auth>=1.0.1 in /opt/conda/lib/python3.8/site-packages (from kubernetes>=12.0.0->kubeflow-katib==0.13.0) (1.35.0) Requirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.8/site-packages (from kubernetes>=12.0.0->kubeflow-katib==0.13.0) (1.3.1) Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /opt/conda/lib/python3.8/site-packages (from kubernetes>=12.0.0->kubeflow-katib==0.13.0) (1.0.1) Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes>=12.0.0->kubeflow-katib==0.13.0) (4.2.4) Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes>=12.0.0->kubeflow-katib==0.13.0) (0.2.8) Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes>=12.0.0->kubeflow-katib==0.13.0) (4.8) Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes>=12.0.0->kubeflow-katib==0.13.0) (0.4.8) Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.8/site-packages (from requests->kubernetes>=12.0.0->kubeflow-katib==0.13.0) (4.0.0) Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests->kubernetes>=12.0.0->kubeflow-katib==0.13.0) (2.10) Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.8/site-packages (from requests-oauthlib->kubernetes>=12.0.0->kubeflow-katib==0.13.0) (3.2.0)
In [2]:
import kfp
import kfp.dsl as dsl
from kfp import components
from kubeflow.katib import ApiClient
from kubeflow.katib import V1beta1ExperimentSpec
from kubeflow.katib import V1beta1AlgorithmSpec
from kubeflow.katib import V1beta1EarlyStoppingSpec
from kubeflow.katib import V1beta1EarlyStoppingSetting
from kubeflow.katib import V1beta1ObjectiveSpec
from kubeflow.katib import V1beta1ParameterSpec
from kubeflow.katib import V1beta1FeasibleSpace
from kubeflow.katib import V1beta1TrialTemplate
from kubeflow.katib import V1beta1TrialParameterSpecIn [3]:
# Experiment name and namespace.
experiment_name = "median-stop"
experiment_namespace = "kubeflow-user-example-com"
# Trial count specification.
max_trial_count = 18
max_failed_trial_count = 3
parallel_trial_count = 2
# Objective specification.
objective=V1beta1ObjectiveSpec(
type="minimize",
goal= 0.001,
objective_metric_name="loss",
)
# Algorithm specification.
algorithm=V1beta1AlgorithmSpec(
algorithm_name="random",
)
# Early Stopping specification.
early_stopping=V1beta1EarlyStoppingSpec(
algorithm_name="medianstop",
algorithm_settings=[
V1beta1EarlyStoppingSetting(
name="min_trials_required",
value="2"
)
]
)
# Experiment search space.
# In this example we tune learning rate, number of layer and optimizer.
# Learning rate has bad feasible space to show more early stopped Trials.
parameters=[
V1beta1ParameterSpec(
name="lr",
parameter_type="double",
feasible_space=V1beta1FeasibleSpace(
min="0.01",
max="0.3"
),
),
V1beta1ParameterSpec(
name="momentum",
parameter_type="double",
feasible_space=V1beta1FeasibleSpace(
min="0.5",
max="0.9"
),
),
]
In [4]:
# JSON template specification for the Trial's Worker Kubernetes Job.
trial_spec={
"apiVersion": "batch/v1",
"kind": "Job",
"spec": {
"template": {
"metadata": {
"labels": {
"sidecar.istio.io/inject": "false"
}
},
"spec": {
"containers": [
{
"name": "training-container",
"image": "docker.io/kubeflowkatib/pytorch-mnist-cpu:v0.14.0",
"command": [
"python3",
"/opt/pytorch-mnist/mnist.py",
"--epochs=1",
"--batch-size=16",
"--lr=${trialParameters.learningRate}",
"--momentum=${trialParameters.momentum}",
]
}
],
"restartPolicy": "Never"
}
}
}
}
# Configure parameters for the Trial template.
# We set the retain parameter to "True" to not clean-up the Trial Job's Kubernetes Pods.
trial_template=V1beta1TrialTemplate(
retain=True,
primary_container_name="training-container",
trial_parameters=[
V1beta1TrialParameterSpec(
name="learningRate",
description="Learning rate for the training model",
reference="lr"
),
V1beta1TrialParameterSpec(
name="momentum",
description="Momentum for the training model",
reference="momentum"
),
],
trial_spec=trial_spec
)In [5]:
experiment_spec=V1beta1ExperimentSpec(
max_trial_count=max_trial_count,
max_failed_trial_count=max_failed_trial_count,
parallel_trial_count=parallel_trial_count,
objective=objective,
algorithm=algorithm,
early_stopping=early_stopping,
parameters=parameters,
trial_template=trial_template
)In [6]:
# Get the Katib launcher.
katib_experiment_launcher_op = components.load_component_from_url(
"https://raw.githubusercontent.com/kubeflow/pipelines/master/components/kubeflow/katib-launcher/component.yaml")
@dsl.pipeline(
name="Launch Katib early stopping Experiment",
description="An example to launch Katib Experiment with early stopping"
)
def median_stop():
# Katib launcher component.
# Experiment Spec should be serialized to a valid Kubernetes object.
op = katib_experiment_launcher_op(
experiment_name=experiment_name,
experiment_namespace=experiment_namespace,
experiment_spec=ApiClient().sanitize_for_serialization(experiment_spec),
experiment_timeout_minutes=60,
delete_finished_experiment=False)
# Output container to print the results.
op_out = dsl.ContainerOp(
name="best-hp",
image="library/bash:4.4.23",
command=["sh", "-c"],
arguments=["echo Best HyperParameters: %s" % op.output],
)In [7]:
# Run the Kubeflow Pipeline in the user's namespace.
kfp.Client().create_run_from_pipeline_func(median_stop, namespace=experiment_namespace, arguments={})Out [7]:
/opt/conda/lib/python3.8/site-packages/kfp/dsl/_container_op.py:1257: FutureWarning: Please create reusable components instead of constructing ContainerOp instances directly. Reusable components are shareable, portable and have compatibility and support guarantees. Please see the documentation: https://www.kubeflow.org/docs/pipelines/sdk/component-development/#writing-your-component-definition-file The components can be created manually (or, in case of python, using kfp.components.create_component_from_func or func_to_container_op) and then loaded using kfp.components.load_component_from_file, load_component_from_uri or load_component_from_text: https://kubeflow-pipelines.readthedocs.io/en/stable/source/kfp.components.html#kfp.components.load_component_from_file warnings.warn(
RunPipelineResult(run_id=bb6689a9-1efa-4fd7-bcb1-09f47bf1e932)
In [ ]: