mirror of https://github.com/hanzo-ml/katib.git synced 2026-09-24 00:31:27 +00:00

Files

T

Andrey VelichkevichandGitHub 268e68e9ab [SDK] Use Katib SDK for E2E Tests (#2075 )

* [SDK] Use Katib SDK for E2E tests

* Fix pvc deletion

* Add list_suggestions API

* Remove wait from edit Experiment function

* Add shell to GitHub action

* Add protobuf package to Katib SDK

* Add Experiment Timeout to 40 min

* Modify SDK Examples

* Fix example text

* Change to custom_api

* Enable verbose logging for Katib E2E

* Use expected condition arg

* Add timeout and delete options

* Modify logging to debug

* Use read API to check resource status

2023-01-16 16:40:59 +00:00

550 KiB

Raw Permalink Blame History

Tune and Train with Kubeflow Katib and Training Operator

In this Notebook we are going to do the following:

Train Tensorflow model using Kubeflow Notebook.
Improve the model HyperParameters with Kubeflow Katib.
Use Multi Worker Mirrored Strategy to distributively train the model with Kubeflow TFJob.

Install Kubeflow Python SDKs

You need to install Tensorflow package and Kubeflow SDKs to run this Notebook.

In [ ]:

!pip install tensorflow==2.9.1

# TODO (andreyvelich): Change to release version when SDK with the new APIs is published.
!pip install git+https://github.com/kubeflow/katib.git#subdirectory=sdk/python/v1beta1
!pip install git+https://github.com/kubeflow/training-operator.git#subdirectory=sdk/python

Create Train Script for CNN Model

This is simple Convolutional Neural Network (CNN) model for recognizing hand-written digits using MNIST Dataset.

In [1]:

def train_mnist_model(parameters):
    import tensorflow as tf
    import numpy as np
    import logging

    logging.basicConfig(
        format="%(asctime)s %(levelname)-8s %(message)s",
        datefmt="%Y-%m-%dT%H:%M:%SZ",
        level=logging.INFO,
    )
    logging.info("--------------------------------------------------------------------------------------")
    logging.info(f"Input Parameters: {parameters}")
    logging.info("--------------------------------------------------------------------------------------\n\n")


    # Get HyperParameters from the input params dict.
    lr = float(parameters["lr"])
    num_epoch = int(parameters["num_epoch"])

    # Set dist parameters and strategy.
    is_dist = parameters["is_dist"]
    num_workers = parameters["num_workers"]
    batch_size_per_worker = 64
    batch_size_global = batch_size_per_worker * num_workers
    strategy = tf.distribute.MultiWorkerMirroredStrategy(
        communication_options=tf.distribute.experimental.CommunicationOptions(
            implementation=tf.distribute.experimental.CollectiveCommunication.RING
        )
    )

    # Callback class for logging training.
    # Katib parses metrics in this format: <metric-name>=<metric-value>.
    class CustomCallback(tf.keras.callbacks.Callback):
        def on_epoch_end(self, epoch, logs=None):
            logging.info(
                "Epoch {}/{}. accuracy={:.4f} - loss={:.4f}".format(
                    epoch+1, num_epoch, logs["accuracy"], logs["loss"]
                )
            )

    # Prepare MNIST Dataset.
    def mnist_dataset(batch_size):
        (x_train, y_train), _ = tf.keras.datasets.mnist.load_data()
        x_train = x_train / np.float32(255)
        y_train = y_train.astype(np.int64)
        train_dataset = (
            tf.data.Dataset.from_tensor_slices((x_train, y_train))
            .shuffle(60000)
            .repeat()
            .batch(batch_size)
        )
        return train_dataset

    # Build and compile CNN Model.
    def build_and_compile_cnn_model():
        model = tf.keras.Sequential(
            [
                tf.keras.layers.InputLayer(input_shape=(28, 28)),
                tf.keras.layers.Reshape(target_shape=(28, 28, 1)),
                tf.keras.layers.Conv2D(32, 3, activation="relu"),
                tf.keras.layers.Flatten(),
                tf.keras.layers.Dense(128, activation="relu"),
                tf.keras.layers.Dense(10),
            ]
        )
        model.compile(
            loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
            optimizer=tf.keras.optimizers.SGD(learning_rate=lr),
            metrics=["accuracy"],
        )
        return model
    
    # Download Dataset.
    dataset = mnist_dataset(batch_size_global)

    # For dist strategy we should build model under scope().
    if is_dist:
        logging.info("Running Distributed Training")
        logging.info("--------------------------------------------------------------------------------------\n\n")
        with strategy.scope():
            model = build_and_compile_cnn_model()
    else:
        logging.info("Running Single Worker Training")
        logging.info("--------------------------------------------------------------------------------------\n\n")
        model = build_and_compile_cnn_model()
    
    # Start Training.
    model.fit(
        dataset,
        epochs=num_epoch,
        steps_per_epoch=70,
        callbacks=[CustomCallback()],
        verbose=0,
    )

Run Training Locally in the Notebook

We are going to download MNIST Dataset and start local training.

Also, set Epochs = 2 to reduce training time and avoid CPU overload.

In [2]:

# Set Parameters for Local Training.
parameters = {
    "lr": "0.1",
    "num_epoch": "2",
    "is_dist": False,
    "num_workers": 1
}

# Train Model locally in the Notebook.
train_mnist_model(parameters)

2022-09-12T18:47:45Z INFO     --------------------------------------------------------------------------------------
2022-09-12T18:47:45Z INFO     Input Parameters: {'lr': '0.1', 'num_epoch': '2', 'is_dist': False, 'num_workers': 1}
2022-09-12T18:47:45Z INFO     --------------------------------------------------------------------------------------

INFO:tensorflow:Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.RING

2022-09-12T18:47:45Z INFO     Single-worker MultiWorkerMirroredStrategy with local_devices = ('/device:CPU:0',), communication = CommunicationImplementation.RING

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11490434/11490434 [==============================] - 1s 0us/step

2022-09-12T18:47:51Z INFO     Running Single Worker Training
2022-09-12T18:47:51Z INFO     --------------------------------------------------------------------------------------


2022-09-12T18:49:24Z INFO     Epoch 1/2. accuracy=0.7310 - loss=0.8630
2022-09-12T18:50:45Z INFO     Epoch 2/2. accuracy=0.8862 - loss=0.3731

Start Model Tuning with Katib

If you want to improve your model, you can run HyperParameter tuning with Katib.

The following example uses Covariance Matrix Adaptation Evolution Strategy (CMA-ES) algorithm to tune HyperParameters.

We are going to tune learning rate and number of epochs.

In [13]:

import kubeflow.katib as katib

# Set parameters with their distribution for HyperParameter Tuning with Katib.
parameters = {
    "lr": katib.search.double(min=0.1, max=0.2),
    "num_epoch": katib.search.int(min=10, max=15),
    "is_dist": False,
    "num_workers": 1
}

# Start the Katib Experiment.
exp_name = "tune-mnist"
katib_client = katib.KatibClient()

katib_client.tune(
    name=exp_name,
    objective=train_mnist_model, # Objective function.
    parameters=parameters, # HyperParameters to tune.
    algorithm_name="cmaes", # Alorithm to use.
    objective_metric_name="accuracy", # Katib is going to optimize "accuracy".
    additional_metric_names=["loss"], # Katib is going to collect these metrics in addition to the objective metric.
    max_trial_count=12, # Trial Threshold.
    parallel_trial_count=2,
)

Experiment tune-mnist has been created

Katib Experiment tune-mnist link here

Access to Katib UI

You can check created Experiment in the Katib UI.

Get the Best HyperParameters from the Katib Experiment

You can get the best HyperParameters from the most optimal Katib Trial.

In [14]:

status = katib_client.is_experiment_succeeded(exp_name)
print(f"Katib Experiment is Succeeded: {status}\n")

best_hps = katib_client.get_optimal_hyperparameters(exp_name)

if best_hps != None:
    print("Current Optimal Trial\n")
    print(best_hps)
    
    for hp in best_hps.parameter_assignments:
        if hp.name == "lr":
            best_lr = hp.value
        else:
            best_num_epoch = hp.value

Katib Experiment is Succeeded: True

Current Optimal Trial

{'best_trial_name': 'tune-mnist-l8xvxzvj',
 'observation': {'metrics': [{'latest': '0.9694',
                              'max': '0.9694',
                              'min': '0.7203',
                              'name': 'accuracy'},
                             {'latest': '0.1007',
                              'max': '0.9082',
                              'min': '0.1007',
                              'name': 'loss'}]},
 'parameter_assignments': [{'name': 'num_epoch', 'value': '13'},
                           {'name': 'lr', 'value': '0.16377224201308005'}]}

Start Distributive Training with TFJob

Use the best HyperParameters (learning rate and number of epochs) from the Katib Experiment and run the TFJob.

In [15]:

from kubeflow.training import TFJobClient

# Set Parameters for Distributed Training with TFJob.
parameters = {
    "lr": best_lr,
    "num_epoch": best_num_epoch,
    "is_dist": True,
    "num_workers": 5
}

# Start TFJob Training.
tfjob_name = "train-mnist"
tfjob_client = TFJobClient()

tfjob_client.create_tfjob_from_func(
    name=tfjob_name,
    func=train_mnist_model,
    parameters=parameters, # Input parameters for the train function.
    num_worker_replicas=5, # How many TFJob Workers will be run.
)

2022-09-12T19:08:51Z INFO     TFJob train-mnist has been created

Get TFJob Status and Training Logs

You can check the TFJob status and logs.

In [16]:

print(f"TFJob status: {tfjob_client.get_job_status(tfjob_name)}")

TFJob status: Succeeded

In [17]:

tfjob_client.get_logs(name=tfjob_name, master=True, follow=True)

2022-09-12T19:10:11Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:53Z INFO     --------------------------------------------------------------------------------------
2022-09-12T19:10:11Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:53Z INFO     Input Parameters: {'lr': '0.17016692449867332', 'num_epoch': '13', 'is_dist': True, 'num_workers': 5}
2022-09-12T19:10:11Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:53Z INFO     --------------------------------------------------------------------------------------
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: 2022-09-12 19:08:53.988515: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: 2022-09-12 19:08:54.008619: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:272] Initialize GrpcChannelCache for job worker -> {0 -> train-mnist-worker-0.kubeflow-andrey.svc:2222, 1 -> train-mnist-worker-1.kubeflow-andrey.svc:2222, 2 -> train-mnist-worker-2.kubeflow-andrey.svc:2222, 3 -> train-mnist-worker-3.kubeflow-andrey.svc:2222, 4 -> train-mnist-worker-4.kubeflow-andrey.svc:2222}
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: 2022-09-12 19:08:54.008700: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:272] Initialize GrpcChannelCache for job worker -> {0 -> train-mnist-worker-0.kubeflow-andrey.svc:2222, 1 -> train-mnist-worker-1.kubeflow-andrey.svc:2222, 2 -> train-mnist-worker-2.kubeflow-andrey.svc:2222, 3 -> train-mnist-worker-3.kubeflow-andrey.svc:2222, 4 -> train-mnist-worker-4.kubeflow-andrey.svc:2222}
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: 2022-09-12 19:08:54.009579: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:438] Started server with target: grpc://train-mnist-worker-0.kubeflow-andrey.svc:2222
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:4/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:0']
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:54Z INFO     Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:4/device:CPU:0', '/job:worker/replica:0/task:2/device:CPU:0', '/job:worker/replica:0/task:3/device:CPU:0', '/job:worker/replica:0/task:1/device:CPU:0']
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: INFO:tensorflow:Check health not enabled.
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:54Z INFO     Check health not enabled.
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['train-mnist-worker-0.kubeflow-andrey.svc:2222', 'train-mnist-worker-1.kubeflow-andrey.svc:2222', 'train-mnist-worker-2.kubeflow-andrey.svc:2222', 'train-mnist-worker-3.kubeflow-andrey.svc:2222', 'train-mnist-worker-4.kubeflow-andrey.svc:2222']}, task_type = 'worker', task_id = 0, num_workers = 5, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.RING
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:54Z INFO     MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['train-mnist-worker-0.kubeflow-andrey.svc:2222', 'train-mnist-worker-1.kubeflow-andrey.svc:2222', 'train-mnist-worker-2.kubeflow-andrey.svc:2222', 'train-mnist-worker-3.kubeflow-andrey.svc:2222', 'train-mnist-worker-4.kubeflow-andrey.svc:2222']}, task_type = 'worker', task_id = 0, num_workers = 5, local_devices = ('/job:worker/task:0/device:CPU:0',), communication = CommunicationImplementation.RING
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11490434/11490434 [==============================] - 0s 0us/step
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:56Z INFO     Running Distributed Training
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:56Z INFO     --------------------------------------------------------------------------------------
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: 2022-09-12 19:08:56.666389: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:776] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Found an unshardable source dataset: name: "TensorSliceDataset/_2"
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: op: "TensorSliceDataset"
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: input: "Placeholder/_0"
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: input: "Placeholder/_1"
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: attr {
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:   key: "Toutput_types"
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:   value {
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:     list {
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:       type: DT_FLOAT
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:       type: DT_INT64
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:     }
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:   }
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: }
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: attr {
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:   key: "_cardinality"
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:   value {
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:     i: 60000
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:   }
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: }
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: attr {
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:   key: "is_files"
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:   value {
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:     b: false
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:   }
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: }
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: attr {
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:   key: "metadata"
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:   value {
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:     s: "\n\024TensorSliceDataset:0"
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:   }
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: }
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]: attr {
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:   key: "output_shapes"
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:   value {
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:     list {
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:       shape {
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:         dim {
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:           size: 28
2022-09-12T19:10:12Z INFO     [Pod train-mnist-worker-0]:         }
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:         dim {
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:           size: 28
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:         }
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:       }
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:       shape {
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:       }
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:     }
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:   }
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: }
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: experimental_type {
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:   type_id: TFT_PRODUCT
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:   args {
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:     type_id: TFT_DATASET
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:     args {
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:       type_id: TFT_PRODUCT
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:       args {
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:         type_id: TFT_TENSOR
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:         args {
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:           type_id: TFT_FLOAT
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:         }
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:       }
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:       args {
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:         type_id: TFT_TENSOR
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:         args {
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:           type_id: TFT_INT64
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:         }
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:       }
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:     }
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]:   }
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: }
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: 2022-09-12 19:08:56.901683: W tensorflow/core/framework/dataset.cc:768] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: INFO:tensorflow:Collective all_reduce tensors: 6 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:57Z INFO     Collective all_reduce tensors: 6 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:57Z INFO     Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:57Z INFO     Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:57Z INFO     Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:57Z INFO     Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:57Z INFO     Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: INFO:tensorflow:Collective all_reduce tensors: 6 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:58Z INFO     Collective all_reduce tensors: 6 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:58Z INFO     Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:58Z INFO     Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:58Z INFO     Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: INFO:tensorflow:Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:13Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:08:58Z INFO     Collective all_reduce tensors: 1 all_reduces, num_devices = 1, group_size = 5, implementation = CommunicationImplementation.RING, num_packs = 1
2022-09-12T19:10:14Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:09:04Z INFO     Epoch 1/13. accuracy=0.7755 - loss=0.7565
2022-09-12T19:10:14Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:09:09Z INFO     Epoch 2/13. accuracy=0.9104 - loss=0.2964
2022-09-12T19:10:14Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:09:13Z INFO     Epoch 3/13. accuracy=0.9371 - loss=0.2100
2022-09-12T19:10:14Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:09:18Z INFO     Epoch 4/13. accuracy=0.9475 - loss=0.1756
2022-09-12T19:10:14Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:09:23Z INFO     Epoch 5/13. accuracy=0.9505 - loss=0.1612
2022-09-12T19:10:14Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:09:27Z INFO     Epoch 6/13. accuracy=0.9608 - loss=0.1309
2022-09-12T19:10:14Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:09:32Z INFO     Epoch 7/13. accuracy=0.9613 - loss=0.1298
2022-09-12T19:10:14Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:09:37Z INFO     Epoch 8/13. accuracy=0.9645 - loss=0.1165
2022-09-12T19:10:14Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:09:41Z INFO     Epoch 9/13. accuracy=0.9717 - loss=0.0962
2022-09-12T19:10:14Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:09:46Z INFO     Epoch 10/13. accuracy=0.9719 - loss=0.0920
2022-09-12T19:10:14Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:09:51Z INFO     Epoch 11/13. accuracy=0.9743 - loss=0.0873
2022-09-12T19:10:14Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:09:55Z INFO     Epoch 12/13. accuracy=0.9751 - loss=0.0831
2022-09-12T19:10:14Z INFO     [Pod train-mnist-worker-0]: 2022-09-12T19:10:00Z INFO     Epoch 13/13. accuracy=0.9765 - loss=0.0803

Delete Katib Experiment and TFJob

When jobs are finished, you can delete the resources.

In [18]:

katib_client.delete_experiment(exp_name)

Experiment tune-mnist has been deleted

In [19]:

tfjob_client.delete(tfjob_name)

2022-09-12T19:10:19Z INFO     TFJob train-mnist has been deleted