kserve/docs/samples/client/kfserving_sdk_v1beta1_sample.ipynb

Sample for KFServing SDK v1beta1

This is a sample for KFServing SDK v1beta1.

The notebook shows how to use KFServing SDK to create, get and delete InferenceService.

In [11]:

from kubernetes import client
from kfserving import KFServingClient
from kfserving import constants
from kfserving import utils
from kfserving import V1beta1InferenceService
from kfserving import V1beta1InferenceServiceSpec
from kfserving import V1beta1PredictorSpec
from kfserving import V1beta1TFServingSpec

Define namespace where InferenceService needs to be deployed to. If not specified, below function defines namespace to the current one where SDK is running in the cluster, otherwise it will deploy to default namespace.

In [12]:

# namespace = utils.get_default_target_namespace()
namespace = "kfserving-test"

Define InferenceService

Firstly define default endpoint spec, and then define the inferenceservice basic on the endpoint spec.

In [13]:

api_version = constants.KFSERVING_GROUP + "/" + kfserving_version

isvc = V1beta1InferenceService(
    api_version=api_version,
    kind=constants.KFSERVING_KIND,
    metadata=client.V1ObjectMeta(name="flower-sample", namespace=namespace),
    spec=V1beta1InferenceServiceSpec(
        predictor=V1beta1PredictorSpec(
            tensorflow=(
                V1beta1TFServingSpec(
                    storage_uri="gs://kfserving-examples/models/tensorflow/flowers"
                )
            )
        )
    ),
)

Create InferenceService

Call KFServingClient to create InferenceService.

In [15]:

KFServing = KFServingClient()
KFServing.create(isvc)

Out [15]:

{'apiVersion': 'serving.kserve.io/v1beta1',
 'kind': 'InferenceService',
 'metadata': {'creationTimestamp': '2021-01-18T00:31:35Z',
  'generation': 1,
  'name': 'flower-sample',
  'namespace': 'kfserving-test',
  'resourceVersion': '283999021',
  'selfLink': '/apis/serving.kserve.io/v1beta1/namespaces/kfserving-test/inferenceservices/flower-sample',
  'uid': 'd074779a-e0d0-4612-b9c7-a7da69002683'},
 'spec': {'predictor': {'tensorflow': {'name': 'kfserving-container',
    'resources': {'limits': {'cpu': '1', 'memory': '2Gi'},
     'requests': {'cpu': '1', 'memory': '2Gi'}},
    'runtimeVersion': '1.14.0',
    'storageUri': 'gs://kfserving-examples/models/tensorflow/flowers'}}}}

Check the InferenceService

In [17]:

KFServing.get("flower-sample", namespace=namespace, watch=True, timeout_seconds=120)

NAME                 READY                           PREV                    LATEST URL                                                              
flower-sample        True                               0                       100 http://flower-sample.kfserving-test.example.com                  

Patch the InferenceService and define Canary Traffic Percent

In [18]:

isvc = V1beta1InferenceService(
    api_version=api_version,
    kind=constants.KFSERVING_KIND,
    metadata=client.V1ObjectMeta(name="flower-sample", namespace=namespace),
    spec=V1beta1InferenceServiceSpec(
        predictor=V1beta1PredictorSpec(
            canary_traffic_percent=20,
            tensorflow=(
                V1beta1TFServingSpec(
                    storage_uri="gs://kfserving-examples/models/tensorflow/flowers-2"
                )
            ),
        )
    ),
)

KFServing.patch("flower-sample", isvc, namespace=namespace)

Out [18]:

{'apiVersion': 'serving.kserve.io/v1beta1',
 'kind': 'InferenceService',
 'metadata': {'creationTimestamp': '2021-01-18T00:31:35Z',
  'finalizers': ['inferenceservice.finalizers'],
  'generation': 2,
  'name': 'flower-sample',
  'namespace': 'kfserving-test',
  'resourceVersion': '283999615',
  'selfLink': '/apis/serving.kserve.io/v1beta1/namespaces/kfserving-test/inferenceservices/flower-sample',
  'uid': 'd074779a-e0d0-4612-b9c7-a7da69002683'},
 'spec': {'predictor': {'canaryTrafficPercent': 20,
   'tensorflow': {'name': 'kfserving-container',
    'resources': {'limits': {'cpu': '1', 'memory': '2Gi'},
     'requests': {'cpu': '1', 'memory': '2Gi'}},
    'runtimeVersion': '1.14.0',
    'storageUri': 'gs://kfserving-examples/models/tensorflow/flowers-2'}}},
 'status': {'address': {'url': 'http://flower-sample.kfserving-test.svc.cluster.local/v1/models/flower-sample:predict'},
  'components': {'predictor': {'address': {'url': 'http://flower-sample-predictor-default.kfserving-test.svc.cluster.local'},
    'latestCreatedRevision': 'flower-sample-predictor-default-fg4d6',
    'latestReadyRevision': 'flower-sample-predictor-default-fg4d6',
    'latestRolledoutRevision': 'flower-sample-predictor-default-fg4d6',
    'traffic': [{'latestRevision': True,
      'percent': 100,
      'revisionName': 'flower-sample-predictor-default-fg4d6',
      'tag': 'latest',
      'url': 'http://latest-flower-sample-predictor-default.kfserving-test.example.com'}],
    'url': 'http://flower-sample-predictor-default.kfserving-test.example.com'}},
  'conditions': [{'lastTransitionTime': '2021-01-18T00:31:55Z',
    'status': 'True',
    'type': 'IngressReady'},
   {'lastTransitionTime': '2021-01-18T00:31:55Z',
    'severity': 'Info',
    'status': 'True',
    'type': 'PredictorConfigurationReady'},
   {'lastTransitionTime': '2021-01-18T00:31:55Z',
    'status': 'True',
    'type': 'PredictorReady'},
   {'lastTransitionTime': '2021-01-18T00:31:51Z',
    'severity': 'Info',
    'status': 'True',
    'type': 'PredictorRouteReady'},
   {'lastTransitionTime': '2021-01-18T00:31:55Z',
    'status': 'True',
    'type': 'Ready'}],
  'url': 'http://flower-sample.kfserving-test.example.com'}}

Check the InferenceService after Patching

In [19]:

KFServing.wait_isvc_ready("flower-sample", namespace=namespace)

In [21]:

KFServing.get("flower-sample", namespace=namespace, watch=True)

NAME                 READY                           PREV                    LATEST URL                                                              
flower-sample        True                              80                        20 http://flower-sample.kfserving-test.example.com                  

Delete the InferenceService

In [10]:

KFServing.delete("flower-sample", namespace=namespace)

Out [10]:

{'apiVersion': 'serving.kserve.io/v1alpha2',
 'kind': 'InferenceService',
 'metadata': {'creationTimestamp': '2021-01-18T00:21:55Z',
  'deletionGracePeriodSeconds': 0,
  'deletionTimestamp': '2021-01-18T00:24:37Z',
  'finalizers': ['inferenceservice.finalizers'],
  'generation': 3,
  'name': 'flower-sample',
  'namespace': 'kfserving-test',
  'resourceVersion': '283995283',
  'selfLink': '/apis/serving.kserve.io/v1alpha2/namespaces/kfserving-test/inferenceservices/flower-sample',
  'uid': 'd2b1aeb1-8029-41fc-a614-1ed65949a797'},
 'spec': {'canaryTrafficPercent': 20,
  'default': {'predictor': {'tensorflow': {'resources': {'limits': {'cpu': '1',
       'memory': '2Gi'},
      'requests': {'cpu': '1', 'memory': '2Gi'}},
     'runtimeVersion': '1.14.0',
     'storageUri': 'gs://kfserving-examples/models/tensorflow/flowers-2'}}}},
 'status': {}}

In [ ]: