kserve/docs/samples/v1beta1/custom/torchserve/logger/logger_demo.ipynb

KfServing Logger demo

We create a message dumper Knative service to print out CloudEvents it receives:

In [1]:

!pygmentize message-dumper.yaml

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: message-dumper
spec:
  template:
    spec:
      containers:
      - image: gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display

In [53]:

!kubectl apply -f message-dumper.yaml -n default

service.serving.knative.dev/message-dumper created

Create a Torchserve model with associated logger to push events to the message logger URL

In [1]:

!pygmentize torchserve-logger.yaml

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "torchserve-custom"
spec:
  predictor:
    minReplicas: 1      
    logger:
      url: http://message-dumper.default.svc.cluster.local
      mode: all
    containers:
      - image: {username}/torchserve:latest
        name: torchserve-container

In [19]:

!kubectl apply -f torchserve-logger.yaml -n kfserving-test

inferenceservice.serving.kserve.io/torchserve-custom created

In [20]:

CLUSTER_IPS = !(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
CLUSTER_IP = CLUSTER_IPS[0]
print(CLUSTER_IP)

a881f5a8c676a41edbccdb0a394a80d6-2069247558.us-west-2.elb.amazonaws.com

In [21]:

SERVICE_HOSTNAMES = !(kubectl get inferenceservice torchserve-custom -n kfserving-test -o jsonpath='{.status.url}' | cut -d "/" -f 3)
SERVICE_HOSTNAME = SERVICE_HOSTNAMES[0]
print(SERVICE_HOSTNAME)

torchserve-custom.kfserving-test.a881f5a8c676a41edbccdb0a394a80d6-2069247558.us-west-2.elb.amazonaws.com

In [11]:

!wget https://raw.githubusercontent.com/pytorch/serve/master/examples/image_classifier/mnist/test_data/1.png

--2020-12-09 19:05:29--  https://raw.githubusercontent.com/pytorch/serve/master/examples/image_classifier/mnist/test_data/1.png
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 199.232.252.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|199.232.252.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 167 [image/png]
Saving to: ‘1.png.2’

1.png.2             100%[===================>]     167  --.-KB/s    in 0s      

2020-12-09 19:05:30 (4.25 MB/s) - ‘1.png.2’ saved [167/167]

In [19]:

!echo $SERVICE_HOSTNAME

torchserve-custom.kfserving-test.a881f5a8c676a41edbccdb0a394a80d6-2069247558.us-west-2.elb.amazonaws.com

In [22]:

!curl -v -H "Host: $SERVICE_HOSTNAME" http://$CLUSTER_IP/predictions/mnist -T 1.png

*   Trying 52.37.128.207...
* Connected to a881f5a8c676a41edbccdb0a394a80d6-2069247558.us-west-2.elb.amazonaws.com (52.37.128.207) port 80 (#0)








* We are completely uploaded and fine







* Connection #0 to host a881f5a8c676a41edbccdb0a394a80d6-2069247558.us-west-2.elb.amazonaws.com left intact
1

In [23]:

!kubectl logs $(kubectl get pod -l serving.knative.dev/configuration=message-dumper -n default -o jsonpath='{.items[0].metadata.name}') -c user-container

☁️  cloudevents.Event
Validation: valid
Context Attributes,
  specversion: 1.0
  type: org.kubeflow.serving.inference.request
  source: http://localhost:9081/
  id: 842e083d-ffb1-4f90-9649-14d291afa271
  time: 2020-12-09T13:41:29.604504965Z
  datacontenttype: application/json
Extensions,
  endpoint: 
  inferenceservicename: torchserve-custom
  namespace: kfserving-test
  traceparent: 00-c404939670665fd4630adc993d99fe43-a53901ad0192f760-00
Data,


IHDR����������Wf�H���nIDATx�c`�`��$�r�o�n�)���$��߻��e��q�'y��G?<�rpJ����[2�N��p����?�%=����-���Hj���_�l����%U�I
����)I-���J4ԩ�����IEND�B`�
☁️  cloudevents.Event
Validation: valid
Context Attributes,
  specversion: 1.0
  type: org.kubeflow.serving.inference.response
  source: http://localhost:9081/
  id: 842e083d-ffb1-4f90-9649-14d291afa271
  time: 2020-12-09T13:41:34.829648499Z
  datacontenttype: application/json
Extensions,
  endpoint: 
  inferenceservicename: torchserve-custom
  namespace: kfserving-test
  traceparent: 00-396af54bc5b3decdb7dc0afc27022b54-1f385ad3861c25b4-00
Data,
  1

In [18]:

!kubectl delete -f torchserve-logger.yaml -n kfserving-test

inferenceservice.serving.kserve.io "torchserve-custom" deleted

In [14]:

!kubectl delete -f message-dumper.yaml -n kfserving-test

service.serving.knative.dev "message-dumper" deleted

In [ ]: