kserve/docs/samples/v1beta1/onnx

Deploy InferenceService with ONNX model

Setup

1. Your ~/.kube/config should point to a cluster with KServe installed.
2. Your cluster's Istio Ingress gateway must be network accessible.

Create the InferenceService

Apply the CRD

kubectl apply -f onnx.yaml 

Expected Output

$ inferenceservice.serving.kserve.io/style-sample configured

Run a sample inference

1. Setup env vars The first step is to determine the ingress IP and ports and set INGRESS_HOST and INGRESS_PORT

export MODEL_NAME=onnx-model
export ISVC_NAME=style-sample
export SERVICE_HOSTNAME=$(kubectl get inferenceservice ${ISVC_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3)

2. Verify the service is healthy

curl -v -H "Host:${SERVICE_HOSTNAME}" http://localhost:8080/v2/models/${MODEL_NAME}

3. Install dependencies

pip install -r requirements.txt

4. Run the sample notebook in jupyter

jupyter notebook

Uploading your own model

The sample model for the example in this readme is already uploaded and available for use. However if you would like to modify the example to use your own ONNX model, all you need to do is to upload your model as model.onnx to S3, GCS or an Azure Blob.