kserve/docs/samples/performance-testing/README.md

HTTP Performance test

This tutorial demonstrates performance testing for KServe inference services using HTTP. We will use Iter8 for generating load and validating service-level objectives (SLOs) for the inference service. Iter8 is an open-source Kubernetes release optimizer that makes it easy to ensure that your ML models perform well and maximize business value.

---

Performance testing of KServe inference services using gRPC is described here. Canary testing using Prometheus metrics is described here. This tutorial focuses on performance testing of KServe inference services with HTTP endpoints. The main steps in this tutorial are:

1. Deploy an InferenceService
2. Launch an Iter8 experiment
3. View experiment report

---

Deploy an InferenceService

Create an InferenceService that exposes an HTTP port. The following serves the Scikit-learn irisv2 model:

cat <<EOF | kubectl apply -f -
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "sklearn-irisv2"
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      runtime: kserve-mlserver
      storageUri: "gs://seldon-models/sklearn/mms/lr_model"
EOF

Verify that your inference service is ready.

kubectl wait --for=condition=Ready --timeout=600s isvc/sklearn-irisv2

Install Iter8 CLI

Install the Iter8 CLI using brew as follows. You can also install using pre-built binaries as described here.

brew tap iter8-tools/iter8
brew install [email protected]

Launch an Iter8 experiment

Iter8 introduces the notion of an experiment that makes it easy to verify that your inference service is ready, generate load for the inference, collect latency and error-related metrics, and assess SLOs for performance validation. Launch the Iter8 experiment inside the Kubernetes cluster.

iter8 k launch \
--set "tasks={ready,http,assess}" \
--set ready.isvc=sklearn-irisv2 \
--set ready.timeout=600s \
--set http.url=http://sklearn-irisv2.default.svc.cluster.local/v2/models/sklearn-irisv2/infer \
--set http.payloadURL=https://gist.githubusercontent.com/kalantar/d2dd03e8ebff2c57c3cfa992b44a54ad/raw/97a0480d0dfb1deef56af73a0dd31c80dc9b71f4/sklearn-irisv2-input.json \
--set http.contentType="application/json" \
--set http.warmupNumRequests=50 \
--set assess.SLOs.upper.http/latency-mean=500 \
--set assess.SLOs.upper.http/latency-p90=1000 \
--set assess.SLOs.upper.http/error-count=0 \
--set runner=job

More about this Iter8 experiment

1. This experiment consists of three tasks, namely, ready, http, and assess.

   • The ready task checks if the sklearn-irisv2 inference service exists and is ready to serve user requests.

   • The http task sends requests to the cluster-local HTTP service whose URL is http://sklearn-irisv2.default.svc.cluster.local/v2/models/sklearn-irisv2/infer, and collects Iter8's built-in HTTP load test metrics. As part of these requests, it uses the JSON data at this payloadURL as the payload.

   • The assess task verifies if the app satisfies the specified SLOs: i) the mean latency of the service does not exceed 500 msec, ii) the 90th percentile latency of the service does not exceed 1000 msec, and iii) there are no errors (4xx or 5xx response codes) in the responses.

2. This is a single-loop in which all the previously mentioned tasks will run once and the experiment will finish.

View experiment report

iter8 k report -o html > report.html # view in a browser

Below is a sample HTML report.

You can also assert various conditions about the outcome of the experiment and view the execution logs for the experiment.

Cleanup

Delete the Iter8 experiment and KServe inference service.

iter8 k delete
kubectl delete isvc sklearn-irisv2

---

This tutorial just scratches the surface of Iter8 experimentation capabilities. For more features (for example, automatically sending a notification to Slack or GitHub with experiment results), please see Iter8 documentation.