mirror of https://github.com/hanzo-ml/kserve.git synced 2026-10-30 20:53:49 +00:00

Files

T

1375a961ca Fix spelling (#1970 )

* annotations
* appended
* arbitrary
* architecture
* archive
* associated
* builtin
* character
* classification
* container
* control
* credentials
* downloader
* egress
* entities
* explain
* explanation
* forwarding
* grafana
* heuristic
* huggingface
* identified
* image
* include
* instances
* interpreting
* knative
* kubernetes
* licenses
* lightgbm
* module
* overridden
* patched
* pmmlserver
* prediction
* predictions
* prerequisites
* previous
* provides
* queries
* reconcile
* request
* resource
* specified
* storage
* tabular
* tokenization
* transformer
* transition
* true
* unknown
* variabilize
* workflows
* wrapper

Signed-off-by: Josh Soref <[email protected]>

Co-authored-by: Josh Soref <[email protected]>

2022-01-02 06:53:47 -08:00

8.1 KiB

Raw Permalink Blame History

Benchmark of Multi-Model Serving

The following stress and performance tests have been executed to support that Multi-Model Serving is a solution to the model deployment scalability problem as discussed in the MMS Guide.

Stress Test

To determine the maximum amount of models that can be deployed

Environment Setup:
- kfserving: v0.5.1
- knative: v0.17.0
- istio: v1.7.x
Cluster limitation:
- Maximum 2048 IP Addresses
- Maximum 0.55 TiB memory per worker node
- Maximum ~2000 kubernetes services resources
  - Limited by istio ingress gateway
- Maximum 660 pods on worker nodes
  - Maximum 110 pods per worker node
  - 6 total worker nodes
- ETCD size of 65 MiB
Inference Service:
- apiVersion: serving.kserve.io/v1beta1
- concurrency: 1
- minReplicas: 1
- Triton model server
- Queue_proxy: CPU 1, Memory 500Mi
- Traditional Model Serving Predictor Spec: CPU 100m, Memory 100Mi
- Multi-Model Serving Predictor Spec: CPU 10, Memory 10Gi
Model:
- Simple string
- Size of 700B
- Tensorflow framework

Baseline:

Test Summary:
- The traditional model serving was able to deploy 352 InferenceServices
- Multi-model serving was able to deploy ~10K models using 6 InferenceService which equates to 2800% more models loaded
- Overall, Multi-Model Serving utilises less CPU and memory while deploying more
  - The traditional model serving used a total of 387 CPU and 211 GB
  - Multi-model serving used a total of 66 CPU and 63 GB
  - Comparing the queue-proxy usage
    - The traditional model serving queue-proxies used 352 CPU and 176 GB
    - Multi-model serving queue-proxies used 6 CPU and 3 GB
- Adding/removing models in multi-model serving causes new configmap versions to be generated without ETCD getting the chance to remove the previous versions. This can exhaust the memory of ETCD in case of deploying thousands of models
  - Should consider an external storage to store models

Performance Test

To compare the latency between traditional model serving and multi-model serving

Environment Setup:
- kfserving: v0.5.1
- knative: v0.17.0
- istio: v1.7.x
Inference Service:
- apiVersion: serving.kserve.io/v1beta1
- concurrency: 1
- minReplicas: 1
- Triton model server
- Queue_proxy: 1 CPU and 500Mi
- Simple String:
  - Traditional Model Serving Predictor Spec: CPU 1, Memory 16GB, GPU 1
  - Multi-Model Serving Predictor Spec: CPU 1, Memory 16GB, GPU 1
- Bert:
  - Traditional Model Serving Predictor Spec: CPU 1, Memory 16GB, GPU 1
  - Multi-Model Serving Predictor Spec: CPU 1, Memory 16GB, GPU 1
Vegeta loading test agent:
- Simple String (CPU):
  - model size of 700B
  - testing duration 1 min
  - Queries per second (QPS): 5/sec, 10/sec, 20/sec, 30/sec, 40/sec, 50/sec, 100/sec
- Bert (GPU):
  - testing duration 1 min
  - Queries per second (QPS): 5/sec, 10/sec, 20/sec, 30/sec, 40/sec

Simple String test (CPU)

Test Summary:
- The traditional model serving using 5 inferenceService is able to handle up to 2500 queries per second without scaling
- Multi-model serving with 1 inferenceservice and 5 trainedModels starts to scale up at 500 queries per second
- The relative latency between traditional and multi-model serving seems comparable when not scaling up
- Multi-model serving seems to be more sensitive to autoscaling

Test Results:

Traditional model serving with 5 inferenceServices

Client side QPS	QPS per InferenceService (model)	mean	p50	p90	p95	p99	max
25	5	3.763	3.669	4.089	4.222	4.511	43.877
50	10	3.78	3.546	3.936	4.097	6.956	220.716
100	20	3.594	3.496	3.875	4.008	4.438	84.529
150	30	3.59	3.294	3.642	3.744	6.191	260.923
200	40	3.508	3.169	3.535	3.631	5.659	205.389
250	50	3.882	3.288	3.559	3.642	9.411	302.492
500	100	4.282	3.086	3.471	3.604	52.807	324.815
1000	200	6.266	3.062	3.436	6.7	116.774	308.413
1500	300	10.365	2.828	5.901	55.38	183.41	1039
2000	400	9.338	2.783	4.288	32.912	178.837	1058
2500	500	11.796	2.719	8.509	64.447	217.911	1043

Multi-model serving with 1 inferenceService and 5 TrainedModels

Client side QPS	QPS per InferenceService	QPS per TrainedModel	mean	p50	p90	p95	p99	max
25	25	5	4.453	3.85	4.181	4.282	26.465	138.671
50	50	10	4.081	3.676	3.934	4.048	11.653	107.775
100	100	20	5.051	3.304	3.625	3.772	72.291	193.219
150	150	30	4.403	3.399	3.684	3.823	46.981	119.896
200	200	40	6.071	3.296	3.63	8.587	90.514	203.785
250	250	50	6.846	3.223	3.575	23.825	100.167	351.26
500	500	100	260.202	2.846	1298	2025	2241	2475

At 500 QPS the multi-model serving starts to scale up which causes the latency to dramatically increase. The benchmark for this case will stop at 500 QPS because 500+ QPS will lead to pods scaling which is not what the focus should be on.

Bert test (GPU)

Test Summary:
- The traditional model serving using 5 inferenceService is able to handle up to 200 queries per second without scaling
- Multi-model serving with 1 inferenceservice and 5 trainedModels is able to handle up to 40 queries per second without scaling
- The latency is relatively the same for traditional and multi-model serving
- Multi-model serving seems to be more sensitive to autoscaling