hanzoai/ROCm
2
0
Fork 0
mirror of https://github.com/hanzoai/ROCm.git synced 2026-09-30 20:37:25 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update vLLM benchmarking guide (#4347)

Browse Source 
* update vllm-benchmark

fix hlist overflow

update standalone benchmarking options

update list of models

fix typo and model name

unnecessary duplicate info

update formatting

update vllm benchmark guide

- remove Llama 2 FP8
- add Jais 13B
- update commands

update docker pull tag

update MAD available models

remove extra mad models not relevant to vllm

update PyTorch version

add changelog

add model names to .wordlist.txt

* Update docs/how-to/rocm-for-ai/inference/vllm-benchmark.rst

Co-authored-by: Pratik Basyal <[email protected]>

* Update docs/how-to/rocm-for-ai/inference/vllm-benchmark.rst

Co-authored-by: Pratik Basyal <[email protected]>

* Update docs/how-to/rocm-for-ai/inference/vllm-benchmark.rst

Co-authored-by: Pratik Basyal <[email protected]>

* fix typo

* update link

* fix link text

* change changelog to previous versions

* fix typo

* remove "for"

---------

Co-authored-by: Pratik Basyal <[email protected]>
This commit is contained in:
Peter Park
2025-02-05 17:18:35 -05:00
committed by GitHub
co-authored by Pratik Basyal
parent 9fce0757d3
commit 54e6d4d689
2 changed files with 177 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.wordlist.txt
+3
View File
@@ -74,6 +74,7 @@ Conda
ConnectX ConnectX
CuPy CuPy
Dashboarding Dashboarding
DBRX
DDR DDR
DF DF
DGEMM DGEMM
@@ -92,6 +93,7 @@ DataFrame
DataLoader DataLoader
DataParallel DataParallel
Debian Debian
DeepSeek
DeepSpeed DeepSpeed
Dependabot Dependabot
Deprecations Deprecations
@@ -129,6 +131,7 @@ GDS
GEMM GEMM
GEMMs GEMMs
GFortran GFortran
Gemma
GiB GiB
GIM GIM
GL GL
docs/how-to/rocm-for-ai/inference/vllm-benchmark.rst
+174 -103
View File
@@ -10,49 +10,22 @@ LLM inference performance validation on AMD Instinct MI300X
.. _vllm-benchmark-unified-docker: .. _vllm-benchmark-unified-docker:
The `ROCm vLLM Docker <https://hub.docker.com/r/rocm/vllm/tags>`_ image offers The `ROCm vLLM Docker <https://hub.docker.com/r/rocm/vllm/tags>`_ image offers
a prebuilt, optimized environment designed for validating large language model a prebuilt, optimized environment for validating large language model (LLM)
(LLM) inference performance on the AMD Instinct™ MI300X accelerator. This inference performance on the AMD Instinct™ MI300X accelerator. This ROCm vLLM
ROCm vLLM Docker image integrates vLLM and PyTorch tailored specifically for the Docker image integrates vLLM and PyTorch tailored specifically for the MI300X
MI300X accelerator and includes the following components: accelerator and includes the following components:
* `ROCm 6.2.1 <https://github.com/ROCm/ROCm>`_ * `ROCm 6.3.1 <https://github.com/ROCm/ROCm>`_
* `vLLM 0.6.4 <https://docs.vllm.ai/en/latest>`_ * `vLLM 0.6.6 <https://docs.vllm.ai/en/latest>`_
* `PyTorch 2.5.0 <https://github.com/pytorch/pytorch>`_ * `PyTorch 2.7.0 (2.7.0a0+git3a58512) <https://github.com/pytorch/pytorch>`_
* Tuning files (in CSV format)
With this Docker image, you can quickly validate the expected inference With this Docker image, you can quickly validate the expected inference
performance numbers on the MI300X accelerator. This topic also provides tips on performance numbers for the MI300X accelerator. This topic also provides tips on
optimizing performance with popular AI models. optimizing performance with popular AI models. For more information, see the lists of
:ref:`available models for MAD-integrated benchmarking <vllm-benchmark-mad-models>`
.. hlist:: and :ref:`standalone benchmarking <vllm-benchmark-standalone-options>`.
:columns: 6
* Llama 3.1 8B
* Llama 3.1 70B
* Llama 3.1 405B
* Llama 2 7B
* Llama 2 70B
* Mixtral 8x7B
* Mixtral 8x22B
* Mixtral 7B
* Qwen2 7B
* Qwen2 72B
* JAIS 13B
* JAIS 30B
.. _vllm-benchmark-vllm: .. _vllm-benchmark-vllm:
@@ -91,9 +64,9 @@ MI300X accelerator with the prebuilt vLLM Docker image.
.. code-block:: shell .. code-block:: shell
docker pull rocm/vllm:rocm6.2_mi300_ubuntu20.04_py3.9_vllm_0.6.4 docker pull rocm/vllm:rocm6.3.1_mi300_ubuntu22.04_py3.12_vllm_0.6.6
Once setup is complete, you can choose between two options to reproduce the Once the setup is complete, choose between two options to reproduce the
benchmark results: benchmark results:
- :ref:`MAD-integrated benchmarking <vllm-benchmark-mad>` - :ref:`MAD-integrated benchmarking <vllm-benchmark-mad>`
@@ -130,45 +103,89 @@ Although the following models are preconfigured to collect latency and
throughput performance data, you can also change the benchmarking parameters. throughput performance data, you can also change the benchmarking parameters.
Refer to the :ref:`Standalone benchmarking <vllm-benchmark-standalone>` section. Refer to the :ref:`Standalone benchmarking <vllm-benchmark-standalone>` section.
.. _vllm-benchmark-mad-models:
Available models Available models
---------------- ----------------
.. hlist:: .. list-table::
:columns: 3 :header-rows: 1
:widths: 2, 3
* ``pyt_vllm_llama-3.1-8b`` * - Model name
- Tag
* ``pyt_vllm_llama-3.1-70b`` * - `Llama 3.1 8B <https://huggingface.co/meta-llama/Llama-3.1-8B>`_
- ``pyt_vllm_llama-3.1-8b``
* ``pyt_vllm_llama-3.1-405b`` * - `Llama 3.1 70B <https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct>`_
- ``pyt_vllm_llama-3.1-70b``
* ``pyt_vllm_llama-2-7b`` * - `Llama 3.1 405B <https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct>`_
- ``pyt_vllm_llama-3.1-405b``
* ``pyt_vllm_llama-2-70b`` * - `Llama 3.2 11B Vision <https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct>`_
- ``pyt_vllm_llama-3.2-11b-vision-instruct``
* ``pyt_vllm_mixtral-8x7b`` * - `Llama 2 7B <https://huggingface.co/meta-llama/Llama-2-7b-chat-hf>`_
- ``pyt_vllm_llama-2-7b``
* ``pyt_vllm_mixtral-8x22b`` * - `Llama 2 70B <https://huggingface.co/meta-llama/Llama-2-70b-chat-hf>`_
- ``pyt_vllm_llama-2-70b``
* ``pyt_vllm_mistral-7b`` * - `Mixtral MoE 8x7B <https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1>`_
- ``pyt_vllm_mixtral-8x7b``
* ``pyt_vllm_qwen2-7b`` * - `Mixtral MoE 8x22B <https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1>`_
- ``pyt_vllm_mixtral-8x22b``
* ``pyt_vllm_qwen2-72b`` * - `Mistral 7B <https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3>`_
- ``pyt_vllm_mistral-7b``
* ``pyt_vllm_jais-13b`` * - `Qwen2 7B <https://huggingface.co/Qwen/Qwen2-7B-Instruct>`_
- ``pyt_vllm_qwen2-7b``
* ``pyt_vllm_jais-30b`` * - `Qwen2 72B <https://huggingface.co/Qwen/Qwen2-72B-Instruct>`_
- ``pyt_vllm_qwen2-72b``
* ``pyt_vllm_llama-3.1-8b_fp8`` * - `JAIS 13B <https://huggingface.co/core42/jais-13b-chat>`_
- ``pyt_vllm_jais-13b``
* ``pyt_vllm_llama-3.1-70b_fp8`` * - `JAIS 30B <https://huggingface.co/core42/jais-30b-chat-v3>`_
- ``pyt_vllm_jais-30b``
* ``pyt_vllm_llama-3.1-405b_fp8`` * - `DBRX Instruct <https://huggingface.co/databricks/dbrx-instruct>`_
- ``pyt_vllm_dbrx-instruct``
* ``pyt_vllm_mixtral-8x7b_fp8`` * - `Gemma 2 27B <https://huggingface.co/google/gemma-2-27b>`_
- ``pyt_vllm_gemma-2-27b``
* ``pyt_vllm_mixtral-8x22b_fp8`` * - `C4AI Command R+ 08-2024 <https://huggingface.co/CohereForAI/c4ai-command-r-plus-08-2024>`_
- ``pyt_vllm_c4ai-command-r-plus-08-2024``
* - `DeepSeek MoE 16B <https://huggingface.co/deepseek-ai/deepseek-moe-16b-chat>`_
- ``pyt_vllm_deepseek-moe-16b-chat``
* - `Llama 3.1 70B FP8 <https://huggingface.co/amd/Llama-3.1-70B-Instruct-FP8-KV>`_
- ``pyt_vllm_llama-3.1-70b_fp8``
* - `Llama 3.1 405B FP8 <https://huggingface.co/amd/Llama-3.1-405B-Instruct-FP8-KV>`_
- ``pyt_vllm_llama-3.1-405b_fp8``
* - `Mixtral MoE 8x7B FP8 <https://huggingface.co/amd/Mixtral-8x7B-Instruct-v0.1-FP8-KV>`_
- ``pyt_vllm_mixtral-8x7b_fp8``
* - `Mixtral MoE 8x22B FP8 <https://huggingface.co/amd/Mixtral-8x22B-Instruct-v0.1-FP8-KV>`_
- ``pyt_vllm_mixtral-8x22b_fp8``
* - `Mistral 7B FP8 <https://huggingface.co/amd/Mistral-7B-v0.1-FP8-KV>`_
- ``pyt_vllm_mistral-7b_fp8``
* - `DBRX Instruct FP8 <https://huggingface.co/amd/dbrx-instruct-FP8-KV>`_
- ``pyt_vllm_dbrx_fp8``
* - `C4AI Command R+ 08-2024 FP8 <https://huggingface.co/amd/c4ai-command-r-plus-FP8-KV>`_
- ``pyt_vllm_command-r-plus_fp8``
.. _vllm-benchmark-standalone: .. _vllm-benchmark-standalone:
@@ -181,8 +198,8 @@ snippet.
.. code-block:: .. code-block::
docker pull rocm/vllm:rocm6.2_mi300_ubuntu20.04_py3.9_vllm_0.6.4 docker pull rocm/vllm:rocm6.3.1_mi300_ubuntu22.04_py3.12_vllm_0.6.6
docker run -it --device=/dev/kfd --device=/dev/dri --group-add video --shm-size 128G --security-opt seccomp=unconfined --security-opt apparmor=unconfined --cap-add=SYS_PTRACE -v $(pwd):/workspace --env HUGGINGFACE_HUB_CACHE=/workspace --name vllm_v0.6.4 rocm/vllm:rocm6.2_mi300_ubuntu20.04_py3.9_vllm_0.6.4 docker run -it --device=/dev/kfd --device=/dev/dri --group-add video --shm-size 16G --security-opt seccomp=unconfined --security-opt apparmor=unconfined --cap-add=SYS_PTRACE -v $(pwd):/workspace --env HUGGINGFACE_HUB_CACHE=/workspace --name vllm_v0.6.6 rocm/vllm:rocm6.3.1_mi300_ubuntu22.04_py3.12_vllm_0.6.6
In the Docker container, clone the ROCm MAD repository and navigate to the In the Docker container, clone the ROCm MAD repository and navigate to the
benchmark scripts directory at ``~/MAD/scripts/vllm``. benchmark scripts directory at ``~/MAD/scripts/vllm``.
@@ -224,8 +241,8 @@ See the :ref:`examples <vllm-benchmark-run-benchmark>` for more information.
.. _vllm-benchmark-standalone-options: .. _vllm-benchmark-standalone-options:
Options Options and available models
------- ----------------------------
.. list-table:: .. list-table::
:header-rows: 1 :header-rows: 1
@@ -248,72 +265,100 @@ Options
- Measure both throughput and latency - Measure both throughput and latency
* - ``$model_repo`` * - ``$model_repo``
- ``meta-llama/Meta-Llama-3.1-8B-Instruct`` - ``meta-llama/Llama-3.1-8B-Instruct``
- Llama 3.1 8B - `Llama 3.1 8B <https://huggingface.co/meta-llama/Llama-3.1-8B>`_
* - (``float16``) * - (``float16``)
- ``meta-llama/Meta-Llama-3.1-70B-Instruct`` - ``meta-llama/Llama-3.1-70B-Instruct``
- Llama 3.1 70B - `Llama 3.1 70B <https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct>`_
* - * -
- ``meta-llama/Meta-Llama-3.1-405B-Instruct`` - ``meta-llama/Llama-3.1-405B-Instruct``
- Llama 3.1 405B - `Llama 3.1 405B <https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct>`_
* -
- ``meta-llama/Llama-3.2-11B-Vision-Instruct``
- `Llama 3.2 11B Vision <https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct>`_
* - * -
- ``meta-llama/Llama-2-7b-chat-hf`` - ``meta-llama/Llama-2-7b-chat-hf``
- Llama 2 7B - `Llama 2 7B <https://huggingface.co/meta-llama/Llama-2-7b-chat-hf>`_
* - * -
- ``meta-llama/Llama-2-70b-chat-hf`` - ``meta-llama/Llama-2-70b-chat-hf``
- Llama 2 70B - `Llama 2 7B <https://huggingface.co/meta-llama/Llama-2-70b-chat-hf>`_
* - * -
- ``mistralai/Mixtral-8x7B-Instruct-v0.1`` - ``mistralai/Mixtral-8x7B-Instruct-v0.1``
- Mixtral 8x7B - `Mixtral MoE 8x7B <https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1>`_
* - * -
- ``mistralai/Mixtral-8x22B-Instruct-v0.1`` - ``mistralai/Mixtral-8x22B-Instruct-v0.1``
- Mixtral 8x22B - `Mixtral MoE 8x22B <https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1>`_
* - * -
- ``mistralai/Mistral-7B-Instruct-v0.3`` - ``mistralai/Mistral-7B-Instruct-v0.3``
- Mixtral 7B - `Mistral 7B <https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3>`_
* - * -
- ``Qwen/Qwen2-7B-Instruct`` - ``Qwen/Qwen2-7B-Instruct``
- Qwen2 7B - `Qwen2 7B <https://huggingface.co/Qwen/Qwen2-7B-Instruct>`_
* - * -
- ``Qwen/Qwen2-72B-Instruct`` - ``Qwen/Qwen2-72B-Instruct``
- Qwen2 72B - `Qwen2 72B <https://huggingface.co/Qwen/Qwen2-72B-Instruct>`_
* - * -
- ``core42/jais-13b-chat`` - ``core42/jais-13b-chat``
- JAIS 13B - `JAIS 13B <https://huggingface.co/core42/jais-13b-chat>`_
* - * -
- ``core42/jais-30b-chat-v3`` - ``core42/jais-30b-chat-v3``
- JAIS 30B - `JAIS 30B <https://huggingface.co/core42/jais-30b-chat-v3>`_
* - ``$model_repo``
- ``amd/Meta-Llama-3.1-8B-Instruct-FP8-KV``
- Llama 3.1 8B
* - (``float8``)
- ``amd/Meta-Llama-3.1-70B-Instruct-FP8-KV``
- Llama 3.1 70B
* - * -
- ``amd/Meta-Llama-3.1-405B-Instruct-FP8-KV`` - ``databricks/dbrx-instruct``
- Llama 3.1 405B - `DBRX Instruct <https://huggingface.co/databricks/dbrx-instruct>`_
* -
- ``google/gemma-2-27b``
- `Gemma 2 27B <https://huggingface.co/google/gemma-2-27b>`_
* -
- ``CohereForAI/c4ai-command-r-plus-08-2024``
- `C4AI Command R+ 08-2024 <https://huggingface.co/CohereForAI/c4ai-command-r-plus-08-2024>`_
* -
- ``deepseek-ai/deepseek-moe-16b-chat``
- `DeepSeek MoE 16B <https://huggingface.co/deepseek-ai/deepseek-moe-16b-chat>`_
* - ``$model_repo``
- ``amd/Llama-3.1-70B-Instruct-FP8-KV``
- `Llama 3.1 70B FP8 <https://huggingface.co/amd/Llama-3.1-70B-Instruct-FP8-KV>`_
* - (``float8``)
- ``amd/Llama-3.1-405B-Instruct-FP8-KV``
- `Llama 3.1 405B FP8 <https://huggingface.co/amd/Llama-3.1-405B-Instruct-FP8-KV>`_
* - * -
- ``amd/Mixtral-8x7B-Instruct-v0.1-FP8-KV`` - ``amd/Mixtral-8x7B-Instruct-v0.1-FP8-KV``
- Mixtral 8x7B - `Mixtral MoE 8x7B FP8 <https://huggingface.co/amd/Mixtral-8x7B-Instruct-v0.1-FP8-KV>`_
* - * -
- ``amd/Mixtral-8x22B-Instruct-v0.1-FP8-KV`` - ``amd/Mixtral-8x22B-Instruct-v0.1-FP8-KV``
- Mixtral 8x22B - `Mixtral MoE 8x22B FP8 <https://huggingface.co/amd/Mixtral-8x22B-Instruct-v0.1-FP8-KV>`_
* -
- ``amd/Mistral-7B-v0.1-FP8-KV``
- `Mistral 7B FP8 <https://huggingface.co/amd/Mistral-7B-v0.1-FP8-KV>`_
* -
- ``amd/dbrx-instruct-FP8-KV``
- `DBRX Instruct FP8 <https://huggingface.co/amd/dbrx-instruct-FP8-KV>`_
* -
- ``amd/c4ai-command-r-plus-FP8-KV``
- `C4AI Command R+ 08-2024 FP8 <https://huggingface.co/amd/c4ai-command-r-plus-FP8-KV>`_
* - ``$num_gpu`` * - ``$num_gpu``
- 1 or 8 - 1 or 8
@@ -335,34 +380,34 @@ options and their descriptions.
Example 1: latency benchmark Example 1: latency benchmark
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Use this command to benchmark the latency of the Llama 3.1 8B model on one GPU with the ``float16`` and ``float8`` data types. Use this command to benchmark the latency of the Llama 3.1 70B model on eight GPUs with the ``float16`` and ``float8`` data types.
.. code-block:: .. code-block::
./vllm_benchmark_report.sh -s latency -m meta-llama/Meta-Llama-3.1-8B-Instruct -g 1 -d float16 ./vllm_benchmark_report.sh -s latency -m meta-llama/Llama-3.1-70B-Instruct -g 8 -d float16
./vllm_benchmark_report.sh -s latency -m amd/Meta-Llama-3.1-8B-Instruct-FP8-KV -g 1 -d float8 ./vllm_benchmark_report.sh -s latency -m amd/Llama-3.1-70B-Instruct-FP8-KV -g 8 -d float8
Find the latency reports at: Find the latency reports at:
- ``./reports_float16/summary/Meta-Llama-3.1-8B-Instruct_latency_report.csv`` - ``./reports_float16/summary/Llama-3.1-70B-Instruct_latency_report.csv``
- ``./reports_float8/summary/Meta-Llama-3.1-8B-Instruct-FP8-KV_latency_report.csv`` - ``./reports_float8/summary/Llama-3.1-70B-Instruct-FP8-KV_latency_report.csv``
Example 2: throughput benchmark Example 2: throughput benchmark
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Use this command to benchmark the throughput of the Llama 3.1 8B model on one GPU with the ``float16`` and ``float8`` data types. Use this command to benchmark the throughput of the Llama 3.1 70B model on eight GPUs with the ``float16`` and ``float8`` data types.
.. code-block:: shell .. code-block:: shell
./vllm_benchmark_report.sh -s throughput -m meta-llama/Meta-Llama-3.1-8B-Instruct -g 1 -d float16 ./vllm_benchmark_report.sh -s throughput -m meta-llama/Llama-3.1-70B-Instruct -g 8 -d float16
./vllm_benchmark_report.sh -s throughput -m amd/Meta-Llama-3.1-8B-Instruct-FP8-KV -g 1 -d float8 ./vllm_benchmark_report.sh -s throughput -m amd/Llama-3.1-70B-Instruct-FP8-KV -g 8 -d float8
Find the throughput reports at: Find the throughput reports at:
- ``./reports_float16/summary/Meta-Llama-3.1-8B-Instruct_throughput_report.csv`` - ``./reports_float16/summary/Llama-3.1-70B-Instruct_throughput_report.csv``
- ``./reports_float8/summary/Meta-Llama-3.1-8B-Instruct-FP8-KV_throughput_report.csv`` - ``./reports_float8/summary/Llama-3.1-70B-Instruct-FP8-KV_throughput_report.csv``
.. raw:: html .. raw:: html
@@ -394,7 +439,7 @@ Further reading
MI300X accelerators, see :doc:`../../system-optimization/mi300x`. MI300X accelerators, see :doc:`../../system-optimization/mi300x`.
- To learn how to run LLM models from Hugging Face or your own model, see - To learn how to run LLM models from Hugging Face or your own model, see
:doc:`Using ROCm for AI <../index>`. :doc:`Running models from Hugging Face <hugging-face-models>`.
- To learn how to optimize inference on LLMs, see - To learn how to optimize inference on LLMs, see
:doc:`Inference optimization <../inference-optimization/index>`. :doc:`Inference optimization <../inference-optimization/index>`.
@@ -402,6 +447,32 @@ Further reading
- To learn how to fine-tune LLMs, see - To learn how to fine-tune LLMs, see
:doc:`Fine-tuning LLMs <../fine-tuning/index>`. :doc:`Fine-tuning LLMs <../fine-tuning/index>`.
- To compare with the previous version of the ROCm vLLM Docker image for performance validation, refer to Previous versions
`LLM inference performance validation on AMD Instinct MI300X (ROCm 6.2.0) <https://rocm.docs.amd.com/en/docs-6.2.0/how-to/performance-validation/mi300x/vllm-benchmark.html>`_. =================
This table lists previous versions of the ROCm vLLM Docker image for inference
performance validation. For detailed information about available models for
benchmarking, see the version-specific documentation.
.. list-table::
:header-rows: 1
:stub-columns: 1
* - ROCm version
- vLLM version
- PyTorch version
- Resources
* - 6.2.1
- 0.6.4
- 2.5.0
-
* `Documentation <https://rocm.docs.amd.com/en/docs-6.3.0/how-to/performance-validation/mi300x/vllm-benchmark.html>`_
* `Docker Hub <https://hub.docker.com/layers/rocm/vllm/rocm6.2_mi300_ubuntu20.04_py3.9_vllm_0.6.4/images/sha256-ccbb74cc9e7adecb8f7bdab9555f7ac6fc73adb580836c2a35ca96ff471890d8>`_
* - 6.2.0
- 0.4.3
- 2.4.0
-
* `Documentation <https://rocm.docs.amd.com/en/docs-6.2.0/how-to/performance-validation/mi300x/vllm-benchmark.html>`_
* `Docker Hub <https://hub.docker.com/layers/rocm/vllm/rocm6.2_mi300_ubuntu22.04_py3.9_vllm_7c5fd50/images/sha256-9e4dd4788a794c3d346d7d0ba452ae5e92d39b8dfac438b2af8efdc7f15d22c0>`_