hanzoai/ROCm
2
0
Fork 0
mirror of https://github.com/hanzoai/ROCm.git synced 2026-09-30 20:37:25 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix ExLlama-v2 code snippet (#3281)

Browse Source
This commit is contained in:
Peter Park
2024-06-12 17:03:04 -04:00
committed by GitHub
parent 73caef5cef
commit d29c61e8a8
1 changed files with 5 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/how-to/llm-fine-tuning-optimization/model-quantization.rst
+5 -5
View File
@@ -154,13 +154,13 @@ kernels by configuring the ``exllama_config`` parameter as the following.
.. code-block:: python
from transformers import AutoModelForCausalLM, GPTQConfig
pretrained_model_dir = "meta-llama/Llama-2-7b"
gptq_config = GPTQConfig(bits=4, exllama_config={"version":2})
#pretrained_model_dir = "meta-llama/Llama-2-7b"
base_model_name = "NousResearch/Llama-2-7b-hf"
gptq_config = GPTQConfig(bits=4, dataset="c4", exllama_config={"version":2})
quantized_model = AutoModelForCausalLM.from_pretrained(
base_model_name,
device_map="auto",
base_model_name,
device_map="auto",
quantization_config=gptq_config)
bitsandbytes
============