mirror of
https://github.com/zenlm/zen-blog.git
synced 2026-11-02 17:28:53 +00:00
zeekay
a30ff37468
b3b6eb0 ran a find-and-replace over the posts forked from the QwenLM
blog that turned "Qwen2" and "Qwen2.5" into "zen": Qwen2-57B-A14B read
as zen7B-A14B, Qwen2-72B as zen-72B, Qwen2AudioForConditionalGeneration
as a class that does not exist, and links such as
github.com/QwenLM/Qwen2 and the qianwen-res image paths pointed at
nothing. Every token is put back from the original post, aligned
against the rewritten one: 536 lines across 29 posts, and 116 of the
example files under public/cases.
These 44 posts are about another lab's releases and now say so with
"upstream: true" in their frontmatter. Two Zen posts lose the
base-model names they carried, and the Zen 3 post stops calling itself
the next generation.
93 lines
4.7 KiB
Plaintext
93 lines
4.7 KiB
Plaintext
---
|
|
title: "Qwen1.5-110B: The First 100B+ Model of the Qwen1.5 Series"
|
|
date: "2024-04-25"
|
|
description: "Recently we have witnessed a burst of large-scale models with over 100 billion parameters in the opensource community. These models have demonstrated remarkable performance in both benchmark evaluation and chatbot arena. Today, we release the first 100B+ model of the Qwen1.5 series, Qwen1.5-110B, wh"
|
|
upstream: true
|
|
---
|
|
|
|
<LinkButton href="https://github.com/QwenLM/Qwen1.5" label="GITHUB" external />
|
|
<LinkButton href="https://huggingface.co/Qwen" label="HUGGING FACE" external />
|
|
<LinkButton href="https://modelscope.cn/organization/qwen" label="MODELSCOPE" external />
|
|
<LinkButton href="https://huggingface.co/spaces/Qwen/Qwen1.5-110B-Chat-Demo" label="DEMO" external />
|
|
<LinkButton href="https://discord.gg/yPEP2vHTu4" label="DISCORD" external />
|
|
|
|
# Introduction
|
|
|
|
|
|
Recently we have witnessed a burst of large-scale models with over 100 billion parameters in the opensource community. These models have demonstrated remarkable performance in both benchmark evaluation and chatbot arena. Today, we release the first 100B+ model of the Qwen1.5 series, Qwen1.5-110B, which achieves comparable performance with Meta-Llama3-70B in the base model evaluation, and outstanding performance in the chat evaluation, including MT-Bench and AlpacaEval 2.0.
|
|
|
|
|
|
# Model Features
|
|
Qwen1.5-110B is similar to other Qwen1.5 models and built with the same Transformer decoder architecture. It consists of grouped query attention (GQA) and it can be efficient in model serving. The model supports the context length 32K tokens, and the model is still multilingual, supporting a large number of languages including English, Chinese, French, Spanish, German, Russian, Korean, Japanese, Vietnamese, Arabic, etc.
|
|
|
|
# Model Quality
|
|
|
|
We conduct a series of evaluations for the base language models, and we compare with Meta-Llama3-70B, the recent SOTA language model as well as Mixtral-8x22B.
|
|
|
|
| | Qwen1.5-110B | Qwen1.5-72B | Llama-3-70B | Mixtral-8x22B |
|
|
| :----------- | :--: | :--: | :----: | :---: |
|
|
| MMLU | 80.4 | 77.5 | 79.5 | 77.8 |
|
|
| TheoremQA | 34.9 | 29.3 | 32.0 | 35.9 |
|
|
| GPQA | 35.9 | 36.3 | 36.4 | 34.3 |
|
|
| Hellaswag | 87.5 | 86.0 | 88.0 | 88.7 |
|
|
| BBH | 74.8 | 65.5 | 76.6 | 69.2 |
|
|
| ARC-C | 69.6 | 65.9 | 68.8 | 70.7 |
|
|
| GSM8K | 85.4 | 79.5 | 79.2 | 78.6 |
|
|
| MATH | 49.6 | 34.1 | 41.0 | 41.7 |
|
|
| HumanEval | 52.4 | 41.5 | 45.7 | 45.1 |
|
|
| MBPP | 58.1 | 53.4 | 55.1 | 71.2 |
|
|
|
|
The above results show that the new 110B model is at least competitive with the Llama-3-70B model in terms of base capabilities. In terms of this model, we did not change the pretraining and posttraining recipes drastically, and thus we believe that the performance improvement compared with 72B comes from increasing model size.
|
|
|
|
We also test the chat models on MT-Bench and AlpacaEval 2.0.
|
|
|
|
<table>
|
|
<tr>
|
|
<th rowspan="2">Models</th>
|
|
<th colspan="1">MT-Bench</th>
|
|
<th colspan="1">AlpacaEval 2.0</th>
|
|
</tr>
|
|
<tr>
|
|
<th>Avg. Score</th><th>LC Win Rate</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Llama-3-70B-Instruct</td>
|
|
<td>8.85</td>
|
|
<td>34.40</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Qwen1.5-72B-Chat</td>
|
|
<td>8.61</td>
|
|
<td>36.60</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Qwen1.5-110B-Chat</td>
|
|
<td>8.88</td>
|
|
<td>43.90</td>
|
|
</tr>
|
|
</table>
|
|
|
|
Compared with the previously released 72B model, on the two benchmark evaluation for chat models the 110B performs significantly better. The consistent improvement in the evaluation indicates that stronger and larger base language models can lead to better chat models even without changing the post-training recipes much.
|
|
|
|
|
|
# Develop with Qwen1.5-110B
|
|
|
|
We advise you to read our blog for [Qwen1.5](https://qwenlm.github.io/blog/qwen1.5/) to figure out the usages with Transformers, vLLM, llama.cpp, Ollama, LMStudio, SkyPilot, Axolotl, LLaMA-Factory, etc.
|
|
|
|
|
|
# Conclusion
|
|
|
|
The Qwen1.5-110B is the largest model in the Qwen1.5 series, and it is also the first one with over 100 billion parameters in the series. It demonstrates competitive performance against the very recently released SOTA model Llama-3-70B and it is significantly better than the 72B model. This tells us that there is still a lot of room in model size scaling for better performance. While the releease of Llama-3 indicates the significance of data scaling to an extremely large scale, we believe we can get the best of both worlds by scaling both data and model size in our future release. Stay tuned for Qwen2!
|
|
|
|
|
|
# Citation
|
|
|
|
```
|
|
@misc{qwen1.5,
|
|
title = {Introducing Qwen1.5},
|
|
url = {https://qwenlm.github.io/blog/qwen1.5/},
|
|
author = {Qwen Team},
|
|
month = {February},
|
|
year = {2024}
|
|
}
|
|
``` |