mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
4.6 KiB
4.6 KiB
Qwen 2 Vision Model: Qwen2-VL Collection
Hanzo Engine supports the Qwen2-VL multimodal model family. ISQ quantization is supported to allow running the model with less memory requirements.
UQFF quantizations are also available.
Note: When using device mapping or model topology, only the text model and its layers will be managed. This is because it contains most of the model parameters. The text model has 28 layers.
Quick Start
hanzo run -m Qwen/Qwen2-VL-2B-Instruct --isq 4 --image photo.jpg -i "Describe this image"
Input Formats
The Python and HTTP APIs support sending inputs as:
- Images: URL, path to a local file, or base64 encoded string (via
image_url)
The Rust SDK takes images from the image crate.
HTTP API
- Start the server
hanzo serve -m Qwen/Qwen2-VL-2B-Instruct --isq 4 -p 1234
- Send a request
from openai import OpenAI
client = OpenAI(api_key="foobar", base_url="http://localhost:1234/v1/")
completion = client.chat.completions.create(
model="default",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://www.nhmagazine.com/content/uploads/2019/05/mtwashingtonFranconia-2-19-18-108-Edit-Edit.jpg"
},
},
{
"type": "text",
"text": "What is this?",
},
],
},
],
max_tokens=256,
)
print(completion.choices[0].message.content)
- You can find an example of encoding the image via base64 here.
- You can find an example of loading an image locally here.
Python SDK
You can find this example here.
from hanzo import Runner, Which, ChatCompletionRequest, MultimodalArchitecture
MODEL_ID = "Qwen/Qwen2-VL-2B-Instruct"
runner = Runner(
which=Which.MultimodalPlain(
model_id=MODEL_ID,
arch=MultimodalArchitecture.Qwen2VL,
),
)
res = runner.send_chat_completion_request(
ChatCompletionRequest(
model="default",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://www.nhmagazine.com/content/uploads/2019/05/mtwashingtonFranconia-2-19-18-108-Edit-Edit.jpg"
},
},
{
"type": "text",
"text": "What is this?",
},
],
}
],
max_tokens=256,
)
)
print(res.choices[0].message.content)
print(res.usage)
- You can find an example of encoding the image via base64 here.
- You can find an example of loading an image locally here.
Rust SDK
You can find this example here.
use anyhow::Result;
use hanzo::{IsqType, TextMessageRole, MultimodalMessages, MultimodalModelBuilder};
const MODEL_ID: &str = "Qwen/Qwen2-VL-2B-Instruct";
#[tokio::main]
async fn main() -> Result<()> {
let model =
MultimodalModelBuilder::new(MODEL_ID)
.with_isq(IsqType::Q4K)
.with_logging()
.build()
.await?;
let bytes = match reqwest::blocking::get(
"https://www.nhmagazine.com/content/uploads/2019/05/mtwashingtonFranconia-2-19-18-108-Edit-Edit.jpg",
) {
Ok(http_resp) => http_resp.bytes()?.to_vec(),
Err(e) => anyhow::bail!(e),
};
let image = image::load_from_memory(&bytes)?;
let messages = MultimodalMessages::new().add_image_message(
TextMessageRole::User,
"What is this?",
vec![image],
);
let response = model.send_chat_request(messages).await?;
println!("{}", response.choices[0].message.content.as_ref().unwrap());
dbg!(
response.usage.avg_prompt_tok_per_sec,
response.usage.avg_compl_tok_per_sec
);
Ok(())
}