Files
engine/examples/server/qwen3_vl.py
Eric BuehlerandGitHub 2896a7fa04 Implement Qwen 3 VL! (#1657)
* Vision model loading

* Progress

* Works

* Fix a small bug

* Updated get_rope_index

* Dont upcast to f32

* Dead code

* Cleanup

* Use sdpa in vision

* optimized deepstack

* UQFF support

* Paged attn support

* Finalize

* Fix ci

* Clippy
2025-10-22 23:38:59 -04:00

31 lines
804 B
Python

from openai import OpenAI
client = OpenAI(api_key="foobar", base_url="http://localhost:1234/v1/")
completion = client.chat.completions.create(
model="default",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://www.garden-treasures.com/cdn/shop/products/IMG_6245.jpg"
},
},
{
"type": "text",
"text": "What type of flower is this? Give some fun facts.",
},
],
},
],
max_tokens=256,
frequency_penalty=1.0,
top_p=0.1,
temperature=0,
)
resp = completion.choices[0].message.content
print(resp)