Files
engine/examples/server/phi4mm_audio.py
Eric BuehlerandGitHub 5537355ce6 v0.7.0 preparation: mistralrs-cli, revamped documentation, readme update (#1848)
* Add cli

* Add quantize command

* Works

* Not interactive mode for quantize

* Not prefix cache for quantize

* Fix scheduler off by one bug

* Add config toml

* Add ui, doctor, tune

* Add static files

* Better message

* Fix max batch size default

* Remove dead code

* Format

* Store git rev

* Fix a few bugs with ui upload

* Clippy

* Better error on quantized file missing

* Better error on quantized file missing

* Better error

* Tweak doctor cehcks

* Format

* Add cli docs

* Better tue and doctor

* Tweaks

* Handle max context

* Remove concurrent

* Add more subcomands

* Remove prune

* Fix command

* Fix handling of bench prompt

* Fix prompt timings

* Add version and help

* Clarify interactive

* Cleaner

* Show latency

* Show toml

* Fix default

* Also log for toml

* Better toml docs

* Format for device layres

* Change from >=75 to >=80 for FA

* Add install scripts

* Update all docs

* Apply prefix in server too

* Redo readme

* Fix default port

* Fix image tokens

* Mark old crates as deprecated and warn

* Mention cli

* Finalize readme

* Add doctor

* Fix install script

* Fix curl

* Add nccl feature

* Tweak readme

* Fixes for install scripts

* Fix confusing HF error with auto, fix timings for prompt

* Switch serve models

* Unify hf paths + token resolution

* Add config link, tweak quants docs

* Update typos

* Fix block metadata bug

* Fix tagline

* All models extract logitcs before lmhead

* Show guide to quickstart or restart

* Add banner

* Tweaks

* More tweaks

* Contributing and credits

* Simplify python docs

* Add mdbook, deploy workflow

* Add banner

* Fix deploy

* Dark theme to ayu

* Add python sdk to main docs

* Add python sdk to main docs

* Link to crates.io

* Default run

* Update to point at readme

* Remove dead configs

* Update version, ring feature too

* Fix images rendering, better docs for features

* Fix version bump

* Clippy

* Dont doc mistralrs-cli

* Fix link warn
2026-01-27 15:58:40 -05:00

38 lines
1.1 KiB
Python
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
from openai import OpenAI
# Point the client to the locally running server
client = OpenAI(api_key="foobar", base_url="http://localhost:1234/v1/")
# Remote assets – feel free to swap for anything else
IMAGE_URL = "https://www.allaboutbirds.org/guide/assets/og/528129121-1200px.jpg"
AUDIO_URL = "https://upload.wikimedia.org/wikipedia/commons/4/42/Bird_singing.ogg"
completion = client.chat.completions.create(
model="default",
messages=[
{
"role": "user",
"content": [
{ # Audio clip
"type": "audio_url",
"audio_url": {"url": AUDIO_URL},
},
{ # Image
"type": "image_url",
"image_url": {"url": IMAGE_URL},
},
{
"type": "text",
"text": "Describe in detail what is happening, referencing both what you hear and what you see.",
},
],
}
],
max_tokens=256,
temperature=0.2,
top_p=0.9,
)
print(completion.choices[0].message.content)