mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
* Add cli * Add quantize command * Works * Not interactive mode for quantize * Not prefix cache for quantize * Fix scheduler off by one bug * Add config toml * Add ui, doctor, tune * Add static files * Better message * Fix max batch size default * Remove dead code * Format * Store git rev * Fix a few bugs with ui upload * Clippy * Better error on quantized file missing * Better error on quantized file missing * Better error * Tweak doctor cehcks * Format * Add cli docs * Better tue and doctor * Tweaks * Handle max context * Remove concurrent * Add more subcomands * Remove prune * Fix command * Fix handling of bench prompt * Fix prompt timings * Add version and help * Clarify interactive * Cleaner * Show latency * Show toml * Fix default * Also log for toml * Better toml docs * Format for device layres * Change from >=75 to >=80 for FA * Add install scripts * Update all docs * Apply prefix in server too * Redo readme * Fix default port * Fix image tokens * Mark old crates as deprecated and warn * Mention cli * Finalize readme * Add doctor * Fix install script * Fix curl * Add nccl feature * Tweak readme * Fixes for install scripts * Fix confusing HF error with auto, fix timings for prompt * Switch serve models * Unify hf paths + token resolution * Add config link, tweak quants docs * Update typos * Fix block metadata bug * Fix tagline * All models extract logitcs before lmhead * Show guide to quickstart or restart * Add banner * Tweaks * More tweaks * Contributing and credits * Simplify python docs * Add mdbook, deploy workflow * Add banner * Fix deploy * Dark theme to ayu * Add python sdk to main docs * Add python sdk to main docs * Link to crates.io * Default run * Update to point at readme * Remove dead configs * Update version, ring feature too * Fix images rendering, better docs for features * Fix version bump * Clippy * Dont doc mistralrs-cli * Fix link warn
38 lines
1.1 KiB
Python
38 lines
1.1 KiB
Python
from openai import OpenAI
|
||
|
||
|
||
# Point the client to the locally running server
|
||
client = OpenAI(api_key="foobar", base_url="http://localhost:1234/v1/")
|
||
|
||
# Remote assets – feel free to swap for anything else
|
||
IMAGE_URL = "https://www.allaboutbirds.org/guide/assets/og/528129121-1200px.jpg"
|
||
AUDIO_URL = "https://upload.wikimedia.org/wikipedia/commons/4/42/Bird_singing.ogg"
|
||
|
||
completion = client.chat.completions.create(
|
||
model="default",
|
||
messages=[
|
||
{
|
||
"role": "user",
|
||
"content": [
|
||
{ # Audio clip
|
||
"type": "audio_url",
|
||
"audio_url": {"url": AUDIO_URL},
|
||
},
|
||
{ # Image
|
||
"type": "image_url",
|
||
"image_url": {"url": IMAGE_URL},
|
||
},
|
||
{
|
||
"type": "text",
|
||
"text": "Describe in detail what is happening, referencing both what you hear and what you see.",
|
||
},
|
||
],
|
||
}
|
||
],
|
||
max_tokens=256,
|
||
temperature=0.2,
|
||
top_p=0.9,
|
||
)
|
||
|
||
print(completion.choices[0].message.content)
|