Replace all remaining references to the upstream project/author/framework brands with hanzo / hanzoai across source, comments, docs, examples, and build scripts. - Project spellings mistral.rs / mistral-rs / mistralrs -> hanzo (and the hanzo-* crate/binary names in Dockerfiles, Makefile, install.ps1, .gitignore). - Author EricLBuehler / EricBuehler / EricB / Eric Buehler -> hanzoai / Hanzo AI (incl. Cargo `authors` and LICENSE copyright). - URLs: github.com/EricLBuehler/mistral.rs -> github.com/hanzoai/engine; ericlbuehler.github.io/mistral.rs -> hanzoai.github.io/engine; github.com/huggingface/candle -> github.com/hanzoai/ml; docs.rs/candle-* -> docs.rs/hanzo-ml; ghcr.io image -> ghcr.io/hanzoai/engine. - Env var prefix MISTRALRS_ -> HANZO_ (producers and consumers kept in sync, incl. the build-script-injected HANZO_GIT_REVISION). - ROCm static lib libmistralrsrocm.a / mistralrsrocm -> libhanzorocm.a / hanzorocm. - localStorage keys mistralrs_settings -> hanzo_settings. Deliberately preserved: real model architectures (struct Mistral, models/mistral.rs file, Mixtral/Llama/Gemma/Qwen/Phi), real model-org HF repo-ids (mistralai/, meta-llama/, google/, Qwen/, microsoft/), the `use hanzo_ml as candle` code aliases and `candle::` identifiers, and the upstream-author HF data repo-id suffix in "hanzoai/mistralrs_tests". Both `cargo check` and `cargo check --features vulkan` pass with 0 errors. Co-authored-by: Hanzo Dev <[email protected]>
6.6 KiB
Gemma 4 Model Family: Collection
Gemma 4 is a multimodal model that supports text, vision (image), video, and audio input with text output. It builds on the Gemma family with full multimodal capabilities across all four input modalities.
Pre-quantized UQFF models are available in the hanzo-community Gemma 4 collection.
Video support: Non-GIF video formats (mp4, avi, mov, etc.) require FFmpeg to be installed. See VIDEO.md for installation instructions and details.
Quick Start
hanzo run -m google/gemma-4-E4B-it --isq 8 --image photo.jpg -i "Describe this image"
With a video:
hanzo run -m google/gemma-4-E4B-it --isq 8 --video video.mp4 -i "Describe this video in detail."
With audio:
hanzo run -m google/gemma-4-E4B-it --isq 8 --audio audio.mp3 -i "Transcribe this fully."
Input Formats
The Python and HTTP APIs support sending inputs as:
- Images: URL, path to a local file, or base64 encoded string (via
image_url) - Videos: URL or path to a local file (via
video_url) - Audio: URL or path to a local file (via
audio_url)
The Rust SDK takes images from the image crate, audio from AudioInput, and video from VideoInput.
HTTP API
- Start the server
hanzo serve -m google/gemma-4-E4B-it --isq 8 -p 1234
- Send a request
from openai import OpenAI
client = OpenAI(api_key="foobar", base_url="http://localhost:1234/v1/")
completion = client.chat.completions.create(
model="default",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://www.nhmagazine.com/content/uploads/2019/05/mtwashingtonFranconia-2-19-18-108-Edit-Edit.jpg"
},
},
{
"type": "text",
"text": "What is this?",
},
],
},
],
max_tokens=256,
)
print(completion.choices[0].message.content)
Video example:
from openai import OpenAI
client = OpenAI(api_key="foobar", base_url="http://localhost:1234/v1/")
completion = client.chat.completions.create(
model="default",
messages=[
{
"role": "user",
"content": [
{
"type": "video_url",
"video_url": {
"url": "path/to/video.mp4"
},
},
{
"type": "text",
"text": "What happens in this video?",
},
],
}
],
max_tokens=256,
)
print(completion.choices[0].message.content)
Combined image + video + audio example:
from openai import OpenAI
client = OpenAI(api_key="foobar", base_url="http://localhost:1234/v1/")
completion = client.chat.completions.create(
model="default",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://www.nhmagazine.com/content/uploads/2019/05/mtwashingtonFranconia-2-19-18-108-Edit-Edit.jpg"
},
},
{
"type": "video_url",
"video_url": {
"url": "path/to/video.mp4"
},
},
{
"type": "audio_url",
"audio_url": {
"url": "path/to/audio.wav"
},
},
{
"type": "text",
"text": "Describe what you see in the image and video, and what you hear in the audio.",
},
],
},
],
max_tokens=256,
)
print(completion.choices[0].message.content)
- You can find an example of encoding the image via base64 here.
- You can find an example of loading an image locally here.
Python SDK
You can find this example here.
from hanzo import Runner, Which, ChatCompletionRequest, MultimodalArchitecture
runner = Runner(
which=Which.MultimodalPlain(
model_id="google/gemma-4-E4B-it",
arch=MultimodalArchitecture.Gemma4,
),
)
res = runner.send_chat_completion_request(
ChatCompletionRequest(
model="default",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://www.nhmagazine.com/content/uploads/2019/05/mtwashingtonFranconia-2-19-18-108-Edit-Edit.jpg"
},
},
{
"type": "text",
"text": "What is this?",
},
],
}
],
max_tokens=256,
)
)
print(res.choices[0].message.content)
print(res.usage)
Rust SDK
You can find this example here.
This is a minimal example of running the Gemma 4 model with a video input. Video decoding uses the parse_video_url helper from hanzo-server-core, which handles FFmpeg decoding and frame sampling automatically.
use anyhow::Result;
use hanzo::{IsqType, TextMessageRole, MultimodalMessages, MultimodalModelBuilder};
use hanzo_server_core::video::parse_video_url;
#[tokio::main]
async fn main() -> Result<()> {
let model =
MultimodalModelBuilder::new("google/gemma-4-E4B-it")
.with_isq(IsqType::Q4K)
.with_logging()
.build()
.await?;
let video = parse_video_url("path/to/video.mp4", None).await?;
let messages = MultimodalMessages::new().add_video_message(
TextMessageRole::User,
"What happens in this video?",
vec![video],
);
let response = model.send_chat_request(messages).await?;
println!("{}", response.choices[0].message.content.as_ref().unwrap());
dbg!(
response.usage.avg_prompt_tok_per_sec,
response.usage.avg_compl_tok_per_sec
);
Ok(())
}