Files
zeekayandHanzo Dev 02b3f00cef chore: complete brand scrub of upstream mistral.rs lineage to hanzo/hanzoai
Replace all remaining references to the upstream project/author/framework
brands with hanzo / hanzoai across source, comments, docs, examples, and
build scripts.

- Project spellings mistral.rs / mistral-rs / mistralrs -> hanzo (and the
  hanzo-* crate/binary names in Dockerfiles, Makefile, install.ps1, .gitignore).
- Author EricLBuehler / EricBuehler / EricB / Eric Buehler -> hanzoai / Hanzo AI
  (incl. Cargo `authors` and LICENSE copyright).
- URLs: github.com/EricLBuehler/mistral.rs -> github.com/hanzoai/engine;
  ericlbuehler.github.io/mistral.rs -> hanzoai.github.io/engine;
  github.com/huggingface/candle -> github.com/hanzoai/ml;
  docs.rs/candle-* -> docs.rs/hanzo-ml; ghcr.io image -> ghcr.io/hanzoai/engine.
- Env var prefix MISTRALRS_ -> HANZO_ (producers and consumers kept in sync,
  incl. the build-script-injected HANZO_GIT_REVISION).
- ROCm static lib libmistralrsrocm.a / mistralrsrocm -> libhanzorocm.a / hanzorocm.
- localStorage keys mistralrs_settings -> hanzo_settings.

Deliberately preserved: real model architectures (struct Mistral, models/mistral.rs
file, Mixtral/Llama/Gemma/Qwen/Phi), real model-org HF repo-ids (mistralai/,
meta-llama/, google/, Qwen/, microsoft/), the `use hanzo_ml as candle` code
aliases and `candle::` identifiers, and the upstream-author HF data repo-id
suffix in "hanzoai/mistralrs_tests".

Both `cargo check` and `cargo check --features vulkan` pass with 0 errors.

Co-authored-by: Hanzo Dev <[email protected]>
2026-06-01 11:53:12 -07:00

61 lines
1.7 KiB
Rust

//! Load a pre-quantized UQFF text model.
//!
//! Run with: `cargo run --release --example uqff -p hanzo`
use anyhow::Result;
use hanzo::{
PagedAttentionMetaBuilder, RequestBuilder, TextMessageRole, TextMessages, UqffTextModelBuilder,
};
#[tokio::main]
async fn main() -> Result<()> {
let model = UqffTextModelBuilder::new(
"hanzoai/Phi-3.5-mini-instruct-UQFF",
vec!["phi3.5-mini-instruct-q8_0.uqff".into()],
)
.into_inner()
.with_logging()
.with_paged_attn(PagedAttentionMetaBuilder::default().build()?)
.build()
.await?;
let messages = TextMessages::new()
.add_message(
TextMessageRole::System,
"You are an AI agent with a specialty in programming.",
)
.add_message(
TextMessageRole::User,
"Hello! How are you? Please write generic binary search function in Rust.",
);
let response = model.send_chat_request(messages).await?;
println!("{}", response.choices[0].message.content.as_ref().unwrap());
dbg!(
response.usage.avg_prompt_tok_per_sec,
response.usage.avg_compl_tok_per_sec
);
// Next example: Return some logprobs with the `RequestBuilder`, which enables higher configurability.
let request = RequestBuilder::new().return_logprobs(true).add_message(
TextMessageRole::User,
"Please write a mathematical equation where a few numbers are added.",
);
let response = model.send_chat_request(request).await?;
println!(
"Logprobs: {:?}",
&response.choices[0]
.logprobs
.as_ref()
.unwrap()
.content
.as_ref()
.unwrap()[0..3]
);
Ok(())
}