mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
The endpoint reported zero cache usage every turn: the engine carried no cached-token count, so a prefix cache serving most of a long conversation was invisible to the client. Usage now carries cached_prompt_tokens, filled in get_usage from the sequence group, which takes it from prefix_cache_len: set by the prefix cacher on the non-paged path and by the scheduler prefix-cache lookup on the paged one. Anthropic reports it as cache_read_input_tokens and subtracts it from input_tokens, because the two have to sum to the prompt and prompt_len counts the whole prompt; without the subtraction a cached turn counts its prefix twice. The responses endpoint keeps cached tokens inside input_tokens and breaks them out under input_tokens_details, which is the OpenAI shape rather than the Anthropic one. The tests cover the mapping and the message_start wire shape. A live cache hit needs a warm prefix cache and a model. Co-authored-by: Hanzo Dev <[email protected]>
hanzo — Blazing-Fast LLM Inference in Rust
The Rust SDK for hanzo, a high-performance LLM inference engine supporting text, multimodal, speech, image generation, and embedding models.
API Docs | GitHub | Examples | Discord
Quick Start
use hanzo::{IsqBits, ModelBuilder, TextMessages, TextMessageRole};
#[tokio::main]
async fn main() -> hanzo::error::Result<()> {
let model = ModelBuilder::new("Qwen/Qwen3-4B")
.with_auto_isq(IsqBits::Four)
.build()
.await?;
let response = model.chat("What is Rust's ownership model?").await?;
println!("{response}");
Ok(())
}
Capabilities
| Capability | Builder | Example |
|---|---|---|
| Any model (auto-detect) | ModelBuilder |
examples/getting_started/text_generation/ |
| Text generation | TextModelBuilder |
examples/getting_started/text_generation/ |
| Multimodal (image+text) | MultimodalModelBuilder |
examples/getting_started/multimodal/ |
| GGUF quantized models | GgufModelBuilder |
examples/getting_started/gguf/ |
| Image generation | DiffusionModelBuilder |
examples/models/diffusion/ |
| Speech synthesis | SpeechModelBuilder |
examples/models/speech/ |
| Embeddings | EmbeddingModelBuilder |
examples/getting_started/embedding/ |
| Structured output | Model::generate_structured |
examples/advanced/json_schema/ |
| Tool calling | Tool, ToolChoice |
examples/advanced/tools/ |
| Agents | AgentBuilder |
examples/advanced/agent/ |
| LoRA / X-LoRA | LoraModelBuilder, XLoraModelBuilder |
examples/advanced/lora/ |
| AnyMoE | AnyMoeModelBuilder |
examples/advanced/anymoe/ |
| MCP client | McpClientConfig |
examples/advanced/mcp_client/ |
Choosing a Request Type
| Type | Use When | Sampling |
|---|---|---|
TextMessages |
Simple text-only chat | Deterministic |
MultimodalMessages |
Prompt includes images or audio | Deterministic |
RequestBuilder |
Tools, logprobs, custom sampling, constraints, adapters, or web search | Configurable |
TextMessages and MultimodalMessages convert into RequestBuilder via Into<RequestBuilder>.
Feature Flags
| Flag | Effect |
|---|---|
cuda |
CUDA GPU support |
flash-attn |
Flash Attention 2 kernels (requires cuda) |
cudnn |
cuDNN acceleration (requires cuda) |
nccl |
Multi-GPU via NCCL (requires cuda and NCCL) |
metal |
Apple Metal GPU support |
accelerate |
Apple Accelerate framework |
mkl |
Intel MKL acceleration |
The default feature set (no flags) builds with pure Rust — no C compiler or system libraries required.
License
MIT