Files
zeekayandHanzo Dev 9bbe297a87 feat(server): report prompt-cache reads on /v1/messages and /v1/responses
The endpoint reported zero cache usage every turn: the engine carried no cached-token count, so a prefix cache serving most of a long conversation was invisible to the client. Usage now carries cached_prompt_tokens, filled in get_usage from the sequence group, which takes it from prefix_cache_len: set by the prefix cacher on the non-paged path and by the scheduler prefix-cache lookup on the paged one.

Anthropic reports it as cache_read_input_tokens and subtracts it from input_tokens, because the two have to sum to the prompt and prompt_len counts the whole prompt; without the subtraction a cached turn counts its prefix twice. The responses endpoint keeps cached tokens inside input_tokens and breaks them out under input_tokens_details, which is the OpenAI shape rather than the Anthropic one.

The tests cover the mapping and the message_start wire shape. A live cache hit needs a warm prefix cache and a model.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-15 20:43:13 -07:00
..

hanzo — Blazing-Fast LLM Inference in Rust

The Rust SDK for hanzo, a high-performance LLM inference engine supporting text, multimodal, speech, image generation, and embedding models.

API Docs | GitHub | Examples | Discord

Quick Start

use hanzo::{IsqBits, ModelBuilder, TextMessages, TextMessageRole};

#[tokio::main]
async fn main() -> hanzo::error::Result<()> {
    let model = ModelBuilder::new("Qwen/Qwen3-4B")
        .with_auto_isq(IsqBits::Four)
        .build()
        .await?;

    let response = model.chat("What is Rust's ownership model?").await?;
    println!("{response}");
    Ok(())
}

Capabilities

Capability Builder Example
Any model (auto-detect) ModelBuilder examples/getting_started/text_generation/
Text generation TextModelBuilder examples/getting_started/text_generation/
Multimodal (image+text) MultimodalModelBuilder examples/getting_started/multimodal/
GGUF quantized models GgufModelBuilder examples/getting_started/gguf/
Image generation DiffusionModelBuilder examples/models/diffusion/
Speech synthesis SpeechModelBuilder examples/models/speech/
Embeddings EmbeddingModelBuilder examples/getting_started/embedding/
Structured output Model::generate_structured examples/advanced/json_schema/
Tool calling Tool, ToolChoice examples/advanced/tools/
Agents AgentBuilder examples/advanced/agent/
LoRA / X-LoRA LoraModelBuilder, XLoraModelBuilder examples/advanced/lora/
AnyMoE AnyMoeModelBuilder examples/advanced/anymoe/
MCP client McpClientConfig examples/advanced/mcp_client/

Choosing a Request Type

Type Use When Sampling
TextMessages Simple text-only chat Deterministic
MultimodalMessages Prompt includes images or audio Deterministic
RequestBuilder Tools, logprobs, custom sampling, constraints, adapters, or web search Configurable

TextMessages and MultimodalMessages convert into RequestBuilder via Into<RequestBuilder>.

Feature Flags

Flag Effect
cuda CUDA GPU support
flash-attn Flash Attention 2 kernels (requires cuda)
cudnn cuDNN acceleration (requires cuda)
nccl Multi-GPU via NCCL (requires cuda and NCCL)
metal Apple Metal GPU support
accelerate Apple Accelerate framework
mkl Intel MKL acceleration

The default feature set (no flags) builds with pure Rust — no C compiler or system libraries required.

License

MIT