Replace all remaining references to the upstream project/author/framework brands with hanzo / hanzoai across source, comments, docs, examples, and build scripts. - Project spellings mistral.rs / mistral-rs / mistralrs -> hanzo (and the hanzo-* crate/binary names in Dockerfiles, Makefile, install.ps1, .gitignore). - Author EricLBuehler / EricBuehler / EricB / Eric Buehler -> hanzoai / Hanzo AI (incl. Cargo `authors` and LICENSE copyright). - URLs: github.com/EricLBuehler/mistral.rs -> github.com/hanzoai/engine; ericlbuehler.github.io/mistral.rs -> hanzoai.github.io/engine; github.com/huggingface/candle -> github.com/hanzoai/ml; docs.rs/candle-* -> docs.rs/hanzo-ml; ghcr.io image -> ghcr.io/hanzoai/engine. - Env var prefix MISTRALRS_ -> HANZO_ (producers and consumers kept in sync, incl. the build-script-injected HANZO_GIT_REVISION). - ROCm static lib libmistralrsrocm.a / mistralrsrocm -> libhanzorocm.a / hanzorocm. - localStorage keys mistralrs_settings -> hanzo_settings. Deliberately preserved: real model architectures (struct Mistral, models/mistral.rs file, Mixtral/Llama/Gemma/Qwen/Phi), real model-org HF repo-ids (mistralai/, meta-llama/, google/, Qwen/, microsoft/), the `use hanzo_ml as candle` code aliases and `candle::` identifiers, and the upstream-author HF data repo-id suffix in "hanzoai/mistralrs_tests". Both `cargo check` and `cargo check --features vulkan` pass with 0 errors. Co-authored-by: Hanzo Dev <[email protected]>
26 KiB
Agentic Features Guide
hanzo can execute tools on behalf of the model in a server-side loop, eliminating client round-trips. This guide walks through each agentic capability from simplest to most advanced.
Give a local model web search in one command:
hanzo run --enable-search -m Qwen/Qwen3-4B
Or use the standard OpenAI Python SDK. No custom client needed:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:1234/v1/", api_key="foobar")
response = client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": "What happened in the news today?"}],
web_search_options={},
)
| I want to... | Use... | Section |
|---|---|---|
| Add search to any model, zero code | Web Search | Jump |
| Run my own functions server-side | Tool Callbacks | Jump |
| Build a full agent in Rust | Agent Builder | Jump |
| Connect to external tool servers | MCP Client | Jump |
| Dispatch tools over HTTP in production | Tool Dispatch URLs | Jump |
Table of Contents
- Overview
- Build a Full Agent in 5 Minutes
- Web Search
- Tool Callbacks
- Agent Builder (Rust SDK)
- MCP Client
- Tool Dispatch URLs
- Common Configuration
- Combining Features
- Further Reading
Overview
What makes a request "agentic"
In basic tool calling, the model generates a tool call, the server returns it to your code, and you execute the tool yourself before sending the result back. This is the standard OpenAI flow.
In agentic tool calling, the server executes tools automatically and feeds results back to the model in a loop. No client round-trips needed. You get a final text answer instead of intermediate tool calls.
For basic tool calling details, see TOOL_CALLING.md.
The agentic loop
- Model generates a response. If it includes tool calls, continue; otherwise return the response.
- Server dispatches each tool call to the appropriate handler.
- Tool results are appended to the conversation as
toolmessages. - Model generates again with the updated conversation.
- Repeat until the model produces a final text response or
max_tool_roundsis reached.
Dispatch order
When the model calls a tool, the server tries these handlers in order:
- Built-in search tools:
search_the_webandwebsite_content_extractor(if web search is enabled) - Registered callbacks: tool callbacks from the Python/Rust SDK or auto-registered MCP tools
- Tool dispatch URL: POSTs the tool call to your HTTP endpoint
- No handler found: the loop stops and the un-executed tool call is returned to the client
Supported models
Agentic features work with any model that supports tool calling. See TOOL_CALLING.md for the full list.
Web Search
The simplest agentic feature: flip one flag and the model can search the web. hanzo uses DuckDuckGo for search and EmbeddingGemma for result reranking. No tool schemas needed from the user; the search tools are injected automatically.
Enabling web search
CLI:
hanzo run --enable-search --isq 4 -m google/gemma-4-E4B-it
HTTP API:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:1234/v1/", api_key="foobar")
response = client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": "What is the weather forecast for Boston?"}],
tool_choice="auto",
max_tokens=1024,
web_search_options={},
)
print(response.choices[0].message.content)
Python SDK:
from hanzo import Runner, Which, Architecture, ChatCompletionRequest, WebSearchOptions
runner = Runner(
which=Which.Plain(model_id="google/gemma-4-E4B-it"),
enable_search=True,
)
res = runner.send_chat_completion_request(
ChatCompletionRequest(
model="default",
messages=[{"role": "user", "content": "What is the weather forecast for Boston?"}],
max_tokens=256,
web_search_options=WebSearchOptions(),
)
)
print(res.choices[0].message.content)
Rust SDK:
use hanzo::{
IsqBits, ModelBuilder, RequestBuilder, SearchEmbeddingModel,
TextMessageRole, TextMessages, WebSearchOptions,
};
let model = ModelBuilder::new("google/gemma-4-E4B-it")
.with_auto_isq(IsqBits::Four)
.with_search(SearchEmbeddingModel::default())
.build()
.await?;
let messages = TextMessages::new()
.add_message(TextMessageRole::User, "What is the weather forecast for Boston?");
let request = RequestBuilder::from(messages)
.with_web_search_options(WebSearchOptions::default());
let response = model.send_chat_request(request).await?;
println!("{}", response.choices[0].message.content.as_ref().unwrap());
Custom search callbacks
Override the default DuckDuckGo search with your own function. The callback receives a query and returns a list of results with title, description, url, and content fields.
Python SDK:
def my_search(query: str) -> list[dict[str, str]]:
return [{"title": "Result", "description": "...", "url": "https://...", "content": "..."}]
runner = Runner(
which=Which.Plain(model_id="google/gemma-4-E4B-it"),
enable_search=True,
search_callback=my_search,
)
Rust SDK:
let model = ModelBuilder::new("google/gemma-4-E4B-it")
.with_auto_isq(IsqBits::Four)
.with_search_callback(Arc::new(|params: &hanzo::SearchFunctionParameters| {
// Return Vec<SearchResult> with title, description, url, content
my_custom_search(¶ms.query)
}))
.build()
.await?;
For full details, see WEB_SEARCH.md.
Tool Callbacks
Register named functions at model-build time. When the model calls a tool by that name during the agentic loop, your function runs server-side. Set max_tool_rounds on the request to activate the loop.
Tool callbacks: Python SDK
import json
from hanzo import Runner, Which, Architecture, ChatCompletionRequest, ToolChoice
def tool_callback(name: str, args: dict) -> str:
"""Dispatch tool calls to local implementations."""
if name == "get_weather":
city = args.get("city", "unknown")
return json.dumps({"city": city, "temp": 22, "condition": "Sunny"})
if name == "calculate":
expression = args.get("expression", "0")
return json.dumps({"result": eval(expression)})
return json.dumps({"error": f"Unknown tool: {name}"})
runner = Runner(
which=Which.Plain(model_id="Qwen/Qwen3-4B", arch=Architecture.Qwen3),
tool_callbacks={"get_weather": tool_callback, "calculate": tool_callback},
)
tools = [
json.dumps({
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city.",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string", "description": "City name"}},
"required": ["city"],
},
"strict": True,
},
}),
json.dumps({
"type": "function",
"function": {
"name": "calculate",
"description": "Evaluate a math expression.",
"parameters": {
"type": "object",
"properties": {"expression": {"type": "string", "description": "Expression"}},
"required": ["expression"],
},
"strict": True,
},
}),
]
request = ChatCompletionRequest(
messages=[{"role": "user", "content": "What's the weather in Tokyo? Also calculate 42 * 17."}],
model="default",
tool_schemas=tools,
tool_choice=ToolChoice.Auto,
max_tool_rounds=5,
)
response = runner.send_chat_completion_request(request)
print(response.choices[0].message.content)
Tool callbacks: Rust SDK
use hanzo::{
CalledFunction, IsqBits, ModelBuilder, RequestBuilder,
TextMessageRole, TextMessages, Tool, ToolChoice, ToolType,
};
use std::sync::Arc;
let model = ModelBuilder::new("Qwen/Qwen3-4B")
.with_auto_isq(IsqBits::Four)
.with_tool_callback(
"get_weather",
Arc::new(|f: &CalledFunction| {
let args: serde_json::Value = serde_json::from_str(&f.arguments)?;
let city = args["city"].as_str().unwrap_or("unknown");
Ok(format!(r#"{{"city":"{}","temp":22,"condition":"Sunny"}}"#, city))
}),
)
.build()
.await?;
let tool = Tool {
tp: ToolType::Function,
function: hanzo::Function {
description: Some("Get the current weather for a city.".to_string()),
name: "get_weather".to_string(),
parameters: Some(std::collections::HashMap::from([(
"city".to_string(),
serde_json::json!({"type": "string", "description": "City name"}),
)])),
strict: Some(true),
},
};
let messages = TextMessages::new()
.add_message(TextMessageRole::User, "What's the weather in Tokyo?");
let request = RequestBuilder::from(messages)
.set_tools(vec![tool])
.set_tool_choice(ToolChoice::Auto)
.set_max_tool_rounds(5);
let response = model.send_chat_request(request).await?;
println!("{}", response.choices[0].message.content.as_ref().unwrap());
Agent Builder (Rust SDK)
The Agent Builder is a Rust SDK-only abstraction that wraps the model, tool definitions, and the agentic loop into a single Agent object. It provides a higher-level API compared to RequestBuilder with manual tool callbacks.
The #[tool] macro
Annotate a function with #[tool] to generate tool definitions and callbacks automatically. The macro produces three items: <name>_tool(), <name>_callback(), and <name>_tool_with_callback().
Both sync and async functions are supported. Sync tools run in spawn_blocking; async tools run natively.
use hanzo::tool;
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
#[derive(Debug, Serialize, Deserialize, JsonSchema)]
struct WeatherInfo {
temperature: f32,
conditions: String,
humidity: u8,
}
#[tool(description = "Get the current weather for a location")]
fn get_weather(
#[description = "The city name to get weather for"] city: String,
#[description = "Temperature unit: 'celsius' or 'fahrenheit'"]
#[default = "celsius"]
unit: Option<String>,
) -> Result<WeatherInfo> {
let temp = match unit.as_deref() {
Some("fahrenheit") => 72.5,
_ => 22.5,
};
Ok(WeatherInfo {
temperature: temp,
conditions: format!("Sunny with clear skies in {}", city),
humidity: 45,
})
}
// Async tools are also supported:
#[tool(description = "Search the web for information on a topic")]
async fn web_search(
#[description = "The search query"] query: String,
#[description = "Maximum number of results"]
#[default = 3u32]
max_results: Option<u32>,
) -> Result<Vec<SearchResult>> {
// async implementation...
}
Building an agent
use hanzo::{AgentBuilder, IsqBits, ModelBuilder, PagedAttentionMetaBuilder};
let model = ModelBuilder::new("Qwen/Qwen3-4B")
.with_auto_isq(IsqBits::Four)
.with_paged_attn(PagedAttentionMetaBuilder::default().build()?)
.build()
.await?;
let agent = AgentBuilder::new(model)
.with_system_prompt(
"You are a helpful assistant with access to weather and web search tools.",
)
.with_max_iterations(5)
.with_parallel_tool_execution(true)
.register_tool(get_weather_tool_with_callback())
.register_tool(web_search_tool_with_callback())
.build();
Non-streaming execution
let response = agent.run("What's the weather in Boston?").await?;
if let Some(text) = &response.final_response {
println!("{}", text);
}
println!("Iterations: {}", response.iterations);
println!("Stop reason: {:?}", response.stop_reason);
for (i, step) in response.steps.iter().enumerate() {
println!("Step {}: {} tool call(s)", i + 1, step.tool_calls.len());
for call in &step.tool_calls {
println!(" - {}: {}", call.function.name, call.function.arguments);
}
}
AgentStopReason tells you why the agent stopped:
TextResponse: model produced a final text answerMaxIterations: hit the iteration limitNoAction: model produced no responseError(e): an error occurred
Streaming execution
use hanzo::AgentEvent;
use std::io::Write;
let mut stream = agent.run_stream("What's the weather in Boston?").await?;
while let Some(event) = stream.next().await {
match event {
AgentEvent::TextDelta(text) => {
print!("{}", text);
std::io::stdout().flush()?;
}
AgentEvent::ToolCallsStart(calls) => {
println!("\n[Calling {} tool(s)...]", calls.len());
for call in &calls {
println!(" - {}: {}", call.function.name, call.function.arguments);
}
}
AgentEvent::ToolResult(result) => {
let status = if result.result.is_ok() { "OK" } else { "ERROR" };
println!(" [Tool {} completed: {}]", result.tool_name, status);
}
AgentEvent::ToolCallsComplete => {
println!("[All tools completed, continuing...]");
}
AgentEvent::Complete(response) => {
println!("\nDone in {} iteration(s)", response.iterations);
}
}
}
MCP Client
Connect to external MCP servers to give the model access to tools like filesystem operations, databases, APIs, and more. Tools are auto-discovered from connected servers at startup.
Three transport types are supported:
- Process: local command (stdin/stdout communication)
- HTTP: remote JSON-RPC over HTTP
- WebSocket: remote real-time communication
Quick start with the CLI
Create mcp-config.json:
{
"servers": [
{
"name": "Filesystem Tools",
"source": {
"type": "Process",
"command": "npx",
"args": ["@modelcontextprotocol/server-filesystem", "."]
}
}
]
}
Start the server:
hanzo serve -p 1234 --mcp-config mcp-config.json --max-tool-rounds 5 -m Qwen/Qwen3-4B
MCP tools are now available to the model automatically.
MCP: Rust SDK
use hanzo::{
IsqBits, McpClientConfig, McpServerConfig, McpServerSource,
ModelBuilder, TextMessageRole, TextMessages,
};
let mcp_config = McpClientConfig {
servers: vec![McpServerConfig {
name: "Filesystem Tools".to_string(),
source: McpServerSource::Process {
command: "npx".to_string(),
args: vec![
"@modelcontextprotocol/server-filesystem".to_string(),
".".to_string(),
],
work_dir: None,
env: None,
},
..Default::default()
}],
..Default::default()
};
let model = ModelBuilder::new("Qwen/Qwen3-4B")
.with_auto_isq(IsqBits::Four)
.with_mcp_client(mcp_config)
.build()
.await?;
let messages = TextMessages::new()
.add_message(TextMessageRole::User, "List the files in the current directory.");
let response = model.send_chat_request(messages).await?;
println!("{}", response.choices[0].message.content.as_ref().unwrap());
MCP: Python SDK
import asyncio
import hanzo
mcp_config = hanzo.McpClientConfigPy(
servers=[
hanzo.McpServerConfigPy(
name="Filesystem Tools",
source=hanzo.McpServerSourcePy.Process(
command="npx",
args=["@modelcontextprotocol/server-filesystem", "."],
work_dir=None,
env=None,
),
)
]
)
runner = hanzo.Runner(
which=hanzo.Which.Plain(
model_id="Qwen/Qwen3-4B",
arch=hanzo.Architecture.Qwen3,
),
mcp_client_config=mcp_config,
)
request = hanzo.ChatCompletionRequest(
model="default",
messages=[{"role": "user", "content": "List the files in the current directory."}],
max_tokens=1000,
tool_choice="auto",
)
response = asyncio.run(runner.send_chat_completion_request(request))
print(response.choices[0].message.content)
MCP: HTTP API
Once the server is started with --mcp-config, MCP tools are available to all requests:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:1234/v1", api_key="placeholder")
response = client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": "List the files in the current directory."}],
tool_choice="auto",
extra_body={"max_tool_rounds": 5},
)
print(response.choices[0].message.content)
Multi-server and authentication
Connect to multiple MCP servers with different transports. Use tool_prefix to avoid naming conflicts and bearer_token for authentication.
let mcp_config = McpClientConfig {
servers: vec![
McpServerConfig {
name: "Filesystem Tools".to_string(),
source: McpServerSource::Process {
command: "npx".to_string(),
args: vec!["@modelcontextprotocol/server-filesystem".to_string(), ".".to_string()],
work_dir: None,
env: None,
},
tool_prefix: Some("fs".to_string()),
..Default::default()
},
McpServerConfig {
id: "hf_server".to_string(),
name: "Hugging Face MCP".to_string(),
source: McpServerSource::Http {
url: "https://hf.co/mcp".to_string(),
timeout_secs: Some(30),
headers: None,
},
tool_prefix: Some("hf".to_string()),
bearer_token: Some("hf_xxx".to_string()),
..Default::default()
},
McpServerConfig {
id: "ws_server".to_string(),
name: "WebSocket Server".to_string(),
source: McpServerSource::WebSocket {
url: "wss://api.example.com/mcp".to_string(),
timeout_secs: Some(30),
headers: None,
},
tool_prefix: Some("ws".to_string()),
..Default::default()
},
],
auto_register_tools: true,
tool_timeout_secs: Some(30),
max_concurrent_calls: Some(5),
};
For full MCP configuration details, see MCP/client.md, MCP/configuration.md, and MCP/transports.md.
Tool Dispatch URLs
The tool dispatch URL lets the server POST unhandled tool calls to your HTTP endpoint. This is how HTTP API users get server-side tool execution without registering SDK-level callbacks or MCP servers.
Security model
To prevent SSRF, tool_dispatch_url cannot be set per-request via the HTTP API. It can only be configured:
- Server-side via the
--tool-dispatch-urlCLI flag - Per-request in trusted Rust/Python SDK code
Setting up a tool server
The server POSTs to your endpoint with this format:
Request:
{"name": "get_weather", "arguments": {"city": "Tokyo"}}
Response:
{"content": "Sunny, 22C"}
The response can also be a bare string instead of a JSON object.
Here is a minimal Python tool server:
import json
from http.server import HTTPServer, BaseHTTPRequestHandler
def get_weather(args):
city = args.get("city", "unknown")
return f"Sunny, 22C in {city}"
TOOLS = {"get_weather": get_weather}
class ToolHandler(BaseHTTPRequestHandler):
def do_POST(self, *_args):
body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
name = body["name"]
result = TOOLS.get(name, lambda a: f"Unknown tool: {name}")(body.get("arguments", {}))
response = json.dumps({"content": result}).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(response)))
self.end_headers()
self.wfile.write(response)
HTTPServer(("localhost", 8787), ToolHandler).serve_forever()
Tool dispatch configuration
CLI:
hanzo serve -p 1234 \
--tool-dispatch-url http://localhost:8787/tools \
--max-tool-rounds 5 \
-m Qwen/Qwen3-4B
HTTP API (with the server started above):
from openai import OpenAI
client = OpenAI(base_url="http://localhost:1234/v1", api_key="placeholder")
response = client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": "What's the weather in Tokyo and London?"}],
tools=[{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city.",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
"strict": True,
},
}],
tool_choice="auto",
extra_body={"max_tool_rounds": 5},
)
print(response.choices[0].message.content)
Rust SDK (per-request, trusted code):
let request = RequestBuilder::from(messages)
.set_tools(tools)
.set_tool_choice(ToolChoice::Auto)
.set_tool_dispatch_url("http://localhost:8787/tools")
.set_max_tool_rounds(5);
Python SDK (per-request, trusted code):
request = ChatCompletionRequest(
messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
model="default",
tool_schemas=tools,
tool_choice=ToolChoice.Auto,
tool_dispatch_url="http://localhost:8787/tools",
max_tool_rounds=5,
)
Common Configuration
max_tool_rounds
Controls how many iterations the agentic loop can run. The loop terminates early if the model stops calling tools.
| Setting | Value |
|---|---|
| Default (unset) | 16 (safety cap) |
| CLI flag | --max-tool-rounds <N> |
| Per-request (HTTP) | "max_tool_rounds": N in request body |
| Per-request (Rust) | .set_max_tool_rounds(N) |
| Per-request (Python) | max_tool_rounds=N |
Per-request values override the server default.
Grammar enforcement and strict mode
When tools are provided, hanzo automatically constrains output to valid tool call syntax using llguidance. No configuration needed.
For stricter argument validation, set "strict": true on the function definition. This enforces the tool's JSON schema on the generated arguments: only declared property names, correct types, valid enum values, and required fields. See TOOL_CALLING.md for details.
Streaming support
All agentic features (web search, callbacks, MCP, dispatch URLs) support streaming. Tool call chunks are forwarded to the client during the loop so you can observe which tools are being called in real time.
Combining Features
You can enable multiple agentic features on the same model. The dispatch order governs which handler runs for each tool call.
Rust SDK:
let model = ModelBuilder::new("google/gemma-4-E4B-it")
.with_auto_isq(IsqBits::Four)
.with_search(SearchEmbeddingModel::default()) // Built-in web search
.with_tool_callback("local_db", Arc::new(db_cb)) // Custom callback
.with_mcp_client(mcp_config) // MCP servers
.build()
.await?;
CLI:
hanzo serve -p 1234 \
--enable-search \
--mcp-config mcp-config.json \
--tool-dispatch-url http://localhost:8787/tools \
--max-tool-rounds 10 \
-m google/gemma-4-E4B-it
Further Reading
| Topic | Link |
|---|---|
| Tool calling reference | TOOL_CALLING.md |
| Web search reference | WEB_SEARCH.md |
| MCP client reference | MCP/client.md |
| MCP configuration | MCP/configuration.md |
| MCP transports | MCP/transports.md |
| MCP advanced usage | MCP/advanced.md |
Examples
Rust SDK:
Python SDK:
HTTP API: