Files
engine/examples/server/anthropic_tool_calling.py
zeekayandHanzo Dev 42e34c2d41 refactor: name only Hanzo Engine in code, comments and docs
Every mention of another inference engine is rewritten to say what our code
does, or the math it computes. Reference values are "the reference value
from the published config" or "the recorded reference output", line
citations into another engine's source become the rule itself, comparisons
with other engines are dropped, and asides about other engines' bugs go.

Identifiers follow:
- CUDA: hanzo-paged-attn's `namespace vllm` is `pagedattn`, hanzo-engine's
  MoE helpers `moe_dtype` and `moe_detail`, the rotary kernels'
  `hanzo_rotary`; the shuffle/ldg macros are HANZO_*.
- GPTQ: the shuffled-layout path (`reconstruct_exllama`, `use_exllama`) is
  `reconstruct_shuffled` / `is_shuffled`, on the kernel and the FFI side.
- Tests: `*_matches_vllm` and friends name what they compare against (the
  reference fixture, the tile rule).
- MiniCPM-o's vision-LLM embedding is `vlm_embedding`; the rotary custom op
  is `rotary-embedding`.
- hanzo-bench's board scores ours against a "rival" whose name comes from
  the manifest: `rival_<tag>.json`, `--rival*` manifest flags, `rival t/s`
  columns; dossier.sh and spec_decode.sh take `--rival-bench`.
- install.ps1 installs this repository's hanzo-cli, as install.sh does,
  with HANZO_INSTALL_NCCL / HANZO_INSTALL_NO_NCCL.
- bench_http.py reads only the hanzo: metric families.

Kept, because they are what they are: LICENSE, LICENSE-APACHE, NOTICE and
tools/upstream-map.sh; each ported file's copyright/licence/provenance
header; Cargo.toml dependency names; data identifiers (Hugging Face repo
IDs, fixture file names and fixture metadata keys the tests read, on-disk
paths, executable names the GPU and engine guards match on a host); model
and format names.

LLM.md says the rule once, under "One name: Hanzo Engine".


Co-authored-by: Hanzo Dev <[email protected]>
2026-10-16 20:59:50 -07:00

107 lines
2.6 KiB
Python

"""
Basic client-side tool calling with the Anthropic Messages API.
Run the server:
hanzo-engine serve -p 1234 --quant 4 -m Qwen/Qwen3-4B
Then run:
python3 examples/server/anthropic_tool_calling.py
"""
import json
import os
import urllib.request
BASE_URL = os.environ.get("HANZO_BASE_URL", "http://localhost:1234")
API_KEY = os.environ.get("ANTHROPIC_API_KEY", "local")
def post(path, payload):
request = urllib.request.Request(
f"{BASE_URL}{path}",
data=json.dumps(payload).encode("utf-8"),
headers={
"content-type": "application/json",
"x-api-key": API_KEY,
"anthropic-version": "2023-06-01",
},
method="POST",
)
with urllib.request.urlopen(request) as response:
return json.loads(response.read().decode("utf-8"))
def get_weather(city):
data = {"tokyo": "Sunny, 22C", "london": "Cloudy, 15C"}
return data.get(city.lower(), f"Unknown city: {city}")
tools = [
{
"name": "get_weather",
"description": "Get the current weather for a city.",
"input_schema": {
"type": "object",
"properties": {
"city": {"type": "string", "description": "City name"},
},
"required": ["city"],
},
}
]
messages = [{"role": "user", "content": "What's the weather in Tokyo?"}]
# Step 1: Model generates a tool call.
first = post(
"/v1/messages",
{
"model": "default",
"max_tokens": 256,
"messages": messages,
"tools": tools,
"tool_choice": {"type": "auto"},
},
)
tool_uses = [block for block in first["content"] if block["type"] == "tool_use"]
if not tool_uses:
raise RuntimeError(f"Expected a tool_use block, got: {json.dumps(first, indent=2)}")
tool_use = tool_uses[0]
print(f"Model wants to call: {tool_use['name']}")
# Step 2: Execute the tool locally.
result = get_weather(tool_use["input"]["city"])
print(f"Tool result: {result}")
# Step 3: Send the result back.
messages.append({"role": "assistant", "content": first["content"]})
messages.append(
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": tool_use["id"],
"content": result,
}
],
}
)
# Step 4: Model produces the final answer.
second = post(
"/v1/messages",
{
"model": "default",
"max_tokens": 256,
"messages": messages,
"tools": tools,
},
)
for block in second["content"]:
if block["type"] == "text":
print(block["text"])