Files
skills/public/ml-llm-model-routing.html
zeekayandHanzo Dev 9e10c1c10e Each llms index points at the other two
hanzoskills.com/llms.txt now opens with the install line, the API
reference index and the company index, so an agent that lands on any
one of the three can find the rest.

robots.txt claimed 537 documents while llms.txt counts 920. The number
is gone from robots.txt; llms.txt derives it.

public/ is regenerated from scripts/site.py, which it had fallen
behind: the copy-prompt menu and agent bar were in the generator and
not in the committed pages. tests/site.mjs passes against the result.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-11 00:27:59 -07:00

731 lines
36 KiB
HTML

<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>ml-llm-model-routing</title>
<meta name="color-scheme" content="dark">
<style>
/* Committed to one visual world: a dark instrument panel. No light variant —
the dot field and the hairline rules only read on black, and a naive
inversion would wash both out. Every colour is painted explicitly so the page
never borrows a host background. */
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
:root{
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
--font-zen-pixel-square:'Zen Pixel Square',monospace;
--mono:var(--font-zen-mono);
--sans:var(--font-zen-sans);
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
--r:999px; --r-soft:14px;
}
*{box-sizing:border-box}
html{-webkit-text-size-adjust:100%}
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
/* The dot field. A halftone made of one repeating radial-gradient, faded out
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
/* No overflow clip here: the dot field is inset to the box already, and a clip
would cut off the copy-prompt menu. */
.hero{position:relative;border-bottom:1px solid var(--line)}
.hero::before{content:"";position:absolute;inset:0;
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
background-size:7px 7px;
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
pointer-events:none}
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
nav{border-bottom:1px solid var(--line)}
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
justify-content:space-between;align-items:center;gap:1rem}
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--ink);text-decoration:none;
display:inline-flex;align-items:center;gap:.6rem}
.brand i{font-style:normal;color:var(--faint)}
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
text-transform:uppercase;color:var(--soft);text-decoration:none}
nav a.up:hover{color:var(--ink)}
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
h1 em{font-style:normal;color:var(--faint)}
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
line-height:1.2;font-weight:400}
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
.btn.solid:hover{background:#d8d8d4}
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
/* Copy-prompt selector. Each row copies the prompt for its own target. */
.copy{position:relative;display:inline-flex}
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
padding:.4rem;display:flex;flex-direction:column;
box-shadow:0 12px 32px rgba(0,0,0,.75)}
.menu[hidden]{display:none}
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
white-space:nowrap}
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
.mi:hover .mark{color:var(--ink)}
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
letter-spacing:.1em;text-transform:uppercase}
/* Works-with bar: one scrolling row, name and mark per agent. */
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
display:flex;align-items:center}
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
-webkit-overflow-scrolling:touch;
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
.bar::-webkit-scrollbar{display:none}
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
text-decoration:none;background:var(--ground)}
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
.agent .mark{width:.95rem;height:.95rem}
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
p{margin:0 0 1rem;max-width:70ch}
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
li{margin:.22rem 0}
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
text-decoration-color:var(--line-hi)}
a:hover{text-decoration-color:var(--ink)}
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
display:block;overflow-x:auto}
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
.curl{display:block;background:var(--panel);border:1px solid var(--line);
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
white-space:pre;line-height:1.9}
.curl b{color:var(--ink);font-weight:400}
.curl i{font-style:normal;color:var(--faint)}
/* index */
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
#q:focus{outline:none;border-color:var(--soft)}
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
text-transform:uppercase;margin:0 0 2rem}
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
padding-bottom:.45rem}
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
border-radius:var(--r-soft);text-decoration:none}
.row:hover{background:#0a0a0a}
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
color:var(--faint);font-size:.85rem}
footer a{color:var(--soft)}
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
margin:0 0 2rem;font-size:.85rem}
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
border-bottom:1px solid var(--line);padding-bottom:.4rem}
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
.fmap a:hover{text-decoration:underline}
.fmap div{color:var(--faint);line-height:1.5}
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
@media (max-width:640px){
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
.btns{gap:.5rem}
.btn{padding:.75rem 1.15rem;font-size:.68rem}
.copy,.copy .btn{width:100%;justify-content:center}
.menu{left:0;right:0;min-width:0}
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
.strip b{border-right:0;padding-right:0}
.bar{width:100%}
}
</style>
<link rel="alternate" type="text/markdown" href="/ml-llm-model-routing.md">
<link rel="icon" href="/favicon.svg">
<link rel="canonical" href="https://hanzoskills.com/ml-llm-model-routing">
</head>
<!-- Reading this with a machine? Every page here is also plain markdown.
This one: https://hanzoskills.com/ml-llm-model-routing.md
Index: https://hanzoskills.com/llms.txt
Corpus: https://hanzoskills.com/all.md
Append .md to any path for the markdown twin. -->
<body>
<nav><div class="in">
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
<a class="up" href="/">All skills</a>
</div></nav>
<div class="wrap">
<h1>ml-llm-model-routing</h1><p class="lede">Intelligent LLM routing with RouteLLM, semantic routing, and cost optimization</p><h1>LLM Model Routing</h1>
<p><strong>Scope</strong>: Cost-effective LLM routing, RouteLLM framework, semantic routing, rule-based strategies, observability <strong>Lines</strong>: ~450 <strong>Last Updated</strong>: 2025-10-26</p>
<h2>When to Use This Skill</h2>
<p>Activate this skill when:</p>
<ul><li>Reducing LLM costs while maintaining quality (up to 85% reduction)</li><li>Building applications with mixed complexity queries</li><li>Implementing multi-model architectures for different use cases</li><li>Optimizing response latency and quality tradeoffs</li><li>Routing based on query intent, complexity, or domain</li><li>Instrumenting routing decisions with observability tools</li><li>Deploying production LLM systems with cost constraints</li></ul>
<h2>Core Concepts</h2>
<h3>The Routing Problem</h3>
<p><strong>Challenge</strong>: Strong models (GPT-4o, Claude 3.5 Sonnet) are expensive but not always necessary. Weak models (GPT-4o mini, Gemini Flash) are cheap but limited.</p>
<p><strong>Solution</strong>: Route queries dynamically to appropriate models based on complexity, intent, or requirements.</p>
<p><strong>Benefits</strong>:</p>
<ul><li>85%+ cost reduction on benchmarks (MT Bench)</li><li>45% cost reduction on MMLU</li><li>35% cost reduction on GSM8K</li><li>Maintained quality equivalent to always using strong models</li><li>Improved average latency for simple queries</li></ul>
<h3>Routing Strategies</h3>
<p><strong>1. Learning-Based Routing (RouteLLM)</strong>:</p>
<ul><li>Trained on preference data (human/GPT-4 judgments)</li><li>Predicts which model pair will satisfy quality thresholds</li><li>Generalizes across different strong/weak model pairs</li></ul>
<p><strong>2. Semantic Routing</strong>:</p>
<ul><li>Uses embeddings to match queries to specialized models</li><li>Routes based on semantic similarity to reference prompts</li><li>Enables domain-specific model assignment</li></ul>
<p><strong>3. Rule-Based Routing</strong>:</p>
<ul><li>Deterministic rules based on query metadata</li><li>Simple heuristics (length, keywords, patterns)</li><li>Fastest routing decision time</li></ul>
<p><strong>4. Hybrid Routing</strong>:</p>
<ul><li>Combines multiple strategies for robustness</li><li>Fallback chains for reliability</li><li>Context-aware routing logic</li></ul>
<h3>RouteLLM Framework</h3>
<p><strong>Architecture</strong>: Published at ICLR 2025, RouteLLM is an open-source framework from UC Berkeley and Anyscale for cost-effective LLM routing based on preference data.</p>
<p><strong>Key Components</strong>:</p>
<ul><li><strong>Router Models</strong>: Trained classifiers that predict model selection</li><li><strong>Preference Data</strong>: Training data from Chatbot Arena, human feedback</li><li><strong>Model Pairs</strong>: Strong model (GPT-4) + weak model (Mixtral, GPT-3.5)</li><li><strong>Threshold Tuning</strong>: Adjustable quality/cost tradeoff</li></ul>
<p><strong>Router Types</strong>:</p>
<ol><li><strong>sw_ranking</strong> (Elo-based):</li></ol>
<ul><li>Weighted Elo calculation for routing decisions</li><li>Fast, interpretable, no training required</li><li>Good baseline performance</li></ul>
<ol><li><strong>bert</strong> (BERT classifier):</li></ol>
<ul><li>BERT encoder trained on preference data</li><li>Balances speed and accuracy</li><li>Recommended for production use</li></ul>
<ol><li><strong>causal_llm</strong> (LLM classifier):</li></ol>
<ul><li>Fine-tuned LLM as router (e.g., Llama)</li><li>Highest accuracy but slower</li><li>Best for quality-critical applications</li></ul>
<ol><li><strong>matrix_factorization</strong>:</li></ol>
<ul><li>Low-rank factorization of preference matrix</li><li>Memory efficient</li><li>Suitable for large-scale deployments</li></ul>
<p><strong>Performance</strong>:</p>
<ul><li>Matches commercial routers (Martian, Not Diamond)</li><li>40%+ cheaper than commercial alternatives</li><li>Generalizes to new model pairs without retraining</li></ul>
<h3>Semantic Routing Architecture</h3>
<p><strong>Components</strong>:</p>
<ul><li><strong>Encoder</strong>: Embedding model (BERT, Sentence Transformers, OpenAI)</li><li><strong>Route Definitions</strong>: Named routes with example prompts</li><li><strong>Similarity Matching</strong>: Cosine similarity to route examples</li><li><strong>Threshold</strong>: Minimum similarity for route assignment</li></ul>
<p><strong>Modern Implementations (2025)</strong>:</p>
<ol><li><strong>vLLM Semantic Router</strong>:</li></ol>
<ul><li>Rust-based, high-concurrency routing</li><li>ModernBERT classifier for intent detection</li><li>Zero-copy inference with Hugging Face Candle</li><li>Mixture-of-Models (MoM) architecture</li></ul>
<ol><li><strong>Red Hat Semantic Processor</strong>:</li></ol>
<ul><li>Envoy ExtProc integration</li><li>Hybrid Rust + Golang implementation</li><li>Semantic cache with response reuse</li><li>~90% accuracy without LLM inference</li></ul>
<ol><li><strong>Aurelio Labs Semantic Router</strong>:</li></ol>
<ul><li>Multi-modal support (text, images)</li><li>Easy encoder integration (Cohere, OpenAI, HuggingFace)</li><li>Layer-based decision making</li></ul>
<p><strong>Advantages</strong>:</p>
<ul><li>Sub-millisecond routing decisions</li><li>No LLM call required for routing</li><li>Handles multi-domain applications</li><li>Supports caching and reuse</li></ul>
<hr>
<h2>Implementation Patterns</h2>
<h3>RouteLLM: Basic Setup</h3>
<pre><code class="lang-python">from routellm.controller import Controller
# Initialize router with model pair
router = Controller(
routers=[&quot;bert&quot;], # or [&quot;sw_ranking&quot;, &quot;causal_llm&quot;, &quot;mf&quot;]
strong_model=&quot;gpt-4-1106-preview&quot;,
weak_model=&quot;mixtral-8x7b-instruct-v0.1&quot;,
api_base=&quot;http://localhost:8000/v1&quot;, # Optional: custom endpoint
)
# Route a query
query = &quot;Explain quantum entanglement&quot;
response = router.route(
model=&quot;router-bert-0.5&quot;, # router-{type}-{threshold}
messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: query}]
)
print(f&quot;Routed to: {response.model}&quot;)
print(f&quot;Response: {response.choices[0].message.content}&quot;)</code></pre>
<h2>Hanzo SDK Integration</h2>
<h3>Automatic Model Routing with Hanzo</h3>
<p><strong>Instead of manual RouteLLM implementation</strong>, use Hanzo SDK&#x27;s built-in intelligent routing:</p>
<pre><code class="lang-python">from hanzo import Hanzo
# Initialize with hybrid mode (local + cloud with automatic routing)
hanzo = Hanzo(
inference_mode=&#x27;hybrid&#x27;, # Combines local + cloud models
auto_route=True, # Enable intelligent routing
cost_optimize=True # Optimize for cost
)
# Automatic routing based on query complexity
response = hanzo.chat.completions.create(
messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: query}],
# No need to specify model - Hanzo routes automatically
)
# Hanzo automatically:
# 1. Analyzes query complexity
# 2. Checks local models (Hanzo Node) first for privacy
# 3. Routes to cloud only if needed
# 4. Caches results for repeated queries
# 5. Tracks costs and latency
print(f&quot;Routed to: {response.model}&quot;)
print(f&quot;Cost: ${response.usage.cost:.4f}&quot;)
print(f&quot;Latency: {response.latency_ms}ms&quot;)</code></pre>
<h3>Hanzo Routing Strategies</h3>
<pre><code class="lang-python"># 1. Privacy-First Routing (local whenever possible)
hanzo = Hanzo(
inference_mode=&#x27;hybrid&#x27;,
prefer_local=True, # Use Hanzo Node first
node_url=&#x27;http://localhost:8080&#x27;
)
# Simple queries stay local (privacy + no cost)
response = hanzo.chat.completions.create(
messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;What is 2+2?&quot;}]
)
# Routed to: local-llama-3-8b
# Complex queries go to cloud
response = hanzo.chat.completions.create(
messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Analyze quantum cryptography&quot;}]
)
# Routed to: claude-3.5-sonnet
# 2. Cost-Optimized Routing
hanzo = Hanzo(
inference_mode=&#x27;cloud&#x27;,
cost_limit_per_request=0.01, # Max $0.01 per request
fallback_on_limit=True # Use cheaper model if exceeded
)
# 3. Latency-Optimized Routing
hanzo = Hanzo(
inference_mode=&#x27;hybrid&#x27;,
max_latency_ms=500, # Sub-500ms required
prefer_local=True # Local models are fastest
)
# 4. Quality-Focused Routing
hanzo = Hanzo(
inference_mode=&#x27;cloud&#x27;,
quality_tier=&#x27;high&#x27;, # Only top-tier models
models=[&#x27;gpt-4&#x27;, &#x27;claude-3.5-sonnet&#x27;]
)</code></pre>
<h3>Hanzo MCP Tool for Agents</h3>
<p>Expose Hanzo routing as MCP tool for agentic workflows:</p>
<pre><code class="lang-typescript">import { MCPServer, Tool } from &#x27;@hanzo/mcp&#x27;
import { Hanzo } from &#x27;hanzo&#x27; // Python SDK
const hanzo = new Hanzo({
inference_mode: &#x27;hybrid&#x27;,
auto_route: true
})
const hanzoRouteTool: Tool = {
name: &#x27;hanzo_route_and_infer&#x27;,
description: &#x27;Automatically route query to best model and generate response&#x27;,
parameters: {
prompt: { type: &#x27;string&#x27;, required: true },
strategy: {
type: &#x27;string&#x27;,
enum: [&#x27;cost&#x27;, &#x27;latency&#x27;, &#x27;quality&#x27;, &#x27;privacy&#x27;],
default: &#x27;cost&#x27;
}
},
async execute({ prompt, strategy }) {
const response = await hanzo.chat.completions.create({
messages: [{ role: &#x27;user&#x27;, content: prompt }],
routing_strategy: strategy
})
return {
text: response.choices[0].message.content,
model: response.model,
cost: response.usage.cost,
latency_ms: response.latency_ms
}
}
}</code></pre>
<h3>Hanzo vs Manual RouteLLM</h3>
<p>| Feature | Manual RouteLLM | Hanzo SDK | |---------|-----------------|-----------| | <strong>Setup</strong> | 50+ lines config | 3 lines | | <strong>Local Models</strong> | Not supported | Built-in (Hanzo Node) | | <strong>Privacy</strong> | Manual logic | Automatic (local-first) | | <strong>Cost Tracking</strong> | Custom implementation | Built-in | | <strong>Caching</strong> | Manual | Automatic | | <strong>MCP Integration</strong> | Custom wrapper | Native support | | <strong>Fallback</strong> | Manual chains | Automatic | | <strong>Monitoring</strong> | Custom Prometheus | Built-in dashboard |</p>
<p><strong>When to Use What</strong>:</p>
<ul><li><strong>Hanzo SDK</strong>: Most use cases (simpler, more features)</li><li><strong>Manual RouteLLM</strong>: Research, custom training, non-Hanzo stack</li></ul>
<h3>RouteLLM: Threshold Tuning</h3>
<pre><code class="lang-python"># Threshold controls quality/cost tradeoff
# Higher threshold = more weak model usage = lower cost
# Lower threshold = more strong model usage = higher quality
thresholds = [0.3, 0.5, 0.7, 0.9]
for threshold in thresholds:
router_name = f&quot;router-bert-{threshold}&quot;
response = router.route(
model=router_name,
messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Simple math: 2+2&quot;}]
)
print(f&quot;Threshold {threshold}: {response.model}&quot;)
# Results:
# 0.3 -&gt; gpt-4 (conservative, high quality)
# 0.5 -&gt; mixtral (balanced)
# 0.7 -&gt; mixtral (aggressive cost savings)
# 0.9 -&gt; mixtral (maximum savings)</code></pre>
<h3>RouteLLM: Evaluation Mode</h3>
<pre><code class="lang-python">import pandas as pd
from routellm.routers import ROUTER_CLS
# Load router for evaluation
router_cls = ROUTER_CLS[&quot;bert&quot;]
router = router_cls(
strong_model_name=&quot;gpt-4-1106-preview&quot;,
weak_model_name=&quot;mixtral-8x7b-instruct-v0.1&quot;
)
# Evaluate on dataset
test_queries = [
&quot;What is 2+2?&quot;,
&quot;Explain the implications of quantum computing on cryptography&quot;,
&quot;Name the capital of France&quot;,
&quot;Analyze the socioeconomic factors in the French Revolution&quot;
]
decisions = []
for query in test_queries:
# Get routing decision (0-1 score)
score = router.route(query)
selected = &quot;strong&quot; if score &lt; 0.5 else &quot;weak&quot;
decisions.append({
&quot;query&quot;: query[:50],
&quot;score&quot;: score,
&quot;model&quot;: selected
})
df = pd.DataFrame(decisions)
print(df)
# Calculate cost savings
weak_ratio = (df[&quot;model&quot;] == &quot;weak&quot;).mean()
print(f&quot;Weak model usage: {weak_ratio:.1%}&quot;)
print(f&quot;Estimated savings: {weak_ratio * 0.95:.1%}&quot;) # 95% cheaper</code></pre>
<h3>Semantic Routing with Aurelio</h3>
<pre><code class="lang-python">from semantic_router import Route, RouteLayer
from semantic_router.encoders import OpenAIEncoder
# Define routes with examples
code_route = Route(
name=&quot;code&quot;,
utterances=[
&quot;How do I write a function in Python?&quot;,
&quot;Debug this code snippet&quot;,
&quot;Explain this algorithm&quot;,
&quot;Write a sorting function&quot;
]
)
general_route = Route(
name=&quot;general&quot;,
utterances=[
&quot;What is the weather today?&quot;,
&quot;Tell me a joke&quot;,
&quot;What&#x27;s the capital of France?&quot;,
&quot;Who won the World Series?&quot;
]
)
reasoning_route = Route(
name=&quot;reasoning&quot;,
utterances=[
&quot;Solve this complex math problem&quot;,
&quot;Analyze the implications of this policy&quot;,
&quot;Compare and contrast these approaches&quot;,
&quot;Provide a detailed ethical analysis&quot;
]
)
# Initialize encoder and layer
encoder = OpenAIEncoder()
route_layer = RouteLayer(
encoder=encoder,
routes=[code_route, general_route, reasoning_route]
)
# Route queries
queries = [
&quot;How do I reverse a string in JavaScript?&quot;,
&quot;What&#x27;s 2+2?&quot;,
&quot;Explain the philosophical implications of AI consciousness&quot;
]
for query in queries:
route = route_layer(query)
print(f&quot;Query: {query[:50]}&quot;)
print(f&quot;Route: {route.name if route else &#x27;fallback&#x27;}\n&quot;)
# Map routes to models
model_map = {
&quot;code&quot;: &quot;claude-3.5-sonnet&quot;, # Best at coding
&quot;general&quot;: &quot;gpt-4o-mini&quot;, # Fast and cheap
&quot;reasoning&quot;: &quot;gpt-4o&quot;, # Strong reasoning
None: &quot;gpt-4o-mini&quot; # Default fallback
}
def route_and_call(query: str):
route = route_layer(query)
model = model_map.get(route.name if route else None)
print(f&quot;Routing to: {model}&quot;)
# Call selected model...</code></pre>
<h3>Rule-Based Routing</h3>
<pre><code class="lang-python">import re
from typing import Literal
ModelType = Literal[&quot;strong&quot;, &quot;weak&quot;, &quot;specialized&quot;]
class RuleBasedRouter:
def __init__(
self,
strong_model: str = &quot;gpt-4o&quot;,
weak_model: str = &quot;gpt-4o-mini&quot;,
code_model: str = &quot;claude-3.5-sonnet&quot;
):
self.strong = strong_model
self.weak = weak_model
self.code = code_model
def route(self, query: str, context: dict = None) -&gt; str:
&quot;&quot;&quot;Route based on heuristics&quot;&quot;&quot;
query_lower = query.lower()
# Rule 1: Code-related queries -&gt; specialized model
code_keywords = [&quot;code&quot;, &quot;function&quot;, &quot;debug&quot;, &quot;implement&quot;, &quot;algorithm&quot;]
if any(kw in query_lower for kw in code_keywords):
return self.code
# Rule 2: Simple factual queries -&gt; weak model
simple_patterns = [
r&quot;^what is \w+\?$&quot;,
r&quot;^who is \w+\?$&quot;,
r&quot;^when did \w+&quot;,
r&quot;^\d+[\+\-\*/]\d+&quot;, # Simple math
]
if any(re.match(p, query_lower) for p in simple_patterns):
return self.weak
# Rule 3: Long, complex queries -&gt; strong model
if len(query) &gt; 500:
return self.strong
# Rule 4: Reasoning keywords -&gt; strong model
reasoning_keywords = [
&quot;analyze&quot;, &quot;implications&quot;, &quot;compare&quot;, &quot;evaluate&quot;,
&quot;critique&quot;, &quot;synthesize&quot;, &quot;ethical&quot;, &quot;philosophical&quot;
]
if any(kw in query_lower for kw in reasoning_keywords):
return self.strong
# Rule 5: Context indicates multi-turn conversation -&gt; strong model
if context and context.get(&quot;turn_count&quot;, 0) &gt; 3:
return self.strong
# Default: weak model
return self.weak
# Usage
router = RuleBasedRouter()
queries = [
&quot;Write a binary search function&quot;, # -&gt; code model
&quot;What is the capital of France?&quot;, # -&gt; weak
&quot;2+2&quot;, # -&gt; weak
&quot;Analyze the ethical implications of AI in healthcare&quot; # -&gt; strong
]
for q in queries:
model = router.route(q)
print(f&quot;{q[:50]:50} -&gt; {model}&quot;)</code></pre>
<h3>Hybrid Routing with Fallback</h3>
<pre><code class="lang-python">from typing import Optional
import openai
class HybridRouter:
def __init__(self):
self.semantic_router = route_layer # From earlier example
self.rule_router = RuleBasedRouter()
def route(
self,
query: str,
use_semantic: bool = True,
use_rules: bool = True
) -&gt; str:
&quot;&quot;&quot;Hybrid routing with fallback chain&quot;&quot;&quot;
# Try semantic routing first (most accurate)
if use_semantic:
route = self.semantic_router(query)
if route and route.name:
return model_map.get(route.name)
# Fallback to rule-based routing
if use_rules:
return self.rule_router.route(query)
# Final fallback
return &quot;gpt-4o-mini&quot;
def route_with_retry(self, query: str, max_retries: int = 2):
&quot;&quot;&quot;Route with fallback on errors&quot;&quot;&quot;
models = [
self.route(query), # Primary
&quot;gpt-4o&quot;, # Fallback 1
&quot;claude-3.5-sonnet&quot; # Fallback 2
]
for i, model in enumerate(models[:max_retries + 1]):
try:
response = openai.ChatCompletion.create(
model=model,
messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: query}],
timeout=30
)
return {
&quot;model&quot;: model,
&quot;response&quot;: response.choices[0].message.content,
&quot;attempt&quot;: i + 1
}
except Exception as e:
print(f&quot;Model {model} failed: {e}&quot;)
if i == len(models) - 1:
raise
continue
# Usage
hybrid = HybridRouter()
result = hybrid.route_with_retry(
&quot;Explain quantum entanglement to a 10-year-old&quot;
)
print(f&quot;Used model: {result[&#x27;model&#x27;]} (attempt {result[&#x27;attempt&#x27;]})&quot;)</code></pre>
<h3>Arize Phoenix Observability</h3>
<pre><code class="lang-python">import phoenix as px
from phoenix.trace import using_project
from opentelemetry import trace
# Start Phoenix
px.launch_app()
# Instrument routing
tracer = trace.get_tracer(__name__)
@using_project(&quot;llm-routing&quot;)
def route_and_call(query: str):
with tracer.start_as_current_span(&quot;routing_decision&quot;) as span:
# Route
router = HybridRouter()
selected_model = router.route(query)
# Log routing decision
span.set_attribute(&quot;query&quot;, query)
span.set_attribute(&quot;selected_model&quot;, selected_model)
span.set_attribute(&quot;routing_strategy&quot;, &quot;hybrid&quot;)
# Call model
with tracer.start_as_current_span(&quot;llm_call&quot;) as call_span:
call_span.set_attribute(&quot;model&quot;, selected_model)
response = openai.ChatCompletion.create(
model=selected_model,
messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: query}]
)
# Log metrics
call_span.set_attribute(&quot;tokens_used&quot;, response.usage.total_tokens)
call_span.set_attribute(&quot;cost_usd&quot;, calculate_cost(response, selected_model))
return response
# View routing patterns in Phoenix UI
# Navigate to http://localhost:6006</code></pre>
<hr>
<h2>Quick Reference</h2>
<h3>Model Strengths by Category (2025)</h3>
<pre><code>Category | Best Models | Use Case
--------------|--------------------------|----------------------------------
Reasoning | GPT-4o, Claude 3.5/4 | Complex analysis, multi-step logic
Coding | Claude 3.5 Sonnet | Code generation, debugging
Speed | Gemini 2.5 Flash | 370 tok/s, real-time applications
Cost | GPT-4o mini, Gemini Flash| Simple queries, high volume
Multimodal | GPT-4o, Gemini 2.5 | Vision + text tasks
Long Context | Claude 3.5 (200k) | Document analysis, large codebases</code></pre>
<h3>Cost Comparison (per million tokens)</h3>
<pre><code>Model | Input | Output | Use Case
-------------------------|----------|----------|------------------
GPT-4o | $5.00 | $20.00 | General strong
GPT-4o mini | $0.15 | $0.60 | General weak
Claude 3.5 Sonnet | $3.00 | $15.00 | Coding strong
Claude 3 Haiku | $0.25 | $1.25 | Fast weak
Gemini 2.5 Flash | $0.10 | $0.40 | Cheapest strong
DeepSeek V3 | $0.30 | $1.00 | Open-source alt</code></pre>
<h3>Routing Decision Tree</h3>
<pre><code>Query characteristics -&gt; Recommended strategy
────────────────────────────────────────────────
High volume, mixed complexity -&gt; RouteLLM
Domain-specific (code, legal) -&gt; Semantic routing
Latency critical -&gt; Rule-based
Quality critical -&gt; Always strong model
Budget constrained -&gt; RouteLLM threshold 0.7+
Multi-turn conversations -&gt; Session-aware routing</code></pre>
<hr>
<h2>Anti-Patterns</h2>
<p>❌ <strong>Always using the strongest model</strong>: Wastes 10-50x cost on simple queries ✅ Route based on complexity; use weak models for 60-80% of queries</p>
<p>❌ <strong>No observability on routing decisions</strong>: Can&#x27;t optimize or debug ✅ Use Arize Phoenix to trace routing logic and model performance</p>
<p>❌ <strong>Static routing rules that don&#x27;t adapt</strong>: Performance degrades over time ✅ Continuously evaluate routing accuracy; retrain or adjust thresholds</p>
<p>❌ <strong>Ignoring latency in routing overhead</strong>: Routing adds 100ms+ delay ✅ Use cached semantic embeddings; pre-compute route assignments</p>
<p>❌ <strong>Single point of failure in routing</strong>: Entire system fails if router breaks ✅ Implement fallback chains; default to mid-tier model on router errors</p>
<p>❌ <strong>Not tracking cost savings</strong>: Can&#x27;t justify routing complexity ✅ Log model selection, token usage, and calculate cost deltas</p>
<p>❌ <strong>Over-engineering for simple use cases</strong>: Rule-based is fine for 2-3 models ✅ Start simple; add RouteLLM/semantic routing when scaling to 5+ models</p>
<p>❌ <strong>Training routers on outdated model pairs</strong>: New models shift capabilities ✅ Regularly re-evaluate with current model benchmarks (GPT-4o, Claude 3.5)</p>
<hr>
<h2>Related Skills</h2>
<ul><li><code>llm-model-selection.md</code> - Choosing models for specific capabilities</li><li><code>multi-model-orchestration.md</code> - Coordinating multiple models in pipelines</li><li><code>modal-web-endpoints.md</code> - Deploying routing logic as FastAPI endpoints</li><li><code>api-rate-limiting.md</code> - Rate limiting across multiple model providers</li><li><code>observability-distributed-tracing.md</code> - Tracing multi-model systems</li><li><code>api-error-handling.md</code> - Handling provider failures and fallbacks</li></ul>
<hr>
<p><strong>Last Updated</strong>: 2025-10-26 <strong>Format Version</strong>: 1.0 (Atomic)</p>
<footer>
<div class="fmap">
<div><b>Build</b>
<a href="https://hanzo.app">hanzo.app</a> app builder
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
<div><b>Run</b>
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
<div><b>Account</b>
<a href="https://hanzo.id">hanzo.id</a> sign in
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
<div><b>Company</b>
<a href="https://hanzo.ai">hanzo.ai</a> canonical
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
</div>
<div class="fend"><a href="/ml-llm-model-routing.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
</footer>
</div></body></html>