mirror of
https://github.com/hanzoai/skills.git
synced 2026-11-19 00:47:36 +00:00
hanzoskills.com/llms.txt now opens with the install line, the API reference index and the company index, so an agent that lands on any one of the three can find the rest. robots.txt claimed 537 documents while llms.txt counts 920. The number is gone from robots.txt; llms.txt derives it. public/ is regenerated from scripts/site.py, which it had fallen behind: the copy-prompt menu and agent bar were in the generator and not in the committed pages. tests/site.mjs passes against the result. Co-authored-by: Hanzo Dev <[email protected]>
731 lines
36 KiB
HTML
731 lines
36 KiB
HTML
<!doctype html>
|
|
<html lang="en"><head>
|
|
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
<title>ml-llm-model-routing</title>
|
|
<meta name="color-scheme" content="dark">
|
|
<style>
|
|
/* Committed to one visual world: a dark instrument panel. No light variant —
|
|
the dot field and the hairline rules only read on black, and a naive
|
|
inversion would wash both out. Every colour is painted explicitly so the page
|
|
never borrows a host background. */
|
|
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
|
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
|
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
|
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
|
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
|
|
|
|
:root{
|
|
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
|
|
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
|
|
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
|
|
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
|
|
--font-zen-pixel-square:'Zen Pixel Square',monospace;
|
|
--mono:var(--font-zen-mono);
|
|
--sans:var(--font-zen-sans);
|
|
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
|
|
--r:999px; --r-soft:14px;
|
|
}
|
|
*{box-sizing:border-box}
|
|
html{-webkit-text-size-adjust:100%}
|
|
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
|
|
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
|
|
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
|
|
|
|
/* The dot field. A halftone made of one repeating radial-gradient, faded out
|
|
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
|
|
/* No overflow clip here: the dot field is inset to the box already, and a clip
|
|
would cut off the copy-prompt menu. */
|
|
.hero{position:relative;border-bottom:1px solid var(--line)}
|
|
.hero::before{content:"";position:absolute;inset:0;
|
|
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
|
|
background-size:7px 7px;
|
|
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
|
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
|
pointer-events:none}
|
|
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
|
|
|
|
nav{border-bottom:1px solid var(--line)}
|
|
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
|
|
justify-content:space-between;align-items:center;gap:1rem}
|
|
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
|
|
text-transform:uppercase;color:var(--ink);text-decoration:none;
|
|
display:inline-flex;align-items:center;gap:.6rem}
|
|
.brand i{font-style:normal;color:var(--faint)}
|
|
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
|
|
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
|
|
text-transform:uppercase;color:var(--soft);text-decoration:none}
|
|
nav a.up:hover{color:var(--ink)}
|
|
|
|
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
|
|
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
|
|
h1 em{font-style:normal;color:var(--faint)}
|
|
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
|
|
|
|
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
|
|
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
|
|
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
|
|
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
|
|
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
|
|
line-height:1.2;font-weight:400}
|
|
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
|
|
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
|
|
.btn.solid:hover{background:#d8d8d4}
|
|
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
|
|
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
|
|
|
|
/* Copy-prompt selector. Each row copies the prompt for its own target. */
|
|
.copy{position:relative;display:inline-flex}
|
|
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
|
|
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
|
|
padding:.4rem;display:flex;flex-direction:column;
|
|
box-shadow:0 12px 32px rgba(0,0,0,.75)}
|
|
.menu[hidden]{display:none}
|
|
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
|
|
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
|
|
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
|
|
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
|
|
white-space:nowrap}
|
|
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
|
|
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
|
|
.mi:hover .mark{color:var(--ink)}
|
|
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
|
|
letter-spacing:.1em;text-transform:uppercase}
|
|
|
|
/* Works-with bar: one scrolling row, name and mark per agent. */
|
|
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
|
|
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
|
|
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
|
|
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
|
|
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
|
|
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
|
|
display:flex;align-items:center}
|
|
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
|
|
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
|
|
-webkit-overflow-scrolling:touch;
|
|
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
|
|
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
|
|
.bar::-webkit-scrollbar{display:none}
|
|
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
|
|
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
|
|
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
|
|
text-decoration:none;background:var(--ground)}
|
|
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
|
|
.agent .mark{width:.95rem;height:.95rem}
|
|
|
|
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
|
|
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
|
|
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
|
|
p{margin:0 0 1rem;max-width:70ch}
|
|
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
|
|
li{margin:.22rem 0}
|
|
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
|
|
text-decoration-color:var(--line-hi)}
|
|
a:hover{text-decoration-color:var(--ink)}
|
|
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
|
|
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
|
|
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
|
|
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
|
|
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
|
|
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
|
|
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
|
|
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
|
|
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
|
|
display:block;overflow-x:auto}
|
|
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
|
|
|
|
.curl{display:block;background:var(--panel);border:1px solid var(--line);
|
|
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
|
|
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
|
|
white-space:pre;line-height:1.9}
|
|
.curl b{color:var(--ink);font-weight:400}
|
|
.curl i{font-style:normal;color:var(--faint)}
|
|
|
|
/* index */
|
|
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
|
|
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
|
|
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
|
|
#q:focus{outline:none;border-color:var(--soft)}
|
|
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
|
|
text-transform:uppercase;margin:0 0 2rem}
|
|
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
|
|
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
|
|
padding-bottom:.45rem}
|
|
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
|
|
border-radius:var(--r-soft);text-decoration:none}
|
|
.row:hover{background:#0a0a0a}
|
|
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
|
|
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
|
|
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
|
|
color:var(--faint);font-size:.85rem}
|
|
footer a{color:var(--soft)}
|
|
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
|
|
margin:0 0 2rem;font-size:.85rem}
|
|
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
|
|
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
|
|
border-bottom:1px solid var(--line);padding-bottom:.4rem}
|
|
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
|
|
.fmap a:hover{text-decoration:underline}
|
|
.fmap div{color:var(--faint);line-height:1.5}
|
|
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
|
|
|
|
@media (max-width:640px){
|
|
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
|
|
.btns{gap:.5rem}
|
|
.btn{padding:.75rem 1.15rem;font-size:.68rem}
|
|
.copy,.copy .btn{width:100%;justify-content:center}
|
|
.menu{left:0;right:0;min-width:0}
|
|
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
|
|
.strip b{border-right:0;padding-right:0}
|
|
.bar{width:100%}
|
|
}
|
|
</style>
|
|
<link rel="alternate" type="text/markdown" href="/ml-llm-model-routing.md">
|
|
|
|
<link rel="icon" href="/favicon.svg">
|
|
<link rel="canonical" href="https://hanzoskills.com/ml-llm-model-routing">
|
|
</head>
|
|
<!-- Reading this with a machine? Every page here is also plain markdown.
|
|
This one: https://hanzoskills.com/ml-llm-model-routing.md
|
|
Index: https://hanzoskills.com/llms.txt
|
|
Corpus: https://hanzoskills.com/all.md
|
|
Append .md to any path for the markdown twin. -->
|
|
<body>
|
|
<nav><div class="in">
|
|
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
|
|
<a class="up" href="/">All skills</a>
|
|
</div></nav>
|
|
|
|
<div class="wrap">
|
|
<h1>ml-llm-model-routing</h1><p class="lede">Intelligent LLM routing with RouteLLM, semantic routing, and cost optimization</p><h1>LLM Model Routing</h1>
|
|
<p><strong>Scope</strong>: Cost-effective LLM routing, RouteLLM framework, semantic routing, rule-based strategies, observability <strong>Lines</strong>: ~450 <strong>Last Updated</strong>: 2025-10-26</p>
|
|
<h2>When to Use This Skill</h2>
|
|
<p>Activate this skill when:</p>
|
|
<ul><li>Reducing LLM costs while maintaining quality (up to 85% reduction)</li><li>Building applications with mixed complexity queries</li><li>Implementing multi-model architectures for different use cases</li><li>Optimizing response latency and quality tradeoffs</li><li>Routing based on query intent, complexity, or domain</li><li>Instrumenting routing decisions with observability tools</li><li>Deploying production LLM systems with cost constraints</li></ul>
|
|
<h2>Core Concepts</h2>
|
|
<h3>The Routing Problem</h3>
|
|
<p><strong>Challenge</strong>: Strong models (GPT-4o, Claude 3.5 Sonnet) are expensive but not always necessary. Weak models (GPT-4o mini, Gemini Flash) are cheap but limited.</p>
|
|
<p><strong>Solution</strong>: Route queries dynamically to appropriate models based on complexity, intent, or requirements.</p>
|
|
<p><strong>Benefits</strong>:</p>
|
|
<ul><li>85%+ cost reduction on benchmarks (MT Bench)</li><li>45% cost reduction on MMLU</li><li>35% cost reduction on GSM8K</li><li>Maintained quality equivalent to always using strong models</li><li>Improved average latency for simple queries</li></ul>
|
|
<h3>Routing Strategies</h3>
|
|
<p><strong>1. Learning-Based Routing (RouteLLM)</strong>:</p>
|
|
<ul><li>Trained on preference data (human/GPT-4 judgments)</li><li>Predicts which model pair will satisfy quality thresholds</li><li>Generalizes across different strong/weak model pairs</li></ul>
|
|
<p><strong>2. Semantic Routing</strong>:</p>
|
|
<ul><li>Uses embeddings to match queries to specialized models</li><li>Routes based on semantic similarity to reference prompts</li><li>Enables domain-specific model assignment</li></ul>
|
|
<p><strong>3. Rule-Based Routing</strong>:</p>
|
|
<ul><li>Deterministic rules based on query metadata</li><li>Simple heuristics (length, keywords, patterns)</li><li>Fastest routing decision time</li></ul>
|
|
<p><strong>4. Hybrid Routing</strong>:</p>
|
|
<ul><li>Combines multiple strategies for robustness</li><li>Fallback chains for reliability</li><li>Context-aware routing logic</li></ul>
|
|
<h3>RouteLLM Framework</h3>
|
|
<p><strong>Architecture</strong>: Published at ICLR 2025, RouteLLM is an open-source framework from UC Berkeley and Anyscale for cost-effective LLM routing based on preference data.</p>
|
|
<p><strong>Key Components</strong>:</p>
|
|
<ul><li><strong>Router Models</strong>: Trained classifiers that predict model selection</li><li><strong>Preference Data</strong>: Training data from Chatbot Arena, human feedback</li><li><strong>Model Pairs</strong>: Strong model (GPT-4) + weak model (Mixtral, GPT-3.5)</li><li><strong>Threshold Tuning</strong>: Adjustable quality/cost tradeoff</li></ul>
|
|
<p><strong>Router Types</strong>:</p>
|
|
<ol><li><strong>sw_ranking</strong> (Elo-based):</li></ol>
|
|
<ul><li>Weighted Elo calculation for routing decisions</li><li>Fast, interpretable, no training required</li><li>Good baseline performance</li></ul>
|
|
<ol><li><strong>bert</strong> (BERT classifier):</li></ol>
|
|
<ul><li>BERT encoder trained on preference data</li><li>Balances speed and accuracy</li><li>Recommended for production use</li></ul>
|
|
<ol><li><strong>causal_llm</strong> (LLM classifier):</li></ol>
|
|
<ul><li>Fine-tuned LLM as router (e.g., Llama)</li><li>Highest accuracy but slower</li><li>Best for quality-critical applications</li></ul>
|
|
<ol><li><strong>matrix_factorization</strong>:</li></ol>
|
|
<ul><li>Low-rank factorization of preference matrix</li><li>Memory efficient</li><li>Suitable for large-scale deployments</li></ul>
|
|
<p><strong>Performance</strong>:</p>
|
|
<ul><li>Matches commercial routers (Martian, Not Diamond)</li><li>40%+ cheaper than commercial alternatives</li><li>Generalizes to new model pairs without retraining</li></ul>
|
|
<h3>Semantic Routing Architecture</h3>
|
|
<p><strong>Components</strong>:</p>
|
|
<ul><li><strong>Encoder</strong>: Embedding model (BERT, Sentence Transformers, OpenAI)</li><li><strong>Route Definitions</strong>: Named routes with example prompts</li><li><strong>Similarity Matching</strong>: Cosine similarity to route examples</li><li><strong>Threshold</strong>: Minimum similarity for route assignment</li></ul>
|
|
<p><strong>Modern Implementations (2025)</strong>:</p>
|
|
<ol><li><strong>vLLM Semantic Router</strong>:</li></ol>
|
|
<ul><li>Rust-based, high-concurrency routing</li><li>ModernBERT classifier for intent detection</li><li>Zero-copy inference with Hugging Face Candle</li><li>Mixture-of-Models (MoM) architecture</li></ul>
|
|
<ol><li><strong>Red Hat Semantic Processor</strong>:</li></ol>
|
|
<ul><li>Envoy ExtProc integration</li><li>Hybrid Rust + Golang implementation</li><li>Semantic cache with response reuse</li><li>~90% accuracy without LLM inference</li></ul>
|
|
<ol><li><strong>Aurelio Labs Semantic Router</strong>:</li></ol>
|
|
<ul><li>Multi-modal support (text, images)</li><li>Easy encoder integration (Cohere, OpenAI, HuggingFace)</li><li>Layer-based decision making</li></ul>
|
|
<p><strong>Advantages</strong>:</p>
|
|
<ul><li>Sub-millisecond routing decisions</li><li>No LLM call required for routing</li><li>Handles multi-domain applications</li><li>Supports caching and reuse</li></ul>
|
|
<hr>
|
|
<h2>Implementation Patterns</h2>
|
|
<h3>RouteLLM: Basic Setup</h3>
|
|
<pre><code class="lang-python">from routellm.controller import Controller
|
|
|
|
# Initialize router with model pair
|
|
router = Controller(
|
|
routers=["bert"], # or ["sw_ranking", "causal_llm", "mf"]
|
|
strong_model="gpt-4-1106-preview",
|
|
weak_model="mixtral-8x7b-instruct-v0.1",
|
|
api_base="http://localhost:8000/v1", # Optional: custom endpoint
|
|
)
|
|
|
|
# Route a query
|
|
query = "Explain quantum entanglement"
|
|
response = router.route(
|
|
model="router-bert-0.5", # router-{type}-{threshold}
|
|
messages=[{"role": "user", "content": query}]
|
|
)
|
|
|
|
print(f"Routed to: {response.model}")
|
|
print(f"Response: {response.choices[0].message.content}")</code></pre>
|
|
<h2>Hanzo SDK Integration</h2>
|
|
<h3>Automatic Model Routing with Hanzo</h3>
|
|
<p><strong>Instead of manual RouteLLM implementation</strong>, use Hanzo SDK's built-in intelligent routing:</p>
|
|
<pre><code class="lang-python">from hanzo import Hanzo
|
|
|
|
# Initialize with hybrid mode (local + cloud with automatic routing)
|
|
hanzo = Hanzo(
|
|
inference_mode='hybrid', # Combines local + cloud models
|
|
auto_route=True, # Enable intelligent routing
|
|
cost_optimize=True # Optimize for cost
|
|
)
|
|
|
|
# Automatic routing based on query complexity
|
|
response = hanzo.chat.completions.create(
|
|
messages=[{"role": "user", "content": query}],
|
|
# No need to specify model - Hanzo routes automatically
|
|
)
|
|
|
|
# Hanzo automatically:
|
|
# 1. Analyzes query complexity
|
|
# 2. Checks local models (Hanzo Node) first for privacy
|
|
# 3. Routes to cloud only if needed
|
|
# 4. Caches results for repeated queries
|
|
# 5. Tracks costs and latency
|
|
print(f"Routed to: {response.model}")
|
|
print(f"Cost: ${response.usage.cost:.4f}")
|
|
print(f"Latency: {response.latency_ms}ms")</code></pre>
|
|
<h3>Hanzo Routing Strategies</h3>
|
|
<pre><code class="lang-python"># 1. Privacy-First Routing (local whenever possible)
|
|
hanzo = Hanzo(
|
|
inference_mode='hybrid',
|
|
prefer_local=True, # Use Hanzo Node first
|
|
node_url='http://localhost:8080'
|
|
)
|
|
|
|
# Simple queries stay local (privacy + no cost)
|
|
response = hanzo.chat.completions.create(
|
|
messages=[{"role": "user", "content": "What is 2+2?"}]
|
|
)
|
|
# Routed to: local-llama-3-8b
|
|
|
|
# Complex queries go to cloud
|
|
response = hanzo.chat.completions.create(
|
|
messages=[{"role": "user", "content": "Analyze quantum cryptography"}]
|
|
)
|
|
# Routed to: claude-3.5-sonnet
|
|
|
|
# 2. Cost-Optimized Routing
|
|
hanzo = Hanzo(
|
|
inference_mode='cloud',
|
|
cost_limit_per_request=0.01, # Max $0.01 per request
|
|
fallback_on_limit=True # Use cheaper model if exceeded
|
|
)
|
|
|
|
# 3. Latency-Optimized Routing
|
|
hanzo = Hanzo(
|
|
inference_mode='hybrid',
|
|
max_latency_ms=500, # Sub-500ms required
|
|
prefer_local=True # Local models are fastest
|
|
)
|
|
|
|
# 4. Quality-Focused Routing
|
|
hanzo = Hanzo(
|
|
inference_mode='cloud',
|
|
quality_tier='high', # Only top-tier models
|
|
models=['gpt-4', 'claude-3.5-sonnet']
|
|
)</code></pre>
|
|
<h3>Hanzo MCP Tool for Agents</h3>
|
|
<p>Expose Hanzo routing as MCP tool for agentic workflows:</p>
|
|
<pre><code class="lang-typescript">import { MCPServer, Tool } from '@hanzo/mcp'
|
|
import { Hanzo } from 'hanzo' // Python SDK
|
|
|
|
const hanzo = new Hanzo({
|
|
inference_mode: 'hybrid',
|
|
auto_route: true
|
|
})
|
|
|
|
const hanzoRouteTool: Tool = {
|
|
name: 'hanzo_route_and_infer',
|
|
description: 'Automatically route query to best model and generate response',
|
|
parameters: {
|
|
prompt: { type: 'string', required: true },
|
|
strategy: {
|
|
type: 'string',
|
|
enum: ['cost', 'latency', 'quality', 'privacy'],
|
|
default: 'cost'
|
|
}
|
|
},
|
|
async execute({ prompt, strategy }) {
|
|
const response = await hanzo.chat.completions.create({
|
|
messages: [{ role: 'user', content: prompt }],
|
|
routing_strategy: strategy
|
|
})
|
|
|
|
return {
|
|
text: response.choices[0].message.content,
|
|
model: response.model,
|
|
cost: response.usage.cost,
|
|
latency_ms: response.latency_ms
|
|
}
|
|
}
|
|
}</code></pre>
|
|
<h3>Hanzo vs Manual RouteLLM</h3>
|
|
<p>| Feature | Manual RouteLLM | Hanzo SDK | |---------|-----------------|-----------| | <strong>Setup</strong> | 50+ lines config | 3 lines | | <strong>Local Models</strong> | Not supported | Built-in (Hanzo Node) | | <strong>Privacy</strong> | Manual logic | Automatic (local-first) | | <strong>Cost Tracking</strong> | Custom implementation | Built-in | | <strong>Caching</strong> | Manual | Automatic | | <strong>MCP Integration</strong> | Custom wrapper | Native support | | <strong>Fallback</strong> | Manual chains | Automatic | | <strong>Monitoring</strong> | Custom Prometheus | Built-in dashboard |</p>
|
|
<p><strong>When to Use What</strong>:</p>
|
|
<ul><li><strong>Hanzo SDK</strong>: Most use cases (simpler, more features)</li><li><strong>Manual RouteLLM</strong>: Research, custom training, non-Hanzo stack</li></ul>
|
|
<h3>RouteLLM: Threshold Tuning</h3>
|
|
<pre><code class="lang-python"># Threshold controls quality/cost tradeoff
|
|
# Higher threshold = more weak model usage = lower cost
|
|
# Lower threshold = more strong model usage = higher quality
|
|
|
|
thresholds = [0.3, 0.5, 0.7, 0.9]
|
|
|
|
for threshold in thresholds:
|
|
router_name = f"router-bert-{threshold}"
|
|
response = router.route(
|
|
model=router_name,
|
|
messages=[{"role": "user", "content": "Simple math: 2+2"}]
|
|
)
|
|
print(f"Threshold {threshold}: {response.model}")
|
|
|
|
# Results:
|
|
# 0.3 -> gpt-4 (conservative, high quality)
|
|
# 0.5 -> mixtral (balanced)
|
|
# 0.7 -> mixtral (aggressive cost savings)
|
|
# 0.9 -> mixtral (maximum savings)</code></pre>
|
|
<h3>RouteLLM: Evaluation Mode</h3>
|
|
<pre><code class="lang-python">import pandas as pd
|
|
from routellm.routers import ROUTER_CLS
|
|
|
|
# Load router for evaluation
|
|
router_cls = ROUTER_CLS["bert"]
|
|
router = router_cls(
|
|
strong_model_name="gpt-4-1106-preview",
|
|
weak_model_name="mixtral-8x7b-instruct-v0.1"
|
|
)
|
|
|
|
# Evaluate on dataset
|
|
test_queries = [
|
|
"What is 2+2?",
|
|
"Explain the implications of quantum computing on cryptography",
|
|
"Name the capital of France",
|
|
"Analyze the socioeconomic factors in the French Revolution"
|
|
]
|
|
|
|
decisions = []
|
|
for query in test_queries:
|
|
# Get routing decision (0-1 score)
|
|
score = router.route(query)
|
|
selected = "strong" if score < 0.5 else "weak"
|
|
decisions.append({
|
|
"query": query[:50],
|
|
"score": score,
|
|
"model": selected
|
|
})
|
|
|
|
df = pd.DataFrame(decisions)
|
|
print(df)
|
|
|
|
# Calculate cost savings
|
|
weak_ratio = (df["model"] == "weak").mean()
|
|
print(f"Weak model usage: {weak_ratio:.1%}")
|
|
print(f"Estimated savings: {weak_ratio * 0.95:.1%}") # 95% cheaper</code></pre>
|
|
<h3>Semantic Routing with Aurelio</h3>
|
|
<pre><code class="lang-python">from semantic_router import Route, RouteLayer
|
|
from semantic_router.encoders import OpenAIEncoder
|
|
|
|
# Define routes with examples
|
|
code_route = Route(
|
|
name="code",
|
|
utterances=[
|
|
"How do I write a function in Python?",
|
|
"Debug this code snippet",
|
|
"Explain this algorithm",
|
|
"Write a sorting function"
|
|
]
|
|
)
|
|
|
|
general_route = Route(
|
|
name="general",
|
|
utterances=[
|
|
"What is the weather today?",
|
|
"Tell me a joke",
|
|
"What's the capital of France?",
|
|
"Who won the World Series?"
|
|
]
|
|
)
|
|
|
|
reasoning_route = Route(
|
|
name="reasoning",
|
|
utterances=[
|
|
"Solve this complex math problem",
|
|
"Analyze the implications of this policy",
|
|
"Compare and contrast these approaches",
|
|
"Provide a detailed ethical analysis"
|
|
]
|
|
)
|
|
|
|
# Initialize encoder and layer
|
|
encoder = OpenAIEncoder()
|
|
route_layer = RouteLayer(
|
|
encoder=encoder,
|
|
routes=[code_route, general_route, reasoning_route]
|
|
)
|
|
|
|
# Route queries
|
|
queries = [
|
|
"How do I reverse a string in JavaScript?",
|
|
"What's 2+2?",
|
|
"Explain the philosophical implications of AI consciousness"
|
|
]
|
|
|
|
for query in queries:
|
|
route = route_layer(query)
|
|
print(f"Query: {query[:50]}")
|
|
print(f"Route: {route.name if route else 'fallback'}\n")
|
|
|
|
# Map routes to models
|
|
model_map = {
|
|
"code": "claude-3.5-sonnet", # Best at coding
|
|
"general": "gpt-4o-mini", # Fast and cheap
|
|
"reasoning": "gpt-4o", # Strong reasoning
|
|
None: "gpt-4o-mini" # Default fallback
|
|
}
|
|
|
|
def route_and_call(query: str):
|
|
route = route_layer(query)
|
|
model = model_map.get(route.name if route else None)
|
|
print(f"Routing to: {model}")
|
|
# Call selected model...</code></pre>
|
|
<h3>Rule-Based Routing</h3>
|
|
<pre><code class="lang-python">import re
|
|
from typing import Literal
|
|
|
|
ModelType = Literal["strong", "weak", "specialized"]
|
|
|
|
class RuleBasedRouter:
|
|
def __init__(
|
|
self,
|
|
strong_model: str = "gpt-4o",
|
|
weak_model: str = "gpt-4o-mini",
|
|
code_model: str = "claude-3.5-sonnet"
|
|
):
|
|
self.strong = strong_model
|
|
self.weak = weak_model
|
|
self.code = code_model
|
|
|
|
def route(self, query: str, context: dict = None) -> str:
|
|
"""Route based on heuristics"""
|
|
query_lower = query.lower()
|
|
|
|
# Rule 1: Code-related queries -> specialized model
|
|
code_keywords = ["code", "function", "debug", "implement", "algorithm"]
|
|
if any(kw in query_lower for kw in code_keywords):
|
|
return self.code
|
|
|
|
# Rule 2: Simple factual queries -> weak model
|
|
simple_patterns = [
|
|
r"^what is \w+\?$",
|
|
r"^who is \w+\?$",
|
|
r"^when did \w+",
|
|
r"^\d+[\+\-\*/]\d+", # Simple math
|
|
]
|
|
if any(re.match(p, query_lower) for p in simple_patterns):
|
|
return self.weak
|
|
|
|
# Rule 3: Long, complex queries -> strong model
|
|
if len(query) > 500:
|
|
return self.strong
|
|
|
|
# Rule 4: Reasoning keywords -> strong model
|
|
reasoning_keywords = [
|
|
"analyze", "implications", "compare", "evaluate",
|
|
"critique", "synthesize", "ethical", "philosophical"
|
|
]
|
|
if any(kw in query_lower for kw in reasoning_keywords):
|
|
return self.strong
|
|
|
|
# Rule 5: Context indicates multi-turn conversation -> strong model
|
|
if context and context.get("turn_count", 0) > 3:
|
|
return self.strong
|
|
|
|
# Default: weak model
|
|
return self.weak
|
|
|
|
# Usage
|
|
router = RuleBasedRouter()
|
|
|
|
queries = [
|
|
"Write a binary search function", # -> code model
|
|
"What is the capital of France?", # -> weak
|
|
"2+2", # -> weak
|
|
"Analyze the ethical implications of AI in healthcare" # -> strong
|
|
]
|
|
|
|
for q in queries:
|
|
model = router.route(q)
|
|
print(f"{q[:50]:50} -> {model}")</code></pre>
|
|
<h3>Hybrid Routing with Fallback</h3>
|
|
<pre><code class="lang-python">from typing import Optional
|
|
import openai
|
|
|
|
class HybridRouter:
|
|
def __init__(self):
|
|
self.semantic_router = route_layer # From earlier example
|
|
self.rule_router = RuleBasedRouter()
|
|
|
|
def route(
|
|
self,
|
|
query: str,
|
|
use_semantic: bool = True,
|
|
use_rules: bool = True
|
|
) -> str:
|
|
"""Hybrid routing with fallback chain"""
|
|
|
|
# Try semantic routing first (most accurate)
|
|
if use_semantic:
|
|
route = self.semantic_router(query)
|
|
if route and route.name:
|
|
return model_map.get(route.name)
|
|
|
|
# Fallback to rule-based routing
|
|
if use_rules:
|
|
return self.rule_router.route(query)
|
|
|
|
# Final fallback
|
|
return "gpt-4o-mini"
|
|
|
|
def route_with_retry(self, query: str, max_retries: int = 2):
|
|
"""Route with fallback on errors"""
|
|
models = [
|
|
self.route(query), # Primary
|
|
"gpt-4o", # Fallback 1
|
|
"claude-3.5-sonnet" # Fallback 2
|
|
]
|
|
|
|
for i, model in enumerate(models[:max_retries + 1]):
|
|
try:
|
|
response = openai.ChatCompletion.create(
|
|
model=model,
|
|
messages=[{"role": "user", "content": query}],
|
|
timeout=30
|
|
)
|
|
return {
|
|
"model": model,
|
|
"response": response.choices[0].message.content,
|
|
"attempt": i + 1
|
|
}
|
|
except Exception as e:
|
|
print(f"Model {model} failed: {e}")
|
|
if i == len(models) - 1:
|
|
raise
|
|
continue
|
|
|
|
# Usage
|
|
hybrid = HybridRouter()
|
|
result = hybrid.route_with_retry(
|
|
"Explain quantum entanglement to a 10-year-old"
|
|
)
|
|
print(f"Used model: {result['model']} (attempt {result['attempt']})")</code></pre>
|
|
<h3>Arize Phoenix Observability</h3>
|
|
<pre><code class="lang-python">import phoenix as px
|
|
from phoenix.trace import using_project
|
|
from opentelemetry import trace
|
|
|
|
# Start Phoenix
|
|
px.launch_app()
|
|
|
|
# Instrument routing
|
|
tracer = trace.get_tracer(__name__)
|
|
|
|
@using_project("llm-routing")
|
|
def route_and_call(query: str):
|
|
with tracer.start_as_current_span("routing_decision") as span:
|
|
# Route
|
|
router = HybridRouter()
|
|
selected_model = router.route(query)
|
|
|
|
# Log routing decision
|
|
span.set_attribute("query", query)
|
|
span.set_attribute("selected_model", selected_model)
|
|
span.set_attribute("routing_strategy", "hybrid")
|
|
|
|
# Call model
|
|
with tracer.start_as_current_span("llm_call") as call_span:
|
|
call_span.set_attribute("model", selected_model)
|
|
|
|
response = openai.ChatCompletion.create(
|
|
model=selected_model,
|
|
messages=[{"role": "user", "content": query}]
|
|
)
|
|
|
|
# Log metrics
|
|
call_span.set_attribute("tokens_used", response.usage.total_tokens)
|
|
call_span.set_attribute("cost_usd", calculate_cost(response, selected_model))
|
|
|
|
return response
|
|
|
|
# View routing patterns in Phoenix UI
|
|
# Navigate to http://localhost:6006</code></pre>
|
|
<hr>
|
|
<h2>Quick Reference</h2>
|
|
<h3>Model Strengths by Category (2025)</h3>
|
|
<pre><code>Category | Best Models | Use Case
|
|
--------------|--------------------------|----------------------------------
|
|
Reasoning | GPT-4o, Claude 3.5/4 | Complex analysis, multi-step logic
|
|
Coding | Claude 3.5 Sonnet | Code generation, debugging
|
|
Speed | Gemini 2.5 Flash | 370 tok/s, real-time applications
|
|
Cost | GPT-4o mini, Gemini Flash| Simple queries, high volume
|
|
Multimodal | GPT-4o, Gemini 2.5 | Vision + text tasks
|
|
Long Context | Claude 3.5 (200k) | Document analysis, large codebases</code></pre>
|
|
<h3>Cost Comparison (per million tokens)</h3>
|
|
<pre><code>Model | Input | Output | Use Case
|
|
-------------------------|----------|----------|------------------
|
|
GPT-4o | $5.00 | $20.00 | General strong
|
|
GPT-4o mini | $0.15 | $0.60 | General weak
|
|
Claude 3.5 Sonnet | $3.00 | $15.00 | Coding strong
|
|
Claude 3 Haiku | $0.25 | $1.25 | Fast weak
|
|
Gemini 2.5 Flash | $0.10 | $0.40 | Cheapest strong
|
|
DeepSeek V3 | $0.30 | $1.00 | Open-source alt</code></pre>
|
|
<h3>Routing Decision Tree</h3>
|
|
<pre><code>Query characteristics -> Recommended strategy
|
|
────────────────────────────────────────────────
|
|
High volume, mixed complexity -> RouteLLM
|
|
Domain-specific (code, legal) -> Semantic routing
|
|
Latency critical -> Rule-based
|
|
Quality critical -> Always strong model
|
|
Budget constrained -> RouteLLM threshold 0.7+
|
|
Multi-turn conversations -> Session-aware routing</code></pre>
|
|
<hr>
|
|
<h2>Anti-Patterns</h2>
|
|
<p>❌ <strong>Always using the strongest model</strong>: Wastes 10-50x cost on simple queries ✅ Route based on complexity; use weak models for 60-80% of queries</p>
|
|
<p>❌ <strong>No observability on routing decisions</strong>: Can't optimize or debug ✅ Use Arize Phoenix to trace routing logic and model performance</p>
|
|
<p>❌ <strong>Static routing rules that don't adapt</strong>: Performance degrades over time ✅ Continuously evaluate routing accuracy; retrain or adjust thresholds</p>
|
|
<p>❌ <strong>Ignoring latency in routing overhead</strong>: Routing adds 100ms+ delay ✅ Use cached semantic embeddings; pre-compute route assignments</p>
|
|
<p>❌ <strong>Single point of failure in routing</strong>: Entire system fails if router breaks ✅ Implement fallback chains; default to mid-tier model on router errors</p>
|
|
<p>❌ <strong>Not tracking cost savings</strong>: Can't justify routing complexity ✅ Log model selection, token usage, and calculate cost deltas</p>
|
|
<p>❌ <strong>Over-engineering for simple use cases</strong>: Rule-based is fine for 2-3 models ✅ Start simple; add RouteLLM/semantic routing when scaling to 5+ models</p>
|
|
<p>❌ <strong>Training routers on outdated model pairs</strong>: New models shift capabilities ✅ Regularly re-evaluate with current model benchmarks (GPT-4o, Claude 3.5)</p>
|
|
<hr>
|
|
<h2>Related Skills</h2>
|
|
<ul><li><code>llm-model-selection.md</code> - Choosing models for specific capabilities</li><li><code>multi-model-orchestration.md</code> - Coordinating multiple models in pipelines</li><li><code>modal-web-endpoints.md</code> - Deploying routing logic as FastAPI endpoints</li><li><code>api-rate-limiting.md</code> - Rate limiting across multiple model providers</li><li><code>observability-distributed-tracing.md</code> - Tracing multi-model systems</li><li><code>api-error-handling.md</code> - Handling provider failures and fallbacks</li></ul>
|
|
<hr>
|
|
<p><strong>Last Updated</strong>: 2025-10-26 <strong>Format Version</strong>: 1.0 (Atomic)</p>
|
|
<footer>
|
|
<div class="fmap">
|
|
<div><b>Build</b>
|
|
<a href="https://hanzo.app">hanzo.app</a> app builder
|
|
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
|
|
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
|
|
<div><b>Run</b>
|
|
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
|
|
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
|
|
<div><b>Account</b>
|
|
<a href="https://hanzo.id">hanzo.id</a> sign in
|
|
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
|
|
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
|
|
<div><b>Company</b>
|
|
<a href="https://hanzo.ai">hanzo.ai</a> canonical
|
|
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
|
|
</div>
|
|
<div class="fend"><a href="/ml-llm-model-routing.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
|
|
</footer>
|
|
</div></body></html> |