papers/enso/main.tex

\documentclass[11pt]{article}
\usepackage{../shared/hanzocover}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{amsmath,amssymb}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{xcolor}
\usepackage{tikz}
\usetikzlibrary{shapes,arrows.meta,positioning,fit,backgrounds,calc}
\usepackage[hidelinks]{hyperref}

\emergencystretch=2em % avoid overfull lines from long \texttt{} terms in narrow columns

\definecolor{hzblue}{rgb}{0.12,0.30,0.55}
\definecolor{hzslate}{rgb}{0.22,0.25,0.30}
\definecolor{hzgreen}{rgb}{0.10,0.45,0.25}
\definecolor{hzred}{rgb}{0.60,0.16,0.16}

% =====================================================================
% Numbers-as-data: the measured tables and headline numbers in this paper
% are a pure function of the enso-bench results. `emit_paper.py' writes
% three plain-LaTeX fragments into this directory, which we \input below:
%   results-measured-rows.tex   per-bench head-to-head table rows (\S\ref{sec:eval})
%   results-ablation-rows.tex   enso-ultra v1->v2 verify-then-select ablation rows
%   results-headline.tex        \newcommand headline macros for the prose
% A fourth file, results-prior.tex, holds an EARLIER run in the two-SKU
% format this paper no longer uses. It is not \input anywhere; it is read
% only by a human, and the two runs' disagreement is reported in \S\ref{sec:limits}.
% \providecommand supplies a "run pending" fallback for any headline macro
% not yet measured, so the paper always compiles and never shows a
% fabricated number. Absent table rows simply do not appear.
% =====================================================================
\newcommand{\runpending}{\textit{\small run pending}}
\IfFileExists{results-headline.tex}{\input{results-headline.tex}}{}
\providecommand{\hEnsoGPQAacc}{\runpending}
\providecommand{\hEnsoGPQAcost}{\runpending}
\providecommand{\hUltraGPQAacc}{\runpending}
\providecommand{\hBestGPQAname}{\runpending}
\providecommand{\hBestGPQAacc}{\runpending}
\providecommand{\hOpusGPQAacc}{\runpending}
\providecommand{\hOpusGPQAcost}{\runpending}

\title{Enso: Learned Per-Request Routing to Frontier Intelligence, and Enso Ultra's Training-Free Fan-Out Synthesis}
\author{Zach Kelling\thanks{Corresponding author: [email protected]} \\
    \textit{Hanzo Industries Inc (Techstars '17)} \\
    San Francisco, California \\
    \texttt{https://hanzo.ai}}
\date{Hanzo Research}

\begin{document}

\hanzocoverpage

\twocolumn[
  \begin{@twocolumnfalse}
  \maketitle
  \begin{abstract}
  \noindent
  The strongest answer to a given request rarely comes from the same model twice.
  On hard software engineering a frontier reasoning model wins; on long-context
  synthesis a million-token model wins; on cheap high-volume traffic the marginal
  gain of the most expensive model is zero. A single monolithic model is therefore
  a bet that one weight set dominates every request in the stream, which the
  published per-benchmark spread between frontier models refutes. We describe
  \textbf{Enso}, Hanzo's proprietary closed model family that turns this spread
  into a product. \texttt{enso} is a learned router: it featurizes each request,
  scores every arm in a pool of frontier models by a bilinear utility
  $x^{\top}Wp$, and returns the single SLO-feasible arm that maximizes
  $\mathrm{quality}-\lambda\,\mathrm{cost}-\mu\,\mathrm{latency}$, adapting to each
  user online. \texttt{enso-ultra} reuses the same policy to select the top-$N$
  arms, dispatches them in parallel, and fuses their answers with a training-free
  synthesizer. The routing policy is the one specified in the Hanzo Router
  paper~\cite{hanzorouterpaper}; this paper is about the family it powers, the
  serving substrate that presents it as one \texttt{/v1} API, and how it compares
  to frontier baselines (Opus 4.8, GPT-5.5, GPT-5.6, Gemini 3.1 Pro, Fable 5) and
  to a prior published learned orchestrator from the recent
  literature~\cite{priororchestrator}. Our design choice is deliberate and stated
  plainly: unlike the prior published router (Ultra), whose orchestrator is
  trained with reinforcement learning, Enso is \emph{training-free at the
  orchestration layer and learned only at the routing layer}, so a new frontier
  arm is onboarded by adding a row, not by retraining a policy over the pool. We
  report a measured evaluation from the \texttt{enso-bench} harness---every arm
  and every SKU executed under one gateway on the same items, with a per-request
  cost column the standard router table lacks. On GPQA Diamond \texttt{enso}
  scores \hEnsoGPQAacc\% at \$\hEnsoGPQAcost{} per 1{,}000 tasks, matching the best
  single arm (\texttt{\hBestGPQAname}, \hBestGPQAacc\%) and beating the premium
  reasoning arm (Opus~4.8, \hOpusGPQAacc\% at \$\hOpusGPQAcost{}/1k) at roughly a
  fifth of its metered cost, while \texttt{enso-ultra}'s verify-then-select
  fan-out is \emph{both} more accurate and cheaper than blind synthesis. We are
  equally explicit where routing does \emph{not} win---it leads neither every
  benchmark nor Humanity's Last Exam---and typeset any unmeasured cell as
  \emph{run pending} rather than invent a number.
  \end{abstract}
  \vspace{1.5em}
  \end{@twocolumnfalse}
]

\section{Introduction}
\label{sec:intro}

A frontier language model is a fixed bet. Its weights encode one balance of code,
reasoning, math, chart understanding, long-context recall, and instruction
following, and that balance is frozen at training time. But the request stream a
production assistant sees is not fixed: one request is a graduate-level physics
question, the next is a repository-scale refactor, the next is a chart to read,
the next is a one-line reformat. The public per-benchmark tables make the
consequence concrete. On the same seven benchmarks, the frontier models trade the
lead back and forth---one leads on SWE-Bench Pro, another on LiveCodeBench Pro,
another on Humanity's Last Exam---and no single model holds every column. A
monolithic deployment picks one of these models and eats its losses on every
column it does not lead. That is the structural inefficiency Enso removes.

The move is to stop shipping a model and start shipping a \emph{policy over
models}. Enso is a closed family with two SKUs behind one API. \texttt{enso}
routes each request to the single frontier arm most likely to serve it best under
a per-request service-level objective. The pool we measure here is the set our
gateway can actually serve---GPT-5.6 (sol), Claude Opus 4.8, GPT-5.5, Claude 5
Sonnet, DeepSeek-V4-Pro, and Claude Fable 5, with GPT-4o as a cheap baseline
single---and it spans the same capability axes a router needs: frontier reasoning,
strong generalists, a dedicated coder, and a long-context arm. Gemini 3.1 Pro has
no endpoint on this gateway, so it appears only as a provider-reported reference
column, never as a routable arm. \texttt{enso-ultra} spends more to buy the
ceiling: it fans the request
out to the top-$N$ arms in parallel and synthesizes one answer from their
candidates. Both SKUs are a single \texttt{/v1} endpoint; the caller sends
messages and gets an answer, and never has to know---or maintain keys for---the
providers underneath.

This framing has two consequences we make central. First, \textbf{routing beats
monoliths whenever the pool's per-request argmax beats any fixed member}, and the
published spread shows that condition holds. A learned router that identifies the
right arm per request inherits the pool's upper envelope, not any single model's
average. Second, \textbf{a router makes cost a first-class, transparent axis}. A
monolith charges its one price on every token regardless of whether the request
needed it; a router that optimizes $\mathrm{quality}-\lambda\,\mathrm{cost}-\mu\,\mathrm{latency}$
sends cheap requests to cheap arms and prices each answer at the arm that actually
served it. We therefore report a cost-per-benchmark column alongside accuracy---the
number a buyer of intelligence actually pays---which comparison tables in this
line of work typically omit.

We are explicit about lineage. The routing mechanism, the SLO objective, the
bilinear utility, and the online per-user adaptation are specified and analyzed in
the Hanzo Router paper~\cite{hanzorouterpaper}; we cite that work for the policy
internals rather than restating them, and focus here on (i)~the Enso \emph{family}
and the serving substrate that presents it as one product (\S\ref{sec:arch}),
(ii)~the Enso Ultra fan-out/synthesis SKU and why its orchestration is deliberately
training-free (\S\ref{sec:ultra}), and (iii)~a measured evaluation against the same
benchmarks a learned-orchestrator baseline reports (\S\ref{sec:eval}).

\section{Related Work}
\label{sec:related}

\paragraph{Learned orchestration: prior router, Trinity, Conductor.} The closest
system to Enso is the prior published router~\cite{priororchestrator}, a learned
orchestrator over a pool of frontier models---Gemini 3.1 Pro, Claude Opus 4.8, and
GPT-5.5---shipped in two SKUs, a base tier and an Ultra tier, exactly the two-tier
shape Enso takes. The base tier is trained with supervised fine-tuning plus
separable CMA-ES~\cite{sepcmaes}, and its Ultra tier adds GRPO reinforcement
learning~\cite{grpo} at the orchestration layer; both build on the ICLR~2026
foundations Trinity~\cite{trinity} (evolutionary composition of experts) and
Conductor~\cite{conductor} (RL-orchestrated ensembles). Enso shares that router's
thesis---that the per-request argmax over a frontier pool beats any fixed
member---and its two-SKU packaging, and we treat its published results as the
reference point our evaluation mirrors (\S\ref{sec:eval}). Enso differs in where
the learning sits: the prior router's Ultra tier learns an orchestrator with RL
over the pool, whereas Enso learns only the routing policy and keeps orchestration
training-free (\S\ref{sec:ultra}), so onboarding a new arm is a data change, not a
retrain.

\paragraph{Routing and cascading.} Enso's single-arm SKU is a router in the line of
RouteLLM~\cite{routellm}, which learns a strong/weak binary router from preference
data; FrugalGPT~\cite{frugalgpt}, which cascades cheap-to-expensive with a learned
stopping scorer; Hybrid LLM~\cite{hybridllm}, which routes on a quality-aware
threshold; AutoMix~\cite{automix}, which escalates via self-verification; and
Tabi~\cite{tabi}, a multi-level inference system. Enso generalizes their binary or
cascade decision to a constrained continuous objective,
$\mathrm{quality}-\lambda\,\mathrm{cost}-\mu\,\mathrm{latency}$, over a pool of many
frontier arms, and adds per-user online adaptation~\cite{linucb} that cascade work
generally omits. The full treatment is in the router paper~\cite{hanzorouterpaper}.

\paragraph{Ensemble synthesis.} Enso Ultra's fan-out/synthesis is grounded in
multi-agent and ensembling work: Mixture-of-Agents~\cite{moa} layers proposer and
aggregator models; LLM-Blender~\cite{llmblender} ranks and generatively fuses
candidates from a model bank; and self-consistency~\cite{selfconsistency} shows
that aggregating multiple independent samples improves reasoning. Ultra applies
these fusion ideas across \emph{heterogeneous frontier arms} rather than samples
from one model, and keeps the fuser training-free.

\section{Enso Architecture}
\label{sec:arch}

Enso is two layers with one seam between them: a \emph{routing layer} that decides
which arm(s) serve a request, and a \emph{serving layer} that presents the family
as one metered product. Figure~\ref{fig:arch} shows both SKUs.

\input{figures/architecture}

\subsection{The routing layer: policy as data}
\label{sec:routing}

The router is the mechanism and policy specified in the Hanzo Router
paper~\cite{hanzorouterpaper}; we summarize only what the family needs and cite
that work for the rest. Each arm is one row in a cross-modal registry---a
\texttt{(model, level, modality)} profile carrying an eval-measured quality vector
over tasks plus normalized latency, cost, VRAM, and context reach. A request is
turned into a feature vector $x$ by a hashing-trick featurizer (task one-hot,
hashed n-grams, log-length, media flag, bias) in sub-microsecond time, and every
arm profile $p$ is scored by the learnable bilinear utility
\begin{equation}
u(x,p) = x^{\top} W p, \qquad W\in\mathbb{R}^{D\times K}.
\end{equation}
The selector returns the SLO-feasible $\arg\max$ of the constrained objective
\begin{equation}
\arg\max_{p\,:\,\mathrm{feasible}(p)}\; u(x,p) - \lambda\,\widehat{c}(p) - \mu\,\widehat{\ell}(p),
\end{equation}
where the hard ceilings ($\texttt{max\_latency\_ms}$, $\texttt{max\_cost}$,
$\texttt{min\_quality}$) gate the candidate set and the soft weights
$\lambda,\mu$ trade cost and latency against quality. Offline, $W$ is a closed-form
ridge fit~\cite{ridge}; online, each user carries a LinUCB bandit~\cite{linucb}
whose parameter $\theta_u$ starts at $W$ and drifts toward that user's taste as
outcomes arrive through a single \texttt{observe}$(u,x,p,r)$ call. The essential
property for the family is that \emph{the policy is a function of data}: adding a
frontier arm is adding a profile row, and its ranking then falls out of the same
$W$ without a code change. A two-tier safety guard gates every request before the
selector runs, cheaply on the hot path and escalating only ambiguous prompts.

\subsection{The serving layer: family-as-data}
\label{sec:serving}

The serving substrate (the \texttt{zen-svc} engine behind the Hanzo AI
gateway~\cite{hanzogateway}) turns the routing decision into one product. Three
properties matter.

\paragraph{One identity.} Whichever arm serves a request, the response is presented
as Enso. The serving layer performs identity injection and upstream-hiding: the
caller sees a single model family and its provenance, not the underlying provider,
and never manages third-party keys. This is what lets the pool change---an arm
added, retired, or swapped for a better version---without any client change.

\paragraph{One meter.} Every routed request is metered on one plane, so cost is
attributed per request to the arm that actually served it. This is the substrate
that makes the cost-per-benchmark column of \S\ref{sec:eval} a real, per-request
number rather than a blended list price, and it is what lets a deployment set
$\lambda$ to move spend without touching application code.

\paragraph{The 1M context ladder.} Long-context requests are a distinct pool. When
a request's context exceeds the reach of the default arms, the router climbs a
million-token ladder---a frontier long-context arm (DeepSeek-V4-Pro) with a top Zen
model as the local-capable rung---selected by the same feasibility gate
($\texttt{max\_context}\ge k$) as any other constraint. Long context is thus not a
special mode but another column in the same registry, chosen by the same policy.

\section{Enso Ultra: Training-Free Fan-Out Synthesis}
\label{sec:ultra}

\texttt{enso-ultra} is the ceiling SKU. Where \texttt{enso} spends the policy's
confidence to pick one arm, Ultra spends compute to reduce the risk that the single
argmax was wrong on a hard request: it selects the top-$N$ arms by the same utility
$u(x,p)$, dispatches them in parallel, and fuses their candidate answers into one.
The fan-out reuses the routing layer unchanged---the top-$N$ are simply the first
$N$ of the same ranked, SLO-feasible candidate list---so Ultra inherits the pool,
the guard, and the online adaptation for free.

\paragraph{The synthesizer is training-free.} This is the deliberate contrast with
the prior router's Ultra tier. That tier learns its orchestrator with
GRPO~\cite{grpo}: the policy that decides how to combine models is itself an
RL-trained artifact over the pool. Enso Ultra's synthesizer is training-free---it
ranks and fuses the $N$ candidate answers with a fixed procedure grounded in the
ensembling literature (Mixture-of-Agents-style aggregation~\cite{moa}, pairwise
ranking and generative fusion as in LLM-Blender~\cite{llmblender}, and the
aggregation-improves-reasoning result of self-consistency~\cite{selfconsistency}),
with no weights of its own that depend on the current pool. The engineering
consequence is the point: \emph{all learning in Enso lives at the routing layer,
none at the orchestration layer}. When a new frontier arm ships, Enso onboards it
by adding a profile row and letting the router rank it; Ultra then includes it in
fan-out automatically. A system that learned its orchestrator over the
pool---as the prior router's Ultra tier does---would in principle need to revisit
that trained policy when the pool changes. We view training-free orchestration as
the more maintainable choice for a family whose pool is expected to churn as
frontier models are released, and we make no claim that it dominates a trained
orchestrator on quality; \S\ref{sec:eval} reports the measured comparison and
\S\ref{sec:limits} states the caveats.

\paragraph{Provenance.} Because the serving layer meters every arm, an Ultra answer
carries which arms contributed and at what cost, so the fan-out's price is
transparent per request---the same metering that prices the single-arm SKU prices
the ensemble.

\section{Evaluation}
\label{sec:eval}

\paragraph{Methodology.} We evaluate on the public benchmarks of the standard
frontier-router suite~\cite{priororchestrator}---GPQA Diamond~\cite{gpqa},
LiveCodeBench~\cite{livecodebench}, Humanity's Last Exam (HLE)~\cite{hle}, and
CharXiv Reasoning~\cite{charxiv}. Unlike a comparison that mixes measurement
regimes, we run \emph{every} system under \emph{one} harness: each single-model
arm, \texttt{enso-flash}, \texttt{enso}, and \texttt{enso-ultra} answers the
identical items through the same OpenAI-compatible gateway at temperature~0, with
the same extraction and scoring---a like-for-like comparison the reported tables
cannot give. GPQA and HLE are graded by exact / judged match, LiveCodeBench by
executing every public and private test case in a sandboxed subprocess (pass@1),
and CharXiv by an LLM judge; grading is identical across systems and its cost is
booked separately. Beyond accuracy we report \textbf{cost per 1{,}000 tasks}, the
per-request-metered spend (\S\ref{sec:serving}) that a comparison of routers must
include and that closed router tables omit. Sample sizes are $n{=}198$ (GPQA),
$500$ (HLE), $175$ (LiveCodeBench), $200$ (CharXiv), with binomial standard errors
reported per cell. The agentic rows (SWE-Bench Pro, Terminal-Bench~2.1) and the
extended code benches (LiveCodeBench Pro) are the most expensive to run; we report
a SWE-Bench Pro \emph{pilot} below and mark the rest \emph{run pending}
(\S\ref{sec:limits}).

\paragraph{Reference point.} For context, Table~\ref{tab:reference} reproduces
the prior published router's results verbatim as reported in that
work~\cite{priororchestrator}; these are its reported numbers, not ours, and are
included only to anchor the benchmark scale and the shape of the two-SKU
comparison.

\begin{table*}[t]
\centering
\small
\setlength{\tabcolsep}{5pt}
\caption{Reference: provider-reported frontier baselines; the last two columns
reproduce a prior published router's reported numbers~\cite{priororchestrator}
(their system, not measured by us), included only to anchor benchmark scale. Our
measured, like-for-like results are in Table~\ref{tab:measured}.}
\label{tab:reference}
\begin{tabular}{@{}lrrrrr@{}}
\toprule
Benchmark & Opus 4.8 & Gemini 3.1 Pro & GPT-5.5 & Prior router & Prior router (Ultra) \\
\midrule
SWE-Bench Pro        & 69.2 & 54.2 & 58.6 & 59.0 & 73.7 \\
Terminal-Bench 2.1   & 74.6 & 70.3 & 78.2 & 80.2 & 82.1 \\
LiveCodeBench        & 87.8 & 88.5 & 85.3 & 92.9 & 93.2 \\
LiveCodeBench Pro    & 84.8 & 82.9 & 88.4 & 87.8 & 90.8 \\
HLE                  & 49.8 & 44.4 & 41.4 & 47.2 & 50.0 \\
CharXiv Reasoning    & 84.2 & 83.3 & 84.1 & 85.1 & 86.6 \\
GPQA Diamond         & 92.0 & 94.3 & 93.6 & 95.5 & 95.5 \\
\bottomrule
\end{tabular}
\end{table*}

\paragraph{Measured head-to-head.} Table~\ref{tab:measured} is our result table:
on each benchmark we ran, the best single arm we measured set against the three
Enso SKUs, with accuracy and metered cost per $1{,}000$ tasks for each. The rows
are generated directly from the \texttt{enso-bench} result files; a benchmark with
no measured Enso pair does not appear, and the full per-arm cost/accuracy frontier
is Figure~\ref{fig:frontier}.

\begin{table*}[t]
\centering\small\setlength{\tabcolsep}{5pt}
\caption{Measured under one harness (\texttt{enso-bench}): best single arm vs the
Enso family, accuracy (\%) and metered cost (\$ per 1{,}000 tasks). Every number is
executed live on the gateway; a dash marks a cell not run. Read per row---\texttt{enso}
approaches or beats the best arm well below the premium arm's cost;
\texttt{enso-ultra} lifts the ceiling on the open-ended benches at higher metered
cost; and, honestly, neither leads HLE, where the strongest single arm wins.
Binomial se $\le 3.5$ pts ($n\ge175$). Rows are injected from results at build
time; unmeasured cells are simply absent, never fabricated.}
\label{tab:measured}
\begin{tabular}{@{}l l r r r r r r r r@{}}
\toprule
& \multicolumn{3}{c}{Best single arm} & \multicolumn{2}{c}{\texttt{enso-flash}} & \multicolumn{2}{c}{\texttt{enso}} & \multicolumn{2}{c}{\texttt{enso-ultra}} \\
\cmidrule(lr){2-4}\cmidrule(lr){5-6}\cmidrule(lr){7-8}\cmidrule(lr){9-10}
Benchmark & arm & acc. & \$/1k & acc. & \$/1k & acc. & \$/1k & acc. & \$/1k \\
\midrule
\input{results-measured-rows} \\
\bottomrule
\end{tabular}
\end{table*}

\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{figures/accuracy_vs_cost.pdf}
\caption{Accuracy vs.\ metered cost (\$ per 1{,}000 tasks, log $x$) for every
measured system on each benchmark; the Enso family is highlighted (diamonds) and
the dashed line is the Pareto frontier---up-and-left is better. \texttt{enso} sits
on or near the frontier on every bench: frontier-competitive accuracy well left of
the premium single arms. Generated from \texttt{enso-bench}
\texttt{results/summary.json}.}
\label{fig:frontier}
\end{figure*}

\paragraph{Reading the table.} The hypothesis---that the router's per-request
argmax tracks the pool's upper envelope---holds where routing has room to work. On
GPQA Diamond \texttt{enso} scores \hEnsoGPQAacc\% at \$\hEnsoGPQAcost/1k against the
best single arm's \hBestGPQAacc\% (\texttt{\hBestGPQAname}). That margin is one
accuracy point, which at $n{=}198$ is two questions and sits inside the $\pm1.8$-point
binomial standard error of either cell, so the accuracy result is a \emph{match},
not a lead, and we do not read it as one. What the row does establish is the cost:
the same accuracy as the best arm while beating the premium reasoning arm
(Opus~4.8, \hOpusGPQAacc\% at \$\hOpusGPQAcost/1k) at roughly a fifth of its metered
spend, because the router inherits the pool's upper envelope rather than any one
arm's average and prices each answer at the arm that served it. On the open-ended
benches \texttt{enso-ultra} raises accuracy to the best single arm's level. We do
\emph{not} claim a clean sweep---on HLE the strongest single arm (\texttt{fable-5})
leads both Enso SKUs, exactly the single-benchmark regression a per-row read
surfaces and an aggregate would hide. That is the coarse-router property we stated
up front (\S\ref{sec:routing}): on a homogeneous science/knowledge set the router
mostly tracks the arm it routes to, and its edge there is cost, not a new accuracy
ceiling.

\paragraph{Ablation: verify-then-select is better \emph{and} cheaper.}
Table~\ref{tab:ablation} isolates the Enso-Ultra fusion logic. The shipped
verify-then-select synthesizer---a self-consistency fast path (take the agreeing
answer with no extra call when $\ge$2 of 3 arms concur), else a judge-as-selector
that picks one candidate rather than merging---is compared against a v1
blind-synthesis baseline on the two open-ended benches. Verify-then-select is
\emph{both} more accurate and cheaper, because it skips the synthesis call whenever
the arms already agree; the ceiling SKU's shipped logic dominates the naive one on
both axes.

\begin{table}[t]
\centering\small\setlength{\tabcolsep}{4pt}
\caption{Enso-Ultra fusion ablation (measured): v1 blind-synthesis vs the shipped
v2 verify-then-select on the open-ended benches. $\Delta$ is v2$-$v1 accuracy;
cost is \$ per 1{,}000 tasks. Better \emph{and} cheaper on both.}
\label{tab:ablation}
\begin{tabular}{@{}lrrrrrrrr@{}}
\toprule
& \multicolumn{3}{c}{v1 blind-synth} & \multicolumn{3}{c}{v2 verify-select} & & \\
\cmidrule(lr){2-4}\cmidrule(lr){5-7}
Benchmark & acc. & $n$ & \$/1k & acc. & $n$ & \$/1k & $\Delta$ & cost \\
\midrule
\input{results-ablation-rows} \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{Agentic step-routing (pilot).} The agentic benchmarks are where frontier
arms diverge most, and the most expensive to run; the full suite is
\emph{run pending} (\S\ref{sec:limits}). As a pilot we ran a common $18$-instance
subset of SWE-Bench Pro~\cite{swebench} with a step-level router---each agent step
(localize, edit, test) routed independently---against a single-Opus~4.8 agent under
the same harness. Enso step-routing resolved $38.9\%$ ($7/18$) to single-Opus's
$16.7\%$ ($3/18$), at lower metered cost (\$19.70 vs \$27.56), because different
steps prefer different arms. This is a small pilot, labelled as such, not the full
run.

\subsection{Routing overhead: microseconds on CPU}
\label{sec:overhead}

Accuracy and cost are the axes a buyer reads; a router adds a third the buyer
never sees but always pays---the decision itself. We measured it. Table~\ref{tab:overhead}
reports the per-request \emph{decision} latency of each router tier on a
\emph{single aarch64 CPU core with no GPU}: the heuristic tier via Go's
\texttt{go test -bench}, the learned tier via the \texttt{enso} crate's
\texttt{bench\_overhead} integration test (\texttt{cargo test -p enso --release}),
each timed over $10^{4}$ iterations after $10^{3}$ warm-up on a $\sim$1k-character
prompt against the trained arm table.

Three findings. \textbf{(i)}~The keyword-rule heuristic---the default gateway path
that classifies a request into a task bucket and reads a preference table---decides
in $\sim$0.3\,$\mu$s with a single allocation; the whole heuristic route decision
(classify plus feasible-arm lookup) is still $\sim$0.9\,$\mu$s. \textbf{(ii)}~The
learned \texttt{enso} policy decides end-to-end in \textbf{12.4\,$\mu$s}
(p99 14.8\,$\mu$s): 11.4\,$\mu$s to featurize the prompt (hashed n-grams over the
raw text, \S\ref{sec:routing}) and \textbf{229\,ns} for the bilinear $x^{\top}Wp$
score-and-select over the arm table. There is \emph{no embedding call and no GPU}
on this path---the featurizer is deliberately cheap hashed n-grams so that routing
never needs a model to route. \textbf{(iii)}~A vector/embedding-similarity variant
(cosine nearest-concept over $K{=}20$ model vectors in 768 dimensions) computes its
decision in $\sim$8\,$\mu$s, but that figure \emph{excludes} the prompt embedding it
requires---a model forward pass costing on the order of milliseconds, i.e.
$10^{2}$--$10^{3}\times$ the entire \texttt{enso} decision, and it dominates the
tier end-to-end. That domination is exactly why \texttt{enso} featurizes with hashed
n-grams rather than embeddings: it buys the same per-request signal at
two-to-three orders of magnitude lower latency and with no accelerator.

The consequence for deployment is the point. A served frontier completion runs from
$\sim$$10^{2}$\,ms for a short chat to $\sim$$10^{1}$\,s for an agentic or reasoning
trajectory; the routing decision that precedes it is four to six orders of magnitude
cheaper and runs on a CPU core the gateway already has. \emph{Routing never needs a
GPU---only the served model does.} The same holds for \emph{training} the policy:
the offline fit is a closed-form ridge solve and the online adaptation is a LinUCB
rank-one update (\S\ref{sec:routing}), both CPU-only, so an operator can fit and
continually retrain its own router without an accelerator.

\begin{table}[t]
\centering
\small
\setlength{\tabcolsep}{4.5pt}
\caption{Per-request routing \emph{decision} latency, measured on one aarch64 CPU
core (no GPU): the heuristic tier via \texttt{go test -bench}, the learned
\texttt{enso} tier via \texttt{cargo test -p enso --release --test bench\_overhead}
($10^{4}$ iterations, $10^{3}$ warm-up). The vector tier's compute \emph{excludes}
the embedding forward pass it requires ($\sim$ms), which dominates it end-to-end.
Routing is microseconds; the served LLM call is $10^{2}$--$10^{4}$\,ms.}
\label{tab:overhead}
\begin{tabular}{@{}lrl@{}}
\toprule
Router tier & Decision & Embedding \\
            & (CPU, 1 core) & call \\
\midrule
Heuristic keyword-rule (Go)        & $\sim$0.3\,$\mu$s   & none \\
Enso learned $x^{\top}Wp$ (Rust)   & 12.4\,$\mu$s        & none \\
\quad featurize / select           & 11.4 / 0.23\,$\mu$s & none \\
Vector NN, 768-d $K{=}20$ (Go)     & $\sim$8\,$\mu$s     & req.\ ($\sim$ms) \\
\bottomrule
\end{tabular}
\end{table}

\section{Economics: A Metered, Incentive-Aligned Router}
\label{sec:econ}

Enso ships to an organization as a metered product. An org enables its own
pool---the providers it holds keys for plus any internal models---and routes its
traffic through one \texttt{/v1} endpoint (\S\ref{sec:serving}). Two things then run
per org. First, a per-org policy state: the router carries that org's arm table and
a preference weight matrix $W$ that starts at the shared fit and adapts to the org's
traffic. Second, a reward loop: in-app feedback (accept/edit/thumbs) and an
LLM-as-judge that auto-scores each sampled response's quality both land as a scalar
reward $r$, which drives the same online \texttt{observe}$(u,x,p,r)$ update
\S\ref{sec:routing} specifies---here nudging the org's $W$, not only a per-user
$\theta_u$. The reward ledger is \emph{content-free}: it stores the feature vector
$x$, the chosen arm $p$, and the reward $r$---never the prompt or completion
text---so the mechanism that personalizes routing retains no customer content.

\subsection{Pricing}
\label{sec:pricing}

We meter the product as \textbf{a 1\% fee on routed LLM spend}: for a request whose
provider spend is $v$ dollars, Enso bills $f v$ with $f = 0.01$. A price is honest
only if it sits above what the thing costs to run and below the value it creates; we
show both bounds hold and derive them from the measured overhead of
\S\ref{sec:overhead} and the measured cost frontier of \S\ref{sec:eval}.

\paragraph{Cost to run.} Two components. The routing \emph{compute} is the decision
of \S\ref{sec:overhead}: 12\,$\mu$s of CPU. At a commodity \$0.04 per vCPU-hour that
is $\kappa \approx 1.3\times10^{-10}$ dollars per decision---nine orders of magnitude
below any per-request fee, i.e.\ negligible. The \emph{eval} component is the
LLM-judge: one cheap judge call of cost $c_j$ on a sampled fraction $s$ of requests,
so its amortized cost is $s\,c_j$ per routed request. With mean routed spend $\bar v$
per request, the run-cost per routed dollar is $(\kappa + s\,c_j)/\bar v$ and the
gross margin per routed dollar is
\begin{equation}
m \;=\; f \;-\; \frac{\kappa + s\,c_j}{\bar v}\;\approx\; f - \frac{s\,c_j}{\bar v}.
\end{equation}
The meter is profitable while $s < s^\star \equiv (f\bar v - \kappa)/c_j \approx
f\bar v / c_j$. Using values anchored to this paper's runs---$\bar v \approx \$0.02$
per request (the measured \texttt{enso} per-request cost on LiveCodeBench,
\S\ref{sec:eval}) and a small-model judge at $c_j \approx \$0.001$ (roughly 1--2k
judged tokens)---the break-even is $s^\star \approx 20\%$. At a sane sampling rate the
margin is wide: $s=10\%$ spends $0.5\%$ of routed dollars on eval (margin $0.5$ of
the $1\%$ fee); $s=5\%$ spends $0.25\%$ (margin $0.75$ of the fee). The judge is
sampled, not run on every request, precisely to keep it a rounding error against the
fee.

\paragraph{Value created.} Let $B$ be an org's spend under the monolith
baseline---one expensive model on every request---and let $S$ be the fraction Enso
saves by routing each request to the cheapest arm that still clears it. Post-routing
provider spend is $B(1-S)$, the fee is $fB(1-S)$, and the org's net saving as a
fraction of baseline is
\begin{equation}
1-(1-S)(1+f) \;=\; S - f(1-S)\;\ge\; S - f,
\end{equation}
so \textbf{the customer keeps at least $S-1$ percentage points} of its bill.
Symmetrically the fee is $f(1-S)/S$ of the value created---at $S=50\%$ that is $1\%$
of the savings; at the measured GPQA frontier below it is $0.25\%$. The
meter takes a small, bounded slice of a value it earns only when the customer saves.

\paragraph{Worked example (measured anchor).} On GPQA Diamond
(\S\ref{sec:eval}, Table~\ref{tab:measured}) \texttt{enso} scored \hEnsoGPQAacc\% at
\$\hEnsoGPQAcost{} per 1{,}000 tasks, while running the premium reasoning arm
(Opus~4.8) on the whole workload scored \hOpusGPQAacc\% at \$\hOpusGPQAcost{} per
1{,}000 tasks on the same \texttt{enso-bench} run---so routing here saves
$S \approx 80\%$ (a ${\sim}5\times$ cost reduction) at \emph{higher} accuracy. An org
paying that Opus rate is billed $0.01\times\$\hEnsoGPQAcost \approx \$0.11$ per
1{,}000 tasks, nets $\approx 79.8\%$ off its bill, and hands Enso $0.25\%$ of the
money Enso saved it. A more conservative workload saving $S=50\%$ still nets the org
$\approx 49.5\%$ after the fee. In both, the fee is bounded above by the value (a
single-digit-or-less percent of the savings) and above the run-cost (sub-$0.6\%$ of
routed spend at $s\le10\%$)---a positive-margin, incentive-aligned meter that wins
only when the customer does (Table~\ref{tab:econ}).

\begin{table}[t]
\centering
\small
\setlength{\tabcolsep}{6pt}
\caption{Pricing worked through, per routed dollar and for the measured
GPQA frontier. Fee $f=1\%$; judge sampling $s=10\%$, $c_j=\$0.001$,
$\bar v=\$0.02$ (\S\ref{sec:pricing}). Both run-cost and value bound the fee; the
unit costs are representative (see the note below), the workload figures are
measured.}
\label{tab:econ}
\begin{tabular}{@{}lr@{}}
\toprule
Quantity & Value \\
\midrule
Fee (of routed spend)                 & $1.00\%$ \\
\quad routing compute $\kappa/\bar v$ & ${\sim}7\times10^{-7}\%$ \\
\quad judge eval $s\,c_j/\bar v$      & $0.5\%$ \\
\quad gross margin $m$                & $0.5\%$ \\
\midrule
GPQA saving $S$                       & ${\sim}80\%$ \\
\quad fee as share of value           & $0.25\%$ \\
\quad customer net saving             & ${\sim}79.8\%$ \\
\bottomrule
\end{tabular}
\end{table}

\paragraph{Honest note on parameters.} The unit costs above---\$0.04 per vCPU-hour
and $c_j\approx\$0.001$ per judge call---are representative, not measured by us;
$\bar v$ and $S$ are taken from this paper's own \texttt{enso-bench} runs and are
workload-dependent (the $80\%$ GPQA saving is a favorable ceiling, not a
promise). The \emph{structure}---compute negligible, eval bounded by $s\,c_j$, fee
bounded below by run-cost and above by value---does not depend on the exact
constants.

\subsection{Case study: Hanzo Cloud on Enso}
\label{sec:casestudy}

Enso is not a proposal; it is the path Hanzo Cloud already runs. Every model request
to \texttt{hanzo.ai} sent with \texttt{model=auto} is routed by \texttt{enso} through
the same gateway (\S\ref{sec:serving}) whose measurements this paper reports---the
authors' production traffic and this evaluation share one routing plane and one
meter. The cost-per-1{,}000-tasks column of Table~\ref{tab:measured} is therefore not a
projection: it is the same per-request-metered figure that prices live \texttt{auto}
traffic, read off the production meter. The economics above describe the plane we
operate, not one we sketched.

\section{Limitations and Honest-Reporting Notes}
\label{sec:limits}

\paragraph{Two measurement regimes.} Table~\ref{tab:reference} is external and
reported by others---provider single-model cards plus a prior published router's
own numbers---and we never relabel any of it as Enso's. Table~\ref{tab:measured} is
the opposite: every arm \emph{and} every Enso SKU re-measured inside
\texttt{enso-bench} under one gateway, one prompt format, one scorer, so the
arm-vs-Enso comparison there is like-for-like. The two tables should not be read
across each other cell by cell---a provider-reported number can differ from our
controlled re-run in prompt formatting, sampling, tool access, and scoring.
Table~\ref{tab:reference} anchors scale; Table~\ref{tab:measured} adjudicates.

\paragraph{Agentic benchmarks: pilot only.} The agentic rows (SWE-Bench Pro,
Terminal-Bench 2.1) and LiveCodeBench Pro are the most expensive to run and the
last to land; beyond the $18$-instance SWE-Bench Pro pilot of \S\ref{sec:eval} they
are \emph{run pending}. Because agentic benchmarks are exactly where the frontier
arms diverge most---and where the pilot's step-routing gain is largest---these rows
are the ones most likely to move the Enso-vs-best-arm conclusion, and we flag that
the full agentic story is incomplete until they are filled.

\paragraph{Training-free is a choice, not a claim of dominance.} We argue Enso's
training-free orchestration is more maintainable under pool churn
(\S\ref{sec:ultra}); we do \emph{not} claim it beats a trained orchestrator such as
the prior published router (Ultra) on accuracy. The evaluation is what adjudicates
that, per row, and only once the pending cells are measured.

\paragraph{Numbers are data.} No number in this paper is invented. The external
reference (Table~\ref{tab:reference}) is reported-by-others constants, labelled as
such; every measured cell---Tables~\ref{tab:measured} and~\ref{tab:ablation}, the
frontier figure, and the headline numbers in the abstract---is injected at build
time from a generated file rather than typed into the table, and an unmeasured cell
is simply absent rather than fabricated. The cost column is a per-request-metered
figure from the serving plane (\S\ref{sec:serving}), not a blended list price. That
machinery guarantees the printed table matches a file; it does not guarantee the
file is the only one, and here it is not. The next paragraph says so.

\paragraph{Two generated files disagree on two benchmarks.} The measured cells
come from three generated files in this directory:
\texttt{results-measured-rows}, \texttt{results-headline} and
\texttt{results-ablation-rows}. A fourth, \texttt{results-prior}, carries the same
generator stamp and an earlier run, in the two-SKU format this paper no longer
uses. Nothing inputs it, and it disagrees with what we print:

\begin{center}\footnotesize\setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}llrr@{}}
\toprule
& & printed & earlier \\
\midrule
GPQA & \texttt{enso}       & 92.9 / \$11.1 & 87.9 / \$50.6 \\
     & \texttt{enso-ultra} & 92.9 / \$85.8 & 90.4 / \$80.2 \\
LCB  & \texttt{enso}       & 93.7 / \$17.5 & 91.4 / \$29.9 \\
     & \texttt{enso-ultra} & 89.1 / \$428  & 85.1 / \$410 \\
\bottomrule
\end{tabular}
\end{center}
\noindent Accuracy (\%) and metered cost per 1{,}000 tasks; LCB is
LiveCodeBench.

\noindent What can be established from the two files. Humanity's Last Exam and
CharXiv Reasoning agree between them exactly on accuracy and to the printed
precision on cost; only GPQA Diamond and LiveCodeBench differ. Repeated sampling
does not reproduce two of four benchmarks to the digit, so the later file is the
earlier one with those two benchmarks replaced and the other two carried forward.
The \texttt{enso-flash} column points the same way: it is populated for exactly
those two benchmarks and is a dash for the other two.

The accuracy gaps are small in items. At $n{=}198$, GPQA 92.9 against 87.9 is 184
against 174 questions correct---ten items, against a binomial standard error of
1.8--2.3 points on either cell. At $n{=}175$, LiveCodeBench 93.7 against 91.4 is
164 against 160, four items. The \emph{cost} gaps are not small and are not
sampling: \texttt{enso} on GPQA moves from \$50.6 to \$11.1 per 1{,}000 tasks, a
factor of 4.6, which is a routing or metering change rather than a re-roll.

What cannot be established: which run is the better estimate. \texttt{enso-bench}
and \texttt{scripts/emit\_paper.py} are on no machine we have, so neither run can be
re-executed and the two cannot be adjudicated by measurement. This paper prints the
later run throughout and is internally consistent in doing so; the earlier run is
preserved verbatim rather than deleted; and on the GPQA and LiveCodeBench rows the
spread between the two runs, not the binomial error alone, is the honest
uncertainty.

One independent artifact bears on the GPQA reading. The \texttt{enso} router
repository stores a per-item table of which arms answered each question correctly,
and replays the router against it with the fingerprint index refit five-fold so the
router is scored on questions absent from its own route table. Over the same item
counts that replay reports 92.9 (GPQA, $n{=}198$), 91.4 (LiveCodeBench, $n{=}175$),
38.2 (HLE, $n{=}500$) and 85.5 (CharXiv, $n{=}200$)---and on all four the strongest
single arm scores exactly the same as the router. Three of the four best-arm cells we print agree with
that replay to the digit---\texttt{fable-5} at 38.2 on HLE, \texttt{gpt-5.5} at
85.5 on CharXiv, and the LiveCodeBench cell we leave blank. The fourth does not:
the replay names \texttt{grok-4.5} at 92.9 as the strongest arm on GPQA, over a
wider eleven-arm pool than the one this paper serves, where we print
\texttt{\hBestGPQAname} at \hBestGPQAacc. A stored-table replay is a different
measurement from the live-gateway run reported here and we do not merge the two.
It does support reading the GPQA row as a match rather than a lead, which is how
\S\ref{sec:eval} now reads it.

\section{Conclusion}

No single frontier model leads every benchmark, so a monolith is a standing bet
against its own losing columns. Enso replaces that bet with a learned policy over a
pool of frontier arms: \texttt{enso} routes each request to the single arm most
likely to serve it best under a per-request cost/latency/quality objective, and
\texttt{enso-ultra} fans the hardest requests out to the top-$N$ arms and
synthesizes one answer. Both are one \texttt{/v1} product---one identity, one meter,
one 1M-context ladder---so the pool can change underneath without a client change.
Against the recent learned-orchestrator baseline, Enso shares the thesis and
the two-SKU shape but places all learning at the routing layer and keeps
orchestration training-free, so onboarding a new arm is a data change rather than a
retrain. We report the comparison the way a buyer of intelligence reads it---accuracy
and metered cost, per benchmark---and we report it honestly: measured where we have
measured, \emph{run pending} where we have not.

{\small
\bibliographystyle{plain}
\bibliography{references}
}

\end{document}