papers/zen/zen-mixture-of-experts.tex

\documentclass[11pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{amsmath,amsfonts,amssymb}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{listings}
\usepackage{color}
\usepackage{booktabs}
\usepackage{float}
\usepackage{geometry}
\geometry{margin=1in}
\definecolor{zenblue}{RGB}{41,121,255}
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}

\title{\textbf{Zen MoDE: Mixture of Diverse Experts}\\
\large Complexity-Aware Routing over Frozen Experts Harvested from Open Models}
\author{Zen LM Research, Hanzo AI\\
\texttt{[email protected]}}
\date{2026}

\begin{document}
\maketitle

\begin{abstract}
We present Zen MoDE (Mixture of Diverse Experts), the architecture underlying zen5.
MoDE inverts the usual economics of frontier training: rather than train experts, it
\emph{harvests} them. Expert modules are extracted from the largest open-weight models
available---spanning six architecturally distinct families---and served frozen. The only
trained parameters are a complexity estimator, a cross-architecture alignment layer, and
a per-tier router: roughly 394M parameters, or 0.013\% of the assembled pool. A
complexity-aware hierarchical router estimates task difficulty from a prompt's first
tokens and activates only the tier that task warrants, so a greeting costs 0.8B active
parameters and a proof costs 100B+. Because the experts are frozen and already trained,
the method reaches frontier scale for the cost of training a small router. This paper
describes the expert pool, the alignment and routing design, and the training procedure.
Evaluation is in progress and reported separately; we publish no benchmark numbers here
that we have not measured.
\end{abstract}

\section{Introduction}

Scaling a frontier model conventionally means training one: a single architecture, one
pretraining run, and a compute bill that grows with parameter count. Mixture-of-Experts
reduces the \emph{inference} cost of that model by activating a subset of its
parameters, but the experts are still trained together, from scratch, within one family.

The open-weight ecosystem now offers a different opportunity. Several organizations have
each spent enormous compute training large models with genuinely different inductive
biases---Gated DeltaNet, Lightning Attention, GLM MoE, DeepseekV3-style MoE, FP8 MoE.
Those weights are published. The expensive artifact---a trained expert---already exists;
what is missing is a way to use experts from different families together.

Zen MoDE is that mechanism. It treats published models as an \textbf{expert pool} to be
harvested rather than a baseline to be beaten, and trains only the machinery needed to
route across them:

\begin{enumerate}
  \item \textbf{Diverse, not distilled.} Experts are taken \emph{as-is} from six model
        families and never fine-tuned. Their diversity is the point: models trained by
        different groups, on different data, with different attention mechanisms learn
        different representations, and a router can exploit that.
  \item \textbf{Complexity-aware routing.} A lightweight estimator predicts task
        difficulty from the first 64 tokens and selects a tier. Standard MoE routes
        every token through the same-size network; MoDE varies the network.
  \item \textbf{Frozen experts, tiny trainable surface.} $\sim$394M trained parameters
        against a $\sim$3.1T pool---two to three orders of magnitude less training
        compute than pretraining a model of comparable capability.
\end{enumerate}

\section{Expert Pool}

Experts are harvested from open-weight models spanning six text families and four
additional modalities. Each source contributes its FFN/MoE blocks; weights are frozen at
extraction and never updated.

\begin{table}[H]
\centering
\caption{Text expert sources. Tier is the complexity band the source serves.}
\begin{tabular}{llrrl}
\toprule
\textbf{Source} & \textbf{Architecture} & \textbf{Total} & \textbf{Active} & \textbf{Tier} \\
\midrule
Qwen3.5-0.8B & Gated DeltaNet (dense)       & 0.8B  & 0.8B & T0 --- trivial \\
Qwen3.5-9B   & Gated DeltaNet (dense)       & 9B    & 9B   & T1 --- standard \\
MiniMax-M2.5 & Lightning Attention MoE      & 230B  & 10B  & T2 --- complex \\
GLM-5        & GLM MoE                      & 744B  & 40B  & T3 --- advanced \\
Kimi K2.5    & DeepseekV3 MoE (384 experts) & 1.04T & 32B  & T4 --- frontier \\
Ling-1T      & FP8 MoE                      & 1T    & 50B  & T4 --- frontier \\
\bottomrule
\end{tabular}
\end{table}

\begin{table}[H]
\centering
\caption{Multimodal expert sources.}
\begin{tabular}{lllr}
\toprule
\textbf{Modality} & \textbf{Source} & \textbf{Architecture} & \textbf{Total} \\
\midrule
Vision & Qwen3-VL-30B          & Vision-language transformer   & 30B \\
Video  & Wan2.2                & MoE diffusion (3D causal VAE) & $\sim$14B \\
Video  & CogVideoX-5B          & Expert transformer + 3D-RoPE  & 5B \\
3D     & TRELLIS.2 (Microsoft) & Rectified flow DiT + SC-VAE   & 4B \\
Audio  & Qwen3-Omni            & Thinker--Talker omni MoE      & 30B \\
\bottomrule
\end{tabular}
\end{table}

The assembled pool is $\sim$3.1T total parameters across 1000+ experts, of which at most
$\sim$189B are active on any request. Every source is used under its own license and is
credited, with that license, in the repository's NOTICE.

\section{Complexity-Aware Hierarchical Routing}

\subsection{The estimator}

Standard MoE applies the same computation to every token: a linear router selects
top-$K$ experts from one pool, and depth is fixed whether the prompt is a greeting or a
research question. MoDE first asks how hard the task is.

A 207M-parameter estimator reads the first 64 tokens and predicts a complexity tier
$t \in \{T0,\dots,T4\}$. Only that tier's experts are activated.

\begin{table}[H]
\centering
\caption{Complexity tiers and their activation budget.}
\begin{tabular}{llll}
\toprule
\textbf{Tier} & \textbf{Active} & \textbf{Latency target} & \textbf{Example} \\
\midrule
T0 & 0.8B      & $<$50ms  & ``Hello, how are you?'' \\
T1 & 9B        & $<$200ms & ``Summarize this article'' \\
T2 & 10--40B   & $<$2s    & ``Compare Keynesian vs.\ monetarist economics'' \\
T3 & 40--80B   & $<$5s    & ``Derive Black--Scholes from first principles'' \\
T4 & 80--100B+ & $<$10s   & ``Design a novel consensus algorithm'' \\
\bottomrule
\end{tabular}
\end{table}

The estimator is deliberately small: it must be cheap enough that running it on every
request is free relative to the compute it saves. Misprediction is recoverable---see
adaptive escalation.

\subsection{Cross-architecture alignment}

Experts from different families do not share a representation space. A 134M-parameter
alignment layer projects each source's hidden states into a shared space, which is what
permits a Gated DeltaNet expert and a DeepseekV3 MoE expert to serve the same request.
This layer, the estimator, and the router are the only trained components.

\subsection{Router}

The 52M-parameter MoDERouter performs per-tier expert selection. It adopts two published
results rather than reinventing them:

\begin{itemize}
  \item \textbf{ReMoE ReLU routing} (ICLR 2025)---ReLU gating gives an adaptive expert
        count per token instead of a fixed top-$K$, so token difficulty varies the
        expert budget \emph{within} a tier as the estimator varies it \emph{across}
        tiers.
  \item \textbf{MoE++ zero-computation experts} (ICLR 2025, oral)---zero, copy, and
        constant experts let trivial tokens bypass FFN compute entirely, reported
        upstream at 1.1--2.1$\times$ throughput.
\end{itemize}

\subsection{Adaptive escalation}

A tier chosen from 64 tokens can be wrong: a prompt may look simple and turn out to
require reasoning. When generation confidence drops below threshold, MoDE promotes the
request mid-generation to a higher tier. The estimator therefore need not be right, only
cheap and approximately right; escalation makes its error recoverable rather than fatal.

\section{Trainable Surface}

\begin{table}[H]
\centering
\caption{Trained parameters. Everything else is frozen.}
\begin{tabular}{lrl}
\toprule
\textbf{Component} & \textbf{Parameters} & \textbf{Role} \\
\midrule
ComplexityEstimator & 207M & Predicts the task's tier \\
AlignmentLayer      & 134M & Projects diverse architectures to a shared space \\
MoDERouter          & 52M  & Per-tier expert routing (ReLU gating) \\
\midrule
\textbf{Total} & \textbf{$\sim$394M} & \textbf{0.013\% of the $\sim$3.1T pool} \\
\bottomrule
\end{tabular}
\end{table}

This is the central economic claim, and it follows from the design rather than from a
measurement: if the experts are frozen and already trained, the training cost of the
assembled model is the training cost of the router.

\section{Training}

Training proceeds in five phases:

\begin{enumerate}
  \item \textbf{Expert extraction}---harvest FFN/MoE blocks from each source model.
  \item \textbf{Alignment pre-training}---learn the cross-architecture projections.
  \item \textbf{Router training}---train the complexity estimator and per-tier routers.
  \item \textbf{Integration}---joint fine-tuning of the trainable surface, experts frozen.
  \item \textbf{Omnimodal fusion}---add the vision, video, 3D, and audio experts.
\end{enumerate}

Hanzo's contribution on top of the harvested experts is confined to identity training,
agentic-data fine-tuning, and abliteration. The experts' capabilities are the source
models' own, and we claim none of them.

\section{Lineup}

\begin{table}[H]
\centering
\caption{zen5 configurations. Active parameters are a range because the tier is chosen per request.}
\begin{tabular}{lrll}
\toprule
\textbf{Model} & \textbf{Total} & \textbf{Active} & \textbf{Target} \\
\midrule
zen5       & 750B & 0.8--50B   & General \\
zen5-coder & 1.8T & 0.8--80B   & Code \\
zen5-omni  & 2.5T & 0.8--100B  & Omnimodal \\
zen5-max   & 3.1T & 0.8--100B+ & Frontier \\
\bottomrule
\end{tabular}
\end{table}

\section{Evaluation}

Evaluation is in progress. Routing accuracy, per-tier latency, and end-task quality are
measured by \texttt{scripts/benchmark.py} in the zen5 repository and will be reported
when the numbers are real. We publish none here. The architecture's claim---frontier
capability at router-training cost---is worth testing honestly, and a fabricated table
would only obscure whether it holds.

\section{Related Work}

MoDE builds directly on published results: ReMoE (ICLR 2025) for ReLU-gated adaptive
expert counts, MoE++ (ICLR 2025, oral) for zero-computation experts, and Transfusion for
unified autoregressive and diffusion modeling in a single architecture. Its novelty is
not any single mechanism but the composition: cross-architecture experts harvested from
independently trained open models, made interoperable by a learned alignment, and
selected by a complexity estimator that varies compute with task difficulty. We are not
aware of an existing model that combines these.

\section*{Availability}

Architecture, extraction, and router-training code: \url{https://github.com/zenlm/zen5}.
Each harvested source is credited, with its license, in that repository's NOTICE.

\end{document}