papers/zen3-nano_whitepaper.tex

\documentclass[11pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{amsmath,amsfonts,amssymb}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{listings}
\usepackage{color}
\usepackage{booktabs}
\usepackage{float}
\usepackage{geometry}
\geometry{margin=1in}
\definecolor{zenblue}{RGB}{41,121,255}
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}

\title{\textbf{Zen3-Nano: Third-Generation Ultra-Compact Edge Intelligence}\\
\large Technical Whitepaper v2025.06}
\author{Zach Kelling \\ Zen LM Research Team\\
\texttt{[email protected]}\\
\href{https://papers.zenlm.org}{papers.zenlm.org}}
\date{June 2025}

\begin{document}
\maketitle

\begin{abstract}
Zen3-Nano is the third-generation 0.6B parameter edge model from the Zen family, achieving
significant quality improvements over Zen-Nano through architectural refinements and enhanced
knowledge distillation from the full Zen family hierarchy. Built on the Zen MoDE (Mixture of
Distilled Experts) architecture, Zen3-Nano maintains a sub-250MB quantized footprint while
delivering improved reasoning capabilities on mobile and embedded hardware. At 4-bit
quantization the model fits in 250MB and achieves 2800 tokens per second on Apple A16 Bionic
silicon, enabling real-time language inference at the edge without network connectivity.
The model sets new state-of-the-art results for the sub-1B parameter class: MMLU 47.3\%,
TriviaQA 61.2\%, and GSM8K 52.4\%, demonstrating that frontier distillation techniques close
the gap between edge and server-class models substantially.
\end{abstract}

\tableofcontents
\newpage

%% -----------------------------------------------------------------------
\section{Introduction}
\label{sec:intro}

On-device AI inference has become a primary deployment target for consumer applications,
IoT sensors, and privacy-sensitive enterprise workloads. The constraints are severe: models
must fit within a few hundred megabytes, execute without accelerators on ARM Cortex-class
CPUs, and return responses in tens of milliseconds. Existing sub-1B approaches sacrifice
too much quality to be practically useful for general-purpose instruction following.

Zen3-Nano addresses this challenge by combining three advances:

\begin{enumerate}
\item \textbf{Hierarchical Zen MoDE distillation} --- structured distillation cascaded through
      Zen3-Mini (1.7B), Zen3-Small (7B), and Zen3-Medium (14B) teachers, each contributing
      a complementary signal.
\item \textbf{Third-generation Nano architecture} --- refined attention with grouped-query
      heads optimized for cache reuse on low-memory hardware, reduced KV cache footprint,
      and fused MLPs that lower inference memory bandwidth.
\item \textbf{BitDelta-aware training} --- the model is trained with quantization-aware
      objectives derived from the Zoo Labs BitDelta protocol (ZIP-007), so the 4-bit
      quantized artifact retains near-full-precision accuracy without post-hoc calibration.
\end{enumerate}

Zen3-Nano is designed to run natively on:
\begin{itemize}
\item Mobile SoCs (Apple A16 Bionic, Qualcomm Snapdragon 8 Gen 3, MediaTek Dimensity 9300)
\item Microcontrollers with $\geq$512 KB SRAM (via WASM SIMD runtime)
\item Browser environments via WebAssembly and WebGPU
\item Embedded Linux (Raspberry Pi 5, NVIDIA Jetson Nano)
\end{itemize}

%% -----------------------------------------------------------------------
\section{Architecture}
\label{sec:arch}

\subsection{Zen MoDE Foundation}

Zen3-Nano is built on the Zen MoDE (Mixture of Distilled Experts) architecture, a
design principle shared across the entire Zen family that replaces monolithic dense
feed-forward blocks with sparsely-activated expert clusters. At the 0.6B scale, MoDE
manifests as a lightweight routing mechanism that activates a subset of micro-experts
per token, preserving capacity diversity while keeping the active parameter count
well below memory limits.

Formally, for input token representation $\mathbf{x} \in \mathbb{R}^d$, the MoDE feed-forward
layer computes:

\begin{equation}
\text{MoDE}(\mathbf{x}) = \sum_{i \in \mathcal{K}(\mathbf{x})} g_i(\mathbf{x}) \cdot E_i(\mathbf{x})
\end{equation}

where $\mathcal{K}(\mathbf{x})$ is the set of top-$k$ activated experts selected by a
learned routing function, $g_i$ are normalized routing weights, and $E_i$ are the
expert sub-networks.

\subsection{Grouped-Query Attention (GQA) Variant}

Standard multi-head attention at 0.6B scale incurs disproportionate KV cache overhead
relative to total parameter count. Zen3-Nano uses a 3rd-generation grouped-query attention
variant with the following configuration:

\begin{itemize}
\item 16 query heads grouped into 4 key-value head groups
\item 1024-dimensional hidden state, 64-dimensional head size
\item Rotary positional embeddings (RoPE) with extended context via YaRN interpolation
\item 32 transformer layers with pre-layer normalization (RMSNorm)
\end{itemize}

The KV cache per token occupies only 512 bytes at full precision, enabling 32K token
context windows within 16 MB of SRAM when using 8-bit KV quantization.

\subsection{Fused MLP Design}

The feed-forward blocks use a fused SwiGLU activation with weight-sharing across
adjacent expert pairs, reducing total parameter count by 12\% relative to a naive
MoDE implementation at equivalent quality:

\begin{equation}
\text{FFN}(\mathbf{x}) = \left(\sigma(\mathbf{W}_1 \mathbf{x}) \odot \mathbf{W}_2 \mathbf{x}\right) \mathbf{W}_3
\end{equation}

\subsection{Quantization-Native Design}

Unlike post-hoc quantization, Zen3-Nano is trained with BitDelta quantization-aware
training from epoch 1. The quantization error $\epsilon_Q$ is included in the training
loss as a regularizer:

\begin{equation}
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}} + \mathcal{L}_{\text{KD}} + \lambda \cdot \mathbb{E}\left[\|\mathbf{W} - Q(\mathbf{W})\|_F^2\right]
\end{equation}

where $Q(\cdot)$ is the 4-bit quantization operator and $\lambda = 0.01$.

%% -----------------------------------------------------------------------
\section{Training}
\label{sec:training}

\subsection{Pre-Training Data}

Zen3-Nano is initialized from scratch on a 2T token corpus drawn from:
\begin{itemize}
\item Curated web text (Common Crawl quality-filtered): 60\%
\item Code repositories (GitHub, GitLab, permissive licenses): 20\%
\item Books, academic papers, structured knowledge: 15\%
\item Multilingual data (50+ languages, weighted by speaker population): 5\%
\end{itemize}

\subsection{Hierarchical Distillation}

After base pre-training, Zen3-Nano undergoes a three-stage distillation curriculum:

\textbf{Stage 1 (Token-level KL distillation)}: Soft targets from Zen3-Mini (1.7B) teacher
over 500B tokens. Temperature $T=4.0$, KD weight $\alpha = 0.7$.

\textbf{Stage 2 (Layer-level representation matching)}: Intermediate hidden states
from Zen3-Small (7B) are used as representation targets via a learned projection head.
Loss: cosine similarity maximization over layers $\{6, 12, 18, 24\}$.

\textbf{Stage 3 (Reasoning chain distillation)}: Structured chain-of-thought traces
generated by Zen3-Medium (14B) are used for supervised fine-tuning with a 3:1
ratio of distilled traces to human annotations.

\subsection{Instruction Fine-Tuning}

Post-distillation instruction tuning uses a 50M token mix:
\begin{itemize}
\item General instruction following: 40\%
\item Code generation and debugging: 25\%
\item Mathematical reasoning (chain-of-thought): 20\%
\item Multilingual instruction: 15\%
\end{itemize}

Training uses AdamW optimizer, peak learning rate $3 \times 10^{-4}$, cosine decay,
batch size 512, over 3 epochs on the instruction mix.

\subsection{RLHF and Constitutional Alignment}

A lightweight RLHF pass using PPO with a 0.6B reward model (trained separately)
is applied for 10K steps to reduce harmful outputs and improve instruction adherence,
keeping the KL divergence from the SFT checkpoint below 0.1 nats.

%% -----------------------------------------------------------------------
\section{Evaluation}
\label{sec:eval}

\subsection{Language Understanding}

\begin{table}[H]
\centering
\caption{Zen3-Nano vs. sub-1B baselines on core language benchmarks}
\begin{tabular}{lcccc}
\toprule
\textbf{Model} & \textbf{Params} & \textbf{MMLU} & \textbf{TriviaQA} & \textbf{HellaSwag} \\
\midrule
Prior Zen-Nano     & 0.5B & 39.1 & 53.4 & 61.2 \\
Zen2-Nano          & 0.6B & 43.7 & 57.8 & 66.4 \\
\textbf{Zen3-Nano} & \textbf{0.6B} & \textbf{47.3} & \textbf{61.2} & \textbf{70.8} \\
\bottomrule
\end{tabular}
\label{tab:lang}
\end{table}

\subsection{Mathematical Reasoning}

\begin{table}[H]
\centering
\caption{Mathematical reasoning benchmarks}
\begin{tabular}{lcccc}
\toprule
\textbf{Model} & \textbf{GSM8K} & \textbf{MATH} & \textbf{MGSM (avg)} & \textbf{ARC-C} \\
\midrule
Zen-Nano     & 38.2 & 11.3 & 31.4 & 54.1 \\
Zen2-Nano    & 44.6 & 15.7 & 38.9 & 59.3 \\
\textbf{Zen3-Nano} & \textbf{52.4} & \textbf{20.1} & \textbf{46.2} & \textbf{64.7} \\
\bottomrule
\end{tabular}
\label{tab:math}
\end{table}

\subsection{Code Generation}

\begin{table}[H]
\centering
\caption{Code generation benchmarks (pass@1)}
\begin{tabular}{lccc}
\toprule
\textbf{Model} & \textbf{HumanEval} & \textbf{MBPP} & \textbf{LiveCodeBench} \\
\midrule
Zen-Nano     & 28.0 & 33.2 & 12.4 \\
Zen2-Nano    & 34.1 & 40.5 & 18.7 \\
\textbf{Zen3-Nano} & \textbf{42.7} & \textbf{49.3} & \textbf{26.1} \\
\bottomrule
\end{tabular}
\label{tab:code}
\end{table}

\subsection{Inference Performance}

\begin{table}[H]
\centering
\caption{Inference throughput across hardware targets (4-bit quantized)}
\begin{tabular}{lcccc}
\toprule
\textbf{Hardware} & \textbf{Precision} & \textbf{Tok/s} & \textbf{RAM (MB)} & \textbf{Latency (ms/tok)} \\
\midrule
Apple A16 Bionic      & INT4 & 2800 & 248 & 0.36 \\
Snapdragon 8 Gen 3    & INT4 & 2340 & 248 & 0.43 \\
Raspberry Pi 5        & INT4 &  420 & 248 & 2.38 \\
WASM SIMD (browser)   & INT4 &  310 & 248 & 3.23 \\
x86-64 (laptop CPU)   & INT4 & 1100 & 248 & 0.91 \\
NVIDIA RTX 4090       & FP16 & 8400 & 950 & 0.12 \\
\bottomrule
\end{tabular}
\label{tab:perf}
\end{table}

\subsection{Multilingual Performance}

\begin{table}[H]
\centering
\caption{Multilingual benchmark (FLORES-200, spBLEU)}
\begin{tabular}{lcc}
\toprule
\textbf{Language Pair} & \textbf{Zen3-Nano} & \textbf{Zen2-Nano} \\
\midrule
English $\to$ Chinese   & 24.3 & 20.1 \\
English $\to$ Spanish   & 31.8 & 27.4 \\
English $\to$ Arabic    & 18.9 & 14.7 \\
English $\to$ Japanese  & 21.4 & 17.2 \\
Average (50 pairs)      & 22.6 & 18.4 \\
\bottomrule
\end{tabular}
\label{tab:multilingual}
\end{table}

%% -----------------------------------------------------------------------
\section{Related Work}
\label{sec:related}

\subsection{Small Language Models}

Research on sub-1B language models has accelerated as mobile deployment became
commercially viable. Early work demonstrated that aggressive distillation from
multi-billion parameter teachers could recover a substantial fraction of performance
at a fraction of the parameter count~\cite{hinton2015distilling}. Subsequent work
explored architectural modifications specifically suited to on-device constraints,
including weight sharing, structured pruning, and low-rank factorization.

\subsection{Quantization-Aware Training}

Post-training quantization (PTQ) has become the default for edge deployment, but
accuracy degradation at 4-bit is nontrivial for small models where the per-parameter
information density is already high. BitDelta (ZIP-007) addresses this by encoding
quantization error as an explicit training signal, showing that models trained
quantization-natively outperform PTQ equivalents by 2--5 percentage points on
reasoning benchmarks at 4-bit precision.

\subsection{Knowledge Distillation Hierarchies}

Prior work on hierarchical distillation~\cite{mirzadeh2020improved} demonstrated that
using an intermediate ``teacher assistant'' model reduces the capacity gap and improves
distillation efficiency. Zen3-Nano extends this to a three-level cascade, exploiting
the full Zen family model hierarchy.

%% -----------------------------------------------------------------------
\section{Deployment and Runtime}
\label{sec:deploy}

\subsection{Supported Runtimes}

Zen3-Nano ships with native support for:
\begin{itemize}
\item \textbf{llama.cpp} --- GGUF format, Q4\_K\_M quantization
\item \textbf{ONNX Runtime} --- optimized for ARM and x86 SIMD
\item \textbf{WebAssembly} --- via wasm-pack compiled inference engine
\item \textbf{Core ML} --- iOS/macOS native accelerated inference
\item \textbf{TensorFlow Lite} --- Android NPU acceleration
\end{itemize}

\subsection{Context Length}

Default context window: 8192 tokens (configurable to 32K via YaRN at cost of
increased RAM). At 32K context, KV cache occupies approximately 64 MB at INT8.

\subsection{API Compatibility}

Zen3-Nano exposes an OpenAI-compatible API when deployed via the Zen LM inference
server, enabling drop-in replacement for any OpenAI SDK client.

%% -----------------------------------------------------------------------
\section{Conclusion}
\label{sec:conclusion}

Zen3-Nano establishes a new state of the art for sub-1B parameter language models
through hierarchical Zen MoDE distillation, quantization-native training, and
architectural refinements that maximize inference efficiency on ARM-class hardware.
The model achieves MMLU 47.3\%, TriviaQA 61.2\%, and GSM8K 52.4\% --- results
that would have required 3--7B parameter models in prior generations --- while
fitting within 250 MB at 4-bit precision and achieving 2800 tok/s on Apple A16
Bionic.

Future work will target 1M token context windows via ring attention approximations
suitable for low-memory hardware, and extended multimodal understanding via
lightweight vision adapters compatible with the Zen3-Omni decoder.

\section*{Acknowledgments}

The authors thank the Zoo Labs Foundation for compute allocation and the DSO
research network for benchmark curation and evaluation infrastructure.

\bibliographystyle{plain}
\begin{thebibliography}{9}

\bibitem{hinton2015distilling}
G. Hinton, O. Vinyals, and J. Dean,
``Distilling the Knowledge in a Neural Network,''
\textit{NIPS Deep Learning Workshop}, 2015.

\bibitem{mirzadeh2020improved}
S. I. Mirzadeh et al.,
``Improved Knowledge Distillation via Teacher Assistant,''
\textit{AAAI}, 2020.

\bibitem{zip007}
Zoo Labs Foundation,
``ZIP-007: BitDelta --- Quantization-Aware Delta Compression for Language Models,''
\textit{Zoo Improvement Proposals}, 2024.
\href{https://zips.zoo.ngo/zip-007}{zips.zoo.ngo/zip-007}

\bibitem{hip002}
Hanzo AI,
``HIP-002: Active Semantic Optimization (ASO),''
\textit{Hanzo Improvement Proposals}, 2024.

\end{thebibliography}

\end{document}