papers/zen/zen3-nano_whitepaper.tex at main

mirror of https://github.com/hanzoai/papers.git synced 2026-10-30 14:58:42 +00:00

Files

T

zeekayandHanzo Dev bc1452e9f0 The zen claims that arithmetic settles

zenlm/papers had a sweeping withdrawal pass, 5deeeaa, and the owner reverted it in
f34f609 for asserting more than it verified. That objection sets the rule here: a
claim changes only where the proof is in the paper itself or in arithmetic, never
because a run could not be found. Twenty-seven of the eighty-one zen papers meet
that bar. Every correction quotes a number the paper already carries or one derived
from its own numbers by a division a reader can repeat.

Twelve papers led with a figure that appears in no row of the table it summarises.
zen-agent averaged GAIA 83.1/68.4/42.3 to 71.2 while its four baseline rows are each
the plain mean of their own three levels; the mean is 64.6. zen-embeddings-retrieval
and zen3-embedding claim MTEB averages of 72.4 and 74.3 over tables that supply their
task counts, where the 56-task weighted means are 67.1 and 67.4.
zen-mathematical-reasoning opened with five figures, none of them any row, and
explained one as best-of-32 sampling when its own sampling table puts best-of-32 at a
different number. zen-dso-protocol claimed 94% of centralized quality and a 78%
bandwidth cut against tables reading 99.4% and 99.8%-and-better, the second of them
the table that claim captions. zen-synthetic-data claimed an 8.4-point MT-Bench gain
on a scale that runs to 10, from a 7.84 baseline, where its table says 0.88.
zen-3d quoted the Chair column as the ShapeNet result. zen-multilingual understated
its own XNLI average. Also zen-hallucination-reduction, zen-distributed-training,
zen-finetuning, zen-legal-ai and zen-coder-flash.

Eight more are settled by arithmetic against the hardware or the parameter count.
zen-nano, zen-eco and zen-scribe each report INT4 needing as much memory as FP16 or
more, while their own FP16 and INT8 rows are exactly two and one bytes per parameter;
zen_family_overview already carried the right values. The two zen-designer papers size
a stated 235B model at 220 GB in FP16, which is a 110B model, and zen-quantization puts
the same model at 470 GB. zen-omni and zen-next recommend cards smaller than the memory
the same row requires. zen-live captions a latency table to an A10G when its own
deployment table assigns that 98 ms to the H100. zen-hardware-optimization calls a
throughput table FP8+INT4 when its cumulative table reaches the identical 31,200 tok/s
through INT4 + FP16.

Six papers share one template whose visual benchmark block bundles two metrics
computed over images a model generates with one computed over text a model emits
about an image. No model produces all three, and the block went to four papers across
four modalities, so each received at least one metric it cannot compute: VQA v2 and a
token rate for two diffusion models, FID and CLIP score for two vision-language models.
Each now keeps the rows its own stated output type can produce.

Throughput is the one category where the defect is the missing condition rather than
the number. Decode reads the active weight set once per token, so a single stream
cannot outrun bandwidth over footprint, and zen4-mini, zen3-nano and zen-coder-flash
all report above their own bound. Batched serving would make those figures legal and
none of the three records a batch size, so the numbers stand and each table now carries
its bound and the arithmetic behind it. AIME gets the same treatment in
zen-mathematical-reasoning: a score that is not a multiple of 1/30 on a 30-problem exam
is legal under averaging, and the caption now says which row has its sample count on
record and which do not.

What was examined and left is in LLM.md with the reason in each case. Training fleets
absent from this estate prove nothing about models trained on rented capacity.
zen-reranker's 98% semantic preservation is uncited rather than contradicted.
zen-safety-evaluation and zen-vision-architecture state no weighting, so their averages
are not determined. zen4-coder-flash puts FP8 on an A100, and Ampere has no FP8 tensor
cores, but its claims there are footprint claims and FP8 weight-only storage does run
on Ampere.

All twenty-seven rebuild under latexmk -Werror. Nineteen have diverged from their
counterparts in zenlm/papers and six exist only here, so those fixes do not reach the
other site. zen-3d and zen-agent were byte-identical across the two repositories until
now, and are the two to mirror first.

Co-authored-by: Hanzo Dev <[email protected]>

2026-09-10 23:54:37 -07:00

384 lines

15 KiB

TeX

Raw Permalink Blame History

 \documentclass[11pt,a4paper]{article}
 \usepackage[utf8]{inputenc}
 \usepackage[T1]{fontenc}
 \usepackage{amsmath,amsfonts,amssymb}
 \usepackage{graphicx}
 \usepackage{hyperref}
 \usepackage{listings}
 \usepackage{color}
 \usepackage{booktabs}
 \usepackage{float}
 \usepackage{geometry}
 \geometry{margin=1in}
 \definecolor{zenblue}{RGB}{41,121,255}
 \hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
 \title{\textbf{Zen3-Nano: Third-Generation Ultra-Compact Edge Intelligence}\\
 \large Technical Whitepaper v2025.06}
 \author{Hanzo AI Research, Zen LM Research Team\\
 \texttt{[email protected]}\\
 \href{https://papers.zenlm.org}{papers.zenlm.org}}
 \date{June 2025}
 \begin{document}
 \maketitle
 \begin{abstract}
 Zen3-Nano is the third-generation 0.6B parameter edge model from the Zen family, achieving
 significant quality improvements over Zen-Nano through architectural refinements and enhanced
 knowledge distillation from the full Zen family hierarchy. Built on the Zen MoDE (Mixture of
 Distilled Experts) architecture, Zen3-Nano maintains a sub-250MB quantized footprint while
 delivering improved reasoning capabilities on mobile and embedded hardware. At 4-bit
 quantization the model fits in 250MB and achieves 2800 tokens per second on Apple A16 Bionic
 silicon, enabling real-time language inference at the edge without network connectivity.
 The model sets new state-of-the-art results for the sub-1B parameter class: MMLU 47.3\%,
 TriviaQA 61.2\%, and GSM8K 52.4\%, demonstrating that frontier distillation techniques close
 the gap between edge and server-class models substantially.
 \end{abstract}
 \tableofcontents
 \newpage
 %% -----------------------------------------------------------------------
 \section{Introduction}
 \label{sec:intro}
 On-device AI inference has become a primary deployment target for consumer applications,
 IoT sensors, and privacy-sensitive enterprise workloads. The constraints are severe: models
 must fit within a few hundred megabytes, execute without accelerators on ARM Cortex-class
 CPUs, and return responses in tens of milliseconds. Existing sub-1B approaches sacrifice
 too much quality to be practically useful for general-purpose instruction following.
 Zen3-Nano addresses this challenge by combining three advances:
 \begin{enumerate}
 \item \textbf{Hierarchical Zen MoDE distillation} --- structured distillation cascaded through
       Zen3-Mini (1.7B), Zen3-Small (7B), and Zen3-Medium (14B) teachers, each contributing
       a complementary signal.
 \item \textbf{Third-generation Nano architecture} --- refined attention with grouped-query
       heads optimized for cache reuse on low-memory hardware, reduced KV cache footprint,
       and fused MLPs that lower inference memory bandwidth.
 \item \textbf{BitDelta-aware training} --- the model is trained with quantization-aware
       objectives derived from the Zoo Labs BitDelta protocol (ZIP-007), so the 4-bit
       quantized artifact retains near-full-precision accuracy without post-hoc calibration.
 \end{enumerate}
 Zen3-Nano is designed to run natively on:
 \begin{itemize}
 \item Mobile SoCs (Apple A16 Bionic, Qualcomm Snapdragon 8 Gen 3, MediaTek Dimensity 9300)
 \item Microcontrollers with $\geq$512 KB SRAM (via WASM SIMD runtime)
 \item Browser environments via WebAssembly and WebGPU
 \item Embedded Linux (Raspberry Pi 5, NVIDIA Jetson Nano)
 \end{itemize}
 %% -----------------------------------------------------------------------
 \section{Architecture}
 \label{sec:arch}
 \subsection{Zen MoDE Foundation}
 Zen3-Nano is built on the Zen MoDE (Mixture of Distilled Experts) architecture, a
 design principle shared across the entire Zen family that replaces monolithic dense
 feed-forward blocks with sparsely-activated expert clusters. At the 0.6B scale, MoDE
 manifests as a lightweight routing mechanism that activates a subset of micro-experts
 per token, preserving capacity diversity while keeping the active parameter count
 well below memory limits.
 Formally, for input token representation $\mathbf{x} \in \mathbb{R}^d$, the MoDE feed-forward
 layer computes:
 \begin{equation}
 \text{MoDE}(\mathbf{x}) = \sum_{i \in \mathcal{K}(\mathbf{x})} g_i(\mathbf{x}) \cdot E_i(\mathbf{x})
 \end{equation}
 where $\mathcal{K}(\mathbf{x})$ is the set of top-$k$ activated experts selected by a
 learned routing function, $g_i$ are normalized routing weights, and $E_i$ are the
 expert sub-networks.
 \subsection{Grouped-Query Attention (GQA) Variant}
 Standard multi-head attention at 0.6B scale incurs disproportionate KV cache overhead
 relative to total parameter count. Zen3-Nano uses a 3rd-generation grouped-query attention
 variant with the following configuration:
 \begin{itemize}
 \item 16 query heads grouped into 4 key-value head groups
 \item 1024-dimensional hidden state, 64-dimensional head size
 \item Rotary positional embeddings (RoPE) with extended context via YaRN interpolation
 \item 32 transformer layers with pre-layer normalization (RMSNorm)
 \end{itemize}
 The KV cache per token occupies only 512 bytes at full precision, enabling 32K token
 context windows within 16 MB of SRAM when using 8-bit KV quantization.
 \subsection{Fused MLP Design}
 The feed-forward blocks use a fused SwiGLU activation with weight-sharing across
 adjacent expert pairs, reducing total parameter count by 12\% relative to a naive
 MoDE implementation at equivalent quality:
 \begin{equation}
 \text{FFN}(\mathbf{x}) = \left(\sigma(\mathbf{W}_1 \mathbf{x}) \odot \mathbf{W}_2 \mathbf{x}\right) \mathbf{W}_3
 \end{equation}
 \subsection{Quantization-Native Design}
 Unlike post-hoc quantization, Zen3-Nano is trained with BitDelta quantization-aware
 training from epoch 1. The quantization error $\epsilon_Q$ is included in the training
 loss as a regularizer:
 \begin{equation}
 \mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}} + \mathcal{L}_{\text{KD}} + \lambda \cdot \mathbb{E}\left[\|\mathbf{W} - Q(\mathbf{W})\|_F^2\right]
 \end{equation}
 where $Q(\cdot)$ is the 4-bit quantization operator and $\lambda = 0.01$.
 %% -----------------------------------------------------------------------
 \section{Training}
 \label{sec:training}
 \subsection{Pre-Training Data}
 Zen3-Nano is initialized from scratch on a 2T token corpus drawn from:
 \begin{itemize}
 \item Curated web text (Common Crawl quality-filtered): 60\%
 \item Code repositories (GitHub, GitLab, permissive licenses): 20\%
 \item Books, academic papers, structured knowledge: 15\%
 \item Multilingual data (50+ languages, weighted by speaker population): 5\%
 \end{itemize}
 \subsection{Hierarchical Distillation}
 After base pre-training, Zen3-Nano undergoes a three-stage distillation curriculum:
 \textbf{Stage 1 (Token-level KL distillation)}: Soft targets from Zen3-Mini (1.7B) teacher
 over 500B tokens. Temperature $T=4.0$, KD weight $\alpha = 0.7$.
 \textbf{Stage 2 (Layer-level representation matching)}: Intermediate hidden states
 from Zen3-Small (7B) are used as representation targets via a learned projection head.
 Loss: cosine similarity maximization over layers $\{6, 12, 18, 24\}$.
 \textbf{Stage 3 (Reasoning chain distillation)}: Structured chain-of-thought traces
 generated by Zen3-Medium (14B) are used for supervised fine-tuning with a 3:1
 ratio of distilled traces to human annotations.
 \subsection{Instruction Fine-Tuning}
 Post-distillation instruction tuning uses a 50M token mix:
 \begin{itemize}
 \item General instruction following: 40\%
 \item Code generation and debugging: 25\%
 \item Mathematical reasoning (chain-of-thought): 20\%
 \item Multilingual instruction: 15\%
 \end{itemize}
 Training uses AdamW optimizer, peak learning rate $3 \times 10^{-4}$, cosine decay,
 batch size 512, over 3 epochs on the instruction mix.
 \subsection{RLHF and Constitutional Alignment}
 A lightweight RLHF pass using PPO with a 0.6B reward model (trained separately)
 is applied for 10K steps to reduce harmful outputs and improve instruction adherence,
 keeping the KL divergence from the SFT checkpoint below 0.1 nats.
 %% -----------------------------------------------------------------------
 \section{Evaluation}
 \label{sec:eval}
 \subsection{Language Understanding}
 \begin{table}[H]
 \centering
 \caption{Zen3-Nano vs. sub-1B baselines on core language benchmarks}
 \begin{tabular}{lcccc}
 \toprule
 \textbf{Model} & \textbf{Params} & \textbf{MMLU} & \textbf{TriviaQA} & \textbf{HellaSwag} \\
 \midrule
 Prior Zen-Nano     & 0.5B & 39.1 & 53.4 & 61.2 \\
 Zen2-Nano          & 0.6B & 43.7 & 57.8 & 66.4 \\
 \textbf{Zen3-Nano} & \textbf{0.6B} & \textbf{47.3} & \textbf{61.2} & \textbf{70.8} \\
 \bottomrule
 \end{tabular}
 \label{tab:lang}
 \end{table}
 \subsection{Mathematical Reasoning}
 \begin{table}[H]
 \centering
 \caption{Mathematical reasoning benchmarks}
 \begin{tabular}{lcccc}
 \toprule
 \textbf{Model} & \textbf{GSM8K} & \textbf{MATH} & \textbf{MGSM (avg)} & \textbf{ARC-C} \\
 \midrule
 Zen-Nano     & 38.2 & 11.3 & 31.4 & 54.1 \\
 Zen2-Nano    & 44.6 & 15.7 & 38.9 & 59.3 \\
 \textbf{Zen3-Nano} & \textbf{52.4} & \textbf{20.1} & \textbf{46.2} & \textbf{64.7} \\
 \bottomrule
 \end{tabular}
 \label{tab:math}
 \end{table}
 \subsection{Code Generation}
 \begin{table}[H]
 \centering
 \caption{Code generation benchmarks (pass@1)}
 \begin{tabular}{lccc}
 \toprule
 \textbf{Model} & \textbf{HumanEval} & \textbf{MBPP} & \textbf{LiveCodeBench} \\
 \midrule
 Zen-Nano     & 28.0 & 33.2 & 12.4 \\
 Zen2-Nano    & 34.1 & 40.5 & 18.7 \\
 \textbf{Zen3-Nano} & \textbf{42.7} & \textbf{49.3} & \textbf{26.1} \\
 \bottomrule
 \end{tabular}
 \label{tab:code}
 \end{table}
 \subsection{Inference Performance}
 \begin{table}[H]
 \centering
 \caption{Inference throughput across hardware targets (4-bit quantized). Batch size
 is not recorded, and the ms/tok column is the reciprocal of the tokens/second column
 rather than a separate measurement. One decode stream reads the resident weights once
 per token, which on the 4090 row (950 MB, 1{,}008 GB/s) bounds a single stream at about
 {,}060 tokens per second.}
 \begin{tabular}{lcccc}
 \toprule
 \textbf{Hardware} & \textbf{Precision} & \textbf{Tok/s} & \textbf{RAM (MB)} & \textbf{Latency (ms/tok)} \\
 \midrule
 Apple A16 Bionic      & INT4 & 2800 & 248 & 0.36 \\
 Snapdragon 8 Gen 3    & INT4 & 2340 & 248 & 0.43 \\
 Raspberry Pi 5        & INT4 &  420 & 248 & 2.38 \\
 WASM SIMD (browser)   & INT4 &  310 & 248 & 3.23 \\
 x86-64 (laptop CPU)   & INT4 & 1100 & 248 & 0.91 \\
 NVIDIA RTX 4090       & FP16 & 8400 & 950 & 0.12 \\
 \bottomrule
 \end{tabular}
 \label{tab:perf}
 \end{table}
 \subsection{Multilingual Performance}
 \begin{table}[H]
 \centering
 \caption{Multilingual benchmark (FLORES-200, spBLEU)}
 \begin{tabular}{lcc}
 \toprule
 \textbf{Language Pair} & \textbf{Zen3-Nano} & \textbf{Zen2-Nano} \\
 \midrule
 English $\to$ Chinese   & 24.3 & 20.1 \\
 English $\to$ Spanish   & 31.8 & 27.4 \\
 English $\to$ Arabic    & 18.9 & 14.7 \\
 English $\to$ Japanese  & 21.4 & 17.2 \\
 Average (50 pairs)      & 22.6 & 18.4 \\
 \bottomrule
 \end{tabular}
 \label{tab:multilingual}
 \end{table}
 %% -----------------------------------------------------------------------
 \section{Related Work}
 \label{sec:related}
 \subsection{Small Language Models}
 Research on sub-1B language models has accelerated as mobile deployment became
 commercially viable. Early work demonstrated that aggressive distillation from
 multi-billion parameter teachers could recover a substantial fraction of performance
 at a fraction of the parameter count~\cite{hinton2015distilling}. Subsequent work
 explored architectural modifications specifically suited to on-device constraints,
 including weight sharing, structured pruning, and low-rank factorization.
 \subsection{Quantization-Aware Training}
 Post-training quantization (PTQ) has become the default for edge deployment, but
 accuracy degradation at 4-bit is nontrivial for small models where the per-parameter
 information density is already high. BitDelta (ZIP-007) addresses this by encoding
 quantization error as an explicit training signal, showing that models trained
 quantization-natively outperform PTQ equivalents by 2--5 percentage points on
 reasoning benchmarks at 4-bit precision.
 \subsection{Knowledge Distillation Hierarchies}
 Prior work on hierarchical distillation~\cite{mirzadeh2020improved} demonstrated that
 using an intermediate ``teacher assistant'' model reduces the capacity gap and improves
 distillation efficiency. Zen3-Nano extends this to a three-level cascade, exploiting
 the full Zen family model hierarchy.
 %% -----------------------------------------------------------------------
 \section{Deployment and Runtime}
 \label{sec:deploy}
 \subsection{Supported Runtimes}
 Zen3-Nano ships with native support for:
 \begin{itemize}
 \item \textbf{llama.cpp} --- GGUF format, Q4\_K\_M quantization
 \item \textbf{ONNX Runtime} --- optimized for ARM and x86 SIMD
 \item \textbf{WebAssembly} --- via wasm-pack compiled inference engine
 \item \textbf{Core ML} --- iOS/macOS native accelerated inference
 \item \textbf{TensorFlow Lite} --- Android NPU acceleration
 \end{itemize}
 \subsection{Context Length}
 Default context window: 8192 tokens (configurable to 32K via YaRN at cost of
 increased RAM). At 32K context, KV cache occupies approximately 64 MB at INT8.
 \subsection{API Compatibility}
 Zen3-Nano exposes an OpenAI-compatible API when deployed via the Zen LM inference
 server, enabling drop-in replacement for any OpenAI SDK client.
 %% -----------------------------------------------------------------------
 \section{Conclusion}
 \label{sec:conclusion}
 Zen3-Nano establishes a new state of the art for sub-1B parameter language models
 through hierarchical Zen MoDE distillation, quantization-native training, and
 architectural refinements that maximize inference efficiency on ARM-class hardware.
 The model achieves MMLU 47.3\%, TriviaQA 61.2\%, and GSM8K 52.4\% --- results
 that would have required 3--7B parameter models in prior generations --- while
 fitting within 250 MB at 4-bit precision and achieving 2800 tok/s on Apple A16
 Bionic.
 Future work will target 1M token context windows via ring attention approximations
 suitable for low-memory hardware, and extended multimodal understanding via
 lightweight vision adapters compatible with the Zen3-Omni decoder.
 \section*{Acknowledgments}
 The authors thank the Zoo Labs Foundation for compute allocation and the DSO
 research network for benchmark curation and evaluation infrastructure.
 \bibliographystyle{plain}
 \begin{thebibliography}{9}
 \bibitem{hinton2015distilling}
 G. Hinton, O. Vinyals, and J. Dean,
 ``Distilling the Knowledge in a Neural Network,''
 \textit{NIPS Deep Learning Workshop}, 2015.
 \bibitem{mirzadeh2020improved}
 S. I. Mirzadeh et al.,
 ``Improved Knowledge Distillation via Teacher Assistant,''
 \textit{AAAI}, 2020.
 \bibitem{zip007}
 Zoo Labs Foundation,
 ``ZIP-007: BitDelta --- Quantization-Aware Delta Compression for Language Models,''
 \textit{Zoo Improvement Proposals}, 2024.
 \href{https://zips.zoo.ngo/zip-007}{zips.zoo.ngo/zip-007}
 \bibitem{hip002}
 Hanzo AI,
 ``HIP-002: Active Semantic Optimization (ASO),''
 \textit{Hanzo Improvement Proposals}, 2024.
 \end{thebibliography}
 \end{document}

384 lines 15 KiB TeX Raw Permalink Blame History

384 lines

15 KiB

TeX

Raw Permalink Blame History