papers/zen/zen3-embedding_whitepaper.tex at main

mirror of https://github.com/hanzoai/papers.git synced 2026-10-29 07:34:31 +00:00

Files

T

zeekayandHanzo Dev bc1452e9f0 The zen claims that arithmetic settles

zenlm/papers had a sweeping withdrawal pass, 5deeeaa, and the owner reverted it in
f34f609 for asserting more than it verified. That objection sets the rule here: a
claim changes only where the proof is in the paper itself or in arithmetic, never
because a run could not be found. Twenty-seven of the eighty-one zen papers meet
that bar. Every correction quotes a number the paper already carries or one derived
from its own numbers by a division a reader can repeat.

Twelve papers led with a figure that appears in no row of the table it summarises.
zen-agent averaged GAIA 83.1/68.4/42.3 to 71.2 while its four baseline rows are each
the plain mean of their own three levels; the mean is 64.6. zen-embeddings-retrieval
and zen3-embedding claim MTEB averages of 72.4 and 74.3 over tables that supply their
task counts, where the 56-task weighted means are 67.1 and 67.4.
zen-mathematical-reasoning opened with five figures, none of them any row, and
explained one as best-of-32 sampling when its own sampling table puts best-of-32 at a
different number. zen-dso-protocol claimed 94% of centralized quality and a 78%
bandwidth cut against tables reading 99.4% and 99.8%-and-better, the second of them
the table that claim captions. zen-synthetic-data claimed an 8.4-point MT-Bench gain
on a scale that runs to 10, from a 7.84 baseline, where its table says 0.88.
zen-3d quoted the Chair column as the ShapeNet result. zen-multilingual understated
its own XNLI average. Also zen-hallucination-reduction, zen-distributed-training,
zen-finetuning, zen-legal-ai and zen-coder-flash.

Eight more are settled by arithmetic against the hardware or the parameter count.
zen-nano, zen-eco and zen-scribe each report INT4 needing as much memory as FP16 or
more, while their own FP16 and INT8 rows are exactly two and one bytes per parameter;
zen_family_overview already carried the right values. The two zen-designer papers size
a stated 235B model at 220 GB in FP16, which is a 110B model, and zen-quantization puts
the same model at 470 GB. zen-omni and zen-next recommend cards smaller than the memory
the same row requires. zen-live captions a latency table to an A10G when its own
deployment table assigns that 98 ms to the H100. zen-hardware-optimization calls a
throughput table FP8+INT4 when its cumulative table reaches the identical 31,200 tok/s
through INT4 + FP16.

Six papers share one template whose visual benchmark block bundles two metrics
computed over images a model generates with one computed over text a model emits
about an image. No model produces all three, and the block went to four papers across
four modalities, so each received at least one metric it cannot compute: VQA v2 and a
token rate for two diffusion models, FID and CLIP score for two vision-language models.
Each now keeps the rows its own stated output type can produce.

Throughput is the one category where the defect is the missing condition rather than
the number. Decode reads the active weight set once per token, so a single stream
cannot outrun bandwidth over footprint, and zen4-mini, zen3-nano and zen-coder-flash
all report above their own bound. Batched serving would make those figures legal and
none of the three records a batch size, so the numbers stand and each table now carries
its bound and the arithmetic behind it. AIME gets the same treatment in
zen-mathematical-reasoning: a score that is not a multiple of 1/30 on a 30-problem exam
is legal under averaging, and the caption now says which row has its sample count on
record and which do not.

What was examined and left is in LLM.md with the reason in each case. Training fleets
absent from this estate prove nothing about models trained on rented capacity.
zen-reranker's 98% semantic preservation is uncited rather than contradicted.
zen-safety-evaluation and zen-vision-architecture state no weighting, so their averages
are not determined. zen4-coder-flash puts FP8 on an A100, and Ampere has no FP8 tensor
cores, but its claims there are footprint claims and FP8 weight-only storage does run
on Ampere.

All twenty-seven rebuild under latexmk -Werror. Nineteen have diverged from their
counterparts in zenlm/papers and six exist only here, so those fixes do not reach the
other site. zen-3d and zen-agent were byte-identical across the two repositories until
now, and are the two to mirror first.

Co-authored-by: Hanzo Dev <[email protected]>

2026-09-10 23:54:37 -07:00

407 lines

15 KiB

TeX

Raw Permalink Blame History

 \documentclass[11pt,a4paper]{article}
 \usepackage[utf8]{inputenc}
 \usepackage[T1]{fontenc}
 \usepackage{amsmath,amsfonts,amssymb}
 \usepackage{graphicx}
 \usepackage{hyperref}
 \usepackage{listings}
 \usepackage{color}
 \usepackage{booktabs}
 \usepackage{float}
 \usepackage{geometry}
 \geometry{margin=1in}
 \definecolor{zenblue}{RGB}{41,121,255}
 \hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
 \title{\textbf{Zen3-Embedding: DSO-Native Semantic Embedding Model}\\
 \large Technical Whitepaper v2025.06}
 \author{Hanzo AI Research, Zen LM Research Team\\
 \texttt{[email protected]}\\
 \href{https://papers.zenlm.org}{papers.zenlm.org}}
 \date{June 2025}
 \begin{document}
 \maketitle
 \begin{abstract}
 Zen3-Embedding produces 7680-dimensional semantic representations optimized for DSO
 (Decentralized Semantic Optimization) retrieval, achieving strong multilingual embedding
 quality across the MTEB benchmark suite while maintaining BitDelta-compatible compression
 for efficient deployment. At 7B parameters, Zen3-Embedding achieves MTEB average 67.4\%,
 BEIR nDCG@10 57.8\% and MIRACL 81.2\%, surpassing prior
 embedding models with substantially smaller parameter counts. The model supports
 + languages through a shared multilingual encoder with language-balanced contrastive
 training, and its 7680-dimensional output space aligns natively with the DSO vector
 index format (ZIP-001), enabling direct integration with decentralized semantic search
 infrastructure without dimension projection overhead.
 \end{abstract}
 \tableofcontents
 \newpage
 %% -----------------------------------------------------------------------
 \section{Introduction}
 \label{sec:intro}
 Dense vector embeddings are the foundation of modern semantic search, retrieval-augmented
 generation (RAG), and knowledge-base indexing. The quality of embeddings determines the
 precision of document retrieval, which directly bounds the quality of RAG-powered
 applications. Yet most deployed embedding models face a trilemma:
 \begin{itemize}
 \item \textbf{Quality vs. size}: High-quality embeddings require large models, but API
       latency and memory constraints favor small models.
 \item \textbf{Generality vs. specialization}: General-purpose embeddings perform
       adequately on all tasks but rarely excel on any specific domain.
 \item \textbf{Monolingual vs. multilingual}: Multilingual models traditionally
       underperform language-specific models on high-resource languages.
 \end{itemize}
 Zen3-Embedding addresses all three through:
 \begin{enumerate}
 \item \textbf{DSO-aligned 7680-dimensional output space} --- The embedding dimension
       is chosen to match the DSO vector index native format (ZIP-001), eliminating
       projection overhead and enabling direct P2P semantic search on the decentralized
       network.
 \item \textbf{Instruction-aware embeddings} --- Following recent instruction-tuned
       embedding work, Zen3-Embedding accepts task instructions that shift the
       embedding geometry toward task-optimal representations without additional
       fine-tuning.
 \item \textbf{BitDelta-compressed index compatibility} --- The 7680-dimensional output
       is compatible with BitDelta (ZIP-007) 1.58-bit scalar quantization, reducing
       index storage from 30.7 KB/vector to 1.5 KB/vector with less than 2\% NDCG degradation.
 \item \textbf{Matryoshka representation learning (MRL)} --- Zen3-Embedding supports
       dimension truncation at 256, 512, 1024, 2048, 3840, and 7680, with each prefix
       independently optimized for retrieval quality.
 \end{enumerate}
 %% -----------------------------------------------------------------------
 \section{Architecture}
 \label{sec:arch}
 \subsection{Bidirectional Zen MoDE Encoder}
 Unlike the autoregressive Zen family language models, Zen3-Embedding uses a
 bidirectional (BERT-style) encoder architecture built on the Zen MoDE foundation.
 Full bidirectional attention allows each token to contextualize from the entire
 sequence, producing richer representations for embedding tasks than causal attention.
 Architecture specification:
 \begin{itemize}
 \item 7.1B parameters total (4.8B active per forward pass with MoDE sparsity)
 \item 48 transformer layers, 4096-dimensional hidden state
 \item 32 attention heads, 128-dimensional head size
 \item 64 experts, top-4 active per token
 \item Maximum input sequence length: 32,768 tokens (full attention)
 \item Longer inputs use hierarchical mean-pooling over 32K chunks
 \end{itemize}
 \subsection{Embedding Head Design}
 The final embedding is produced by a two-component pooling strategy:
 \begin{equation}
 \mathbf{e} = \text{LayerNorm}\left(\mathbf{W}_{\text{proj}} \cdot [\mathbf{h}_{\text{CLS}} \;\|\; \bar{\mathbf{h}}_{\text{mean}}]\right)
 \end{equation}
 where $\mathbf{h}_{\text{CLS}} \in \mathbb{R}^{4096}$ is the \texttt{[CLS]} token
 hidden state and $\bar{\mathbf{h}}_{\text{mean}} \in \mathbb{R}^{4096}$ is the
 mean-pooled sequence representation. The concatenated 8192-dimensional vector is
 projected to 7680 dimensions via $\mathbf{W}_{\text{proj}} \in \mathbb{R}^{7680 \times 8192}$.
 The dual-pooling design is motivated by the complementary nature of CLS (global semantic
 summary) and mean-pooling (compositional token-level semantics): combining both
 outperforms either alone by 1.2--2.4 points on MTEB retrieval tasks.
 \subsection{Matryoshka Representation Learning}
 MRL~\cite{mrl} enables dimension-adaptive embeddings by training the model to
 produce high-quality representations at all prefixes simultaneously:
 \begin{equation}
 \mathcal{L}_{\text{MRL}} = \sum_{d \in \mathcal{D}} w_d \cdot \mathcal{L}_{\text{contrastive}}\left(\mathbf{e}_{:d}, \mathbf{e}^+_{:d}, \{\mathbf{e}^-_i\}_{:d}\right)
 \end{equation}
 where $\mathcal{D} = \{256, 512, 1024, 2048, 3840, 7680\}$ are the supported
 truncation dimensions and $w_d$ is a dimension weight (larger dimensions receive
 higher weight since they have more capacity and are the primary deployment target).
 \subsection{DSO Vector Index Alignment}
 The DSO protocol (ZIP-001) specifies a 7680-dimensional float32 native embedding
 format for peer-to-peer semantic search nodes. Zen3-Embedding outputs are directly
 compatible without projection, enabling:
 \begin{itemize}
 \item Plug-in replacement for DSO network nodes
 \item Direct HNSW index construction from model output
 \item BitDelta scalar quantization (ZIP-007) to 1.58-bit per dimension
 \item Cross-node cosine similarity computation without format conversion
 \end{itemize}
 \subsection{Instruction-Aware Embedding}
 Task instructions are prepended to queries using a structured template:
 \begin{verbatim}
 Instruction: <task_description>
 Query: <query_text>
 \end{verbatim}
 The instruction shifts the attention pattern toward task-relevant dimensions of the
 embedding space. Evaluated instruction types include:
 \begin{itemize}
 \item \texttt{Retrieve relevant documents}: general retrieval
 \item \texttt{Find semantically similar sentences}: semantic similarity
 \item \texttt{Classify the following text}: classification
 \item \texttt{Find duplicates}: deduplication
 \end{itemize}
 Instructions are applied to queries only (not documents), preserving document index reusability.
 %% -----------------------------------------------------------------------
 \section{Training}
 \label{sec:training}
 \subsection{Pre-Training}
 Zen3-Embedding is initialized from the Zen3-Small (7B) base language model, then
 adapted to bidirectional attention by:
 \begin{enumerate}
 \item Removing the causal attention mask
 \item Adding \texttt{[CLS]} and \texttt{[SEP]} special tokens to the vocabulary
 \item Continued pre-training on masked language modeling (MLM) for 100B tokens
 \item Learning rate: $1 \times 10^{-5}$, warm-up 5\%, cosine decay
 \end{enumerate}
 \subsection{Contrastive Training Data}
 The contrastive training corpus contains 2.1B positive (query, document) pairs:
 \begin{itemize}
 \item Web search query-document pairs (mined from search logs): 800M
 \item Academic paper abstract-body pairs: 120M
 \item Question-answer pairs (StackExchange, community QA): 180M
 \item Multilingual parallel corpora (translation pairs as positives): 400M
 \item Code documentation pairs (function signature + docstring): 90M
 \item Synthetic pairs generated by Zen3-Medium (for hard positives): 250M
 \item Curated high-quality pairs (human-annotated): 260M
 \end{itemize}
 Hard negatives are mined using BM25 top-100 retrieval followed by reranking with
 a lightweight cross-encoder to select negatives that are lexically similar but
 semantically distinct (``hard BM25 negatives'').
 \subsection{Contrastive Learning Objective}
 Training uses InfoNCE loss with in-batch negatives plus 8 mined hard negatives per query:
 \begin{equation}
 \mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(\mathbf{q}, \mathbf{d}^+) / \tau)}{\exp(\text{sim}(\mathbf{q}, \mathbf{d}^+) / \tau) + \sum_{j=1}^{N} \exp(\text{sim}(\mathbf{q}, \mathbf{d}^-_j) / \tau)}
 \end{equation}
 where $\text{sim}(\cdot, \cdot)$ is cosine similarity, $\tau = 0.02$ is the
 temperature, $N = 8 + B - 1$ is the total number of negatives ($B$ is batch size,
 mined hard negatives per sample).
 GradCache~\cite{gradcache} is used to enable batch sizes of 32,768 without requiring
 proportional GPU memory, by accumulating gradients from sub-batches.
 \subsection{Multilingual Training}
 Language-balanced sampling ensures no language dominates training:
 \begin{itemize}
 \item High-resource languages (EN, ZH, ES, FR, DE, JA, KO, RU, AR, PT): 35\% total
 \item Mid-resource languages (50 languages): 45\% total
 \item Low-resource languages (50+ languages): 20\% total
 \end{itemize}
 Cross-lingual positive pairs (same meaning, different languages) are sampled at 25\%
 of training data, training the model to produce language-agnostic representations.
 %% -----------------------------------------------------------------------
 \section{Evaluation}
 \label{sec:eval}
 \subsection{MTEB Overview}
 \begin{table}[H]
 \centering
 \caption{MTEB benchmark performance by task category}
 \begin{tabular}{lccc}
 \toprule
 \textbf{Task Category} & \textbf{Zen3-Embedding} & \textbf{Zen2-Embedding} & \textbf{Tasks} \\
 \midrule
 Retrieval          & 57.8 & 53.1 & 15 \\
 Semantic Similarity & 87.4 & 84.2 & 10 \\
 Reranking          & 59.3 & 55.8 & 4  \\
 Classification     & 78.6 & 74.9 & 12 \\
 Clustering         & 51.2 & 47.4 & 11 \\
 Pair Classification & 86.9 & 83.1 & 3  \\
 Summarization      & 30.4 & 28.7 & 1  \\
 \textbf{MTEB Average} & \textbf{67.4} & \textbf{63.6} & 56 \\
 \bottomrule
 \end{tabular}
 \label{tab:mteb}
 \end{table}
 \subsection{BEIR Retrieval}
 \begin{table}[H]
 \centering
 \caption{BEIR retrieval benchmark (nDCG@10)}
 \begin{tabular}{lcc}
 \toprule
 \textbf{Dataset} & \textbf{Zen3-Embedding} & \textbf{Zen2-Embedding} \\
 \midrule
 MSMARCO         & 41.2 & 38.4 \\
 TREC-COVID      & 77.4 & 72.1 \\
 NFCorpus        & 36.8 & 33.9 \\
 NQ              & 52.3 & 48.7 \\
 HotpotQA        & 68.9 & 64.2 \\
 FEVER           & 78.1 & 73.4 \\
 SCIDOCS         & 19.4 & 17.8 \\
 SciFact         & 73.2 & 69.4 \\
 Quora           & 88.4 & 85.1 \\
 DBPedia         & 41.7 & 38.2 \\
 \textbf{BEIR Average} & \textbf{57.8} & \textbf{54.1} \\
 \bottomrule
 \end{tabular}
 \label{tab:beir}
 \end{table}
 \subsection{Multilingual Retrieval}
 \begin{table}[H]
 \centering
 \caption{MIRACL multilingual retrieval (nDCG@10)}
 \begin{tabular}{lcc}
 \toprule
 \textbf{Language} & \textbf{Zen3-Embedding} & \textbf{Zen2-Embedding} \\
 \midrule
 English (EN)   & 87.3 & 83.4 \\
 Chinese (ZH)   & 82.1 & 77.8 \\
 Arabic (AR)    & 78.9 & 73.1 \\
 Spanish (ES)   & 84.2 & 80.6 \\
 Japanese (JA)  & 79.4 & 74.2 \\
 Korean (KO)    & 81.3 & 76.9 \\
 Russian (RU)   & 79.8 & 75.1 \\
 French (FR)    & 83.7 & 79.4 \\
 German (DE)    & 82.4 & 78.2 \\
 Swahili (SW)   & 72.4 & 64.3 \\
 \textbf{Average (18 langs)} & \textbf{81.2} & \textbf{76.4} \\
 \bottomrule
 \end{tabular}
 \label{tab:miracl}
 \end{table}
 \subsection{Matryoshka Dimension Truncation}
 \begin{table}[H]
 \centering
 \caption{MTEB average score vs. embedding dimension (MRL truncation)}
 \begin{tabular}{lcc}
 \toprule
 \textbf{Dimension} & \textbf{MTEB Average} & \textbf{Storage per Vector (KB)} \\
 \midrule
 & 68.2 & 1.0  \\
 & 70.4 & 2.0  \\
 & 71.9 & 4.0  \\
 & 73.1 & 8.0  \\
 & 73.9 & 15.0 \\
 & 74.3 & 30.0 \\
 \textbf{7680 + BitDelta} & \textbf{72.8} & \textbf{1.5} \\
 \bottomrule
 \end{tabular}
 \label{tab:matryoshka}
 \end{table}
 %% -----------------------------------------------------------------------
 \section{Related Work}
 \label{sec:related}
 \subsection{Dense Retrieval}
 Dense passage retrieval~\cite{karpukhin2020dense} established bi-encoder architectures
 as practical alternatives to BM25 for open-domain question answering. Subsequent
 scaling work demonstrated that larger encoders consistently improve retrieval quality.
 Zen3-Embedding scales to 7B parameters while maintaining production-practical latency
 through the MoDE sparse activation pattern (4.8B active parameters per forward pass).
 \subsection{Contrastive Learning for Embeddings}
 SimCSE~\cite{gao2021simcse} demonstrated that contrastive learning with dropout as
 data augmentation substantially improves semantic textual similarity. E5~\cite{wang2022e5}
 and subsequent work showed that web-scale weakly supervised contrastive training
 with curated hard negatives matches or exceeds expensive human-annotated pairs.
 Zen3-Embedding synthesizes these insights at 7B scale.
 \subsection{Matryoshka Representations}
 Matryoshka Representation Learning~\cite{mrl} enables flexible dimension truncation
 with minimal quality loss, supporting diverse downstream deployment constraints.
 Zen3-Embedding is the first 7B-scale model to implement full MRL across 6 dimensions.
 %% -----------------------------------------------------------------------
 \section{Conclusion}
 \label{sec:conclusion}
 Zen3-Embedding delivers frontier retrieval quality at 7B scale with native DSO
 protocol compatibility, achieving MTEB 67.4\%, BEIR 57.8\%, and MIRACL 81.2\%.
 The 7680-dimensional embedding space, Matryoshka truncation support, BitDelta
 index compression, and instruction-aware embedding collectively address the quality,
 efficiency, and deployment flexibility requirements of production semantic search systems.
 Integration with the DSO network (ZIP-001) enables decentralized semantic search
 at scale without centralized index infrastructure, a capability unique to the
 Zen family embedding ecosystem.
 \section*{Acknowledgments}
 The authors thank Zoo Labs Foundation for DSO network integration support and
 MTEB benchmark evaluation infrastructure.
 \bibliographystyle{plain}
 \begin{thebibliography}{9}
 \bibitem{karpukhin2020dense}
 V. Karpukhin et al.,
 ``Dense Passage Retrieval for Open-Domain Question Answering,''
 \textit{EMNLP}, 2020.
 \bibitem{gao2021simcse}
 T. Gao, X. Yao, D. Chen,
 ``SimCSE: Simple Contrastive Learning of Sentence Embeddings,''
 \textit{EMNLP}, 2021.
 \bibitem{wang2022e5}
 L. Wang et al.,
 ``Text Embeddings by Weakly-Supervised Contrastive Pre-Training,''
 \textit{arXiv:2212.03533}, 2022.
 \bibitem{mrl}
 A. Kusupati et al.,
 ``Matryoshka Representation Learning,''
 \textit{NeurIPS}, 2022.
 \bibitem{gradcache}
 L. Gao, J. Callan,
 ``Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup,''
 \textit{RepL4NLP Workshop}, 2021.
 \bibitem{zip001}
 Zoo Labs Foundation,
 ``ZIP-001: Decentralized Semantic Optimization (DSO),''
 \textit{Zoo Improvement Proposals}, 2024.
 \bibitem{zip007}
 Zoo Labs Foundation,
 ``ZIP-007: BitDelta --- Quantization-Aware Delta Compression for Language Models,''
 \textit{Zoo Improvement Proposals}, 2024.
 \end{thebibliography}
 \end{document}

407 lines 15 KiB TeX Raw Permalink Blame History

407 lines

15 KiB

TeX

Raw Permalink Blame History