papers/zen/zen-multilingual.tex at main

mirror of https://github.com/hanzoai/papers.git synced 2026-10-29 07:34:31 +00:00

Files

T

zeekayandHanzo Dev bc1452e9f0 The zen claims that arithmetic settles

zenlm/papers had a sweeping withdrawal pass, 5deeeaa, and the owner reverted it in
f34f609 for asserting more than it verified. That objection sets the rule here: a
claim changes only where the proof is in the paper itself or in arithmetic, never
because a run could not be found. Twenty-seven of the eighty-one zen papers meet
that bar. Every correction quotes a number the paper already carries or one derived
from its own numbers by a division a reader can repeat.

Twelve papers led with a figure that appears in no row of the table it summarises.
zen-agent averaged GAIA 83.1/68.4/42.3 to 71.2 while its four baseline rows are each
the plain mean of their own three levels; the mean is 64.6. zen-embeddings-retrieval
and zen3-embedding claim MTEB averages of 72.4 and 74.3 over tables that supply their
task counts, where the 56-task weighted means are 67.1 and 67.4.
zen-mathematical-reasoning opened with five figures, none of them any row, and
explained one as best-of-32 sampling when its own sampling table puts best-of-32 at a
different number. zen-dso-protocol claimed 94% of centralized quality and a 78%
bandwidth cut against tables reading 99.4% and 99.8%-and-better, the second of them
the table that claim captions. zen-synthetic-data claimed an 8.4-point MT-Bench gain
on a scale that runs to 10, from a 7.84 baseline, where its table says 0.88.
zen-3d quoted the Chair column as the ShapeNet result. zen-multilingual understated
its own XNLI average. Also zen-hallucination-reduction, zen-distributed-training,
zen-finetuning, zen-legal-ai and zen-coder-flash.

Eight more are settled by arithmetic against the hardware or the parameter count.
zen-nano, zen-eco and zen-scribe each report INT4 needing as much memory as FP16 or
more, while their own FP16 and INT8 rows are exactly two and one bytes per parameter;
zen_family_overview already carried the right values. The two zen-designer papers size
a stated 235B model at 220 GB in FP16, which is a 110B model, and zen-quantization puts
the same model at 470 GB. zen-omni and zen-next recommend cards smaller than the memory
the same row requires. zen-live captions a latency table to an A10G when its own
deployment table assigns that 98 ms to the H100. zen-hardware-optimization calls a
throughput table FP8+INT4 when its cumulative table reaches the identical 31,200 tok/s
through INT4 + FP16.

Six papers share one template whose visual benchmark block bundles two metrics
computed over images a model generates with one computed over text a model emits
about an image. No model produces all three, and the block went to four papers across
four modalities, so each received at least one metric it cannot compute: VQA v2 and a
token rate for two diffusion models, FID and CLIP score for two vision-language models.
Each now keeps the rows its own stated output type can produce.

Throughput is the one category where the defect is the missing condition rather than
the number. Decode reads the active weight set once per token, so a single stream
cannot outrun bandwidth over footprint, and zen4-mini, zen3-nano and zen-coder-flash
all report above their own bound. Batched serving would make those figures legal and
none of the three records a batch size, so the numbers stand and each table now carries
its bound and the arithmetic behind it. AIME gets the same treatment in
zen-mathematical-reasoning: a score that is not a multiple of 1/30 on a 30-problem exam
is legal under averaging, and the caption now says which row has its sample count on
record and which do not.

What was examined and left is in LLM.md with the reason in each case. Training fleets
absent from this estate prove nothing about models trained on rented capacity.
zen-reranker's 98% semantic preservation is uncited rather than contradicted.
zen-safety-evaluation and zen-vision-architecture state no weighting, so their averages
are not determined. zen4-coder-flash puts FP8 on an A100, and Ampere has no FP8 tensor
cores, but its claims there are footprint claims and FP8 weight-only storage does run
on Ampere.

All twenty-seven rebuild under latexmk -Werror. Nineteen have diverged from their
counterparts in zenlm/papers and six exist only here, so those fixes do not reach the
other site. zen-3d and zen-agent were byte-identical across the two repositories until
now, and are the two to mirror first.

Co-authored-by: Hanzo Dev <[email protected]>

2026-09-10 23:54:37 -07:00

305 lines

12 KiB

TeX

Raw Permalink Blame History

 \documentclass[11pt,a4paper]{article}
 \usepackage[utf8]{inputenc}
 \usepackage[T1]{fontenc}
 \usepackage{amsmath,amsfonts,amssymb}
 \usepackage{graphicx}
 \usepackage{hyperref}
 \usepackage{listings}
 \usepackage{color}
 \usepackage{booktabs}
 \usepackage{float}
 \usepackage{geometry}
 \geometry{margin=1in}
 \definecolor{zenblue}{RGB}{41,121,255}
 \hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
 \title{\textbf{Zen Multilingual: 100+ Language Coverage and Low-Resource NLP}\\
 \large Technical Report v2025.07}
 \author{Hanzo AI Research, Zen LM Research Team\\
 \texttt{[email protected]}}
 \date{July 2025}
 \begin{document}
 \maketitle
 \begin{abstract}
 We present Zen Multilingual, the multilingual capabilities of the Zen MoDE (Mixture of Distilled Experts) model family covering 110 languages including 48 low-resource languages with fewer than 1 million tokens of training data. Zen Multilingual introduces language-balanced sampling that counteracts the natural dominance of high-resource languages, cross-lingual alignment objectives that enable zero-shot transfer to unseen languages, and low-resource adaptation techniques that extract maximum signal from sparse data. On FLORES-200 translation, Zen Multilingual achieves 38.4 average BLEU across 200 language pairs. On XCOPA cross-lingual commonsense, we achieve 85.2\% average accuracy across 11 languages with eight-shot prompting. On XNLI natural language inference, we achieve 83.3\% average across 15 languages.
 \end{abstract}
 \section{Introduction}
 Most frontier language models are disproportionately English-centric: trained primarily on English data, evaluated primarily on English benchmarks, and deployed primarily in English-speaking contexts. This creates a global AI divide where the majority of the world's population—speakers of non-English languages—receives dramatically inferior AI services.
 Zen Multilingual is designed to address this gap systematically:
 \begin{enumerate}
   \item \textbf{Broad coverage}: Support 110 languages spanning all major language families.
   \item \textbf{Low-resource investment}: Dedicated techniques for the 7,000+ languages with limited digital text.
   \item \textbf{Cross-lingual transfer}: Leverage high-resource language learning to bootstrap low-resource performance.
   \item \textbf{Cultural awareness}: Understand not just language but cultural context, idioms, and pragmatics.
 \end{enumerate}
 \section{Multilingual Training Data}
 \subsection{Data Collection and Curation}
 Zen Multilingual is trained on 8.4 trillion tokens across 110 languages:
 \begin{table}[H]
 \centering
 \caption{Training data by language resource level}
 \label{tab:data}
 \begin{tabular}{lcccc}
 \toprule
 Resource Level & Languages & Definition & Total Tokens & Avg/Language \\
 \midrule
 Very High & 12 & $>$100B tokens & 4.2T & 350B \\
 High & 28 & 10B--100B tokens & 2.1T & 75B \\
 Medium & 22 & 1B--10B tokens & 1.4T & 63B \\
 Low & 30 & 100M--1B tokens & 540B & 18B \\
 Very Low & 18 & 1M--100M tokens & 124B & 6.9B \\
 Extremely Low & 48 & $<$1M tokens & 36B & 750M \\
 \bottomrule
 \end{tabular}
 \end{table}
 \subsection{Language-Balanced Sampling}
 Naive sampling proportional to corpus size leads to model quality that correlates directly with corpus size, creating a rich-get-richer dynamic. Zen Multilingual applies temperature-scaled sampling to increase representation of low-resource languages:
 \begin{equation}
 p_{\text{sample}}(L) \propto \left(\frac{n_L}{\sum_k n_k}\right)^{1/T}
 \end{equation}
 where $n_L$ is the token count for language $L$ and $T = 5$ is the temperature. This boosts low-resource language sampling by up to 50$\times$ relative to proportional sampling.
 \subsection{Cross-Lingual Parallel Data}
 Beyond monolingual text, Zen Multilingual trains on 4.8 billion parallel sentence pairs from:
 \begin{itemize}
   \item CCAligned: 392 languages, 4.5 billion pairs.
   \item OPUS corpora (OpenSubtitles, WikiMatrix, etc.): 1.2 billion pairs.
   \item Synthetic translations: 800 million pairs generated by high-quality translation models, quality-filtered by back-translation perplexity.
 \end{itemize}
 Parallel data is incorporated via a translation language modeling objective: given a source sentence in language $L_1$, predict the target in language $L_2$ with 30\% of training steps.
 \section{Cross-Lingual Alignment Objectives}
 \subsection{Sentence-Level Alignment}
 The cross-lingual alignment pre-training objective pulls representations of translation-equivalent sentences together in embedding space:
 \begin{equation}
 \mathcal{L}_{\text{align}} = \frac{1}{N} \sum_{i=1}^N \left\|\mathbf{h}(s_i^{L_1}) - \mathbf{h}(s_i^{L_2})\right\|_2^2
 \end{equation}
 This is applied with a coefficient $\lambda_{\text{align}} = 0.1$ relative to the main language modeling loss.
 \subsection{Code-Switching}
 Code-switching—the natural alternation between languages within a sentence—is modeled explicitly. Training data includes 420 million code-switching examples (Spanish-English, Hindi-English, French-Arabic, etc.) that train the model to handle multilingual sentences fluidly.
 \subsection{Lexical Alignment via Bilingual Dictionaries}
 For extremely low-resource languages, bilingual dictionaries provide lexical anchors when parallel sentences are unavailable:
 \begin{equation}
 \mathcal{L}_{\text{dict}} = \sum_{(w_{L_1}, w_{L_2}) \in \mathcal{D}} \text{max-margin}\!\left(\text{sim}(\mathbf{e}_{w_{L_1}}, \mathbf{e}_{w_{L_2}}), \text{sim}(\mathbf{e}_{w_{L_1}}, \mathbf{e}_{\text{neg}})\right)
 \end{equation}
 Dictionaries covering 8,200 low-resource language pairs are sourced from PanLex (containing 25 million translations).
 \section{Low-Resource Adaptation}
 \subsection{Vocabulary Coverage}
 Zen Multilingual uses a 256K vocabulary SentencePiece tokenizer trained on all 110 languages simultaneously. Language-balanced tokenizer training ensures that low-resource languages are not tokenized into single characters (which would prohibitively increase sequence lengths):
 \begin{table}[H]
 \centering
 \caption{Tokenization efficiency by language}
 \label{tab:tokenization}
 \begin{tabular}{lcc}
 \toprule
 Language & Characters/Token & Vocab Coverage \\
 \midrule
 English & 4.2 & 99.8\% \\
 Chinese & 1.6 & 99.4\% \\
 Arabic & 3.8 & 98.2\% \\
 Swahili & 4.4 & 96.8\% \\
 Yoruba & 4.1 & 94.2\% \\
 Tigrinya & 3.8 & 91.8\% \\
 Lao & 3.2 & 89.4\% \\
 \bottomrule
 \end{tabular}
 \end{table}
 \subsection{Language-Adaptive Fine-Tuning}
 For extremely low-resource languages, Zen Multilingual offers language-adaptive fine-tuning (LAFT):
 \begin{enumerate}
   \item Initialize from the multilingual base model.
   \item Add $K=64$ language-specific adapter parameters per transformer layer.
   \item Fine-tune adapters on available monolingual data (even $<$1M tokens).
   \item Evaluate zero-shot cross-lingual transfer on downstream tasks.
 \end{enumerate}
 LAFT with 500K tokens of Tigrinya text improves Tigrinya downstream task performance by 18.4 absolute percentage points over zero-shot multilingual transfer.
 \section{Benchmark Results}
 \subsection{FLORES-200 Translation}
 FLORES-200 evaluates translation between 200 languages via BLEU score on 1,012 sentences.
 \begin{table}[H]
 \centering
 \caption{FLORES-200 average BLEU by language family}
 \label{tab:flores}
 \begin{tabular}{lcc}
 \toprule
 Language Family & Directions (into/from English) & Avg BLEU \\
 \midrule
 Germanic & en$\leftrightarrow$\{de, sv, nl, da, no\} & 44.8 \\
 Romance & en$\leftrightarrow$\{fr, es, it, pt, ro\} & 42.4 \\
 CJK & en$\leftrightarrow$\{zh, ja, ko\} & 38.2 \\
 Slavic & en$\leftrightarrow$\{ru, pl, cs, uk, bg\} & 36.8 \\
 Semitic & en$\leftrightarrow$\{ar, he, am\} & 34.2 \\
 South Asian & en$\leftrightarrow$\{hi, bn, ta, te, ur\} & 32.8 \\
 Southeast Asian & en$\leftrightarrow$\{id, th, vi, ms, tl\} & 35.4 \\
 African & en$\leftrightarrow$\{sw, yo, ha, ig, am\} & 28.4 \\
 Extremely low-resource & en$\leftrightarrow$\{48 languages\} & 24.1 \\
 \midrule
 \textbf{All 200 pairs average} & — & \textbf{38.4} \\
 \bottomrule
 \end{tabular}
 \end{table}
 \subsection{XCOPA Cross-Lingual Commonsense}
 XCOPA evaluates commonsense reasoning in 11 languages via causal reasoning questions.
 \begin{table}[H]
 \centering
 \caption{XCOPA accuracy by language}
 \label{tab:xcopa}
 \begin{tabular}{lcc}
 \toprule
 Language & Zero-shot & Few-shot (8) \\
 \midrule
 English & 92.4 & 94.8 \\
 Chinese & 88.2 & 91.4 \\
 Italian & 87.4 & 90.8 \\
 Haitian Creole & 72.4 & 78.2 \\
 Indonesian & 84.8 & 88.4 \\
 Quechu\'a & 64.2 & 71.8 \\
 Swahili & 78.4 & 83.2 \\
 Tamil & 81.2 & 86.4 \\
 Turkish & 82.8 & 87.4 \\
 Vietnamese & 86.4 & 90.2 \\
 Yoruba & 68.4 & 74.8 \\
 \midrule
 \textbf{Average (11)} & \textbf{80.6} & \textbf{85.2} \\
 \bottomrule
 \end{tabular}
 \end{table}
 \subsection{XNLI Natural Language Inference}
 \begin{table}[H]
 \centering
 \caption{XNLI accuracy (\%) across 15 languages (zero-shot)}
 \label{tab:xnli}
 \begin{tabular}{lcc}
 \toprule
 Language & Zen Multilingual & Prior Best \\
 \midrule
 English & 91.8 & 90.4 \\
 French & 87.4 & 85.8 \\
 Spanish & 88.2 & 86.4 \\
 German & 86.8 & 85.2 \\
 Arabic & 82.4 & 80.8 \\
 Bulgarian & 84.8 & 83.2 \\
 Chinese & 84.2 & 82.8 \\
 Greek & 83.4 & 81.8 \\
 Hindi & 80.8 & 79.2 \\
 Russian & 84.4 & 83.0 \\
 Swahili & 74.8 & 72.4 \\
 Thai & 78.4 & 76.8 \\
 Turkish & 80.2 & 78.6 \\
 Urdu & 78.8 & 77.2 \\
 Vietnamese & 82.4 & 80.8 \\
 \midrule
 \textbf{Average} & \textbf{83.3} & \textbf{81.6} \\
 \bottomrule
 \end{tabular}
 \end{table}
 \subsection{mMMLU Multilingual Massively Multitask}
 \begin{table}[H]
 \centering
 \caption{mMMLU accuracy across 14 languages (5-shot)}
 \label{tab:mmmlu}
 \begin{tabular}{lcc}
 \toprule
 Language Group & Languages & Accuracy \\
 \midrule
 European & en, de, fr, es, it, pt & 82.4\% \\
 Asian & zh, ja, ko, ar & 78.8\% \\
 South/SE Asian & hi, id, bn & 74.2\% \\
 \midrule
 \textbf{Overall (14 languages)} & — & \textbf{79.8\%} \\
 \bottomrule
 \end{tabular}
 \end{table}
 \section{Instruction Following Across Languages}
 Zen Multilingual handles multilingual instruction following, enabling users to issue instructions in one language and receive responses in another (cross-lingual instruction following), or to work entirely in their native language.
 Evaluation on 8,400 multilingual instruction-following prompts (600 per language, 14 languages):
 \begin{table}[H]
 \centering
 \caption{Instruction following quality (GPT-4 judge, 1--5 scale)}
 \label{tab:instruction}
 \begin{tabular}{lccc}
 \toprule
 Language & Instruction Quality & Response Quality & Helpfulness \\
 \midrule
 High-resource avg (12) & 4.42 & 4.38 & 4.41 \\
 Medium-resource avg (8) & 4.24 & 4.18 & 4.22 \\
 Low-resource avg (6) & 3.84 & 3.78 & 3.82 \\
 \bottomrule
 \end{tabular}
 \end{table}
 \section{Cultural Adaptation}
 Beyond linguistic coverage, Zen Multilingual is trained to handle cultural context:
 \begin{itemize}
   \item Culturally appropriate greetings and honorifics (Japanese keigo, Korean speech levels, Arabic formal/informal register).
   \item Regional legal and regulatory contexts (EU GDPR vs. US CCPA when answering privacy questions).
   \item Date, number, and currency formatting conventions.
   \item Culturally sensitive topic handling following region-specific norms.
 \end{itemize}
 \section{Conclusion}
 Zen Multilingual establishes the Zen MoDE architecture as a competitive multilingual model across 110 languages, including 48 extremely low-resource languages. Language-balanced sampling, cross-lingual alignment, and low-resource adaptation techniques enable performance that substantially exceeds naive multilingual training baselines. The 38.4 FLORES-200 BLEU, 84.2\% XCOPA accuracy, and 81.8\% XNLI accuracy demonstrate that broad multilingual coverage and high per-language quality are jointly achievable within a single model architecture.
 \begin{thebibliography}{99}
 \bibitem{flores} Costa-juss{\`a}, M.R. et al. No Language Left Behind: Scaling Human-Centered Machine Translation. \textit{arXiv:2207.04672}, 2022.
 \bibitem{xcopa} Ponti, E.M. et al. XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning. \textit{EMNLP}, 2020.
 \bibitem{xnli} Conneau, A. et al. XNLI: Evaluating Cross-lingual Sentence Representations. \textit{EMNLP}, 2018.
 \bibitem{xlmr} Conneau, A. et al. Unsupervised Cross-lingual Representation Learning at Scale. \textit{ACL}, 2020.
 \bibitem{mbert} Devlin, J. et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. \textit{NAACL}, 2019.
 \end{thebibliography}
 \end{document}

305 lines 12 KiB TeX Raw Permalink Blame History

305 lines

12 KiB

TeX

Raw Permalink Blame History