papers/zen/zen-live_whitepaper.tex at main

mirror of https://github.com/hanzoai/papers.git synced 2026-10-29 07:34:31 +00:00

Files

T

zeekayandHanzo Dev bc1452e9f0 The zen claims that arithmetic settles

zenlm/papers had a sweeping withdrawal pass, 5deeeaa, and the owner reverted it in
f34f609 for asserting more than it verified. That objection sets the rule here: a
claim changes only where the proof is in the paper itself or in arithmetic, never
because a run could not be found. Twenty-seven of the eighty-one zen papers meet
that bar. Every correction quotes a number the paper already carries or one derived
from its own numbers by a division a reader can repeat.

Twelve papers led with a figure that appears in no row of the table it summarises.
zen-agent averaged GAIA 83.1/68.4/42.3 to 71.2 while its four baseline rows are each
the plain mean of their own three levels; the mean is 64.6. zen-embeddings-retrieval
and zen3-embedding claim MTEB averages of 72.4 and 74.3 over tables that supply their
task counts, where the 56-task weighted means are 67.1 and 67.4.
zen-mathematical-reasoning opened with five figures, none of them any row, and
explained one as best-of-32 sampling when its own sampling table puts best-of-32 at a
different number. zen-dso-protocol claimed 94% of centralized quality and a 78%
bandwidth cut against tables reading 99.4% and 99.8%-and-better, the second of them
the table that claim captions. zen-synthetic-data claimed an 8.4-point MT-Bench gain
on a scale that runs to 10, from a 7.84 baseline, where its table says 0.88.
zen-3d quoted the Chair column as the ShapeNet result. zen-multilingual understated
its own XNLI average. Also zen-hallucination-reduction, zen-distributed-training,
zen-finetuning, zen-legal-ai and zen-coder-flash.

Eight more are settled by arithmetic against the hardware or the parameter count.
zen-nano, zen-eco and zen-scribe each report INT4 needing as much memory as FP16 or
more, while their own FP16 and INT8 rows are exactly two and one bytes per parameter;
zen_family_overview already carried the right values. The two zen-designer papers size
a stated 235B model at 220 GB in FP16, which is a 110B model, and zen-quantization puts
the same model at 470 GB. zen-omni and zen-next recommend cards smaller than the memory
the same row requires. zen-live captions a latency table to an A10G when its own
deployment table assigns that 98 ms to the H100. zen-hardware-optimization calls a
throughput table FP8+INT4 when its cumulative table reaches the identical 31,200 tok/s
through INT4 + FP16.

Six papers share one template whose visual benchmark block bundles two metrics
computed over images a model generates with one computed over text a model emits
about an image. No model produces all three, and the block went to four papers across
four modalities, so each received at least one metric it cannot compute: VQA v2 and a
token rate for two diffusion models, FID and CLIP score for two vision-language models.
Each now keeps the rows its own stated output type can produce.

Throughput is the one category where the defect is the missing condition rather than
the number. Decode reads the active weight set once per token, so a single stream
cannot outrun bandwidth over footprint, and zen4-mini, zen3-nano and zen-coder-flash
all report above their own bound. Batched serving would make those figures legal and
none of the three records a batch size, so the numbers stand and each table now carries
its bound and the arithmetic behind it. AIME gets the same treatment in
zen-mathematical-reasoning: a score that is not a multiple of 1/30 on a 30-problem exam
is legal under averaging, and the caption now says which row has its sample count on
record and which do not.

What was examined and left is in LLM.md with the reason in each case. Training fleets
absent from this estate prove nothing about models trained on rented capacity.
zen-reranker's 98% semantic preservation is uncited rather than contradicted.
zen-safety-evaluation and zen-vision-architecture state no weighting, so their averages
are not determined. zen4-coder-flash puts FP8 on an A100, and Ampere has no FP8 tensor
cores, but its claims there are footprint claims and FP8 weight-only storage does run
on Ampere.

All twenty-seven rebuild under latexmk -Werror. Nineteen have diverged from their
counterparts in zenlm/papers and six exist only here, so those fixes do not reach the
other site. zen-3d and zen-agent were byte-identical across the two repositories until
now, and are the two to mirror first.

Co-authored-by: Hanzo Dev <[email protected]>

2026-09-10 23:54:37 -07:00

340 lines

16 KiB

TeX

Raw Permalink Blame History

 \documentclass[11pt,a4paper]{article}
 \usepackage[utf8]{inputenc}
 \usepackage[T1]{fontenc}
 \usepackage{amsmath,amsfonts,amssymb}
 \usepackage{graphicx}
 \usepackage{hyperref}
 \usepackage{listings}
 \usepackage{color}
 \usepackage{booktabs}
 \usepackage{float}
 \usepackage{geometry}
 \geometry{margin=1in}
 \definecolor{zenblue}{RGB}{41,121,255}
 \hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
 \title{\textbf{Zen-Live: A Real-Time Conversational AI Model\\
 with Native Turn Management and Voice Interaction}\\[0.5em]
 \large Technical Whitepaper v2025.03}
 \author{Antje Worring, Hanzo AI Research, Zen LM Research Team\\
 \texttt{[email protected]}\\
 \href{https://papers.zenlm.org}{papers.zenlm.org}}
 \date{March 2025}
 \begin{document}
 \maketitle
 \begin{abstract}
 Zen-Live is a 7 billion parameter language model purpose-built for real-time voice and video conversation, delivering P50 first-token latency of 67ms and P95 of 98ms through speculative decoding and early-exit inference mechanisms. Unlike conversational adaptations of general-purpose LLMs, Zen-Live integrates native voice activity detection (VAD), barge-in handling, and turn-taking management directly into the model architecture, eliminating the 80--150ms overhead of external pipeline components. Zen-Live achieves a word error rate (WER) of 4.2\% on conversational speech benchmarks and turn-taking accuracy of 94.3\% on the MultiTurn evaluation suite. This paper describes the model architecture, training methodology, latency optimization techniques, and conversational capability evaluations.
 \end{abstract}
 \tableofcontents
 \newpage
 \section{Introduction}
 Human conversation operates at time constants incompatible with standard LLM inference. Turn-taking decisions occur within 200--400ms of an interlocutor finishing an utterance. Barge-in (interrupting mid-utterance) is expected and natural, requiring the model to gracefully abort synthesis and switch to listening mode. Backchannels (``mm-hmm'', ``right'', ``I see'') are produced every 3--8 seconds during active listening to signal engagement.
 Standard LLM serving pipelines address these requirements through external orchestration: a separate VAD model, turn detection heuristics, and speech synthesis pipeline. This cascade introduces 80--150ms of coordination overhead and creates seams in conversational behavior visible to users.
 Zen-Live integrates all conversational management functions into a single model with three key design principles:
 \begin{enumerate}
   \item \textbf{Latency-first design}: Every architectural choice is evaluated against its impact on first-token latency, the primary user-perceptible quality metric in voice conversation.
   \item \textbf{Native turn management}: Turn-taking, VAD, and barge-in are model-level capabilities, not external pipeline stages.
   \item \textbf{Streaming compatibility}: The model produces usable partial outputs from the first token, enabling streaming synthesis that begins before generation completes.
 \end{enumerate}
 \subsection{Model Overview}
 \begin{table}[H]
 \centering
 \caption{Zen-Live Model Specification}
 \begin{tabular}{ll}
 \toprule
 \textbf{Parameter} & \textbf{Value} \\
 \midrule
 Architecture & Streaming Transformer with early-exit \\
 Total Parameters & 7B \\
 Context Window & 16K tokens (audio + text interleaved) \\
 P50 First-Token Latency & 67ms \\
 P95 First-Token Latency & 98ms \\
 Turn-Taking Accuracy & 94.3\% \\
 Word Error Rate (conversational) & 4.2\% \\
 Supported Languages & 18 \\
 Version & v2025.03 \\
 Release Date & March 2025 \\
 \bottomrule
 \end{tabular}
 \end{table}
 \section{Architecture}
 \subsection{Interleaved Audio-Text Representation}
 Zen-Live processes audio and text in a unified interleaved sequence representation. Audio is encoded as discrete acoustic tokens using a quantized audio codec at 50Hz (one token per 20ms frame). Text tokens use the standard 100K vocabulary. A modality embedding distinguishes token types:
 \begin{equation}
   h_t = \text{Embed}(x_t) + \text{ModalityEmbed}(m_t) + \text{PosEmbed}(t)
 \end{equation}
 where $m_t \in \{\text{audio}, \text{text}\}$ is the modality label. This unified representation allows the model to condition text generation directly on the acoustic input without a separate transcription step.
 \subsection{Early-Exit Inference}
 Early-exit \cite{earlyexit} reduces latency by allowing the model to produce outputs from intermediate layers when confidence is high, avoiding the full forward pass through all layers. Zen-Live implements adaptive early-exit with per-layer confidence heads:
 \begin{equation}
   \hat{y}_l = \text{ExitHead}_l(h_l), \quad c_l = \text{Confidence}_l(h_l)
 \end{equation}
 If $c_l > \tau_{\text{exit}} = 0.92$, the model exits at layer $l$ and returns $\hat{y}_l$ without computing deeper layers. On conversational speech, the mean exit layer is 14.3 out of 32 total layers, reducing effective computation by 55\% compared to full forward passes.
 The confidence threshold $\tau_{\text{exit}}$ is calibrated to maintain $<0.1\%$ accuracy degradation on held-out conversational benchmarks, verified by comparing early-exit outputs to full-pass outputs on 100K evaluation examples.
 \subsection{Speculative Decoding with Dialogue Draft Model}
 Zen-Live uses a 180M parameter dialogue-specialized draft model for speculative decoding. Unlike general-purpose draft models, the dialogue draft model is trained specifically on conversational turn patterns, giving higher acceptance rates for the stereotyped phrases common in conversational AI (greetings, acknowledgments, question frames).
 \begin{equation}
   \beta_{\text{dialogue}} = 0.84, \quad \beta_{\text{general}} = 0.71
 \end{equation}
 The higher acceptance rate yields a 3.1$\times$ effective speedup in conversational contexts, compared to 2.4$\times$ for a general draft model of equivalent size.
 \subsection{Native Voice Activity Detection}
 VAD is implemented as a lightweight auxiliary head on the audio encoder layers, producing a binary activity signal at the frame level (20ms resolution):
 \begin{equation}
   \hat{v}_t = \sigma\left(W_{\text{VAD}} \cdot h_t^{(8)} + b_{\text{VAD}}\right)
 \end{equation}
 where $h_t^{(8)}$ is the hidden state at encoder layer 8. The VAD head adds $<0.1\%$ parameter overhead and produces activity estimates with 6ms latency from the raw audio frame, compared to 25--40ms for external VAD models operating on decoded PCM.
 \subsection{Turn-Taking Architecture}
 Turn-taking prediction determines when it is appropriate to begin a response. Zen-Live implements turn-taking as a classification head that predicts one of three turn states on each VAD-active frame:
 \begin{itemize}
   \item \textbf{HOLD}: Speaker is mid-utterance; continue listening.
   \item \textbf{YIELD}: Speaker has completed their turn; begin response.
   \item \textbf{BACKCHANNEL}: Speaker expects acknowledgment; produce a brief backchannel.
 \end{itemize}
 The turn state predictor is conditioned on both acoustic features (prosodic contour, speaking rate, final phoneme duration) and semantic features (syntactic completeness, question vs. statement framing):
 \begin{equation}
   \hat{t}_{\text{turn}} = \text{softmax}\left(W_t [h_{\text{acoustic}}; h_{\text{semantic}}]\right)
 \end{equation}
 This joint conditioning achieves 94.3\% turn-taking accuracy, compared to 87.1\% for acoustic-only models and 89.4\% for heuristic rule-based systems.
 \subsection{Barge-In Handling}
 Barge-in detection monitors the VAD signal during synthesis. When user speech is detected during model output, Zen-Live executes a graceful interruption protocol:
 \begin{enumerate}
   \item Synthesis halts at the next sentence boundary (if within 200ms) or immediately.
   \item The partial response is retained in the context as an incomplete assistant turn.
   \item Listening mode resumes from the point of interruption.
   \item The model's next response acknowledges the interruption if semantically appropriate.
 \end{enumerate}
 In a held-out evaluation of 500 barge-in events, Zen-Live achieved clean interruption in 97.3\% of cases (synthesis stopped within 1 sentence boundary of the interrupt signal), compared to 82.1\% for systems using external VAD with TCP notification delay.
 \section{Training Methodology}
 \subsection{Conversational Pre-Training Data}
 \begin{table}[H]
 \centering
 \caption{Training Data Composition}
 \begin{tabular}{lrr}
 \toprule
 \textbf{Source} & \textbf{Hours} & \textbf{Fraction} \\
 \midrule
 Telephone conversations (licensed) & 90,000 & 34.6\% \\
 Podcast and interview audio & 60,000 & 23.1\% \\
 Customer service recordings & 40,000 & 15.4\% \\
 Video conference transcripts & 30,000 & 11.5\% \\
 Synthetic dialogue augmentation & 25,000 & 9.6\% \\
 Academic lectures and Q\&A & 15,000 & 5.8\% \\
 \midrule
 \textbf{Total} & \textbf{260,000} & \textbf{100\%} \\
 \bottomrule
 \end{tabular}
 \end{table}
 \subsection{Latency-Aware Training}
 Zen-Live is trained with a latency-aware training objective that penalizes late exit points. In addition to the standard language modeling loss, an early-exit regularization term encourages the model to produce confident predictions from earlier layers:
 \begin{equation}
   \mathcal{L}_{\text{latency}} = \mathcal{L}_{\text{LM}} + \lambda \sum_l w_l \mathcal{L}_{\text{KD}}(l)
 \end{equation}
 where $w_l = (L - l + 1) / L$ weights later layers more heavily (penalizing failures to exit early), and $\mathcal{L}_{\text{KD}}(l)$ is the KL divergence between the full-model output distribution and the early-exit head at layer $l$.
 \subsection{Turn-Taking Fine-Tuning}
 Turn-taking capability is fine-tuned on a corpus of 50K annotated conversational turns, with human annotators labeling each audio frame with HOLD / YIELD / BACKCHANNEL ground truth. The annotation protocol includes inter-annotator agreement calibration; frames with $<80\%$ annotator agreement are excluded from training to avoid learning from ambiguous social cues.
 \section{Evaluation}
 \subsection{Latency Benchmarks}
 \begin{table}[H]
 \centering
 \caption{First-Token Latency Benchmarks (single H100 80GB GPU, FP8).
 The 98ms P95 here is the cloud configuration of Table~\ref{tab:zenlive-deploy};
 the A10G edge configuration measures 113ms.}
 \begin{tabular}{lcccc}
 \toprule
 \textbf{Model} & \textbf{P50} & \textbf{P90} & \textbf{P95} & \textbf{P99} \\
 \midrule
 Zen-Live (7B) & \textbf{67ms} & 88ms & \textbf{98ms} & 134ms \\
 Adapted 7B general model & 143ms & 191ms & 218ms & 287ms \\
 Adapted 3B general model & 89ms & 121ms & 143ms & 194ms \\
 External pipeline (7B + VAD) & 198ms & 261ms & 294ms & 381ms \\
 \bottomrule
 \end{tabular}
 \end{table}
 \subsection{Speech Recognition Accuracy}
 \begin{table}[H]
 \centering
 \caption{Word Error Rate on Conversational Benchmarks}
 \begin{tabular}{lccc}
 \toprule
 \textbf{Model} & \textbf{Conversational WER} & \textbf{Noisy WER} & \textbf{Accented WER} \\
 \midrule
 Zen-Live (7B) & \textbf{4.2\%} & 8.7\% & 7.1\% \\
 Adapted 7B ASR model & 4.8\% & 9.3\% & 8.4\% \\
 External ASR pipeline & 5.1\% & 10.2\% & 9.7\% \\
 \bottomrule
 \end{tabular}
 \end{table}
 \subsection{Turn-Taking Accuracy}
 \begin{table}[H]
 \centering
 \caption{Turn-Taking Accuracy on MultiTurn Benchmark}
 \begin{tabular}{lccc}
 \toprule
 \textbf{Model} & \textbf{YIELD F1} & \textbf{BACKCHANNEL F1} & \textbf{Overall Accuracy} \\
 \midrule
 Zen-Live (7B) & 95.1\% & 91.4\% & \textbf{94.3\%} \\
 Acoustic-only model & 88.3\% & 83.7\% & 87.1\% \\
 Rule-based heuristics & 90.2\% & 85.4\% & 89.4\% \\
 Human agreement baseline & 97.8\% & 94.1\% & 97.1\% \\
 \bottomrule
 \end{tabular}
 \end{table}
 \subsection{Barge-In Performance}
 \begin{table}[H]
 \centering
 \caption{Barge-In Handling Evaluation (500 events)}
 \begin{tabular}{lcc}
 \toprule
 \textbf{Metric} & \textbf{Zen-Live} & \textbf{External VAD Pipeline} \\
 \midrule
 Clean interruption rate & 97.3\% & 82.1\% \\
 Mean synthesis overshoot & 0.4 sentences & 2.1 sentences \\
 Context retention after barge-in & 94.7\% & 81.3\% \\
 Recovery quality (MOS) & 4.1/5 & 3.4/5 \\
 \bottomrule
 \end{tabular}
 \end{table}
 \subsection{Naturalness}
 Human evaluators rated extended 5-minute conversations with Zen-Live on naturalness and engagement:
 \begin{table}[H]
 \centering
 \caption{Conversational Naturalness MOS (200 conversations)}
 \begin{tabular}{lcc}
 \toprule
 \textbf{Dimension} & \textbf{Zen-Live} & \textbf{Adapted General Model} \\
 \midrule
 Response relevance & 4.3/5 & 4.1/5 \\
 Turn timing naturalness & 4.2/5 & 3.3/5 \\
 Backchannel appropriateness & 3.9/5 & 2.7/5 \\
 Interruption handling & 4.1/5 & 2.9/5 \\
 \textbf{Overall conversational MOS} & \textbf{4.1/5} & \textbf{3.3/5} \\
 \bottomrule
 \end{tabular}
 \end{table}
 The largest gains from Zen-Live's native turn management are in timing naturalness (+0.9 MOS) and backchannel appropriateness (+1.2 MOS), dimensions where external pipeline approaches are weakest.
 \section{Deployment}
 \subsection{Inference Hardware}
 \begin{table}[H]
 \centering
 \caption{Deployment Configurations}
 \label{tab:zenlive-deploy}
 \begin{tabular}{llcl}
 \toprule
 \textbf{Config} & \textbf{Hardware} & \textbf{P95 Latency} & \textbf{Concurrent Sessions} \\
 \midrule
 Cloud (FP8) & 1 $\times$ H100 80GB & 98ms & 96 sessions \\
 Edge server (FP8) & 1 $\times$ A10G 24GB & 113ms & 48 sessions \\
 On-device (INT4) & Apple M3 Max & 187ms & 4 sessions \\
 \bottomrule
 \end{tabular}
 \end{table}
 \subsection{API Integration}
 Zen-Live exposes a WebRTC-compatible streaming API with the following modes:
 \begin{itemize}
   \item \textbf{Full duplex}: Bidirectional audio streaming with native barge-in.
   \item \textbf{Half duplex}: Push-to-talk or VAD-gated turn alternation.
   \item \textbf{Transcription-only}: Audio in, text out, for accessibility or logging use cases.
   \item \textbf{Voice synthesis}: Text in, speech out, using the internal TTS component.
 \end{itemize}
 \section{Safety Considerations}
 \subsection{Real-Time Content Filtering}
 Zen-Live integrates a lightweight token-level safety filter (the Zen-Guard-Stream module) that operates with $<2$ms overhead per synthesis step, enabling safe streaming outputs without post-hoc filtering that would introduce latency spikes.
 \subsection{Consent and Privacy}
 All voice recordings processed by Zen-Live are subject to:
 \begin{itemize}
   \item Explicit user consent before voice data processing.
   \item No persistent storage of audio beyond the session context window.
   \item Speaker anonymization in telemetry logs (voice embeddings are discarded; only aggregate quality metrics retained).
 \end{itemize}
 \section{Related Work}
 Real-time conversational AI has been developed through voice assistants \cite{siri,alexa}, dialogue systems \cite{dialog}, and recently through LLM-based voice interfaces \cite{voicellm}. Early-exit inference has been applied to NLP tasks \cite{earlyexit}. Speculative decoding for conversational AI is explored in \cite{spectalk}. Zen-Live unifies these advances into a model architecture trained end-to-end for conversational requirements.
 \section{Conclusion}
 Zen-Live achieves real-time voice conversation with P95 first-token latency of 98ms, turn-taking accuracy of 94.3\%, and WER of 4.2\% by integrating VAD, turn management, and language generation into a single 7B parameter model. The architecture eliminates the coordination overhead of cascaded pipeline approaches and enables more natural conversational behavior, particularly in barge-in handling and backchannel generation. Zen-Live is deployable on a single GPU for cloud applications and on Apple Silicon for on-device voice assistant use cases.
 \begin{thebibliography}{9}
 \bibitem{earlyexit} Schwartz, R. et al. (2020). The Right Tool for the Job: Matching Model and Instance Complexities. ACL 2020.
 \bibitem{siri} Shum, H.-Y. et al. (2018). From Eliza to XiaoIce: Challenges and Opportunities with Social Chatbots. Frontiers of IT and EE.
 \bibitem{alexa} Alexa AI Team. (2018). The Alexa Meaning Representation Language. NAACL 2018.
 \bibitem{dialog} Gao, J. et al. (2019). Neural Approaches to Conversational AI. Foundations and Trends in IR.
 \bibitem{voicellm} Défossez, A. et al. (2024). Moshi: a speech-text foundation model for real-time dialogue. arXiv:2410.00037.
 \bibitem{spectalk} Medina, G. et al. (2024). Speculative Decoding for Conversational AI Systems. arXiv preprint.
 \end{thebibliography}
 \end{document}

340 lines 16 KiB TeX Raw Permalink Blame History

340 lines

16 KiB

TeX

Raw Permalink Blame History