papers/zen/zen-agent.tex at main

mirror of https://github.com/hanzoai/papers.git synced 2026-10-22 17:20:17 +00:00

Files

T

zeekayandHanzo Dev bc1452e9f0 The zen claims that arithmetic settles

zenlm/papers had a sweeping withdrawal pass, 5deeeaa, and the owner reverted it in
f34f609 for asserting more than it verified. That objection sets the rule here: a
claim changes only where the proof is in the paper itself or in arithmetic, never
because a run could not be found. Twenty-seven of the eighty-one zen papers meet
that bar. Every correction quotes a number the paper already carries or one derived
from its own numbers by a division a reader can repeat.

Twelve papers led with a figure that appears in no row of the table it summarises.
zen-agent averaged GAIA 83.1/68.4/42.3 to 71.2 while its four baseline rows are each
the plain mean of their own three levels; the mean is 64.6. zen-embeddings-retrieval
and zen3-embedding claim MTEB averages of 72.4 and 74.3 over tables that supply their
task counts, where the 56-task weighted means are 67.1 and 67.4.
zen-mathematical-reasoning opened with five figures, none of them any row, and
explained one as best-of-32 sampling when its own sampling table puts best-of-32 at a
different number. zen-dso-protocol claimed 94% of centralized quality and a 78%
bandwidth cut against tables reading 99.4% and 99.8%-and-better, the second of them
the table that claim captions. zen-synthetic-data claimed an 8.4-point MT-Bench gain
on a scale that runs to 10, from a 7.84 baseline, where its table says 0.88.
zen-3d quoted the Chair column as the ShapeNet result. zen-multilingual understated
its own XNLI average. Also zen-hallucination-reduction, zen-distributed-training,
zen-finetuning, zen-legal-ai and zen-coder-flash.

Eight more are settled by arithmetic against the hardware or the parameter count.
zen-nano, zen-eco and zen-scribe each report INT4 needing as much memory as FP16 or
more, while their own FP16 and INT8 rows are exactly two and one bytes per parameter;
zen_family_overview already carried the right values. The two zen-designer papers size
a stated 235B model at 220 GB in FP16, which is a 110B model, and zen-quantization puts
the same model at 470 GB. zen-omni and zen-next recommend cards smaller than the memory
the same row requires. zen-live captions a latency table to an A10G when its own
deployment table assigns that 98 ms to the H100. zen-hardware-optimization calls a
throughput table FP8+INT4 when its cumulative table reaches the identical 31,200 tok/s
through INT4 + FP16.

Six papers share one template whose visual benchmark block bundles two metrics
computed over images a model generates with one computed over text a model emits
about an image. No model produces all three, and the block went to four papers across
four modalities, so each received at least one metric it cannot compute: VQA v2 and a
token rate for two diffusion models, FID and CLIP score for two vision-language models.
Each now keeps the rows its own stated output type can produce.

Throughput is the one category where the defect is the missing condition rather than
the number. Decode reads the active weight set once per token, so a single stream
cannot outrun bandwidth over footprint, and zen4-mini, zen3-nano and zen-coder-flash
all report above their own bound. Batched serving would make those figures legal and
none of the three records a batch size, so the numbers stand and each table now carries
its bound and the arithmetic behind it. AIME gets the same treatment in
zen-mathematical-reasoning: a score that is not a multiple of 1/30 on a 30-problem exam
is legal under averaging, and the caption now says which row has its sample count on
record and which do not.

What was examined and left is in LLM.md with the reason in each case. Training fleets
absent from this estate prove nothing about models trained on rented capacity.
zen-reranker's 98% semantic preservation is uncited rather than contradicted.
zen-safety-evaluation and zen-vision-architecture state no weighting, so their averages
are not determined. zen4-coder-flash puts FP8 on an A100, and Ampere has no FP8 tensor
cores, but its claims there are footprint claims and FP8 weight-only storage does run
on Ampere.

All twenty-seven rebuild under latexmk -Werror. Nineteen have diverged from their
counterparts in zenlm/papers and six exist only here, so those fixes do not reach the
other site. zen-3d and zen-agent were byte-identical across the two repositories until
now, and are the two to mirror first.

Co-authored-by: Hanzo Dev <[email protected]>

2026-09-10 23:54:37 -07:00

442 lines

16 KiB

TeX

Raw Permalink Blame History

 \documentclass[11pt,a4paper]{article}
 \usepackage[utf8]{inputenc}
 \usepackage[T1]{fontenc}
 \usepackage{amsmath,amsfonts,amssymb}
 \usepackage{graphicx}
 \usepackage{hyperref}
 \usepackage{listings}
 \usepackage[dvipsnames]{xcolor}
 \usepackage{booktabs}
 \usepackage{float}
 \usepackage{geometry}
 \geometry{margin=1in}
 \definecolor{zenblue}{RGB}{41,121,255}
 \definecolor{zengreen}{RGB}{52,199,89}
 \definecolor{codegray}{RGB}{245,245,245}
 \hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
 \lstset{
     backgroundcolor=\color{codegray},
     basicstyle=\ttfamily\small,
     breaklines=true,
     captionpos=b,
     frame=single,
     numbers=left,
     numberstyle=\tiny\color{gray}
 }
 \title{
     \vspace{-2cm}
     \Large \textbf{Zen AI Model Family} \\
     \vspace{0.5cm}
     \Huge \textbf{Zen-Agent} \\
     \vspace{0.3cm}
     \large Autonomous Task Execution with Long-Horizon Planning \\
     \vspace{0.5cm}
     \normalsize Technical Report v2025.01
 }
 \author{
     Hanzo AI Research Team\thanks{[email protected]} \and
     Zoo Labs Foundation\thanks{[email protected]}
 }
 \date{January 2025}
 \begin{document}
 \maketitle
 \begin{abstract}
 We present \textbf{Zen-Agent}, a 32-billion parameter model trained for autonomous task execution,
 multi-step planning, and compositional tool use. Zen-Agent is built on the Zen MoDE (Mixture of
 Distilled Experts) architecture and fine-tuned on the Zen Agentic Trajectory Dataset---a corpus
 of 12.4 million complete task trajectories spanning software engineering, web navigation, data
 analysis, and scientific reasoning. The model achieves 64.6\% on GAIA (Level 1--3 average),
 .8\% on SWE-bench Verified, and 88.3\% on HumanEval, establishing a new frontier for open
 agentic models at this parameter scale. Zen-Agent introduces \textbf{Hierarchical Task
 Decomposition (HTD)}: an explicit planning module that generates subgoal trees before executing
 any tool calls, substantially reducing failure modes caused by premature commitment to incorrect
 action sequences. The model natively speaks the Model Context Protocol (MCP), enabling
 zero-configuration integration with the Hanzo MCP tool ecosystem of 260+ tools.
 \end{abstract}
 \tableofcontents
 \newpage
 \section{Introduction}
 The transition from conversational AI to agentic AI---systems that take sequences of actions in
 the world to complete long-horizon tasks---demands a qualitative shift in how models are trained
 and evaluated. A conversational model optimizes a single response; an agentic model must plan,
 execute, observe feedback, revise beliefs, and persist toward a goal across potentially hundreds
 of tool-mediated steps.
 Existing large language models, even those capable of impressive single-turn reasoning, fail on
 agentic benchmarks primarily due to:
 \begin{enumerate}
     \item \textbf{Premature commitment}: Taking tool calls before adequately planning the task
     structure, leading to irrecoverable state.
     \item \textbf{Context drift}: Losing track of the original goal as the trajectory lengthens.
     \item \textbf{Tool misuse}: Calling tools with syntactically correct but semantically wrong
     arguments, wasting steps and budget.
     \item \textbf{Error propagation}: Failing to detect and recover from subtask failures before
     they cascade into total task failure.
 \end{enumerate}
 Zen-Agent addresses each failure mode through architectural and training innovations described
 in this report.
 \subsection{Model Summary}
 \begin{table}[H]
 \centering
 \begin{tabular}{ll}
 \toprule
 \textbf{Property} & \textbf{Value} \\
 \midrule
 Parameters & 32B \\
 Architecture & Zen MoDE 32B (8 experts, top-2) \\
 Context Length & 128K tokens \\
 Max Trajectory Length & 512 tool calls \\
 Native Protocols & MCP, OpenAI Tools, Anthropic Tools \\
 Supported Runtimes & Python, Node.js, Docker, Kubernetes \\
 Training Trajectories & 12.4M task trajectories \\
 \bottomrule
 \end{tabular}
 \caption{Zen-Agent Model Specifications}
 \end{table}
 \section{Architecture}
 \subsection{Zen MoDE 32B Backbone}
 Zen-Agent uses the Zen MoDE architecture at 32B scale: 40 transformer layers with 32 attention
 heads, grouped-query attention (GQA with 8 KV heads), and sparse MoE feed-forward networks.
 Each MoE layer maintains 8 experts with top-2 routing, yielding approximately 8B active
 parameters per forward pass while providing 32B parameter capacity for specialist knowledge.
 \begin{table}[H]
 \centering
 \begin{tabular}{ll}
 \toprule
 \textbf{Component} & \textbf{Specification} \\
 \midrule
 Layers & 40 \\
 Attention heads & 32 (GQA, 8 KV heads) \\
 Hidden dimension & 5120 \\
 MoE experts per layer & 8 (top-2 routing) \\
 Active params per token & $\approx$8B of 32B total \\
 Context length & 128K (RoPE, $\theta = 500{,}000$) \\
 Vocabulary & 152,064 tokens \\
 \bottomrule
 \end{tabular}
 \caption{Zen MoDE 32B Architecture}
 \end{table}
 \subsection{Hierarchical Task Decomposition (HTD)}
 Before any tool call is issued, Zen-Agent generates an explicit \textbf{task plan} in a
 structured thinking block. The plan takes the form of a tree of subgoals:
 \begin{lstlisting}[language=Python, caption=HTD Plan Structure]
 {
   "goal": "Fix the authentication bug in PR #1247",
   "subgoals": [
     {
       "id": "sg1",
       "description": "Reproduce the bug",
       "steps": ["checkout branch", "run failing tests", "read error"],
       "depends_on": []
     },
     {
       "id": "sg2",
       "description": "Identify root cause",
       "steps": ["read auth module", "trace call stack"],
       "depends_on": ["sg1"]
     },
     {
       "id": "sg3",
       "description": "Implement fix and verify",
       "steps": ["edit file", "run tests", "verify no regressions"],
       "depends_on": ["sg2"]
     }
   ]
 }
 \end{lstlisting}
 The HTD module is implemented as a constrained decoding prefix: the model always generates a
 \texttt{<plan>} block before the first \texttt{<tool\_call>} block. Training enforces this
 through a curriculum that penalizes trajectories where tool calls precede planning.
 \subsection{Goal Persistence Module}
 To combat context drift over long trajectories, the model maintains a \textbf{goal stack} in
 its working memory: a compact structured summary of the original task and completed subgoals.
 At every 32-step checkpoint, the model re-reads the goal stack and emits a brief
 \texttt{<goal\_check>} block confirming alignment. Divergence from the goal triggers automatic
 backtracking to the last consistent state.
 \subsection{MCP-Native Tool Interface}
 Zen-Agent is the first model trained natively on the Model Context Protocol (MCP) tool call
 format. This means:
 \begin{itemize}
     \item Tool schemas are embedded in the model's training distribution, not injected at inference.
     \item Common MCP tools (bash, file I/O, web browser, code execution) are callable without
     schema prompting.
     \item The model understands MCP error codes and applies appropriate retry strategies.
     \item Tool results are parsed and integrated into the plan without additional prompting.
 \end{itemize}
 \section{Training}
 \subsection{Zen Agentic Trajectory Dataset}
 \begin{table}[H]
 \centering
 \begin{tabular}{lrrl}
 \toprule
 \textbf{Domain} & \textbf{Trajectories} & \textbf{Proportion} & \textbf{Source} \\
 \midrule
 Software engineering & 4,100,000 & 33.1\% & GitHub issues, PRs \\
 Web navigation & 2,800,000 & 22.6\% & Browser sessions \\
 Data analysis & 1,900,000 & 15.3\% & Jupyter notebooks \\
 Scientific reasoning & 1,400,000 & 11.3\% & Research papers \\
 Coding tasks & 1,200,000 & 9.7\% & Competitive coding \\
 System administration & 1,000,000 & 8.1\% & DevOps runbooks \\
 \midrule
 \textbf{Total} & \textbf{12,400,000} & 100\% & \\
 \bottomrule
 \end{tabular}
 \caption{Zen Agentic Trajectory Dataset Composition}
 \end{table}
 Each trajectory record contains: task specification, tool call sequence, tool outputs, final
 answer, success/failure label, and a human-written critique. Trajectories with more than 10
 consecutive failed tool calls were filtered out during dataset curation.
 \subsection{Training Protocol}
 Training proceeds in four stages:
 \textbf{Stage 1 -- SFT on correct trajectories} (50K steps): The model is fine-tuned on the
 \% of trajectories labeled as fully successful, using standard next-token prediction loss
 over the complete trajectory (plan + tool calls + observations + answer).
 \textbf{Stage 2 -- DPO on trajectory pairs} (30K steps): For each task with both successful
 and failed trajectories, we construct preference pairs and apply Direct Preference Optimization
 to increase the probability of success-leading prefixes.
 \textbf{Stage 3 -- GRPO for tool accuracy} (20K steps): Group Relative Policy Optimization
 with a sparse reward signal: $+1$ for task success, $-0.1$ per unnecessary tool call (calls
 that could have been skipped given the information already available in context).
 \textbf{Stage 4 -- Safety fine-tuning} (10K steps): Constitutional AI prompts are used to
 discourage trajectories that involve irreversible destructive actions without explicit user
 confirmation steps.
 \begin{table}[H]
 \centering
 \begin{tabular}{llll}
 \toprule
 \textbf{Stage} & \textbf{Steps} & \textbf{Batch} & \textbf{LR} \\
 \midrule
 SFT & 50K & 64 & 1e-5 \\
 DPO & 30K & 32 & 5e-6 \\
 GRPO & 20K & 16 & 2e-6 \\
 Safety & 10K & 32 & 1e-6 \\
 \bottomrule
 \end{tabular}
 \caption{Zen-Agent Training Configuration}
 \end{table}
 \section{Evaluation}
 \subsection{GAIA Benchmark}
 GAIA measures real-world assistant capability across general AI tasks requiring web search,
 file processing, code execution, and multi-step reasoning. Results below are on the official
 GAIA validation set (Level 1 = easy, Level 3 = hard).
 \begin{table}[H]
 \centering
 \begin{tabular}{lcccc}
 \toprule
 \textbf{Model} & \textbf{Level 1} & \textbf{Level 2} & \textbf{Level 3} & \textbf{Average} \\
 \midrule
 GPT-4o (2024-11) & 73.2 & 52.1 & 28.4 & 51.2 \\
 Claude 3.5 Sonnet & 77.3 & 56.2 & 31.7 & 55.1 \\
 Gemini 1.5 Pro & 70.1 & 48.3 & 26.1 & 48.2 \\
 Open-source SoTA (32B) & 52.7 & 31.4 & 14.2 & 32.8 \\
 \textbf{Zen-Agent 32B} & \textbf{83.1} & \textbf{68.4} & \textbf{42.3} & \textbf{64.6} \\
 \bottomrule
 \end{tabular}
 \caption{GAIA Benchmark Results (\%, higher is better)}
 \end{table}
 \subsection{SWE-bench Verified}
 SWE-bench Verified presents real GitHub issues from popular Python repositories; the model must
 produce a patch that passes the official test suite without seeing the tests.
 \begin{table}[H]
 \centering
 \begin{tabular}{lcc}
 \toprule
 \textbf{Model} & \textbf{Resolve Rate (\%)} & \textbf{Avg. Tool Calls} \\
 \midrule
 GPT-4o (Agentless) & 33.2 & 24.3 \\
 Claude 3.5 Sonnet & 38.1 & 31.7 \\
 SWE-agent + Claude & 39.4 & 48.2 \\
 OpenHands + Claude & 41.0 & 52.1 \\
 \textbf{Zen-Agent 32B} & \textbf{42.8} & \textbf{29.6} \\
 \bottomrule
 \end{tabular}
 \caption{SWE-bench Verified Results}
 \end{table}
 Zen-Agent achieves the highest resolve rate while requiring 30\% fewer tool calls than the
 next-best method, demonstrating the efficiency gain from Hierarchical Task Decomposition.
 \subsection{HumanEval}
 Single-function code generation evaluated by executing generated code against hidden test cases.
 \begin{table}[H]
 \centering
 \begin{tabular}{lc}
 \toprule
 \textbf{Model} & \textbf{pass@1 (\%)} \\
 \midrule
 GPT-4o & 86.4 \\
 Claude 3.5 Haiku & 81.2 \\
 Gemini 1.5 Flash & 78.3 \\
 \textbf{Zen-Agent 32B} & \textbf{88.3} \\
 \bottomrule
 \end{tabular}
 \caption{HumanEval pass@1}
 \end{table}
 \subsection{Tool Use Accuracy}
 We evaluate tool argument correctness on a held-out set of 10,000 tool calls annotated with
 ground-truth arguments. Zen-Agent achieves 91.4\% exact-match accuracy on tool arguments,
 compared to 78.2\% for the best-performing comparable baseline.
 \subsection{Trajectory Efficiency}
 \begin{table}[H]
 \centering
 \begin{tabular}{lccc}
 \toprule
 \textbf{Task Type} & \textbf{Success Rate} & \textbf{Avg Calls} & \textbf{Backtrack Rate} \\
 \midrule
 Code debugging & 84.3\% & 18.2 & 12.1\% \\
 Web research & 91.2\% & 12.7 & 6.3\% \\
 Data analysis & 87.6\% & 22.4 & 9.8\% \\
 File manipulation & 96.1\% & 7.3 & 2.1\% \\
 System administration & 78.4\% & 31.2 & 18.7\% \\
 \bottomrule
 \end{tabular}
 \caption{Zen-Agent Trajectory Efficiency by Task Type}
 \end{table}
 \section{Applications}
 \subsection{Software Engineering Automation}
 Zen-Agent powers the Hanzo Code service, which autonomously handles GitHub issues labeled
 \texttt{zen-agent}: the model reads the issue, explores the codebase, writes a patch, runs
 the test suite, and opens a pull request with a structured description of its changes.
 Average cycle time is under 3 minutes for issues categorized as medium complexity.
 \subsection{Research Automation}
 For scientific research tasks, Zen-Agent can execute multi-hour workflows involving literature
 search (via web tools), data download and preprocessing, statistical analysis (via Python
 execution), and report generation---all from a single natural language task specification.
 \subsection{IT Operations}
 Zen-Agent is deployed in the Hanzo cloud platform for automated incident response: given an
 alert, the model diagnoses the root cause by querying logs, metrics, and configuration state,
 then executes a remediation playbook or escalates with a structured diagnostic report.
 \section{Safety and Constraints}
 \subsection{Action Reversibility Classification}
 Every tool in the Zen MCP ecosystem is annotated with a reversibility label:
 \texttt{reversible}, \texttt{soft-destructive} (data can be recovered), or
 \texttt{hard-destructive} (action is irreversible). Zen-Agent refuses to execute
 \texttt{hard-destructive} actions without an explicit user confirmation step in the trajectory.
 \subsection{Budget Limits}
 Each agent session is configured with maximum tool calls (default 200), maximum wall-clock
 time (default 10 minutes), and maximum cost per session (configurable, default \$0.50).
 The model monitors its own budget and gracefully summarizes progress when approaching limits.
 \subsection{Hallucination Reduction}
 Zen-Agent's training emphasized grounded claims: the model is penalized in GRPO training for
 asserting facts about tool outputs that contradict what was actually returned. On the
 TruthfulQA benchmark, Zen-Agent achieves 81.3\% accuracy, 4.2 points above the backbone
 pre-fine-tuning baseline.
 \section{Integration}
 \begin{lstlisting}[language=Python, caption=Zen-Agent Task Execution]
 from zen import ZenAgent
 agent = ZenAgent(
     model="zenlm/zen-agent-32b",
     tools=["bash", "file", "web", "python"],
     max_steps=100,
     budget_usd=1.0,
 )
 result = agent.run(
     "Analyze the performance regression in the /api/search endpoint "
     "using the last 24h of production logs and propose a fix.",
     context={"log_path": "/var/log/api/search.log"}
 )
 print(result.answer)
 print(f"Steps taken: {result.num_steps}")
 print(f"Cost: ${result.cost_usd:.4f}")
 \end{lstlisting}
 \section{Related Work}
 ReAct \cite{yao2022react} established the interleaved reasoning-action paradigm.
 Tree-of-Thought \cite{yao2023tree} introduced explicit planning over branching action spaces.
 SWE-agent \cite{yang2024swe} specialized for software engineering via shell-based interaction.
 Zen-Agent advances this line through HTD, native MCP integration, and large-scale trajectory
 training at 32B scale with the Zen MoDE architecture.
 \section{Conclusion}
 Zen-Agent demonstrates that agentic capability can be substantially improved by training on
 real task trajectories at scale, enforcing explicit hierarchical planning before action
 execution, and natively integrating with tool ecosystems via MCP. The resulting model achieves
 .6\% GAIA and 42.8\% SWE-bench Verified, establishing a new state of the art for open
 agentic models at 32B scale while using 30\% fewer tool calls than comparable methods.
 Future work will extend Zen-Agent to multi-agent collaboration (where multiple Zen-Agent
 instances coordinate via MCP), longer planning horizons (beyond 512 tool calls), and
 continual learning from deployment feedback via the Zoo DSO protocol.
 \begin{thebibliography}{10}
 \bibitem{yao2022react} S. Yao et al., ``ReAct: Synergizing Reasoning and Acting in Language Models,'' ICLR, 2023.
 \bibitem{yao2023tree} S. Yao et al., ``Tree of Thoughts: Deliberate Problem Solving with Large Language Models,'' NeurIPS, 2023.
 \bibitem{yang2024swe} J. Yang et al., ``SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering,'' NeurIPS, 2024.
 \end{thebibliography}
 \end{document}

442 lines 16 KiB TeX Raw Permalink Blame History

442 lines

16 KiB

TeX

Raw Permalink Blame History