mirror of
https://github.com/hanzoai/papers.git
synced 2026-11-03 02:27:30 +00:00
zeekayandHanzo Dev
bc1452e9f0
zenlm/papers had a sweeping withdrawal pass, 5deeeaa, and the owner reverted it in f34f609 for asserting more than it verified. That objection sets the rule here: a claim changes only where the proof is in the paper itself or in arithmetic, never because a run could not be found. Twenty-seven of the eighty-one zen papers meet that bar. Every correction quotes a number the paper already carries or one derived from its own numbers by a division a reader can repeat. Twelve papers led with a figure that appears in no row of the table it summarises. zen-agent averaged GAIA 83.1/68.4/42.3 to 71.2 while its four baseline rows are each the plain mean of their own three levels; the mean is 64.6. zen-embeddings-retrieval and zen3-embedding claim MTEB averages of 72.4 and 74.3 over tables that supply their task counts, where the 56-task weighted means are 67.1 and 67.4. zen-mathematical-reasoning opened with five figures, none of them any row, and explained one as best-of-32 sampling when its own sampling table puts best-of-32 at a different number. zen-dso-protocol claimed 94% of centralized quality and a 78% bandwidth cut against tables reading 99.4% and 99.8%-and-better, the second of them the table that claim captions. zen-synthetic-data claimed an 8.4-point MT-Bench gain on a scale that runs to 10, from a 7.84 baseline, where its table says 0.88. zen-3d quoted the Chair column as the ShapeNet result. zen-multilingual understated its own XNLI average. Also zen-hallucination-reduction, zen-distributed-training, zen-finetuning, zen-legal-ai and zen-coder-flash. Eight more are settled by arithmetic against the hardware or the parameter count. zen-nano, zen-eco and zen-scribe each report INT4 needing as much memory as FP16 or more, while their own FP16 and INT8 rows are exactly two and one bytes per parameter; zen_family_overview already carried the right values. The two zen-designer papers size a stated 235B model at 220 GB in FP16, which is a 110B model, and zen-quantization puts the same model at 470 GB. zen-omni and zen-next recommend cards smaller than the memory the same row requires. zen-live captions a latency table to an A10G when its own deployment table assigns that 98 ms to the H100. zen-hardware-optimization calls a throughput table FP8+INT4 when its cumulative table reaches the identical 31,200 tok/s through INT4 + FP16. Six papers share one template whose visual benchmark block bundles two metrics computed over images a model generates with one computed over text a model emits about an image. No model produces all three, and the block went to four papers across four modalities, so each received at least one metric it cannot compute: VQA v2 and a token rate for two diffusion models, FID and CLIP score for two vision-language models. Each now keeps the rows its own stated output type can produce. Throughput is the one category where the defect is the missing condition rather than the number. Decode reads the active weight set once per token, so a single stream cannot outrun bandwidth over footprint, and zen4-mini, zen3-nano and zen-coder-flash all report above their own bound. Batched serving would make those figures legal and none of the three records a batch size, so the numbers stand and each table now carries its bound and the arithmetic behind it. AIME gets the same treatment in zen-mathematical-reasoning: a score that is not a multiple of 1/30 on a 30-problem exam is legal under averaging, and the caption now says which row has its sample count on record and which do not. What was examined and left is in LLM.md with the reason in each case. Training fleets absent from this estate prove nothing about models trained on rented capacity. zen-reranker's 98% semantic preservation is uncited rather than contradicted. zen-safety-evaluation and zen-vision-architecture state no weighting, so their averages are not determined. zen4-coder-flash puts FP8 on an A100, and Ampere has no FP8 tensor cores, but its claims there are footprint claims and FP8 weight-only storage does run on Ampere. All twenty-seven rebuild under latexmk -Werror. Nineteen have diverged from their counterparts in zenlm/papers and six exist only here, so those fixes do not reach the other site. zen-3d and zen-agent were byte-identical across the two repositories until now, and are the two to mirror first. Co-authored-by: Hanzo Dev <[email protected]>
294 lines
12 KiB
TeX
294 lines
12 KiB
TeX
\documentclass[11pt,a4paper]{article}
|
|
\usepackage[utf8]{inputenc}
|
|
\usepackage[T1]{fontenc}
|
|
\usepackage{amsmath,amsfonts,amssymb}
|
|
\usepackage{graphicx}
|
|
\usepackage{hyperref}
|
|
\usepackage{listings}
|
|
\usepackage{color}
|
|
\usepackage{booktabs}
|
|
\usepackage{float}
|
|
\usepackage{geometry}
|
|
\geometry{margin=1in}
|
|
\definecolor{zenblue}{RGB}{41,121,255}
|
|
\hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
|
|
|
|
\title{\textbf{Zen Legal: AI-Assisted Legal Research and Analysis}\\
|
|
\large Technical Report v2025.10}
|
|
\author{Hanzo AI Research, Zen LM Research Team\\
|
|
\texttt{[email protected]}}
|
|
\date{October 2025}
|
|
|
|
\begin{document}
|
|
\maketitle
|
|
|
|
\begin{abstract}
|
|
We present Zen Legal, a suite of AI models for legal research, document analysis, and contract review built on the Zen MoDE (Mixture of Distilled Experts) backbone. Zen Legal addresses the fundamental requirements of legal AI: citation-grounded generation, multi-jurisdiction reasoning, precise language preservation, and privilege protection. On standardized legal benchmarks: LegalBench 81.4\%, CUAD contract clause extraction F1 0.923 at 0.938 recall across 41 clause types, and case outcome prediction 74.2\%. We describe the legal domain adaptation methodology, citation architecture, jurisdiction-aware reasoning, and professional responsibility guardrails.
|
|
\end{abstract}
|
|
|
|
\section{Introduction}
|
|
|
|
Legal practice demands precision at a level that distinguishes it from most AI application domains. A misquoted statute, a missed precedent, or a hallucinated citation can undermine a legal argument, expose a client to liability, or trigger bar discipline for the attorney who relied on it.
|
|
|
|
Zen Legal is built on the premise that the primary role of AI in legal contexts is \emph{augmentation}, not replacement—reducing research time from days to hours, identifying overlooked precedents, flagging inconsistencies in contract drafts, and generating first drafts that attorneys review and refine. This framing shapes every design decision:
|
|
|
|
\begin{enumerate}
|
|
\item \textbf{Citation-grounded generation}: Every legal claim cites a specific case, statute, or regulation. Uncited assertions are prohibited.
|
|
\item \textbf{Jurisdictional awareness}: Legal rules vary by jurisdiction; Zen Legal explicitly identifies which jurisdiction's law it is applying and flags conflicts.
|
|
\item \textbf{Uncertainty disclosure}: The model expresses calibrated uncertainty about unsettled legal questions rather than stating contested doctrine as established law.
|
|
\item \textbf{Privilege protection}: Attorney-client privileged content is handled with enhanced access controls and never leaves the client environment.
|
|
\item \textbf{Professional responsibility compliance}: Outputs comply with ABA Model Rules and applicable state professional conduct rules.
|
|
\end{enumerate}
|
|
|
|
\section{Legal Domain Adaptation}
|
|
|
|
\subsection{Training Corpus}
|
|
|
|
Zen Legal is fine-tuned on a 380-billion-token legal corpus:
|
|
|
|
\begin{table}[H]
|
|
\centering
|
|
\caption{Legal training corpus composition}
|
|
\label{tab:corpus}
|
|
\begin{tabular}{lcc}
|
|
\toprule
|
|
Source & Tokens (B) & Coverage \\
|
|
\midrule
|
|
Federal case law (PACER, CourtListener) & 68 & 1789--2025 \\
|
|
State case law (all 50 states) & 84 & Selective post-1970 \\
|
|
Federal statutory code (USC) & 8.4 & Current \\
|
|
Code of Federal Regulations (CFR) & 12 & Current \\
|
|
Law review articles (HeinOnline) & 48 & 1820--2025 \\
|
|
International law (UN, EU, OECD) & 28 & 1945--2025 \\
|
|
Legal contracts and agreements & 92 & Commercial \\
|
|
Bar exam materials and treatises & 14 & Current \\
|
|
Synthetic legal QA pairs & 26 & All jurisdictions \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\subsection{Citation Grounding Architecture}
|
|
|
|
Legal citations require hyper-precise attribution: case name, reporter, volume, page, pinpoint page, court, and year. Zen Legal's citation generation combines:
|
|
|
|
\begin{enumerate}
|
|
\item \textbf{Retrieval}: A dense retrieval index over 48 million legal documents retrieves candidate authorities.
|
|
\item \textbf{Verification}: Retrieved documents are verified against a citation registry that checks existence and accuracy of pinpoint cites.
|
|
\item \textbf{Shepardization}: Citations are checked against a legal validation service to flag overruled, distinguished, or limited precedents.
|
|
\item \textbf{Attribution}: The model generates text with inline citation markers that link to verified source documents.
|
|
\end{enumerate}
|
|
|
|
Citation accuracy (citation exists and supports the stated proposition) reaches 96.8\% on internal held-out evaluation by licensed attorneys.
|
|
|
|
\section{Multi-Jurisdiction Reasoning}
|
|
|
|
\subsection{Jurisdiction Classification}
|
|
|
|
Every legal query is classified by the applicable jurisdiction(s) before substantive analysis:
|
|
|
|
\begin{itemize}
|
|
\item Primary jurisdiction: The jurisdiction whose law governs the issue.
|
|
\item Secondary jurisdictions: Related jurisdictions consulted for persuasive authority.
|
|
\item Conflict detection: When jurisdictions differ, the model explicitly identifies the split.
|
|
\end{itemize}
|
|
|
|
\begin{equation}
|
|
P(j \mid q) = \text{softmax}(\mathbf{W}_j \cdot \text{ZenMoDE}(q))
|
|
\end{equation}
|
|
|
|
Jurisdiction classification accuracy: 97.4\% for US federal/state classification, 94.2\% for international jurisdiction identification.
|
|
|
|
\subsection{Circuit Splits and Unsettled Law}
|
|
|
|
When a question implicates a circuit split or unsettled area of law, Zen Legal:
|
|
|
|
\begin{enumerate}
|
|
\item Identifies the split (e.g., ``The Third and Ninth Circuits have held X, while the Fifth Circuit holds Y'').
|
|
\item States which rule applies in the user's jurisdiction.
|
|
\item Notes any Supreme Court certiorari activity or pending resolution.
|
|
\item Advises the user to verify current status given the dynamic nature of the area.
|
|
\end{enumerate}
|
|
|
|
\section{Contract Analysis}
|
|
|
|
\subsection{CUAD Benchmark Tasks}
|
|
|
|
The Contract Understanding Atticus Dataset (CUAD) includes 41 clause types across commercial contracts. Zen Legal achieves the following results:
|
|
|
|
\begin{table}[H]
|
|
\centering
|
|
\caption{CUAD contract clause extraction F1 (top 15 clause types)}
|
|
\label{tab:cuad}
|
|
\begin{tabular}{lcc}
|
|
\toprule
|
|
Clause Type & F1 & Recall \\
|
|
\midrule
|
|
Governing Law & 0.972 & 0.981 \\
|
|
Limitation of Liability & 0.941 & 0.958 \\
|
|
Indemnification & 0.928 & 0.944 \\
|
|
Non-compete & 0.918 & 0.934 \\
|
|
Termination for Cause & 0.924 & 0.941 \\
|
|
Assignment & 0.931 & 0.948 \\
|
|
IP Ownership & 0.912 & 0.928 \\
|
|
Audit Rights & 0.904 & 0.921 \\
|
|
Change of Control & 0.896 & 0.914 \\
|
|
Warranty Duration & 0.884 & 0.902 \\
|
|
Renewal Term & 0.918 & 0.934 \\
|
|
Most Favored Nation & 0.872 & 0.891 \\
|
|
Exclusivity & 0.868 & 0.888 \\
|
|
Anti-assignment & 0.914 & 0.931 \\
|
|
Force Majeure & 0.894 & 0.912 \\
|
|
\midrule
|
|
\textbf{CUAD Average (41 types)} & \textbf{0.923} & \textbf{0.938} \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\subsection{Contract Risk Scoring}
|
|
|
|
Beyond clause extraction, Zen Legal generates risk assessments for each contract:
|
|
|
|
\begin{itemize}
|
|
\item \textbf{Clause-level risk}: Each extracted clause is scored on a 1--5 risk scale relative to market standard terms.
|
|
\item \textbf{Missing clause detection}: Identifies standard clauses absent from the agreement (e.g., no limitation of liability, no governing law clause).
|
|
\item \textbf{Inconsistency detection}: Flags internally inconsistent provisions (e.g., conflicting notice periods in different sections).
|
|
\item \textbf{Redline suggestions}: Proposes alternative language for high-risk clauses with explanation.
|
|
\end{itemize}
|
|
|
|
\section{Legal Benchmark Results}
|
|
|
|
\subsection{LegalBench}
|
|
|
|
LegalBench is a collaborative benchmark of 162 legal reasoning tasks spanning case law, statutes, and legal documents across diverse legal domains.
|
|
|
|
\begin{table}[H]
|
|
\centering
|
|
\caption{LegalBench results by legal domain}
|
|
\label{tab:legalbench}
|
|
\begin{tabular}{lcc}
|
|
\toprule
|
|
Domain & Tasks & Accuracy \\
|
|
\midrule
|
|
Contract law & 28 & 84.2\% \\
|
|
Constitutional law & 18 & 79.4\% \\
|
|
Administrative law & 14 & 77.8\% \\
|
|
Criminal law & 22 & 82.4\% \\
|
|
Tort law & 16 & 83.8\% \\
|
|
Statutory interpretation & 24 & 80.2\% \\
|
|
International law & 12 & 76.8\% \\
|
|
Procedural law & 28 & 83.4\% \\
|
|
\midrule
|
|
\textbf{LegalBench Overall} & \textbf{162} & \textbf{81.4\%} \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\subsection{Case Outcome Prediction}
|
|
|
|
\begin{table}[H]
|
|
\centering
|
|
\caption{Case outcome prediction accuracy by court level}
|
|
\label{tab:outcome}
|
|
\begin{tabular}{lccc}
|
|
\toprule
|
|
Court & Cases & Accuracy & AUC \\
|
|
\midrule
|
|
U.S. Supreme Court & 8,412 & 78.4\% & 0.841 \\
|
|
U.S. Circuit Courts & 142,841 & 74.8\% & 0.812 \\
|
|
U.S. District Courts & 284,121 & 72.4\% & 0.784 \\
|
|
State Supreme Courts & 48,412 & 71.8\% & 0.778 \\
|
|
\midrule
|
|
\textbf{Overall} & — & \textbf{74.2\%} & \textbf{0.804} \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\subsection{Legal NLP Tasks}
|
|
|
|
\begin{table}[H]
|
|
\centering
|
|
\caption{Legal NLP task results}
|
|
\label{tab:legal_nlp}
|
|
\begin{tabular}{llcc}
|
|
\toprule
|
|
Task & Dataset & Metric & Score \\
|
|
\midrule
|
|
Statute classification & EUR-Lex & Micro-F1 & 0.848 \\
|
|
Legal judgment prediction & ECHR & Accuracy & 81.4\% \\
|
|
Obligation detection & ContractNLI & F1 & 0.872 \\
|
|
Legal argument mining & ECHRArguments & F1 & 0.814 \\
|
|
Regulatory compliance & PrivacyQA & F1 & 0.864 \\
|
|
Legal summarization & Multi-LexSum & ROUGE-L & 0.384 \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\section{Legal Research Workflow}
|
|
|
|
\subsection{Memorandum of Law Generation}
|
|
|
|
Zen Legal generates complete legal memoranda following standard structure:
|
|
|
|
\begin{enumerate}
|
|
\item \textbf{Question Presented}: Precise formulation of the legal question.
|
|
\item \textbf{Brief Answer}: Concise answer with legal conclusion and confidence.
|
|
\item \textbf{Statement of Facts}: Relevant factual background.
|
|
\item \textbf{Discussion}: IRAC (Issue, Rule, Application, Conclusion) analysis with full citations.
|
|
\item \textbf{Conclusion}: Summary of findings and recommended next steps.
|
|
\end{enumerate}
|
|
|
|
Attorney evaluation of generated memoranda (blind comparison vs. associate-drafted):
|
|
|
|
\begin{table}[H]
|
|
\centering
|
|
\caption{Attorney evaluation of AI-generated legal memoranda (N=150)}
|
|
\label{tab:memo_eval}
|
|
\begin{tabular}{lcc}
|
|
\toprule
|
|
Criterion & Zen Legal Score & Associate Avg \\
|
|
\midrule
|
|
Legal accuracy & 4.24 / 5 & 4.18 / 5 \\
|
|
Citation completeness & 4.41 / 5 & 3.84 / 5 \\
|
|
Analytical clarity & 4.08 / 5 & 4.28 / 5 \\
|
|
Conclusion soundness & 4.18 / 5 & 4.22 / 5 \\
|
|
\textbf{Overall} & \textbf{4.23 / 5} & \textbf{4.13 / 5} \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\section{Professional Responsibility Compliance}
|
|
|
|
\subsection{Unauthorized Practice of Law}
|
|
|
|
Zen Legal explicitly frames all outputs as legal research assistance, not legal advice. The system:
|
|
|
|
\begin{itemize}
|
|
\item Appends standard disclaimers: ``This output does not constitute legal advice and does not create an attorney-client relationship.''
|
|
\item Monitors for jurisdiction-specific UPL rules that constrain AI-generated legal content.
|
|
\item For consumer-facing deployments, routes complex matters to licensed attorney review before delivery.
|
|
\end{itemize}
|
|
|
|
\subsection{Competence and Supervision}
|
|
|
|
Per ABA Model Rule 5.3, attorneys using AI tools must supervise the AI's work competently. Zen Legal supports this by:
|
|
|
|
\begin{itemize}
|
|
\item Providing explicit confidence scores on all legal conclusions.
|
|
\item Flagging recently decided cases ($<$6 months) that may not be fully integrated in training data.
|
|
\item Recommending verification steps for time-sensitive or high-stakes matters.
|
|
\end{itemize}
|
|
|
|
\section{Conclusion}
|
|
|
|
Zen Legal demonstrates that rigorous citation grounding, jurisdictional awareness, and professional responsibility compliance can be integrated into a production legal AI system without sacrificing capability. The 81.4\% LegalBench score and the 0.923 CUAD clause-extraction F1 establish Zen Legal as state-of-the-art for AI-assisted legal research, with attorney evaluations confirming practical utility comparable to junior associate work product.
|
|
|
|
\begin{thebibliography}{99}
|
|
\bibitem{legalbench} Guha, N. et al. LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. \textit{NeurIPS}, 2023.
|
|
\bibitem{cuad} Hendrycks, D. et al. CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review. \textit{NeurIPS}, 2021.
|
|
\bibitem{echr} Chalkidis, I. et al. Neural Legal Judgment Prediction in English. \textit{ACL}, 2019.
|
|
\bibitem{contractnli} Koreeda, Y. and Manning, C. ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts. \textit{EMNLP Findings}, 2021.
|
|
\bibitem{multilexsum} Shen, Z. et al. Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits. \textit{NeurIPS}, 2022.
|
|
\end{thebibliography}
|
|
|
|
\end{document}
|