papers/zen/zen-3d.tex at main

mirror of https://github.com/hanzoai/papers.git synced 2026-10-29 04:32:13 +00:00

Files

T

zeekayandHanzo Dev bc1452e9f0 The zen claims that arithmetic settles

zenlm/papers had a sweeping withdrawal pass, 5deeeaa, and the owner reverted it in
f34f609 for asserting more than it verified. That objection sets the rule here: a
claim changes only where the proof is in the paper itself or in arithmetic, never
because a run could not be found. Twenty-seven of the eighty-one zen papers meet
that bar. Every correction quotes a number the paper already carries or one derived
from its own numbers by a division a reader can repeat.

Twelve papers led with a figure that appears in no row of the table it summarises.
zen-agent averaged GAIA 83.1/68.4/42.3 to 71.2 while its four baseline rows are each
the plain mean of their own three levels; the mean is 64.6. zen-embeddings-retrieval
and zen3-embedding claim MTEB averages of 72.4 and 74.3 over tables that supply their
task counts, where the 56-task weighted means are 67.1 and 67.4.
zen-mathematical-reasoning opened with five figures, none of them any row, and
explained one as best-of-32 sampling when its own sampling table puts best-of-32 at a
different number. zen-dso-protocol claimed 94% of centralized quality and a 78%
bandwidth cut against tables reading 99.4% and 99.8%-and-better, the second of them
the table that claim captions. zen-synthetic-data claimed an 8.4-point MT-Bench gain
on a scale that runs to 10, from a 7.84 baseline, where its table says 0.88.
zen-3d quoted the Chair column as the ShapeNet result. zen-multilingual understated
its own XNLI average. Also zen-hallucination-reduction, zen-distributed-training,
zen-finetuning, zen-legal-ai and zen-coder-flash.

Eight more are settled by arithmetic against the hardware or the parameter count.
zen-nano, zen-eco and zen-scribe each report INT4 needing as much memory as FP16 or
more, while their own FP16 and INT8 rows are exactly two and one bytes per parameter;
zen_family_overview already carried the right values. The two zen-designer papers size
a stated 235B model at 220 GB in FP16, which is a 110B model, and zen-quantization puts
the same model at 470 GB. zen-omni and zen-next recommend cards smaller than the memory
the same row requires. zen-live captions a latency table to an A10G when its own
deployment table assigns that 98 ms to the H100. zen-hardware-optimization calls a
throughput table FP8+INT4 when its cumulative table reaches the identical 31,200 tok/s
through INT4 + FP16.

Six papers share one template whose visual benchmark block bundles two metrics
computed over images a model generates with one computed over text a model emits
about an image. No model produces all three, and the block went to four papers across
four modalities, so each received at least one metric it cannot compute: VQA v2 and a
token rate for two diffusion models, FID and CLIP score for two vision-language models.
Each now keeps the rows its own stated output type can produce.

Throughput is the one category where the defect is the missing condition rather than
the number. Decode reads the active weight set once per token, so a single stream
cannot outrun bandwidth over footprint, and zen4-mini, zen3-nano and zen-coder-flash
all report above their own bound. Batched serving would make those figures legal and
none of the three records a batch size, so the numbers stand and each table now carries
its bound and the arithmetic behind it. AIME gets the same treatment in
zen-mathematical-reasoning: a score that is not a multiple of 1/30 on a 30-problem exam
is legal under averaging, and the caption now says which row has its sample count on
record and which do not.

What was examined and left is in LLM.md with the reason in each case. Training fleets
absent from this estate prove nothing about models trained on rented capacity.
zen-reranker's 98% semantic preservation is uncited rather than contradicted.
zen-safety-evaluation and zen-vision-architecture state no weighting, so their averages
are not determined. zen4-coder-flash puts FP8 on an A100, and Ampere has no FP8 tensor
cores, but its claims there are footprint claims and FP8 weight-only storage does run
on Ampere.

All twenty-seven rebuild under latexmk -Werror. Nineteen have diverged from their
counterparts in zenlm/papers and six exist only here, so those fixes do not reach the
other site. zen-3d and zen-agent were byte-identical across the two repositories until
now, and are the two to mirror first.

Co-authored-by: Hanzo Dev <[email protected]>

2026-09-10 23:54:37 -07:00

458 lines

17 KiB

TeX

Raw Permalink Blame History

 \documentclass[11pt,a4paper]{article}
 \usepackage[utf8]{inputenc}
 \usepackage[T1]{fontenc}
 \usepackage{amsmath,amsfonts,amssymb}
 \usepackage{graphicx}
 \usepackage{hyperref}
 \usepackage{listings}
 \usepackage[dvipsnames]{xcolor}
 \usepackage{booktabs}
 \usepackage{float}
 \usepackage{geometry}
 \geometry{margin=1in}
 \definecolor{zenblue}{RGB}{41,121,255}
 \definecolor{zengreen}{RGB}{52,199,89}
 \definecolor{codegray}{RGB}{245,245,245}
 \hypersetup{colorlinks=true,linkcolor=zenblue,urlcolor=zenblue,citecolor=zenblue}
 \lstset{
     backgroundcolor=\color{codegray},
     basicstyle=\ttfamily\small,
     breaklines=true,
     captionpos=b,
     frame=single,
     numbers=left,
     numberstyle=\tiny\color{gray}
 }
 \title{
     \vspace{-2cm}
     \Large \textbf{Zen AI Model Family} \\
     \vspace{0.5cm}
     \Huge \textbf{Zen-3D} \\
     \vspace{0.3cm}
     \large 3D Scene Understanding and Generation at Scale \\
     \vspace{0.5cm}
     \normalsize Technical Report v2025.01
 }
 \author{
     Hanzo AI Research Team\thanks{[email protected]} \and
     Zoo Labs Foundation\thanks{[email protected]}
 }
 \date{January 2025}
 \begin{document}
 \maketitle
 \begin{abstract}
 We present \textbf{Zen-3D}, an 8-billion parameter model specialized for 3D scene understanding,
 mesh generation, and point cloud processing. Built on the Zen MoDE (Mixture of Distilled Experts)
 architecture with domain-adapted 3D tokenization, Zen-3D achieves state-of-the-art results across
 all major 3D benchmarks: 92.2\% mean IoU on ShapeNet, 88.3\% mIoU on ScanNet semantic segmentation,
 and 94.1\% classification accuracy on ModelNet40. Unlike prior work that treats 3D understanding
 as a downstream task grafted onto language models, Zen-3D natively encodes spatial geometry through
 a novel triplane-based 3D tokenizer and dedicated attention mechanisms for volumetric context.
 The model supports interactive scene reconstruction, counterfactual 3D editing, and zero-shot
 generalization to unseen object categories---enabling downstream applications in robotics, augmented
 reality, and generative design at production scale.
 \end{abstract}
 \tableofcontents
 \newpage
 \section{Introduction}
 Three-dimensional scene understanding has long been a frontier capability distinguishing physically
 situated AI from purely linguistic systems. Tasks such as reconstructing a scene from partial
 observations, segmenting semantically meaningful regions in a point cloud, or generating plausible
 D meshes from text descriptions require models that reason natively about geometry, topology, and
 spatial relations---not merely pattern-match against 2D projections.
 Prior multimodal approaches to 3D have taken two broad strategies: (1) projecting 3D data into
 sequences of 2D renders and applying vision-language models, or (2) building bespoke 3D encoders
 that feed compressed representations into a language backbone. Both strategies suffer from either
 information loss in the projection step or a semantic gap between the 3D encoder and the pretrained
 language prior.
 Zen-3D resolves this tension through three contributions:
 \begin{enumerate}
     \item \textbf{Native 3D tokenization}: A triplane decomposition tokenizer that converts point
     clouds and voxel grids into discrete token sequences without lossy projection.
     \item \textbf{Geometry-aware attention}: Modified self-attention with relative 3D position
     encodings, enabling the model to reason about spatial adjacency, occlusion, and scale.
     \item \textbf{Unified generation and understanding}: A single model that handles segmentation,
     reconstruction, and conditional mesh generation through task-specific decoding heads built on
     the shared Zen MoDE backbone.
 \end{enumerate}
 \subsection{Model Overview}
 \begin{table}[H]
 \centering
 \begin{tabular}{ll}
 \toprule
 \textbf{Property} & \textbf{Value} \\
 \midrule
 Parameters & 8B \\
 Architecture & Zen MoDE (Mixture of Distilled Experts) \\
 D Tokenizer & Triplane VQ-VAE (codebook size 16,384) \\
 Context Length & 32,768 tokens (spatial + language) \\
 Max Resolution & 256$^3$ voxels / 100K point clouds \\
 Output Modalities & Mesh, point cloud, segmentation mask, text \\
 Training Data & 47M 3D object-text pairs, 2.1M scene reconstructions \\
 \bottomrule
 \end{tabular}
 \caption{Zen-3D Model Specifications}
 \end{table}
 \section{Architecture}
 \subsection{Triplane 3D Tokenizer}
 Input 3D data---whether a point cloud, a voxel grid, or an implicit surface---is first projected
 onto three orthogonal feature planes ($XY$, $XZ$, $YZ$). Each plane is independently encoded by a
 convolutional backbone and then vector-quantized to produce a sequence of discrete tokens. The three
 plane token sequences are concatenated to form the full 3D token sequence fed to the transformer.
 Formally, given input geometry $\mathcal{G} \in \mathbb{R}^{N \times 3}$:
 \begin{align}
     F_{XY}, F_{XZ}, F_{YZ} &= \text{TriplaneProject}(\mathcal{G}) \\
     T_i &= \text{VQ}(\text{ConvEnc}(F_i)), \quad i \in \{XY, XZ, YZ\} \\
     T_{3D} &= [T_{XY} \; || \; T_{XZ} \; || \; T_{YZ}]
 \end{align}
 The VQ-VAE codebook is trained jointly with the transformer backbone using a commitment loss
 $\mathcal{L}_{commit} = \|z_e - \text{sg}(z_q)\|_2^2$ where $z_e$ is the encoder output and
 $z_q$ is the nearest codebook entry.
 \subsection{Geometry-Aware Attention}
 Standard self-attention treats tokens as an unordered set, discarding geometric structure. We
 introduce \textbf{Spatial Relative Position Encoding (SRPE)}: for any pair of 3D tokens $(i, j)$,
 we compute their approximate 3D centroid distance $d_{ij}$ and inject a learned distance bias into
 the attention logits:
 \begin{equation}
     a_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_k}} + b(d_{ij})
 \end{equation}
 where $b: \mathbb{R}_{\geq 0} \to \mathbb{R}$ is a learned monotone function discretized into 64
 distance buckets covering 0--10 meters in log-scale.
 \subsection{Zen MoDE Backbone}
 The transformer backbone follows the Zen MoDE architecture: 32 transformer layers, each with 16
 attention heads and a mixture-of-experts feed-forward network with 8 experts (top-2 routing per
 token). The 3D token stream and optional text tokens are interleaved via cross-attention at every
 fourth layer, enabling bidirectional grounding between language and geometry.
 \begin{table}[H]
 \centering
 \begin{tabular}{ll}
 \toprule
 \textbf{Component} & \textbf{Configuration} \\
 \midrule
 Transformer layers & 32 \\
 Attention heads & 16 \\
 Hidden dimension & 4096 \\
 FFN type & MoE (8 experts, top-2) \\
 Active parameters per token & $\approx$2.1B of 8B total \\
 Cross-attention layers & Every 4th (8 total) \\
 Positional encoding & Rotary (RoPE) + SRPE \\
 \bottomrule
 \end{tabular}
 \caption{Zen-3D Backbone Architecture}
 \end{table}
 \subsection{Task-Specific Decoding Heads}
 Three decoding heads share the backbone:
 \textbf{Segmentation head}: A lightweight MLP applied to per-token backbone features, producing
 per-point semantic logits over a 200-class ontology (ShapeNet taxonomy + ScanNet classes).
 \textbf{Reconstruction head}: A triplane decoder that inverts the tokenizer, predicting an
 occupancy field from which meshes are extracted via marching cubes at configurable resolution.
 \textbf{Generation head}: A diffusion decoder conditioned on text-derived embeddings, producing
 novel 3D shapes by iteratively denoising a triplane latent.
 \section{Training}
 \subsection{Dataset}
 \begin{table}[H]
 \centering
 \begin{tabular}{lrrl}
 \toprule
 \textbf{Source} & \textbf{Samples} & \textbf{Proportion} & \textbf{Modality} \\
 \midrule
 ShapeNet v2 & 51,300 & 0.1\% & Mesh + text \\
 Objaverse 1.0 & 800,000 & 1.7\% & Mesh + text \\
 Objaverse-XL & 10,200,000 & 21.6\% & Mesh + text \\
 ScanNet v2 & 1,513 scenes & 0.003\% & Point cloud + segmentation \\
 D-FRONT & 18,000 scenes & 0.04\% & Scene mesh + layout \\
 Synthetic augmentation & 36,000,000 & 76.5\% & Rendered point clouds \\
 \midrule
 \textbf{Total} & \textbf{47,070,813} & 100\% & \\
 \bottomrule
 \end{tabular}
 \caption{Zen-3D Training Data Composition}
 \end{table}
 \subsection{Training Protocol}
 Training proceeds in three stages:
 \textbf{Stage 1 -- Tokenizer pretraining} (50K steps): The VQ-VAE is trained standalone on the
 full 3D corpus with reconstruction loss and commitment loss. Codebook utilization is monitored;
 dead entries are re-initialized via exponential moving average updates.
 \textbf{Stage 2 -- Backbone pretraining} (200K steps): The Zen MoDE backbone is pretrained on
 tokenized 3D sequences with a masked-token prediction objective, analogous to masked language
 modeling but over the 3D token space.
 \textbf{Stage 3 -- Task fine-tuning} (100K steps): All three decoding heads are trained jointly
 with task-specific losses: cross-entropy for segmentation, binary cross-entropy occupancy loss for
 reconstruction, and diffusion DDPM loss for generation. Tasks are sampled at ratio 4:3:3.
 \begin{table}[H]
 \centering
 \begin{tabular}{lllll}
 \toprule
 \textbf{Stage} & \textbf{Steps} & \textbf{Batch} & \textbf{LR} & \textbf{Hardware} \\
 \midrule
 Tokenizer & 50K & 512 & 1e-4 & 16$\times$A100 80GB \\
 Backbone & 200K & 256 & 3e-4 & 64$\times$A100 80GB \\
 Fine-tune & 100K & 128 & 5e-5 & 32$\times$A100 80GB \\
 \bottomrule
 \end{tabular}
 \caption{Training Configuration by Stage}
 \end{table}
 \subsection{Data Augmentation}
 To improve robustness to real-world sensor noise, all point cloud inputs during training undergo:
 random dropout (10--30\% of points), Gaussian jitter ($\sigma = 0.01$ m), random rotation about
 the vertical axis, and random scale perturbation ($\pm 10\%$).
 \section{Evaluation}
 \subsection{ShapeNet Object Reconstruction}
 We evaluate 3D reconstruction quality on the ShapeNet v2 test split using Intersection over Union
 (IoU) at 0.5 occupancy threshold.
 \begin{table}[H]
 \centering
 \begin{tabular}{lccccc}
 \toprule
 \textbf{Model} & \textbf{Chair} & \textbf{Table} & \textbf{Lamp} & \textbf{Car} & \textbf{Mean IoU} \\
 \midrule
 ONet & 82.3 & 79.1 & 68.4 & 83.7 & 75.9 \\
 ConvONet & 88.6 & 85.2 & 74.1 & 88.9 & 82.4 \\
 IF-Net & 90.1 & 87.3 & 76.8 & 90.2 & 84.6 \\
 D-LLM & 91.4 & 88.7 & 78.3 & 91.5 & 87.2 \\
 \textbf{Zen-3D} & \textbf{95.2} & \textbf{93.8} & \textbf{84.1} & \textbf{95.7} & \textbf{92.2} \\
 \bottomrule
 \end{tabular}
 \caption{ShapeNet Reconstruction (IoU \%, higher is better)}
 \end{table}
 \subsection{ScanNet Semantic Segmentation}
 Point cloud semantic segmentation on ScanNet v2 validation set (20 classes).
 \begin{table}[H]
 \centering
 \begin{tabular}{lcccc}
 \toprule
 \textbf{Model} & \textbf{mIoU} & \textbf{mAcc} & \textbf{oAcc} & \textbf{Parameters} \\
 \midrule
 PointNet++ & 53.5 & 64.2 & 85.3 & 1.5M \\
 KPConv & 68.4 & 79.1 & 88.1 & 14.3M \\
 PointTransformer & 70.6 & 81.9 & 90.2 & 7.8M \\
 Mask3D & 73.7 & 83.4 & 91.0 & 38.4M \\
 Point-MAE + ScanNet & 80.3 & 87.2 & 91.8 & 22.1M \\
 \textbf{Zen-3D} & \textbf{88.3} & \textbf{92.1} & \textbf{94.6} & 8B \\
 \bottomrule
 \end{tabular}
 \caption{ScanNet Semantic Segmentation (validation, higher is better)}
 \end{table}
 \subsection{ModelNet40 Classification}
 Object classification on ModelNet40 with 1,024 input points, 10-fold cross-validation.
 \begin{table}[H]
 \centering
 \begin{tabular}{lcc}
 \toprule
 \textbf{Model} & \textbf{OA (\%)} & \textbf{mAcc (\%)} \\
 \midrule
 PointNet & 89.2 & 86.2 \\
 PointNet++ & 91.9 & 90.7 \\
 DGCNN & 92.9 & 90.2 \\
 Point-BERT & 93.2 & 90.6 \\
 Point-MAE & 93.8 & 90.6 \\
 PointGPT-L & 94.0 & 91.1 \\
 \textbf{Zen-3D} & \textbf{94.1} & \textbf{91.3} \\
 \bottomrule
 \end{tabular}
 \caption{ModelNet40 Classification (OA = Overall Accuracy)}
 \end{table}
 \subsection{Text-to-3D Generation Quality}
 We evaluate generative quality using the Frechet Point Cloud Distance (FPCD) and CLIP-3D alignment
 score on a held-out set of 5,000 text prompts from the Cap3D dataset.
 \begin{table}[H]
 \centering
 \begin{tabular}{lcc}
 \toprule
 \textbf{Model} & \textbf{FPCD} $\downarrow$ & \textbf{CLIP-3D} $\uparrow$ \\
 \midrule
 Shap-E & 143.2 & 0.312 \\
 Point-E & 128.7 & 0.331 \\
 DreamFusion & 115.4 & 0.347 \\
 One-2-3-45 & 98.3 & 0.369 \\
 \textbf{Zen-3D} & \textbf{71.4} & \textbf{0.403} \\
 \bottomrule
 \end{tabular}
 \caption{Text-to-3D Generation (Cap3D evaluation set)}
 \end{table}
 \subsection{Zero-Shot Generalization}
 To assess zero-shot generalization, we evaluate on the Objaverse-LVIS benchmark (1,156 novel
 categories not seen during training). Zen-3D achieves 61.3\% top-1 classification accuracy,
 compared to 43.7\% for the best prior approach, demonstrating strong zero-shot transfer from
 the large-scale pretraining regime.
 \section{Applications}
 \subsection{Robotics Scene Understanding}
 Zen-3D has been integrated into the Hanzo robotics perception stack, where it operates on
 streaming LiDAR point clouds at 5 Hz (200ms per frame on a single A10G GPU). The model
 simultaneously provides semantic segmentation for grasp planning and 3D occupancy maps for
 collision-free path planning.
 \subsection{Augmented Reality}
 For AR applications, Zen-3D's reconstruction head is used to generate watertight meshes from
 partial RGBD observations in real time, enabling stable AR object placement without depth
 sensor drift artifacts.
 \subsection{Generative 3D Design}
 The text-to-3D generation pipeline enables designers to specify 3D shapes in natural language
 and receive printable mesh outputs in under 10 seconds. Integration with standard CAD tools
 (FreeCAD, Blender) is provided via the Zen SDK.
 \subsection{Scene Counterfactual Editing}
 Zen-3D supports counterfactual scene editing: given a complete scene reconstruction and a natural
 language instruction (e.g., ``remove the chair and place a desk''), the model regenerates the
 affected scene region while maintaining geometric consistency with the unchanged context.
 \section{Inference and Deployment}
 \subsection{Inference Performance}
 \begin{table}[H]
 \centering
 \begin{tabular}{lrrl}
 \toprule
 \textbf{Task} & \textbf{Latency (ms)} & \textbf{Throughput} & \textbf{Hardware} \\
 \midrule
 Segmentation (10K pts) & 45 & 22 scenes/s & A10G \\
 Reconstruction (ShapeNet) & 120 & 8 meshes/s & A10G \\
 Text-to-3D generation & 8,200 & 0.12 shapes/s & A100 \\
 Classification (1K pts) & 12 & 83 objects/s & T4 \\
 \bottomrule
 \end{tabular}
 \caption{Zen-3D Inference Performance}
 \end{table}
 \subsection{Quantization}
 Zen-3D supports INT8 quantization via the Zen SDK with less than 0.5\% degradation on all
 benchmarks. The quantized model fits in 10GB VRAM, enabling deployment on consumer-grade GPUs
 (RTX 3080 and above).
 \subsection{Python API}
 \begin{lstlisting}[language=Python, caption=Zen-3D Inference Example]
 from zen import Zen3D
 import numpy as np
 model = Zen3D.from_pretrained("zenlm/zen-3d-8b")
 # Semantic segmentation from point cloud
 points = np.load("scene.npy")  # (N, 3) float32
 labels = model.segment(points, taxonomy="scannet20")
 # Text-to-3D generation
 mesh = model.generate(
     "a modern office chair with armrests",
     resolution=128,
     num_steps=50
 )
 mesh.export("chair.obj")
 # Scene reconstruction from partial observations
 partial = points[:points.shape[0] // 2]
 complete = model.reconstruct(partial)
 \end{lstlisting}
 \section{Related Work}
 \subsection{3D Representation Learning}
 PointNet \cite{qi2017pointnet} established the paradigm of operating directly on unordered point
 sets via symmetric functions. PointNet++ \cite{qi2017pointnet2} extended this with hierarchical
 local feature learning. Transformer-based approaches (Point-BERT, Point-MAE) adapted masked
 pretraining from NLP to 3D, achieving strong transfer performance.
 \subsection{3D Generation}
 Occupancy Networks \cite{mescheder2019occupancy} and Implicit Functions \cite{park2019deepsdf}
 enabled continuous implicit representations. Diffusion-based generators (Shap-E, Point-E) brought
 generative quality closer to 2D image synthesis. Zen-3D advances this line by conditioning
 generation on the same backbone used for understanding, enabling tighter semantic alignment.
 \subsection{Vision-Language 3D Models}
 D-LLM and related work demonstrated that language model priors can improve 3D understanding.
 Zen-3D builds on this direction but replaces the 2D-projection interface with native 3D
 tokenization, eliminating the projection bottleneck.
 \section{Conclusion}
 Zen-3D demonstrates that a single 8B parameter model, trained with the Zen MoDE architecture
 and a purpose-built triplane tokenizer, can simultaneously achieve state-of-the-art results
 across 3D segmentation (ScanNet mIoU 88.3\%), reconstruction (ShapeNet mean IoU 92.2\%), and
 classification (ModelNet40 94.1\%). The unified backbone enables cross-task transfer that
 benefits all three capabilities simultaneously.
 Future work will extend Zen-3D to dynamic 4D scene understanding (point cloud video), tighter
 integration with the Zen-Omni multimodal backbone for joint 2D+3D understanding, and
 distillation of the 8B model to a sub-1B edge variant for real-time mobile AR applications.
 \begin{thebibliography}{10}
 \bibitem{qi2017pointnet} R. Q. Charles et al., ``PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation,'' CVPR, 2017.
 \bibitem{qi2017pointnet2} C. R. Qi et al., ``PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space,'' NeurIPS, 2017.
 \bibitem{mescheder2019occupancy} L. Mescheder et al., ``Occupancy Networks: Learning 3D Reconstruction in Function Space,'' CVPR, 2019.
 \bibitem{park2019deepsdf} J. J. Park et al., ``DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation,'' CVPR, 2019.
 \end{thebibliography}
 \end{document}

458 lines 17 KiB TeX Raw Permalink Blame History

458 lines

17 KiB

TeX

Raw Permalink Blame History