mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-29 07:34:31 +00:00
zeekay
f9db4f5737
Bylines, paper authorship, correspondence addresses and the founder blurb under each post named an individual. They name the organisation that published the work instead: the brand on posts and pages, the research byline on papers, and research@ in place of a personal address. Both are what these repositories already used elsewhere — nothing new was coined. Co-authors are untouched. Where a paper named someone else alongside, they stay named; only the one byline changed. Bibliography entries, running heads, PDF metadata and affiliations move with it, so a citation still resolves to whoever the cited paper names. Sample and fixture data uses the placeholder name these tests already use elsewhere, so no assertion changed meaning.
74 lines
2.8 KiB
TeX
74 lines
2.8 KiB
TeX
% Hanzo GPU Inference: GPU-Accelerated AI Inference on Hanzo Cloud
|
|
\documentclass[11pt]{article}
|
|
\usepackage{../shared/hanzocover}
|
|
\usepackage{amsmath, amssymb, amsthm}
|
|
\usepackage{mathtools}
|
|
\usepackage{bm}
|
|
\usepackage{graphicx}
|
|
\usepackage{booktabs}
|
|
\usepackage{hyperref}
|
|
\usepackage{enumitem}
|
|
\usepackage{algorithm}
|
|
\usepackage{algpseudocode}
|
|
\usepackage{xcolor}
|
|
\usepackage{float}
|
|
\usepackage{multirow}
|
|
|
|
\hypersetup{colorlinks=true,linkcolor=black,citecolor=blue,urlcolor=blue}
|
|
|
|
\newtheorem{theorem}{Theorem}
|
|
\newtheorem{lemma}[theorem]{Lemma}
|
|
\newtheorem{proposition}[theorem]{Proposition}
|
|
\newtheorem{corollary}[theorem]{Corollary}
|
|
\newtheorem{definition}{Definition}
|
|
\newtheorem{remark}{Remark}
|
|
|
|
\title{HANZO GPU INFERENCE\\[0.3em]
|
|
{\large GPU-Accelerated AI Inference\\on Hanzo Cloud}}
|
|
\author{
|
|
Hanzo AI Research \\
|
|
\textit{Hanzo Industries Inc (Techstars '17)} \\
|
|
\texttt{[email protected]}
|
|
}
|
|
\date{March 2026}
|
|
|
|
\begin{document}
|
|
|
|
\hanzocoverpage
|
|
|
|
% ============================================================================
|
|
% ABSTRACT
|
|
% ============================================================================
|
|
\begin{abstract}
|
|
We present the GPU inference architecture of Hanzo Cloud, a production system
|
|
serving over 47 million inference requests per month across heterogeneous GPU
|
|
clusters. The system addresses four challenges in large-scale model serving:
|
|
(i)~batch inference with adaptive batching that maximizes GPU utilization while
|
|
meeting latency SLAs, (ii)~multi-model GPU sharing via spatial and temporal
|
|
multiplexing on a single device, (iii)~GPU-aware scheduling that accounts for
|
|
memory bandwidth, compute capacity, and thermal constraints, and (iv)~latency
|
|
optimization through speculative decoding, KV-cache management, and continuous
|
|
batching. We describe the architecture deployed on Hanzo ACI nodes, where GPU
|
|
operators stake HANZO tokens and serve Zen foundation models (Qwen3+). Production
|
|
measurements on NVIDIA H100 and A100 clusters demonstrate 3.2$\times$ throughput
|
|
improvement over naive serving, p99 latency under 180ms for 7B-parameter models,
|
|
and 94\% average GPU utilization across the fleet. The system integrates with
|
|
Hanzo Gateway for authentication and routing, Hanzo IAM for tenant isolation,
|
|
and the Candle runtime for Rust-native inference kernels.
|
|
\end{abstract}
|
|
|
|
% ============================================================================
|
|
|
|
\input{sections/01-introduction}
|
|
\input{sections/02-system-architecture}
|
|
\input{sections/03-adaptive-continuous-batching}
|
|
\input{sections/04-multi-model-gpu-sharing}
|
|
\input{sections/05-gpu-aware-cluster-scheduling}
|
|
\input{sections/06-latency-optimization}
|
|
\input{sections/07-production-results}
|
|
\input{sections/08-integration-with-hanzo-infrastructure}
|
|
\input{sections/09-related-work}
|
|
\input{sections/99-conclusion}
|
|
|
|
\end{document}
|