mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-29 07:34:31 +00:00
zeekayandHanzo Dev
b79b939f11
Every author block and the authorship standard said Los Angeles; Hanzo has been in San Francisco since at least 2016. The five catalogue PDFs whose sources changed are rebuilt so papers.hanzo.ai says so too. Co-authored-by: Hanzo Dev <[email protected]>
68 lines
2.7 KiB
TeX
68 lines
2.7 KiB
TeX
\documentclass[11pt]{article}
|
|
\usepackage{../shared/hanzocover}
|
|
\usepackage{amsmath, amssymb, amsthm}
|
|
\usepackage{mathtools}
|
|
\usepackage{graphicx}
|
|
\usepackage{booktabs}
|
|
\usepackage{hyperref}
|
|
\usepackage{enumitem}
|
|
\usepackage{xcolor}
|
|
\usepackage{float}
|
|
\usepackage{multirow}
|
|
\usepackage{listings}
|
|
\input{../shared/lstlang}
|
|
\usepackage[numbers]{natbib}
|
|
\hypersetup{colorlinks=true,linkcolor=black,citecolor=blue,urlcolor=blue}
|
|
|
|
\newtheorem{theorem}{Theorem}
|
|
\newtheorem{lemma}{Lemma}
|
|
\newtheorem{definition}{Definition}
|
|
\newtheorem{proposition}{Proposition}
|
|
|
|
\title{Datastore: Profile-Guided, Post-Link, and Transport\\ Optimization of a Columnar Analytics Engine}
|
|
\author{
|
|
Hanzo AI Research\thanks{Corresponding author: [email protected]} \\
|
|
\textit{Hanzo Industries Inc (Techstars '17)} \\
|
|
San Francisco, California \\
|
|
\texttt{https://hanzo.ai}
|
|
}
|
|
\date{June 2026}
|
|
|
|
\begin{document}
|
|
\hanzocoverpage
|
|
|
|
\begin{abstract}
|
|
We present \emph{Datastore}, Hanzo's analytics database: a fork of the
|
|
ClickHouse columnar engine (Apache~2.0) re-engineered for a leaner resident
|
|
footprint, a smaller binary, and a thinner transport surface. We report three
|
|
results. First, applying profile-guided optimization (PGO), ThinLTO, and the
|
|
BOLT post-link optimizer yields a server whose steady-state resident set is
|
|
$49$--$53$\,MB versus $\sim\!116$\,MB for the upstream official binary --- less
|
|
than half --- while matching or exceeding its throughput on aggregation and
|
|
filter workloads. Second, we trace a long-standing failure of BOLT on AArch64
|
|
to an unexpected root cause: default-enabled XRay instrumentation emits composed
|
|
\texttt{R\_AARCH64\_PREL64} relocations that the AArch64 path of \texttt{llvm-bolt}
|
|
cannot lower, manifesting as a spurious ``signature is not valid'' on every
|
|
ephemeral runner. Disabling XRay --- correct for a release binary regardless ---
|
|
both unblocks BOLT and removes $\sim\!45$\,MB of sled sections, shrinking the
|
|
binary from $945$\,MB to $648$\,MB. Third, we replace the gRPC/ArrowFlight stack
|
|
with ZAP, a Cap'n-Proto-derived transport, and show that carrying native
|
|
columnar blocks as opaque ZAP payloads with promise pipelining flattens
|
|
request latency in chain depth, a $63\times$ improvement at depth~64 over a
|
|
request-response baseline. Resident memory is flat from $10$ to $10{,}000$
|
|
concurrent connections. We close with the implications for hosting Datastore as
|
|
a multi-tenant, provisionable offering on a native PaaS.
|
|
\end{abstract}
|
|
|
|
\input{sections/01-introduction}
|
|
\input{sections/02-architecture}
|
|
\input{sections/03-build-optimization}
|
|
\input{sections/04-xray-bolt}
|
|
\input{sections/05-evaluation}
|
|
\input{sections/07-blockchain-native}
|
|
\input{sections/08-two-engine}
|
|
\input{sections/09-economics}
|
|
\input{sections/06-conclusion}
|
|
|
|
\end{document}
|