mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-29 07:34:31 +00:00
6f7e42d1a4
* paper: Datastore — PGO/ThinLTO/BOLT + ZAP optimization of a columnar engine Covers the ClickHouse fork's memory/binary optimization (49-50MB RSS vs official 116MB, binary 945->648MB), the XRay root-cause for AArch64 BOLT failure (composed R_AARCH64_PREL64 in xray_instr_map), gRPC->ZAP transport with promise pipelining (63x on dependent chains), flat memory to 10k connections, and the multi-tenant/Apache-2.0 resale posture. Pattern-A layout, builds via repo CI. * paper: add blockchain-native section (S-Chain storage, Raft→Quasar PQ, D-Chain VM, TEE) Storage/compute separation PoC (parts on SeaweedFS, verified), leaderless PQ consensus (Quasar/ML-DSA, base-ha precedent), data-system-as-a-chain pattern (graphvm+cevm), and confidential-compute permissionless model. * paper: two-engine decentralized platform + reframe (extends S-Chain, not a new chain) - 08-two-engine: Datastore OLAP + Decentralized SQL (SQLite/K-Chain KEM E2E, Z-Chain FHE, CRDTs) over S-Chain; OLAP/OLTP split along the trust axis. - 07: proven elastic scale-out + zero-copy (2 nodes, 1 copy); ZAP-native transport scorecard (gRPC dead, interserver-HTTP->ZAP, ZK->Quasar); reframed datastore as a co-located service EXTENDING S-Chain (D-Chain is the DEX), not its own public chain. * paper: S-Chain storage economics (metered + Arweave-style endowment + DEX-routed multi-currency public-goods market) - 09-economics: two demand curves (metered ops + perpetual endowment w/ solvency math), bonded provider market (stake/proof/payout/slash), multi-currency fees routed through the DEX (D-Chain) -> swapped to LUX, bootstrap inflation tapering to fees. Permissionless public good, no central leader. - 07: reframed datastore as inheriting S-Chain's permissionless public posture (not operator-run/centralized). * paper: F-Chain FHE (luxfi/fhe, real) + Z-Chain ZK rollups + E2E post-quantum (P3Q/ML-DSA/KEM) --------- Co-authored-by: Hanzo Dev <[email protected]>
68 lines
2.7 KiB
TeX
68 lines
2.7 KiB
TeX
\documentclass[11pt]{article}
|
|
\usepackage{../shared/hanzocover}
|
|
\usepackage{amsmath, amssymb, amsthm}
|
|
\usepackage{mathtools}
|
|
\usepackage{graphicx}
|
|
\usepackage{booktabs}
|
|
\usepackage{hyperref}
|
|
\usepackage{enumitem}
|
|
\usepackage{xcolor}
|
|
\usepackage{float}
|
|
\usepackage{multirow}
|
|
\usepackage{listings}
|
|
\input{../shared/lstlang}
|
|
\usepackage[numbers]{natbib}
|
|
\hypersetup{colorlinks=true,linkcolor=black,citecolor=blue,urlcolor=blue}
|
|
|
|
\newtheorem{theorem}{Theorem}
|
|
\newtheorem{lemma}{Lemma}
|
|
\newtheorem{definition}{Definition}
|
|
\newtheorem{proposition}{Proposition}
|
|
|
|
\title{Datastore: Profile-Guided, Post-Link, and Transport\\ Optimization of a Columnar Analytics Engine}
|
|
\author{
|
|
Zach Kelling\thanks{Corresponding author: [email protected]} \\
|
|
\textit{Hanzo Industries Inc (Techstars '17)} \\
|
|
Los Angeles, California \\
|
|
\texttt{https://hanzo.ai}
|
|
}
|
|
\date{June 2026}
|
|
|
|
\begin{document}
|
|
\hanzocoverpage
|
|
|
|
\begin{abstract}
|
|
We present \emph{Datastore}, Hanzo's analytics database: a fork of the
|
|
ClickHouse columnar engine (Apache~2.0) re-engineered for a leaner resident
|
|
footprint, a smaller binary, and a thinner transport surface. We report three
|
|
results. First, applying profile-guided optimization (PGO), ThinLTO, and the
|
|
BOLT post-link optimizer yields a server whose steady-state resident set is
|
|
$49$--$53$\,MB versus $\sim\!116$\,MB for the upstream official binary --- less
|
|
than half --- while matching or exceeding its throughput on aggregation and
|
|
filter workloads. Second, we trace a long-standing failure of BOLT on AArch64
|
|
to an unexpected root cause: default-enabled XRay instrumentation emits composed
|
|
\texttt{R\_AARCH64\_PREL64} relocations that the AArch64 path of \texttt{llvm-bolt}
|
|
cannot lower, manifesting as a spurious ``signature is not valid'' on every
|
|
ephemeral runner. Disabling XRay --- correct for a release binary regardless ---
|
|
both unblocks BOLT and removes $\sim\!45$\,MB of sled sections, shrinking the
|
|
binary from $945$\,MB to $648$\,MB. Third, we replace the gRPC/ArrowFlight stack
|
|
with ZAP, a Cap'n-Proto-derived transport, and show that carrying native
|
|
columnar blocks as opaque ZAP payloads with promise pipelining flattens
|
|
request latency in chain depth, a $63\times$ improvement at depth~64 over a
|
|
request-response baseline. Resident memory is flat from $10$ to $10{,}000$
|
|
concurrent connections. We close with the implications for hosting Datastore as
|
|
a multi-tenant, provisionable offering on a native PaaS.
|
|
\end{abstract}
|
|
|
|
\input{sections/01-introduction}
|
|
\input{sections/02-architecture}
|
|
\input{sections/03-build-optimization}
|
|
\input{sections/04-xray-bolt}
|
|
\input{sections/05-evaluation}
|
|
\input{sections/07-blockchain-native}
|
|
\input{sections/08-two-engine}
|
|
\input{sections/09-economics}
|
|
\input{sections/06-conclusion}
|
|
|
|
\end{document}
|