mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
paper: Datastore — PGO/ThinLTO/BOLT + ZAP optimization (#5)
* paper: Datastore — PGO/ThinLTO/BOLT + ZAP optimization of a columnar engine Covers the ClickHouse fork's memory/binary optimization (49-50MB RSS vs official 116MB, binary 945->648MB), the XRay root-cause for AArch64 BOLT failure (composed R_AARCH64_PREL64 in xray_instr_map), gRPC->ZAP transport with promise pipelining (63x on dependent chains), flat memory to 10k connections, and the multi-tenant/Apache-2.0 resale posture. Pattern-A layout, builds via repo CI. * paper: add blockchain-native section (S-Chain storage, Raft→Quasar PQ, D-Chain VM, TEE) Storage/compute separation PoC (parts on SeaweedFS, verified), leaderless PQ consensus (Quasar/ML-DSA, base-ha precedent), data-system-as-a-chain pattern (graphvm+cevm), and confidential-compute permissionless model. * paper: two-engine decentralized platform + reframe (extends S-Chain, not a new chain) - 08-two-engine: Datastore OLAP + Decentralized SQL (SQLite/K-Chain KEM E2E, Z-Chain FHE, CRDTs) over S-Chain; OLAP/OLTP split along the trust axis. - 07: proven elastic scale-out + zero-copy (2 nodes, 1 copy); ZAP-native transport scorecard (gRPC dead, interserver-HTTP->ZAP, ZK->Quasar); reframed datastore as a co-located service EXTENDING S-Chain (D-Chain is the DEX), not its own public chain. * paper: S-Chain storage economics (metered + Arweave-style endowment + DEX-routed multi-currency public-goods market) - 09-economics: two demand curves (metered ops + perpetual endowment w/ solvency math), bonded provider market (stake/proof/payout/slash), multi-currency fees routed through the DEX (D-Chain) -> swapped to LUX, bootstrap inflation tapering to fees. Permissionless public good, no central leader. - 07: reframed datastore as inheriting S-Chain's permissionless public posture (not operator-run/centralized). * paper: F-Chain FHE (luxfi/fhe, real) + Z-Chain ZK rollups + E2E post-quantum (P3Q/ML-DSA/KEM) --------- Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
committed by
GitHub
GitHub
co-authored by
Hanzo Dev
Hanzo Dev
parent
1437832a94
commit
6f7e42d1a4
@@ -0,0 +1,67 @@
|
||||
\documentclass[11pt]{article}
|
||||
\usepackage{../shared/hanzocover}
|
||||
\usepackage{amsmath, amssymb, amsthm}
|
||||
\usepackage{mathtools}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{hyperref}
|
||||
\usepackage{enumitem}
|
||||
\usepackage{xcolor}
|
||||
\usepackage{float}
|
||||
\usepackage{multirow}
|
||||
\usepackage{listings}
|
||||
\input{../shared/lstlang}
|
||||
\usepackage[numbers]{natbib}
|
||||
\hypersetup{colorlinks=true,linkcolor=black,citecolor=blue,urlcolor=blue}
|
||||
|
||||
\newtheorem{theorem}{Theorem}
|
||||
\newtheorem{lemma}{Lemma}
|
||||
\newtheorem{definition}{Definition}
|
||||
\newtheorem{proposition}{Proposition}
|
||||
|
||||
\title{Datastore: Profile-Guided, Post-Link, and Transport\\ Optimization of a Columnar Analytics Engine}
|
||||
\author{
|
||||
Zach Kelling\thanks{Corresponding author: [email protected]} \\
|
||||
\textit{Hanzo Industries Inc (Techstars '17)} \\
|
||||
Los Angeles, California \\
|
||||
\texttt{https://hanzo.ai}
|
||||
}
|
||||
\date{June 2026}
|
||||
|
||||
\begin{document}
|
||||
\hanzocoverpage
|
||||
|
||||
\begin{abstract}
|
||||
We present \emph{Datastore}, Hanzo's analytics database: a fork of the
|
||||
ClickHouse columnar engine (Apache~2.0) re-engineered for a leaner resident
|
||||
footprint, a smaller binary, and a thinner transport surface. We report three
|
||||
results. First, applying profile-guided optimization (PGO), ThinLTO, and the
|
||||
BOLT post-link optimizer yields a server whose steady-state resident set is
|
||||
$49$--$53$\,MB versus $\sim\!116$\,MB for the upstream official binary --- less
|
||||
than half --- while matching or exceeding its throughput on aggregation and
|
||||
filter workloads. Second, we trace a long-standing failure of BOLT on AArch64
|
||||
to an unexpected root cause: default-enabled XRay instrumentation emits composed
|
||||
\texttt{R\_AARCH64\_PREL64} relocations that the AArch64 path of \texttt{llvm-bolt}
|
||||
cannot lower, manifesting as a spurious ``signature is not valid'' on every
|
||||
ephemeral runner. Disabling XRay --- correct for a release binary regardless ---
|
||||
both unblocks BOLT and removes $\sim\!45$\,MB of sled sections, shrinking the
|
||||
binary from $945$\,MB to $648$\,MB. Third, we replace the gRPC/ArrowFlight stack
|
||||
with ZAP, a Cap'n-Proto-derived transport, and show that carrying native
|
||||
columnar blocks as opaque ZAP payloads with promise pipelining flattens
|
||||
request latency in chain depth, a $63\times$ improvement at depth~64 over a
|
||||
request-response baseline. Resident memory is flat from $10$ to $10{,}000$
|
||||
concurrent connections. We close with the implications for hosting Datastore as
|
||||
a multi-tenant, provisionable offering on a native PaaS.
|
||||
\end{abstract}
|
||||
|
||||
\input{sections/01-introduction}
|
||||
\input{sections/02-architecture}
|
||||
\input{sections/03-build-optimization}
|
||||
\input{sections/04-xray-bolt}
|
||||
\input{sections/05-evaluation}
|
||||
\input{sections/07-blockchain-native}
|
||||
\input{sections/08-two-engine}
|
||||
\input{sections/09-economics}
|
||||
\input{sections/06-conclusion}
|
||||
|
||||
\end{document}
|
||||
@@ -0,0 +1,55 @@
|
||||
\section{Introduction}
|
||||
|
||||
Columnar analytics engines such as ClickHouse~\cite{clickhouse} deliver
|
||||
high scan and aggregation throughput, but a production deployment is
|
||||
dominated by two operational costs that the literature on query execution
|
||||
rarely addresses: the resident memory of an idle-to-moderately-loaded server,
|
||||
and the size and start-up cost of the binary itself. In a multi-tenant hosting
|
||||
setting --- where the operator runs thousands of small-to-medium instances
|
||||
rather than a handful of large ones --- these constants, not asymptotic query
|
||||
complexity, determine density and therefore unit economics.
|
||||
|
||||
\emph{Datastore} is Hanzo's analytics database. It is a fork of ClickHouse,
|
||||
which is distributed under the permissive Apache~2.0 license, re-branded and
|
||||
re-engineered with three goals: (i) a substantially smaller resident footprint
|
||||
and binary at equal or better query throughput; (ii) a transport layer free of
|
||||
the gRPC/protobuf stack; and (iii) a build that is reproducible and
|
||||
self-hosted end to end. This paper reports the engineering and the
|
||||
measurements behind those goals.
|
||||
|
||||
Our contributions are:
|
||||
|
||||
\begin{itemize}[leftmargin=1.5em]
|
||||
\item A build pipeline combining profile-guided optimization (PGO),
|
||||
ThinLTO, and the BOLT~\cite{bolt} post-link optimizer that produces a
|
||||
server binary with $49$--$53$\,MB resident set --- less than half the
|
||||
$\sim\!116$\,MB of the upstream official binary (which itself ships
|
||||
PGO+LTO+BOLT) --- while matching or exceeding its throughput on
|
||||
aggregation and filter workloads (Section~\ref{sec:eval}).
|
||||
|
||||
\item A root-cause analysis (Section~\ref{sec:xray}) showing that the
|
||||
decade-old difficulty of running BOLT on AArch64 ClickHouse binaries is, in
|
||||
our case, not a deficiency of \texttt{llvm-bolt} per se but a consequence of
|
||||
default-enabled XRay instrumentation. XRay emits composed
|
||||
\texttt{R\_AARCH64\_PREL64} relocations (we observe a single group of
|
||||
$62{,}692$ entries in \texttt{xray\_instr\_map}) whose lowering is
|
||||
unimplemented on the AArch64 path, surfacing downstream as an invalid OAuth
|
||||
signature on the self-hosted runner rather than as a BOLT error. Disabling
|
||||
XRay --- the correct choice for a release binary --- removes
|
||||
$\sim\!45$\,MB of sled sections \emph{and} unblocks BOLT.
|
||||
|
||||
\item A transport substitution (Section~\ref{sec:arch}) that removes gRPC,
|
||||
protobuf, and ArrowFlight and carries ClickHouse-native columnar blocks as
|
||||
opaque payloads over ZAP, a Cap'n-Proto-derived RPC with promise
|
||||
pipelining. On dependent-call chains, end-to-end latency becomes flat in
|
||||
chain depth: a $63\times$ improvement at depth~64 over a request-response
|
||||
baseline at $0.5$\,ms round-trip.
|
||||
\end{itemize}
|
||||
|
||||
A guiding constraint throughout is that Datastore must remain wire- and
|
||||
SQL-compatible with ClickHouse, so that the migration of an existing
|
||||
deployment is a change of image reference rather than a change of client.
|
||||
Section~\ref{sec:eval} confirms apples-to-apples parity: built with identical
|
||||
flags, the fork and upstream are statistically indistinguishable in memory and
|
||||
throughput, isolating every reported gain to the optimization pipeline rather
|
||||
than to the fork.
|
||||
@@ -0,0 +1,69 @@
|
||||
\section{Architecture}
|
||||
\label{sec:arch}
|
||||
|
||||
Datastore retains the ClickHouse execution core --- the vectorized,
|
||||
block-at-a-time query pipeline over \texttt{MergeTree} storage --- and changes
|
||||
three surfaces: branding, transport, and the build. We summarize the first two
|
||||
here; the build is the subject of Sections~\ref{sec:build} and~\ref{sec:xray}.
|
||||
|
||||
\subsection{Compatibility-preserving rebrand}
|
||||
|
||||
The fork renames the product across symbols, strings, on-disk paths, wire
|
||||
headers, and binary names (\texttt{datastore-server}, \texttt{datastore-client},
|
||||
\texttt{datastore-local}) while preserving the \texttt{DB::} C++ namespace and
|
||||
every externally observable protocol. The native TCP protocol (port~9000), the
|
||||
HTTP interface (port~8123), and the SQL dialect are byte-for-byte compatible
|
||||
with upstream, so an existing client or driver connects unchanged and a
|
||||
deployment migrates by changing an image reference. This compatibility is a
|
||||
hard requirement: it is what makes the optimization work in later sections
|
||||
\emph{drop-in} rather than a migration project.
|
||||
|
||||
\subsection{Removing gRPC}
|
||||
|
||||
Upstream exposes an optional gRPC interface and an Arrow Flight endpoint, both
|
||||
of which pull in gRPC, protobuf, and the associated code generation and
|
||||
re-encoding machinery. Datastore removes them entirely: the \texttt{contrib/grpc}
|
||||
submodule, \texttt{GRPCServer}, the \texttt{ServerType::GRPC} adapter, the
|
||||
\texttt{USE\_GRPC} paths, and the Flight server (which is itself gRPC-based and
|
||||
was gated behind \texttt{ENABLE\_ARROW\_FLIGHT}). The motivation is twofold:
|
||||
gRPC contributes materially to binary size and dependency surface, and its
|
||||
request-response model is a poor fit for the dependent-call patterns analytics
|
||||
clients actually issue. After removal, \texttt{grpc::} symbols are absent from
|
||||
the binary.
|
||||
|
||||
\subsection{ZAP transport and the opaque-block decision}
|
||||
|
||||
In place of gRPC we introduce \emph{ZAP}, an RPC derived from Cap'n
|
||||
Proto~\cite{capnproto}. ZAP provides two properties we want: zero-copy access to
|
||||
wire-resident structures, and \emph{promise pipelining} --- the ability to
|
||||
issue a call against the (not-yet-returned) result of a previous call, so that
|
||||
a dependent chain of $n$ requests costs one network round trip rather than $n$.
|
||||
|
||||
A naive ZAP integration would re-encode each result row as a ZAP message. We
|
||||
deliberately do not do this. ClickHouse's native format is already a compact,
|
||||
typed, columnar \emph{block}; re-encoding it as a richly-typed ZAP structure
|
||||
would (i) inflate memory, since zero-copy messages are materialized in an arena
|
||||
whose size tracks the largest in-flight result, and (ii) risk the Cap'n Proto
|
||||
traversal-limit and amplification guards on wide results. Instead, ZAP carries
|
||||
the native columnar block as an \emph{opaque, streamed payload}: the transport
|
||||
provides framing, flow control, and pipelining, while the block itself remains
|
||||
the engine's own bounded, streamable representation. This keeps the memory
|
||||
behavior of a query identical to the native protocol (Section~\ref{sec:eval}
|
||||
shows resident memory flat to $10{,}000$ connections) while exposing
|
||||
pipelining to clients that want it.
|
||||
|
||||
We are explicit about scope: at the time of writing, ZAP is wired as a
|
||||
transport and the pipelining benefit is measured at the transport layer
|
||||
(Section~\ref{sec:eval}), but the full SQL-over-ZAP service is not yet the
|
||||
primary query path --- the native TCP handler remains primary. ZAP-native query
|
||||
execution and a ZAP-native coordination layer are future work.
|
||||
|
||||
\subsection{Coordination}
|
||||
|
||||
Replication and distributed DDL are coordinated by the embedded Keeper, a
|
||||
ZooKeeper-API-compatible service built on the NuRaft Raft library. This is a
|
||||
crash-fault-tolerant, strongly-consistent (CP) coordinator for trusted nodes;
|
||||
it is distinct from, and should not be conflated with, the Byzantine,
|
||||
DAG-native consensus families used in blockchain settings. Keeper is not on the
|
||||
data-scan path --- it manages metadata, replica state, and merge/mutation
|
||||
assignment --- and it retains NuRaft's own transport.
|
||||
@@ -0,0 +1,57 @@
|
||||
\section{Build Optimization}
|
||||
\label{sec:build}
|
||||
|
||||
Datastore's binary is built with three composable techniques: profile-guided
|
||||
optimization (PGO), ThinLTO, and the BOLT post-link optimizer. We describe the
|
||||
pipeline and the constraints that order it; Section~\ref{sec:eval} reports the
|
||||
effect of each stage.
|
||||
|
||||
\subsection{Pipeline}
|
||||
|
||||
The canonical optimized build (\texttt{utils/pgo-build/build-optimized.sh})
|
||||
proceeds in three phases:
|
||||
|
||||
\begin{enumerate}[leftmargin=1.5em]
|
||||
\item \textbf{Instrumented build.} Compile with clang IR instrumentation
|
||||
(\texttt{-fprofile-generate}). Because IR-level PGO instrumentation is
|
||||
incompatible with ThinLTO, this phase runs with LTO disabled.
|
||||
\item \textbf{Profile collection.} Run a representative OLAP workload ---
|
||||
inserts into a \texttt{MergeTree} table followed by aggregation, top-$k$,
|
||||
filter, time-bucketing, join, window, array, and sub-query reads --- against
|
||||
the instrumented binary, then merge the raw counters with
|
||||
\texttt{llvm-profdata}.
|
||||
\item \textbf{Optimized build.} Recompile with \texttt{-fprofile-use} and
|
||||
ThinLTO enabled. Functions whose control-flow hash no longer matches the
|
||||
recorded profile (expected, since instrumentation ran without LTO) are
|
||||
demoted from \texttt{-Werror} to a warning so the build does not abort on
|
||||
benign profile staleness.
|
||||
\end{enumerate}
|
||||
|
||||
The resulting binary is optionally passed to BOLT, which re-lays-out hot code
|
||||
using a branch profile, separating hot and cold basic blocks and reordering
|
||||
functions for instruction-cache locality. BOLT requires the linker to emit
|
||||
relocations (\texttt{-Wl,--emit-relocs}), which we gate behind a build flag.
|
||||
|
||||
\subsection{A subtle thread-pool deadlock in instrumented collection}
|
||||
|
||||
A practical obstacle: running the \emph{instrumented} binary against a
|
||||
large insert deadlocked. The \texttt{-fprofile-generate} runtime serializes
|
||||
access to per-function counters, and this serialization interacts badly with
|
||||
the engine's background merge thread pools under a high-volume
|
||||
\texttt{MergeTree} insert. We resolve this by collecting the profile with
|
||||
bounded threads (\texttt{max\_insert\_threads=1},
|
||||
\texttt{background\_pool\_size=1}) over a smaller but representative row count.
|
||||
PGO requires that the profile cover the hot \emph{code paths}, not a
|
||||
production-scale data volume, so this does not degrade profile quality; it
|
||||
completes in seconds rather than deadlocking.
|
||||
|
||||
\subsection{Profile representativeness and matched comparison}
|
||||
|
||||
A PGO build is only as good as its profile. For a fair comparison we hold the
|
||||
profile fixed across the fork and upstream, and --- crucially --- we regenerate
|
||||
the profile against the \emph{same} build configuration the binary will ship
|
||||
with. An early result used a profile collected from an XRay-instrumented build
|
||||
to optimize a non-XRay binary; the control-flow-hash mismatch on hot functions
|
||||
silently discarded counts for exactly the aggregation kernels we cared about,
|
||||
understating the aggregation result. The shipped recipe collects the profile
|
||||
from the final, XRay-free configuration, eliminating the mismatch.
|
||||
@@ -0,0 +1,68 @@
|
||||
\section{The XRay--BOLT Interaction on AArch64}
|
||||
\label{sec:xray}
|
||||
|
||||
Running BOLT on an AArch64 ClickHouse-derived binary has a reputation for
|
||||
fragility. We present a root-cause analysis that reframes the problem: the
|
||||
failure we observed is not, at bottom, a defect of \texttt{llvm-bolt}, but a
|
||||
consequence of an instrumentation feature that a release binary should not
|
||||
carry in the first place.
|
||||
|
||||
\subsection{Symptom}
|
||||
|
||||
On AArch64, BOLT-processed binaries failed at run time. In our deployment the
|
||||
failure surfaced indirectly: ephemeral CI runners launched from such binaries
|
||||
reported \texttt{VssOAuthTokenRequestException: The signature is not valid}
|
||||
during the broker handshake and never came online --- a misleading symptom
|
||||
several layers removed from the true cause. Direct BOLT invocation, when forced,
|
||||
failed earlier with \texttt{error: expected relocatable expression} during
|
||||
emission, and an assertion build of \texttt{llvm-bolt} (v22.1.1) pinpointed the
|
||||
failure to
|
||||
\begin{center}
|
||||
\texttt{Relocation::getComposeOpcodeFor():}\\
|
||||
\texttt{assert(Arch == Triple::riscv64 \&\& "only implemented for RISC-V")}
|
||||
\end{center}
|
||||
|
||||
\subsection{Root cause}
|
||||
|
||||
The function that BOLT uses to \emph{compose} paired relocations into a single
|
||||
MC expression is implemented only for RISC-V. On AArch64 it is reached by
|
||||
composed \texttt{R\_AARCH64\_PREL64} relocations --- PC-relative 64-bit data
|
||||
references of the form $S + A - P$. Instrumenting the binary revealed that the
|
||||
overwhelming majority of these composed groups, including one pathological group
|
||||
of \textbf{$62{,}692$ relocations sharing a single offset}, lived in the
|
||||
\texttt{xray\_instr\_map} section. In other words, the relocations BOLT could
|
||||
not lower were emitted by \textbf{XRay instrumentation}, which was enabled by
|
||||
default.
|
||||
|
||||
XRay~\cite{xray} is a function-call tracing facility: it inserts entry/exit
|
||||
``sleds'' at every function and a side table (\texttt{xray\_instr\_map},
|
||||
\texttt{xray\_fn\_idx}) of PC-relative pointers into them. We measured these
|
||||
sections at $\sim\!45$\,MB in the datastore binary. A release database server
|
||||
has no business carrying always-on tracing sleds; the feature is appropriate
|
||||
for a profiling build, opt-in, and nowhere else.
|
||||
|
||||
\subsection{Fix}
|
||||
|
||||
The defect upstream was a default: \texttt{ENABLE\_XRAY} was \texttt{ON}.
|
||||
Turning it off (with the coverage build that legitimately needs XRay
|
||||
re-enabling it locally) has three effects:
|
||||
|
||||
\begin{enumerate}[leftmargin=1.5em]
|
||||
\item The \texttt{xray\_instr\_map} section disappears, taking the
|
||||
unlowerable composed \texttt{R\_AARCH64\_PREL64} relocations with it. BOLT
|
||||
then emits cleanly on AArch64 and the resulting binary runs correctly.
|
||||
\item The binary shrinks from $945$\,MB to $648$\,MB after BOLT --- a $31\%$
|
||||
reduction --- before any other consideration.
|
||||
\item Per-function sled overhead is removed from the hot path.
|
||||
\end{enumerate}
|
||||
|
||||
A residual, genuinely-upstreamable robustness fix remains --- teaching
|
||||
\texttt{getComposeOpcodeFor} the AArch64 cases so that \emph{any} future
|
||||
composed \texttt{PREL64} (not only XRay's) lowers correctly --- but it is not
|
||||
required for Datastore, because Datastore should not emit those relocations at
|
||||
all. Two operational notes for reproducing BOLT on the large AArch64 binary:
|
||||
\texttt{llvm-bolt} recurses deeply during veneer insertion and overflows the
|
||||
default $8$\,MB stack, so it must run under a raised stack limit
|
||||
(\texttt{ulimit -s 2{,}097{,}152}); and the GB10 firmware we tested does not
|
||||
expose the SPE table, so branch profiles are collected with cycle sampling and
|
||||
\texttt{perf2bolt --nl} rather than via LBR/SPE.
|
||||
@@ -0,0 +1,93 @@
|
||||
\section{Evaluation}
|
||||
\label{sec:eval}
|
||||
|
||||
All measurements are on a single Grace-class AArch64 host (Cortex-X925,
|
||||
$20$ cores, $121$\,GB RAM, Linux). Each server is driven over its native
|
||||
protocol against a $10$M-row \texttt{MergeTree} table
|
||||
(\texttt{id UInt64, user\_id UInt32, ts DateTime, value Float64, category
|
||||
LowCardinality(String)}). We report steady-state resident set (RSS, from
|
||||
\texttt{/proc/<pid>/status}) and queries-per-second. ``Official'' is the
|
||||
upstream binary, which ships PGO+LTO+BOLT.
|
||||
|
||||
\subsection{The fork costs nothing}
|
||||
|
||||
Built with identical flags ($-O3$, no PGO/LTO), Datastore and upstream are
|
||||
statistically indistinguishable: RSS within $1$\,MB ($143$ vs $144$\,MB) and
|
||||
throughput within noise across aggregation, top-$k$, and filter at concurrencies
|
||||
$1$ through $64$. This establishes that every subsequent gain is attributable to
|
||||
the optimization pipeline, not to the fork.
|
||||
|
||||
\subsection{Effect of each optimization stage}
|
||||
|
||||
Table~\ref{tab:stages} reports resident set and binary size as the pipeline is
|
||||
applied. ThinLTO alone already brings the resident set below the official
|
||||
binary's; PGO halves it again via hot/cold splitting; removing XRay and applying
|
||||
BOLT delivers the final size and a further locality gain.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\small
|
||||
\begin{tabular}{@{}lrr@{}}
|
||||
\toprule
|
||||
Configuration & RSS & Binary \\
|
||||
\midrule
|
||||
Datastore $-O3$ & $143$\,MB & $709$\,MB \\
|
||||
\quad + ThinLTO & $113$\,MB & $684$\,MB \\
|
||||
\quad + PGO & $53$\,MB & --- \\
|
||||
\quad + no-XRay, + BOLT & $\mathbf{49}$--$\mathbf{50}$\,MB & $\mathbf{648}$\,MB \\
|
||||
\midrule
|
||||
Official (PGO+LTO+BOLT) & $116$\,MB & --- \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Resident set and binary size by optimization stage. ThinLTO alone
|
||||
($113$\,MB) already beats the official binary's $116$\,MB; the full pipeline
|
||||
reaches $49$--$50$\,MB, $\sim\!57\%$ below official.}
|
||||
\label{tab:stages}
|
||||
\end{table}
|
||||
|
||||
\subsection{Throughput at parity memory}
|
||||
|
||||
At equal correctness the optimized Datastore matches or beats upstream on the
|
||||
read mix. On the stable mid-concurrency point ($c{=}4$, $10$M rows) the
|
||||
no-XRay PGO+LTO+BOLT build leads upstream on filters by $25$--$40\%$
|
||||
($397$ vs $284$ QPS), on top-$k$ by $10$--$30\%$, and tracks or exceeds it on
|
||||
aggregation, while using less than half the resident memory. We caution that
|
||||
single-shot QPS on a shared host carries contention noise; the memory result,
|
||||
by contrast, is invariant to load and reproduces on every run.
|
||||
|
||||
\subsection{Memory is flat in connection count}
|
||||
|
||||
To isolate per-connection cost we issue a trivial query across $10$ to
|
||||
$10{,}000$ concurrent clients (Table~\ref{tab:scale}). Resident set is flat:
|
||||
$10{,}000$ connections cost the same memory as $10$. This is the bounded-state
|
||||
connection model --- the server pools threads and holds negligible per-connection
|
||||
state --- and Datastore is $\sim\!15$\,MB leaner than upstream at every point.
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\small
|
||||
\begin{tabular}{@{}lrr@{}}
|
||||
\toprule
|
||||
Concurrent clients & Datastore RSS & Official RSS \\
|
||||
\midrule
|
||||
$10$ & $114$\,MB & $129$\,MB \\
|
||||
$100$ & $113$\,MB & $129$\,MB \\
|
||||
$1{,}000$ & $113$\,MB & $128$\,MB \\
|
||||
$10{,}000$& $\mathbf{113}$\,MB & $128$\,MB \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Resident set vs.\ concurrent connection count. Per-connection memory is
|
||||
effectively zero; $10{,}000$ clients cost the same as $10$. (Measured on an
|
||||
LTO build before the PGO/BOLT stages of Table~\ref{tab:stages}.)}
|
||||
\label{tab:scale}
|
||||
\end{table}
|
||||
|
||||
\subsection{ZAP pipelining flattens dependent-chain latency}
|
||||
|
||||
We measure end-to-end latency of a dependent chain of $n$ calls --- each input
|
||||
the previous output --- at a $0.5$\,ms round trip, comparing ZAP promise
|
||||
pipelining against a request-response baseline. The baseline grows linearly in
|
||||
$n$; ZAP is flat, since the whole chain resolves in one logical round trip. At
|
||||
depth $64$ this is a $63\times$ reduction in latency. This is the property that
|
||||
motivates ZAP for analytics clients that issue dependent queries (build a
|
||||
filtered view, then aggregate it, then top-$k$ it) rather than independent ones.
|
||||
@@ -0,0 +1,76 @@
|
||||
\section{Deployment and Multi-Tenancy}
|
||||
|
||||
Datastore is built to be hosted as a provisionable, multi-tenant offering on a
|
||||
native PaaS. Its compatibility surface makes the two analytics products Hanzo
|
||||
operates --- a privacy-preserving web-analytics service and a product-analytics
|
||||
platform --- migrate by configuration alone: both already isolate tenants by a
|
||||
column key (\texttt{organization\_id} and \texttt{team\_id} respectively) over a
|
||||
shared database and schema, so pointing them at a Datastore instance is a change
|
||||
of host, not of data model. The leaner resident footprint compounds directly in
|
||||
this setting: at $\sim\!50$\,MB per instance instead of $\sim\!116$\,MB, an
|
||||
operator more than doubles the number of tenant instances per node at fixed
|
||||
memory, which is the dimension that governs the unit economics of a managed
|
||||
service.
|
||||
|
||||
Because the engine derives from ClickHouse under the permissive Apache~2.0
|
||||
license, Datastore is freely re-brandable, hostable, and resellable --- as are
|
||||
the adjacent Hanzo data products on equally permissive upstreams (PostgreSQL,
|
||||
pgvector, SQLite, PocketBase, and the Valkey-derived key-value store). This
|
||||
licensing posture is what makes a commercial managed-database offering built on
|
||||
these components viable without copyleft entanglement.
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
We have shown that the operationally dominant costs of a columnar analytics
|
||||
server --- resident memory and binary size --- yield substantially to a
|
||||
disciplined build pipeline. PGO, ThinLTO, and BOLT together produce a Datastore
|
||||
server at less than half the resident footprint of the upstream official binary
|
||||
with equal or better read throughput, and memory that is flat from $10$ to
|
||||
$10{,}000$ connections. Along the way we identified that default-enabled XRay
|
||||
instrumentation was both bloating the binary by $\sim\!45$\,MB and, through
|
||||
unlowerable composed \texttt{R\_AARCH64\_PREL64} relocations, silently blocking
|
||||
BOLT on AArch64 --- a fix that is one disabled default. Finally, replacing gRPC
|
||||
with a Cap'n-Proto-derived transport that carries native columnar blocks as
|
||||
opaque payloads exposes promise pipelining to clients, flattening dependent-chain
|
||||
latency by up to $63\times$ without disturbing the engine's bounded memory
|
||||
behavior.
|
||||
|
||||
The work points to two clear next steps: promoting ZAP from a transport to the
|
||||
primary, ZAP-native query path, and extending the same posture to the
|
||||
coordination layer. Both are constrained by the same principle that guided this
|
||||
work --- preserve compatibility, pay only for what the workload uses, and let the
|
||||
measurements, not the defaults, decide what ships.
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
|
||||
\bibitem{clickhouse}
|
||||
ClickHouse, Inc.,
|
||||
\newblock \textit{ClickHouse: Open-Source Column-Oriented DBMS for Real-Time
|
||||
Analytics}.
|
||||
\newblock \texttt{https://clickhouse.com}, Apache License 2.0.
|
||||
|
||||
\bibitem{bolt}
|
||||
M.~Panchenko, R.~Auler, B.~Nell, and G.~Ottoni,
|
||||
\newblock ``BOLT: A Practical Binary Optimizer for Data Centers and Beyond,''
|
||||
\newblock \textit{CGO}, 2019.
|
||||
|
||||
\bibitem{xray}
|
||||
D.~M.~Berris, A.~Veitch, N.~Lewycky, et~al.,
|
||||
\newblock ``XRay: A Function Call Tracing System,''
|
||||
\newblock Google Technical Report, 2016.
|
||||
|
||||
\bibitem{capnproto}
|
||||
K.~Varda,
|
||||
\newblock \textit{Cap'n Proto: Cerealization Protocol}.
|
||||
\newblock \texttt{https://capnproto.org}.
|
||||
|
||||
\bibitem{pgo}
|
||||
\newblock ``Profile Guided Optimization,'' LLVM/Clang Documentation.
|
||||
\newblock \texttt{https://clang.llvm.org/docs/UsersManual.html\#profile-guided-optimization}.
|
||||
|
||||
\bibitem{thinlto}
|
||||
T.~Johnson, M.~Amini, and X.~D.~Li,
|
||||
\newblock ``ThinLTO: Scalable and Incremental LTO,''
|
||||
\newblock \textit{CGO}, 2017.
|
||||
|
||||
\end{thebibliography}
|
||||
@@ -0,0 +1,161 @@
|
||||
\section{Toward a Blockchain-Native Datastore}
|
||||
\label{sec:chain}
|
||||
|
||||
The optimizations of Sections~\ref{sec:build}--\ref{sec:eval} make Datastore a
|
||||
better single-node engine. This section sketches a more ambitious trajectory:
|
||||
operating Datastore as a self-hosted service that \emph{extends S-Chain} --- the
|
||||
network's decentralized object-storage substrate --- co-located on the very same
|
||||
nodes that serve S-Chain, with storage and compute separated over that shared
|
||||
substrate. We are deliberate about what this is \emph{not}: Datastore is not a
|
||||
new public chain with its own letter and validator economy (the network's chain
|
||||
letters are otherwise assigned, e.g.\ the DEX). It is a query/compute engine
|
||||
layered on S-Chain, using S-Chain for durable storage and the nodes' existing
|
||||
Quasar consensus for coordination. Crucially, that does not make it centralized:
|
||||
S-Chain is itself permissionless public-goods infrastructure
|
||||
(Section~\ref{sec:economics}), run by an open, bonded node set with no central
|
||||
leader, so Datastore inherits that posture --- it runs wherever S-Chain runs,
|
||||
operated by whoever runs the nodes (including, e.g., LLM-consensus quorums),
|
||||
rather than by us. Not its own chain, but not centralized either. We report a
|
||||
working proof-of-concept for the foundational layer and the validated
|
||||
integration path for the rest.
|
||||
|
||||
\subsection{Separation of storage and compute}
|
||||
|
||||
The dominant cloud-analytics architecture (Snowflake, Databricks, ClickHouse
|
||||
Cloud) separates a durable object-storage tier from stateless, elastic compute.
|
||||
ClickHouse's \texttt{ReplicatedMergeTree} already factors cleanly along this
|
||||
line: a small ordered \emph{replication log} and large, content-addressed
|
||||
\emph{data parts}. We exploit this by pointing Datastore's MergeTree storage
|
||||
policy at an S3-compatible object store --- here SeaweedFS, the storage engine
|
||||
behind the network's ``S-Chain.''
|
||||
|
||||
\paragraph{Proof of concept.} We loaded a $5$M-row MergeTree table whose
|
||||
storage policy targets SeaweedFS over S3. Verified outcome: all five parts
|
||||
report disk \texttt{seaweed}; the local data directory holds $24$\,KB of
|
||||
metadata; SeaweedFS holds $101$ content-addressed objects; and queries execute
|
||||
correctly against the object-backed table (aggregation $25$\,ms, filter
|
||||
$7$\,ms over $5$M rows). Resident memory was $68$\,MB versus $130$\,MB for an
|
||||
upstream binary on local disk.
|
||||
|
||||
\paragraph{Honest cost.} Object storage loses a warm-cache drag race to local
|
||||
disk by $2$--$4\times$ on single queries --- a round trip versus a page-cache
|
||||
hit. The win is not raw warm latency; it is (i) a stateless compute tier
|
||||
($24$\,KB local vs.\ $99$\,MB), enabling $N$ disposable compute nodes over one
|
||||
copy of data --- which a local-disk engine structurally cannot do --- and (ii)
|
||||
roughly half the resident memory. A local filesystem cache over the object disk
|
||||
(the ClickHouse-Cloud technique) recovers most of the warm-latency gap.
|
||||
|
||||
\paragraph{Elastic scale-out, verified.} The stateless-compute claim is the one
|
||||
a local-disk engine cannot make, so we verified it directly. With a
|
||||
\texttt{ReplicatedMergeTree} over the S-Chain object store and zero-copy
|
||||
replication enabled, we ran two independent compute nodes coordinated by Keeper:
|
||||
node~1 inserted $5$M rows; node~2, started afterward, served the identical
|
||||
$5{,}000{,}000$ rows. Critically, the object count in S-Chain rose from $101$ to
|
||||
$102$ --- a single manifest object, \emph{not} a second copy of the data ---
|
||||
confirming that the two replicas share one physical copy via zero-copy. Two
|
||||
compute nodes, one copy of data on decentralized storage: the elastic-compute
|
||||
property, demonstrated. (The single non-obvious requirement was configuring the
|
||||
inter-node part-transfer endpoint; see Section~\ref{sec:zapnative}.)
|
||||
|
||||
\subsection{Replacing Raft with leaderless post-quantum consensus}
|
||||
|
||||
Coordination of the replication log is performed today by an embedded
|
||||
Keeper on NuRaft (Raft): crash-fault-tolerant, leader-based, classical
|
||||
cryptography. Replacing the Raft backend with the network's leaderless
|
||||
consensus (Quasar) yields three properties at once: leaderless resilience (no
|
||||
single writer to fail over), Byzantine fault tolerance over a bonded validator
|
||||
set, and --- because Quasar finalizes with ML-DSA --- \emph{post-quantum
|
||||
finality}. This is not speculative: a sibling system already runs SQLite under
|
||||
the same leaderless Quasar coordination (writer-pin, all nodes equal
|
||||
validators), establishing the pattern. The realistic integration is a
|
||||
coordination service that speaks the ZooKeeper API Datastore already expects but
|
||||
backs the ordered log with Quasar, so the engine itself is unchanged. The result
|
||||
is, to our knowledge, the first \emph{post-quantum-native} columnar OLAP engine.
|
||||
|
||||
\subsection{Extending S-Chain, not minting a new chain}
|
||||
|
||||
Co-locating Datastore with S-Chain rather than standing up a separate chain is
|
||||
the simpler and more honest design: bulk columnar data is far too large to pass
|
||||
through consensus (the network caps a gossiped block at $2$\,MB), so a
|
||||
``datastore chain'' would in any case put \emph{only} small commitments on-chain
|
||||
and keep the data off-chain --- which is exactly what running beside S-Chain
|
||||
already gives us, without a new validator set or chain letter. The division of
|
||||
labor is therefore: the columnar \emph{parts} are content-addressed objects in
|
||||
S-Chain storage; the small \emph{commitments} (part-manifest roots, DDL,
|
||||
schema/version pointers) are ordered by the nodes' existing Quasar consensus; and
|
||||
reads are served by the co-located Datastore compute engine. Two on-disk
|
||||
precedents make the engine integration concrete without inventing anything: a
|
||||
sibling component already binds a C++ execution engine behind a thin Go shim via
|
||||
FFI (``valid as long as state roots match''), and another already serves a
|
||||
read-only query surface over a handler while keeping bulk state out of blocks.
|
||||
Datastore reuses both shapes as a \emph{service on S-Chain nodes} --- the C++
|
||||
engine for execution, a query surface for reads --- with ZAP
|
||||
(Section~\ref{sec:arch}) as the wire throughout. Should a workload genuinely want
|
||||
a public, permissionless analytics chain later (an LLM-consensus quorum
|
||||
publishing verifiable aggregates, say), the same commitments-on-chain /
|
||||
parts-off-chain split promotes to a first-class chain unchanged; but that is an
|
||||
option, not the default.
|
||||
|
||||
\subsection{Permissionless operation with confidential compute}
|
||||
|
||||
Open, untrusted query execution is the residual hard problem: a serving node's
|
||||
\texttt{SELECT} result is not verifiable by re-execution for arbitrary SQL, and
|
||||
ZK-SQL is not practical at OLAP scale. The pragmatic resolution is
|
||||
attestation rather than proof: run compute inside a hardware trusted execution
|
||||
environment (e.g.\ confidential-compute GPUs), so a remote party verifies an
|
||||
attestation of the enclave rather than re-deriving the query. Combined with
|
||||
content-addressed parts (integrity) and Quasar-ordered manifests (a verifiable
|
||||
write history), this yields a system that is open to read and use, bonded to
|
||||
write and serve, and private under attestation --- without requiring verifiable
|
||||
arbitrary-query mathematics.
|
||||
|
||||
\subsection{One transport: ZAP-native, end to end}
|
||||
\label{sec:zapnative}
|
||||
|
||||
A conventional analytics deployment carries three distinct wire protocols: a
|
||||
client protocol, a gRPC service surface, an inter-node replication transport
|
||||
(HTTP), and a ZooKeeper wire for coordination. Our objective is to collapse all
|
||||
of these onto one substrate --- ZAP, the Cap'n-Proto-derived transport with
|
||||
zero-copy framing and promise pipelining (Section~\ref{sec:arch}). The current
|
||||
state and the remaining work:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\small
|
||||
\begin{tabular}{@{}lll@{}}
|
||||
\toprule
|
||||
Wire & Today & ZAP-native target \\
|
||||
\midrule
|
||||
gRPC service & \textbf{removed} & --- (gone) \\
|
||||
Inter-node parts & HTTP (DataPartsExch.) & ZAP \\
|
||||
Coordination log & ZooKeeper / Keeper & Quasar (ZAP) \\
|
||||
Client query & native TCP / ZAP stub & SQL-over-ZAP \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Collapsing four wires onto ZAP. gRPC is already gone; the inter-node
|
||||
HTTP part-transfer and the ZooKeeper coordination wire are the remaining
|
||||
targets.}
|
||||
\label{tab:zapnative}
|
||||
\end{table}
|
||||
|
||||
The gRPC surface and ArrowFlight are already deleted: the binary contains no
|
||||
\texttt{GRPCServer} and no \texttt{USE\_GRPC} code paths. The inter-node
|
||||
part-transfer --- the \texttt{InterserverIOHTTPHandler}/\texttt{DataPartsExchange}
|
||||
that the scale-out result above relies on --- runs over HTTP today and is the
|
||||
first migration to ZAP, since part fetch is exactly the kind of large,
|
||||
streamable, pipelineable transfer ZAP's opaque-payload model was designed for.
|
||||
The coordination wire (the $17$-file ZooKeeper client talking to Keeper) is the
|
||||
second: replaced by the Quasar-backed coordination service of
|
||||
Section~\ref{sec:chain}, ``ZooKeeper'' disappears as a wire while the engine's
|
||||
expectations are preserved. When both land, alongside the SQL-over-ZAP query
|
||||
path, the platform is one transport deep --- no HTTP, no gRPC, no ZooKeeper ---
|
||||
which is also precisely the precondition for the unified two-engine platform of
|
||||
Section~\ref{sec:twoengine}.
|
||||
|
||||
\paragraph{Phasing.} The product engine ships today (Sections~\ref{sec:build}--%
|
||||
\ref{sec:eval}); storage/compute separation \emph{and elastic scale-out with
|
||||
zero-copy} over S-Chain are demonstrated above; the Raft$\rightarrow$Quasar
|
||||
coordination swap, the inter-node-HTTP$\rightarrow$ZAP migration, and the chain
|
||||
VM are the near-term build, each with an on-disk precedent; verifiable/attested
|
||||
permissionless reads are the research frontier. Each phase preserves
|
||||
compatibility and pays only for what the workload uses.
|
||||
@@ -0,0 +1,114 @@
|
||||
\section{A Two-Engine Decentralized Data Platform}
|
||||
\label{sec:twoengine}
|
||||
|
||||
The blockchain-native trajectory of Section~\ref{sec:chain} addresses analytics.
|
||||
But analytics is only half of a data platform: the other half is the
|
||||
transactional, per-user, per-project record store. These two workloads have
|
||||
opposite shapes --- columnar scan-and-aggregate versus row-oriented
|
||||
point-read-and-write --- and, crucially, opposite \emph{trust} requirements.
|
||||
Aggregate analytics can run server-side (fast, shared, TEE-attested); a user's
|
||||
own per-project records should be private end-to-end, visible to no operator at
|
||||
all. We therefore propose two complementary engines over one decentralized
|
||||
storage substrate, both native to the Lux network and both speaking ZAP.
|
||||
|
||||
\subsection{Engine 1: Datastore --- scalable analytics (OLAP)}
|
||||
|
||||
Datastore is the columnar engine of this paper: horizontally-scalable,
|
||||
\emph{server-side} compute over the shared S-Chain object store
|
||||
(Section~\ref{sec:chain}), demonstrated to scale out to multiple stateless
|
||||
compute replicas sharing a single zero-copy copy of data
|
||||
(Section~\ref{sec:eval}). Its privacy model is \emph{attestation}: compute runs
|
||||
in a hardware trusted execution environment, so a client verifies an enclave
|
||||
attestation rather than trusting the operator. This is the right model for
|
||||
operational and aggregate analytics, where throughput and elastic scale dominate
|
||||
and the data is the tenant's own aggregate.
|
||||
|
||||
\subsection{Engine 2: Decentralized SQL --- private per-user records (OLTP)}
|
||||
|
||||
The second engine is a SQLite-based transactional store --- one database file
|
||||
per user or per project, the file being the unit of tenancy --- already running
|
||||
under leaderless Quasar coordination in the sibling base-ha system. To this
|
||||
foundation we add three primitives, each backed by an existing Lux chain, to
|
||||
make it a \emph{decentralized, end-to-end-private} SQL engine:
|
||||
|
||||
\begin{description}[leftmargin=1.2em,style=nextline]
|
||||
\item[Client-held keys (K-Chain KEM).] Each user/project is issued a
|
||||
post-quantum key-encapsulation keypair from the K-Chain. The SQLite database
|
||||
--- at page or WAL granularity --- is encrypted under a data key wrapped by
|
||||
the user's KEM public key. The operator stores only ciphertext on S-Chain;
|
||||
\emph{decryption happens only on the client}. No operator, validator, or
|
||||
storage provider ever observes plaintext.
|
||||
\item[Compute on ciphertext (F-Chain FHE, Z-Chain rollups).] To answer
|
||||
queries without decrypting, predicates and aggregates evaluate homomorphically
|
||||
against the encrypted pages. This is not hypothetical: the FHE compute is a
|
||||
shipping lattice-based library (\texttt{luxfi/fhe}, with TFHE bootstrapping)
|
||||
exposed as the F-Chain, and the FHE-CRDT combination --- homomorphic operations
|
||||
that also merge conflict-free --- already exists. Because FHE evaluation is
|
||||
expensive, it is scaled and made publicly verifiable by Z-Chain ZK rollups:
|
||||
many encrypted operations are batched off-chain and a single succinct proof,
|
||||
posted to the rollup, attests that the batch was evaluated correctly over the
|
||||
committed ciphertext. The server thus returns an encrypted result the client
|
||||
alone can open, with a rollup proof that it was computed honestly --- a public,
|
||||
permissionless confidential-compute path that needs no trusted server.
|
||||
\item[Coordinator-free merge (CRDTs).] Multi-device and multi-writer updates
|
||||
reconcile through conflict-free replicated data types, so concurrent writes
|
||||
merge deterministically without a consensus round on the write path. Quasar is
|
||||
used only for the coarse coordination (key rotation, file-version commitments
|
||||
on S-Chain), not for every row write.
|
||||
\end{description}
|
||||
|
||||
The result is ``decentralized SQL'': a per-user record store that is private by
|
||||
construction (E2E encryption), available without a trusted server (content on
|
||||
S-Chain, keys on the client), and writable offline (CRDT merge), with the chain
|
||||
providing only what chains are good at --- ordered commitments, key custody, and
|
||||
verifiable compute.
|
||||
|
||||
\subsection{Why both, and how they compose}
|
||||
|
||||
The two engines are not redundant; they are the OLAP/OLTP split re-cast along the
|
||||
trust axis (Table~\ref{tab:twoengine}). Datastore answers ``what are the
|
||||
aggregate trends across the fleet'' at scale; Decentralized SQL holds ``this
|
||||
user's project data'' privately. They share one substrate and one transport:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\small
|
||||
\begin{tabular}{@{}lll@{}}
|
||||
\toprule
|
||||
& \textbf{Datastore} & \textbf{Decentralized SQL} \\
|
||||
\midrule
|
||||
Shape & columnar OLAP & row OLTP (SQLite) \\
|
||||
Scope & aggregate / fleet & per-user / project \\
|
||||
Compute & server-side (TEE) & client + FHE on server \\
|
||||
Privacy & attested enclave & E2E, keys client-held \\
|
||||
Scale & horizontal compute & per-file, CRDT merge \\
|
||||
Consensus & Quasar (manifests) & Quasar (keys/versions) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Two engines, one platform: the OLAP/OLTP split re-cast along the trust
|
||||
axis. Both ride S-Chain storage, K-Chain keys, and ZAP transport.}
|
||||
\label{tab:twoengine}
|
||||
\end{table}
|
||||
|
||||
\paragraph{The shared stack.} \textbf{S-Chain} (the Hanzo S3 / SeaweedFS fork)
|
||||
is the decentralized storage tier for \emph{both} engines --- Datastore parts and
|
||||
encrypted SQLite files are both content-addressed objects. \textbf{K-Chain}
|
||||
custodies the post-quantum keys (KEM). \textbf{F-Chain} runs the FHE compute
|
||||
(\texttt{luxfi/fhe} / TFHE), and \textbf{Z-Chain} provides the ZK rollups that
|
||||
scale and publicly verify it. \textbf{Quasar} provides post-quantum, leaderless
|
||||
finality for the commitments of both engines. And \textbf{ZAP} --- the
|
||||
Cap'n-Proto-derived transport that already replaced gRPC --- is the single wire
|
||||
for client-to-engine queries, engine-to-S3 part transfer, and
|
||||
engine-to-consensus coordination, so the entire platform is one transport deep
|
||||
rather than the usual stack of HTTP, gRPC, and a ZooKeeper wire.
|
||||
|
||||
\paragraph{Post-quantum end to end.} A property worth stating explicitly because
|
||||
it holds at \emph{every} layer, not one: the consensus finalizes with ML-DSA
|
||||
(Quasar), the per-user data keys are post-quantum KEM (K-Chain), the
|
||||
classical-primitive precompiles are gated behind a post-quantum profile (P3Q),
|
||||
and the confidential compute is lattice-based FHE (F-Chain). There is no
|
||||
classical-cryptography weak link in the path from a client's key to a finalized
|
||||
commitment. The endpoint of the program in this paper is therefore a platform
|
||||
that is, simultaneously, one transport deep (no internal HTTP, no gRPC, no
|
||||
ZooKeeper --- only ZAP and Quasar) and post-quantum native end to end, over
|
||||
decentralized S-Chain storage.
|
||||
@@ -0,0 +1,103 @@
|
||||
\section{The S-Chain Storage Market}
|
||||
\label{sec:economics}
|
||||
|
||||
A decentralized storage substrate needs an economic model, both to price the
|
||||
service to customers and to compensate the operators who physically store and
|
||||
serve bytes. S-Chain needs two pricing models, because it serves two kinds of
|
||||
data with opposite lifetimes, and a provider market underneath both.
|
||||
|
||||
\subsection{Two demand curves: metered and perpetual}
|
||||
|
||||
\paragraph{Metered (operational).} Hot, mutable, operational data --- a
|
||||
Datastore tenant's active parts, a project's live SQLite file --- is priced
|
||||
pay-as-you-go. This model exists today: the platform meters storage at a
|
||||
per-gigabyte-month rate (with finer per-second and per-hour granularity) plus
|
||||
egress, billed against a prepaid organization wallet, comparable to commodity
|
||||
object-store pricing. It suits data whose value tracks its recency and whose
|
||||
owner expects a recurring bill.
|
||||
|
||||
\paragraph{Perpetual (archival).} A second class of data should be stored
|
||||
\emph{once} and kept \emph{forever} --- audit logs, published datasets, the
|
||||
verifiable analytics commitments of Section~\ref{sec:chain}, content a user
|
||||
wants to outlive their subscription. Recurring billing is the wrong instrument
|
||||
here: it makes permanence contingent on an unbroken payment stream. The
|
||||
Arweave-style alternative is an \emph{endowment}: the writer pays a single
|
||||
upfront fee, a conservative fraction is reserved to cover immediate storage, and
|
||||
the remainder capitalizes an endowment whose yield funds storage in perpetuity.
|
||||
The model is solvent under one empirical assumption --- that the real cost of
|
||||
storing a byte declines over time (the historical $\sim\!30\%$/year decline in
|
||||
\$/GB) faster than the endowment is drawn down. Concretely, for a one-time fee
|
||||
$F$, an annual real storage cost $c_0$ per byte declining at rate $r$, and an
|
||||
endowment real return $\rho$, perpetual storage is funded when
|
||||
\[
|
||||
F \;\ge\; c_0 \sum_{t=0}^{\infty} \frac{(1-r)^t}{(1+\rho)^t}
|
||||
\;=\; c_0\,\frac{1+\rho}{\rho + r},
|
||||
\]
|
||||
a finite sum precisely because $r>0$. Pricing $F$ at a deliberately pessimistic
|
||||
$r$ (and re-pricing new writes as hardware costs fall) is what makes
|
||||
``pay once, stored for two centuries'' actuarially sound rather than a promise.
|
||||
This endowment does not exist yet and is the concrete economic build for S-Chain.
|
||||
|
||||
\subsection{The provider market underneath}
|
||||
|
||||
Both demand curves pay into a supply side: independent operators who run S-Chain
|
||||
storage nodes (the Hanzo S3 / SeaweedFS volume servers) and earn for storing and
|
||||
serving content-addressed objects. The accountability surface is already modeled
|
||||
--- the network's service-node registry carries provider registrations, service
|
||||
endpoints, and SLA/uptime records --- so the missing piece is the incentive
|
||||
layer that turns those records into payments and penalties:
|
||||
|
||||
\begin{itemize}[leftmargin=1.4em]
|
||||
\item \textbf{Bonded providers.} An operator stakes a bond to join a storage
|
||||
pool, consistent with the network's bonded-signer model elsewhere; this gates
|
||||
a permissioned-but-open supply set rather than fully anonymous storage.
|
||||
\item \textbf{Proof of storage / availability.} Providers periodically prove
|
||||
they still hold the objects they are paid for (a challenge--response over
|
||||
content-addressed chunks), recorded as the registry's uptime/SLA proofs.
|
||||
\item \textbf{Payout and slashing.} The metered revenue and the endowment
|
||||
yield flow to providers in proportion to proven, served storage; failed
|
||||
availability proofs slash the bond. This is the same accounting that the
|
||||
platform's existing usage-metering and wallet rails perform for compute,
|
||||
extended to storage providers.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{A permissionless public good, paid in any currency}
|
||||
|
||||
The market above is deliberately \emph{not} operator-run or
|
||||
us-in-control: S-Chain is intended as permissionless public-goods
|
||||
infrastructure that web3 applications build on with no central leader, the way
|
||||
one builds on a base layer rather than on a vendor. Two mechanisms make that
|
||||
economically coherent.
|
||||
|
||||
\paragraph{Multi-currency fees routed through the DEX.} Applications should not
|
||||
be forced to hold the network's base asset to pay for storage and compute. The
|
||||
network's native DEX (the D-Chain) is the exchange layer: any L2/L3 lists its
|
||||
coin into the DEX, and its applications pay infrastructure fees in \emph{their
|
||||
own} currency, which is swapped natively --- through the relevant liquidity pool
|
||||
(e.g.\ a ZOO/LUX pool) --- back into LUX to settle operator payouts. A storage
|
||||
or compute provider is thus paid in the base asset while a tenant spends the
|
||||
token it already holds; the DEX absorbs the currency conversion as a routine
|
||||
swap. The fee economy is multi-currency at the edge and single-currency at
|
||||
settlement.
|
||||
|
||||
\paragraph{Bootstrap inflation, then fees.} A storage market has a cold-start
|
||||
problem: providers must be paid before fee volume exists. The resolution is a
|
||||
\emph{temporary} inflation subsidy to seed provider rewards, tapering to zero as
|
||||
fee revenue grows --- explicitly temporary because the base asset is
|
||||
deflationary and cannot subsidize forever. Steady state is fee-funded: metered
|
||||
leases and endowment yield, denominated in any listed currency and settled in
|
||||
LUX, pay the bonded providers. Inflation is the on-ramp, not the engine.
|
||||
|
||||
\subsection{Why this closes the platform}
|
||||
|
||||
The economics complete the architecture rather than decorate it. The two query
|
||||
engines of Section~\ref{sec:twoengine} both reduce, at the storage layer, to
|
||||
content-addressed objects on S-Chain --- Datastore parts and encrypted SQLite
|
||||
pages alike --- so a single market prices and endows \emph{all} platform data
|
||||
uniformly: meter the hot working set, endow the permanent record, pay the
|
||||
bonded providers who hold it, and let the same Quasar consensus that orders the
|
||||
data's commitments also order the market's payments. A customer buys ``storage''
|
||||
once; the platform decides per-object whether that is a recurring lease or a
|
||||
perpetual endowment; and neither Datastore nor the decentralized SQL engine
|
||||
needs its own billing system, because both are tenants of the one S-Chain
|
||||
market.
|
||||
Reference in New Issue
Block a user