mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-30 14:58:42 +00:00
6f7e42d1a4
* paper: Datastore — PGO/ThinLTO/BOLT + ZAP optimization of a columnar engine Covers the ClickHouse fork's memory/binary optimization (49-50MB RSS vs official 116MB, binary 945->648MB), the XRay root-cause for AArch64 BOLT failure (composed R_AARCH64_PREL64 in xray_instr_map), gRPC->ZAP transport with promise pipelining (63x on dependent chains), flat memory to 10k connections, and the multi-tenant/Apache-2.0 resale posture. Pattern-A layout, builds via repo CI. * paper: add blockchain-native section (S-Chain storage, Raft→Quasar PQ, D-Chain VM, TEE) Storage/compute separation PoC (parts on SeaweedFS, verified), leaderless PQ consensus (Quasar/ML-DSA, base-ha precedent), data-system-as-a-chain pattern (graphvm+cevm), and confidential-compute permissionless model. * paper: two-engine decentralized platform + reframe (extends S-Chain, not a new chain) - 08-two-engine: Datastore OLAP + Decentralized SQL (SQLite/K-Chain KEM E2E, Z-Chain FHE, CRDTs) over S-Chain; OLAP/OLTP split along the trust axis. - 07: proven elastic scale-out + zero-copy (2 nodes, 1 copy); ZAP-native transport scorecard (gRPC dead, interserver-HTTP->ZAP, ZK->Quasar); reframed datastore as a co-located service EXTENDING S-Chain (D-Chain is the DEX), not its own public chain. * paper: S-Chain storage economics (metered + Arweave-style endowment + DEX-routed multi-currency public-goods market) - 09-economics: two demand curves (metered ops + perpetual endowment w/ solvency math), bonded provider market (stake/proof/payout/slash), multi-currency fees routed through the DEX (D-Chain) -> swapped to LUX, bootstrap inflation tapering to fees. Permissionless public good, no central leader. - 07: reframed datastore as inheriting S-Chain's permissionless public posture (not operator-run/centralized). * paper: F-Chain FHE (luxfi/fhe, real) + Z-Chain ZK rollups + E2E post-quantum (P3Q/ML-DSA/KEM) --------- Co-authored-by: Hanzo Dev <[email protected]>
77 lines
3.5 KiB
TeX
77 lines
3.5 KiB
TeX
\section{Deployment and Multi-Tenancy}
|
|
|
|
Datastore is built to be hosted as a provisionable, multi-tenant offering on a
|
|
native PaaS. Its compatibility surface makes the two analytics products Hanzo
|
|
operates --- a privacy-preserving web-analytics service and a product-analytics
|
|
platform --- migrate by configuration alone: both already isolate tenants by a
|
|
column key (\texttt{organization\_id} and \texttt{team\_id} respectively) over a
|
|
shared database and schema, so pointing them at a Datastore instance is a change
|
|
of host, not of data model. The leaner resident footprint compounds directly in
|
|
this setting: at $\sim\!50$\,MB per instance instead of $\sim\!116$\,MB, an
|
|
operator more than doubles the number of tenant instances per node at fixed
|
|
memory, which is the dimension that governs the unit economics of a managed
|
|
service.
|
|
|
|
Because the engine derives from ClickHouse under the permissive Apache~2.0
|
|
license, Datastore is freely re-brandable, hostable, and resellable --- as are
|
|
the adjacent Hanzo data products on equally permissive upstreams (PostgreSQL,
|
|
pgvector, SQLite, PocketBase, and the Valkey-derived key-value store). This
|
|
licensing posture is what makes a commercial managed-database offering built on
|
|
these components viable without copyleft entanglement.
|
|
|
|
\section{Conclusion}
|
|
|
|
We have shown that the operationally dominant costs of a columnar analytics
|
|
server --- resident memory and binary size --- yield substantially to a
|
|
disciplined build pipeline. PGO, ThinLTO, and BOLT together produce a Datastore
|
|
server at less than half the resident footprint of the upstream official binary
|
|
with equal or better read throughput, and memory that is flat from $10$ to
|
|
$10{,}000$ connections. Along the way we identified that default-enabled XRay
|
|
instrumentation was both bloating the binary by $\sim\!45$\,MB and, through
|
|
unlowerable composed \texttt{R\_AARCH64\_PREL64} relocations, silently blocking
|
|
BOLT on AArch64 --- a fix that is one disabled default. Finally, replacing gRPC
|
|
with a Cap'n-Proto-derived transport that carries native columnar blocks as
|
|
opaque payloads exposes promise pipelining to clients, flattening dependent-chain
|
|
latency by up to $63\times$ without disturbing the engine's bounded memory
|
|
behavior.
|
|
|
|
The work points to two clear next steps: promoting ZAP from a transport to the
|
|
primary, ZAP-native query path, and extending the same posture to the
|
|
coordination layer. Both are constrained by the same principle that guided this
|
|
work --- preserve compatibility, pay only for what the workload uses, and let the
|
|
measurements, not the defaults, decide what ships.
|
|
|
|
\begin{thebibliography}{9}
|
|
|
|
\bibitem{clickhouse}
|
|
ClickHouse, Inc.,
|
|
\newblock \textit{ClickHouse: Open-Source Column-Oriented DBMS for Real-Time
|
|
Analytics}.
|
|
\newblock \texttt{https://clickhouse.com}, Apache License 2.0.
|
|
|
|
\bibitem{bolt}
|
|
M.~Panchenko, R.~Auler, B.~Nell, and G.~Ottoni,
|
|
\newblock ``BOLT: A Practical Binary Optimizer for Data Centers and Beyond,''
|
|
\newblock \textit{CGO}, 2019.
|
|
|
|
\bibitem{xray}
|
|
D.~M.~Berris, A.~Veitch, N.~Lewycky, et~al.,
|
|
\newblock ``XRay: A Function Call Tracing System,''
|
|
\newblock Google Technical Report, 2016.
|
|
|
|
\bibitem{capnproto}
|
|
K.~Varda,
|
|
\newblock \textit{Cap'n Proto: Cerealization Protocol}.
|
|
\newblock \texttt{https://capnproto.org}.
|
|
|
|
\bibitem{pgo}
|
|
\newblock ``Profile Guided Optimization,'' LLVM/Clang Documentation.
|
|
\newblock \texttt{https://clang.llvm.org/docs/UsersManual.html\#profile-guided-optimization}.
|
|
|
|
\bibitem{thinlto}
|
|
T.~Johnson, M.~Amini, and X.~D.~Li,
|
|
\newblock ``ThinLTO: Scalable and Incremental LTO,''
|
|
\newblock \textit{CGO}, 2017.
|
|
|
|
\end{thebibliography}
|