mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
The product papers, held to what was measured
All ninety-six papers from 86d8915 were audited for numbers presented as
results. Three hundred and fifty-three turn out to be presented as measurements
with nothing behind them, and not one paper in the batch cites a harness, a run,
a checkpoint, a seed or a date. The thirteen product papers at the root are the
part of that this commit fixes.
Each carried a production-operations figure in its abstract — 200+ models across
48 GPUs at 99.9% availability, 2M authentications daily over 1,200 tenants,
$240M processed at 99.97%, 15TB across 400 tenants, three years of ledger
operation without a discrepancy, a 65% cut in mean-time-to-diagnosis, per-request
overhead against nginx and Caddy, an iPhone 15 Pro latency table. None of the
repositories behind them holds a benchmark that produces those numbers; no
deployment in this estate records those volumes; the hardware two of them name
does not exist here; and several papers are dated years before their own code was
first committed.
A design target can be relabelled. These could not, because a target is a claim
about the future and these were claims about a past that did not happen. So they
are removed, and every paper now carries a short section naming what it said,
why it is gone, and what run would put a number back. Nothing was replaced with a
number that was not measured either.
Two papers are different. pubsub and ledger ship harnesses that measure exactly
the quantities their tables report — the NATS benchmark suite in one, a
parallelism-flagged TPS harness in the other. Those tables stay, with their
provenance stated: not traceable to a recorded run, and here is what would make
them so. The gap is a run, not a mechanism.
The eighty-one zen/ papers hold the other 321 and are left alone. zenlm/papers
already had a sweeping withdrawal pass reverted for asserting more than it
verified, the models are largely real, and several of these are template-
generated with one pretraining sentence filled in for different models — that
needs a decision about the family, not an edit per paper. LLM.md records the
finding, the shapes it takes, and the two zen papers that already do it right.
Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
@@ -101,3 +101,63 @@ The website is a curated, externally checkable catalog rather than an automatic
|
||||
every TeX file. Publishing a new paper therefore requires both its card in
|
||||
`site/src/config/papers.ts` and its compiled artifact in `site/public/pdfs/`. The root
|
||||
`pdfs/` directory remains CI output and is ignored.
|
||||
|
||||
## The 86d8915 batch
|
||||
|
||||
Ninety-six papers landed in one commit — fifteen product papers, plus a sync of
|
||||
fifty-eight from `zenlm/papers`. Two of the fifteen (`hanzo-base`,
|
||||
`hanzo-operate-computer`) were replaced in `dbf6fc9` by rewritten papers under
|
||||
`hanzo-base-runtime/` and `hanzo-operative/`. The other ninety-four were audited
|
||||
for numbers presented as measurements: every quantitative performance claim was
|
||||
put in one of four places — traceable to a run, a harness or a commit; someone
|
||||
else's published number, attributed; a target or a model, labelled as one in the
|
||||
text; or presented as a result with nothing behind it.
|
||||
|
||||
**353 claims are in the last place, and none is in the first.** No paper in the
|
||||
batch names an evaluation harness, a run identifier, a checkpoint, a seed or a
|
||||
date. None cites `bench/brain` (hanzoai/cloud) or `bench/locomo`
|
||||
(hanzoai/semantic), the two harnesses in the estate that declare their splits and
|
||||
publish bootstrap intervals, nor `hanzoai/benchmarks`, which backs
|
||||
`hanzo-cloud-network-bench.tex`. That paper is what the house standard looks
|
||||
like: two machines named, the tuning published as one idempotent script, raw
|
||||
numbers in a public repo. The batch does not follow it.
|
||||
|
||||
**The thirteen product papers at the repository root are corrected.** Each
|
||||
carried a production-operations figure in its abstract — 48 GPUs and 99.9%
|
||||
availability, 2M authentications daily across 1,200 tenants, \$240M processed at
|
||||
99.97%, 15TB across 400 tenants, three years of ledger operation without a
|
||||
discrepancy — for services whose repositories hold no benchmark, whose
|
||||
deployments record no such volume, and several of which were first committed
|
||||
years after the paper's own date. Those claims cannot be relabelled as targets,
|
||||
because a target is a statement about the future and these were statements about
|
||||
a past that did not happen, so they are removed and each paper now carries a
|
||||
short section saying what was removed and why. Two are different: `pubsub` and
|
||||
`ledger` ship harnesses that measure exactly the quantities their tables report,
|
||||
so those tables stay with their provenance stated. The gap there is a run, not a
|
||||
mechanism.
|
||||
|
||||
**The eighty-one `zen/` papers are not corrected and need a decision.** They hold
|
||||
321 of the 353 defects, in three recurring shapes: training and inference
|
||||
attributed to H100 and A100 fleets of 8 to 16,384 GPUs; efficiency, energy and
|
||||
carbon figures with no baseline named; and abstracts whose headline number
|
||||
contradicts the paper's own table. Some are checkable without any estate fact —
|
||||
decode throughput above the memory-bandwidth ceiling of the part it names, FP8 on
|
||||
Ampere, AIME scores that are not multiples of 1/30 on a 30-problem exam, INT4
|
||||
requiring more memory than FP16.
|
||||
|
||||
They are left alone deliberately. `zenlm/papers` already had a sweeping
|
||||
withdrawal pass, `5deeeaa`, which the owner reverted in `f34f609`: *"That commit
|
||||
asserted it; nothing verified it... a paper that disclaims its own results is
|
||||
worse than one that under-cites them. Where these papers are thin it is in
|
||||
provenance — the runs need citing, not retracting."* The subject models are
|
||||
mostly real; 78 are published under the `zenlm` org. What is needed is a decision
|
||||
about a family of papers — several of which are template-generated, with the same
|
||||
pretraining sentence and the same energy table filled in for different models —
|
||||
and that is not an edit to make one paper at a time. Two of the eighty-one show
|
||||
what the rest could be: `zen-mixture-of-experts.tex` states plainly that it
|
||||
publishes no number it has not measured and names its harness, and
|
||||
`zen-coder_whitepaper.tex` reports no scores and labels its costs as estimates.
|
||||
|
||||
Note also that 69 of the 86 `zen/*.tex` here have diverged from their
|
||||
counterparts in `zenlm/papers`, so the two sites publish different text under the
|
||||
same titles. A fix applied in one place does not reach the other.
|
||||
|
||||
Binary file not shown.
+23
-18
@@ -45,7 +45,7 @@
|
||||
\hanzocoverpage
|
||||
|
||||
\begin{abstract}
|
||||
We present Hanzo Commerce, a payment orchestration platform that unifies multiple payment processors---Stripe, Square, PayPal, Adyen, and Braintree---alongside cryptocurrency payment channels under a single transactional API. The system employs Temporal-based durable workflows for payment lifecycle management, ensuring exactly-once processing semantics even under partial failures. Commerce introduces processor-agnostic payment intents, automatic failover across processors, and a double-entry ledger for financial reconciliation. Deployed since January 2017, the platform has processed over \$240M in transactions with a 99.97\% success rate and zero double-charge incidents. We describe the orchestration architecture, the processor abstraction layer, and the reconciliation system that maintains financial integrity across heterogeneous payment backends.
|
||||
We present Hanzo Commerce, a payment orchestration platform that unifies multiple payment processors---Stripe, Square, PayPal, Adyen, and Braintree---alongside cryptocurrency payment channels under a single transactional API. The system employs Temporal-based durable workflows for payment lifecycle management, ensuring exactly-once processing semantics even under partial failures. Commerce introduces processor-agnostic payment intents, automatic failover across processors, and a double-entry ledger for financial reconciliation. This paper describes the orchestration design and reports no operating figures; Section~\ref{sec:figures} records the deployment history, transaction volume and success rate an earlier version reported and why they are withdrawn. We describe the orchestration architecture, the processor abstraction layer, and the reconciliation system that maintains financial integrity across heterogeneous payment backends.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -181,22 +181,27 @@ Commerce supports on-chain payments on Ethereum, Lux C-Chain, and Bitcoin. Crypt
|
||||
|
||||
Commerce is implemented in 35,000 lines of Go with a React admin dashboard. Processor adapters total approximately 2,000 lines each. The Temporal worker runs as a separate deployment with 3 replicas for high availability.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Processing volume by processor (2023).}
|
||||
\label{tab:volume}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Processor} & \textbf{Volume (\$M)} & \textbf{Success \%} \\
|
||||
\midrule
|
||||
Stripe & 142.3 & 99.98 \\
|
||||
Adyen & 58.7 & 99.95 \\
|
||||
PayPal & 22.1 & 99.94 \\
|
||||
Crypto (LUX) & 12.4 & 99.99 \\
|
||||
Square & 4.8 & 99.96 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\section{Status of the operating figures}
|
||||
\label{sec:figures}
|
||||
|
||||
An earlier version of this paper carried a table titled ``Processing volume by
|
||||
processor (2023)'' attributing \$142.3M at 99.98\% success to Stripe, \$58.7M to
|
||||
Adyen, \$22.1M to PayPal, \$12.4M to a crypto channel and \$4.8M to Square. The
|
||||
abstract and conclusion stated that the platform had been deployed since January
|
||||
2017 and had processed over \$240M at a 99.97\% success rate with zero
|
||||
double-charge incidents.
|
||||
|
||||
None of it has a source. No ledger export, settlement report or reconciliation
|
||||
output stands behind the volume or the success rates; there is no incident
|
||||
record behind ``zero double-charge incidents''; and the deployment date is not
|
||||
one this codebase can support. Financial volume is the one class of claim that
|
||||
should never appear in a paper without a reconciled source, so these are removed
|
||||
outright rather than restated as approximate.
|
||||
|
||||
Nothing here is hard to establish for a system that is actually processing
|
||||
payments: the double-entry ledger the paper describes is itself the source, and
|
||||
a period, a currency and a reconciliation run would produce every figure above
|
||||
with provenance attached.
|
||||
|
||||
\section{Security}
|
||||
|
||||
@@ -210,7 +215,7 @@ Square & 4.8 & 99.96 \\
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Hanzo Commerce provides a unified payment orchestration layer that abstracts the complexity of multi-processor payment processing. Temporal-based durable workflows ensure exactly-once semantics and automatic compensation. Intelligent routing optimizes for success rate, latency, and cost. The system has processed over \$240M with a 99.97\% success rate, demonstrating that payment orchestration can achieve higher reliability than direct processor integration through cross-processor failover.
|
||||
Hanzo Commerce provides a unified payment orchestration layer that abstracts the complexity of multi-processor payment processing. Temporal-based durable workflows ensure exactly-once semantics and automatic compensation. Routing selects among processors on success rate, latency and cost. The argument for orchestration over a direct processor integration is structural --- a failed authorization at one processor can be retried at another under the same idempotency key, which a single integration cannot do --- and this paper does not quantify what that is worth (Section~\ref{sec:figures}).
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{10}
|
||||
|
||||
Binary file not shown.
+22
-18
@@ -45,7 +45,7 @@
|
||||
\hanzocoverpage
|
||||
|
||||
\begin{abstract}
|
||||
We present Hanzo Console, a unified web dashboard for managing AI infrastructure across the Hanzo platform. Console provides a single pane of glass for service deployments, real-time metrics, log aggregation, secret management, database administration, and team management. The dashboard integrates with every Hanzo infrastructure service---IAM, KMS, Engine, Storage, PubSub, and Ledger---through a consistent API gateway pattern, presenting a cohesive management experience. Console supports multi-tenant RBAC with organization-scoped views, real-time metric streaming via WebSocket, and deployment management with one-click rollbacks. Serving 1,200+ organizations, Console reduces mean-time-to-diagnosis by 65\% compared to CLI-based workflows by surfacing correlated metrics, logs, and traces in a unified interface.
|
||||
We present Hanzo Console, a unified web dashboard for managing AI infrastructure across the Hanzo platform. Console provides a single pane of glass for service deployments, real-time metrics, log aggregation, secret management, database administration, and team management. The dashboard integrates with every Hanzo infrastructure service---IAM, KMS, Engine, Storage, PubSub, and Ledger---through a consistent API gateway pattern, presenting a cohesive management experience. Console supports multi-tenant RBAC with organization-scoped views, real-time metric streaming via WebSocket, and deployment management with one-click rollbacks. Console surfaces correlated metrics, logs and traces in one interface, which is the property it is built for: an operator following an incident does not change tools between the symptom and its cause. This paper reports no measurements of that benefit or of the dashboard itself; Section~\ref{sec:figures} records the figures an earlier version reported and why they are withdrawn.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -195,22 +195,26 @@ Alert rules are evaluated server-side by the metrics pipeline. Notifications are
|
||||
|
||||
Console is implemented in 42,000 lines of TypeScript (React frontend) with a 5,000-line Go BFF (Backend for Frontend) that handles WebSocket multiplexing and metric aggregation. The frontend uses Radix UI primitives for accessibility compliance.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Dashboard load time by view (ms).}
|
||||
\label{tab:load}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{View} & \textbf{Initial} & \textbf{Cached} \\
|
||||
\midrule
|
||||
Overview & 420 & 85 \\
|
||||
Engine models & 380 & 70 \\
|
||||
KMS secrets & 310 & 60 \\
|
||||
PubSub streams & 350 & 75 \\
|
||||
Deployments & 290 & 55 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\section{Status of the performance figures}
|
||||
\label{sec:figures}
|
||||
|
||||
An earlier version of this paper carried a table titled ``Dashboard load time by
|
||||
view (ms)'' giving initial and cached load times for five views, from 420/85\,ms
|
||||
on the overview to 290/55\,ms on deployments. The abstract stated that Console
|
||||
serves more than 1{,}200 organizations and reduces mean-time-to-diagnosis by
|
||||
65\% against CLI-based workflows.
|
||||
|
||||
None of it was measured. The load-time table named no browser, no network
|
||||
profile, no device and no sample count, and no trace or Lighthouse output exists
|
||||
in \texttt{hanzoai/console}. The 65\% figure has no study behind it: no incident
|
||||
dataset, no baseline definition of the CLI workflow it is compared against, and
|
||||
no sample. The organization count describes an operating history the console
|
||||
does not have. All three are removed.
|
||||
|
||||
Load time is measurable today --- a Lighthouse run against a named build on a
|
||||
declared network profile answers it. Mean-time-to-diagnosis is a harder claim
|
||||
and needs an incident record with both arms, which is the reason it should not
|
||||
be asserted until one exists.
|
||||
|
||||
\section{Security}
|
||||
|
||||
@@ -224,7 +228,7 @@ Deployments & 290 & 55 \\
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Hanzo Console provides a unified management interface for the entire Hanzo infrastructure stack. By integrating deeply with each service rather than providing a generic dashboard framework, Console surfaces domain-specific insights---GPU utilization for Engine, consumer lag for PubSub, balance reconciliation for Ledger---in a coherent interface. Real-time metric streaming, deployment management, and configurable alerting reduce operational overhead and mean-time-to-diagnosis for platform operators.
|
||||
Hanzo Console provides a unified management interface for the entire Hanzo infrastructure stack. By integrating deeply with each service rather than providing a generic dashboard framework, Console surfaces domain-specific insights---GPU utilization for Engine, consumer lag for PubSub, balance reconciliation for Ledger---in a coherent interface. Real-time metric streaming, deployment management and configurable alerting are aimed at reducing operational overhead and mean-time-to-diagnosis; by how much is not established here (Section~\ref{sec:figures}).
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{10}
|
||||
|
||||
Binary file not shown.
+24
-18
@@ -45,7 +45,7 @@
|
||||
\hanzocoverpage
|
||||
|
||||
\begin{abstract}
|
||||
We present Hanzo Edge, a Rust-based inference runtime for deploying AI models on mobile devices, edge servers, and IoT hardware. Edge supports ONNX model loading with automatic optimization for target hardware, including Apple Neural Engine (ANE) on iOS/macOS, GPU compute via Metal and Vulkan, and SIMD-accelerated CPU inference. The runtime implements dynamic quantization (INT8/INT4), operator fusion, and memory-mapped model loading to minimize memory footprint and startup latency. On an iPhone 15 Pro, Edge achieves 42 tokens per second for a 3B parameter language model and 8ms latency for a MobileNet-V3 image classifier. We describe the hardware abstraction layer, the quantization pipeline, and the model optimization passes that enable production AI inference on resource-constrained devices.
|
||||
We present Hanzo Edge, a Rust-based inference runtime for deploying AI models on mobile devices, edge servers, and IoT hardware. Edge supports ONNX model loading with automatic optimization for target hardware, including Apple Neural Engine (ANE) on iOS/macOS, GPU compute via Metal and Vulkan, and SIMD-accelerated CPU inference. The runtime implements dynamic quantization (INT8/INT4), operator fusion, and memory-mapped model loading to minimize memory footprint and startup latency. This paper describes those mechanisms and reports no measurements; Section~\ref{sec:figures} records the on-device latency figures an earlier version reported and why they are withdrawn. We describe the hardware abstraction layer, the quantization pipeline, and the model optimization passes that enable production AI inference on resource-constrained devices.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -59,7 +59,7 @@ Hanzo Edge provides a unified Rust runtime that accepts ONNX models and automati
|
||||
\item A hardware abstraction layer supporting ANE, Metal, Vulkan, and SIMD CPU backends with automatic backend selection.
|
||||
\item Dynamic INT8 and INT4 quantization with calibration-free activation range estimation.
|
||||
\item Operator fusion passes that combine convolution-batchnorm-relu and attention patterns into single fused kernels.
|
||||
\item Memory-mapped model loading with lazy initialization, reducing cold-start time to under 50ms for models up to 3B parameters.
|
||||
\item Memory-mapped model loading with lazy initialization, so that start-up touches only the pages a first forward pass needs rather than the whole file.
|
||||
\end{itemize}
|
||||
|
||||
\section{Architecture}
|
||||
@@ -171,21 +171,27 @@ Operations unsupported by ANE (custom activations, dynamic shapes) fall back to
|
||||
|
||||
Edge is implemented in 22,000 lines of Rust with platform-specific backends: 3,000 lines of Metal shader code (Apple), 2,500 lines of Vulkan compute shaders (Android/Linux), and 1,500 lines of NEON/AVX intrinsics (CPU).
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Inference latency on iPhone 15 Pro.}
|
||||
\label{tab:edge-perf}
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
\textbf{Model} & \textbf{Params} & \textbf{Latency} & \textbf{Memory} \\
|
||||
\midrule
|
||||
MobileNet-V3 & 5.4M & 8ms & 22MB \\
|
||||
Whisper-small & 244M & 1.2s/30s & 480MB \\
|
||||
Qwen3-3B (INT4) & 3B & 24ms/tok & 1.8GB \\
|
||||
CLIP-ViT-B/32 & 150M & 18ms & 310MB \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\section{Status of the performance figures}
|
||||
\label{sec:figures}
|
||||
|
||||
An earlier version of this paper carried a table titled ``Inference latency on
|
||||
iPhone 15 Pro'' reporting 8\,ms for MobileNet-V3, 1.2\,s per 30\,s of audio for
|
||||
Whisper-small, 24\,ms per token for Qwen3-3B at INT4 and 18\,ms for
|
||||
CLIP-ViT-B/32, with memory footprints for each. The abstract restated the same
|
||||
run as 42 tokens per second and 8\,ms, and the contributions list and conclusion
|
||||
claimed a cold start under 50\,ms.
|
||||
|
||||
None of it was measured, and the runtime the table describes is not the one that
|
||||
exists. \texttt{hanzoai/edge} is a Candle-based Rust crate with no iOS target,
|
||||
no Core~ML or Neural Engine backend, no ONNX loader, and none of the four models
|
||||
named in the table; its own \texttt{bench} subcommand reports time-to-first-token
|
||||
and tokens per second for an already-loaded model and never measures load time.
|
||||
The figures cannot be relabelled as targets for a runtime that has no path to
|
||||
the device they name, so they are removed.
|
||||
|
||||
The sections that follow describe the hardware abstraction layer, the
|
||||
quantization pipeline and the optimization passes as a design. Which of them are
|
||||
implemented today, and what they cost on a named device, are both open.
|
||||
|
||||
\section{Security}
|
||||
|
||||
@@ -199,7 +205,7 @@ CLIP-ViT-B/32 & 150M & 18ms & 310MB \\
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Hanzo Edge enables production AI inference on resource-constrained devices by combining a Rust runtime with automatic hardware dispatch, dynamic quantization, and operator fusion. The memory-mapped model format and lazy initialization achieve sub-50ms cold start times, while INT4 quantization fits 3B parameter models within mobile memory constraints. By running inference entirely on-device, Edge preserves user privacy and eliminates network latency.
|
||||
Hanzo Edge is a design for on-device AI inference: a Rust runtime with hardware dispatch, dynamic quantization and operator fusion, a memory-mapped model format to keep start-up proportional to what is touched, and INT4 quantization to fit 3B-parameter models inside a mobile memory budget. Running inference on-device removes the network from the path and keeps input data on the device. What the design costs in latency and memory on a named device is not established here; Section~\ref{sec:figures} says why the previous figures were withdrawn.
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{10}
|
||||
|
||||
Binary file not shown.
+37
-25
@@ -45,7 +45,7 @@
|
||||
\hanzocoverpage
|
||||
|
||||
\begin{abstract}
|
||||
We present Hanzo Engine, a production ML inference engine implemented in Rust that serves multiple models concurrently with dynamic batching, intelligent GPU scheduling, and A/B traffic routing. Engine manages a fleet of GPU workers, automatically loading and unloading models based on traffic patterns, and implements continuous batching for transformer models to maximize GPU utilization. The system serves 200+ models in production across 48 GPUs with a median time-to-first-token (TTFT) of 45ms for language models and 99.9\% availability. Key innovations include: (1) a priority-aware GPU scheduler that co-locates complementary workloads, (2) speculative model preloading based on traffic prediction, and (3) a unified serving protocol supporting both streaming and batch inference.
|
||||
We describe the design of Hanzo Engine, an ML inference orchestration layer implemented in Rust that serves multiple models concurrently with dynamic batching, GPU scheduling, and A/B traffic routing. Engine manages a fleet of GPU workers, loading and unloading models according to traffic, and applies continuous batching to transformer models to raise GPU utilization. The design has three parts: (1) a priority-aware GPU scheduler that co-locates models with complementary memory and compute profiles, (2) speculative model preloading driven by traffic prediction, and (3) a unified serving protocol covering streaming and batch inference. This paper describes a design and reports no measurements. An earlier version quoted serving latency, throughput, GPU utilization and availability figures for a 48-GPU production fleet; those figures were never measured and no such fleet exists. Section~\ref{sec:figures} records what they were and what would be needed to replace them.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -57,8 +57,8 @@ Hanzo Engine operates at the orchestration layer above individual model servers.
|
||||
\paragraph{Contributions.}
|
||||
\begin{itemize}
|
||||
\item A priority-aware GPU scheduler that co-locates models with complementary memory and compute profiles on shared GPUs.
|
||||
\item Continuous batching with iteration-level scheduling for transformer models, achieving 3.2x throughput improvement over static batching.
|
||||
\item Speculative model preloading using traffic prediction to reduce cold-start latency from 30s to under 200ms.
|
||||
\item Continuous batching with iteration-level scheduling for transformer models, following Orca~\cite{yu2022}, so that padding waste does not scale with the longest sequence in the batch.
|
||||
\item Speculative model preloading using traffic prediction, which moves the cold-start cost off the request path when the prediction is correct.
|
||||
\item A unified serving protocol supporting streaming tokens, batch embeddings, and multimodal inputs.
|
||||
\end{itemize}
|
||||
|
||||
@@ -76,9 +76,9 @@ Hanzo Engine operates at the orchestration layer above individual model servers.
|
||||
\node[box] (api) {Serving\\API};
|
||||
\node[box, below=of api] (router) {Model\\Router};
|
||||
\node[box, below=of router] (scheduler) {GPU\\Scheduler};
|
||||
\node[box, below left=0.6cm and 0.3cm of scheduler] (w1) {Worker 1\\(8xA100)};
|
||||
\node[box, below=of scheduler] (w2) {Worker 2\\(8xA100)};
|
||||
\node[box, below right=0.6cm and 0.3cm of scheduler] (w3) {Worker 3\\(8xH100)};
|
||||
\node[box, below left=0.6cm and 0.3cm of scheduler] (w1) {GPU\\Worker 1};
|
||||
\node[box, below=of scheduler] (w2) {GPU\\Worker 2};
|
||||
\node[box, below right=0.6cm and 0.3cm of scheduler] (w3) {GPU\\Worker $n$};
|
||||
\node[box, right=1.5cm of router] (registry) {Model\\Registry};
|
||||
\node[box, left=1.5cm of router] (ab) {A/B\\Router};
|
||||
|
||||
@@ -160,11 +160,11 @@ impl ContinuousBatcher {
|
||||
}
|
||||
\end{lstlisting}
|
||||
|
||||
This achieves 3.2x higher throughput than static batching by eliminating padding waste.
|
||||
The mechanism removes the padding waste that static batching incurs when a batch is held open for its longest sequence. Orca~\cite{yu2022} and vLLM~\cite{kwon2023} report the size of that effect for their own systems and workloads; we have not measured it for this one.
|
||||
|
||||
\subsection{Speculative Preloading}
|
||||
|
||||
Engine tracks per-model request rates with exponential moving averages and preloads models predicted to receive traffic within the next 5 minutes. The predictor uses a simple ARIMA model trained on the last 24 hours of traffic data. Preloading reduces cold-start latency from 30s (model load from storage) to under 200ms (already in GPU memory).
|
||||
Engine tracks per-model request rates with exponential moving averages and preloads models predicted to receive traffic within the next 5 minutes. The predictor uses a simple ARIMA model over the last 24 hours of traffic. A correct prediction replaces a cold start --- reading weights from storage into GPU memory --- with a cache hit on a model already resident. The size of that difference is set by model size and storage bandwidth and is not reported here; a wrong prediction costs the memory the preloaded model occupies.
|
||||
|
||||
\subsection{A/B Routing}
|
||||
|
||||
@@ -183,23 +183,32 @@ Results are logged with the model version for offline analysis of quality metric
|
||||
|
||||
\section{Implementation}
|
||||
|
||||
Engine is implemented in 35,000 lines of Rust. Worker communication uses gRPC with streaming for token-by-token delivery. Model loading uses \texttt{safetensors} format with memory-mapped I/O.
|
||||
Engine is implemented in Rust. Worker communication uses gRPC with streaming for token-by-token delivery. Model loading uses \texttt{safetensors} format with memory-mapped I/O.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Serving performance (Qwen3-72B, 8xH100).}
|
||||
\label{tab:engine-perf}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Static Batch} & \textbf{Continuous} \\
|
||||
\midrule
|
||||
TTFT p50 (ms) & 52 & 45 \\
|
||||
TTFT p99 (ms) & 280 & 120 \\
|
||||
Throughput (tok/s) & 4,200 & 13,400 \\
|
||||
GPU utilization & 42\% & 87\% \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\section{Status of the performance figures}
|
||||
\label{sec:figures}
|
||||
|
||||
An earlier version of this paper carried a table titled ``Serving performance
|
||||
(Qwen3-72B, 8xH100)'' reporting TTFT p50 of 52\,ms under static batching against
|
||||
45\,ms under continuous batching, p99 of 280\,ms against 120\,ms, throughput of
|
||||
4{,}200 against 13{,}400\,tok/s, and GPU utilization of 42\% against 87\%. The
|
||||
abstract and conclusion carried the same numbers together with ``200+ models in
|
||||
production across 48 GPUs'' and ``99.9\% availability,'' and the contributions
|
||||
list stated a $3.2\times$ throughput gain and a cold-start reduction from 30\,s
|
||||
to under 200\,ms.
|
||||
|
||||
None of it was measured. There is no 8-GPU H100 node and no 48-GPU fleet in this
|
||||
estate, so the table could not be relabelled as a design target without also
|
||||
changing the machine it names; there is no serving deployment behind the model
|
||||
count or the availability figure; and the paper contains no experiment
|
||||
corresponding to the two ratios. The figures are removed rather than restated.
|
||||
|
||||
The measurement they stood in for is available to make. \texttt{hanzoai/engine}
|
||||
ships a benchmark command (\texttt{hanzo bench}) that reports prompt and decode
|
||||
throughput for a named model and accelerator, and the house convention for
|
||||
reporting such a run --- declared hardware, a fixed handler and payload, raw
|
||||
artifacts published alongside --- is set out in the Hanzo cloud data plane
|
||||
benchmark~\cite{clouddataplane}. What is missing is the run, not the harness.
|
||||
|
||||
\section{Security}
|
||||
|
||||
@@ -215,7 +224,7 @@ GPU utilization & 42\% & 87\% \\
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Hanzo Engine provides production ML inference orchestration that maximizes GPU utilization through continuous batching, intelligent scheduling, and speculative preloading. By operating at the orchestration layer above individual model servers, Engine manages the complexity of multi-model serving---model lifecycle, GPU allocation, traffic routing, and A/B experiments---while maintaining sub-50ms TTFT for language models at 99.9\% availability.
|
||||
Hanzo Engine is a design for ML inference orchestration that raises GPU utilization through continuous batching, priority-aware scheduling, and speculative preloading. By operating above individual model servers, Engine takes on the parts of multi-model serving that a single-model server leaves open---model lifecycle, GPU allocation, traffic routing, and A/B experiments. Whether the design delivers the utilization it is built for is an open question here; Section~\ref{sec:figures} says what would answer it.
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{10}
|
||||
@@ -232,6 +241,9 @@ NVIDIA. Triton Inference Server. 2020.
|
||||
\bibitem{yu2022}
|
||||
G. Yu et al. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI, 2022.
|
||||
|
||||
\bibitem{clouddataplane}
|
||||
Hanzo AI Research. The Hanzo Cloud Data Plane: Benchmarking \texttt{zip}, ZAP, and the Road to a Million Requests per Second. Hanzo Industries, 2026.
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
+24
-20
@@ -45,7 +45,7 @@
|
||||
\hanzocoverpage
|
||||
|
||||
\begin{abstract}
|
||||
We present Hanzo Gateway, a high-performance API gateway built on the KrakenD framework that serves as the unified entry point for all Hanzo platform services. The gateway implements stateless JWT validation against Hanzo IAM, per-tenant rate limiting with sliding window counters, automatic circuit breaking for downstream failures, and request/response transformation pipelines. Processing over 147 endpoint definitions, the gateway achieves a median latency overhead of 1.2ms per request while handling 45,000 requests per second on a single node. We describe the declarative configuration model, the plugin architecture for custom middleware, and the operational patterns that enable zero-downtime configuration reloads in production.
|
||||
We present Hanzo Gateway, a high-performance API gateway built on the KrakenD framework that serves as the unified entry point for all Hanzo platform services. The gateway implements stateless JWT validation against Hanzo IAM, per-tenant rate limiting with sliding window counters, automatic circuit breaking for downstream failures, and request/response transformation pipelines. The gateway carries 147 endpoint definitions. This paper describes that configuration model and the middleware it drives, and reports no measurements; Section~\ref{sec:figures} records the per-request overhead and throughput figures an earlier version reported and why they are withdrawn. We describe the declarative configuration model, the plugin architecture for custom middleware, and the operational patterns that enable zero-downtime configuration reloads in production.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -194,24 +194,25 @@ The gateway is built on KrakenD Community Edition with custom Go plugins for Han
|
||||
|
||||
\paragraph{Observability.} Every request generates a Prometheus histogram observation keyed by endpoint, method, status code, and backend. Custom metrics track JWT validation latency, rate limit rejections, and circuit breaker state transitions.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Gateway overhead per request (ms).}
|
||||
\label{tab:overhead}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Component} & \textbf{p50} & \textbf{p99} \\
|
||||
\midrule
|
||||
TLS + CORS & 0.2 & 0.5 \\
|
||||
JWT validation & 0.3 & 0.8 \\
|
||||
Rate limit check & 0.05 & 0.1 \\
|
||||
Routing & 0.1 & 0.2 \\
|
||||
Response merge & 0.15 & 0.4 \\
|
||||
\midrule
|
||||
\textbf{Total overhead} & \textbf{0.8} & \textbf{2.0} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\section{Status of the performance figures}
|
||||
\label{sec:figures}
|
||||
|
||||
An earlier version of this paper carried a table titled ``Gateway overhead per
|
||||
request (ms)'' attributing p50/p99 of 0.2/0.5\,ms to TLS and CORS, 0.3/0.8\,ms
|
||||
to JWT validation, 0.05/0.1\,ms to the rate-limit check, 0.1/0.2\,ms to routing
|
||||
and 0.15/0.4\,ms to response merging, totalling 0.8/2.0\,ms. The abstract gave a
|
||||
median overhead of 1.2\,ms and 45{,}000 requests per second on a single node,
|
||||
and the conclusion claimed under 2\,ms on the critical path.
|
||||
|
||||
None of it was measured. \texttt{hanzoai/gateway} contains no benchmark
|
||||
function, no load generator configuration and no captured result; the table
|
||||
named no hardware, no concurrency and no request count; and the abstract's
|
||||
1.2\,ms median contradicted the table's own 0.8\,ms total in the same document.
|
||||
|
||||
A per-component breakdown of this kind is exactly what a profile produces, and
|
||||
the middleware chain it describes is real. What is missing is the profile, the
|
||||
load generator, and the machine --- reported the way the Hanzo cloud data plane
|
||||
benchmark~\cite{clouddataplane} reports them.
|
||||
|
||||
\section{Security}
|
||||
|
||||
@@ -225,7 +226,7 @@ Response merge & 0.15 & 0.4 \\
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Hanzo Gateway provides a stateless, high-performance API gateway that adds less than 2ms of overhead to the critical request path. The declarative configuration model enables rapid endpoint management without code changes, while the circuit breaker and rate limiting systems protect backend services from cascading failures. The stateless architecture enables simple horizontal scaling---adding capacity requires only deploying additional replicas.
|
||||
Hanzo Gateway is a stateless API gateway whose per-request work is bounded by design: signature verification against a cached JWKS, a counter increment, and a route lookup, with no datastore on the path. The declarative configuration model enables endpoint management without code changes, while the circuit breaker and rate limiting systems protect backend services from cascading failures. What that costs per request is not established here (Section~\ref{sec:figures}). The stateless architecture enables simple horizontal scaling---adding capacity requires only deploying additional replicas.
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{10}
|
||||
@@ -242,6 +243,9 @@ M. Jones et al. JSON Web Token (JWT). RFC 7519, 2015.
|
||||
\bibitem{sliding2019}
|
||||
K. Veeraraghavan et al. Sliding Window Rate Limiters. ACM SoCC, 2019.
|
||||
|
||||
\bibitem{clouddataplane}
|
||||
Hanzo AI Research. The Hanzo Cloud Data Plane: Benchmarking \texttt{zip}, ZAP, and the Road to a Million Requests per Second. Hanzo Industries, 2026.
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
+28
-19
@@ -45,7 +45,7 @@
|
||||
\hanzocoverpage
|
||||
|
||||
\begin{abstract}
|
||||
We present Hanzo IAM, a multi-tenant identity and access management platform that serves as the unified authentication layer for the Hanzo ecosystem. IAM supports OAuth~2.0, OpenID Connect, SAML~2.0, LDAP, and WebAuthn protocols, integrating with over 40 social identity providers. The system implements Casbin-based role-based access control (RBAC) with per-organization policy engines, SCIM~2.0 provisioning for enterprise directory synchronization, and TOTP/WebAuthn multi-factor authentication. IAM processes over 2 million authentication requests daily across 1,200+ tenant organizations with a median authentication latency of 18ms. All credentials are stored using Argon2id hashing with per-tenant salt derivation, and session tokens employ cryptographic binding to prevent token theft.
|
||||
We present Hanzo IAM, a multi-tenant identity and access management platform that serves as the unified authentication layer for the Hanzo ecosystem. IAM supports OAuth~2.0, OpenID Connect, SAML~2.0, LDAP, and WebAuthn protocols, integrating with over 40 social identity providers. The system implements Casbin-based role-based access control (RBAC) with per-organization policy engines, SCIM~2.0 provisioning for enterprise directory synchronization, and TOTP/WebAuthn multi-factor authentication. This paper describes the protocol surface, the policy model and the credential handling, and reports no measurements; Section~\ref{sec:figures} records the request volume, tenant count and authentication latency an earlier version reported and why they are withdrawn. All credentials are stored using Argon2id hashing with per-tenant salt derivation, and session tokens employ cryptographic binding to prevent token theft.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -160,7 +160,7 @@ m = r.sub == p.sub && r.org == p.org
|
||||
&& r.act == p.act
|
||||
\end{lstlisting}
|
||||
|
||||
Policies support role hierarchies, resource wildcards, and time-based constraints. Policy evaluation adds less than 0.1ms to request latency due to in-memory caching with write-through invalidation.
|
||||
Policies support role hierarchies, resource wildcards, and time-based constraints. Policy decisions are served from an in-memory cache with write-through invalidation, so an evaluation is a map lookup rather than a datastore round trip; the cost of that lookup is not measured here (Section~\ref{sec:figures}).
|
||||
|
||||
\subsection{SCIM Provisioning}
|
||||
|
||||
@@ -178,22 +178,28 @@ IAM is implemented in 38,000 lines of Go with zero CGo dependencies. The identit
|
||||
|
||||
\paragraph{Rate Limiting.} Authentication endpoints are rate-limited per IP and per account. Failed login attempts trigger exponential backoff: 1s, 2s, 4s, up to 5 minutes. After 10 consecutive failures, the account is locked for 30 minutes with email notification.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Authentication latency (ms) by protocol.}
|
||||
\label{tab:latency}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Protocol} & \textbf{p50} & \textbf{p99} \\
|
||||
\midrule
|
||||
OIDC token issue & 12 & 28 \\
|
||||
JWT validation & 0.3 & 0.8 \\
|
||||
SAML assertion & 18 & 45 \\
|
||||
WebAuthn verify & 8 & 22 \\
|
||||
RBAC check & 0.05 & 0.1 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\section{Status of the performance figures}
|
||||
\label{sec:figures}
|
||||
|
||||
An earlier version of this paper carried a table titled ``Authentication latency
|
||||
(ms) by protocol'' giving p50/p99 of 12/28\,ms for OIDC token issue, 0.3/0.8\,ms
|
||||
for JWT validation, 18/45\,ms for SAML assertion, 8/22\,ms for WebAuthn verify
|
||||
and 0.05/0.1\,ms for an RBAC check. The abstract restated these as a median
|
||||
authentication latency of 18\,ms over more than two million requests daily
|
||||
across 1{,}200 or more tenant organizations, and the policy section stated that
|
||||
evaluation adds less than 0.1\,ms.
|
||||
|
||||
None of it was measured. The table named no hardware, no load generator and no
|
||||
request count; \texttt{hanzoai/iam} contains no benchmark function and no load
|
||||
test; and no deployment record in this estate counts authentication requests or
|
||||
tenant organizations. The daily-volume and tenant figures in particular describe
|
||||
an operating history the service does not have, so they cannot be relabelled as
|
||||
targets and are removed.
|
||||
|
||||
The latency figures are the cheapest of these to establish honestly, because
|
||||
each protocol path is a function call away from a Go benchmark in the same
|
||||
repository. What is missing is the run and the machine it ran on, reported the
|
||||
way the Hanzo cloud data plane benchmark~\cite{clouddataplane} reports one.
|
||||
|
||||
\section{Security}
|
||||
\label{sec:security}
|
||||
@@ -214,7 +220,7 @@ By 2019 the limitations were clear: organizations wanted SSO with their existing
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Hanzo IAM provides a unified identity platform that combines authentication, authorization, and user management in a single service. By supporting the major identity protocols and 40+ social providers, IAM eliminates the need to manage multiple identity systems. The Casbin-based policy engine provides fine-grained access control, and native SCIM provisioning enables enterprise directory integration.
|
||||
Hanzo IAM is a unified identity platform that combines authentication, authorization, and user management in a single service. By supporting the major identity protocols and 40+ social providers, IAM removes the need to run several identity systems side by side. The Casbin-based policy engine provides fine-grained access control, and native SCIM provisioning enables enterprise directory integration. This paper reports no latency or volume measurements; Section~\ref{sec:figures} says which were withdrawn and what would replace them.
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{10}
|
||||
@@ -234,6 +240,9 @@ N. Sakimura et al. OpenID Connect Core 1.0. OpenID Foundation, 2014.
|
||||
\bibitem{webauthn2019}
|
||||
J. Hodges et al. Web Authentication: An API for accessing Public Key Credentials. W3C, 2019.
|
||||
|
||||
\bibitem{clouddataplane}
|
||||
Hanzo AI Research. The Hanzo Cloud Data Plane: Benchmarking \texttt{zip}, ZAP, and the Road to a Million Requests per Second. Hanzo Industries, 2026.
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Binary file not shown.
+22
-18
@@ -45,7 +45,7 @@
|
||||
\hanzocoverpage
|
||||
|
||||
\begin{abstract}
|
||||
We present Hanzo Ingress, a Kubernetes-native L7 reverse proxy built on Traefik that provides automatic TLS certificate management, a custom static file serving plugin, and multi-namespace routing for the Hanzo platform. Ingress replaces traditional web servers (nginx, Caddy) with a single ingress controller that handles TLS termination via Let's Encrypt, serves static assets from in-cluster volumes, and routes traffic to backend services using IngressRoute custom resources. The static file plugin eliminates the need for separate web server containers, reducing pod count by 40\% in typical deployments. We describe the middleware chain architecture, the certificate resolver pipeline, and operational patterns for zero-downtime deployments across 200+ services.
|
||||
We present Hanzo Ingress, a Kubernetes-native L7 reverse proxy built on Traefik that provides automatic TLS certificate management, a custom static file serving plugin, and multi-namespace routing for the Hanzo platform. Ingress replaces traditional web servers (nginx, Caddy) with a single ingress controller that handles TLS termination via Let's Encrypt, serves static assets from in-cluster volumes, and routes traffic to backend services using IngressRoute custom resources. The static file plugin removes the separate web-server container a static site would otherwise need, so a deployment that served assets from its own pod serves them from the ingress instead. By how much that lowers pod count across the platform is not measured here (Section~\ref{sec:figures}). We describe the middleware chain architecture, the certificate resolver pipeline, and operational patterns for zero-downtime deployments across 200+ services.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -169,24 +169,28 @@ Traefik's cross-namespace service references allow a single IngressRoute to rout
|
||||
|
||||
The static file plugin is implemented in Go as a Traefik middleware plugin (approximately 800 lines). It uses \texttt{http.FileServer} internally with custom logic for SPA fallback, cache headers, and compression.
|
||||
|
||||
\paragraph{Resource Usage.} Each Traefik instance consumes approximately 50MB of memory and 0.1 CPU cores at idle, scaling to 200MB and 0.5 cores under 10,000 concurrent connections.
|
||||
\paragraph{Resource Usage.} Memory and CPU scale with the number of live connections and the size of the routing table rather than with request rate, since neither TLS session state nor the static-file cache is per-request. Concrete figures for a named workload are not reported here (Section~\ref{sec:figures}).
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Latency overhead comparison (ms, p99).}
|
||||
\label{tab:compare}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Ingress} & \textbf{API Route} & \textbf{Static File} \\
|
||||
\midrule
|
||||
nginx + sidecar & 1.8 & 2.2 \\
|
||||
Caddy & 1.5 & 1.8 \\
|
||||
Hanzo Ingress & 1.2 & 0.8 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\section{Status of the performance figures}
|
||||
\label{sec:figures}
|
||||
|
||||
The static file plugin achieves lower latency than sidecar-based approaches because files are served directly from the ingress process without an additional network hop (Table~\ref{tab:compare}).
|
||||
An earlier version of this paper carried a table titled ``Latency overhead
|
||||
comparison (ms, p99)'' placing Hanzo Ingress at 1.2\,ms on an API route and
|
||||
0.8\,ms on a static file against 1.8/2.2\,ms for nginx with a sidecar and
|
||||
1.5/1.8\,ms for Caddy, and an abstract claiming a 40\% reduction in pod count.
|
||||
|
||||
None of it was measured. The comparison named no load generator, no hardware, no
|
||||
request count and no configuration for either competitor, which makes it a claim
|
||||
about two third-party products with nothing behind it --- the most exposed kind
|
||||
of number a paper can carry. The 40\% figure defined neither ``typical
|
||||
deployment'' nor a before-and-after pod inventory. Both are removed.
|
||||
|
||||
The structural argument the table was meant to support does not need it: serving
|
||||
a static asset from the ingress rather than from a pod behind it removes a
|
||||
network hop and a container from the path. That is a statement about the
|
||||
topology, and it holds whatever the measured difference turns out to be.
|
||||
|
||||
A sidecar-based approach pays for a second process and the hop to reach it on every static request. The plugin serves the file from the process that already terminated the connection, so neither cost appears.
|
||||
|
||||
\section{Security}
|
||||
|
||||
@@ -200,7 +204,7 @@ The static file plugin achieves lower latency than sidecar-based approaches beca
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Hanzo Ingress provides a unified ingress layer that eliminates the need for separate web server containers in Kubernetes deployments. The static file plugin serves SPAs and assets directly from the ingress controller, reducing pod count and operational complexity. Automatic TLS management, a composable middleware chain, and multi-namespace routing make it the single point of configuration for all external traffic entering the Hanzo platform.
|
||||
Hanzo Ingress provides a unified ingress layer that eliminates the need for separate web server containers in Kubernetes deployments. The static file plugin serves SPAs and assets directly from the ingress controller, which removes a container and a hop from the path an asset takes. Automatic TLS management, a composable middleware chain, and multi-namespace routing make it the single point of configuration for all external traffic entering the Hanzo platform.
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{10}
|
||||
|
||||
Binary file not shown.
@@ -45,7 +45,7 @@
|
||||
\hanzocoverpage
|
||||
|
||||
\begin{abstract}
|
||||
We present Hanzo Insights, a privacy-first product analytics platform built on the Umami framework with Hanzo Datastore---our Apache-2.0 ClickHouse fork---as the analytical backend. Insights provides real-time event tracking, funnel analysis, cohort retention, and feature flag management without collecting personally identifiable information (PII). The system processes over 50 million events daily with sub-second query latency on dashboards spanning 90-day windows. Key innovations include: (1) a cookie-less visitor identification scheme using salted hashing of request attributes, (2) a feature flag engine with percentage rollouts, user targeting, and A/B experiment support, and (3) a streaming ingestion pipeline with at-most-once deduplication. Insights replaces Google Analytics and LaunchDarkly in the Hanzo stack, eliminating third-party data sharing while providing equivalent functionality.
|
||||
We present Hanzo Insights, a privacy-first product analytics platform built on the Umami framework with Hanzo Datastore---our Apache-2.0 ClickHouse fork---as the analytical backend. Insights provides real-time event tracking, funnel analysis, cohort retention, and feature flag management without collecting personally identifiable information (PII). This paper describes the ingestion path, the identification scheme and the flag engine, and reports no measurements; Section~\ref{sec:figures} records the event volume and query latency an earlier version reported and why they are withdrawn. Key innovations include: (1) a cookie-less visitor identification scheme using salted hashing of request attributes, (2) a feature flag engine with percentage rollouts, user targeting, and A/B experiment support, and (3) a streaming ingestion pipeline with at-most-once deduplication. Insights replaces Google Analytics and LaunchDarkly in the Hanzo stack, eliminating third-party data sharing while providing equivalent functionality.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -176,22 +176,23 @@ The dashboard displays live metrics using Server-Sent Events (SSE). The Real-Tim
|
||||
|
||||
Insights is implemented as a Go service (18,000 lines) with a React dashboard (12,000 lines). The collection endpoint processes events in batches of 1,000, inserting into Datastore via the native protocol for maximum throughput.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Query latency on 90-day window (ms).}
|
||||
\label{tab:query}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Query} & \textbf{p50} & \textbf{p99} \\
|
||||
\midrule
|
||||
Dashboard overview & 45 & 180 \\
|
||||
Funnel (4 steps) & 120 & 450 \\
|
||||
Cohort retention & 200 & 800 \\
|
||||
Custom event count & 25 & 90 \\
|
||||
Feature flag impact & 150 & 520 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\section{Status of the performance figures}
|
||||
\label{sec:figures}
|
||||
|
||||
An earlier version of this paper carried a table titled ``Query latency on
|
||||
90-day window (ms)'' giving p50/p99 from 45/180\,ms on the dashboard overview to
|
||||
200/800\,ms on cohort retention, and an abstract stating that the system
|
||||
processes over 50 million events daily with sub-second query latency.
|
||||
|
||||
None of it was measured. The table named no cluster, no row count and no query
|
||||
log; there is no deployment record for Insights in this estate, so the daily
|
||||
event volume describes traffic that was never observed; and a latency figure for
|
||||
a 90-day window is meaningless without the number of rows in that window, which
|
||||
the paper never gives.
|
||||
|
||||
A query-latency table is straightforward to produce once a corpus is fixed:
|
||||
generate a known event volume, state the cluster, and report the same five
|
||||
queries with their row counts. That run does not exist yet.
|
||||
|
||||
\section{Security}
|
||||
|
||||
@@ -205,7 +206,7 @@ Feature flag impact & 150 & 520 \\
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Hanzo Insights demonstrates that privacy-first analytics need not sacrifice functionality. By combining cookie-less visitor identification with Datastore's columnar storage, Insights provides funnel analysis, cohort retention, and feature flag management with sub-second query latency---all without storing personally identifiable information. The integrated feature flag engine eliminates the need for a separate service, reducing infrastructure complexity while enabling data-driven product decisions.
|
||||
Hanzo Insights demonstrates that privacy-first analytics need not sacrifice functionality. By combining cookie-less visitor identification with Datastore's columnar storage, Insights provides funnel analysis, cohort retention and feature flag management without storing personally identifiable information. Its query latency is a function of the event volume in the window and the columnar layout underneath, and is not characterized here (Section~\ref{sec:figures}). The integrated feature flag engine eliminates the need for a separate service, reducing infrastructure complexity while enabling data-driven product decisions.
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{10}
|
||||
|
||||
Binary file not shown.
+22
-17
@@ -45,7 +45,7 @@
|
||||
\hanzocoverpage
|
||||
|
||||
\begin{abstract}
|
||||
We present Hanzo KMS, a secrets management platform built on the Infisical framework that provides centralized secret storage, per-environment rotation policies, and a comprehensive audit trail. KMS implements envelope encryption with AES-256-GCM, role-based access control scoped to projects and environments, automatic secret rotation for databases and API keys, and a Kubernetes operator that synchronizes secrets to cluster namespaces. The system manages over 12,000 secrets across 180 projects with zero plaintext secret exposure since deployment. We describe the encryption architecture, the rotation pipeline, and the Kubernetes integration that eliminates the need for secrets in configuration files or environment variables committed to source control.
|
||||
We present Hanzo KMS, a secrets management platform built on the Infisical framework that provides centralized secret storage, per-environment rotation policies, and a comprehensive audit trail. KMS implements envelope encryption with AES-256-GCM, role-based access control scoped to projects and environments, automatic secret rotation for databases and API keys, and a Kubernetes operator that synchronizes secrets to cluster namespaces. This paper describes the encryption architecture, the rotation pipeline and the Kubernetes integration, and reports no operating figures; Section~\ref{sec:figures} records the inventory and latency figures an earlier version reported and why they are withdrawn. We describe the encryption architecture, the rotation pipeline, and the Kubernetes integration that eliminates the need for secrets in configuration files or environment variables committed to source control.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -171,21 +171,26 @@ KMS is implemented as a Go service (22,000 lines) with a React admin dashboard.
|
||||
|
||||
\paragraph{Storage.} Secret metadata is stored in PostgreSQL. Encrypted secret values are stored separately in a dedicated secrets table with column-level encryption. The HSM interface supports AWS CloudHSM, YubiHSM, and software-based PKCS\#11 tokens.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Secret retrieval latency (ms).}
|
||||
\label{tab:kms-perf}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Operation} & \textbf{p50} & \textbf{p99} \\
|
||||
\midrule
|
||||
Get single secret & 3.2 & 8.1 \\
|
||||
Get project secrets & 12.4 & 28.3 \\
|
||||
Create/update secret & 8.7 & 22.1 \\
|
||||
Rotate credential & 450 & 1,200 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\section{Status of the operating figures}
|
||||
\label{sec:figures}
|
||||
|
||||
An earlier version of this paper carried a table titled ``Secret retrieval
|
||||
latency (ms)'' giving p50/p99 of 3.2/8.1\,ms to fetch one secret, 12.4/28.3\,ms
|
||||
for a project's secrets, 8.7/22.1\,ms to create or update one and
|
||||
450/1{,}200\,ms to rotate a credential, and an abstract reporting more than
|
||||
12{,}000 secrets across 180 projects with zero plaintext exposure since
|
||||
deployment.
|
||||
|
||||
None of it was measured. There is no benchmark function or load test in
|
||||
\texttt{hanzoai/kms}, and the table named no hardware, no datastore
|
||||
configuration and no sample count. The inventory figures count nothing that is
|
||||
recorded anywhere, and ``zero plaintext exposure since deployment'' is a
|
||||
security claim that would have to rest on an audit-log review; no such review
|
||||
exists. All are removed.
|
||||
|
||||
Retrieval latency is worth establishing here for a specific reason: KMS sits on
|
||||
the start-up path of every service that needs a secret, so its p99 is somebody
|
||||
else's cold start. That makes it a number to measure rather than to assert.
|
||||
|
||||
\section{Security}
|
||||
|
||||
@@ -201,7 +206,7 @@ Rotate credential & 450 & 1,200 \\
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Hanzo KMS eliminates the need for secrets in source control, environment files, or manually-created Kubernetes secrets. The envelope encryption model limits the impact of key compromise, automatic rotation reduces the window of credential exposure, and the comprehensive audit trail provides accountability for every secret access. The Kubernetes operator closes the delivery gap, ensuring that secrets flow from KMS to running applications without human intervention.
|
||||
Hanzo KMS removes the need for secrets in source control, environment files, or manually-created Kubernetes secrets. The envelope encryption model limits the impact of key compromise, automatic rotation reduces the window of credential exposure, and the comprehensive audit trail provides accountability for every secret access. The Kubernetes operator closes the delivery gap, ensuring that secrets flow from KMS to running applications without human intervention.
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{10}
|
||||
|
||||
Binary file not shown.
@@ -45,7 +45,7 @@
|
||||
\hanzocoverpage
|
||||
|
||||
\begin{abstract}
|
||||
We present Hanzo Ledger, a double-entry financial ledger that provides programmable transaction definitions via the Numscript domain-specific language (DSL), multi-currency support with automatic exchange rate resolution, and real-time balance computation with serializable isolation. Ledger maintains the fundamental accounting invariant---every transaction has balanced debits and credits---while supporting complex financial workflows such as split payments, escrow, revenue recognition, and multi-party settlements. The system processes 8,000 transactions per second with sub-5ms commit latency on PostgreSQL, maintaining zero balance discrepancies across 3 years of production operation. We describe the Numscript language, the transaction engine, and the reconciliation system.
|
||||
We present Hanzo Ledger, a double-entry financial ledger that provides programmable transaction definitions via the Numscript domain-specific language (DSL), multi-currency support with automatic exchange rate resolution, and real-time balance computation with serializable isolation. Ledger maintains the fundamental accounting invariant---every transaction has balanced debits and credits---while supporting complex financial workflows such as split payments, escrow, revenue recognition, and multi-party settlements. Section~\ref{sec:figures} reports transaction throughput and commit latency on PostgreSQL, together with the provenance those figures do and do not have. We describe the Numscript language, the transaction engine, and the reconciliation system.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -187,7 +187,7 @@ Ledger is implemented in 18,000 lines of Go. The Numscript compiler is a recursi
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Transaction throughput and latency.}
|
||||
\caption{Transaction throughput and latency. Not traceable to a recorded run; see Section~\ref{sec:figures}.}
|
||||
\label{tab:ledger-perf}
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
@@ -201,6 +201,28 @@ Balance query & 0.3 & 0.8 & 32,000 \\
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Status of the performance figures}
|
||||
\label{sec:figures}
|
||||
|
||||
The figures above are not traceable to a recorded run. The table names no
|
||||
hardware, no PostgreSQL configuration, no concurrency and no duration, and no
|
||||
result file in \texttt{hanzoai/ledger} contains these numbers.
|
||||
|
||||
What distinguishes them from an invented table is that the harness which would
|
||||
produce them is already in the repository: \texttt{test/performance/pkg/write}
|
||||
and \texttt{test/performance/pkg/read} are a parallelism-flagged TPS harness
|
||||
driven by two benchmark workflows in CI. The gap is the run and its
|
||||
declaration, not the mechanism.
|
||||
|
||||
One claim that appeared alongside these figures is withdrawn rather than
|
||||
qualified. An earlier version stated that the system maintained ``zero balance
|
||||
discrepancies across 3 years of production operation.'' No reconciliation
|
||||
record, incident log or monitoring export stands behind it, and there is no
|
||||
three-year operating history for this ledger to observe. The invariant the
|
||||
paper describes --- that debits equal credits within a transaction --- is
|
||||
enforced by the engine and is a property of the code; whether it has held
|
||||
across three years of live traffic is a separate, unevidenced claim.
|
||||
|
||||
\section{Security}
|
||||
|
||||
\paragraph{Immutable Audit Trail.} All transactions are append-only. Modifications are represented as new corrective transactions (reversals), never as mutations to existing records.
|
||||
@@ -213,7 +235,7 @@ Balance query & 0.3 & 0.8 & 32,000 \\
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Hanzo Ledger provides a financial data infrastructure that enforces the double-entry invariant at the system level, eliminating an entire class of accounting bugs. The Numscript DSL enables complex financial workflows---split payments, escrow, multi-currency conversion---to be expressed declaratively and executed atomically. Three years of production operation with zero balance discrepancies demonstrates the effectiveness of the approach.
|
||||
Hanzo Ledger provides a financial data infrastructure that enforces the double-entry invariant at the system level, eliminating an entire class of accounting bugs. The Numscript DSL enables complex financial workflows---split payments, escrow, multi-currency conversion---to be expressed declaratively and executed atomically. Enforcing the invariant in the engine rather than in each caller is what makes that class of bug unrepresentable; Section~\ref{sec:figures} states what this paper does and does not establish about the system's behaviour in operation.
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{10}
|
||||
|
||||
Binary file not shown.
+22
-2
@@ -45,7 +45,7 @@
|
||||
\hanzocoverpage
|
||||
|
||||
\begin{abstract}
|
||||
We present Hanzo PubSub, an event streaming platform built on NATS and JetStream that provides at-least-once delivery, dead letter queues, schema validation, and multi-tenant topic isolation for the Hanzo microservice ecosystem. PubSub replaces direct HTTP calls between services with asynchronous event streams, reducing coupling and enabling event-driven architectures. The system processes 120,000 events per second with a median publish-to-consume latency of 1.8ms. Key design decisions include: (1) tenant-prefixed subject namespaces for isolation without separate NATS clusters, (2) CloudEvents~\cite{cloudevents2019} as the canonical event envelope for cross-service interoperability, and (3) automatic dead letter routing with configurable retry policies. We describe the event schema design, the delivery guarantee implementation, and the operational patterns for managing event streams in a multi-tenant platform.
|
||||
We present Hanzo PubSub, an event streaming platform built on NATS and JetStream that provides at-least-once delivery, dead letter queues, schema validation, and multi-tenant topic isolation for the Hanzo microservice ecosystem. PubSub replaces direct HTTP calls between services with asynchronous event streams, reducing coupling and enabling event-driven architectures. Section~\ref{sec:figures} reports throughput and publish-to-consume latency by delivery mode, together with the provenance those figures do and do not have. Key design decisions include: (1) tenant-prefixed subject namespaces for isolation without separate NATS clusters, (2) CloudEvents~\cite{cloudevents2019} as the canonical event envelope for cross-service interoperability, and (3) automatic dead letter routing with configurable retry policies. We describe the event schema design, the delivery guarantee implementation, and the operational patterns for managing event streams in a multi-tenant platform.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -202,7 +202,7 @@ PubSub is deployed as a 3-node NATS cluster with JetStream enabled. The schema v
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Event processing performance.}
|
||||
\caption{Event processing by delivery mode. Not traceable to a recorded run; see Section~\ref{sec:figures}.}
|
||||
\label{tab:pubsub-perf}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
@@ -215,6 +215,26 @@ Effectively-once & 2.4 & 95,000 \\
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{Status of the performance figures}
|
||||
\label{sec:figures}
|
||||
|
||||
The figures above are not traceable to a recorded run. The table names no
|
||||
hardware, no message size, no client count and no duration, and no result file
|
||||
in \texttt{hanzoai/pubsub} contains these numbers.
|
||||
|
||||
What distinguishes them from an invented table is that the harness which would
|
||||
produce them is already in the repository: \texttt{hanzoai/pubsub} carries the
|
||||
upstream NATS benchmark suite --- \texttt{server/core\_benchmarks\_test.go},
|
||||
\texttt{server/jetstream\_benchmark\_test.go} and \texttt{test/bench\_test.go} ---
|
||||
which measures exactly publish-to-consume latency and throughput by delivery
|
||||
mode. The only committed results in that suite are upstream runs on 2015 and
|
||||
2017 iMacs, and they are not these numbers.
|
||||
|
||||
So the gap here is provenance, not mechanism. Until a run is cited, with the
|
||||
machine, the payload size and the client count declared, the table should be
|
||||
read as unverified: the ordering across delivery modes follows from what each
|
||||
mode has to do, while the absolute values stand on nothing.
|
||||
|
||||
\section{Security}
|
||||
|
||||
\paragraph{Authentication.} NATS connections require TLS client certificates issued by the Hanzo internal CA. Service identities are extracted from the certificate CN.
|
||||
|
||||
Binary file not shown.
+22
-18
@@ -45,7 +45,7 @@
|
||||
\hanzocoverpage
|
||||
|
||||
\begin{abstract}
|
||||
We present Hanzo Storage, an S3-compatible object storage gateway that provides per-tenant encryption, signed URL generation, quota management, and multi-backend support across AWS S3, Google Cloud Storage, and Cloudflare R2. Storage acts as a proxy layer that intercepts S3 API calls, enforces tenant isolation via Hanzo IAM, encrypts objects with per-tenant keys from Hanzo KMS, and routes requests to the configured backend. The system handles 15TB of stored objects across 400+ tenants with a median upload latency overhead of 8ms for encryption and policy enforcement. We describe the encryption pipeline, the quota enforcement mechanism, and the backend abstraction that enables transparent migration between storage providers.
|
||||
We present Hanzo Storage, an S3-compatible object storage gateway that provides per-tenant encryption, signed URL generation, quota management, and multi-backend support across AWS S3, Google Cloud Storage, and Cloudflare R2. Storage acts as a proxy layer that intercepts S3 API calls, enforces tenant isolation via Hanzo IAM, encrypts objects with per-tenant keys from Hanzo KMS, and routes requests to the configured backend. This paper describes the encryption pipeline, the quota mechanism and the backend abstraction, and reports no measurements; Section~\ref{sec:figures} records the stored volume, tenant count and latency overhead an earlier version reported and why they are withdrawn. We describe the encryption pipeline, the quota enforcement mechanism, and the backend abstraction that enables transparent migration between storage providers.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -179,23 +179,27 @@ Backends are configured per tenant, enabling cost optimization (R2 for egress-he
|
||||
|
||||
Storage is implemented in 12,000 lines of Go. The S3 API compatibility layer uses the \texttt{s3-gw} library, supporting the 12 operations that cover 98\% of real-world S3 API usage.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Upload latency overhead (ms) by object size.}
|
||||
\label{tab:upload}
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
\textbf{Object Size} & \textbf{Encrypt} & \textbf{Auth+Quota} & \textbf{Total} \\
|
||||
\midrule
|
||||
1KB & 0.1 & 2.0 & 2.1 \\
|
||||
1MB & 3.2 & 2.0 & 5.2 \\
|
||||
100MB & 45 & 2.0 & 47 \\
|
||||
1GB & 380 & 2.0 & 382 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\section{Status of the performance figures}
|
||||
\label{sec:figures}
|
||||
|
||||
Encryption overhead scales linearly with object size (Table~\ref{tab:upload}). For objects under 10MB (the majority of uploads), total overhead is under 10ms.
|
||||
An earlier version of this paper carried a table titled ``Upload latency
|
||||
overhead (ms) by object size'' attributing 0.1/2.0/2.1\,ms to a 1\,KB object and
|
||||
380/2.0/382\,ms to a 1\,GB object across encryption and authorization, and an
|
||||
abstract reporting 15\,TB stored across more than 400 tenants at a median
|
||||
overhead of 8\,ms.
|
||||
|
||||
None of it was measured. The table named no CPU, said nothing about whether
|
||||
AES-NI was available, and gave no sample count; the 8\,ms median in the abstract
|
||||
appears nowhere in the table; and no capacity or tenant inventory for this
|
||||
service exists in the estate, so the stored volume and tenant count describe an
|
||||
operating history that was never observed.
|
||||
|
||||
The encryption column is the one figure here that is cheap to establish and
|
||||
worth establishing, because it is the tax this gateway adds over talking to the
|
||||
backend directly: an AES-256-GCM throughput measurement on a named CPU, with and
|
||||
without AES-NI, answers it for every object size at once.
|
||||
|
||||
Encryption cost scales linearly with object size, while the authorization and quota check is a fixed cost per request. Below some object size the fixed cost dominates and the gateway is indistinguishable from the backend; above it the cipher does. Where that crossover falls depends on the CPU and is not measured here.
|
||||
|
||||
\section{Security}
|
||||
|
||||
@@ -209,7 +213,7 @@ Encryption overhead scales linearly with object size (Table~\ref{tab:upload}). F
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Hanzo Storage provides a multi-tenant object storage gateway that adds per-tenant encryption, quota management, and backend portability to standard S3-compatible storage. By operating as a proxy rather than a storage engine, Storage leverages the reliability and scalability of established cloud storage providers while adding the isolation and encryption guarantees required for a multi-tenant platform.
|
||||
Hanzo Storage provides a multi-tenant object storage gateway that adds per-tenant encryption, quota management, and backend portability to standard S3-compatible storage. By operating as a proxy rather than a storage engine, Storage inherits the durability of the backing provider while adding the isolation and encryption a multi-tenant platform requires. What the proxy costs on the upload path is not established here (Section~\ref{sec:figures}).
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{10}
|
||||
|
||||
Binary file not shown.
+31
-21
@@ -45,7 +45,7 @@
|
||||
\hanzocoverpage
|
||||
|
||||
\begin{abstract}
|
||||
We present Hanzo Vector, a high-performance vector search engine implemented in Rust, designed for retrieval-augmented generation (RAG) pipelines and semantic search in AI applications. Vector implements Hierarchical Navigable Small World (HNSW) graphs with configurable distance metrics (cosine, Euclidean, dot product), scalar and product quantization for memory efficiency, and filtered search with pre- and post-filtering strategies. The engine supports collections of up to 100 million vectors with 99\% recall at 1ms p95 latency for 1536-dimensional embeddings. Key innovations include: (1) a hybrid quantization scheme that selects scalar or product quantization per segment based on dimensionality, (2) filtered HNSW traversal that prunes the search graph using metadata predicates during navigation, and (3) a write-ahead log with snapshot isolation for concurrent reads and writes.
|
||||
We present Hanzo Vector, a high-performance vector search engine implemented in Rust, designed for retrieval-augmented generation (RAG) pipelines and semantic search in AI applications. Vector implements Hierarchical Navigable Small World (HNSW) graphs with configurable distance metrics (cosine, Euclidean, dot product), scalar and product quantization for memory efficiency, and filtered search with pre- and post-filtering strategies. The engine is designed for collections in the tens of millions of vectors at embedding dimensions around 1536. This paper describes that design and reports no measurements; Section~\ref{sec:figures} records the recall and latency figures an earlier version reported and why they are withdrawn. Key design elements are: (1) a hybrid quantization scheme that selects scalar or product quantization per segment based on dimensionality, (2) filtered HNSW traversal that prunes the search graph using metadata predicates during navigation, and (3) a write-ahead log with snapshot isolation for concurrent reads and writes.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
@@ -59,7 +59,7 @@ Hanzo Vector is designed as an embeddable Rust library with an optional gRPC ser
|
||||
\item An HNSW implementation with segment-level adaptive quantization selecting between scalar and product quantization based on dimensionality and recall requirements.
|
||||
\item Filtered HNSW traversal integrating metadata predicates into graph navigation, avoiding the recall degradation of post-filtering.
|
||||
\item A WAL-based storage engine with snapshot isolation enabling concurrent reads and writes without lock contention.
|
||||
\item Benchmarks demonstrating 99\% recall at 1ms p95 latency on 10M vectors with 1536 dimensions.
|
||||
\item A statement of the recall and latency properties the design is built for, and of what would have to be run to establish them (Section~\ref{sec:figures}).
|
||||
\end{itemize}
|
||||
|
||||
\section{Architecture}
|
||||
@@ -163,25 +163,32 @@ The WAL-based storage engine enables concurrent reads and writes. New vectors ar
|
||||
|
||||
Vector is implemented in 28,000 lines of Rust. The HNSW index uses \texttt{unsafe} Rust for SIMD-accelerated distance computations (AVX2 on x86, NEON on ARM). The gRPC API uses \texttt{tonic}; the HTTP API uses \texttt{axum}.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Search performance (10M vectors, 1536-dim).}
|
||||
\label{tab:search}
|
||||
\begin{tabular}{lrrr}
|
||||
\toprule
|
||||
\textbf{Config} & \textbf{Recall@10} & \textbf{p50 (ms)} & \textbf{p95 (ms)} \\
|
||||
\midrule
|
||||
Full precision & 99.5\% & 0.6 & 1.0 \\
|
||||
SQ (uint8) & 98.8\% & 0.3 & 0.6 \\
|
||||
PQ (m=96, k=256) & 97.2\% & 0.2 & 0.4 \\
|
||||
\midrule
|
||||
+ Filter (10\%) & 99.1\% & 0.8 & 1.4 \\
|
||||
+ Filter (1\%) & 98.5\% & 1.2 & 2.1 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
\paragraph{Memory Usage.} At 10M vectors of 1536 dimensions, full-precision storage is 61\,GB and scalar quantization to \texttt{uint8} is 15\,GB; both follow from the element width. Product quantization at $m=96$, $k=256$ stores 96 bytes per vector, or 0.96\,GB for the codes plus the codebooks.
|
||||
|
||||
\paragraph{Memory Usage.} Full precision: 61GB for 10M$\times$1536. SQ: 15GB. PQ: 2.4GB.
|
||||
\section{Status of the performance figures}
|
||||
\label{sec:figures}
|
||||
|
||||
An earlier version of this paper carried a table titled ``Search performance
|
||||
(10M vectors, 1536-dim)'' reporting recall@10 of 99.5\% at 0.6/1.0\,ms p50/p95
|
||||
for full precision, 98.8\% at 0.3/0.6\,ms under scalar quantization and 97.2\%
|
||||
at 0.2/0.4\,ms under product quantization, with two filtered rows. The abstract
|
||||
and conclusion restated the same figures as 99\% recall at 1\,ms p95 on
|
||||
collections of up to 100 million vectors, and the contributions list called them
|
||||
benchmarks.
|
||||
|
||||
Nothing produced them. The table named no hardware, no query count, and no
|
||||
recall ground truth; there is no benchmark harness in \texttt{hanzoai/vector},
|
||||
no ANN benchmark configuration, and no dataset --- SIFT, GIST, DEEP1B or
|
||||
otherwise --- against which recall could have been computed. The figures are
|
||||
removed rather than relabelled as targets, because a recall--latency pair is
|
||||
only meaningful against a stated dataset, index build and query set, and this
|
||||
paper states none of them.
|
||||
|
||||
Establishing them is a bounded piece of work: fix a public dataset and query
|
||||
set, build the index at declared \texttt{ef\_construction} and $M$, sweep
|
||||
\texttt{ef\_search}, and report the recall--latency curve with the machine
|
||||
named. The house convention for reporting such a run is set out in the Hanzo
|
||||
cloud data plane benchmark~\cite{clouddataplane}.
|
||||
|
||||
\section{Security}
|
||||
|
||||
@@ -195,7 +202,7 @@ PQ (m=96, k=256) & 97.2\% & 0.2 & 0.4 \\
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
Hanzo Vector provides an embeddable, high-performance vector search engine designed for AI workloads. Adaptive quantization, filtered HNSW traversal, and WAL-based storage enable 99\% recall at sub-millisecond latency on collections of tens of millions of vectors. The Rust implementation ensures predictable performance without garbage collection pauses, making Vector suitable for latency-sensitive RAG pipelines.
|
||||
Hanzo Vector is an embeddable vector search engine designed for AI workloads. Adaptive quantization, filtered HNSW traversal and WAL-based storage are the three mechanisms it combines, and each addresses a specific cost: memory per vector, recall loss under restrictive filters, and lock contention between readers and writers. What the combination is worth in recall and latency is not established here; Section~\ref{sec:figures} says what would establish it.
|
||||
|
||||
\bibliographystyle{plain}
|
||||
\begin{thebibliography}{10}
|
||||
@@ -212,6 +219,9 @@ Qdrant. Vector Search Engine. 2021.
|
||||
\bibitem{jegou2011}
|
||||
H. J\'{e}gou et al. Product Quantization for Nearest Neighbor Search. IEEE TPAMI, 2011.
|
||||
|
||||
\bibitem{clouddataplane}
|
||||
Hanzo AI Research. The Hanzo Cloud Data Plane: Benchmarking \texttt{zip}, ZAP, and the Road to a Million Requests per Second. Hanzo Industries, 2026.
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
|
||||
Reference in New Issue
Block a user