papers/empirical-benchmark/main.tex

\documentclass[11pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath, amsthm, amssymb}
\usepackage{booktabs}
\usepackage{listings}
\usepackage{xcolor}
\usepackage{hyperref}

\hypersetup{
  colorlinks=true,
  linkcolor=blue,
  urlcolor=blue,
}

\title{\texorpdfstring{ZAP and HTTP/1.1 on Loopback: \\
\large An Allocation and Wire-Size Baseline, and a Correction}%
{ZAP and HTTP/1.1 on Loopback: An Allocation and Wire-Size Baseline, and a Correction}}

\author{ZAP Protocol Authors\\\href{https://zap-proto.io}{zap-proto.io}}

\date{2026-09-10}

% =====================================================================
% Every measured number below comes from results.tex, which emit.py
% writes from bench-results.txt. main.tex holds no measured literal, so
% a figure cannot drift from the harness output and a column cannot
% exist without a line behind it -- which is the defect \S1 is about.
%   ./emit.py ../../bench/bench-results.txt
% =====================================================================
\input{results}

\begin{document}

\maketitle

\begin{abstract}
We measure the ZAP wire against HTTP/1.1 on a loopback echo workload at \nWorkloads{}
payload sizes. Five arms answer the identical request: Go's \texttt{net/http},
\texttt{fasthttp} over HTTP/1.1, ZAP-HTTP (\texttt{fasthttp} objects on the ZAP
wire), a native ZAP typed message built with the published
\texttt{zap-proto/go} runtime, and a length-prefixed byte echo that bounds the
others from below. The \texttt{fasthttp} arm is what makes the result readable:
its object model is far cheaper than \texttt{net/http}'s on \emph{any} wire, so a
ZAP-against-\texttt{net/http} ratio moves for two reasons at once. Held against
that matched baseline, ZAP-HTTP is allocation-free in steady state and allocates
fewer bytes than \texttt{fasthttp} at every size (\zapTiny{} vs.\ \fastTiny{}
B/req at 16-byte bodies, \zapLarge{} vs.\ \fastLarge{} at 64\,KiB), and its frame
is $\wireZapLo$--$\wireZapHi$\% larger than
the equivalent HTTP/1.1 request rather than the 86\% larger the previous revision
of this paper measured. The native arm is the correction. This paper previously
reported that native ZAP allocates exactly its payload in exactly one allocation
per request, up to $622\times$ less memory than \texttt{net/http}; those figures
came from an arm with no stored output whose implementation was a bespoke TCP byte
echo importing nothing from any ZAP package. Rebuilt on \texttt{zap-proto/go}
v1.8.3 and re-measured, a native round trip allocates \emph{six} objects and
two heap copies of the payload, because \texttt{ObjectBuilder.SetBytes}
heap-copies the payload before copying it into the message buffer. At 64\,KiB that
is \natLarge{} B/req against \texttt{fasthttp}'s \fastLarge.
\ifconcMeasured%
Throughput was measured at a load average of \loadLo--\loadHi{} across ten
cores, where one arm's sequential rate varies by a factor of \spreadFactor{}
between repetitions of the identical loop. Under concurrency both ZAP arms beat
\texttt{net/http} by $\zapVsStdLo$--$\zapVsStdHi\times$ in every paired
repetition; against \texttt{fasthttp} the ratio brackets $1.0$ in both
directions, and this machine cannot separate them.%
\else%
The stored run reports \emph{no} throughput: the harness refuses to time
anything on a machine whose load average exceeds its core count, and this one
began at \loadStart{} on ten cores. The sequential rates it does print vary by a
factor of \spreadFactor{} between repetitions of the identical loop, which is
what that refusal is for. We print no rate rather than one taken under that.%
\fi{} Every number here is a line in
\texttt{bench-results.txt} at \href{https://github.com/zap-proto/bench}{zap-proto/bench};
this paper's tables and quoted figures are generated from that file rather than
typed, so a column cannot outlive the output behind it.
\end{abstract}

\section{What this paper previously claimed, and why it changed}
\label{sec:correction}

The previous revision reported that native ZAP allocates ``exactly what it
sends'' --- a 16-byte payload producing 16 bytes of allocation --- in exactly one
allocation per request, for $622\times$ less memory and $115\times$ fewer
allocations than \texttt{net/http}, at $1.85$--$3.68\times$ its throughput. Three
things were wrong with that, and it is worth being precise about which.

\paragraph{The arm had no stored output.} The harness commits its raw output to
\texttt{bench-results.txt}. That file was written by the initial commit
(\texttt{b468d54}) at 20:33:54; the native arm arrived eight minutes later
(\texttt{a9766c4}, ``native-ZAP RPC bench + memory pressure test'') at 20:41:04,
and the file was never regenerated. Every number in the stored output came from a
two-arm run --- \texttt{HTTP/1.1+JSON} and \texttt{ZAP-binary} --- in which the
string \emph{Native} does not appear. The third column of every table in the
previous revision had no line behind it. \emph{(Hashes are as of this writing;
that history has been rewritten once, so the commit messages and times identify
them more durably than the hashes do.)}

\paragraph{The arm was not ZAP.} More seriously, the code that would have produced
those numbers was a hand-written TCP server that read a 4-byte length prefix and
wrote the same bytes back without looking at them. It imported no ZAP package and
carried no message format, no schema, and no headers, while the HTTP arm it was
compared against constructed a request, parsed a header set, and returned a
JSON-shaped body. The paper described this arm as ``length-prefixed frames,
schema-typed bodies''. It had no typed field of any kind. Re-running it would have
produced numbers, but they would have measured the cost of copying bytes, not the
cost of a protocol. $622$ is $9954/16$: an HTTP allocation floor divided by a
payload size, which is arithmetic on the workload rather than a property of
anything.

\paragraph{And for a long time the harness did not compile.} \texttt{zaphttp}
moved from \texttt{net/http} to \texttt{fasthttp} objects, \texttt{NewTransport}
became \texttt{Dial}, and \texttt{MarshalRequest} changed signature. Checked out at
the commit contemporaneous with the published numbers, the harness still fails to
build against today's \texttt{zap-proto/http}:
\begin{quote}\ttfamily\small
cannot use http.HandlerFunc(echoHandler)\\
as fasthttp.RequestHandler
\end{quote}
A benchmark that does
not build is a benchmark nobody can check, which is how the first two problems
survived. A later commit did migrate it to \texttt{fasthttp} and made the two HTTP
arms share one handler --- but it kept the byte echo as the native arm, so the
second problem survived the migration that fixed the third.

We therefore finished the job rather than deleting the claim. The native arm is
now a real ZAP message built with the published \texttt{zap-proto/go} runtime
(\S\ref{sec:arms}); a \texttt{net/http} arm was restored, because that is the
baseline the published claim was made against; a matched \texttt{fasthttp} arm
sits between them so the transport comparison is not confounded by the object
model; and everything was re-measured. The results that follow are worse for the
native arm than what was published and better for the adapter. We keep the byte
echo as an explicitly labelled lower bound, because the gap between it and the
native arm is exactly the quantity the previous revision hid.

\section{Five arms}
\label{sec:arms}

\begin{description}
  \item[\texttt{net/http}] Go stdlib client and server, keep-alive, connection
    pooling. The everywhere baseline, and the only arm whose library offers no
    request pooling: it builds a fresh \texttt{http.Request} per call because
    there is no other way to use it.
  \item[\texttt{fasthttp}] \texttt{fasthttp} client and server over HTTP/1.1,
    requests and responses acquired from and released to its pools. Same object
    model as the ZAP-HTTP arm, same wire as \texttt{net/http}. This arm exists so
    that the difference between it and ZAP-HTTP is the wire and nothing else.
  \item[ZAP-HTTP] \texttt{zap-proto/http}: HTTP request and response semantics on
    the ZAP wire, driving the identical \texttt{fasthttp} handler.
  \item[Native ZAP] A typed \texttt{zap-proto/go} v1.8.3 message carrying the body
    as a \texttt{bytes} field, the header set as a variable-entry list, and a
    status word, over 4-byte length-prefixed frames. The server parses the
    request, reads the body and every header, and builds a response echoing the
    body and the \texttt{X-} headers --- the same work the HTTP handler does.
    Reads are zero-copy views into the frame; each end reuses one builder per
    connection.
  \item[Floor] The same framing carrying raw bytes, echoed without being read.
    No message format, no schema, no headers, no dispatch. This is a lower bound
    on transport cost and is not a protocol; no number from this arm describes
    ZAP.
\end{description}

\section{Methodology}

\paragraph{Hardware and conditions.} Apple M1 Max, 10 cores, macOS Darwin 25.6.0,
Go 1.26.5, loopback only. \textbf{The machine was not quiet}: \texttt{run.sh}
records \texttt{uptime} before and after each run, and the stored run began at a
one-minute load average of \loadStart{} and ended at \loadEnd, on ten cores. Allocation
counts and wire sizes are counts and do not move with load --- they repeat across
three repetitions, and the \texttt{net/http} allocation count reproduces the May
2026 run on the same machine to within 0.1\% (115.4 vs.\ 115.3 allocations per
request at the 16-byte workload). Throughput is a rate and does move. The widest
swing in the stored run is the \spreadArm{} arm on the \spreadWorkload{} body:
\spreadValues{} \texttt{req/s} across \nReps{} repetitions of the identical loop,
a factor of \spreadFactor. We report the rates, we
report the load they were taken under, and we draw no conclusion from the
sequential ones.

\paragraph{Workloads.} \NWorkloads{} bodies: tiny (16\,B, 4 headers), small
(256\,B, 8), chat (\bodyChat\,B of chat-completion-shaped JSON, 12), medium
(4\,KiB, 12), large (64\,KiB, 8). Echo handler, warm connection. The chat body is the shape the
transport is actually deployed under; the other four span the range around it.

\paragraph{Driver.} 50 warm-up requests, two forced \texttt{runtime.GC()}, then
\memReqs{} requests with \texttt{MemStats} snapshotted around the window; \nReps{}
repetitions, medians reported. That snapshot is process-wide, so every figure
covers the client and the server together --- a round trip, not one end.

\paragraph{Fairness, and one asymmetry we did not remove.} Every arm ships the
same body and the same header set over a warm connection and consumes the whole
response. Each client is driven the way its library is meant to be driven, which
is itself part of what is measured. One asymmetry is deliberate: the native arm
flattens its header set to bytes once, outside the request loop, while the HTTP
arms walk a string-keyed map on every call. A typed protocol has no such map, and
charging it for one would be charging it for a data structure it does not have.

\paragraph{Reproduce.}
\begin{lstlisting}[basicstyle=\ttfamily\small,frame=single,backgroundcolor=\color{gray!10}]
git clone https://github.com/zap-proto/bench
cd bench && ./run.sh > bench-results.txt 2>&1
\end{lstlisting}
The file in the repository is that command's complete output, machine line
included. A claim with no matching line in it has no evidence behind it.

\section{Results}

\subsection{Bytes allocated per request}

\begin{table}[ht]
\centering
\begin{tabular}{lrrrrr}
\toprule
Workload & \texttt{net/http} & \texttt{fasthttp} & ZAP-HTTP & Native ZAP & Floor \\
\midrule
\allocBytesRows
\bottomrule
\end{tabular}
\caption{Bytes allocated per round trip, \texttt{MemStats.TotalAlloc} delta over
\memReqs{} requests, median of \nReps. \texttt{net/http} carries a $\approx$10\,KiB
floor independent of payload; the two pooled HTTP-object arms carry effectively
none; the native arm carries twice the payload (\S\ref{sec:builder}).}
\end{table}

Two readings. Against \texttt{net/http}, ZAP-HTTP's advantage is enormous and
mostly uninformative: dividing \stdTiny{} by \zapTiny{} gives a four-digit ratio whose
denominator is a rounding artifact, which is the same mistake $622\times$ was.
Report the absolutes. Against the matched \texttt{fasthttp} baseline the honest
statement is small and solid: ZAP-HTTP allocates fewer bytes per request than
\texttt{fasthttp} on the identical handler at every size --- \zapTiny{} against
\fastTiny{} at 16 bytes, \zapLarge{} against \fastLarge{} at 64\,KiB. Both are allocation-free in steady state; the
difference is residual, and the wire is not costing anything.

\subsection{Allocations per request}

\begin{table}[ht]
\centering
\begin{tabular}{lrrrrr}
\toprule
Workload & \texttt{net/http} & \texttt{fasthttp} & ZAP-HTTP & Native ZAP & Floor \\
\midrule
\allocCountRows
\bottomrule
\end{tabular}
\caption{Heap allocations per round trip, median of \nReps; identical in all \nReps
repetitions to one decimal place.}
\end{table}

\texttt{net/http} makes \allocsTiny--\allocsMedium{} allocations per request and
the count barely depends on payload size: it is the cost of the request object, the URL, the header
map, and the body reader. The native arm makes six, against the one this paper
previously claimed.

The zeros want reading carefully. The harness reports one decimal place, so a
\texttt{fasthttp} or ZAP-HTTP zero means \emph{below 0.05 allocations per
request} --- one allocation every twenty or more requests, not none ever. The
residual bytes in the previous table say the same thing: a few bytes per request
amortized over \memReqs{} is a handful of allocations across the whole window, not a
per-request cost. ``Allocation-free in steady state'' is the accurate claim;
``zero allocations, ever'' is not, and a table with a decimal place in it should
not be read as making the second one.

\subsection{Garbage collection over \texorpdfstring{\memReqs}{5000} requests}

\begin{table}[ht]
\centering
\begin{tabular}{lrrrrr}
\toprule
Workload & \texttt{net/http} & \texttt{fasthttp} & ZAP-HTTP & Native ZAP & Floor \\
\midrule
\gcRows
\bottomrule
\end{tabular}
\caption{GC cycles during the measured window, median of \nReps.}
\end{table}

The zero column is the operationally interesting one. Allocation rate drives GC
frequency and GC frequency drives tail latency under load, and both pooled HTTP
arms reach zero cycles at every payload size. The native arm does not: at 64\,KiB
it triggers a third as many cycles as \texttt{net/http}, not none.

\subsection{Wire bytes}

\begin{table}[ht]
\centering
\begin{tabular}{lrrr}
\toprule
Workload & HTTP/1.1 & ZAP-HTTP & Native ZAP \\
\midrule
\wireRows
\bottomrule
\end{tabular}
\caption{One serialized request, in bytes. Percentages are against the HTTP/1.1
column.}
\end{table}

ZAP is not a wire-size win on this workload. The ZAP-HTTP frame is larger than the
equivalent HTTP/1.1 request at every size --- $\wireZapLo$ to $\wireZapHi$\%,
worst where the headers dominate and converging to parity as the body does. The
native message is the only one that ever comes out smaller, and only at the
16-byte workload ($\wireNatLo$\%), where it drops the request line and the header
names it does not need; above that it runs up to $\wireNatHi$\% larger, because
each header entry carries a 4-byte length prefix where HTTP spends a colon and a
newline. On the one realistic body here --- the \bodyChat-byte chat payload ---
ZAP-HTTP costs \wireZapChat{} extra bytes on the wire and native ZAP
\wireNatChat.

This is a substantial improvement on the previous revision, which measured the
ZAP-HTTP frame at 452 bytes against HTTP's 243 --- 86\% larger --- and printed it
as ``0.54x ($-86.0\%$ smaller)''. The codec has tightened from $+86\%$ to
$+11.9\%$ at the same workload. The reporting bug is fixed too: the harness now
names the direction, so a ratio cannot be read backwards.

\subsection{Throughput}

\ifconcMeasured
\begin{table}[ht]
\centering
\begin{tabular}{lrrrr}
\toprule
Workload & \texttt{net/http} & \texttt{fasthttp} & ZAP-HTTP & Native ZAP \\
\midrule
\concRows
\bottomrule
\end{tabular}
\caption{Requests per second, \nWorkers{} concurrent workers, \concReqs{}
requests, median of \nReps{} repetitions. Taken at a load average between
\loadLo{} and \loadHi{} on ten cores; the absolute rates are not a statement
about the hardware, and neither are the margins.}
\end{table}

The repetitions are what make this table readable, and they separate one claim we
will defend from one we will not. Comparing each arm against \texttt{net/http}
\emph{within} the same repetition, so both share the load of that moment, both ZAP
arms clear it in every pairing, by $\zapVsStdLo$ to $\zapVsStdHi\times$. That gap
never closes and never inverts, and it is what a stdlib service would see if it
moved.

Against the matched \texttt{fasthttp} baseline there is no result. The paired
ratios are \pairedZapSmall{} at 256\,B and \pairedZapMedium{} at 4\,KiB for
ZAP-HTTP, and \pairedNatSmall{} and \pairedNatMedium{} for the native arm. Both
brackets straddle $1.0$ in both directions, so the wire and the object model
cannot be separated on this machine: whatever ZAP-HTTP saves in allocation does
not show up as throughput against a client that was already allocation-free.
Reporting a median here would manufacture a winner out of three numbers that
disagree about who won.

\else
\textbf{The stored run measured no throughput.} The harness refuses to time
anything on a machine whose load average exceeds its core count, and this run
began at \loadStart{} on ten cores, so both rate sections read \emph{NOT
MEASURED} and there is no table here. That is the correct behaviour and the
reason the section is empty rather than populated from an older run: a rate
measured under that much contention describes the contention.

What a quiet run of this harness has shown, and what we would expect it to show
again: both ZAP arms clear \texttt{net/http} by several times in every paired
repetition under concurrency, while against the matched \texttt{fasthttp}
baseline the paired ratios straddle $1.0$ in both directions. The first is a
gap an allocation profile predicts and that no repetition inverted; the second
is not a result. Neither is quoted as a number here, because this run has none.
\fi

We make no claim at all from the sequential rates either. The previous revision's
$1.85$--$3.68\times$ throughput claim, and its bandwidth table (1\,233\,MB/s
native against HTTP's 519), are neither confirmed nor refuted here; they
are unmeasurable on this machine in this state. We withdraw them rather than
restate them from output that cannot carry them, and they want a re-run on an idle
host.

\section{Where the native arm's allocations are}
\label{sec:builder}

The native figures are not noise and they have an exact form. Writing $B$ for the
body size, the measured bytes per round trip are
\[
  \natTiny = 2B + \natKTiny,\quad \natSmall = 2B + \natKSmall,\quad
  \natMedium = 2B + \natKMedium,\quad \natLarge = 2B + \natKLarge
\]
at $B = \bodyTiny, \bodySmall, \bodyMedium, \bodyLarge$ --- two copies of the
payload plus a fixed \natKTiny{} bytes of builder scaffolding, at every size, in
six allocations. (At 64\,KiB the constant reads \natKTiny{} to \natKLarge{} across
repetitions; the other three repeat to the byte.)

The chat workload confirms the mechanism rather than breaking it. That body is
\bodyChat{} bytes and the measured figure is \natChat, not $2(\bodyChat) +
\natKTiny$. The excess over the constant is \natKChat{} rather than \natKTiny,
and the difference between them is twice the gap from \bodyChat{} to 896 --- Go's
next slice size class. The copy is a slice allocation, so it rounds up: the
relation is two \emph{allocations} of the payload, not two copies of exactly $B$
bytes. Two of them, for a round trip with two encodes, is one heap copy per
encode.

The source says the same thing. \texttt{ObjectBuilder.SetBytes} in
\texttt{zap-proto/go} v1.8.3 defers a variable-length field by storing
\texttt{append([]byte(nil), v...)} --- a fresh heap allocation holding the whole
payload --- and \texttt{Finish} then copies that buffer a second time into the
message. The builder's own buffer is reused across requests and does not grow
after warm-up; the per-request cost is the deferred copy.

The floor arm settles what this is not. It runs the same framing over the same
buffered connection and reports 0.0 bytes per request at every size, on the same
one-decimal scale that makes the pooled HTTP arms read as zero. So the $2B$ is the
message format's encode path, not the framing and not the socket.

The fix is mechanical rather than structural --- append directly into the
builder's buffer and patch the offset afterwards, which is what the deferred entry
already does one step later --- and it is in neither v1.1.0 nor the current v1.8.3.
We first measured this against v1.1.0 and re-measured against v1.8.3 when the
harness was bumped: $2B + 192$ and six allocations, to the byte, on both. It is a
property of the released runtime rather than of a stale tag, and an independent
measurement of the same code path in a third tag reports it too~\cite{cloudbench}.

\section{What this paper does not measure}

\paragraph{Authentication and post-quantum transport.} The benchmark runs without
TLS or any handshake. ZAP's hybrid KEM~\cite{pqhybrid} is a connection-setup cost
amortized across a warm connection, but we have not measured it, and nothing here
is a statement about it.

\paragraph{A quiet machine.} Everything in \S\ref{sec:correction} about the
previous revision applies to us if we report a rate taken at load \loadHi{} as though it
were a hardware capability. The allocation and wire columns stand on their own;
the throughput columns want a re-run on an idle host and should be read as
provisional until they get one.

\paragraph{A real network.} Loopback removes packet loss, the NIC, and the
kernel's receive path. Allocation counts are indifferent to all three; throughput
is not, and on a saturated link the codec stops being the bottleneck at
all~\cite{cloudbench}.

\paragraph{Multiplexed streams.} Sequential single-connection RPC only. The case
where a binary transport has structural room that HTTP/1.1 does not --- thousands
of logical streams over one connection --- is \texttt{zap-proto/ws}, and is not
exercised here.

\section{Conclusion}

On the identical handler and against a baseline with the same object model,
ZAP-HTTP is allocation-free per request, allocates fewer bytes than
\texttt{fasthttp} at every payload size, triggers no garbage collection over \memReqs
requests, and carries a frame $\wireZapLo$--$\wireZapHi$\% larger than the
equivalent HTTP/1.1 request. Against Go's \texttt{net/http} it removes a
10\,KiB, \allocsTiny-allocation
per-request floor\ifconcMeasured{} and serves
$\zapVsStdLo$--$\zapVsStdHi\times$ more requests in every paired repetition under
contention\fi. That floor is the number a stdlib service would actually see, and
it is worth more than the four-digit ratio the floor produces when divided by a
rounding artifact. Against \texttt{fasthttp}, which already has no such floor,
the throughput margin is not resolvable on this machine and we do not claim one.

The native path is where this paper was wrong and is now worse than advertised.
Built on the published runtime it allocates six objects and two heap copies of the
payload per round trip, not one object and the payload exactly, and against
\texttt{fasthttp} its throughput straddles parity in both directions. The $622\times$, $115\times$
and $1.85$--$3.68\times$ figures this paper carried are withdrawn: the first two
were arithmetic on a payload size against an arm that was not ZAP, and the third
was never measurable from the output on disk. What replaces them is smaller, and
it is in the file.

\bibliographystyle{plain}
\begin{thebibliography}{9}
\bibitem{zerocopy} ZAP Protocol Authors. \emph{Zero-Copy Throughput
of the ZAP Wire Format}. zap-proto/papers, 2026.
\href{https://github.com/zap-proto/papers/tree/main/zero-copy-throughput}
{zap-proto/papers/zero-copy-throughput}.

\bibitem{pqhybrid} ZAP Protocol Authors. \emph{Z-Wing: A Triple-Hybrid KEM for
the ZAP Handshake}. zap-proto/papers, 2026.
\href{https://github.com/zap-proto/papers/tree/main/pq-hybrid-kem}
{zap-proto/papers/pq-hybrid-kem}.

\bibitem{cloudbench} Hanzo AI Research. \emph{The Hanzo Cloud Data Plane:
Benchmarking \texttt{zip}, ZAP, and the Road to a Million Requests per Second}.
2026. Independently reports the same deferred-tail copy in a later
\texttt{zap-proto/go} tag, and reports both transports tying once a 2.5\,GbE link
saturates.

\bibitem{rfc9110} R. Fielding, M. Nottingham, and J. Reschke.
\emph{HTTP Semantics}. RFC 9110, June 2022.
\end{thebibliography}

\end{document}