mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-29 07:34:31 +00:00
zeekayandHanzo Dev
1d1e7be5ab
Both report benchmarks that already live in hanzoai/cloud under bench/, so the numbers have a script behind them rather than a spreadsheet. hanzo-dormant-agents takes apart the estimate that costs an agent as a process. State, execution and isolation are separable and differ by five orders of magnitude: a dormant agent is 477 bytes on disk and resumes in 0.034 ms, a parked loop is 601 bytes of heap, and the isolation boundary — the expensive one — is only owed by an agent actually running. The paper argues the container is worth its price where it is needed and says so; the claim is about which agents are charged for it. hanzo-multi-hop-retrieval is a negative result. On LoCoMo a dense retriever finds every required turn for 78.2% of single-hop questions and 22.7% of multi-hop ones, but finds at least one required turn about 80% of the time in both — so it understands the questions and cannot reach all of the answer. Four retrieval policies over one fixed index move that by at most 0.7 points and two make it worse. A diagnostic says why it was never going to work: 95.4% of multi-hop evidence spans sessions and the median worst-ranked gold turn sits at rank 67, so the missing signal is linkage rather than similarity. Publishing the dead end is the useful part. Each paper carries results-data.tex, generated from the runs, and prints XXX in place of any constant that file does not supply — a number cannot drift from its measurement without the drift showing on the page. Both compile under the CI gate's flags, -Werror included. INDEX.md is regenerated by make index, which also picks up the papers added since it was last written and the header its current generator emits. Co-authored-by: Hanzo Dev <[email protected]>
58 lines
2.9 KiB
TeX
58 lines
2.9 KiB
TeX
\section{Method}
|
|
\label{sec:method}
|
|
|
|
\subsection{Host and provenance}
|
|
|
|
Every measurement in this paper was taken on one machine: an Apple M-series
|
|
laptop, \texttt{darwin/arm64}, 16 cores, Go 1.27.1, Node 26.5.0, on 2026-09-07.
|
|
A laptop is a deliberate choice. The claim under test is that a fleet of a
|
|
million agents does not require special hardware, and a result taken on special
|
|
hardware could not support it.
|
|
|
|
Numbers are labelled throughout. \meas{} was produced by a named script on that
|
|
host. \modeled{} was computed from measured constants and stated assumptions.
|
|
\cited{} came from a published source, with the source given. No number in this
|
|
paper is any other kind.
|
|
|
|
Each measured constant reaches the text through a macro defined in
|
|
\texttt{results-data.tex}, which is generated from the runs. If that file is
|
|
absent the paper prints \ph{} rather than a stale literal, so a number cannot
|
|
drift away from the run that produced it without the drift being visible on the
|
|
page.
|
|
|
|
\subsection{What each benchmark does}
|
|
|
|
The four scripts live in \texttt{hanzoai/cloud} under \texttt{bench/} and each
|
|
re-runs to reproduce its table.
|
|
|
|
\begin{description}[leftmargin=1.2cm,style=nextline]
|
|
\item[\texttt{fleet/fleet.mjs}] writes \rFleetAgents{} dormant agent records
|
|
into SQLite through the production schema (100{,}000 tenants $\times$ 10
|
|
agents), then measures the file, the write rate, and the latency of resuming
|
|
one agent both in-process and through a cold process spawn.
|
|
\item[\texttt{goroutine/main.go}] spawns \rFleetAgents{} goroutines, each parked
|
|
on a channel as an agent loop would be, and measures spawn latency, heap
|
|
growth against \texttt{runtime.MemStats} after a forced GC, and the time to
|
|
wake all of them. It also instantiates the three in-process language runtimes.
|
|
\item[\texttt{sandbox/sandbox.mjs}] measures cold start for an in-process V8
|
|
context, a fresh OCI container, and a command dispatched into an already-warm
|
|
container, 20 runs each.
|
|
\item[\texttt{pricing/pricing.mjs}] converts the measured volumes into a monthly
|
|
bill at published cloud rates.
|
|
\end{description}
|
|
|
|
\subsection{Repetition, and one method note that changed the numbers}
|
|
|
|
Memory results are deterministic here and are reported as a single figure: the
|
|
per-goroutine cost came out at exactly \rGoPer{}\,bytes on all five runs.
|
|
Latency results are not deterministic, and are reported as a median with the
|
|
observed range, never as a best-of.
|
|
|
|
The one trap worth recording is that the first goroutine measurements were taken
|
|
under \texttt{go run}, which compiles the program as part of the invocation.
|
|
That inflated spawn latency by roughly a third (253\,ns against a median of
|
|
\rGoSpawn{}\,ns from a built binary) and inflated the Python context figure by
|
|
about 15\%. \emph{Benchmark a built artifact.} The rows published here come from
|
|
\texttt{go build} output, run five times, serialized so that no two benchmarks
|
|
contend for the same cores.
|