mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-29 07:34:31 +00:00
zeekayandHanzo Dev
e57d01a884
Every measured constant in hanzo-dormant-agents came from bench lanes that have since been re-run, and two of them from a lane that was deleted. The paper is the last consumer to be corrected and the one where it matters most. The pooled row was the problem. Section 6 reported 37.54 ms for "a command dispatched into a container that is already up" and built a paragraph on it — "pooling collapses most of the gap, and is the only tier that matters at fleet scale" — which the conclusion then turned into policy: pool the expensive boundary so a fleet pays the pooled number in steady state. That row was docker exec, on a runtime this system does not use. The runtime it does use has no exec, no attach and no pool. Every call boots. So the table is hanzo-vm now: 309 ms cold, 335 p95, and 311 ms from a checkpoint — which is the same number, because a checkpoint carries the filesystem an earlier run left and the kernel boots either way. The pooling paragraph is replaced by what is true: there is no warm tier, the published resume figures elsewhere are memory snapshots and ours is not, and setting the two side by side would claim a mechanism by borrowing its name. The cost of not having it is stated instead — about 3x on a whole agent call. Section 6 also said our figure "compares favourably to the published cold-start figures". At 309 ms it does not: it is slower than the under-200 ms published for Firecracker sandboxes, faster than the container platforms, and all of those were taken by their authors on their own hardware. The paper says that now, including that quoting their hardware back at them is not a defence. The rest are re-measurements: a goroutine spawns in 125 ns (the 187 was a fresh binary's first run, which is 1.65x its own steady state), a million wake in 107 ms, a V8 context is 0.13 ms, an isolate 0.35 ms, wazero 7.8 us, goja 2.4 us, gpython 25.0 us. The isolation ratio is 2377x, not 1016x. Rebuilt: 12 pages, no stale figure survives in the PDF. Co-authored-by: Hanzo Dev <[email protected]>
79 lines
4.2 KiB
TeX
79 lines
4.2 KiB
TeX
\section{Isolation: what a boundary costs, and what it buys}
|
|
\label{sec:isolation}
|
|
|
|
Sandbox comparisons are usually published as one column of milliseconds, and
|
|
that column conflates two primitives that are not alternatives. We measured both
|
|
on the same host, 20 runs each.
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\begin{tabular}{lrrl}
|
|
\toprule
|
|
\textbf{Primitive} & \textbf{median} & \textbf{p95} & \textbf{What it can run} \\
|
|
\midrule
|
|
V8 context, in-process & \rVeight{}\,ms & \rVeightP{}\,ms & JavaScript. No shell, no pip, no pytest. \\
|
|
microVM, cold boot & \rVm{}\,ms & \rVmP{}\,ms & Any language, any binary, a real kernel and filesystem. \\
|
|
microVM, from a checkpoint & \rVmCkpt{}\,ms & \rVmCkptP{}\,ms & As above, on the disk an earlier run left behind. \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\caption{Cold start by isolation primitive, 20 runs each. The measured ratio
|
|
between a microVM boot and an in-process V8 context is
|
|
\rVmRatio{}$\times$.}
|
|
\label{tab:sandbox}
|
|
\end{table}
|
|
|
|
\paragraph{The ratio is real and it is not a defect.}
|
|
\rVmRatio{}$\times$ is what a kernel-enforced boundary costs: a kernel to boot,
|
|
a root image to attach, a virtual network interface, and a machine to shut down. A V8 context
|
|
is a data structure in a process that is already running, so it starts in
|
|
\rVeight{}\,ms and offers a boundary that is exactly as strong as the embedding
|
|
process's correctness. These buy different things and the choice is per workload,
|
|
not per millisecond.
|
|
|
|
The workload decides it, and for agents the workload is frequently unambiguous.
|
|
An agent asked to run a project's test suite needs a package manager, a
|
|
filesystem and a shell. No JavaScript isolate provides those at any latency. The
|
|
right comparison for that workload is the second row against other people's
|
|
second rows, and \emph{we do not win it}. \rVm{}\,ms is slower than the
|
|
under-200\,ms cold start published for Firecracker-based sandboxes, and faster
|
|
than the container-based platforms whose published figures run to
|
|
seconds~\cite{e2b,modal,cfworkers}. Those figures were taken by their authors on
|
|
their own hardware under protocols we did not run; that is the reason none of
|
|
these numbers settles the question, and it is not an argument that ours is
|
|
better than it measures.
|
|
|
|
\paragraph{A checkpoint saves the disk, not the boot, and there is no pooled
|
|
tier.}
|
|
An earlier version of this section reported a pooled row at $37.5$\,ms and argued
|
|
that it, rather than the cold number, was what a fleet experiences. That row was
|
|
a command dispatched into an already-running OCI container, measured on a runtime
|
|
this system does not use. The runtime it does use offers no such mode: a VM is
|
|
booted, a command runs, the machine stops. Starting \emph{from a checkpoint} is
|
|
the nearest thing and it is not the same thing --- \rVmCkpt{}\,ms against
|
|
\rVm{}\,ms for a cold boot, which is no difference, because a checkpoint carries
|
|
the filesystem a previous run left and the kernel boots either way.
|
|
|
|
The published resume figures for other microVM platforms are memory snapshots,
|
|
which is a different mechanism~\cite{e2b}. Setting our checkpoint number beside
|
|
theirs would claim a capability by borrowing the name of one. The honest position
|
|
is that every call here pays a boot, and that the cost of \emph{not} having a
|
|
warm tier is the difference between the two rows of Table~\ref{tab:sandbox} and
|
|
the first row: roughly $3\times$ on a whole agent call once search and embedding
|
|
are counted with it.
|
|
|
|
\paragraph{Three tiers, chosen by threat model.}
|
|
Taken with \S\ref{sec:code}, the measurements describe a ladder, and the sensible
|
|
policy is to take the cheapest rung the threat model permits:
|
|
|
|
\begin{enumerate}[leftmargin=*]
|
|
\item \textbf{In-process call} --- our own code, no boundary, nanoseconds.
|
|
\item \textbf{In-process runtime} --- tenant code that is not adversarial, a
|
|
capability boundary, \rWazero{}--\rGpython{}\,\textmu s (\S\ref{sec:code}).
|
|
\item \textbf{microVM} --- arbitrary binaries, an adversarial tenant, or a real
|
|
filesystem: \rVm{}\,ms, on every call, because there is no warm tier.
|
|
\end{enumerate}
|
|
|
|
A platform that offers only the third rung charges every workload the price of
|
|
the most demanding one. That is the cost this paper is trying to make visible ---
|
|
not the container itself, which is worth its price when it is needed.
|