mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
feat(distributed-inference): cross-vendor TP result + open-problems companion
Four papers on heterogeneous distributed LLM inference: - hanzo-heterogeneous-distributed-inference: cross-vendor (AMD ROCm gfx1151 + NVIDIA GB10 CUDA) tensor-parallel decode over a 2.5GbE cable, coherent at 50.4 tok/s (TTFT 0.09s, 213 tok/s prefill). Root cause of prior corruption proven: per-backend auto-dtype (ROCm->f16, CUDA->bf16) reinterpreted raw on the wire. Fixed by a canonical f32 ring wire + world_size-1 accumulating all-reduce (correct for any N>=2; spurious power-of-two gate removed). Same-vendor f16+f16 control matches token-for-token. - hanzo-distributed-open-problems: NEW companion enumerating 8 open research problems (sampling-determinism boundaries, quant-aware TP sharding, PP layer-assignment ILP, adaptive speculation under RTT/acceptance drift, the prefill launch-overhead wall, KV wire compression, the geo decode latency floor, and cheap verifiable inference), each cross-referenced to the paper that owns the surrounding result. - hanzo-distributed-decode-latency, hanzo-consensus-zap-compute: consistency pass (canonical f32 wire terminology; link goodput unified to 280 MB/s = 2.24 Gbit/s; precise f16-vs-bf16 root-cause wording). INDEX.md regenerated via `make index`; besides the 4 above it also indexes 6 pre-existing root papers the previous index had missed. All four compile clean with pdflatex (10/10/13/9 pages).
This commit is contained in:
@@ -42,14 +42,18 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
|
||||
| `hanzo-api-gateway` | ✓ | `hanzo-api-gateway/hanzo-api-gateway.tex` |
|
||||
| `hanzo-aso` | ✓ | `hanzo-aso/hanzo-aso.tex` |
|
||||
| `hanzo-base-runtime` | ✓ | `hanzo-base-runtime/hanzo-base-runtime.tex` |
|
||||
| `hanzo-bitexact-oracle` | ✓ | `hanzo-bitexact-oracle.tex` |
|
||||
| `hanzo-blockchain-platform` | ✓ | `hanzo-blockchain-platform/hanzo-blockchain-platform.tex` |
|
||||
| `hanzo-candle` | ✓ | `hanzo-candle/hanzo-candle.tex` |
|
||||
| `hanzo-chat` | ✓ | `hanzo-chat/hanzo-chat.tex` |
|
||||
| `hanzo-checkout` | ✓ | `hanzo-checkout/hanzo-checkout.tex` |
|
||||
| `hanzo-cloud-infrastructure` | ✓ | `hanzo-cloud-infrastructure/hanzo-cloud-infrastructure.tex` |
|
||||
| `hanzo-collapse-methodology` | ✓ | `hanzo-collapse-methodology.tex` |
|
||||
| `hanzo-commerce-payments` | ✓ | `hanzo-commerce-payments.tex` |
|
||||
| `hanzo-comptime-autotune` | ✓ | `hanzo-comptime-autotune.tex` |
|
||||
| `hanzo-consensus-ai` | ✓ | `hanzo-consensus-ai/hanzo-consensus-ai.tex` |
|
||||
| `hanzo-consensus-plugin-platform` | — | `hanzo-consensus-plugin-platform/main.tex` |
|
||||
| `hanzo-consensus-zap-compute` | ✓ | `hanzo-consensus-zap-compute.tex` |
|
||||
| `hanzo-console-dashboard` | ✓ | `hanzo-console-dashboard.tex` |
|
||||
| `hanzo-continuous-learning-privacy` | ✓ | `hanzo-continuous-learning-privacy.tex` |
|
||||
| `hanzo-cross-backend-inference` | ✓ | `hanzo-cross-backend-inference.tex` |
|
||||
@@ -58,6 +62,8 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
|
||||
| `hanzo-datastore` | — | `hanzo-datastore/hanzo-datastore.tex` |
|
||||
| `hanzo-deepseek-v4-dspark` | — | `hanzo-deepseek-v4-dspark.tex` |
|
||||
| `hanzo-digital-securities-integration` | ✓ | `hanzo-digital-securities-integration/hanzo-digital-securities-integration.tex` |
|
||||
| `hanzo-distributed-decode-latency` | ✓ | `hanzo-distributed-decode-latency.tex` |
|
||||
| `hanzo-distributed-open-problems` | ✓ | `hanzo-distributed-open-problems.tex` |
|
||||
| `hanzo-dso` | ✓ | `hanzo-dso/hanzo-dso.tex` |
|
||||
| `hanzo-edge-inference` | ✓ | `hanzo-edge-inference.tex` |
|
||||
| `hanzo-encryption-security` | ✓ | `hanzo-encryption-security/hanzo-encryption-security.tex` |
|
||||
@@ -67,9 +73,11 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
|
||||
| `hanzo-fhe-inference` | ✓ | `hanzo-fhe-inference/hanzo-fhe-inference.tex` |
|
||||
| `hanzo-flow` | ✓ | `hanzo-flow/hanzo-flow.tex` |
|
||||
| `hanzo-formal-verification` | ✓ | `hanzo-formal-verification/hanzo-formal-verification.tex` |
|
||||
| `hanzo-fusion-is-composition` | ✓ | `hanzo-fusion-is-composition.tex` |
|
||||
| `hanzo-gateway-krakend` | ✓ | `hanzo-gateway-krakend.tex` |
|
||||
| `hanzo-gpu-inference` | ✓ | `hanzo-gpu-inference/hanzo-gpu-inference.tex` |
|
||||
| `hanzo-harness-hacking` | ✓ | `hanzo-harness-hacking/hanzo-harness-hacking.tex` |
|
||||
| `hanzo-heterogeneous-distributed-inference` | ✓ | `hanzo-heterogeneous-distributed-inference.tex` |
|
||||
| `hanzo-hmm` | ✓ | `hanzo-hmm/hanzo-hmm.tex` |
|
||||
| `hanzo-iam-platform` | ✓ | `hanzo-iam-platform.tex` |
|
||||
| `hanzo-ico-infrastructure` | ✓ | `hanzo-ico-infrastructure/hanzo-ico-infrastructure.tex` |
|
||||
@@ -77,6 +85,8 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
|
||||
| `hanzo-inference-stack` | ✓ | `hanzo-inference-stack.tex` |
|
||||
| `hanzo-ingress-proxy` | ✓ | `hanzo-ingress-proxy.tex` |
|
||||
| `hanzo-insights-analytics` | ✓ | `hanzo-insights-analytics.tex` |
|
||||
| `hanzo-int8-mmq-negative-result` | ✓ | `hanzo-int8-mmq-negative-result.tex` |
|
||||
| `hanzo-intrinsic-islands` | ✓ | `hanzo-intrinsic-islands.tex` |
|
||||
| `hanzo-jin-architecture` | ✓ | `hanzo-jin-architecture/hanzo-jin-architecture.tex` |
|
||||
| `hanzo-jin` | ✓ | `hanzo-jin/hanzo-jin.tex` |
|
||||
| `hanzo-kernel-dsl` | ✓ | `hanzo-kernel-dsl.tex` |
|
||||
@@ -203,4 +213,4 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
|
||||
| `ZEN_WHITEPAPER_2025` | ✓ | `zen/ZEN_WHITEPAPER_2025.tex` |
|
||||
| `zen-world` | ✓ | `zen/zen-world.tex` |
|
||||
|
||||
**Total**: 198 papers, 192 with compiled PDFs
|
||||
**Total**: 208 papers, 202 with compiled PDFs
|
||||
|
||||
Binary file not shown.
@@ -0,0 +1,714 @@
|
||||
\documentclass[11pt]{article}
|
||||
\input{shared/kernel-preamble.tex}
|
||||
\usepackage{amsthm}
|
||||
|
||||
\newtheorem{proposition}{Proposition}
|
||||
|
||||
% Honesty labels: distinguish what is built from what is designed.
|
||||
\newcommand{\built}{\textcolor{str}{\small\textbf{[built]}}}
|
||||
\newcommand{\partialb}{\textcolor{kw}{\small\textbf{[partial]}}}
|
||||
\newcommand{\design}{\textcolor{cmt}{\small\textbf{[design]}}}
|
||||
|
||||
\title{\textbf{The Three-Plane Compute Fabric:\\
|
||||
Consensus-Managed, ZAP-Transported\\ Decentralized AI Inference}}
|
||||
\author{Hanzo AI --- ML Systems}
|
||||
\date{Architecture Blueprint --- \emph{One Source, Every Backend} series, 2026}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
A heterogeneous GPU fleet already runs a single model across ROCm, CUDA, and Metal
|
||||
today: the Hanzo engine's socket-ring shards a model tensor- and pipeline-parallel across
|
||||
mismatched accelerators over a canonical \code{f32} wire, and a \code{hanzo cluster}
|
||||
command onboards a worker by scanning a QR token. But that ring is configured by a static
|
||||
JSON file handed out of band, and its bytes cross the wire as \emph{plaintext TCP} between
|
||||
nodes that never authenticate each other. It is a LAN appliance, not a network. This paper
|
||||
is the blueprint for the missing two-thirds: turning that ring into a permissionless,
|
||||
byzantine-fault-tolerant AI inference \emph{network} that any developer --- or any
|
||||
public-cloud GPU miner --- can join and be paid for useful work. The thesis is one of
|
||||
\emph{decomplection}: a serving cluster has three concerns that monolithic stacks braid
|
||||
together and we hold apart as three orthogonal, independently-complete planes. The
|
||||
\textbf{control plane} is a Lux \emph{Quasar} consensus chain: membership, permissionless
|
||||
miner admission, layer placement, and config distribution become BFT-agreed state, with the
|
||||
Photon committee-selection primitive (Fisher--Yates over a luminance-weighted population)
|
||||
reused to sample GPU nodes and luminance re-bound from consensus-vote success to
|
||||
\emph{validated inference contribution}. The \textbf{transport} is \emph{ZAP}, the Lux
|
||||
zero-copy wire, whose post-quantum handshake (\code{WrapPQ} over an authenticated
|
||||
\code{Session} exposing a verified 32-byte \code{PeerID}) carries \emph{both} consensus
|
||||
control messages \emph{and} tensor activations over one PQ-authenticated channel --- retiring
|
||||
the plaintext ring sockets and converting the QR join token from a bearer secret into a
|
||||
handshake credential. The \textbf{data plane} is the existing inference ring, unchanged in
|
||||
kind: it receives its topology from consensus instead of a JSON file and moves its tensors
|
||||
over ZAP instead of raw TCP. We are scrupulous about the built/designed line. Built and
|
||||
cited to source: the cross-vendor ring, the \code{hanzo cluster} scan-to-join CLI, the ZAP
|
||||
Go wire with its PQ handshake, the Quasar engine with Photon/Wave/Focus and BLS$+$Pulsar$+$ML-DSA
|
||||
threshold signing, and the ZAP measurements ($17\times$ serialization and $345\times$ parse
|
||||
versus JSON-RPC, $\sim$$4.5\times$ versus gRPC with zero marshal/unmarshal). Designed, and
|
||||
labelled as such: consensus-driven membership, the ZAP-swapped ring transport, miner
|
||||
admission, and result verification by threshold-signing over inference outputs. No benchmark
|
||||
here is invented; every measured number is attributed to the code that produced it. The
|
||||
contribution is the architecture and its honest status, offered as a blueprint the wider
|
||||
AI-infrastructure community can build on.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction}
|
||||
\label{sec:intro}
|
||||
|
||||
The dominant shape of a model-serving stack is a monolith: one operator, one datacenter, one
|
||||
process tree that owns scheduling, transport, and tensor math all at once. vLLM, TGI, and
|
||||
Triton each fuse ``who is in the cluster,'' ``how bytes move between ranks,'' and ``what the
|
||||
GPUs compute'' into a single trusted binary on a private network. That fusion is fine when the
|
||||
operator owns every GPU. It fails the moment the fleet is \emph{heterogeneous} (a ROCm mini-PC,
|
||||
a rented CUDA instance, an Apple laptop), \emph{permissionless} (a stranger's GPU wants to
|
||||
contribute and be paid), or \emph{untrusted} (the node holding your activations is not yours).
|
||||
None of those three failures is a math problem; each is a \emph{concerns} problem. The cluster
|
||||
braids together things that want to be separate.
|
||||
|
||||
This paper takes the opposite stance, the one our engineering culture states as a law: one and
|
||||
only one way to do everything, composable and orthogonal. A decentralized inference cluster has
|
||||
exactly three concerns, and we give each its own plane:
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=2pt]
|
||||
\item \textbf{Control} --- who is in the cluster, who may join, which node holds which layers,
|
||||
and what the run configuration is. This is agreement among mutually distrusting parties: a
|
||||
\emph{consensus} problem.
|
||||
\item \textbf{Transport} --- how a byte gets from one node to another, and how each end knows
|
||||
who the other is. This is an \emph{authenticated wire} problem.
|
||||
\item \textbf{Data} --- the actual tensor exchange: activations forward, gradients or
|
||||
all-reduce sideways. This is a \emph{numerics and topology} problem.
|
||||
\end{itemize}
|
||||
The claim is that these three are genuinely orthogonal, that Hanzo and Lux already ship a
|
||||
best-in-class primitive for each, and that composing them --- rather than reinventing a
|
||||
monolith --- yields something no monolith can be: a GPU network anyone can join. The control
|
||||
primitive is Lux \emph{Quasar} consensus~\cite{quasar,luxconsensus}; the transport is
|
||||
\emph{ZAP}~\cite{zap,lp201}; the data plane is the Hanzo cross-backend inference
|
||||
engine~\cite{infstack}.
|
||||
|
||||
\paragraph{The honesty frame.} This is an architecture paper, and its credibility rests on not
|
||||
blurring what runs today with what is designed. We adopt three labels, used inline throughout
|
||||
and collected in the status table of \S\ref{sec:status}: \built{} for code on \code{main},
|
||||
cited to its file; \partialb{} for a primitive that is compiled and exercised but not yet wired
|
||||
end-to-end; and \design{} for a proposed composition. The measured numbers we quote (ring
|
||||
parity across vendors, ZAP throughput and latency, Photon and luminance costs) are all \built{},
|
||||
each attributed to the benchmark that produced it. The network-level assembly --- consensus as
|
||||
the live scheduler, the ring running over ZAP, miner admission and payment --- is \design{},
|
||||
and never dressed as more.
|
||||
|
||||
\paragraph{Contribution.} We (i) inventory the built substrate with source citations
|
||||
(\S\ref{sec:today}); (ii) state the three-plane architecture and prove its concerns are
|
||||
separable (\S\ref{sec:planes}); (iii) give the threat model and a result-integrity scheme for
|
||||
byzantine miners (\S\ref{sec:threat}); (iv) bind the economic layer to the existing luminance
|
||||
primitive (\S\ref{sec:econ}); (v) lay out a staged migration from the static-config LAN cluster
|
||||
to a public network (\S\ref{sec:migration}); and (vi) publish an honest built-versus-designed
|
||||
status table (\S\ref{sec:status}) and the limitations (\S\ref{sec:limits}).
|
||||
|
||||
\section{The substrate that exists today}
|
||||
\label{sec:today}
|
||||
|
||||
Before proposing a network we account for what is already on \code{main}. Every claim in this
|
||||
section is \built{} and cites the file that implements it. The point of the inventory is that
|
||||
the three planes are not vaporware waiting to be written: each has a mature primitive, and the
|
||||
work is composition, not invention.
|
||||
|
||||
\subsection{Data plane: the heterogeneous inference ring}
|
||||
\label{sec:ring}
|
||||
|
||||
The Hanzo engine already runs one model split across several GPUs of \emph{different vendors}.
|
||||
Its \code{distributed} module (\code{hanzo-engine/src/distributed.rs},
|
||||
\code{hanzo-quant/src/distributed/}) provides two backends behind one \code{Comm} abstraction:
|
||||
NCCL for homogeneous NVIDIA, and a portable \emph{ring} backend for everything else. The ring
|
||||
does tensor-parallel and pipeline-parallel sharding by exchanging activations with its
|
||||
right-hand neighbor, rank $r$ dialing rank $(r{+}1) \bmod N$ in an all-reduce cycle
|
||||
(\code{hanzo-quant/src/distributed/socket.rs}). Cross-vendor generation works because the ring
|
||||
wire is a \emph{canonical} \code{f32} format and reductions take a backend-independent
|
||||
CPU-\code{f32} path with a canonical tie-break (\code{hanzo-engine/src/ops.rs}), so a ROCm box
|
||||
and a CUDA box agree bit-for-bit on the reduced tensor rather than each rank reinterpreting the
|
||||
other's raw 16-bit words in the wrong \code{f16}-versus-\code{bf16} layout. The topology is a static struct read from a JSON file
|
||||
named by the \code{RING\_CONFIG} environment variable (Listing~\ref{lst:ringcfg}): master
|
||||
address, per-rank ports, this rank's index, and the world size. The cross-backend engine
|
||||
underneath --- five backends, the full GGUF quant zoo, decode parity with \code{llama.cpp} --- is
|
||||
the subject of the companion stack paper~\cite{infstack}; here it is simply the data plane.
|
||||
|
||||
\begin{lstlisting}[caption={The data-plane topology today: a static \code{RING\_CONFIG} JSON.
|
||||
Every node needs the full member list up front; nothing authenticates a peer.},label={lst:ringcfg}]
|
||||
{ "master_ip": "10.0.0.1", "master_port": 8000,
|
||||
"port": 8001, "right_ip": "10.0.0.2", "right_port": 8001,
|
||||
"rank": 0, "world_size": 4 }
|
||||
\end{lstlisting}
|
||||
|
||||
Two properties of this substrate matter for everything below. First, it is \emph{correct across
|
||||
vendors} --- the hard part, cross-backend numeric agreement, is solved. Second, it is
|
||||
\emph{unauthenticated plaintext}: the ring exchanges raw bytes over \code{std::net::TcpStream}
|
||||
with \code{write\_all}/\code{read\_exact} and ships its control messages as \code{serde\_json}
|
||||
over the same socket. It is safe only on a trusted private mesh. That is the gap the transport
|
||||
plane closes.
|
||||
|
||||
\subsection{The onboarding UX: \code{hanzo cluster} scan-to-join}
|
||||
\label{sec:clustercli}
|
||||
|
||||
Handing every node a JSON file out of band does not scale past a lab. The \code{hanzo cluster}
|
||||
subcommand (\code{hanzo-cli/src/cluster.rs}) already replaces it with a scan-to-join flow.
|
||||
\code{hanzo cluster init} on the head (rank 0) enumerates the ring, prints a \emph{join token}
|
||||
and its QR code, and launches; \code{hanzo cluster join} on a worker decodes the token and
|
||||
launches as a ring daemon. The token is a \code{bs58}-encoded JSON \code{JoinToken}
|
||||
(Listing~\ref{lst:token}) --- a nonce, the head address, the world size, the ordered ring
|
||||
addresses, and the model --- compact enough to fit in one QR or one paste. This is the correct
|
||||
\emph{shape} of onboarding: a worker needs one string to join. But observe what the token is
|
||||
today: a \emph{bearer credential in cleartext}. Anyone who photographs the QR can join the ring,
|
||||
and the ring cannot tell an invited worker from an intruder, because the token carries topology,
|
||||
not identity. Holding that observation is the whole of \S\ref{sec:threat}.
|
||||
|
||||
\begin{lstlisting}[language=Rust,caption={The join token (\code{hanzo-cli/src/cluster.rs}):
|
||||
\code{bs58(JSON)} so it fits one QR. It is topology, not identity --- a bearer secret today.},label={lst:token}]
|
||||
struct JoinToken {
|
||||
nonce: u64, // ring id, printed {:016x}
|
||||
master: String, // rank-0 (head) reachable IP
|
||||
world_size: usize,
|
||||
addrs: Option<Vec<String>>, // full ordered ring [rank0..rankN-1]
|
||||
model: String,
|
||||
}
|
||||
\end{lstlisting}
|
||||
|
||||
\subsection{Transport: ZAP, the zero-copy post-quantum wire}
|
||||
\label{sec:zap}
|
||||
|
||||
ZAP (Zero-copy Application Protocol, \code{github.com/luxfi/zap}) is the Lux universal
|
||||
wire~\cite{zap,lp201}. Its serialization is zero-copy: a message is read directly out of the
|
||||
network buffer with no parse and no allocation (measured \built{} parse $2.9$~ns/op at zero
|
||||
allocations; build $44$~ns/op). Against the protocols an AI stack would otherwise reach for,
|
||||
the wire eliminates the marshal/unmarshal step entirely: $17\times$ faster serialization and
|
||||
$345\times$ faster parse than JSON-RPC (the MCP transport), $\sim$$157\times$ faster than
|
||||
protobuf on deserialization ($21$~ns versus $3231$~ns, zero allocations), and $\sim$$4.5\times$
|
||||
faster than gRPC with the whole codec layer deleted rather than optimized~\cite{zap,lp201,luxconsensus}.
|
||||
End-to-end it sustains $114$K transactions/s on a single connection and $376$K across $20$
|
||||
parallel connections~\cite{luxconsensus}.
|
||||
|
||||
For a public network the decisive feature is not speed but \emph{authenticated identity}. ZAP's
|
||||
PQ layer (\code{conn\_pq.go}) wraps an established connection \emph{after} a post-quantum
|
||||
handshake: \code{handshake.Initiator.Run(tcp)} produces a \code{Session}, and \code{WrapPQ(tcp,
|
||||
sess)} returns an ordinary \code{net.Conn} that AEAD-frames every write, zeroes its keys on
|
||||
close, and --- critically --- exposes \code{PeerID() [32]byte}, ``the verified peer identity for
|
||||
callers that need to route on it (e.g.\ validator-set membership checks).'' That one method is
|
||||
the hinge of the entire security argument (\S\ref{sec:threat}): the transport already hands the
|
||||
control plane a cryptographically verified identity for the node on the other end.
|
||||
|
||||
Inside the Hanzo engine, ZAP is a first-class module (\code{hanzo-engine/src/zap/}). Its
|
||||
message and adapter surface is compiled and exercised in-process --- \code{ZapInferenceServer}
|
||||
translates ZAP \code{ChatRequest}s into the engine's real \code{Request} channel and streams
|
||||
\code{ChatChunk}s back, satisfying the same \code{ZapInference} trait a networked client would
|
||||
(\code{server.rs}, \code{client.rs}). We mark this \partialb{}, honestly: the out-of-process
|
||||
\code{zap://} transport is a documented stub that carries no \code{capnp} build dependency yet
|
||||
(``flipping on the wire is mechanical,'' \code{transport.rs}), and a few methods
|
||||
(\code{tokenize}, \code{detokenize}, \code{re\_isq}) return ``not yet wired.'' The wire exists
|
||||
in Go and is production-deployed across Lux consensus, MPC, and DEX systems; the engine-side
|
||||
Rust surface is built but not yet turned on over a socket.
|
||||
|
||||
\subsection{Control: the Quasar consensus engine}
|
||||
\label{sec:quasar}
|
||||
|
||||
Lux \emph{Quasar} (\code{github.com/luxfi/consensus}) is a DAG-native, post-quantum BFT
|
||||
consensus engine~\cite{quasar,luxconsensus}. It is built from small, orthogonal sub-protocols,
|
||||
four of which this paper reuses directly:
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=2pt]
|
||||
\item \textbf{Photon} (\code{protocol/photon}) selects a $K$-of-$N$ committee each round by
|
||||
Fisher--Yates shuffle over the node population, drawing with a rejection-sampled uniform integer
|
||||
(\code{emitter.go}) to remove the modulo bias that a grinding adversary could exploit on
|
||||
committee sampling. The population is weighted by \emph{luminance}.
|
||||
\item \textbf{Luminance} (\code{photon/luminance.go}) is a per-node reputation in ``lux'' units:
|
||||
each node starts at $100$, a successful vote multiplies its brightness by $1.1$ (capped at
|
||||
$1000$), a failure by $0.9$ (floored at $10$), and \code{Brightness} normalizes to a selection
|
||||
weight in $[0.1, 10]$. It is a cheap, bounded, exponential reputation --- a brightness update
|
||||
costs $72$~ns \built{}.
|
||||
\item \textbf{Wave} (\code{protocol/wave}) runs per-round threshold voting with Fast
|
||||
Probabilistic Consensus; a vote round costs $3.38$~\textmu s and a $K$-of-$N$ Photon selection
|
||||
$3.03$~\textmu s \built{}.
|
||||
\item \textbf{Quasar} (\code{protocol/quasar}) seals decisions with threshold signatures across
|
||||
three independent cryptographic paths --- BLS12-381 (48-byte aggregate), Pulsar Ring-LWE
|
||||
threshold, and ML-DSA-65 (FIPS 204, rolled to a 192-byte Groth16 proof) --- each independently
|
||||
toggleable, all three in parallel under \code{TripleSignRound1}. A QuasarCert verifies in
|
||||
$\sim$$2$--$5$~ms on CPU, GPU-batchable $10$--$100\times$~\cite{luxconsensus}.
|
||||
\end{itemize}
|
||||
Quasar's own inter-node transport is already ZAP, not gRPC or libp2p (\code{consensus/LLM.md}).
|
||||
Control and transport are, in the Lux stack, \emph{already} composed; what is missing is binding
|
||||
that composition to the \emph{GPU} population instead of the validator population.
|
||||
|
||||
\section{The three-plane architecture}
|
||||
\label{sec:planes}
|
||||
|
||||
The blueprint is a single sentence: \emph{run the inference ring's topology as Quasar consensus
|
||||
state, and move every byte --- control and tensor alike --- over ZAP.} Figure~\ref{fig:planes}
|
||||
is the whole system. What follows gives each plane its interface and states, as a proposition,
|
||||
why the three do not leak into each other.
|
||||
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
\small
|
||||
\begin{tabular}{@{}l@{}}
|
||||
\toprule
|
||||
\textbf{Control plane} --- Quasar consensus (BFT, post-quantum) \hfill \design{} \\
|
||||
\quad Photon committee over GPU nodes $\cdot$ luminance = useful-work weight \\
|
||||
\quad membership $\cdot$ miner admission $\cdot$ layer placement $\cdot$ config $\cdot$ result attest \\
|
||||
\midrule
|
||||
\textbf{Transport plane} --- ZAP PQ wire \hfill \built{}\,/\,\partialb{} \\
|
||||
\quad \code{WrapPQ} session $\cdot$ verified \code{PeerID} $\cdot$ AEAD frames \\
|
||||
\quad one channel for BOTH control messages AND tensor activations \\
|
||||
\midrule
|
||||
\textbf{Data plane} --- Hanzo inference ring \hfill \built{} \\
|
||||
\quad tensor/pipeline parallel $\cdot$ canonical \code{f32} all-reduce $\cdot$ 5 backends \\
|
||||
\quad topology handed down by control; bytes carried by transport \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Three orthogonal planes. Each is independently complete and separately citable to
|
||||
code; the network is their composition, not a new monolith. Labels mark built versus designed
|
||||
per \S\ref{sec:status}.}
|
||||
\label{fig:planes}
|
||||
\end{figure}
|
||||
|
||||
\subsection{Control plane: consensus as the scheduler}
|
||||
\label{sec:control}
|
||||
|
||||
The control plane replaces the static \code{RING\_CONFIG} JSON (Listing~\ref{lst:ringcfg}) and
|
||||
the out-of-band QR token with \emph{agreed state on a Quasar chain}. The chain's VM state
|
||||
\emph{is} the cluster's control surface \design{}:
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=2pt]
|
||||
\item \textbf{Membership} --- the set of GPU nodes, each identified by its ZAP \code{PeerID}
|
||||
(an ML-DSA public key), with advertised capability (backend, VRAM, resident models, measured
|
||||
decode/prefill throughput). Joining and leaving are transactions, so the member list is
|
||||
BFT-agreed rather than hand-distributed.
|
||||
\item \textbf{Miner admission} --- a permissionless join. A public-cloud GPU submits a join
|
||||
transaction carrying its \code{PeerID} and a stake bond; admission is a consensus decision, not
|
||||
an operator's SSH session. This is the step that turns ``my four boxes'' into ``anyone's GPU''
|
||||
(\S\ref{sec:econ}).
|
||||
\item \textbf{Placement / routing} --- which node holds which contiguous layer range of which
|
||||
model. Pipeline-parallel assignment (node $A$ holds layers $0$--$7$, $B$ holds $8$--$15$) and
|
||||
the ring order become an on-chain placement map. A request is routed by reading that map, not by
|
||||
trusting a JSON file.
|
||||
\item \textbf{Config distribution} --- model id, quantization, sampling defaults, ring
|
||||
\code{world\_size}: the fields of \code{JoinToken} and \code{RingConfig} become chain state, so
|
||||
every node reads an identical, agreed configuration.
|
||||
\end{itemize}
|
||||
|
||||
The mechanism is not new code so much as a \emph{re-population} of Photon. Photon selects a
|
||||
committee by weighted Fisher--Yates over a node set; point it at the \emph{GPU} population and it
|
||||
selects a committee of GPU nodes --- to ratify a placement, to admit a miner, or to verify a
|
||||
result (\S\ref{sec:threat}). Luminance, today a function of consensus-vote success, is re-bound
|
||||
to \emph{validated inference contribution}: a node whose outputs pass verification brightens and
|
||||
is preferentially selected and rewarded; a node that stalls or returns wrong tensors dims and is
|
||||
sampled out. The primitive is unchanged --- an exponential, bounded, cheap reputation --- only
|
||||
the event that drives \code{Illuminate(id, success)} moves from ``voted with the quorum'' to
|
||||
``did correct inference work.'' Crucially, \emph{writes to consensus are rare and
|
||||
slow-changing}: placement and membership shift on the order of nodes joining and models loading,
|
||||
not on the order of tokens generated. The hot path --- a forward pass through the ring --- never
|
||||
touches consensus.
|
||||
|
||||
\subsection{Transport plane: one PQ wire for control and tensors}
|
||||
\label{sec:transport}
|
||||
|
||||
Every inter-node byte moves over ZAP. This is deliberately \emph{one} wire for two kinds of
|
||||
traffic that a monolith would split across two stacks:
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=2pt]
|
||||
\item \textbf{Control messages} --- consensus votes, placement updates, admission requests ---
|
||||
which Quasar already speaks over ZAP.
|
||||
\item \textbf{Tensor data} --- the activation exchange and all-reduce that the ring performs
|
||||
every micro-step --- which today crosses raw TCP and, under this design, crosses the same
|
||||
authenticated \code{WrapPQ} \code{net.Conn} instead.
|
||||
\end{itemize}
|
||||
Unifying them on one wire is the payoff of decomplection: the ring does not need to know it is on
|
||||
a network of strangers, only that it holds a \code{net.Conn}; the transport does not need to know
|
||||
whether the bytes are a vote or an activation. ZAP's zero-copy framing is what makes carrying
|
||||
tensors over an authenticated session affordable --- there is no marshal/unmarshal tax on the
|
||||
activation exchange, only an AEAD seal over a buffer the ring already owns (the $64$~KiB
|
||||
\code{MaxPQRecord} chunking of \code{conn\_pq.go} maps naturally onto activation-sized writes).
|
||||
The identity that the handshake establishes, \code{PeerID}, is the same value the control plane
|
||||
keys membership on: the node you complete a ZAP handshake with \emph{is} the node consensus
|
||||
admitted, by construction, with no separate PKI to reconcile.
|
||||
|
||||
\subsection{Data plane: unchanged in kind}
|
||||
\label{sec:data}
|
||||
|
||||
The data plane is the least-changed and that is the point. The ring's numerics --- canonical
|
||||
\code{f32} reduction, tensor/pipeline sharding, the five-backend kernels --- are correct today
|
||||
(\S\ref{sec:ring}) and stay exactly as they are. Two wires change underneath them: the topology
|
||||
arrives from the control plane's placement map rather than \code{RingConfig::load()}, and the
|
||||
\code{TcpStream} is a ZAP \code{net.Conn} rather than a bare socket. The ring code that computes
|
||||
does not move. This is composition over rewrite: we are not building a decentralized inference
|
||||
engine, we are giving the inference engine we have a control plane and an authenticated
|
||||
transport.
|
||||
|
||||
\subsection{Why three planes, stated precisely}
|
||||
\label{sec:whythree}
|
||||
|
||||
\begin{proposition}[Orthogonality of the three planes]
|
||||
\label{prop:ortho}
|
||||
The control, transport, and data planes can each be replaced or scaled independently without
|
||||
modifying the other two, because they share only two narrow, typed interfaces: the control plane
|
||||
emits a \emph{placement map} keyed by \code{PeerID}, and the transport plane emits an
|
||||
authenticated \code{net.Conn} whose \code{PeerID} matches. The data plane consumes both and
|
||||
exposes neither consensus nor cryptography; the transport consumes an identity and exposes a
|
||||
byte stream, knowing nothing of tensors; the control consumes admission and contribution events
|
||||
and exposes agreed state, knowing nothing of wire framing.
|
||||
\end{proposition}
|
||||
|
||||
\begin{proof}[Argument]
|
||||
By interface inspection. The data plane's dependencies are exactly (i) a topology --- rank,
|
||||
world size, neighbor address, layer range --- and (ii) a \code{net.Conn} per neighbor. Neither
|
||||
names a consensus type or a cipher. Swap Quasar for any state machine that produces the same
|
||||
placement map and the ring is unaffected; swap ZAP for any authenticated \code{net.Conn} with a
|
||||
stable peer identity and the ring is unaffected; swap the ring's kernels (add a backend, change
|
||||
the quant) and neither consensus nor transport observes it. The only cross-plane invariant is
|
||||
that the \code{PeerID} the transport authenticates equals the \code{PeerID} the control plane
|
||||
admitted --- a single equality, checked once at connection setup, which is precisely what
|
||||
\code{WrapPQ}'s \code{PeerID()} was built to support (``validator-set membership checks,''
|
||||
\code{conn\_pq.go}). Three concerns, two interfaces, one shared value. \qed
|
||||
\end{proof}
|
||||
|
||||
The contrast with a monolithic serving stack is now sharp. In vLLM or Triton, ``who is in the
|
||||
cluster'' is a launch-time flag baked into the process, ``how bytes move'' is a hardcoded NCCL
|
||||
or gRPC channel with no peer authentication, and ``what computes'' is the same binary --- the
|
||||
three concerns are one artifact, so you cannot add a permissionless member, cannot authenticate
|
||||
a stranger's GPU, and cannot verify a result without forking the whole thing. Our planes are
|
||||
three artifacts with two seams. That is the difference between a cluster and a network.
|
||||
|
||||
\section{Threat model and result integrity}
|
||||
\label{sec:threat}
|
||||
|
||||
A permissionless network inherits adversaries a private cluster never has. We state the model,
|
||||
the gaps in today's substrate, and the design that closes them.
|
||||
|
||||
\paragraph{Assets and adversaries.} The assets are: the integrity of an inference result (the
|
||||
tokens a user is billed for must be the model's real output), the confidentiality and
|
||||
authenticity of activations in flight, the correctness of the placement map, and the fairness of
|
||||
reward. The adversaries are: a network attacker on the path between nodes (public-cloud tenants
|
||||
share fabric); an \emph{intruder} who obtained a join token; and a \emph{byzantine miner} who
|
||||
joined honestly but returns wrong, fabricated, or lazily-cached tensors to earn reward without
|
||||
doing the work.
|
||||
|
||||
\paragraph{Gap 1: the wire is plaintext (\built{} problem).} Today the ring is raw TCP
|
||||
(\S\ref{sec:ring}). A path attacker can read activations, inject a forged tensor into the
|
||||
all-reduce, or man-in-the-middle a neighbor. \textbf{Closure \design{}:} the ZAP PQ handshake.
|
||||
After \code{WrapPQ}, every frame is AEAD-sealed and every peer is a verified ML-DSA identity;
|
||||
injection and eavesdropping both fail against a post-quantum authenticated channel. The tensor
|
||||
plane inherits the same wire security consensus already relies on.
|
||||
|
||||
\paragraph{Gap 2: the join token is a bearer secret (\built{} problem).} The QR token
|
||||
(Listing~\ref{lst:token}) is cleartext topology; whoever holds it joins (\S\ref{sec:clustercli}).
|
||||
\textbf{Closure \design{}:} make the token a \emph{handshake credential}, not the thing that
|
||||
grants access. Under the design the token carries the ring's \code{PeerID} set (or a commitment
|
||||
to it); a worker presenting the token must then \emph{complete a ZAP PQ handshake} proving
|
||||
possession of an admitted ML-DSA key. A photographed QR no longer joins anyone --- it only names
|
||||
who is allowed to, and access is gated on the handshake the intruder cannot forge. The token
|
||||
degrades from a secret to a public address book, which is the correct security posture: identity
|
||||
is the credential, topology is not.
|
||||
|
||||
\paragraph{Gap 3: a miner may lie about its work (the hard one).} Authentication proves
|
||||
\emph{who} sent a tensor, not that the tensor is \emph{correct}. A byzantine miner holding layers
|
||||
$8$--$15$ can return plausible-looking garbage and collect reward. This is the genuinely new
|
||||
problem, and consensus is the tool. We propose three composable mechanisms, in increasing cost,
|
||||
scheduled by the control plane \design{}:
|
||||
\begin{enumerate}[leftmargin=1.5em,itemsep=2pt]
|
||||
\item \textbf{Threshold attestation of outputs.} A completed inference (or a placement-epoch's
|
||||
batch of them) is sealed with a QuasarCert: a Photon committee of GPU nodes threshold-signs the
|
||||
$(\text{request}, \text{output})$ pair with BLS$+$Pulsar$+$ML-DSA (\S\ref{sec:quasar}). A billed
|
||||
result carries a certificate that a supermajority attested to it, giving the user cryptographic
|
||||
non-repudiation and making a single lying miner insufficient to certify a bad output.
|
||||
\item \textbf{Redundant re-execution / spot checks.} The scheduler occasionally assigns the same
|
||||
layer range to two independent miners and compares --- cheap because the ring wire is already
|
||||
\emph{canonical} \code{f32} (\S\ref{sec:ring}), so two honest miners on different vendors produce
|
||||
\emph{bit-identical} tensors and any disagreement is a proof of fault, not a rounding artifact.
|
||||
The canonical-\code{f32} property, built for cross-vendor correctness, doubles as the substrate
|
||||
for verifiable redundancy. A miner caught disagreeing with the honest majority is dimmed by
|
||||
luminance and slashed.
|
||||
\item \textbf{Reputation as the economic backstop.} Luminance (\S\ref{sec:quasar}) makes
|
||||
correct work compound and incorrect work decay: a node that fails verification multiplies its
|
||||
brightness by $0.9$ and is sampled into fewer committees and paid for less work, converging its
|
||||
influence and income toward zero. Reputation is the slow, cheap layer under the fast, expensive
|
||||
cryptographic ones.
|
||||
\end{enumerate}
|
||||
Byzantine fault tolerance is a property the control plane already has (Quasar tolerates $<1/3$
|
||||
adversarial stake); the contribution here is \emph{extending it from block finality to inference
|
||||
finality} by threshold-signing over outputs and exploiting canonical \code{f32} for cheap
|
||||
re-execution. We are explicit that this is \design{}: the threshold-signing primitive is built,
|
||||
its application to inference outputs is proposed.
|
||||
|
||||
\paragraph{Non-goal: confidentiality from the operator.} This design authenticates and verifies;
|
||||
it does not, by itself, hide activations from the miner that computes them. Workloads needing
|
||||
that require a hardware TEE tier (H100/H200 confidential computing, TDX, SEV-SNP), an orthogonal
|
||||
extension developed in the tenant-cloud companion~\cite{tenantcloud} and out of scope here.
|
||||
|
||||
\section{The economic layer: luminance as useful-work weighting}
|
||||
\label{sec:econ}
|
||||
|
||||
A permissionless network needs an incentive, and the incentive must be \emph{useful-work},
|
||||
because paying for wasted computation (proof-of-work hashing) is exactly what an inference
|
||||
network should not do. The luminance primitive is already a useful-work meter in miniature: it
|
||||
rewards nodes for correct participation and decays nodes for failure. The economic design
|
||||
\design{} binds three quantities to it:
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=2pt]
|
||||
\item \textbf{Admission by stake.} A miner joins by bonding stake in a transaction
|
||||
(\S\ref{sec:control}). Stake is the sybil cost --- spinning up a thousand fake GPUs is a
|
||||
thousand bonds --- and the slashing target when verification (\S\ref{sec:threat}) catches a
|
||||
fault.
|
||||
\item \textbf{Reward by validated contribution.} Payment for an inference accrues to the miners
|
||||
whose outputs were attested, weighted by luminance and by measured work (tokens $\times$ model
|
||||
size $\times$ layer share). Because luminance rises with correct work and falls with incorrect
|
||||
work, expected income tracks \emph{honest throughput}, and a lazy or lying miner earns less than
|
||||
its electricity costs.
|
||||
\item \textbf{Routing by brightness.} The Photon committee that ratifies placement and verifies
|
||||
results draws with luminance weight, so brighter (more reliably correct) nodes are preferentially
|
||||
given work --- reputation and revenue reinforce, which is what makes ``contribute a GPU, get
|
||||
paid'' a stable equilibrium rather than a race to the bottom.
|
||||
\end{itemize}
|
||||
The mechanism reuses one existing primitive (luminance) for reputation, one existing primitive
|
||||
(stake-weighted BFT) for sybil-resistance and slashing, and one existing primitive (Photon) for
|
||||
work routing. The realized system is ``any dev, any miner, contributes a GPU and is paid for
|
||||
inference work,'' with the payment denominated in the network's native token and the fairness
|
||||
enforced by the same consensus that finalizes the placement map. This is the point of contact
|
||||
with the sovereign-cloud economics of the tenant-cloud companion~\cite{tenantcloud}, which
|
||||
develops the wallet, staking, and settlement primitives this layer would sit on.
|
||||
|
||||
\section{Migration path: from LAN appliance to public network}
|
||||
\label{sec:migration}
|
||||
|
||||
The design is only credible if there is a continuous path from what runs today to the network,
|
||||
with no flag day. There is, in four stages, each shippable and independently valuable.
|
||||
|
||||
\paragraph{Stage 0 (today, \built{}).} Static \code{RING\_CONFIG} JSON, QR join token, raw-TCP
|
||||
ring, canonical \code{f32} across vendors. Works on a trusted LAN; this is the baseline.
|
||||
|
||||
\paragraph{Stage 1 --- authenticate the wire (\design{}, smallest step).} Replace the ring's
|
||||
\code{TcpStream} with a ZAP \code{WrapPQ} \code{net.Conn} and make the join token gate on a PQ
|
||||
handshake (\S\ref{sec:threat}, Gaps 1--2). No consensus yet: the topology is still the static
|
||||
map, but the wire is now authenticated and encrypted and the token is a credential. This alone
|
||||
converts the appliance into something safe to run across untrusted networks, and it exercises the
|
||||
engine's ZAP module end-to-end --- turning the \partialb{} \code{zap://} transport on
|
||||
(\S\ref{sec:zap}) is precisely this step.
|
||||
|
||||
\paragraph{Stage 2 --- consensus owns the map (\design{}).} Stand up a Quasar chain whose state
|
||||
is the membership and placement map. Nodes read topology from the chain instead of
|
||||
\code{RingConfig::load()}; \code{hanzo cluster join} becomes an admission transaction rather than
|
||||
a local launch. Placement is still operator-proposed and consensus-ratified (permissioned
|
||||
membership, BFT config). The ring code is untouched; only the source of its topology moves from
|
||||
file to chain.
|
||||
|
||||
\paragraph{Stage 3 --- permissionless miners and paid work (\design{}).} Open admission to
|
||||
staked strangers, bind luminance to validated inference contribution, and turn on result
|
||||
attestation and spot-check re-execution (\S\ref{sec:threat}--\S\ref{sec:econ}). This is the full
|
||||
network: anyone bonds, joins, is placed by consensus, serves layers over authenticated ZAP, is
|
||||
verified by a Photon committee, and is paid by luminance-weighted contribution.
|
||||
|
||||
The staging matters because each stage stands alone. Stage 1 is worth shipping even if Stages
|
||||
2--3 never do (it fixes a real security gap). Stage 2 is worth shipping for a single operator's
|
||||
multi-datacenter fleet (BFT config, no static files) without any economics. Only Stage 3 needs
|
||||
the token and slashing machinery. There is no rewrite between stages --- the same ring, the same
|
||||
ZAP, more of the control plane each time.
|
||||
|
||||
\section{Connection to the latency-physics companion}
|
||||
\label{sec:latency}
|
||||
|
||||
A pipeline-parallel ring over a wide-area network has a latency budget a single box does not: each
|
||||
pipeline stage adds a network hop to the critical path of every token. The companion
|
||||
latency-physics analysis~\cite{latency} argues that this budget is spendable precisely because
|
||||
two techniques compose over ZAP. First, \textbf{pipeline parallelism hides transport latency
|
||||
under compute}: with enough micro-batches in flight, a stage's forward pass overlaps the next
|
||||
stage's activation transfer, so the ring's throughput approaches the slowest stage's compute rate
|
||||
rather than the sum of hop latencies --- and ZAP's zero-copy framing minimizes the per-hop
|
||||
transport term to an AEAD seal, not a marshal. Second, \textbf{speculative decoding amortizes the
|
||||
round-trip}: a draft model proposes several tokens that the ring verifies in one pipelined pass,
|
||||
so the wide-area round-trip is paid once per \emph{accepted block} of tokens instead of once per
|
||||
token. The engine already factors speculative decoding behind one \code{SpeculativeProposer}
|
||||
trait~\cite{infstack}; over a consensus-managed ZAP ring, the draft can even run on a nearby,
|
||||
brighter node while the target pipeline spans the fleet. The three-plane fabric is what makes
|
||||
these composable: the data plane does the pipelining and speculation, the transport plane makes
|
||||
each hop cheap and authenticated, and the control plane places draft and target where the latency
|
||||
budget is best --- decomplected, so each optimization lives in exactly one plane. We cite the
|
||||
companion for the quantitative budget; here it is enough that the architecture does not fight the
|
||||
physics.
|
||||
|
||||
\section{Status: built, partial, and designed}
|
||||
\label{sec:status}
|
||||
|
||||
Table~\ref{tab:status} is the honest ledger. It is the table to read first: it says exactly which
|
||||
sentences of this paper are code on \code{main} and which are blueprint. The pattern is
|
||||
deliberate --- every \emph{primitive} is built and cited; every \emph{composition} into the
|
||||
network is designed. That is what makes this a buildable blueprint rather than a wish.
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\small
|
||||
\setlength{\tabcolsep}{5pt}
|
||||
\begin{tabular}{@{}p{4.9cm}p{1.5cm}p{1.5cm}p{5.0cm}@{}}
|
||||
\toprule
|
||||
\textbf{Component} & \textbf{Plane} & \textbf{Status} & \textbf{Evidence / source} \\
|
||||
\midrule
|
||||
Cross-vendor inference ring (TP/PP, canonical \code{f32}) & Data & \built{} &
|
||||
\code{distributed.rs}, \code{distributed/socket.rs}, \code{ops.rs}~\cite{infstack} \\
|
||||
NCCL $+$ portable ring backends & Data & \built{} & \code{hanzo-quant/src/distributed/} \\
|
||||
\code{hanzo cluster} scan-to-join (bs58/QR token) & Data & \built{} &
|
||||
\code{hanzo-cli/src/cluster.rs} \\
|
||||
Five-backend engine, decode parity, quant zoo & Data & \built{} &
|
||||
companion stack paper~\cite{infstack} \\
|
||||
ZAP Go wire (zero-copy, $17\times$/$345\times$ vs JSON-RPC) & Transport & \built{} &
|
||||
\code{luxfi/zap}~\cite{zap,lp201} \\
|
||||
ZAP PQ handshake, \code{WrapPQ}, verified \code{PeerID} & Transport & \built{} &
|
||||
\code{conn\_pq.go} \\
|
||||
ZAP engine module (in-process adapter surface) & Transport & \partialb{} &
|
||||
\code{engine/src/zap/*.rs} (server/client) \\
|
||||
\quad out-of-process \code{zap://} (capnp wire) & Transport & \partialb{} &
|
||||
\code{engine/src/zap/transport.rs} (stub) \\
|
||||
Quasar BFT engine; Photon, Wave, Focus & Control & \built{} &
|
||||
\code{luxfi/consensus}~\cite{quasar,luxconsensus} \\
|
||||
Luminance reputation (100 base, $\times1.1$/$\times0.9$) & Control & \built{} &
|
||||
\code{photon/luminance.go} \\
|
||||
BLS$+$Pulsar$+$ML-DSA threshold signing (QuasarCert) & Control & \built{} &
|
||||
\code{protocol/quasar}~\cite{luxconsensus} \\
|
||||
\midrule
|
||||
Ring transport over ZAP (retire raw TCP) & Transport & \design{} & \S\ref{sec:transport}, Stage 1 \\
|
||||
Join token as PQ handshake credential & Transport & \design{} & \S\ref{sec:threat}, Gap 2 \\
|
||||
Consensus-owned membership \& placement map & Control & \design{} & \S\ref{sec:control}, Stage 2 \\
|
||||
Permissionless miner admission (stake bond) & Control & \design{} & \S\ref{sec:econ}, Stage 3 \\
|
||||
Luminance $\gets$ validated inference contribution & Control & \design{} & \S\ref{sec:control} \\
|
||||
Result verification (threshold attest $+$ spot check) & Control & \design{} & \S\ref{sec:threat}, Gap 3 \\
|
||||
Luminance-weighted reward / routing & Control & \design{} & \S\ref{sec:econ} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Built-versus-designed ledger. Every primitive (upper block) is code on \code{main},
|
||||
cited to its file with its measured cost where relevant; every network composition (lower block)
|
||||
is design, cited to the section that specifies it and the migration stage that ships it.}
|
||||
\label{tab:status}
|
||||
\end{table}
|
||||
|
||||
\paragraph{Reproducing the built numbers.} The ZAP figures ($2.9$~ns parse, $17\times$/$345\times$
|
||||
versus JSON-RPC, $114$K/$376$K TPS) reproduce with \code{go test -bench=. -benchmem} in
|
||||
\code{luxfi/zap} and \code{luxfi/consensus/bench/}; the Photon/Wave/luminance costs ($3.03$~\textmu s,
|
||||
$3.38$~\textmu s, $72$~ns) reproduce with \code{GOWORK=off go test -bench=. ./bench/} in
|
||||
\code{luxfi/consensus}, measured on Apple M1 Max. The cross-vendor ring runs with a
|
||||
\code{RING\_CONFIG} JSON per rank and \code{hanzo cluster init}/\code{join}; the cross-backend
|
||||
engine numbers are reproduced by the protocol in the companion stack paper~\cite{infstack}. None
|
||||
of these numbers is produced by this paper; each is attributed to its source so the blueprint's
|
||||
foundation is auditable.
|
||||
|
||||
\section{Limitations}
|
||||
\label{sec:limits}
|
||||
|
||||
The rigor is the persuasion, so the costs are as plain as the benefits.
|
||||
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=2pt]
|
||||
\item \textbf{The network is a composition, not a shipped network.} The primitives are built and
|
||||
cited; the assembly of \S\ref{sec:planes}--\S\ref{sec:econ} is \design{}. Nothing here should be
|
||||
read as ``Hanzo runs a permissionless GPU network today.'' It runs a cross-vendor ring on a
|
||||
trusted mesh (Stage 0) and ships every primitive the network needs.
|
||||
|
||||
\item \textbf{The ZAP engine wire is not yet turned on.} The Rust ZAP module is \partialb{}: its
|
||||
in-process surface is exercised, but the \code{zap://} out-of-process transport is a documented
|
||||
stub and a few methods return ``not yet wired'' (\S\ref{sec:zap}). Stage 1 depends on completing
|
||||
it. The Go ZAP wire it mirrors is production-deployed, so the risk is integration, not invention.
|
||||
|
||||
\item \textbf{Wide-area tensor transport is unproven at scale.} The ring was built for a LAN.
|
||||
Pipeline parallelism over a wide-area network is argued to be latency-tolerant
|
||||
(\S\ref{sec:latency}) but not measured here; the companion latency paper carries that budget, and
|
||||
a real deployment must validate it. Bandwidth, not just latency, may bound activation exchange
|
||||
for large hidden dimensions.
|
||||
|
||||
\item \textbf{Result verification adds cost.} Threshold attestation and redundant re-execution
|
||||
(\S\ref{sec:threat}) are not free: a QuasarCert verifies in $2$--$5$~ms and spot-check redundancy
|
||||
duplicates work on sampled requests. The scheme is designed to make the \emph{expected} overhead
|
||||
small (attest per epoch, spot-check a fraction) but it is a real tax the monolithic single-trust
|
||||
model does not pay --- the price of permissionlessness.
|
||||
|
||||
\item \textbf{Confidentiality from the operator is out of scope.} This design authenticates and
|
||||
verifies; it does not hide activations from the computing miner without a TEE tier
|
||||
(\S\ref{sec:threat}). Sensitive workloads need that orthogonal extension.
|
||||
|
||||
\item \textbf{Luminance re-binding is a policy change with game-theoretic edges.} Moving the
|
||||
\code{Illuminate} trigger from vote-success to inference-contribution is a one-line mechanism
|
||||
change but a large incentive-design question: collusion among a committee, grinding on the
|
||||
verification lottery, and reward-share disputes across a pipeline are real and not solved by the
|
||||
primitive alone. We name them; we do not claim to have priced them.
|
||||
\end{itemize}
|
||||
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
The heterogeneous GPU ring already solves the hard, unglamorous problem --- one model computing
|
||||
correctly across ROCm, CUDA, and Metal, bit-for-bit, over a canonical wire. What it lacks to
|
||||
become a network is not more numerics but two planes it does not have: a control plane to decide
|
||||
membership and placement among parties who do not trust each other, and a transport plane to
|
||||
authenticate the strangers it talks to. Both already exist as best-in-class Lux primitives ---
|
||||
Quasar consensus and the ZAP post-quantum wire --- and the entire contribution of this paper is
|
||||
to hold the three concerns apart and compose them: consensus schedules, ZAP carries, the ring
|
||||
computes. That decomplection is what a monolith cannot do and what makes ``any developer, any
|
||||
miner, contributes a GPU and is paid for inference work'' a buildable system rather than a slogan.
|
||||
We have drawn the line between built and designed down the middle of every claim: the ring, the
|
||||
scan-to-join CLI, the ZAP wire with its verified \code{PeerID}, the Quasar engine with luminance
|
||||
and threshold signing are code on \code{main}, cited and measured; the network that composes them
|
||||
--- consensus-owned membership, the ZAP-transported ring, permissionless admission, and result
|
||||
verification by threshold-signing over outputs --- is design, staged for a migration with no flag
|
||||
day. The primitives are done. The composition is the blueprint. We publish it so the rest of the
|
||||
AI-infrastructure community can build on the same three planes.
|
||||
|
||||
\begin{thebibliography}{20}
|
||||
\small
|
||||
|
||||
\bibitem{infstack}
|
||||
Hanzo AI --- ML Systems. \emph{Hanzo ML and Hanzo Engine: A Unified Multi-Backend, Multi-Modal
|
||||
Inference Stack}. Hanzo AI white paper, 2026 (companion; the data-plane engine, cross-vendor ring,
|
||||
and five-backend decode parity).
|
||||
|
||||
\bibitem{tenantcloud}
|
||||
Hanzo AI Research. \emph{The Unified Sovereign-Tenant Cloud: Linear Shared-Nothing Scaling via
|
||||
Per-Tenant SQLite and In-Process Composition}. Hanzo Industries white paper, 2026 (companion; the
|
||||
\code{hanzo.network} fabric, wallet/staking, TEE trust tiers, and PQ identity primitives).
|
||||
|
||||
\bibitem{latency}
|
||||
Hanzo AI --- ML Systems. \emph{The Latency Physics of Decentralized Decode: Pipeline Parallelism
|
||||
and Speculative Decoding over ZAP}. Hanzo AI white paper, 2026 (companion; the wide-area latency
|
||||
budget referenced in \S\ref{sec:latency}).
|
||||
|
||||
\bibitem{quasar}
|
||||
Lux Industries. \emph{Quasar: Post-Quantum DAG-Native BFT Consensus} (LP-020 / LP-105). Lux
|
||||
Network, 2026. \url{https://github.com/luxfi/consensus}.
|
||||
|
||||
\bibitem{luxconsensus}
|
||||
Lux Industries. \emph{luxfi/consensus}: the Quasar engine --- \code{protocol/photon} (Fisher--Yates
|
||||
committee selection, luminance), \code{protocol/wave} (FPC threshold voting), \code{protocol/focus},
|
||||
\code{protocol/quasar} (BLS $+$ Pulsar $+$ ML-DSA threshold signing). Source and benchmarks
|
||||
(\code{bench/}, \code{README.md}, \code{LLM.md}). \url{https://github.com/luxfi/consensus}.
|
||||
|
||||
\bibitem{zap}
|
||||
Lux Industries. \emph{ZAP: Zero-copy Application Protocol} --- the universal wire.
|
||||
\code{github.com/luxfi/zap} (\code{zap.go}, \code{conn\_pq.go} PQ handshake / \code{WrapPQ} /
|
||||
\code{PeerID}, \code{README.md} benchmarks).
|
||||
|
||||
\bibitem{lp201}
|
||||
Lux Industries. \emph{LP-201: ZAP as the Universal Wire} and \emph{LP-200: The ZAP Stack}. Lux
|
||||
Proposals, 2026. \url{https://github.com/luxfi/lps} (the $17\times$/$345\times$ vs JSON-RPC and
|
||||
$\sim$$4.5\times$ vs gRPC measurements).
|
||||
|
||||
\bibitem{mldsa}
|
||||
National Institute of Standards and Technology. \emph{FIPS 204: Module-Lattice-Based Digital
|
||||
Signature Standard (ML-DSA)}. NIST, 2024.
|
||||
|
||||
\bibitem{mlkem}
|
||||
National Institute of Standards and Technology. \emph{FIPS 203: Module-Lattice-Based
|
||||
Key-Encapsulation Mechanism Standard (ML-KEM)}. NIST, 2024.
|
||||
|
||||
\bibitem{bls}
|
||||
D. Boneh, B. Lynn, and H. Shacham. Short Signatures from the Weil Pairing. \emph{Journal of
|
||||
Cryptology}, 17(4):297--319, 2004.
|
||||
|
||||
\bibitem{petals}
|
||||
A. Borzunov et al. \emph{Petals: Collaborative Inference and Fine-tuning of Large Models}. ACL
|
||||
System Demonstrations, 2023 (decentralized inference over volunteer GPUs; no BFT control plane or
|
||||
PQ-authenticated transport).
|
||||
|
||||
\bibitem{kwon2023}
|
||||
W. Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention.
|
||||
\emph{SOSP}, 2023 (the monolithic single-operator serving baseline).
|
||||
|
||||
\bibitem{pinkas}
|
||||
B. Pinkas and M. Reiter, and the committee-grinding threat that motivates rejection-sampled
|
||||
uniform selection in \code{photon/emitter.go}. (Bias-free sortition under a grinding adversary.)
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
@@ -0,0 +1,570 @@
|
||||
\documentclass[11pt]{article}
|
||||
\input{shared/kernel-preamble.tex}
|
||||
\usepackage{pgfplots}
|
||||
\pgfplotsset{compat=1.16}
|
||||
|
||||
\newcommand{\Tring}{\ensuremath{T_{\mathrm{ring}}}}
|
||||
\newcommand{\dmodel}{\ensuremath{d_{\mathrm{model}}}}
|
||||
\newcommand{\Gam}{\ensuremath{\Gamma}}
|
||||
|
||||
\title{\textbf{The Latency Physics of Geo-Distributed\\ Autoregressive Decode}}
|
||||
\author{Hanzo AI --- ML Systems}
|
||||
\date{An analytical systems paper --- companion to the cross-backend inference measurements}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Splitting a large language model across machines that sit far apart is often imagined as
|
||||
``BitTorrent for AI'': pool everyone's memory and the model runs. For \emph{loading} weights and
|
||||
for \emph{batched throughput} that intuition is right. For \emph{single-stream, interactive decode}
|
||||
it is exactly wrong, and this paper derives why. We separate the two costs of a node boundary.
|
||||
\textbf{Bandwidth is trivial}: the only tensor that crosses a pipeline boundary is the hidden state
|
||||
for the in-flight tokens, \dmodel${}\times n_{\mathrm{tok}}\times b$ bytes --- $\sim$32\,KiB for one
|
||||
token of a 700B-class model --- which serializes in $\approx$0.1\,ms over our measured 280\,MB/s
|
||||
wired mesh, and the KV cache never crosses at all. \textbf{Latency is the wall}: autoregressive
|
||||
decode is a strict dependency chain, so each generated token must traverse every node once, and the
|
||||
next token cannot begin until the current one returns. Per-token decode rate is therefore
|
||||
$r=1/(C+\Tring)$, where $C$ is the summed compute and $\Tring=\sum_i \ell_i$ is the one-way latency
|
||||
of a full loop around the ring. Because \Tring{} adds to a per-token denominator, a geo-distributed
|
||||
ring (Hong Kong\,$\to$\,Dublin\,$\to$\,New York, $\Tring\!\approx\!280$\,ms) floors single-stream
|
||||
decode at $\approx$3\,tok/s \emph{regardless of GPU speed} --- the wall is the speed of light in
|
||||
fiber, not the accelerator. The headline result is the cure: \textbf{speculative decoding converts
|
||||
$K$ sequential network traversals into one}. A local draft model proposes $K$ tokens at full LAN
|
||||
speed; the distributed model verifies all $K$ in a single pipeline pass; at acceptance rate $\alpha$
|
||||
the effective rate is $r_{\mathrm{spec}}=\frac{1-\alpha^{K+1}}{1-\alpha}\big/(Kc_d+C+\Tring)$, which
|
||||
recovers 12--21\,tok/s on the same 280\,ms ring --- interactive speeds from an intercontinental
|
||||
deployment. We close with four placement laws, including a derivation that pipeline parallelism
|
||||
($O(\text{nodes})$ hops/token) strictly dominates tensor parallelism ($O(\text{layers})$
|
||||
all-reduces/token) over any slow link by a factor of $\sim\!4L/N$. Every model here is analytical
|
||||
and labelled as such; the one measured number is the 280\,MB/s link.
|
||||
\end{abstract}
|
||||
|
||||
\section{Two regimes: dependency-bound vs.\ throughput-bound}
|
||||
\label{sec:dichotomy}
|
||||
|
||||
The whole paper turns on one distinction, so we state it first.
|
||||
|
||||
A computation is \textbf{throughput-bound} when its pieces are independent: they can be done in any
|
||||
order, in parallel, and the only thing that matters is aggregate bytes-or-flops per second. Network
|
||||
\emph{latency} barely touches such a workload --- a slow link means each piece arrives a little
|
||||
later, but no piece waits on another, so the pipe stays full and throughput is set by
|
||||
\emph{bandwidth}. Downloading a file is the canonical case; so is a batch of a thousand independent
|
||||
inference requests.
|
||||
|
||||
A computation is \textbf{dependency-bound} when its pieces form a chain: piece $t+1$ cannot start
|
||||
until piece $t$ finishes. Now latency is on the critical path of \emph{every} step and cannot be
|
||||
hidden, because there is no independent work to overlap it with. Bandwidth is almost irrelevant ---
|
||||
the messages are tiny --- and the wall is the round-trip time.
|
||||
|
||||
Autoregressive decode is the purest dependency chain in production machine learning. Token $t+1$ is
|
||||
sampled from the model's output after token $t$ has been fed back as input; \emph{within} a token,
|
||||
layer $\ell+1$ consumes the output of layer $\ell$. There is exactly one token in flight for a
|
||||
single stream, so there is nothing to overlap. Distributing that chain across nodes does not add
|
||||
parallelism to a single stream --- it adds hops to the chain. This is why the ``pool everyone's
|
||||
GPUs'' picture, which is a throughput picture, gives the wrong answer for interactive decode, which
|
||||
is a dependency problem. The rest of the paper makes both halves quantitative.
|
||||
|
||||
\paragraph{Parameters and provenance.}
|
||||
Table~\ref{tab:params} lists the symbols and the reference values used in the worked numbers.
|
||||
\emph{One} value is measured --- the 280\,MB/s link goodput, from iperf3 on our direct 2.5\,GbE
|
||||
mesh. The representative one-way fiber latencies are physical (Section~\ref{sec:wall}) and are not
|
||||
our measurements; the model geometry (\dmodel, $L$) is that of a public 700B-class dense/MoE model;
|
||||
the local decode ceiling $r_0$ and draft rate $r_d$ are stated parameters, not claims about a
|
||||
specific box. Every rate, table, and plot downstream is a closed-form evaluation of the models in
|
||||
this paper, not an experiment.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\small
|
||||
\caption{Symbols and reference values. Only $W$ is measured.}
|
||||
\label{tab:params}
|
||||
\begin{tabular}{l l l l}
|
||||
\toprule
|
||||
Symbol & Meaning & Reference value & Provenance\\
|
||||
\midrule
|
||||
$N$ & pipeline nodes (stages) & 3 & scenario\\
|
||||
$L$ & transformer layers & $\sim$100 & 700B-class model\\
|
||||
\dmodel & hidden size & 16384 & 700B-class model\\
|
||||
$b$ & activation bytes/element (bf16) & 2 & format\\
|
||||
$W$ & per-link goodput & \textbf{280\,MB/s} & \textbf{measured (2.5\,GbE)}\\
|
||||
$\ell_i$ & one-way latency of hop $i$ & 40--130\,ms (geo) & physical (fiber)\\
|
||||
$\Tring$ & ring latency $\sum_{i=1}^N \ell_i$ & 280\,ms (geo) & physical (fiber)\\
|
||||
$C=1/r_0$ & target compute per decode step & 33\,ms ($r_0{=}30$) & parameter\\
|
||||
$c_d=1/r_d$& local draft time per token & 5\,ms ($r_d{=}200$) & parameter\\
|
||||
$\alpha$ & speculative acceptance rate & 0.6--0.9 & parameter\\
|
||||
$K$ & draft length (tokens/round) & 1--24 & swept\\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\section{The crossing tensor: bandwidth is trivial}
|
||||
\label{sec:bandwidth}
|
||||
|
||||
In pipeline parallelism (PP) the model is partitioned by \emph{layer}: node $i$ owns a contiguous
|
||||
stack of $L/N$ layers. The only data that crosses the boundary between node $i$ and node $i{+}1$ is
|
||||
the residual-stream hidden state at that layer cut --- one activation tensor of shape
|
||||
$[\,n_{\mathrm{tok}},\ \dmodel\,]$ in the activation dtype. Its size is
|
||||
\begin{equation}
|
||||
B_{\mathrm{hop}} \;=\; n_{\mathrm{tok}}\cdot \dmodel \cdot b \quad\text{bytes},
|
||||
\label{eq:bhop}
|
||||
\end{equation}
|
||||
and the time to push it across a link of goodput $W$ is $t_{\mathrm{xmit}}=B_{\mathrm{hop}}/W$.
|
||||
Three properties of Eq.~\eqref{eq:bhop} decide the paper.
|
||||
|
||||
\paragraph{It is independent of model depth.} $B_{\mathrm{hop}}$ contains no $L$. Only the
|
||||
\emph{width} \dmodel{} and the number of in-flight tokens appear, because exactly one activation
|
||||
crosses each cut no matter how many layers sit behind it. A deeper model costs more
|
||||
\emph{compute} per node but not more \emph{bytes} per hop. This is the structural reason PP is the
|
||||
right WAN topology (Section~\ref{sec:pptp}).
|
||||
|
||||
\paragraph{The KV cache never crosses the wire.} Each node holds the keys and values for
|
||||
\emph{its own} layers and reuses them locally on every step. The KV cache is the large, context-
|
||||
growing state; it stays put. What crosses is only the small, fixed-size residual vector. Distributing
|
||||
the model does not distribute the KV traffic --- there is none across the boundary.
|
||||
|
||||
\paragraph{For decode it is a rounding error.} For a single decoded token, $n_{\mathrm{tok}}=1$.
|
||||
With the 700B-class width $\dmodel=16384$ and bf16 ($b=2$),
|
||||
\[
|
||||
B_{\mathrm{hop}} = 1\cdot 16384 \cdot 2 = 32{,}768~\text{bytes} = 32~\text{KiB},
|
||||
\qquad
|
||||
t_{\mathrm{xmit}} = \frac{32{,}768}{280\times10^6} \approx 0.117~\text{ms}
|
||||
\]
|
||||
over our \emph{measured} 280\,MB/s mesh. Even at a serving-scale aggregate of 1000\,tok/s each hop
|
||||
carries $\sim$32\,MB/s --- about a tenth of a single 2.5\,GbE link, and nothing on a datacenter WAN.
|
||||
\textbf{Bandwidth is never the constraint for decode.} (It can matter for large-prompt prefill,
|
||||
where $n_{\mathrm{tok}}=P$: a 2000-token prompt crosses 65.5\,MB per hop, $\approx$234\,ms to
|
||||
serialize over 280\,MB/s --- but that is a bulk, streamable, latency-insensitive transfer that a
|
||||
fatter pipe fixes. It is a throughput cost, not a wall; Section~\ref{sec:regimes}.)
|
||||
|
||||
\section{The wall: latency floors single-stream decode}
|
||||
\label{sec:wall}
|
||||
|
||||
Now the cost that does not go away. Generating one token drives the hidden state through all $N$
|
||||
nodes in sequence: node~1 embeds and runs its layers, hands off to node~2, and so on to node~$N$,
|
||||
which produces the final hidden state; the model head samples the next token, which must return to
|
||||
node~1 to begin the next step. Counting hops on the critical path: $N-1$ forward hand-offs plus one
|
||||
\emph{return} hop that closes the loop, i.e.\ $N$ inter-node latencies per token. Writing $\ell_i$
|
||||
for the one-way latency of hop $i$ and $c_i$ for node $i$'s compute on its layer slice,
|
||||
\begin{equation}
|
||||
T_{\mathrm{tok}} \;=\; \underbrace{\sum_{i=1}^{N} c_i}_{\textstyle C}
|
||||
\;+\; \underbrace{\sum_{i=1}^{N} \ell_i}_{\textstyle \Tring},
|
||||
\qquad
|
||||
r \;=\; \frac{1}{T_{\mathrm{tok}}} \;=\; \frac{1}{C + \Tring}.
|
||||
\label{eq:wall}
|
||||
\end{equation}
|
||||
Two facts about Eq.~\eqref{eq:wall} carry the argument.
|
||||
|
||||
\paragraph{PP gives a single stream zero compute speedup --- only added latency.} For one in-flight
|
||||
token, only one node computes at a time; the other $N-1$ idle (the pipeline is empty). So the total
|
||||
compute $C=\sum_i c_i$ equals the monolithic single-node decode compute; splitting the model does
|
||||
not make one stream's arithmetic faster. What splitting \emph{adds} is $\Tring$. PP buys memory
|
||||
capacity (a 700B model that fits nowhere else) and, with batching, throughput
|
||||
(Section~\ref{sec:regimes}) --- never single-stream latency.
|
||||
|
||||
\paragraph{Co-locate the turn so the ring closes in exactly $N$ hops.} Keep the embedding, the first
|
||||
layers, the LM head, and the sampler on the same entry node. Then the ``sample $\to$ embed next
|
||||
token'' turnaround happens locally, with no network step, and the return is a single hop. Split the
|
||||
head from the embedding and you pay an extra hop every token (hidden state to the sampler node, then
|
||||
the token id back to the embedder). One node owns both ends of the ring.
|
||||
|
||||
\paragraph{The geo table.} Fix a fast local ceiling $r_0=30$\,tok/s ($C=33$\,ms) and sweep the ring
|
||||
latency. Table~\ref{tab:geo} and Figure~\ref{fig:wall} evaluate Eq.~\eqref{eq:wall}.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\small
|
||||
\caption{Single-stream decode vs.\ ring latency, from $r=1/(C+\Tring)$ with $C=33$\,ms
|
||||
($r_0=30$\,tok/s). Analytical.}
|
||||
\label{tab:geo}
|
||||
\begin{tabular}{l r r r}
|
||||
\toprule
|
||||
Deployment (3-node ring) & $\Tring$ (ms) & decode (tok/s) & \% of $r_0$\\
|
||||
\midrule
|
||||
Single node (no network) & 0 & 30.0 & 100\%\\
|
||||
Rack LAN ($3\times0.1$\,ms) & 0.3 & 29.7 & 99\%\\
|
||||
Same metro ($3\times0.7$\,ms) & 2 & 28.3 & 94\%\\
|
||||
Regional, one continent ($3\times7$\,ms) & 20 & 18.8 & 63\%\\
|
||||
Continental US ($3\times20$\,ms) & 60 & 10.7 & 36\%\\
|
||||
Two-continent ($3\times50$\,ms) & 150 & 5.5 & 18\%\\
|
||||
Global ring HK$\to$Dublin$\to$NY & 280 & \textbf{3.2} & \textbf{11\%}\\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\begin{figure}[h]
|
||||
\centering
|
||||
\begin{tikzpicture}
|
||||
\begin{axis}[
|
||||
width=0.86\linewidth, height=6.2cm,
|
||||
xmode=log, log basis x=10,
|
||||
xlabel={ring latency $\Tring$ (ms, log scale)},
|
||||
ylabel={single-stream decode (tok/s)},
|
||||
xmin=0.1, xmax=600, ymin=0, ymax=125,
|
||||
grid=both, grid style={gray!18},
|
||||
legend pos=north east, legend cell align=left,
|
||||
every axis plot/.append style={very thick},
|
||||
]
|
||||
\addplot[blue] coordinates {
|
||||
(0.1,118.58)(0.2,117.19)(0.3,115.83)(0.5,113.21)(1,107.14)(2,96.77)(5,75.00)(10,54.55)(20,35.29)(40,20.69)(60,14.63)(100,9.23)(150,6.32)(200,4.80)(280,3.47)(400,2.45)(600,1.64)};
|
||||
\addlegendentry{fast GPUs, $r_0{=}120$}
|
||||
\addplot[red] coordinates {
|
||||
(0.1,29.91)(0.2,29.82)(0.3,29.73)(0.5,29.56)(1,29.13)(2,28.30)(5,26.09)(10,23.08)(20,18.75)(40,13.64)(60,10.71)(100,7.50)(150,5.45)(200,4.29)(280,3.19)(400,2.31)(600,1.58)};
|
||||
\addlegendentry{$r_0{=}30$}
|
||||
\draw[dashed,gray] (axis cs:280,0) -- (axis cs:280,3.47);
|
||||
\node[anchor=south,font=\scriptsize,gray] at (axis cs:280,3.9) {global ring};
|
||||
\node[anchor=west,font=\scriptsize] at (axis cs:1.2,90) {LAN};
|
||||
\node[anchor=west,font=\scriptsize] at (axis cs:22,42) {regional};
|
||||
\node[anchor=west,font=\scriptsize] at (axis cs:110,20) {inter-continental};
|
||||
\end{axis}
|
||||
\end{tikzpicture}
|
||||
\caption{The wall. Two accelerator ceilings, $r_0=30$ and $r_0=120$\,tok/s, collapse onto the
|
||||
\emph{same} curve as $\Tring$ grows: past $\sim$100\,ms the two are within $\sim$2\,tok/s. Beyond a
|
||||
few tens of milliseconds of ring latency, single-stream decode is set by the network, not the GPU.
|
||||
Analytical ($r=1/(C+\Tring)$).}
|
||||
\label{fig:wall}
|
||||
\end{figure}
|
||||
|
||||
\paragraph{The GPU stops mattering.} The most important reading of Figure~\ref{fig:wall} is the
|
||||
convergence of the two curves. Hold $\Tring=280$\,ms and vary the local ceiling: $r_0=15\to
|
||||
2.9$, $30\to3.2$, $60\to3.4$, $120\to3.5$, $1000\to3.6$\,tok/s. A $67\times$ faster accelerator buys
|
||||
a $1.2\times$ faster stream. Once the ring is long, $r\to 1/\Tring$ and the accelerator is a rounding
|
||||
error. Buying H100s to serve one interactive user over a global ring is spending on the wrong axis.
|
||||
|
||||
\paragraph{Why it is \emph{physics}.} $\Tring$ has a hard floor: light in fiber travels at
|
||||
$c/n\approx2.04\times10^{5}$\,km/s ($n\approx1.47$), i.e.\ $\approx4.9\,\mu$s/km one-way, so a
|
||||
real (non-geodesic) route accrues $\gtrsim5$--$7$\,ms per 1000\,km. Hong Kong to New York is
|
||||
$\sim$13{,}000\,km great-circle; no switch, protocol, or budget reduces that hop below $\sim$110\,ms
|
||||
one-way. The 130/40/110\,ms legs of our example ring are representative real fiber latencies, and
|
||||
they are irreducible. This is the sense in which the title is literal: the wall is a property of
|
||||
space and light, and the systems question is not how to break it but how to \emph{stop paying it
|
||||
per token} --- which is Section~\ref{sec:spec}.
|
||||
|
||||
\section{``BitTorrent for AI'' is half-right}
|
||||
\label{sec:bittorrent}
|
||||
|
||||
The appeal of swarm inference --- Petals-style pooling of volunteer GPUs across the internet ---
|
||||
rests on an analogy to BitTorrent~\cite{petals}. The analogy is exact where it applies and
|
||||
misleading where it does not, and the dividing line is precisely Section~\ref{sec:dichotomy}'s.
|
||||
|
||||
BitTorrent is fast because a file's chunks are \emph{independent}. A peer fetches them in any order,
|
||||
from many sources at once; a high-latency peer just means a given chunk lands a little later, never
|
||||
that another chunk waits. File download is throughput-bound and latency-insensitive --- the ideal
|
||||
case for pooling far-flung capacity. \textbf{Everything about serving a distributed model that is
|
||||
throughput-bound inherits this good behaviour}: shipping the weights to the nodes, prefilling a long
|
||||
prompt, and running a large batch of independent requests (Section~\ref{sec:regimes}). For these, a
|
||||
geographically spread swarm is genuinely attractive.
|
||||
|
||||
Single-stream decode is the opposite object. Its ``chunks'' --- the per-token forward passes --- are
|
||||
\emph{maximally dependent}: each is the input to the next. There is no order but the one order, no
|
||||
parallel sources, nothing to overlap the latency against. Pooling distant GPUs does not add
|
||||
throughput to one stream; it lengthens the dependency chain by the distance between the pools. The
|
||||
BitTorrent intuition, applied here, predicts the exact opposite of what happens. So the slogan is
|
||||
half-right: \textbf{BitTorrent for AI works for weight distribution, prefill, and batched
|
||||
throughput, and breaks for interactive single-stream latency} --- and it breaks hardest precisely
|
||||
where users feel it, in the token-by-token cadence of a chat.
|
||||
|
||||
\section{The regime map: where geo-distribution wins}
|
||||
\label{sec:regimes}
|
||||
|
||||
The same 280\,ms ring is cheap or catastrophic depending on how many tokens amortize it.
|
||||
|
||||
\paragraph{Prefill / TTFT amortizes the traversal over the whole prompt.} A prompt of $P$ tokens
|
||||
goes through the pipeline in \emph{one} pass. The ring latency is paid once, not per token:
|
||||
\begin{equation}
|
||||
T_{\mathrm{TTFT}} \approx C_{\mathrm{prefill}}(P) + \Tring .
|
||||
\label{eq:ttft}
|
||||
\end{equation}
|
||||
For a 2000-token prompt whose prefill compute is $\sim$1--2\,s on a big model, a $+$280\,ms ring is a
|
||||
15--30\% tax on time-to-first-token --- noticeable, not fatal. Per prompt-token the traversal is
|
||||
$280\,\text{ms}/2000\approx0.14$\,ms: amortized to nothing. Prefill is throughput-bound (Section~
|
||||
\ref{sec:bittorrent}); a geo ring is fine for it.
|
||||
|
||||
\paragraph{Batched throughput fills the pipe.} With continuous batching, $N$ pipeline stages run on
|
||||
$N$ different requests at once. Each \emph{individual} stream still crawls at
|
||||
Eq.~\eqref{eq:wall}'s $r=1/(C+\Tring)$, but the \emph{aggregate} scales with pipeline depth toward
|
||||
$\approx N\!\cdot\!(\text{per-stage rate})$ until a stage saturates. This is the BitTorrent regime
|
||||
done right: many independent streams are the independent chunks that keep the pipe full. High
|
||||
aggregate tok/s, high per-stream latency --- excellent for offline/batch, useless as a proxy for one
|
||||
user's experience.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\small
|
||||
\caption{The regime map. Geo-distribution is bandwidth-cheap everywhere; it is latency-cheap only
|
||||
where many tokens share one traversal.}
|
||||
\label{tab:regime}
|
||||
\begin{tabular}{l l l l}
|
||||
\toprule
|
||||
Workload & Bound by & Geo ring 280\,ms? & Why\\
|
||||
\midrule
|
||||
Weight distribution & throughput & \textbf{great} & independent chunks (BitTorrent)\\
|
||||
Prefill / TTFT & throughput & \textbf{fine} & one traversal per prompt (Eq.~\ref{eq:ttft})\\
|
||||
Batched serving (aggregate) & throughput & \textbf{great} & depth fills the pipe, $\sim\!N\times$\\
|
||||
\textbf{Single-stream decode} & \textbf{dependency} & \textbf{hard} & one traversal \emph{per token} (Eq.~\ref{eq:wall})\\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The map (Table~\ref{tab:regime}) is the practical statement of the thesis: only the bottom row is
|
||||
hard, and it is hard because it is the only dependency-bound row. The next section attacks exactly
|
||||
that row.
|
||||
|
||||
\section{The cure: speculative decoding amortizes the traversal}
|
||||
\label{sec:spec}
|
||||
|
||||
The wall is that decode pays $\Tring$ \emph{per token}. Speculative decoding~\cite{leviathan,chen}
|
||||
changes the denominator: it makes the network pay $\Tring$ \emph{per round}, where a round produces
|
||||
several tokens. It is the one lever that attacks the dependency structure itself rather than the
|
||||
constants.
|
||||
|
||||
\paragraph{Mechanism.} A small \emph{draft} model runs \emph{locally} on the entry node --- no
|
||||
network in its loop --- and autoregressively proposes $K$ candidate tokens. Those $K$ tokens are
|
||||
then fed as a length-$K$ sequence to the large distributed \emph{target} model, which verifies them
|
||||
in a \emph{single} pipeline pass, prefill-style: the $K$ candidates flow through the ring together,
|
||||
one traversal, not $K$. The target accepts the longest correct prefix under the standard
|
||||
speculative-sampling rule (which preserves the target's exact output distribution) and corrects the
|
||||
first mismatch. The expensive thing --- the trip around the ring --- happens once per round instead
|
||||
of once per token.
|
||||
|
||||
\paragraph{Expected tokens per round.} Let each draft token be accepted independently with
|
||||
probability $\alpha$ (the empirical acceptance rate). The number of accepted drafts $A$ satisfies
|
||||
$\Pr[A\ge j]=\alpha^{j}$ for $j\le K$, so
|
||||
$\mathbb{E}[A]=\sum_{j=1}^{K}\alpha^{j}=\alpha\frac{1-\alpha^{K}}{1-\alpha}$. The target always
|
||||
contributes one further token (the correction on the first reject, or a bonus token if all $K$ are
|
||||
accepted), giving the closed form of Leviathan et al.~\cite{leviathan}:
|
||||
\begin{equation}
|
||||
\Gam(\alpha,K)\;=\;1+\mathbb{E}[A]\;=\;\frac{1-\alpha^{K+1}}{1-\alpha}
|
||||
\qquad\text{expected tokens per round.}
|
||||
\label{eq:gamma}
|
||||
\end{equation}
|
||||
|
||||
\paragraph{Effective rate.} A round costs one local draft of $K$ tokens plus one target
|
||||
verification pass (one traversal of the ring):
|
||||
\begin{equation}
|
||||
T_{\mathrm{round}} \;=\; \underbrace{K\,c_d}_{\text{local draft}}
|
||||
\;+\; \underbrace{C_{\mathrm{verify}}(K) + \Tring}_{\text{one target traversal}},
|
||||
\qquad
|
||||
r_{\mathrm{spec}}(\alpha,K)\;=\;\frac{\Gam(\alpha,K)}{T_{\mathrm{round}}}.
|
||||
\label{eq:rspec}
|
||||
\end{equation}
|
||||
In the memory-bandwidth-bound decode regime, verifying $K\!\le\!16$ tokens streams the weights the
|
||||
\emph{same} once as a single decode step, so $C_{\mathrm{verify}}(K)\approx C$ --- the reason
|
||||
speculation is nearly free on the compute side.\footnote{For large $K$ the verify pass eventually
|
||||
becomes compute-bound (prefill-like) and $C_{\mathrm{verify}}(K)$ grows with $K$; that is one of the
|
||||
two forces creating an optimal $K$ below, the other being the geometric decay $\alpha^{K}$ of the
|
||||
marginal acceptance. We take $C_{\mathrm{verify}}\!\approx\!C$ over the plotted range $K\le24$, which
|
||||
is accurate for a 700B-class model where 24 tokens is still far below the arithmetic-intensity
|
||||
knee.} Substituting the closed form,
|
||||
\begin{equation}
|
||||
\boxed{\;
|
||||
r_{\mathrm{spec}}(\alpha,K)\;=\;\frac{1}{\,K\,c_d + C + \Tring\,}\cdot\frac{1-\alpha^{K+1}}{1-\alpha}
|
||||
\;}
|
||||
\label{eq:boxed}
|
||||
\end{equation}
|
||||
The structure is the whole point: $\Tring$ sits in the denominator once, divided into by
|
||||
$\Gam(\alpha,K)$ tokens. The effective per-token network cost falls from $\Tring$ to
|
||||
$\Tring/\Gam(\alpha,K)$.
|
||||
|
||||
\paragraph{Numbers on the global ring.} Take the intercontinental $\Tring=280$\,ms, target
|
||||
$C=33$\,ms, and a fast local draft $c_d=5$\,ms ($r_d=200$\,tok/s). Baseline single-stream decode is
|
||||
$1/(0.033+0.280)=3.2$\,tok/s. Table~\ref{tab:spec} and Figure~\ref{fig:spec} evaluate
|
||||
Eq.~\eqref{eq:boxed}.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\small
|
||||
\caption{Speculative decode on the 280\,ms global ring, from Eq.~\eqref{eq:boxed}
|
||||
($C{=}33$\,ms, $c_d{=}5$\,ms). $\Gam$ = expected tokens/round; $r$ = effective tok/s.
|
||||
Baseline (no speculation) $=3.2$\,tok/s. Analytical.}
|
||||
\label{tab:spec}
|
||||
\begin{tabular}{c cc cc cc c}
|
||||
\toprule
|
||||
& \multicolumn{2}{c}{$K=4$} & \multicolumn{2}{c}{$K=8$} & \multicolumn{2}{c}{$K=16$} & best\\
|
||||
\cmidrule(lr){2-3}\cmidrule(lr){4-5}\cmidrule(lr){6-7}
|
||||
$\alpha$ & $\Gam$ & $r$ & $\Gam$ & $r$ & $\Gam$ & $r$ & tok/s ($K^{\star}$)\\
|
||||
\midrule
|
||||
0.6 & 2.31 & 6.9 & 2.47 & 7.0 & 2.50 & 6.4 & 7.1 ($K{=}6$)\\
|
||||
0.7 & 2.77 & 8.3 & 3.20 & 9.1 & 3.33 & 8.5 & 9.1 ($K{=}8$)\\
|
||||
0.8 & 3.36 & 10.1 & 4.33 & 12.3 & 4.89 & 12.4 & 12.7 ($K{=}12$)\\
|
||||
0.9 & 4.10 & 12.3 & 6.13 & 17.3 & 8.33 & 21.2 & 21.6 ($K{=}21$)\\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\begin{figure}[h]
|
||||
\centering
|
||||
\begin{tikzpicture}
|
||||
\begin{axis}[
|
||||
width=0.86\linewidth, height=6.4cm,
|
||||
xlabel={draft length $K$ (tokens/round)},
|
||||
ylabel={effective decode (tok/s)},
|
||||
xmin=1, xmax=24, ymin=0, ymax=24,
|
||||
grid=both, grid style={gray!18},
|
||||
legend pos=north west, legend cell align=left,
|
||||
every axis plot/.append style={very thick},
|
||||
]
|
||||
\addplot[black!70,dashed,thick] coordinates {(1,3.19)(24,3.19)};
|
||||
\addlegendentry{baseline (no spec.), 3.2}
|
||||
\addplot[red] coordinates {(1,5.97)(2,8.38)(3,10.47)(4,12.29)(5,13.85)(6,15.20)(7,16.35)(8,17.34)(9,18.18)(10,18.89)(11,19.48)(12,19.98)(13,20.38)(14,20.72)(15,20.98)(16,21.18)(17,21.34)(18,21.44)(19,21.51)(20,21.55)(21,21.55)(22,21.53)(23,21.48)(24,21.42)};
|
||||
\addlegendentry{$\alpha=0.9$}
|
||||
\addplot[orange] coordinates {(1,5.65)(2,7.55)(3,8.99)(4,10.08)(5,10.90)(6,11.51)(7,11.95)(8,12.25)(9,12.46)(10,12.58)(11,12.64)(12,12.66)(13,12.63)(14,12.58)(15,12.51)(16,12.43)(17,12.33)(18,12.22)(19,12.10)(20,11.99)(21,11.86)(22,11.74)(23,11.62)(24,11.49)};
|
||||
\addlegendentry{$\alpha=0.8$}
|
||||
\addplot[teal] coordinates {(1,5.34)(2,6.77)(3,7.71)(4,8.32)(5,8.69)(6,8.91)(7,9.02)(8,9.05)(9,9.04)(10,8.99)(11,8.92)(12,8.84)(13,8.75)(14,8.65)(15,8.56)(16,8.45)(17,8.35)(18,8.26)(19,8.16)(20,8.06)(21,7.97)(22,7.87)(23,7.78)(24,7.69)};
|
||||
\addlegendentry{$\alpha=0.7$}
|
||||
\addplot[blue!55] coordinates {(1,5.03)(2,6.06)(3,6.63)(4,6.92)(5,7.04)(6,7.08)(7,7.06)(8,7.00)(9,6.93)(10,6.86)(11,6.77)(12,6.69)(13,6.60)(14,6.52)(15,6.44)(16,6.35)(17,6.28)(18,6.20)(19,6.12)(20,6.05)(21,5.98)(22,5.91)(23,5.84)(24,5.77)};
|
||||
\addlegendentry{$\alpha=0.6$}
|
||||
\end{axis}
|
||||
\end{tikzpicture}
|
||||
\caption{The cure. Effective single-stream decode on the \emph{same} 280\,ms global ring, from
|
||||
Eq.~\eqref{eq:boxed}. Speculation lifts the 3.2\,tok/s baseline (dashed) to 7--21\,tok/s. Each curve
|
||||
has an optimum $K$: too small wastes the traversal, too large is drowned by the geometric decay
|
||||
$\alpha^K$ and the growing draft cost $Kc_d$. Higher acceptance $\alpha$ both raises the ceiling and
|
||||
pushes $K^{\star}$ out. Analytical.}
|
||||
\label{fig:spec}
|
||||
\end{figure}
|
||||
|
||||
\paragraph{Reading the result.} On a ring that floors naive decode at 3.2\,tok/s, a draft with
|
||||
acceptance $\alpha=0.8$ recovers $\sim$12.7\,tok/s ($4.0\times$) and $\alpha=0.9$ recovers
|
||||
$\sim$21.6\,tok/s ($6.8\times$) --- from a bad slideshow to a readable stream, on the identical
|
||||
intercontinental deployment, with no faster hardware and no shorter cable. The mechanism is not a
|
||||
constant-factor tuning; it is a change in what the network is charged for. Naive decode pays one
|
||||
traversal per emitted token; speculation pays one traversal per $\Gam(\alpha,K)$ emitted tokens, and
|
||||
$\Gam$ is exactly the knob acceptance gives you. This is why speculative decoding is \emph{the}
|
||||
lever for interactive geo-distributed serving, not one option among several: it is the only cure
|
||||
that divides the wall instead of chipping at it.
|
||||
|
||||
\paragraph{The optimum is real and worth hitting.} Each curve in Figure~\ref{fig:spec} peaks. Below
|
||||
$K^{\star}$ the fixed $\Tring$ is shared over too few tokens; above it, marginal drafts are accepted
|
||||
with probability $\alpha^{K}\!\to\!0$ while still costing draft time $c_d$ and, eventually, verify
|
||||
compute. The peak drifts right with $\alpha$ ($K^{\star}\!:\,6\to8\to12\to21$ for
|
||||
$\alpha=0.6\to0.9$) because a better draft keeps paying off for longer. A deployment should tune $K$
|
||||
to its measured acceptance and ring, not to a fixed default --- the difference between $K{=}4$ and
|
||||
$K^{\star}$ is $\sim$1.7$\times$ at $\alpha=0.9$.
|
||||
|
||||
\section{Placement laws}
|
||||
\label{sec:placement}
|
||||
|
||||
Four heuristics fall directly out of the models above; the first two are strategy, the last two are
|
||||
free.
|
||||
|
||||
\begin{enumerate}[leftmargin=1.5em,itemsep=3pt]
|
||||
\item \textbf{Regional-first placement.} $\Tring$ dominates $r$ (Eq.~\ref{eq:wall}), so pack the
|
||||
whole pipeline into one region before spanning regions. A same-metro ring ($\Tring\!\approx\!2$\,ms)
|
||||
is 94\% of local; a global ring is 11\%. Span a WAN only when memory capacity forces it, and when you
|
||||
do, expect to pay for speculation (Section~\ref{sec:spec}) to stay interactive. LAN neighbours are
|
||||
near-native; distant neighbours are a different regime.
|
||||
\item \textbf{Speculate across every slow boundary.} Any ring above a few tens of milliseconds
|
||||
should run a local draft. It is the only lever that touches the $\Tring$-per-token term, and it
|
||||
composes with placement: regional-first shortens $\Tring$, speculation amortizes whatever remains.
|
||||
\item \textbf{Co-locate embed $+$ first layers $+$ head $+$ sampler on the entry node} (free). Makes
|
||||
the sample$\to$embed turnaround local and closes the ring in exactly $N$ hops instead of $N{+}1$
|
||||
(Section~\ref{sec:wall}). One node owns both ends.
|
||||
\item \textbf{Keep the KV cache local to each node's layers} (free, and automatic in PP). The KV
|
||||
cache never crosses the wire; only the 32\,KiB hidden state does (Section~\ref{sec:bandwidth}). Do
|
||||
nothing that would ship KV state between nodes per step --- it is the large state, and it has no
|
||||
reason to move.
|
||||
\end{enumerate}
|
||||
|
||||
\section{Why pipeline-, not tensor-, parallel over slow links}
|
||||
\label{sec:pptp}
|
||||
|
||||
The choice of \emph{how} to split the model is not free, and over a WAN it is decisive. The two
|
||||
options generate fundamentally different network traffic per token.
|
||||
|
||||
\textbf{Pipeline parallel (PP)} splits by layer. Per token, the hidden state crosses each node
|
||||
boundary once: $N$ point-to-point hops on the critical path, each moving the 32\,KiB of
|
||||
Eq.~\eqref{eq:bhop}. Critical-path network latency per token $=\Tring=\sum_{i=1}^N\ell_i$.
|
||||
|
||||
\textbf{Tensor parallel (TP)} splits each layer's matrices across nodes~\cite{megatron}. Every layer
|
||||
must \emph{all-reduce} partial results across all $N$ nodes to reconstruct the activation --- roughly
|
||||
two collectives per layer (one after attention, one after the MLP). That is $\sim\!2L$ global
|
||||
synchronizations per token, each of which, even counted optimistically as a single round-trip
|
||||
($\ge\!2\bar\ell$; a ring all-reduce is worse, $2(N{-}1)$ serial steps), puts a barrier on the
|
||||
critical path. Critical-path network latency per token $\gtrsim 2L\cdot2\bar\ell=4L\bar\ell$.
|
||||
|
||||
\paragraph{The traffic law.} Comparing the two, with $\bar\ell$ the mean one-way hop and
|
||||
$\Tring\approx N\bar\ell$:
|
||||
\begin{equation}
|
||||
\frac{(\text{TP latency/token})}{(\text{PP latency/token})}
|
||||
\;\gtrsim\;\frac{4L\bar\ell}{N\bar\ell}\;=\;\frac{4L}{N}.
|
||||
\label{eq:pptp}
|
||||
\end{equation}
|
||||
\textbf{PP costs $O(\text{nodes})$ hops per token; TP costs $O(\text{layers})$ collectives per
|
||||
token.} Since a model always has far more layers than nodes ($L\gg N$) and each TP collective is a
|
||||
global barrier rather than a single hand-off, PP wins by a wide, structural margin: for $L=100$,
|
||||
$N=3$, Eq.~\eqref{eq:pptp} gives $\gtrsim\!130\times$ less critical-path latency, and it is a lower
|
||||
bound (a realistic ring all-reduce adds another $\sim(N{-}1)$ factor). TP also moves more bytes
|
||||
($\sim\!2\dmodel b$ per node per collective, $\times 2L$), but latency alone settles it. TP belongs
|
||||
\emph{inside} a node or an NVLink domain, where $\bar\ell$ is nanoseconds; across any link where
|
||||
$\bar\ell$ is milliseconds, TP multiplies the wall by $\sim\!4L/N$ and PP is the only sane topology.
|
||||
This is the structural companion to Section~\ref{sec:bandwidth}: PP's crossing tensor is
|
||||
depth-independent precisely because it hands off once per node, and that is the same property that
|
||||
makes it latency-cheap.
|
||||
|
||||
\section{What this paper claims, and what it does not}
|
||||
\label{sec:honesty}
|
||||
|
||||
This is an analysis paper, and the analytical models \emph{are} the contribution --- so the line
|
||||
between derived and measured is drawn explicitly.
|
||||
|
||||
\textbf{Measured (one number).} The 280\,MB/s link goodput is real: iperf3 over a direct 2.5\,GbE
|
||||
cable on our mesh, sustained TCP ($\approx$2.24\,Gbit/s). It anchors the bandwidth claim in
|
||||
Section~\ref{sec:bandwidth} and nothing else.
|
||||
|
||||
\textbf{Physical (not our measurement).} The one-way fiber latencies (40--130\,ms legs, 280\,ms
|
||||
ring) are representative of real intercontinental routes and are bounded below by the speed of light
|
||||
in fiber ($\approx$4.9\,$\mu$s/km). We cite the floor, not a traceroute we ran.
|
||||
|
||||
\textbf{Parameters (stated, not claimed).} $r_0=30$\,tok/s, $r_d=200$\,tok/s, $\alpha\in[0.6,0.9]$,
|
||||
$\dmodel=16384$, $L\approx100$. These are reference values chosen to be representative of a
|
||||
700B-class model with a small local draft; every table and plot is a closed-form evaluation at these
|
||||
values, and the \emph{shape} of every result (the $1/(C+\Tring)$ wall, the $\Tring/\Gam$
|
||||
amortization, the $4L/N$ law) is independent of the specific numbers.
|
||||
|
||||
\textbf{Not claimed.} We report no end-to-end wall-clock from a running geo-distributed deployment;
|
||||
no measured acceptance rates for a specific draft/target pair; no serving-throughput sweep; no
|
||||
overlap/async-speculation optimizations (which only improve the cure). Real acceptance depends on
|
||||
the draft, the target, and the prompt distribution, and real rings have jitter and packet loss the
|
||||
constant-$\ell$ model omits. The models predict \emph{regimes and ratios}, and those are what the
|
||||
paper stands behind.
|
||||
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
Distributing a large language model across machines is bandwidth-cheap and latency-expensive, and
|
||||
the two costs live in different regimes. Bandwidth is trivial because the only tensor crossing a
|
||||
pipeline boundary is a $\sim$32\,KiB hidden state --- depth-independent, KV-cache-free,
|
||||
$\sim$0.1\,ms on a commodity link. Latency is a wall because autoregressive decode is a strict
|
||||
dependency chain: every token traverses every node, the next cannot start until this one returns,
|
||||
and so single-stream decode is floored at $1/(C+\Tring)$ --- $\approx$3\,tok/s on a global ring,
|
||||
\emph{independent of the accelerator}, because the wall is the speed of light in fiber. The
|
||||
``BitTorrent for AI'' picture is right for the throughput-bound work (weights, prefill, batch) and
|
||||
wrong for the one dependency-bound workload (interactive decode) --- and that is the only workload
|
||||
that is hard. The cure is to change what the network is charged for: speculative decoding turns $K$
|
||||
sequential traversals into one, dividing the wall by the accepted-tokens-per-round $\Gam(\alpha,K)$
|
||||
and recovering 12--21\,tok/s on the identical 280\,ms ring. Combined with regional-first placement, a
|
||||
co-located ring turn, node-local KV, and pipeline- (never tensor-) parallelism over the slow links,
|
||||
geo-distributed inference is not a contradiction --- it is a latency-engineering problem with a known
|
||||
solution. The physics sets the floor; speculation decides how often you pay it.
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\small
|
||||
\bibitem{leviathan} Y.~Leviathan, M.~Kalman, Y.~Matias.
|
||||
\emph{Fast Inference from Transformers via Speculative Decoding.} ICML 2023.
|
||||
\bibitem{chen} C.~Chen, S.~Borgeaud, G.~Irving, J.-B.~Lespiau, L.~Sifre, J.~Jumper.
|
||||
\emph{Accelerating Large Language Model Decoding with Speculative Sampling.} arXiv:2302.01318, 2023.
|
||||
\bibitem{petals} A.~Borzunov, D.~Baranchuk, T.~Dettmers, M.~Ryabinin, et al.
|
||||
\emph{Petals: Collaborative Inference and Fine-tuning of Large Models.} ACL 2023 (System Demonstrations).
|
||||
\bibitem{gpipe} Y.~Huang, Y.~Cheng, A.~Bapna, et al.
|
||||
\emph{GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism.} NeurIPS 2019.
|
||||
\bibitem{megatron} M.~Shoeybi, M.~Patwary, R.~Puri, P.~LeGresley, J.~Casper, B.~Catanzaro.
|
||||
\emph{Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.}
|
||||
arXiv:1909.08053, 2019.
|
||||
\bibitem{pipedream} D.~Narayanan, A.~Harlap, A.~Phanishayee, et al.
|
||||
\emph{PipeDream: Generalized Pipeline Parallelism for DNN Training.} SOSP 2019.
|
||||
\bibitem{xbackend} Hanzo AI. \emph{Edge LLM Inference Across Commodity Accelerators: hanzo-engine at
|
||||
llama.cpp Decode Parity on Every Backend.} hanzo-engine white paper (companion, measured).
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
@@ -0,0 +1,414 @@
|
||||
\documentclass[11pt]{article}
|
||||
\input{shared/kernel-preamble.tex}
|
||||
|
||||
\newcommand{\Tring}{\ensuremath{T_{\mathrm{ring}}}}
|
||||
|
||||
\title{\textbf{Open Problems in Heterogeneous\\ Distributed Inference}}
|
||||
\author{Hanzo AI --- ML Systems}
|
||||
\date{Research-agenda companion to the distributed-inference series}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Three companion papers report a working heterogeneous distributed-inference stack. One engine shards a
|
||||
model tensor-parallel across an AMD/NVIDIA boundary over commodity ethernet and decodes
|
||||
coherently~\cite{het}; an analytical model derives the latency physics of geo-distributed autoregressive
|
||||
decode and the speculative-decoding cure~\cite{latency}; and a three-plane blueprint turns the
|
||||
trusted-LAN ring into a permissionless, consensus-managed, post-quantum-authenticated
|
||||
network~\cite{consensus}. Building them surfaced eight problems a homogeneous, single-trust,
|
||||
datacenter-local stack never has to confront, because each is an assumption that only \emph{heterogeneity},
|
||||
\emph{wide-area distribution}, or \emph{permissionlessness} puts under load. This paper states the eight
|
||||
precisely --- with the formal object where one exists --- and ties each to the companion that owns the
|
||||
relevant result, so the boundary between what is measured, what is designed, and what is open stays sharp.
|
||||
The eight are: (1) sampling-determinism boundaries in replicated-state inference; (2) quant-aware
|
||||
tensor-parallel sharding on block-quantized weights; (3) pipeline parallelism as the format-agnostic
|
||||
primitive and its layer-assignment ILP; (4) speculative decoding as a latency amortizer, made adaptive
|
||||
under round-trip time and acceptance drift; (5) the prefill launch-overhead wall, attacked by fixed-shape
|
||||
graph capture versus compiler-level fusion; (6) KV-cache wire compression for prefill migration and its
|
||||
cross-hop error accumulation; (7) the geo-distributed decode latency floor and the placement/replication
|
||||
optimization that follows from it; and (8) cheap verifiable inference for a permissionless compute
|
||||
network. None is a tuning parameter; each is a research question, and several compose --- the layer
|
||||
schedule of (3) and the geo-placement of (7) are one joint optimization, and the determinism boundary of
|
||||
(1) and the cheap verification of (8) share the canonical-\code{f32} substrate. We publish the agenda so
|
||||
the problems can be attacked on the same terms.
|
||||
\end{abstract}
|
||||
|
||||
\section{Introduction: where the assumptions break}
|
||||
\label{sec:intro}
|
||||
|
||||
A model-serving stack built for one datacenter, one vendor, and one operator gets to assume a great deal:
|
||||
every rank runs the same silicon and the same numeric format, the network between ranks is a nanosecond-
|
||||
scale NVLink fabric, and every node is trusted to compute what it claims. The distributed-inference series
|
||||
removes those assumptions one at a time --- the fleet becomes vendor-mixed~\cite{het}, the links become
|
||||
intercontinental~\cite{latency}, and the nodes become strangers who must be paid and
|
||||
verified~\cite{consensus} --- and each removal exposes a problem the assumption had been hiding.
|
||||
|
||||
This paper is the ledger of what those removals expose. It is deliberately \emph{not} a results paper: it
|
||||
states problems, not solutions, and its contribution is precision --- each problem is pinned to the
|
||||
companion that owns the surrounding result, given its formal shape where it has one, and separated from
|
||||
the parts already solved so that no claim is duplicated across the series. We adopt the series' discipline
|
||||
of labelling measured, designed, and open, and here everything is open by construction; the job is to say
|
||||
\emph{exactly} what is open and why the environment forces it.
|
||||
|
||||
The eight problems fall into four families. \textbf{Correctness under replication} (\S\ref{sec:determinism}):
|
||||
when several ranks must agree on a sampled token, where must their arithmetic be bit-identical?
|
||||
\textbf{The right primitive and its schedule} (\S\ref{sec:quantshard}--\S\ref{sec:pp}): tensor parallelism
|
||||
is correct but coupled to dequantized weights and to a per-layer collective; pipeline parallelism is
|
||||
format-agnostic but demands a hard scheduling decision. \textbf{Paying the wire} (\S\ref{sec:spec}--\S\ref{sec:geofloor}):
|
||||
the latency of a wide-area hop is on the critical path of every token, and the levers are speculation,
|
||||
compression, and placement. \textbf{Trust without an operator} (\S\ref{sec:verify}): a permissionless node
|
||||
can lie about its work, and full re-execution is too expensive to be the answer. We take them in that
|
||||
order.
|
||||
|
||||
\section{Problem 1: determinism boundaries in replicated-state inference}
|
||||
\label{sec:determinism}
|
||||
|
||||
\paragraph{Statement.} In any inference topology where more than one rank holds replicated state that must
|
||||
stay coherent --- most sharply, the tensor-parallel ring of~\cite{het}, where each rank keeps its own KV
|
||||
cache and advances it token by token --- every rank must sample the \emph{same} next token, or the caches
|
||||
diverge and the streams silently fork. The cross-vendor ring makes the \emph{reduced activations} agree
|
||||
bit-for-bit by exchanging them in a canonical \code{f32} wire and reducing with a canonical
|
||||
tie-break~\cite{het}. But the final logits are computed \emph{after} the last reduction, locally, in each
|
||||
rank's own 16-bit compute dtype (\code{f16} on the ROCm rank, \code{bf16} on the CUDA rank). Sampling is
|
||||
therefore \emph{outside} the agreement the wire guarantees: at a near-argmax tie, \code{f16} and
|
||||
\code{bf16} rounding can select different tokens, and the ranks desync.
|
||||
|
||||
\paragraph{The formal object.} Call the \emph{agreement set} $\mathcal{A}$ the minimal set of tensors that
|
||||
must be bit-identical across replicas for state to stay coherent. The canonical-\code{f32} reduce places
|
||||
the post-attention and post-MLP activations in $\mathcal{A}$. Coherent decode additionally requires the
|
||||
\emph{sampled token id} $\in\mathcal{A}$; the open question is the cheapest way to put it there. Two
|
||||
mechanisms bound the design space:
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=2pt]
|
||||
\item \textbf{Broadcast the decision.} One head rank samples and broadcasts the token id; the others do not
|
||||
sample at all. Cost: one small ($\le 8$-byte) broadcast per token on the critical path. Correct by
|
||||
construction for \emph{any} sampler (greedy, temperature, top-$p$, min-$p$), because there is exactly one
|
||||
draw.
|
||||
\item \textbf{Canonicalize the computation.} Every rank computes logits in canonical \code{f32} with a
|
||||
deterministic tie-break and a shared RNG stream, so all ranks sample identically without communicating.
|
||||
Cost: an \code{f32} logit head on every rank and a synchronized RNG; fragile the moment any op on the path
|
||||
is not bit-reproducible across vendors.
|
||||
\end{itemize}
|
||||
The trade is broadcast latency versus redundant canonical compute, and it interacts with the sampler:
|
||||
stochastic sampling needs a \emph{shared seed and a shared draw}, which effectively forces the broadcast
|
||||
mechanism, so the ``canonicalize'' route is really only clean for greedy decode.
|
||||
|
||||
\paragraph{What is open.} A general theory of determinism boundaries for replicated-state inference: which
|
||||
ops must be canonical, the cost of canonicalization versus decision-broadcast as a function of vocabulary
|
||||
size and ring width, and the extension to speculative decoding (\S\ref{sec:spec}), where the boundary must
|
||||
hold not for one token but for a verified block of $K$. This is the correctness dual of the verification
|
||||
problem of \S\ref{sec:verify}: the same canonical-\code{f32} property that makes two honest ranks agree is
|
||||
what lets a third party \emph{check} that they did.
|
||||
|
||||
\section{Problem 2: quant-aware tensor-parallel sharding}
|
||||
\label{sec:quantshard}
|
||||
|
||||
\paragraph{Statement.} Tensor parallelism shards each weight matrix along rows (column-parallel) or columns
|
||||
(row-parallel), giving each rank a contiguous slice of the matrix~\cite{megatron}. The block-quantized
|
||||
formats that make edge inference fit in commodity memory --- GGUF $k$-quants and their kin --- store
|
||||
weights in \emph{superblocks}: a run of (typically) 256 weights sharing a block scale, with sub-block
|
||||
scales and a packed low-bit mantissa. A naive row/column shard cuts \emph{through} a superblock, so a
|
||||
rank's slice no longer contains the shared scale its weights are quantized against. The only way to shard
|
||||
such a layout with an arbitrary cut is to dequantize the whole matrix, slice, and requantize per shard ---
|
||||
which reintroduces the dequant--requant loss the format exists to avoid, and defeats the memory saving by
|
||||
materializing the dense matrix. This is why the shipping TP path in~\cite{het} is wired only through the
|
||||
dequantized (safetensors) model definitions and cannot shard a GGUF model at all.
|
||||
|
||||
\paragraph{What is open.} \emph{Quant-aware sharding}: a split that respects superblock boundaries. If the
|
||||
shard count divides the matrix dimension into superblock-aligned pieces (or the matrix is padded to make
|
||||
it so), each rank receives whole superblocks with their scales intact and no dequantization is needed. The
|
||||
open questions are the constraints and their cost: which (hidden size, head count, shard count, block size)
|
||||
combinations admit an aligned split; how much padding waste an arbitrary combination incurs; whether a
|
||||
format with per-shard scale replication can be sharded losslessly at any cut; and whether the analysis
|
||||
differs for row- versus column-parallel layers, which cut different axes of the superblock grid. The prize
|
||||
is TP over quantized weights --- but note that pipeline parallelism (\S\ref{sec:pp}) sidesteps the entire
|
||||
problem, because it ships \emph{activations}, not weight slices, across the seam and each node loads its
|
||||
layer range in whatever quantization it likes. Problem~2 is worth solving only where TP is worth keeping:
|
||||
inside a node or an NVLink domain, for a matrix too wide to hold on one accelerator.
|
||||
|
||||
\section{Problem 3: pipeline parallelism as the universal primitive, and its layer-assignment ILP}
|
||||
\label{sec:pp}
|
||||
|
||||
\paragraph{The engineering result (in progress).} The heterogeneous-cluster analysis of~\cite{het} argues,
|
||||
and the latency analysis of~\cite{latency} proves under a wide-area cost model, that \emph{pipeline
|
||||
parallelism} (PP) is the right primitive for a heterogeneous fleet: it is format-agnostic (\S\ref{sec:quantshard}),
|
||||
carries only an activation tensor across each node boundary rather than an all-reduce per layer, imposes no
|
||||
head-count-divides-node-count constraint, and reduces the sampling determinism boundary
|
||||
(\S\ref{sec:determinism}) to a single head sampler. Implementing PP as the default multi-node primitive in
|
||||
the same engine is an engineering task under way in the codebase; it is not the open problem. The open
|
||||
problem is what PP forces you to decide that TP does not.
|
||||
|
||||
\paragraph{The open problem: optimal layer-range assignment.} PP must cut the $L$-layer model into $N$
|
||||
contiguous ranges and assign each to a node, under nodes that differ in \emph{compute}, \emph{memory}, and
|
||||
\emph{link bandwidth} simultaneously. Let node $j$ have compute throughput $\rho_j$ (layers/s-equivalent)
|
||||
and memory capacity $M_j$; let layer $l$ have compute weight $w_l$ and resident footprint $m_l$ (weights
|
||||
plus its KV cache). A pipeline is a permutation $\pi$ of nodes onto stages and a contiguous partition
|
||||
$0=c_0<c_1<\dots<c_N=L$ assigning layers $[c_{j-1},c_j)$ to stage $j$. Steady-state pipelined throughput is
|
||||
set by the slowest stage, so the assignment problem is
|
||||
\begin{equation}
|
||||
\min_{\pi,\;c}\;\max_{1\le j\le N}\;
|
||||
\max\!\left(\underbrace{\frac{1}{\rho_{\pi(j)}}\sum_{l=c_{j-1}}^{c_j-1} w_l}_{\text{stage compute}},\;
|
||||
\underbrace{\frac{B_{\mathrm{hop}}}{b_{\pi(j),\pi(j+1)}}}_{\text{activation transmit}}\right)
|
||||
\quad\text{s.t.}\quad
|
||||
\sum_{l=c_{j-1}}^{c_j-1} m_l \le M_{\pi(j)}\ \ \forall j,
|
||||
\label{eq:ilp}
|
||||
\end{equation}
|
||||
where $B_{\mathrm{hop}}$ is the per-hop activation size and $b_{\cdot,\cdot}$ the link bandwidth between
|
||||
consecutive stages. For a \emph{fixed} node ordering $\pi$ the inner problem --- partition a weighted
|
||||
sequence into $N$ contiguous parts minimizing the bottleneck part under per-part capacity --- is solvable
|
||||
exactly by dynamic programming or parametric search. The difficulty is the \emph{joint} choice of ordering
|
||||
$\pi$ (which node is adjacent to which, hence which link carries each hop) together with the partition and,
|
||||
in the geo setting (\S\ref{sec:geofloor}), the placement of nodes in space: that couples an assignment, a
|
||||
sequencing, and a facility-location decision, and is NP-hard in general.
|
||||
|
||||
\paragraph{What is open.} A practical solver: good heuristics with a bounded approximation ratio for
|
||||
Eq.~\eqref{eq:ilp}; online \emph{re-balancing} as nodes join, leave, or thermally throttle (the membership
|
||||
of a permissionless network~\cite{consensus} is not static); and the memory-vs-compute tension when the
|
||||
throughput-optimal cut violates a small node's capacity. The single-stream \emph{latency} objective ---
|
||||
$\sum_j$ stage compute $+\,\Tring$ rather than the bottleneck --- is the variant~\cite{latency} analyses;
|
||||
Eq.~\eqref{eq:ilp} is its throughput dual, and a real scheduler must weigh the two. Prior PP scheduling
|
||||
work assumes homogeneous stages~\cite{gpipe} or homogeneous datacenter nodes~\cite{pipedream}; the open
|
||||
case is genuinely heterogeneous $\rho_j$, $M_j$, and $b_{jk}$ at once.
|
||||
|
||||
\section{Problem 4: speculative decoding as a geo-distributed latency amortizer, made adaptive}
|
||||
\label{sec:spec}
|
||||
|
||||
\paragraph{What the companion owns.} The latency paper~\cite{latency} derives the speculative-decoding
|
||||
cure in full: a local draft proposes $K$ tokens, the distributed target verifies all $K$ in one pipeline
|
||||
pass, and at acceptance rate $\alpha$ the effective rate is
|
||||
$r_{\mathrm{spec}}(\alpha,K)=\frac{1-\alpha^{K+1}}{1-\alpha}\big/(Kc_d+C+\Tring)$, which recovers
|
||||
interactive speeds on an intercontinental ring by paying the round-trip once per accepted \emph{block}
|
||||
rather than once per token. That derivation, its optimal $K^\star$ for fixed $\alpha$, and the numbers are
|
||||
the companion's result and are not restated here.
|
||||
|
||||
\paragraph{What is open.} The companion treats $\alpha$ as a fixed parameter and $K$ as a tuned constant.
|
||||
In deployment neither holds. The acceptance rate $\alpha$ \emph{drifts} within a single generation --- it
|
||||
is high on boilerplate and low at genuine decision points, and it depends on prompt, temperature, and how
|
||||
far the draft and target have diverged in context --- so the throughput-optimal $K$ is a moving target,
|
||||
not a launch-time flag. The open problem is an \emph{online} controller that adapts $K$ to the measured
|
||||
acceptance and the measured ring latency $\Tring$: estimate $\alpha$ from a trailing window of
|
||||
accept/reject outcomes, react to $\Tring$ jitter and packet loss that the constant-$\ell$ model omits, and
|
||||
set $K$ to the running optimum with a stability guarantee (a controller that chases a noisy $\alpha$ can
|
||||
oscillate). Two couplings make it harder and are themselves open: the determinism boundary of
|
||||
\S\ref{sec:determinism} must hold across the \emph{whole verified block}, not one token, so a replicated
|
||||
target must agree on the accept/reject cut; and the draft can itself be placed on a nearby, more reliable
|
||||
node~\cite{consensus}, making draft placement part of the scheduling problem of \S\ref{sec:pp}. The
|
||||
genuinely empirical unknown --- measured $\alpha$ for real draft/target pairs on real prompt distributions,
|
||||
which the companion explicitly does not claim --- gates any adaptive scheme.
|
||||
|
||||
\section{Problem 5: the prefill launch-overhead wall}
|
||||
\label{sec:prefill}
|
||||
|
||||
\paragraph{Statement.} On identical mathematics, hanzo-engine prefill reaches a lower GPU utilization than
|
||||
\code{llama.cpp} not because any kernel is slow but because there are too many kernels. Profiled on an
|
||||
NVIDIA GB10 (node \emph{spark}) running a 35B-class mixture-of-experts prefill, the engine sustains
|
||||
$\approx$79.3\% GPU utilization across $\approx$33{,}000 kernel launches per step, against
|
||||
\code{llama.cpp}'s $\approx$97.3\% across $\approx$3{,}000 --- an order of magnitude more launches for the
|
||||
same work. Per-kernel \emph{busy} time is already at parity; the gap is entirely the idle interstice
|
||||
between launches, i.e.\ launch and scheduling overhead, not arithmetic. Decode does not have this problem,
|
||||
because the decode path is captured into a CUDA graph that replays the whole token as one submission; the
|
||||
prefill path is not, because mixture-of-experts routing is \emph{data-dependent} --- which experts fire
|
||||
depends on the tokens --- so the shape of the work changes per step and a naive static graph capture does
|
||||
not apply.
|
||||
|
||||
\paragraph{Two attacks, and they compose.} There are two structurally different ways to collapse the launch
|
||||
count, and they are not exclusive:
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=2pt]
|
||||
\item \textbf{Fixed-shape chunked graph capture.} Quantize the data-dependent dimension: route to a fixed
|
||||
capacity per expert (pad or drop to a chunk size), so the per-step work has a bounded set of shapes, and
|
||||
capture a CUDA graph per shape. This keeps the many small kernels but pays their launch cost \emph{once}
|
||||
at capture and replays them as one submission, exactly as decode already does. It is a runtime technique
|
||||
and inherits the graph machinery the engine has.
|
||||
\item \textbf{Compiler-level DAG fusion.} Reduce the \emph{number} of kernels rather than the cost of
|
||||
launching them: fuse elementwise and reduction chains into fewer, fatter kernels at compile time, so there
|
||||
is less to launch in the first place. This is the subject of the kernel-fusion companion~\cite{fuse},
|
||||
where fusion is shown to be composition; extending it from local elementwise chains to the full prefill DAG
|
||||
(across the MoE gate, the expert GEMMs, and the combine) is the open compiler problem.
|
||||
\end{itemize}
|
||||
Graph capture attacks the \emph{launch cost}; fusion attacks the \emph{launch count}; applied together they
|
||||
multiply. The decisive reason to invest in fusion as well as capture is portability: graph capture is a
|
||||
CUDA (and partial ROCm) facility, but fusion is a property of the kernel \emph{source} and therefore
|
||||
transfers to every backend the engine targets --- Metal, Vulkan, and ROCm, where graph support is weaker or
|
||||
absent~\cite{backend}. The open problem is the fusion \emph{partitioner}: given the prefill DAG, choose the
|
||||
kernel boundaries that minimize launches subject to register and shared-memory limits, and do it once,
|
||||
portably, from the one source.
|
||||
|
||||
\section{Problem 6: KV-cache wire compression for prefill migration}
|
||||
\label{sec:kvcompress}
|
||||
|
||||
\paragraph{Statement.} In steady-state pipeline decode the KV cache never crosses the wire --- each node
|
||||
keeps the keys and values for its own layers~\cite{latency}. The exception is \emph{migration}: moving a
|
||||
prefilled context between nodes, as in disaggregated prefill/decode, session hand-off, or re-placement when
|
||||
the schedule of \S\ref{sec:pp} changes. Then the KV cache --- the large, context-growing state --- must be
|
||||
shipped, and its size, not the activation's, sets the transfer cost. The engine already has an FP8 KV
|
||||
representation, so the obvious move is to ship KV in FP8 and halve the bytes.
|
||||
|
||||
\paragraph{What is open.} Error accumulation across hops. A single FP8 round-trip of the KV cache is a
|
||||
tolerable, well-studied quantization. The open question is what happens when compression is applied
|
||||
\emph{repeatedly} --- a context migrated across several nodes over its lifetime, or a design that keeps KV
|
||||
in FP8 resident and re-quantizes on each write --- so that quantization error compounds hop over hop and
|
||||
layer over layer. Formally: bound the accumulated error of $H$ successive FP8 (de)compressions of a KV
|
||||
cache as a function of $H$, context length, and layer depth, and identify the schemes that arrest it ---
|
||||
error feedback (carry the residual forward), selective precision (keep the attention-sink and recent tokens
|
||||
in higher precision), or a migration protocol that recompresses from a retained high-precision anchor
|
||||
rather than from the last lossy copy. This is the compression dual of the determinism boundary
|
||||
(\S\ref{sec:determinism}): the all-reduce must be canonical \code{f32} because it is summed every layer,
|
||||
but the KV cache may tolerate FP8 precisely because it is \emph{read}, not reduced --- and the open problem
|
||||
is exactly how much, and for how many hops.
|
||||
|
||||
\section{Problem 7: the geo-distributed decode latency floor and the placement it dictates}
|
||||
\label{sec:geofloor}
|
||||
|
||||
\paragraph{What the companion owns.} The latency paper~\cite{latency} establishes the floor: single-stream
|
||||
autoregressive decode must traverse every pipeline stage per token, so its rate is $1/(C+\Tring)$ where
|
||||
$\Tring$ is the one-way loop latency, and because $\Tring$ is bounded below by the speed of light in fiber
|
||||
($\approx$4.9\,$\mu$s/km), an intercontinental ring floors decode at a few tok/s \emph{regardless of
|
||||
accelerator speed}. The companion also states the levers as placement laws --- regional-first placement,
|
||||
speculate across every slow boundary, co-locate the ring turn, keep KV node-local --- and proves pipeline
|
||||
parallelism dominates tensor parallelism over any slow link by $\sim\!4L/N$. The floor, the laws, and the
|
||||
$4L/N$ result are the companion's and are not restated.
|
||||
|
||||
\paragraph{What is open.} The companion's laws are heuristics, stated and argued but not optimized or
|
||||
measured end-to-end. Two things remain. First, \emph{empirical validation}: the companion is explicit that
|
||||
it reports no wall-clock from a running geo deployment; a real multi-continent ring, with real jitter and
|
||||
loss, must confirm the $1/(C+\Tring)$ floor and the speculative recovery. Second, and deeper, the
|
||||
\emph{placement and replication optimization} the floor implies. If one ring cannot serve a global user
|
||||
base at interactive latency, the answer is \emph{regional replicas} --- multiple rings, each latency-local
|
||||
to a set of users, with requests routed to the nearest --- and choosing how many rings, in which regions,
|
||||
holding which shards, to minimize the population's tail latency under a memory and cost budget is a
|
||||
facility-location problem entangled with the layer-assignment ILP of \S\ref{sec:pp}: where a node sits in
|
||||
space changes both which users it is near and which link it contributes to its ring. Formalizing that
|
||||
joint placement--partition problem, and the routing that assigns each request to a ring, is open, and it is
|
||||
the geo-scale version of Problem~3.
|
||||
|
||||
\section{Problem 8: cheap verifiable inference for a permissionless compute network}
|
||||
\label{sec:verify}
|
||||
|
||||
\paragraph{What the companion owns.} The three-plane blueprint~\cite{consensus} builds the transport and
|
||||
control layers of a permissionless inference network: a post-quantum-authenticated ZAP wire that proves
|
||||
\emph{who} sent a tensor, a Quasar consensus control plane that admits staked miners and agrees the
|
||||
placement map, and a Photon committee that can threshold-sign an $(\text{request},\text{output})$ pair.
|
||||
For result integrity it proposes three composable mechanisms --- threshold attestation of outputs,
|
||||
redundant re-execution with the canonical-\code{f32} property as a bit-exact cross-vendor oracle, and
|
||||
luminance reputation as the economic backstop --- and labels them, honestly, as design. Authentication and
|
||||
committee formation are solved there and not reopened here.
|
||||
|
||||
\paragraph{What is open: verification that is cheaper than the work.} Authentication proves who sent a
|
||||
tensor, not that the tensor is \emph{correct}, and the blueprint's strongest integrity mechanism ---
|
||||
redundant re-execution --- costs a full second computation on every checked request. That is fine as a
|
||||
sampled spot check but cannot be the common path: a network whose verification costs as much as its
|
||||
inference has no margin to pay honest miners. The open problem is \emph{sub-linear} verification of correct
|
||||
inference. Candidate directions, each unproven for transformer decode:
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=2pt]
|
||||
\item \textbf{Sampling-based logit audits.} A verifier recomputes a \emph{random subset} of the claimed
|
||||
logits (a few positions, a few layers) and checks them against the miner's committed output, catching a
|
||||
cheating miner with high probability at a fraction of the compute. The open question is the soundness
|
||||
--- how large a subset defeats an adversary who corrupts only the tokens a user cares about --- and how it
|
||||
composes with the canonical-\code{f32} determinism (\S\ref{sec:determinism}) that makes a recomputed logit
|
||||
comparable bit-for-bit in the first place.
|
||||
\item \textbf{Redundant-execution economics.} Treat verification as a game: honest computation must be the
|
||||
profit-maximizing strategy under a spot-check probability $p$, a stake bond, and a slashing penalty. The
|
||||
open problem is to price $p$, the bond, and the reward so that the \emph{expected} cost of cheating exceeds
|
||||
its gain for a rational miner, while keeping the honest miner's expected verification tax small --- and to
|
||||
do so against collusion among a committee and grinding on the verification lottery, threats the blueprint
|
||||
names but does not price.
|
||||
\end{itemize}
|
||||
The unifying observation is that the canonical-\code{f32} wire built for cross-vendor \emph{correctness}
|
||||
(\S\ref{sec:determinism}, \cite{het}) is the same substrate that makes inference \emph{checkable}: two
|
||||
honest nodes agree bit-for-bit, so any disagreement is a proof of fault rather than a rounding artifact.
|
||||
Turning that from a full-re-execution check into a sampled, economically-sound one is the last open
|
||||
problem, and the one that decides whether a permissionless inference network can exist at all.
|
||||
|
||||
\section{Synthesis}
|
||||
\label{sec:synthesis}
|
||||
|
||||
The eight problems are not independent, and Table~\ref{tab:map} states their dependencies as plainly as
|
||||
their statements. Three throughlines run across them.
|
||||
|
||||
\emph{Canonical \code{f32} is load-bearing twice.} The same property that makes an \code{f16} rank and a
|
||||
\code{bf16} rank agree on a reduced activation (\S\ref{sec:determinism}) is what lets a third party check a
|
||||
miner's work without a rounding dispute (\S\ref{sec:verify}). Correctness under replication and verifiable
|
||||
correctness are the same substrate seen from two sides.
|
||||
|
||||
\emph{Scheduling is one problem at two scales.} The layer-assignment ILP of \S\ref{sec:pp} and the
|
||||
regional-replica placement of \S\ref{sec:geofloor} are the LAN and geo faces of a single
|
||||
partition-and-place optimization; solving one in isolation leaves the other's coupling on the table, and a
|
||||
real deployment sets both at once, with speculative-draft placement (\S\ref{sec:spec}) folded in.
|
||||
|
||||
\emph{Pipeline parallelism dissolves more than it creates.} It sidesteps quant-aware sharding
|
||||
(\S\ref{sec:quantshard}), collapses the determinism boundary to one sampler (\S\ref{sec:determinism}), and
|
||||
makes the graph-capture asymmetry a local choice (\S\ref{sec:prefill}) --- but it buys those with the
|
||||
scheduling problem of \S\ref{sec:pp} and the migration cost of \S\ref{sec:kvcompress}. The net is
|
||||
favourable, which is why the series argues for it; the residue is real, which is why it is here.
|
||||
|
||||
Finally, the register. Some of these are engineering with a known shape --- adaptive $K$ (\S\ref{sec:spec}),
|
||||
fixed-shape graph capture (\S\ref{sec:prefill}), FP8 migration (\S\ref{sec:kvcompress}) --- and will yield
|
||||
to careful implementation and measurement. Others are genuinely research --- the determinism-boundary
|
||||
theory (\S\ref{sec:determinism}), the heterogeneous scheduling ILP (\S\ref{sec:pp}), sub-linear
|
||||
verification (\S\ref{sec:verify}) --- and may not have clean answers. We have tried to say which is which,
|
||||
because the most useful thing an open-problems paper can do is not overclaim the difficulty of the easy
|
||||
ones or underclaim the depth of the hard ones.
|
||||
|
||||
\begin{table}[t]
|
||||
\centering
|
||||
\small
|
||||
\setlength{\tabcolsep}{5pt}
|
||||
\begin{tabular}{@{}p{0.4cm}p{4.5cm}p{2.2cm}p{5.1cm}@{}}
|
||||
\toprule
|
||||
\# & \textbf{Problem} & \textbf{Exposed by} & \textbf{What is open (owning companion)} \\
|
||||
\midrule
|
||||
1 & Sampling-determinism boundary & heterogeneity & minimal agreement set; broadcast vs.\ canonicalize cost~\cite{het} \\
|
||||
2 & Quant-aware TP sharding & heterogeneity & superblock-aligned splits without dequant--requant~\cite{het} \\
|
||||
3 & PP layer-assignment ILP & heterogeneity & Eq.~\eqref{eq:ilp}: heuristics, online rebalance~\cite{het,latency} \\
|
||||
4 & Adaptive speculation & wide area & online $K$ under drifting $\alpha$, jittering $\Tring$~\cite{latency} \\
|
||||
5 & Prefill launch-overhead wall & (single-node) & fusion partitioner; portable, composes with capture~\cite{fuse,backend} \\
|
||||
6 & KV wire compression & wide area & cross-hop FP8 error accumulation bound~\cite{latency} \\
|
||||
7 & Geo decode latency floor & wide area & empirical floor; regional-replica placement~\cite{latency} \\
|
||||
8 & Cheap verifiable inference & permissionless & sub-linear logit audits; slashing economics~\cite{consensus} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{The eight open problems, the environment property that exposes each, and the open kernel with the
|
||||
companion that owns the surrounding result. Problems 3 and 7 are one optimization at two scales; problems 1
|
||||
and 8 share the canonical-\code{f32} substrate. Problem 5 is exposed already single-node but is sharpest
|
||||
when prefill is distributed.}
|
||||
\label{tab:map}
|
||||
\end{table}
|
||||
|
||||
\section*{Acknowledgement of scope}
|
||||
\addcontentsline{toc}{section}{Acknowledgement of scope}
|
||||
This paper contributes problem statements, not solutions, and every result it leans on belongs to a cited
|
||||
companion: the cross-vendor decode measurement and its residuals~\cite{het}, the latency floor and the
|
||||
speculative cure~\cite{latency}, and the three-plane network with its verification design~\cite{consensus}.
|
||||
Where a formal object appears --- the agreement set (\S\ref{sec:determinism}), the assignment ILP
|
||||
Eq.~\eqref{eq:ilp}, the error-accumulation bound (\S\ref{sec:kvcompress}) --- it is stated to make the open
|
||||
question precise, not claimed as solved. The measured numbers quoted (the launch-overhead profile of
|
||||
\S\ref{sec:prefill}) are attributed to the hardware that produced them (NVIDIA GB10) and to the companion
|
||||
that carries the surrounding stack~\cite{backend}.
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\small
|
||||
\bibitem{het} Hanzo AI --- ML Systems. \emph{One Cluster, Every Vendor: Cross-Vendor Tensor-Parallel LLM
|
||||
Inference over Commodity Ethernet.} Distributed-inference companion (measured).
|
||||
\bibitem{latency} Hanzo AI --- ML Systems. \emph{The Latency Physics of Geo-Distributed Autoregressive
|
||||
Decode.} Distributed-inference companion (analytical).
|
||||
\bibitem{consensus} Hanzo AI --- ML Systems. \emph{The Three-Plane Compute Fabric: Consensus-Managed,
|
||||
ZAP-Transported Decentralized AI Inference.} Distributed-inference companion (architecture).
|
||||
\bibitem{backend} Hanzo AI. \emph{Edge LLM Inference Across Commodity Accelerators: hanzo-engine at
|
||||
llama.cpp Decode Parity on Every Backend.} Paper 05 (measured).
|
||||
\bibitem{fuse} Hanzo AI. \emph{Fusion Is Composition: Kernel Fusion as the Functor Law.} Paper 07.1.
|
||||
\bibitem{megatron} M.\ Shoeybi, M.\ Patwary, R.\ Puri, P.\ LeGresley, J.\ Casper, B.\ Catanzaro.
|
||||
\emph{Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.}
|
||||
arXiv:1909.08053, 2019.
|
||||
\bibitem{gpipe} Y.\ Huang et al. \emph{GPipe: Efficient Training of Giant Neural Networks using Pipeline
|
||||
Parallelism.} NeurIPS, 2019.
|
||||
\bibitem{pipedream} D.\ Narayanan, A.\ Harlap, A.\ Phanishayee, et al. \emph{PipeDream: Generalized
|
||||
Pipeline Parallelism for DNN Training.} SOSP, 2019.
|
||||
\bibitem{petals} A.\ Borzunov et al. \emph{Petals: Collaborative Inference and Fine-tuning of Large
|
||||
Models.} arXiv:2209.01188, 2022.
|
||||
\bibitem{leviathan} Y.\ Leviathan, M.\ Kalman, Y.\ Matias. \emph{Fast Inference from Transformers via
|
||||
Speculative Decoding.} ICML, 2023.
|
||||
\bibitem{chen} C.\ Chen, S.\ Borgeaud, G.\ Irving, J.-B.\ Lespiau, L.\ Sifre, J.\ Jumper. \emph{Accelerating
|
||||
Large Language Model Decoding with Speculative Sampling.} arXiv:2302.01318, 2023.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
@@ -0,0 +1,468 @@
|
||||
\documentclass[11pt]{article}
|
||||
\input{shared/kernel-preamble.tex}
|
||||
|
||||
\title{\textbf{One Cluster, Every Vendor:\\ Cross-Vendor Tensor-Parallel LLM Inference over Commodity Ethernet}}
|
||||
\author{Hanzo AI --- ML Systems}
|
||||
\date{Distributed companion to the \emph{One Source, Every Backend} series --- extends Paper 05 across the wire}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Real accelerator fleets are vendor-mixed: an AMD APU, an NVIDIA dev kit, and a Mac sit on the same
|
||||
desk, and no collective-communication library spans them --- NCCL is NVIDIA-only, RCCL is AMD-only, and
|
||||
neither will open a channel to the other. This paper reports a working result, with its limits stated
|
||||
plainly. One inference engine (\code{hanzo-engine}, Rust, one codebase, per-backend kernels) runs
|
||||
on three vendor-diverse accelerators --- AMD ROCm gfx1151 (a ``Strix Halo'' APU, node \emph{evo}), NVIDIA
|
||||
CUDA sm\_121 (a GB10 Blackwell, node \emph{spark}), and Apple Metal M4 Max (node \emph{dbc}). We make its
|
||||
Megatron-style tensor-parallel (TP) collectives \emph{vendor-agnostic} by host-staging every exchange
|
||||
--- \code{device}$\to$\code{CPU}$\to$\code{TCP}$\to$\code{CPU}$\to$\code{device} --- so any device that
|
||||
can copy to host storage can join the ring, which is exactly what NCCL/RCCL cannot do across a vendor
|
||||
boundary. On a two-node ring (\emph{evo} ROCm $+$ \emph{spark} CUDA) over a 2.5GbE direct link, the
|
||||
sharded engine \textbf{builds, connects, shards a \code{world\_size}$=$2 tensor-parallel model, and
|
||||
generates fully coherent multi-token output --- chain-of-thought and a correct answer --- across the
|
||||
AMD/NVIDIA boundary at 50.4~tok/s decode} (TTFT 0.09s, 213~tok/s prefill). To our knowledge this is the
|
||||
first reported cross-vendor (AMD$+$NVIDIA) tensor-parallel LLM inference over commodity ethernet in a
|
||||
single engine.
|
||||
|
||||
We are equally explicit about the engineering, because the honest failures are the paper's most useful
|
||||
content. The socket-ring path had only ever been \emph{compiled}, never \emph{run} (only the NCCL path
|
||||
was exercised), and bringing it up cross-vendor surfaced four heterogeneity bugs --- a \code{len}$^2$
|
||||
byte-count overflow, a missing ROCm host-staging arm that meant the ring never even compiled for AMD, a
|
||||
10s handshake timeout that raced heterogeneous startup, and --- the subtle one that blocked coherent
|
||||
decode --- a \emph{heterogeneous-dtype-on-the-wire} hazard, where the ROCm rank auto-selects \code{f16}
|
||||
and the CUDA rank auto-selects \code{bf16} and each reinterpreted its neighbour's 16-bit words as the
|
||||
wrong floating-point layout, corrupting the reduction from the first layer. The cure is a canonical
|
||||
\code{f32} ring wire (a lossless superset of both) plus a \code{world\_size}$-$1 accumulating all-reduce
|
||||
correct for any \code{N}$\geq$2 and the removal of a spurious power-of-two world-size restriction; all
|
||||
four fixes are merged on \code{main}, and a same-vendor control (both ranks \code{f16}) matches the
|
||||
cross-vendor run token-for-token, isolating the wire dtype as the sole cross-vendor variable. The
|
||||
measured cost of the primitive is itself a result: 50.4~tok/s is $\approx$3.3$\times$ slower than the same
|
||||
model single-node (168~tok/s), and the gap is \emph{all} synchronization --- an all-reduce every layer,
|
||||
over the wire. That measured tax is the empirical case for \textbf{pipeline parallelism} --- contiguous
|
||||
layer ranges, activations-only on the wire (\code{O(nodes)} hops, not \code{O(layers)} all-reduces),
|
||||
format-agnostic, one generation loop driving stateless workers --- as the structurally better primitive
|
||||
for heterogeneous clusters: the numeric fixes made TP \emph{correct}, not well-matched. Three residuals
|
||||
stay honestly open, and we develop them, with five further problems, as a research agenda in the
|
||||
companion~\cite{openproblems}: the two ranks still sample independently from their own-dtype logits (a
|
||||
near-tie argmax can diverge and desync the KV caches), the CUDA rank must run graphless to stay lockstep
|
||||
with the graphless ROCm rank, and TP is still coupled to the dequantized (safetensors) weight path.
|
||||
\end{abstract}
|
||||
|
||||
\section{The problem: fleets are vendor-mixed, and the collective libraries are not}
|
||||
\label{sec:problem}
|
||||
|
||||
The single-node premise of this series is that the interesting inference target is a commodity box a
|
||||
customer already owns~\cite{backend}. The multi-node reality is that they own \emph{several}, and not
|
||||
from one vendor. A team that bought an AMD ``Strix Halo'' mini-PC last quarter, a Grace-Blackwell dev kit
|
||||
this quarter, and codes on a MacBook has three capable accelerators totalling well over a hundred
|
||||
gigabytes of unified memory --- enough to hold a model no single one of them can --- sitting idle on a
|
||||
LAN. The obstacle is not the models and not the math. It is that \emph{the collective-communication
|
||||
layer is single-vendor by construction}.
|
||||
|
||||
Tensor parallelism~\cite{megatron} shards each weight matrix across ranks and, twice per transformer
|
||||
block (after attention, after the MLP), performs an \code{all\_reduce} to sum the partial results back
|
||||
into a full activation. That \code{all\_reduce} is the whole game, and in practice it is delegated to a
|
||||
vendor library: \textbf{NCCL}~\cite{nccl} on NVIDIA, \textbf{RCCL}~\cite{rccl} on AMD. Both are excellent
|
||||
and both are closed at the vendor boundary: NCCL speaks to NVIDIA GPUs over NVLink/PCIe/GPUDirect and
|
||||
has no concept of an AMD peer; RCCL is the symmetric mirror. There is no NCCL$\leftrightarrow$RCCL
|
||||
channel, no cross-vendor GPUDirect, and no shared handle either side will accept. So the textbook path to
|
||||
running one model across \emph{evo} and \emph{spark} together does not merely underperform --- it does
|
||||
not exist.
|
||||
|
||||
The engine already dodges the \emph{first} half of this problem. \code{hanzo-engine} is one Rust codebase
|
||||
that lowers to CUDA, ROCm, Metal, Vulkan, and CPU behind a single model loader, reaching \code{llama.cpp}
|
||||
decode parity on all three vendors from that one build~\cite{backend}; the kernel layer that makes ``one
|
||||
source, every backend'' hold is the subject of the companion DSL papers~\cite{umbrella,fuse,oracle}. What
|
||||
was missing is the \emph{second} half: a way for those per-vendor engines to talk to \emph{each other}
|
||||
inside one cluster. This paper builds that channel, runs it across a vendor boundary for the first time,
|
||||
and reports --- without varnish --- exactly how far it gets and what it cost to get there.
|
||||
|
||||
\section{The design: a host-staged socket ring makes the collective vendor-agnostic}
|
||||
\label{sec:design}
|
||||
|
||||
The engine carries a unified \code{Comm} abstraction with three backends behind one enum: \code{Nccl}
|
||||
(the fast same-vendor NVIDIA path), \code{Ring} (a socket-based collective), and \code{Dummy}
|
||||
(\code{world\_size}$=$1). The idea that makes cross-vendor work possible is in the \code{Ring} backend,
|
||||
and it is deliberately humble: \textbf{stage every tensor through host memory and move the bytes over
|
||||
TCP.}
|
||||
|
||||
Concretely, a ring \code{sum\_all\_reduce} does not touch a vendor collective at all. It pulls the
|
||||
tensor's storage to the CPU, converts it to a canonical \code{f32} payload, exchanges the bytes with its
|
||||
ring neighbours over ordinary TCP sockets, sums in \code{f32}, and copies the reduced result back to the
|
||||
device in its native dtype (Listing~\ref{lst:stage}). The single load-bearing property is that the
|
||||
exchange happens in \code{CpuStorage}, which \emph{every} backend can produce --- CUDA, Metal, and ROCm
|
||||
each implement \code{to\_cpu\_storage()}. The device that produced the tensor is irrelevant by the time
|
||||
the bytes reach the socket. \emph{Why} the wire dtype is a canonical \code{f32} rather than the raw
|
||||
device bytes --- the difference between coherent and garbage output across vendors --- is the subject of
|
||||
\S\ref{sec:rootcause}; here it is simply the shipped design.
|
||||
|
||||
\begin{lstlisting}[language=Rust,caption={The vendor-agnostic collective, condensed from the shipping
|
||||
code. The reduce goes device$\to$host$\to$canonical \code{f32}$\to$TCP$\to$\code{f32} sum$\to$device; the
|
||||
\code{match} over \code{Storage} is the only place the backend is named, and every arm lands in the same
|
||||
\code{CpuStorage}. The \code{cpu\_to\_f32} stage is what makes an \code{f16} rank and a \code{bf16} rank
|
||||
agree (\S\ref{sec:rootcause}). NCCL, by contrast, would keep the whole exchange on-device and refuse a
|
||||
foreign-vendor peer.},label={lst:stage}]
|
||||
fn sum_all_reduce(&self, xs: &Tensor) -> Result<Tensor> {
|
||||
let storage = xs.storage_and_layout().0;
|
||||
let host = match &*storage { // device -> host (any backend)
|
||||
Storage::Cpu(s) => s.clone(),
|
||||
Storage::Cuda(s) => s.to_cpu_storage()?,
|
||||
Storage::Metal(s) => s.to_cpu_storage()?,
|
||||
#[cfg(feature = "rocm")]
|
||||
Storage::Rocm(s) => s.to_cpu_storage()?, // was missing: ring never built for AMD (Bug 2)
|
||||
};
|
||||
let wire = cpu_to_f32(&host)?; // CANONICAL wire dtype: f16|bf16|f32 -> f32
|
||||
let sum = self.ring_exchange(&wire, xs.dims())?; // world_size-1 accumulating steps, sum in f32
|
||||
Tensor::from_slice(&sum, xs.dims(), xs.device())? // host -> device
|
||||
.to_dtype(xs.dtype()) // back to this rank's native compute dtype
|
||||
}
|
||||
\end{lstlisting}
|
||||
|
||||
This is not free. NCCL keeps the reduction on-device and rides NVLink; the host-staged ring pays two
|
||||
extra device$\leftrightarrow$host copies per collective, a widening cast to \code{f32}, and then rides a
|
||||
2.5GbE link whose measured goodput is $\approx$280~MB/s (\S\ref{sec:repro}). That is the correct trade for
|
||||
the problem: we spend bandwidth to buy \emph{vendor-agnosticism}, because the alternative on a mixed fleet
|
||||
is not a faster collective, it is \emph{no} collective. The ring is selected purely by configuration ---
|
||||
\code{use\_ring()} is true when the \code{ring} feature is built and a \code{RING\_CONFIG} path is present
|
||||
in the environment (\S\ref{sec:repro}) --- so the same engine binary is a single-node server, an NCCL
|
||||
rank, or a socket-ring rank depending only on how it is launched.
|
||||
|
||||
\section{Compiles $\ne$ runs: an unexercised path was broken three ways}
|
||||
\label{sec:compiles}
|
||||
|
||||
Here is the lesson that generalizes past this codebase. \emph{The ring path had been written, had type-
|
||||
checked, had shipped in every build for months --- and had never once been run.} All multi-GPU testing
|
||||
had gone through NCCL (the same-vendor path), so the socket ring was dead code that happened to compile.
|
||||
The first time it was asked to carry a real tensor across a real vendor boundary, it failed three
|
||||
separate ways, each a different species of the gap between ``the compiler accepted it'' and ``it does the
|
||||
thing.'' All three are fixed on \code{main} (commit \code{63cc4c7ac}); the fourth, subtler failure ---
|
||||
visible only once the generated tokens came out wrong --- is \S\ref{sec:rootcause}.
|
||||
|
||||
\paragraph{Bug 1 --- a \code{len}$^2$ byte count (a logic error the types could not catch).}
|
||||
The wire code computed its transfer size as
|
||||
\begin{center}
|
||||
\code{let nbytes = x.len() * std::mem::size\_of\_val(x);}
|
||||
\end{center}
|
||||
This looks reasonable and is badly wrong. For a slice \code{x:\ \&[T]}, \code{size\_of\_val(x)} is
|
||||
\emph{already} the size of the whole referent --- \code{x.len()}\,$\times$\,\code{size\_of::<T>()} --- so
|
||||
multiplying by \code{x.len()} again yields \code{len}$^2$\,$\times$\,\code{elem\_size}, overshooting the
|
||||
true byte count by a factor of \code{len}. The subsequent \code{from\_raw\_parts(ptr, nbytes)} then
|
||||
fabricates a byte slice spanning far past the end of the allocation, and \code{write\_all} walks off the
|
||||
buffer and hits \code{EFAULT}. It is invisible for a one-element tensor (\code{len}$^2 =$ \code{len}) and
|
||||
fatal for every real activation. The fix is one token --- delete the \code{x.len()\ *} --- but the point
|
||||
is that the type system was content the entire time: both operands are \code{usize}, the product is a
|
||||
\code{usize}, and the meaning is nonsense. Only \emph{running} it with \code{len}$>$1 reveals it.
|
||||
|
||||
\paragraph{Bug 2 --- the ring never compiled for AMD (a hole hidden by a feature flag).}
|
||||
The device$\to$host \code{match} in Listing~\ref{lst:stage} had arms for \code{Cpu}, \code{Cuda}, and
|
||||
\code{Metal} but \emph{not} \code{Rocm}. Because \code{Storage::Rocm} only exists when the \code{rocm}
|
||||
feature is enabled, and no one had ever built \code{ring} $+$ \code{rocm} together, the non-exhaustive
|
||||
\code{match} was never forced into existence. The moment you try to build the one configuration a
|
||||
heterogeneous ROCm$+$CUDA cluster needs, it fails to compile. The fix adds the two
|
||||
\code{\#[cfg(feature = "rocm")]} arms (one in \code{sum\_all\_reduce}, one in \code{all\_gather}) that
|
||||
stage a ROCm tensor to host exactly like the others. This is the subtler lesson: a feature-gated code
|
||||
path is not merely untested, it may be \emph{unbuilt}, and a green CI that never enables that flag
|
||||
combination proves nothing about it.
|
||||
|
||||
\paragraph{Bug 3 --- a handshake that assumed both nodes were the same speed.}
|
||||
The ring's neighbour-connect loop gave up after 10 seconds. On a homogeneous cluster every rank boots in
|
||||
lockstep and 10s is generous. On a \emph{heterogeneous} cluster the nodes do not load at the same rate ---
|
||||
a Blackwell dev kit and an APU parse weights, warm kernels, and reach the barrier at visibly different
|
||||
times --- so the faster node would time out waiting for the slower one and abort the whole ring. Raising
|
||||
the timeout to 120s (and matching the panic message) lets asynchronous, uneven startup settle. The
|
||||
assumption ``all ranks are interchangeable, so they arrive together'' is exactly the kind of invariant
|
||||
that is silently true on one vendor and silently false across three.
|
||||
|
||||
None of these three is deep. That is the message. A distributed path can be well-typed, code-reviewed,
|
||||
and shipped, and still be dead on arrival in three independent ways, because ``it compiles'' certifies
|
||||
none of the things that actually matter at run time --- the arithmetic of a byte count, the existence of a
|
||||
match arm under a flag nobody set, or a timing assumption baked into a constant. And a fourth bug, subtler
|
||||
than all three and invisible until the generated tokens themselves came out wrong, waited past these; it
|
||||
is the root cause of cross-vendor incoherence, and its fix is what turns prefill into coherent decode
|
||||
(\S\ref{sec:rootcause}).
|
||||
|
||||
\section{Result: coherent decode across the AMD/NVIDIA boundary}
|
||||
\label{sec:result}
|
||||
|
||||
With the four heterogeneity bugs fixed --- three mechanical (\S\ref{sec:compiles}), one numeric
|
||||
(\S\ref{sec:rootcause}) --- the cross-vendor ring generates coherent output. On a two-node ring ---
|
||||
\emph{evo} (AMD ROCm gfx1151) as rank~0, \emph{spark} (NVIDIA CUDA sm\_121) as rank~1 --- connected by a
|
||||
2.5GbE direct cable, the sharded engine:
|
||||
|
||||
\begin{enumerate}[leftmargin=1.5em,itemsep=1pt]
|
||||
\item \textbf{builds} with \code{ring} $+$ \code{rocm} on the AMD node and \code{ring} $+$ \code{cuda} on
|
||||
the NVIDIA node (Bug~2 was the blocker);
|
||||
\item \textbf{connects} the ring --- each rank binds its listen port, dials its right neighbour, and
|
||||
accepts its left, the handshake surviving the uneven boot (Bug~3);
|
||||
\item \textbf{shards} a Qwen3-0.6B at \code{world\_size}$=$2, each node holding half of every parallelized
|
||||
weight matrix via the \code{ColumnParallelLayer}/\code{RowParallelLayer} split; and
|
||||
\item \textbf{decodes coherently} --- running the host-staged \code{f32} \code{all\_reduce} across the
|
||||
vendor boundary at every layer --- emitting a chain-of-thought and a correct final answer in lockstep, at
|
||||
\textbf{50.4~tok/s decode} (TTFT 0.09s, 213~tok/s prefill).
|
||||
\end{enumerate}
|
||||
|
||||
The two ranks run \emph{different} 16-bit compute dtypes --- \emph{evo} in \code{f16}, \emph{spark} in
|
||||
\code{bf16} (\S\ref{sec:rootcause}) --- and still agree, because the reduction crosses the wire in
|
||||
canonical \code{f32}. To our knowledge this is the first reported cross-vendor (AMD$+$NVIDIA)
|
||||
tensor-parallel LLM inference over commodity ethernet in a single engine: two different-vendor
|
||||
accelerators jointly executing one tensor-parallel forward \emph{and} autoregressive decode, over a
|
||||
commodity cable, with no shared-vendor collective (NCCL/RCCL) between them --- the \code{all\_reduce} that
|
||||
both libraries refuse across the boundary runs to completion on plain TCP.
|
||||
|
||||
\paragraph{Controls isolate the wire, not the model.} A \emph{same}-vendor control --- both ranks forced
|
||||
to \code{f16}, so the raw-bytes shortcut and the canonical-\code{f32} wire coincide --- produces the same
|
||||
tokens as the cross-vendor run, which pins the dtype wire (\S\ref{sec:rootcause}), not the ring
|
||||
mechanism, as the only thing heterogeneity changed. A \emph{single}-node control --- the same Qwen3-0.6B
|
||||
on \emph{spark} alone --- generates the same answer at 168~tok/s decode and 464~tok/s prefill. So the
|
||||
cross-vendor 50.4~tok/s is $\approx$3.3$\times$ slower than single-node decode, and the entire gap is
|
||||
synchronization: a host-staged \code{all\_reduce} every layer, over the wire, is the only added work. The
|
||||
model is coherent and the kernels are at parity; what the ring costs is the wire.
|
||||
|
||||
\section{Root cause and fix: the heterogeneous-dtype-on-the-wire hazard}
|
||||
\label{sec:rootcause}
|
||||
|
||||
The fourth bug was invisible until the tokens themselves came out wrong, and it is the one worth dwelling
|
||||
on, because it is the trap waiting in every host-staged cross-vendor collective. We state the cause, the
|
||||
fix, and the residuals the fix leaves.
|
||||
|
||||
\subsection{The cause: two ranks legitimately pick different 16-bit formats}
|
||||
\label{sec:dtype}
|
||||
|
||||
The engine picks its compute dtype \emph{per node} from the \emph{local} device. The selector
|
||||
(\code{determine\_auto\_dtype\_all}) returns \code{f16} when it sees a ROCm gfx1151 device --- RDNA3.5 has
|
||||
fast \code{f16} matrix cores (WMMA) but no fast \code{bf16} matmul --- and \code{bf16} for the CUDA
|
||||
Blackwell path. So on a mixed ring the two ranks legitimately and independently choose \emph{different}
|
||||
16-bit formats: \emph{evo} runs the model in \code{f16}, \emph{spark} in \code{bf16}. The original
|
||||
host-staged ring shipped \emph{raw device-dtype bytes} and the receiver reinterpreted them as \emph{its
|
||||
own} local dtype. Since \code{f16} and \code{bf16} share a 16-bit container but split it differently ---
|
||||
5-bit exponent / 10-bit mantissa versus 8-bit exponent / 7-bit mantissa --- every value crossing the
|
||||
boundary was silently re-read with the wrong exponent/mantissa layout. The bit pattern was intact; its
|
||||
\emph{interpretation} was wrong, which is worse, because nothing errored: the \code{all\_reduce} summed
|
||||
correctly-typed nonsense into correctly-typed nonsense, corrupting activations from the first layer.
|
||||
Notably, a \emph{same}-vendor ring never exposed this --- both ranks pick the same dtype and the raw-bytes
|
||||
shortcut accidentally works --- which is precisely why the hazard survived months of same-vendor NCCL and
|
||||
ring testing until the first cross-vendor run, and why the same-vendor control (\S\ref{sec:result}) is the
|
||||
clean isolator.
|
||||
|
||||
\subsection{The fix: a canonical \code{f32} wire and an \code{N}-node accumulating reduce}
|
||||
\label{sec:fix}
|
||||
|
||||
The remedy is structural, not a patch: the wire needs a \emph{canonical} dtype both ranks agree on
|
||||
independent of their local compute dtype. \code{f32} is the natural choice --- a lossless superset of both
|
||||
\code{f16} and \code{bf16}, and higher-precision accumulation for the reduction besides. The shipped
|
||||
reduce (Listing~\ref{lst:stage}) stages each device tensor to host, converts to \code{f32}
|
||||
(\code{cpu\_to\_f32}), exchanges and sums in \code{f32}, and converts the result back to the receiver's
|
||||
native device dtype. The 16-bit formats now meet only \emph{inside} each node; the wire is always
|
||||
\code{f32}, so both ranks reduce the same values bit-for-bit regardless of vendor, and the reduction is
|
||||
carried out in the CPU-\code{f32} path with a canonical tie-break so a ROCm box and a CUDA box do not
|
||||
diverge on rounding order.
|
||||
|
||||
The same pass closed a latent correctness bound. The original \code{sum\_all\_reduce} performed a
|
||||
\emph{single} write-right / read-left exchange and summed the neighbour's shard --- a complete all-reduce
|
||||
for two ranks, but for \code{N}$>$2 rank $i$ only ever incorporates rank $i{-}1$, so ranks $i{-}2,
|
||||
i{-}3,\dots$ never reach it. The fix replaces the single exchange with the textbook \code{world\_size}$-$1
|
||||
accumulating ring (each rank forwards and sums its running shard for \code{N}$-$1 steps), correct for any
|
||||
\code{N}$\geq$2, and removes a spurious power-of-two world-size restriction inherited from the NCCL path
|
||||
(\code{is\_power\_of\_two}) that had made four the smallest cluster past two nodes the ring would even
|
||||
admit. The all-reduce is now algorithmically correct for any \code{N}$\geq$2; the \emph{measured}
|
||||
cross-vendor demonstration in this paper is the two-node \emph{evo}$+$\emph{spark} ring, and multi-node
|
||||
(\code{N}$>$2, e.g.\ adding \emph{dbc}/Metal) cross-vendor runs are the natural next validation, not a
|
||||
claimed measurement here.
|
||||
|
||||
\subsection{Residuals: three problems the fix leaves open}
|
||||
\label{sec:residual}
|
||||
|
||||
Coherent cross-vendor decode holds, but three residuals bound the result. We state each crisply and
|
||||
develop all three, alongside five further problems, as a research agenda in the companion~\cite{openproblems}.
|
||||
|
||||
\paragraph{Sampling divergence (a determinism boundary).} The \emph{reduced activations} now agree
|
||||
bit-for-bit, but each rank still computes its final logits on its own device in its own 16-bit dtype and
|
||||
samples \emph{locally}. In the demonstrated runs the argmax stays in lockstep; a near-argmax-tie could in
|
||||
principle resolve differently under \code{f16} versus \code{bf16} rounding and desync the per-rank KV
|
||||
caches. The robust cure --- the head rank samples and broadcasts the token id, so every rank advances on
|
||||
the same token --- removes the divergence by construction; it is a determinism-boundary problem in
|
||||
replicated-state distributed inference, not a numeric one.
|
||||
|
||||
\paragraph{Graphless lockstep (an execution-symmetry assumption).} The two ranks run \emph{two}
|
||||
generation loops kept in phase by the per-token \code{all\_reduce} barrier. To stay in phase the CUDA rank
|
||||
must run \emph{graphless}, matching the graphless ROCm rank and forgoing the $\approx$1.5$\times$ decode
|
||||
speedup CUDA-graph capture would give~\cite{backend}. Decode is coherent this way, but per-token
|
||||
cross-node lockstep between nodes that legitimately want different execution strategies is fragile --- an
|
||||
argument for pipeline parallelism (\S\ref{sec:pipeline}), where each node's execution is a purely local
|
||||
choice, rather than a bug to patch.
|
||||
|
||||
\paragraph{Format coupling (a sharding limitation).} The TP collectives are wired through the
|
||||
non-quantized (safetensors) model path --- \code{ColumnParallelLayer} and \code{RowParallelLayer} live in
|
||||
the normal model definitions --- and \emph{not} through the GGUF quantized path. So even with the
|
||||
numerics correct, TP cannot yet shard the very block-quantized formats that make edge inference fit in
|
||||
commodity memory. Quant-aware sharding (splitting on superblock boundaries without a dequant--requant
|
||||
loss) is an open problem~\cite{openproblems}; pipeline parallelism sidesteps it (\S\ref{sec:pipeline})
|
||||
because activations, not weights, cross the seam.
|
||||
|
||||
\section{The case for pipeline parallelism}
|
||||
\label{sec:pipeline}
|
||||
|
||||
The dtype hazard and the \code{N}-node reduce are fixed, so the tensor-parallel ring on \code{main} is now
|
||||
\emph{correct}. But correct is not the same as well-matched, and the residuals of \S\ref{sec:residual}
|
||||
together with the measured synchronization tax point the same way. Tensor parallelism was built for a
|
||||
homogeneous, tightly-coupled, high-bandwidth, single-vendor machine, and every place it strains on our
|
||||
cluster is a place it assumes that machine: a per-layer all-reduce assumes cheap bandwidth
|
||||
(\S\ref{sec:design}) --- the measured $\approx$3.3$\times$ decode tax is exactly this bill; per-token
|
||||
lockstep assumes symmetric execution (\S\ref{sec:residual}); the parallel layers assume dequantized
|
||||
weights (\S\ref{sec:residual}). Fixing the numerics made the primitive correct; it did not make it the
|
||||
right primitive for a heterogeneous fleet.
|
||||
|
||||
\textbf{Pipeline parallelism} inverts the mismatch. Partition the model into contiguous \emph{layer
|
||||
ranges}, one range per node; a token's hidden state flows through node~0's layers, crosses the wire once
|
||||
to node~1, flows through its layers, and so on. The consequences line up against the strains almost
|
||||
one-to-one:
|
||||
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=2pt]
|
||||
\item \textbf{Activations-only on the wire.} What crosses a node boundary is a single hidden-state tensor,
|
||||
once per boundary, forward --- not an \code{all\_reduce} of every layer's output twice per block. On a
|
||||
$\approx$280~MB/s link that is the difference between a per-token tax of \code{O(1)} small transfers and
|
||||
\code{O(layers)} full-width collectives; it is the same $\approx$3.3$\times$ tax converted from
|
||||
per-layer to per-node.
|
||||
\item \textbf{One canonical cast, not a per-op reinterpretation.} The activation is a single tensor at
|
||||
each boundary, so agreeing on one wire dtype (\S\ref{sec:dtype}) is one cast at one seam per node, not a
|
||||
property every collective must carry. The heterogeneous-dtype hazard shrinks from pervasive to pointwise.
|
||||
\item \textbf{Format-agnostic.} Activations are just floats; the boundary does not know or care whether the
|
||||
node that produced them ran a safetensors or a GGUF model, on which vendor, with which kernel. Pipeline
|
||||
parallelism sidesteps the format coupling of \S\ref{sec:residual} for free, so a mixed cluster can run
|
||||
quantized shards.
|
||||
\item \textbf{One generation loop, stateless workers.} The head node runs the single sampling/generation
|
||||
loop; the others are stateless layer-executors that receive an activation, run their layers (holding only
|
||||
their own KV cache), and forward. There is no second generation loop to keep in phase and no cross-node
|
||||
per-token lockstep, so the CUDA-graph asymmetry (\S\ref{sec:residual}) becomes a purely \emph{local}
|
||||
choice --- each node may capture graphs for its own layers or not, and nobody desynchronizes --- and the
|
||||
sampling divergence collapses to the head's single sampler.
|
||||
\end{itemize}
|
||||
|
||||
Pipeline parallelism is not a free lunch --- for a single decode stream it adds pipeline-bubble latency
|
||||
proportional to the number of stages, and throughput needs micro-batching to fill the
|
||||
pipe~\cite{gpipe}. Its wide-area latency budget --- one network hop per stage on the critical path of
|
||||
every token --- is the subject of the companion latency analysis~\cite{latency}, which shows the budget
|
||||
is spendable precisely because pipelining overlaps transport under compute and speculative decoding
|
||||
amortizes the round-trip. But its \emph{coupling} to the cluster is minimal and its assumptions are the
|
||||
ones a heterogeneous fleet can actually honour. The right primitive for one-model-across-many-vendors is
|
||||
the one that puts the least on the wire and demands the least agreement between ranks, and on both axes
|
||||
that is pipeline, not tensor, parallelism. The cross-vendor decode result (\S\ref{sec:result}) proves the
|
||||
\emph{channel} works across the boundary; the argument here is about which \emph{traffic pattern} to run
|
||||
over it, and it is under construction in the same codebase.
|
||||
|
||||
\section{Reproducibility}
|
||||
\label{sec:repro}
|
||||
|
||||
\paragraph{Hardware and link.} \emph{evo}: AMD Ryzen AI Max+ 395 ``Strix Halo,'' Radeon 8060S iGPU
|
||||
(gfx1151, RDNA3.5), ROCm. \emph{spark}: NVIDIA GB10 Grace-Blackwell (sm\_121), CUDA. \emph{dbc}: Apple
|
||||
M4 Max, Metal. \emph{evo}$\leftrightarrow$\emph{spark} is a direct 2.5GbE cable (static
|
||||
\code{192.168.77.0/24}), whose measured sustained-TCP goodput is $\approx$280~MB/s ($\approx$2.24~Gbit/s);
|
||||
its wide-area latency budget is analysed in the companion~\cite{latency}. The three-node substrate is a
|
||||
wired ethernet mesh with \emph{spark} kernel-IP-forwarding the closing \emph{evo}$\leftrightarrow$\emph{dbc}
|
||||
edge, so no switch or router is required.
|
||||
|
||||
\paragraph{Build.} Each node builds the \emph{same} engine source with its own backend feature plus
|
||||
\code{ring}: \code{--features "cuda ring"} on \emph{spark}, \code{--features "rocm ring"} on \emph{evo}.
|
||||
The three mechanical fixes are at commit \code{63cc4c7ac}; the canonical-\code{f32} wire and the
|
||||
\code{world\_size}$-$1 reduce are on \code{main} (\code{hanzo-quant/src/distributed/},
|
||||
\code{hanzo-engine/src/ops.rs}).
|
||||
|
||||
\paragraph{Ring configuration.} The ring backend is activated by pointing \code{RING\_CONFIG} at a JSON
|
||||
file per node; rank $r$ binds \code{port} and dials its right neighbour at
|
||||
\code{right\_ip:right\_port}. A two-node \emph{evo}(rank~0)$+$\emph{spark}(rank~1) ring:
|
||||
|
||||
\begin{lstlisting}[language=Rust,caption={\code{RING\_CONFIG} for a 2-node cross-vendor ring. Rank~0
|
||||
(evo) listens on 9000 and dials spark; rank~1 (spark) listens on 9001 and dials back to evo, closing the
|
||||
ring.},label={lst:ring}]
|
||||
// evo (rank 0, 192.168.77.1): RING_CONFIG=/etc/hanzo/ring0.json
|
||||
{ "master_ip": null, "master_port": 9002,
|
||||
"port": 9000, "right_ip": "192.168.77.2", "right_port": 9001,
|
||||
"rank": 0, "world_size": 2 }
|
||||
|
||||
// spark (rank 1, 192.168.77.2): RING_CONFIG=/etc/hanzo/ring1.json
|
||||
{ "master_ip": "192.168.77.1", "master_port": 9002,
|
||||
"port": 9001, "right_ip": "192.168.77.1", "right_port": 9000,
|
||||
"rank": 1, "world_size": 2 }
|
||||
\end{lstlisting}
|
||||
|
||||
\noindent A \code{hanzo cluster init} / \code{hanzo cluster join <token>} pair wraps this JSON in a
|
||||
scan-to-join token (base58 of the topology plus a terminal QR code) so the operator never hand-writes the
|
||||
ports; the token bakes in the peer list and each worker self-assigns its rank. Both ranks launch the
|
||||
ordinary \code{hanzo serve} with \code{RING\_CONFIG} set and the same model.
|
||||
|
||||
\paragraph{What is measured vs.\ open.} Measured on the real 2.5GbE link: cross-vendor build, connect,
|
||||
\code{world\_size}$=$2 shard, and coherent multi-token \emph{decode} across the AMD/NVIDIA boundary
|
||||
(50.4~tok/s decode, TTFT 0.09s, 213~tok/s prefill on Qwen3-0.6B); the same-vendor \code{f16}$+$\code{f16}
|
||||
control matching token-for-token; and the single-node \emph{spark} control (168~tok/s decode, 464~tok/s
|
||||
prefill). Open (\S\ref{sec:residual}, \cite{openproblems}): head-broadcast sampling to close the
|
||||
determinism boundary, \code{N}$>$2 cross-vendor runs, GGUF-path TP, and the migration to pipeline
|
||||
parallelism. No projected or synthetic number appears in this paper.
|
||||
|
||||
\section{Related work and contribution}
|
||||
\label{sec:related}
|
||||
|
||||
Tensor parallelism is Megatron-LM~\cite{megatron}; pipeline parallelism is GPipe and its
|
||||
successors~\cite{gpipe}; the on-device collectives are NCCL~\cite{nccl} and RCCL~\cite{rccl}, each
|
||||
single-vendor by design. Distributed inference over commodity/heterogeneous networks has been explored
|
||||
most prominently by Petals~\cite{petals}, which runs pipeline-parallel shards across volunteer nodes over
|
||||
the public internet --- close in spirit to \S\ref{sec:pipeline}, and evidence that the activations-only
|
||||
pattern is the one that survives a slow, loose network. What we add is narrow and, to our knowledge, not
|
||||
previously shown: a \emph{single} inference engine, one source lowered to three vendors' kernels, running
|
||||
a \emph{tensor-parallel} forward \emph{and coherent decode} across an \emph{AMD$+$NVIDIA} boundary on
|
||||
\emph{commodity ethernet} with \emph{no} shared-vendor collective --- and an unusually explicit account of
|
||||
the failure modes that a host-staged cross-vendor collective exposes (the \code{len}$^2$ overflow, the
|
||||
unbuilt ROCm arm, the startup-race timeout, and the heterogeneous-dtype wire). The design contribution is
|
||||
the observation, made concrete by those failures, that heterogeneity is a stress test for homogeneity
|
||||
assumptions, and that pipeline parallelism minimizes exactly the assumptions that break. This paper is the
|
||||
distributed extension of the ``one source, every backend'' thesis~\cite{backend,umbrella}: having made one
|
||||
engine run on every backend, we make those backends run in one cluster. Two companions extend the arc: the
|
||||
latency-physics analysis~\cite{latency} carries the wide-area budget for \S\ref{sec:pipeline}, and the
|
||||
three-plane compute-fabric blueprint~\cite{consensus} turns the trusted-LAN ring into a permissionless,
|
||||
consensus-managed, PQ-authenticated network. The open questions this result raises are collected as a
|
||||
research agenda in the open-problems companion~\cite{openproblems}.
|
||||
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
No collective library crosses a vendor boundary, so a mixed AMD/NVIDIA/Apple fleet cannot run one model
|
||||
together through the standard path. Host-staging the collective through CPU memory and TCP makes the
|
||||
channel vendor-agnostic, and on that channel one engine --- built from a single source for three vendors'
|
||||
kernels --- shards a model \code{world\_size}$=$2 across an AMD/NVIDIA boundary over a 2.5GbE cable and
|
||||
\emph{decodes coherently} at 50.4~tok/s, which we believe is a first. The road to that result ran through
|
||||
three bugs in a path that compiled for months but had never run --- a reminder that ``it builds''
|
||||
certifies almost nothing about a distributed system --- and one deeper hazard, a heterogeneous-dtype wire
|
||||
that a same-vendor cluster can never expose, cured by a canonical \code{f32} payload and an accumulating
|
||||
\code{world\_size}$-$1 all-reduce correct for any \code{N}$\geq$2. The measured $\approx$3.3$\times$
|
||||
synchronization tax, and the residuals that remain --- local sampling that can diverge at a near-tie,
|
||||
graphless lockstep between two generation loops, and TP's coupling to the dequantized weight path --- all
|
||||
point one way: toward pipeline parallelism, whose activations-only, one-generation-loop, format-agnostic
|
||||
shape asks of a heterogeneous cluster only what such a cluster can actually give. The channel is open
|
||||
across the boundary and coherent over it; the companions drive the right traffic~\cite{latency}, harden
|
||||
it into a network~\cite{consensus}, and enumerate what is left to solve~\cite{openproblems}.
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\small
|
||||
\bibitem{backend} Hanzo AI. \emph{Edge LLM Inference Across Commodity Accelerators: Measured AMD, NVIDIA,
|
||||
and Apple Performance, and hanzo-engine at llama.cpp Decode Parity on Every Backend.} Paper 05.
|
||||
\bibitem{umbrella} Hanzo AI. \emph{One Source, Every Backend: The hanzo-kernel DSL --- Thesis and Guided
|
||||
Tour.} Paper 07 (umbrella).
|
||||
\bibitem{fuse} Hanzo AI. \emph{Fusion Is Composition: Kernel Fusion as the Functor Law.} Paper 07.1.
|
||||
\bibitem{oracle} Hanzo AI. \emph{One Oracle, Every Backend: The CPU Bit-Exactness Law.} Paper 07.4.
|
||||
\bibitem{latency} Hanzo AI --- ML Systems. \emph{The Latency Physics of Geo-Distributed Autoregressive
|
||||
Decode.} Distributed-inference companion (analytical).
|
||||
\bibitem{consensus} Hanzo AI --- ML Systems. \emph{The Three-Plane Compute Fabric: Consensus-Managed,
|
||||
ZAP-Transported Decentralized AI Inference.} Distributed-inference companion (architecture).
|
||||
\bibitem{openproblems} Hanzo AI --- ML Systems. \emph{Open Problems in Heterogeneous Distributed
|
||||
Inference.} Distributed-inference companion (research agenda).
|
||||
\bibitem{megatron} M.\ Shoeybi, M.\ Patwary, R.\ Puri, P.\ LeGresley, J.\ Casper, B.\ Catanzaro.
|
||||
\emph{Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.} arXiv:1909.08053, 2019.
|
||||
\bibitem{gpipe} Y.\ Huang et al. \emph{GPipe: Efficient Training of Giant Neural Networks using Pipeline
|
||||
Parallelism.} NeurIPS, 2019.
|
||||
\bibitem{petals} A.\ Borzunov et al. \emph{Petals: Collaborative Inference and Fine-tuning of Large
|
||||
Models.} arXiv:2209.01188, 2022.
|
||||
\bibitem{nccl} NVIDIA. \emph{NCCL: NVIDIA Collective Communications Library.}
|
||||
\bibitem{rccl} AMD. \emph{RCCL: ROCm Communication Collectives Library.}
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Reference in New Issue
Block a user