feat(distributed-inference): cross-vendor TP result + open-problems companion

Four papers on heterogeneous distributed LLM inference: - hanzo-heterogeneous-distributed-inference: cross-vendor (AMD ROCm gfx1151 + NVIDIA GB10 CUDA) tensor-parallel decode over a 2.5GbE cable, coherent at 50.4 tok/s (TTFT 0.09s, 213 tok/s prefill). Root cause of prior corruption proven: per-backend auto-dtype (ROCm->f16, CUDA->bf16) reinterpreted raw on the wire. Fixed by a canonical f32 ring wire + world_size-1 accumulating all-reduce (correct for any N>=2; spurious power-of-two gate removed). Same-vendor f16+f16 control matches token-for-token. - hanzo-distributed-open-problems: NEW companion enumerating 8 open research problems (sampling-determinism boundaries, quant-aware TP sharding, PP layer-assignment ILP, adaptive speculation under RTT/acceptance drift, the prefill launch-overhead wall, KV wire compression, the geo decode latency floor, and cheap verifiable inference), each cross-referenced to the paper that owns the surrounding result. - hanzo-distributed-decode-latency, hanzo-consensus-zap-compute: consistency pass (canonical f32 wire terminology; link goodput unified to 280 MB/s = 2.24 Gbit/s; precise f16-vs-bf16 root-cause wording). INDEX.md regenerated via `make index`; besides the 4 above it also indexes 6 pre-existing root papers the previous index had missed. All four compile clean with pdflatex (10/10/13/9 pages).
2026-10-22 17:20:17 +00:00 · 2026-07-09 16:31:20 -07:00
parent 7fcaab6495
commit 7fdca28595
9 changed files with 2177 additions and 1 deletions
@@ -42,14 +42,18 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
 | `hanzo-api-gateway` | ✓ | `hanzo-api-gateway/hanzo-api-gateway.tex` |
 | `hanzo-aso` | ✓ | `hanzo-aso/hanzo-aso.tex` |
 | `hanzo-base-runtime` | ✓ | `hanzo-base-runtime/hanzo-base-runtime.tex` |
 | `hanzo-bitexact-oracle` | ✓ | `hanzo-bitexact-oracle.tex` |
 | `hanzo-blockchain-platform` | ✓ | `hanzo-blockchain-platform/hanzo-blockchain-platform.tex` |
 | `hanzo-candle` | ✓ | `hanzo-candle/hanzo-candle.tex` |
 | `hanzo-chat` | ✓ | `hanzo-chat/hanzo-chat.tex` |
 | `hanzo-checkout` | ✓ | `hanzo-checkout/hanzo-checkout.tex` |
 | `hanzo-cloud-infrastructure` | ✓ | `hanzo-cloud-infrastructure/hanzo-cloud-infrastructure.tex` |
 | `hanzo-collapse-methodology` | ✓ | `hanzo-collapse-methodology.tex` |
 | `hanzo-commerce-payments` | ✓ | `hanzo-commerce-payments.tex` |
 | `hanzo-comptime-autotune` | ✓ | `hanzo-comptime-autotune.tex` |
 | `hanzo-consensus-ai` | ✓ | `hanzo-consensus-ai/hanzo-consensus-ai.tex` |
 | `hanzo-consensus-plugin-platform` | — | `hanzo-consensus-plugin-platform/main.tex` |
 | `hanzo-consensus-zap-compute` | ✓ | `hanzo-consensus-zap-compute.tex` |
 | `hanzo-console-dashboard` | ✓ | `hanzo-console-dashboard.tex` |
 | `hanzo-continuous-learning-privacy` | ✓ | `hanzo-continuous-learning-privacy.tex` |
 | `hanzo-cross-backend-inference` | ✓ | `hanzo-cross-backend-inference.tex` |
@@ -58,6 +62,8 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
 | `hanzo-datastore` | — | `hanzo-datastore/hanzo-datastore.tex` |
 | `hanzo-deepseek-v4-dspark` | — | `hanzo-deepseek-v4-dspark.tex` |
 | `hanzo-digital-securities-integration` | ✓ | `hanzo-digital-securities-integration/hanzo-digital-securities-integration.tex` |
 | `hanzo-distributed-decode-latency` | ✓ | `hanzo-distributed-decode-latency.tex` |
 | `hanzo-distributed-open-problems` | ✓ | `hanzo-distributed-open-problems.tex` |
 | `hanzo-dso` | ✓ | `hanzo-dso/hanzo-dso.tex` |
 | `hanzo-edge-inference` | ✓ | `hanzo-edge-inference.tex` |
 | `hanzo-encryption-security` | ✓ | `hanzo-encryption-security/hanzo-encryption-security.tex` |
@@ -67,9 +73,11 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
 | `hanzo-fhe-inference` | ✓ | `hanzo-fhe-inference/hanzo-fhe-inference.tex` |
 | `hanzo-flow` | ✓ | `hanzo-flow/hanzo-flow.tex` |
 | `hanzo-formal-verification` | ✓ | `hanzo-formal-verification/hanzo-formal-verification.tex` |
 | `hanzo-fusion-is-composition` | ✓ | `hanzo-fusion-is-composition.tex` |
 | `hanzo-gateway-krakend` | ✓ | `hanzo-gateway-krakend.tex` |
 | `hanzo-gpu-inference` | ✓ | `hanzo-gpu-inference/hanzo-gpu-inference.tex` |
 | `hanzo-harness-hacking` | ✓ | `hanzo-harness-hacking/hanzo-harness-hacking.tex` |
 | `hanzo-heterogeneous-distributed-inference` | ✓ | `hanzo-heterogeneous-distributed-inference.tex` |
 | `hanzo-hmm` | ✓ | `hanzo-hmm/hanzo-hmm.tex` |
 | `hanzo-iam-platform` | ✓ | `hanzo-iam-platform.tex` |
 | `hanzo-ico-infrastructure` | ✓ | `hanzo-ico-infrastructure/hanzo-ico-infrastructure.tex` |
@@ -77,6 +85,8 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
 | `hanzo-inference-stack` | ✓ | `hanzo-inference-stack.tex` |
 | `hanzo-ingress-proxy` | ✓ | `hanzo-ingress-proxy.tex` |
 | `hanzo-insights-analytics` | ✓ | `hanzo-insights-analytics.tex` |
 | `hanzo-int8-mmq-negative-result` | ✓ | `hanzo-int8-mmq-negative-result.tex` |
 | `hanzo-intrinsic-islands` | ✓ | `hanzo-intrinsic-islands.tex` |
 | `hanzo-jin-architecture` | ✓ | `hanzo-jin-architecture/hanzo-jin-architecture.tex` |
 | `hanzo-jin` | ✓ | `hanzo-jin/hanzo-jin.tex` |
 | `hanzo-kernel-dsl` | ✓ | `hanzo-kernel-dsl.tex` |
@@ -203,4 +213,4 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
 | `ZEN_WHITEPAPER_2025` | ✓ | `zen/ZEN_WHITEPAPER_2025.tex` |
 | `zen-world` | ✓ | `zen/zen-world.tex` |
-**Total**: 198 papers, 192 with compiled PDFs
+**Total**: 208 papers, 202 with compiled PDFs
@@ -0,0 +1,714 @@
 \documentclass[11pt]{article}
 \input{shared/kernel-preamble.tex}
 \usepackage{amsthm}
 \newtheorem{proposition}{Proposition}
 % Honesty labels: distinguish what is built from what is designed.
 \newcommand{\built}{\textcolor{str}{\small\textbf{[built]}}}
 \newcommand{\partialb}{\textcolor{kw}{\small\textbf{[partial]}}}
 \newcommand{\design}{\textcolor{cmt}{\small\textbf{[design]}}}
 \title{\textbf{The Three-Plane Compute Fabric:\\
 Consensus-Managed, ZAP-Transported\\ Decentralized AI Inference}}
 \author{Hanzo AI --- ML Systems}
 \date{Architecture Blueprint --- \emph{One Source, Every Backend} series, 2026}
 \begin{document}
 \maketitle
 \begin{abstract}
 A heterogeneous GPU fleet already runs a single model across ROCm, CUDA, and Metal
 today: the Hanzo engine's socket-ring shards a model tensor- and pipeline-parallel across
 mismatched accelerators over a canonical \code{f32} wire, and a \code{hanzo cluster}
 command onboards a worker by scanning a QR token. But that ring is configured by a static
 JSON file handed out of band, and its bytes cross the wire as \emph{plaintext TCP} between
 nodes that never authenticate each other. It is a LAN appliance, not a network. This paper
 is the blueprint for the missing two-thirds: turning that ring into a permissionless,
 byzantine-fault-tolerant AI inference \emph{network} that any developer --- or any
 public-cloud GPU miner --- can join and be paid for useful work. The thesis is one of
 \emph{decomplection}: a serving cluster has three concerns that monolithic stacks braid
 together and we hold apart as three orthogonal, independently-complete planes. The
 \textbf{control plane} is a Lux \emph{Quasar} consensus chain: membership, permissionless
 miner admission, layer placement, and config distribution become BFT-agreed state, with the
 Photon committee-selection primitive (Fisher--Yates over a luminance-weighted population)
 reused to sample GPU nodes and luminance re-bound from consensus-vote success to
 \emph{validated inference contribution}. The \textbf{transport} is \emph{ZAP}, the Lux
 zero-copy wire, whose post-quantum handshake (\code{WrapPQ} over an authenticated
 \code{Session} exposing a verified 32-byte \code{PeerID}) carries \emph{both} consensus
 control messages \emph{and} tensor activations over one PQ-authenticated channel --- retiring
 the plaintext ring sockets and converting the QR join token from a bearer secret into a
 handshake credential. The \textbf{data plane} is the existing inference ring, unchanged in
 kind: it receives its topology from consensus instead of a JSON file and moves its tensors
 over ZAP instead of raw TCP. We are scrupulous about the built/designed line. Built and
 cited to source: the cross-vendor ring, the \code{hanzo cluster} scan-to-join CLI, the ZAP
 Go wire with its PQ handshake, the Quasar engine with Photon/Wave/Focus and BLS$+$Pulsar$+$ML-DSA
 threshold signing, and the ZAP measurements ($17\times$ serialization and $345\times$ parse
 versus JSON-RPC, $\sim$$4.5\times$ versus gRPC with zero marshal/unmarshal). Designed, and
 labelled as such: consensus-driven membership, the ZAP-swapped ring transport, miner
 admission, and result verification by threshold-signing over inference outputs. No benchmark
 here is invented; every measured number is attributed to the code that produced it. The
 contribution is the architecture and its honest status, offered as a blueprint the wider
 AI-infrastructure community can build on.
 \end{abstract}
 \section{Introduction}
 \label{sec:intro}
 The dominant shape of a model-serving stack is a monolith: one operator, one datacenter, one
 process tree that owns scheduling, transport, and tensor math all at once. vLLM, TGI, and
 Triton each fuse ``who is in the cluster,'' ``how bytes move between ranks,'' and ``what the
 GPUs compute'' into a single trusted binary on a private network. That fusion is fine when the
 operator owns every GPU. It fails the moment the fleet is \emph{heterogeneous} (a ROCm mini-PC,
 a rented CUDA instance, an Apple laptop), \emph{permissionless} (a stranger's GPU wants to
 contribute and be paid), or \emph{untrusted} (the node holding your activations is not yours).
 None of those three failures is a math problem; each is a \emph{concerns} problem. The cluster
 braids together things that want to be separate.
 This paper takes the opposite stance, the one our engineering culture states as a law: one and
 only one way to do everything, composable and orthogonal. A decentralized inference cluster has
 exactly three concerns, and we give each its own plane:
 \begin{itemize}[leftmargin=1.4em,itemsep=2pt]
 \item \textbf{Control} --- who is in the cluster, who may join, which node holds which layers,
 and what the run configuration is. This is agreement among mutually distrusting parties: a
 \emph{consensus} problem.
 \item \textbf{Transport} --- how a byte gets from one node to another, and how each end knows
 who the other is. This is an \emph{authenticated wire} problem.
 \item \textbf{Data} --- the actual tensor exchange: activations forward, gradients or
 all-reduce sideways. This is a \emph{numerics and topology} problem.
 \end{itemize}
 The claim is that these three are genuinely orthogonal, that Hanzo and Lux already ship a
 best-in-class primitive for each, and that composing them --- rather than reinventing a
 monolith --- yields something no monolith can be: a GPU network anyone can join. The control
 primitive is Lux \emph{Quasar} consensus~\cite{quasar,luxconsensus}; the transport is
 \emph{ZAP}~\cite{zap,lp201}; the data plane is the Hanzo cross-backend inference
 engine~\cite{infstack}.
 \paragraph{The honesty frame.} This is an architecture paper, and its credibility rests on not
 blurring what runs today with what is designed. We adopt three labels, used inline throughout
 and collected in the status table of \S\ref{sec:status}: \built{} for code on \code{main},
 cited to its file; \partialb{} for a primitive that is compiled and exercised but not yet wired
 end-to-end; and \design{} for a proposed composition. The measured numbers we quote (ring
 parity across vendors, ZAP throughput and latency, Photon and luminance costs) are all \built{},
 each attributed to the benchmark that produced it. The network-level assembly --- consensus as
 the live scheduler, the ring running over ZAP, miner admission and payment --- is \design{},
 and never dressed as more.
 \paragraph{Contribution.} We (i) inventory the built substrate with source citations
 (\S\ref{sec:today}); (ii) state the three-plane architecture and prove its concerns are
 separable (\S\ref{sec:planes}); (iii) give the threat model and a result-integrity scheme for
 byzantine miners (\S\ref{sec:threat}); (iv) bind the economic layer to the existing luminance
 primitive (\S\ref{sec:econ}); (v) lay out a staged migration from the static-config LAN cluster
 to a public network (\S\ref{sec:migration}); and (vi) publish an honest built-versus-designed
 status table (\S\ref{sec:status}) and the limitations (\S\ref{sec:limits}).
 \section{The substrate that exists today}
 \label{sec:today}
 Before proposing a network we account for what is already on \code{main}. Every claim in this
 section is \built{} and cites the file that implements it. The point of the inventory is that
 the three planes are not vaporware waiting to be written: each has a mature primitive, and the
 work is composition, not invention.
 \subsection{Data plane: the heterogeneous inference ring}
 \label{sec:ring}
 The Hanzo engine already runs one model split across several GPUs of \emph{different vendors}.
 Its \code{distributed} module (\code{hanzo-engine/src/distributed.rs},
 \code{hanzo-quant/src/distributed/}) provides two backends behind one \code{Comm} abstraction:
 NCCL for homogeneous NVIDIA, and a portable \emph{ring} backend for everything else. The ring
 does tensor-parallel and pipeline-parallel sharding by exchanging activations with its
 right-hand neighbor, rank $r$ dialing rank $(r{+}1) \bmod N$ in an all-reduce cycle
 (\code{hanzo-quant/src/distributed/socket.rs}). Cross-vendor generation works because the ring
 wire is a \emph{canonical} \code{f32} format and reductions take a backend-independent
 CPU-\code{f32} path with a canonical tie-break (\code{hanzo-engine/src/ops.rs}), so a ROCm box
 and a CUDA box agree bit-for-bit on the reduced tensor rather than each rank reinterpreting the
 other's raw 16-bit words in the wrong \code{f16}-versus-\code{bf16} layout. The topology is a static struct read from a JSON file
 named by the \code{RING\_CONFIG} environment variable (Listing~\ref{lst:ringcfg}): master
 address, per-rank ports, this rank's index, and the world size. The cross-backend engine
 underneath --- five backends, the full GGUF quant zoo, decode parity with \code{llama.cpp} --- is
 the subject of the companion stack paper~\cite{infstack}; here it is simply the data plane.
 \begin{lstlisting}[caption={The data-plane topology today: a static \code{RING\_CONFIG} JSON.
 Every node needs the full member list up front; nothing authenticates a peer.},label={lst:ringcfg}]
 { "master_ip": "10.0.0.1", "master_port": 8000,
  "port": 8001, "right_ip": "10.0.0.2", "right_port": 8001,
  "rank": 0, "world_size": 4 }
 \end{lstlisting}
 Two properties of this substrate matter for everything below. First, it is \emph{correct across
 vendors} --- the hard part, cross-backend numeric agreement, is solved. Second, it is
 \emph{unauthenticated plaintext}: the ring exchanges raw bytes over \code{std::net::TcpStream}
 with \code{write\_all}/\code{read\_exact} and ships its control messages as \code{serde\_json}
 over the same socket. It is safe only on a trusted private mesh. That is the gap the transport
 plane closes.
 \subsection{The onboarding UX: \code{hanzo cluster} scan-to-join}
 \label{sec:clustercli}
 Handing every node a JSON file out of band does not scale past a lab. The \code{hanzo cluster}
 subcommand (\code{hanzo-cli/src/cluster.rs}) already replaces it with a scan-to-join flow.
 \code{hanzo cluster init} on the head (rank 0) enumerates the ring, prints a \emph{join token}
 and its QR code, and launches; \code{hanzo cluster join} on a worker decodes the token and
 launches as a ring daemon. The token is a \code{bs58}-encoded JSON \code{JoinToken}
 (Listing~\ref{lst:token}) --- a nonce, the head address, the world size, the ordered ring
 addresses, and the model --- compact enough to fit in one QR or one paste. This is the correct
 \emph{shape} of onboarding: a worker needs one string to join. But observe what the token is
 today: a \emph{bearer credential in cleartext}. Anyone who photographs the QR can join the ring,
 and the ring cannot tell an invited worker from an intruder, because the token carries topology,
 not identity. Holding that observation is the whole of \S\ref{sec:threat}.
 \begin{lstlisting}[language=Rust,caption={The join token (\code{hanzo-cli/src/cluster.rs}):
 \code{bs58(JSON)} so it fits one QR. It is topology, not identity --- a bearer secret today.},label={lst:token}]
 struct JoinToken {
    nonce: u64,           // ring id, printed {:016x}
    master: String,       // rank-0 (head) reachable IP
    world_size: usize,
    addrs: Option<Vec<String>>, // full ordered ring [rank0..rankN-1]
    model: String,
 }
 \end{lstlisting}
 \subsection{Transport: ZAP, the zero-copy post-quantum wire}
 \label{sec:zap}
 ZAP (Zero-copy Application Protocol, \code{github.com/luxfi/zap}) is the Lux universal
 wire~\cite{zap,lp201}. Its serialization is zero-copy: a message is read directly out of the
 network buffer with no parse and no allocation (measured \built{} parse $2.9$~ns/op at zero
 allocations; build $44$~ns/op). Against the protocols an AI stack would otherwise reach for,
 the wire eliminates the marshal/unmarshal step entirely: $17\times$ faster serialization and
 $345\times$ faster parse than JSON-RPC (the MCP transport), $\sim$$157\times$ faster than
 protobuf on deserialization ($21$~ns versus $3231$~ns, zero allocations), and $\sim$$4.5\times$
 faster than gRPC with the whole codec layer deleted rather than optimized~\cite{zap,lp201,luxconsensus}.
 End-to-end it sustains $114$K transactions/s on a single connection and $376$K across $20$
 parallel connections~\cite{luxconsensus}.
 For a public network the decisive feature is not speed but \emph{authenticated identity}. ZAP's
 PQ layer (\code{conn\_pq.go}) wraps an established connection \emph{after} a post-quantum
 handshake: \code{handshake.Initiator.Run(tcp)} produces a \code{Session}, and \code{WrapPQ(tcp,
 sess)} returns an ordinary \code{net.Conn} that AEAD-frames every write, zeroes its keys on
 close, and --- critically --- exposes \code{PeerID() [32]byte}, ``the verified peer identity for
 callers that need to route on it (e.g.\ validator-set membership checks).'' That one method is
 the hinge of the entire security argument (\S\ref{sec:threat}): the transport already hands the
 control plane a cryptographically verified identity for the node on the other end.
 Inside the Hanzo engine, ZAP is a first-class module (\code{hanzo-engine/src/zap/}). Its
 message and adapter surface is compiled and exercised in-process --- \code{ZapInferenceServer}
 translates ZAP \code{ChatRequest}s into the engine's real \code{Request} channel and streams
 \code{ChatChunk}s back, satisfying the same \code{ZapInference} trait a networked client would
 (\code{server.rs}, \code{client.rs}). We mark this \partialb{}, honestly: the out-of-process
 \code{zap://} transport is a documented stub that carries no \code{capnp} build dependency yet
 (``flipping on the wire is mechanical,'' \code{transport.rs}), and a few methods
 (\code{tokenize}, \code{detokenize}, \code{re\_isq}) return ``not yet wired.'' The wire exists
 in Go and is production-deployed across Lux consensus, MPC, and DEX systems; the engine-side
 Rust surface is built but not yet turned on over a socket.
 \subsection{Control: the Quasar consensus engine}
 \label{sec:quasar}
 Lux \emph{Quasar} (\code{github.com/luxfi/consensus}) is a DAG-native, post-quantum BFT
 consensus engine~\cite{quasar,luxconsensus}. It is built from small, orthogonal sub-protocols,
 four of which this paper reuses directly:
 \begin{itemize}[leftmargin=1.4em,itemsep=2pt]
 \item \textbf{Photon} (\code{protocol/photon}) selects a $K$-of-$N$ committee each round by
 Fisher--Yates shuffle over the node population, drawing with a rejection-sampled uniform integer
 (\code{emitter.go}) to remove the modulo bias that a grinding adversary could exploit on
 committee sampling. The population is weighted by \emph{luminance}.
 \item \textbf{Luminance} (\code{photon/luminance.go}) is a per-node reputation in ``lux'' units:
 each node starts at $100$, a successful vote multiplies its brightness by $1.1$ (capped at
 $1000$), a failure by $0.9$ (floored at $10$), and \code{Brightness} normalizes to a selection
 weight in $[0.1, 10]$. It is a cheap, bounded, exponential reputation --- a brightness update
 costs $72$~ns \built{}.
 \item \textbf{Wave} (\code{protocol/wave}) runs per-round threshold voting with Fast
 Probabilistic Consensus; a vote round costs $3.38$~\textmu s and a $K$-of-$N$ Photon selection
 $3.03$~\textmu s \built{}.
 \item \textbf{Quasar} (\code{protocol/quasar}) seals decisions with threshold signatures across
 three independent cryptographic paths --- BLS12-381 (48-byte aggregate), Pulsar Ring-LWE
 threshold, and ML-DSA-65 (FIPS 204, rolled to a 192-byte Groth16 proof) --- each independently
 toggleable, all three in parallel under \code{TripleSignRound1}. A QuasarCert verifies in
 $\sim$$2$--$5$~ms on CPU, GPU-batchable $10$--$100\times$~\cite{luxconsensus}.
 \end{itemize}
 Quasar's own inter-node transport is already ZAP, not gRPC or libp2p (\code{consensus/LLM.md}).
 Control and transport are, in the Lux stack, \emph{already} composed; what is missing is binding
 that composition to the \emph{GPU} population instead of the validator population.
 \section{The three-plane architecture}
 \label{sec:planes}
 The blueprint is a single sentence: \emph{run the inference ring's topology as Quasar consensus
 state, and move every byte --- control and tensor alike --- over ZAP.} Figure~\ref{fig:planes}
 is the whole system. What follows gives each plane its interface and states, as a proposition,
 why the three do not leak into each other.
 \begin{figure}[t]
 \centering
 \small
 \begin{tabular}{@{}l@{}}
 \toprule
 \textbf{Control plane} --- Quasar consensus (BFT, post-quantum) \hfill \design{} \\
 \quad Photon committee over GPU nodes $\cdot$ luminance = useful-work weight \\
 \quad membership $\cdot$ miner admission $\cdot$ layer placement $\cdot$ config $\cdot$ result attest \\
 \midrule
 \textbf{Transport plane} --- ZAP PQ wire \hfill \built{}\,/\,\partialb{} \\
 \quad \code{WrapPQ} session $\cdot$ verified \code{PeerID} $\cdot$ AEAD frames \\
 \quad one channel for BOTH control messages AND tensor activations \\
 \midrule
 \textbf{Data plane} --- Hanzo inference ring \hfill \built{} \\
 \quad tensor/pipeline parallel $\cdot$ canonical \code{f32} all-reduce $\cdot$ 5 backends \\
 \quad topology handed down by control; bytes carried by transport \\
 \bottomrule
 \end{tabular}
 \caption{Three orthogonal planes. Each is independently complete and separately citable to
 code; the network is their composition, not a new monolith. Labels mark built versus designed
 per \S\ref{sec:status}.}
 \label{fig:planes}
 \end{figure}
 \subsection{Control plane: consensus as the scheduler}
 \label{sec:control}
 The control plane replaces the static \code{RING\_CONFIG} JSON (Listing~\ref{lst:ringcfg}) and
 the out-of-band QR token with \emph{agreed state on a Quasar chain}. The chain's VM state
 \emph{is} the cluster's control surface \design{}:
 \begin{itemize}[leftmargin=1.4em,itemsep=2pt]
 \item \textbf{Membership} --- the set of GPU nodes, each identified by its ZAP \code{PeerID}
 (an ML-DSA public key), with advertised capability (backend, VRAM, resident models, measured
 decode/prefill throughput). Joining and leaving are transactions, so the member list is
 BFT-agreed rather than hand-distributed.
 \item \textbf{Miner admission} --- a permissionless join. A public-cloud GPU submits a join
 transaction carrying its \code{PeerID} and a stake bond; admission is a consensus decision, not
 an operator's SSH session. This is the step that turns ``my four boxes'' into ``anyone's GPU''
 (\S\ref{sec:econ}).
 \item \textbf{Placement / routing} --- which node holds which contiguous layer range of which
 model. Pipeline-parallel assignment (node $A$ holds layers $0$--$7$, $B$ holds $8$--$15$) and
 the ring order become an on-chain placement map. A request is routed by reading that map, not by
 trusting a JSON file.
 \item \textbf{Config distribution} --- model id, quantization, sampling defaults, ring
 \code{world\_size}: the fields of \code{JoinToken} and \code{RingConfig} become chain state, so
 every node reads an identical, agreed configuration.
 \end{itemize}
 The mechanism is not new code so much as a \emph{re-population} of Photon. Photon selects a
 committee by weighted Fisher--Yates over a node set; point it at the \emph{GPU} population and it
 selects a committee of GPU nodes --- to ratify a placement, to admit a miner, or to verify a
 result (\S\ref{sec:threat}). Luminance, today a function of consensus-vote success, is re-bound
 to \emph{validated inference contribution}: a node whose outputs pass verification brightens and
 is preferentially selected and rewarded; a node that stalls or returns wrong tensors dims and is
 sampled out. The primitive is unchanged --- an exponential, bounded, cheap reputation --- only
 the event that drives \code{Illuminate(id, success)} moves from ``voted with the quorum'' to
 ``did correct inference work.'' Crucially, \emph{writes to consensus are rare and
 slow-changing}: placement and membership shift on the order of nodes joining and models loading,
 not on the order of tokens generated. The hot path --- a forward pass through the ring --- never
 touches consensus.
 \subsection{Transport plane: one PQ wire for control and tensors}
 \label{sec:transport}
 Every inter-node byte moves over ZAP. This is deliberately \emph{one} wire for two kinds of
 traffic that a monolith would split across two stacks:
 \begin{itemize}[leftmargin=1.4em,itemsep=2pt]
 \item \textbf{Control messages} --- consensus votes, placement updates, admission requests ---
 which Quasar already speaks over ZAP.
 \item \textbf{Tensor data} --- the activation exchange and all-reduce that the ring performs
 every micro-step --- which today crosses raw TCP and, under this design, crosses the same
 authenticated \code{WrapPQ} \code{net.Conn} instead.
 \end{itemize}
 Unifying them on one wire is the payoff of decomplection: the ring does not need to know it is on
 a network of strangers, only that it holds a \code{net.Conn}; the transport does not need to know
 whether the bytes are a vote or an activation. ZAP's zero-copy framing is what makes carrying
 tensors over an authenticated session affordable --- there is no marshal/unmarshal tax on the
 activation exchange, only an AEAD seal over a buffer the ring already owns (the $64$~KiB
 \code{MaxPQRecord} chunking of \code{conn\_pq.go} maps naturally onto activation-sized writes).
 The identity that the handshake establishes, \code{PeerID}, is the same value the control plane
 keys membership on: the node you complete a ZAP handshake with \emph{is} the node consensus
 admitted, by construction, with no separate PKI to reconcile.
 \subsection{Data plane: unchanged in kind}
 \label{sec:data}
 The data plane is the least-changed and that is the point. The ring's numerics --- canonical
 \code{f32} reduction, tensor/pipeline sharding, the five-backend kernels --- are correct today
 (\S\ref{sec:ring}) and stay exactly as they are. Two wires change underneath them: the topology
 arrives from the control plane's placement map rather than \code{RingConfig::load()}, and the
 \code{TcpStream} is a ZAP \code{net.Conn} rather than a bare socket. The ring code that computes
 does not move. This is composition over rewrite: we are not building a decentralized inference
 engine, we are giving the inference engine we have a control plane and an authenticated
 transport.
 \subsection{Why three planes, stated precisely}
 \label{sec:whythree}
 \begin{proposition}[Orthogonality of the three planes]
 \label{prop:ortho}
 The control, transport, and data planes can each be replaced or scaled independently without
 modifying the other two, because they share only two narrow, typed interfaces: the control plane
 emits a \emph{placement map} keyed by \code{PeerID}, and the transport plane emits an
 authenticated \code{net.Conn} whose \code{PeerID} matches. The data plane consumes both and
 exposes neither consensus nor cryptography; the transport consumes an identity and exposes a
 byte stream, knowing nothing of tensors; the control consumes admission and contribution events
 and exposes agreed state, knowing nothing of wire framing.
 \end{proposition}
 \begin{proof}[Argument]
 By interface inspection. The data plane's dependencies are exactly (i) a topology --- rank,
 world size, neighbor address, layer range --- and (ii) a \code{net.Conn} per neighbor. Neither
 names a consensus type or a cipher. Swap Quasar for any state machine that produces the same
 placement map and the ring is unaffected; swap ZAP for any authenticated \code{net.Conn} with a
 stable peer identity and the ring is unaffected; swap the ring's kernels (add a backend, change
 the quant) and neither consensus nor transport observes it. The only cross-plane invariant is
 that the \code{PeerID} the transport authenticates equals the \code{PeerID} the control plane
 admitted --- a single equality, checked once at connection setup, which is precisely what
 \code{WrapPQ}'s \code{PeerID()} was built to support (``validator-set membership checks,''
 \code{conn\_pq.go}). Three concerns, two interfaces, one shared value. \qed
 \end{proof}
 The contrast with a monolithic serving stack is now sharp. In vLLM or Triton, ``who is in the
 cluster'' is a launch-time flag baked into the process, ``how bytes move'' is a hardcoded NCCL
 or gRPC channel with no peer authentication, and ``what computes'' is the same binary --- the
 three concerns are one artifact, so you cannot add a permissionless member, cannot authenticate
 a stranger's GPU, and cannot verify a result without forking the whole thing. Our planes are
 three artifacts with two seams. That is the difference between a cluster and a network.
 \section{Threat model and result integrity}
 \label{sec:threat}
 A permissionless network inherits adversaries a private cluster never has. We state the model,
 the gaps in today's substrate, and the design that closes them.
 \paragraph{Assets and adversaries.} The assets are: the integrity of an inference result (the
 tokens a user is billed for must be the model's real output), the confidentiality and
 authenticity of activations in flight, the correctness of the placement map, and the fairness of
 reward. The adversaries are: a network attacker on the path between nodes (public-cloud tenants
 share fabric); an \emph{intruder} who obtained a join token; and a \emph{byzantine miner} who
 joined honestly but returns wrong, fabricated, or lazily-cached tensors to earn reward without
 doing the work.
 \paragraph{Gap 1: the wire is plaintext (\built{} problem).} Today the ring is raw TCP
 (\S\ref{sec:ring}). A path attacker can read activations, inject a forged tensor into the
 all-reduce, or man-in-the-middle a neighbor. \textbf{Closure \design{}:} the ZAP PQ handshake.
 After \code{WrapPQ}, every frame is AEAD-sealed and every peer is a verified ML-DSA identity;
 injection and eavesdropping both fail against a post-quantum authenticated channel. The tensor
 plane inherits the same wire security consensus already relies on.
 \paragraph{Gap 2: the join token is a bearer secret (\built{} problem).} The QR token
 (Listing~\ref{lst:token}) is cleartext topology; whoever holds it joins (\S\ref{sec:clustercli}).
 \textbf{Closure \design{}:} make the token a \emph{handshake credential}, not the thing that
 grants access. Under the design the token carries the ring's \code{PeerID} set (or a commitment
 to it); a worker presenting the token must then \emph{complete a ZAP PQ handshake} proving
 possession of an admitted ML-DSA key. A photographed QR no longer joins anyone --- it only names
 who is allowed to, and access is gated on the handshake the intruder cannot forge. The token
 degrades from a secret to a public address book, which is the correct security posture: identity
 is the credential, topology is not.
 \paragraph{Gap 3: a miner may lie about its work (the hard one).} Authentication proves
 \emph{who} sent a tensor, not that the tensor is \emph{correct}. A byzantine miner holding layers
 $8$--$15$ can return plausible-looking garbage and collect reward. This is the genuinely new
 problem, and consensus is the tool. We propose three composable mechanisms, in increasing cost,
 scheduled by the control plane \design{}:
 \begin{enumerate}[leftmargin=1.5em,itemsep=2pt]
 \item \textbf{Threshold attestation of outputs.} A completed inference (or a placement-epoch's
 batch of them) is sealed with a QuasarCert: a Photon committee of GPU nodes threshold-signs the
 $(\text{request}, \text{output})$ pair with BLS$+$Pulsar$+$ML-DSA (\S\ref{sec:quasar}). A billed
 result carries a certificate that a supermajority attested to it, giving the user cryptographic
 non-repudiation and making a single lying miner insufficient to certify a bad output.
 \item \textbf{Redundant re-execution / spot checks.} The scheduler occasionally assigns the same
 layer range to two independent miners and compares --- cheap because the ring wire is already
 \emph{canonical} \code{f32} (\S\ref{sec:ring}), so two honest miners on different vendors produce
 \emph{bit-identical} tensors and any disagreement is a proof of fault, not a rounding artifact.
 The canonical-\code{f32} property, built for cross-vendor correctness, doubles as the substrate
 for verifiable redundancy. A miner caught disagreeing with the honest majority is dimmed by
 luminance and slashed.
 \item \textbf{Reputation as the economic backstop.} Luminance (\S\ref{sec:quasar}) makes
 correct work compound and incorrect work decay: a node that fails verification multiplies its
 brightness by $0.9$ and is sampled into fewer committees and paid for less work, converging its
 influence and income toward zero. Reputation is the slow, cheap layer under the fast, expensive
 cryptographic ones.
 \end{enumerate}
 Byzantine fault tolerance is a property the control plane already has (Quasar tolerates $<1/3$
 adversarial stake); the contribution here is \emph{extending it from block finality to inference
 finality} by threshold-signing over outputs and exploiting canonical \code{f32} for cheap
 re-execution. We are explicit that this is \design{}: the threshold-signing primitive is built,
 its application to inference outputs is proposed.
 \paragraph{Non-goal: confidentiality from the operator.} This design authenticates and verifies;
 it does not, by itself, hide activations from the miner that computes them. Workloads needing
 that require a hardware TEE tier (H100/H200 confidential computing, TDX, SEV-SNP), an orthogonal
 extension developed in the tenant-cloud companion~\cite{tenantcloud} and out of scope here.
 \section{The economic layer: luminance as useful-work weighting}
 \label{sec:econ}
 A permissionless network needs an incentive, and the incentive must be \emph{useful-work},
 because paying for wasted computation (proof-of-work hashing) is exactly what an inference
 network should not do. The luminance primitive is already a useful-work meter in miniature: it
 rewards nodes for correct participation and decays nodes for failure. The economic design
 \design{} binds three quantities to it:
 \begin{itemize}[leftmargin=1.4em,itemsep=2pt]
 \item \textbf{Admission by stake.} A miner joins by bonding stake in a transaction
 (\S\ref{sec:control}). Stake is the sybil cost --- spinning up a thousand fake GPUs is a
 thousand bonds --- and the slashing target when verification (\S\ref{sec:threat}) catches a
 fault.
 \item \textbf{Reward by validated contribution.} Payment for an inference accrues to the miners
 whose outputs were attested, weighted by luminance and by measured work (tokens $\times$ model
 size $\times$ layer share). Because luminance rises with correct work and falls with incorrect
 work, expected income tracks \emph{honest throughput}, and a lazy or lying miner earns less than
 its electricity costs.
 \item \textbf{Routing by brightness.} The Photon committee that ratifies placement and verifies
 results draws with luminance weight, so brighter (more reliably correct) nodes are preferentially
 given work --- reputation and revenue reinforce, which is what makes ``contribute a GPU, get
 paid'' a stable equilibrium rather than a race to the bottom.
 \end{itemize}
 The mechanism reuses one existing primitive (luminance) for reputation, one existing primitive
 (stake-weighted BFT) for sybil-resistance and slashing, and one existing primitive (Photon) for
 work routing. The realized system is ``any dev, any miner, contributes a GPU and is paid for
 inference work,'' with the payment denominated in the network's native token and the fairness
 enforced by the same consensus that finalizes the placement map. This is the point of contact
 with the sovereign-cloud economics of the tenant-cloud companion~\cite{tenantcloud}, which
 develops the wallet, staking, and settlement primitives this layer would sit on.
 \section{Migration path: from LAN appliance to public network}
 \label{sec:migration}
 The design is only credible if there is a continuous path from what runs today to the network,
 with no flag day. There is, in four stages, each shippable and independently valuable.
 \paragraph{Stage 0 (today, \built{}).} Static \code{RING\_CONFIG} JSON, QR join token, raw-TCP
 ring, canonical \code{f32} across vendors. Works on a trusted LAN; this is the baseline.
 \paragraph{Stage 1 --- authenticate the wire (\design{}, smallest step).} Replace the ring's
 \code{TcpStream} with a ZAP \code{WrapPQ} \code{net.Conn} and make the join token gate on a PQ
 handshake (\S\ref{sec:threat}, Gaps 1--2). No consensus yet: the topology is still the static
 map, but the wire is now authenticated and encrypted and the token is a credential. This alone
 converts the appliance into something safe to run across untrusted networks, and it exercises the
 engine's ZAP module end-to-end --- turning the \partialb{} \code{zap://} transport on
 (\S\ref{sec:zap}) is precisely this step.
 \paragraph{Stage 2 --- consensus owns the map (\design{}).} Stand up a Quasar chain whose state
 is the membership and placement map. Nodes read topology from the chain instead of
 \code{RingConfig::load()}; \code{hanzo cluster join} becomes an admission transaction rather than
 a local launch. Placement is still operator-proposed and consensus-ratified (permissioned
 membership, BFT config). The ring code is untouched; only the source of its topology moves from
 file to chain.
 \paragraph{Stage 3 --- permissionless miners and paid work (\design{}).} Open admission to
 staked strangers, bind luminance to validated inference contribution, and turn on result
 attestation and spot-check re-execution (\S\ref{sec:threat}--\S\ref{sec:econ}). This is the full
 network: anyone bonds, joins, is placed by consensus, serves layers over authenticated ZAP, is
 verified by a Photon committee, and is paid by luminance-weighted contribution.
 The staging matters because each stage stands alone. Stage 1 is worth shipping even if Stages
 2--3 never do (it fixes a real security gap). Stage 2 is worth shipping for a single operator's
 multi-datacenter fleet (BFT config, no static files) without any economics. Only Stage 3 needs
 the token and slashing machinery. There is no rewrite between stages --- the same ring, the same
 ZAP, more of the control plane each time.
 \section{Connection to the latency-physics companion}
 \label{sec:latency}
 A pipeline-parallel ring over a wide-area network has a latency budget a single box does not: each
 pipeline stage adds a network hop to the critical path of every token. The companion
 latency-physics analysis~\cite{latency} argues that this budget is spendable precisely because
 two techniques compose over ZAP. First, \textbf{pipeline parallelism hides transport latency
 under compute}: with enough micro-batches in flight, a stage's forward pass overlaps the next
 stage's activation transfer, so the ring's throughput approaches the slowest stage's compute rate
 rather than the sum of hop latencies --- and ZAP's zero-copy framing minimizes the per-hop
 transport term to an AEAD seal, not a marshal. Second, \textbf{speculative decoding amortizes the
 round-trip}: a draft model proposes several tokens that the ring verifies in one pipelined pass,
 so the wide-area round-trip is paid once per \emph{accepted block} of tokens instead of once per
 token. The engine already factors speculative decoding behind one \code{SpeculativeProposer}
 trait~\cite{infstack}; over a consensus-managed ZAP ring, the draft can even run on a nearby,
 brighter node while the target pipeline spans the fleet. The three-plane fabric is what makes
 these composable: the data plane does the pipelining and speculation, the transport plane makes
 each hop cheap and authenticated, and the control plane places draft and target where the latency
 budget is best --- decomplected, so each optimization lives in exactly one plane. We cite the
 companion for the quantitative budget; here it is enough that the architecture does not fight the
 physics.
 \section{Status: built, partial, and designed}
 \label{sec:status}
 Table~\ref{tab:status} is the honest ledger. It is the table to read first: it says exactly which
 sentences of this paper are code on \code{main} and which are blueprint. The pattern is
 deliberate --- every \emph{primitive} is built and cited; every \emph{composition} into the
 network is designed. That is what makes this a buildable blueprint rather than a wish.
 \begin{table}[t]
 \centering
 \small
 \setlength{\tabcolsep}{5pt}
 \begin{tabular}{@{}p{4.9cm}p{1.5cm}p{1.5cm}p{5.0cm}@{}}
 \toprule
 \textbf{Component} & \textbf{Plane} & \textbf{Status} & \textbf{Evidence / source} \\
 \midrule
 Cross-vendor inference ring (TP/PP, canonical \code{f32}) & Data & \built{} &
 \code{distributed.rs}, \code{distributed/socket.rs}, \code{ops.rs}~\cite{infstack} \\
 NCCL $+$ portable ring backends & Data & \built{} & \code{hanzo-quant/src/distributed/} \\
 \code{hanzo cluster} scan-to-join (bs58/QR token) & Data & \built{} &
 \code{hanzo-cli/src/cluster.rs} \\
 Five-backend engine, decode parity, quant zoo & Data & \built{} &
 companion stack paper~\cite{infstack} \\
 ZAP Go wire (zero-copy, $17\times$/$345\times$ vs JSON-RPC) & Transport & \built{} &
 \code{luxfi/zap}~\cite{zap,lp201} \\
 ZAP PQ handshake, \code{WrapPQ}, verified \code{PeerID} & Transport & \built{} &
 \code{conn\_pq.go} \\
 ZAP engine module (in-process adapter surface) & Transport & \partialb{} &
 \code{engine/src/zap/*.rs} (server/client) \\
 \quad out-of-process \code{zap://} (capnp wire) & Transport & \partialb{} &
 \code{engine/src/zap/transport.rs} (stub) \\
 Quasar BFT engine; Photon, Wave, Focus & Control & \built{} &
 \code{luxfi/consensus}~\cite{quasar,luxconsensus} \\
 Luminance reputation (100 base, $\times1.1$/$\times0.9$) & Control & \built{} &
 \code{photon/luminance.go} \\
 BLS$+$Pulsar$+$ML-DSA threshold signing (QuasarCert) & Control & \built{} &
 \code{protocol/quasar}~\cite{luxconsensus} \\
 \midrule
 Ring transport over ZAP (retire raw TCP) & Transport & \design{} & \S\ref{sec:transport}, Stage 1 \\
 Join token as PQ handshake credential & Transport & \design{} & \S\ref{sec:threat}, Gap 2 \\
 Consensus-owned membership \& placement map & Control & \design{} & \S\ref{sec:control}, Stage 2 \\
 Permissionless miner admission (stake bond) & Control & \design{} & \S\ref{sec:econ}, Stage 3 \\
 Luminance $\gets$ validated inference contribution & Control & \design{} & \S\ref{sec:control} \\
 Result verification (threshold attest $+$ spot check) & Control & \design{} & \S\ref{sec:threat}, Gap 3 \\
 Luminance-weighted reward / routing & Control & \design{} & \S\ref{sec:econ} \\
 \bottomrule
 \end{tabular}
 \caption{Built-versus-designed ledger. Every primitive (upper block) is code on \code{main},
 cited to its file with its measured cost where relevant; every network composition (lower block)
 is design, cited to the section that specifies it and the migration stage that ships it.}
 \label{tab:status}
 \end{table}
 \paragraph{Reproducing the built numbers.} The ZAP figures ($2.9$~ns parse, $17\times$/$345\times$
 versus JSON-RPC, $114$K/$376$K TPS) reproduce with \code{go test -bench=. -benchmem} in
 \code{luxfi/zap} and \code{luxfi/consensus/bench/}; the Photon/Wave/luminance costs ($3.03$~\textmu s,
 $3.38$~\textmu s, $72$~ns) reproduce with \code{GOWORK=off go test -bench=. ./bench/} in
 \code{luxfi/consensus}, measured on Apple M1 Max. The cross-vendor ring runs with a
 \code{RING\_CONFIG} JSON per rank and \code{hanzo cluster init}/\code{join}; the cross-backend
 engine numbers are reproduced by the protocol in the companion stack paper~\cite{infstack}. None
 of these numbers is produced by this paper; each is attributed to its source so the blueprint's
 foundation is auditable.
 \section{Limitations}
 \label{sec:limits}
 The rigor is the persuasion, so the costs are as plain as the benefits.
 \begin{itemize}[leftmargin=1.4em,itemsep=2pt]
 \item \textbf{The network is a composition, not a shipped network.} The primitives are built and
 cited; the assembly of \S\ref{sec:planes}--\S\ref{sec:econ} is \design{}. Nothing here should be
 read as ``Hanzo runs a permissionless GPU network today.'' It runs a cross-vendor ring on a
 trusted mesh (Stage 0) and ships every primitive the network needs.
 \item \textbf{The ZAP engine wire is not yet turned on.} The Rust ZAP module is \partialb{}: its
 in-process surface is exercised, but the \code{zap://} out-of-process transport is a documented
 stub and a few methods return ``not yet wired'' (\S\ref{sec:zap}). Stage 1 depends on completing
 it. The Go ZAP wire it mirrors is production-deployed, so the risk is integration, not invention.
 \item \textbf{Wide-area tensor transport is unproven at scale.} The ring was built for a LAN.
 Pipeline parallelism over a wide-area network is argued to be latency-tolerant
 (\S\ref{sec:latency}) but not measured here; the companion latency paper carries that budget, and
 a real deployment must validate it. Bandwidth, not just latency, may bound activation exchange
 for large hidden dimensions.
 \item \textbf{Result verification adds cost.} Threshold attestation and redundant re-execution
 (\S\ref{sec:threat}) are not free: a QuasarCert verifies in $2$--$5$~ms and spot-check redundancy
 duplicates work on sampled requests. The scheme is designed to make the \emph{expected} overhead
 small (attest per epoch, spot-check a fraction) but it is a real tax the monolithic single-trust
 model does not pay --- the price of permissionlessness.
 \item \textbf{Confidentiality from the operator is out of scope.} This design authenticates and
 verifies; it does not hide activations from the computing miner without a TEE tier
 (\S\ref{sec:threat}). Sensitive workloads need that orthogonal extension.
 \item \textbf{Luminance re-binding is a policy change with game-theoretic edges.} Moving the
 \code{Illuminate} trigger from vote-success to inference-contribution is a one-line mechanism
 change but a large incentive-design question: collusion among a committee, grinding on the
 verification lottery, and reward-share disputes across a pipeline are real and not solved by the
 primitive alone. We name them; we do not claim to have priced them.
 \end{itemize}
 \section{Conclusion}
 \label{sec:conclusion}
 The heterogeneous GPU ring already solves the hard, unglamorous problem --- one model computing
 correctly across ROCm, CUDA, and Metal, bit-for-bit, over a canonical wire. What it lacks to
 become a network is not more numerics but two planes it does not have: a control plane to decide
 membership and placement among parties who do not trust each other, and a transport plane to
 authenticate the strangers it talks to. Both already exist as best-in-class Lux primitives ---
 Quasar consensus and the ZAP post-quantum wire --- and the entire contribution of this paper is
 to hold the three concerns apart and compose them: consensus schedules, ZAP carries, the ring
 computes. That decomplection is what a monolith cannot do and what makes ``any developer, any
 miner, contributes a GPU and is paid for inference work'' a buildable system rather than a slogan.
 We have drawn the line between built and designed down the middle of every claim: the ring, the
 scan-to-join CLI, the ZAP wire with its verified \code{PeerID}, the Quasar engine with luminance
 and threshold signing are code on \code{main}, cited and measured; the network that composes them
 --- consensus-owned membership, the ZAP-transported ring, permissionless admission, and result
 verification by threshold-signing over outputs --- is design, staged for a migration with no flag
 day. The primitives are done. The composition is the blueprint. We publish it so the rest of the
 AI-infrastructure community can build on the same three planes.
 \begin{thebibliography}{20}
 \small
 \bibitem{infstack}
 Hanzo AI --- ML Systems. \emph{Hanzo ML and Hanzo Engine: A Unified Multi-Backend, Multi-Modal
 Inference Stack}. Hanzo AI white paper, 2026 (companion; the data-plane engine, cross-vendor ring,
 and five-backend decode parity).
 \bibitem{tenantcloud}
 Hanzo AI Research. \emph{The Unified Sovereign-Tenant Cloud: Linear Shared-Nothing Scaling via
 Per-Tenant SQLite and In-Process Composition}. Hanzo Industries white paper, 2026 (companion; the
 \code{hanzo.network} fabric, wallet/staking, TEE trust tiers, and PQ identity primitives).
 \bibitem{latency}
 Hanzo AI --- ML Systems. \emph{The Latency Physics of Decentralized Decode: Pipeline Parallelism
 and Speculative Decoding over ZAP}. Hanzo AI white paper, 2026 (companion; the wide-area latency
 budget referenced in \S\ref{sec:latency}).
 \bibitem{quasar}
 Lux Industries. \emph{Quasar: Post-Quantum DAG-Native BFT Consensus} (LP-020 / LP-105). Lux
 Network, 2026. \url{https://github.com/luxfi/consensus}.
 \bibitem{luxconsensus}
 Lux Industries. \emph{luxfi/consensus}: the Quasar engine --- \code{protocol/photon} (Fisher--Yates
 committee selection, luminance), \code{protocol/wave} (FPC threshold voting), \code{protocol/focus},
 \code{protocol/quasar} (BLS $+$ Pulsar $+$ ML-DSA threshold signing). Source and benchmarks
 (\code{bench/}, \code{README.md}, \code{LLM.md}). \url{https://github.com/luxfi/consensus}.
 \bibitem{zap}
 Lux Industries. \emph{ZAP: Zero-copy Application Protocol} --- the universal wire.
 \code{github.com/luxfi/zap} (\code{zap.go}, \code{conn\_pq.go} PQ handshake / \code{WrapPQ} /
 \code{PeerID}, \code{README.md} benchmarks).
 \bibitem{lp201}
 Lux Industries. \emph{LP-201: ZAP as the Universal Wire} and \emph{LP-200: The ZAP Stack}. Lux
 Proposals, 2026. \url{https://github.com/luxfi/lps} (the $17\times$/$345\times$ vs JSON-RPC and
 $\sim$$4.5\times$ vs gRPC measurements).
 \bibitem{mldsa}
 National Institute of Standards and Technology. \emph{FIPS 204: Module-Lattice-Based Digital
 Signature Standard (ML-DSA)}. NIST, 2024.
 \bibitem{mlkem}
 National Institute of Standards and Technology. \emph{FIPS 203: Module-Lattice-Based
 Key-Encapsulation Mechanism Standard (ML-KEM)}. NIST, 2024.
 \bibitem{bls}
 D. Boneh, B. Lynn, and H. Shacham. Short Signatures from the Weil Pairing. \emph{Journal of
 Cryptology}, 17(4):297--319, 2004.
 \bibitem{petals}
 A. Borzunov et al. \emph{Petals: Collaborative Inference and Fine-tuning of Large Models}. ACL
 System Demonstrations, 2023 (decentralized inference over volunteer GPUs; no BFT control plane or
 PQ-authenticated transport).
 \bibitem{kwon2023}
 W. Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention.
 \emph{SOSP}, 2023 (the monolithic single-operator serving baseline).
 \bibitem{pinkas}
 B. Pinkas and M. Reiter, and the committee-grinding threat that motivates rejection-sampled
 uniform selection in \code{photon/emitter.go}. (Bias-free sortition under a grinding adversary.)
 \end{thebibliography}
 \end{document}
@@ -0,0 +1,570 @@
 \documentclass[11pt]{article}
 \input{shared/kernel-preamble.tex}
 \usepackage{pgfplots}
 \pgfplotsset{compat=1.16}
 \newcommand{\Tring}{\ensuremath{T_{\mathrm{ring}}}}
 \newcommand{\dmodel}{\ensuremath{d_{\mathrm{model}}}}
 \newcommand{\Gam}{\ensuremath{\Gamma}}
 \title{\textbf{The Latency Physics of Geo-Distributed\\ Autoregressive Decode}}
 \author{Hanzo AI --- ML Systems}
 \date{An analytical systems paper --- companion to the cross-backend inference measurements}
 \begin{document}
 \maketitle
 \begin{abstract}
 Splitting a large language model across machines that sit far apart is often imagined as
 ``BitTorrent for AI'': pool everyone's memory and the model runs. For \emph{loading} weights and
 for \emph{batched throughput} that intuition is right. For \emph{single-stream, interactive decode}
 it is exactly wrong, and this paper derives why. We separate the two costs of a node boundary.
 \textbf{Bandwidth is trivial}: the only tensor that crosses a pipeline boundary is the hidden state
 for the in-flight tokens, \dmodel${}\times n_{\mathrm{tok}}\times b$ bytes --- $\sim$32\,KiB for one
 token of a 700B-class model --- which serializes in $\approx$0.1\,ms over our measured 280\,MB/s
 wired mesh, and the KV cache never crosses at all. \textbf{Latency is the wall}: autoregressive
 decode is a strict dependency chain, so each generated token must traverse every node once, and the
 next token cannot begin until the current one returns. Per-token decode rate is therefore
 $r=1/(C+\Tring)$, where $C$ is the summed compute and $\Tring=\sum_i \ell_i$ is the one-way latency
 of a full loop around the ring. Because \Tring{} adds to a per-token denominator, a geo-distributed
 ring (Hong Kong\,$\to$\,Dublin\,$\to$\,New York, $\Tring\!\approx\!280$\,ms) floors single-stream
 decode at $\approx$3\,tok/s \emph{regardless of GPU speed} --- the wall is the speed of light in
 fiber, not the accelerator. The headline result is the cure: \textbf{speculative decoding converts
 $K$ sequential network traversals into one}. A local draft model proposes $K$ tokens at full LAN
 speed; the distributed model verifies all $K$ in a single pipeline pass; at acceptance rate $\alpha$
 the effective rate is $r_{\mathrm{spec}}=\frac{1-\alpha^{K+1}}{1-\alpha}\big/(Kc_d+C+\Tring)$, which
 recovers 12--21\,tok/s on the same 280\,ms ring --- interactive speeds from an intercontinental
 deployment. We close with four placement laws, including a derivation that pipeline parallelism
 ($O(\text{nodes})$ hops/token) strictly dominates tensor parallelism ($O(\text{layers})$
 all-reduces/token) over any slow link by a factor of $\sim\!4L/N$. Every model here is analytical
 and labelled as such; the one measured number is the 280\,MB/s link.
 \end{abstract}
 \section{Two regimes: dependency-bound vs.\ throughput-bound}
 \label{sec:dichotomy}
 The whole paper turns on one distinction, so we state it first.
 A computation is \textbf{throughput-bound} when its pieces are independent: they can be done in any
 order, in parallel, and the only thing that matters is aggregate bytes-or-flops per second. Network
 \emph{latency} barely touches such a workload --- a slow link means each piece arrives a little
 later, but no piece waits on another, so the pipe stays full and throughput is set by
 \emph{bandwidth}. Downloading a file is the canonical case; so is a batch of a thousand independent
 inference requests.
 A computation is \textbf{dependency-bound} when its pieces form a chain: piece $t+1$ cannot start
 until piece $t$ finishes. Now latency is on the critical path of \emph{every} step and cannot be
 hidden, because there is no independent work to overlap it with. Bandwidth is almost irrelevant ---
 the messages are tiny --- and the wall is the round-trip time.
 Autoregressive decode is the purest dependency chain in production machine learning. Token $t+1$ is
 sampled from the model's output after token $t$ has been fed back as input; \emph{within} a token,
 layer $\ell+1$ consumes the output of layer $\ell$. There is exactly one token in flight for a
 single stream, so there is nothing to overlap. Distributing that chain across nodes does not add
 parallelism to a single stream --- it adds hops to the chain. This is why the ``pool everyone's
 GPUs'' picture, which is a throughput picture, gives the wrong answer for interactive decode, which
 is a dependency problem. The rest of the paper makes both halves quantitative.
 \paragraph{Parameters and provenance.}
 Table~\ref{tab:params} lists the symbols and the reference values used in the worked numbers.
 \emph{One} value is measured --- the 280\,MB/s link goodput, from iperf3 on our direct 2.5\,GbE
 mesh. The representative one-way fiber latencies are physical (Section~\ref{sec:wall}) and are not
 our measurements; the model geometry (\dmodel, $L$) is that of a public 700B-class dense/MoE model;
 the local decode ceiling $r_0$ and draft rate $r_d$ are stated parameters, not claims about a
 specific box. Every rate, table, and plot downstream is a closed-form evaluation of the models in
 this paper, not an experiment.
 \begin{table}[h]
 \centering
 \small
 \caption{Symbols and reference values. Only $W$ is measured.}
 \label{tab:params}
 \begin{tabular}{l l l l}
 \toprule
 Symbol & Meaning & Reference value & Provenance\\
 \midrule
 $N$        & pipeline nodes (stages)                 & 3                 & scenario\\
 $L$        & transformer layers                      & $\sim$100         & 700B-class model\\
 \dmodel    & hidden size                             & 16384             & 700B-class model\\
 $b$        & activation bytes/element (bf16)         & 2                 & format\\
 $W$        & per-link goodput                        & \textbf{280\,MB/s} & \textbf{measured (2.5\,GbE)}\\
 $\ell_i$   & one-way latency of hop $i$              & 40--130\,ms (geo) & physical (fiber)\\
 $\Tring$   & ring latency $\sum_{i=1}^N \ell_i$      & 280\,ms (geo)     & physical (fiber)\\
 $C=1/r_0$  & target compute per decode step          & 33\,ms ($r_0{=}30$) & parameter\\
 $c_d=1/r_d$& local draft time per token              & 5\,ms ($r_d{=}200$) & parameter\\
 $\alpha$   & speculative acceptance rate             & 0.6--0.9          & parameter\\
 $K$        & draft length (tokens/round)             & 1--24             & swept\\
 \bottomrule
 \end{tabular}
 \end{table}
 \section{The crossing tensor: bandwidth is trivial}
 \label{sec:bandwidth}
 In pipeline parallelism (PP) the model is partitioned by \emph{layer}: node $i$ owns a contiguous
 stack of $L/N$ layers. The only data that crosses the boundary between node $i$ and node $i{+}1$ is
 the residual-stream hidden state at that layer cut --- one activation tensor of shape
 $[\,n_{\mathrm{tok}},\ \dmodel\,]$ in the activation dtype. Its size is
 \begin{equation}
 B_{\mathrm{hop}} \;=\; n_{\mathrm{tok}}\cdot \dmodel \cdot b \quad\text{bytes},
 \label{eq:bhop}
 \end{equation}
 and the time to push it across a link of goodput $W$ is $t_{\mathrm{xmit}}=B_{\mathrm{hop}}/W$.
 Three properties of Eq.~\eqref{eq:bhop} decide the paper.
 \paragraph{It is independent of model depth.} $B_{\mathrm{hop}}$ contains no $L$. Only the
 \emph{width} \dmodel{} and the number of in-flight tokens appear, because exactly one activation
 crosses each cut no matter how many layers sit behind it. A deeper model costs more
 \emph{compute} per node but not more \emph{bytes} per hop. This is the structural reason PP is the
 right WAN topology (Section~\ref{sec:pptp}).
 \paragraph{The KV cache never crosses the wire.} Each node holds the keys and values for
 \emph{its own} layers and reuses them locally on every step. The KV cache is the large, context-
 growing state; it stays put. What crosses is only the small, fixed-size residual vector. Distributing
 the model does not distribute the KV traffic --- there is none across the boundary.
 \paragraph{For decode it is a rounding error.} For a single decoded token, $n_{\mathrm{tok}}=1$.
 With the 700B-class width $\dmodel=16384$ and bf16 ($b=2$),
 \[
 B_{\mathrm{hop}} = 1\cdot 16384 \cdot 2 = 32{,}768~\text{bytes} = 32~\text{KiB},
 \qquad
 t_{\mathrm{xmit}} = \frac{32{,}768}{280\times10^6} \approx 0.117~\text{ms}
 \]
 over our \emph{measured} 280\,MB/s mesh. Even at a serving-scale aggregate of 1000\,tok/s each hop
 carries $\sim$32\,MB/s --- about a tenth of a single 2.5\,GbE link, and nothing on a datacenter WAN.
 \textbf{Bandwidth is never the constraint for decode.} (It can matter for large-prompt prefill,
 where $n_{\mathrm{tok}}=P$: a 2000-token prompt crosses 65.5\,MB per hop, $\approx$234\,ms to
 serialize over 280\,MB/s --- but that is a bulk, streamable, latency-insensitive transfer that a
 fatter pipe fixes. It is a throughput cost, not a wall; Section~\ref{sec:regimes}.)
 \section{The wall: latency floors single-stream decode}
 \label{sec:wall}
 Now the cost that does not go away. Generating one token drives the hidden state through all $N$
 nodes in sequence: node~1 embeds and runs its layers, hands off to node~2, and so on to node~$N$,
 which produces the final hidden state; the model head samples the next token, which must return to
 node~1 to begin the next step. Counting hops on the critical path: $N-1$ forward hand-offs plus one
 \emph{return} hop that closes the loop, i.e.\ $N$ inter-node latencies per token. Writing $\ell_i$
 for the one-way latency of hop $i$ and $c_i$ for node $i$'s compute on its layer slice,
 \begin{equation}
 T_{\mathrm{tok}} \;=\; \underbrace{\sum_{i=1}^{N} c_i}_{\textstyle C}
 \;+\; \underbrace{\sum_{i=1}^{N} \ell_i}_{\textstyle \Tring},
 \qquad
 r \;=\; \frac{1}{T_{\mathrm{tok}}} \;=\; \frac{1}{C + \Tring}.
 \label{eq:wall}
 \end{equation}
 Two facts about Eq.~\eqref{eq:wall} carry the argument.
 \paragraph{PP gives a single stream zero compute speedup --- only added latency.} For one in-flight
 token, only one node computes at a time; the other $N-1$ idle (the pipeline is empty). So the total
 compute $C=\sum_i c_i$ equals the monolithic single-node decode compute; splitting the model does
 not make one stream's arithmetic faster. What splitting \emph{adds} is $\Tring$. PP buys memory
 capacity (a 700B model that fits nowhere else) and, with batching, throughput
 (Section~\ref{sec:regimes}) --- never single-stream latency.
 \paragraph{Co-locate the turn so the ring closes in exactly $N$ hops.} Keep the embedding, the first
 layers, the LM head, and the sampler on the same entry node. Then the ``sample $\to$ embed next
 token'' turnaround happens locally, with no network step, and the return is a single hop. Split the
 head from the embedding and you pay an extra hop every token (hidden state to the sampler node, then
 the token id back to the embedder). One node owns both ends of the ring.
 \paragraph{The geo table.} Fix a fast local ceiling $r_0=30$\,tok/s ($C=33$\,ms) and sweep the ring
 latency. Table~\ref{tab:geo} and Figure~\ref{fig:wall} evaluate Eq.~\eqref{eq:wall}.
 \begin{table}[h]
 \centering
 \small
 \caption{Single-stream decode vs.\ ring latency, from $r=1/(C+\Tring)$ with $C=33$\,ms
 ($r_0=30$\,tok/s). Analytical.}
 \label{tab:geo}
 \begin{tabular}{l r r r}
 \toprule
 Deployment (3-node ring) & $\Tring$ (ms) & decode (tok/s) & \% of $r_0$\\
 \midrule
 Single node (no network)          & 0     & 30.0 & 100\%\\
 Rack LAN ($3\times0.1$\,ms)       & 0.3   & 29.7 & 99\%\\
 Same metro ($3\times0.7$\,ms)     & 2     & 28.3 & 94\%\\
 Regional, one continent ($3\times7$\,ms) & 20 & 18.8 & 63\%\\
 Continental US ($3\times20$\,ms)  & 60    & 10.7 & 36\%\\
 Two-continent ($3\times50$\,ms)   & 150   & 5.5  & 18\%\\
 Global ring HK$\to$Dublin$\to$NY  & 280   & \textbf{3.2} & \textbf{11\%}\\
 \bottomrule
 \end{tabular}
 \end{table}
 \begin{figure}[h]
 \centering
 \begin{tikzpicture}
 \begin{axis}[
  width=0.86\linewidth, height=6.2cm,
  xmode=log, log basis x=10,
  xlabel={ring latency $\Tring$ (ms, log scale)},
  ylabel={single-stream decode (tok/s)},
  xmin=0.1, xmax=600, ymin=0, ymax=125,
  grid=both, grid style={gray!18},
  legend pos=north east, legend cell align=left,
  every axis plot/.append style={very thick},
 ]
 \addplot[blue] coordinates {
 (0.1,118.58)(0.2,117.19)(0.3,115.83)(0.5,113.21)(1,107.14)(2,96.77)(5,75.00)(10,54.55)(20,35.29)(40,20.69)(60,14.63)(100,9.23)(150,6.32)(200,4.80)(280,3.47)(400,2.45)(600,1.64)};
 \addlegendentry{fast GPUs, $r_0{=}120$}
 \addplot[red] coordinates {
 (0.1,29.91)(0.2,29.82)(0.3,29.73)(0.5,29.56)(1,29.13)(2,28.30)(5,26.09)(10,23.08)(20,18.75)(40,13.64)(60,10.71)(100,7.50)(150,5.45)(200,4.29)(280,3.19)(400,2.31)(600,1.58)};
 \addlegendentry{$r_0{=}30$}
 \draw[dashed,gray] (axis cs:280,0) -- (axis cs:280,3.47);
 \node[anchor=south,font=\scriptsize,gray] at (axis cs:280,3.9) {global ring};
 \node[anchor=west,font=\scriptsize] at (axis cs:1.2,90) {LAN};
 \node[anchor=west,font=\scriptsize] at (axis cs:22,42) {regional};
 \node[anchor=west,font=\scriptsize] at (axis cs:110,20) {inter-continental};
 \end{axis}
 \end{tikzpicture}
 \caption{The wall. Two accelerator ceilings, $r_0=30$ and $r_0=120$\,tok/s, collapse onto the
 \emph{same} curve as $\Tring$ grows: past $\sim$100\,ms the two are within $\sim$2\,tok/s. Beyond a
 few tens of milliseconds of ring latency, single-stream decode is set by the network, not the GPU.
 Analytical ($r=1/(C+\Tring)$).}
 \label{fig:wall}
 \end{figure}
 \paragraph{The GPU stops mattering.} The most important reading of Figure~\ref{fig:wall} is the
 convergence of the two curves. Hold $\Tring=280$\,ms and vary the local ceiling: $r_0=15\to
 2.9$, $30\to3.2$, $60\to3.4$, $120\to3.5$, $1000\to3.6$\,tok/s. A $67\times$ faster accelerator buys
 a $1.2\times$ faster stream. Once the ring is long, $r\to 1/\Tring$ and the accelerator is a rounding
 error. Buying H100s to serve one interactive user over a global ring is spending on the wrong axis.
 \paragraph{Why it is \emph{physics}.} $\Tring$ has a hard floor: light in fiber travels at
 $c/n\approx2.04\times10^{5}$\,km/s ($n\approx1.47$), i.e.\ $\approx4.9\,\mu$s/km one-way, so a
 real (non-geodesic) route accrues $\gtrsim5$--$7$\,ms per 1000\,km. Hong Kong to New York is
 $\sim$13{,}000\,km great-circle; no switch, protocol, or budget reduces that hop below $\sim$110\,ms
 one-way. The 130/40/110\,ms legs of our example ring are representative real fiber latencies, and
 they are irreducible. This is the sense in which the title is literal: the wall is a property of
 space and light, and the systems question is not how to break it but how to \emph{stop paying it
 per token} --- which is Section~\ref{sec:spec}.
 \section{``BitTorrent for AI'' is half-right}
 \label{sec:bittorrent}
 The appeal of swarm inference --- Petals-style pooling of volunteer GPUs across the internet ---
 rests on an analogy to BitTorrent~\cite{petals}. The analogy is exact where it applies and
 misleading where it does not, and the dividing line is precisely Section~\ref{sec:dichotomy}'s.
 BitTorrent is fast because a file's chunks are \emph{independent}. A peer fetches them in any order,
 from many sources at once; a high-latency peer just means a given chunk lands a little later, never
 that another chunk waits. File download is throughput-bound and latency-insensitive --- the ideal
 case for pooling far-flung capacity. \textbf{Everything about serving a distributed model that is
 throughput-bound inherits this good behaviour}: shipping the weights to the nodes, prefilling a long
 prompt, and running a large batch of independent requests (Section~\ref{sec:regimes}). For these, a
 geographically spread swarm is genuinely attractive.
 Single-stream decode is the opposite object. Its ``chunks'' --- the per-token forward passes --- are
 \emph{maximally dependent}: each is the input to the next. There is no order but the one order, no
 parallel sources, nothing to overlap the latency against. Pooling distant GPUs does not add
 throughput to one stream; it lengthens the dependency chain by the distance between the pools. The
 BitTorrent intuition, applied here, predicts the exact opposite of what happens. So the slogan is
 half-right: \textbf{BitTorrent for AI works for weight distribution, prefill, and batched
 throughput, and breaks for interactive single-stream latency} --- and it breaks hardest precisely
 where users feel it, in the token-by-token cadence of a chat.
 \section{The regime map: where geo-distribution wins}
 \label{sec:regimes}
 The same 280\,ms ring is cheap or catastrophic depending on how many tokens amortize it.
 \paragraph{Prefill / TTFT amortizes the traversal over the whole prompt.} A prompt of $P$ tokens
 goes through the pipeline in \emph{one} pass. The ring latency is paid once, not per token:
 \begin{equation}
 T_{\mathrm{TTFT}} \approx C_{\mathrm{prefill}}(P) + \Tring .
 \label{eq:ttft}
 \end{equation}
 For a 2000-token prompt whose prefill compute is $\sim$1--2\,s on a big model, a $+$280\,ms ring is a
 15--30\% tax on time-to-first-token --- noticeable, not fatal. Per prompt-token the traversal is
 $280\,\text{ms}/2000\approx0.14$\,ms: amortized to nothing. Prefill is throughput-bound (Section~
 \ref{sec:bittorrent}); a geo ring is fine for it.
 \paragraph{Batched throughput fills the pipe.} With continuous batching, $N$ pipeline stages run on
 $N$ different requests at once. Each \emph{individual} stream still crawls at
 Eq.~\eqref{eq:wall}'s $r=1/(C+\Tring)$, but the \emph{aggregate} scales with pipeline depth toward
 $\approx N\!\cdot\!(\text{per-stage rate})$ until a stage saturates. This is the BitTorrent regime
 done right: many independent streams are the independent chunks that keep the pipe full. High
 aggregate tok/s, high per-stream latency --- excellent for offline/batch, useless as a proxy for one
 user's experience.
 \begin{table}[h]
 \centering
 \small
 \caption{The regime map. Geo-distribution is bandwidth-cheap everywhere; it is latency-cheap only
 where many tokens share one traversal.}
 \label{tab:regime}
 \begin{tabular}{l l l l}
 \toprule
 Workload & Bound by & Geo ring 280\,ms? & Why\\
 \midrule
 Weight distribution      & throughput & \textbf{great}     & independent chunks (BitTorrent)\\
 Prefill / TTFT           & throughput & \textbf{fine}      & one traversal per prompt (Eq.~\ref{eq:ttft})\\
 Batched serving (aggregate) & throughput & \textbf{great}  & depth fills the pipe, $\sim\!N\times$\\
 \textbf{Single-stream decode} & \textbf{dependency} & \textbf{hard} & one traversal \emph{per token} (Eq.~\ref{eq:wall})\\
 \bottomrule
 \end{tabular}
 \end{table}
 The map (Table~\ref{tab:regime}) is the practical statement of the thesis: only the bottom row is
 hard, and it is hard because it is the only dependency-bound row. The next section attacks exactly
 that row.
 \section{The cure: speculative decoding amortizes the traversal}
 \label{sec:spec}
 The wall is that decode pays $\Tring$ \emph{per token}. Speculative decoding~\cite{leviathan,chen}
 changes the denominator: it makes the network pay $\Tring$ \emph{per round}, where a round produces
 several tokens. It is the one lever that attacks the dependency structure itself rather than the
 constants.
 \paragraph{Mechanism.} A small \emph{draft} model runs \emph{locally} on the entry node --- no
 network in its loop --- and autoregressively proposes $K$ candidate tokens. Those $K$ tokens are
 then fed as a length-$K$ sequence to the large distributed \emph{target} model, which verifies them
 in a \emph{single} pipeline pass, prefill-style: the $K$ candidates flow through the ring together,
 one traversal, not $K$. The target accepts the longest correct prefix under the standard
 speculative-sampling rule (which preserves the target's exact output distribution) and corrects the
 first mismatch. The expensive thing --- the trip around the ring --- happens once per round instead
 of once per token.
 \paragraph{Expected tokens per round.} Let each draft token be accepted independently with
 probability $\alpha$ (the empirical acceptance rate). The number of accepted drafts $A$ satisfies
 $\Pr[A\ge j]=\alpha^{j}$ for $j\le K$, so
 $\mathbb{E}[A]=\sum_{j=1}^{K}\alpha^{j}=\alpha\frac{1-\alpha^{K}}{1-\alpha}$. The target always
 contributes one further token (the correction on the first reject, or a bonus token if all $K$ are
 accepted), giving the closed form of Leviathan et al.~\cite{leviathan}:
 \begin{equation}
 \Gam(\alpha,K)\;=\;1+\mathbb{E}[A]\;=\;\frac{1-\alpha^{K+1}}{1-\alpha}
 \qquad\text{expected tokens per round.}
 \label{eq:gamma}
 \end{equation}
 \paragraph{Effective rate.} A round costs one local draft of $K$ tokens plus one target
 verification pass (one traversal of the ring):
 \begin{equation}
 T_{\mathrm{round}} \;=\; \underbrace{K\,c_d}_{\text{local draft}}
 \;+\; \underbrace{C_{\mathrm{verify}}(K) + \Tring}_{\text{one target traversal}},
 \qquad
 r_{\mathrm{spec}}(\alpha,K)\;=\;\frac{\Gam(\alpha,K)}{T_{\mathrm{round}}}.
 \label{eq:rspec}
 \end{equation}
 In the memory-bandwidth-bound decode regime, verifying $K\!\le\!16$ tokens streams the weights the
 \emph{same} once as a single decode step, so $C_{\mathrm{verify}}(K)\approx C$ --- the reason
 speculation is nearly free on the compute side.\footnote{For large $K$ the verify pass eventually
 becomes compute-bound (prefill-like) and $C_{\mathrm{verify}}(K)$ grows with $K$; that is one of the
 two forces creating an optimal $K$ below, the other being the geometric decay $\alpha^{K}$ of the
 marginal acceptance. We take $C_{\mathrm{verify}}\!\approx\!C$ over the plotted range $K\le24$, which
 is accurate for a 700B-class model where 24 tokens is still far below the arithmetic-intensity
 knee.} Substituting the closed form,
 \begin{equation}
 \boxed{\;
 r_{\mathrm{spec}}(\alpha,K)\;=\;\frac{1}{\,K\,c_d + C + \Tring\,}\cdot\frac{1-\alpha^{K+1}}{1-\alpha}
 \;}
 \label{eq:boxed}
 \end{equation}
 The structure is the whole point: $\Tring$ sits in the denominator once, divided into by
 $\Gam(\alpha,K)$ tokens. The effective per-token network cost falls from $\Tring$ to
 $\Tring/\Gam(\alpha,K)$.
 \paragraph{Numbers on the global ring.} Take the intercontinental $\Tring=280$\,ms, target
 $C=33$\,ms, and a fast local draft $c_d=5$\,ms ($r_d=200$\,tok/s). Baseline single-stream decode is
 $1/(0.033+0.280)=3.2$\,tok/s. Table~\ref{tab:spec} and Figure~\ref{fig:spec} evaluate
 Eq.~\eqref{eq:boxed}.
 \begin{table}[h]
 \centering
 \small
 \caption{Speculative decode on the 280\,ms global ring, from Eq.~\eqref{eq:boxed}
 ($C{=}33$\,ms, $c_d{=}5$\,ms). $\Gam$ = expected tokens/round; $r$ = effective tok/s.
 Baseline (no speculation) $=3.2$\,tok/s. Analytical.}
 \label{tab:spec}
 \begin{tabular}{c cc cc cc c}
 \toprule
 & \multicolumn{2}{c}{$K=4$} & \multicolumn{2}{c}{$K=8$} & \multicolumn{2}{c}{$K=16$} & best\\
 \cmidrule(lr){2-3}\cmidrule(lr){4-5}\cmidrule(lr){6-7}
 $\alpha$ & $\Gam$ & $r$ & $\Gam$ & $r$ & $\Gam$ & $r$ & tok/s ($K^{\star}$)\\
 \midrule
 0.6 & 2.31 & 6.9 & 2.47 & 7.0 & 2.50 & 6.4 & 7.1 ($K{=}6$)\\
 0.7 & 2.77 & 8.3 & 3.20 & 9.1 & 3.33 & 8.5 & 9.1 ($K{=}8$)\\
 0.8 & 3.36 & 10.1 & 4.33 & 12.3 & 4.89 & 12.4 & 12.7 ($K{=}12$)\\
 0.9 & 4.10 & 12.3 & 6.13 & 17.3 & 8.33 & 21.2 & 21.6 ($K{=}21$)\\
 \bottomrule
 \end{tabular}
 \end{table}
 \begin{figure}[h]
 \centering
 \begin{tikzpicture}
 \begin{axis}[
  width=0.86\linewidth, height=6.4cm,
  xlabel={draft length $K$ (tokens/round)},
  ylabel={effective decode (tok/s)},
  xmin=1, xmax=24, ymin=0, ymax=24,
  grid=both, grid style={gray!18},
  legend pos=north west, legend cell align=left,
  every axis plot/.append style={very thick},
 ]
 \addplot[black!70,dashed,thick] coordinates {(1,3.19)(24,3.19)};
 \addlegendentry{baseline (no spec.), 3.2}
 \addplot[red] coordinates {(1,5.97)(2,8.38)(3,10.47)(4,12.29)(5,13.85)(6,15.20)(7,16.35)(8,17.34)(9,18.18)(10,18.89)(11,19.48)(12,19.98)(13,20.38)(14,20.72)(15,20.98)(16,21.18)(17,21.34)(18,21.44)(19,21.51)(20,21.55)(21,21.55)(22,21.53)(23,21.48)(24,21.42)};
 \addlegendentry{$\alpha=0.9$}
 \addplot[orange] coordinates {(1,5.65)(2,7.55)(3,8.99)(4,10.08)(5,10.90)(6,11.51)(7,11.95)(8,12.25)(9,12.46)(10,12.58)(11,12.64)(12,12.66)(13,12.63)(14,12.58)(15,12.51)(16,12.43)(17,12.33)(18,12.22)(19,12.10)(20,11.99)(21,11.86)(22,11.74)(23,11.62)(24,11.49)};
 \addlegendentry{$\alpha=0.8$}
 \addplot[teal] coordinates {(1,5.34)(2,6.77)(3,7.71)(4,8.32)(5,8.69)(6,8.91)(7,9.02)(8,9.05)(9,9.04)(10,8.99)(11,8.92)(12,8.84)(13,8.75)(14,8.65)(15,8.56)(16,8.45)(17,8.35)(18,8.26)(19,8.16)(20,8.06)(21,7.97)(22,7.87)(23,7.78)(24,7.69)};
 \addlegendentry{$\alpha=0.7$}
 \addplot[blue!55] coordinates {(1,5.03)(2,6.06)(3,6.63)(4,6.92)(5,7.04)(6,7.08)(7,7.06)(8,7.00)(9,6.93)(10,6.86)(11,6.77)(12,6.69)(13,6.60)(14,6.52)(15,6.44)(16,6.35)(17,6.28)(18,6.20)(19,6.12)(20,6.05)(21,5.98)(22,5.91)(23,5.84)(24,5.77)};
 \addlegendentry{$\alpha=0.6$}
 \end{axis}
 \end{tikzpicture}
 \caption{The cure. Effective single-stream decode on the \emph{same} 280\,ms global ring, from
 Eq.~\eqref{eq:boxed}. Speculation lifts the 3.2\,tok/s baseline (dashed) to 7--21\,tok/s. Each curve
 has an optimum $K$: too small wastes the traversal, too large is drowned by the geometric decay
 $\alpha^K$ and the growing draft cost $Kc_d$. Higher acceptance $\alpha$ both raises the ceiling and
 pushes $K^{\star}$ out. Analytical.}
 \label{fig:spec}
 \end{figure}
 \paragraph{Reading the result.} On a ring that floors naive decode at 3.2\,tok/s, a draft with
 acceptance $\alpha=0.8$ recovers $\sim$12.7\,tok/s ($4.0\times$) and $\alpha=0.9$ recovers
 $\sim$21.6\,tok/s ($6.8\times$) --- from a bad slideshow to a readable stream, on the identical
 intercontinental deployment, with no faster hardware and no shorter cable. The mechanism is not a
 constant-factor tuning; it is a change in what the network is charged for. Naive decode pays one
 traversal per emitted token; speculation pays one traversal per $\Gam(\alpha,K)$ emitted tokens, and
 $\Gam$ is exactly the knob acceptance gives you. This is why speculative decoding is \emph{the}
 lever for interactive geo-distributed serving, not one option among several: it is the only cure
 that divides the wall instead of chipping at it.
 \paragraph{The optimum is real and worth hitting.} Each curve in Figure~\ref{fig:spec} peaks. Below
 $K^{\star}$ the fixed $\Tring$ is shared over too few tokens; above it, marginal drafts are accepted
 with probability $\alpha^{K}\!\to\!0$ while still costing draft time $c_d$ and, eventually, verify
 compute. The peak drifts right with $\alpha$ ($K^{\star}\!:\,6\to8\to12\to21$ for
 $\alpha=0.6\to0.9$) because a better draft keeps paying off for longer. A deployment should tune $K$
 to its measured acceptance and ring, not to a fixed default --- the difference between $K{=}4$ and
 $K^{\star}$ is $\sim$1.7$\times$ at $\alpha=0.9$.
 \section{Placement laws}
 \label{sec:placement}
 Four heuristics fall directly out of the models above; the first two are strategy, the last two are
 free.
 \begin{enumerate}[leftmargin=1.5em,itemsep=3pt]
 \item \textbf{Regional-first placement.} $\Tring$ dominates $r$ (Eq.~\ref{eq:wall}), so pack the
 whole pipeline into one region before spanning regions. A same-metro ring ($\Tring\!\approx\!2$\,ms)
 is 94\% of local; a global ring is 11\%. Span a WAN only when memory capacity forces it, and when you
 do, expect to pay for speculation (Section~\ref{sec:spec}) to stay interactive. LAN neighbours are
 near-native; distant neighbours are a different regime.
 \item \textbf{Speculate across every slow boundary.} Any ring above a few tens of milliseconds
 should run a local draft. It is the only lever that touches the $\Tring$-per-token term, and it
 composes with placement: regional-first shortens $\Tring$, speculation amortizes whatever remains.
 \item \textbf{Co-locate embed $+$ first layers $+$ head $+$ sampler on the entry node} (free). Makes
 the sample$\to$embed turnaround local and closes the ring in exactly $N$ hops instead of $N{+}1$
 (Section~\ref{sec:wall}). One node owns both ends.
 \item \textbf{Keep the KV cache local to each node's layers} (free, and automatic in PP). The KV
 cache never crosses the wire; only the 32\,KiB hidden state does (Section~\ref{sec:bandwidth}). Do
 nothing that would ship KV state between nodes per step --- it is the large state, and it has no
 reason to move.
 \end{enumerate}
 \section{Why pipeline-, not tensor-, parallel over slow links}
 \label{sec:pptp}
 The choice of \emph{how} to split the model is not free, and over a WAN it is decisive. The two
 options generate fundamentally different network traffic per token.
 \textbf{Pipeline parallel (PP)} splits by layer. Per token, the hidden state crosses each node
 boundary once: $N$ point-to-point hops on the critical path, each moving the 32\,KiB of
 Eq.~\eqref{eq:bhop}. Critical-path network latency per token $=\Tring=\sum_{i=1}^N\ell_i$.
 \textbf{Tensor parallel (TP)} splits each layer's matrices across nodes~\cite{megatron}. Every layer
 must \emph{all-reduce} partial results across all $N$ nodes to reconstruct the activation --- roughly
 two collectives per layer (one after attention, one after the MLP). That is $\sim\!2L$ global
 synchronizations per token, each of which, even counted optimistically as a single round-trip
 ($\ge\!2\bar\ell$; a ring all-reduce is worse, $2(N{-}1)$ serial steps), puts a barrier on the
 critical path. Critical-path network latency per token $\gtrsim 2L\cdot2\bar\ell=4L\bar\ell$.
 \paragraph{The traffic law.} Comparing the two, with $\bar\ell$ the mean one-way hop and
 $\Tring\approx N\bar\ell$:
 \begin{equation}
 \frac{(\text{TP latency/token})}{(\text{PP latency/token})}
 \;\gtrsim\;\frac{4L\bar\ell}{N\bar\ell}\;=\;\frac{4L}{N}.
 \label{eq:pptp}
 \end{equation}
 \textbf{PP costs $O(\text{nodes})$ hops per token; TP costs $O(\text{layers})$ collectives per
 token.} Since a model always has far more layers than nodes ($L\gg N$) and each TP collective is a
 global barrier rather than a single hand-off, PP wins by a wide, structural margin: for $L=100$,
 $N=3$, Eq.~\eqref{eq:pptp} gives $\gtrsim\!130\times$ less critical-path latency, and it is a lower
 bound (a realistic ring all-reduce adds another $\sim(N{-}1)$ factor). TP also moves more bytes
 ($\sim\!2\dmodel b$ per node per collective, $\times 2L$), but latency alone settles it. TP belongs
 \emph{inside} a node or an NVLink domain, where $\bar\ell$ is nanoseconds; across any link where
 $\bar\ell$ is milliseconds, TP multiplies the wall by $\sim\!4L/N$ and PP is the only sane topology.
 This is the structural companion to Section~\ref{sec:bandwidth}: PP's crossing tensor is
 depth-independent precisely because it hands off once per node, and that is the same property that
 makes it latency-cheap.
 \section{What this paper claims, and what it does not}
 \label{sec:honesty}
 This is an analysis paper, and the analytical models \emph{are} the contribution --- so the line
 between derived and measured is drawn explicitly.
 \textbf{Measured (one number).} The 280\,MB/s link goodput is real: iperf3 over a direct 2.5\,GbE
 cable on our mesh, sustained TCP ($\approx$2.24\,Gbit/s). It anchors the bandwidth claim in
 Section~\ref{sec:bandwidth} and nothing else.
 \textbf{Physical (not our measurement).} The one-way fiber latencies (40--130\,ms legs, 280\,ms
 ring) are representative of real intercontinental routes and are bounded below by the speed of light
 in fiber ($\approx$4.9\,$\mu$s/km). We cite the floor, not a traceroute we ran.
 \textbf{Parameters (stated, not claimed).} $r_0=30$\,tok/s, $r_d=200$\,tok/s, $\alpha\in[0.6,0.9]$,
 $\dmodel=16384$, $L\approx100$. These are reference values chosen to be representative of a
 700B-class model with a small local draft; every table and plot is a closed-form evaluation at these
 values, and the \emph{shape} of every result (the $1/(C+\Tring)$ wall, the $\Tring/\Gam$
 amortization, the $4L/N$ law) is independent of the specific numbers.
 \textbf{Not claimed.} We report no end-to-end wall-clock from a running geo-distributed deployment;
 no measured acceptance rates for a specific draft/target pair; no serving-throughput sweep; no
 overlap/async-speculation optimizations (which only improve the cure). Real acceptance depends on
 the draft, the target, and the prompt distribution, and real rings have jitter and packet loss the
 constant-$\ell$ model omits. The models predict \emph{regimes and ratios}, and those are what the
 paper stands behind.
 \section{Conclusion}
 \label{sec:conclusion}
 Distributing a large language model across machines is bandwidth-cheap and latency-expensive, and
 the two costs live in different regimes. Bandwidth is trivial because the only tensor crossing a
 pipeline boundary is a $\sim$32\,KiB hidden state --- depth-independent, KV-cache-free,
 $\sim$0.1\,ms on a commodity link. Latency is a wall because autoregressive decode is a strict
 dependency chain: every token traverses every node, the next cannot start until this one returns,
 and so single-stream decode is floored at $1/(C+\Tring)$ --- $\approx$3\,tok/s on a global ring,
 \emph{independent of the accelerator}, because the wall is the speed of light in fiber. The
 ``BitTorrent for AI'' picture is right for the throughput-bound work (weights, prefill, batch) and
 wrong for the one dependency-bound workload (interactive decode) --- and that is the only workload
 that is hard. The cure is to change what the network is charged for: speculative decoding turns $K$
 sequential traversals into one, dividing the wall by the accepted-tokens-per-round $\Gam(\alpha,K)$
 and recovering 12--21\,tok/s on the identical 280\,ms ring. Combined with regional-first placement, a
 co-located ring turn, node-local KV, and pipeline- (never tensor-) parallelism over the slow links,
 geo-distributed inference is not a contradiction --- it is a latency-engineering problem with a known
 solution. The physics sets the floor; speculation decides how often you pay it.
 \begin{thebibliography}{9}
 \small
 \bibitem{leviathan} Y.~Leviathan, M.~Kalman, Y.~Matias.
 \emph{Fast Inference from Transformers via Speculative Decoding.} ICML 2023.
 \bibitem{chen} C.~Chen, S.~Borgeaud, G.~Irving, J.-B.~Lespiau, L.~Sifre, J.~Jumper.
 \emph{Accelerating Large Language Model Decoding with Speculative Sampling.} arXiv:2302.01318, 2023.
 \bibitem{petals} A.~Borzunov, D.~Baranchuk, T.~Dettmers, M.~Ryabinin, et al.
 \emph{Petals: Collaborative Inference and Fine-tuning of Large Models.} ACL 2023 (System Demonstrations).
 \bibitem{gpipe} Y.~Huang, Y.~Cheng, A.~Bapna, et al.
 \emph{GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism.} NeurIPS 2019.
 \bibitem{megatron} M.~Shoeybi, M.~Patwary, R.~Puri, P.~LeGresley, J.~Casper, B.~Catanzaro.
 \emph{Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.}
 arXiv:1909.08053, 2019.
 \bibitem{pipedream} D.~Narayanan, A.~Harlap, A.~Phanishayee, et al.
 \emph{PipeDream: Generalized Pipeline Parallelism for DNN Training.} SOSP 2019.
 \bibitem{xbackend} Hanzo AI. \emph{Edge LLM Inference Across Commodity Accelerators: hanzo-engine at
 llama.cpp Decode Parity on Every Backend.} hanzo-engine white paper (companion, measured).
 \end{thebibliography}
 \end{document}
@@ -0,0 +1,414 @@
 \documentclass[11pt]{article}
 \input{shared/kernel-preamble.tex}
 \newcommand{\Tring}{\ensuremath{T_{\mathrm{ring}}}}
 \title{\textbf{Open Problems in Heterogeneous\\ Distributed Inference}}
 \author{Hanzo AI --- ML Systems}
 \date{Research-agenda companion to the distributed-inference series}
 \begin{document}
 \maketitle
 \begin{abstract}
 Three companion papers report a working heterogeneous distributed-inference stack. One engine shards a
 model tensor-parallel across an AMD/NVIDIA boundary over commodity ethernet and decodes
 coherently~\cite{het}; an analytical model derives the latency physics of geo-distributed autoregressive
 decode and the speculative-decoding cure~\cite{latency}; and a three-plane blueprint turns the
 trusted-LAN ring into a permissionless, consensus-managed, post-quantum-authenticated
 network~\cite{consensus}. Building them surfaced eight problems a homogeneous, single-trust,
 datacenter-local stack never has to confront, because each is an assumption that only \emph{heterogeneity},
 \emph{wide-area distribution}, or \emph{permissionlessness} puts under load. This paper states the eight
 precisely --- with the formal object where one exists --- and ties each to the companion that owns the
 relevant result, so the boundary between what is measured, what is designed, and what is open stays sharp.
 The eight are: (1) sampling-determinism boundaries in replicated-state inference; (2) quant-aware
 tensor-parallel sharding on block-quantized weights; (3) pipeline parallelism as the format-agnostic
 primitive and its layer-assignment ILP; (4) speculative decoding as a latency amortizer, made adaptive
 under round-trip time and acceptance drift; (5) the prefill launch-overhead wall, attacked by fixed-shape
 graph capture versus compiler-level fusion; (6) KV-cache wire compression for prefill migration and its
 cross-hop error accumulation; (7) the geo-distributed decode latency floor and the placement/replication
 optimization that follows from it; and (8) cheap verifiable inference for a permissionless compute
 network. None is a tuning parameter; each is a research question, and several compose --- the layer
 schedule of (3) and the geo-placement of (7) are one joint optimization, and the determinism boundary of
 (1) and the cheap verification of (8) share the canonical-\code{f32} substrate. We publish the agenda so
 the problems can be attacked on the same terms.
 \end{abstract}
 \section{Introduction: where the assumptions break}
 \label{sec:intro}
 A model-serving stack built for one datacenter, one vendor, and one operator gets to assume a great deal:
 every rank runs the same silicon and the same numeric format, the network between ranks is a nanosecond-
 scale NVLink fabric, and every node is trusted to compute what it claims. The distributed-inference series
 removes those assumptions one at a time --- the fleet becomes vendor-mixed~\cite{het}, the links become
 intercontinental~\cite{latency}, and the nodes become strangers who must be paid and
 verified~\cite{consensus} --- and each removal exposes a problem the assumption had been hiding.
 This paper is the ledger of what those removals expose. It is deliberately \emph{not} a results paper: it
 states problems, not solutions, and its contribution is precision --- each problem is pinned to the
 companion that owns the surrounding result, given its formal shape where it has one, and separated from
 the parts already solved so that no claim is duplicated across the series. We adopt the series' discipline
 of labelling measured, designed, and open, and here everything is open by construction; the job is to say
 \emph{exactly} what is open and why the environment forces it.
 The eight problems fall into four families. \textbf{Correctness under replication} (\S\ref{sec:determinism}):
 when several ranks must agree on a sampled token, where must their arithmetic be bit-identical?
 \textbf{The right primitive and its schedule} (\S\ref{sec:quantshard}--\S\ref{sec:pp}): tensor parallelism
 is correct but coupled to dequantized weights and to a per-layer collective; pipeline parallelism is
 format-agnostic but demands a hard scheduling decision. \textbf{Paying the wire} (\S\ref{sec:spec}--\S\ref{sec:geofloor}):
 the latency of a wide-area hop is on the critical path of every token, and the levers are speculation,
 compression, and placement. \textbf{Trust without an operator} (\S\ref{sec:verify}): a permissionless node
 can lie about its work, and full re-execution is too expensive to be the answer. We take them in that
 order.
 \section{Problem 1: determinism boundaries in replicated-state inference}
 \label{sec:determinism}
 \paragraph{Statement.} In any inference topology where more than one rank holds replicated state that must
 stay coherent --- most sharply, the tensor-parallel ring of~\cite{het}, where each rank keeps its own KV
 cache and advances it token by token --- every rank must sample the \emph{same} next token, or the caches
 diverge and the streams silently fork. The cross-vendor ring makes the \emph{reduced activations} agree
 bit-for-bit by exchanging them in a canonical \code{f32} wire and reducing with a canonical
 tie-break~\cite{het}. But the final logits are computed \emph{after} the last reduction, locally, in each
 rank's own 16-bit compute dtype (\code{f16} on the ROCm rank, \code{bf16} on the CUDA rank). Sampling is
 therefore \emph{outside} the agreement the wire guarantees: at a near-argmax tie, \code{f16} and
 \code{bf16} rounding can select different tokens, and the ranks desync.
 \paragraph{The formal object.} Call the \emph{agreement set} $\mathcal{A}$ the minimal set of tensors that
 must be bit-identical across replicas for state to stay coherent. The canonical-\code{f32} reduce places
 the post-attention and post-MLP activations in $\mathcal{A}$. Coherent decode additionally requires the
 \emph{sampled token id} $\in\mathcal{A}$; the open question is the cheapest way to put it there. Two
 mechanisms bound the design space:
 \begin{itemize}[leftmargin=1.4em,itemsep=2pt]
 \item \textbf{Broadcast the decision.} One head rank samples and broadcasts the token id; the others do not
 sample at all. Cost: one small ($\le 8$-byte) broadcast per token on the critical path. Correct by
 construction for \emph{any} sampler (greedy, temperature, top-$p$, min-$p$), because there is exactly one
 draw.
 \item \textbf{Canonicalize the computation.} Every rank computes logits in canonical \code{f32} with a
 deterministic tie-break and a shared RNG stream, so all ranks sample identically without communicating.
 Cost: an \code{f32} logit head on every rank and a synchronized RNG; fragile the moment any op on the path
 is not bit-reproducible across vendors.
 \end{itemize}
 The trade is broadcast latency versus redundant canonical compute, and it interacts with the sampler:
 stochastic sampling needs a \emph{shared seed and a shared draw}, which effectively forces the broadcast
 mechanism, so the ``canonicalize'' route is really only clean for greedy decode.
 \paragraph{What is open.} A general theory of determinism boundaries for replicated-state inference: which
 ops must be canonical, the cost of canonicalization versus decision-broadcast as a function of vocabulary
 size and ring width, and the extension to speculative decoding (\S\ref{sec:spec}), where the boundary must
 hold not for one token but for a verified block of $K$. This is the correctness dual of the verification
 problem of \S\ref{sec:verify}: the same canonical-\code{f32} property that makes two honest ranks agree is
 what lets a third party \emph{check} that they did.
 \section{Problem 2: quant-aware tensor-parallel sharding}
 \label{sec:quantshard}
 \paragraph{Statement.} Tensor parallelism shards each weight matrix along rows (column-parallel) or columns
 (row-parallel), giving each rank a contiguous slice of the matrix~\cite{megatron}. The block-quantized
 formats that make edge inference fit in commodity memory --- GGUF $k$-quants and their kin --- store
 weights in \emph{superblocks}: a run of (typically) 256 weights sharing a block scale, with sub-block
 scales and a packed low-bit mantissa. A naive row/column shard cuts \emph{through} a superblock, so a
 rank's slice no longer contains the shared scale its weights are quantized against. The only way to shard
 such a layout with an arbitrary cut is to dequantize the whole matrix, slice, and requantize per shard ---
 which reintroduces the dequant--requant loss the format exists to avoid, and defeats the memory saving by
 materializing the dense matrix. This is why the shipping TP path in~\cite{het} is wired only through the
 dequantized (safetensors) model definitions and cannot shard a GGUF model at all.
 \paragraph{What is open.} \emph{Quant-aware sharding}: a split that respects superblock boundaries. If the
 shard count divides the matrix dimension into superblock-aligned pieces (or the matrix is padded to make
 it so), each rank receives whole superblocks with their scales intact and no dequantization is needed. The
 open questions are the constraints and their cost: which (hidden size, head count, shard count, block size)
 combinations admit an aligned split; how much padding waste an arbitrary combination incurs; whether a
 format with per-shard scale replication can be sharded losslessly at any cut; and whether the analysis
 differs for row- versus column-parallel layers, which cut different axes of the superblock grid. The prize
 is TP over quantized weights --- but note that pipeline parallelism (\S\ref{sec:pp}) sidesteps the entire
 problem, because it ships \emph{activations}, not weight slices, across the seam and each node loads its
 layer range in whatever quantization it likes. Problem~2 is worth solving only where TP is worth keeping:
 inside a node or an NVLink domain, for a matrix too wide to hold on one accelerator.
 \section{Problem 3: pipeline parallelism as the universal primitive, and its layer-assignment ILP}
 \label{sec:pp}
 \paragraph{The engineering result (in progress).} The heterogeneous-cluster analysis of~\cite{het} argues,
 and the latency analysis of~\cite{latency} proves under a wide-area cost model, that \emph{pipeline
 parallelism} (PP) is the right primitive for a heterogeneous fleet: it is format-agnostic (\S\ref{sec:quantshard}),
 carries only an activation tensor across each node boundary rather than an all-reduce per layer, imposes no
 head-count-divides-node-count constraint, and reduces the sampling determinism boundary
 (\S\ref{sec:determinism}) to a single head sampler. Implementing PP as the default multi-node primitive in
 the same engine is an engineering task under way in the codebase; it is not the open problem. The open
 problem is what PP forces you to decide that TP does not.
 \paragraph{The open problem: optimal layer-range assignment.} PP must cut the $L$-layer model into $N$
 contiguous ranges and assign each to a node, under nodes that differ in \emph{compute}, \emph{memory}, and
 \emph{link bandwidth} simultaneously. Let node $j$ have compute throughput $\rho_j$ (layers/s-equivalent)
 and memory capacity $M_j$; let layer $l$ have compute weight $w_l$ and resident footprint $m_l$ (weights
 plus its KV cache). A pipeline is a permutation $\pi$ of nodes onto stages and a contiguous partition
 $0=c_0<c_1<\dots<c_N=L$ assigning layers $[c_{j-1},c_j)$ to stage $j$. Steady-state pipelined throughput is
 set by the slowest stage, so the assignment problem is
 \begin{equation}
 \min_{\pi,\;c}\;\max_{1\le j\le N}\;
 \max\!\left(\underbrace{\frac{1}{\rho_{\pi(j)}}\sum_{l=c_{j-1}}^{c_j-1} w_l}_{\text{stage compute}},\;
 \underbrace{\frac{B_{\mathrm{hop}}}{b_{\pi(j),\pi(j+1)}}}_{\text{activation transmit}}\right)
 \quad\text{s.t.}\quad
 \sum_{l=c_{j-1}}^{c_j-1} m_l \le M_{\pi(j)}\ \ \forall j,
 \label{eq:ilp}
 \end{equation}
 where $B_{\mathrm{hop}}$ is the per-hop activation size and $b_{\cdot,\cdot}$ the link bandwidth between
 consecutive stages. For a \emph{fixed} node ordering $\pi$ the inner problem --- partition a weighted
 sequence into $N$ contiguous parts minimizing the bottleneck part under per-part capacity --- is solvable
 exactly by dynamic programming or parametric search. The difficulty is the \emph{joint} choice of ordering
 $\pi$ (which node is adjacent to which, hence which link carries each hop) together with the partition and,
 in the geo setting (\S\ref{sec:geofloor}), the placement of nodes in space: that couples an assignment, a
 sequencing, and a facility-location decision, and is NP-hard in general.
 \paragraph{What is open.} A practical solver: good heuristics with a bounded approximation ratio for
 Eq.~\eqref{eq:ilp}; online \emph{re-balancing} as nodes join, leave, or thermally throttle (the membership
 of a permissionless network~\cite{consensus} is not static); and the memory-vs-compute tension when the
 throughput-optimal cut violates a small node's capacity. The single-stream \emph{latency} objective ---
 $\sum_j$ stage compute $+\,\Tring$ rather than the bottleneck --- is the variant~\cite{latency} analyses;
 Eq.~\eqref{eq:ilp} is its throughput dual, and a real scheduler must weigh the two. Prior PP scheduling
 work assumes homogeneous stages~\cite{gpipe} or homogeneous datacenter nodes~\cite{pipedream}; the open
 case is genuinely heterogeneous $\rho_j$, $M_j$, and $b_{jk}$ at once.
 \section{Problem 4: speculative decoding as a geo-distributed latency amortizer, made adaptive}
 \label{sec:spec}
 \paragraph{What the companion owns.} The latency paper~\cite{latency} derives the speculative-decoding
 cure in full: a local draft proposes $K$ tokens, the distributed target verifies all $K$ in one pipeline
 pass, and at acceptance rate $\alpha$ the effective rate is
 $r_{\mathrm{spec}}(\alpha,K)=\frac{1-\alpha^{K+1}}{1-\alpha}\big/(Kc_d+C+\Tring)$, which recovers
 interactive speeds on an intercontinental ring by paying the round-trip once per accepted \emph{block}
 rather than once per token. That derivation, its optimal $K^\star$ for fixed $\alpha$, and the numbers are
 the companion's result and are not restated here.
 \paragraph{What is open.} The companion treats $\alpha$ as a fixed parameter and $K$ as a tuned constant.
 In deployment neither holds. The acceptance rate $\alpha$ \emph{drifts} within a single generation --- it
 is high on boilerplate and low at genuine decision points, and it depends on prompt, temperature, and how
 far the draft and target have diverged in context --- so the throughput-optimal $K$ is a moving target,
 not a launch-time flag. The open problem is an \emph{online} controller that adapts $K$ to the measured
 acceptance and the measured ring latency $\Tring$: estimate $\alpha$ from a trailing window of
 accept/reject outcomes, react to $\Tring$ jitter and packet loss that the constant-$\ell$ model omits, and
 set $K$ to the running optimum with a stability guarantee (a controller that chases a noisy $\alpha$ can
 oscillate). Two couplings make it harder and are themselves open: the determinism boundary of
 \S\ref{sec:determinism} must hold across the \emph{whole verified block}, not one token, so a replicated
 target must agree on the accept/reject cut; and the draft can itself be placed on a nearby, more reliable
 node~\cite{consensus}, making draft placement part of the scheduling problem of \S\ref{sec:pp}. The
 genuinely empirical unknown --- measured $\alpha$ for real draft/target pairs on real prompt distributions,
 which the companion explicitly does not claim --- gates any adaptive scheme.
 \section{Problem 5: the prefill launch-overhead wall}
 \label{sec:prefill}
 \paragraph{Statement.} On identical mathematics, hanzo-engine prefill reaches a lower GPU utilization than
 \code{llama.cpp} not because any kernel is slow but because there are too many kernels. Profiled on an
 NVIDIA GB10 (node \emph{spark}) running a 35B-class mixture-of-experts prefill, the engine sustains
 $\approx$79.3\% GPU utilization across $\approx$33{,}000 kernel launches per step, against
 \code{llama.cpp}'s $\approx$97.3\% across $\approx$3{,}000 --- an order of magnitude more launches for the
 same work. Per-kernel \emph{busy} time is already at parity; the gap is entirely the idle interstice
 between launches, i.e.\ launch and scheduling overhead, not arithmetic. Decode does not have this problem,
 because the decode path is captured into a CUDA graph that replays the whole token as one submission; the
 prefill path is not, because mixture-of-experts routing is \emph{data-dependent} --- which experts fire
 depends on the tokens --- so the shape of the work changes per step and a naive static graph capture does
 not apply.
 \paragraph{Two attacks, and they compose.} There are two structurally different ways to collapse the launch
 count, and they are not exclusive:
 \begin{itemize}[leftmargin=1.4em,itemsep=2pt]
 \item \textbf{Fixed-shape chunked graph capture.} Quantize the data-dependent dimension: route to a fixed
 capacity per expert (pad or drop to a chunk size), so the per-step work has a bounded set of shapes, and
 capture a CUDA graph per shape. This keeps the many small kernels but pays their launch cost \emph{once}
 at capture and replays them as one submission, exactly as decode already does. It is a runtime technique
 and inherits the graph machinery the engine has.
 \item \textbf{Compiler-level DAG fusion.} Reduce the \emph{number} of kernels rather than the cost of
 launching them: fuse elementwise and reduction chains into fewer, fatter kernels at compile time, so there
 is less to launch in the first place. This is the subject of the kernel-fusion companion~\cite{fuse},
 where fusion is shown to be composition; extending it from local elementwise chains to the full prefill DAG
 (across the MoE gate, the expert GEMMs, and the combine) is the open compiler problem.
 \end{itemize}
 Graph capture attacks the \emph{launch cost}; fusion attacks the \emph{launch count}; applied together they
 multiply. The decisive reason to invest in fusion as well as capture is portability: graph capture is a
 CUDA (and partial ROCm) facility, but fusion is a property of the kernel \emph{source} and therefore
 transfers to every backend the engine targets --- Metal, Vulkan, and ROCm, where graph support is weaker or
 absent~\cite{backend}. The open problem is the fusion \emph{partitioner}: given the prefill DAG, choose the
 kernel boundaries that minimize launches subject to register and shared-memory limits, and do it once,
 portably, from the one source.
 \section{Problem 6: KV-cache wire compression for prefill migration}
 \label{sec:kvcompress}
 \paragraph{Statement.} In steady-state pipeline decode the KV cache never crosses the wire --- each node
 keeps the keys and values for its own layers~\cite{latency}. The exception is \emph{migration}: moving a
 prefilled context between nodes, as in disaggregated prefill/decode, session hand-off, or re-placement when
 the schedule of \S\ref{sec:pp} changes. Then the KV cache --- the large, context-growing state --- must be
 shipped, and its size, not the activation's, sets the transfer cost. The engine already has an FP8 KV
 representation, so the obvious move is to ship KV in FP8 and halve the bytes.
 \paragraph{What is open.} Error accumulation across hops. A single FP8 round-trip of the KV cache is a
 tolerable, well-studied quantization. The open question is what happens when compression is applied
 \emph{repeatedly} --- a context migrated across several nodes over its lifetime, or a design that keeps KV
 in FP8 resident and re-quantizes on each write --- so that quantization error compounds hop over hop and
 layer over layer. Formally: bound the accumulated error of $H$ successive FP8 (de)compressions of a KV
 cache as a function of $H$, context length, and layer depth, and identify the schemes that arrest it ---
 error feedback (carry the residual forward), selective precision (keep the attention-sink and recent tokens
 in higher precision), or a migration protocol that recompresses from a retained high-precision anchor
 rather than from the last lossy copy. This is the compression dual of the determinism boundary
 (\S\ref{sec:determinism}): the all-reduce must be canonical \code{f32} because it is summed every layer,
 but the KV cache may tolerate FP8 precisely because it is \emph{read}, not reduced --- and the open problem
 is exactly how much, and for how many hops.
 \section{Problem 7: the geo-distributed decode latency floor and the placement it dictates}
 \label{sec:geofloor}
 \paragraph{What the companion owns.} The latency paper~\cite{latency} establishes the floor: single-stream
 autoregressive decode must traverse every pipeline stage per token, so its rate is $1/(C+\Tring)$ where
 $\Tring$ is the one-way loop latency, and because $\Tring$ is bounded below by the speed of light in fiber
 ($\approx$4.9\,$\mu$s/km), an intercontinental ring floors decode at a few tok/s \emph{regardless of
 accelerator speed}. The companion also states the levers as placement laws --- regional-first placement,
 speculate across every slow boundary, co-locate the ring turn, keep KV node-local --- and proves pipeline
 parallelism dominates tensor parallelism over any slow link by $\sim\!4L/N$. The floor, the laws, and the
 $4L/N$ result are the companion's and are not restated.
 \paragraph{What is open.} The companion's laws are heuristics, stated and argued but not optimized or
 measured end-to-end. Two things remain. First, \emph{empirical validation}: the companion is explicit that
 it reports no wall-clock from a running geo deployment; a real multi-continent ring, with real jitter and
 loss, must confirm the $1/(C+\Tring)$ floor and the speculative recovery. Second, and deeper, the
 \emph{placement and replication optimization} the floor implies. If one ring cannot serve a global user
 base at interactive latency, the answer is \emph{regional replicas} --- multiple rings, each latency-local
 to a set of users, with requests routed to the nearest --- and choosing how many rings, in which regions,
 holding which shards, to minimize the population's tail latency under a memory and cost budget is a
 facility-location problem entangled with the layer-assignment ILP of \S\ref{sec:pp}: where a node sits in
 space changes both which users it is near and which link it contributes to its ring. Formalizing that
 joint placement--partition problem, and the routing that assigns each request to a ring, is open, and it is
 the geo-scale version of Problem~3.
 \section{Problem 8: cheap verifiable inference for a permissionless compute network}
 \label{sec:verify}
 \paragraph{What the companion owns.} The three-plane blueprint~\cite{consensus} builds the transport and
 control layers of a permissionless inference network: a post-quantum-authenticated ZAP wire that proves
 \emph{who} sent a tensor, a Quasar consensus control plane that admits staked miners and agrees the
 placement map, and a Photon committee that can threshold-sign an $(\text{request},\text{output})$ pair.
 For result integrity it proposes three composable mechanisms --- threshold attestation of outputs,
 redundant re-execution with the canonical-\code{f32} property as a bit-exact cross-vendor oracle, and
 luminance reputation as the economic backstop --- and labels them, honestly, as design. Authentication and
 committee formation are solved there and not reopened here.
 \paragraph{What is open: verification that is cheaper than the work.} Authentication proves who sent a
 tensor, not that the tensor is \emph{correct}, and the blueprint's strongest integrity mechanism ---
 redundant re-execution --- costs a full second computation on every checked request. That is fine as a
 sampled spot check but cannot be the common path: a network whose verification costs as much as its
 inference has no margin to pay honest miners. The open problem is \emph{sub-linear} verification of correct
 inference. Candidate directions, each unproven for transformer decode:
 \begin{itemize}[leftmargin=1.4em,itemsep=2pt]
 \item \textbf{Sampling-based logit audits.} A verifier recomputes a \emph{random subset} of the claimed
 logits (a few positions, a few layers) and checks them against the miner's committed output, catching a
 cheating miner with high probability at a fraction of the compute. The open question is the soundness
 --- how large a subset defeats an adversary who corrupts only the tokens a user cares about --- and how it
 composes with the canonical-\code{f32} determinism (\S\ref{sec:determinism}) that makes a recomputed logit
 comparable bit-for-bit in the first place.
 \item \textbf{Redundant-execution economics.} Treat verification as a game: honest computation must be the
 profit-maximizing strategy under a spot-check probability $p$, a stake bond, and a slashing penalty. The
 open problem is to price $p$, the bond, and the reward so that the \emph{expected} cost of cheating exceeds
 its gain for a rational miner, while keeping the honest miner's expected verification tax small --- and to
 do so against collusion among a committee and grinding on the verification lottery, threats the blueprint
 names but does not price.
 \end{itemize}
 The unifying observation is that the canonical-\code{f32} wire built for cross-vendor \emph{correctness}
 (\S\ref{sec:determinism}, \cite{het}) is the same substrate that makes inference \emph{checkable}: two
 honest nodes agree bit-for-bit, so any disagreement is a proof of fault rather than a rounding artifact.
 Turning that from a full-re-execution check into a sampled, economically-sound one is the last open
 problem, and the one that decides whether a permissionless inference network can exist at all.
 \section{Synthesis}
 \label{sec:synthesis}
 The eight problems are not independent, and Table~\ref{tab:map} states their dependencies as plainly as
 their statements. Three throughlines run across them.
 \emph{Canonical \code{f32} is load-bearing twice.} The same property that makes an \code{f16} rank and a
 \code{bf16} rank agree on a reduced activation (\S\ref{sec:determinism}) is what lets a third party check a
 miner's work without a rounding dispute (\S\ref{sec:verify}). Correctness under replication and verifiable
 correctness are the same substrate seen from two sides.
 \emph{Scheduling is one problem at two scales.} The layer-assignment ILP of \S\ref{sec:pp} and the
 regional-replica placement of \S\ref{sec:geofloor} are the LAN and geo faces of a single
 partition-and-place optimization; solving one in isolation leaves the other's coupling on the table, and a
 real deployment sets both at once, with speculative-draft placement (\S\ref{sec:spec}) folded in.
 \emph{Pipeline parallelism dissolves more than it creates.} It sidesteps quant-aware sharding
 (\S\ref{sec:quantshard}), collapses the determinism boundary to one sampler (\S\ref{sec:determinism}), and
 makes the graph-capture asymmetry a local choice (\S\ref{sec:prefill}) --- but it buys those with the
 scheduling problem of \S\ref{sec:pp} and the migration cost of \S\ref{sec:kvcompress}. The net is
 favourable, which is why the series argues for it; the residue is real, which is why it is here.
 Finally, the register. Some of these are engineering with a known shape --- adaptive $K$ (\S\ref{sec:spec}),
 fixed-shape graph capture (\S\ref{sec:prefill}), FP8 migration (\S\ref{sec:kvcompress}) --- and will yield
 to careful implementation and measurement. Others are genuinely research --- the determinism-boundary
 theory (\S\ref{sec:determinism}), the heterogeneous scheduling ILP (\S\ref{sec:pp}), sub-linear
 verification (\S\ref{sec:verify}) --- and may not have clean answers. We have tried to say which is which,
 because the most useful thing an open-problems paper can do is not overclaim the difficulty of the easy
 ones or underclaim the depth of the hard ones.
 \begin{table}[t]
 \centering
 \small
 \setlength{\tabcolsep}{5pt}
 \begin{tabular}{@{}p{0.4cm}p{4.5cm}p{2.2cm}p{5.1cm}@{}}
 \toprule
 \# & \textbf{Problem} & \textbf{Exposed by} & \textbf{What is open (owning companion)} \\
 \midrule
 1 & Sampling-determinism boundary & heterogeneity & minimal agreement set; broadcast vs.\ canonicalize cost~\cite{het} \\
 2 & Quant-aware TP sharding & heterogeneity & superblock-aligned splits without dequant--requant~\cite{het} \\
 3 & PP layer-assignment ILP & heterogeneity & Eq.~\eqref{eq:ilp}: heuristics, online rebalance~\cite{het,latency} \\
 4 & Adaptive speculation & wide area & online $K$ under drifting $\alpha$, jittering $\Tring$~\cite{latency} \\
 5 & Prefill launch-overhead wall & (single-node) & fusion partitioner; portable, composes with capture~\cite{fuse,backend} \\
 6 & KV wire compression & wide area & cross-hop FP8 error accumulation bound~\cite{latency} \\
 7 & Geo decode latency floor & wide area & empirical floor; regional-replica placement~\cite{latency} \\
 8 & Cheap verifiable inference & permissionless & sub-linear logit audits; slashing economics~\cite{consensus} \\
 \bottomrule
 \end{tabular}
 \caption{The eight open problems, the environment property that exposes each, and the open kernel with the
 companion that owns the surrounding result. Problems 3 and 7 are one optimization at two scales; problems 1
 and 8 share the canonical-\code{f32} substrate. Problem 5 is exposed already single-node but is sharpest
 when prefill is distributed.}
 \label{tab:map}
 \end{table}
 \section*{Acknowledgement of scope}
 \addcontentsline{toc}{section}{Acknowledgement of scope}
 This paper contributes problem statements, not solutions, and every result it leans on belongs to a cited
 companion: the cross-vendor decode measurement and its residuals~\cite{het}, the latency floor and the
 speculative cure~\cite{latency}, and the three-plane network with its verification design~\cite{consensus}.
 Where a formal object appears --- the agreement set (\S\ref{sec:determinism}), the assignment ILP
 Eq.~\eqref{eq:ilp}, the error-accumulation bound (\S\ref{sec:kvcompress}) --- it is stated to make the open
 question precise, not claimed as solved. The measured numbers quoted (the launch-overhead profile of
 \S\ref{sec:prefill}) are attributed to the hardware that produced them (NVIDIA GB10) and to the companion
 that carries the surrounding stack~\cite{backend}.
 \begin{thebibliography}{9}
 \small
 \bibitem{het} Hanzo AI --- ML Systems. \emph{One Cluster, Every Vendor: Cross-Vendor Tensor-Parallel LLM
 Inference over Commodity Ethernet.} Distributed-inference companion (measured).
 \bibitem{latency} Hanzo AI --- ML Systems. \emph{The Latency Physics of Geo-Distributed Autoregressive
 Decode.} Distributed-inference companion (analytical).
 \bibitem{consensus} Hanzo AI --- ML Systems. \emph{The Three-Plane Compute Fabric: Consensus-Managed,
 ZAP-Transported Decentralized AI Inference.} Distributed-inference companion (architecture).
 \bibitem{backend} Hanzo AI. \emph{Edge LLM Inference Across Commodity Accelerators: hanzo-engine at
 llama.cpp Decode Parity on Every Backend.} Paper 05 (measured).
 \bibitem{fuse} Hanzo AI. \emph{Fusion Is Composition: Kernel Fusion as the Functor Law.} Paper 07.1.
 \bibitem{megatron} M.\ Shoeybi, M.\ Patwary, R.\ Puri, P.\ LeGresley, J.\ Casper, B.\ Catanzaro.
 \emph{Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.}
 arXiv:1909.08053, 2019.
 \bibitem{gpipe} Y.\ Huang et al. \emph{GPipe: Efficient Training of Giant Neural Networks using Pipeline
 Parallelism.} NeurIPS, 2019.
 \bibitem{pipedream} D.\ Narayanan, A.\ Harlap, A.\ Phanishayee, et al. \emph{PipeDream: Generalized
 Pipeline Parallelism for DNN Training.} SOSP, 2019.
 \bibitem{petals} A.\ Borzunov et al. \emph{Petals: Collaborative Inference and Fine-tuning of Large
 Models.} arXiv:2209.01188, 2022.
 \bibitem{leviathan} Y.\ Leviathan, M.\ Kalman, Y.\ Matias. \emph{Fast Inference from Transformers via
 Speculative Decoding.} ICML, 2023.
 \bibitem{chen} C.\ Chen, S.\ Borgeaud, G.\ Irving, J.-B.\ Lespiau, L.\ Sifre, J.\ Jumper. \emph{Accelerating
 Large Language Model Decoding with Speculative Sampling.} arXiv:2302.01318, 2023.
 \end{thebibliography}
 \end{document}
@@ -0,0 +1,468 @@
 \documentclass[11pt]{article}
 \input{shared/kernel-preamble.tex}
 \title{\textbf{One Cluster, Every Vendor:\\ Cross-Vendor Tensor-Parallel LLM Inference over Commodity Ethernet}}
 \author{Hanzo AI --- ML Systems}
 \date{Distributed companion to the \emph{One Source, Every Backend} series --- extends Paper 05 across the wire}
 \begin{document}
 \maketitle
 \begin{abstract}
 Real accelerator fleets are vendor-mixed: an AMD APU, an NVIDIA dev kit, and a Mac sit on the same
 desk, and no collective-communication library spans them --- NCCL is NVIDIA-only, RCCL is AMD-only, and
 neither will open a channel to the other. This paper reports a working result, with its limits stated
 plainly. One inference engine (\code{hanzo-engine}, Rust, one codebase, per-backend kernels) runs
 on three vendor-diverse accelerators --- AMD ROCm gfx1151 (a ``Strix Halo'' APU, node \emph{evo}), NVIDIA
 CUDA sm\_121 (a GB10 Blackwell, node \emph{spark}), and Apple Metal M4 Max (node \emph{dbc}). We make its
 Megatron-style tensor-parallel (TP) collectives \emph{vendor-agnostic} by host-staging every exchange
 --- \code{device}$\to$\code{CPU}$\to$\code{TCP}$\to$\code{CPU}$\to$\code{device} --- so any device that
 can copy to host storage can join the ring, which is exactly what NCCL/RCCL cannot do across a vendor
 boundary. On a two-node ring (\emph{evo} ROCm $+$ \emph{spark} CUDA) over a 2.5GbE direct link, the
 sharded engine \textbf{builds, connects, shards a \code{world\_size}$=$2 tensor-parallel model, and
 generates fully coherent multi-token output --- chain-of-thought and a correct answer --- across the
 AMD/NVIDIA boundary at 50.4~tok/s decode} (TTFT 0.09s, 213~tok/s prefill). To our knowledge this is the
 first reported cross-vendor (AMD$+$NVIDIA) tensor-parallel LLM inference over commodity ethernet in a
 single engine.
 We are equally explicit about the engineering, because the honest failures are the paper's most useful
 content. The socket-ring path had only ever been \emph{compiled}, never \emph{run} (only the NCCL path
 was exercised), and bringing it up cross-vendor surfaced four heterogeneity bugs --- a \code{len}$^2$
 byte-count overflow, a missing ROCm host-staging arm that meant the ring never even compiled for AMD, a
 10s handshake timeout that raced heterogeneous startup, and --- the subtle one that blocked coherent
 decode --- a \emph{heterogeneous-dtype-on-the-wire} hazard, where the ROCm rank auto-selects \code{f16}
 and the CUDA rank auto-selects \code{bf16} and each reinterpreted its neighbour's 16-bit words as the
 wrong floating-point layout, corrupting the reduction from the first layer. The cure is a canonical
 \code{f32} ring wire (a lossless superset of both) plus a \code{world\_size}$-$1 accumulating all-reduce
 correct for any \code{N}$\geq$2 and the removal of a spurious power-of-two world-size restriction; all
 four fixes are merged on \code{main}, and a same-vendor control (both ranks \code{f16}) matches the
 cross-vendor run token-for-token, isolating the wire dtype as the sole cross-vendor variable. The
 measured cost of the primitive is itself a result: 50.4~tok/s is $\approx$3.3$\times$ slower than the same
 model single-node (168~tok/s), and the gap is \emph{all} synchronization --- an all-reduce every layer,
 over the wire. That measured tax is the empirical case for \textbf{pipeline parallelism} --- contiguous
 layer ranges, activations-only on the wire (\code{O(nodes)} hops, not \code{O(layers)} all-reduces),
 format-agnostic, one generation loop driving stateless workers --- as the structurally better primitive
 for heterogeneous clusters: the numeric fixes made TP \emph{correct}, not well-matched. Three residuals
 stay honestly open, and we develop them, with five further problems, as a research agenda in the
 companion~\cite{openproblems}: the two ranks still sample independently from their own-dtype logits (a
 near-tie argmax can diverge and desync the KV caches), the CUDA rank must run graphless to stay lockstep
 with the graphless ROCm rank, and TP is still coupled to the dequantized (safetensors) weight path.
 \end{abstract}
 \section{The problem: fleets are vendor-mixed, and the collective libraries are not}
 \label{sec:problem}
 The single-node premise of this series is that the interesting inference target is a commodity box a
 customer already owns~\cite{backend}. The multi-node reality is that they own \emph{several}, and not
 from one vendor. A team that bought an AMD ``Strix Halo'' mini-PC last quarter, a Grace-Blackwell dev kit
 this quarter, and codes on a MacBook has three capable accelerators totalling well over a hundred
 gigabytes of unified memory --- enough to hold a model no single one of them can --- sitting idle on a
 LAN. The obstacle is not the models and not the math. It is that \emph{the collective-communication
 layer is single-vendor by construction}.
 Tensor parallelism~\cite{megatron} shards each weight matrix across ranks and, twice per transformer
 block (after attention, after the MLP), performs an \code{all\_reduce} to sum the partial results back
 into a full activation. That \code{all\_reduce} is the whole game, and in practice it is delegated to a
 vendor library: \textbf{NCCL}~\cite{nccl} on NVIDIA, \textbf{RCCL}~\cite{rccl} on AMD. Both are excellent
 and both are closed at the vendor boundary: NCCL speaks to NVIDIA GPUs over NVLink/PCIe/GPUDirect and
 has no concept of an AMD peer; RCCL is the symmetric mirror. There is no NCCL$\leftrightarrow$RCCL
 channel, no cross-vendor GPUDirect, and no shared handle either side will accept. So the textbook path to
 running one model across \emph{evo} and \emph{spark} together does not merely underperform --- it does
 not exist.
 The engine already dodges the \emph{first} half of this problem. \code{hanzo-engine} is one Rust codebase
 that lowers to CUDA, ROCm, Metal, Vulkan, and CPU behind a single model loader, reaching \code{llama.cpp}
 decode parity on all three vendors from that one build~\cite{backend}; the kernel layer that makes ``one
 source, every backend'' hold is the subject of the companion DSL papers~\cite{umbrella,fuse,oracle}. What
 was missing is the \emph{second} half: a way for those per-vendor engines to talk to \emph{each other}
 inside one cluster. This paper builds that channel, runs it across a vendor boundary for the first time,
 and reports --- without varnish --- exactly how far it gets and what it cost to get there.
 \section{The design: a host-staged socket ring makes the collective vendor-agnostic}
 \label{sec:design}
 The engine carries a unified \code{Comm} abstraction with three backends behind one enum: \code{Nccl}
 (the fast same-vendor NVIDIA path), \code{Ring} (a socket-based collective), and \code{Dummy}
 (\code{world\_size}$=$1). The idea that makes cross-vendor work possible is in the \code{Ring} backend,
 and it is deliberately humble: \textbf{stage every tensor through host memory and move the bytes over
 TCP.}
 Concretely, a ring \code{sum\_all\_reduce} does not touch a vendor collective at all. It pulls the
 tensor's storage to the CPU, converts it to a canonical \code{f32} payload, exchanges the bytes with its
 ring neighbours over ordinary TCP sockets, sums in \code{f32}, and copies the reduced result back to the
 device in its native dtype (Listing~\ref{lst:stage}). The single load-bearing property is that the
 exchange happens in \code{CpuStorage}, which \emph{every} backend can produce --- CUDA, Metal, and ROCm
 each implement \code{to\_cpu\_storage()}. The device that produced the tensor is irrelevant by the time
 the bytes reach the socket. \emph{Why} the wire dtype is a canonical \code{f32} rather than the raw
 device bytes --- the difference between coherent and garbage output across vendors --- is the subject of
 \S\ref{sec:rootcause}; here it is simply the shipped design.
 \begin{lstlisting}[language=Rust,caption={The vendor-agnostic collective, condensed from the shipping
 code. The reduce goes device$\to$host$\to$canonical \code{f32}$\to$TCP$\to$\code{f32} sum$\to$device; the
 \code{match} over \code{Storage} is the only place the backend is named, and every arm lands in the same
 \code{CpuStorage}. The \code{cpu\_to\_f32} stage is what makes an \code{f16} rank and a \code{bf16} rank
 agree (\S\ref{sec:rootcause}). NCCL, by contrast, would keep the whole exchange on-device and refuse a
 foreign-vendor peer.},label={lst:stage}]
 fn sum_all_reduce(&self, xs: &Tensor) -> Result<Tensor> {
    let storage = xs.storage_and_layout().0;
    let host = match &*storage {              // device -> host (any backend)
        Storage::Cpu(s)   => s.clone(),
        Storage::Cuda(s)  => s.to_cpu_storage()?,
        Storage::Metal(s) => s.to_cpu_storage()?,
        #[cfg(feature = "rocm")]
        Storage::Rocm(s)  => s.to_cpu_storage()?, // was missing: ring never built for AMD (Bug 2)
    };
    let wire = cpu_to_f32(&host)?;            // CANONICAL wire dtype: f16|bf16|f32 -> f32
    let sum  = self.ring_exchange(&wire, xs.dims())?; // world_size-1 accumulating steps, sum in f32
    Tensor::from_slice(&sum, xs.dims(), xs.device())? // host -> device
        .to_dtype(xs.dtype())                 // back to this rank's native compute dtype
 }
 \end{lstlisting}
 This is not free. NCCL keeps the reduction on-device and rides NVLink; the host-staged ring pays two
 extra device$\leftrightarrow$host copies per collective, a widening cast to \code{f32}, and then rides a
 2.5GbE link whose measured goodput is $\approx$280~MB/s (\S\ref{sec:repro}). That is the correct trade for
 the problem: we spend bandwidth to buy \emph{vendor-agnosticism}, because the alternative on a mixed fleet
 is not a faster collective, it is \emph{no} collective. The ring is selected purely by configuration ---
 \code{use\_ring()} is true when the \code{ring} feature is built and a \code{RING\_CONFIG} path is present
 in the environment (\S\ref{sec:repro}) --- so the same engine binary is a single-node server, an NCCL
 rank, or a socket-ring rank depending only on how it is launched.
 \section{Compiles $\ne$ runs: an unexercised path was broken three ways}
 \label{sec:compiles}
 Here is the lesson that generalizes past this codebase. \emph{The ring path had been written, had type-
 checked, had shipped in every build for months --- and had never once been run.} All multi-GPU testing
 had gone through NCCL (the same-vendor path), so the socket ring was dead code that happened to compile.
 The first time it was asked to carry a real tensor across a real vendor boundary, it failed three
 separate ways, each a different species of the gap between ``the compiler accepted it'' and ``it does the
 thing.'' All three are fixed on \code{main} (commit \code{63cc4c7ac}); the fourth, subtler failure ---
 visible only once the generated tokens came out wrong --- is \S\ref{sec:rootcause}.
 \paragraph{Bug 1 --- a \code{len}$^2$ byte count (a logic error the types could not catch).}
 The wire code computed its transfer size as
 \begin{center}
 \code{let nbytes = x.len() * std::mem::size\_of\_val(x);}
 \end{center}
 This looks reasonable and is badly wrong. For a slice \code{x:\ \&[T]}, \code{size\_of\_val(x)} is
 \emph{already} the size of the whole referent --- \code{x.len()}\,$\times$\,\code{size\_of::<T>()} --- so
 multiplying by \code{x.len()} again yields \code{len}$^2$\,$\times$\,\code{elem\_size}, overshooting the
 true byte count by a factor of \code{len}. The subsequent \code{from\_raw\_parts(ptr, nbytes)} then
 fabricates a byte slice spanning far past the end of the allocation, and \code{write\_all} walks off the
 buffer and hits \code{EFAULT}. It is invisible for a one-element tensor (\code{len}$^2 =$ \code{len}) and
 fatal for every real activation. The fix is one token --- delete the \code{x.len()\ *} --- but the point
 is that the type system was content the entire time: both operands are \code{usize}, the product is a
 \code{usize}, and the meaning is nonsense. Only \emph{running} it with \code{len}$>$1 reveals it.
 \paragraph{Bug 2 --- the ring never compiled for AMD (a hole hidden by a feature flag).}
 The device$\to$host \code{match} in Listing~\ref{lst:stage} had arms for \code{Cpu}, \code{Cuda}, and
 \code{Metal} but \emph{not} \code{Rocm}. Because \code{Storage::Rocm} only exists when the \code{rocm}
 feature is enabled, and no one had ever built \code{ring} $+$ \code{rocm} together, the non-exhaustive
 \code{match} was never forced into existence. The moment you try to build the one configuration a
 heterogeneous ROCm$+$CUDA cluster needs, it fails to compile. The fix adds the two
 \code{\#[cfg(feature = "rocm")]} arms (one in \code{sum\_all\_reduce}, one in \code{all\_gather}) that
 stage a ROCm tensor to host exactly like the others. This is the subtler lesson: a feature-gated code
 path is not merely untested, it may be \emph{unbuilt}, and a green CI that never enables that flag
 combination proves nothing about it.
 \paragraph{Bug 3 --- a handshake that assumed both nodes were the same speed.}
 The ring's neighbour-connect loop gave up after 10 seconds. On a homogeneous cluster every rank boots in
 lockstep and 10s is generous. On a \emph{heterogeneous} cluster the nodes do not load at the same rate ---
 a Blackwell dev kit and an APU parse weights, warm kernels, and reach the barrier at visibly different
 times --- so the faster node would time out waiting for the slower one and abort the whole ring. Raising
 the timeout to 120s (and matching the panic message) lets asynchronous, uneven startup settle. The
 assumption ``all ranks are interchangeable, so they arrive together'' is exactly the kind of invariant
 that is silently true on one vendor and silently false across three.
 None of these three is deep. That is the message. A distributed path can be well-typed, code-reviewed,
 and shipped, and still be dead on arrival in three independent ways, because ``it compiles'' certifies
 none of the things that actually matter at run time --- the arithmetic of a byte count, the existence of a
 match arm under a flag nobody set, or a timing assumption baked into a constant. And a fourth bug, subtler
 than all three and invisible until the generated tokens themselves came out wrong, waited past these; it
 is the root cause of cross-vendor incoherence, and its fix is what turns prefill into coherent decode
 (\S\ref{sec:rootcause}).
 \section{Result: coherent decode across the AMD/NVIDIA boundary}
 \label{sec:result}
 With the four heterogeneity bugs fixed --- three mechanical (\S\ref{sec:compiles}), one numeric
 (\S\ref{sec:rootcause}) --- the cross-vendor ring generates coherent output. On a two-node ring ---
 \emph{evo} (AMD ROCm gfx1151) as rank~0, \emph{spark} (NVIDIA CUDA sm\_121) as rank~1 --- connected by a
 2.5GbE direct cable, the sharded engine:
 \begin{enumerate}[leftmargin=1.5em,itemsep=1pt]
 \item \textbf{builds} with \code{ring} $+$ \code{rocm} on the AMD node and \code{ring} $+$ \code{cuda} on
 the NVIDIA node (Bug~2 was the blocker);
 \item \textbf{connects} the ring --- each rank binds its listen port, dials its right neighbour, and
 accepts its left, the handshake surviving the uneven boot (Bug~3);
 \item \textbf{shards} a Qwen3-0.6B at \code{world\_size}$=$2, each node holding half of every parallelized
 weight matrix via the \code{ColumnParallelLayer}/\code{RowParallelLayer} split; and
 \item \textbf{decodes coherently} --- running the host-staged \code{f32} \code{all\_reduce} across the
 vendor boundary at every layer --- emitting a chain-of-thought and a correct final answer in lockstep, at
 \textbf{50.4~tok/s decode} (TTFT 0.09s, 213~tok/s prefill).
 \end{enumerate}
 The two ranks run \emph{different} 16-bit compute dtypes --- \emph{evo} in \code{f16}, \emph{spark} in
 \code{bf16} (\S\ref{sec:rootcause}) --- and still agree, because the reduction crosses the wire in
 canonical \code{f32}. To our knowledge this is the first reported cross-vendor (AMD$+$NVIDIA)
 tensor-parallel LLM inference over commodity ethernet in a single engine: two different-vendor
 accelerators jointly executing one tensor-parallel forward \emph{and} autoregressive decode, over a
 commodity cable, with no shared-vendor collective (NCCL/RCCL) between them --- the \code{all\_reduce} that
 both libraries refuse across the boundary runs to completion on plain TCP.
 \paragraph{Controls isolate the wire, not the model.} A \emph{same}-vendor control --- both ranks forced
 to \code{f16}, so the raw-bytes shortcut and the canonical-\code{f32} wire coincide --- produces the same
 tokens as the cross-vendor run, which pins the dtype wire (\S\ref{sec:rootcause}), not the ring
 mechanism, as the only thing heterogeneity changed. A \emph{single}-node control --- the same Qwen3-0.6B
 on \emph{spark} alone --- generates the same answer at 168~tok/s decode and 464~tok/s prefill. So the
 cross-vendor 50.4~tok/s is $\approx$3.3$\times$ slower than single-node decode, and the entire gap is
 synchronization: a host-staged \code{all\_reduce} every layer, over the wire, is the only added work. The
 model is coherent and the kernels are at parity; what the ring costs is the wire.
 \section{Root cause and fix: the heterogeneous-dtype-on-the-wire hazard}
 \label{sec:rootcause}
 The fourth bug was invisible until the tokens themselves came out wrong, and it is the one worth dwelling
 on, because it is the trap waiting in every host-staged cross-vendor collective. We state the cause, the
 fix, and the residuals the fix leaves.
 \subsection{The cause: two ranks legitimately pick different 16-bit formats}
 \label{sec:dtype}
 The engine picks its compute dtype \emph{per node} from the \emph{local} device. The selector
 (\code{determine\_auto\_dtype\_all}) returns \code{f16} when it sees a ROCm gfx1151 device --- RDNA3.5 has
 fast \code{f16} matrix cores (WMMA) but no fast \code{bf16} matmul --- and \code{bf16} for the CUDA
 Blackwell path. So on a mixed ring the two ranks legitimately and independently choose \emph{different}
 16-bit formats: \emph{evo} runs the model in \code{f16}, \emph{spark} in \code{bf16}. The original
 host-staged ring shipped \emph{raw device-dtype bytes} and the receiver reinterpreted them as \emph{its
 own} local dtype. Since \code{f16} and \code{bf16} share a 16-bit container but split it differently ---
 5-bit exponent / 10-bit mantissa versus 8-bit exponent / 7-bit mantissa --- every value crossing the
 boundary was silently re-read with the wrong exponent/mantissa layout. The bit pattern was intact; its
 \emph{interpretation} was wrong, which is worse, because nothing errored: the \code{all\_reduce} summed
 correctly-typed nonsense into correctly-typed nonsense, corrupting activations from the first layer.
 Notably, a \emph{same}-vendor ring never exposed this --- both ranks pick the same dtype and the raw-bytes
 shortcut accidentally works --- which is precisely why the hazard survived months of same-vendor NCCL and
 ring testing until the first cross-vendor run, and why the same-vendor control (\S\ref{sec:result}) is the
 clean isolator.
 \subsection{The fix: a canonical \code{f32} wire and an \code{N}-node accumulating reduce}
 \label{sec:fix}
 The remedy is structural, not a patch: the wire needs a \emph{canonical} dtype both ranks agree on
 independent of their local compute dtype. \code{f32} is the natural choice --- a lossless superset of both
 \code{f16} and \code{bf16}, and higher-precision accumulation for the reduction besides. The shipped
 reduce (Listing~\ref{lst:stage}) stages each device tensor to host, converts to \code{f32}
 (\code{cpu\_to\_f32}), exchanges and sums in \code{f32}, and converts the result back to the receiver's
 native device dtype. The 16-bit formats now meet only \emph{inside} each node; the wire is always
 \code{f32}, so both ranks reduce the same values bit-for-bit regardless of vendor, and the reduction is
 carried out in the CPU-\code{f32} path with a canonical tie-break so a ROCm box and a CUDA box do not
 diverge on rounding order.
 The same pass closed a latent correctness bound. The original \code{sum\_all\_reduce} performed a
 \emph{single} write-right / read-left exchange and summed the neighbour's shard --- a complete all-reduce
 for two ranks, but for \code{N}$>$2 rank $i$ only ever incorporates rank $i{-}1$, so ranks $i{-}2,
 i{-}3,\dots$ never reach it. The fix replaces the single exchange with the textbook \code{world\_size}$-$1
 accumulating ring (each rank forwards and sums its running shard for \code{N}$-$1 steps), correct for any
 \code{N}$\geq$2, and removes a spurious power-of-two world-size restriction inherited from the NCCL path
 (\code{is\_power\_of\_two}) that had made four the smallest cluster past two nodes the ring would even
 admit. The all-reduce is now algorithmically correct for any \code{N}$\geq$2; the \emph{measured}
 cross-vendor demonstration in this paper is the two-node \emph{evo}$+$\emph{spark} ring, and multi-node
 (\code{N}$>$2, e.g.\ adding \emph{dbc}/Metal) cross-vendor runs are the natural next validation, not a
 claimed measurement here.
 \subsection{Residuals: three problems the fix leaves open}
 \label{sec:residual}
 Coherent cross-vendor decode holds, but three residuals bound the result. We state each crisply and
 develop all three, alongside five further problems, as a research agenda in the companion~\cite{openproblems}.
 \paragraph{Sampling divergence (a determinism boundary).} The \emph{reduced activations} now agree
 bit-for-bit, but each rank still computes its final logits on its own device in its own 16-bit dtype and
 samples \emph{locally}. In the demonstrated runs the argmax stays in lockstep; a near-argmax-tie could in
 principle resolve differently under \code{f16} versus \code{bf16} rounding and desync the per-rank KV
 caches. The robust cure --- the head rank samples and broadcasts the token id, so every rank advances on
 the same token --- removes the divergence by construction; it is a determinism-boundary problem in
 replicated-state distributed inference, not a numeric one.
 \paragraph{Graphless lockstep (an execution-symmetry assumption).} The two ranks run \emph{two}
 generation loops kept in phase by the per-token \code{all\_reduce} barrier. To stay in phase the CUDA rank
 must run \emph{graphless}, matching the graphless ROCm rank and forgoing the $\approx$1.5$\times$ decode
 speedup CUDA-graph capture would give~\cite{backend}. Decode is coherent this way, but per-token
 cross-node lockstep between nodes that legitimately want different execution strategies is fragile --- an
 argument for pipeline parallelism (\S\ref{sec:pipeline}), where each node's execution is a purely local
 choice, rather than a bug to patch.
 \paragraph{Format coupling (a sharding limitation).} The TP collectives are wired through the
 non-quantized (safetensors) model path --- \code{ColumnParallelLayer} and \code{RowParallelLayer} live in
 the normal model definitions --- and \emph{not} through the GGUF quantized path. So even with the
 numerics correct, TP cannot yet shard the very block-quantized formats that make edge inference fit in
 commodity memory. Quant-aware sharding (splitting on superblock boundaries without a dequant--requant
 loss) is an open problem~\cite{openproblems}; pipeline parallelism sidesteps it (\S\ref{sec:pipeline})
 because activations, not weights, cross the seam.
 \section{The case for pipeline parallelism}
 \label{sec:pipeline}
 The dtype hazard and the \code{N}-node reduce are fixed, so the tensor-parallel ring on \code{main} is now
 \emph{correct}. But correct is not the same as well-matched, and the residuals of \S\ref{sec:residual}
 together with the measured synchronization tax point the same way. Tensor parallelism was built for a
 homogeneous, tightly-coupled, high-bandwidth, single-vendor machine, and every place it strains on our
 cluster is a place it assumes that machine: a per-layer all-reduce assumes cheap bandwidth
 (\S\ref{sec:design}) --- the measured $\approx$3.3$\times$ decode tax is exactly this bill; per-token
 lockstep assumes symmetric execution (\S\ref{sec:residual}); the parallel layers assume dequantized
 weights (\S\ref{sec:residual}). Fixing the numerics made the primitive correct; it did not make it the
 right primitive for a heterogeneous fleet.
 \textbf{Pipeline parallelism} inverts the mismatch. Partition the model into contiguous \emph{layer
 ranges}, one range per node; a token's hidden state flows through node~0's layers, crosses the wire once
 to node~1, flows through its layers, and so on. The consequences line up against the strains almost
 one-to-one:
 \begin{itemize}[leftmargin=1.4em,itemsep=2pt]
 \item \textbf{Activations-only on the wire.} What crosses a node boundary is a single hidden-state tensor,
 once per boundary, forward --- not an \code{all\_reduce} of every layer's output twice per block. On a
 $\approx$280~MB/s link that is the difference between a per-token tax of \code{O(1)} small transfers and
 \code{O(layers)} full-width collectives; it is the same $\approx$3.3$\times$ tax converted from
 per-layer to per-node.
 \item \textbf{One canonical cast, not a per-op reinterpretation.} The activation is a single tensor at
 each boundary, so agreeing on one wire dtype (\S\ref{sec:dtype}) is one cast at one seam per node, not a
 property every collective must carry. The heterogeneous-dtype hazard shrinks from pervasive to pointwise.
 \item \textbf{Format-agnostic.} Activations are just floats; the boundary does not know or care whether the
 node that produced them ran a safetensors or a GGUF model, on which vendor, with which kernel. Pipeline
 parallelism sidesteps the format coupling of \S\ref{sec:residual} for free, so a mixed cluster can run
 quantized shards.
 \item \textbf{One generation loop, stateless workers.} The head node runs the single sampling/generation
 loop; the others are stateless layer-executors that receive an activation, run their layers (holding only
 their own KV cache), and forward. There is no second generation loop to keep in phase and no cross-node
 per-token lockstep, so the CUDA-graph asymmetry (\S\ref{sec:residual}) becomes a purely \emph{local}
 choice --- each node may capture graphs for its own layers or not, and nobody desynchronizes --- and the
 sampling divergence collapses to the head's single sampler.
 \end{itemize}
 Pipeline parallelism is not a free lunch --- for a single decode stream it adds pipeline-bubble latency
 proportional to the number of stages, and throughput needs micro-batching to fill the
 pipe~\cite{gpipe}. Its wide-area latency budget --- one network hop per stage on the critical path of
 every token --- is the subject of the companion latency analysis~\cite{latency}, which shows the budget
 is spendable precisely because pipelining overlaps transport under compute and speculative decoding
 amortizes the round-trip. But its \emph{coupling} to the cluster is minimal and its assumptions are the
 ones a heterogeneous fleet can actually honour. The right primitive for one-model-across-many-vendors is
 the one that puts the least on the wire and demands the least agreement between ranks, and on both axes
 that is pipeline, not tensor, parallelism. The cross-vendor decode result (\S\ref{sec:result}) proves the
 \emph{channel} works across the boundary; the argument here is about which \emph{traffic pattern} to run
 over it, and it is under construction in the same codebase.
 \section{Reproducibility}
 \label{sec:repro}
 \paragraph{Hardware and link.} \emph{evo}: AMD Ryzen AI Max+ 395 ``Strix Halo,'' Radeon 8060S iGPU
 (gfx1151, RDNA3.5), ROCm. \emph{spark}: NVIDIA GB10 Grace-Blackwell (sm\_121), CUDA. \emph{dbc}: Apple
 M4 Max, Metal. \emph{evo}$\leftrightarrow$\emph{spark} is a direct 2.5GbE cable (static
 \code{192.168.77.0/24}), whose measured sustained-TCP goodput is $\approx$280~MB/s ($\approx$2.24~Gbit/s);
 its wide-area latency budget is analysed in the companion~\cite{latency}. The three-node substrate is a
 wired ethernet mesh with \emph{spark} kernel-IP-forwarding the closing \emph{evo}$\leftrightarrow$\emph{dbc}
 edge, so no switch or router is required.
 \paragraph{Build.} Each node builds the \emph{same} engine source with its own backend feature plus
 \code{ring}: \code{--features "cuda ring"} on \emph{spark}, \code{--features "rocm ring"} on \emph{evo}.
 The three mechanical fixes are at commit \code{63cc4c7ac}; the canonical-\code{f32} wire and the
 \code{world\_size}$-$1 reduce are on \code{main} (\code{hanzo-quant/src/distributed/},
 \code{hanzo-engine/src/ops.rs}).
 \paragraph{Ring configuration.} The ring backend is activated by pointing \code{RING\_CONFIG} at a JSON
 file per node; rank $r$ binds \code{port} and dials its right neighbour at
 \code{right\_ip:right\_port}. A two-node \emph{evo}(rank~0)$+$\emph{spark}(rank~1) ring:
 \begin{lstlisting}[language=Rust,caption={\code{RING\_CONFIG} for a 2-node cross-vendor ring. Rank~0
 (evo) listens on 9000 and dials spark; rank~1 (spark) listens on 9001 and dials back to evo, closing the
 ring.},label={lst:ring}]
 // evo  (rank 0, 192.168.77.1):  RING_CONFIG=/etc/hanzo/ring0.json
 { "master_ip": null,           "master_port": 9002,
  "port": 9000, "right_ip": "192.168.77.2", "right_port": 9001,
  "rank": 0, "world_size": 2 }
 // spark (rank 1, 192.168.77.2):  RING_CONFIG=/etc/hanzo/ring1.json
 { "master_ip": "192.168.77.1", "master_port": 9002,
  "port": 9001, "right_ip": "192.168.77.1", "right_port": 9000,
  "rank": 1, "world_size": 2 }
 \end{lstlisting}
 \noindent A \code{hanzo cluster init} / \code{hanzo cluster join <token>} pair wraps this JSON in a
 scan-to-join token (base58 of the topology plus a terminal QR code) so the operator never hand-writes the
 ports; the token bakes in the peer list and each worker self-assigns its rank. Both ranks launch the
 ordinary \code{hanzo serve} with \code{RING\_CONFIG} set and the same model.
 \paragraph{What is measured vs.\ open.} Measured on the real 2.5GbE link: cross-vendor build, connect,
 \code{world\_size}$=$2 shard, and coherent multi-token \emph{decode} across the AMD/NVIDIA boundary
 (50.4~tok/s decode, TTFT 0.09s, 213~tok/s prefill on Qwen3-0.6B); the same-vendor \code{f16}$+$\code{f16}
 control matching token-for-token; and the single-node \emph{spark} control (168~tok/s decode, 464~tok/s
 prefill). Open (\S\ref{sec:residual}, \cite{openproblems}): head-broadcast sampling to close the
 determinism boundary, \code{N}$>$2 cross-vendor runs, GGUF-path TP, and the migration to pipeline
 parallelism. No projected or synthetic number appears in this paper.
 \section{Related work and contribution}
 \label{sec:related}
 Tensor parallelism is Megatron-LM~\cite{megatron}; pipeline parallelism is GPipe and its
 successors~\cite{gpipe}; the on-device collectives are NCCL~\cite{nccl} and RCCL~\cite{rccl}, each
 single-vendor by design. Distributed inference over commodity/heterogeneous networks has been explored
 most prominently by Petals~\cite{petals}, which runs pipeline-parallel shards across volunteer nodes over
 the public internet --- close in spirit to \S\ref{sec:pipeline}, and evidence that the activations-only
 pattern is the one that survives a slow, loose network. What we add is narrow and, to our knowledge, not
 previously shown: a \emph{single} inference engine, one source lowered to three vendors' kernels, running
 a \emph{tensor-parallel} forward \emph{and coherent decode} across an \emph{AMD$+$NVIDIA} boundary on
 \emph{commodity ethernet} with \emph{no} shared-vendor collective --- and an unusually explicit account of
 the failure modes that a host-staged cross-vendor collective exposes (the \code{len}$^2$ overflow, the
 unbuilt ROCm arm, the startup-race timeout, and the heterogeneous-dtype wire). The design contribution is
 the observation, made concrete by those failures, that heterogeneity is a stress test for homogeneity
 assumptions, and that pipeline parallelism minimizes exactly the assumptions that break. This paper is the
 distributed extension of the ``one source, every backend'' thesis~\cite{backend,umbrella}: having made one
 engine run on every backend, we make those backends run in one cluster. Two companions extend the arc: the
 latency-physics analysis~\cite{latency} carries the wide-area budget for \S\ref{sec:pipeline}, and the
 three-plane compute-fabric blueprint~\cite{consensus} turns the trusted-LAN ring into a permissionless,
 consensus-managed, PQ-authenticated network. The open questions this result raises are collected as a
 research agenda in the open-problems companion~\cite{openproblems}.
 \section{Conclusion}
 \label{sec:conclusion}
 No collective library crosses a vendor boundary, so a mixed AMD/NVIDIA/Apple fleet cannot run one model
 together through the standard path. Host-staging the collective through CPU memory and TCP makes the
 channel vendor-agnostic, and on that channel one engine --- built from a single source for three vendors'
 kernels --- shards a model \code{world\_size}$=$2 across an AMD/NVIDIA boundary over a 2.5GbE cable and
 \emph{decodes coherently} at 50.4~tok/s, which we believe is a first. The road to that result ran through
 three bugs in a path that compiled for months but had never run --- a reminder that ``it builds''
 certifies almost nothing about a distributed system --- and one deeper hazard, a heterogeneous-dtype wire
 that a same-vendor cluster can never expose, cured by a canonical \code{f32} payload and an accumulating
 \code{world\_size}$-$1 all-reduce correct for any \code{N}$\geq$2. The measured $\approx$3.3$\times$
 synchronization tax, and the residuals that remain --- local sampling that can diverge at a near-tie,
 graphless lockstep between two generation loops, and TP's coupling to the dequantized weight path --- all
 point one way: toward pipeline parallelism, whose activations-only, one-generation-loop, format-agnostic
 shape asks of a heterogeneous cluster only what such a cluster can actually give. The channel is open
 across the boundary and coherent over it; the companions drive the right traffic~\cite{latency}, harden
 it into a network~\cite{consensus}, and enumerate what is left to solve~\cite{openproblems}.
 \begin{thebibliography}{9}
 \small
 \bibitem{backend} Hanzo AI. \emph{Edge LLM Inference Across Commodity Accelerators: Measured AMD, NVIDIA,
 and Apple Performance, and hanzo-engine at llama.cpp Decode Parity on Every Backend.} Paper 05.
 \bibitem{umbrella} Hanzo AI. \emph{One Source, Every Backend: The hanzo-kernel DSL --- Thesis and Guided
 Tour.} Paper 07 (umbrella).
 \bibitem{fuse} Hanzo AI. \emph{Fusion Is Composition: Kernel Fusion as the Functor Law.} Paper 07.1.
 \bibitem{oracle} Hanzo AI. \emph{One Oracle, Every Backend: The CPU Bit-Exactness Law.} Paper 07.4.
 \bibitem{latency} Hanzo AI --- ML Systems. \emph{The Latency Physics of Geo-Distributed Autoregressive
 Decode.} Distributed-inference companion (analytical).
 \bibitem{consensus} Hanzo AI --- ML Systems. \emph{The Three-Plane Compute Fabric: Consensus-Managed,
 ZAP-Transported Decentralized AI Inference.} Distributed-inference companion (architecture).
 \bibitem{openproblems} Hanzo AI --- ML Systems. \emph{Open Problems in Heterogeneous Distributed
 Inference.} Distributed-inference companion (research agenda).
 \bibitem{megatron} M.\ Shoeybi, M.\ Patwary, R.\ Puri, P.\ LeGresley, J.\ Casper, B.\ Catanzaro.
 \emph{Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.} arXiv:1909.08053, 2019.
 \bibitem{gpipe} Y.\ Huang et al. \emph{GPipe: Efficient Training of Giant Neural Networks using Pipeline
 Parallelism.} NeurIPS, 2019.
 \bibitem{petals} A.\ Borzunov et al. \emph{Petals: Collaborative Inference and Fine-tuning of Large
 Models.} arXiv:2209.01188, 2022.
 \bibitem{nccl} NVIDIA. \emph{NCCL: NVIDIA Collective Communications Library.}
 \bibitem{rccl} AMD. \emph{RCCL: ROCm Communication Collectives Library.}
 \end{thebibliography}
 \end{document}