mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-29 07:34:31 +00:00
51 lines
2.0 KiB
TeX
51 lines
2.0 KiB
TeX
\section{Edge Inference}
|
|
|
|
\label{sec:edge}
|
|
% ============================================================================
|
|
|
|
Not all inference should happen in the cloud. For latency-sensitive
|
|
applications (real-time object detection, voice assistants, on-device
|
|
classification), Hanzo AI Cloud supports edge inference via Hanzo
|
|
Edge~\cite{hanzoedge}.
|
|
|
|
\subsection{On-Device Runtime}
|
|
|
|
Hanzo Edge is a Candle-based inference runtime for mobile and IoT devices. It
|
|
supports:
|
|
|
|
\begin{itemize}[leftmargin=1.1em]
|
|
\item \textbf{Apple Neural Engine}: CoreML integration for ANE-accelerated
|
|
inference on iPhone/iPad/Mac.
|
|
\item \textbf{GPU compute}: Metal (Apple), Vulkan (Android/Linux) for
|
|
GPU-accelerated inference.
|
|
\item \textbf{CPU fallback}: NEON (ARM) and AVX-512 (x86) SIMD-optimized
|
|
CPU inference for devices without GPU access.
|
|
\item \textbf{Dynamic quantization}: INT8/INT4 quantization applied at
|
|
model load time, adapting to available memory.
|
|
\end{itemize}
|
|
|
|
\subsection{CRDT Result Sync}
|
|
|
|
Edge inference results sync to the user's Base instance via the CRDT protocol
|
|
described in the Base runtime paper~\cite{hanzobaseruntime}. A mobile device
|
|
runs inference locally, writes the result to its local Base vault, and the
|
|
result syncs to the user's other devices and cloud Base instance on next
|
|
connection. This enables offline inference with eventual consistency.
|
|
|
|
\subsection{Model Distribution}
|
|
|
|
Edge models are distributed from the Hanzo model registry via a CDN-backed
|
|
download with integrity verification:
|
|
|
|
\begin{equation}
|
|
\text{verify}(\text{model}_{\text{local}}) :
|
|
\text{SHA3-256}(\text{model}_{\text{local}}) \stackrel{?}{=}
|
|
\text{model\_id}_{\text{A-Chain}}
|
|
\end{equation}
|
|
|
|
The device downloads the model once, verifies its hash against the A-Chain
|
|
registry entry, and caches it locally. Model updates are delta-compressed
|
|
(binary diff of quantized weights) to minimize bandwidth.
|
|
|
|
% ============================================================================
|