papers/hanzo-jin-architecture/sections/02-background.tex

\section{Background}


\subsection{Joint Embedding Predictive Architectures}

JEPA~\cite{jepa2022} learns representations by predicting masked patch embeddings rather than reconstructing raw pixels. Given an image $x$, a context encoder $f_\theta$ produces embeddings for visible patches, and a predictor $g_\phi$ predicts embeddings for masked patches. The target encoder $f_{\bar{\theta}}$ (EMA of context encoder) provides prediction targets:

\begin{equation}
    \mathcal{L}_{\text{JEPA}} = \|g_\phi(f_\theta(x_{\text{vis}})) - f_{\bar{\theta}}(x_{\text{mask}})\|^2
\end{equation}

JEPA avoids reconstruction collapse by operating in embedding space, enabling learning of abstract features without pixel-level supervision.

\subsection{Mixture of Experts}

Sparse MoE~\cite{moe2017} routes tokens through a subset of expert networks:

\begin{equation}
    y = \sum_{i=1}^{K} g_i(x) \cdot E_i(x)
\end{equation}

where $g_i(x)$ is the gating weight for expert $E_i$, with typically $K \ll N$ experts active per token. This enables parameter scaling without proportional compute increase.

\subsection{Diffusion Models}

Diffusion models~\cite{ddpm2020} learn to reverse a noise process:

\begin{equation}
    p_\theta(x_{t-1}|x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \Sigma_\theta(x_t, t))
\end{equation}

Recent work shows diffusion transformers (DiT)~\cite{dit2023} can generate high-quality images with better scaling properties than U-Nets.