papers/embedding-7680/sections/03-dimensionality-analysis.tex

\section{Dimensionality Analysis}

\label{sec:analysis}

\subsection{Information-Theoretic Perspective}

The embedding dimension bounds the information capacity:

\begin{theorem}[Embedding Capacity]
An embedding space $\mathbb{R}^d$ with precision $p$ bits per dimension can distinguish at most $2^{pd}$ distinct semantic concepts.
\end{theorem}

For 7680 dimensions with 16-bit precision: $2^{16 \times 7680} \approx 10^{37,000}$ distinguishable embeddings---vastly exceeding any practical vocabulary.

\subsection{Semantic Preservation}

We measure semantic preservation as the correlation between original model similarity and projected similarity:

\begin{definition}[Semantic Preservation]
For source embeddings $\phi_S \in \mathbb{R}^{d_S}$ projected to $\phi_T \in \mathbb{R}^{d_T}$:
\begin{equation}
\text{SP}(d_S \to d_T) = \text{Corr}\left(\text{sim}(\phi_S, \phi_S'), \text{sim}(\phi_T, \phi_T')\right)
\end{equation}
\end{definition}

\begin{table}[h]
\centering
\caption{Semantic preservation for different target dimensions}
\label{tab:preservation}
\begin{tabular}{lcccc}
\toprule
\textbf{Source} & \textbf{4096-dim} & \textbf{6144-dim} & \textbf{7680-dim} & \textbf{8192-dim} \\
\midrule
DeepSeek-V3 (7168) & 94.2\% & 97.1\% & \textbf{98.3\%} & 98.5\% \\
Qwen3-72B (8192) & 91.8\% & 95.4\% & \textbf{97.8\%} & 100\% \\
Llama-3.3-70B (8192) & 91.5\% & 95.2\% & \textbf{97.6\%} & 100\% \\
Qwen3-32B (5120) & 97.1\% & 98.9\% & \textbf{99.2\%} & 99.3\% \\
\bottomrule
\end{tabular}
\end{table}

7680 dimensions preserves $>$97\% semantics from all major frontier models.

\subsection{Compression Efficiency}

Larger dimensions require more storage. We analyze compression ratios:

\begin{equation}
\text{Compression Ratio} = \frac{\text{Original Size}}{\text{Compressed Size}}
\end{equation}

\begin{table}[h]
\centering
\caption{BitDelta compression efficiency by dimension}
\label{tab:compression}
\begin{tabular}{rccc}
\toprule
\textbf{Dimensions} & \textbf{Original (bytes)} & \textbf{Compressed} & \textbf{Ratio} \\
\midrule
4,096 & 16,384 & 542 & 30.2$\times$ \\
6,144 & 24,576 & 789 & 31.1$\times$ \\
\textbf{7,680} & \textbf{30,720} & \textbf{964} & \textbf{31.87$\times$} \\
8,192 & 32,768 & 1,047 & 31.3$\times$ \\
\bottomrule
\end{tabular}
\end{table}

7680 dimensions achieves optimal compression ratio due to favorable factorization properties.