papers/hanzo-llm-gateway/sections/02-background-and-related-work.tex

\section{Background and Related Work}

\label{sec:background}

\subsection{LLM Provider Landscape}

The LLM provider ecosystem can be categorized along several dimensions. \textit{First-party providers} (OpenAI, Anthropic, Google) train and serve their own models. \textit{Cloud providers} (AWS Bedrock, Azure OpenAI, Google Vertex AI) offer managed access to both first-party and third-party models. \textit{Inference platforms} (Together AI, Fireworks, Groq, Replicate) specialize in serving open-weight models with optimized infrastructure. \textit{Self-hosted solutions} (vLLM~\cite{kwon2023vllm}, TGI~\cite{tgi2023}, Ollama) enable on-premises deployment.

Each category exhibits distinct operational characteristics. First-party providers offer the highest model quality but with opaque capacity management. Cloud providers offer enterprise compliance features but with additional latency from abstraction layers. Inference platforms offer cost-competitive serving of open models but with variable availability. Self-hosted solutions offer full control but require significant infrastructure investment.

\subsection{API Incompatibilities}

Despite OpenAI's de facto standard position, significant incompatibilities persist across providers:

\begin{itemize}[leftmargin=*]
    \item \textbf{Request format}: Anthropic uses XML-tagged system prompts; Google uses \texttt{contents} instead of \texttt{messages}; Cohere uses \texttt{chat\_history} and \texttt{message}.
    \item \textbf{Streaming}: OpenAI uses Server-Sent Events (SSE) with \texttt{data: [DONE]} termination; Anthropic uses typed SSE events (\texttt{content\_block\_delta}); Google uses JSON lines.
    \item \textbf{Function calling}: Tool/function schemas differ in parameter naming, response format, and multi-tool invocation semantics.
    \item \textbf{Error codes}: HTTP status codes and error payloads are provider-specific, complicating unified error handling.
    \item \textbf{Rate limiting}: Headers (\texttt{x-ratelimit-*}, \texttt{retry-after}) and strategies (token bucket, sliding window, concurrency limits) vary.
\end{itemize}

\subsection{Related Work}

LiteLLM~\cite{litellm2023} provides a Python SDK for multi-provider access but lacks server-side routing, caching, and cost tracking. Portkey~\cite{portkey2024} offers a commercial gateway with reliability features but limited routing intelligence. Martian~\cite{martian2024} focuses on model selection via quality benchmarks but does not address infrastructure concerns. Our gateway combines comprehensive provider normalization with production-grade routing, caching, and observability.