papers/hanzo-llm-gateway/sections/11-discussion.tex

\section{Discussion}

\label{sec:discussion}

\subsection{Limitations}

\paragraph{Semantic cache precision.} While 97.2\% precision is high, the 2.8\% false positive rate means approximately 1 in 36 cache hits returns a subtly incorrect response. For safety-critical applications, we recommend disabling semantic caching or reducing the similarity threshold.

\paragraph{Provider-specific features.} Some provider-specific features (e.g., Anthropic's extended thinking, OpenAI's structured outputs with \texttt{strict: true}) require pass-through handling that bypasses normalization. The gateway supports a \texttt{provider\_params} escape hatch for such cases, but this breaks provider-agnostic routing.

\paragraph{Real-time pricing.} Cost tracking relies on a pricing database that must be manually updated when providers change prices. We are developing automated price scraping and notification systems to reduce this operational burden.

\subsection{Future Work}

\begin{itemize}[leftmargin=*]
    \item \textbf{Learned routing}: Training a lightweight model to predict optimal provider selection based on prompt features, replacing the heuristic scoring functions.
    \item \textbf{Speculative execution}: Dispatching requests to multiple providers simultaneously and returning the first successful response, trading cost for latency.
    \item \textbf{Prompt optimization}: Automatically reformatting prompts for provider-specific strengths (e.g., adding chain-of-thought markers for models that benefit from them).
    \item \textbf{Federated deployment}: Multi-region gateway instances with global routing for compliance and latency optimization.
\end{itemize}