papers/hanzo-llm-gateway/sections/07-observability.tex

\section{Observability}

\label{sec:observability}

\subsection{Metrics}

The gateway exports Prometheus-compatible metrics at four levels:

\begin{table}[t]
\centering
\caption{Key gateway metrics}
\label{tab:metrics}
\begin{tabular}{@{}ll@{}}
\toprule
\textbf{Metric} & \textbf{Labels} \\
\midrule
\texttt{llm\_request\_duration} & provider, model, status \\
\texttt{llm\_tokens\_total} & provider, model, direction \\
\texttt{llm\_cost\_usd} & provider, model, org \\
\texttt{llm\_cache\_hits} & model, cache\_tier \\
\texttt{llm\_errors\_total} & provider, error\_type \\
\texttt{llm\_ttft\_seconds} & provider, model \\
\texttt{llm\_provider\_health} & provider, endpoint \\
\texttt{llm\_queue\_depth} & priority\_level \\
\bottomrule
\end{tabular}
\end{table}

Time-to-first-token (TTFT) is tracked separately from total latency, as it directly impacts perceived responsiveness for streaming applications.

\subsection{Distributed Tracing}

Each request generates an OpenTelemetry trace span with the following structure:

\begin{enumerate}[leftmargin=*]
    \item \texttt{gateway.request}: Root span covering full request lifecycle.
    \item \texttt{gateway.route}: Routing decision with selected provider and reason.
    \item \texttt{gateway.cache}: Cache lookup with hit/miss result.
    \item \texttt{gateway.transform}: Request/response transformation.
    \item \texttt{gateway.provider}: Upstream provider call with full timing.
    \item \texttt{gateway.stream}: Streaming duration and chunk count.
\end{enumerate}

Traces are propagated using W3C Trace Context headers, enabling end-to-end tracing from client applications through the gateway to provider endpoints.

\subsection{Audit Logging}

All requests are logged with configurable verbosity. At the default level, logs include request metadata (model, provider, tokens, cost, latency) without prompt content. At elevated verbosity levels, full prompts and responses are logged with PII redaction applied via configurable regex patterns.