papers/hanzo-webhooks/sections/09-monitoring-and-observability.tex

\section{Monitoring and Observability}


\subsection{Metrics}

Key metrics tracked:

\begin{itemize}
    \item \texttt{webhook\_deliveries\_total}: Counter by status, event type
    \item \texttt{webhook\_delivery\_latency}: Histogram of delivery time
    \item \texttt{webhook\_retry\_queue\_size}: Gauge of pending retries
    \item \texttt{webhook\_endpoint\_health}: Gauge per endpoint
\end{itemize}

\subsection{Alerting}

Alerts trigger on:

\begin{itemize}
    \item First-attempt success rate < 95\%
    \item Retry queue size > 100,000
    \item Dead letter queue growth > 100/hour
    \item Delivery latency P99 > 10 seconds
\end{itemize}

\subsection{Dashboard}

Merchants access delivery status via dashboard:

\begin{lstlisting}[caption=Delivery Status API]
GET /v1/webhooks/{id}/deliveries
{
  "data": [
    {
      "id": "del_abc123",
      "event": "evt_xyz789",
      "status": "delivered",
      "attempts": 1,
      "delivered_at": "2019-06-15T10:30:00Z",
      "response_code": 200,
      "response_time_ms": 234
    },
    ...
  ]
}
\end{lstlisting}