papers/hip-0302-replicate/sections/10-operational-concerns.tex

\section{Operational Concerns}


\subsection{Monitoring}

\texttt{replicate} exposes Prometheus metrics on port 9090:

\begin{table}[H]
\centering
\begin{tabular}{@{}ll@{}}
\toprule
\textbf{Metric} & \textbf{Description} \\
\midrule
\texttt{replicate\_wal\_bytes\_total}      & Total WAL bytes replicated \\
\texttt{replicate\_wal\_segments\_total}   & Total WAL segments uploaded \\
\texttt{replicate\_snapshot\_bytes\_total} & Total snapshot bytes uploaded \\
\texttt{replicate\_restore\_duration\_seconds} & Last restore duration \\
\texttt{replicate\_replication\_lag\_seconds}  & Time since last successful upload \\
\bottomrule
\end{tabular}
\caption{Prometheus metrics.}
\label{tab:metrics}
\end{table}

An alert fires if \texttt{replicate\_replication\_lag\_seconds} exceeds 10
seconds for any service.

\subsection{Failure Modes}

\begin{itemize}[nosep]
    \item \textbf{S3 unavailable.} \texttt{replicate} buffers WAL frames
    locally and retries with exponential backoff. The application continues
    operating. RPO degrades to the duration of the S3 outage.

    \item \textbf{Age identity missing.} The init container fails to restore.
    The pod enters CrashLoopBackOff. Alert fires. Operator provisions the
    identity from KMS.

    \item \textbf{Corrupt replica.} \texttt{replicate} validates snapshots
    via checksum on restore. A corrupt snapshot triggers fallback to the
    previous generation.
\end{itemize}

% ============================================================================
% 11. POST-QUANTUM SECURITY
% ============================================================================