mirror of
https://github.com/lux-apps/papers.git
synced 2026-10-30 18:15:21 +00:00
zeekay
abf99d05fc
Restructured from flat .tex/.pdf files into per-paper subdirectories. Added Antje Worring as co-author on all papers. Synced PDFs to pdfs/ for download site.
58 lines
2.2 KiB
TeX
58 lines
2.2 KiB
TeX
\section{Operational Considerations}
|
|
\label{sec:ops}
|
|
%% ============================================================
|
|
|
|
\subsection{Monitoring}
|
|
|
|
The replicator exposes Prometheus metrics on \texttt{:9091/metrics}:
|
|
|
|
\begin{itemize}[nosep]
|
|
\item \texttt{replicate\_wal\_bytes\_total} --- total WAL bytes shipped.
|
|
\item \texttt{replicate\_wal\_segments\_total} --- total WAL segments
|
|
uploaded.
|
|
\item \texttt{replicate\_snapshot\_bytes\_total} --- total snapshot
|
|
bytes uploaded.
|
|
\item \texttt{replicate\_last\_sync\_seconds} --- time since last
|
|
successful sync.
|
|
\item \texttt{replicate\_restore\_duration\_seconds} --- histogram
|
|
of restore times.
|
|
\item \texttt{replicate\_encryption\_errors\_total} --- age
|
|
encryption failures.
|
|
\end{itemize}
|
|
|
|
An alert fires if \texttt{replicate\_last\_sync\_seconds} exceeds
|
|
$3\times$ the configured sync interval.
|
|
|
|
\subsection{Disaster Recovery}
|
|
|
|
\begin{enumerate}[nosep]
|
|
\item \textbf{Pod restart}: Init container restores from S3.
|
|
Automatic. No data loss.
|
|
\item \textbf{Node failure}: Pod rescheduled to new node. Init
|
|
container restores from S3. No data loss.
|
|
\item \textbf{S3 bucket loss}: Restore from S3 cross-region
|
|
replication (configured separately). RPO = cross-region
|
|
replication lag (typically < 15\,min).
|
|
\item \textbf{Master key loss}: Unrecoverable without KMS backup.
|
|
KMS itself uses Shamir secret sharing for its unseal key.
|
|
\item \textbf{Corrupted WAL}: Replicator detects checksum mismatch,
|
|
falls back to last valid snapshot, alerts operator.
|
|
\end{enumerate}
|
|
|
|
\subsection{Cost Model}
|
|
|
|
S3 storage cost for a typical ATS deployment (500 orgs, avg 50\,MB
|
|
per org DB, hourly snapshots, 1\,s WAL sync):
|
|
|
|
\begin{itemize}[nosep]
|
|
\item Snapshot storage: $500 \times 50\,\text{MB} \times 24 = 600\,\text{GB/day}$
|
|
(with 24h retention). At \$0.023/GB: \$13.80/day.
|
|
\item WAL storage: approximately $10\times$ DB size per day with
|
|
30-day retention. At \$0.023/GB: \$2.88/day.
|
|
\item Total: approximately \$500/month for 500 orgs with full
|
|
durability. Lifecycle policies garbage-collect snapshots older
|
|
than 30 days.
|
|
\end{itemize}
|
|
|
|
%% ============================================================
|