mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
paper: HIP-0302 add post-quantum considerations section
Add harvest-now-decrypt-later threat analysis and X-Wing hybrid KEM migration roadmap (X25519 + ML-KEM-768) for age encryption layer in Hanzo Replicate proposal.
This commit is contained in:
@@ -0,0 +1,885 @@
|
||||
% HIP-0302: Hanzo Replicate — Encrypted SQLite Durability for Base Services
|
||||
\documentclass[11pt]{article}
|
||||
\usepackage{shared/hanzocover}
|
||||
\usepackage[utf8]{inputenc}
|
||||
\usepackage{amsmath,amssymb}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{hyperref}
|
||||
\usepackage{enumitem}
|
||||
\usepackage{xcolor}
|
||||
\usepackage{float}
|
||||
\usepackage{multirow}
|
||||
\usepackage{listings}
|
||||
\input{shared/lstlang}
|
||||
\usepackage{tikz}
|
||||
\usetikzlibrary{shapes,arrows,positioning,fit}
|
||||
|
||||
\hypersetup{colorlinks=true,linkcolor=black,citecolor=blue,urlcolor=blue}
|
||||
|
||||
\definecolor{codegreen}{rgb}{0,0.6,0}
|
||||
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
|
||||
\definecolor{codepurple}{rgb}{0.58,0,0.82}
|
||||
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}
|
||||
|
||||
\lstdefinestyle{mystyle}{
|
||||
backgroundcolor=\color{backcolour},
|
||||
commentstyle=\color{codegreen},
|
||||
keywordstyle=\color{codepurple},
|
||||
numberstyle=\tiny\color{codegray},
|
||||
stringstyle=\color{codegreen},
|
||||
basicstyle=\ttfamily\footnotesize,
|
||||
breakatwhitespace=false,
|
||||
breaklines=true,
|
||||
captionpos=b,
|
||||
keepspaces=true,
|
||||
numbers=left,
|
||||
numbersep=5pt,
|
||||
showspaces=false,
|
||||
showstringspaces=false,
|
||||
showtabs=false,
|
||||
tabsize=2
|
||||
}
|
||||
\lstset{style=mystyle}
|
||||
|
||||
\title{HIP-0302: Hanzo Replicate\\[0.3em]
|
||||
{\large Encrypted SQLite Durability\\for Base Services}}
|
||||
\author{
|
||||
Zach Kelling \\
|
||||
\textit{Hanzo Industries Inc (Techstars '17)} \\
|
||||
\texttt{[email protected]}
|
||||
}
|
||||
\date{April 2026}
|
||||
|
||||
\begin{document}
|
||||
\hanzocoverpage
|
||||
|
||||
% ============================================================================
|
||||
% METADATA
|
||||
% ============================================================================
|
||||
\noindent
|
||||
\begin{tabular}{@{}ll}
|
||||
\textbf{HIP} & 0302 \\
|
||||
\textbf{Title} & Hanzo Replicate --- Encrypted SQLite Durability for Base Services \\
|
||||
\textbf{Author} & Zach Kelling (\texttt{[email protected]}) \\
|
||||
\textbf{Type} & Standards Track \\
|
||||
\textbf{Category} & Infrastructure \\
|
||||
\textbf{Status} & Draft \\
|
||||
\textbf{Created} & 2026-04-09 \\
|
||||
\textbf{Requires} & HIP-0027 (KMS), HIP-0032 (Object Storage), HIP-0065 (Backup \& DR) \\
|
||||
\end{tabular}
|
||||
|
||||
\vspace{1em}
|
||||
|
||||
% ============================================================================
|
||||
% ABSTRACT
|
||||
% ============================================================================
|
||||
\begin{abstract}
|
||||
This proposal specifies how all Hanzo Base-powered services achieve durable
|
||||
state persistence through continuous encrypted WAL replication to
|
||||
S3-compatible storage. The system comprises two packages:
|
||||
\texttt{hanzoai/replicate} (a rebranded Litestream with age encryption
|
||||
support) and \texttt{hanzoai/age} (X25519/Curve25519 file encryption). Every
|
||||
Base service---IAM, KMS, ATS, BD, TA, Tasks, and all future
|
||||
services---deploys \texttt{replicate} as a Kubernetes sidecar that
|
||||
continuously streams encrypted WAL frames to \texttt{hanzoai/s3}. The result
|
||||
is sub-second RPO, seconds-scale RTO, and zero dependence on centralized
|
||||
databases or persistent volume durability.
|
||||
\end{abstract}
|
||||
|
||||
% ============================================================================
|
||||
% 1. MOTIVATION
|
||||
% ============================================================================
|
||||
\section{Motivation}
|
||||
|
||||
Hanzo Base (HIP Base Runtime) packages an encrypted SQLite database per
|
||||
organization into a single Go binary. This architecture delivers
|
||||
sub-millisecond CRUD latency, per-tenant isolation by construction, and zero
|
||||
external database dependencies. It is the right architecture for sovereign
|
||||
applications.
|
||||
|
||||
It has one gap: durability on ephemeral infrastructure.
|
||||
|
||||
Kubernetes pods are ephemeral. A node drain, an OOM kill, a failed health
|
||||
check, or a routine deployment all destroy local state. Current Base
|
||||
deployments use either \texttt{emptyDir} volumes (state lost on every
|
||||
restart) or PersistentVolumeClaims (state survives restart but not node
|
||||
failure, and PVCs introduce scheduling constraints that conflict with
|
||||
horizontal scaling).
|
||||
|
||||
The consequences are concrete:
|
||||
|
||||
\begin{enumerate}[nosep]
|
||||
\item \textbf{Data loss on restart.} An \texttt{emptyDir} deployment of
|
||||
ATS loses all trade records, order history, and portfolio state when the
|
||||
pod restarts. This is unacceptable for a regulated securities exchange.
|
||||
|
||||
\item \textbf{Node affinity locks.} A PVC-backed deployment pins a pod to
|
||||
one node. If that node fails, the pod cannot reschedule until the volume
|
||||
is manually detached or the cloud provider's volume controller completes
|
||||
failover---typically 2--10 minutes on GKE.
|
||||
|
||||
\item \textbf{No disaster recovery.} Neither \texttt{emptyDir} nor PVC
|
||||
provides off-site backup. A storage controller bug, a zone outage, or
|
||||
accidental \texttt{kubectl delete pvc} destroys all state with no
|
||||
recovery path.
|
||||
|
||||
\item \textbf{No cross-region replication.} PVCs are zone-scoped. A zone
|
||||
failure takes down both the pod and its data. Regional redundancy requires
|
||||
an external replication mechanism.
|
||||
\end{enumerate}
|
||||
|
||||
This HIP solves all four problems with a single mechanism: continuous WAL
|
||||
replication to S3-compatible object storage, encrypted with age before
|
||||
leaving the pod.
|
||||
|
||||
% ============================================================================
|
||||
% 2. BACKGROUND
|
||||
% ============================================================================
|
||||
\section{Background}
|
||||
|
||||
\subsection{Hanzo Base Architecture}
|
||||
|
||||
Hanzo Base runs one SQLite database per organization. Each database is
|
||||
encrypted at rest with sqlcipher (AES-256-CBC, PBKDF2-derived key). The
|
||||
encryption key is stored in Hanzo KMS (HIP-0027) and fetched at startup.
|
||||
Base exposes a REST API, a realtime SSE subscription layer, and a JavaScript
|
||||
plugin runtime (Goja VM). All data operations go through SQLite's WAL
|
||||
(Write-Ahead Log) mode.
|
||||
|
||||
WAL mode is critical to this proposal. In WAL mode, SQLite writes all
|
||||
changes to a separate WAL file before checkpointing them into the main
|
||||
database. This creates a sequential, append-only stream of changes---exactly
|
||||
the primitive that continuous replication requires.
|
||||
|
||||
\subsection{Litestream}
|
||||
|
||||
Litestream~\cite{litestream} is an open-source tool by Ben Johnson that
|
||||
continuously replicates SQLite databases to S3-compatible storage. It works
|
||||
by monitoring the SQLite WAL file and streaming new frames to a replica as
|
||||
they are written. On restore, it downloads the most recent snapshot and
|
||||
replays WAL frames to reconstruct the database to its last replicated state.
|
||||
|
||||
Litestream operates as a separate process alongside the application. It
|
||||
requires no application code changes, no special SQLite build, and no
|
||||
coordination protocol. It reads the WAL file directly from the filesystem.
|
||||
|
||||
\subsection{Beeper's Encryption Fork}
|
||||
|
||||
Beeper (acquired by Automattic, 2024) forked Litestream to add age
|
||||
encryption for their Matrix bridge infrastructure. Their fork encrypts each
|
||||
WAL segment and snapshot with an age recipient before uploading to S3. This
|
||||
ensures that the S3 operator (or any party with bucket access) cannot read
|
||||
the replicated data without the age identity (private key).
|
||||
|
||||
Beeper's fork demonstrated the viability of encrypted Litestream replication
|
||||
in production at scale (millions of Matrix bridge databases). However, it
|
||||
was never upstreamed and has diverged from mainline Litestream.
|
||||
|
||||
\subsection{age Encryption}
|
||||
|
||||
age~\cite{age} is a file encryption tool by Filippo Valsorda designed as a
|
||||
modern replacement for PGP. It uses X25519 for key agreement, ChaCha20-Poly1305
|
||||
for authenticated encryption, and HKDF for key derivation. age has three
|
||||
properties that make it suitable for this use case:
|
||||
|
||||
\begin{enumerate}[nosep]
|
||||
\item \textbf{Small keys.} An age identity (private key) is a single
|
||||
62-character string. A recipient (public key) is a single 59-character
|
||||
string. Both fit in a Kubernetes Secret without base64 bloat.
|
||||
|
||||
\item \textbf{Composable recipients.} A file can be encrypted to multiple
|
||||
recipients. This supports key rotation (encrypt to both old and new keys)
|
||||
and escrow (encrypt to both service key and recovery key).
|
||||
|
||||
\item \textbf{No configuration.} age has no cipher suites, no key
|
||||
formats, no modes. There is one algorithm. There is one way.
|
||||
\end{enumerate}
|
||||
|
||||
% ============================================================================
|
||||
% 3. SPECIFICATION
|
||||
% ============================================================================
|
||||
\section{Specification}
|
||||
|
||||
\subsection{Components}
|
||||
|
||||
The replication system consists of three components:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{@{}lll@{}}
|
||||
\toprule
|
||||
\textbf{Component} & \textbf{Image} & \textbf{Purpose} \\
|
||||
\midrule
|
||||
\texttt{hanzoai/replicate} & \texttt{ghcr.io/hanzoai/replicate} & WAL replication + age encryption \\
|
||||
\texttt{hanzoai/age} & \texttt{ghcr.io/hanzoai/age} & X25519 key generation \\
|
||||
\texttt{hanzoai/s3} & \texttt{ghcr.io/hanzoai/s3} & S3-compatible object storage \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{System components.}
|
||||
\label{tab:components}
|
||||
\end{table}
|
||||
|
||||
\texttt{hanzoai/replicate} is a fork of Litestream incorporating Beeper's
|
||||
age encryption patches, rebased onto mainline Litestream, and maintained as
|
||||
part of the Hanzo ecosystem. The binary is called \texttt{replicate}. It
|
||||
accepts the same configuration format as Litestream with one addition: an
|
||||
\texttt{age} block specifying identities and recipients.
|
||||
|
||||
\texttt{hanzoai/age} is a container image packaging the \texttt{age-keygen}
|
||||
binary for key generation in init containers and CI pipelines.
|
||||
|
||||
\texttt{hanzoai/s3} (HIP-0032) provides the S3-compatible storage backend.
|
||||
|
||||
\subsection{Replication Protocol}
|
||||
|
||||
The replication lifecycle for a single SQLite database:
|
||||
|
||||
\begin{enumerate}[nosep]
|
||||
\item \textbf{Startup.} \texttt{replicate} reads the configuration file,
|
||||
connects to the S3 endpoint, and checks for an existing replica.
|
||||
|
||||
\item \textbf{Restore (if needed).} If the local database does not exist
|
||||
and a replica exists in S3, \texttt{replicate} downloads the most recent
|
||||
snapshot, decrypts it with the age identity, replays all subsequent WAL
|
||||
segments (each decrypted individually), and writes the restored database
|
||||
to the configured path.
|
||||
|
||||
\item \textbf{Monitor.} \texttt{replicate} monitors the SQLite WAL file
|
||||
using \texttt{inotify} (Linux) or \texttt{kqueue} (macOS). When new
|
||||
frames are written, it reads them from the WAL.
|
||||
|
||||
\item \textbf{Encrypt.} Each WAL segment is encrypted with the
|
||||
configured age recipients before upload. Encryption happens in memory.
|
||||
Plaintext WAL data never touches S3.
|
||||
|
||||
\item \textbf{Upload.} The encrypted WAL segment is uploaded to S3 at the
|
||||
configured path. The upload is atomic (multipart with commit).
|
||||
|
||||
\item \textbf{Snapshot.} Periodically (default: every 24 hours),
|
||||
\texttt{replicate} takes a full snapshot of the database, encrypts it,
|
||||
and uploads it to S3. Old snapshots are retained according to the
|
||||
configured retention policy.
|
||||
|
||||
\item \textbf{Shutdown.} On SIGTERM, \texttt{replicate} flushes any
|
||||
pending WAL frames, uploads them, and exits cleanly. Kubernetes sends
|
||||
SIGTERM before SIGKILL, so graceful shutdown is the normal path.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Encryption Layers}
|
||||
|
||||
Data is protected by three independent encryption layers:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{@{}llll@{}}
|
||||
\toprule
|
||||
\textbf{Layer} & \textbf{Mechanism} & \textbf{Scope} & \textbf{Protects Against} \\
|
||||
\midrule
|
||||
1. Disk & sqlcipher (AES-256-CBC) & Database at rest & Local filesystem access \\
|
||||
2. S3 & age (X25519 + ChaCha20) & Replicas in storage & S3 operator, bucket leak \\
|
||||
3. Wire & TLS 1.3 & In transit & Network interception \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Encryption layers. Each layer is independent---compromise of one
|
||||
does not compromise the others.}
|
||||
\label{tab:encryption}
|
||||
\end{table}
|
||||
|
||||
Layer 1 (sqlcipher) is managed by Hanzo Base. Layer 2 (age) is managed by
|
||||
\texttt{replicate}. Layer 3 (TLS) is managed by the S3 endpoint. This HIP
|
||||
specifies Layer 2.
|
||||
|
||||
\subsection{Recovery Objectives}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{@{}lll@{}}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Target} & \textbf{Mechanism} \\
|
||||
\midrule
|
||||
RPO (Recovery Point Objective) & $\leq$ 1 second & Continuous WAL streaming \\
|
||||
RTO (Recovery Time Objective) & $\leq$ 30 seconds & Snapshot restore + WAL replay \\
|
||||
Retention & 72 hours WAL, 30 days snapshots & S3 lifecycle policy \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Recovery objectives.}
|
||||
\label{tab:recovery}
|
||||
\end{table}
|
||||
|
||||
The 1-second RPO is achieved by the default \texttt{sync-interval} of 1
|
||||
second. WAL frames generated within the last sync interval may be lost on
|
||||
unclean shutdown (SIGKILL without preceding SIGTERM). In practice,
|
||||
Kubernetes provides a 30-second grace period between SIGTERM and SIGKILL, so
|
||||
data loss on clean shutdown is zero.
|
||||
|
||||
The 30-second RTO accounts for S3 download latency for the most recent
|
||||
snapshot (typically 1--5 seconds for databases under 1~GiB) plus WAL replay
|
||||
time (linear in the number of frames since the last snapshot).
|
||||
|
||||
% ============================================================================
|
||||
% 4. CONFIGURATION
|
||||
% ============================================================================
|
||||
\section{Configuration}
|
||||
|
||||
\subsection{Replicate Configuration File}
|
||||
|
||||
The configuration file follows the Litestream YAML format with the addition
|
||||
of age encryption fields.
|
||||
|
||||
\begin{lstlisting}[language=YAML,caption={replicate.yml --- canonical configuration.}]
|
||||
dbs:
|
||||
- path: /data/base/data.db
|
||||
replicas:
|
||||
- type: s3
|
||||
bucket: hanzo-replicate
|
||||
path: ${SERVICE}/${NAMESPACE}/${POD_NAME}
|
||||
endpoint: s3.hanzo.ai
|
||||
region: us-east-1
|
||||
sync-interval: 1s
|
||||
snapshot-interval: 24h
|
||||
retention: 72h
|
||||
retention-check-interval: 1h
|
||||
age:
|
||||
identities: /secrets/age/identity.txt
|
||||
recipients: /secrets/age/recipients.txt
|
||||
\end{lstlisting}
|
||||
|
||||
\subsection{Environment Variables}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{@{}lll@{}}
|
||||
\toprule
|
||||
\textbf{Variable} & \textbf{Required} & \textbf{Purpose} \\
|
||||
\midrule
|
||||
\texttt{LITESTREAM\_ACCESS\_KEY\_ID} & Yes & S3 access key \\
|
||||
\texttt{LITESTREAM\_SECRET\_ACCESS\_KEY} & Yes & S3 secret key \\
|
||||
\texttt{SERVICE} & Yes & Service name (ats, iam, kms, ...) \\
|
||||
\texttt{NAMESPACE} & Yes & K8s namespace \\
|
||||
\texttt{POD\_NAME} & Yes & K8s pod name (from fieldRef) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Required environment variables.}
|
||||
\label{tab:env}
|
||||
\end{table}
|
||||
|
||||
S3 credentials are stored in Hanzo KMS (HIP-0027) and injected as
|
||||
Kubernetes Secrets. They are never committed to manifests.
|
||||
|
||||
\subsection{S3 Path Structure}
|
||||
|
||||
Replicas are organized by service, namespace, and pod:
|
||||
|
||||
\begin{lstlisting}[language=bash,caption={S3 path layout.}]
|
||||
s3://hanzo-replicate/
|
||||
ats/acme/ats-0/
|
||||
generations/
|
||||
00000001/ # snapshot + WAL segments
|
||||
00000002/
|
||||
iam/acme/iam-0/
|
||||
generations/
|
||||
00000001/
|
||||
kms/acme/kms-0/
|
||||
generations/
|
||||
00000001/
|
||||
\end{lstlisting}
|
||||
|
||||
The three-level prefix (\texttt{service/namespace/pod}) ensures per-tenant
|
||||
isolation. No service can read another service's replicas without the
|
||||
corresponding age identity.
|
||||
|
||||
% ============================================================================
|
||||
% 5. KEY MANAGEMENT
|
||||
% ============================================================================
|
||||
\section{Key Management}
|
||||
|
||||
\subsection{Key Hierarchy}
|
||||
|
||||
\begin{figure}[H]
|
||||
\centering
|
||||
\begin{tikzpicture}[
|
||||
node distance=1.2cm,
|
||||
box/.style={rectangle, draw, minimum width=3cm, minimum height=0.7cm,
|
||||
align=center, font=\footnotesize},
|
||||
arrow/.style={->, thick}
|
||||
]
|
||||
\node[box] (master) {Master Key\\(Hanzo KMS)};
|
||||
\node[box, below left=1.2cm and 0.5cm of master] (iam-key) {IAM Identity\\(age)};
|
||||
\node[box, below=1.2cm of master] (ats-key) {ATS Identity\\(age)};
|
||||
\node[box, below right=1.2cm and 0.5cm of master] (kms-key) {KMS Identity\\(age)};
|
||||
\node[box, below=1.2cm of iam-key] (iam-recip) {IAM Recipients};
|
||||
\node[box, below=1.2cm of ats-key] (ats-recip) {ATS Recipients};
|
||||
\node[box, below=1.2cm of kms-key] (kms-recip) {KMS Recipients};
|
||||
|
||||
\draw[arrow] (master) -- (iam-key) node[midway,left,font=\scriptsize] {HKDF};
|
||||
\draw[arrow] (master) -- (ats-key) node[midway,left,font=\scriptsize] {HKDF};
|
||||
\draw[arrow] (master) -- (kms-key) node[midway,right,font=\scriptsize] {HKDF};
|
||||
\draw[arrow] (iam-key) -- (iam-recip);
|
||||
\draw[arrow] (ats-key) -- (ats-recip);
|
||||
\draw[arrow] (kms-key) -- (kms-recip);
|
||||
\end{tikzpicture}
|
||||
\caption{Key derivation hierarchy. Per-service age identities are derived
|
||||
from the master key via HKDF.}
|
||||
\label{fig:keys}
|
||||
\end{figure}
|
||||
|
||||
\begin{enumerate}[nosep]
|
||||
\item \textbf{Master encryption key.} Stored in Hanzo KMS (HIP-0027).
|
||||
This is the root of trust for all replicate encryption. It is a 256-bit
|
||||
symmetric key used as HKDF input keying material.
|
||||
|
||||
\item \textbf{Per-service age keypairs.} Derived from the master key via
|
||||
HKDF with the service name as the info parameter:
|
||||
\texttt{HKDF-SHA256(master, salt="hanzo-replicate", info=service\_name)}.
|
||||
The derived 32 bytes are used as the X25519 private key (age identity).
|
||||
The corresponding public key is the age recipient.
|
||||
|
||||
\item \textbf{Recipients file.} Contains the public keys (recipients) of
|
||||
all parties authorized to decrypt the replica. For normal operation, this
|
||||
is the service's own recipient. For disaster recovery, this includes a
|
||||
recovery recipient held offline.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Key Rotation}
|
||||
|
||||
Key rotation proceeds without downtime:
|
||||
|
||||
\begin{enumerate}[nosep]
|
||||
\item Generate a new age keypair (derived from a new HKDF info or a
|
||||
rotated master key).
|
||||
\item Add the new recipient to the recipients file. The old recipient
|
||||
remains.
|
||||
\item Restart the \texttt{replicate} sidecar. New WAL segments and
|
||||
snapshots are encrypted to both old and new recipients.
|
||||
\item After the retention period expires (72 hours for WAL, 30 days for
|
||||
snapshots), all surviving replicas are encrypted to the new recipient.
|
||||
\item Remove the old recipient from the recipients file.
|
||||
\item The old identity is retained in KMS for historical reads. It is
|
||||
marked as \texttt{rotated} but never deleted.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Emergency Recovery}
|
||||
|
||||
Full restore requires two keys: the master key from Hanzo KMS and the
|
||||
per-service age identity derived from it. The recovery procedure:
|
||||
|
||||
\begin{enumerate}[nosep]
|
||||
\item Retrieve the master key from KMS (requires KMS admin
|
||||
authentication).
|
||||
\item Derive the service identity:
|
||||
\texttt{age-keygen -o identity.txt} from the HKDF output.
|
||||
\item Run \texttt{replicate restore -config replicate.yml
|
||||
/data/base/data.db}.
|
||||
\item The restored database is encrypted with sqlcipher. The sqlcipher key
|
||||
is also in KMS.
|
||||
\end{enumerate}
|
||||
|
||||
If KMS itself is unavailable (catastrophic failure), the KMS database is
|
||||
restored first using the offline recovery identity (stored in a physical
|
||||
safe or HSM).
|
||||
|
||||
% ============================================================================
|
||||
% 6. DEPLOYMENT
|
||||
% ============================================================================
|
||||
\section{Deployment}
|
||||
|
||||
\subsection{Kubernetes Sidecar Pattern}
|
||||
|
||||
Every Base service deploys \texttt{replicate} as an init container (restore)
|
||||
and a sidecar container (continuous replication).
|
||||
|
||||
\begin{lstlisting}[language=YAML,caption={Kubernetes pod spec (abbreviated).}]
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: ats
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
initContainers:
|
||||
- name: replicate-restore
|
||||
image: ghcr.io/hanzoai/replicate:latest
|
||||
args:
|
||||
- restore
|
||||
- -if-db-not-exists
|
||||
- -if-replica-exists
|
||||
- -config
|
||||
- /etc/replicate/replicate.yml
|
||||
- /data/base/data.db
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data/base
|
||||
- name: replicate-config
|
||||
mountPath: /etc/replicate
|
||||
- name: age-secrets
|
||||
mountPath: /secrets/age
|
||||
readOnly: true
|
||||
resources:
|
||||
requests: { memory: 16Mi, cpu: 10m }
|
||||
limits: { memory: 64Mi, cpu: 100m }
|
||||
containers:
|
||||
- name: base
|
||||
image: ghcr.io/hanzoai/ats:latest
|
||||
# ... service container
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data/base
|
||||
- name: replicate
|
||||
image: ghcr.io/hanzoai/replicate:latest
|
||||
args:
|
||||
- replicate
|
||||
- -config
|
||||
- /etc/replicate/replicate.yml
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data/base
|
||||
- name: replicate-config
|
||||
mountPath: /etc/replicate
|
||||
- name: age-secrets
|
||||
mountPath: /secrets/age
|
||||
readOnly: true
|
||||
env:
|
||||
- name: LITESTREAM_ACCESS_KEY_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: s3-credentials
|
||||
key: access-key-id
|
||||
- name: LITESTREAM_SECRET_ACCESS_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: s3-credentials
|
||||
key: secret-access-key
|
||||
resources:
|
||||
requests: { memory: 16Mi, cpu: 10m }
|
||||
limits: { memory: 64Mi, cpu: 100m }
|
||||
volumes:
|
||||
- name: data
|
||||
emptyDir: {}
|
||||
- name: replicate-config
|
||||
configMap:
|
||||
name: replicate-config
|
||||
- name: age-secrets
|
||||
secret:
|
||||
secretName: replicate-age-keys
|
||||
\end{lstlisting}
|
||||
|
||||
Note: the \texttt{data} volume is \texttt{emptyDir}. PVCs are no longer
|
||||
required. On pod startup, the init container restores the database from S3.
|
||||
During operation, the sidecar continuously replicates. On pod termination,
|
||||
the sidecar flushes pending WAL frames before exiting.
|
||||
|
||||
\subsection{Resource Requirements}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{@{}lllll@{}}
|
||||
\toprule
|
||||
\textbf{Container} & \textbf{CPU Req} & \textbf{CPU Lim} & \textbf{Mem Req} & \textbf{Mem Lim} \\
|
||||
\midrule
|
||||
replicate-restore (init) & 10m & 100m & 16Mi & 64Mi \\
|
||||
replicate (sidecar) & 10m & 100m & 16Mi & 64Mi \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Resource requirements. The sidecar is lightweight---it reads the WAL file
|
||||
and uploads compressed, encrypted chunks.}
|
||||
\label{tab:resources}
|
||||
\end{table}
|
||||
|
||||
\subsection{Multi-Database Services}
|
||||
|
||||
Services with per-org databases (ATS, BD, TA) replicate multiple SQLite
|
||||
files. The \texttt{replicate.yml} configuration supports multiple
|
||||
\texttt{dbs} entries. A filesystem watcher discovers new databases created at
|
||||
runtime and adds them to the replication set dynamically.
|
||||
|
||||
\begin{lstlisting}[language=YAML,caption={Multi-database configuration.}]
|
||||
dbs:
|
||||
- path: /data/base/data.db
|
||||
replicas:
|
||||
- type: s3
|
||||
bucket: hanzo-replicate
|
||||
path: ats/acme/ats-0/system
|
||||
# ... same age config
|
||||
- path: /data/base/orgs/*.db
|
||||
replicas:
|
||||
- type: s3
|
||||
bucket: hanzo-replicate
|
||||
path: ats/acme/ats-0/orgs
|
||||
# ... same age config
|
||||
\end{lstlisting}
|
||||
|
||||
% ============================================================================
|
||||
% 7. AFFECTED SERVICES
|
||||
% ============================================================================
|
||||
\section{Affected Services}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{@{}lllll@{}}
|
||||
\toprule
|
||||
\textbf{Service} & \textbf{Database(s)} & \textbf{Typical Size} & \textbf{PVC} & \textbf{Replicate} \\
|
||||
\midrule
|
||||
IAM & iam.db & 50--200 MiB & 1 GiB & Yes \\
|
||||
KMS & kms.db (sqlcipher) & 10--50 MiB & 1 GiB & Yes \\
|
||||
ATS & data.db + per-org .db & 1--20 GiB & 20 GiB & Yes \\
|
||||
BD & data.db + per-org .db & 1--10 GiB & 20 GiB & Yes \\
|
||||
TA & data.db + per-org .db & 500 MiB--5 GiB & 20 GiB & Yes \\
|
||||
Tasks & tasks.db & 100--500 MiB & 5 GiB & Yes \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Affected services. The PVC column shows the current allocation; with
|
||||
replicate, PVCs are replaced by \texttt{emptyDir} and the listed sizes
|
||||
become the S3 storage consumption.}
|
||||
\label{tab:services}
|
||||
\end{table}
|
||||
|
||||
All services listed above are \textbf{required} to deploy the replicate
|
||||
sidecar. New Base services created after this HIP is finalized inherit the
|
||||
requirement automatically.
|
||||
|
||||
% ============================================================================
|
||||
% 8. SECURITY
|
||||
% ============================================================================
|
||||
\section{Security Analysis}
|
||||
|
||||
\subsection{Threat Model}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{@{}lll@{}}
|
||||
\toprule
|
||||
\textbf{Threat} & \textbf{Mitigation} & \textbf{Layer} \\
|
||||
\midrule
|
||||
S3 bucket compromise & age encryption & Layer 2 \\
|
||||
S3 operator reads replicas & age encryption & Layer 2 \\
|
||||
Network interception & TLS 1.3 to S3 endpoint & Layer 3 \\
|
||||
Pod filesystem access & sqlcipher encryption & Layer 1 \\
|
||||
Age identity theft & KMS access control, HKDF & Key mgmt \\
|
||||
Key loss & Offline recovery identity & Key mgmt \\
|
||||
Replay attack on WAL & Generation counters, HMAC & Protocol \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Threat model and mitigations.}
|
||||
\label{tab:threats}
|
||||
\end{table}
|
||||
|
||||
\subsection{Isolation Guarantees}
|
||||
|
||||
Per-org isolation is maintained at three levels:
|
||||
|
||||
\begin{enumerate}[nosep]
|
||||
\item \textbf{Database level.} Each org has a separate SQLite file with a
|
||||
separate sqlcipher key. Compromise of one org's sqlcipher key does not
|
||||
expose another org's data.
|
||||
|
||||
\item \textbf{S3 path level.} Each org's replica is stored at a distinct
|
||||
S3 prefix. S3 bucket policies restrict access by prefix.
|
||||
|
||||
\item \textbf{Encryption level.} Each service has its own age identity.
|
||||
The ATS age identity cannot decrypt IAM replicas. Cross-service access
|
||||
requires the master key.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{What This HIP Does Not Cover}
|
||||
|
||||
\begin{itemize}[nosep]
|
||||
\item \textbf{Application-level backup testing.} HIP-0065 specifies
|
||||
automated restore verification. This HIP defers to HIP-0065 for backup
|
||||
testing schedules and alerting.
|
||||
|
||||
\item \textbf{Cross-region replication.} S3 bucket replication to a
|
||||
second region is an operational concern configured at the
|
||||
\texttt{hanzoai/s3} level (HIP-0032), not at the replicate level.
|
||||
|
||||
\item \textbf{PostgreSQL replication.} Services using PostgreSQL
|
||||
(HIP-0029) use WAL-G or pgBackRest. This HIP applies only to SQLite
|
||||
databases managed by Hanzo Base.
|
||||
\end{itemize}
|
||||
|
||||
% ============================================================================
|
||||
% 9. MIGRATION
|
||||
% ============================================================================
|
||||
\section{Migration}
|
||||
|
||||
\subsection{Existing PVC-Backed Services}
|
||||
|
||||
Services currently using PVCs migrate to replicate in three steps:
|
||||
|
||||
\begin{enumerate}[nosep]
|
||||
\item \textbf{Bootstrap.} Deploy the replicate sidecar alongside the
|
||||
existing PVC-backed pod. The sidecar begins replicating the PVC-resident
|
||||
database to S3. Wait for the initial snapshot to complete.
|
||||
|
||||
\item \textbf{Cutover.} Redeploy the StatefulSet with \texttt{emptyDir}
|
||||
instead of PVC. The init container restores from S3. Verify the restored
|
||||
database matches the PVC-resident database (row count, checksum).
|
||||
|
||||
\item \textbf{Cleanup.} Delete the PVCs. Update the service's Kustomize
|
||||
overlay to remove PVC-related resources.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{New Services}
|
||||
|
||||
New Base services created after this HIP include the replicate sidecar in
|
||||
their initial Kustomize base. No migration required.
|
||||
|
||||
% ============================================================================
|
||||
% 10. OPERATIONAL CONCERNS
|
||||
% ============================================================================
|
||||
\section{Operational Concerns}
|
||||
|
||||
\subsection{Monitoring}
|
||||
|
||||
\texttt{replicate} exposes Prometheus metrics on port 9090:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{@{}ll@{}}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Description} \\
|
||||
\midrule
|
||||
\texttt{replicate\_wal\_bytes\_total} & Total WAL bytes replicated \\
|
||||
\texttt{replicate\_wal\_segments\_total} & Total WAL segments uploaded \\
|
||||
\texttt{replicate\_snapshot\_bytes\_total} & Total snapshot bytes uploaded \\
|
||||
\texttt{replicate\_restore\_duration\_seconds} & Last restore duration \\
|
||||
\texttt{replicate\_replication\_lag\_seconds} & Time since last successful upload \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Prometheus metrics.}
|
||||
\label{tab:metrics}
|
||||
\end{table}
|
||||
|
||||
An alert fires if \texttt{replicate\_replication\_lag\_seconds} exceeds 10
|
||||
seconds for any service.
|
||||
|
||||
\subsection{Failure Modes}
|
||||
|
||||
\begin{itemize}[nosep]
|
||||
\item \textbf{S3 unavailable.} \texttt{replicate} buffers WAL frames
|
||||
locally and retries with exponential backoff. The application continues
|
||||
operating. RPO degrades to the duration of the S3 outage.
|
||||
|
||||
\item \textbf{Age identity missing.} The init container fails to restore.
|
||||
The pod enters CrashLoopBackOff. Alert fires. Operator provisions the
|
||||
identity from KMS.
|
||||
|
||||
\item \textbf{Corrupt replica.} \texttt{replicate} validates snapshots
|
||||
via checksum on restore. A corrupt snapshot triggers fallback to the
|
||||
previous generation.
|
||||
\end{itemize}
|
||||
|
||||
% ============================================================================
|
||||
% 11. POST-QUANTUM CONSIDERATIONS
|
||||
% ============================================================================
|
||||
\section{Post-Quantum Considerations}
|
||||
|
||||
The age encryption layer (Layer~2) currently uses X25519 (Curve25519)
|
||||
key agreement, which is not post-quantum safe. A sufficiently powerful
|
||||
quantum computer running Shor's algorithm could recover X25519 private
|
||||
keys from public keys, retroactively decrypting all age-encrypted S3
|
||||
objects.
|
||||
|
||||
\subsection{Harvest-Now-Decrypt-Later Threat}
|
||||
|
||||
S3 replicas are long-lived. Snapshots are retained for 30 days by
|
||||
default; compliance-critical buckets may retain data for years.
|
||||
An adversary who captures encrypted S3 objects today could decrypt
|
||||
them once a cryptographically relevant quantum computer (CRQC) becomes
|
||||
available. This ``harvest-now-decrypt-later'' attack is the primary
|
||||
quantum threat to this replication standard.
|
||||
|
||||
\subsection{Mitigation: X-Wing Hybrid KEM}
|
||||
|
||||
The mitigation is a hybrid Key Encapsulation Mechanism (KEM) called
|
||||
X-Wing~\cite{xwing}, which combines X25519 with ML-KEM-768 (NIST FIPS
|
||||
203~\cite{fips203}). If quantum computers never materialize, the X25519
|
||||
component provides security. If they do, the ML-KEM-768 component
|
||||
provides post-quantum security. Both must be broken to compromise the
|
||||
hybrid.
|
||||
|
||||
\texttt{lux/crypto} already implements ML-KEM at
|
||||
\texttt{encryption/mlkem.go}. The \texttt{hanzoai/age} fork will add
|
||||
X-Wing as a new recipient type.
|
||||
|
||||
\subsection{Migration Phases}
|
||||
|
||||
\begin{description}
|
||||
\item[Phase 1 (current):] X25519 recipients. Adequate for operational
|
||||
data with limited shelf life. AES-256 at the sqlcipher layer
|
||||
(Layer~1) already provides 128-bit post-quantum security via Grover's
|
||||
bound, protecting on-disk data independently.
|
||||
\item[Phase 2 (planned):] X-Wing hybrid recipients
|
||||
(\texttt{age1xwing1...}) in \texttt{hanzoai/age}. Backward
|
||||
compatible---old X25519 recipients continue to work. New replicas
|
||||
encrypted to both X-Wing and X25519 recipients during transition.
|
||||
\item[Phase 3 (future):] Pure ML-KEM-768 or ML-KEM-1024 recipients
|
||||
when the ecosystem matures and X25519 is considered deprecated.
|
||||
\end{description}
|
||||
|
||||
\subsection{Three-Layer Post-Quantum Status}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{@{}llll@{}}
|
||||
\toprule
|
||||
\textbf{Layer} & \textbf{Algorithm} & \textbf{PQ Status} & \textbf{PQ Path} \\
|
||||
\midrule
|
||||
1. Disk & AES-256 (sqlcipher) & Safe (128-bit PQ) & No change needed \\
|
||||
2. S3 & age X25519 & Vulnerable & X-Wing (Phase 2) \\
|
||||
3. Wire & TLS 1.3 & Depends on suite & PQ TLS when available \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Post-quantum status of each encryption layer.}
|
||||
\label{tab:pq-status}
|
||||
\end{table}
|
||||
|
||||
% ============================================================================
|
||||
% 12. REFERENCES
|
||||
% ============================================================================
|
||||
\section{References}
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
|
||||
\bibitem{litestream}
|
||||
B.~Johnson, ``Litestream --- Streaming replication for SQLite,''
|
||||
\url{https://litestream.io}, 2021.
|
||||
|
||||
\bibitem{age}
|
||||
F.~Valsorda, ``age --- A simple, modern and secure encryption tool,''
|
||||
\url{https://age-encryption.org}, 2019.
|
||||
|
||||
\bibitem{sqlcipher}
|
||||
Zetetic LLC, ``SQLCipher --- Full Database Encryption for SQLite,''
|
||||
\url{https://www.zetetic.net/sqlcipher/}, 2008.
|
||||
|
||||
\bibitem{beeper-litestream}
|
||||
Beeper, ``Litestream fork with age encryption,''
|
||||
\url{https://github.com/nicolo-ribaudo/litestream-age}, 2023.
|
||||
|
||||
\bibitem{hip-base}
|
||||
Z.~Kelling, ``Hanzo Base: The Single-Binary Runtime for Sovereign
|
||||
Applications,'' Hanzo Industries, March 2026.
|
||||
|
||||
\bibitem{hip-0027}
|
||||
Hanzo AI Team, ``HIP-0027: Secrets Management Standard,'' 2025.
|
||||
|
||||
\bibitem{hip-0032}
|
||||
Hanzo AI Team, ``HIP-0032: Object Storage Standard,'' 2025.
|
||||
|
||||
\bibitem{hip-0065}
|
||||
Hanzo AI Team, ``HIP-0065: Backup \& Disaster Recovery Standard,'' 2026.
|
||||
|
||||
\bibitem{xwing}
|
||||
M.~Connolly, ``X-Wing: The Hybrid KEM You've Been Looking For,''
|
||||
Internet-Draft, IETF CFRG,
|
||||
\url{https://www.ietf.org/archive/id/draft-connolly-cfrg-xwing-kem-05.html},
|
||||
2024.
|
||||
|
||||
\bibitem{fips203}
|
||||
NIST, ``Module-Lattice-Based Key-Encapsulation Mechanism Standard
|
||||
(ML-KEM),'' FIPS~203, August 2024.
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Reference in New Issue
Block a user