mirror of
https://github.com/hanzoai/skills.git
synced 2026-11-19 00:47:36 +00:00
hanzoskills.com/llms.txt now opens with the install line, the API reference index and the company index, so an agent that lands on any one of the three can find the rest. robots.txt claimed 537 documents while llms.txt counts 920. The number is gone from robots.txt; llms.txt derives it. public/ is regenerated from scripts/site.py, which it had fallen behind: the copy-prompt menu and agent bar were in the generator and not in the committed pages. tests/site.mjs passes against the result. Co-authored-by: Hanzo Dev <[email protected]>
381 lines
35 KiB
HTML
381 lines
35 KiB
HTML
<!doctype html>
|
||
<html lang="en"><head>
|
||
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||
<title>hip-0107</title>
|
||
<meta name="color-scheme" content="dark">
|
||
<style>
|
||
/* Committed to one visual world: a dark instrument panel. No light variant —
|
||
the dot field and the hairline rules only read on black, and a naive
|
||
inversion would wash both out. Every colour is painted explicitly so the page
|
||
never borrows a host background. */
|
||
@font-face{font-family:'Zen';src:url('/fonts/Zen-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
||
@font-face{font-family:'Zen';src:url('/fonts/Zen-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
||
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Variable.woff2') format('woff2');font-weight:100 900;font-style:normal;font-display:swap}
|
||
@font-face{font-family:'Zen Mono';src:url('/fonts/ZenMono-Italic-Variable.woff2') format('woff2');font-weight:100 900;font-style:italic;font-display:swap}
|
||
@font-face{font-family:'Zen Pixel Square';src:url('/fonts/ZenPixel-Square.woff2') format('woff2');font-style:normal;font-display:swap}
|
||
|
||
:root{
|
||
--ink:#f4f4f2; --soft:#9c9c98; --faint:#6a6a66; --ground:#000; --panel:#0a0a0a;
|
||
--line:#232323; --line-hi:#3a3a3a; --accent:#f4f4f2;
|
||
--font-zen-sans:'Zen',-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
|
||
--font-zen-mono:'Zen Mono',ui-monospace,"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;
|
||
--font-zen-pixel-square:'Zen Pixel Square',monospace;
|
||
--mono:var(--font-zen-mono);
|
||
--sans:var(--font-zen-sans);
|
||
/* Controls are pills and surfaces are soft, as on hanzo.ai. */
|
||
--r:999px; --r-soft:14px;
|
||
}
|
||
*{box-sizing:border-box}
|
||
html{-webkit-text-size-adjust:100%}
|
||
body{margin:0;background:var(--ground);color:var(--ink);font-family:var(--sans);
|
||
font-size:16px;line-height:1.65;-webkit-font-smoothing:antialiased}
|
||
.wrap{max-width:60rem;margin:0 auto;padding:0 1.75rem}
|
||
|
||
/* The dot field. A halftone made of one repeating radial-gradient, faded out
|
||
along the diagonal by a mask so it reads as a wave rather than wallpaper. */
|
||
/* No overflow clip here: the dot field is inset to the box already, and a clip
|
||
would cut off the copy-prompt menu. */
|
||
.hero{position:relative;border-bottom:1px solid var(--line)}
|
||
.hero::before{content:"";position:absolute;inset:0;
|
||
background-image:radial-gradient(var(--line-hi) 1px, transparent 1px);
|
||
background-size:7px 7px;
|
||
-webkit-mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
||
mask-image:linear-gradient(115deg,#000 0%,rgba(0,0,0,.55) 38%,transparent 72%);
|
||
pointer-events:none}
|
||
.hero .wrap{position:relative;padding-top:5.5rem;padding-bottom:4rem}
|
||
|
||
nav{border-bottom:1px solid var(--line)}
|
||
nav .in{max-width:60rem;margin:0 auto;padding:1.1rem 1.75rem;display:flex;
|
||
justify-content:space-between;align-items:center;gap:1rem}
|
||
.brand{font-family:var(--mono);font-size:.85rem;letter-spacing:.14em;
|
||
text-transform:uppercase;color:var(--ink);text-decoration:none;
|
||
display:inline-flex;align-items:center;gap:.6rem}
|
||
.brand i{font-style:normal;color:var(--faint)}
|
||
.mark{width:1.15em;height:1.15em;display:block;flex:0 0 auto}
|
||
nav a.up{font-family:var(--mono);font-size:.72rem;letter-spacing:.14em;
|
||
text-transform:uppercase;color:var(--soft);text-decoration:none}
|
||
nav a.up:hover{color:var(--ink)}
|
||
|
||
h1{font-family:var(--mono);font-size:clamp(2rem,5.4vw,3.4rem);line-height:1.12;
|
||
letter-spacing:-.02em;margin:0 0 1.1rem;font-weight:500;text-wrap:balance}
|
||
h1 em{font-style:normal;color:var(--faint)}
|
||
.lede{color:var(--soft);font-size:1.05rem;max-width:52ch;margin:0 0 2.25rem}
|
||
|
||
.btns{display:flex;flex-wrap:wrap;gap:.7rem;margin:0 0 .5rem;align-items:center}
|
||
.btn{display:inline-flex;align-items:center;gap:.5rem;font-family:var(--mono);
|
||
font-size:.72rem;letter-spacing:.14em;text-transform:uppercase;
|
||
padding:.8rem 1.4rem;border:1px solid var(--line-hi);border-radius:var(--r);
|
||
color:var(--ink);text-decoration:none;background:transparent;cursor:pointer;
|
||
line-height:1.2;font-weight:400}
|
||
.btn:hover{background:#141414;text-decoration:none;border-color:var(--soft)}
|
||
.btn.solid{background:var(--ink);color:#000;border-color:var(--ink)}
|
||
.btn.solid:hover{background:#d8d8d4}
|
||
.btn .chev{width:.7em;height:.7em;flex:0 0 auto;transition:transform .15s ease}
|
||
.btn[aria-expanded="true"] .chev{transform:rotate(180deg)}
|
||
|
||
/* Copy-prompt selector. Each row copies the prompt for its own target. */
|
||
.copy{position:relative;display:inline-flex}
|
||
.menu{position:absolute;top:calc(100% + .5rem);left:0;z-index:20;min-width:21rem;
|
||
background:var(--panel);border:1px solid var(--line-hi);border-radius:var(--r-soft);
|
||
padding:.4rem;display:flex;flex-direction:column;
|
||
box-shadow:0 12px 32px rgba(0,0,0,.75)}
|
||
.menu[hidden]{display:none}
|
||
.mi{display:flex;align-items:center;gap:.7rem;width:100%;text-align:left;
|
||
padding:.62rem .8rem;border:0;border-radius:calc(var(--r-soft) - .35rem);
|
||
background:transparent;color:var(--ink);font-family:var(--mono);font-size:.76rem;
|
||
letter-spacing:.04em;text-decoration:none;cursor:pointer;line-height:1.3;
|
||
white-space:nowrap}
|
||
.mi:hover,.mi:focus-visible{background:#181818;outline:none}
|
||
.mi .mark{width:1.05rem;height:1.05rem;color:var(--soft)}
|
||
.mi:hover .mark{color:var(--ink)}
|
||
.mi u{margin-left:auto;color:var(--faint);text-decoration:none;font-size:.68rem;
|
||
letter-spacing:.1em;text-transform:uppercase}
|
||
|
||
/* Works-with bar: one scrolling row, name and mark per agent. */
|
||
.strip{border-bottom:1px solid var(--line);background:var(--panel)}
|
||
.strip .in{max-width:60rem;margin:0 auto;padding:.9rem 1.75rem;display:flex;
|
||
align-items:center;gap:1.25rem;font-family:var(--mono);font-size:.72rem;
|
||
letter-spacing:.1em;text-transform:uppercase;color:var(--faint)}
|
||
.strip b{color:var(--soft);font-weight:400;flex:0 0 auto;
|
||
border-right:1px solid var(--line);padding-right:1.25rem;align-self:stretch;
|
||
display:flex;align-items:center}
|
||
.bar{display:flex;align-items:center;gap:.5rem;flex:1 1 auto;min-width:0;
|
||
overflow-x:auto;scrollbar-width:none;scroll-snap-type:x proximity;
|
||
-webkit-overflow-scrolling:touch;
|
||
-webkit-mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent);
|
||
mask-image:linear-gradient(90deg,#000 calc(100% - 3rem),transparent)}
|
||
.bar::-webkit-scrollbar{display:none}
|
||
.agent{display:inline-flex;align-items:center;gap:.45rem;flex:0 0 auto;
|
||
padding:.4rem .85rem;border:1px solid var(--line);border-radius:var(--r);
|
||
color:var(--soft);white-space:nowrap;scroll-snap-align:start;
|
||
text-decoration:none;background:var(--ground)}
|
||
.agent:hover{color:var(--ink);border-color:var(--line-hi);text-decoration:none}
|
||
.agent .mark{width:.95rem;height:.95rem}
|
||
|
||
h2{font-family:var(--mono);font-size:1.1rem;letter-spacing:-.01em;margin:2.75rem 0 .75rem;font-weight:500}
|
||
h3{font-size:1rem;margin:2rem 0 .5rem;font-weight:600}
|
||
h4,h5,h6{font-size:.94rem;margin:1.5rem 0 .5rem;font-weight:600}
|
||
p{margin:0 0 1rem;max-width:70ch}
|
||
ul,ol{margin:0 0 1rem;padding-left:1.35rem;max-width:70ch}
|
||
li{margin:.22rem 0}
|
||
a{color:var(--ink);text-decoration:underline;text-underline-offset:3px;
|
||
text-decoration-color:var(--line-hi)}
|
||
a:hover{text-decoration-color:var(--ink)}
|
||
a:focus-visible{outline:1px solid var(--ink);outline-offset:3px}
|
||
code{font-family:var(--mono);font-size:.85em;background:var(--panel);
|
||
border:1px solid var(--line);border-radius:5px;padding:.1em .36em}
|
||
pre{background:var(--panel);border:1px solid var(--line);padding:1rem 1.1rem;
|
||
border-radius:var(--r-soft);overflow-x:auto;margin:0 0 1.25rem}
|
||
pre code{background:none;border:0;padding:0;font-size:.82rem;line-height:1.7}
|
||
blockquote{margin:0 0 1rem;padding-left:1rem;border-left:1px solid var(--line-hi);color:var(--soft)}
|
||
hr{border:0;border-top:1px solid var(--line);margin:2.5rem 0}
|
||
table{border-collapse:collapse;width:100%;margin:0 0 1.25rem;font-size:.92rem;
|
||
display:block;overflow-x:auto}
|
||
th,td{border:1px solid var(--line);padding:.5rem .7rem;text-align:left}
|
||
|
||
.curl{display:block;background:var(--panel);border:1px solid var(--line);
|
||
border-radius:var(--r-soft);padding:.95rem 1.1rem;font-family:var(--mono);
|
||
font-size:.8rem;color:var(--soft);overflow-x:auto;margin:0 0 2.5rem;
|
||
white-space:pre;line-height:1.9}
|
||
.curl b{color:var(--ink);font-weight:400}
|
||
.curl i{font-style:normal;color:var(--faint)}
|
||
|
||
/* index */
|
||
#q{width:100%;padding:.85rem 1.2rem;font-family:var(--mono);font-size:.85rem;color:var(--ink);
|
||
background:var(--panel);border:1px solid var(--line);border-radius:var(--r);margin:0 0 .6rem}
|
||
#q::placeholder{color:var(--faint);text-transform:uppercase;letter-spacing:.1em;font-size:.72rem}
|
||
#q:focus{outline:none;border-color:var(--soft)}
|
||
#count{color:var(--faint);font-family:var(--mono);font-size:.72rem;letter-spacing:.1em;
|
||
text-transform:uppercase;margin:0 0 2rem}
|
||
.cat{margin:2.5rem 0 .5rem;font-family:var(--mono);font-size:.7rem;letter-spacing:.16em;
|
||
text-transform:uppercase;color:var(--faint);border-bottom:1px solid var(--line);
|
||
padding-bottom:.45rem}
|
||
.row{display:block;padding:.6rem .9rem;margin:0 -.9rem;border-bottom:1px solid var(--line);
|
||
border-radius:var(--r-soft);text-decoration:none}
|
||
.row:hover{background:#0a0a0a}
|
||
.nm{font-family:var(--mono);font-size:.84rem;color:var(--ink);display:block}
|
||
.ds{color:var(--soft);font-size:.9rem;display:block;margin-top:.12rem}
|
||
footer{margin-top:4rem;padding:2rem 0 4rem;border-top:1px solid var(--line);
|
||
color:var(--faint);font-size:.85rem}
|
||
footer a{color:var(--soft)}
|
||
.fmap{display:grid;gap:1.75rem;grid-template-columns:repeat(auto-fit,minmax(11rem,1fr));
|
||
margin:0 0 2rem;font-size:.85rem}
|
||
.fmap b{display:block;font-family:var(--mono);font-size:.68rem;letter-spacing:.16em;
|
||
text-transform:uppercase;color:var(--faint);font-weight:400;margin:0 0 .6rem;
|
||
border-bottom:1px solid var(--line);padding-bottom:.4rem}
|
||
.fmap a{display:inline-block;color:var(--ink);text-decoration:none;margin:.3rem .35rem 0 0}
|
||
.fmap a:hover{text-decoration:underline}
|
||
.fmap div{color:var(--faint);line-height:1.5}
|
||
.fend{border-top:1px solid var(--line);padding-top:1.25rem;font-size:.82rem}
|
||
|
||
@media (max-width:640px){
|
||
.hero .wrap{padding-top:3.25rem;padding-bottom:2.75rem}
|
||
.btns{gap:.5rem}
|
||
.btn{padding:.75rem 1.15rem;font-size:.68rem}
|
||
.copy,.copy .btn{width:100%;justify-content:center}
|
||
.menu{left:0;right:0;min-width:0}
|
||
.strip .in{flex-direction:column;align-items:flex-start;gap:.6rem;padding:.9rem 1.75rem}
|
||
.strip b{border-right:0;padding-right:0}
|
||
.bar{width:100%}
|
||
}
|
||
</style>
|
||
<link rel="alternate" type="text/markdown" href="/hip-0107.md">
|
||
|
||
<link rel="icon" href="/favicon.svg">
|
||
<link rel="canonical" href="https://hanzoskills.com/hip-0107">
|
||
</head>
|
||
<!-- Reading this with a machine? Every page here is also plain markdown.
|
||
This one: https://hanzoskills.com/hip-0107.md
|
||
Index: https://hanzoskills.com/llms.txt
|
||
Corpus: https://hanzoskills.com/all.md
|
||
Append .md to any path for the markdown twin. -->
|
||
<body>
|
||
<nav><div class="in">
|
||
<a class="brand" href="/"><svg class="mark" viewBox="0 0 67 67" fill="currentColor" aria-hidden="true"><path d="M22.21 67V44.6369H0V67H22.21Z"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z"/><path d="M22.21 0H0V22.3184H22.21V0Z"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z"/></svg> HANZO<i>SKILLS</i></a>
|
||
<a class="up" href="/">All skills</a>
|
||
</div></nav>
|
||
|
||
<div class="wrap">
|
||
<h1>hip-0107</h1><p class="lede">HIP-107: Streaming Replication over VFS. Status Final. Hanzo architectural specification.</p><h1>HIP-0107: Streaming Replication over VFS</h1>
|
||
<h2>Abstract</h2>
|
||
<p>This proposal defines the <strong>unified streaming replication contract</strong> for every durable state in Hanzo. All sources of incremental updates — SQLite WAL, ZapDB log, blockchain state, generic append-only logs — ship frames through a <strong>single pipeline</strong> built from three orthogonal pieces:</p>
|
||
<ol><li><strong>One source-adapter interface</strong> (<code>replicate.Source</code>) — pulls</li></ol>
|
||
<p> frames from a backing engine.</p>
|
||
<ol><li><strong>One sink interface</strong> (<code>hanzo/vfs.Backend</code>) — the only object-store</li></ol>
|
||
<p> abstraction in Hanzo. All sink-side bytes are <code>vfs.Write</code> calls.</p>
|
||
<ol><li><strong>One encryption layer</strong> (<code>luxfi/age</code>) — frames are age-encrypted at</li></ol>
|
||
<p> the boundary; the master key lives in KMS per HIP-0027.</p>
|
||
<p>The standalone <code>hanzoai/zapdb-replicator</code> sidecar is deprecated and folded into <code>hanzoai/replicate</code> as a source adapter. The per-backend sink code in <code>replicate/s3</code>, <code>replicate/gs</code>, <code>replicate/abs</code>, <code>replicate/nats</code>, <code>replicate/oss</code>, <code>replicate/sftp</code>, <code>replicate/webdav</code> collapses into <code>hanzo/vfs</code>. <strong>One way to do everything.</strong></p>
|
||
<p>This HIP <strong>does not supersede</strong> HIP-0302; HIP-0302 stays Final as the spec for the SQLite-WAL and ZapDB substrate (encryption keys, recovery objectives, K8s sidecar pattern, PQ-safe age v1.3.0+ recipients). HIP-0107 sits on top: it unifies the transport so the SQLite backend, the ZapDB backend, and any future backend route bytes through the same pipeline.</p>
|
||
<p><strong>Two mechanisms, one substrate (the reconciliation).</strong> Hanzo grew TWO SQLite-HA implementations: <code>hanzoai/replicate</code> (a Litestream fork — continuous streaming WAL → object store, files stored as age-encrypted <code>.zap.age</code>) and <code>hanzoai/vfs/replica</code> (whole-object <code>VACUUM INTO</code> snapshots plus the piece Litestream lacks: a coordinator-free single-writer election). That overlap is a one-and-one-way violation. This HIP resolves it, and names the winners — one home per concern: <strong>streaming replication (<code>hanzoai/replicate</code>) is the canonical durability/replication transport; HRW single-writer election is <code>hanzoai/ha</code> (extracted from <code>vfs/replica</code>, which had kept storage-agnostic election inside a SQLite library — its own one-and-one-way violation); the <code>VACUUM INTO</code> snapshot / hydrate / adopt primitives stay in <code>hanzoai/vfs/replica</code>.</strong> <code>ha</code> decides WHO writes; <code>vfs/replica</code> decides HOW state ships. The whole-object snapshot path is superseded by streaming for the replication path; election (now <code>ha</code>) and the hydrate/adopt primitives stay. See "HA-SQLite substrate" and "Relationship to HIP-0302 and hanzoai/replicate" below.</p>
|
||
<h2>Specification</h2>
|
||
<h3>Architecture</h3>
|
||
<pre><code> ┌───────────────────────────────────────────┐
|
||
│ hanzoai/replicate │
|
||
│ │
|
||
┌─────────┐ │ ┌──────────────┐ ┌─────────────────┐ │ ┌────────────┐
|
||
│ Source │──┼─▶│ frame-pump │───▶│ age-encrypt │──┼─▶│ vfs.Backend│
|
||
│ adapter │ │ │ (one impl) │ │ (luxfi/age) │ │ │ (sink) │
|
||
└─────────┘ │ └──────────────┘ └─────────────────┘ │ └────────────┘
|
||
└───────────────────────────────────────────┘</code></pre>
|
||
<p>Four canonical source adapters ship with <code>hanzoai/replicate</code>:</p>
|
||
<p>| Source | Engine | Frame definition | |---|---|---| | <code>source/sqlite-wal</code> | SQLite WAL (the original Litestream surface) | WAL frames, header per Litestream protocol | | <code>source/zapdb-log</code> | ZapDB incremental backup | ZAP frame format (magic <code>0x5A415001</code>, defined in HIP-0302 §"ZapDB Streaming Replication") | | <code>source/chain-state</code> | Lux primary network, regulated EVM L1, Z-Chain, Hanzo's L1 | per-block state delta + snapshot frames | | <code>source/generic-log</code> | Any append-only log file or Unix socket | length-prefixed records |</p>
|
||
<p>All four implement:</p>
|
||
<pre><code class="lang-go">package replicate
|
||
|
||
type Source interface {
|
||
Name() string
|
||
// PullFrames yields frames as the source produces them.
|
||
// The returned channel closes when the source quiesces.
|
||
PullFrames(ctx context.Context) (<-chan Frame, error)
|
||
// Restore reads frames in order and applies them to the destination.
|
||
Restore(ctx context.Context, frames <-chan Frame, dst string) error
|
||
}
|
||
|
||
type Frame struct {
|
||
ID uint64 // monotonic per-source
|
||
Kind FrameKind // snapshot | delta | tombstone
|
||
Payload []byte // engine-specific (WAL bytes, ZAP frame, block delta, log record)
|
||
Checksum uint32 // CRC-32 over Payload
|
||
Timestamp time.Time
|
||
}</code></pre>
|
||
<h3>Sink: hanzo/vfs is the only object store</h3>
|
||
<p>The per-backend code currently in <code>hanzoai/replicate/{s3,gs,abs,nats, oss,sftp,webdav}</code> <strong>moves to <code>hanzo/vfs</code></strong> as standard <code>vfs.Backend</code> implementations. After this HIP lands:</p>
|
||
<ul><li><code>hanzoai/replicate</code> has <strong>zero direct cloud-SDK imports</strong>. The only</li></ul>
|
||
<p> storage interaction is <code>vfs.Open(uri)</code> + <code>vfs.Backend.Write(...)</code>.</p>
|
||
<ul><li>Adding a new sink backend is <strong>one PR to <code>hanzo/vfs</code></strong>. The</li></ul>
|
||
<p> replication pipeline picks it up for free.</p>
|
||
<ul><li>A deployment configures replication via a single <code>vfs://</code> URI:</li></ul>
|
||
<p> <code>vfs://gs/hanzo-replicate-mainnet/iam/<instance>/...</code>.</p>
|
||
<h3>Encryption boundary</h3>
|
||
<p>Age encryption happens <strong>once, at the frame boundary</strong>, before the sink call. The recipient set per HIP-0302:</p>
|
||
<ul><li>ML-KEM-768 + X25519 hybrid (<code>age1pq</code> prefix), age v1.3.0+</li><li>Per-service keypair derived from a KMS-stored master key via HKDF</li></ul>
|
||
<p> with <code>hanzo:replicate:</code> domain separator</p>
|
||
<ul><li>HIP-0302 recovery objectives apply unchanged (RPO 1s SQLite / 500ms</li></ul>
|
||
<p> ZapDB; RTO 30s / 15s)</p>
|
||
<p>The sink layer sees opaque ciphertext. The sink layer <strong>must not</strong> inspect frame contents.</p>
|
||
<h3>Frame format (canonical)</h3>
|
||
<p>The on-disk frame format is the ZAP frame already defined in HIP-0302 §"ZapDB Streaming Replication":</p>
|
||
<pre><code>Frame header (16 bytes):
|
||
[0:4] magic = 0x5A415001 ("ZAP\x01")
|
||
[4:8] frame_id = monotonic uint32
|
||
[8:12] length = payload length in bytes
|
||
[12:14] checksum = CRC-16 of payload
|
||
[14:15] flags = 0x01=snapshot, 0x02=delta, 0x04=compressed
|
||
[15:16] reserved = 0x00</code></pre>
|
||
<p>The SQLite-WAL source wraps WAL frames in this envelope. The chain-state source wraps block deltas. The generic-log source wraps records. <strong>One on-disk format across all sources.</strong></p>
|
||
<h3>Multi-tenant bucket layout</h3>
|
||
<p>The KMS-derived master key is per-deployment. Each tenant org gets a distinct key path:</p>
|
||
<pre><code>vfs://<backend>/<bucket>/replicate/<org>/<service>/<source>/
|
||
├── snapshots/<frame_id>.snap.zap.age
|
||
├── deltas/<start>_<end>.delta.zap.age
|
||
└── latest</code></pre>
|
||
<ul><li><code><org></code> comes from JWT <code>owner</code> claim (HIP-0026 unified identity).</li><li><code><service></code> is the subsystem name (<code>iam</code>, <code>base</code>, <code>kms</code>, ...).</li><li><code><source></code> is one of <code>sqlite-wal</code>, <code>zapdb-log</code>, <code>chain-state</code>,</li></ul>
|
||
<p> <code>generic-log</code>.</p>
|
||
<ul><li><code>latest</code> is a small pointer object naming the most recent frame.</li></ul>
|
||
<h3>Restore protocol</h3>
|
||
<p>On pod startup, the K8s init container per HIP-0302 invokes:</p>
|
||
<pre><code class="lang-bash">hanzo-replicate restore \
|
||
--source=sqlite-wal \
|
||
--vfs=vfs://gs/hanzo-replicate-mainnet \
|
||
--org=hanzo \
|
||
--service=iam \
|
||
--dst=/data/iam.db \
|
||
--age-key-ref=kms://hanzo/superbase/master</code></pre>
|
||
<p>The same binary handles all four source types via the <code>--source</code> flag. Restore drains the latest snapshot frame, then applies delta frames in monotonic order, then resumes streaming.</p>
|
||
<h2>HA-SQLite substrate: single-writer election + adoption contract (shipped)</h2>
|
||
<p>The streaming pipeline above answers "how do bytes get durable." It does NOT answer "which replica may write." Litestream — and therefore <code>hanzoai/replicate</code> — assumes a single writer and does not elect one. <code>hanzoai/ha</code> supplies exactly that missing half — coordinator-free single-writer election — and <code>hanzoai/vfs/replica</code> supplies the primitives a service needs to adopt HA without changing its own storage library. Together they are the shared substrate every Hanzo service imports instead of re-inventing durability: <code>ha</code> decides WHO writes, <code>vfs/replica</code> decides HOW state ships.</p>
|
||
<h3>Single-writer election (Rendezvous / HRW), no coordinator — <code>hanzoai/ha</code></h3>
|
||
<p><code>hanzoai/ha</code> answers, identically on every replica and with no network round-trip, "who is the single writer for key K?" via Highest-Random-Weight hashing over the live membership set (K is any ownership unit — an org id, a shard, a singleton job name):</p>
|
||
<ul><li><code>Owner(key, members)</code> — deterministic winner; `weight =</li></ul>
|
||
<p> SHA-256(key ‖ 0x00 ‖ memberID)[:8]` as a big-endian uint64, tie-broken on member ID so the result is order-independent.</p>
|
||
<ul><li><code>IsOwner(key, selfID, members)</code> — the per-write hot-path gate.</li><li><code>Replicas(key, members, n)</code> — owner first, then ordered failover</li></ul>
|
||
<p> successors, so a reader can pre-warm the key it is next in line to own.</p>
|
||
<ul><li><code>Membership</code> (seam) + <code>Static</code> — the live-set source: a cluster impl</li></ul>
|
||
<p> (<code>ha/k8s</code>, roadmap) or single-process <code>Static</code> for dev and tests.</p>
|
||
<ul><li><strong>Fail-closed:</strong> empty membership yields <code>ok=false</code> — never a wrong</li></ul>
|
||
<p> writer. No election protocol, no lock service, no service discovery, <strong>no SQL, no KV.</strong> Pure Go, stdlib only.</p>
|
||
<p>Ownership is PER-ORG: the owning replica writes ALL of an org's DBs (root + every per-project + per-user file) so an org's data has locality and moves as one unit on failover. Reads are served from any replica's vfs-synced copy.</p>
|
||
<h3>Adoption contract</h3>
|
||
<p>Every adopter wires the same three lines at its per-org store seam:</p>
|
||
<ol><li><strong>hydrate-on-open</strong> — <code>Pull()</code> (or <code>RestoreFile</code>) the latest durable</li></ol>
|
||
<p> copy before first use of an org DB, so a fresh or replaced pod starts from committed state.</p>
|
||
<ol><li><strong>single-writer</strong> — <code>IsOwner(org, self, members)</code> gates writes; a</li></ol>
|
||
<p> non-owner serves stale-tolerant reads (Pull-then-read) and forwards writes to the owner.</p>
|
||
<ol><li><strong>post-commit ship/stream</strong> — the owner ships after a write burst (the</li></ol>
|
||
<p> shipped <code>Replicator.PushLoop</code>), superseded on the streaming path by <code>replicate</code> continuous WAL shipping.</p>
|
||
<h3>Shipped primitives (<code>hanzoai/vfs/replica</code>)</h3>
|
||
<ul><li><code>SQLiteDB</code> (<code>sqlite.go</code>) — the real per-org handle: <code>Snapshot</code> = `VACUUM</li></ul>
|
||
<p> INTO<code> a temp (forces a WAL checkpoint, one transaction-consistent copy, no torn pages, no </code>-wal<code>/</code>-shm<code> to ship); </code>Restore<code> = write-temp → close → drop </code>-wal<code>/</code>-shm<code> → atomic rename → reopen (a crash mid-restore leaves the old DB intact). CGO-free (</code>modernc.org/sqlite`), same durability pragmas (WAL / NORMAL / foreign_keys / busy_timeout) as base and visor.</p>
|
||
<ul><li><code>SnapshotFile</code> / <code>RestoreFile</code> — handle-less adoption primitives so an</li></ul>
|
||
<p> xorm / GORM service adopts HA without a handle conflict (restore runs BEFORE the service opens its own engine).</p>
|
||
<ul><li><code>BackendStore</code> (<code>store.go</code>) — adapts any <code>vfs</code> backend (<code>file://</code> dev,</li></ul>
|
||
<p> <code>s3://</code> SeaweedFS prod) to the <code>Store</code>, with a cheap <code><key>.ver</code> content-version sidecar so a reader skips redundant pulls; <code>Get</code> recomputes the version authoritatively, so a stale sidecar degrades to a redundant pull, never a wrong restore.</p>
|
||
<ul><li>Per-org envelope encryption — <code>WithEncryption(Cipher, org)</code> seals every</li></ul>
|
||
<p> pushed snapshot with the org's KMS-derived key, orgID bound as AAD so a blob cannot be replayed under another org (HIP-0302 recipients).</p>
|
||
<ul><li><code>DBPath(org, scope, service)</code> — the canonical HIP-0302 object layout</li></ul>
|
||
<p> (<code>orgs/<org>/[scope/]<service>.db</code>), computed identically on every replica; <code>Version(data)</code> — the one content hash (SHA-256) both the <code>Replicator</code> and any <code>Store</code> use.</p>
|
||
<h3>Live adoption</h3>
|
||
<p><strong>visor is the reference adopter and it is LIVE.</strong> <code>object/store_replica.go</code> opts in via <code>REPLICA_STORE</code> (<code>s3://…</code> SeaweedFS in prod, <code>file://</code> in test; unset = local-only, nil replicator, today's behavior with zero regression). It hydrates-on-open before xorm opens each org DB and runs a per-org ship loop; the success path logs, verbatim (<code>store_replica.go:128</code>):</p>
|
||
<pre><code>visor HA: org "_global" shipping to object store (durable)</code></pre>
|
||
<p>with explicit push-FAILED / push-recovered logs so a durability feature can never fail invisibly. <code>hanzoai/cloud</code> consolidated its own former <code>internal/org</code> replicator + election onto this same package (#11a3ae6 — re-exported as thin aliases in <code>internal/org/shared.go</code>), so cloud and visor now run ONE implementation; election has since been extracted again into <code>hanzoai/ha</code> (aliases repointed there; <code>vfs/replica</code> is now replication-only). Rolling the same adoption to iam / commerce / base is the remaining work.</p>
|
||
<h2>Relationship to HIP-0302 and hanzoai/replicate</h2>
|
||
<p>HIP-0302 stays Final: it owns the encryption recipients, recovery objectives, and the sidecar / emptyDir pattern. HIP-0107 owns the transport and the write-coordination on top. The two SQLite-HA mechanisms reconcile as follows — <strong>this is the decision, stated plainly:</strong></p>
|
||
<p>| Concern | Winner | Why | |---|---|---| | Durability / replication transport | <strong><code>hanzoai/replicate</code></strong> (Litestream fork) | Continuous streaming WAL → object store (converted to immutable LTX, shipped as age-encrypted <code>.zap.age</code>) is strictly better than whole-object snapshots: lower RPO, incremental bytes, no periodic full-DB rewrite. CGO-free (<code>modernc.org/sqlite</code>). Canonical replication path. | | Single-writer coordination | <strong><code>hanzoai/ha</code></strong> | Litestream / <code>replicate</code> do NOT elect a writer. HRW / Rendezvous election is the genuinely-novel piece — coordinator-free, fail-closed, no SQL/KV. Extracted from <code>vfs/replica</code> so a non-SQLite singleton (cron, queue drain) elects without importing a storage library. | | Adoption / hydrate / restore | <strong><code>hanzoai/vfs/replica</code></strong> (<code>SnapshotFile</code> / <code>RestoreFile</code>) | Handle-less primitives let any service hydrate-on-open and adopt HA regardless of its own DB library. | | Whole-object <code>VACUUM INTO</code> snapshot | <strong>superseded for replication</strong> | Kept as the consistent-copy primitive behind <code>SnapshotFile</code> and as the currently-shipped ship path; replaced by streaming as the durability transport as services migrate. |</p>
|
||
<p><strong>Honest status.</strong> The election, the adoption primitives, and the visor live path are shipped. The convergence — every adopter shipping via <code>replicate</code> streaming rather than the <code>Replicator</code> whole-object push, and folding <code>zapdb-replicator</code> + the per-backend sinks into the one <code>source-adapter → age → vfs.Backend</code> pipeline (the four-source design above) — is the reconciliation roadmap this HIP ratifies, not a claim that it is already uniform. The one-and-one-way violation (two overlapping SQLite-HA libraries) is closed by DECISION here; the code convergence follows the phases below.</p>
|
||
<h2>Implementation phases</h2>
|
||
<p>| Phase | Work | Effort | |---|---|---| | 1 | Move <code>replicate/{s3,gs,abs,nats,oss,sftp,webdav}</code> per-backend code into <code>hanzo/vfs</code>. Existing <code>replicate.Sink</code> interface deletes; callers switch to <code>vfs.Backend</code>. | 3–4 days | | 2 | Fold <code>hanzoai/zapdb-replicator</code> into <code>hanzoai/replicate</code> as <code>source/zapdb-log</code>. Deprecate <code>ghcr.io/hanzoai/zapdb-replicator</code> (publish a final <code>vN+1.0.0-deprecated</code> tag pointing deployers at <code>ghcr.io/hanzoai/replicate</code>). | 2 days | | 3 | Implement <code>source/chain-state</code> for Lux primary network. Reuse for regulated EVM L1 and any other Lux-derived L1. | 3–5 days for Lux primary; ~1–2 days per derived L1. | | 4 | Implement <code>source/generic-log</code> for arbitrary append-only logs (Unix socket or file). | 1 day | | 5 | Update K8s manifests across <code>~/work/hanzo/universe/infra/k8s/</code> to swap <code>zapdb-replicator</code> sidecars for <code>replicate --source=zapdb-log</code>. Bump <code>hanzoai/replicate</code> to the version that ships all four sources. | 1 day per service touched |</p>
|
||
<h2>Non-goals</h2>
|
||
<ul><li><strong><code>hanzoai/datastore</code> (Hanzo Datastore fork).</strong> Datastore already uses</li></ul>
|
||
<p> ReplicatedMergeTree + S3-disk for its own replication. HIP-0107 does not touch it. The two systems share an S3 bucket via <code>vfs</code> prefix separation: <code>s3://bucket/datastore/...</code> is Datastore-owned; <code>s3://bucket/replicate/...</code> is HIP-0107-owned. No collision.</p>
|
||
<ul><li><strong>Cross-region replication patterns.</strong> Active-active, geo-failover,</li></ul>
|
||
<p> read replicas across regions — separate concern. HIP-0107 ships frames to one <code>vfs.Backend</code>; what the operator chooses for the region/zone topology of that backend is outside scope.</p>
|
||
<ul><li><strong>Real-time CDC for non-WAL sources.</strong> Hanzo Base's per-record</li></ul>
|
||
<p> hooks (HIP-0105) carry their own change-data-capture stream. That is a logical-event pipeline; HIP-0107 is a byte-level frame pipeline. They do not converge.</p>
|
||
<ul><li><strong>Snapshot retention / GC policy.</strong> HIP-0302 already defines RPO,</li></ul>
|
||
<p> RTO, and retention windows. HIP-0107 inherits them unchanged.</p>
|
||
<ul><li><strong>Replacing HIP-0302's encryption + recovery contract.</strong> HIP-0302</li></ul>
|
||
<p> stays Final. HIP-0107 adds the unified transport on top.</p>
|
||
<h2>Open questions</h2>
|
||
<ol><li><strong>vfs URI scheme for non-S3 backends.</strong> <code>vfs://gs/...</code>,</li></ol>
|
||
<p> <code>vfs://abs/...</code>, <code>vfs://sftp/...</code> — confirm the URI prefix matches the existing <code>hanzo/vfs</code> convention before Phase 1.</p>
|
||
<ol><li><strong>Chain-state delta granularity.</strong> Per-block is the obvious</li></ol>
|
||
<p> default. Some Lux-derived L1s have 200ms block times — frames at 5 Hz per chain may stress the sink. Suggest configurable batching, default = 1 frame per N blocks where N is tuned per chain.</p>
|
||
<ol><li><strong>Generic-log fan-in.</strong> A single replicate process pumping N</li></ol>
|
||
<p> generic logs sharing one <code>Source</code> — likely fine; benchmark before committing to a goroutine-per-source model.</p>
|
||
<ol><li><strong>Backpressure when vfs.Backend is slow.</strong> Today</li></ol>
|
||
<p> <code>zapdb-replicator</code> drops oldest unsent frame on bounded-queue overflow. Confirm this is the policy we want across all four sources, or whether <code>chain-state</code> (consensus-critical) needs block-on-full instead.</p>
|
||
<h2>References</h2>
|
||
<ul><li>HIP-0027 — Secrets Management Standard (the KMS that holds the</li></ul>
|
||
<p> age master key per service/per org)</p>
|
||
<ul><li>HIP-0302 — Hanzo Replicate: Encrypted SQLite + ZapDB Durability</li></ul>
|
||
<p> for Base Services (the predecessor for the SQLite/ZapDB substrate; HIP-0107 adds the unified pipeline on top — HIP-0302 stays Final)</p>
|
||
<ul><li>HIP-0405 — S3 CRD (the <code>hanzoai/s3</code> workload behind the <code>s3://</code></li></ul>
|
||
<p> sink this pipeline writes to in production)</p>
|
||
<ul><li>Reference implementations:</li><li><code>~/work/hanzo/ha</code> (SHIPPED) — <code>github.com/hanzoai/ha</code>, HRW single-writer</li></ul>
|
||
<p> election (<code>Owner</code>/<code>IsOwner</code>/<code>Replicas</code>/<code>Member</code>) + the <code>Membership</code> seam and <code>Static</code>. Pure Go, stdlib only; election with no storage dependency.</p>
|
||
<ul><li><code>~/work/hanzo/vfs/replica</code> (SHIPPED) — the <code>Replicator</code> (<code>replica.go</code>),</li></ul>
|
||
<p> the real <code>SQLiteDB</code> snapshot/restore + <code>SnapshotFile</code>/<code>RestoreFile</code> adoption primitives (<code>sqlite.go</code>), and <code>BackendStore</code> over a <code>vfs</code> backend (<code>store.go</code>). Replication only; election lives in <code>hanzoai/ha</code>.</p>
|
||
<ul><li><code>~/work/hanzo/replicate</code> — <code>github.com/hanzoai/replicate</code>, the</li></ul>
|
||
<p> Litestream fork: streaming WAL → immutable LTX → object store, files stored as age-encrypted <code>.zap.age</code>; the canonical replication transport.</p>
|
||
<ul><li><code>~/work/hanzo/vfs</code> — the sink backends (<code>vfs.Backend</code>).</li><li>Litestream protocol (SQLite WAL framing): https://litestream.io</li><li>age v1.3.0+ PQ-hybrid recipients: https://age-encryption.org</li></ul>
|
||
<footer>
|
||
<div class="fmap">
|
||
<div><b>Build</b>
|
||
<a href="https://hanzo.app">hanzo.app</a> app builder
|
||
<a href="https://hanzo.codes">hanzo.codes</a> code with an agent
|
||
<a href="https://hanzoskills.com">hanzoskills.com</a> what an agent reads</div>
|
||
<div><b>Run</b>
|
||
<a href="https://console.hanzo.ai">console.hanzo.ai</a> cloud console
|
||
<a href="https://api.hanzo.ai">api.hanzo.ai</a> the API</div>
|
||
<div><b>Account</b>
|
||
<a href="https://hanzo.id">hanzo.id</a> sign in
|
||
<a href="https://pay.hanzo.ai">pay.hanzo.ai</a> top up
|
||
<a href="https://billing.hanzo.ai">billing.hanzo.ai</a> invoices and usage</div>
|
||
<div><b>Company</b>
|
||
<a href="https://hanzo.ai">hanzo.ai</a> canonical
|
||
<a href="https://hanzo.ventures">hanzo.ventures</a> ventures</div>
|
||
</div>
|
||
<div class="fend"><a href="/hip-0107.md">This page as plain text</a> · <a href="/llms.txt">Index for agents</a></div>
|
||
</footer>
|
||
</div></body></html> |