base/docs/NETWORK.md

Base Network

Seamless, PQ-native, consensus-replicated HA for every Base app. One mechanism. One env flag. Replica count N ∈ {1,2,3,…} works identically on a laptop (docker compose) and on k8s (via ~/work/hanzo/operator). No FUSE, no LiteFS, no Consul.

Durability and replication come from luxfi/consensus/protocol/quasar — the same DAG consensus that runs Lux validators — applied to SQLite WAL frames, one DAG per shard. Archive tier is optional cold storage on ~/work/hanzo/storage (hanzoai/s3) or GCS for PITR.

Problem

Base apps (ATS, BD, TA, IAM, KMS, AML, …) each carry SQLite files — often per-user or per-org — and need HA with real durability, no split-brain, and uniform ops across dev / test / main.

Pre-existing options fall short:

• Raw StatefulSet + RWO PVC, replicas > 1 → split-brain, silent DB divergence (current mainnet ATS/TA state).
• LiteFS + Consul → three moving parts (FUSE, LTX, Consul) and a single-primary election we don't need; we already have quasar.
• Litestream + single writer → backup, not HA.
• Postgres → violates Base's "embedded SQLite by default".

Goal

• Any Base app opts in with one env: BASE_NETWORK=quasar.
• BASE_REPLICATION=1 works (standalone, zero consensus overhead, still ships to archive).
• BASE_REPLICATION=N works (full quasar quorum, k = ⌊N/2⌋+1).
• docker compose up = true HA locally.
• Operator emits the env surface + HPA/KEDA from one CR field.

Design

Data model

• A Base app owns many shards. Shard key = user_id | org_id | <custom header>.
• Each shard = one SQLite file on the pod's local PVC (hot cache).
• Each shard is replicated across BASE_REPLICATION network members (that shard's quasar subset). Assignment = consistent-hash over live members.
• A shard's finalized WAL-frame DAG IS its durability tier. Archive is a cheap, optional second tier.

Shard hierarchy + per-shard encryption

Multi-tenant Base apps produce many shards. The model is two-level physical files plus a fixed logical consensus grouping:

• Physical shard = one SQLite file. Files exist at two scopes:
  • platform — _platform.db (rare; e.g. IAM app registry).
  • org — <org_id>.db (tenant-wide state).
  • user — <org_id>/<user_id>.db (per-customer state). A single request may touch multiple scopes in one transaction; Base opens the right handle based on the route's declared scope.
• Virtual shard (vshard) = consensus group. A fixed-cardinality ring (default 256, power-of-two) is hashed over at connection time: vshard = hash(shardKey) mod 256. Each vshard has a quasar validator subset of size BASE_REPLICATION. All physical files whose shardKey maps to vshard V replicate together. This keeps consensus metadata O(256) instead of O(customers).
• Assignment of physical pods to vshards = consistent-hash over live network members. Adding a pod moves ~1/N of the vshards to the new pod; physical files on those vshards migrate via quasar tail-sync (not a file copy — new pod subscribes to the vshard's DAG and materialises from frames).

Per-file encryption (SQLCipher, opt-in):

• BASE_ENCRYPT=sqlcipher activates per-file page-level encryption.
• Each physical file has its own 256-bit Data Encryption Key (DEK).
• DEKs are wrapped by a per-org Key Encryption Key (KEK) held in KMS; KEK is wrapped by a platform KEK-of-KEKs. On open(shard):
  1. If <shard>.dekwrap exists on disk: fetch KEK from KMS, unwrap DEK.
  2. If not: generate DEK, wrap with KEK, persist <shard>.dekwrap.
• DEKs cached in memory, per-shard, with LRU eviction (default 4096 hot files). Cold files drop their DEK; re-fetch on next access costs one KMS round-trip.
• WAL frames leaving the pod carry ciphertext pages only — quasar replication, gateway transit, and archive storage never see plaintext. PQ sig from quasar is over the ciphertext.
• Right-to-be-forgotten = delete the .dekwrap (+ quasar tombstone for the vshard's frame range). Ciphertext becomes unreadable globally, including in archive segments.

Env additions for this layer:

var	values	notes
BASE_VSHARDS	256 default, power-of-two	fixed at deploy; resharding is a separate tool.
BASE_ENCRYPT	sqlcipher | off (default)	opt-in page encryption.
BASE_KMS_URL	https://kms.<env>.<your-domain>	for KEK fetch/wrap.
BASE_KEK_SCOPE	platform | org (default)	grain of the KEK.
BASE_DEK_CACHE_SIZE	4096 default	LRU cap for unwrapped DEKs in RAM.

Write path

client ──► gateway (base-network://svc)
            │   JWT→shardKey, consistent-hash → writer pod
            ▼
         writer pod
            │   handle request, SQLite commit
            ▼
         sqlite3_wal_hook fires frame F
            │
            ▼
         network.SubmitFrame(shardID, F)
            │   quasar vertex, PQ-signed
            ▼
         quasar finalize (k-of-N ack)
            │
            ├──► apply callback: write F to local SQLite (writer + replicas)
            └──► optional: archive validator appends F to S3/GCS segment

No round-trip to a coordinator; no FUSE; no separate leader-election service. A finalized frame IS the commit.

Read path

• Any pod in the shard's subset can serve reads.
• Gateway sets a txseq cookie with the last-seen finalized seq.
• Target pod checks localSeq ≥ txseq:
  • yes → serve.
  • no → either block briefly on quasar tail (typical < 100 ms) or 307 to a caught-up peer.

Membership

• Every Base pod is a quasar validator.
• Peers discovered from BASE_PEERS seed list (CSV of DNS), gossip thereafter.
• In k8s: headless Service, peers = pod-0…N-1.svc.cluster.local:9999.
• In compose: static DNS names.
• Heartbeat TTL 1.5 s. On miss: the pod's shards are reassigned across survivors; new assignments sync the DAG from peers before accepting writes.

Replication factor

N	quasar mode	k (acks needed)	failure mode
1	standalone	1	No HA. Archive = durability.
2	pair	2	1-down blocks writes, reads survive.
3+	quorum	⌊N/2⌋+1	Byzantine-safe, PQ-signed.

Same code path for every N; only the k constant changes.

Archive tier (optional)

A witness-only validator runs alongside as BASE_NODE_ROLE=archive. It does not vote. It subscribes to finalized frames and appends per-shard WAL segments to object storage. Layout:

<scheme>://<bucket>/<svc>/<shard>/<seq-prefix>/<segment>.lbn

.lbn = Lux Base Network segment — a compact length-prefixed list of PQ-signed, quasar-finalized frames. Replay any prefix ⇒ SQLite at that txseq. PITR for free.

Backend is pluggable via github.com/hanzoai/s3 client surface:

• S3 (s3://…) — MinIO-protocol, used by ~/work/hanzo/storage.
• GCS (gs://…) — native client, used for production in GCP-hosted clusters.
• off — archive disabled; durability = quasar DAG only.

Env surface (one shape, every Base app)

var	values	notes
BASE_NETWORK	quasar | standalone (default)	standalone = today's behaviour, no network.
BASE_SHARD_KEY	user_id | org_id | <header name>	required when BASE_NETWORK=quasar.
BASE_REPLICATION	1 | 2 | 3 | …	≤ BASE_PEERS count.
BASE_PEERS	CSV of host:port DNS	operator-emitted in k8s; explicit in compose.
BASE_NODE_ROLE	validator (default) | archive
BASE_ARCHIVE	gs://… | s3://… | off (default)
BASE_LISTEN_HTTP	:8090 default	Base HTTP.
BASE_LISTEN_P2P	:9999 default	quasar p2p port.
BASE_SHARD_BACKLOG_MAX	bytes (64 MiB default)	R6 per-shard archive backlog cap; drop-oldest beyond.
BASE_SHARD_BACKLOG_SEGMENTS	integer (100 000 default)	R6 segment-count cap; first-to-hit with MAX drops.
BASE_TLS_CA	path to PEM bundle	R5 mTLS CA for peer verification. Unset ⇒ no TLS.
BASE_TLS_SERVER_CERT	path to PEM	R5 server cert presented to inbound peers.
BASE_TLS_SERVER_KEY	path to PEM	R5 server key.
BASE_TLS_ALLOWED_SANS	CSV of DNS SANs	R5 SAN allowlist. Typically = BASE_PEERS stripped of :9999.

R1–R8 fix notes

• R1 (frame/envelope shardID binding): inbound envelopes whose inner Frame.ShardID disagrees with Envelope.ShardID are rejected at the transport boundary; the apply loop re-checks as defence-in-depth.
• R2 (localSeq monotonic): Shard.localSeq advances strictly by +1 per finalised frame. Out-of-order frames buffer (cap 1024). An attacker-forged high-Seq frame stays in the buffer forever because its predecessors never arrive, so the gateway's txseq check cannot be fooled into reporting "caught up".
• R3 (archive authorship): segments are LBN2 — Ed25519 signature over body || CRC32 || pubkey. Writers refuse to encode without a signer; readers refuse to decode without a verifier; LBN1 hard- rejected as a downgrade vector.
• R4 (StatefulSet by default): when network.enabled: true, the operator emits a StatefulSet (spec.network.workload: StatefulSet default) so BASE_PEERS pod-ordinal DNS resolves. Deployment is permitted only at replicas == 1.
• R5 (mTLS on p2p): TLSConfig wires Ed25519/EC mTLS with SAN pinning against the BASE_PEERS list. Transport wire-format is ZAP (LP-200) over QUIC; TLSConfig.ServerConfig() / ClientConfig() land the crypto surface.
• R6 (backlog caps): per-shard cap 64 MiB / 100 k segments, drop-oldest with base_shard_backlog_drops_total metric.
• R7 (HPA workload kind): operator threads workload kind into the HPA / KEDA scaleTargetRef.kind so a StatefulSet-backed service autoscales correctly.
• R8 (nanos-suffixed object keys): objectKey(…, startSeq, nanos) produces unique keys per flush; Range dedupes by (startSeq + frameIndex) when multiple segments overlap.

Layout

~/work/hanzo/base/network/ (new package)

network/
  network.go    // NewNetwork(cfg) → *Network; Start/Stop; public API below.
  config.go     // env parsing, defaults, validation.
  node.go       // Node identity (stable from pod name or $HOSTNAME).
  quasar.go     // thin wrapper over luxfi/consensus/protocol/quasar.
  wal.go        // install sqlite3_wal_hook; frame serialise/PQ-sign.
  apply.go      // on-finalize callback → sqlite_apply().
  shard.go      // Shard struct: assignment, local cache, txseq.
  router.go     // consistent-hash ring; who owns shardID?
  archive.go    // BASE_NODE_ROLE=archive loop → hanzoai/s3 or GCS.
  metrics.go    // Prom: base_shards, base_wal_lag_bytes,
                //       base_hot_shards, base_lease_contentions,
                //       base_archive_lag_bytes, ...
  network_test.go
  wal_test.go
  router_test.go
  integration_test.go  // docker compose N=3 smoke: write, crash a node, read, restore.

Public API that Base calls from core/base.go on startup:

package network

type Network interface {
    Start(ctx context.Context) error
    Stop(ctx context.Context) error

    // Installed by Base onto the sqlite *Conn; intercepts commits.
    InstallWALHook(conn *sqlite.Conn, shardID string) error

    // Given a shardID, returns the current writer endpoint plus
    // whether it's the local pod. Gateway uses this.
    WriterFor(shardID string) (endpoint string, local bool)

    // Any member in the shard subset. Gateway uses this for reads.
    MembersFor(shardID string) []string

    // Metrics plumbing.
    Metrics() *Metrics
}

func New(cfg Config) (Network, error)

Base itself (core/base.go) gains:

if net := network.FromEnv(); net.Enabled() {
    b.net = net
    b.db.OnConnect(func(c *sqlite.Conn, shardID string) error {
        return net.InstallWALHook(c, shardID)
    })
}

That is all the coupling. Rest of Base code is unchanged.

~/work/hanzo/operator/ (reconciler update)

Add BaseNetworkSpec struct reused across Base-ish CRDs:

pub struct BaseNetworkSpec {
    pub enabled: bool,
    pub shard_key: String,            // user_id | org_id | <hdr>
    pub replication: u8,              // ≤ replicas
    pub archive: Option<String>,      // gs://… | s3://… | off
    pub autoscale: Option<Autoscale>,
}

pub struct Autoscale {
    pub min: u8,
    pub max: u8,
    pub cpu_target: Option<u8>,
    pub memory_target: Option<u8>,
    pub hot_shards_target: Option<u32>,
}

Added to: ServiceSpec, IAMSpec, KMSSpec, and any other downstream Base-backed CRDs.

New controller.rs helper:

reconcile_base_network(api, spec, meta) -> ();

emits, per replica-managed workload:

• Headless Service (clusterIP: None) for pod-ordinal peer DNS.
• Env on pods: BASE_NETWORK, BASE_SHARD_KEY, BASE_REPLICATION, BASE_PEERS (computed from spec.replicas + headless svc), BASE_ARCHIVE, BASE_NODE_ROLE=validator.
• Optional archive sidecar Deployment (BASE_NODE_ROLE=archive) when archive != off.
• HPA (cpu, memory) and KEDA ScaledObject (Prom metrics base_hot_shards, base_wal_lag_bytes) when autoscale set.
• PodDisruptionBudget: minAvailable = k where k = ⌊N/2⌋+1 for N≥3, N-1 for N=2, 0 for N=1.

CR shape:

spec:
  replicas: 3
  network:
    enabled: true
    shardKey: user_id
    replication: 3
    archive: gs://hanzo-base-wal/svc
    autoscale:
      min: 3
      max: 7
      cpuTarget: 70
      memoryTarget: 75
      hotShardsTarget: 500

~/work/hanzo/gateway/ (new upstream type)

base-network://<service> upstream.

• Parses JWT sub or configured header → shardID.
• Polls GET /-/base/members on the service's headless Service every 5 s; caches member list.
• Consistent-hashes shardID → owner pod.
• Writes → owner pod. Reads → random pick from MembersFor(shardID), propagating txseq cookie.
• On 307 from a not-yet-caught-up replica, retries to another member.
• Passes X-Base-Shard: <shardID> upstream for logging.

Config example:

upstreams:
  - name: my-svc
    type: base-network
    service: my-svc.hanzo.svc:8090
    shardKey: user_id
    shardKeySource: jwt.sub

Local dev — docker compose parity

~/work/hanzo/base/compose.network.yml:

services:
  base-a: &node
    image: ghcr.io/hanzoai/base:dev
    environment:
      BASE_NETWORK: quasar
      BASE_SHARD_KEY: user_id
      BASE_REPLICATION: 3
      BASE_PEERS: base-a:9999,base-b:9999,base-c:9999
      BASE_ARCHIVE: off
    ports: ["18090:8090"]
  base-b:
    <<: *node
    ports: ["18091:8090"]
  base-c:
    <<: *node
    ports: ["18092:8090"]

docker compose -f compose.network.yml up → real 3-node HA locally. Kill one container, writes still succeed, reads still succeed, DB never diverges. Same code path as prod.

Acceptance tests

1. Single-node: N=1, archive=off → behaves exactly like today. No regression in core/base_test.go.
2. Docker compose N=3: write 1000 rows, kill base-b, read all 1000 from base-a and base-c. Bring base-b back, verify it catches up (txseq matches). No row loss, no duplication.
3. Archive PITR: write 1000 rows, record txseq, write 100 more. Restore to the first txseq from GCS archive; DB has exactly 1000.
4. k8s operator: apply LiquidBD with network.replication: 3. Verify: headless svc, 3 pods with BASE_PEERS env, HPA, PDB minAvailable=2, archive sidecar when archive != off.
5. Autoscale: drive base_hot_shards > hotShardsTarget via synthetic load; verify KEDA scales Deployment up, new pod joins the network, existing shards rebalance.
6. Gateway: two users in different shards produce writes on different owner pods; both succeed; read-your-writes respected via txseq cookie.

Non-goals

• FUSE. Nothing mounts a fuse filesystem.
• LiteFS. No import, no fork. Pure quasar WAL-hook.
• 3rd-party SQLite processes. This is Base-internal; external apps don't get the WAL-hook.
• Cross-app shard sharing. Each Base service has its own network.

Deprecations

• ~/work/hanzo/base-ha is superseded. After base/network lands and ATS/BD/TA/IAM/KMS/AML migrate, archive base-ha to the deprecations page. The repo stays read-only for history.