base/docs/NETWORK.md

Base Network

PQ-native, consensus-replicated HA for every Base app. One mechanism. One env flag. Replica count N ∈ {1,2,3,…} works identically on a laptop (docker compose) and on k8s (via ~/work/hanzo/operator). No FUSE, no LiteFS, no Consul.

Durability and replication come from luxfi/consensus/protocol/quasar — the same DAG consensus that runs Lux validators — applied to SQLite WAL frames, one DAG per shard. Archive tier is optional cold storage in S3 (s3://) for PITR.

Problem

Base apps (IAM, KMS, AML, and every app built on Base) each carry SQLite files — often per-user or per-org — and need HA with real durability, no split-brain, and uniform ops across dev / test / main.

Pre-existing options fall short:

• Raw StatefulSet + RWO PVC, replicas > 1 → split-brain, silent DB divergence.
• LiteFS + Consul → three moving parts (FUSE, LTX, Consul) and a single-primary election we don't need; we already have quasar.
• Litestream + single writer → backup, not HA.
• Postgres → violates Base's "embedded SQLite by default".

Goal

• Any Base app opts in with one env: BASE_NETWORK=quasar.
• BASE_REPLICATION=1 works (standalone, zero consensus overhead, still ships to archive).
• BASE_REPLICATION=N works (full quasar quorum, k = ⌊N/2⌋+1).
• docker compose up = true HA locally.
• Operator emits the env surface + HPA/KEDA from one CR field.

Design

Data model

• A Base app owns many shards. Shard key = user_id | org_id | <custom header>.
• Each shard = one SQLite file on the pod's local PVC (hot cache).
• Each shard is replicated across BASE_REPLICATION network members (that shard's quasar subset). Assignment = consistent-hash over live members.
• A shard's finalized WAL-frame DAG IS its durability tier. Archive is a cheap, optional second tier.

Shard hierarchy + per-shard encryption

Multi-tenant Base apps produce many shards. The model is two-level physical files plus a fixed logical consensus grouping:

• Physical shard = one SQLite file. Files exist at two scopes:
  • platform — _platform.db (rare; e.g. IAM app registry).
  • org — <org_id>.db (tenant-wide state).
  • user — <org_id>/<user_id>.db (per-customer state). A single request may touch multiple scopes in one transaction; Base opens the right handle based on the route's declared scope.
• Virtual shard (vshard) = consensus group. A fixed-cardinality ring (default 256, power-of-two) is hashed over at connection time: vshard = hash(shardKey) mod 256. Each vshard has a quasar validator subset of size BASE_REPLICATION. All physical files whose shardKey maps to vshard V replicate together. This keeps consensus metadata O(256) instead of O(customers).
• Assignment of physical pods to vshards = consistent-hash over live network members. Adding a pod moves ~1/N of the vshards to the new pod; physical files on those vshards migrate via quasar tail-sync (not a file copy — new pod subscribes to the vshard's DAG and materialises from frames).

Encryption at rest is decided a layer up, and not here.

This section used to describe a per-file scheme with its own DEKs, .dekwrap files beside each shard, an LRU of unwrapped keys and a KEK-of-KEKs — and an env table for driving it. None of that was ever built, and the variables the table listed (BASE_VSHARDS, BASE_KMS_URL, BASE_KEK_SCOPE, BASE_DEK_CACHE_SIZE) are read by no code in this repo. A knob that does nothing is worse than a missing one: it reads as a control an operator has already exercised.

What is real: an org's Base opens under a key derived for that org (plugins/org, hanzoai/cek), so one tenant's file is unreadable with another's key. The process's own database takes its protection from the volume it sits on. BASE_ENCRYPT is refused at startup rather than ignored, because setting it and seeing no complaint is how a deployment comes to believe it has page encryption.

A frame leaving a pod therefore carries whatever the page held. Treat the replication plane and the archive bucket as holding the same data the database does.

Write path

client ──► gateway (base-network://svc)
            │   JWT→shardKey, consistent-hash → writer pod
            ▼
         writer pod
            │   handle request, SQLite commit
            ▼
         sqlite3_wal_hook fires frame F
            │
            ▼
         network.SubmitFrame(shardID, F)
            │   quasar vertex, PQ-signed
            ▼
         quasar finalize (k-of-N ack)
            │
            ├──► apply callback: write F to local SQLite (writer + replicas)
            └──► optional: archive validator appends F to S3/GCS segment

No round-trip to a coordinator; no FUSE; no separate leader-election service. A finalized frame IS the commit.

Read path

• Any pod in the shard's subset can serve reads.
• Gateway sets a txseq cookie with the last-seen finalized seq.
• Target pod checks localSeq ≥ txseq:
  • yes → serve.
  • no → either block briefly on quasar tail (typical < 100 ms) or 307 to a caught-up peer.

Membership

• Every Base pod is a quasar validator.
• Peers discovered from BASE_PEERS seed list (CSV of DNS), gossip thereafter.
• In k8s: headless Service, peers = pod-0…N-1.svc.cluster.local:9999.
• In compose: static DNS names.
• Heartbeat TTL 1.5 s. On miss: the pod's shards are reassigned across survivors; new assignments sync the DAG from peers before accepting writes.

Replication factor

N	quasar mode	k (acks needed)	failure mode
1	standalone	1	No HA. Archive = durability.
2	pair	2	1-down blocks writes, reads survive.
3+	quorum	⌊N/2⌋+1	Byzantine-safe, PQ-signed.

Same code path for every N; only the k constant changes.

Archive tier (optional)

A witness-only validator runs alongside as BASE_NODE_ROLE=archive. It does not vote. It subscribes to finalized frames and appends per-shard WAL segments to object storage. Layout:

<scheme>://<bucket>/<svc>/<shard>/<seq-prefix>/<segment>.lbn

.lbn = Lux Base Network segment — a compact length-prefixed list of PQ-signed, quasar-finalized frames. Replay any prefix ⇒ SQLite at that txseq. PITR for free.

The backend is S3 (s3://bucket/prefix), served by hanzoai/s3 or AWS. off disables the archive, leaving the quasar DAG as the only durability.

Env surface (one shape, every Base app)

var	values	notes
BASE_NETWORK	quasar | standalone (default)	standalone = today's behaviour, no network.
BASE_SHARD_KEY	user_id | org_id | header:<Name>	required when BASE_NETWORK=quasar. Names the shard's ONE source: a field on the verified identity, or the named request header.
BASE_REPLICATION	1 | 2 | 3 | …	≤ BASE_PEERS count.
BASE_PEERS	CSV of host:port DNS	operator-emitted in k8s; explicit in compose.
BASE_NODE_ROLE	validator (default) | archive
BASE_ARCHIVE	s3://bucket/prefix | off (default)
BASE_LISTEN_HTTP	:8090 default	Read, but opens nothing; serve --http sets the HTTP address.
BASE_LISTEN_P2P	host:port, :9999 default	quasar p2p listener. The host is honoured: :9999 is every interface, 127.0.0.1:9999 loopback only.

The R6 backlog caps are constants in network/archive.go: 64 MiB and 100,000 segments per shard. No variable changes them.

The peer transport presents no certificate.

Four names used to stand in this table — BASE_TLS_CA, BASE_TLS_SERVER_CERT, BASE_TLS_SERVER_KEY, BASE_TLS_ALLOWED_SANS — against a config surface that no request ever reached. They are refused at startup now, for the reason BASE_ENCRYPT is: an operator who mounts a keypair, sets one of these and sees no complaint has been told that peers authenticate.

Wiring it needs two things this repo does not have. The transport is a luxfi/zap node, which takes one *tls.Config and uses it both for the listener it binds and for every peer it dials — and Go dials with the config verbatim, so one config cannot name the several peers it is dialling. That is upstream. Certificates also need an issuer: the estate's cert-manager builds internal CAs with pod-DNS SANs elsewhere in the cluster, so the pattern exists, but nothing issues one for a Base pod today.

Until both land, peers authenticate by reachability — see the trust-set note below.

What the transport and archive enforce

The R-numbers are the labels the env surface above uses; each is glossed here so the table reads on its own.

• R1 (frame/envelope shardID binding): inbound envelopes whose inner Frame.ShardID disagrees with Envelope.ShardID are rejected at the transport boundary; the apply loop re-checks as defence-in-depth.
• R2 (localSeq monotonic): Shard.localSeq advances strictly by +1 per finalised frame. Out-of-order frames buffer (cap 1024). A frame arriving far ahead of the sequence waits there for predecessors that never come, so it cannot advance localSeq and the gateway's txseq check cannot read as "caught up".
• R3 (archive authorship): segments are LBN2 — Ed25519 signature over body || CRC32 || pubkey. Writers refuse to encode without a signer; readers refuse to decode without a verifier; LBN1 hard- rejected as a downgrade vector.
• R4 (StatefulSet by default): when network.enabled: true, the operator emits a StatefulSet (spec.network.workload: StatefulSet default) so BASE_PEERS pod-ordinal DNS resolves. Deployment is permitted only at replicas == 1.
• R5 (peer identity): not held. The transport establishes none, and the config surface that read as though it did is gone — see the note under the env table, and the trust-set property below.
• R6 (backlog caps): per-shard cap 64 MiB / 100 k segments, drop-oldest with base_shard_backlog_drops_total metric.
• R7 (HPA workload kind): operator threads workload kind into the HPA / KEDA scaleTargetRef.kind so a StatefulSet-backed service autoscales correctly.
• R8 (nanos-suffixed object keys): objectKey(…, startSeq, nanos) produces unique keys per flush; Range dedupes by (startSeq + frameIndex) when multiple segments overlap.

What this layer leaves to you

Standing properties of the design, each with the thing you do about it.

• BASE_PEERS is the trust set, and reachability is the whole of it. The transport carries no peer identity, so a peer is whatever answers at one of those addresses, and the NodeID it states in the handshake is its own word. Every entry may submit frames for any shard this node owns; there is no second, finer authority inside the set. Keep the list to peers of the same service in the same namespace, and put a NetworkPolicy in front of BASE_LISTEN_P2P — a cluster that allows all in-namespace ingress puts that port in reach of every pod in it. Treat adding an entry as granting write access to the group's shards.

• BASE_SHARD_KEY: header:<Name> lets the client choose its shard. A header is whatever the caller sends. It exists because compose dev needs a shard key with no token, and it is the only form that reads one — every other form reads the verified identity, so a request carrying none resolves no shard and runs local. Use user_id or org_id in anything but local dev. The shard picks which pod serves a write, so a caller choosing its own picks its own writer.

• BASE_ENCRYPT is refused. Setting it stops the process with a message naming what actually protects data at rest, rather than starting quietly and leaving the operator to assume. The encryption that is real is a layer up: per-org Base shards open under a per-org key. Put the data directory on an encrypted volume.

• A shard costs an engine. Each holds a 1024-entry channel, and nothing has been exercised past single-digit shard counts. Pick a shard key whose cardinality you can bound — org_id over user_id where a tenant is the natural unit — and size before you scale.

• network/attack_vectors_test.go is the executable half of this section. The tests that pass are the defences listed above; the tests that skip name what this package does not defend, and print themselves on every CI run. The network-attack-suite gate in hanzo.yml holds the two apart — a defence that stopped holding cannot become a skip, and a marker cannot quietly become an assertion.

Layout

~/work/hanzo/base/network/ (new package)

network/
  network.go    // NewNetwork(cfg) → *Network; Start/Stop; public API below.
  config.go     // env parsing, defaults, validation.
  node.go       // Node identity (stable from pod name or $HOSTNAME).
  quasar.go     // thin wrapper over luxfi/consensus/protocol/quasar.
  wal.go        // install sqlite3_wal_hook; frame serialise/PQ-sign.
  apply.go      // on-finalize callback → sqlite_apply().
  shard.go      // Shard struct: assignment, local cache, txseq.
  router.go     // consistent-hash ring; who owns shardID?
  archive.go    // BASE_NODE_ROLE=archive loop → hanzoai/s3 or GCS.
  metrics.go    // Prom: base_shards, base_wal_lag_bytes,
                //       base_hot_shards, base_lease_contentions,
                //       base_archive_lag_bytes, ...
  network_test.go
  wal_test.go
  router_test.go
  integration_test.go  // docker compose N=3 smoke: write, crash a node, read, restore.

Public API that Base calls from core/base.go on startup:

package network

type Network interface {
    Start(ctx context.Context) error
    Stop(ctx context.Context) error

    // Installed by Base onto the sqlite *Conn; intercepts commits.
    InstallWALHook(conn *sqlite.Conn, shardID string) error

    // Given a shardID, returns the current writer endpoint plus
    // whether it's the local pod. Gateway uses this.
    WriterFor(shardID string) (endpoint string, local bool)

    // Any member in the shard subset. Gateway uses this for reads.
    MembersFor(shardID string) []string

    // Metrics plumbing.
    Metrics() *Metrics
}

func New(cfg Config) (Network, error)

Base itself (core/base.go) gains:

if net := network.FromEnv(); net.Enabled() {
    b.net = net
    b.db.OnConnect(func(c *sqlite.Conn, shardID string) error {
        return net.InstallWALHook(c, shardID)
    })
}

That is all the coupling. Rest of Base code is unchanged.

~/work/hanzo/operator/ (reconciler update)

Add BaseNetworkSpec struct reused across Base-ish CRDs:

pub struct BaseNetworkSpec {
    pub enabled: bool,
    pub shard_key: String,            // user_id | org_id | <hdr>
    pub replication: u8,              // ≤ replicas
    pub archive: Option<String>,      // gs://… | s3://… | off
    pub autoscale: Option<Autoscale>,
}

pub struct Autoscale {
    pub min: u8,
    pub max: u8,
    pub cpu_target: Option<u8>,
    pub memory_target: Option<u8>,
    pub hot_shards_target: Option<u32>,
}

Added to: ServiceSpec, IAMSpec, KMSSpec, and any other downstream Base-backed CRDs.

New controller.rs helper:

reconcile_base_network(api, spec, meta) -> ();

emits, per replica-managed workload:

• Headless Service (clusterIP: None) for pod-ordinal peer DNS.
• Env on pods: BASE_NETWORK, BASE_SHARD_KEY, BASE_REPLICATION, BASE_PEERS (computed from spec.replicas + headless svc), BASE_ARCHIVE, BASE_NODE_ROLE=validator.
• Optional archive sidecar Deployment (BASE_NODE_ROLE=archive) when archive != off.
• HPA (cpu, memory) and KEDA ScaledObject (Prom metrics base_hot_shards, base_wal_lag_bytes) when autoscale set.
• PodDisruptionBudget: minAvailable = k where k = ⌊N/2⌋+1 for N≥3, N-1 for N=2, 0 for N=1.

CR shape:

spec:
  replicas: 3
  network:
    enabled: true
    shardKey: user_id
    replication: 3
    archive: gs://hanzo-base-wal/svc
    autoscale:
      min: 3
      max: 7
      cpuTarget: 70
      memoryTarget: 75
      hotShardsTarget: 500

~/work/hanzo/gateway/ (new upstream type)

base-network://<service> upstream.

• Parses JWT sub or configured header → shardID.
• Polls GET /-/base/members on the service's headless Service every 5 s; caches member list.
• Consistent-hashes shardID → owner pod.
• Writes → owner pod. Reads → random pick from MembersFor(shardID), propagating txseq cookie.
• On 307 from a not-yet-caught-up replica, retries to another member.
• Passes X-Base-Shard: <shardID> upstream for logging.

Config example:

upstreams:
  - name: my-svc
    type: base-network
    service: my-svc.hanzo.svc:8090
    shardKey: user_id
    shardKeySource: jwt.sub

Local dev — docker compose parity

~/work/hanzo/base/compose.network.yml:

services:
  base-a: &node
    image: ghcr.io/hanzoai/base:dev
    environment:
      BASE_NETWORK: quasar
      BASE_SHARD_KEY: user_id
      BASE_REPLICATION: 3
      BASE_PEERS: base-a:9999,base-b:9999,base-c:9999
      BASE_ARCHIVE: off
    ports: ["18090:8090"]
  base-b:
    <<: *node
    ports: ["18091:8090"]
  base-c:
    <<: *node
    ports: ["18092:8090"]

docker compose -f compose.network.yml up → real 3-node HA locally. Kill one container, writes still succeed, reads still succeed, DB never diverges. Same code path as prod.

Acceptance tests

• Single-node: N=1, archive=off → behaves exactly like today. No regression in core/base_test.go.
• Docker compose N=3: write 1000 rows, kill base-b, read all 1000 from base-a and base-c. Bring base-b back, verify it catches up (txseq matches). No row loss, no duplication.
• Archive PITR: write 1000 rows, record txseq, write 100 more. Restore to the first txseq from the S3 archive; DB has exactly 1000.
• k8s operator: apply a Base-backed CR with network.replication: 3. Verify: headless svc, 3 pods with BASE_PEERS env, HPA, PDB minAvailable=2, archive sidecar when archive != off.
• Autoscale: drive base_hot_shards > hotShardsTarget via synthetic load; verify KEDA scales Deployment up, new pod joins the network, existing shards rebalance.
• Gateway: two users in different shards produce writes on different owner pods; both succeed; read-your-writes respected via txseq cookie.

Non-goals

• FUSE. Nothing mounts a fuse filesystem.
• LiteFS. No import, no fork. Pure quasar WAL-hook.
• 3rd-party SQLite processes. This is Base-internal; external apps don't get the WAL-hook.
• Cross-app shard sharing. Each Base service has its own network.

Deprecations

• base-ha is superseded. Once base/network has landed and the services on it have migrated, archive base-ha to the deprecations page. The repo stays read-only for history.