mirror of
https://github.com/luxfi/database.git
synced 2026-10-30 15:01:18 +00:00
zeekayandHanzo Dev
fb16ece6a3
Replicates a zapdb to S3 via hanzoai/vfs with no keyspace scan in steady state and post-quantum encryption out of the box. - CDC incrementals via db.Subscribe (the WAL tail), not Backup(since)'s O(total) scan — a 100-key delta out of 200k keys: 2.76us/19KB vs 20ms/1GB (~7000x). - Physical SST-copy snapshots (sparse, hole-aware): 134ms/25MB vs logical 158ms/1.25GB/2.2M allocs. Hydrated before Badger opens; incrementals replay on top. - Streaming codec: Backup -> zstd -> age into a pooled buffer (peak mem = compressed). - Post-quantum (ML-KEM-768 + X25519 via luxfi/age v1.5.0), client-side before S3. Out of the box via REPLICATE_AGE_KEY_FILE (generate+persist, recipient derived). - Per-DB streams (REPLICATE_AUTO_ON_OPEN) for subprocess VM plugins; backer/restorer split (REPLICATE_BACKUP / REPLICATE_SOURCE_ORDINAL) for multi-node. - Public imports only (hanzoai/vfs v0.4.1, no replace). Verified live on an EVM C-chain end-to-end incl. disaster-recovery exact-match. Canonical KV schema (zapdb/zapdb.zap) + zero-copy codec (zapkv.go). Renames engine alias badger->zdb. Co-authored-by: Hanzo Dev <[email protected]>
41 lines
1.9 KiB
Plaintext
41 lines
1.9 KiB
Plaintext
# ZAP ZapDB backup schema v1
|
|
#
|
|
# Canonical, language-agnostic wire format for a ZapDB (Badger fork) backup
|
|
# stream: a length-framed sequence of KV records. This is the SINGLE SOURCE OF
|
|
# TRUTH that replaces the hand-rolled sequential encoder in
|
|
# github.com/luxfi/zapdb/pb/marshal_zap.go (which is "zap" in name only — a
|
|
# sequential [u32 len][bytes]… layout with no fixed offsets and no zero-copy
|
|
# reads).
|
|
#
|
|
# A KV here is a true ZAP struct: a fixed 48-byte header of scalars and
|
|
# (offset:u32, length:u32) pointer pairs, with the variable-length key / value /
|
|
# meta payloads living in the heap area that follows. A reader resolves any field
|
|
# in O(1) from its compile-time offset — no scanning, no per-field allocation.
|
|
#
|
|
# Backup framing (unchanged from the current stream so a bucket stays mixed-
|
|
# readable during migration): each record is `len(KV):u64-LE || KV bytes`,
|
|
# concatenated; EOF terminates. The fixed-header/heap/pointer LAYOUT rules this
|
|
# schema is written against live in github.com/zap-proto/zap-spec (SPEC.md); this
|
|
# file is ZapDB's own schema, colocated with its reference codec (zapkv.go).
|
|
#
|
|
# Wire stability: this is the v1 ZapDB wire format. Per the project's
|
|
# "no backwards compatibility, only forwards perfection" rule, a shape change
|
|
# means a new struct name, not a silent offset shift.
|
|
|
|
package zapdb
|
|
|
|
# A single key/value version entry. Fixed header = 48 bytes; heap follows.
|
|
struct KV {
|
|
Version u64 @0 # Badger commit timestamp (monotonic version)
|
|
ExpiresAt u64 @8 # unix-seconds TTL, 0 = never
|
|
Key bytes @16 # (offset:u32, length:u32) into heap
|
|
Value bytes @24 # (offset:u32, length:u32) into heap
|
|
UserMeta bytes @32 # (offset:u32, length:u32) into heap — usually 1 byte
|
|
Meta bytes @40 # (offset:u32, length:u32) into heap
|
|
}
|
|
|
|
# A batch of KV records (the unit Badger's Stream framework emits per key range).
|
|
struct KVList {
|
|
Kv list<KV> @0
|
|
}
|