zeekay 45d49ecf38 test+docs: production-readiness for VFS-backed stateful workloads ...

Adds five Go tests covering the four code-level concerns that gate
using VFS as the durable store for stateful services like luxd /
zapdb:

  TestCrashRecovery_Synced_Persists                   (#1)
  TestCrashRecovery_UnsyncedWriteIsLost_NotCorrupted  (#1)
  TestCompaction_SustainedWrite                       (#4, opt-in)
  TestBackendOutage_PutErrorBubblesUp                 (#5)
  TestBackendOutage_GetErrorBubblesUp                 (#5)
  TestKeyRotation_MultiRecipientAllowsRollingRotation (#6)

Adds docs/PRODUCTION-READINESS.md addressing all seven concerns —
the four with code, plus three operational runbooks (#2 empty-cache
bootstrap timing, #3 warm-cache bootstrap timing, #7 rollback path
to PVC-local state).

Verdict for blockchain state: VFS is production-grade as the
finalized-block archive tier — finalized blocks are immutable, so
they stream into VFS once (no per-write fsync overhead) and dedup
content-addressed across replicas. Hot tier (zapdb WAL + memtable)
stays on local SSD. The §4 6.4 fsync'd-ops/s benchmark is the
worst case (per-write Sync); the finalized-SST stream pattern looks
nothing like this and is bounded by network bandwidth not metadata
churn.

#2 and #3 are deliberately runbook-only — bootstrap timing needs
a real cluster + S3 backend to measure honestly, not a Go test.
#7 ditto — the procedure is straightforward (vfsd export → tar →
PVC → restore) but execution is operational.

2026-06-02 22:05:19 -07:00

..

PRODUCTION-READINESS.md

test+docs: production-readiness for VFS-backed stateful workloads

2026-06-02 22:05:19 -07:00