ha: single-writer election, decomplected from vfs

HRW (Rendezvous) single-writer election as its own coordination primitive
— no coordinator, lock service, Postgres, or Redis. Answers "who is the
single writer for key K" identically on every replica.

Extracted from hanzoai/vfs/replica/owner.go (HIP-0107), which was
storage-agnostic pure Go with no vfs-internal callers. Election (who
writes) and replication (how state ships) are orthogonal; this is the cut.
Generalizes orgID -> key (any ownership unit).

API: Member, Owner, IsOwner, Replicas + the Membership seam + Static.
Zero deps (stdlib crypto/sha256, encoding/binary, context).

LICENSE

@@ -0,0 +1,17 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+Copyright 2026 Hanzo AI, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

LLM.md

@@ -0,0 +1,33 @@
+# LLM.md — ha
+
+`github.com/hanzoai/ha` — Hanzo's HA coordination primitive. Single-writer election
+over a live membership set via Rendezvous (HRW) hashing. No coordinator, lock
+service, Postgres, or Redis. Pure Go, stdlib-only (`crypto/sha256`,
+`encoding/binary`, `context`), zero external deps.
+
+## What it is / is not
+- **Is:** the answer to "who is the single writer for key K right now", identical on
+  every replica. `Owner` / `IsOwner` / `Replicas` + the `Membership` seam + `Static`.
+- **Is NOT:** storage, replication, or a lock service. It decides WHO writes, never
+  HOW state is stored/shipped. That is `hanzoai/vfs` (SQLite + object store), which
+  composes this for its writer gate.
+
+## Provenance
+Extracted verbatim from `hanzoai/vfs/replica/owner.go` (HIP-0107), which is
+storage-agnostic pure Go and had no vfs-internal callers — it was hosted in a SQLite
+library only for historical reasons. Promoting it to its own package is the
+decomplection: coordination (this) vs replication (vfs) are orthogonal concerns.
+The `key` argument generalizes the original `orgID` (any stable ownership unit: org,
+shard, or singleton job name).
+
+## Consumers
+- `hanzoai/visor` billing (exactly-once hourly metering): `ha.IsOwner("_global", …)`
+  gates lease claims so exactly one replica bills per wall-clock hour.
+- Adoption target (HIP-0107): iam / commerce / base and any singleton workload.
+
+## Roadmap
+- `ha/k8s` (Phase 2): the cluster `Membership` — lists Running+Ready pods by label
+  via the in-cluster K8s API. Currently lives as `k8sMembership` inside visor; folding
+  it here makes the K8s lister the ONE shared source (no per-service copy).
+
+Update THIS file for notes; do not add scratch summary files.

README.md

@@ -0,0 +1,76 @@
+# ha
+
+Hanzo's high-availability coordination primitive: **single-writer election over a
+live membership set** — no coordinator, no lock service, no Postgres, no Redis.
+
+It answers one question, identically on every replica:
+
+> which replica is the single writer for key `K` right now?
+
+via Rendezvous (Highest-Random-Weight) hashing. `N` stateless replicas agree on one
+owner per key and fail over deterministically, with no recomputation and nothing to
+coordinate.
+
+## Why it is its own package
+
+`ha` decides **who writes**. It never touches **how** state is stored or replicated
+— that is [`hanzoai/vfs`](https://github.com/hanzoai/vfs) (per-org SQLite + object
+store) or any other backing. Keeping election separate means a cron tick, a queue
+consumer, or any singleton can be made exactly-once **without importing a storage
+library** to get it. One primitive, one home, every consumer.
+
+```
+ha            who is the single writer          ← this package (pure Go, zero deps)
+vfs           how the SQLite state replicates    → composes ha for its writer gate
+your service  what the single writer does        → cron, billing sweep, queue drain
+```
+
+## Use
+
+```go
+import "github.com/hanzoai/ha"
+
+// The per-write hot path: only the elected owner of key proceeds.
+if ha.IsOwner(orgID, self, members) {
+    // ... this replica is the single writer for orgID; do the write.
+}
+```
+
+`members` comes from a `Membership` — the injectable seam:
+
+```go
+// Single process / local dev / tests: itself is the sole writer.
+m := ha.Static(podName)
+
+// Production: a cluster source (e.g. hanzoai/ha/k8s) lists live, ready peers.
+self := m.Self()
+members, err := m.Members(ctx)
+if err != nil || len(members) == 0 {
+    // FAIL CLOSED — unknown or empty membership has no safe owner; skip.
+    return
+}
+if ha.IsOwner(orgID, self, members) { /* write */ }
+```
+
+## API
+
+| Symbol | Purpose |
+|--------|---------|
+| `Member{ID, Addr}` | one replica; `ID` is the stable HRW identity, `Addr` for write-forwarding |
+| `Owner(key, members) (Member, bool)` | the elected single writer for `key`; `ok=false` on empty set (fail-closed) |
+| `IsOwner(key, self, members) bool` | the per-write gate every replica runs |
+| `Replicas(key, members, n) []Member` | owner first, then ordered failover successors (pre-warm order) |
+| `Membership` | seam: `Self()` + `Members(ctx)` — the live set election elects over |
+| `Static(id)` | single-process `Membership` (one member, itself, never an error) |
+
+## Guarantees
+
+- **Deterministic & order-independent** — same `(key, members)` ⇒ same owner on
+  every replica, regardless of set order (ties break on `ID`).
+- **Fail-closed** — empty membership yields no owner; a consumer that cannot read
+  its peers must skip, never assume ownership.
+- **Even spread** — distinct keys distribute across replicas (HRW), not all pinned
+  to one, so ownership (and its write load) balances.
+- **Cheap** — a per-write check is a handful of SHA-256s; see `BenchmarkOwnerElection`.
+
+Apache-2.0.

VERSION

@@ -0,0 +1 @@
+0.1.0

doc.go

@@ -0,0 +1,22 @@
+// Copyright 2025 Hanzo AI Inc. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+
+// Package ha is Hanzo's high-availability coordination primitive: single-writer
+// election over a live membership set, with no coordinator, no lock service, no
+// Postgres, no Redis. It answers one question identically on every replica —
+// "which replica is the single writer for key K right now?" — via Rendezvous
+// (Highest-Random-Weight) hashing, so N stateless replicas agree on one owner per
+// key and fail over deterministically with no recomputation.
+//
+// ha decides WHO writes; it never touches HOW state is stored or replicated. That
+// separation is the point: compose ha with hanzoai/vfs (per-org SQLite + object
+// store) for HA storage, or with a cron tick, a queue consumer, or any singleton
+// that must run exactly once across a multi-replica Deployment. The same election
+// guards all of them.
+//
+// The seam is Membership: election is a pure function of (key, []Member), and a
+// Membership supplies the live set. Wire a cluster source (hanzoai/ha/k8s) in
+// production, or Static for a single process and tests.
+package ha

election.go

@@ -0,0 +1,104 @@
+// Copyright 2025 Hanzo AI Inc. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+
+package ha
+
+// Single-writer election WITHOUT a coordinator. Answers one question, identically
+// on every replica: "which replica is the single writer for key K?" via Rendezvous
+// (Highest-Random-Weight) hashing over the live membership set. No election
+// protocol, no lock service, no service discovery, no Postgres, no Redis.
+//
+// The key is any stable string that names a unit of ownership — an org id (the
+// owning replica writes ALL of that org's databases, so the org's data has locality
+// and moves as one unit on failover), a shard id, or a singleton job name. Reads are
+// served from any replica; only writes are gated. A unit that outgrows one writer is
+// a future sub-shard concern (hash key+shard); one-key-one-owner is the correct,
+// simple default and the natural isolation boundary.
+
+import (
+	"crypto/sha256"
+	"encoding/binary"
+)
+
+// Member is one replica in the live membership set. ID must be stable for the
+// life of the replica (e.g. its pod name / a persistent node id): HRW weights
+// derive from it. Addr is the reachable address for write-forwarding (host:port).
+type Member struct {
+	ID   string
+	Addr string
+}
+
+// Owner returns the replica that owns the writer for key, or ok=false when members
+// is empty (fail-closed — never a wrong writer). Deterministic: the same
+// (key, members) yields the same Owner on every replica, independent of order.
+func Owner(key string, members []Member) (Member, bool) {
+	if len(members) == 0 {
+		return Member{}, false
+	}
+	best := members[0]
+	bestScore := weight(key, best.ID)
+	for _, m := range members[1:] {
+		s := weight(key, m.ID)
+		// Tie-break on ID so the result is order-independent.
+		if s > bestScore || (s == bestScore && m.ID < best.ID) {
+			best, bestScore = m, s
+		}
+	}
+	return best, true
+}
+
+// IsOwner reports whether selfID owns the writer for key under members — the
+// per-write hot-path check every replica runs.
+func IsOwner(key, selfID string, members []Member) bool {
+	o, ok := Owner(key, members)
+	return ok && o.ID == selfID
+}
+
+// Replicas returns key's owner first, then ordered failover successors. On owner
+// loss the next replica becomes owner with no recomputation, so a reader can
+// pre-warm the state for the keys it is next-in-line to own.
+func Replicas(key string, members []Member, n int) []Member {
+	if n <= 0 || len(members) == 0 {
+		return nil
+	}
+	type scored struct {
+		m Member
+		s uint64
+	}
+	all := make([]scored, len(members))
+	for i, m := range members {
+		all[i] = scored{m, weight(key, m.ID)}
+	}
+	for i := 1; i < len(all); i++ {
+		for j := i; j > 0; j-- {
+			a, b := all[j], all[j-1]
+			less := a.s > b.s || (a.s == b.s && a.m.ID < b.m.ID)
+			if !less {
+				break
+			}
+			all[j], all[j-1] = all[j-1], all[j]
+		}
+	}
+	if n > len(all) {
+		n = len(all)
+	}
+	out := make([]Member, n)
+	for i := 0; i < n; i++ {
+		out[i] = all[i].m
+	}
+	return out
+}
+
+// weight is the HRW score for (key, member): the first 8 bytes of
+// SHA-256(key || 0x00 || memberID) as a big-endian uint64. The NUL separator
+// stops ("ab","c") and ("a","bc") from colliding.
+func weight(key, memberID string) uint64 {
+	h := sha256.New()
+	_, _ = h.Write([]byte(key))
+	_, _ = h.Write([]byte{0})
+	_, _ = h.Write([]byte(memberID))
+	sum := h.Sum(nil)
+	return binary.BigEndian.Uint64(sum[:8])
+}

election_test.go

@@ -0,0 +1,90 @@
+// Copyright 2025 Hanzo AI Inc. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+
+package ha
+
+import (
+	"fmt"
+	"testing"
+)
+
+// TestOwnerDeterministicAndExclusive proves the HRW election picks exactly one
+// owner, order-independently, and every member agrees.
+func TestOwnerDeterministicAndExclusive(t *testing.T) {
+	members := []Member{{ID: "r3"}, {ID: "r1"}, {ID: "r2"}}
+	reordered := []Member{{ID: "r1"}, {ID: "r2"}, {ID: "r3"}}
+
+	o1, ok1 := Owner("acme", members)
+	o2, ok2 := Owner("acme", reordered)
+	if !ok1 || !ok2 || o1.ID != o2.ID {
+		t.Fatalf("owner not order-independent: %q vs %q", o1.ID, o2.ID)
+	}
+
+	// Exactly one member is the owner across the set.
+	owners := 0
+	for _, m := range members {
+		if IsOwner("acme", m.ID, members) {
+			owners++
+		}
+	}
+	if owners != 1 {
+		t.Fatalf("IsOwner true for %d members, want exactly 1", owners)
+	}
+
+	// Empty membership fails closed (no owner).
+	if _, ok := Owner("acme", nil); ok {
+		t.Fatal("empty membership must have no owner")
+	}
+
+	// Distinct keys spread across replicas (not all pinned to one).
+	seen := map[string]bool{}
+	for _, key := range []string{"a", "b", "c", "d", "e", "f", "g", "h"} {
+		o, _ := Owner(key, members)
+		seen[o.ID] = true
+	}
+	if len(seen) < 2 {
+		t.Fatalf("HRW pinned all keys to %d replica(s); want spread", len(seen))
+	}
+}
+
+// TestReplicasOwnerFirstThenFailover proves Replicas ranks the owner first and
+// returns an ordered, bounded successor list — the failover pre-warm order.
+func TestReplicasOwnerFirstThenFailover(t *testing.T) {
+	members := []Member{{ID: "r1"}, {ID: "r2"}, {ID: "r3"}, {ID: "r4"}}
+
+	// The head of Replicas is exactly the elected Owner.
+	owner, _ := Owner("acme", members)
+	rep := Replicas("acme", members, 3)
+	if len(rep) != 3 {
+		t.Fatalf("Replicas returned %d, want 3", len(rep))
+	}
+	if rep[0].ID != owner.ID {
+		t.Fatalf("Replicas[0]=%q, want owner %q", rep[0].ID, owner.ID)
+	}
+
+	// No duplicates, and n is clamped to the member count.
+	if got := Replicas("acme", members, 99); len(got) != len(members) {
+		t.Fatalf("Replicas clamp: got %d, want %d", len(got), len(members))
+	}
+	if got := Replicas("acme", members, 0); got != nil {
+		t.Fatalf("Replicas(n=0)=%v, want nil", got)
+	}
+	if got := Replicas("acme", nil, 3); got != nil {
+		t.Fatalf("Replicas(empty)=%v, want nil", got)
+	}
+}
+
+// BenchmarkOwnerElection measures the per-write HRW single-writer check (the hot path
+// gating every write when scaled to N replicas).
+func BenchmarkOwnerElection(b *testing.B) {
+	members := make([]Member, 16)
+	for i := range members {
+		members[i] = Member{ID: fmt.Sprintf("replica-%02d", i)}
+	}
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		_ = IsOwner("some-org-slug", "replica-07", members)
+	}
+}

go.mod

@@ -0,0 +1,3 @@
+module github.com/hanzoai/ha
+
+go 1.26.4

membership.go

@@ -0,0 +1,39 @@
+// Copyright 2025 Hanzo AI Inc. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+
+package ha
+
+import "context"
+
+// Membership is the seam election elects over: the live, writer-eligible replica
+// set plus this replica's own stable ID. It is the ONE injectable source every
+// consumer shares — production wires a cluster source (e.g. hanzoai/ha/k8s), tests
+// and single-process mode wire Static.
+//
+// A consumer MUST fail CLOSED on a Members error or an empty set: election over an
+// unknown or empty membership has no safe owner, so the caller skips rather than
+// risk two writers. A nil error with a non-empty set is the only "proceed" signal.
+type Membership interface {
+	// Self is this replica's stable ID — the value HRW weights derive from. It must
+	// be stable for the replica's life (e.g. its pod name / a persistent node id).
+	Self() string
+	// Members is the live writer-eligible set, or an error the caller fails closed
+	// on. A nil error with a non-empty set is the only signal safe to proceed on.
+	Members(ctx context.Context) ([]Member, error)
+}
+
+// Static is the single-process Membership: one member, itself, never an error — the
+// sole process is the sole writer. It is the correct source for local dev, a
+// standalone binary, and tests, and the safe default when no cluster source is
+// wired (a single process is exactly-once by construction). id is this member's ID.
+func Static(id string) Membership { return static(id) }
+
+type static string
+
+func (s static) Self() string { return string(s) }
+
+func (s static) Members(context.Context) ([]Member, error) {
+	return []Member{{ID: string(s)}}, nil
+}

membership_test.go

@@ -0,0 +1,32 @@
+// Copyright 2025 Hanzo AI Inc. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+
+package ha
+
+import (
+	"context"
+	"testing"
+)
+
+// TestStaticIsSelfOwner proves single-process Membership always elects itself: the
+// sole member owns every key, so a standalone binary is exactly-once by construction.
+func TestStaticIsSelfOwner(t *testing.T) {
+	m := Static("solo")
+	if m.Self() != "solo" {
+		t.Fatalf("Self()=%q, want solo", m.Self())
+	}
+	members, err := m.Members(context.Background())
+	if err != nil {
+		t.Fatalf("Members: %v", err)
+	}
+	if len(members) != 1 || members[0].ID != "solo" {
+		t.Fatalf("Members=%v, want [{solo}]", members)
+	}
+	for _, key := range []string{"a", "b", "c"} {
+		if !IsOwner(key, m.Self(), members) {
+			t.Fatalf("static member must own every key; missed %q", key)
+		}
+	}
+}