mirror of
https://github.com/hanzoai/replicate.git
synced 2026-10-30 18:23:20 +00:00
zeekayandGitHub
36ef0867fd
* file: add file:// Leaser for filesystem-backed CAS Mirrors the S3 Leaser CAS semantics using POSIX O_EXCL for initial creation and content-hash ETag matching for renew/release. Atomic rename for the cutover. Same Leaser interface contract — callers swap backends without code changes. Use cases: per-process / single-node leases for dev + tests, container-local coordination where S3 is overkill, and cross-process tests of leader/standby patterns without spinning up MinIO. Tests cover: fresh acquire, expired-lease takeover, contended acquire (LeaseExistsError), renew bumps expiry + etag without touching generation, renew loses race to a stealer (ErrLeaseNotHeld), release roundtrip, release after steal (ErrLeaseNotHeld), and concurrent acquire with exactly one winner across 32 goroutines. * file: harden Leaser — MaxTTL clamp + foreign-owner renew guard Closes red findings #1 (HIGH) and #4 (MEDIUM) on the per-market shard layer's lease coordinator. #1 — Server-side MaxTTL clamp: - Add Leaser.MaxTTL (default 30s when zero). - Acquire + Renew reject TTL > MaxTTL with ErrTTLExceedsMax. Without this, one malicious or buggy caller pins the lease for an arbitrary duration regardless of liveness. - readLease rejects an on-disk ExpiresAt > now + MaxTTL with ErrLeaseCorrupt — defense in depth against a writer that bypassed the API. #4 — Owner check in RenewLease: - ETag CAS alone permits a caller who read another Leaser's ETag from disk to craft a renew call carrying any Owner string. We now read the on-disk record first and reject when existing.Owner != l.Owner with ErrLeaseNotHeld. ETag CAS remains primary; Owner check is belt-and-suspenders. Tests: - TestLeaser_RejectsTTLBeyondMax (acquire, renew, zero MaxTTL falls back to default) - TestLeaser_ReadRejectsCorruptedTTL (hand-write 1h ExpiresAt, Acquire rejects) - TestLeaser_RejectsRenewByForeignOwner (foreign Owner with stolen ETag → ErrLeaseNotHeld) All existing leaser tests continue to pass. * leaser: surface MaxLeaseTTL() on the interface Heartbeat schedulers depend on the leaser's TTL ceiling to validate that their renew cadence is at least 2x tighter than the server-side bound. Without surfacing it, callers either hardcode the bound or compute it from the leaser type — both brittle. The struct field stays MaxTTL (configuration knob); the interface method is MaxLeaseTTL() to avoid the Go name-collision rule ("field and method with the same name"). file:// and s3:// both default to DefaultLeaseMaxTTL (30s) when unset. * dblock: expose SQLite-compatible byte-range lock as a public package External callers that mirror Replica.applyLTXFile (e.g. shard-layer exclusive-lock primitive so their apply pipeline interlocks with SQLite's own locking and with the upstream applier. dblock.LockExclusive / dblock.Unlock target the SQLite pending-byte (0x40000000) and shared-region addresses, matching the existing internal.LockFileExclusive contract used by Replica.applyLTXFile. The internal/lock_*.go shims now forward to dblock so there is one canonical implementation across the module. Behavior identical to the previous internal/ implementation — file-level tests in file/ pass. * chore: bump go to 1.26.4 Picks up upstream security fixes for crypto/x509, mime, net/textproto, crypto/fips140. No dependency bumps — only the toolchain directive. go test -race ./... is green. * chore: fix gofmt + goimports drift across repo Pre-existing formatting drift on main exposed by go-imports-repo pre-commit hook. Pure whitespace alignment + import ordering — no logic changes. Touches 16 files unrelated to the file:// Leaser PR.
61 lines
1.8 KiB
Go
61 lines
1.8 KiB
Go
//go:build aix || darwin || dragonfly || freebsd || linux || netbsd || openbsd || solaris
|
|
|
|
// Package dblock exposes the SQLite-compatible byte-range lock used
|
|
// by Replica.applyLTXFile. External callers that mirror the LTX apply
|
|
// pipeline (e.g. shard-layer standby readers) take the same lock so
|
|
// they interlock with SQLite's own locking and with each other.
|
|
//
|
|
// The pending-byte + shared-region ranges are SQLite-internal
|
|
// addresses (sqlitePendingByte=0x40000000); locking those bytes
|
|
// blocks any SQLite reader/writer process that opens the same file,
|
|
// matching the granularity used by replicate's internal applyLTXFile.
|
|
package dblock
|
|
|
|
import (
|
|
"os"
|
|
|
|
"golang.org/x/sys/unix"
|
|
)
|
|
|
|
const (
|
|
sqlitePendingByte = 0x40000000
|
|
sqliteSharedFirst = sqlitePendingByte + 2
|
|
sqliteSharedSize = 510
|
|
)
|
|
|
|
// LockExclusive acquires the SQLite-compatible exclusive lock on f.
|
|
// Returns an error if the lock cannot be acquired (already held by
|
|
// another process, fd is invalid, etc).
|
|
func LockExclusive(f *os.File) error {
|
|
fd := int(f.Fd())
|
|
if err := setLock(fd, unix.F_WRLCK, sqlitePendingByte, 1); err != nil {
|
|
return err
|
|
}
|
|
if err := setLock(fd, unix.F_WRLCK, sqliteSharedFirst, sqliteSharedSize); err != nil {
|
|
_ = setLock(fd, unix.F_UNLCK, sqlitePendingByte, 1)
|
|
return err
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// Unlock releases the exclusive lock acquired by LockExclusive.
|
|
func Unlock(f *os.File) error {
|
|
fd := int(f.Fd())
|
|
err1 := setLock(fd, unix.F_UNLCK, sqliteSharedFirst, sqliteSharedSize)
|
|
err2 := setLock(fd, unix.F_UNLCK, sqlitePendingByte, 1)
|
|
if err1 != nil {
|
|
return err1
|
|
}
|
|
return err2
|
|
}
|
|
|
|
func setLock(fd int, lockType int16, start, length int64) error {
|
|
flock := unix.Flock_t{
|
|
Type: lockType,
|
|
Whence: 0,
|
|
Start: start,
|
|
Len: length,
|
|
}
|
|
return unix.FcntlFlock(uintptr(fd), unix.F_SETLKW, &flock)
|
|
}
|