mirror of
https://github.com/hanzoai/compute.git
synced 2026-10-30 14:50:41 +00:00
zeekayandGitHub
846ef09e7a
Closes Red's CRITICAL double-charge finding. Root cause: under the Base backend the
_global coord DB is pod-local SQLite, so an insert-once lease wins independently on
each replica → every machine debited N×/hour once billing is on.
Fix = two composed guarantees:
1. Single-writer gate (object/coordinator.go): only the HRW-elected owner of the
coord key runs the metering sweep and performs lease claims
(ClaimMeterHour/ClaimBillingUnit/AdvanceCostCursor). Same deterministic HRW
(Rendezvous) primitive the HA-SQLite substrate uses — no coordinator, no
Postgres, no Redis. Fail-closed: unknown ownership → not owner → skip (never
risk a duplicate debit). At replicas:1 the sole pod always elects itself; at
replicas:N only the owner bills while all N serve reads.
2. Durable lease history across handoff (store_replica.go): the coord DB hydrates
before a claim and ships after, so a post-flip new owner can't re-claim an hour.
ClaimMeterHour/ClaimBillingUnit re-check ownership, so the ticker gate is a
clean-skip optimization, not the sole guard.
Tests (exactly-once proofs, all green): concurrent ClaimMeterHour/ClaimBillingUnit
→ exactly one wins; single-replica-always-owner; exactly-one-owner-across-replicas;
leadership-handoff-no-reclaim (meter + billing unit); non-owner-does-not-claim.
Billing stays OFF when COMMERCE_SERVICE_TOKEN unset.
74 lines
3.5 KiB
Go
74 lines
3.5 KiB
Go
// Copyright 2025 Hanzo Industries Inc. All Rights Reserved.
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
package object
|
|
|
|
import (
|
|
"time"
|
|
)
|
|
|
|
// MeterLease is a single-flight lease for the hourly compute-metering sweep.
|
|
// Its whole purpose is a MONEY-SAFETY invariant: the visor Deployment runs
|
|
// replicas: 2 with no leader election, so without a lease BOTH replicas would run
|
|
// service.MeterRunningMachines every hour and debit every running machine twice
|
|
// (the per-machine hour-bucketed RequestID is only a dedup HINT — commerce's
|
|
// RecordUsage does NOT dedup the withdraw transaction on requestId, so the client
|
|
// key alone does not stop a duplicate debit). The lease makes exactly one replica
|
|
// perform the sweep per wall-clock hour, cluster-wide.
|
|
//
|
|
// The mechanism is the same insert-once-wins primitive the rest of object uses:
|
|
// Hour is the PK, so only the FIRST Insert for a given hour succeeds; a concurrent
|
|
// replica's Insert fails the unique-key constraint and that replica skips the
|
|
// hour. It survives a mid-hour restart too — a restarted replica finds the hour
|
|
// already claimed and does not re-sweep (so a rollout cannot re-bill the hour).
|
|
type MeterLease struct {
|
|
Hour string `xorm:"varchar(12) notnull pk" json:"hour"` // UTC "YYYYMMDDHH" bucket — the billable unit.
|
|
CreatedTime string `xorm:"varchar(100)" json:"createdTime"`
|
|
}
|
|
|
|
// meterLeaseTTL bounds how long claimed-hour rows are retained. One day is far
|
|
// longer than any sweep, so pruning never races an in-flight claim, and the table
|
|
// stays tiny.
|
|
const meterLeaseTTL = 24 * time.Hour
|
|
|
|
// ClaimMeterHour attempts to claim the hourly metering sweep for the wall-clock
|
|
// hour containing now, cluster-wide. It returns true to EXACTLY ONE caller per
|
|
// hour (the elected owner whose Insert wins the Hour primary key); every other
|
|
// replica — a non-owner, or any re-entry within the same hour after a restart —
|
|
// gets false and must skip the sweep.
|
|
//
|
|
// Exactly-once is the shared claimLease primitive: the single-writer gate (only the
|
|
// elected `_global` owner claims), a hydrate of the prior owner's leases before the
|
|
// insert, and a synchronous ship of the winning claim before the caller debits. Any
|
|
// failure (not owner, hydrate/ship failed, PK lost, DB error) yields false: fail SAFE
|
|
// for a paid product means NOT sweeping (a missed hour is reconciled; a duplicate debit
|
|
// is not). Best-effort prune of stale rows runs only for the winner, so it costs
|
|
// nothing on the hot skip path.
|
|
func ClaimMeterHour(now time.Time) bool {
|
|
if !claimLease(&MeterLease{
|
|
Hour: now.UTC().Format("2006010215"),
|
|
CreatedTime: now.UTC().Format(time.RFC3339),
|
|
}) {
|
|
return false
|
|
}
|
|
pruneMeterLeases(now.Add(-meterLeaseTTL))
|
|
return true
|
|
}
|
|
|
|
// pruneMeterLeases deletes lease rows older than cutoff. Best-effort: a failure
|
|
// only leaves stale rows, never causes a double debit.
|
|
func pruneMeterLeases(cutoff time.Time) {
|
|
_, _ = Shared().Where("hour < ?", cutoff.UTC().Format("2006010215")).Delete(&MeterLease{})
|
|
}
|