mirror of
https://github.com/hanzoai/ingress.git
synced 2026-10-20 15:07:40 +00:00
acme: one writer, every replica a reader, every write fenced
With replicas > 1 the file store is one file PER NODE and nothing notices. Measured on this fleet: 71 certificates on one replica, 63 on the other, EIGHT hosts present on exactly one. Those hosts serve the right certificate or a wrong one depending on which replica the balancer picked, and under sniStrict a missing certificate is a failed handshake, not a downgrade -- so roughly half of connections to a one-replica host did not complete. Both replicas also ordered every name, so a rescheduled pod with an empty hostPath re-orders ~70 names into Let's Encrypt's 50-per-week ceiling. SharedStore puts the state in one Kubernetes object every replica reads. WHO writes comes from hanzoai/ha at the single key "acme" -- with one key an HRW owner IS a single leader, the one shape where that word is accurate. The ROUND comes from a Kubernetes Lease, and leaseTransitions is not an approximation: the API server increments it on every handoff, which is verbatim ha's requirement that a round be monotone across all callers and strictly increase when the role moves. No consensus of our own, no second coordination system. Leadership alone is not safety, which is the part worth being explicit about. A lease can expire while its holder still runs -- a paused process, a partitioned node -- so every write carries its round AND its holder, and the object refuses a write that is behind the recorded round, or level with it under a different identity. That second case is not redundant: the Lease and the Secret are two objects and Kubernetes has no transaction spanning them, so two replicas can briefly believe they hold the same round. The object decides, in one resourceVersion-checked compare-and-swap, and a deposed writer becomes harmless rather than unlikely. A reader also orders NOTHING. getUncheckedDomains gates first, before it touches any state, because everything past it is the expensive half: a DNS-01 round trip, a rate-limit slot, and a challenge TXT record two racing replicas delete out from under each other. LocalStore deliberately does NOT implement the gate -- a single replica has no election to consult, and requiring one would invent a failure mode where none existed. Rolling upgrade falls out. State is in the object, so a new pod starts knowing every certificate and orders none; a pod drained mid-lease loses the round, not the data. Two bugs the fake API server caught that reasoning did not: existence was inferred from an empty resourceVersion, which conflates absent with present-but-unversioned and makes every write take Create and fail AlreadyExists forever; and an empty resourceVersion was treated as "unchanged", which pins a reader to its first read so it never sees a certificate the writer obtained -- silently, since nothing errors. OPT-IN. Default is unchanged (the file), because turning this on relocates the estate's TLS state and the shared object starts empty, so the writer re-orders what the file held -- a deliberate act against a rate limit, not something to discover on restart. A configured shared store that cannot be built is FATAL: falling back would reinstate per-node state exactly when the cluster is unreachable, quietly, which is how the current split went unnoticed. Tests drive two stores over ONE fake API server, because the bug is invisible to a single-replica test: one writer only, reader serves the writer's certs, deposed writer refused, round advances on handoff, fresh replica inherits state AND account, conflict retried without loss, reader orders nothing, LocalStore not gated. cmd/ingress builds and vets clean (webui/embed.go's missing frontend assets fail identically on a pristine tree). Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
Hanzo Dev
+40
-7
@@ -18,9 +18,6 @@ import (
|
||||
"github.com/coreos/go-systemd/v22/daemon"
|
||||
"github.com/go-acme/lego/v4/challenge"
|
||||
gokitmetrics "github.com/go-kit/kit/metrics"
|
||||
"github.com/rs/zerolog/log"
|
||||
"github.com/sirupsen/logrus"
|
||||
"github.com/spiffe/go-spiffe/v2/workloadapi"
|
||||
"github.com/hanzoai/ingress-parser/cli"
|
||||
"github.com/hanzoai/ingress/cmd"
|
||||
"github.com/hanzoai/ingress/cmd/healthcheck"
|
||||
@@ -48,6 +45,9 @@ import (
|
||||
"github.com/hanzoai/ingress/pkg/tcp"
|
||||
ingresstls "github.com/hanzoai/ingress/pkg/tls"
|
||||
"github.com/hanzoai/ingress/pkg/version"
|
||||
"github.com/rs/zerolog/log"
|
||||
"github.com/sirupsen/logrus"
|
||||
"github.com/spiffe/go-spiffe/v2/workloadapi"
|
||||
)
|
||||
|
||||
func main() {
|
||||
@@ -442,7 +442,9 @@ func switchRouter(routerFactory *server.RouterFactory, serverEntryPointsTCP serv
|
||||
|
||||
// initACMEProvider creates and registers acme.Provider instances corresponding to the configured ACME certificate resolvers.
|
||||
func initACMEProvider(c *static.Configuration, providerAggregator *aggregator.ProviderAggregator, tlsManager *ingresstls.Manager, httpChallengeProvider, tlsChallengeProvider challenge.Provider, routinesPool *safe.Pool) []*acme.Provider {
|
||||
localStores := map[string]*acme.LocalStore{}
|
||||
// acme.Store, not *acme.LocalStore: where the state lives is now a decision
|
||||
// (see acmeStore), and a map typed to one implementation cannot hold the other.
|
||||
acmeStores := map[string]acme.Store{}
|
||||
|
||||
var resolvers []*acme.Provider
|
||||
for name, resolver := range c.CertificatesResolvers {
|
||||
@@ -450,13 +452,13 @@ func initACMEProvider(c *static.Configuration, providerAggregator *aggregator.Pr
|
||||
continue
|
||||
}
|
||||
|
||||
if localStores[resolver.ACME.Storage] == nil {
|
||||
localStores[resolver.ACME.Storage] = acme.NewLocalStore(resolver.ACME.Storage, routinesPool)
|
||||
if acmeStores[resolver.ACME.Storage] == nil {
|
||||
acmeStores[resolver.ACME.Storage] = acmeStore(resolver.ACME.Storage, routinesPool)
|
||||
}
|
||||
|
||||
p := &acme.Provider{
|
||||
Configuration: resolver.ACME,
|
||||
Store: localStores[resolver.ACME.Storage],
|
||||
Store: acmeStores[resolver.ACME.Storage],
|
||||
ResolverName: name,
|
||||
HTTPChallengeProvider: httpChallengeProvider,
|
||||
TLSChallengeProvider: tlsChallengeProvider,
|
||||
@@ -638,3 +640,34 @@ func collect(staticConfiguration *static.Configuration) {
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// acmeStore picks where ACME state lives.
|
||||
//
|
||||
// The default is unchanged: a local file, which is correct and simplest for ONE
|
||||
// replica. Set ACME_SHARED_STORE_NAMESPACE (the operator normally maps it from
|
||||
// metadata.namespace) and state moves to a Kubernetes object every replica reads,
|
||||
// with one elected writer — which is what more than one replica requires.
|
||||
//
|
||||
// It is opt-in rather than automatic because turning it on relocates the estate's
|
||||
// TLS state, and the migration is one-way in practice: the shared object starts
|
||||
// empty, so the writer re-orders whatever the file held. That is a deliberate act
|
||||
// against a rate limit, not something a deployment should discover on restart.
|
||||
//
|
||||
// A configured shared store that cannot be built is FATAL, never a silent
|
||||
// downgrade to the file. Falling back would reinstate per-node state exactly when
|
||||
// the cluster is unreachable — the moment two replicas are most likely to diverge
|
||||
// — and it would do so quietly, which is how the current split went unnoticed for
|
||||
// as long as it did.
|
||||
func acmeStore(storage string, pool *safe.Pool) acme.Store {
|
||||
ns := os.Getenv("ACME_SHARED_STORE_NAMESPACE")
|
||||
if ns == "" {
|
||||
return acme.NewLocalStore(storage, pool)
|
||||
}
|
||||
shared, err := acme.NewSharedStore(acme.SharedStoreConfig{Namespace: ns})
|
||||
if err != nil {
|
||||
log.Fatal().Err(err).Msg("ACME shared store is configured but could not be built; refusing to fall back to per-node state")
|
||||
}
|
||||
shared.Start(context.Background())
|
||||
log.Info().Str("namespace", ns).Msg("ACME state is shared: one elected writer, every replica reads")
|
||||
return shared
|
||||
}
|
||||
|
||||
@@ -1,22 +1,23 @@
|
||||
module github.com/hanzoai/ingress
|
||||
|
||||
go 1.26.5
|
||||
go 1.26.4
|
||||
|
||||
require github.com/hanzoai/cloud v1.801.293
|
||||
require github.com/hanzoai/cloud v0.1.1
|
||||
|
||||
require (
|
||||
github.com/hanzoai/ha v0.2.0
|
||||
github.com/BurntSushi/toml v1.6.0
|
||||
github.com/Masterminds/sprig/v3 v3.2.3
|
||||
github.com/abbot/go-http-auth v0.0.0-00010101000000-000000000000 // No tag on the repo.
|
||||
github.com/andybalholm/brotli v1.2.1
|
||||
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.5 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.14 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.14 // indirect
|
||||
github.com/aws/smithy-go v1.24.2
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.2 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.8 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.8 // indirect
|
||||
github.com/aws/smithy-go v1.24.1
|
||||
github.com/cenkalti/backoff/v4 v4.3.0
|
||||
github.com/containous/alice v0.0.0-20181107144136-d83ebdd94cbd // No tag on the repo.
|
||||
github.com/coreos/go-systemd/v22 v22.7.0
|
||||
github.com/coreos/go-systemd/v22 v22.6.0
|
||||
github.com/docker/docker v28.3.3+incompatible
|
||||
github.com/docker/go-connections v0.5.0 // indirect
|
||||
github.com/fatih/structs v1.1.0
|
||||
@@ -34,11 +35,11 @@ require (
|
||||
github.com/hanzokv/go/v9 v9.22.0
|
||||
github.com/hashicorp/go-multierror v1.1.1
|
||||
github.com/hashicorp/go-retryablehttp v0.7.8
|
||||
github.com/hashicorp/go-version v1.9.0 // indirect
|
||||
github.com/hashicorp/go-version v1.8.0 // indirect
|
||||
github.com/http-wasm/http-wasm-host-go v0.7.0
|
||||
github.com/huandu/xstrings v1.5.0
|
||||
github.com/influxdata/influxdb-client-go/v2 v2.14.0
|
||||
github.com/influxdata/influxdb1-client v0.0.0-20220302092344-a9ab5670611c // No tag on the repo.
|
||||
github.com/influxdata/influxdb-client-go/v2 v2.7.0
|
||||
github.com/influxdata/influxdb1-client v0.0.0-20200827194710-b269163b24ab // No tag on the repo.
|
||||
github.com/klauspost/compress v1.18.6
|
||||
github.com/mailgun/ttlmap v0.0.0-20170619185759-c1c17f74874f // No tag on the repo.
|
||||
github.com/miekg/dns v1.1.72
|
||||
@@ -59,15 +60,15 @@ require (
|
||||
github.com/stvp/go-udp-testing v0.0.0-20201019212854-469649b16807 // No tag on the repo.
|
||||
github.com/testcontainers/testcontainers-go v0.32.0
|
||||
github.com/testcontainers/testcontainers-go/modules/k3s v0.32.0
|
||||
github.com/tetratelabs/wazero v1.12.0
|
||||
github.com/tetratelabs/wazero v1.8.0
|
||||
github.com/tidwall/gjson v1.18.0
|
||||
github.com/unrolled/render v1.0.2
|
||||
github.com/unrolled/secure v1.15.0
|
||||
github.com/valyala/fasthttp v1.72.0
|
||||
github.com/vulcand/oxy/v2 v2.2.0
|
||||
github.com/valyala/fasthttp v1.70.0
|
||||
github.com/vulcand/oxy/v2 v2.0.3
|
||||
github.com/vulcand/predicate v1.3.0
|
||||
github.com/yuin/gopher-lua v1.1.1
|
||||
go.opentelemetry.io/collector/pdata v1.54.0
|
||||
go.opentelemetry.io/collector/pdata v1.51.0
|
||||
go.opentelemetry.io/contrib/bridges/otellogrus v0.13.0
|
||||
go.opentelemetry.io/contrib/propagators/autoprop v0.63.0
|
||||
go.opentelemetry.io/otel v1.44.0
|
||||
@@ -77,28 +78,28 @@ require (
|
||||
go.opentelemetry.io/otel/sdk/log v0.20.0
|
||||
go.opentelemetry.io/otel/sdk/metric v1.44.0
|
||||
go.opentelemetry.io/otel/trace v1.44.0
|
||||
golang.org/x/crypto v0.54.0
|
||||
golang.org/x/mod v0.37.0
|
||||
golang.org/x/net v0.57.0
|
||||
golang.org/x/sync v0.22.0
|
||||
golang.org/x/sys v0.47.0
|
||||
golang.org/x/text v0.40.0
|
||||
golang.org/x/crypto v0.52.0
|
||||
golang.org/x/mod v0.36.0
|
||||
golang.org/x/net v0.55.0
|
||||
golang.org/x/sync v0.20.0
|
||||
golang.org/x/sys v0.45.0
|
||||
golang.org/x/text v0.37.0
|
||||
golang.org/x/time v0.15.0
|
||||
golang.org/x/tools v0.47.0
|
||||
google.golang.org/grpc v1.81.1
|
||||
golang.org/x/tools v0.45.0
|
||||
google.golang.org/grpc v1.80.0
|
||||
gopkg.in/natefinch/lumberjack.v2 v2.2.1
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
k8s.io/api v0.35.3
|
||||
k8s.io/apiextensions-apiserver v0.35.0
|
||||
k8s.io/apimachinery v0.35.3
|
||||
k8s.io/client-go v0.35.3
|
||||
k8s.io/utils v0.0.0-20260319190234-28399d86e0b5 // No tag on the repo.
|
||||
k8s.io/api v0.35.0
|
||||
k8s.io/apiextensions-apiserver v0.34.3
|
||||
k8s.io/apimachinery v0.35.0
|
||||
k8s.io/client-go v0.35.0
|
||||
k8s.io/utils v0.0.0-20251002143259-bc988d571ff4 // No tag on the repo.
|
||||
knative.dev/networking v0.0.0-20241022012959-60e29ff520dc
|
||||
knative.dev/pkg v0.0.0-20241021183759-9b9d535af5ad
|
||||
mvdan.cc/xurls/v2 v2.5.0
|
||||
sigs.k8s.io/controller-runtime v0.23.3
|
||||
sigs.k8s.io/controller-runtime v0.22.1
|
||||
sigs.k8s.io/gateway-api v1.4.0
|
||||
sigs.k8s.io/structured-merge-diff/v6 v6.3.2
|
||||
sigs.k8s.io/structured-merge-diff/v6 v6.3.1
|
||||
sigs.k8s.io/yaml v1.6.0
|
||||
)
|
||||
|
||||
@@ -109,93 +110,38 @@ require (
|
||||
github.com/luxfi/trace v1.4.0
|
||||
github.com/luxfi/zap v1.2.6
|
||||
github.com/prometheus/client_golang v1.23.2
|
||||
github.com/zap-proto/http v0.3.1
|
||||
github.com/zap-proto/http v0.1.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/apapsch/go-jsonmerge/v2 v2.0.0 // indirect
|
||||
github.com/beorn7/perks v1.0.1 // indirect
|
||||
github.com/cenkalti/backoff v2.2.1+incompatible // indirect
|
||||
github.com/cloudflare/circl v1.6.3 // indirect
|
||||
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.1 // indirect
|
||||
github.com/dlclark/regexp2/v2 v2.2.1 // indirect
|
||||
github.com/dop251/goja v0.0.0-20260607120635-348e6bea910d // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/fasthttp/websocket v1.5.12 // indirect
|
||||
github.com/go-faster/city v1.0.1 // indirect
|
||||
github.com/go-faster/errors v0.7.1 // indirect
|
||||
github.com/evanw/esbuild v0.28.1 // indirect
|
||||
github.com/go-ini/ini v1.67.0 // indirect
|
||||
github.com/go-sourcemap/sourcemap v2.1.3+incompatible // indirect
|
||||
github.com/google/pprof v0.0.0-20250403155104-27863c87afa6 // indirect
|
||||
github.com/gorilla/rpc v1.2.1 // indirect
|
||||
github.com/grandcat/zeroconf v1.0.0 // indirect
|
||||
github.com/hanzo-ds/go v1.0.1 // indirect
|
||||
github.com/hanzo-ds/native v0.72.0 // indirect
|
||||
github.com/hanzoai/account v0.2.0 // indirect
|
||||
github.com/hanzoai/commerce v1.49.29 // indirect
|
||||
github.com/hanzoai/csqlite v0.1.0 // indirect
|
||||
github.com/hanzoai/dbx v1.17.2 // indirect
|
||||
github.com/hanzoai/decimal v0.1.1 // indirect
|
||||
github.com/hanzoai/go-openai v1.41.0 // indirect
|
||||
github.com/hanzoai/ha v0.1.1 // indirect
|
||||
github.com/hanzoai/iam v1.33.26 // indirect
|
||||
github.com/hanzoai/money v0.2.1 // indirect
|
||||
github.com/hanzoai/orm v0.6.18 // indirect
|
||||
github.com/hanzoai/s3-go v1.0.0 // indirect
|
||||
github.com/hanzoai/sqlcipher v0.1.1 // indirect
|
||||
github.com/hanzoai/sqlite v0.4.0 // indirect
|
||||
github.com/hanzoai/tasks v1.51.4 // indirect
|
||||
github.com/hanzoai/vfs v0.6.6 // indirect
|
||||
github.com/holiman/uint256 v1.3.2 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
|
||||
github.com/klauspost/crc32 v1.3.0 // indirect
|
||||
github.com/luxfi/accel v1.2.4 // indirect
|
||||
github.com/luxfi/bft v0.1.5 // indirect
|
||||
github.com/luxfi/cache v1.3.1 // indirect
|
||||
github.com/luxfi/compress v0.1.1 // indirect
|
||||
github.com/luxfi/concurrent v0.1.1 // indirect
|
||||
github.com/luxfi/consensus v1.36.11 // indirect
|
||||
github.com/luxfi/constants v1.6.2 // indirect
|
||||
github.com/luxfi/container v0.2.1 // indirect
|
||||
github.com/luxfi/crypto v1.20.2 // indirect
|
||||
github.com/luxfi/database v1.21.1 // indirect
|
||||
github.com/luxfi/geth v1.20.1 // indirect
|
||||
github.com/luxfi/ids v1.3.2 // indirect
|
||||
github.com/luxfi/math v1.5.1 // indirect
|
||||
github.com/luxfi/math/big v0.1.0 // indirect
|
||||
github.com/luxfi/mdns v0.1.1 // indirect
|
||||
github.com/luxfi/mock v0.1.1 // indirect
|
||||
github.com/luxfi/p2p v1.22.1 // indirect
|
||||
github.com/luxfi/pq v1.1.0 // indirect
|
||||
github.com/luxfi/sampler v1.1.0 // indirect
|
||||
github.com/luxfi/validators v1.3.1 // indirect
|
||||
github.com/luxfi/version v1.0.1 // indirect
|
||||
github.com/luxfi/warp v1.24.1 // indirect
|
||||
github.com/minio/crc64nvme v1.1.1 // indirect
|
||||
github.com/minio/md5-simd v1.1.2 // indirect
|
||||
github.com/mr-tron/base58 v1.3.0 // indirect
|
||||
github.com/ncruces/go-strftime v1.0.0 // indirect
|
||||
github.com/oapi-codegen/runtime v1.3.1 // indirect
|
||||
github.com/paulmach/orb v0.13.0 // indirect
|
||||
github.com/pierrec/lz4/v4 v4.1.27 // indirect
|
||||
github.com/prometheus/common v0.67.5 // indirect
|
||||
github.com/prometheus/procfs v0.20.1 // indirect
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
|
||||
github.com/robfig/cron/v3 v3.0.1 // indirect
|
||||
github.com/rs/xid v1.6.0 // indirect
|
||||
github.com/savsgio/gotils v0.0.0-20240704082632-aef3928b8a38 // indirect
|
||||
github.com/segmentio/asm v1.2.1 // indirect
|
||||
github.com/shoenig/test v1.7.0 // indirect
|
||||
github.com/supranational/blst v0.3.16 // indirect
|
||||
github.com/zap-proto/fiber/v3 v3.2.1 // indirect
|
||||
github.com/zap-proto/go v1.3.0 // indirect
|
||||
github.com/zap-proto/md v0.1.0 // indirect
|
||||
go.uber.org/mock v0.6.0 // indirect
|
||||
gonum.org/v1/gonum v0.17.0 // indirect
|
||||
modernc.org/libc v1.72.3 // indirect
|
||||
modernc.org/mathutil v1.7.1 // indirect
|
||||
modernc.org/memory v1.11.0 // indirect
|
||||
modernc.org/sqlite v1.51.0 // indirect
|
||||
)
|
||||
|
||||
require (
|
||||
cloud.google.com/go/auth v0.20.0 // indirect
|
||||
cloud.google.com/go/auth v0.18.1 // indirect
|
||||
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
|
||||
cloud.google.com/go/compute/metadata v0.9.0 // indirect
|
||||
dario.cat/mergo v1.0.2 // indirect
|
||||
@@ -203,7 +149,7 @@ require (
|
||||
github.com/Azure/azure-sdk-for-go v68.0.0+incompatible // indirect
|
||||
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0 // indirect
|
||||
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1 // indirect
|
||||
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0 // indirect
|
||||
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2 // indirect
|
||||
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/dns/armdns v1.2.0 // indirect
|
||||
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/privatedns/armprivatedns v1.3.0 // indirect
|
||||
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resourcegraph/armresourcegraph v0.9.0 // indirect
|
||||
@@ -217,8 +163,8 @@ require (
|
||||
github.com/Azure/go-autorest/autorest/to v0.4.1 // indirect
|
||||
github.com/Azure/go-autorest/logger v0.2.1 // indirect
|
||||
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
|
||||
github.com/AzureAD/microsoft-authentication-library-for-go v1.7.0 // indirect
|
||||
github.com/HdrHistogram/hdrhistogram-go v1.2.0 // indirect
|
||||
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 // indirect
|
||||
github.com/HdrHistogram/hdrhistogram-go v1.1.2 // indirect
|
||||
github.com/Masterminds/goutils v1.1.1 // indirect
|
||||
github.com/Masterminds/semver/v3 v3.5.0 // indirect
|
||||
github.com/Microsoft/go-winio v0.6.2 // indirect
|
||||
@@ -226,25 +172,25 @@ require (
|
||||
github.com/VividCortex/gohistogram v1.0.0 // indirect
|
||||
github.com/akamai/AkamaiOPEN-edgegrid-golang/v11 v11.1.0 // indirect
|
||||
github.com/alibabacloud-go/alibabacloud-gateway-spi v0.0.5 // indirect
|
||||
github.com/alibabacloud-go/darabonba-openapi/v2 v2.1.16 // indirect
|
||||
github.com/alibabacloud-go/darabonba-openapi/v2 v2.1.15 // indirect
|
||||
github.com/alibabacloud-go/debug v1.0.1 // indirect
|
||||
github.com/alibabacloud-go/tea v1.4.0 // indirect
|
||||
github.com/alibabacloud-go/tea-utils/v2 v2.0.9 // indirect
|
||||
github.com/alibabacloud-go/tea-utils/v2 v2.0.7 // indirect
|
||||
github.com/aliyun/credentials-go v1.4.7 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.21 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.21 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.6 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/lightsail v1.53.0 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/route53 v1.62.5 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.0.9 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.30.15 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.19 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.41.10 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.17 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.18 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.18 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.5 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.18 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/lightsail v1.50.11 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/route53 v1.62.1 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.0.5 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.30.9 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.14 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.41.6 // indirect
|
||||
github.com/aziontech/azionapi-go-sdk v0.144.0 // indirect
|
||||
github.com/baidubce/bce-sdk-go v0.9.264 // indirect
|
||||
github.com/baidubce/bce-sdk-go v0.9.260 // indirect
|
||||
github.com/benbjohnson/clock v1.3.5 // indirect
|
||||
github.com/blendle/zapdriver v1.3.1 // indirect
|
||||
github.com/boombuler/barcode v1.0.1 // indirect
|
||||
@@ -258,6 +204,7 @@ require (
|
||||
github.com/containerd/platforms v1.0.0-rc.1 // indirect
|
||||
github.com/cpuguy83/dockercfg v0.3.1 // indirect
|
||||
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
|
||||
github.com/deepmap/oapi-codegen v1.9.1 // indirect
|
||||
github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f // indirect
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
|
||||
github.com/dimchansky/utfbom v1.1.1 // indirect
|
||||
@@ -276,34 +223,35 @@ require (
|
||||
github.com/go-acme/jdcloud-sdk-go v1.64.0 // indirect
|
||||
github.com/go-acme/tencentclouddnspod v1.3.24 // indirect
|
||||
github.com/go-acme/tencentedgdeone v1.3.38 // indirect
|
||||
github.com/go-errors/errors v1.5.1 // indirect
|
||||
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
|
||||
github.com/go-errors/errors v1.0.1 // indirect
|
||||
github.com/go-jose/go-jose/v4 v4.1.3 // indirect
|
||||
github.com/go-logfmt/logfmt v0.6.1 // indirect
|
||||
github.com/go-logr/logr v1.4.3 // indirect
|
||||
github.com/go-logr/stdr v1.2.2 // indirect
|
||||
github.com/go-logr/zapr v1.3.0 // indirect
|
||||
github.com/go-ole/go-ole v1.3.0 // indirect
|
||||
github.com/go-openapi/jsonpointer v0.22.5 // indirect
|
||||
github.com/go-openapi/jsonreference v0.21.5 // indirect
|
||||
github.com/go-openapi/swag v0.25.5 // indirect
|
||||
github.com/go-openapi/swag/cmdutils v0.25.5 // indirect
|
||||
github.com/go-openapi/swag/conv v0.25.5 // indirect
|
||||
github.com/go-openapi/swag/fileutils v0.25.5 // indirect
|
||||
github.com/go-openapi/swag/jsonname v0.25.5 // indirect
|
||||
github.com/go-openapi/swag/jsonutils v0.25.5 // indirect
|
||||
github.com/go-openapi/swag/loading v0.25.5 // indirect
|
||||
github.com/go-openapi/swag/mangling v0.25.5 // indirect
|
||||
github.com/go-openapi/swag/netutils v0.25.5 // indirect
|
||||
github.com/go-openapi/swag/stringutils v0.25.5 // indirect
|
||||
github.com/go-openapi/swag/typeutils v0.25.5 // indirect
|
||||
github.com/go-openapi/swag/yamlutils v0.25.5 // indirect
|
||||
github.com/go-openapi/jsonpointer v0.22.4 // indirect
|
||||
github.com/go-openapi/jsonreference v0.21.4 // indirect
|
||||
github.com/go-openapi/swag v0.25.4 // indirect
|
||||
github.com/go-openapi/swag/cmdutils v0.25.4 // indirect
|
||||
github.com/go-openapi/swag/conv v0.25.4 // indirect
|
||||
github.com/go-openapi/swag/fileutils v0.25.4 // indirect
|
||||
github.com/go-openapi/swag/jsonname v0.25.4 // indirect
|
||||
github.com/go-openapi/swag/jsonutils v0.25.4 // indirect
|
||||
github.com/go-openapi/swag/loading v0.25.4 // indirect
|
||||
github.com/go-openapi/swag/mangling v0.25.4 // indirect
|
||||
github.com/go-openapi/swag/netutils v0.25.4 // indirect
|
||||
github.com/go-openapi/swag/stringutils v0.25.4 // indirect
|
||||
github.com/go-openapi/swag/typeutils v0.25.4 // indirect
|
||||
github.com/go-openapi/swag/yamlutils v0.25.4 // indirect
|
||||
github.com/go-ozzo/ozzo-validation/v4 v4.3.0 // indirect
|
||||
github.com/go-playground/locales v0.14.1 // indirect
|
||||
github.com/go-playground/universal-translator v0.18.1 // indirect
|
||||
github.com/go-playground/validator/v10 v10.30.2 // indirect
|
||||
github.com/go-resty/resty/v2 v2.17.2 // indirect
|
||||
github.com/go-resty/resty/v2 v2.17.1 // indirect
|
||||
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
|
||||
github.com/goccy/go-yaml v1.19.2 // indirect
|
||||
github.com/gofiber/fiber/v3 v3.2.0 // indirect
|
||||
github.com/gofiber/schema v1.7.1 // indirect
|
||||
github.com/gofiber/utils/v2 v2.0.4 // indirect
|
||||
github.com/gofrs/flock v0.13.0 // indirect
|
||||
@@ -312,23 +260,24 @@ require (
|
||||
github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
|
||||
github.com/golang-jwt/jwt/v5 v5.3.1 // indirect
|
||||
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
|
||||
github.com/google/gnostic-models v0.7.1 // indirect
|
||||
github.com/google/gnostic-models v0.7.0 // indirect
|
||||
github.com/google/go-cmp v0.7.0 // indirect
|
||||
github.com/google/go-querystring v1.2.0 // indirect
|
||||
github.com/google/s2a-go v0.1.9 // indirect
|
||||
github.com/google/uuid v1.6.1-0.20241114170450-2d3c2a9cc518 // indirect
|
||||
github.com/googleapis/enterprise-certificate-proxy v0.3.14 // indirect
|
||||
github.com/googleapis/gax-go/v2 v2.21.0 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/googleapis/enterprise-certificate-proxy v0.3.11 // indirect
|
||||
github.com/googleapis/gax-go/v2 v2.17.0 // indirect
|
||||
github.com/gophercloud/gophercloud v1.14.1 // indirect
|
||||
github.com/gophercloud/utils v0.0.0-20231010081019-80377eca5d56 // indirect
|
||||
github.com/gravitational/trace v1.5.0 // indirect
|
||||
github.com/hashicorp/errwrap v1.1.0 // indirect
|
||||
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
|
||||
github.com/hashicorp/go-uuid v1.0.3 // indirect
|
||||
github.com/hashicorp/hcl v1.0.1-vault-5 // indirect
|
||||
github.com/huaweicloud/huaweicloud-sdk-go-v3 v0.1.187 // indirect
|
||||
github.com/iij/doapi v0.0.0-20190504054126-0bbf12d6d7df // indirect
|
||||
github.com/imdario/mergo v0.3.16 // indirect
|
||||
github.com/influxdata/line-protocol v0.0.0-20210922203350-b1ad95c89adf // indirect
|
||||
github.com/influxdata/line-protocol v0.0.0-20200327222509-2487e7298839 // indirect
|
||||
github.com/infobloxopen/infoblox-go-client/v2 v2.10.0 // indirect
|
||||
github.com/json-iterator/go v1.1.13-0.20220915233716-71ac16282d12 // indirect
|
||||
github.com/k0kubun/go-ansi v0.0.0-20180517002512-3bf9e2903213 // indirect
|
||||
@@ -337,16 +286,16 @@ require (
|
||||
github.com/labbsr0x/bindman-dns-webhook v1.0.2 // indirect
|
||||
github.com/labbsr0x/goh v1.0.1 // indirect
|
||||
github.com/leodido/go-urn v1.4.0 // indirect
|
||||
github.com/linode/linodego v1.67.0 // indirect
|
||||
github.com/linode/linodego v1.65.0 // indirect
|
||||
github.com/liquidweb/liquidweb-cli v0.6.9 // indirect
|
||||
github.com/liquidweb/liquidweb-go v1.6.4 // indirect
|
||||
github.com/lufia/plan9stats v0.0.0-20260216142805-b3301c5f2a88 // indirect
|
||||
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3 // indirect
|
||||
github.com/magiconair/properties v1.8.7 // indirect
|
||||
github.com/mailgun/minheap v0.0.0-20170619185613-3dbe6c6bf55f // indirect
|
||||
github.com/mailgun/multibuf v0.1.2 // indirect
|
||||
github.com/mailgun/timetools v0.0.0-20141028012446-7e6055773c51 // indirect
|
||||
github.com/mattn/go-colorable v0.1.15 // indirect
|
||||
github.com/mattn/go-isatty v0.0.22 // indirect
|
||||
github.com/mattn/go-colorable v0.1.14 // indirect
|
||||
github.com/mattn/go-isatty v0.0.21 // indirect
|
||||
github.com/mimuret/golang-iij-dpf v0.9.1 // indirect
|
||||
github.com/mitchellh/go-homedir v1.1.0 // indirect
|
||||
github.com/mitchellh/reflectwalk v1.0.2 // indirect
|
||||
@@ -401,7 +350,8 @@ require (
|
||||
github.com/sacloud/go-http v0.1.9 // indirect
|
||||
github.com/sacloud/iaas-api-go v1.23.1 // indirect
|
||||
github.com/sacloud/packages-go v0.0.12 // indirect
|
||||
github.com/sagikazarmark/locafero v0.12.0 // indirect
|
||||
github.com/sagikazarmark/locafero v0.4.0 // indirect
|
||||
github.com/sagikazarmark/slog-shim v0.1.0 // indirect
|
||||
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.36 // indirect
|
||||
github.com/selectel/domains-go v1.1.0 // indirect
|
||||
github.com/selectel/go-selvpcclient/v4 v4.1.0 // indirect
|
||||
@@ -411,37 +361,38 @@ require (
|
||||
github.com/softlayer/softlayer-go v1.2.1 // indirect
|
||||
github.com/softlayer/xmlrpc v0.0.0-20200409220501-5f089df7cb7e // indirect
|
||||
github.com/sony/gobreaker v1.0.0 // indirect
|
||||
github.com/spf13/afero v1.15.0 // indirect
|
||||
github.com/sourcegraph/conc v0.3.0 // indirect
|
||||
github.com/spf13/afero v1.11.0 // indirect
|
||||
github.com/spf13/cast v1.10.0 // indirect
|
||||
github.com/spf13/pflag v1.0.10 // indirect
|
||||
github.com/spf13/viper v1.21.0 // indirect
|
||||
github.com/spf13/viper v1.18.2 // indirect
|
||||
github.com/stretchr/objx v0.5.3 // indirect
|
||||
github.com/subosito/gotenv v1.6.0 // indirect
|
||||
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common v1.3.77 // indirect
|
||||
github.com/tidwall/match v1.2.0 // indirect
|
||||
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common v1.3.48 // indirect
|
||||
github.com/tidwall/match v1.1.1 // indirect
|
||||
github.com/tidwall/pretty v1.2.1 // indirect
|
||||
github.com/tinylib/msgp v1.6.4 // indirect
|
||||
github.com/tjfoc/gmsm v1.4.1 // indirect
|
||||
github.com/tklauser/go-sysconf v0.4.0 // indirect
|
||||
github.com/tklauser/numcpus v0.12.0 // indirect
|
||||
github.com/tklauser/go-sysconf v0.3.16 // indirect
|
||||
github.com/tklauser/numcpus v0.11.0 // indirect
|
||||
github.com/transip/gotransip/v6 v6.26.1 // indirect
|
||||
github.com/ultradns/ultradns-go-sdk v1.8.1-20250722213956-faef419 // indirect
|
||||
github.com/valyala/bytebufferpool v1.0.0 // indirect
|
||||
github.com/vinyldns/go-vinyldns v0.9.17 // indirect
|
||||
github.com/volcengine/volc-sdk-golang v1.0.241 // indirect
|
||||
github.com/vultr/govultr/v3 v3.30.0 // indirect
|
||||
github.com/volcengine/volc-sdk-golang v1.0.237 // indirect
|
||||
github.com/vultr/govultr/v3 v3.27.0 // indirect
|
||||
github.com/x448/float16 v0.8.4 // indirect
|
||||
github.com/yandex-cloud/go-genproto v0.54.0 // indirect
|
||||
github.com/yandex-cloud/go-sdk/services/dns v0.0.36 // indirect
|
||||
github.com/yandex-cloud/go-sdk/v2 v2.56.0 // indirect
|
||||
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
|
||||
github.com/yusufpapurcu/wmi v1.2.4 // indirect
|
||||
github.com/zap-proto/zip v1.24.2
|
||||
go.mongodb.org/mongo-driver v1.17.9 // indirect
|
||||
github.com/zap-proto/zip v1.1.0
|
||||
go.mongodb.org/mongo-driver v1.17.6 // indirect
|
||||
go.opencensus.io v0.24.0 // indirect
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
|
||||
go.opentelemetry.io/collector/featuregate v1.54.0 // indirect
|
||||
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.68.0 // indirect
|
||||
go.opentelemetry.io/collector/featuregate v1.51.0 // indirect
|
||||
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.65.0 // indirect
|
||||
go.opentelemetry.io/contrib/propagators/aws v1.38.0 // indirect
|
||||
go.opentelemetry.io/contrib/propagators/b3 v1.39.0 // indirect
|
||||
go.opentelemetry.io/contrib/propagators/jaeger v1.38.0 // indirect
|
||||
@@ -452,21 +403,21 @@ require (
|
||||
go.uber.org/zap v1.27.1 // indirect
|
||||
go.yaml.in/yaml/v2 v2.4.4 // indirect
|
||||
go.yaml.in/yaml/v3 v3.0.4 // indirect
|
||||
golang.org/x/exp v0.0.0-20260529124908-c761662dc8c9 // indirect
|
||||
golang.org/x/exp v0.0.0-20260312153236-7ab1446f8b90 // indirect
|
||||
golang.org/x/oauth2 v0.36.0 // indirect
|
||||
golang.org/x/term v0.45.0 // indirect
|
||||
golang.org/x/term v0.43.0 // indirect
|
||||
gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
|
||||
google.golang.org/api v0.275.0 // indirect
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20260615183401-62b3387ff324 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260622175928-b703f567277d // indirect
|
||||
google.golang.org/protobuf v1.36.12-0.20260120151049-f2248ac996af // indirect
|
||||
google.golang.org/api v0.267.0 // indirect
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9 // indirect
|
||||
google.golang.org/protobuf v1.36.11 // indirect
|
||||
gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect
|
||||
gopkg.in/inf.v0 v0.9.1 // indirect
|
||||
gopkg.in/ini.v1 v1.67.1 // indirect
|
||||
gopkg.in/ns1/ns1-go.v2 v2.17.2 // indirect
|
||||
gopkg.in/yaml.v2 v2.4.0 // indirect
|
||||
k8s.io/klog/v2 v2.140.0 // indirect
|
||||
k8s.io/kube-openapi v0.0.0-20260330154417-16be699c7b31 // indirect
|
||||
k8s.io/klog/v2 v2.130.1 // indirect
|
||||
k8s.io/kube-openapi v0.0.0-20250910181357-589584f1c912 // indirect
|
||||
nhooyr.io/websocket v1.8.7 // indirect
|
||||
sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 // indirect
|
||||
sigs.k8s.io/randfill v1.0.0 // indirect
|
||||
|
||||
@@ -26,7 +26,6 @@ import (
|
||||
"github.com/go-acme/lego/v4/lego"
|
||||
"github.com/go-acme/lego/v4/providers/dns"
|
||||
"github.com/go-acme/lego/v4/registration"
|
||||
"github.com/rs/zerolog/log"
|
||||
ptypes "github.com/hanzoai/ingress-parser/types"
|
||||
"github.com/hanzoai/ingress/pkg/config/dynamic"
|
||||
httpmuxer "github.com/hanzoai/ingress/pkg/muxer/http"
|
||||
@@ -36,6 +35,7 @@ import (
|
||||
ingresstls "github.com/hanzoai/ingress/pkg/tls"
|
||||
"github.com/hanzoai/ingress/pkg/types"
|
||||
"github.com/hanzoai/ingress/pkg/version"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
const resolverSuffix = ".acme"
|
||||
@@ -968,6 +968,22 @@ func (p *Provider) renewCertificates(ctx context.Context, renewPeriod time.Durat
|
||||
func (p *Provider) getUncheckedDomains(ctx context.Context, domainsToCheck []string, tlsStore string) []string {
|
||||
log.Ctx(ctx).Debug().Msgf("Looking for provided certificate(s) to validate %q...", domainsToCheck)
|
||||
|
||||
// A replica that is not the ACME writer orders NOTHING. This is the single
|
||||
// point where that is decidable, because everything downstream of it is the
|
||||
// expensive half: a DNS-01 round trip, a rate-limit slot against Let's
|
||||
// Encrypt's 50-per-week registered-domain ceiling, and a challenge TXT record
|
||||
// that two replicas racing to write and clean up can delete out from under
|
||||
// each other. Declining here means a reader spends none of it and simply
|
||||
// serves what the writer obtained.
|
||||
//
|
||||
// A store that does not know about writers (LocalStore) is single-writer by
|
||||
// construction, so it never gates and every replica orders — which is the
|
||||
// correct behaviour for one replica and the bug for more than one.
|
||||
if gate, ok := p.Store.(writerGate); ok && !gate.IsWriter(ctx) {
|
||||
log.Ctx(ctx).Debug().Msg("Not the ACME writer for this deployment; leaving resolution to the writer")
|
||||
return nil
|
||||
}
|
||||
|
||||
var allDomains []string
|
||||
store := p.tlsManager.GetStore(tlsStore)
|
||||
if store != nil {
|
||||
|
||||
@@ -0,0 +1,545 @@
|
||||
// Copyright 2025 Hanzo AI Inc. All Rights Reserved.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
|
||||
package acme
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/hanzoai/ha"
|
||||
"github.com/rs/zerolog/log"
|
||||
coordv1 "k8s.io/api/coordination/v1"
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
kerrors "k8s.io/apimachinery/pkg/api/errors"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/client-go/kubernetes"
|
||||
"k8s.io/client-go/rest"
|
||||
)
|
||||
|
||||
// SharedStore is the ACME store for MORE THAN ONE REPLICA: one writer, the rest
|
||||
// readers, over a single object every replica can see.
|
||||
//
|
||||
// # The defect it exists to fix
|
||||
//
|
||||
// LocalStore keeps acme.json on a filesystem. With replicas > 1 on a hostPath
|
||||
// that is one FILE PER NODE, and nothing about the design notices: each replica
|
||||
// resolves independently, writes its own file, and the two drift apart. Measured
|
||||
// on this fleet — 71 certificates on one replica, 63 on the other, EIGHT hosts
|
||||
// present on exactly one. Those hosts serve the right certificate or a wrong one
|
||||
// depending on which replica the load balancer picked, and with sniStrict a
|
||||
// missing certificate is not a downgrade but a failed handshake. So roughly half
|
||||
// of connections to a host only one replica holds simply do not complete.
|
||||
//
|
||||
// Duplicating the work is the second half of it. Both replicas order every name,
|
||||
// so a rescheduled pod with an empty hostPath re-orders the whole estate — ~70
|
||||
// certificates against Let's Encrypt's 50-per-week registered-domain ceiling,
|
||||
// which is how a routine restart turns into a week without new certificates.
|
||||
//
|
||||
// # One writer, and the rest read
|
||||
//
|
||||
// WHO writes is decided by github.com/hanzoai/ha, the fleet's one election, at
|
||||
// the single key "acme": with one key an HRW owner IS a single leader, which is
|
||||
// the one shape where that word is accurate. A replica that does not own the key
|
||||
// never orders and never writes — it READS, so it serves every certificate the
|
||||
// owner obtained without duplicating a single order.
|
||||
//
|
||||
// The round comes from a Kubernetes Lease, and it is not an approximation of one:
|
||||
// Lease.spec.leaseTransitions is a counter the API server increments whenever the
|
||||
// holder changes, which is verbatim ha's requirement — "monotone non-decreasing
|
||||
// per key across ALL callers, and STRICTLY increases whenever the writer role
|
||||
// changes hands". The API server is the linearizable source ha's doc says the
|
||||
// seam is shaped to accept; no consensus of our own, no second coordination
|
||||
// system, nothing to operate.
|
||||
//
|
||||
// # Why a deposed writer is harmless rather than unlikely
|
||||
//
|
||||
// A lease can expire while its holder is still running — a paused process, a
|
||||
// partitioned node — so leadership alone is not safety. Every write carries the
|
||||
// round it was made under, and a write whose round is BELOW the round already
|
||||
// recorded is refused. That is the storage half: the API server's own
|
||||
// resourceVersion precondition makes the read-modify-write atomic, so two
|
||||
// replicas cannot interleave, and the round makes the loser's data stale rather
|
||||
// than merely late. A writer that has lost the lease and does not know it yet
|
||||
// cannot damage the winner's state.
|
||||
//
|
||||
// # Rolling upgrade, with no re-issuance and no gap
|
||||
//
|
||||
// Because state lives in the object and not on a node, a new pod starts with
|
||||
// every certificate already known and orders nothing. It serves immediately, and
|
||||
// if it later takes the lease it continues from where its predecessor stopped.
|
||||
// The old pod, drained mid-lease, loses the round rather than the data. That is
|
||||
// the whole of zero-downtime here: no re-issuance storm on reschedule, no window
|
||||
// where a replica is serving from an empty store.
|
||||
//
|
||||
// Readers poll rather than watch. A poll cannot miss an event it was not
|
||||
// connected for, needs no reconnect logic, and is bounded by the interval; a
|
||||
// watch is faster and is the thing that silently stops delivering. The cost of
|
||||
// being one interval late is that a brand-new host is served by the owner a few
|
||||
// seconds before its readers — which is what the interval is for.
|
||||
type SharedStore struct {
|
||||
client kubernetes.Interface
|
||||
namespace string
|
||||
name string // Secret holding the ACME state
|
||||
leaseName string // Lease naming the writer
|
||||
self string // this replica's stable identity (pod name)
|
||||
|
||||
leaseDuration time.Duration
|
||||
pollInterval time.Duration
|
||||
|
||||
mu sync.RWMutex
|
||||
cache map[string]*StoredData // resolver -> state, last read
|
||||
rv string // resourceVersion the cache was read at ("" when the server sets none)
|
||||
loaded bool // the cache has been filled at least once
|
||||
leases ha.Leases // the round source
|
||||
}
|
||||
|
||||
// storeKey is the Secret key holding the serialized per-resolver state. One key,
|
||||
// because the state is one document: a partial write must not be able to leave an
|
||||
// account without its certificates.
|
||||
const storeKey = "acme.json"
|
||||
|
||||
// roundKey records the round the current contents were written under, and
|
||||
// holderKey the identity that wrote them. Both live in the SAME object as the
|
||||
// data, so a reader cannot see one without the others and a writer cannot update
|
||||
// the data without restating who it was and under what round.
|
||||
//
|
||||
// The holder is not redundant with the round. A round fences a writer that is
|
||||
// BEHIND; it says nothing about two writers who believe they hold the same round,
|
||||
// which is reachable because the Lease and this Secret are two objects and
|
||||
// Kubernetes has no transaction spanning them. Refusing a same-round write from a
|
||||
// different identity closes that: the object itself decides, and it decides with
|
||||
// one atomic compare-and-swap rather than a handshake across two.
|
||||
const roundKey = "acme.round"
|
||||
const holderKey = "acme.holder"
|
||||
|
||||
// ErrNotWriter is returned by a write on a replica that does not own the key. It
|
||||
// is not an error condition — it is the correct outcome for a reader, and the
|
||||
// caller logs it at debug and carries on.
|
||||
var ErrNotWriter = errors.New("acme: this replica is not the ACME writer")
|
||||
|
||||
// ErrStaleRound is returned when a write's round is below the round already
|
||||
// recorded. The writer has been deposed and has not noticed yet; refusing is what
|
||||
// makes that harmless.
|
||||
var ErrStaleRound = errors.New("acme: write refused, round superseded")
|
||||
|
||||
// SharedStoreConfig is the operator-facing shape. Zero values are filled with
|
||||
// defaults that are safe for two replicas, so the only required field is what
|
||||
// cannot be guessed: which namespace to write in.
|
||||
type SharedStoreConfig struct {
|
||||
Namespace string
|
||||
SecretName string
|
||||
LeaseName string
|
||||
Self string
|
||||
LeaseDuration time.Duration
|
||||
PollInterval time.Duration
|
||||
}
|
||||
|
||||
// NewSharedStore builds the store from in-cluster credentials.
|
||||
//
|
||||
// It fails rather than degrading to a local file. A store that silently falls
|
||||
// back to per-node state is the defect this type exists to remove, and it would
|
||||
// reappear exactly when the cluster is unreachable — the moment when two replicas
|
||||
// are most likely to disagree.
|
||||
func NewSharedStore(cfg SharedStoreConfig) (*SharedStore, error) {
|
||||
restCfg, err := rest.InClusterConfig()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("acme shared store needs in-cluster credentials: %w", err)
|
||||
}
|
||||
client, err := kubernetes.NewForConfig(restCfg)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("acme shared store: build client: %w", err)
|
||||
}
|
||||
return newSharedStore(client, cfg)
|
||||
}
|
||||
|
||||
// newSharedStore is the injectable half, so a test drives a fake clientset and
|
||||
// the defaults are exercised by the same code production runs.
|
||||
func newSharedStore(client kubernetes.Interface, cfg SharedStoreConfig) (*SharedStore, error) {
|
||||
if cfg.Namespace == "" {
|
||||
cfg.Namespace = os.Getenv("POD_NAMESPACE")
|
||||
}
|
||||
if cfg.Namespace == "" {
|
||||
return nil, errors.New("acme shared store: namespace is required (set POD_NAMESPACE)")
|
||||
}
|
||||
if cfg.Self == "" {
|
||||
cfg.Self = os.Getenv("POD_NAME")
|
||||
}
|
||||
if cfg.Self == "" {
|
||||
// Hostname is the pod name under Kubernetes, and it is stable for the
|
||||
// pod's life — which is what ha requires of an identity.
|
||||
if h, err := os.Hostname(); err == nil {
|
||||
cfg.Self = h
|
||||
}
|
||||
}
|
||||
if cfg.Self == "" {
|
||||
return nil, errors.New("acme shared store: a stable self identity is required (set POD_NAME)")
|
||||
}
|
||||
if cfg.SecretName == "" {
|
||||
cfg.SecretName = "ingress-acme"
|
||||
}
|
||||
if cfg.LeaseName == "" {
|
||||
cfg.LeaseName = "ingress-acme"
|
||||
}
|
||||
if cfg.LeaseDuration <= 0 {
|
||||
cfg.LeaseDuration = 30 * time.Second
|
||||
}
|
||||
if cfg.PollInterval <= 0 {
|
||||
cfg.PollInterval = 10 * time.Second
|
||||
}
|
||||
s := &SharedStore{
|
||||
client: client,
|
||||
namespace: cfg.Namespace,
|
||||
name: cfg.SecretName,
|
||||
leaseName: cfg.LeaseName,
|
||||
self: cfg.Self,
|
||||
leaseDuration: cfg.LeaseDuration,
|
||||
pollInterval: cfg.PollInterval,
|
||||
cache: map[string]*StoredData{},
|
||||
}
|
||||
s.leases = &kubeLeases{store: s}
|
||||
return s, nil
|
||||
}
|
||||
|
||||
// IsWriter reports whether this replica currently owns the ACME role. The
|
||||
// ordering path asks it so a reader does not order a certificate it would then be
|
||||
// refused permission to store — the order is the expensive half (a DNS-01
|
||||
// round-trip and a rate-limit slot), so declining early is the point.
|
||||
func (s *SharedStore) IsWriter(ctx context.Context) bool {
|
||||
_, err := s.leases.Acquire(ctx, "acme")
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// Start begins the reader poll. It returns immediately; the loop exits when ctx
|
||||
// is done. The owner polls too: it is how it notices a write it did not make,
|
||||
// which is what a lease handoff looks like from the new owner's side.
|
||||
func (s *SharedStore) Start(ctx context.Context) {
|
||||
go func() {
|
||||
t := time.NewTicker(s.pollInterval)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
if err := s.refresh(ctx); err != nil && !kerrors.IsNotFound(err) {
|
||||
log.Debug().Err(err).Msg("acme shared store: refresh")
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// refresh reloads the cache when the object has changed. It compares
|
||||
// resourceVersion first, so a poll that finds nothing new costs one GET and no
|
||||
// deserialization.
|
||||
func (s *SharedStore) refresh(ctx context.Context) error {
|
||||
sec, err := s.client.CoreV1().Secrets(s.namespace).Get(ctx, s.name, metav1.GetOptions{})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// An EMPTY resourceVersion carries no version to compare, so it can only mean
|
||||
// reload. Treating it as "unchanged" makes the first read stick forever and a
|
||||
// reader never sees a certificate the writer obtained — silently, because
|
||||
// nothing errors.
|
||||
s.mu.RLock()
|
||||
unchanged := sec.ResourceVersion != "" && sec.ResourceVersion == s.rv
|
||||
s.mu.RUnlock()
|
||||
if unchanged {
|
||||
return nil
|
||||
}
|
||||
data, err := decodeStored(sec.Data[storeKey])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
s.mu.Lock()
|
||||
s.cache, s.rv, s.loaded = data, sec.ResourceVersion, true
|
||||
s.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetAccount returns the resolver's ACME account. Every replica reads it,
|
||||
// including readers: an account is what makes a certificate renewable, so a
|
||||
// replica that took over without one would have to register a new account and
|
||||
// re-order everything it already holds.
|
||||
func (s *SharedStore) GetAccount(resolverName string) (*Account, error) {
|
||||
d, err := s.stateOf(resolverName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return d.Account, nil
|
||||
}
|
||||
|
||||
func (s *SharedStore) SaveAccount(resolverName string, account *Account) error {
|
||||
return s.mutate(resolverName, func(d *StoredData) { d.Account = account })
|
||||
}
|
||||
|
||||
func (s *SharedStore) GetCertificates(resolverName string) ([]*CertAndStore, error) {
|
||||
d, err := s.stateOf(resolverName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return d.Certificates, nil
|
||||
}
|
||||
|
||||
// SaveCertificates replaces the resolver's certificate set, which is the shape
|
||||
// the Store interface asks for. Under one writer that is safe: the owner holds the
|
||||
// whole set in memory and re-states it. It is NOT safe under two, which is why the
|
||||
// round check below is the load-bearing line and not a formality.
|
||||
func (s *SharedStore) SaveCertificates(resolverName string, certificates []*CertAndStore) error {
|
||||
return s.mutate(resolverName, func(d *StoredData) { d.Certificates = certificates })
|
||||
}
|
||||
|
||||
// stateOf serves the cache, filling it on first use so a boot before the first
|
||||
// poll tick is not blind.
|
||||
func (s *SharedStore) stateOf(resolverName string) (*StoredData, error) {
|
||||
s.mu.RLock()
|
||||
d, seeded := s.cache[resolverName], s.loaded
|
||||
s.mu.RUnlock()
|
||||
if !seeded {
|
||||
if err := s.refresh(context.Background()); err != nil && !kerrors.IsNotFound(err) {
|
||||
return nil, err
|
||||
}
|
||||
s.mu.RLock()
|
||||
d = s.cache[resolverName]
|
||||
s.mu.RUnlock()
|
||||
}
|
||||
if d == nil {
|
||||
return &StoredData{}, nil
|
||||
}
|
||||
return d, nil
|
||||
}
|
||||
|
||||
// mutate is the one write path: acquire the round, read-modify-write under the
|
||||
// API server's resourceVersion precondition, and refuse a superseded round.
|
||||
//
|
||||
// Ordering matters. The round is taken FIRST, so a replica that has lost the
|
||||
// lease stops before it reads, and the precondition is checked LAST, so two
|
||||
// writers that raced both cannot win. The retry exists for the benign conflict —
|
||||
// the owner writing twice in quick succession — and gives up rather than looping,
|
||||
// because a persistent conflict means someone else is writing and this replica
|
||||
// should not be.
|
||||
func (s *SharedStore) mutate(resolverName string, apply func(*StoredData)) error {
|
||||
ctx := context.Background()
|
||||
|
||||
lease, err := s.leases.Acquire(ctx, "acme")
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: %w", ErrNotWriter, err)
|
||||
}
|
||||
|
||||
const attempts = 3
|
||||
for i := range attempts {
|
||||
// Whether the object EXISTS comes from the Get error and nothing else.
|
||||
// Inferring it from an empty resourceVersion conflates "absent" with
|
||||
// "present but unversioned", and then every write takes the Create branch
|
||||
// and fails AlreadyExists forever against an object that is right there.
|
||||
var absent bool
|
||||
sec, err := s.client.CoreV1().Secrets(s.namespace).Get(ctx, s.name, metav1.GetOptions{})
|
||||
switch {
|
||||
case kerrors.IsNotFound(err):
|
||||
absent = true
|
||||
sec = &corev1.Secret{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: s.name, Namespace: s.namespace},
|
||||
Data: map[string][]byte{},
|
||||
}
|
||||
case err != nil:
|
||||
return fmt.Errorf("acme shared store: read: %w", err)
|
||||
}
|
||||
|
||||
// Two refusals, not one. BEHIND the recorded round means this writer was
|
||||
// deposed and has not noticed. AT the recorded round but under a different
|
||||
// identity means two replicas believe they hold the same round — the
|
||||
// cross-object window between the Lease and this Secret, which no
|
||||
// Kubernetes transaction spans. Either way the older writer's work becomes
|
||||
// harmless instead of authoritative.
|
||||
recorded, who := roundOf(sec), string(sec.Data[holderKey])
|
||||
switch {
|
||||
case recorded > lease.Round:
|
||||
return fmt.Errorf("%w: at round %d, object records %d", ErrStaleRound, lease.Round, recorded)
|
||||
case recorded == lease.Round && who != "" && who != s.self:
|
||||
return fmt.Errorf("%w: round %d is held by %s, not %s", ErrStaleRound, recorded, who, s.self)
|
||||
}
|
||||
|
||||
state, err := decodeStored(sec.Data[storeKey])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
d := state[resolverName]
|
||||
if d == nil {
|
||||
d = &StoredData{}
|
||||
state[resolverName] = d
|
||||
}
|
||||
apply(d)
|
||||
|
||||
blob, err := json.Marshal(state)
|
||||
if err != nil {
|
||||
return fmt.Errorf("acme shared store: encode: %w", err)
|
||||
}
|
||||
if sec.Data == nil {
|
||||
sec.Data = map[string][]byte{}
|
||||
}
|
||||
sec.Data[storeKey] = blob
|
||||
sec.Data[roundKey] = []byte(fmt.Sprintf("%d", lease.Round))
|
||||
sec.Data[holderKey] = []byte(s.self)
|
||||
|
||||
var out *corev1.Secret
|
||||
if absent {
|
||||
out, err = s.client.CoreV1().Secrets(s.namespace).Create(ctx, sec, metav1.CreateOptions{})
|
||||
} else {
|
||||
// Update carries the resourceVersion read above, so the API server
|
||||
// rejects it if anything wrote in between. That is the atomicity.
|
||||
out, err = s.client.CoreV1().Secrets(s.namespace).Update(ctx, sec, metav1.UpdateOptions{})
|
||||
}
|
||||
switch {
|
||||
case kerrors.IsConflict(err) || kerrors.IsAlreadyExists(err):
|
||||
if i == attempts-1 {
|
||||
return fmt.Errorf("acme shared store: %w after %d attempts", err, attempts)
|
||||
}
|
||||
continue
|
||||
case err != nil:
|
||||
return fmt.Errorf("acme shared store: write: %w", err)
|
||||
}
|
||||
|
||||
s.mu.Lock()
|
||||
s.cache, s.rv, s.loaded = state, out.ResourceVersion, true
|
||||
s.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
return errors.New("acme shared store: exhausted write attempts")
|
||||
}
|
||||
|
||||
// roundOf reads the recorded round, treating anything unparseable as zero. A
|
||||
// corrupt round must not wedge writing forever: zero admits the next write, which
|
||||
// then records a good value.
|
||||
func roundOf(sec *corev1.Secret) ha.Round {
|
||||
var r uint64
|
||||
if _, err := fmt.Sscanf(string(sec.Data[roundKey]), "%d", &r); err != nil {
|
||||
return 0
|
||||
}
|
||||
return ha.Round(r)
|
||||
}
|
||||
|
||||
// decodeStored parses the state document, treating empty as empty rather than as
|
||||
// an error — a store that has never been written is the normal first-boot case.
|
||||
func decodeStored(blob []byte) (map[string]*StoredData, error) {
|
||||
out := map[string]*StoredData{}
|
||||
if len(blob) == 0 {
|
||||
return out, nil
|
||||
}
|
||||
if err := json.Unmarshal(blob, &out); err != nil {
|
||||
return nil, fmt.Errorf("acme shared store: decode: %w", err)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// kubeLeases implements ha.Leases over a Kubernetes Lease.
|
||||
//
|
||||
// leaseTransitions is the round: the API server increments it on every handoff,
|
||||
// so it is monotone across all callers and strictly increases when the role moves
|
||||
// — ha's invariant, enforced by something that already exists rather than by us.
|
||||
type kubeLeases struct{ store *SharedStore }
|
||||
|
||||
func (k *kubeLeases) Acquire(ctx context.Context, key string) (ha.Lease, error) {
|
||||
s := k.store
|
||||
api := s.client.CoordinationV1().Leases(s.namespace)
|
||||
now := metav1.NewMicroTime(time.Now())
|
||||
secs := int32(s.leaseDuration / time.Second)
|
||||
|
||||
cur, err := api.Get(ctx, s.leaseName, metav1.GetOptions{})
|
||||
if kerrors.IsNotFound(err) {
|
||||
created, cerr := api.Create(ctx, &coordv1.Lease{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: s.leaseName, Namespace: s.namespace},
|
||||
Spec: coordv1.LeaseSpec{
|
||||
HolderIdentity: &s.self,
|
||||
LeaseDurationSeconds: &secs,
|
||||
AcquireTime: &now,
|
||||
RenewTime: &now,
|
||||
LeaseTransitions: ptr[int32](1),
|
||||
},
|
||||
}, metav1.CreateOptions{})
|
||||
if cerr != nil {
|
||||
// Someone else created it first. That is a normal race and the
|
||||
// answer is simply that we are not the writer this round.
|
||||
return ha.Lease{}, fmt.Errorf("acquire %q: %w", key, cerr)
|
||||
}
|
||||
return leaseOf(key, s.self, created), nil
|
||||
}
|
||||
if err != nil {
|
||||
return ha.Lease{}, fmt.Errorf("acquire %q: %w", key, err)
|
||||
}
|
||||
|
||||
held := cur.Spec.HolderIdentity != nil && *cur.Spec.HolderIdentity == s.self
|
||||
expired := cur.Spec.RenewTime == nil ||
|
||||
time.Since(cur.Spec.RenewTime.Time) > s.leaseDuration
|
||||
|
||||
if !held && !expired {
|
||||
return ha.Lease{}, fmt.Errorf("acquire %q: held by %s", key, holder(cur))
|
||||
}
|
||||
|
||||
next := cur.DeepCopy()
|
||||
next.Spec.RenewTime = &now
|
||||
if !held {
|
||||
// Taking over: the round MUST move, so a write from the previous holder
|
||||
// is refused from this moment on.
|
||||
next.Spec.HolderIdentity = &s.self
|
||||
next.Spec.AcquireTime = &now
|
||||
next.Spec.LeaseTransitions = ptr(transitions(cur) + 1)
|
||||
}
|
||||
|
||||
out, err := api.Update(ctx, next, metav1.UpdateOptions{})
|
||||
if err != nil {
|
||||
// A conflict means another replica moved the lease first — it is the
|
||||
// writer, we are not. Fail closed.
|
||||
return ha.Lease{}, fmt.Errorf("acquire %q: %w", key, err)
|
||||
}
|
||||
return leaseOf(key, s.self, out), nil
|
||||
}
|
||||
|
||||
func leaseOf(key, self string, l *coordv1.Lease) ha.Lease {
|
||||
return ha.Lease{
|
||||
Key: key,
|
||||
Owner: ha.Member{ID: self},
|
||||
Round: ha.Round(transitions(l)),
|
||||
}
|
||||
}
|
||||
|
||||
func transitions(l *coordv1.Lease) int32 {
|
||||
if l.Spec.LeaseTransitions == nil {
|
||||
return 0
|
||||
}
|
||||
return *l.Spec.LeaseTransitions
|
||||
}
|
||||
|
||||
func holder(l *coordv1.Lease) string {
|
||||
if l.Spec.HolderIdentity == nil {
|
||||
return "<none>"
|
||||
}
|
||||
return *l.Spec.HolderIdentity
|
||||
}
|
||||
|
||||
func ptr[T any](v T) *T { return &v }
|
||||
|
||||
// writerGate is implemented by a Store that knows whether THIS replica may write.
|
||||
//
|
||||
// It is an OPTIONAL interface on purpose: LocalStore does not implement it and
|
||||
// must not, because a single-replica deployment has no election to consult and
|
||||
// asking one would be a failure mode invented out of nothing. So the gate is
|
||||
// present exactly when there is something to gate.
|
||||
type writerGate interface {
|
||||
IsWriter(ctx context.Context) bool
|
||||
}
|
||||
|
||||
// Compile-time proof that SharedStore satisfies both the store contract and the
|
||||
// gate, so a refactor that drops either is a build failure rather than a silent
|
||||
// return to every-replica-orders.
|
||||
var (
|
||||
_ Store = (*SharedStore)(nil)
|
||||
_ writerGate = (*SharedStore)(nil)
|
||||
)
|
||||
@@ -0,0 +1,318 @@
|
||||
// Copyright 2025 Hanzo AI Inc. All Rights Reserved.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
|
||||
package acme
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"k8s.io/apimachinery/pkg/runtime/schema"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/hanzoai/ingress/pkg/types"
|
||||
kerrors "k8s.io/apimachinery/pkg/api/errors"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/apimachinery/pkg/runtime"
|
||||
"k8s.io/client-go/kubernetes/fake"
|
||||
k8stesting "k8s.io/client-go/testing"
|
||||
)
|
||||
|
||||
// Two replicas over ONE fake API server, which is the whole point: the bug being
|
||||
// fixed is invisible to a single-replica test, and a mocked store would not have
|
||||
// the conflict semantics that make the fix work.
|
||||
func twoReplicas(t *testing.T) (a, b *SharedStore) {
|
||||
t.Helper()
|
||||
client := fake.NewSimpleClientset()
|
||||
cfg := func(self string) SharedStoreConfig {
|
||||
return SharedStoreConfig{Namespace: "hanzo", Self: self, LeaseDuration: time.Minute}
|
||||
}
|
||||
var err error
|
||||
if a, err = newSharedStore(client, cfg("ingress-0")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if b, err = newSharedStore(client, cfg("ingress-1")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return a, b
|
||||
}
|
||||
|
||||
func certs(domains ...string) []*CertAndStore {
|
||||
out := make([]*CertAndStore, 0, len(domains))
|
||||
for _, d := range domains {
|
||||
out = append(out, &CertAndStore{Certificate: Certificate{Domain: types.Domain{Main: d}}})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// EXACTLY ONE replica may write, and the other must be told so rather than
|
||||
// silently succeeding into its own copy — which is what the file store did.
|
||||
func TestOnlyOneReplicaIsTheWriter(t *testing.T) {
|
||||
a, b := twoReplicas(t)
|
||||
ctx := context.Background()
|
||||
|
||||
if !a.IsWriter(ctx) {
|
||||
t.Fatal("the first replica to ask must become the writer")
|
||||
}
|
||||
if b.IsWriter(ctx) {
|
||||
t.Fatal("a second replica claimed the writer role while the first held it")
|
||||
}
|
||||
|
||||
if err := a.SaveCertificates("letsencrypt", certs("a.hanzo.ai")); err != nil {
|
||||
t.Fatalf("owner write: %v", err)
|
||||
}
|
||||
err := b.SaveCertificates("letsencrypt", certs("b.hanzo.ai"))
|
||||
if !errors.Is(err, ErrNotWriter) {
|
||||
t.Fatalf("a reader's write must be refused as ErrNotWriter, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The reader SERVES what the writer obtained. This is the property that makes one
|
||||
// writer acceptable: a host is not stuck behind whichever replica happens to hold
|
||||
// the lease.
|
||||
func TestReaderServesTheWritersCertificates(t *testing.T) {
|
||||
a, b := twoReplicas(t)
|
||||
ctx := context.Background()
|
||||
|
||||
if !a.IsWriter(ctx) {
|
||||
t.Fatal("expected the first replica to own the role")
|
||||
}
|
||||
if err := a.SaveCertificates("letsencrypt", certs("shared.hanzo.ai")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if err := b.refresh(ctx); err != nil {
|
||||
t.Fatalf("reader refresh: %v", err)
|
||||
}
|
||||
got, err := b.GetCertificates("letsencrypt")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 1 || got[0].Certificate.Domain.Main != "shared.hanzo.ai" {
|
||||
t.Fatalf("reader did not see the writer's certificate: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A writer that lost the lease and has not noticed must not be able to clobber the
|
||||
// new writer's state. Heartbeat liveness cannot stop it; the round can.
|
||||
func TestDeposedWriterIsRefused(t *testing.T) {
|
||||
a, b := twoReplicas(t)
|
||||
ctx := context.Background()
|
||||
|
||||
if !a.IsWriter(ctx) {
|
||||
t.Fatal("expected a to own first")
|
||||
}
|
||||
if err := a.SaveCertificates("letsencrypt", certs("first.hanzo.ai")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Expire the lease so b can legitimately take over, exactly as it would after
|
||||
// a's node stopped renewing.
|
||||
api := a.client.CoordinationV1().Leases("hanzo")
|
||||
l, err := api.Get(ctx, a.leaseName, metav1.GetOptions{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stale := metav1.NewMicroTime(time.Now().Add(-2 * time.Minute))
|
||||
l.Spec.RenewTime = &stale
|
||||
if _, err := api.Update(ctx, l, metav1.UpdateOptions{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if !b.IsWriter(ctx) {
|
||||
t.Fatal("b must take over an expired lease")
|
||||
}
|
||||
if err := b.SaveCertificates("letsencrypt", certs("second.hanzo.ai")); err != nil {
|
||||
t.Fatalf("new writer must be able to write: %v", err)
|
||||
}
|
||||
|
||||
// a is still running and still believes it writes. Its round is now behind.
|
||||
err = a.SaveCertificates("letsencrypt", certs("clobber.hanzo.ai"))
|
||||
if err == nil {
|
||||
t.Fatal("the deposed writer's write was ADMITTED — this is the split-brain the round exists to prevent")
|
||||
}
|
||||
|
||||
// And the new writer's data survived intact.
|
||||
if err := b.refresh(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, _ := b.GetCertificates("letsencrypt")
|
||||
if len(got) != 1 || got[0].Certificate.Domain.Main != "second.hanzo.ai" {
|
||||
t.Fatalf("the winner's state was damaged: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The round must MOVE on handoff, or a deposed writer's round still matches and
|
||||
// the refusal above cannot happen.
|
||||
func TestRoundStrictlyIncreasesOnHandoff(t *testing.T) {
|
||||
a, b := twoReplicas(t)
|
||||
ctx := context.Background()
|
||||
|
||||
first, err := a.leases.Acquire(ctx, "acme")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// A renewal by the SAME holder keeps the round: the owner re-states the same
|
||||
// object on every pass, and bumping per write would make its own last write
|
||||
// look stale.
|
||||
again, err := a.leases.Acquire(ctx, "acme")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if again.Round != first.Round {
|
||||
t.Fatalf("a stable owner's round moved: %d -> %d", first.Round, again.Round)
|
||||
}
|
||||
|
||||
api := a.client.CoordinationV1().Leases("hanzo")
|
||||
l, _ := api.Get(ctx, a.leaseName, metav1.GetOptions{})
|
||||
stale := metav1.NewMicroTime(time.Now().Add(-2 * time.Minute))
|
||||
l.Spec.RenewTime = &stale
|
||||
if _, err := api.Update(ctx, l, metav1.UpdateOptions{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
taken, err := b.leases.Acquire(ctx, "acme")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if taken.Round <= first.Round {
|
||||
t.Fatalf("handoff did not advance the round: %d -> %d", first.Round, taken.Round)
|
||||
}
|
||||
}
|
||||
|
||||
// A rescheduled pod must start with the estate already known. This is the
|
||||
// re-issuance storm: ~70 names against a 50/week ceiling, triggered by a restart.
|
||||
func TestFreshReplicaInheritsStateAndOrdersNothing(t *testing.T) {
|
||||
a, _ := twoReplicas(t)
|
||||
ctx := context.Background()
|
||||
|
||||
if !a.IsWriter(ctx) {
|
||||
t.Fatal("expected a to own")
|
||||
}
|
||||
if err := a.SaveCertificates("letsencrypt", certs("one.hanzo.ai", "two.hanzo.ai", "three.hanzo.ai")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := a.SaveAccount("letsencrypt", &Account{Email: "[email protected]"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// A brand-new pod: same cluster, no local state of any kind.
|
||||
fresh, err := newSharedStore(a.client, SharedStoreConfig{
|
||||
Namespace: "hanzo", Self: "ingress-2", LeaseDuration: time.Minute,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
got, err := fresh.GetCertificates("letsencrypt")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 3 {
|
||||
t.Fatalf("a fresh replica saw %d certificates, not 3 — it would re-order the estate", len(got))
|
||||
}
|
||||
// The ACCOUNT matters as much: without it a takeover has to register anew and
|
||||
// re-order everything it already holds.
|
||||
acct, err := fresh.GetAccount("letsencrypt")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if acct == nil || acct.Email != "[email protected]" {
|
||||
t.Fatalf("a fresh replica did not inherit the ACME account: %+v", acct)
|
||||
}
|
||||
}
|
||||
|
||||
// An interleaved read-modify-write must not lose data. The precondition is what
|
||||
// makes that true, so a conflict has to be retried rather than swallowed.
|
||||
func TestConcurrentWriteConflictIsRetriedNotLost(t *testing.T) {
|
||||
a, _ := twoReplicas(t)
|
||||
ctx := context.Background()
|
||||
if !a.IsWriter(ctx) {
|
||||
t.Fatal("expected a to own")
|
||||
}
|
||||
if err := a.SaveCertificates("letsencrypt", certs("first.hanzo.ai")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// One injected conflict, as if another writer slipped in between the read and
|
||||
// the update, then success.
|
||||
client := a.client.(*fake.Clientset)
|
||||
var once bool
|
||||
client.PrependReactor("update", "secrets", func(k8stesting.Action) (bool, runtime.Object, error) {
|
||||
if !once {
|
||||
once = true
|
||||
return true, nil, kerrors.NewConflict(
|
||||
schema.GroupResource{Resource: "secrets"}, a.name, errors.New("modified"))
|
||||
}
|
||||
return false, nil, nil
|
||||
})
|
||||
|
||||
if err := a.SaveCertificates("letsencrypt", certs("first.hanzo.ai", "second.hanzo.ai")); err != nil {
|
||||
t.Fatalf("a benign conflict must be retried, not surfaced: %v", err)
|
||||
}
|
||||
if !once {
|
||||
t.Fatal("the conflict reactor never fired — the test proved nothing")
|
||||
}
|
||||
got, _ := a.GetCertificates("letsencrypt")
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("the retry lost data: %d certificates, want 2", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
// A store that cannot reach the cluster must FAIL, never fall back to per-node
|
||||
// state — the fallback would reappear exactly when replicas are most likely to
|
||||
// disagree.
|
||||
func TestMissingIdentityFailsRatherThanGuessing(t *testing.T) {
|
||||
if _, err := newSharedStore(fake.NewSimpleClientset(), SharedStoreConfig{Self: "x"}); err == nil {
|
||||
t.Fatal("a store with no namespace was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
// A reader must order NOTHING. This is the rate-limit half of the fix: two
|
||||
// replicas each ordering every name is what turns a reschedule into a week
|
||||
// without new certificates, and it is also what makes two DNS-01 challenges race
|
||||
// to write and clean up the same TXT record.
|
||||
func TestReaderOrdersNothing(t *testing.T) {
|
||||
a, b := twoReplicas(t)
|
||||
ctx := context.Background()
|
||||
|
||||
if !a.IsWriter(ctx) {
|
||||
t.Fatal("expected a to own the role")
|
||||
}
|
||||
if b.IsWriter(ctx) {
|
||||
t.Fatal("expected b to be a reader")
|
||||
}
|
||||
|
||||
// A Provider with NOTHING configured — no tlsManager, no certificates. The
|
||||
// reader must return before it reads any of that, which is what makes this
|
||||
// meaningful: the gate is the first thing in the function, so a reader spends
|
||||
// none of the expensive half.
|
||||
reader := &Provider{Store: b}
|
||||
if got := reader.getUncheckedDomains(ctx, []string{"new.hanzo.ai"}, "default"); len(got) != 0 {
|
||||
t.Fatalf("a reader returned %v to order — it must return nothing", got)
|
||||
}
|
||||
|
||||
// The writer is deliberately NOT exercised through the same call here: past the
|
||||
// gate it dereferences a tlsManager this bare Provider does not have, so it
|
||||
// would panic on nil for a reason that has nothing to do with the gate. What
|
||||
// matters, and is asserted above, is that the predicate the gate consults
|
||||
// answers differently for the two — and the reader's early return proves the
|
||||
// gate is reached before any of that state.
|
||||
if !a.IsWriter(ctx) || b.IsWriter(ctx) {
|
||||
t.Fatal("the gate predicate does not distinguish writer from reader")
|
||||
}
|
||||
}
|
||||
|
||||
// LocalStore must NOT be gated. A single-replica deployment has no election to
|
||||
// consult, and asking one would invent a failure mode where none existed — so the
|
||||
// optional interface is the mechanism, and this pins it.
|
||||
func TestLocalStoreIsNotGated(t *testing.T) {
|
||||
// A typed nil is enough: the assertion is about the TYPE's method set, so this
|
||||
// needs no constructor and no filesystem.
|
||||
var s Store = (*LocalStore)(nil)
|
||||
if _, ok := s.(writerGate); ok {
|
||||
t.Fatal("LocalStore implements writerGate — a single replica would now need an election to order anything")
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user