hanzo-dev
78259297ce
Found by rendering every values file exactly as CD does and comparing against the
live objects — not by reading the files.
THE MECHANISM THAT MATTERS, because `kubectl diff --server-side` does NOT predict
it: a live object applied with client-side `kubectl apply` carries
`kubectl.kubernetes.io/last-applied-configuration`. The FIRST server-side apply
adopts that annotation's field set as the prior owner and then deletes the
annotation — so every field the annotation claimed and the render omits is
PRUNED. Proven the hard way: syncing `lux-system/lux-validator-exporter` dropped
`imagePullSecrets` (the values file said `[]`, live said `ghcr-secret`) and the
new pod took a 401 from ghcr.io on a private image. `kubectl diff` had shown
nothing. The old ReplicaSet kept serving — maxUnavailable 0 — so it was a stuck
rollout, not an outage.
Corrected, each to what live actually runs:
lux-system/lux-validator-exporter, lux-cloud/cloud imagePullSecrets: ghcr-secret
lux-bridge/lux-bridge, lux-mainnet/rpc-tip-gate podSecurityContext runAsUser/
lux-devnet/dchain-edge, lux-devnet/lux-indexer runAsGroup [/runAsNonRoot/
seccompProfile]
The securityContext four are not cosmetic: the chart renders only `fsGroup`
unless `podSecurityContext` is set, so omitting it returns four pods to uid 0.
LANDMINE 1 — lux-mpc/mpc-kv rendered a Deployment while a StatefulSet of the same
name holds the valkey-data PVC. It had ALREADY fired: Service/mpc-kv carried both
mpc-kv-0 and a Deployment pod as endpoints since 2026-03-26, so mpc-api and
mpc-node — live threshold signing — were round-robining their bus between a
persistent store and an ephemeral one. Both measured DBSIZE 0, so nothing was
lost. Now renders the StatefulSet (persistence.enabled + claim name valkey-data,
which is what a bound volume matches on), and the stray Deployment is deleted:
Service/mpc-kv has exactly one endpoint, mpc-kv-0, with all five mpc-node pods
and mpc-api untouched. It is still apiserver-Forbidden — the live claim template
carries labels the chart cannot express and volumeClaimTemplates is atomic and
immutable — but a refused apply changes nothing, where before a sync created a
second workload. Stated in the file, not hidden.
LANDMINE 2 — lux-system/s3 deleted. It described a 0-replica husk AND would have
moved Service/s3's selector off `app: seaweedfs`, whose only endpoint is
seaweedfs-0, to `app: s3`, which matches nothing. Reason and the naming decision
it waits on are in deploy/README.md.
Co-authored-by: Hanzo Dev <[email protected]>
67 lines
2.6 KiB
YAML
67 lines
2.6 KiB
YAML
# lux-mpc/mpc-kv — the Valkey the MPC signers use for coordination.
|
|
#
|
|
# THIS FILE USED TO RENDER A DEPLOYMENT, AND THAT WAS THE LANDMINE. live2values.py
|
|
# generated it from a live Deployment named mpc-kv — but a StatefulSet of the SAME
|
|
# name has held the valkey-data PVC since 2026-03-26, and Service/mpc-kv selects
|
|
# `app: mpc-kv`, so BOTH pods were endpoints. mpc-api and mpc-node — live
|
|
# threshold signing — were round-robining their bus between a persistent store
|
|
# and an ephemeral one. Measured DBSIZE 0 on both, so nothing was lost; the split
|
|
# was real regardless.
|
|
#
|
|
# `persistence` is what makes the chart render a StatefulSet
|
|
# (templates/_helpers.tpl: workloadKind derives from persistence.enabled or
|
|
# volumeClaimTemplates). The claim NAME must stay `valkey-data` — a StatefulSet
|
|
# binds existing volumes by name, and any other value provisions an empty one.
|
|
# strategy Recreate maps to updateStrategy OnDelete, which is right for a
|
|
# singleton on a ReadWriteOnce volume: two of those overlapping is corruption.
|
|
#
|
|
# The stray Deployment is deleted out-of-band. CD cannot do it — ClusterRole
|
|
# `hanzo-cd` withholds `delete` on purpose — and that is the correct division:
|
|
# git declares one workload per name, a human removes the duplicate once.
|
|
#
|
|
# STILL apiserver-REJECTED after this change, and that is an improvement worth
|
|
# stating precisely rather than hiding. The live volumeClaimTemplate carries
|
|
# labels (app: mpc-valkey, app.kubernetes.io/part-of: lux-mpc) that the chart has
|
|
# no way to express; volumeClaimTemplates is an atomic list AND immutable, so the
|
|
# apply is Forbidden. Before this change a sync CREATED a second workload; now a
|
|
# sync is refused and changes nothing. Finishing it needs either a chart that can
|
|
# label a claim template, or an orphan-delete recreate of the StatefulSet — a
|
|
# decision about live MPC coordination, not a values edit.
|
|
replicas: 1
|
|
selectorLabels:
|
|
app: mpc-kv
|
|
containerName: kv
|
|
image:
|
|
repository: ghcr.io/hanzoai/kv
|
|
tag: '9'
|
|
pullPolicy: IfNotPresent
|
|
partOf: lux-mpc
|
|
component: mpc
|
|
resources:
|
|
limits:
|
|
cpu: 200m
|
|
memory: 256Mi
|
|
requests:
|
|
cpu: 50m
|
|
memory: 64Mi
|
|
ports:
|
|
- containerPort: 6379
|
|
name: valkey
|
|
persistence:
|
|
enabled: true
|
|
name: valkey-data
|
|
mountPath: /data
|
|
size: 2Gi
|
|
workload:
|
|
serviceName: mpc-kv-headless
|
|
# Matches the live updateStrategy. Recreate would map to OnDelete and the
|
|
# apiserver rejects an OnDelete strategy while live still carries
|
|
# rollingUpdate.partition — two reasons for one refusal is one too many.
|
|
strategy: RollingUpdate
|
|
topologySpread:
|
|
enabled: false
|
|
lifecycle: null
|
|
terminationGracePeriodSeconds: 30
|
|
defaultReadinessProbe: false
|
|
imagePullSecrets: []
|