bridge/docs/operator-runbook.md

cmd/bridge — Operator Runbook

Operating the unified Lux Bridge binary (cmd/bridge). One image, one port, embedded SPA + native /v1/bridge/* API + background drivers (deposit watcher, MPC signing, broadcast).

This binary replaces the legacy bridge-server (Express + Prisma) and bridge-ui (separate Vite SPA) deployment split.

---

1. Overview

Aspect	Value
Binary	cmd/bridge
Image	ghcr.io/luxfi/bridge:<tag>
Listen port	:8080 (HTTP)
Operator port	:8081 (HTTP, not published by the Ingress)
Health endpoint	GET /health (200 JSON)
Config file	/etc/bridge/networks.yaml
Data directory	/var/lib/lux-bridge (zapdb / Lux Badger v4)
Logger	luxfi/log (JSON, one line per event)
HTTP framework	hanzoai/zip on Fiber v3 / fasthttp
Swap store	luxfi/zapdb (embedded KV, exclusive dir lock)

Routes:

/                          embedded SPA + SPA-routing fallback
/envs.js                   runtime config (window.ENV)
/icon.svg, /logo.svg       per-host brand assets (disk override possible)
/health                    service health + driver stats
/v1/bridge/networks        supported chains (from networks.yaml)
/v1/bridge/tokens          tokens per chain
/v1/bridge/quote           price quote
/v1/bridge/limits          per-token min/max swap caps
/v1/bridge/swaps           create swap
/v1/bridge/swaps/:id       fetch swap

Operator routes, on BRIDGE_ADMIN_ADDR and only there. They identify no caller, and the Ingress publishes the public port alone:

/metrics                   Prometheus exposition
/v1/bridge/swaps           list swaps, with signatures and signed dest txs
/v1/bridge/check-deposit   poll a source-chain RPC for an address balance
/admin/swaps/:id/reset          re-sign a swap      (BRIDGE_ADMIN_ROUTES=true)
/admin/swaps/:id/inject-raw-tx  push an operator's raw tx (same)

Background drivers (all on by default, all toggleable):

Driver	Reads from	Writes to	Default cadence
DepositWatcher	source-chain RPC	swap.State advancement	15s
SigningDriver	MPC daemon	swap.DestRawTx	5s
BroadcastDriver	dest-chain RPC	swap.State = completed	5s

---

2. Build & ship

Local build (smoke test)

go build -o /tmp/lux-bridge ./cmd/bridge
/tmp/lux-bridge --config cmd/bridge/networks.example.yaml --addr :8080
curl -sS localhost:8080/health | jq .

Docker image

The canonical Dockerfile is cmd/bridge/Dockerfile. Three stages:

1. ui-build — node:20-alpine, pnpm install + pnpm -C app/bridge build
2. go-build — golang:1.26.4-alpine, compiles cmd/bridge with embedded SPA
3. runtime — gcr.io/distroless/static-debian12:nonroot, ~25 MB

Build from the repo root (build context must include the pnpm workspace):

docker build \
  -f cmd/bridge/Dockerfile \
  --build-arg VERSION=$(git rev-parse --short HEAD) \
  -t ghcr.io/luxfi/bridge:$(git rev-parse --short HEAD) \
  -t ghcr.io/luxfi/bridge:latest \
  .

Push:

docker push ghcr.io/luxfi/bridge:$(git rev-parse --short HEAD)
docker push ghcr.io/luxfi/bridge:latest

---

3. Deploy to k8s

Manifest: k8s/bridge-deployment.yaml. Apply after the MPC StatefulSet from k8s/mpc-deployment.yaml is healthy (the bridge talks to mpc-api-svc.lux-bridge.svc:8081).

kubectl apply -f k8s/bridge-deployment.yaml
kubectl -n lux-bridge rollout status deployment/lux-bridge

The manifest creates: ConfigMap/lux-bridge-config, Deployment/lux-bridge, Service/lux-bridge, Ingress/lux-bridge, HorizontalPodAutoscaler/lux-bridge, PodDisruptionBudget/lux-bridge.

Required pre-existing resources

Resource	Created by
Namespace/lux-bridge	k8s/mpc-deployment.yaml
Secret/bridge-secrets	KMS sync (see mpc-deployment header)
Service/mpc-api-svc	k8s/mpc-deployment.yaml
TLS cluster issuer	cert-manager install

Rollout

# Pin a specific tag (do not run on :latest in prod)
kubectl -n lux-bridge set image deployment/lux-bridge \
  bridge=ghcr.io/luxfi/bridge:<git-sha>

# Watch the rollout
kubectl -n lux-bridge rollout status deployment/lux-bridge --watch

Rollback

kubectl -n lux-bridge rollout undo deployment/lux-bridge
kubectl -n lux-bridge rollout history deployment/lux-bridge

The deployment uses RollingUpdate with maxUnavailable: 0 — a bad image will not take down traffic; old pods stay up until new pods become Ready.

---

4. Configuration

All flags accept either a CLI argument or the matching env var. Env vars are the standard way to configure inside the container.

Required-ish

Env / Flag	Default	Notes
BRIDGE_CONFIG / --config	/etc/bridge/networks.yaml	Loaded from ConfigMap
BRIDGE_ADDR / --addr	:8080	The listener the Ingress publishes.
BRIDGE_ADMIN_ADDR / --admin-addr	""	The operator listener (:8081 in k8s). Empty serves none of the operator routes — including /metrics. Give it a port the Ingress does not route.
BRIDGE_DATA_DIR / --data-dir	""	Empty = in-memory swap store (lossy on restart). Set to a writable path on a PV for durability — e.g. /var/lib/lux-bridge.
BRIDGE_PROFILE / --profile	classical-compat	strict-pq for internal Lux↔Lux only

Backend wiring

Env / Flag	Default	Behavior when empty
BRIDGE_BACKEND_URL / --backend	""	Reverse proxy to legacy Node backend disabled — all /v1/bridge/* use native handlers
BRIDGE_BCHAIN_URL / --bchain-url	""	BridgeVM native handlers disabled — swap/quote stays on the legacy proxy if --backend is also set
BRIDGE_MPC_URL / --mpc-url	""	MPC keygen disabled — swaps with use_deposit_address=true return 503
BRIDGE_MPC_TOKEN / --mpc-token	""	Required for the live mpcd daemon — every endpoint except /health is bearer-protected
BRIDGE_MPC_IDENTITY_FILE / --mpc-identity-file	""	Convenience: derives the bearer token via SHA-256(seed‖"mpc-internal-api"). Use in dev; prod sets BRIDGE_MPC_TOKEN explicitly
BRIDGE_MPC_ORG_ID / --mpc-org-id	bridge	Multi-tenant key the MPC daemon multiplexes by

Source RPC overrides

BRIDGE_SOURCE_RPC_OVERRIDES / --source-rpc-overrides overrides specific source-chain RPCs at runtime when the package defaults (rpc.sepolia.org etc.) are stale or rate-limited. Format:

NETWORK1=URL1,NETWORK2=URL2

Used by both the deposit watcher (source) and the broadcast driver (destination) — many networks are both a source and a destination.

Example:

BRIDGE_SOURCE_RPC_OVERRIDES="ETHEREUM_SEPOLIA=https://ethereum-sepolia-rpc.publicnode.com,BITCOIN_TESTNET=https://mempool.space/testnet/api"

Driver toggles + cadence

Env / Flag	Default	Effect when set
--disable-deposit-check	off	/v1/bridge/check-deposit returns 404
--disable-deposit-watcher	off	Swaps never advance from user_deposit_pending
--disable-signing-driver	off	Swaps stall in bridge_transfer_pending
--disable-broadcast-driver	off	Signed txs never broadcast
--deposit-watcher-interval	15s	Polling cadence (source-chain RPC load)
--signing-interval	5s	MPC signing polling cadence
--broadcast-interval	5s	Broadcast polling cadence
--broadcast-timeout	15s	Per-RPC broadcast call timeout
--deposit-check-timeout	10s	Per-RPC source-poll timeout
--bchain-timeout	10s	Per-RPC b-chain call timeout
--mpc-timeout	120s	Per-MPC call timeout (matches mpc-wallet.ts)

Static assets

Env / Flag	Default	Effect
BRIDGE_STATIC_DIR / --static	""	Override the embedded SPA from disk (dev only)

---

5. Updating networks.yaml

The supported chains, tokens, limits, and brand are all driven from networks.yaml — mounted from the lux-bridge-config ConfigMap.

# Edit the ConfigMap in place
kubectl -n lux-bridge edit configmap lux-bridge-config

# Restart pods to pick up the change (the binary reads the file once at boot)
kubectl -n lux-bridge rollout restart deployment/lux-bridge

Adding a network requires no code change and no DB migration. The in-binary token registry (internal/tokens) covers detection + calldata for known assets; networks.yaml controls what the SPA and /v1/bridge/networks advertise.

---

6. Health, logs, metrics

Health

kubectl -n lux-bridge port-forward svc/lux-bridge 8080:80 &
curl -sS localhost:8080/health | jq .

Sample response:

{
  "status": "ok",
  "version": "<git-sha>",
  "backend_proxy": false,
  "bchain_rpc": true,
  "mpc_keygen": true,
  "deposit_check": true,
  "deposit_watcher": true,
  "signing_driver": true,
  "broadcast_driver": true,
  "profile": "BRIDGE_CLASSICAL_COMPAT_UNSAFE",
  "post_quantum_end_to_end": false,
  "watcher_stats": { "ticks": 12, "checks": 0, "advances": 0, "errors": 0 },
  "signing_stats": { "ticks": 12, "signs": 0, "errors": 0 },
  "broadcast_stats": { "ticks": 12, "attempts": 0, "successes": 0,
                       "failures": 0, "skipped_no_raw_tx": 0, "list_errors": 0 }
}

Use the driver stats blocks to spot misbehavior:

• watcher.errors rising → upstream source-chain RPC issues
• broadcast.failures rising → destination chain rejecting our txs
• signing.errors rising → MPC daemon misconfigured or down

Logs

JSON one-line-per-event via luxfi/log. Stream:

kubectl -n lux-bridge logs -f deployment/lux-bridge | jq -c

Filter for errors:

kubectl -n lux-bridge logs deployment/lux-bridge --tail=1000 \
  | jq -c 'select(.level=="error")'

Metrics

Prometheus exposition at /metrics on the operator port. The series name the MPC release wallets and report which of them cannot sign, so the scrape target is the admin port, never the published one (k8s/bridge-servicemonitor.yaml). Scraping by pod role instead:

- job_name: lux-bridge
  kubernetes_sd_configs:
    - role: pod
      namespaces: {names: [lux-bridge]}
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      regex: lux-bridge
      action: keep
    - source_labels: [__meta_kubernetes_pod_container_port_name]
      regex: admin
      action: keep

Reading it by hand:

kubectl -n lux-bridge port-forward svc/lux-bridge 8081:8081 &
curl -sS localhost:8081/metrics

---

7. Common operations

Pause the pipeline (incident response)

Disable all background drivers without taking the API down:

kubectl -n lux-bridge set env deployment/lux-bridge \
  BRIDGE_DISABLE_DEPOSIT_WATCHER=true \
  BRIDGE_DISABLE_SIGNING_DRIVER=true \
  BRIDGE_DISABLE_BROADCAST_DRIVER=true

Note: env-var names mirror the CLI flags. To reverse, drop the vars with kubectl set env … BRIDGE_DISABLE_…-.

Drain before maintenance

kubectl -n lux-bridge scale deployment/lux-bridge --replicas=0
# do work
kubectl -n lux-bridge scale deployment/lux-bridge --replicas=2

The PodDisruptionBudget keeps at least 1 pod up during cluster drains.

Force a config reload

kubectl -n lux-bridge rollout restart deployment/lux-bridge

Override an upstream RPC at runtime

kubectl -n lux-bridge set env deployment/lux-bridge \
  BRIDGE_SOURCE_RPC_OVERRIDES="ETHEREUM_SEPOLIA=https://ethereum-sepolia-rpc.publicnode.com"

Add a new token mapping

The binary's in-memory tokens.DefaultRegistry() covers the common bridged assets. To add a new ERC-20 mapping that the deposit watcher + tx assembler should recognize, edit internal/tokens/tokens.go and ship a new image — there is no runtime token-registration endpoint yet.

networks.yaml Token entries control SPA listings and per-token caps only; they do not wire detection or calldata. Detection + calldata live in internal/tokens.

---

8. Troubleshooting

Pod won't start: bridge profile invalid

BRIDGE_PROFILE must be strict-pq or classical-compat. Anything else exits 1 (this is intentional — typo = wrong security posture).

mpc_keygen: false in /health

BRIDGE_MPC_URL is unset or unreachable. Check:

kubectl -n lux-bridge get svc mpc-api-svc
kubectl -n lux-bridge get statefulset mpc-node

Without MPC keygen, swap creates with use_deposit_address=true return 503.

signing_driver: false despite MPC enabled

mchainClient is nil → BRIDGE_MPC_URL was empty at boot. Set it, roll the deployment. If MPC is reachable but signing still errors, check the bearer token (BRIDGE_MPC_TOKEN) and the org id (BRIDGE_MPC_ORG_ID).

Source-chain checks failing

Default RPCs (rpc.sepolia.org, etc.) get rate-limited. Override with BRIDGE_SOURCE_RPC_OVERRIDES. Track watcher_stats.errors in /health to confirm the fix.

Watcher / signing / broadcast all running but no progress

Check the swap is actually in the right state. Use the swap CRUD:

curl -sS localhost:8080/v1/bridge/swaps/<id> | jq .state

State machine:

user_deposit_pending
   → bridge_transfer_pending           (watcher: deposit confirmed)
   → bridge_transfer_pending_signing   (signing: MPC asked)
   → bridge_transfer_pending_broadcast (signing: tx signed)
   → completed                          (broadcast: pushed onchain)

---

9. Persistence + backups

The swap store (internal SwapStore interface) has two implementations:

• InMemoryStore — used when BRIDGE_DATA_DIR is empty. Concurrency-safe, zero ops cost, but every restart drops in-flight swap state. Dev only.
• ZapStore — used when BRIDGE_DATA_DIR is set. Backed by luxfi/zapdb (Lux-flavored Badger v4). LSM-tree KV with WAL durability, atomic db.Update transactions, MVCC concurrency control. Takes an exclusive directory lock — only one bridge pod can hold the data dir at a time.

Single-writer constraint

Because zapdb takes an exclusive dir lock, the k8s Deployment is pinned to:

replicas: 1
strategy: {type: Recreate}

Two pods can't share /var/lib/lux-bridge. The Recreate strategy ensures the old pod fully terminates (and releases the lock) before the new one mounts the PVC. This costs zero-downtime rollouts — expect ~10s of API gap during a deploy. The deposit watcher's 15s poll cadence absorbs this without losing real progress.

Scaling beyond one pod requires migrating swap state to a shared external store (Postgres, hanzoai/base SQLite served behind an API, etc.). Not on the roadmap yet — single-replica covers the bridge's expected load.

Disk layout

Inside /var/lib/lux-bridge:

000000.vlog        value log (immutable)
000001.vlog        ...
000002.vlog        currently being written
000003.sst         level-0 sorted string table
MANIFEST           the LSM tree's index
KEYREGISTRY        encryption-at-rest key registry (we don't use)
LOCK               the exclusive directory lock

Typical size: hundreds of bytes per swap. 1 GB holds millions of swaps. The 20Gi PVC in the manifest is generous head-room.

Backups

zapdb supports a streaming backup primitive. For first-deploy ops, a filesystem snapshot of the PV (cloud snapshot, Velero, etc.) while the pod is paused is sufficient.

Pause-snapshot-resume:

kubectl -n lux-bridge scale deployment/lux-bridge --replicas=0
# wait for pod termination, then snapshot the PV however your cloud does it
kubectl -n lux-bridge scale deployment/lux-bridge --replicas=1

In-flight signing/broadcast that was mid-RPC will rerun on restart — the state machine is idempotent (the watcher re-detects the same deposit; signing checks for an existing signature before re-asking MPC; broadcast checks for the dest tx hash before re-sending).

Drift recovery

If the LOCK file is stale (e.g. a node crash left the file behind), the new pod's NewZapStore() will fail with Cannot acquire dir lock. Inspect the data dir, confirm no other process holds it, then:

# inside the pod — preserves all data, just drops the lock file
rm /var/lib/lux-bridge/LOCK

Restart the pod after.

---

10. Known gaps

These are tracked separately and not operationally fixable:

• Non-EVM broadcasters (BTC, SOL, TON, XRP, DOT) return ErrFamilyNotImplemented. Add per-chain RPC clients to internal/broadcast to extend coverage.
• EIP-1559 gas pricing not implemented in internal/txassembler. EVM txs assemble with legacy (gasPrice) pricing. Most destination chains still accept legacy txs.
• Substrate SS58 address encoding not implemented.
• LP-333 BridgeVM client methods are stubbed — --bchain-url controls whether bchain.Client is constructed, but the handlers it backs are read-only.
• Single-writer swap store. Horizontal scale-out requires migrating to a shared external store.

---

11. Release-wallet signability monitoring

Alert on bridge_release_wallet_signable{network="..."} == 0. A background poller (WalletHealthPoller, cmd/bridge/wallet_health_poller.go) runs a harmless canary sign (arbitrary throwaway digest, discarded — never a real tx) against every minted release wallet on a timer (--wallet-health-poll-interval, default 10m) and caches the result. This exists because the real threshold-MPC cluster's key shares can silently desync on a long-lived wallet with no node restart and no config change — seen directly on 2026-06-11, when a ZOO release wallet went unsignable ~9h after keygen while a fresh wallet on the same cluster signed fine. A real payout would have stalled the exact same way; this poller catches it first. Companion series: bridge_release_wallet_sign_latency_ms{network="..."} (canary sign latency — a rising trend ahead of an outright failure is an early warning) and bridge_release_wallet_last_check_age_seconds{network="..."} (alert if this grows unbounded while bridge_wallet_health_poller_running is 1 — the loop is alive but stuck on a specific network). A network only appears once its release wallet has been minted and checked at least once — absence isn't a 0.

If a wallet goes unsignable: don't retry in place. Re-mint the release wallet (mpcd keygen is not idempotent, so there's no "repair" path) and re-fund it.

Alert rules ship in k8s/bridge-alerts.yaml (PrometheusRule) and the scrape target in k8s/bridge-servicemonitor.yaml (ServiceMonitor) — both plug into the existing o11y / luxfi-monitoring stack (dash.lux.network); no standalone Prometheus needed.

---

12. BTC release confirmation gate

A Bitcoin release is only marked completed once its tx is mined — mempool admission parks the swap in bridge_transfer_pending_confirmation and the broadcast driver polls mempool.space until confirmed. A release that sits unconfirmed past --btc-confirm-timeout (default 30m) is rebuilt at a bumped RBF feerate (+25%+1, compounding, floored via the swap's persisted last_fee_rate); a fee-below-relay-floor reject on submit takes the same rebuild path. Both draw from one ceiling (broadcast_rebuilds, default 5) — past it the swap moves to failed with the reason in last_error, and the deposit needs a sweep.

Watch bridge_btc_confirm_timeouts_total (each increment = one stuck release rebuilt) and bridge_broadcast_rebuilds_total alongside it; sustained growth means BTC network fees are outpacing the bump rate. --disable-btc-confirm-gate restores legacy complete-on-admission.