bridge/docs/operator-runbook.md

# cmd/bridge — Operator Runbook

Operating the unified Lux Bridge binary (`cmd/bridge`). One image,
one port, embedded SPA + native `/v1/bridge/*` API + background drivers
(deposit watcher, MPC signing, broadcast).

This binary replaces the legacy `bridge-server` (Express + Prisma) and
`bridge-ui` (separate Vite SPA) deployment split.

---

## 1. Overview

| Aspect          | Value                                              |
|-----------------|----------------------------------------------------|
| Binary          | `cmd/bridge`                                       |
| Image           | `ghcr.io/luxfi/bridge:<tag>`                       |
| Listen port     | `:8080` (HTTP)                                     |
| Operator port   | `:8081` (HTTP, not published by the Ingress)       |
| Health endpoint | `GET /health` (200 JSON)                           |
| Config file     | `/etc/bridge/networks.yaml`                        |
| Data directory  | `/var/lib/lux-bridge` (zapdb / Lux Badger v4)      |
| Logger          | `luxfi/log` (JSON, one line per event)             |
| HTTP framework  | `hanzoai/zip` on Fiber v3 / fasthttp                |
| Swap store      | `luxfi/zapdb` (embedded KV, exclusive dir lock)    |

Routes:

```
/                          embedded SPA + SPA-routing fallback
/envs.js                   runtime config (window.ENV)
/icon.svg, /logo.svg       per-host brand assets (disk override possible)
/health                    service health + driver stats
/v1/bridge/networks        supported chains (from networks.yaml)
/v1/bridge/tokens          tokens per chain
/v1/bridge/quote           price quote
/v1/bridge/limits          per-token min/max swap caps
/v1/bridge/swaps           create swap
/v1/bridge/swaps/:id       fetch swap
```

Operator routes, on `BRIDGE_ADMIN_ADDR` and only there. They identify no
caller, and the Ingress publishes the public port alone:

```
/metrics                   Prometheus exposition
/v1/bridge/swaps           list swaps, with signatures and signed dest txs
/v1/bridge/check-deposit   poll a source-chain RPC for an address balance
/admin/swaps/:id/reset          re-sign a swap      (BRIDGE_ADMIN_ROUTES=true)
/admin/swaps/:id/inject-raw-tx  push an operator's raw tx (same)
```

Background drivers (all on by default, all toggleable):

| Driver             | Reads from              | Writes to                  | Default cadence |
|--------------------|-------------------------|----------------------------|-----------------|
| DepositWatcher     | source-chain RPC        | `swap.State` advancement   | 15s             |
| SigningDriver      | MPC daemon              | `swap.DestRawTx`           | 5s              |
| BroadcastDriver    | dest-chain RPC          | `swap.State = completed`   | 5s              |

---

## 2. Build & ship

### Local build (smoke test)

```bash
go build -o /tmp/lux-bridge ./cmd/bridge
/tmp/lux-bridge --config cmd/bridge/networks.example.yaml --addr :8080
curl -sS localhost:8080/health | jq .
```

### Docker image

The canonical Dockerfile is `cmd/bridge/Dockerfile`. Three stages:

1. **ui-build** — `node:20-alpine`, `pnpm install` + `pnpm -C app/bridge build`
2. **go-build** — `golang:1.26.4-alpine`, compiles `cmd/bridge` with embedded SPA
3. **runtime** — `gcr.io/distroless/static-debian12:nonroot`, ~25 MB

Build from the repo root (build context must include the pnpm workspace):

```bash
docker build \
  -f cmd/bridge/Dockerfile \
  --build-arg VERSION=$(git rev-parse --short HEAD) \
  -t ghcr.io/luxfi/bridge:$(git rev-parse --short HEAD) \
  -t ghcr.io/luxfi/bridge:latest \
  .
```

Push:

```bash
docker push ghcr.io/luxfi/bridge:$(git rev-parse --short HEAD)
docker push ghcr.io/luxfi/bridge:latest
```

---

## 3. Deploy to k8s

Manifest: `k8s/bridge-deployment.yaml`. Apply after the MPC StatefulSet
from `k8s/mpc-deployment.yaml` is healthy (the bridge talks to
`mpc-api-svc.lux-bridge.svc:8081`).

```bash
kubectl apply -f k8s/bridge-deployment.yaml
kubectl -n lux-bridge rollout status deployment/lux-bridge
```

The manifest creates: `ConfigMap/lux-bridge-config`, `Deployment/lux-bridge`,
`Service/lux-bridge`, `Ingress/lux-bridge`, `HorizontalPodAutoscaler/lux-bridge`,
`PodDisruptionBudget/lux-bridge`.

### Required pre-existing resources

| Resource                       | Created by                            |
|--------------------------------|---------------------------------------|
| `Namespace/lux-bridge`         | `k8s/mpc-deployment.yaml`             |
| `Secret/bridge-secrets`        | KMS sync (see mpc-deployment header)  |
| `Service/mpc-api-svc`          | `k8s/mpc-deployment.yaml`             |
| TLS cluster issuer             | cert-manager install                  |

### Rollout

```bash
# Pin a specific tag (do not run on :latest in prod)
kubectl -n lux-bridge set image deployment/lux-bridge \
  bridge=ghcr.io/luxfi/bridge:<git-sha>

# Watch the rollout
kubectl -n lux-bridge rollout status deployment/lux-bridge --watch
```

### Rollback

```bash
kubectl -n lux-bridge rollout undo deployment/lux-bridge
kubectl -n lux-bridge rollout history deployment/lux-bridge
```

The deployment uses `RollingUpdate` with `maxUnavailable: 0` — a bad image
will not take down traffic; old pods stay up until new pods become Ready.

---

## 4. Configuration

All flags accept either a CLI argument **or** the matching env var.
Env vars are the standard way to configure inside the container.

### Required-ish

| Env / Flag                     | Default                          | Notes |
|--------------------------------|----------------------------------|-------|
| `BRIDGE_CONFIG` / `--config`   | `/etc/bridge/networks.yaml`      | Loaded from ConfigMap |
| `BRIDGE_ADDR` / `--addr`       | `:8080`                          | The listener the Ingress publishes. |
| `BRIDGE_ADMIN_ADDR` / `--admin-addr` | `""`                       | The operator listener (`:8081` in k8s). Empty serves none of the operator routes — including `/metrics`. Give it a port the Ingress does not route. |
| `BRIDGE_DATA_DIR` / `--data-dir` | `""`                          | Empty = in-memory swap store (lossy on restart). Set to a writable path on a PV for durability — e.g. `/var/lib/lux-bridge`. |
| `BRIDGE_PROFILE` / `--profile` | `classical-compat`               | `strict-pq` for internal Lux↔Lux only |

### Backend wiring

| Env / Flag                            | Default | Behavior when empty |
|---------------------------------------|---------|---------------------|
| `BRIDGE_BACKEND_URL` / `--backend`    | `""`    | Reverse proxy to legacy Node backend disabled — all `/v1/bridge/*` use native handlers |
| `BRIDGE_BCHAIN_URL` / `--bchain-url`  | `""`    | BridgeVM native handlers disabled — swap/quote stays on the legacy proxy if `--backend` is also set |
| `BRIDGE_MPC_URL` / `--mpc-url`        | `""`    | MPC keygen disabled — swaps with `use_deposit_address=true` return 503 |
| `BRIDGE_MPC_TOKEN` / `--mpc-token`    | `""`    | Required for the live `mpcd` daemon — every endpoint except `/health` is bearer-protected |
| `BRIDGE_MPC_IDENTITY_FILE` / `--mpc-identity-file` | `""` | Convenience: derives the bearer token via SHA-256(seed‖"mpc-internal-api"). Use in dev; prod sets `BRIDGE_MPC_TOKEN` explicitly |
| `BRIDGE_MPC_ORG_ID` / `--mpc-org-id`  | `bridge`| Multi-tenant key the MPC daemon multiplexes by |

### Source RPC overrides

`BRIDGE_SOURCE_RPC_OVERRIDES` / `--source-rpc-overrides` overrides
specific source-chain RPCs at runtime when the package defaults
(`rpc.sepolia.org` etc.) are stale or rate-limited. Format:

```
NETWORK1=URL1,NETWORK2=URL2
```

Used by *both* the deposit watcher (source) and the broadcast driver
(destination) — many networks are both a source and a destination.

Example:

```
BRIDGE_SOURCE_RPC_OVERRIDES="ETHEREUM_SEPOLIA=https://ethereum-sepolia-rpc.publicnode.com,BITCOIN_TESTNET=https://mempool.space/testnet/api"
```

### Driver toggles + cadence

| Env / Flag                          | Default | Effect when set |
|-------------------------------------|---------|-----------------|
| `--disable-deposit-check`           | off     | `/v1/bridge/check-deposit` returns 404 |
| `--disable-deposit-watcher`         | off     | Swaps never advance from `user_deposit_pending` |
| `--disable-signing-driver`          | off     | Swaps stall in `bridge_transfer_pending` |
| `--disable-broadcast-driver`        | off     | Signed txs never broadcast |
| `--deposit-watcher-interval`        | 15s     | Polling cadence (source-chain RPC load) |
| `--signing-interval`                | 5s      | MPC signing polling cadence |
| `--broadcast-interval`              | 5s      | Broadcast polling cadence |
| `--broadcast-timeout`               | 15s     | Per-RPC broadcast call timeout |
| `--deposit-check-timeout`           | 10s     | Per-RPC source-poll timeout |
| `--bchain-timeout`                  | 10s     | Per-RPC b-chain call timeout |
| `--mpc-timeout`                     | 120s    | Per-MPC call timeout (matches mpc-wallet.ts) |

### Static assets

| Env / Flag                          | Default | Effect |
|-------------------------------------|---------|--------|
| `BRIDGE_STATIC_DIR` / `--static`    | `""`    | Override the embedded SPA from disk (dev only) |

---

## 5. Updating networks.yaml

The supported chains, tokens, limits, and brand are all driven from
`networks.yaml` — mounted from the `lux-bridge-config` ConfigMap.

```bash
# Edit the ConfigMap in place
kubectl -n lux-bridge edit configmap lux-bridge-config

# Restart pods to pick up the change (the binary reads the file once at boot)
kubectl -n lux-bridge rollout restart deployment/lux-bridge
```

Adding a network requires no code change and no DB migration. The
in-binary token registry (`internal/tokens`) covers detection +
calldata for known assets; `networks.yaml` controls what the SPA
and `/v1/bridge/networks` advertise.

---

## 6. Health, logs, metrics

### Health

```bash
kubectl -n lux-bridge port-forward svc/lux-bridge 8080:80 &
curl -sS localhost:8080/health | jq .
```

Sample response:

```json
{
  "status": "ok",
  "version": "<git-sha>",
  "backend_proxy": false,
  "bchain_rpc": true,
  "mpc_keygen": true,
  "deposit_check": true,
  "deposit_watcher": true,
  "signing_driver": true,
  "broadcast_driver": true,
  "profile": "BRIDGE_CLASSICAL_COMPAT_UNSAFE",
  "post_quantum_end_to_end": false,
  "watcher_stats": { "ticks": 12, "checks": 0, "advances": 0, "errors": 0 },
  "signing_stats": { "ticks": 12, "signs": 0, "errors": 0 },
  "broadcast_stats": { "ticks": 12, "attempts": 0, "successes": 0,
                       "failures": 0, "skipped_no_raw_tx": 0, "list_errors": 0 }
}
```

Use the driver stats blocks to spot misbehavior:

- `watcher.errors` rising → upstream source-chain RPC issues
- `broadcast.failures` rising → destination chain rejecting our txs
- `signing.errors` rising → MPC daemon misconfigured or down

### Logs

JSON one-line-per-event via `luxfi/log`. Stream:

```bash
kubectl -n lux-bridge logs -f deployment/lux-bridge | jq -c
```

Filter for errors:

```bash
kubectl -n lux-bridge logs deployment/lux-bridge --tail=1000 \
  | jq -c 'select(.level=="error")'
```

### Metrics

Prometheus exposition at `/metrics` on the operator port. The series name
the MPC release wallets and report which of them cannot sign, so the scrape
target is the `admin` port, never the published one
(`k8s/bridge-servicemonitor.yaml`). Scraping by pod role instead:

```yaml
- job_name: lux-bridge
  kubernetes_sd_configs:
    - role: pod
      namespaces: {names: [lux-bridge]}
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      regex: lux-bridge
      action: keep
    - source_labels: [__meta_kubernetes_pod_container_port_name]
      regex: admin
      action: keep
```

Reading it by hand:

```bash
kubectl -n lux-bridge port-forward svc/lux-bridge 8081:8081 &
curl -sS localhost:8081/metrics
```

---

## 7. Common operations

### Pause the pipeline (incident response)

Disable all background drivers without taking the API down:

```bash
kubectl -n lux-bridge set env deployment/lux-bridge \
  BRIDGE_DISABLE_DEPOSIT_WATCHER=true \
  BRIDGE_DISABLE_SIGNING_DRIVER=true \
  BRIDGE_DISABLE_BROADCAST_DRIVER=true
```

> Note: env-var names mirror the CLI flags. To reverse, drop the vars
> with `kubectl set env … BRIDGE_DISABLE_…-`.

### Drain before maintenance

```bash
kubectl -n lux-bridge scale deployment/lux-bridge --replicas=0
# do work
kubectl -n lux-bridge scale deployment/lux-bridge --replicas=2
```

The PodDisruptionBudget keeps at least 1 pod up during cluster drains.

### Force a config reload

```bash
kubectl -n lux-bridge rollout restart deployment/lux-bridge
```

### Override an upstream RPC at runtime

```bash
kubectl -n lux-bridge set env deployment/lux-bridge \
  BRIDGE_SOURCE_RPC_OVERRIDES="ETHEREUM_SEPOLIA=https://ethereum-sepolia-rpc.publicnode.com"
```

### Add a new token mapping

The binary's in-memory `tokens.DefaultRegistry()` covers the common
bridged assets. To add a new ERC-20 mapping that the deposit watcher +
tx assembler should recognize, edit `internal/tokens/tokens.go` and
ship a new image — there is no runtime token-registration endpoint yet.

`networks.yaml` Token entries control SPA listings and per-token caps
only; they do **not** wire detection or calldata. Detection +
calldata live in `internal/tokens`.

---

## 8. Troubleshooting

### Pod won't start: `bridge profile invalid`

`BRIDGE_PROFILE` must be `strict-pq` or `classical-compat`. Anything
else exits 1 (this is intentional — typo = wrong security posture).

### `mpc_keygen: false` in `/health`

`BRIDGE_MPC_URL` is unset or unreachable. Check:

```bash
kubectl -n lux-bridge get svc mpc-api-svc
kubectl -n lux-bridge get statefulset mpc-node
```

Without MPC keygen, swap creates with `use_deposit_address=true`
return 503.

### `signing_driver: false` despite MPC enabled

`mchainClient` is nil → `BRIDGE_MPC_URL` was empty at boot. Set it,
roll the deployment. If MPC is reachable but signing still errors,
check the bearer token (`BRIDGE_MPC_TOKEN`) and the org id
(`BRIDGE_MPC_ORG_ID`).

### Source-chain checks failing

Default RPCs (`rpc.sepolia.org`, etc.) get rate-limited. Override with
`BRIDGE_SOURCE_RPC_OVERRIDES`. Track `watcher_stats.errors` in
`/health` to confirm the fix.

### Watcher / signing / broadcast all running but no progress

Check the swap is actually in the right state. Use the swap CRUD:

```bash
curl -sS localhost:8080/v1/bridge/swaps/<id> | jq .state
```

State machine:

```
user_deposit_pending
   → bridge_transfer_pending           (watcher: deposit confirmed)
   → bridge_transfer_pending_signing   (signing: MPC asked)
   → bridge_transfer_pending_broadcast (signing: tx signed)
   → completed                          (broadcast: pushed onchain)
```

---

## 9. Persistence + backups

The swap store (`internal SwapStore` interface) has two implementations:

- **`InMemoryStore`** — used when `BRIDGE_DATA_DIR` is empty. Concurrency-safe,
  zero ops cost, but every restart drops in-flight swap state. Dev only.
- **`ZapStore`** — used when `BRIDGE_DATA_DIR` is set. Backed by `luxfi/zapdb`
  (Lux-flavored Badger v4). LSM-tree KV with WAL durability, atomic
  `db.Update` transactions, MVCC concurrency control. Takes an exclusive
  directory lock — only one bridge pod can hold the data dir at a time.

### Single-writer constraint

Because zapdb takes an exclusive dir lock, the k8s Deployment is pinned to:

```yaml
replicas: 1
strategy: {type: Recreate}
```

Two pods can't share `/var/lib/lux-bridge`. The Recreate strategy ensures
the old pod fully terminates (and releases the lock) before the new one
mounts the PVC. This costs zero-downtime rollouts — expect ~10s of API
gap during a deploy. The deposit watcher's 15s poll cadence absorbs this
without losing real progress.

Scaling beyond one pod requires migrating swap state to a shared external
store (Postgres, hanzoai/base SQLite served behind an API, etc.). Not on
the roadmap yet — single-replica covers the bridge's expected load.

### Disk layout

Inside `/var/lib/lux-bridge`:

```
000000.vlog        value log (immutable)
000001.vlog        ...
000002.vlog        currently being written
000003.sst         level-0 sorted string table
MANIFEST           the LSM tree's index
KEYREGISTRY        encryption-at-rest key registry (we don't use)
LOCK               the exclusive directory lock
```

Typical size: hundreds of bytes per swap. 1 GB holds millions of swaps.
The 20Gi PVC in the manifest is generous head-room.

### Backups

zapdb supports a streaming backup primitive. For first-deploy ops, a
filesystem snapshot of the PV (cloud snapshot, Velero, etc.) while the
pod is paused is sufficient.

Pause-snapshot-resume:

```bash
kubectl -n lux-bridge scale deployment/lux-bridge --replicas=0
# wait for pod termination, then snapshot the PV however your cloud does it
kubectl -n lux-bridge scale deployment/lux-bridge --replicas=1
```

In-flight signing/broadcast that was mid-RPC will rerun on restart —
the state machine is idempotent (the watcher re-detects the same
deposit; signing checks for an existing signature before re-asking
MPC; broadcast checks for the dest tx hash before re-sending).

### Drift recovery

If the LOCK file is stale (e.g. a node crash left the file behind),
the new pod's `NewZapStore()` will fail with `Cannot acquire dir
lock`. Inspect the data dir, confirm no other process holds it, then:

```bash
# inside the pod — preserves all data, just drops the lock file
rm /var/lib/lux-bridge/LOCK
```

Restart the pod after.

---

## 10. Known gaps

These are tracked separately and **not** operationally fixable:

- Non-EVM broadcasters (BTC, SOL, TON, XRP, DOT) return
  `ErrFamilyNotImplemented`. Add per-chain RPC clients to
  `internal/broadcast` to extend coverage.
- EIP-1559 gas pricing not implemented in `internal/txassembler`. EVM
  txs assemble with legacy (gasPrice) pricing. Most destination
  chains still accept legacy txs.
- Substrate SS58 address encoding not implemented.
- LP-333 BridgeVM client methods are stubbed — `--bchain-url` controls
  whether `bchain.Client` is constructed, but the handlers it backs
  are read-only.
- Single-writer swap store. Horizontal scale-out requires migrating to
  a shared external store.

---

## 11. Release-wallet signability monitoring

**Alert on `bridge_release_wallet_signable{network="..."} == 0`.** A
background poller (`WalletHealthPoller`, `cmd/bridge/wallet_health_poller.go`)
runs a harmless canary sign (arbitrary throwaway digest, discarded —
never a real tx) against every minted release wallet on a timer
(`--wallet-health-poll-interval`, default 10m) and caches the result.
This exists because the real threshold-MPC cluster's key shares can
silently desync on a long-lived wallet with **no node restart and no
config change** — seen directly on 2026-06-11, when a ZOO release
wallet went unsignable ~9h after keygen while a fresh wallet on the
same cluster signed fine. A real payout would have stalled the exact
same way; this poller catches it first. Companion series:
`bridge_release_wallet_sign_latency_ms{network="..."}` (canary sign
latency — a rising trend ahead of an outright failure is an early
warning) and `bridge_release_wallet_last_check_age_seconds{network="..."}`
(alert if this grows unbounded while `bridge_wallet_health_poller_running`
is 1 — the loop is alive but stuck on a specific network). A network
only appears once its release wallet has been minted and checked at
least once — absence isn't a 0.

If a wallet goes unsignable: don't retry in place. Re-mint the release
wallet (mpcd keygen is not idempotent, so there's no "repair" path)
and re-fund it.

Alert rules ship in `k8s/bridge-alerts.yaml` (PrometheusRule) and the
scrape target in `k8s/bridge-servicemonitor.yaml` (ServiceMonitor) —
both plug into the existing o11y / luxfi-monitoring stack
(dash.lux.network); no standalone Prometheus needed.

---

## 12. BTC release confirmation gate

A Bitcoin release is only marked `completed` once its tx is **mined** —
mempool admission parks the swap in `bridge_transfer_pending_confirmation`
and the broadcast driver polls mempool.space until confirmed. A release
that sits unconfirmed past `--btc-confirm-timeout` (default 30m) is
rebuilt at a bumped RBF feerate (+25%+1, compounding, floored via the
swap's persisted `last_fee_rate`); a fee-below-relay-floor reject on
submit takes the same rebuild path. Both draw from one ceiling
(`broadcast_rebuilds`, default 5) — past it the swap moves to `failed`
with the reason in `last_error`, and the deposit needs a sweep.

Watch `bridge_btc_confirm_timeouts_total` (each increment = one stuck
release rebuilt) and `bridge_broadcast_rebuilds_total` alongside it;
sustained growth means BTC network fees are outpacing the bump rate.
`--disable-btc-confirm-gate` restores legacy complete-on-admission.