mpc/docs/RUNBOOK.md

# MPC Operator Runbook (Outline)

Day-to-day operator reference for a `mpcd` consensus-mode cluster.

This document is intentionally an **outline**. Each section lists the
shape of the procedure and the commands an operator runs; protocol-level
detail (exact reshare sequence, slashing semantics, recovery state
machine) is owned by the maintainers and is marked `TODO(maintainer)`
where it has not been filled in yet. PRs welcome.

> **Scope.** Consensus-mode (`mpcd`) only. Legacy `lux-mpc` clusters are
> covered in `INSTALLATION.md` Appendix A and are not in scope here.

---

## 0. Cluster topology at a glance

- `n` nodes (StatefulSet pods / VMs), each running `mpcd start ...`
- Quorum: `ready_count >= threshold + 1` for signing; **all** peers must
  be ready for keygen of fresh wallets
- Default ports per node:
  - `9999` — P2P ZAP transport (peer ↔ peer)
  - `9800` — internal MPC API (bearer-auth, K8s-internal only)
  - `8081` — dashboard API (JWT-auth, may be ingressed)
  - `9653` — KMS-facing ZAP (consumed by `luxfi/kms`)
  - `7301` — embedded threshold dispatcher (loopback by default)
- Persistent state: `--data` directory (key shares in ZapDB, dashboard
  in SQLite, backups under `backups/`)

---

## 1. Bring-up checklist

Run through this when standing up a new cluster or after a planned full
restart.

- [ ] All node identities (`<data>/keys/<node-id>_identity.json`) backed
      up off-host before first start.
- [ ] HSM password provider reachable (`aws kms decrypt` or equivalent
      smoke test from the host network).
- [ ] `MPC_INTERNAL_API_KEY` set on every node and identical across the
      cluster.
- [ ] `JWT_SECRET` set for the dashboard API.
- [ ] `mpcd start` succeeds on each node — logs show
      `[READY] Node is ready (consensus mode)`.
- [ ] `/healthz` returns `signing_quorum: true` on every node.
- [ ] Smoke test passes (see §8).

---

## 2. Health endpoints and probes

`mpcd` exposes two health surfaces. Both are unauthenticated so that
Kubernetes probes work without secret injection.

- `GET /healthz` (port 9800) — node-level. Returns:
  - `200 status=healthy` — signing quorum holds and every peer is ready
  - `200 status=healthy-reduced` — signing quorum holds, ≥1 non-critical
    peer unreachable
  - `503 status=degraded` — signing quorum lost
- `GET /health` (legacy alias of `/healthz`).

Recommended Kubernetes probes:

- **Liveness:** `/healthz`, period `30s`, failure threshold `5`. Restart
  only on prolonged failure — transient quorum gaps are recoverable.
- **Readiness:** `/healthz`, period `10s`, failure threshold `2`. A 503
  removes the pod from the dashboard service endpoints.

`TODO(maintainer)`: confirm whether a dedicated readiness path is
preferred over reusing `/healthz` for very large (n>7) clusters.

---

## 3. Key rotation (reshare)

The reshare flow rotates per-share material **without** changing the
wallet's public key. Use it on a regular cadence and on personnel
change.

- [ ] Stage the new participant set and threshold.
- [ ] Confirm the cluster is at full quorum (`/healthz` healthy on every
      node) before starting — reshare requires all current participants.
- [ ] Trigger reshare via the dashboard API or by publishing to the
      `mpc:reshare` topic (`ConsensusMPCBackend.TriggerReshare` in
      `cmd/mpcd/main.go`).
- [ ] Verify the new key info on every node (`GET /keys` on port 9800).
- [ ] Retire old share material from the previous participant set once
      every node has acknowledged the new shares.

`TODO(maintainer)`: document the exact wire shape of the reshare event
(org_id, wallet_id, new_threshold, new_participants[]) and the expected
result topic.

---

## 4. Backup and restore

`mpcd` runs a periodic ZapDB backup driven by `backup_period_seconds`
(default 300 s). Backups land in `<data>/backups/` and, when the
`MPC_BACKUP_S3_*` env vars are set, are uploaded to S3 by
`pkg/backup.Manager`.

### Backup

- [ ] Verify the periodic backup is running (`grep "Periodic ZapDB
      backup completed" mpcd.log` on each node).
- [ ] On demand: `curl -X POST -H "Authorization: Bearer
      ${MPC_INTERNAL_API_KEY}" http://<node>:9800/backup`.

### Restore (single node)

- [ ] Stop `mpcd` on the affected node.
- [ ] Restore the latest `<data>/backups/<node-id>.zapdb.bak` file into
      `<data>/db/<node-id>/`.
- [ ] Restore `<data>/keys/<node-id>_identity.json` from off-host backup
      (the identity is **not** in the ZapDB backup).
- [ ] Restart `mpcd`. The node rejoins, repopulates the peer registry,
      and `/healthz` flips back to `healthy`.

`TODO(maintainer)`: describe the restore-from-S3 flow and the
verification step that a restored share is byte-identical to peers'
view of the same wallet.

---

## 5. Add / remove a node without resharing

This procedure changes cluster size **without** rotating shares. It is
appropriate for scaling capacity, not for rotating key material — for
that, use §3 (reshare).

### Add

- [ ] Provision the new pod / VM with a clean `--data` volume.
- [ ] Generate the new node's identity (first start creates it under
      `<data>/keys/`).
- [ ] Update the `--peer` flags on every existing node to include the
      new address.
- [ ] Rolling-restart the existing nodes. Quorum is preserved as long as
      `ready_count >= threshold + 1` throughout the rollout.
- [ ] The new node will not hold shares for existing wallets until a
      reshare is triggered — see §3.

### Remove

- [ ] Confirm the cluster can tolerate `n - 1` (i.e. `n - 1 >= threshold
      + 1`).
- [ ] Drain and stop the node.
- [ ] Remove the node's address from `--peer` on remaining nodes;
      rolling-restart.
- [ ] `TODO(maintainer)`: confirm whether the removed node's share
      material should be cryptographically retired (re-shared away) or
      whether physical destruction is sufficient given the share is
      single-party-useless.

---

## 6. Recovery from share loss

If a single node loses its `--data` volume but the cluster still has
`t-of-n` quorum.

- [ ] Confirm `n - 1 >= threshold` — the cluster can still sign.
- [ ] `TODO(maintainer)`: document whether `pkg/mpc/recovery.go`
      exposes an operator-facing restore-from-quorum flow, or whether
      the only path is reshare into a fresh share for the affected
      node.
- [ ] After recovery, run the smoke test (§8) before returning the node
      to production traffic.

If the cluster has lost **more than `n - t`** nodes (below signing
quorum), restore from off-host backups (§4) is the only option.

---

## 7. Slashing / misbehaviour response

`TODO(maintainer)`: the cluster currently treats peer misbehaviour as a
liveness signal (`ArePeersReady` flips false, `/healthz` may downgrade).
Operator response should cover:

- [ ] Identifying the offending peer from `mpcd` logs (signature
      verification failure, repeated wire-protocol violations).
- [ ] Quarantining the peer (remove from `--peer` set, rolling-restart).
- [ ] Forensic capture of the peer's `<data>/` and recent log window.
- [ ] Re-keying via reshare (§3) once root cause is established.

Document the threshold above which misbehaviour requires share rotation
vs. simple peer removal.

---

## 8. Deployment-validation smoke test

Run after every cluster start, every K8s rolling update, and as a
post-deploy gate in CI.

- [ ] `for n in node0 node1 node2; do curl -fs http://${n}:9800/healthz
      | jq -e '.signing_quorum == true'; done`
- [ ] Keygen a throwaway wallet via the dashboard API or `/keygen` on
      port 9800.
- [ ] Sign a fixed nonce against that wallet.
- [ ] Verify the signature off-cluster (e.g. `secp256k1`/`ed25519`
      verify with the returned public key).
- [ ] Tear down the throwaway wallet.

`TODO(maintainer)`: a packaged `scripts/smoke-test.sh` that wraps the
above into a single exit-coded command would be a useful follow-up;
issue #1 proposed it explicitly.

---

## 9. Log locations and verbosity

| Location | Contents |
|----------|----------|
| stderr (default) | Structured logs from `pkg/logger`. Capture via the systemd journal or the K8s pod log stream. |
| `<data>/backups/` | Periodic ZapDB backups. Not log data — but useful in incident timelines. |
| `<data>/dashboard.db` | Dashboard SQLite (audit/admin trail). |

Increase verbosity with `--log-level debug` or `--debug`. **Do not** run
production with debug logging — wire messages and identities are not
logged, but timing data leaks downstream operator information.

Key audit log lines to grep for:

- `Audit: keygen triggered` — every `/keygen` invocation
- `Audit: backup triggered` — every manual backup
- `[READY] Node is ready` — successful start
- `Periodic ZapDB backup completed successfully` — backup heartbeat
- `Shutdown signal received` — graceful stop