main still called the provider Casdoor in ten places. hanzo.id is served by
github.com/hanzoai/iam, which its own LICENSE states is a clean-room
implementation carrying no third-party licensed source. The Casdoor-derived
server was hanzoai/iam-v1, retired and no part of this stack.
Every fact the text carried is kept. IAM really does stamp iss claims that
openid-client rejects, which is why the NextAuth provider is type: "oauth" and
why setting `issuer` breaks it; the applications really do live in the
iam-init-data configmap. Only the vendor name changed.
iam.py claimed a "Casdoor-compatible" flow. It is a standard flow — RFC 6749 and
OIDC Core — which is what HIP-0111 requires; naming a vendor for a capability
that has an RFC is what the standard exists to stop.
infra/k8s/cloud-deployment.yaml is deleted rather than edited. Its env block
named casdoorEndpoint / casdoorOrganization / casdoorApplication, and nothing
reads them: hanzoai/cloud configures IAM through IAM_URL, IAM_MINT_CLIENT_ID and
CLOUD_IAM_ISSUER, and ships no conf/. Renaming them to IAM_* would have produced
config that looks live and still does nothing. The file is unreferenced by
kustomization.yaml, duplicates declared state that belongs in universe
(infra/k8s/cloud/), pins registry.digitalocean.com instead of ghcr.io/hanzoai,
and pins a floating :latest tag that must never reach a cluster.
Landed directly on main via an isolated worktree. chore/use-datastore-fork is
139/137 divergent from a history rewrite; merging it would re-import that
lineage, and only two commit subjects there are genuinely absent from main.
Co-authored-by: Hanzo Dev <[email protected]>
Settings() raised SettingsError constructing ALLOWED_ORIGINS. The field is
declared list[str] and documented "comma-separated in env", but
pydantic-settings decodes complex types from the environment as JSON before
any validator runs, so the plain `a,b,c` that every Deployment manifest, .env
and compose file writes was fatal — not a bad value, an unconstructable
Settings.
That made every route depending on it 500: require_admin builds an IAMClient,
which calls get_settings(). The lux.cloud fleet overview returned HTTP 500 the
moment the admin gate shipped, and the failure was in the gate, nowhere near
the collector it guarded.
NoDecode hands the raw string to one validator that splits it, for both list
settings. One variable, one format, decided in one place.
Co-authored-by: Hanzo Dev <[email protected]>
lux-operator is scaled to 0, so the LuxNetwork CR's image tag is only a
declaration and can be arbitrarily stale. Mainnet's CR says v1.34.0 — a tag
that does not exist in the registry — while all five nodes answer
info.getNodeVersion with luxd/1.36.2; testnet's says v1.32.12 against a
running 1.36.24.
Reporting the CR tag as `version` would put a fabricated number on the
dashboard. `version` is now the observed value and `image` stays as declared,
so a drift between intent and reality is visible instead of hidden.
Co-authored-by: Hanzo Dev <[email protected]>
luxd serves /v1 only — server/http/server.go pins baseURL="/v1" — so every
/ext call in this repo was a silent no-op. The fleet collector's _rpc()
returns None on non-200, so the lux.cloud dashboard rendered nodes_rpc_ok:0,
version:null and every node rpc_reachable:false while the fleet was in fact
running luxd/1.36.2 (mainnet), 1.36.24 (testnet), 1.36.25 (devnet).
Fixing the path alone would not have helped: the collector addressed each
validator by a hardcoded public LoadBalancer IP, and there are now zero
LoadBalancer Services in lux-mainnet/testnet/devnet. Those IPs time out. So
node addressing is no longer configured — it is derived from the cluster on
every collection, and the one way to reach a validator is the lux-k8s API
server's pod proxy, since luxd RPC is cluster-internal.
That collapses the old MODE A / MODE B split into a single path: read the CR
for the declared shape, the pods for phase/readiness/IP, then each pod's
JSON-RPC through the proxy. One credential authenticates both. Deleting the
endpoint ConfigMap and its 15 baked-in IPs removes the failure mode where
infrastructure moves and a dashboard quietly reports zeros.
Probes move to /v1/health/liveness, not /v1/health: a degraded-but-running
validator answers /v1/health with 503, so probing it would restart nodes that
are merely behind. Any fleet from charts/lux could never have gone Ready.
LuxNode.external_ip becomes .ip — with no public per-pod endpoint, the honest
value is the pod IP.
Avalanche endpoints keep /ext; that is their API, not ours.
Measured, with controls: pod-proxy POST /v1/info -> 200 on all 15 pods;
/ext/health -> 404; /zzz/health -> 404; nonexistent pod -> 404. A test asserts
every probe path is under /v1 and fails if /ext returns.
Co-authored-by: Hanzo Dev <[email protected]>
github.com/bootnode/bootnode is the canonical home. These two diverged from
766da81c with real work on BOTH sides — 21 commits there, 26 here — so this is
a merge, not a fork-kill. Deleting either side would have destroyed shipped
work.
From CANONICAL (bootnode/bootnode):
· Base/SQLite storage, Postgres coupling dropped (5650324)
· Cluster + Node models — models this side never defined
· native CI: hanzo.yml, zero .github/workflows (a698acc, aff0952)
· images to ghcr.io/bootnode/bootnode, its own org (9d0957c)
· luxd fleet dashboard, k8s validator detector
From HERE (hanzoai/bootnode):
· HIP-0111 IAM: SPA PKCE, JWKS token validation (ed27226),
JWT audience enforcement (49eb8cb) — security features canonical lacks
· runtime white-label brand catalog (422f8f1 + today's work)
· static-export build (852bb7c)
· today: toolchain pinned, static export via hanzoai/static, one API
resolver, honest error states
Seven conflicts, each resolved on evidence, not preference:
web/lib/brand.ts OURS — canonical still hardcodes
bootnode → iam.url "https://hanzo.id",
which is the "Sign in to Hanzo ID" leak
fixed today. It also carries a duplicate
`skillsOrg` field.
web/Dockerfile OURS — canonical runs `pnpm config set
dangerously-allow-all-builds true` AND
`--no-frozen-lockfile`: disables the
supply-chain guard and ignores the lockfile.
db/models.py THEIRS — has Cluster + Node and no Postgres dialect.
db/__init__.py THEIRS — matches those models.
.github/workflows/* DELETED— canonical already went native.
api/bootnode/main.py BOTH — took `__version__` from canonical (better
than the hardcoded "2.0.0" here) and kept
the conditional `docs` from b06f09f
(canonical re-adds "docs": "/docs", a URL
production disables, so it 404s).
This drops a386c9b, the Node/MPCWalletModel commit written earlier today. Its
problem did not exist on the canonical side: those models are already there,
and there is no MPC router to need MPCWalletModel. It was written against the
storage layer canonical replaced in June.
Verified on the merged tree, not assumed: `import bootnode.db` succeeds and
exports 14 models, all 27 API routers import, 0 postgresql dialect imports
remain, web tsc reports 0 errors and the production build exits 0 with
localhost:8000 in 0 chunks.
`bootnode/db/__init__.py` imported and re-exported three names that models.py
does not define, so `import bootnode.db` raised ImportError on main. Every
consumer died with it: the e2e job's init_db(), the nodes API, the MPC API, the
keys API, the seed. Prod is unaffected only because it runs image 1.0.1, built
before this landed.
It was invisible because test-api ran
pytest -v --tb=short || echo "No tests yet"
The `|| echo` makes the step exit 0 unconditionally, so CI reported GREEN while
every test module failed to import. That mask is removed; CI can fail honestly.
The consequence was not subtle: docker-release triggers on
`workflow_run: ["CI"] completed`, CI was red, so the image build was SKIPPED
every time — which is why the deployed web image sat at 0.0.1-web from June
while main moved on.
Cluster → renamed OrgCluster at some point; call sites updated.
The `"kind": "Cluster"` CRD strings are correct and untouched.
Node → never existed in ANY revision (checked back to 2026-01-26).
Defined here from the contract the code already declares.
MPCWalletModel → likewise. Distinct from SmartWallet, which is ERC-4337
account abstraction, not threshold signing.
Design, in the terms asked for:
DECOMPLECTED — Node stores DESIRED state only. Sync progress, peer count, block
height, cpu/memory, uptime and the backing pod are facts about the cluster, and
fleet_manager already reads them from the Cluster CR. A copy in this table would
be a second source of truth, stale the instant it is written. That split is why
the model is small; it is the spec half of the spec/status pair the CRD layer
already uses, not a new idea.
TYPED — status/node_type/sync_mode are closed Enums at both the Python and the
column level, not free strings. This class of bug was live: the seed wrote
"running" while NodeStatus offers pending/init/starting/bootstrapping/healthy/
unhealthy/terminated — provably DISJOINT sets, so the dashboard's
`filter(n => n.status === "running")` could never match a backend value and
"active nodes" was structurally always 0. One vocabulary now, defined once in
fleet_models and imported everywhere; seed and both frontend call sites moved
onto it.
SMALL INTERFACE — two properties (`is_serving`, `slug`) and a repr. Everything
else is a column read. No cleverness to undo later.
Flagged, not silently reproduced: MPCWalletModel.encrypted_shares holds key
material in the database. It is encrypted, but the standing rule is that secrets
live in KMS. That column is a migration target, documented in the model.
Verified: `import bootnode.db` succeeds, all three enums resolve, every seed
status value is a member of NodeStatus, web tsc 0 errors, web build exits 0.
The FastAPI app deliberately turns off docs/redoc/openapi in production:
docs_url="/docs" if not settings.is_production else None
but the root endpoint returned `"docs": "/docs"` unconditionally. So the
public API root promised a URL that 404s — api.bootno.de/ says
{"name":"Bootnode API","version":"2.0.0","docs":"/docs","status":"operational"}
while /docs, /redoc and /openapi.json all return {"detail":"Not Found"}.
Anyone reading the root to find the schema followed a dead link, and the
disabled-in-prod decision was correct, so the answer is to tell the truth
rather than re-enable them.
root() now mirrors app.docs_url and omits the key entirely when docs are off,
so the two cannot drift again — there is no second place to remember when the
setting flips. Verified both ways: with docs_url None the key is absent, with
"/docs" it is present.
The API resolved JWKS at {iam_url}/.well-known/jwks.json and exchanged
codes at {iam_url}/oauth/token — measured 401 on both; the canonical
/v1/iam/.well-known/jwks and /v1/iam/oauth/token answer.
brand.json pointed at iam.bootno.de, which does not resolve (525). The
brand's canonical issuer is id.bootno.de, already what web/lib/brand.ts
uses.
HanzoLoginButton hand-rolled a /oauth2/authorize URL that exists on no
IAM deployment. It was imported but never rendered — the login page has
used getIam().signinRedirect() (@hanzo/iam PKCE) since HIP-0111 — so it
is deleted rather than repointed. One way in.
Co-authored-by: Hanzo Dev <[email protected]>
Deny-by-default on the ONE cross-tenant infra route, enforced on both the
standalone fleet_app and the mounted api/fleets router:
unauthenticated -> 401 · invalid token -> 401 · non-admin -> 403 · admin -> 200
Adversarial test exercises the gate end-to-end over real HTTP; only JWKS
verification and the luxd RPC collector are stubbed.
Turn lux.cloud into a live graphical fleet dashboard for the managed luxd
fleet (devnet/testnet/mainnet, 5 validators each on do-sfo3-lux-k8s).
Backend (read-only collector, no mutations, no monolith dependency):
- core/chains/lux_fleet.py: reads the operator LuxNetwork CR (lux.cloud/v1)
for validators/version/plugin-set/phase, lists luxd-N pods + per-pod LB
Services, and queries luxd JSON-RPC per node (info.getNodeVersion/getNodeID/
peers/isBootstrapped, C-Chain eth_blockNumber, P-Chain platform.getHeight),
plus a native D-Chain DEX summary. Aggregates all 3 nets concurrently.
- fleet_app.py: minimal standalone FastAPI exposing GET /v1/fleets/lux/overview
(public, read-only). Reuses the bootnode api image; isolated from the
(currently import-broken) monolith.
- fleets/__init__.py: same overview endpoint also mounted on the main router.
- Uses a READ-ONLY lux-k8s kubeconfig from KMS; only public chain data leaves.
Frontend (Next.js 16, Lux white-label, auto by hostname):
- lib/fleet.ts: typed client + health/status derivations.
- components/lux/fleet-dashboard.tsx: graphical multi-net dashboard — totals,
world topology map, per-network cards (phase, version, plugin-set, peers,
C-Chain height, DEX state, per-chain C/P/X/D bootstrap), expandable per-node
tables (node ID, version, peers, heights). Auto-refresh 15s.
- app/fleet/page.tsx: public /fleet page (the lux.cloud fleet dashboard).
- components/lux/fleet-strip.tsx + page.tsx: live fleet status strip on the
landing hero.
- navbar + dashboard nav: Fleet links; register existing Lux fleet mgmt page.
- brand.ts: add skillsOrg (fixes pre-existing typecheck/build break).
Deploy: infra/k8s/lux-fleet.yaml — lux-fleet Deployment/Service + ExternalSecret
(lux-k8s RO kubeconfig from KMS) + ingress path /v1/fleets/lux/overview on
api.cloud.lux.network.
Verified: collector live against all 3 nets (real heights/peers/versions/DEX);
api + collector tests pass (8/8); web typecheck + production build green;
/fleet renders live data in browser (screenshot).
lux.cloud is the canonical Lux web3 console; its RPC & API-Keys section calls
the control plane (api.cloud.lux.network) from the browser. The production CORS
allowlist had cloud.lux.network but not lux.cloud, so key create/list/revoke
preflights 400'd. Add the canonical console origins to production_origins.
Co-authored-by: Hanzo Dev <[email protected]>
Bootnode could only see nodes it launched as Docker containers; real Lux/Zoo
validators run as K8s workloads behind operator CRDs and were invisible to
GET /v1/nodes. Add a detector that scans a cluster's kubeconfig and adopts
its running validators into the DB.
- core/k8s_nodes.py: K8sNodeDetector scans LuxNetwork (lux.network/v1alpha1
luxnetworks) + ZooChain (zoo.network/v1alpha1 zoochains) CRs across all
namespaces via CustomObjectsApi, enriches each with its luxd/zood pods for
live status, and upserts Cluster + Node rows. Adoption, not recreation:
existing rows (keyed on (cluster_id, pod_name)) are updated in place, so
re-scans are idempotent. New operator chains are one descriptor row.
- db/models.py: add Cluster + Node models (portable JSON/Uuid types, matching
the existing db_nodes.py / seed_nodes.py consumers that imported them).
- api/nodes: merge DB-adopted k8s nodes into list_nodes() alongside Docker
discovery; add POST /v1/nodes/detect and POST /v1/nodes/clusters/{id}/scan
(kubeconfig resolved via the shared fleets path). Reuses existing get_db.
- pyproject: add aiosqlite (config.database_url now defaults to
sqlite+aiosqlite; the driver was missing).
Detector adopt/idempotency + model mapping verified against in-memory SQLite
with faked LuxNetwork/ZooChain CRs and pods; full app OpenAPI builds with the
new routes; 112 existing tests pass (pre-existing test_lux_fleet /
test_chain_config failures unchanged).
bootnode stores its catalog in Base/SQLite — the canonical store, durable via
VFS→SeaweedFS — not Postgres. Make the models dialect-portable and the engine
sqlite-aware:
- models.py: postgres JSONB/UUID → portable sqlalchemy JSON/Uuid
- config.py: database_url is str (not PostgresDsn) so sqlite+aiosqlite:/// DSNs
validate; default = sqlite+aiosqlite:////data/bootnode/bootnode.db
- session.py: normalize sqlite:// → sqlite+aiosqlite://, and skip the sized
pool (pool_size/max_overflow are invalid on sqlite) + check_same_thread off
Postgres DSNs still accepted (legacy/multi-instance) but no longer required.
bootnode is its own GitHub org — its images belong in its own GHCR namespace,
per the never-mix-registries rule. This also removes cross-org friction: the
repo's own GITHUB_TOKEN pushes to ghcr.io/bootnode/* (same org), so once the
arcd App is on the bootnode org, build+push needs no cross-org token.
ghcr.io/luxfi/node:latest is amd64-only; under qemu on arm64 the out-of-process
C-Chain EVM plugin can't complete its go-plugin handshake. The v1.30.14-dchain
tag is multi-arch (amd64+arm64) — verified: arm64 build boots natively, C-Chain
eth_chainId=0x7a69 (31337), no qemu. lux/hanzo/zoo now have working C-Chains on
both local (arm64) and DOKS (amd64).
The api crashed on import after a half-finished model refactor, and the qdrant
healthcheck used a binary not in the image.
- compose.yml: qdrant ships no wget/curl, so its healthcheck always failed and
blocked the api (depends_on service_healthy). Dropped the broken healthcheck and
gate the api on qdrant service_started (qdrant itself serves fine on :6333).
- db/__init__.py: reconcile to the real models — drop removed Node + Cluster
(now CRDs) and MPCWalletModel; export the actual set (User, OrgCluster,
SmartWallet, Subscription, TeamMember, etc.).
- api/mpc + api/keys: MPCWalletModel -> SmartWallet (the renamed model).
API now boots healthy: GET /health 200 {"status":"healthy","version":"2.0.0"},
/docs serving, 42 node/chain/fleet endpoints live (fleets deploy/scale, presets,
pod health/logs). Full local stack up: postgres, redis, clickhouse, qdrant,
meilisearch, api.
Note (follow-up, not blocking boot): api/nodes/db_nodes.py + db/seed_nodes.py
still import removed Node/Cluster models — dead code to remove or port to the
fleet/CRD path; the web frontend build fails on a stale pnpm lockfile.
Co-authored-by: Hanzo Dev <[email protected]>
These two repos share root commit 8999f3c3 ("Initial commit") but
diverged long ago — hanzoai/bootnode evolved the recent stack
(api-go, brand package, etc.) while bootnode-org/main retained the
mainline 2026 history (175 commits we don't have).
Brand-leak fix: this work belongs in github.com/bootnode/bootnode,
not github.com/hanzoai/bootnode. Per the no-cross-brand-leak policy,
merge both histories together and push the result back to
bootnode/bootnode as the canonical home.
After this lands:
- bootnode/bootnode is the canonical repo
- hanzoai/bootnode gets archived with a redirect notice
- @bootnode-io/brand npm package's `repository.url` already points at
bootnode/bootnode (set in the earlier package.json scaffold)
# Conflicts:
# .github/workflows/ci.yml
# api/bootnode/api/__init__.py
# api/bootnode/api/billing/__init__.py
# api/bootnode/config.py
# api/bootnode/core/billing/__init__.py
# api/bootnode/db/models.py
# api/bootnode/main.py
# api/tests/test_billing.py
# web/app/dashboard/infrastructure/page.tsx
# web/app/dashboard/organization/billing/page.tsx
# web/app/dashboard/organization/page.tsx
# web/app/dashboard/page.tsx
# web/app/layout.tsx
# web/components/navbar.tsx
# web/components/unified-navigation.tsx
# web/lib/brand.ts
# web/public/logo/zoo-icon.svg
# web/public/logo/zoo-logo-white.svg
# web/public/logo/zoo-logo.svg
# web/tsconfig.tsbuildinfo
- Add Explorer product page with branded block explorer deployment, contract
verification, Etherscan-compatible API, and multi-chain support
- Add MPC product page with deep Lux MPC integration: CGGMP21 (ECDSA),
FROST (EdDSA), LSS resharing, 15+ chain support, threshold signing API
- Update landing page with 6-product grid (Node, Data, Wallets, Explorer,
MPC, Rollups) and 8 API cards including Explorer API and MPC Signing
- Update navbar and command menu with Explorer and MPC products
- Add Lux fleet management dashboard, observability, and services pages
- Add AI chat component and unified navigation with org switcher
- Add Kubernetes operator (Rust) for multi-chain node lifecycle management
- Add Lux Helm charts with genesis configs for devnet/testnet/mainnet
- Add docker-release.yml CI/CD pipeline for automated DOKS deployment
- Add monitoring, network policies, web deployments, and ingress updates
- Refactor API: fleet management, billing tracker, observability endpoints
- Update brand config, API client, and infrastructure manifests
The K8s DATABASE_URL env var uses postgresql:// which makes
create_async_engine try to load psycopg2 (sync driver). Add
_ensure_async_dsn() to convert postgresql:// and postgres:// to
postgresql+asyncpg:// at engine creation time. asyncpg is already
in dependencies.
capnp.AsyncIoStream.create_server() must be called from within an active
KJ event loop context. Wrap the entire serve lifecycle in a background
asyncio Task that runs under `async with capnp.kj_loop()`, fixing the
startup error: "The KJ event-loop is not running on this thread".
The server now signals readiness via `_running=True` after bind and
the caller polls for up to 1 s before raising if startup fails.
- Remove hardcoded Secret from bootnode-deployment.yaml (conflicted with ExternalSecret)
- Fix Postgres/Redis pods to read passwords from KMS-managed bootnode-secrets
- Expand ExternalSecret from 7 to 30 secrets with UPPERCASE_UNDERSCORE keys matching Pydantic
- Enable hanzo-kms SDK as optional dependency for direct KMS access outside K8s
- Improve inject_secrets() startup logging to indicate active secrets path
- Rewrite secrets.yaml as documentation-only with flow diagram and setup guide
- Fix compose.yml Bitcoin healthcheck to use env vars instead of hardcoded password
- Replace eval() with json.loads() in usage tracker (security fix)
- Webhook verification fails-closed in production when secret missing
- Add Commerce env vars to K8s deployment (URL, API key, webhook secret)
- Expose ZAP port 9999 in Dockerfile and K8s service
- Replace Stripe env vars with Square in compose.yml
- Update privacy/terms pages: Stripe→Square via Hanzo Commerce
- Add full billing API client module to web/lib/api.ts (17 endpoints)
- Fix hardcoded localhost URLs in dashboard/analytics/organization pages
- Add production webhook rejection test (57 tests passing)
P0: config.py defined hanzo_commerce_* but billing code accessed
settings.commerce_* — causes AttributeError at runtime. Renamed fields
with AliasChoices for backward compat with HANZO_COMMERCE_* env vars.
P1: deps.py silently swallowed JWT errors (bare pass) — now logs them.
Virtual API key used fake UUID for missing projects — now raises 404.
Updated secrets.yaml to replace stale Stripe references with Commerce.
Update K8s manifests (Ingress hosts, CRD API groups, CORS origins),
ZAP schemas, pyproject.toml URLs, SDK docs import paths, e2e test
emails, and K8s annotations to use the canonical bootno.de domain.
- OAuth callback accepts redirect_uri from frontend to derive correct
IAM client_id per network (all use lux-web3 shared app)
- Token verification accepts tokens from any known IAM app audience
- Frontend sends redirect_uri in callback POST body
- Login page uses proper /login/oauth/authorize URL with client_id
- Brand configs unified to use lux-web3 IAM app for all networks
- Removed dead code (duplicate get_current_user, unreachable methods)
- IAM init data updated with cloud redirect URIs for pars/zoo/hanzo
The networks module imported get_current_user from bootnode.api.deps
where it doesn't exist — it's in bootnode.core.iam. This broke the
entire API startup (uvicorn can't import bootnode.main:app).
New /v1/networks endpoint orchestrates deploying a complete
white-label blockchain cloud portal:
- K8s deployment + service for bootnode-web with brand env vars
- Ingress with TLS + CORS for cloud.{domain} and api.cloud.{domain}
- IAM integration for SSO
- Scale and teardown endpoints
CI deploy step now restarts all cloud-web deployments (lux, pars,
zoo, hanzo) when new images are pushed.
The CORSMiddleware was initialized with only bootnode.dev origin.
Added all cloud.*.network domains and ALLOWED_ORIGINS env var
for extending origins without code changes.
Updated all container image references across compose, K8s manifests,
cloud deploy scripts, and tests to use the ghcr.io/hanzoai/bootnode
registry format.
- Link Hanzo IAM users to Commerce customers via hanzo_id
- Auto-create Commerce customer on first billing access
- New unified billing endpoints:
- GET /billing/account - unified user with IAM + Commerce data
- GET /billing/account/subscriptions - all subscriptions
- GET /billing/account/invoices - all invoices
- GET /billing/account/payment-methods - payment methods
- POST /billing/account/sync - sync IAM to Commerce
- POST /billing/account/subscribe - create subscription
- Support for hanzo.id, zoo.id, lux.id, pars.id orgs
zeekayandHanzo Dev