The previous commit stopped the pass WRITING a row per swap. It still READ
every swap row in the table, every 30 seconds, into a map.
That is the same shape of mistake one layer down. A live chain's swap table
grows without limit — the Lux C-Chain's already spans blocks 53..1,096,496 in
a 732 MB store — so the cost of a pass grows with chain age forever, and the
allocation churn lands on a process sharing 4Gi with six other chains.
storage.SwapsRaw is replaced by RecentSwapsRaw(limit), which uses the indexed
timestamp column to take the newest `limit` rows in SQL rather than filtering
in Go. The pass takes the newest 50,000.
Where that bound binds, the volume figure is the volume of those 50,000 trades
and the log line says so outright. A partial sum silently presented as a total
is worse than a smaller number that is honest about its window.
Co-authored-by: Hanzo Dev <[email protected]>
Deployed v1.2.10 to a six-chain explorer and watched one chain produce
aggregates while two others produced nothing at all — no result line, no error
line. Two distinct defects, both found by that silence.
1. The pass wrote O(swaps) rows per interval.
valueSwaps rewrote every swap's own amountUSD, one serialized INSERT OR
REPLACE each, every 30s. A chain with a real trade history never finished a
pass, so it never wrote its pool/factory aggregates either — while a quiet
chain beside it looked perfectly healthy. Volume is a per-pool rollup and
belongs on the pool: valueSwaps now READS the swap rows and writes none.
(Per-swap amountUSD stays "0". It is a separate gap, and the honest fix
values a swap at the price of its OWN block, not at today's.)
2. A V3 Swap emits int256, and the handler read it unsigned.
One leg of every V3 swap is negative — the token leaving the pool. Reading
it unsigned persisted ~2^256 in place of a small negative number. Nothing
had ever added those amounts up, so the corruption was invisible until the
volume rollup landed and reported $4.07e72 of trading on Zoo.
handleSwapV3 now uses decodeInt256, the same decoder the V4 handler already
used for the same reason. Rows the old build already wrote are healed on
read rather than by a re-index: a stored leg at or above 2^255 IS the
two's-complement encoding of a negative int256, and the V2 handler's
difference of two uint112 reserves can never reach that magnitude, so the
boundary is unambiguous.
Both were invisible for the same reason, so the third change is observability:
- Config.Label names the indexer ("cchain/amm") and every indexer log line
now carries it. One process runs seven of these; a line that does not say
whose it is cannot be acted on.
- Every revalue exit logs, including "read no balances", and the result line
reports valued/total pools and elapsed time. An indexer that reports
nothing was indistinguishable from one that was never scheduled.
- Each pass runs under a 60s budget, so a pass that cannot finish is
cancelled and says so instead of wedging its chain forever.
Tests pin all three: the wrapped-negative leg valuing identically to the signed
one, handleSwapV3 persisting "-2500", and revalue leaving swap rows untouched.
Co-authored-by: Hanzo Dev <[email protected]>
Every USD field the exchange reads was an empty string. `pools`, `tokens` and
`factories` all resolved — with real addresses, real symbols, real fee tiers —
and then reported totalValueLockedUSD:"", volumeUSD:"", token0Price:"". The
landing page's TVL / 24h-volume / active-pool header had nothing to render, so
it rendered nothing.
Nothing was ever writing those fields. processLog's Sync/Mint/Burn arm was an
explicit no-op ("storage for these types not yet wired"), and bumpFactory's
comment deferred the USD aggregates to a "price oracle / day data" that does not
exist. Sync is the only event carrying V2 reserves, so with it inert there was
no path — at any block height — from the chain to a TVL number.
Fixed by adding the missing concern rather than by decoding one more event:
indexer/valuation.go is the ONE owner of the derived USD aggregates. It reads
balanceOf(pool) for both legs of every registered pool in a single JSON-RPC
batch, anchors stablecoins at $1 by symbol, and relaxes prices outward through
the pool graph — each unknown token taking its quote from the DEEPEST pool
that already has a priced side, which is both deterministic and the least
manipulable choice when pools disagree. A token no path connects to a stable
stays unpriced and contributes nothing: unknown is reported as unknown.
Reserves are chain STATE, not chain HISTORY, so this is a pass beside the log
poller, not a handler inside it. That distinction is the whole point:
- Sync exists only on V2. balanceOf answers the same question for V2, V3 and
V4 identically.
- An event-sourced TVL only converges by replaying all history, and a pool
whose last Sync is a million blocks back reads empty until it trades again.
On an idle chain it never trades again.
- It is one fact, so it gets one writer. The Sync/Mint/Burn arm stays inert on
purpose and now says why.
poolCount moves to the same pass for the same reason: it is derived from the
store, not accumulated by the create handlers, so it can no longer drift across
a restart, a re-index or a duplicate create. bumpFactory keeps the one field
that genuinely is a running total of events — txCount.
Fails closed: an unreachable RPC aborts the pass rather than overwriting good
aggregates with zeros.
Tests: valuation_test.go drives a stub RPC shaped like the live mainnet book —
a stablecoin pair anchoring a WLUX pair — and asserts derived TVL, spot prices,
per-token TVL, factory poolCount, the deeper-quote rule, non-clobbering of
fields the pass does not own, and the RPC-blip guard. Full suite green.
Co-authored-by: Hanzo Dev <[email protected]>
handleInitializeV4 now sets the DEX Market symbol to the human BASE/QUOTE pair
(from the two currencies' ERC20 symbols) and marks assetsBound=true when both
resolve to a clean symbol. This is the exact shape the public exchange-api
real-asset gate (dexMarkets.ts isStructurallyReal) requires: without it every
native 0x9999 market was stripped (poolId-hex symbol failed the clean-pair check)
and the markets surface was empty despite real on-chain DEXFills. A token with no
identifiable symbol yields no pair -> the market stays unbound and hidden.
One place computes a market's display identity, read by every downstream surface
(explorer /dex, exchange-api gate, FE). New TDD: BindsCleanPairSymbol + UnnamedTokenStaysUnbound.
The D-Chain read surface (luxfi/dex pkg/dchain/read.go:50-53) serves
dex_get_{markets,trades,orders,book}; the CLOB source still called the old
clob_get_* names -> 404 -> empty dex subgraph while real fills matched
on-chain (lux.exchange showed no markets though the chain was correct).
Proven: GOWORK=off go test ./indexer/ -run TestCLOB green.
go-sqlite3 v2.0.3+incompatible is a deleted upstream tag (unknown revision),
breaking every standalone graph build. Pin to v1.14.47 (latest v1.14 patch;
the cgo driver store_sqlite.go imports). Regenerate go.sum first-party hashes
stale from the org tag re-move (luxfi/age etc.).
The dex subgraph indexer only consumed EVM 0x9999 settlement logs (the
cross-chain settlement view). On a native-DEX deployment the trade engine
IS the D-Chain (dexvm): a trade is a consensus state transition matched at
Block.Verify, not an EVM event, so native fills never appear on the EVM RPC
and the dex subgraph stays empty (markets:null).
Add CLOBSource (indexer/clob.go): an orthogonal, READ-ONLY poller of the
D-Chain committed-state read RPC clob_get_{markets,trades,orders} that writes
the same Market/Fill/Order entities the existing dex resolvers serve. Enabled
by Config.DexRPC; empty preserves EVM-only behaviour. The two sources share
only the store and the entity shapes, so this is purely additive — the amm
(EVM) subgraph is untouched.
- markets <- clob_get_markets (per-market authoritative book summary; bound-
market hex symbol decoded to its human pair, e.g. LUX/LUSD)
- fills <- clob_get_trades?since=<h> (global height-ordered log; keyed
height:seq -> idempotent; incremental by accepted height)
- orders <- clob_get_orders?market=<id> (resting book per market)
Tests (httptest doubles serving the exact read.go JSON shapes captured live
from lux-devnet): markets/trades/orders ingest + symbol decode, incremental
cursor + idempotent re-read, and EVM-AMM no-regression with DexRPC set.
Token entities were seeded with placeholder metadata (symbol=addr[:8],
name=full address, decimals=18) — no contract read happened, so the AMM
graph returned every token as its own address. Add erc20.go: read
symbol()/name()/decimals() via the existing eth_call RPC (canonical ABI
string + legacy bytes32 fallback, garbage-decimals guard), memoised once
per token (revert→placeholder cached too, never re-probed). Route all four
seed sites through a cache-first seedToken with a cold-start no-downgrade
guard. BackfillTokens (opt-in BACKFILL_TOKENS=1) enriches existing
placeholder rows in place — ≤3 eth_calls per token, no re-sync.
Red re-verify found the V4 AMM-spoof gate left a parallel bypass.
MEDIUM — V2/V3 AMM handlers were UNGATED: handlePairCreated, handlePoolCreated,
handleSwapV2, handleSwapV3 accepted events from ANY emitter — a parallel path to
seed phantom pairs/pools/swaps and inflate the SAME public Factory/TVL/volume
aggregates the now-gated V4 handlers (0x9999) block. Mirror the V4 gate:
- PairCreated/PoolCreated now require the canonical FactoryV2/FactoryV3 emitter
(the AMM trust root), exactly as V4 requires 0x9999.
- V2/V3 Swaps are emitted by the pair/pool, not the factory, so the gate is
"is the emitter a pool the canonical factory created?" (isKnownPool). The
factory authorizes a pool; only that pool's Swaps are honored. A rogue
factory's PairCreated is rejected, so its pools are never registered, so its
Swaps are rejected — the whole parallel path is closed.
- Pair/pool keys lower-cased at seed so the Swap-side lookup is RPC-casing safe.
- Factory addresses configurable (Config.FactoryV2/V3 + FACTORY_V2/V3 env),
default to canonical Lux mainnet (LuxMainnet.Factory{V2,V3}); main wires env.
INFO — the per-namespace resolver limit lacked the `if limit < 1 { limit = 100 }`
floor that resolvers/dex has, so a negative/zero `first` flowed to ListByType
(`first:-1` = "no limit" in SQLite — an unbounded scan). Brought every pl()/
parseLimit() copy to the dex-canonical body (floor <1->100 + cap 1000); fhe and
platform also gained the previously-missing 1000 cap. All namespaces now identical.
Tests (go test ./...): indexer + resolvers/{dex,exchange,fhe,platform} green.
- TestHandleV2_RejectsNonFactory, TestHandleV3_RejectsNonFactory: rogue
PairCreated/PoolCreated/Swap seeds nothing and does not inflate the factory
aggregate; the canonical-source event IS honored end-to-end.
- TestPL_ClampsFirst / TestParseLimit_ClampsFirst (exchange/fhe/platform):
negative/zero floored to 100, capped at 1000.
Three confirmed findings in the explorer/graph indexer:
- MEDIUM: handleInitializeV4/handleSwapV4 seeded AMM pools/tokens/swaps from
ANY emitting contract, so any contract could emit lookalike V4 events to
create phantom pools/tokens and inflate the explorer's public TVL/volume
aggregates. Gate both to the canonical 0x9999 PoolManager (idx.isPoolManager),
mirroring handleDEXFill. The now-always-true inner Market guard in
handleInitializeV4 is de-nested (one path).
- LOW (DoS): unguarded fixed-offset slices (token0[:8], l.Address[:8]) panic on
a short/malformed RPC log, and the poll goroutine had no recover() -> one bad
log crashed graphd. Route those symbol slices through the length-checked
shortAddr helper, and wrap each poll in pollGuarded() which recovers any panic
into an error so the loop continues.
- LOW (query-DoS): resolvers/dex parseLimit had no max cap. Clamp <1 -> 100 and
cap min(limit, 1000), matching every other resolver namespace.
Tests:
- TestHandleV4_RejectsNonPoolManager: rogue Initialize/Swap seeds nothing; the
same event from 0x9999 is honored.
- TestPoll_MalformedLogDoesNotCrash: malformed logs don't panic processLog, and
pollGuarded converts a poll panic into an error.
- TestParseLimit_Caps: default 100, floor 1->100, hard cap 1000.
The classic [self-hosted, linux, amd64] triple is serviced by fleet arcd hosts
that are not reliably available (v1.2.3 queued 30+ min). Use the in-cluster
lux-build autoscalingrunnerset (same as node + exchange-api). amd64-only — QEMU
arm64 pushed builds to ~2h. Artifact identical to v1.2.3; CI-plumbing only.
Item 4: the 3-consecutive-poll hysteresis alone could be satisfied by a deeply-stale
load-balanced RPC backend held >reorgDepth behind for the full window (~6s), wiping a
still-canonical chain and re-indexing from genesis (DoS amplification).
Keep the streak as a CHEAP PRE-FILTER, but gate the actual rewind on a deterministic
signal: the chain's genesis (block 0) hash. Every node serves block 0 regardless of
how far behind its head is, so an unchanged genesis hash deterministically rules out
a re-genesis even on a deeply-lagging backend (unlike a cursor by-number/by-hash probe,
which a backend whose head is below the cursor cannot serve and would false-trip).
Honors the no-eth_getCode rule (only eth_getBlockByNumber 0x0 header is read). The
probe is a struct field (genesisHashFn) so tests drive canonicality without a live RPC;
baseline captured lazily before any reset decision. Fail-closed: a probe error or
missing baseline never wipes.
3 existing regenesis tests updated to the new signature/semantics + 3 new
(deeply-behind-but-canonical => NO wipe; changed genesis => wipe; probe-fails => NO
wipe). Full indexer+engine suites green.
Red MEDIUM (FIX C): maybeResetForRegenesis rewound to StartBlock on the FIRST poll
where head << cursor, so an idle chain (eth_blockNumber=0x0, 'Building block') or a
briefly-lagging RPC backend triggered a full re-index from genesis (DoS amplification).
Add hysteresis: require the low head to persist across regenesisConfirmations
consecutive polls; a recovered/normal head clears the streak. A genuine relaunch holds
low for many polls; a transient dip recovers in one or two.
Red MEDIUM: bumpFactory wrote SeedFactoryData with empty TVL/volume, and SeedFactory is
INSERT OR REPLACE, so every pool-create/swap clobbered totalVolumeUSD/totalValueLockedUSD
(emptying the landing-page header). Carry both USD aggregates through the
read-modify-write (asString helper).
Red LOW: handleInitializeV4 now MERGES with any volume stub a prior fill accumulated
(don't reset volume24h/tradeCount to 0 when enriching with the token pair/fee).
decodeUint256 guards the SetString ok bool (malformed hex -> clean 0, not a half-parsed
amount).
Tests: regenesis requires sustained low head; idle-low-head-that-recovers does NOT wipe;
shallow reorg no reset; Initialize builds a rich Market (pair+fee), merges stub volume;
bumpFactory preserves TVL/volume; decodeInt256 signed. New engine end-to-end: the dex
schema resolves the FE markets/fills query shapes against indexed entities, amm does NOT
(the exchange-api routing rationale, mirroring Red's live finding). Full graph suite
green (CGO_ENABLED=1).
Replaces non-canonical scale-set / org-prefixed labels with the
existing labels every arcd host registers with. Matches evo for
amd64 and spark for arm64. No new labels added.
Cross-org runs-on dispatch failure: hanzo-build-linux-amd64 scale set is
registered to github.com/hanzoai and silently rejects luxfi/* jobs.
Switch to lux-build (githubConfigUrl=github.com/luxfi, max 20).
Less-typing canonical short path. With GRAPH_PREFIX=/v1/graph the
binary now serves both /v1/graph/graphql and /v1/graph/ql. Same
handlers, same payload shape.
Less-typing canonical short path. With GRAPH_PREFIX=/v1/graph the
binary now serves both /v1/graph/graphql and /v1/graph/ql. Same
handlers, same payload shape.
zeekayandHanzo Dev