A client pages a pool's history as "the N trades before this time". The bound
was applied by FilterResults after the database had already taken the newest N,
so every page after the first threw its rows away and came back empty. It sits
in the WHERE now, on the same index the pool does.
Co-authored-by: Hanzo Dev <[email protected]>
GetSwaps compares pool COLLATE NOCASE and orders by timestamp. SQLite consults an
index only when its collation matches the comparison's, so idx_swaps_pool — built
BINARY — was never used for that read: one quiet pool's last trades was a scan of
every swap on the chain, seconds per pool, and a token page asks once per pool it
trades in. Seventeen pools took forty seconds through the exchange-api and timed
out on the way.
The index now names the collation and carries the timestamp, so the read is a
range on the index in the order it is asked for. The old one is dropped on Init,
so a store that already built it stops carrying it.
Co-authored-by: Hanzo Dev <[email protected]>
Valuing the history fixed the protocol's total and left every pool reporting the
window a pass had just read, so a landing page claimed $206,718 in all-time
volume above a list of pools claiming $4,159. Two sums of the same trades, taken
in different places over different sets, and the reader has no way to tell which
is which.
They are one question sliced two ways. The store groups the swap table by pool
in a single scan — the pool column has been indexed since a pool's own trade
history needed it — and the pass folds that: each pool takes its row, each token
takes the rows of the pools holding it, and the protocol takes the sum. What the
pass valued this time around stays in the log, where it says what the pass did
rather than what the chain has done.
Co-authored-by: Hanzo Dev <[email protected]>
Promoting the trade value to a column made the sum honest and, in doing so,
showed what it was a sum of: 1,018,429 trades on this chain, of which the newest
50,000 had ever been priced. The other 95% were stored at zero — so the all-time
total was $4,048 where the newest window alone was $4,159, and every day before
that window was missing from every chart.
A pass now reads both ends of what it has not seen. The head keeps arriving
trades priced; the tail walks forward from a mark and covers the table a batch at
a time until it meets the head, after which it costs a query that returns
nothing. The mark records having LOOKED, not having priced: a trade whose tokens
reach no stablecoin can never be priced, and a mark that waited for one would
stop at it forever. It advances only once the values are on disk.
Days are written whole. A day row is replaced by the pass that writes it, and
both reads cut somewhere, so the day a cut lands in is dropped and comes back
from the other side. A read that came back short of its limit was not cut and
keeps everything — the case where the whole table fits in one window, which is
every small chain.
Co-authored-by: Hanzo Dev <[email protected]>
v1.2.47 replaced one window sum with the day series added up. Those rows are
folded from the same window, so their sum is that window again — a change that
read correctly and moved nothing. Live proof: the day series summed to exactly
the pass's own figure, and its trade counts to exactly the window size.
The value of a trade has always lived inside the swap's JSON document, where SQL
cannot reach it — this driver's amalgamation carries no JSON1, so json_extract
is not an escape either. So amountUSD joins timestamp and pool as a column: the
queryable projection of the document, written by both paths that touch a swap,
summed by one reader. What the protocol has traded is now one scan of one column
and covers every trade ever indexed.
A database that predates the column gets it on the start that introduces it,
filled from the rows themselves. Empty, it would have zeroed every trade older
than the current window — the fix erasing the number it exists to correct — so
that is the case the test pins, and the ALTER failing is how a later start knows
the fill is done.
txCount is left where it is. It counts every interaction, mints and burns among
them, which is a different question from what has traded.
Co-authored-by: Hanzo Dev <[email protected]>
Asking the store for one pool's last N trades read the N newest swaps on the
whole chain and then kept the ones that matched. The limit belongs to the
database and ran first, so the filter was filtering what the limit had already
thrown away.
One pair does most of the trading on a live chain and fills any recent window by
itself. On the Lux C-Chain that is WLUX/LZOO, and every other pool answered with
nothing — LETH's 145,386 trades, LSOL's 52,266, LBTC's 14,857, all present in
the store and unreachable through the query that asks for them. The pool column
has been indexed for exactly this since the table was created.
The test gives a quiet pool five trades and buries it under five hundred. Before
this it returns none of them.
Co-authored-by: Hanzo Dev <[email protected]>
A swap written by an older build holds its block NUMBER where a time belongs.
healSwapTimes exists to date those rows, and it asked for them with
RecentSwapsRaw — the newest rows by timestamp.
A block number is small beside a unix second. So every row needing repair sorts
behind every row already correct, and once a full window of dated rows existed
the heal saw nothing stale, reported success, and stopped. On the Lux C-Chain
that is 435,444 swaps across seventeen pools indexed and dated to 1970, sorting
below everything on every newest-first list the exchange draws: the token page
showed no trades for LETH, LBTC, LSOL, LTON, LPOL, LBNB and the rest while their
trades sat in the store the whole time.
Rows are now asked for by the property that identifies them, and the heal
repeats while a pass is still dating rows rather than stopping after one window.
The test builds the production shape — one undated row under a full window of
dated ones — and fails against the old read, which dates nothing and calls it
success.
Co-authored-by: Hanzo Dev <[email protected]>
A cursor is only meaningful for the trust root it was read with. Change the
factory and every PoolCreated it ever emitted is already behind the cursor, so
the new address is never seen and the old rows stay — the store keeps answering
for a book that is not on this chain.
Zoo mainnet sat that way. Its declared V3 factory held no code; the real one, at
the address Lux uses, has twelve pools. Correcting the address alone would have
changed nothing, because the cursor was at the tip.
So the source is part of the store's identity, exactly as the genesis hash is
part of the chain's, and a change to either rewinds. An unrecorded source is
unknown rather than different: every store written before this has none, and
reading that as a change would re-read a million blocks on Lux to rebuild rows
that are already right.
Co-authored-by: Hanzo Dev <[email protected]>
v1.2.28 moved off upstream mattn for a real reason — mattn compiles its own
copy of the amalgamation, and a binary that links it beside the house C
defines every sqlite3_* symbol twice. But it moved to hanzoai/sqlite, which
registers the driver under the name "sqlite", and modernc.org/sqlite registers
that name too. modernc is in every binary that stores here: hanzoai/replicate
imports it, and replicate.go sits in this package. Two packages, one name,
and database/sql panics in init — before main runs. Every build from v1.2.28
on could not start.
csqlite is the same house C under the name "sqlite3", which nothing else in
this graph takes. One amalgamation, two names, no argument.
Co-authored-by: Hanzo Dev <[email protected]>
Every already-indexed token had a symbol and no supply, so market cap and
fully diluted value printed a dash however much the asset traded. The read
existed; nothing ever reached it. The cold-start guard treated a real symbol
as proof the row was complete, so a token seen once was never asked about
again — and the row it adopted dropped the supply field on the way through.
One predicate now says whether a row still has something to ask the chain:
a real symbol AND a supply. One resolver answers it — store first, contract
for the rest, memoised per address — and one merge folds a fresh read over
what is already known, so a reverting call never erases a symbol and never
erases the value locked, volume and price the valuation pass put on the same
row. That last one was live: SeedToken is INSERT OR REPLACE and the write was
built from the contract read alone.
And one unit. The contract reports base units, the native coin's genesis
figure is already whole, and both landed in the same field — a client that
scales one and not the other is a valuation 10^18 out. The field is whole
tokens now, converted where it is read, exactly: integer division and the
remainder, because float64 gives up four orders of magnitude below a
two-trillion-token supply.
The backfill is what a start does rather than a flag to remember.
Co-authored-by: Hanzo Dev <[email protected]>
Three packages were linked: hanzoai/csqlite (the C), hanzoai/sqlite (the house
driver, which carries that C), and upstream mattn/go-sqlite3, which compiles its
own copy of the same amalgamation. Every sqlite3_* symbol and every cgo
trampoline was therefore defined twice, and the link failed — on darwin. On
linux it survived, because that linker is laxer about duplicates, which is the
only reason CI never showed it.
hanzoai/sqlite is the one already standard here, so mattn goes. The driver it
registers is named "sqlite", not "sqlite3".
Storage, indexer and engine tests pass, and the binary links.
Co-authored-by: Hanzo Dev <[email protected]>
Market cap was blank because nothing knew what was locked. The chain knows: the
P-Chain keeps the native token's whole supply and the part bonded to validators,
and answers in nLUX. Lux today is 13,272,095,200 total with 2,500,000,002
staked — 18.8% that exists and cannot be sold.
So market cap is price times supply less stake, and fully diluted value is price
times supply, and both come from the chain rather than a list somebody maintains.
Read every pass, because supply grows with each reward and the staked figure
moves with each delegation.
The endpoint is derived from the EVM one — the two chains sit beside each other
under the same node — so there is nothing to configure and nothing to drift.
A chain with no validators answers zero staked, which is the true answer for it.
Co-authored-by: Hanzo Dev <[email protected]>
TotalSupply was in the schema and nothing ever filled it, so a token page could
price an asset to six decimals and still print a dash where fully diluted value
belongs. It is one more eth_call beside symbol, name and decimals, cached the
same way — per token, once — and read at head, so it follows a mint or a burn.
Kept as text: a uint256 fits no float, and the client divides by decimals to
display it anyway.
The call-count guards move 3 → 4. What they protect is that this is per token
and not per log, which the cache loop beside them still proves.
Co-authored-by: Hanzo Dev <[email protected]>
json_set belongs to SQLite's JSON1 extension, which this driver's amalgamation
is not always built with — present on a developer's machine, absent in the
image. The statement prepared cleanly in every test and failed on every pass in
production: "no such function: json_set", on all 50,000 swaps, every 30 seconds.
The field is set in Go now and the row written whole, so it does not depend on
how SQLite was compiled. Fifty thousand rows take 372ms.
Co-authored-by: Hanzo Dev <[email protected]>
The write reported nothing, so when it changed no rows in production the pass
still printed a clean summary and every swap kept reading zero under a pool
claiming thousands in volume. It now returns rows-changed and any error, and
the valuation line carries both.
Co-authored-by: Hanzo Dev <[email protected]>
Every test in this repo failed to link on macOS: 283 duplicate SQLite symbols.
The module required an ORM no source file imports, and it brought a second copy
of the C amalgamation alongside the driver's own. Nothing here used it.
With it gone the suite runs on the same driver the indexer and explorer load,
which is the point — one copy of SQLite per binary, and one driver across the
stack.
Co-authored-by: Hanzo Dev <[email protected]>
A pool reported thousands of dollars of volume over a list of trades that each
said they were worth nothing. The value was already computed — the pass folds
it into the pool total — and then dropped, because writing it back had once
sent every row out as its own committed statement and a chain with real trade
history never finished a pass. Behind one transaction fifty thousand rows take
a quarter of a second, so the trades now carry the figures that add up to the
total above them.
Also swaps SQLite for the pure-Go build. The C driver ships its own copy of the
amalgamation, and anything else in the binary carrying one collided with it at
link time — 283 duplicate symbols, which took down every test in the repo on
macOS. With that gone the suite runs again, and it immediately found two
fixtures that never created their tables: the seeds were writing into nothing
and reporting no error.
Co-authored-by: Hanzo Dev <[email protected]>
Every price chart on the site read "Missing chart data". Not because the
trades were missing — they were on disk, and the swaps query serves them —
but because tokenDayDatas, poolDayDatas and uniswapDayDatas each answered
[] from a hardcoded stub, and a client draws nothing under three points.
Underneath that was a clock problem. eth_getLogs says which block a log was
in and never when, so a swap's `timestamp` held the block NUMBER — trade
history dated every fill to January 1970, and a day series was not merely
empty but impossible. The poll now reads the header of each block it saw an
event in, and declines to advance past one it could not read: a block is
indexed when it has been read whole. Rows an older build wrote are dated
from the same headers, which is what lets the series cover the history
already indexed rather than starting the day it ships.
The series itself is a second fold over the trade stream the valuation pass
already builds — one read, one valuation, two questions. A pool's candle is
its own execution ratio, exact for every day. A token's is that ratio carried
to USD through the other leg, so it is exact against a stablecoin and exact
in shape against a volatile one. Value locked is not in the trades at all, so
each day records it while it is today.
Recomputing the whole history every thirty seconds must not rewrite it, so a
day is written only when it moves: in the steady state that is one day per
subject that traded.
Filtering learned two things a chart needs. `where: {token: "0x…"}` names a
reference, and the row carries the token itself so the client can select
through it — so a filter now compares against the referenced id. And an
address arrives checksummed as often as lower-cased, so exact matches on text
ignore case.
pairDayDatas and the hour series stay stubs. Nothing asks for them.
Tests: 18 deliberate mutations — an off-by-one day boundary, high taking the
smallest price, the where filter dropped, the limit applied before the sort,
trades ordered by id, a log index read as decimal, a candle priced off its own
leg, and the call sites that write the series and read the block times removed
outright — each one turns a test red.
Co-authored-by: Hanzo Dev <[email protected]>
A store getter that answers with a list built it from `var result
[]interface{}` — a nil slice — so "nothing matched" marshalled to JSON
`null` instead of `[]`.
On a GraphQL wire those are opposite diagnoses. `{"data":{"markets":[]}}`
says "deployed, healthy, nothing has happened yet"; `{"data":{"markets":
null}}` is the signature of a subgraph that is absent, unsubscribed or
erroring behind a 200. The `dex` subgraph shipped the second while being
the first: mounted at /v1/graph/cchain/dex, subscribed to the 0x9999
DEXFill topic and caught up to the chain head, it reported itself dead on
every query purely because no fill had been emitted yet — leaving no way
to tell a quiet DEX from a broken pipe.
Six getters were affected (ListByType, GetTokens, GetFactories, GetPools,
GetPairs via GetPools, GetSwaps) plus the shared FilterResults tail. Every
other list getter in both backends already returned []interface{}{}; this
restores the convention rather than introducing one.
The regression test is reflective — it walks every (interface{}, error)
getter on *Store and fails any that answers with a nil slice — so a getter
added later cannot reintroduce the bug by being forgotten in a hand-written
list. 26 getters are covered.
Co-authored-by: Hanzo Dev <[email protected]>
The previous commit stopped the pass WRITING a row per swap. It still READ
every swap row in the table, every 30 seconds, into a map.
That is the same shape of mistake one layer down. A live chain's swap table
grows without limit — the Lux C-Chain's already spans blocks 53..1,096,496 in
a 732 MB store — so the cost of a pass grows with chain age forever, and the
allocation churn lands on a process sharing 4Gi with six other chains.
storage.SwapsRaw is replaced by RecentSwapsRaw(limit), which uses the indexed
timestamp column to take the newest `limit` rows in SQL rather than filtering
in Go. The pass takes the newest 50,000.
Where that bound binds, the volume figure is the volume of those 50,000 trades
and the log line says so outright. A partial sum silently presented as a total
is worse than a smaller number that is honest about its window.
Co-authored-by: Hanzo Dev <[email protected]>
Every USD field the exchange reads was an empty string. `pools`, `tokens` and
`factories` all resolved — with real addresses, real symbols, real fee tiers —
and then reported totalValueLockedUSD:"", volumeUSD:"", token0Price:"". The
landing page's TVL / 24h-volume / active-pool header had nothing to render, so
it rendered nothing.
Nothing was ever writing those fields. processLog's Sync/Mint/Burn arm was an
explicit no-op ("storage for these types not yet wired"), and bumpFactory's
comment deferred the USD aggregates to a "price oracle / day data" that does not
exist. Sync is the only event carrying V2 reserves, so with it inert there was
no path — at any block height — from the chain to a TVL number.
Fixed by adding the missing concern rather than by decoding one more event:
indexer/valuation.go is the ONE owner of the derived USD aggregates. It reads
balanceOf(pool) for both legs of every registered pool in a single JSON-RPC
batch, anchors stablecoins at $1 by symbol, and relaxes prices outward through
the pool graph — each unknown token taking its quote from the DEEPEST pool
that already has a priced side, which is both deterministic and the least
manipulable choice when pools disagree. A token no path connects to a stable
stays unpriced and contributes nothing: unknown is reported as unknown.
Reserves are chain STATE, not chain HISTORY, so this is a pass beside the log
poller, not a handler inside it. That distinction is the whole point:
- Sync exists only on V2. balanceOf answers the same question for V2, V3 and
V4 identically.
- An event-sourced TVL only converges by replaying all history, and a pool
whose last Sync is a million blocks back reads empty until it trades again.
On an idle chain it never trades again.
- It is one fact, so it gets one writer. The Sync/Mint/Burn arm stays inert on
purpose and now says why.
poolCount moves to the same pass for the same reason: it is derived from the
store, not accumulated by the create handlers, so it can no longer drift across
a restart, a re-index or a duplicate create. bumpFactory keeps the one field
that genuinely is a running total of events — txCount.
Fails closed: an unreachable RPC aborts the pass rather than overwriting good
aggregates with zeros.
Tests: valuation_test.go drives a stub RPC shaped like the live mainnet book —
a stablecoin pair anchoring a WLUX pair — and asserts derived TVL, spot prices,
per-token TVL, factory poolCount, the deeper-quote rule, non-clobbering of
fields the pass does not own, and the RPC-blip guard. Full suite green.
Co-authored-by: Hanzo Dev <[email protected]>
zeekayandHanzo Dev