lux/indexer
2
0
Fork 0
mirror of https://github.com/luxfi/indexer.git synced 2026-09-24 00:25:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

An NFT transfer says which NFT

Browse Source 
The indexer had the item id all along and threw it away at the edge.
formatTokenTransfer wrote every row in the ERC-20 shape — an amount and a
null decimals — so the three Lux Genesis mints, which moved ids 0, 1 and 2,
came back as three identical rows and no page could name what had moved.
The id now rides in `total`, whose shape follows the token type the way
luxfi/explore's Erc721TotalPayload / Erc1155TotalPayload already read it.

/tokens/{addr}/instances was a hardcoded empty list and /instances/{id} was
an unregistered route, so a collection had no items and an item had no page.
Both are real now. The item list comes from the transfer table rather than
the balance table: balances hold only what someone holds now, so a burned
item would vanish from a collection that still minted it, while its last
transfer always exists and names its current owner.

ERC-1155 was not indexed at all. It is now, single and batch — a batch log
yields one row per (id, amount) pair, so no id is folded away.

Underneath, one decoder. There were two, and they disagreed: the daemon's
path read ERC-20 and ERC-721 only, spelled types "ERC-721", and stored the
raw amount hex; the adapter's path also read ERC-1155 and ERC-404 but
spelled types "ERC721" and stored ids two different ways depending on which
event carried them. decodeTransfers is now the only place a log becomes a
transfer, with one spelling, decimal amounts, and one stored id shape.
applyERC20BalanceDelta and applyERC721BalanceDelta collapse into move(),
which is one function because the three token kinds differ only in what the
id column holds.

An item's tokenURI is read on chain and recorded; the document behind it is
not fetched. An indexer that waits on an IPFS gateway stops indexing, and
the document is not ours and can change. uri_state distinguishes the three
answers that were previously one blank: the chain gave a URI, the chain
names none, or we have not asked. image_url and metadata stay null rather
than assert media nobody read.

Verified against C-Chain 96369: Lux Genesis
(0x9e04fc57c20b2ee45627c4aa280eb471f2ca6ea5, totalSupply 3) indexed from the
live RPC now reports ids 0, 1, 2 on its transfers and lists three items with
their owners, matching eth_getLogs topics[3] and ownerOf.

migrations/004_nft_marketplace.sql stays unapplied and now says why: no
marketplace contract is deployed on any Lux chain, so its tables could only
ever be empty. LLM.md's EVM parity list claimed Seaport/LooksRare/Blur/
X2Y2/Rarible/Zora coverage for a package nothing imports; it now says what
runs.

Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
zeekay
2026-08-29 13:35:20 -07:00
co-authored by Hanzo Dev
parent abb63c79b4
commit 3d6388d6ab
10 changed files with 1079 additions and 309 deletions
Download Patch File Download Diff File Expand all files Collapse all files
LLM.md
+18 -3
View File
@@ -143,18 +143,33 @@ explorer --chain-name="My Chain" --coin=MYC --chain-id=12345
## EVM Feature Parity
All EVM explorer features implemented in Go:
What the Go indexer covers, and where it stops:
- Block indexing (consensus, uncle, EIP-4844 blobs)
- Transaction types 0-3 (legacy, access list, EIP-1559, EIP-4844)
- Internal transaction tracing (3 tracers: callTracer, jsTracer, parityTracer)
- Token tracking: ERC-20, ERC-721, ERC-1155 (batch), historical balances
- Token tracking: ERC-20, ERC-721, ERC-1155 (single + batch), ERC-404. One
decoder reads them all — `evm/decodeTransfers` — so a movement has one
reading whichever indexing path saw the block. ERC-20 and ERC-721 share the
Transfer signature and are told apart by topic arity: the ERC-721 id is
indexed and rides in topics[3].
- NFT items: `/tokens/{addr}/instances` and `/instances/{id}` — id, current
owner (the destination of the item's last transfer), and the URI the
contract gives it. The URI is an eth_call; the document behind it is not
fetched, so `uri_state` says whether the chain answered ("ok"), named
nothing ("absent"), or has not been asked ("unread"). `image_url` and
`metadata` stay null rather than assert media nobody read.
- Contract verification: Solidity (all versions), Vyper, Standard JSON
- Proxy detection: EIP-1967, EIP-1822, EIP-1167, Gnosis Safe, OpenZeppelin
- Account Abstraction: ERC-4337 UserOps, bundlers, paymasters
- MEV detection: sandwich, frontrun, backrun, liquidation
- DeFi: AMM V2/V3, perps, lending, staking, bridges, CLOB orderbook
- NFT marketplaces: Seaport, LooksRare, Blur, X2Y2, Rarible, Zora
- `evm/defi` additionally carries event signatures for Seaport, LooksRare,
Blur, X2Y2, Rarible and Zora. Nothing imports the package and no Lux chain
has a marketplace contract deployed, so no marketplace is indexed today.
`migrations/004_nft_marketplace.sql` is that same unbuilt venue's schema and
is not applied — the live API reads the SQLite tables `evm/indexer.go`
creates.
- Charts, statistics, gas oracle
- User accounts, API keys, watchlists, address tags
evm/adapter.go
+36 -201
View File
@@ -1836,9 +1836,9 @@ func (a *Adapter) updateTokenBalances(ctx context.Context, db *sql.DB, transfer
var err error
switch transfer.TokenType {
case "ERC20":
case TypeERC20:
balance, err = a.GetTokenBalance(ctx, transfer.TokenAddress, transfer.From, blockNumber)
case "ERC721":
case TypeERC721:
// For ERC721, balance is 0 or 1 based on ownership
owner, ownerErr := a.GetERC721Owner(ctx, transfer.TokenAddress, hexToBigInt(transfer.TokenID))
if ownerErr == nil && strings.EqualFold(owner, transfer.From) {
@@ -1847,7 +1847,7 @@ func (a *Adapter) updateTokenBalances(ctx context.Context, db *sql.DB, transfer
balance = big.NewInt(0)
}
err = ownerErr
case "ERC1155":
case TypeERC1155:
balance, err = a.GetERC1155Balance(ctx, transfer.TokenAddress, transfer.From, hexToBigInt(transfer.TokenID))
}
@@ -1871,9 +1871,9 @@ func (a *Adapter) updateTokenBalances(ctx context.Context, db *sql.DB, transfer
var err error
switch transfer.TokenType {
case "ERC20":
case TypeERC20:
balance, err = a.GetTokenBalance(ctx, transfer.TokenAddress, transfer.To, blockNumber)
case "ERC721":
case TypeERC721:
owner, ownerErr := a.GetERC721Owner(ctx, transfer.TokenAddress, hexToBigInt(transfer.TokenID))
if ownerErr == nil && strings.EqualFold(owner, transfer.To) {
balance = big.NewInt(1)
@@ -1881,7 +1881,7 @@ func (a *Adapter) updateTokenBalances(ctx context.Context, db *sql.DB, transfer
balance = big.NewInt(0)
}
err = ownerErr
case "ERC1155":
case TypeERC1155:
balance, err = a.GetERC1155Balance(ctx, transfer.TokenAddress, transfer.To, hexToBigInt(transfer.TokenID))
}
@@ -1965,211 +1965,46 @@ func (a *Adapter) UpdateExtendedStats(ctx context.Context, db *sql.DB) error {
return err
}
// parseTokenTransfers extracts token transfers from a log
// parseTokenTransfers extracts token movements from a log on the Adapter
// path. Transfer events are read by decodeTransfers, the one decoder both
// indexing paths share; the wrapper mint/burn events below are this path's
// own and are not transfers in the ERC sense — a WETH deposit creates supply
// rather than moving it.
func parseTokenTransfers(log Log, timestamp time.Time) []TokenTransfer {
var transfers []TokenTransfer
if len(log.Topics) == 0 {
return transfers
if t := decodeTransfers(log, timestamp); len(t) > 0 {
return t
}
topic0 := log.Topics[0]
switch topic0 {
case TopicTransferERC20:
// ERC20/ERC721 Transfer(address,address,uint256)
if len(log.Topics) >= 3 {
from := topicToAddress(log.Topics[1])
to := topicToAddress(log.Topics[2])
var value, tokenID string
tokenType := "ERC20"
if len(log.Topics) == 4 {
// ERC721: indexed tokenId in topic[3]
tokenType = "ERC721"
tokenID = log.Topics[3]
value = "1"
} else {
// ERC20: value in data
value = hexToBigInt(log.Data).String()
}
transfers = append(transfers, TokenTransfer{
ID: fmt.Sprintf("%s-%d", log.TxHash, log.LogIndex),
TxHash: log.TxHash,
LogIndex: log.LogIndex,
BlockNumber: log.BlockNumber,
TokenAddress: log.Address,
TokenType: tokenType,
From: from,
To: to,
Value: value,
TokenID: tokenID,
Timestamp: timestamp,
})
} else if len(log.Topics) == 1 {
// Non-standard ERC-721: all params in data (from, to, tokenId) = 3 x 32 bytes
data := strings.TrimPrefix(log.Data, "0x")
if len(data) == 192 {
from := topicToAddress("0x" + data[:64])
to := topicToAddress("0x" + data[64:128])
tokenID := hexToBigInt("0x" + data[128:192]).String()
transfers = append(transfers, TokenTransfer{
ID: fmt.Sprintf("%s-%d", log.TxHash, log.LogIndex),
TxHash: log.TxHash,
LogIndex: log.LogIndex,
BlockNumber: log.BlockNumber,
TokenAddress: log.Address,
TokenType: "ERC721",
From: from,
To: to,
Value: "1",
TokenID: tokenID,
Timestamp: timestamp,
})
}
}
case TopicTransferSingle:
// ERC1155 TransferSingle(address operator, address from, address to, uint256 id, uint256 value)
if len(log.Topics) >= 4 {
data := strings.TrimPrefix(log.Data, "0x")
if len(data) >= 128 {
from := topicToAddress(log.Topics[2])
to := topicToAddress(log.Topics[3])
tokenID := hexToBigInt("0x" + data[:64]).String()
value := hexToBigInt("0x" + data[64:128]).String()
transfers = append(transfers, TokenTransfer{
ID: fmt.Sprintf("%s-%d", log.TxHash, log.LogIndex),
TxHash: log.TxHash,
LogIndex: log.LogIndex,
BlockNumber: log.BlockNumber,
TokenAddress: log.Address,
TokenType: "ERC1155",
From: from,
To: to,
Value: value,
TokenID: tokenID,
Timestamp: timestamp,
})
}
}
case TopicTransferBatch:
// ERC1155 TransferBatch(address operator, address from, address to, uint256[] ids, uint256[] values)
if len(log.Topics) >= 4 {
from := topicToAddress(log.Topics[2])
to := topicToAddress(log.Topics[3])
ids, values := decodeBatchData(log.Data)
for i := range ids {
val := "0"
if i < len(values) {
val = values[i]
}
transfers = append(transfers, TokenTransfer{
ID: fmt.Sprintf("%s-%d-%d", log.TxHash, log.LogIndex, i),
TxHash: log.TxHash,
LogIndex: log.LogIndex,
BlockNumber: log.BlockNumber,
TokenAddress: log.Address,
TokenType: "ERC1155",
From: from,
To: to,
Value: val,
TokenID: ids[i],
Timestamp: timestamp,
})
}
}
case TopicERC404ERC20Transfer:
// ERC-404 ERC20Transfer(address,address,uint256)
if len(log.Topics) >= 3 {
from := topicToAddress(log.Topics[1])
to := topicToAddress(log.Topics[2])
value := hexToBigInt(log.Data).String()
transfers = append(transfers, TokenTransfer{
ID: fmt.Sprintf("%s-%d", log.TxHash, log.LogIndex),
TxHash: log.TxHash,
LogIndex: log.LogIndex,
BlockNumber: log.BlockNumber,
TokenAddress: log.Address,
TokenType: "ERC20",
From: from,
To: to,
Value: value,
Timestamp: timestamp,
})
}
case TopicERC404ERC721Transfer:
// ERC-404 ERC721Transfer — from/to indexed, tokenId in data
if len(log.Topics) >= 3 {
from := topicToAddress(log.Topics[1])
to := topicToAddress(log.Topics[2])
tokenID := hexToBigInt(log.Data).String()
transfers = append(transfers, TokenTransfer{
ID: fmt.Sprintf("%s-%d", log.TxHash, log.LogIndex),
TxHash: log.TxHash,
LogIndex: log.LogIndex,
BlockNumber: log.BlockNumber,
TokenAddress: log.Address,
TokenType: "ERC721",
From: from,
To: to,
Value: "1",
TokenID: tokenID,
Timestamp: timestamp,
})
}
if len(log.Topics) == 0 {
return nil
}
zero := "0x0000000000000000000000000000000000000000"
wrap := func(from, to string) []TokenTransfer {
return []TokenTransfer{{
ID: fmt.Sprintf("%s-%d", log.TxHash, log.LogIndex),
TxHash: log.TxHash,
LogIndex: log.LogIndex,
BlockNumber: log.BlockNumber,
TokenAddress: log.Address,
TokenType: TypeERC20,
From: from,
To: to,
Value: hexToBigInt(log.Data).String(),
Timestamp: timestamp,
}}
}
switch log.Topics[0] {
case TopicWETHDeposit:
// Deposit(address indexed dst, uint256 wad)
if len(log.Topics) >= 2 {
to := topicToAddress(log.Topics[1])
value := hexToBigInt(log.Data).String()
transfers = append(transfers, TokenTransfer{
ID: fmt.Sprintf("%s-%d", log.TxHash, log.LogIndex),
TxHash: log.TxHash,
LogIndex: log.LogIndex,
BlockNumber: log.BlockNumber,
TokenAddress: log.Address,
TokenType: "ERC20",
From: "0x0000000000000000000000000000000000000000",
To: to,
Value: value,
Timestamp: timestamp,
})
return wrap(zero, topicToAddress(log.Topics[1]))
}
case TopicWETHWithdrawal:
// Withdrawal(address indexed src, uint256 wad)
if len(log.Topics) >= 2 {
from := topicToAddress(log.Topics[1])
value := hexToBigInt(log.Data).String()
transfers = append(transfers, TokenTransfer{
ID: fmt.Sprintf("%s-%d", log.TxHash, log.LogIndex),
TxHash: log.TxHash,
LogIndex: log.LogIndex,
BlockNumber: log.BlockNumber,
TokenAddress: log.Address,
TokenType: "ERC20",
From: from,
To: "0x0000000000000000000000000000000000000000",
Value: value,
Timestamp: timestamp,
})
return wrap(topicToAddress(log.Topics[1]), zero)
}
}
return transfers
return nil
}
// decodeBatchData ABI-decodes (uint256[], uint256[]) from hex log data.
@@ -2275,7 +2110,7 @@ func (a *Adapter) GetCode(ctx context.Context, address string) (string, error) {
func (a *Adapter) GetTokenInfo(ctx context.Context, tokenAddress string) (*Token, error) {
token := &Token{
Address: strings.ToLower(tokenAddress),
TokenType: "ERC20",
TokenType: TypeERC20,
CreatedAt: time.Now(),
UpdatedAt: time.Now(),
}
evm/adapter_test.go
+18 -18
View File
@@ -594,7 +594,7 @@ func TestGetTokenInfo(t *testing.T) {
if token.Decimals != 18 {
t.Errorf("Decimals = %d, want 18", token.Decimals)
}
if token.TokenType != "ERC20" {
if token.TokenType != TypeERC20 {
t.Errorf("TokenType = %q, want ERC20", token.TokenType)
}
}
@@ -681,7 +681,7 @@ func TestTokenTransferStruct(t *testing.T) {
LogIndex: 0,
BlockNumber: 100,
TokenAddress: "0xtoken",
TokenType: "ERC20",
TokenType: TypeERC20,
From: "0xfrom",
To: "0xto",
Value: "1000",
@@ -715,7 +715,7 @@ func TestTokenStruct(t *testing.T) {
Symbol: "TST",
Decimals: 18,
TotalSupply: "1000000000000000000000000",
TokenType: "ERC20",
TokenType: TypeERC20,
HolderCount: 1000,
TxCount: 5000,
CreatedAt: time.Now(),
@@ -1041,7 +1041,7 @@ func TestParseTokenTransfersERC20(t *testing.T) {
}
transfer := transfers[0]
if transfer.TokenType != "ERC20" {
if transfer.TokenType != TypeERC20 {
t.Errorf("TokenType = %q, want ERC20", transfer.TokenType)
}
if transfer.From != "0xfrom1234567890abcdef1234567890abcdef1234" {
@@ -1076,7 +1076,7 @@ func TestParseTokenTransfersERC721(t *testing.T) {
}
transfer := transfers[0]
if transfer.TokenType != "ERC721" {
if transfer.TokenType != TypeERC721 {
t.Errorf("TokenType = %q, want ERC721", transfer.TokenType)
}
if transfer.Value != "1" {
@@ -2112,7 +2112,7 @@ func TestParseTokenTransfersERC1155Single(t *testing.T) {
}
transfer := transfers[0]
if transfer.TokenType != "ERC1155" {
if transfer.TokenType != TypeERC1155 {
t.Errorf("TokenType = %q, want ERC1155", transfer.TokenType)
}
if transfer.TokenAddress != "0xerc1155token" {
@@ -2174,10 +2174,10 @@ func TestParseTokenTransfersERC1155Batch(t *testing.T) {
t.Fatalf("expected 3 transfers, got %d", len(transfers))
}
wantIDs := []string{"1", "2", "3"}
wantIDs := []string{hexWord("1"), hexWord("2"), hexWord("3")}
wantValues := []string{"10", "20", "30"}
for i, tr := range transfers {
if tr.TokenType != "ERC1155" {
if tr.TokenType != TypeERC1155 {
t.Errorf("[%d] TokenType = %q, want ERC1155", i, tr.TokenType)
}
if tr.TokenID != wantIDs[i] {
@@ -2225,8 +2225,8 @@ func TestParseTokenTransfersERC1155SingleFixed(t *testing.T) {
if len(transfers) != 1 {
t.Fatalf("expected 1 transfer, got %d", len(transfers))
}
if transfers[0].TokenID != "5" {
t.Errorf("TokenID = %q, want 5", transfers[0].TokenID)
if transfers[0].TokenID != hexWord("5") {
t.Errorf("TokenID = %q, want %q", transfers[0].TokenID, hexWord("5"))
}
if transfers[0].Value != "100" {
t.Errorf("Value = %q, want 100", transfers[0].Value)
@@ -2263,14 +2263,14 @@ func TestParseTokenTransfersNonStandardERC721(t *testing.T) {
t.Fatalf("expected 1 transfer, got %d", len(transfers))
}
tr := transfers[0]
if tr.TokenType != "ERC721" {
if tr.TokenType != TypeERC721 {
t.Errorf("TokenType = %q, want ERC721", tr.TokenType)
}
if tr.Value != "1" {
t.Errorf("Value = %q, want 1", tr.Value)
}
if tr.TokenID != "7" {
t.Errorf("TokenID = %q, want 7", tr.TokenID)
if tr.TokenID != hexWord("7") {
t.Errorf("TokenID = %q, want %q", tr.TokenID, hexWord("7"))
}
if tr.From != "0x1111111111111111111111111111111111111111" {
t.Errorf("From = %q", tr.From)
@@ -2300,7 +2300,7 @@ func TestParseTokenTransfersERC404(t *testing.T) {
if len(transfers) != 1 {
t.Fatalf("expected 1 transfer, got %d", len(transfers))
}
if transfers[0].TokenType != "ERC20" {
if transfers[0].TokenType != TypeERC20 {
t.Errorf("TokenType = %q, want ERC20", transfers[0].TokenType)
}
if transfers[0].Value != "1000000000000000000" {
@@ -2321,14 +2321,14 @@ func TestParseTokenTransfersERC404(t *testing.T) {
if len(transfers) != 1 {
t.Fatalf("expected 1 transfer, got %d", len(transfers))
}
if transfers[0].TokenType != "ERC721" {
if transfers[0].TokenType != TypeERC721 {
t.Errorf("TokenType = %q, want ERC721", transfers[0].TokenType)
}
if transfers[0].Value != "1" {
t.Errorf("Value = %q, want 1", transfers[0].Value)
}
if transfers[0].TokenID != "66" {
t.Errorf("TokenID = %q, want 66", transfers[0].TokenID)
if transfers[0].TokenID != hexWord("66") {
t.Errorf("TokenID = %q, want %q", transfers[0].TokenID, hexWord("66"))
}
})
}
@@ -2352,7 +2352,7 @@ func TestParseTokenTransfersWETH(t *testing.T) {
t.Fatalf("expected 1 transfer, got %d", len(transfers))
}
tr := transfers[0]
if tr.TokenType != "ERC20" {
if tr.TokenType != TypeERC20 {
t.Errorf("TokenType = %q, want ERC20", tr.TokenType)
}
if tr.From != "0x0000000000000000000000000000000000000000" {
evm/indexer.go
+167 -79
View File
@@ -245,11 +245,10 @@ func (idx *Indexer) Init(ctx context.Context) error {
},
},
{
// Per-(token, holder) balance state. Maintained on every
// detected Transfer log via applyERC20BalanceDelta /
// applyERC721BalanceDelta. token_id is "" for ERC-20
// (one row per holder per token) or the hex tokenId for
// ERC-721 (one row per NFT, owner stored in `address`).
// Who holds what right now. Maintained by move() on every
// decoded transfer. token_id is "" for ERC-20 (one row per
// holder per token) and the token id as a 32-byte hex word
// for ERC-721 and ERC-1155 (owner in `address`).
Name: "evm_token_balances",
Columns: []storage.Column{
// Composite PK (token_address, address, token_id) is
@@ -265,6 +264,23 @@ func (idx *Indexer) Init(ctx context.Context) error {
{Name: "updated_at", Type: storage.TypeTimestamp, Default: "CURRENT_TIMESTAMP"},
},
},
{
// One row per NFT — the item itself, as distinct from who
// holds it. Ownership moves and lives in
// evm_token_balances; the id and the URI the contract gives
// it belong to the item and outlive any holder, including
// after a burn leaves it with none.
Name: "evm_token_instances",
Columns: []storage.Column{
{Name: "token_address", Type: storage.TypeText, Nullable: false, Primary: true},
{Name: "token_id", Type: storage.TypeText, Nullable: false, Primary: true},
{Name: "uri", Type: storage.TypeText, Default: "''"},
// What the chain answered when asked: "ok" or "absent".
// No row means unread. See readURI.
{Name: "uri_state", Type: storage.TypeText, Default: "''"},
{Name: "updated_at", Type: storage.TypeTimestamp, Default: "CURRENT_TIMESTAMP"},
},
},
},
Indexes: []storage.Index{
{Name: "idx_evm_blocks_number", Table: "evm_blocks", Columns: []string{"number"}},
@@ -283,6 +299,8 @@ func (idx *Indexer) Init(ctx context.Context) error {
{Name: "idx_evm_logs_topic0", Table: "evm_logs", Columns: []string{"topic0"}},
{Name: "idx_evm_token_balances_token", Table: "evm_token_balances", Columns: []string{"token_address"}},
{Name: "idx_evm_token_balances_addr", Table: "evm_token_balances", Columns: []string{"address"}},
{Name: "idx_evm_token_instances_token", Table: "evm_token_instances", Columns: []string{"token_address"}},
{Name: "idx_evm_token_transfers_item", Table: "evm_token_transfers", Columns: []string{"token_address", "token_id"}},
},
}
@@ -516,6 +534,9 @@ func (idx *Indexer) indexBlock(ctx context.Context, blockNum uint64) error {
// row in evm_tokens with the metadata best-effort resolved via eth_call
// (name/symbol/decimals).
tokensSeen := make(map[string]string) // address → token_type
// The (collection, id) pairs this block moved. Each one is an item whose
// tokenURI we go on to read — see readURI.
itemsSeen := make(map[item]struct{})
for i, txHash := range block.Transactions {
tx, logs, err := idx.adapter.GetTransactionReceipt(ctx, txHash)
if err != nil || tx == nil {
@@ -559,34 +580,35 @@ func (idx *Indexer) indexBlock(ctx context.Context, blockNum uint64) error {
l.Address, topic0, topic1, topic2, topic3, l.Data,
block.Timestamp, time.Now())
// Topic-0 of ERC-20 Transfer is identical to ERC-721 Transfer;
// distinguish on topic count (ERC-20 has 3 topics, ERC-721 has 4).
if topic0 == TopicTransferERC20 {
from := topicToAddr(topic1)
to := topicToAddr(topic2)
tokenLo := strings.ToLower(l.Address)
if len(l.Topics) == 3 {
// ERC-20: value is the data field (uint256 hex)
_ = idx.store.Exec(ctx, idx.upsertTokenTransferSQL(),
logID, l.TxHash, int64(l.LogIndex), int64(block.Number),
l.Address, "ERC-20", from, to, l.Data, "",
block.Timestamp, time.Now())
tokensSeen[tokenLo] = "ERC-20"
// Update per-holder balance state. l.Data is the
// 32-byte uint256 value. We move it from `from` to
// `to`, skipping the zero-address (mint source +
// burn sink — those aren't real holders).
idx.applyERC20BalanceDelta(ctx, tokenLo, from, to, l.Data)
} else if len(l.Topics) == 4 {
// ERC-721: token_id in topic3, value is "1"
_ = idx.store.Exec(ctx, idx.upsertTokenTransferSQL(),
logID, l.TxHash, int64(l.LogIndex), int64(block.Number),
l.Address, "ERC-721", from, to, "1", topic3,
block.Timestamp, time.Now())
tokensSeen[tokenLo] = "ERC-721"
// For ERC-721 the entire token_id moves: delete
// from the old owner, insert for the new owner.
idx.applyERC721BalanceDelta(ctx, tokenLo, from, to, topic3)
// Every token movement in this log, decoded once by
// decodeTransfers — ERC-20, ERC-721, ERC-1155 single and
// batch. A batch log yields one row per (id, amount) pair,
// so no id is folded away.
for _, t := range decodeTransfers(Log{
TxHash: l.TxHash, LogIndex: l.LogIndex, BlockNumber: l.BlockNumber,
Address: l.Address, Topics: l.Topics, Data: l.Data,
}, block.Timestamp) {
_ = idx.store.Exec(ctx, idx.upsertTokenTransferSQL(),
t.ID, t.TxHash, int64(t.LogIndex), int64(t.BlockNumber),
t.TokenAddress, t.TokenType, t.From, t.To, t.Value, t.TokenID,
block.Timestamp, time.Now())
tokenLo := strings.ToLower(t.TokenAddress)
tokensSeen[tokenLo] = t.TokenType
// Per-holder state. ERC-20 moves an amount between two
// running totals; an ERC-721 id moves whole, so the row
// leaves the sender and arrives at the recipient; an
// ERC-1155 id has an amount per holder, so it moves like
// ERC-20 but keyed by id as well.
switch t.TokenType {
case TypeERC20:
idx.move(ctx, tokenLo, t.From, t.To, "", t.Value, TypeERC20)
case TypeERC721:
idx.move(ctx, tokenLo, t.From, t.To, t.TokenID, "1", TypeERC721)
itemsSeen[item{tokenLo, t.TokenID, TypeERC721}] = struct{}{}
case TypeERC1155:
idx.move(ctx, tokenLo, t.From, t.To, t.TokenID, t.Value, TypeERC1155)
itemsSeen[item{tokenLo, t.TokenID, TypeERC1155}] = struct{}{}
}
}
}
@@ -621,6 +643,12 @@ func (idx *Indexer) indexBlock(ctx context.Context, blockNum uint64) error {
time.Now(), time.Now())
}
// Read tokenURI for every item this block moved. See readURI for why
// the document behind the URI is left to the client.
for it := range itemsSeen {
idx.readURI(ctx, it)
}
// Resolve is_contract for everything we haven't already pinned as
// a contract via tx.ContractAddress. Anvil-local RPCs return code
// for any address in O(1); cost is bounded by unique addresses per
@@ -1231,18 +1259,6 @@ func (idx *Indexer) resolveTokenMeta(ctx context.Context, addr, tokenType string
return info.Name, info.Symbol, info.Decimals, totalSupply
}
// topicToAddr extracts the trailing 20-byte address from a 32-byte indexed
// topic. ERC-20 Transfer indexes `from` + `to` as topics in their
// uint256-padded form: a topic looks like 0x0000…<20-byte addr>. Returns
// "" for empty / malformed input.
func topicToAddr(topic string) string {
t := strings.TrimPrefix(topic, "0x")
if len(t) < 40 {
return ""
}
return "0x" + strings.ToLower(t[len(t)-40:])
}
// upsertBalanceSQL returns the correct upsert SQL for evm_token_balances.
//
// Schema: (token_address, address, token_id, value, token_type, updated_at)
@@ -1290,49 +1306,121 @@ func (idx *Indexer) readBalance(ctx context.Context, token, holder, tokenID stri
return n
}
// applyERC20BalanceDelta moves `valueHex` (32-byte uint256, hex with "0x"
// prefix optional) from `fromAddr` to `toAddr` in evm_token_balances. The
// zero address is excluded (it's the conventional mint source + burn sink,
// not a real holder).
func (idx *Indexer) applyERC20BalanceDelta(ctx context.Context, token, fromAddr, toAddr, valueHex string) {
v, ok := new(big.Int).SetString(strings.TrimPrefix(valueHex, "0x"), 16)
// move applies one token movement to evm_token_balances, the table that
// answers "who holds what right now".
//
// It is one function for all three token kinds because they differ only in
// what the id column holds. An ERC-20 holding is (token, holder) with an
// empty id and a running amount. An ERC-1155 holding is (token, holder, id)
// with a running amount. An ERC-721 holding is the same with the amount
// always 1, so the id simply arrives at the recipient and leaves the sender.
//
// A holding that reaches zero is deleted rather than kept as a "0" row: a row
// that says nothing is held is not a holding, and every reader already has to
// filter it out.
//
// The zero address is skipped on both sides. It is the conventional mint
// source and burn sink, not a holder, and crediting it would put every token
// ever minted in one enormous fictional account.
func (idx *Indexer) move(ctx context.Context, token, from, to, id, amount, kind string) {
v, ok := new(big.Int).SetString(amount, 10)
if !ok || v.Sign() == 0 {
return
}
now := time.Now()
zero := "0x0000000000000000000000000000000000000000"
if fromAddr != "" && fromAddr != zero {
cur := idx.readBalance(ctx, token, fromAddr, "")
newV := new(big.Int).Sub(cur, v)
if newV.Sign() < 0 {
newV = big.NewInt(0)
const zero = "0x0000000000000000000000000000000000000000"
write := func(holder string, n *big.Int) {
if n.Sign() <= 0 {
_ = idx.store.Exec(ctx,
"DELETE FROM evm_token_balances WHERE token_address = ? AND address = ? AND token_id = ?",
token, holder, id)
return
}
_ = idx.store.Exec(ctx, idx.upsertBalanceSQL(),
token, fromAddr, "", newV.String(), "ERC-20", now)
_ = idx.store.Exec(ctx, idx.upsertBalanceSQL(), token, holder, id, n.String(), kind, now)
}
if toAddr != "" && toAddr != zero {
cur := idx.readBalance(ctx, token, toAddr, "")
newV := new(big.Int).Add(cur, v)
_ = idx.store.Exec(ctx, idx.upsertBalanceSQL(),
token, toAddr, "", newV.String(), "ERC-20", now)
if from != "" && from != zero {
write(from, new(big.Int).Sub(idx.readBalance(ctx, token, from, id), v))
}
if to != "" && to != zero {
write(to, new(big.Int).Add(idx.readBalance(ctx, token, to, id), v))
}
}
// applyERC721BalanceDelta moves a single NFT (tokenId from topic3) between
// owners. Each ERC-721 row represents "this address owns tokenId X" with
// value="1". On transfer we remove the source row and write a destination
// row.
func (idx *Indexer) applyERC721BalanceDelta(ctx context.Context, token, fromAddr, toAddr, tokenIDHex string) {
now := time.Now()
zero := "0x0000000000000000000000000000000000000000"
if fromAddr != "" && fromAddr != zero {
_ = idx.store.Exec(ctx,
"DELETE FROM evm_token_balances WHERE token_address = ? AND address = ? AND token_id = ?",
token, fromAddr, tokenIDHex,
)
// item is one NFT: a collection address and a token id, which together are
// the only thing that identifies a single collectible.
type item struct {
token string
id string
kind string
}
// readURI records what the contract says the URI of an item is.
//
// The URI is an on-chain read — tokenURI(uint256) for ERC-721, uri(uint256)
// for ERC-1155 — so it costs one eth_call against the node we are already
// reading receipts from, and it cannot stall.
//
// The document at the far end is not fetched here, and that is deliberate.
// It usually lives on IPFS or someone's CDN, where a slow or dead gateway
// would hold the block cursor open behind it; an indexer that waits on
// third-party HTTP stops indexing. It is also not our data and can change
// under us. The client asking for it can time out on its own without the
// chain falling behind, so the URI is what we store and the fetch is the
// caller's.
//
// What is recorded is therefore what the chain answered:
//
// ok the contract returned a URI
// absent the call reverted or returned nothing — this contract does not
// name a URI for this id
//
// and no row at all means we have not asked, which the API reports as
// unread. An item with no picture is never silently the same as one whose
// contract has no URI to give.
//
// The read happens once, when the item is first seen. A contract that
// reveals later by changing tokenURI is not picked up until the item next
// moves; nothing on chain announces a reveal.
func (idx *Indexer) readURI(ctx context.Context, it item) {
if it.token == "" || it.id == "" {
return
}
if toAddr != "" && toAddr != zero {
_ = idx.store.Exec(ctx, idx.upsertBalanceSQL(),
token, toAddr, tokenIDHex, "1", "ERC-721", now)
if rows, err := idx.store.Query(ctx,
"SELECT 1 FROM evm_token_instances WHERE token_address = ? AND token_id = ? LIMIT 1",
it.token, it.id,
); err == nil && len(rows) > 0 {
return
}
// tokenURI(uint256) for ERC-721, uri(uint256) for ERC-1155.
selector := "0xc87b56dd"
if it.kind == TypeERC1155 {
selector = "0x0e89341c"
}
uri, state := "", "absent"
if out, err := idx.adapter.callContract(ctx, it.token, selector+strings.TrimPrefix(it.id, "0x")); err == nil {
if u := decodeString(out); u != "" {
uri, state = u, "ok"
}
}
_ = idx.store.Exec(ctx, idx.upsertInstanceSQL(), it.token, it.id, uri, state, time.Now())
}
// upsertInstanceSQL returns the upsert for evm_token_instances, keyed on the
// item — (token_address, token_id).
func (idx *Indexer) upsertInstanceSQL() string {
switch idx.store.Backend() {
case storage.BackendPostgres:
return `INSERT INTO evm_token_instances (token_address, token_id, uri, uri_state, updated_at)
VALUES ($1,$2,$3,$4,$5)
ON CONFLICT (token_address, token_id) DO UPDATE SET
uri = EXCLUDED.uri, uri_state = EXCLUDED.uri_state, updated_at = EXCLUDED.updated_at`
default:
return `INSERT INTO evm_token_instances (token_address, token_id, uri, uri_state, updated_at)
VALUES (?,?,?,?,?)
ON CONFLICT (token_address, token_id) DO UPDATE SET
uri = excluded.uri, uri_state = excluded.uri_state, updated_at = excluded.updated_at`
}
}
evm/transfer.go
+164
View File
@@ -0,0 +1,164 @@
package evm
import (
"fmt"
"math/big"
"strings"
"time"
)
// decodeTransfers reads the token movements out of one log.
//
// It is the only place in this package that turns an event into a
// TokenTransfer. Both indexing paths call it — Indexer.indexBlock, which the
// daemon runs, and Adapter.ProcessBlock — so a chain fact has one reading
// regardless of which path saw the block.
//
// Telling ERC-20 from ERC-721 is the whole trick, and it is not the topic
// hash: Transfer(address,address,uint256) is the same signature for both, so
// topic0 is identical. The difference is arity. ERC-721 declares the id
// `indexed`, so it rides in topics[3] and the log carries four topics; ERC-20
// leaves the amount unindexed, so it rides in data and the log carries three.
// Read the arity and both token types stay right.
//
// ERC-1155 is a different event entirely — TransferSingle carries one
// (id, amount) pair in data, TransferBatch carries two arrays — so it needs
// no arity trick, only its own decoding.
//
// Three representation rules hold for every transfer this returns:
//
// TokenType is hyphenated ("ERC-721"), the spelling the API and the
// explorer's TokenType union use.
// Value is decimal. An amount is a number; the hex in the log is transport.
// TokenID is the 32-byte hex word the chain encoded, which is how it is
// stored — see hexWord. The API converts it to decimal at the edge.
//
// Wrapper mint/burn (WETH Deposit/Withdrawal) is not a transfer event and is
// not decoded here; Adapter handles those on its own path.
func decodeTransfers(l Log, ts time.Time) []TokenTransfer {
if len(l.Topics) == 0 {
return nil
}
at := func(i int) string {
if i < len(l.Topics) {
return l.Topics[i]
}
return ""
}
one := func(kind, from, to, value, id string) []TokenTransfer {
return []TokenTransfer{{
ID: fmt.Sprintf("%s-%d", l.TxHash, l.LogIndex),
TxHash: l.TxHash,
LogIndex: l.LogIndex,
BlockNumber: l.BlockNumber,
TokenAddress: l.Address,
TokenType: kind,
From: from,
To: to,
Value: value,
TokenID: id,
Timestamp: ts,
}}
}
switch l.Topics[0] {
case TopicTransferERC20:
switch len(l.Topics) {
case 4:
// ERC-721: the id is indexed, so it is topics[3].
return one(TypeERC721, topicToAddress(at(1)), topicToAddress(at(2)), "1", hexWord(at(3)))
case 3:
// ERC-20: the amount is not indexed, so it is data.
return one(TypeERC20, topicToAddress(at(1)), topicToAddress(at(2)), hexToBigInt(l.Data).String(), "")
case 1:
// Some early ERC-721s indexed nothing and put all three
// arguments in data. Three words and nothing else is the
// signature of that shape.
d := strings.TrimPrefix(l.Data, "0x")
if len(d) == 192 {
return one(TypeERC721,
topicToAddress("0x"+d[:64]), topicToAddress("0x"+d[64:128]),
"1", hexWord("0x"+d[128:192]))
}
}
case TopicTransferSingle:
// TransferSingle(address operator, address from, address to,
// uint256 id, uint256 value) — operator is topics[1].
d := strings.TrimPrefix(l.Data, "0x")
if len(l.Topics) >= 4 && len(d) >= 128 {
return one(TypeERC1155, topicToAddress(at(2)), topicToAddress(at(3)),
hexToBigInt("0x"+d[64:128]).String(), hexWord("0x"+d[:64]))
}
case TopicTransferBatch:
// TransferBatch(..., uint256[] ids, uint256[] values) — one movement
// per pair, each needing its own row so no id is lost to the batch.
if len(l.Topics) >= 4 {
ids, values := decodeBatchData(l.Data)
from, to := topicToAddress(at(2)), topicToAddress(at(3))
out := make([]TokenTransfer, 0, len(ids))
for i, id := range ids {
v := "0"
if i < len(values) {
v = values[i]
}
t := one(TypeERC1155, from, to, v, hexWord(id))[0]
t.ID = fmt.Sprintf("%s-%d-%d", l.TxHash, l.LogIndex, i)
out = append(out, t)
}
return out
}
case TopicERC404ERC20Transfer:
if len(l.Topics) >= 3 {
return one(TypeERC20, topicToAddress(at(1)), topicToAddress(at(2)), hexToBigInt(l.Data).String(), "")
}
case TopicERC404ERC721Transfer:
// from/to indexed, id in data.
if len(l.Topics) >= 3 {
return one(TypeERC721, topicToAddress(at(1)), topicToAddress(at(2)), "1", hexWord(l.Data))
}
}
return nil
}
// Token type spellings. Hyphenated, because that is what the API emits and
// what luxfi/explore's TokenType union reads.
const (
TypeERC20 = "ERC-20"
TypeERC721 = "ERC-721"
TypeERC1155 = "ERC-1155"
)
// hexWord renders a uint256 as the 32-byte hex word the chain encodes it as:
// "0x" and 64 lower-case hex digits. It accepts either form — a hex word
// straight off a topic, or the decimal an ABI array decoder produced — so a
// token id has one stored shape whichever event carried it.
//
// The stored shape is the chain's own encoding rather than decimal because
// evm_token_balances keys ownership on (token_address, address, token_id):
// a fixed-width word sorts numerically, cannot collide, and needs no
// canonicalization pass over rows already written. Readers convert to decimal
// at the API edge, where people see it.
func hexWord(s string) string {
s = strings.TrimSpace(s)
if s == "" {
return ""
}
n, ok := new(big.Int).SetString(strings.TrimPrefix(s, "0x"), base(s))
if !ok || n.Sign() < 0 {
return ""
}
return fmt.Sprintf("0x%064x", n)
}
func base(s string) int {
if strings.HasPrefix(s, "0x") {
return 16
}
return 10
}
evm/transfer_test.go
+167
View File
@@ -0,0 +1,167 @@
package evm
import (
"testing"
"time"
)
// The three logs the Lux Genesis collection emitted when it minted, copied
// from eth_getLogs against C-Chain (chain 96369) at
// 0x9e04fc57c20b2ee45627c4aa280eb471f2ca6ea5. Real logs, so a decoder that
// only works on hand-written ones fails here.
var genesisMints = []Log{
{
TxHash: "0xac5b05e51c38ebfec88a743a0f2815d5082da105a679035ecea73a07e80717e3", BlockNumber: 1095748,
Address: "0x9e04fc57c20b2ee45627c4aa280eb471f2ca6ea5",
Topics: []string{
"0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",
"0x0000000000000000000000000000000000000000000000000000000000000000",
"0x00000000000000000000000055be906ece0552797752e2894d9d7ef952414def",
"0x0000000000000000000000000000000000000000000000000000000000000000",
},
Data: "0x",
},
{
TxHash: "0x10f128e73ab9a1c0fedc9b459870a7cc5e8f45c2ee2a384f13be5ed7e3dbbb8a", BlockNumber: 1095749,
Address: "0x9e04fc57c20b2ee45627c4aa280eb471f2ca6ea5",
Topics: []string{
"0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",
"0x0000000000000000000000000000000000000000000000000000000000000000",
"0x00000000000000000000000055be906ece0552797752e2894d9d7ef952414def",
"0x0000000000000000000000000000000000000000000000000000000000000001",
},
Data: "0x",
},
{
TxHash: "0xb5a96c04e0b9f8520bc6992d1e98e409eee0ca82f8f28460d304843a02ccff8d", BlockNumber: 1095750,
Address: "0x9e04fc57c20b2ee45627c4aa280eb471f2ca6ea5",
Topics: []string{
"0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",
"0x0000000000000000000000000000000000000000000000000000000000000000",
"0x00000000000000000000000067bd7c7c3dbc53b8bde7d9888f90961d7af3d1a6",
"0x0000000000000000000000000000000000000000000000000000000000000002",
},
Data: "0x",
},
}
// The whole collection is three items with ids 0, 1 and 2. If the decoder
// drops the id, all three of these rows read alike and the collection has no
// items — the failure this test exists to catch.
func TestGenesisMintsKeepTheirIDs(t *testing.T) {
wantOwner := []string{
"0x55be906ece0552797752e2894d9d7ef952414def",
"0x55be906ece0552797752e2894d9d7ef952414def",
"0x67bd7c7c3dbc53b8bde7d9888f90961d7af3d1a6",
}
seen := map[string]bool{}
for i, l := range genesisMints {
got := decodeTransfers(l, time.Unix(0, 0))
if len(got) != 1 {
t.Fatalf("log %d: got %d transfers, want 1", i, len(got))
}
tr := got[0]
if tr.TokenType != TypeERC721 {
t.Errorf("log %d: type %q, want %q", i, tr.TokenType, TypeERC721)
}
if want := hexWord(string(rune('0' + i))); tr.TokenID != want {
t.Errorf("log %d: id %q, want %q", i, tr.TokenID, want)
}
if tr.To != wantOwner[i] {
t.Errorf("log %d: to %q, want %q", i, tr.To, wantOwner[i])
}
if tr.Value != "1" {
t.Errorf("log %d: value %q, want 1", i, tr.Value)
}
if seen[tr.TokenID] {
t.Errorf("log %d: id %q already seen — ids are being collapsed", i, tr.TokenID)
}
seen[tr.TokenID] = true
}
}
// ERC-20 and ERC-721 Transfer share a signature, so topic0 cannot tell them
// apart. Arity can: the ERC-721 id is indexed and the ERC-20 amount is not.
func TestArityTellsTheTokenKindApart(t *testing.T) {
const sig = "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"
from := "0x0000000000000000000000001111111111111111111111111111111111111111"
to := "0x0000000000000000000000002222222222222222222222222222222222222222"
amount := "0x0000000000000000000000000000000000000000000000000de0b6b3a7640000"
fungible := decodeTransfers(Log{Topics: []string{sig, from, to}, Data: amount}, time.Unix(0, 0))
if len(fungible) != 1 || fungible[0].TokenType != TypeERC20 {
t.Fatalf("three topics should read as %s, got %+v", TypeERC20, fungible)
}
if fungible[0].Value != "1000000000000000000" {
t.Errorf("amount %q, want decimal 1000000000000000000", fungible[0].Value)
}
if fungible[0].TokenID != "" {
t.Errorf("an ERC-20 row carries no item id, got %q", fungible[0].TokenID)
}
item := decodeTransfers(Log{Topics: []string{sig, from, to, amount}, Data: "0x"}, time.Unix(0, 0))
if len(item) != 1 || item[0].TokenType != TypeERC721 {
t.Fatalf("four topics should read as %s, got %+v", TypeERC721, item)
}
if item[0].TokenID != amount {
t.Errorf("id %q, want the fourth topic %q", item[0].TokenID, amount)
}
}
// A batch moves several ids in one log. Folding them into a single row would
// lose every id but one.
func TestBatchYieldsARowPerItem(t *testing.T) {
data := "0x" +
"0000000000000000000000000000000000000000000000000000000000000040" + // ids offset
"00000000000000000000000000000000000000000000000000000000000000a0" + // values offset
"0000000000000000000000000000000000000000000000000000000000000002" + // len(ids)
"0000000000000000000000000000000000000000000000000000000000000007" +
"0000000000000000000000000000000000000000000000000000000000000008" +
"0000000000000000000000000000000000000000000000000000000000000002" + // len(values)
"0000000000000000000000000000000000000000000000000000000000000003" +
"0000000000000000000000000000000000000000000000000000000000000004"
got := decodeTransfers(Log{
Topics: []string{
TopicTransferBatch,
"0x0000000000000000000000009999999999999999999999999999999999999999",
"0x0000000000000000000000001111111111111111111111111111111111111111",
"0x0000000000000000000000002222222222222222222222222222222222222222",
},
Data: data,
}, time.Unix(0, 0))
if len(got) != 2 {
t.Fatalf("got %d rows, want one per id", len(got))
}
for i, want := range []struct{ id, value string }{{"7", "3"}, {"8", "4"}} {
if got[i].TokenID != hexWord(want.id) {
t.Errorf("[%d] id %q, want %q", i, got[i].TokenID, hexWord(want.id))
}
if got[i].Value != want.value {
t.Errorf("[%d] amount %q, want %q", i, got[i].Value, want.value)
}
if got[i].TokenType != TypeERC1155 {
t.Errorf("[%d] type %q, want %q", i, got[i].TokenType, TypeERC1155)
}
}
if got[0].ID == got[1].ID {
t.Errorf("both rows share the primary key %q — one would overwrite the other", got[0].ID)
}
}
// hexWord is what makes an id from a topic and an id from an ABI array the
// same stored value.
func TestHexWordTakesEitherEncoding(t *testing.T) {
want := "0x0000000000000000000000000000000000000000000000000000000000000042"
for _, in := range []string{"66", "0x42", want} {
if got := hexWord(in); got != want {
t.Errorf("hexWord(%q) = %q, want %q", in, got, want)
}
}
for _, in := range []string{"", " ", "0xzz", "-1"} {
if got := hexWord(in); got != "" {
t.Errorf("hexWord(%q) = %q, want empty", in, got)
}
}
}
explorer/format.go
+63 -2
View File
@@ -380,18 +380,79 @@ func formatLog(l map[string]any) map[string]any {
}
}
// tokenID renders a stored token id as the decimal number people use.
//
// The indexer stores an id as the chain encoded it — a 32-byte hex word off
// topics[3] — because that is what keys ownership. Nobody says they own
// "0x0000…0002", so the conversion happens here, once, at the edge. A decimal
// already in the column is passed through, so a store written by either
// indexing path reads the same. Empty means the row is not an NFT.
func tokenID(v any) any {
s, _ := v.(string)
if b, ok := v.([]byte); ok {
s = string(b)
}
s = strings.TrimSpace(s)
if s == "" {
return nil
}
n, ok := new(big.Int).SetString(strings.TrimPrefix(s, "0x"), idBase(s))
if !ok {
return nil
}
return n.String()
}
// tokenKey is tokenID inverted: the decimal id a caller puts in a URL, back
// into the 32-byte hex word the store is keyed on. Returns "" for anything
// that is not a uint256.
func tokenKey(s string) string {
n, ok := new(big.Int).SetString(strings.TrimPrefix(strings.TrimSpace(s), "0x"), idBase(s))
if !ok || n.Sign() < 0 || n.BitLen() > 256 {
return ""
}
return fmt.Sprintf("0x%064x", n)
}
func idBase(s string) int {
if strings.HasPrefix(strings.TrimSpace(s), "0x") {
return 16
}
return 10
}
// formatTokenTransfer formats a token transfer row.
//
// `total` carries the amount for a fungible token and the item id for an
// NFT, which is why its shape follows the token type rather than being one
// fixed set of keys — the same split luxfi/explore reads as Erc20TotalPayload
// / Erc721TotalPayload / Erc1155TotalPayload. An ERC-721 row has no amount
// worth printing (it is always one) and an id that is the whole point; an
// ERC-20 row is the reverse. Flattening the two into an amount-shaped
// payload is what turned every NFT movement into an identical row.
func formatTokenTransfer(t map[string]any) map[string]any {
// Column-spelling variants between luxfi/indexer evm_token_transfers
// ("tx_hash", "value") and Blockscout-legacy ("transaction_hash",
// "amount"). Fall through both so the response works against either.
kind, _ := col(t, "token_type", "type").(string)
id := tokenID(col(t, "token_id"))
total := map[string]any{"value": fmtNum(col(t, "value", "amount")), "decimals": nil}
switch {
case kind == "ERC-721" || kind == "ERC721":
total = map[string]any{"token_id": id, "token_instance": nil}
case kind == "ERC-1155" || kind == "ERC1155":
total["token_id"] = id
total["token_instance"] = nil
}
return map[string]any{
"from": map[string]any{"hash": bytesToHex(col(t, "from_addr", "from_address_hash", "from_address"))},
"to": map[string]any{"hash": bytesToHex(col(t, "to_addr", "to_address_hash", "to_address"))},
// Same TokenInfo contract as formatToken, narrowed to what a transfer
// row carries — so the key is address_hash here too.
"token": map[string]any{"address_hash": bytesToHex(t["token_address"]), "type": t["token_type"]},
"total": map[string]any{"value": fmtNum(col(t, "value", "amount")), "decimals": nil},
"token": map[string]any{"address_hash": bytesToHex(col(t, "token_address", "token_contract_address_hash")), "type": col(t, "token_type", "type")},
"total": total,
"log_index": t["log_index"],
"block_number": t["block_number"],
"transaction_hash": bytesToHex(col(t, "tx_hash", "transaction_hash")),
explorer/instance_test.go
+277
View File
@@ -0,0 +1,277 @@
package explorer
import (
"database/sql"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"path/filepath"
"testing"
)
// The Lux Genesis collection as it stands on C-Chain: three items, minted to
// two addresses, ids 0, 1 and 2. Rows are exactly what the indexer wrote for
// blocks 1095748-1095750, ids in the 32-byte form the chain encoded them in.
//
// These tests pin the two things a collection page cannot exist without: a
// transfer that says which item moved, and a list of the items themselves.
func genesisDB(t *testing.T, withInstances bool) string {
t.Helper()
path := filepath.Join(t.TempDir(), "indexer.db")
db, err := sql.Open("sqlite3", "file:"+path+"?_journal_mode=WAL")
if err != nil {
t.Fatalf("open: %v", err)
}
defer db.Close()
const nft = "0x9e04fc57c20b2ee45627c4aa280eb471f2ca6ea5"
id := func(n int) string { return fmt.Sprintf("0x%064x", n) }
stmts := []string{
`CREATE TABLE evm_blocks (number INTEGER PRIMARY KEY, hash TEXT, parent_hash TEXT, nonce TEXT,
miner TEXT, difficulty TEXT, total_difficulty TEXT, size INTEGER, gas_limit INTEGER,
gas_used INTEGER, base_fee TEXT, timestamp TIMESTAMP, tx_count INTEGER)`,
`CREATE TABLE evm_transactions (hash TEXT PRIMARY KEY, block_number INTEGER, timestamp TIMESTAMP)`,
`CREATE TABLE evm_addresses (hash TEXT PRIMARY KEY, tx_count INTEGER DEFAULT 0)`,
`CREATE TABLE evm_tokens (address TEXT PRIMARY KEY, name TEXT, symbol TEXT, decimals INTEGER,
total_supply TEXT, token_type TEXT, holder_count INTEGER DEFAULT 0)`,
`CREATE TABLE evm_token_transfers (id TEXT PRIMARY KEY, tx_hash TEXT, log_index INTEGER,
block_number INTEGER, token_address TEXT, token_type TEXT, from_addr TEXT, to_addr TEXT,
value TEXT, token_id TEXT, timestamp TIMESTAMP)`,
`CREATE TABLE evm_token_balances (token_address TEXT, address TEXT, token_id TEXT DEFAULT '',
value TEXT DEFAULT '0', token_type TEXT DEFAULT '', PRIMARY KEY (token_address, address, token_id))`,
`CREATE TABLE evm_logs (id TEXT PRIMARY KEY, address TEXT, block_number INTEGER)`,
fmt.Sprintf(`INSERT INTO evm_tokens VALUES ('%s','Lux Genesis','GENESIS',0,'3','ERC-721',3)`, nft),
}
if withInstances {
stmts = append(stmts,
`CREATE TABLE evm_token_instances (token_address TEXT, token_id TEXT, uri TEXT DEFAULT '',
uri_state TEXT DEFAULT '', PRIMARY KEY (token_address, token_id))`,
// Read from the contract: it concatenates its base onto a value
// that is already a full URL. Recorded as the chain gave it.
fmt.Sprintf(`INSERT INTO evm_token_instances VALUES
('%s','%s','https://lux.town/nfts/https://lux.town/nfts/validator.mov','ok'),
('%s','%s','https://lux.town/nfts/https://lux.town/nfts/validator.mov','ok')`,
nft, id(0), nft, id(1)),
)
}
mints := []struct {
tx string
block int
to string
id int
}{
{"0xac5b05e51c38ebfec88a743a0f2815d5082da105a679035ecea73a07e80717e3", 1095748, "0x55be906ece0552797752e2894d9d7ef952414def", 0},
{"0x10f128e73ab9a1c0fedc9b459870a7cc5e8f45c2ee2a384f13be5ed7e3dbbb8a", 1095749, "0x55be906ece0552797752e2894d9d7ef952414def", 1},
{"0xb5a96c04e0b9f8520bc6992d1e98e409eee0ca82f8f28460d304843a02ccff8d", 1095750, "0x67bd7c7c3dbc53b8bde7d9888f90961d7af3d1a6", 2},
}
for _, m := range mints {
stmts = append(stmts,
fmt.Sprintf(`INSERT INTO evm_token_transfers VALUES ('%s-0','%s',0,%d,'%s','ERC-721',
'0x0000000000000000000000000000000000000000','%s','1','%s',datetime('now'))`,
m.tx, m.tx, m.block, nft, m.to, id(m.id)),
fmt.Sprintf(`INSERT INTO evm_token_balances VALUES ('%s','%s','%s','1','ERC-721')`, nft, m.to, id(m.id)),
)
}
for _, s := range stmts {
if _, err := db.Exec(s); err != nil {
t.Fatalf("exec %.60s: %v", s, err)
}
}
return path
}
func genesisServer(t *testing.T, withInstances bool) *httptest.Server {
t.Helper()
srv, err := NewStandaloneServer(Config{
IndexerDBPath: genesisDB(t, withInstances),
ChainID: 96369,
ChainName: "Lux C-Chain",
CoinSymbol: "LUX",
})
if err != nil {
t.Fatalf("NewStandaloneServer: %v", err)
}
t.Cleanup(srv.Close)
ts := httptest.NewServer(srv.Handler())
t.Cleanup(ts.Close)
return ts
}
func fetch(t *testing.T, ts *httptest.Server, path string) (map[string]any, int) {
t.Helper()
resp, err := http.Get(ts.URL + path)
if err != nil {
t.Fatalf("GET %s: %v", path, err)
}
defer resp.Body.Close()
var out map[string]any
json.NewDecoder(resp.Body).Decode(&out)
return out, resp.StatusCode
}
func rows(t *testing.T, body map[string]any) []map[string]any {
t.Helper()
raw, ok := body["items"].([]any)
if !ok {
t.Fatalf("no items array in %v", body)
}
out := make([]map[string]any, len(raw))
for i, v := range raw {
out[i] = v.(map[string]any)
}
return out
}
const genesisAddr = "0x9e04fc57c20b2ee45627c4aa280eb471f2ca6ea5"
// Three ERC-721 movements are three different items. A response that gives
// them all the same payload is a feed of "NFT #?" rows.
func TestTransfersCarryTheItemID(t *testing.T) {
ts := genesisServer(t, true)
got := rows(t, mustGet(t, ts, "/v1/explorer/token-transfers"))
if len(got) != 3 {
t.Fatalf("got %d transfers, want 3", len(got))
}
seen := map[string]bool{}
for _, row := range got {
total, ok := row["total"].(map[string]any)
if !ok {
t.Fatalf("no total on %v", row)
}
id, ok := total["token_id"].(string)
if !ok || id == "" {
t.Fatalf("token_id missing from %v — the item id is being dropped", total)
}
if seen[id] {
t.Errorf("id %q appears twice; every mint moved a different item", id)
}
seen[id] = true
// An ERC-721 payload names the item, not an amount.
if _, ok := total["value"]; ok {
t.Errorf("ERC-721 total should carry no amount, got %v", total)
}
}
for _, want := range []string{"0", "1", "2"} {
if !seen[want] {
t.Errorf("id %q missing; chain says the collection minted 0, 1 and 2", want)
}
}
}
// Every item of the collection, each owned by whoever received it last.
func TestInstancesListTheCollection(t *testing.T) {
ts := genesisServer(t, true)
got := rows(t, mustGet(t, ts, "/v1/explorer/tokens/"+genesisAddr+"/instances"))
if len(got) != 3 {
t.Fatalf("got %d items, want the 3 the chain minted", len(got))
}
wantOwner := map[string]string{
"0": "0x55be906ece0552797752e2894d9d7ef952414def",
"1": "0x55be906ece0552797752e2894d9d7ef952414def",
"2": "0x67bd7c7c3dbc53b8bde7d9888f90961d7af3d1a6",
}
for _, it := range got {
id, _ := it["id"].(string)
owner, ok := it["owner"].(map[string]any)
if !ok {
t.Fatalf("item %q has no owner", id)
}
if got, want := owner["hash"], wantOwner[id]; got != want {
t.Errorf("item %q owner %v, want %v", id, got, want)
}
if it["is_unique"] != true {
t.Errorf("item %q: an ERC-721 id is held by one address", id)
}
// We store the URI and never fetch what is behind it, so there is
// no image to name and claiming one would be inventing it.
if it["image_url"] != nil || it["metadata"] != nil {
t.Errorf("item %q asserts media it never fetched: %v", id, it)
}
}
}
// An item we read a URI for, an item we did not, and the difference between
// them stated rather than blurred into one silent blank.
func TestURIStateSaysWhichKindOfNothing(t *testing.T) {
ts := genesisServer(t, true)
state := map[string]any{}
uri := map[string]any{}
for _, it := range rows(t, mustGet(t, ts, "/v1/explorer/tokens/"+genesisAddr+"/instances")) {
id, _ := it["id"].(string)
state[id] = it["uri_state"]
uri[id] = it["uri"]
}
if state["0"] != "ok" || uri["0"] == nil {
t.Errorf("item 0 was read from the contract: state %v uri %v", state["0"], uri["0"])
}
if state["2"] != "unread" {
t.Errorf("item 2 was never asked about, want state unread, got %v", state["2"])
}
if uri["2"] != nil {
t.Errorf("item 2 has no URI on record, got %v", uri["2"])
}
}
// A store with no instances table still lists items; it just has no URI for
// any of them. Missing machinery is not the same as an empty collection.
func TestInstancesListWithoutTheURITable(t *testing.T) {
ts := genesisServer(t, false)
got := rows(t, mustGet(t, ts, "/v1/explorer/tokens/"+genesisAddr+"/instances"))
if len(got) != 3 {
t.Fatalf("got %d items, want 3", len(got))
}
for _, it := range got {
if it["uri_state"] != "unread" {
t.Errorf("item %v: want unread, got %v", it["id"], it["uri_state"])
}
}
}
// One item by its own id, which is the page a marketplace links to.
func TestOneItemByID(t *testing.T) {
ts := genesisServer(t, true)
body, code := fetch(t, ts, "/v1/explorer/tokens/"+genesisAddr+"/instances/2")
if code != 200 {
t.Fatalf("status %d, want 200", code)
}
if body["id"] != "2" {
t.Errorf("id %v, want 2", body["id"])
}
owner, _ := body["owner"].(map[string]any)
if owner["hash"] != "0x67bd7c7c3dbc53b8bde7d9888f90961d7af3d1a6" {
t.Errorf("owner %v", owner)
}
if _, code := fetch(t, ts, "/v1/explorer/tokens/"+genesisAddr+"/instances/9"); code != 404 {
t.Errorf("id 9 was never minted, want 404, got %d", code)
}
if _, code := fetch(t, ts, "/v1/explorer/tokens/"+genesisAddr+"/instances/nope"); code != 400 {
t.Errorf("a non-numeric id is not a token id, want 400, got %d", code)
}
}
// A portfolio names the items held, not the encoding they are stored in.
func TestPortfolioIDsAreDecimal(t *testing.T) {
ts := genesisServer(t, true)
got := rows(t, mustGet(t, ts, "/v1/explorer/addresses/0x55be906ece0552797752e2894d9d7ef952414def/tokens"))
if len(got) != 2 {
t.Fatalf("got %d holdings, want 2", len(got))
}
for _, h := range got {
id, _ := h["token_id"].(string)
if id != "0" && id != "1" {
t.Errorf("token_id %q, want a decimal id", id)
}
}
}
func mustGet(t *testing.T, ts *httptest.Server, path string) map[string]any {
t.Helper()
body, code := fetch(t, ts, path)
if code != 200 {
t.Fatalf("GET %s: status %d", path, code)
}
return body
}
explorer/standalone.go
+161 -4
View File
@@ -109,6 +109,16 @@ type tableNames struct {
// rather than count zero rows and call that an answer.
balTokenCol, transferTokenCol string
// transferToCol / transferTypeCol are the recipient and token-kind
// columns on the transfers table, spelled "to_addr"/"token_type" by
// luxfi/indexer and "to_address_hash"/"token_type" by Blockscout-legacy.
transferToCol, transferTypeCol string
// instances is the per-item table evm_token_instances, which holds the
// URI the contract gives an NFT. Empty means no such table: items still
// list, and their URI reads as unread rather than as absent.
instances string
// platform (P-Chain / linear) variant. When platform is true the /blocks
// and /main-page/blocks routes serve the P-Chain block handlers and the
// /validators route is enabled. Set by detectTables when pchain_blocks
@@ -291,6 +301,9 @@ func (s *StandaloneServer) detectTables() {
s.t.balances = s.detectTable(s.t.balances)
s.t.balTokenCol = s.detectColumn(s.t.balances, "token_address", "token_contract_address_hash")
s.t.transferTokenCol = s.detectColumn(s.t.transfers, "token_address", "token_contract_address_hash")
s.t.transferToCol = s.detectColumn(s.t.transfers, "to_addr", "to_address_hash", "to_address")
s.t.transferTypeCol = s.detectColumn(s.t.transfers, "token_type", "type")
s.t.instances = s.detectTable("evm_token_instances", "token_instances")
if c := s.detectColumn(s.t.tokens, "address", "contract_address", "contract_address_hash", "address_hash"); c != "" {
s.t.tokenAddrCol = c
}
@@ -428,7 +441,8 @@ func (s *StandaloneServer) routes() {
m.HandleFunc("GET "+p+"/tokens/{addr}", s.j(s.getToken))
m.HandleFunc("GET "+p+"/tokens/{addr}/holders", s.j(s.tokenHolders))
m.HandleFunc("GET "+p+"/tokens/{addr}/transfers", s.j(s.tokenTransfers))
m.HandleFunc("GET "+p+"/tokens/{addr}/instances", s.j(s.emptyList))
m.HandleFunc("GET "+p+"/tokens/{addr}/instances", s.j(s.tokenInstances))
m.HandleFunc("GET "+p+"/tokens/{addr}/instances/{id}", s.j(s.tokenInstance))
m.HandleFunc("GET "+p+"/tokens/{addr}/counters", s.j(s.tokenCounters))
m.HandleFunc("GET "+p+"/tokens/{addr}/distribution", s.j(s.tokenDistribution))
m.HandleFunc("GET "+p+"/smart-contracts", s.j(s.listContracts))
@@ -1485,7 +1499,7 @@ func (s *StandaloneServer) addrTokens(r *http.Request) (any, int) {
"decimals": fmtNum(b["decimals"]),
},
"value": fmtNum(b["value"]),
"token_id": b["token_id"],
"token_id": tokenID(b["token_id"]),
}
}
return paginatedResponse{Items: items}, 200
@@ -1602,8 +1616,151 @@ func (s *StandaloneServer) allInternalTxs(r *http.Request) (any, int) {
return paginatedResponse{Items: items}, 200
}
func (s *StandaloneServer) emptyList(r *http.Request) (any, int) {
return ep(), 200
// itemQuery selects the items of one collection: every distinct token id the
// collection has ever moved, each carrying the destination of its own most
// recent transfer.
//
// That destination is the current owner, which is what makes the transfer
// table the right source rather than the balance table. Balances hold only
// what someone holds now, so a burned item — sent to the zero address and
// held by nobody — would vanish from a collection that still minted it. The
// last transfer always exists.
//
// suffix narrows the set (one item, or a page of them) and is server-built,
// never caller text.
func (s *StandaloneServer) itemQuery(suffix string) string {
uri := "'' AS uri, '' AS uri_state"
join := ""
if s.t.instances != "" {
uri = "i.uri AS uri, i.uri_state AS uri_state"
join = fmt.Sprintf("LEFT JOIN %s i ON LOWER(i.token_address) = LOWER(t.%s) AND i.token_id = t.token_id",
s.t.instances, s.t.transferTokenCol)
}
return fmt.Sprintf(`SELECT token_id, owner, kind, uri, uri_state FROM (
SELECT t.token_id AS token_id, t.%s AS owner, t.%s AS kind, %s,
ROW_NUMBER() OVER (PARTITION BY t.token_id
ORDER BY t.block_number DESC, t.log_index DESC) AS rn
FROM %s t %s
WHERE LOWER(t.%s) = ? AND t.token_id IS NOT NULL AND t.token_id != ''
) WHERE rn = 1 %s`,
s.t.transferToCol, s.t.transferTypeCol, uri, s.t.transfers, join, s.t.transferTokenCol, suffix)
}
// formatInstance renders one item in the TokenInstance shape luxfi/explore
// reads.
//
// image_url, animation_url, metadata and thumbnails are null because we
// store the URI, not the document behind it — see the indexer's readURI.
// Naming the URI as an image would be inventing a picture we have never
// seen. uri_state says which kind of nothing this is: "ok" with a URI we
// read, "absent" when the contract names none, "unread" when we have not
// asked.
//
// An ERC-1155 id is held by many addresses at once, so it has no single
// owner; that field is null and /tokens/{addr}/holders answers instead.
func formatInstance(m map[string]any, token map[string]any) map[string]any {
id := tokenID(m["token_id"])
kind, _ := m["kind"].(string)
unique := kind != "ERC-1155" && kind != "ERC1155"
uri, _ := col(m, "uri").(string)
state, _ := col(m, "uri_state").(string)
if state == "" {
state = "unread"
}
var owner any
var holder any
if unique {
if h := bytesToHex(m["owner"]); h != "" {
owner = map[string]any{"hash": h}
holder = h
}
}
var uriOut any
if uri != "" {
uriOut = uri
}
return map[string]any{
"id": id,
"is_unique": unique,
"owner": owner,
"holder_address_hash": holder,
"token": token,
"uri": uriOut,
"uri_state": state,
"metadata": nil,
"image_url": nil,
"animation_url": nil,
"external_app_url": nil,
"thumbnails": nil,
}
}
// tokenInfo loads the collection row once, so every item in the response
// carries the same TokenInfo instead of the page making N lookups.
func (s *StandaloneServer) tokenInfo(r *http.Request, addr string) map[string]any {
rows, err := s.q(r, s.tokenSelect(fmt.Sprintf("WHERE LOWER(t.%s) = ?", s.t.tokenAddrCol)), addr)
if err != nil {
return nil
}
defer rows.Close()
maps, _ := scanMaps(rows)
if len(maps) == 0 {
return nil
}
return formatToken(maps[0])
}
// tokenInstances lists the items of a collection.
func (s *StandaloneServer) tokenInstances(r *http.Request) (any, int) {
addr := r.PathValue("addr")
if !isValidHexAddr(addr) {
return ep(), 400
}
addr = strings.ToLower(addr)
if s.t.transferToCol == "" || s.t.transferTypeCol == "" || s.t.transferTokenCol == "" {
return ep(), 200
}
rows, err := s.q(r, s.itemQuery(fmt.Sprintf("ORDER BY token_id LIMIT %d", lim(r))), addr)
if err != nil {
return ep(), 200
}
defer rows.Close()
maps, _ := scanMaps(rows)
token := s.tokenInfo(r, addr)
items := make([]map[string]any, len(maps))
for i, m := range maps {
items[i] = formatInstance(m, token)
}
return paginatedResponse{Items: items}, 200
}
// tokenInstance serves one item by its decimal id.
func (s *StandaloneServer) tokenInstance(r *http.Request) (any, int) {
addr := r.PathValue("addr")
if !isValidHexAddr(addr) {
return ep(), 400
}
key := tokenKey(r.PathValue("id"))
if key == "" {
return map[string]any{"message": "invalid token id"}, 400
}
addr = strings.ToLower(addr)
if s.t.transferToCol == "" || s.t.transferTypeCol == "" || s.t.transferTokenCol == "" {
return map[string]any{"message": "not found"}, 404
}
rows, err := s.q(r, s.itemQuery("AND token_id = ?"), addr, key)
if err != nil {
return map[string]any{"message": "not found"}, 404
}
defer rows.Close()
maps, _ := scanMaps(rows)
if len(maps) == 0 {
return map[string]any{"message": "not found"}, 404
}
return formatInstance(maps[0], s.tokenInfo(r, addr)), 200
}
// ---- CSV Exports ----
migrations/004_nft_marketplace.sql
+8 -2
View File
@@ -1,7 +1,13 @@
-- NFT Marketplace Tables Migration
-- Version: 1.0.0
-- Date: 2025-01-24
-- Description: Adds NFT marketplace indexing for OpenSea Seaport, LooksRare, Blur, Rarible, X2Y2, Zora
--
-- NOT APPLIED. No marketplace contract is deployed on any Lux chain, so there
-- are no sales to index and these tables would only ever be empty. The live
-- API reads the SQLite tables evm/indexer.go creates, not this Postgres
-- schema. Applying it is a decision to take once a venue exists.
--
-- NFT items themselves are indexed and served today; see evm_token_instances
-- and /tokens/{addr}/instances.
--------------------------------------------------------------------------------
-- NFT Orders (Listings, Bids, Offers)