Consensus-correctness fix for the native DEX settlement VM. Root cause of the
launch-blocking forks: order size/price were IEEE-754 float64 on the ZAP wire, so
every validator re-derived the settlement integer via uint64(float)/int64(float*1e8),
whose out-of-range result is implementation-defined per the Go spec and DIFFERS BY
CPU ARCHITECTURE (arm64 saturates to 2^64-1; amd64 wraps to 2^63). Two honest
validators on different archs then derived different reserves/balances/fills from the
SAME ordered block.
F1/F3/F5 — exact-integer size/price end to end:
- pkg/zapwire: Place/Submit/Fill carry price as uint64 fixed-point (×1e8 = PriceInt)
and size as uint64 atomic base units. No float64 on the wire. Removed PutFloat64/
Float64. Decode is exact bytes->uint64 (deterministic on every arch).
- pkg/dchain/execute: decode the exact integers, set the authoritative PriceUnits/
SizeUnits lane; the float Price/Size become derived projections only (never a
source of a consensus value). Reject price==0/size==0/price>maxPriceUnits.
- pkg/dex/persist: the DEXOrder/DEXTrade row stores the EXACT PriceInt (in
DEXPrice.Integer) and EXACT base units (Quantity/Remaining); RowToOrder restores
the integer lane verbatim (identity — no float round-trip) and Filled=Qty-Remaining.
Removed floatToFixedQty/fixedQtyToFloat/floatToFixedPrice (the non-idempotent
round(size*1e8)/Q64.64 encoding that clamped at ~1.845e11 and forked on restart).
- pkg/dex matcher: cross and level-key on the exact integer PriceUnits.
- pkg/dchain/settle: orderLock/floorsToZeroLock take exact integers; removed
sizeToUnits/priceToInt float conversions and the dead restoreIntegerLane.
F2 — commit the custody ledger into the execution root:
- state.ComposeRoot/ExecutionRoot gain a ledgerRoot term (Merkle over balance:/
locked:/orderasset: rows). Validators can no longer finalize agreeing on the book
while holding different money.
F4 — release the ceil-lock/floor-charge residual on full fill:
- decrementMakerReserves now closes a fully-filled maker's reserve and returns the
ceil(size*price)-floor residual to the owner (a filled order can never be
cancelled), through the same unlock path a cancel uses. No more permanently
frozen quote.
PROVEN: the two-phase cross-arch harness (arm64 build / amd64 Rosetta verify) now
derives byte-identical execRoots AND byte-identical custody ledgers for the
identical block (uint64(1e20) still differs at the language level — 0xffff.. vs
0x8000.. — but the exact-integer wire freezes the value in the bytes so no
per-validator float->int conversion remains). Red guards flipped to assert the fix;
conservation/byzantine/batch-parity/KAT/e2e all green (go test ./... -count=1),
go vet clean.
OUT OF SCOPE (flagged, NOT fixed here): redteam_unbacked_deposit_test documents a
SEPARATE critical finding (F9) — dex_deposit credits an arbitrary self-signed amount
with no C-side backing proof, exposed on the public RPC. That needs an access-control/
custody-model decision, not a mechanical fix, and is left for a follow-up.
Co-authored-by: Hanzo Dev <[email protected]>
The matching engine (pkg/lx, 45.5K LOC incl. tests) and the deterministic
core layered on it (pkg/dex) are one matcher spread across two packages
(dex imported lx in 12 files; lx imported neither). Fold lx into dex and
drop the "lx" name entirely — pkg/dex is now the single matcher package.
Pure rename/fold, ZERO logic change. Behavior proven identical: the merged
pkg/dex runs 457 top-level tests (431 former-lx + 26 former-dex) and 489
test funcs (468 + 21), all green — matching, batch (BatchMatchEqualsSequential,
run-to-run determinism, tile boundary), GPU/CPU parity, truncation, and CL
swap math all pass unchanged.
What moved:
- git mv all 101 pkg/lx/*.go + 1 .c into pkg/dex/ (package lx -> package dex).
- lx's swap_test.go -> swap_math_test.go (only filename collision with dex's
existing swap_test.go).
- Rewrote imports repo-wide: 54 pkg/lx import lines removed; 42 external
importers (api, client, consensus, dchain, dpdk, lxgpu, marketdata, test/,
tests/) repointed to pkg/dex with lx.X -> dex.X; the 12 in-package dex files
had lx.X -> bare X. 717 selector lines rewritten. Stale pkg/lx comment refs
updated (pkg/lxgpu left intact).
Collisions resolved (2, both package-level vars):
- ErrNoLiquidity: lx's pool-level error (dexError, used only in swap.go, never
referenced via qualifier) renamed ErrPoolNoLiquidity to match the existing
ErrPool* convention; dex's router-level ErrNoLiquidity (public, has errors.Is
tests) kept.
- priceMultiplierBig: identical after fold (both big.NewInt(PriceMultiplier));
deduped by deleting dex's copy.
Not folded — pkg/orderbook: it is NOT the GPU seam (that is orderbook_gpu.go,
which lives in the engine and folded in here). pkg/orderbook is a zero-importer,
placeholder-cgo island whose 13 core domain identifiers (Order, OrderBook, Trade,
OrderType, OrderStatus, PriceLevel, Buy, Sell, Market, Limit, Filled,
PartiallyFilled, NewOrderBook) collide incompatibly with the engine's. Folding
it verbatim breaks the build; folding via rename reintroduces the exact parallel
duplication this fold removes. Left for owner decision (recommend deletion as
dead code) — see refactor notes.
Follow-on (separate repos, not touched): 19 files under luxfi/dex import
pkg/lx externally — lux/precompile/dex, lux/precompile-dex/dex,
lux/consensus/examples/01-simple-bridge, lux/_consensus-v12537/examples,
lux/cex/pkg/engine — repoint to pkg/dex after this lands.
Verified: CGO_ENABLED=0 GOWORK=off go build ./... clean; go vet ./... clean;
go test ./... -count=1 all green (22 ok / 0 fail).
Co-authored-by: Hanzo Dev <[email protected]>
dexd is now the ONLY command in the repo. One binary, two run modes:
dexd run as a luxd rpcchainvm PLUGIN. luxd execs the plugin from
dexd plugin its PluginDir with NO positional args (engine address arrives
via the runtime.EngineAddressKey env var) and probes the
version with `dexd version`. No-subcommand == plugin is
REQUIRED to preserve that contract — byte-for-byte the old
cmd/dchain behavior (rpc.Serve over the plugin transport).
dexd run run the STANDALONE D-Chain venue: open the on-disk zapdb,
dexd standalone serve the frozen dex_* surface over ZAP, and drive the
consensus sealer loop (WaitForEvent->BuildBlock->Verify->
Accept). -addr/-db flags preserved from the old cmd/dvenue.
Relocated logic (engine code NOT duplicated — pkg/dchain + pkg/lx untouched):
cmd/dchain/{main,runner} -> internal/plugin (plugin.Run)
cmd/dvenue/{main,engine_*} -> internal/standalone (standalone.Run + engineName)
cmd/dexd/main.go = thin run-mode dispatcher
Removed cmd dirs (12): dchain, dvenue, demo, dex-api-server, dex-server,
dexfillproof, dexseed, fix-gateway, gateway, monitor, persistent-server,
price-feed. Removed committed build artifacts: gateway, dex-server-linux.
Build/ship ONLY dexd:
- Dockerfile: canonical CPU image (CGO=0, ghcr.io/luxfi/dex) running
`dexd run`; no GPU/MLX required.
- Dockerfile.dvenue: GPU operator bundle repointed cmd/dvenue -> cmd/dexd.
- Deleted Dockerfile.{gateway,devnet,minimal,dvenue-cpu} and
docker/{Dockerfile,Dockerfile.cuda,backend/Dockerfile,compose-simple.yml}.
- Makefile rewritten to build/run/ship dexd only.
- docker.yml: dropped the dex-gateway image; dex-venue-image.yml: dvenue->dexd.
- Deleted legacy scripts that built removed/long-gone cmds; fixed deploy.sh +
dev-setup.sh; .gitignore ignores root build binaries.
Also included (coupled, environment-applied, settled): the in-progress
clob -> dex/orderbook terminology refactor across pkg/* + sdk/* + tests/*
(identifier + file renames, e.g. RegisterCLOB->RegisterDEX,
source_clob.go->source_orderbook.go). The relocated standalone venue calls the
renamed API, so it lands in the same consistent snapshot.
Verified (GOWORK=off CGO_ENABLED=0): `go build ./cmd/dexd`, `go build ./...`,
`go vet ./...` all exit 0; `go test ./pkg/dchain` (renamed RegisterDEX socket
path) green. GPU/cgo path not compiled locally (no CUDA); its rename mirrors
the verified nocgo twins and preserves the C ABI symbol names.
Adds the missing SERVER side of the FIX trading path and a unified e2e
proving REAL fills over all four DEX access paths against ONE book.
The decomplection: the D-Chain CLOB (pkg/dchain) is the single matcher,
reached over the single frozen clob_* ZAP surface (pkg/zapwire). Every
access path is a thin wire dialect onto that surface — none owns a
matcher. This commit adds the two missing dialects (FIX, a focused WS
venue handler) and the shared seam they translate through.
New:
- pkg/venue: ONE Venue interface (ensure/place/cancel/submit = the
clob_* ops) + ONE production ZAP-backed impl. The seam FIX, WS, and
the 0x9010 precompile all route through. Dials the SAME rpc.Listen
socket dvenue serves.
- pkg/fix: dependency-free FIX 4.4 codec (BodyLength tag 9 + CheckSum
tag 10 computed AND validated; spec-correct emit/parse) + a TCP
acceptor (Logon/Heartbeat/TestRequest/Logout, monotonic seq,
NewOrderSingle->clob_submit->ExecutionReport, OrderCancelRequest->
clob_cancel). Pure protocol front-end onto Venue; no matcher.
- pkg/api/venue_ws.go: minimal gorilla/websocket /ws handler that
translates a JSON order to Venue.Submit and streams fills back —
the WS dialect onto the SAME venue (kept orthogonal to the existing
TradingEngine product WS server).
- cmd/fix-gateway: runs the acceptor in front of a live venue
(-venue clob_* endpoint), the production form of the FIX path.
- pkg/dchain/fourpath_e2e_test.go: stands up ONE venue + ONE LUX/LUSD
book and drives a real limit-order -> match -> FILL over EACH path
(V4 precompile engine_zap frame | ZAP native | FIX | WS) against the
SAME resting asks, asserting real consensus fills, per-path custody
changes, and GLOBAL conservation (available+locked invariant).
V4 leg: exercised at the engine_zap WIRE boundary — the exact 66-byte
clob_submit frame engine_zap.Swap emits (asserted byte-identical to
zapwire.EncodeSubmit) + the V4 BalanceDelta sign convention derived from
the consensus fills. A live luxd EVM (StateDB replay-idempotency, gas,
PoolManager) is what the full on-chain precompile call additionally
needs; the matcher boundary is proven in-process here.
No deps added; no existing files modified (go.mod/go.sum untouched).
Builds CGO_ENABLED=0; race-clean.
NEW pkg/zapwire: pure-Go leaf holding the frozen CLOB ZAP frame (Submit=66/Place=65/
Cancel=40/EnsureMarket=32/Ack=24, FillWireSize=17, float64 codec) so the public precompile
imports the wire format without the cgo/GPU engine. SubmitMarketable (IOC taker): removed
time.Now()+atomic.AddUint64 from the consensus path; trade ts derives from block-derived
order ts; fills carry maker+taker identity. zap_server.go docstring demoted authority->
gateway. nocgo: verifyOrdersCPU returns nil for empty. Verified pkg/zapwire 6/6, full ./... green.
- Fixed insertBid/insertAsk infinite loop when encountering orphaned order IDs
- Added pos++ before continue to ensure loop termination with stale entries
- Added comprehensive edge case tests for orphaned/nil orders in orderbook
- Added extensive tests for pkg/client handlers and channel accessors
- Added tests for websocket server in pkg/api
Coverage improvements:
- pkg/orderbook: 86.5% → 90.5%
- pkg/client: 47.7% → 64.2%
This fixes a critical bug where orphaned entries in bids/asks lists could
cause infinite loops during order insertion.
- Add missing OrderBook interface implementations
- Fix go vet warnings (unreachable code, redundant newline)
- Add build tags for long-running tests
- Create proper Go and C++ orderbook implementations
- Fix clearinghouse test skipping properly
- Format all Go code with gofmt
- Ensure all packages build successfully
- All tests pass with -short flag
- Fix PriceLevels field access (use private field priceLevels)
- Add missing math import in orderbook_extended.go
- Remove unused log import from websocket_server.go
- Fix MarginEngine method calls (use CheckLiquidations instead of GetAllPositions)
- Fix SimpleVault active status check (use member count instead)
- Fix ClearingHouse field name (clearinghouse not clearingHouse)
- Comment out incomplete orderBook integration in vault_simple.go
- Fix broadcast channel type mismatch (convert Message to []byte)
zeekayandHanzo Dev