A TxDeposit MINTS ledger balance, but its authorization was the DEPOSITOR'S OWN
signature (verifyTxSignature required signer == the body's beneficiary), and
handleDeposit exposed it on the same public ZAP route as dex_place/dex_submit. So
any authenticated principal could self-sign a deposit and mint unlimited unbacked
balance, then withdraw it (redteam_unbacked_deposit_test proved 500B minted).
OPTION B (gate to the trusted bridge authority) — chosen over Option A (route all
deposits through the atomic import) because: (1) the trustless atomic import
(TxImport, atomic.go) is only wired in production (runtime + shared memory) and
credits a DIFFERENT account axis (crossChainAccount(C-address)) than the DEX
tx-auth account (Account16) — routing all deposits through it would require wiring
shared memory + reconciling account models across 17 test files; (2) tx.go:47
already documents TxDeposit as the bridge/proxy-credit path — this implements that
intent correctly (the bug was the missing authority check); (3) the trustless
atomic import remains UNCHANGED and available for cross-chain swaps. Both backing
mechanisms now coexist: trustless import for swaps, authority-gated deposit for
bridge-custodied funds.
The fix:
- VM gains depositAuthority (a CONSENSUS config param, vmConfig.DepositAuthority
hex, decoded at Initialize). ZERO => no authority => EVERY TxDeposit rejected
fail-closed (value then enters only via the backed atomic import).
- authorizeTx (the authoritative consensus gate): a TxDeposit is accepted ONLY when
the recovered signer == depositAuthority; else rejected (no mint, no nonce burn).
- verifyTxSignature: a deposit no longer requires signer==beneficiary (the signer is
the authority; the beneficiary is bound via the signed digest). Withdraw is
UNCHANGED — still signer==beneficiary (a user withdraws their own funds).
- submitTx: boundary pre-screen rejects a non-authority deposit before the mempool.
Backing invariant enforced: Σ(ledger credits via TxDeposit) ≤ Σ(bridge-authorized
deposits); combined with the import's Σ(credits) ≤ Σ(consumed C->D objects), NO path
credits the ledger without backing. The authority gate also HARDENS RED #4: a
stranded/self-signed deposit that lands in a later block is rejected, so it cannot
mint; withdraw is untouched (debits own available, clamped — no burn).
TDD: TestRedteam_UnbackedSelfSignedDepositMints now PASSES (self-signed deposit
rejected at boundary + consensus; minted 0, withdrew 0). New
TestBackedAuthoritySignedDepositCredits proves the legitimate authority-signed
deposit credits + withdraws correctly. Test funding migrated to an authority-signed
helper (fundAcct/fundTx); auth_test nonce expectations updated (a funded account's
first own tx is nonce 0 — funding no longer consumes the user's nonce). ALL suites
green (go test ./... -count=1), go vet clean; the F1-F5 cross-arch execRoots + ledgers
still match byte-for-byte across arm64/amd64.
KNOWN FOLLOW-UPS (tracked): (a) chains/dexvm + precompile/dex are separate modules
that also encode the ZAP frame — they need the same exact-integer wire bump to
interoperate; (b) F7 — retract the inflated 12.76B/9.13B/5.60B "ord/s" GPU benchmark
numbers in ~/work/lux/papers (they measure all-cross synthetic fills/s ~71-85x
inflated; the paper's own text already caps a single GPU below 10^9 fills/s).
Co-authored-by: Hanzo Dev <[email protected]>
Consensus-correctness fix for the native DEX settlement VM. Root cause of the
launch-blocking forks: order size/price were IEEE-754 float64 on the ZAP wire, so
every validator re-derived the settlement integer via uint64(float)/int64(float*1e8),
whose out-of-range result is implementation-defined per the Go spec and DIFFERS BY
CPU ARCHITECTURE (arm64 saturates to 2^64-1; amd64 wraps to 2^63). Two honest
validators on different archs then derived different reserves/balances/fills from the
SAME ordered block.
F1/F3/F5 — exact-integer size/price end to end:
- pkg/zapwire: Place/Submit/Fill carry price as uint64 fixed-point (×1e8 = PriceInt)
and size as uint64 atomic base units. No float64 on the wire. Removed PutFloat64/
Float64. Decode is exact bytes->uint64 (deterministic on every arch).
- pkg/dchain/execute: decode the exact integers, set the authoritative PriceUnits/
SizeUnits lane; the float Price/Size become derived projections only (never a
source of a consensus value). Reject price==0/size==0/price>maxPriceUnits.
- pkg/dex/persist: the DEXOrder/DEXTrade row stores the EXACT PriceInt (in
DEXPrice.Integer) and EXACT base units (Quantity/Remaining); RowToOrder restores
the integer lane verbatim (identity — no float round-trip) and Filled=Qty-Remaining.
Removed floatToFixedQty/fixedQtyToFloat/floatToFixedPrice (the non-idempotent
round(size*1e8)/Q64.64 encoding that clamped at ~1.845e11 and forked on restart).
- pkg/dex matcher: cross and level-key on the exact integer PriceUnits.
- pkg/dchain/settle: orderLock/floorsToZeroLock take exact integers; removed
sizeToUnits/priceToInt float conversions and the dead restoreIntegerLane.
F2 — commit the custody ledger into the execution root:
- state.ComposeRoot/ExecutionRoot gain a ledgerRoot term (Merkle over balance:/
locked:/orderasset: rows). Validators can no longer finalize agreeing on the book
while holding different money.
F4 — release the ceil-lock/floor-charge residual on full fill:
- decrementMakerReserves now closes a fully-filled maker's reserve and returns the
ceil(size*price)-floor residual to the owner (a filled order can never be
cancelled), through the same unlock path a cancel uses. No more permanently
frozen quote.
PROVEN: the two-phase cross-arch harness (arm64 build / amd64 Rosetta verify) now
derives byte-identical execRoots AND byte-identical custody ledgers for the
identical block (uint64(1e20) still differs at the language level — 0xffff.. vs
0x8000.. — but the exact-integer wire freezes the value in the bytes so no
per-validator float->int conversion remains). Red guards flipped to assert the fix;
conservation/byzantine/batch-parity/KAT/e2e all green (go test ./... -count=1),
go vet clean.
OUT OF SCOPE (flagged, NOT fixed here): redteam_unbacked_deposit_test documents a
SEPARATE critical finding (F9) — dex_deposit credits an arbitrary self-signed amount
with no C-side backing proof, exposed on the public RPC. That needs an access-control/
custody-model decision, not a mechanical fix, and is left for a follow-up.
Co-authored-by: Hanzo Dev <[email protected]>
The matching engine (pkg/lx, 45.5K LOC incl. tests) and the deterministic
core layered on it (pkg/dex) are one matcher spread across two packages
(dex imported lx in 12 files; lx imported neither). Fold lx into dex and
drop the "lx" name entirely — pkg/dex is now the single matcher package.
Pure rename/fold, ZERO logic change. Behavior proven identical: the merged
pkg/dex runs 457 top-level tests (431 former-lx + 26 former-dex) and 489
test funcs (468 + 21), all green — matching, batch (BatchMatchEqualsSequential,
run-to-run determinism, tile boundary), GPU/CPU parity, truncation, and CL
swap math all pass unchanged.
What moved:
- git mv all 101 pkg/lx/*.go + 1 .c into pkg/dex/ (package lx -> package dex).
- lx's swap_test.go -> swap_math_test.go (only filename collision with dex's
existing swap_test.go).
- Rewrote imports repo-wide: 54 pkg/lx import lines removed; 42 external
importers (api, client, consensus, dchain, dpdk, lxgpu, marketdata, test/,
tests/) repointed to pkg/dex with lx.X -> dex.X; the 12 in-package dex files
had lx.X -> bare X. 717 selector lines rewritten. Stale pkg/lx comment refs
updated (pkg/lxgpu left intact).
Collisions resolved (2, both package-level vars):
- ErrNoLiquidity: lx's pool-level error (dexError, used only in swap.go, never
referenced via qualifier) renamed ErrPoolNoLiquidity to match the existing
ErrPool* convention; dex's router-level ErrNoLiquidity (public, has errors.Is
tests) kept.
- priceMultiplierBig: identical after fold (both big.NewInt(PriceMultiplier));
deduped by deleting dex's copy.
Not folded — pkg/orderbook: it is NOT the GPU seam (that is orderbook_gpu.go,
which lives in the engine and folded in here). pkg/orderbook is a zero-importer,
placeholder-cgo island whose 13 core domain identifiers (Order, OrderBook, Trade,
OrderType, OrderStatus, PriceLevel, Buy, Sell, Market, Limit, Filled,
PartiallyFilled, NewOrderBook) collide incompatibly with the engine's. Folding
it verbatim breaks the build; folding via rename reintroduces the exact parallel
duplication this fold removes. Left for owner decision (recommend deletion as
dead code) — see refactor notes.
Follow-on (separate repos, not touched): 19 files under luxfi/dex import
pkg/lx externally — lux/precompile/dex, lux/precompile-dex/dex,
lux/consensus/examples/01-simple-bridge, lux/_consensus-v12537/examples,
lux/cex/pkg/engine — repoint to pkg/dex after this lands.
Verified: CGO_ENABLED=0 GOWORK=off go build ./... clean; go vet ./... clean;
go test ./... -count=1 all green (22 ok / 0 fail).
Co-authored-by: Hanzo Dev <[email protected]>
Fixes the fail-safe liveness edge the Byzantine suite found: a self-cross-heavy
stream could wedge one later submit at BuildBlock with "resting order missing
full settlement identity (orderuser)" / "insufficient locked balance" (seed 42,
~step 153; same class as seed-4242).
Root cause: pkg/lx/orderbook.go tryMatchImmediateLocked's self-trade skip removed
the self-crossed maker from the IN-MEMORY book (removeOrder + delete Orders +
ordersMap.Delete) without a matching consensus state write. The in-memory book
feeds the execution root via BookToRows(ob), but the durable order:/orderuser:/
reserve rows were left untouched — so the two diverged, and a later cross tripped
the settle-time invariants.
Fix (one and one way — route through the existing cancel-persist path):
- pkg/lx/orderbook.go: the matcher RECORDS each self-cancelled maker id
(ob.selfCanceled, reset per match) and exposes DrainSelfCanceled(). No fill is
produced; self-trade prevention is unchanged.
- pkg/dchain/execute.go: applySubmit drains the self-cancelled ids into
applyResult.SelfCanceled.
- pkg/dchain/block.go: extract releaseOrderReserve() — the single canonical
cancel-persist primitive (release reserve locked->available, delete reserve +
orderuser: rows, fail-closed on missing identity). TxCancel now calls it; a
submit's self-cancelled makers call it too, and execute deletes their order:
rows in lockstep. In-memory and durable state now move together within the block.
Properties preserved:
- Determinism: the self-cross block's execRoot is UNCHANGED (BookToRows already
excluded the maker); every validator drops the same maker + releases the same
reserve. No fork.
- Conservation-exact: the released reserve moves locked->available; sum unchanged,
no mint/burn.
- Self-trade prevention: self-crosses still never trade (zero fills).
- Anti-theft: owner resolved exclusively via the orderuser: index, fail-closed.
TDD (CGO_ENABLED=0 GOWORK=off GOFLAGS=-mod=mod):
- TestByzantineConservation_AdversarialMatcherStream: 1280 ops, 10 -> 0 fail-safe
refusals (now asserts zero as a regression guard); I=A+L + no-fork every op.
- TestByzantineMatcher_SelfCrossCancelsRestingMakerConsistently: new pin — the
partial-fill -> self-cross -> external-cross wedge shape now cancels the maker
(8 locked base -> available), later crosses run clean, collateral withdrawable.
- Full pkg/lx, pkg/dchain, pkg/consensus suites green; chains/dexvm unaffected.
Co-authored-by: Hanzo Dev <[email protected]>
Add deterministic, seeded (no wall-clock) Byzantine-fault tests across the
two settlement-critical layers: the D-Chain CLOB matcher (single source of
truth, re-executed by every validator) and the quorum-finality boundary.
pkg/dchain/byzantine_matcher_test.go — matcher safety invariants + the
forged-execRoot keystone:
- self-cross never trades an account against itself (no value move)
- price-time priority: best price fills first; worse maker untouched
- over-fill clamped to min(order qty, resting liquidity)
- cancelled/withdrawn order is never filled (no phantom liquidity)
- a byzantine proposer claiming ANY execRoot other than the honest
re-derivation is rejected by every validator; last-accepted unchanged
(subsumes conservation/self-cross/over-fill/phantom fabrication)
pkg/dchain/byzantine_determinism_test.go — determinism under reordering:
- N independent validators replaying the same block bytes stay in lockstep
(identical execRoot + balances) across a fill
- re-verify is a pure function of block bytes (no local wall-clock)
- the consensus timestamp is bound into the execRoot: a MITM timestamp
tamper is rejected (block is a tamper-evident unit)
pkg/dchain/byzantine_conservation_property_test.go — conservation + no-fork
under adversarial matcher load (8 seeds x 160 ops): self-cross, deep
multi-price books, oversized takers, cancel-then-cross; asserts
I=A+L per asset and proposer/validator agreement after every op. Surfaces
(counts, does not hide) fail-safe matcher refusals and proves the market
recovers (no permanent wedge). See FINDING below.
pkg/consensus/byzantine_quorum_test.go — f<n/3 finality boundary with REAL
Ed25519 validator signatures:
- executable BFT bound: a safe+live quorum exists iff f<=floor((n-1)/3)
- n=4 tolerates 1, FAILS at 2; n=7 tolerates 2, FAILS at 3; n=10 3/4
- equivocation: no double-finalize at f*; safety unsalvageable beyond
- withholding: live at f*, stalls beyond (safety over liveness)
- forged/outsider/tampered votes never contribute to a quorum
FINDING (liveness, fail-safe — NOT a safety break): a self-cross-heavy
stream can drive the matcher to REFUSE one later submit at build
("resting order missing full settlement identity (orderuser)" /
"insufficient locked balance"). Root cause: pkg/lx/orderbook.go
tryMatchImmediateLocked self-trade skip mutates the in-memory book
(removeOrder + delete Orders/ordersMap) without a matching consensus write,
so the in-mem book and persisted orderuser:/reserve rows transiently
diverge. It is FAIL-SAFE (no mint, no burn, no fork; conservation intact)
and PER-TX (the market recovers on the next clean cross). Deterministic
repro: seed 42, ~step 153, market LUX/LUSD.
Co-authored-by: Hanzo Dev <[email protected]>
The GPU backend was a build-time PLACE: five mutually-exclusive build-tag
files (orderbook_gpu_{cuda,hip,vulkan,metal,stub}.go), each redefining
gpuMatchOrder + dexBackendLabel and each #cgo pkg-config'ing its own
per-vendor bundle (lux-dex-orderbook-{cuda,hip,metal,vulkan}). Which kernel
you got was frozen at compile time by -tags.
Make it a runtime VALUE. Collapse the five into the single cgo file
orderbook_gpu.go (//go:build cgo, no os/vendor tags), which links one
unified pkg-config bundle, lux-gpu, and calls the shared C ABI in dex_gpu.h:
int lux_gpu_dex_match_order(...) // rc 0 ok, -2 unsupported -> CPU
const char* lux_gpu_backend_name() // runtime-selected: cuda|hip|metal|cpu
liblux_gpu picks the best device at runtime (accel SelectBestBackend). The
MatchOrderGPU dispatcher, errOrderBookGPUUnsupported sentinel, GPU_DISABLE
policy, and byte-parity padding normalization are unchanged. dexBackendLabel
now reports the runtime backend via C.GoString(C.lux_gpu_backend_name()).
The !cgo CPU fallback (orderbook_nogpu.go) and MatchOrderCPU oracle are
untouched; the seam is now exactly cgo vs !cgo, nothing finer.
Verified: CGO_ENABLED=0 full pkg-config build+vet+`go test -run Match|Batch`
green. cgo path built+linked+run against a throwaway stub lux-gpu.pc (ABI
type-checks; rc -2 -> CPU fallback; backend label wired) — real GPU kernel
parity is CI-verified against liblux_gpu on the arcd runners.
Co-authored-by: Hanzo Dev <[email protected]>
Replace one-cross-per-dispatch matching with a batch matcher that crosses
many independent books in a single call. Orders for different books share
no state and are matched concurrently on a bounded worker pool; orders for
the same book stay strictly sequential so price-time priority / FIFO — the
determinism invariant — is never reordered. Batches larger than a tile
(BatchTileSize=65536 books) are split into sequential tiles so no single
dispatch is unbounded (an unbounded multi-million-book dispatch once hung a
GPU).
The per-book cross reuses the existing MatchOrderCPU oracle (one matching
primitive, batched — not a second copy), on the flat DEXOrder/DEXTrade ABI
the GPU kernel and the shadow gate already use, so a GPU batch backend slots
behind the same BatchMatcher interface: flatten a tile into one device
buffer, one GPU block per book, dispatch once per tile. The CPU matcher is
the byte-for-byte authority a GPU backend is compared against, exactly as
MatchOrderCPU is the authority for MatchOrderGPU.
Determinism contract, proved by batch_match_test.go against an independent
one-at-a-time sequential reference: the fills, remaining, and post-cross
book state are byte-identical to the per-order path for the same input
ordering — regardless of worker scheduling or tile size. Race-clean under
-race. CPU speedup on a match-bound block (2048 books x 128 depth x 8
sweeping takers): 11.8ms -> 4.5ms (2.6x) on 10 cores.
Co-authored-by: Hanzo Dev <[email protected]>
CORRECTION 2 (AMM GPU rip): the constant-product xy=k AMM runs on-chain in
the EVM/consensus — GPU-accelerating it was a hallucination. Remove the GPU
AMM surface: pkg/lx/amm_gpu{,_cuda,_metal,_stub}.go + amm_nogpu.go (the
BatchEvalConstantProduct dispatcher) + amm_gpu_test.go; strip the AMM bridge
(AMMSwap, fnAMMSwap, call_amm_xyk_batch, LuxAmmReservePair) from pkg/lxgpu so
it bridges only the order-book path; drop the obsolete GPU_Batch sub-bench.
PRESERVED: pkg/lx/amm.go (ConstantProductOut/ReservePair/BatchEvalConstantProductCPU,
the 128-bit-exact CPU math) — it is the deterministic pricing used by the DEX
router's AMM source (pkg/dex/source_amm.go), untouched.
CORRECTION 1 (clob→dex): rename Go identifiers clobGPUMatch→dexGPUMatch,
clobShadowInput→dexShadowInput, runCLOBShadow→runDEXShadow,
captureCLOBShadowLocked→captureDEXShadowLocked, clobBackendLabel→dexBackendLabel,
clobScenario→dexScenario, TestCLOB*→TestDEX*, MatchDivergence.Kind "clob"→"dex".
CORRECTION 3 (host drivers): add orderbook_gpu_hip.go (cgo && linux && hip &&
!vulkan) and orderbook_gpu_vulkan.go (cgo && linux && vulkan), mirroring the
CUDA binding; narrow orderbook_gpu_cuda.go to (cgo && linux && !hip && !vulkan)
so exactly one backend compiles per linux tag combo (verified via go list).
Verified: CGO_ENABLED=0 go build ./... GREEN; go test ./pkg/lx ./pkg/dex
./pkg/dchain GREEN; gofmt clean; cgo bridge compiles; build-tag exclusivity
confirmed for ''/hip/vulkan.
orderbook_gpu_metal.go (cgo && darwin) links the single-thread Metal
matcher (luxcpp/dex/gpu/metal/dex_orderbook_driver.mm) via the
lux-dex-orderbook-metal pkg-config bundle. The kernel walks book_indices
in one deterministic GPU lane -- byte-identical to MatchOrderCPU by
construction, the property matcher-at-Verify requires.
Proven on M1 Max: the 76-scenario determinism corpus + the existing
GPU/CPU parity test pass with backend=metal (real device), GPU==CPU
byte-identical over fills + remaining + resulting book.
Honest perf (M1 Max): the single-cross flat ABI is dispatch-bound, so the
1-lane Metal kernel is ~400x slower than the CPU matcher (500us vs ~1-4us).
The GPU win lives in batching many books per dispatch, not one serial
cross; the kernel is wired for correctness/determinism + defense-in-depth
today. Narrowed orderbook_gpu_stub.go to !darwin (Metal now owns darwin).
Closes the DRIVE gap in the native C<->D settlement seam: the EXECUTION half
(atomic.go executeImport/executeExport) was real and unit-tested, but NOTHING in
production generated TxImport/TxExport — only the e2e harness, playing the keeper
by hand. So a real 0x9999 SubmitSwapIntent staged a C->D object that nothing
imported: no taker funding, no crossing, no D->C settlement object, no DEXFill.
drive.go is the proposer-side DRIVE — the keeper, but in-consensus. During normal
block production BuildBlock now:
- IMPORT: enumerates the committed C->D intent objects in shared memory (tagged
with the fixed SeamPendingTrait so they are discoverable owner-agnostically),
filters those already imported (committed seamintent: escrow), sorts by
intentID, and prepends a TxImport for each. Double-import-guarded by the
escrow filter AND executeImport's own replay-reject.
- EXPORT: enumerates the open intent escrows whose owner now holds realized
OUTPUT in committed state (the output gate => no premature refund, no
export-before-trade), and appends a TxExport binding owner/asset/amount/spent
exactly as 0x9999 ImportSettlement reads (byte-identical seam wire). The
spent witness = original principal - leftover input. execute then CLOSES the
escrow once the taker's cross-chain account is drained (one-shot per swap).
Consensus-safe BY DESIGN (the seam's own note, atomic.go:47-55): both legs are
pure, deterministic functions of committed state (shared memory + the escrow
index + the ledger) — no wallclock, no mempool, no nondeterministic iteration, no
external I/O — so every validator re-derives the identical effects in execute and
a divergent proposer is rejected on the tx root. Enumeration results are sorted
into a total order so two proposers on the same committed partition build
byte-identical blocks. Keeper-less and co-located-correct: no external venue.
The seam wire and executeImport/executeExport are REUSED unchanged. autoDriveSeam
is enabled exactly when the seam is wired (runtime shared memory + resolved
C-Chain id); BuildBlock builds a pure-seam block on an empty mempool and
WaitForEvent signals seam work so the engine drives it.
Tests (drive through the REAL BuildBlock, not manual mempool.Add):
- TestDrive_AutonomousImportCrossExport: the VM's OWN BuildBlock imports the
intent, the taker's signed order crosses, the VM's OWN BuildBlock exports the
D->C object the modeled 0x9999 Phase B consumes once; conservation holds;
escrow closed.
- TestDrive_NoExportBeforeImportOrTrade, TestDrive_ReplayImportedOnce,
TestDrive_DeterministicOrdering (two proposers => identical blocks).
- The existing manual-drive seam mechanics tests stay green WITH the drive on,
proving the autonomous drive emits byte-identical txs to a correct keeper
(deterministic convergence + tx dedup).
Companion (coordinated cutover, NOT in this commit): precompile/dex
native_staging.go collectRange must add SeamPendingTrait to the swap intent's
Element.Traits (one line) so a production 0x9999 intent is discoverable here;
pinned to the drive.go constant by the existing wire-golden discipline. The e2e
models that corrected flush. dex v1.9.2 readiness.
- TestCLOBShadow_GPUMatchesCPU_Corpus: 76 scenarios (empty book, full/
partial fills, multi-level sweep, price barriers, exact-cross boundary,
300-deep book past a GPU block, market sweep, 64 random seeds) assert
GPU == CPU byte-identical over fills + remaining + resulting book.
- TestCLOBGate_FailClosed_DivergentGPU: a corrupt GPU (off by one unit) is
discarded, the CPU authority is committed unchanged, a divergence is
recorded -- proving a divergent GPU result can never reach state.
- TestCLOBGate_NoFork_CPUvsGPU: EngineCPU and EngineGPUVerified commit
byte-identical fills (the anti-fork property).
- clobBackendLabel() per backend so the corpus log states GPU vs fallback.
Wire a GPU CLOB matcher into the dexvm match path (applySubmit ->
SubmitMarketableVerified) behind a determinism gate that runs the GPU
kernel as a CHECKED SHADOW of the pure-Go authority:
- EngineCPU (default): byte-for-byte the original SubmitMarketable.
- EngineGPUVerified (node-local opt-in, LUX_DEX_MATCH_ENGINE=gpu-verified):
capture the pre-cross opposite side atomically with the match, run the
GPU kernel on those exact inputs, byte-compare fills+remaining+book
against the CPU authority, record the outcome, and commit the CPU
result REGARDLESS. A GPU result that differs by one byte is recorded as
a divergence and discarded -- structurally impossible to commit.
Because the committed bytes are always the CPU authority's, a mixed
GPU/CPU validator set cannot fork. Divergences surface as a metric
(MatchShadowSnapshot) and a loud log (node ERROR via the VM).
Port the atomic import/export into the native dexvm matcher so the in-luxd
D-Chain PRODUCES and CONSUMES the same primary-network shared-memory objects
the 0x9999 precompile speaks — the owner's chosen model (D matches under its
own consensus; C-Chain 0x9999 settles by consuming a node-local atomic object
D produces; no BLS receipt, no external venue).
- atomic.go: the seam. encodeSeamObject/decodeSeamObject reproduce the
precompile's 69-byte wire (rail|owner|asset|amount|spent) BYTE-FOR-BYTE, and
DeriveIntentID its id derivation — ONE wire across precompile/proxy/native.
executeImport consumes a C->D object (Get under the C chain + shared-memory
Remove) and funds the taker's native D account; executeExport debits the
matched proceeds and writes a D->C object keyed exactly as ImportSettlement
reads it, bound to the per-intent escrow (owner/asset/remaining) so Phase B's
owner/asset/amount + per-taker-cap checks pass. commitSeamAtomic applies the
removes/puts atomically with the block overlay (the platformvm acceptor
pattern). No root change needed: the in-consensus matcher makes the atomic
ops a pure function of the already-rooted txs (unlike the proxy's live venue).
- vm.go: capture init.Runtime -> shared-memory handle + C-Chain id.
- tx.go: TxImport (intentID) / TxExport (intentID|asset|amount|spent), both
authorized by the cross-chain object, not a signature.
- block.go: execute drives the seam in the account-scoped lane; Accept commits
via commitSeamAtomic.
- atomic_seam_e2e_test.go: two-phase e2e over REAL shared memory — Phase A intent
-> native D import + a real crossing under D consensus -> native D export ->
Phase B (the exact 0x9999 ImportSettlement consume) credits once; replay
rejected; over-export beyond the locked principal rejected; conservation holds.
Plus a golden-wire test pinning encodeSeamObject to the SAME cross-repo golden
the precompile pins ImportSettlement against (native == precompile == proxy).
go.sum: re-record re-tagged luxfi/[email protected] (stale hash blocked the build).
go.mod: luxfi/runtime indirect -> direct (atomic.go uses it).
Adds the dex_get_nonce read so a signed client (luxfi/maker) resyncs its nonce to the venue's expected-next on restart/redeploy — without it a fresh process starts at nonce 0 and is rejected once the account has transacted. Also hex-encodes orderJSON.User: the raw 16-byte settlement id is not valid UTF-8, so the prior string emit was lossy (json -> U+FFFD) and an owner could never round-trip; hex matches dex_get_nonce's account field (one account encoding) and lets the maker's startup reconcile cancel its own stale orders. Re-realigns luxfi/[email protected] go.sum (re-tagged upstream; keeps reverting). Verified: dchain tests green.
Saves the GPU-venue build work: pkg/lx/orderbook_cuda*→orderbook_gpu*, the
clob→gpu cgo binding rename needed for the GB10 build, and cmd/dexfill (the
real-fill ZAP client). Checkpoint — build-verify pending.
The public permissionless DEX must be provably KYC-free. Compliance/KYC
belongs in lux-private (dex/compliance/sovereign), never in luxfi/dex.
- delete compliance.go + compliance_types.go: EligibilityChecker.CanTrade
pre-trade gate, StandardEligibilityChecker (KYC/AML/accreditation/
jurisdiction enforcement), InvestorProfile/InvestorStatus, PEPStatus,
TransferRestriction, ExemptionType, ComplianceError + codes.
- delete market_config.go: the SecuritiesMarket wrapper existed solely to
gate AddOrder on EligibilityChecker.CanTrade. Zero production consumers —
the permissionless engine uses AdvancedOrderBook directly.
- funds.go: remove preCheck compliance hook + AML/sanctions/compliance
fields + ComplianceGates; permissionless default is FundsStatusApproved.
- reconciliation.go: relocate neutral AssetClass + ReconciliationMode types
here (cross-venue metadata, not gates); drop CEXComplianceID /
ReportedToRegulator / RegulatoryReportID fields.
- tests updated to match.
Green (CGO off): go build ./..., go vet + go test ./pkg/lx ./pkg/dchain.
dexd is now the ONLY command in the repo. One binary, two run modes:
dexd run as a luxd rpcchainvm PLUGIN. luxd execs the plugin from
dexd plugin its PluginDir with NO positional args (engine address arrives
via the runtime.EngineAddressKey env var) and probes the
version with `dexd version`. No-subcommand == plugin is
REQUIRED to preserve that contract — byte-for-byte the old
cmd/dchain behavior (rpc.Serve over the plugin transport).
dexd run run the STANDALONE D-Chain venue: open the on-disk zapdb,
dexd standalone serve the frozen dex_* surface over ZAP, and drive the
consensus sealer loop (WaitForEvent->BuildBlock->Verify->
Accept). -addr/-db flags preserved from the old cmd/dvenue.
Relocated logic (engine code NOT duplicated — pkg/dchain + pkg/lx untouched):
cmd/dchain/{main,runner} -> internal/plugin (plugin.Run)
cmd/dvenue/{main,engine_*} -> internal/standalone (standalone.Run + engineName)
cmd/dexd/main.go = thin run-mode dispatcher
Removed cmd dirs (12): dchain, dvenue, demo, dex-api-server, dex-server,
dexfillproof, dexseed, fix-gateway, gateway, monitor, persistent-server,
price-feed. Removed committed build artifacts: gateway, dex-server-linux.
Build/ship ONLY dexd:
- Dockerfile: canonical CPU image (CGO=0, ghcr.io/luxfi/dex) running
`dexd run`; no GPU/MLX required.
- Dockerfile.dvenue: GPU operator bundle repointed cmd/dvenue -> cmd/dexd.
- Deleted Dockerfile.{gateway,devnet,minimal,dvenue-cpu} and
docker/{Dockerfile,Dockerfile.cuda,backend/Dockerfile,compose-simple.yml}.
- Makefile rewritten to build/run/ship dexd only.
- docker.yml: dropped the dex-gateway image; dex-venue-image.yml: dvenue->dexd.
- Deleted legacy scripts that built removed/long-gone cmds; fixed deploy.sh +
dev-setup.sh; .gitignore ignores root build binaries.
Also included (coupled, environment-applied, settled): the in-progress
clob -> dex/orderbook terminology refactor across pkg/* + sdk/* + tests/*
(identifier + file renames, e.g. RegisterCLOB->RegisterDEX,
source_clob.go->source_orderbook.go). The relocated standalone venue calls the
renamed API, so it lands in the same consistent snapshot.
Verified (GOWORK=off CGO_ENABLED=0): `go build ./cmd/dexd`, `go build ./...`,
`go vet ./...` all exit 0; `go test ./pkg/dchain` (renamed RegisterDEX socket
path) green. GPU/cgo path not compiled locally (no CUDA); its rename mirrors
the verified nocgo twins and preserves the C ABI symbol names.
The package lived at dex/pkg/dexcore (package dexcore) with 'dexcore:' error
prefixes, while every sibling is short (api, client, dchain, consensus) and every
other dex error says 'dex:'. The 'core' qualifier is redundant inside the dex
module. Rename pkg/dexcore → pkg/dex (package dex), unify error prefixes
dexcore: → dex:, update the 5 importers (dexseed, dexfillproof). Tests green
(pkg/dex, 1.398s); dvenue + dexvm plugin rebuild; e2e swap re-proven (seed 12
orders + cross → fill @ 12.5125).
The venue V of the HTLC atomic-swap design: a pure relay that serves a
maker-signed quote book and sequences a swap session between taker U (the
secret-holder) and a CEX-backed maker M, validating each typed signed
transition — role bindings (HTLC_A recipient=U/refund=M, HTLC_B
recipient=M/refund=U), the cross-leg hashlock, and the timeout ordering
T_B > T_A + Δ (maker locks first/shorter; spec Δ table: Lux<->BTC 2h,
EVM<->EVM 5m).
Non-custody is STRUCTURAL: the Coordinator holds no private key of any
asset, is the recipient/refund of no HTLC, and has no fund-moving path —
TestNonCustody reflects over it and fails if any field is a key type. After
the hashlock commit V is irrelevant to safety; its crashing or lying can
only deny service or relay a stale quote.
Composes pkg/htlc for the shared hashlock/preimage and authenticates every
message through crypto.Ecrecover directly (recoverAddr) with its own venue
domain tags — a leaf, so the relay never pulls the GPU CLOB matcher. Builds
and passes go test -race off a non-GPU box.
The single place a Lux cross-chain atomic swap is constructed across all
three legs, composed from one shared primitive:
- secret.go — Secret [32]byte, Hashlock() = sha256(secret), constant-time
VerifyPreimage. THE source of the SHA-256 hashlock every leg shares (one
32-byte preimage unlocks BTC, the LP-90A0 swap precompile, and the EVM
LXSwapHTLC.sol); keccak is explicitly excluded.
- bitcoin.go — the fixed P2WSH HTLC script (OP_IF SHA256/EQUALVERIFY/CHECKSIG
// OP_ELSE CLTV/DROP/CHECKSIG), P2WSHAddress, and claim/refund witnesses
signed with real BIP143 segwit-v0 sighash via btcd txscript (no hand-rolled
sighash; low-S RFC6979 sigs).
- evm.go — lock/claim/refund calldata; selectors DERIVED from the canonical
ABI signatures and asserted equal to the precompile's 0x4da2c728 /
0x84cc9dfb / 0x7249fbb6.
Tests (-race) include real txscript.Engine execution proofs: claim spends
with the correct preimage and fails with a wrong one; refund spends at/after
locktime and fails before / with CLTV disabled. btcd pinned v0.24.2 (v0.26.0
split chaincfg/wire/txscript into /v2 modules and breaks the build).
A rested maker carries an 8-byte-truncated User (from persist.go userToUint64);
a fresh taker carries the full 16-byte identity. The marketable matcher's raw
User== self-trade check missed the same-account self-cross, desyncing the int/
float quantity lanes -> maker reserve drained + orderuser row deleted -> later
cross stranded settlement -> conservation broke. Add selfTradeKey() (one fold,
used by both place + marketable paths) and skip-and-cancel the self-maker leg.
settle.go arithmetic untouched. seed 4242 re-included + conserves.
Every state-mutating D-Chain tx now carries a signature + per-account nonce,
verified in consensus before any ledger mutation: recovered signer must equal
the asserted 16-byte user, and the digest binds tx type + nonce + full body
(so the withdraw ref and cancel orderId are signed). secp256k1 + ML-DSA-65,
fail-closed. Previously any caller of the venue RPC could move anyone's funds.
Conservation, identity-width, and ownership invariants preserved; the
signed-order primitive that existed without a caller is now the auth boundary.
Tests across ingest/read/zapingest/e2e adapted to sign; no checks weakened.
poolIDForSymbol was defined only in localnet_e2e_test.go (//go:build !cgo) but
read_test.go (no build tag) used it, so the package test binary failed to
compile under CGO_ENABLED=1 (undefined: poolIDForSymbol). Move the helper to an
untagged poolid_helper_test.go — it is pure-Go (a byte copy) and belongs in both
test binaries. Both CGO_ENABLED=1 (read/purity) and CGO_ENABLED=0 (venue/4-path/
ingest/conservation) suites now compile and pass.
go.sum: luxfi/zap v0.8.10 content hash updated to the canonical registry hash
(h1:QKNTAsenkke+...) after an upstream re-tag; 'go mod verify' = all modules
verified. Regenerated, not bypassed.
Structural regression lock for the consensus write-path fix (luxfi/consensus
fix/fetch-on-unknown-vote). The wedge was a SHARED-engine bug, not a dexvm bug:
the dexvm has — and must keep — ZERO networking/gossip primitives. It parses,
verifies, accepts, and matches blocks under consensus but NEVER propagates them;
propagation, vote distribution, and ancestor fetch live in luxfi/consensus.
TestDChainVM_IsPureBlockChainVM walks the package's own non-test .go files and
fails (with file:line) if any references AppGossip, SendAppGossip, PushQuery,
PullQuery, SendChits, common.Sender, or RequestAncestors. Verified to FAIL on an
injected sentinel and PASS on the real tree (12 source files, zero hits).
CGO_ENABLED=0 go test ./pkg/dchain/... green.
The D-Chain DEX wire methods now use the dex_ namespace (dex_place,
dex_submit, dex_cancel, dex_ensure_market, dex_open_market, dex_deposit,
dex_withdraw, dex_get_markets/book/orders/trades) per the luxd
service-prefix convention (eth_/avm./platform.). On the D-Chain the DEX
IS the CLOB (the AMM lives in C-Chain contracts, not this VM), so the
public wire exposes the chain's purpose, not the internal engine name.
The internal Go identifiers (clobMethods, MethodPlace, the CLOB engine
name) are unchanged — only the on-wire string VALUES move. Covers BOTH
transports via the one clobMethods table (ZAP RegisterCLOB + HTTP
ingest.go httpHandlers) plus read.go GET endpoints. Forwards-only (DEX
not live yet). dexseed + dexfillproof clients aligned.
zapwire + dchain tests green; four-path + venue E2E pass.
The serve goroutine deliberately uses context.Background() (its lifetime is
the VM's, cancelled by Shutdown via zapIngestCancel, not the short-lived
Initialize ctx), so the ctx parameter was never used. Remove it — no slop.
Functionally identical to v1.5.17; the param carried no behavior.
The in-luxd native D-Chain plugin served HTTP only (the dev left the ZAP
socket to standalone cmd/dvenue: handler.go said 'the in-luxd plugin does
NOT use this path'). Owner transport decision: the ZAP socket is the
CANONICAL native ingestion in-luxd (the HFT path — binary wire, streaming,
co-located on the pod), HTTP stays as the compat surface (web/exchange-api/
debug). Both fold over the ONE clobMethods core; transport is orthogonal to
consensus, so it is safe to serve both — an order from either transport hits
the identical submitTx -> mempool.Add -> Verify-match path, and validators
replay from accepted block BYTES, never the wire.
zapingest.go: the VM owns its transports. Initialize parses init.Config for
an optional zapIngestAddr and, when set, opens rpc.Listen co-located in the
plugin process bound to clobMethods (reusing RegisterCLOB, the existing
socket seam) — served concurrently, closed on Shutdown. Enabled by DATA (the
chain config the operator/genesis sets), NOT a build tag or forked entry
point: one binary, one core, the transport set is a per-environment knob.
Empty addr => HTTP-compat-only (in-process tests + cmd/dvenue, which runs its
own socket, are unaffected). Malformed config fails fast at Initialize.
TDD (all !cgo):
- TestZAPIngestCrossingOrdersMatchInBlock: an order over the VM's OWN
config-started socket (no manual RegisterCLOB) -> consensus -> committed
fill.
- TestZAPIngestSharesOneCoreWithHTTP: maker rests over ZAP, taker crosses
over HTTP against the SAME book -> proves one core, two transports.
- TestParseConfig / TestNoZAPIngestWhenUnset / TestZAPIngestStopsOnShutdown:
config contract (forwards-only), no-op path, clean idempotent teardown.
Additive: RegisterCLOB unchanged; CreateHandlers/ingest.go unchanged; the
runner (cmd/dchain) unchanged — the socket is a VM concern driven by config.
The mess was three layers defending a self-inflicted footgun:
(a) cmd/clobverify + cmd/fourpath-live SEED ascii-of-symbol synthetic
markets (poolID = ASCII of "LUX/LUSD"), orders with no orderuser row;
(b) internal/localguard refused those tools off-loopback;
(c) VM.assertOrderUserCoverage BRICKED the chain at boot if a custody
market had a resting order with no orderuser row.
Decomplect-perfect (NOT another guard):
1. DELETE cmd/clobverify + cmd/fourpath-live. Real markets come from
permissionless OpenMarket + real ERC-20s + the maker; the real-market
e2e supersedes them. cmd/dexseed (canonical, structurally synthetic-safe)
stays.
2. DELETE internal/localguard (+ test) — it existed ONLY to fence the
deleted seeders.
3. The settlement-identity invariant already holds BY CONSTRUCTION at order
creation: lockOrderSpend writes orderuser: and putOrderRow writes order:
to the SAME versiondb overlay, committed in ONE atomic Block.Accept
overlay.Commit(); an unfunded place never rests. So a resting custody
order without its orderuser row is UNREPRESENTABLE in committed state ->
the boot scan was provably dead -> DELETE VM.assertOrderUserCoverage.
The single remaining settlement-identity guard is the consensus-time
fail-closed in settleOrderEffects (ErrMissingSettlementUser), which can
never fire on honest committed state.
TDD:
- TestCommittedStateNeverHasOrderWithoutIdentity: drives the real commit
path (rest + reject + fill); committed order: set == orderuser: set.
- TestRestartDoesNotBrickBoot: hand-corrupts state the commit path cannot
produce; boot now SUCCEEDS (no brick); consensus-time guard still fails
closed when a fill touches the corrupt order.
- All prior settlement-identity regression tests still pass.
Result: zero guards, illegal state unrepresentable. Removes the footgun
instead of guarding the footgun we created.
HIGH-1/HIGH-2 (integrity): assetid.go was UNTRACKED — it defines AssetKind, the
{EVMNative,ERC20,UTXO} consts, EVMNativeMarker, DeriveAssetID, and CanonicalRefFor:
the foundational real-asset identity primitive the whole value path keys on. A fresh
worktree at the prior dexcore commit failed to build (undefined: AssetKind ×5 in
place.go/resolver.go). Commit it (+ assetid_test.go) so pkg/dexcore COMPILES and the
value-path branch is reproducible.
assetid.go derives the consensus-native 32-byte AssetID from where an asset really
lives — (networkID, sourceChainID, kind, canonicalRef) — via a length-prefixed,
domain-separated ("lux:dex:asset:v1") SHA-256 fold, byte-identical to
chains/dexvm/registry.DeriveAssetID, so a swap-derived id (this home) and a registered
id (the registry) name the SAME asset by the SAME id, by construction.
MED-1 (golden KAT): assetid_test.go pins AssetIDGoldenVectors — fixed
(networkID, chainID, kind, ref) → expected 32-byte id for all three kinds (incl UTXO).
The mirror test in chains/dexvm/registry asserts the IDENTICAL bytes, locking the
byte-identity the resolve↔register equivalence depends on; a drift in either home's
preimage discipline fails CI in both. Domain-separation test confirms ERC20 vs
EVM_NATIVE never collide and a zero-address ERC-20 / empty source chain is refused.
Verified: CGO_ENABLED=0 go build ./pkg/dexcore clean; full dexcore suite green.
The value-path admission port becomes PERMISSIONLESS + PUBLIC per the owner's clean
model: an asset trades IFF its canonical identity verifies on-chain, with NO
manifest-membership requirement.
- resolver.go: AssetResolver.ResolveEnabled -> ResolveAsset (proves canonical identity
permissionlessly; refuses only malformed/wrong-network/explicitly-disabled/synthetic,
never 'not pre-registered'). RequireEnabledAsset -> RequireResolvedAsset.
ErrAssetNotAdmitted -> ErrAssetNotResolved (identity-resolve refusal). The EXTCODESIZE
OnChainAssetVerifier is now documented as the AUTHORITATIVE reality check (a synthetic
asset has no code -> ErrAssetNotOnChain). Fail-closed errors (no resolver/verifier)
unchanged.
- place.go OpenMarketChecked: same two orthogonal steps (resolve identity -> verify
on-chain reality), now permissionless: any REAL (base, quote) pair opens; only
malformed/synthetic/disabled refused. Value keying (left-pad) unchanged.
- resolver_test.go: replace the membership-allowlist fakeResolver with a permissionless
resolver. New positive proof (fails without the change): a REAL unregistered ERC-20
opens a market. Synthetic base now refused by the reality gate (ErrAssetNotOnChain),
preserving the safety property via the authoritative check, not an allowlist. Disabled
(kill switch) and malformed refused at resolve; nil resolver/verifier fail closed.
DeriveAssetID untouched (byte-identical). Conservation keying untouched.
The CLOB source's SettleFills credits the taker's dexcore available with the proceeds;
the AMM source did not (only SpendLocked + journal record), leaving the two sources
asymmetric on the D-surface. A caller draining the taker's transient by the journal
amount then over-drained for an AMM leg (the proceeds were never in available). Credit
the taker's available in the AMM source too, so after ANY swap the taker's available
holds exactly the journal-recorded proceeds. Conservation holds across (ledger + pool
reserves): input left the taker's locked into the pool base reserve; output left the
pool quote reserve into the taker's available — both legs balance to the unit.
The cEVM home's taker is a 20-byte EVM address; the prior 16-byte AccountID would
fold a 20-byte address lossily, so two distinct addresses sharing a leading 16-byte
prefix could collide as one dexcore account and drain each other's resting-order
proceeds (cross-user drain on the account axis). Widen AccountID to the full 32-byte
injective settlement identity both homes map into without a lossy fold: the cEVM
left-pads the 20-byte address (accountFromAddress), dchain left-pads its 16-byte wire
user. 32 bytes is birthday-safe at 2^128 and injective for every real identity.
matcherHint renders the full 32 account bytes as the matcher's User/UserID string —
the hint is ONLY a matching collision handle, never authoritative for value (settlement
resolves the taker from the explicit AccountID and the maker from the orderuser: row).
Replaces the trimTrailingZeros round-trip (no longer needed: settlement never reads the
hint back).
The caller-agnostic core both DEX homes call (the standalone D-Chain VM over a
versiondb overlay, and the in-process 0x9999 cEVM precompile over 0x9999 storage),
so there is ONE settlement algorithm and ONE router in ONE place:
- ledger.go: custody schema + conserving balance primitives (available/locked) over
a generic Store, full-injective AccountID(16)/AssetID(32) keys.
- settle.go: integer-exact fill conservation + fail-closed maker identity
(orderuser: index, ErrMissingSettlementUser, never the 8-byte handle).
- router.go: the on-chain smart-order-router — CLOB-first, AMM-fallthrough,
best-execution across V4+AMM with split; taker-authenticated MEV floor. Source
abstraction leaves room for an RFQ/external-CEX source LATER without reshaping.
RouteClass{PublicCLOB,FHEPrivate} reserves the staged private-pool route.
- journal.go: the JOINT JOURNAL — C-surface deltas recorded for the caller to mirror
onto EVM balances; D-surface writes go straight to the Store. Both-or-neither.
- source_clob.go / source_amm.go: the two liquidity sources (in-process matcher;
in-cEVM V2/V3 constant-product, 128-bit exact).
- place.go: maker resting-order place/cancel (custody-gated, fail-closed cancel).
- book.go / root.go: book schema + rebuild; RFC-6962 DEX execution root.
REAL assets only (assetID = left-pad of the 20-byte token address); no synthetic
ticker ids anywhere. 16 tests pass race-clean: conservation, fail-closed ownership
(handle-collision non-value-bearing), deterministic replay across independent
stores, two-layer MEV floor, CLOB/AMM routing + split, BUY-requires-limit.
Reword comparison docs, comments, and design prose to Lux's own terms
(native DEX core, EVM adapter / 0x9999 V4 PoolManager, native HFT surface,
one consensus-finalized C+D transition). Rename the test broker-venue label
and oracle source weight to neutral names. Behavior-preserving: no logic,
ABI, or wire format changes.
The D-Chain VM exposed only write methods (clob_place/submit/cancel/...) over
its HTTP surface; there was no way to read committed chain state, so a fill
proven on the proposer could not be verified as replicated across validators,
and native fills (trade: rows) could not feed markets-display.
Add a READ surface (read.go) orthogonal to the write surface (handler.go):
- clob_get_trades?since=<h>&limit=<n> committed trade: log (markets feed +
cross-validator replication probe)
- clob_get_orders?market=<hex> resting order: rows for a market
- clob_get_markets every market: id/symbol/assets/summary
- clob_get_book?market=<hex>&depth=<n> aggregated bid/ask ladder + totals
Reads are JSON, GET, served via CreateHandlers on the SAME /ext/bc/<D>/dex/
route group as the writes. They read committed state directly under vm.mu —
NO mempool, NO block, NO consensus round — so a read can never change a fill or
balance (proven: TestRead_ZeroConsensusImpact). Every response carries the
accepted-block head (height, lastAccepted, root) so a cross-validator diff is
anchored to an exact block: two nodes at the same height+root MUST return
identical trades.
Trades/markets read durable rows (the committed truth, identical on every node
at a height); orders/book read the in-RAM book (a deterministic fold of the
committed order: rows). The 80-byte trade row carries no poolID (frozen GPU
ABI), so clob_get_trades is the GLOBAL height-ordered log — exactly right for a
single-market devnet (the verification case) and for an indexer that scopes
fills to a market by the order ids it tracks.
State helpers (state.go): readTrades (height-ordered durable scan), listMarkets.
lx: export DEXPrice.Float() + FixedQtyToFloat (one conversion; the internal
fixedPriceToFloat/fixedQtyToFloat now delegate).
TDD (read_test.go, all green, race-clean):
- TradesAfterCommittedFill: a matched cross is queryable with right px/size/side
- MarketsAndOrdersAndBook: the three other endpoints reflect committed state
- ZeroConsensusImpact: 25 reads advance neither height nor mempool
- CrossValidatorIdenticalTrades: two INDEPENDENT VMs (separate DBs) that
Accept the SAME block bytes return BYTE-IDENTICAL trades JSON + same root —
the in-process model of the live 5-validator replication question
- GenesisEmpty / BadParams: clean empty answers; boundary validation (400/405)
Removed the temporary zzz_fillrepro_test.go (its header said DELETE after;
superseded by read_test.go's committed-fill coverage).
The luxd plugin transport (vm/rpc) resolves Block.Accept by ID via VM.GetBlock —
unlike Block.Verify, which carries the full block bytes (ParseBlock). The dexvm's
GetBlock only indexed ACCEPTED blocks, so GetBlock(builtID) for a block the engine
had just built returned ErrNotFound; the engine's self-finalize Accept
(finalizeOwnProposal) became a silent no-op, the clob_* submitTx waiter
(handler.go) hung forever, and the consensus forwarder wedged inside the hung
Accept so NO further D-Chain blocks were ever built. The EVM survives the same
ID-only Accept because geth resolves built-but-unaccepted blocks from its block
cache; the dexvm had no equivalent.
Add a processingBlocks index (built/verified-but-unaccepted blocks): BuildBlock
and ParseBlock record into it (ParseBlock returns the stable instance so Verify's
stashed overlay is the one Accept commits), GetBlock consults it after
acceptedBlocks, and Accept/Reject prune it. Smallest change that makes the
plugin Accept(ID) round-trip resolve the same instance built/verified — no
transport, engine, or matcher change.
TDD: TestVMAcceptViaGetBlock drives the exact plugin path BuildBlock ->
GetBlock(id) -> Accept (proven to FAIL without the lookup: 'not found, the
production stall'); TestVMParseVerifyAcceptByID covers the follower
ParseBlock -> Verify -> GetBlock(id) -> Accept. Full pkg/dchain suite green
CGO=0. Every prior test accepted the *Block from BuildBlock directly, skipping
the ID-resolve path the live node uses.
The block store is the in-RAM acceptedBlocks map; only bootstrapGenesis and
Accept populate it. On restart loadHead restored the head META (lastAccepted id,
height, root) but NOT the block object, so once the chain advanced past genesis
the engine's GetBlock(lastAccepted) — called immediately after Initialize —
returned ErrNotFound and luxd failed VM init with 'zap initialize: remote error:
get last accepted block: not found', taking the D-Chain down on the first
validator to restart.
Persist the head block's canonical bytes (metaHeadBlock) in the SAME atomic
overlay/batch as the head meta (Accept) and in bootstrapGenesis. loadHead now
reconstructs the head block via parseBlock (id/height asserted to match the head
pointer) and puts it back into acceptedBlocks. A legacy DB with no persisted head
block recovers genesis from the deterministic genesisBlock() (DRY with
bootstrapGenesis) and errors loudly for a non-genesis head rather than silently
resetting the chain.
TDD: TestVMRestartGetBlockHead advances two blocks, restarts (fresh VM, same DB),
and asserts GetBlock(head) returns the block — the exact engine call that failed;
proven to FAIL without the acceptedBlocks repopulation ('not found'), pass with.
TestVMRestartLegacyGenesisDB covers the no-metaHeadBlock genesis recovery. Full
pkg/dchain suite green CGO=0.
The in-luxd native D-Chain VM (pkg/dchain) bootstrapped consensus but never
exposed order ingestion: VM.CreateHandlers returned an empty map, so luxd
mounted nothing and /ext/bc/D/* 404'd. RegisterCLOB (the clob_* -> submitTx ->
mempool -> consensus -> Verify-match path) was only reachable over the ZAP
socket transport, which only cmd/dvenue (the abandoned standalone venue) wires.
Wire ingestion the idiomatic luxd way: CreateHandlers now returns one HTTP
handler per CLOB method (/dex/clob_place, /dex/clob_submit, ...), each adapting
the EXACT same rpc.RawHandler RegisterCLOB binds (one method->handler table,
two transports — vm.clobMethods). luxd mounts each as an exact route under
/ext/bc/<chainID> (and the D alias) through the node's own router. An order
POSTed to /ext/bc/D/dex/clob_submit flows submitTx -> mempool.Add -> consensus
PendingTxs -> BuildBlock -> Block.Verify (the matcher) -> Block.Accept, and the
response is the CONSENSUS-COMPUTED outcome. No external matcher, no relay, no
side port, no node image change — the chain is the matcher.
TDD: TestIngestCrossingOrdersMatchInBlock proves two crossing orders submitted
over the HTTP handler produce a fill committed as D-Chain state (trade: row,
height advances). CGO=0; full pkg/dchain suite green; plugin cross-builds
linux/amd64.
SubmitMarketable and MatchOrders derived their returned fills by re-slicing
the shared trade log (ob.Trades[startOffset:]) with an offset captured BEFORE
matching. ob.Trades self-truncates at 100k entries (keeps the last 50k), so a
match that crosses that boundary mid-match trims the log and leaves the
pre-match offset pointing past the end -> panic 'slice bounds out of range'.
The log feeds settlement, so fills had to stay exact.
tryMatchImmediateLocked / MatchOrders now capture each fill as it is produced,
independent of ob.Trades, and return that slice. Decomplects 'this match's
fills' (a value the caller needs) from 'the trade log' (a bounded-retention
buffer): correct across the truncation boundary by construction, deterministic
across validators.
Adds orderbook_truncation_test.go: three regression cases that cross the
boundary through the production match path (single submits, one sweeping
submit, legacy MatchOrders) with exact-fill and value-conservation asserts.
They panic on the old code, pass on the fixed code. No sleeps, no stubs.
Also: 0x9999 D-matches/C-settles docs (LLM.md, README, ARCHITECTURE),
venue-canonical naming, fourpath-live proof tool, drop dead snowman-api key
from run-lux-testnet.sh.
Adds the missing SERVER side of the FIX trading path and a unified e2e
proving REAL fills over all four DEX access paths against ONE book.
The decomplection: the D-Chain CLOB (pkg/dchain) is the single matcher,
reached over the single frozen clob_* ZAP surface (pkg/zapwire). Every
access path is a thin wire dialect onto that surface — none owns a
matcher. This commit adds the two missing dialects (FIX, a focused WS
venue handler) and the shared seam they translate through.
New:
- pkg/venue: ONE Venue interface (ensure/place/cancel/submit = the
clob_* ops) + ONE production ZAP-backed impl. The seam FIX, WS, and
the 0x9010 precompile all route through. Dials the SAME rpc.Listen
socket dvenue serves.
- pkg/fix: dependency-free FIX 4.4 codec (BodyLength tag 9 + CheckSum
tag 10 computed AND validated; spec-correct emit/parse) + a TCP
acceptor (Logon/Heartbeat/TestRequest/Logout, monotonic seq,
NewOrderSingle->clob_submit->ExecutionReport, OrderCancelRequest->
clob_cancel). Pure protocol front-end onto Venue; no matcher.
- pkg/api/venue_ws.go: minimal gorilla/websocket /ws handler that
translates a JSON order to Venue.Submit and streams fills back —
the WS dialect onto the SAME venue (kept orthogonal to the existing
TradingEngine product WS server).
- cmd/fix-gateway: runs the acceptor in front of a live venue
(-venue clob_* endpoint), the production form of the FIX path.
- pkg/dchain/fourpath_e2e_test.go: stands up ONE venue + ONE LUX/LUSD
book and drives a real limit-order -> match -> FILL over EACH path
(V4 precompile engine_zap frame | ZAP native | FIX | WS) against the
SAME resting asks, asserting real consensus fills, per-path custody
changes, and GLOBAL conservation (available+locked invariant).
V4 leg: exercised at the engine_zap WIRE boundary — the exact 66-byte
clob_submit frame engine_zap.Swap emits (asserted byte-identical to
zapwire.EncodeSubmit) + the V4 BalanceDelta sign convention derived from
the consensus fills. A live luxd EVM (StateDB replay-idempotency, gas,
PoolManager) is what the full on-chain precompile call additionally
needs; the matcher boundary is proven in-process here.
No deps added; no existing files modified (go.mod/go.sum untouched).
Builds CGO_ENABLED=0; race-clean.
Drop 31 demo/benchmark/experiment one-off cmds whose coverage lives in
package tests (bench-all, dag-network, dclient, dexe2e, e2e-fix-zmq,
fix-benchmark, hybrid-auto, latency-benchmark, memory-*, mlx-memory-analysis,
multi-node, multi-node-test, nats-*, perf-test, stress-test, test-client,
zmq-*, dvenue-cancel, dvenue-custody, venueprobe, empty api-server/server/trader).
Keep production binaries: dvenue, dchain, dex-server, dex-api-server, gateway,
price-feed, monitor, persistent-server, demo (each referenced by a
Dockerfile/Makefile/script/k8s manifest or a documented entrypoint).
Consolidate cmd/dvenue + cmd/dvenue-cpu into one tag-free cmd/dvenue: the two
mains were byte-identical except a build tag and a log label; the GPU-vs-CPU
matcher is already selected inside pkg/lx by its own cgo-tagged files, so the
main needs no tag. CGO_ENABLED alone picks the matcher; one binary builds in
both modes. engine_{cgo,nocgo}.go carry only the engine display label. dvenue
now builds under the default CGO_ENABLED=0 gate (was tag-excluded before).
Dockerfile.dvenue: drop the dvenue-custody build+copy (cmd removed).
Makefile build-tools: point at real tools (price-feed, monitor) instead of the
removed cmd/trader and never-existent cmd/lx-metrics.
pkg/price: fix the three CChainSource tests. They were gated on requireLuxd
(a luxd TCP dial) but CChainSource never dials luxd — fetch() is pure simulate().
The gate skipped only when luxd was down, masking the real defect on hosts where
luxd runs. Rewrite the tests to drive the source's true contract deterministically
via src.poll() (no ticker race) and assert against the symbols the source actually
polls (cchainTokens), plus not-found for untracked symbols. Remove the dead
requireLuxd helper and its net import.
orderuser->full 16-byte user is the only settlement identity; 8-byte UserID is a matcher/STP handle only, never a money key. Missing orderuser fails closed (ErrMissingSettlementUser), no matcher-handle fallback. Ownership invariant harness alongside conservation (catches value-conserving theft). txHash-ref custody idempotency closes vault-drain/strand. Reject zero-lock orders; persist orderuser before book insert; restart asserts coverage.
bootVenue opened the on-disk chainstate via badgerdb.New; use zapdb.New
(same signature, same on-disk format). Bump luxfi/vm v1.2.3→v1.2.4 to pick
up the matching rpc/zap zapdb harness. Transitive api v1.0.12→v1.0.15,
warp v1.18.6→v1.19.2 (v1.x). go build+vet ./pkg/dchain/ clean.
zeekayandHanzo Dev