mirror of
https://github.com/hanzoai/datastore.git
synced 2026-09-26 13:55:07 +00:00
Raúl MarínandClaude Opus 4.8
a73921741a
`IntersectOrExceptTransform` handled the ALL (multiset) variants by hashing every row with `SipHash` into a `UInt128` and using that digest as the key of a `HashMap<UInt128, UInt64>`. This was ~7-10x slower than the DISTINCT path: per-row hashing plus the fat 128-bit-keyed map dominated (large allocations, resizes, page faults), and storing only the digest risked silent collisions corrupting results. Use the same `SetVariants` dispatch the DISTINCT path uses, but with a counting map keyed on the real row value. For fixed and string keys this avoids per-row hashing entirely (the value is the key) and uses a narrower cell; only exotic keys fall back to the previous 128-bit hash. The multiset-count algorithm is unchanged, so semantics are preserved, and keying on real values removes the collision risk. Add `CountingSet` to `SetVariants`: a `HashMap<Key, UInt64>` analogue of the set, with the consecutive-keys cache disabled so the mapped count is incremented in place. On `SELECT count() FROM (numbers INTERSECT numbers)` over 40M rows this drops from ~12s to ~1.6s, matching `INTERSECT DISTINCT`. Add a performance test. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
15 lines
880 B
XML
15 lines
880 B
XML
<test>
|
|
<!-- INTERSECT ALL / EXCEPT ALL build a multiset (occurrence counts) of the right side
|
|
and decrement it while filtering the left side. The counting hash map is keyed on the
|
|
row value, exercising the key64 / keys128 / key_string dispatch. -->
|
|
|
|
<!-- Single UInt64 key (key64). -->
|
|
<query>SELECT count() FROM (SELECT number FROM numbers(2500000) INTERSECT ALL SELECT number FROM numbers(2500000))</query>
|
|
|
|
<!-- Two fixed columns packed into a 128-bit key (keys128). -->
|
|
<query>SELECT count() FROM (SELECT number AS a, number + 1 AS b FROM numbers(2000000) INTERSECT ALL SELECT number AS a, number + 1 AS b FROM numbers(2000000))</query>
|
|
|
|
<!-- String key (key_string). -->
|
|
<query>SELECT count() FROM (SELECT toString(number) AS s FROM numbers(2000000) EXCEPT ALL SELECT toString(number) AS s FROM numbers(1000000))</query>
|
|
</test>
|