mirror of
https://github.com/hanzoai/datastore.git
synced 2026-09-26 13:55:07 +00:00
Nikita TaranovandClaude Opus 4.8
79fdabf86a
A join on a single non-nullable LowCardinality(T) key materialized the key
column on both the build and probe sides, discarding the dictionary. Add
dictionary-aware maps (low_cardinality_key{8,16,32,64,string,fixed_string}) so
the key column stays dictionary-encoded: the probe deduplicates the hash-table
lookup per dictionary index within a block, and the key is not materialized.
- New map variants + a HashJoin-local LowCardinalityKeyGetterForJoin in
KeyGetter.h that wraps a base method over the dictionary's nested column. It is
const-correct on the probe side, produces an offset-carrying FindResult (for
JoinUsedFlags), caches a pointer into the stable hash-table cell (not a copy,
which would dangle in the lazy output), and does not deduplicate on the build
side (every row must be appended).
- chooseMethod detects a single non-nullable LowCardinality key; the key column
is kept un-materialized at the build and probe sites for those maps.
LowCardinality(Nullable(T)), composite keys and two-level maps fall back to the
existing materialized path.
~2-3x faster for string keys (hashjoin_low_cardinality_key.xml); neutral for
numeric LowCardinality, where the materialized baseline is already cheap.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
37 lines
2.5 KiB
XML
37 lines
2.5 KiB
XML
<test>
|
|
<settings>
|
|
<max_bytes_ratio_before_external_join>0</max_bytes_ratio_before_external_join>
|
|
<join_algorithm>hash</join_algorithm>
|
|
<query_plan_join_swap_table>false</query_plan_join_swap_table>
|
|
</settings>
|
|
|
|
<!--
|
|
Joins on a single non-nullable LowCardinality key now use a dictionary-aware hash map: the
|
|
key column is not materialized, and the probe deduplicates the hash-table lookup per
|
|
dictionary index within a block. The probe table has many rows but few distinct keys, so the
|
|
per-block dictionary has far fewer entries than the block size; with swap pinned off the large
|
|
table is the probe side. The benefit is concentrated on string keys, where the materialized
|
|
baseline pays for variable-length data and string hashing/comparison.
|
|
-->
|
|
|
|
<create_query>CREATE TABLE lck_str_build (k LowCardinality(String), w UInt64) ENGINE = Memory</create_query>
|
|
<create_query>CREATE TABLE lck_str_probe (k LowCardinality(String), v UInt64) ENGINE = Memory</create_query>
|
|
<create_query>CREATE TABLE lck_fs_build (k LowCardinality(FixedString(16)), w UInt64) ENGINE = Memory</create_query>
|
|
<create_query>CREATE TABLE lck_fs_probe (k LowCardinality(FixedString(16)), v UInt64) ENGINE = Memory</create_query>
|
|
|
|
<fill_query>INSERT INTO lck_str_build SELECT toString(number), number FROM numbers_mt(1000)</fill_query>
|
|
<fill_query>INSERT INTO lck_str_probe SELECT toString(number % 1000), number FROM numbers_mt(50000000)</fill_query>
|
|
<fill_query>INSERT INTO lck_fs_build SELECT toFixedString(leftPad(toString(number), 16, '0'), 16), number FROM numbers_mt(1000)</fill_query>
|
|
<fill_query>INSERT INTO lck_fs_probe SELECT toFixedString(leftPad(toString(number % 1000), 16, '0'), 16), number FROM numbers_mt(50000000)</fill_query>
|
|
|
|
<!-- The build keys are unique, so ALL INNER matches each of the 50M probe rows exactly once and
|
|
drives the full probe through the dictionary-aware key getter. -->
|
|
<query>SELECT sum(l.v) FROM lck_str_probe AS l ALL INNER JOIN lck_str_build AS r ON l.k = r.k FORMAT Null</query>
|
|
<query>SELECT sum(l.v) FROM lck_fs_probe AS l ALL INNER JOIN lck_fs_build AS r ON l.k = r.k FORMAT Null</query>
|
|
|
|
<drop_query>DROP TABLE IF EXISTS lck_str_build</drop_query>
|
|
<drop_query>DROP TABLE IF EXISTS lck_str_probe</drop_query>
|
|
<drop_query>DROP TABLE IF EXISTS lck_fs_build</drop_query>
|
|
<drop_query>DROP TABLE IF EXISTS lck_fs_probe</drop_query>
|
|
</test>
|