mirror of
https://github.com/hanzoai/datastore.git
synced 2026-09-26 13:55:07 +00:00
Alexey MilovidovandClaude Opus 4.8
783fcdb5ea
The new `native_arrow_ipc` performance test failed the `Performance Comparison` job (`amd_release` 3/6 and `arm_release` 6/6), which made `Mergeable Check` and the overall `PR` check red. Two independent problems, both self-contained in the test: 1. The write benchmark used `SELECT * FROM na_src FORMAT Arrow`, which does not measure Arrow encoding at all. The perf runner drives queries over the native TCP protocol (`clickhouse_driver`), which ignores the query's `FORMAT` clause and sends `Native` blocks back to the client, so the query really measured client-side row deserialization of 20M rows (~58s) instead of the encoder. That blew past the 15s `max_execution_time` of a measured run (`Code: 159 Timeout exceeded`) and aborted the whole test. Verified with `clickhouse_driver`: `FORMAT Arrow` (2.71s) is identical to a plain `SELECT *` (2.69s) and unlike `FORMAT Null` (0.018s). The encode is now measured by `INSERT INTO na_arrow_out SELECT * FROM na_src` where `na_arrow_out` is `File(Arrow, '/dev/null')` — the encoder runs server-side and the output is discarded — the same idiom used by `select_format.xml`. 2. Every query pinned `input_format_arrow_use_native_reader` / `output_format_arrow_use_native_writer`, which are added by this pull request and therefore do not exist on the compared master server. The prewarm on that server failed with `Code: 115 ... is neither a builtin setting`, and any non-empty per-test `*-err.log` is folded into `run-errors.tsv`, which `report.py` counts unconditionally as an error and fails the job. The queries now run on the default path (native on this branch, the Apache Arrow library on master), so nothing errors on the reference server and the comparison directly measures the speedup of this change. Validated end to end through `tests/performance/scripts/perf.py`: exit code 0, empty `err.log`, write ~1.70s and read ~1.10s per run for 20M rows on the library path, both far under the limit (native is faster still). CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=106522&sha=5d9447cb8cea70d101d8e3f1683402b07ec84440&name_0=PR PR: https://github.com/ClickHouse/ClickHouse/pull/106522 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
68 lines
3.1 KiB
XML
68 lines
3.1 KiB
XML
<test>
|
|
<settings>
|
|
<!-- Single-threaded so the measurement reflects encode/decode cost, not pipeline parallelism. -->
|
|
<max_threads>1</max_threads>
|
|
</settings>
|
|
|
|
<!-- The queries deliberately do NOT set input_format_arrow_use_native_reader /
|
|
output_format_arrow_use_native_writer: they exercise the default path, which is the native
|
|
implementation on this branch and the Apache Arrow library on the compared master build, so
|
|
the comparison directly measures the speedup of this change. Pinning those settings would also
|
|
make every query fail on the master server (the settings do not exist there yet), which the
|
|
perf runner reports as an error and fails the whole test. -->
|
|
|
|
<!-- Source data covering the encoder/decoder paths that are not a plain memcpy: nullable columns
|
|
(validity bitmap), Bool (bit packing), Date (uint16->int32 widening), DateTime64 (rescale),
|
|
Decimal (sign extension), String, LowCardinality, Array. ~1/3 nulls exercises the validity path. -->
|
|
<create_query>
|
|
CREATE TABLE na_src
|
|
(
|
|
i64 Int64,
|
|
n_i64 Nullable(Int64),
|
|
n_u32 Nullable(UInt32),
|
|
b Bool,
|
|
d Date,
|
|
dt64 DateTime64(3),
|
|
dec Decimal64(4),
|
|
s String,
|
|
lc LowCardinality(String),
|
|
arr Array(Int64)
|
|
) ENGINE = Memory
|
|
</create_query>
|
|
<fill_query>
|
|
INSERT INTO na_src
|
|
SELECT
|
|
number,
|
|
if(number % 3 = 0, NULL, number),
|
|
if(number % 5 = 0, NULL, toUInt32(number)),
|
|
number % 2,
|
|
toDate(number % 40000),
|
|
toDateTime64(number / 1000, 3),
|
|
toDecimal64(number % 100000, 4),
|
|
toString(number % 100000),
|
|
toString(number % 1000),
|
|
range(number % 5)
|
|
FROM numbers(20000000)
|
|
</fill_query>
|
|
|
|
<!-- A File(Arrow) table holding the same data, for the read (decode) benchmarks. -->
|
|
<create_query>CREATE TABLE na_arrow ENGINE = File(Arrow) AS na_src</create_query>
|
|
<fill_query>INSERT INTO na_arrow SELECT * FROM na_src</fill_query>
|
|
|
|
<!-- A File(Arrow) sink pointed at /dev/null for the write (encode) benchmark: the INSERT drives
|
|
the Arrow encoder server-side and the output is discarded. A plain "SELECT ... FORMAT Arrow"
|
|
would not measure encoding here, because the perf runner uses the native TCP protocol, which
|
|
ignores the query's FORMAT clause and sends Native blocks back to the client. -->
|
|
<create_query>CREATE TABLE na_arrow_out ENGINE = File(Arrow, '/dev/null') AS na_src</create_query>
|
|
|
|
<!-- WRITE / encode. -->
|
|
<query>INSERT INTO na_arrow_out SELECT * FROM na_src SETTINGS output_format_arrow_compression_method = 'none'</query>
|
|
|
|
<!-- READ / decode. -->
|
|
<query>SELECT * FROM na_arrow FORMAT Null</query>
|
|
|
|
<drop_query>DROP TABLE IF EXISTS na_src</drop_query>
|
|
<drop_query>DROP TABLE IF EXISTS na_arrow</drop_query>
|
|
<drop_query>DROP TABLE IF EXISTS na_arrow_out</drop_query>
|
|
</test>
|