Integrate zxc (https://github.com/hellobertrand/zxc), an asymmetric LZ codec
built for very fast decompression, as a new general-purpose codec
`CODEC(ZXC[(level)])` (levels 1..7, default 3, compression method byte 0x9e).
`ZXC` trades slow compression for fast decompression at a ratio between `LZ4`
and `ZSTD`, which suits the compress-once/read-many pattern of analytical
columns and decodes especially fast on modern ARM cores.
- `contrib/zxc` submodule plus a curated `contrib/zxc-cmake` build that mirrors
zxc's per-ISA variant objects with runtime dispatch, without its own
`-march=native`/LTO/PGO defaults (ClickHouse manages its own flags).
- `CompressionCodecZXC` uses zxc's one-shot frame API with zxc's internal
checksum disabled, since ClickHouse already checksums every compressed block.
- Documentation and a stateless roundtrip test (all levels plus error handling).
Note: `contrib/zxc` points at the upstream repository as a placeholder; before
merging it must be replaced with a fork under github.com/ClickHouse.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Two safety fixes against crafted input and misuse, plus the corresponding
contrib/sz3 fork bump:
- Lossy codecs (SZ3) reinterpret the raw bytes as floating-point values, so they
must not be applied where the column data type is unknown. CompressionFactory
now rejects a lossy codec built without a data type (after the only_generic
filter, so structural substreams are unaffected). This closes the silent
corruption of the marks and primary key when 'marks_compression_codec' or
'primary_key_compression_codec' is set to SZ3.
- Before decompressing, the codec parses and validates the SZ3 configuration:
only the interpolation/Lorenzo algorithms and the lossless fallback are
accepted (a crafted block can otherwise dispatch into ALGO_BIOMD/ALGO_BIOMDXTC
or the OpenMP path, whose decoders are not hardened), the element count must
match the trusted uncompressed size, and OpenMP decompression is rejected.
The contrib/sz3 fork is bumped to add SZ_load_config (parse the config without
dispatching) and bounds checks on the generic decompressor, the quantizer and
predictors, and the Huffman tree reconstruction.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This reverts the removal of the SZ3 lossy floating-point compression
codec done in https://github.com/ClickHouse/ClickHouse/pull/84358
(itself a revert of https://github.com/ClickHouse/ClickHouse/pull/83088),
and adapts it to the current `master`, finishing the work that the
fuzzer crashes (https://github.com/ClickHouse/ClickHouse/issues/84345)
had left unfinished.
`SZ3` is an experimental, error-bounded lossy codec for `Float32`,
`Float64`, `Array(Float32)` and `Array(Float64)` columns. It requires
`SET allow_experimental_codecs = 1`.
Changes needed to revive it on top of `master`:
- Method byte: the original `SZ3` used `0x9c`, but that byte is now
taken by the `ALP` codec, so `SZ3` is assigned the next free byte
`0x9d`. The `SZ3` was never part of a stable release, so no persisted
data depends on the old byte.
- `doDecompressData` now returns `UInt32` (the base class signature
changed) and the codec registration was moved to the shared
`registerCompressionCodecs.h`.
- Re-integrated the vector-dimension and lossy-offset handling into the
refactored `MergeTreeDataPartWriterCompact`/`Wide`/`OnDisk`.
- `flake.nix` and the `aspell` dictionary entries are not restored,
because both were removed from `master` in the meantime.
- `SZ3` is intentionally not re-added to the BuzzHouse codec fuzzer: its
codec generator is now type-aware and only emits lossless codecs, and
`SZ3` is lossy, which would produce false oracle mismatches.
Hardening of `doDecompressData` against corrupted input (the reason for
the original revert):
- The trusted `uncompressed_size` is now enforced. The previous code
ignored it and let SZ3 write `config.num` (taken from the untrusted
compressed data) elements into `dest`, which is only `uncompressed_size`
bytes long - a heap buffer overflow. The decompressed element count is
now validated against `uncompressed_size` before decompression.
- The float width is validated and `uncompressed_size` must be a multiple
of it.
- The serialized `SZ3::Config` is parsed from a zero-padded copy of the
data, after validating the dimension count and bit width, to avoid
out-of-bounds reads, and `config.size_est()` is checked against the
compressed size to prevent the payload-size underflow inside
`SZ_decompress`.
Closes: https://github.com/ClickHouse/ClickHouse/issues/84345
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The std-container lint for `src/Compression` requires the `-WithMemoryTracking`
container aliases. Use `VectorWithMemoryTracking` and `UnorderedMapWithMemoryTracking`
for the additions made for `system.documentation`.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Extend `system.documentation` with a `source` column holding the path of the
source file where each entity's documentation is defined, relative to the
repository root. The path is captured automatically at the construction site of
every `Documentation`/`FunctionDocumentation` object via `std::source_location`
(and normalized to be repository-relative regardless of build path mapping), and
is provided explicitly for the entity groups documented in a single source file.
Also document additional kinds of entities: compression codecs (`Compression
Codec`), profile events (`Profile Event`), current metrics (`Current Metric`),
asynchronous metrics (`Asynchronous Metric`), and the system tables themselves
(`System Table`). For system tables the documentation is the table comment
together with the list of its columns (name, type and description); for settings
of any kind the documentation now also includes the type and default value.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Swap std::vector, std::map, std::set, std::unordered_map, std::list to the
corresponding *WithMemoryTracking aliases in src/Compression. Mark a few
spots as STYLE_CHECK_ALLOW_STD_CONTAINERS where the type is bound to
IFactoryWithAliases::InnerMap covariance or referenced in a documentation
comment block. Add src/Compression to the directories linted by
check_cpp.sh.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Add `static` to file-local functions and forward declarations
for cross-TU functions to satisfy `-Wmissing-prototypes`.
This ensures all functions have visible prototypes before their
definitions, improving code quality and catching accidental
linkage issues.
Reference: https://github.com/ClickHouse/ClickHouse/issues/76944
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Move `getAllRegisteredNames` implementations from headers to .cpp files
for `CompressionCodecFactory`, `MergeTreeIndexFactory`,
`DictionaryFactory`, and `DictionarySourceFactory`.
- Fix double-counting of opening parenthesis when a dictionary value
is a function call — the deferred bareword handler now emits the
paren and continues instead of falling through.
- Add test cases for function calls in dict values, nested parens,
and dict keywords outside dictionary context.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add a new virtual function getNotes that returns documentation for the codec taken from:
https://kb.altinity.com/altinity-kb-schema-design/codecs/
* src/Compression/ICompressionCodec.h (getData) : New interface function. And update in all derived classes.
* src/Storages/System/StorageSystemCodecs.{h,cpp} : New files with the new table class.
* src/Storages/System/StorageSystemCodecs.{h,cpp} (getMutableColumns) : New function to get columns information accessing the private members in CompressionCodecFactory
* src/Storages/System/attachSystemTables.cpp : Attach the new table to system.
