Note, I've tried to use librseq, but looks like it does not worth it,
for read it may get up to 16%, while contended atomic vs uncontended is
30x.
Though rseq can also help with contention between write and read sides,
but it is also neglectible, it is only around 40ns win for p99+, while
complete read of all counters (for metric_log) is 2us vs 38us, yes, it
is 19x more, but it is still only 38us, and unlikely the contention is a
problem there.
v2: switch users to shared counter - 30x faster
v3: use NPROC over MAX_CPUs
v4: remove atomic pointer in ctor
v5: rename shard → per_cpu
v6: cache-line-align ProfileEvents per-CPU rows
v7: Add benchmark for ProfileEvents under 500 threads × 50 groups
v8: Drop librseq from ProfileEvents per-CPU sharding (does not give lots of benefit)
per-cpu atomic
==============
Benchmark Time CPU Iterations UserCounters...
---------------------------------------------------------------------------------------------------------
BM_ProfileEvents/1000/process_time/real_time 0.948 ms 26.1 ms 694 groups=50 incs/thread/step=1.44092 items_per_second=527.21M/s ns/inc=1.89678ns threads=500 users=1
BM_ProfileEvents/10000/process_time/real_time 5.13 ms 213 ms 110 groups=50 incs/thread/step=90.9091 items_per_second=975.257M/s ns/inc=1025.37ps threads=500 users=1
BM_ProfileEvents/100000/process_time/real_time 38.3 ms 1862 ms 18 groups=50 incs/thread/step=5.55556k items_per_second=1.30592G/s ns/inc=765.745ps threads=500 users=1
rseq
==============
Benchmark Time CPU Iterations UserCounters...
---------------------------------------------------------------------------------------------------------
BM_ProfileEvents/1000/process_time/real_time 0.875 ms 22.3 ms 753 groups=50 incs/thread/step=1.32802 items_per_second=571.651M/s ns/inc=1.74932ns threads=500 users=1
BM_ProfileEvents/10000/process_time/real_time 4.64 ms 175 ms 156 groups=50 incs/thread/step=64.1026 items_per_second=1076.95M/s ns/inc=928.552ps threads=500 users=1
BM_ProfileEvents/100000/process_time/real_time 34.6 ms 1576 ms 20 groups=50 incs/thread/step=5k items_per_second=1.44688G/s ns/inc=691.142ps threads=500 users=1
v9: Add user_profile_events_per_cpu server setting
v10: Cache-line-align the per-user ProfileEvents counter allocation
v11: Resolve sched_getcpu once per logical ProfileEvents increment
v12: More benchmarks
Squashes prior branch commits:
- Make `to_throw` settings changeable without restart
- Respect settings in local `clickhouse`
- Add runtime settings update test
- Extend tests and respect `server_settings`
- Condense duplicated `Context` code with X-macros
- Fix style check and warning format messages
getEffectiveSampleProbability reads only ThreadStatus's cached sample config, so it
goes stale whenever the effective tracker parent/config changes without a refresh:
threads parented to total_memory_tracker with no query context, the initiating thread
after ProcessList::insert applies the query's SETTINGS (e.g. memory_profiler_sample_probability
in the SQL SETTINGS clause), detach reparenting back to total_memory_tracker, the main
thread after the server applies total_memory_tracker_sample_probability (set after
MainThreadStatus was created), and MemoryTrackerSwitcher scopes — each kept a stale
value and stopped emitting MemorySample. Re-resolve the cache at every such point:
ThreadStatus construction, applyGlobalSettings (every attach), applyQuerySettings,
ProcessList::insert, detachFromGroup, MemoryTrackerSwitcher (save/re-resolve/restore),
and Server after the total-sampler setters.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Threads defer (de)allocations in a thread-local untracked_memory and flush to the
global MemoryTracker only past max_untracked_memory, so it undercounts by up to
nthreads * max_untracked_memory — tens of GiB with thousands of threads, enough to
overshoot max_server_memory_usage and OOM.
Bound it per-CPU instead. Each CPU keeps two counters, allocated (>= 0) and freed
(<= 0); a thread folds its untracked_memory into the counter for its sign, batched by
per_cpu_untracked_memory_thread_buffer (32 KiB), and must flush once a counter leaves
[-max_per_cpu_untracked_memory, max_per_cpu_untracked_memory]. The deferred total is
then bounded by ncpu * max_per_cpu_untracked_memory regardless of thread count, parked
threads included. The two sides are independent so a deferred free cannot admit a
deferred allocation (which the free would then strand untracked on its own flush).
max_per_cpu_untracked_memory (default 8 MiB, 0 disables) is a new server setting;
max_untracked_memory stays per-thread. Linux only (needs sched_getcpu).
v2: reimplemented — v1 bounded via sum % SLICE on one global counter, which cannot
bound the per-CPU deferred bytes; replaced with the PerCPUMemory class.
v3: two counters instead of one signed net — a signed net let a deferred free cancel
a deferred allocation, stranding it untracked when the free flushed.
v4: restore the per-CPU contribution unconditionally on a failed flush; a gated
re-reserve could drop still-live bytes if another thread took the budget meanwhile.
v5: release the per-CPU contribution only in the caller, not in reserve().
v6: Rename PerCPUMemory::reserve to sync (it sets an absolute value, not a reservation)
v7: Rename net_per_cpu member to slots
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Resolve include-region conflicts in programs/local/LocalServer.cpp and
programs/server/Server.cpp: master added <Functions/pointInPolygon.h>
and this branch added <Functions/UserDefined/UserDefinedSQLFunctionFactory.h>
at the same location; keep both (both are used in each file).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Branch was 1157 commits behind master and the only CI red was the chronic
master-wide `Hung check failed, possible deadlock found` flake (13-70/day on
master itself; this PR is a compile-time no-op under `SANITIZER` so it cannot
affect the `arm_asan_ubsan` stress run). Re-merge to refresh CI on the current
master and to re-run the AI review against the already-updated PR description.
The merge conflicted only in `src/Common/MemoryWorker.cpp`: master evolved the
`memory_worker_dynamic_hard_limit` apply block to skip adjustments smaller than
`min_change_ratio` (1%) of the current limit, still naming the local
`current_hard_limit`. The branch had renamed that local to `installed_hard_limit`
to avoid shadowing the `current_hard_limit` the speculative-reserve block reads
earlier in `updateResidentMemoryThread` (`-Wshadow`/`-Werror`). Kept master's new
threshold logic verbatim and renamed its local to `installed_hard_limit`; the
`LOG_TRACE` already references `installed_hard_limit`. No behavior change.
Verified the resolved `MemoryWorker.cpp` compiles cleanly (`-fsyntax-only`,
`-Werror -Wall -Wextra`) against the merged tree.
The merge with master brought in the multistage distributed queries
feature, whose `ExchangeServer` startup code in `Server.cpp` throws
`Exception(ErrorCodes::NETWORK_ERROR, ...)`. This branch had removed the
`extern const int NETWORK_ERROR;` declaration from `Server.cpp` because
its only previous use (in `Server::createServer`) was extracted into
`src/Server/createServer.cpp`. Restore the declaration so the new use
compiles.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Third slice of the UNIQUE KEY stack. Lands the per-part delete-bitmap
storage layer on top of PR-2's DeleteBitmap + DeleteBitmapFileOps
primitives, aligned with the CSN-based transaction model documented
in design/txn/transaction_consistency.md.
Why a separate bitmap store. UNIQUE KEY needs to mark logically-deleted
rows without rewriting the part. Each commit publishes one cumulative
bitmap per touched part, tagged by a monotonic per-partition version.
Readers pin a snapshot version and consult the bitmap with
`max version ≤ snapshot` for each part — that's the snapshot-isolation
primitive for UK reads, INSERT-time probe, and merge-time row
resolution. This PR delivers the storage and read/write API for that
bitmap; the transaction-control plane (version allocation, partition
mutex, query snapshot, manifest, recovery) lands in later PRs.
What's in this PR:
- MergeTreeBitmapStore — one per table. Atomic install at a version
(strictly monotonic; throws LOGICAL_ERROR otherwise), snapshot-pinned
read returning (shared_ptr<const DeleteBitmap>, chosen_version),
(V_b, V_next)-rule GC, and dropPart for parts leaving the active
set. Keeps an in-memory version index per part (rebuilt from disk on
first access) so the hot read path is map lookup + binary search,
not a syscall.
- DeleteBitmapCache — process-wide cache for deserialized bitmaps,
keyed by stable part identity + version. Configurable via
unique_key_bitmap_cache_* settings, wired through Context startup/
reload, observable via DeleteBitmapCache* metrics and
UniqueKeyBitmap* events.
- IMergeTreeDataPart — cache-identity accessor for callers building
cache keys; delete_bitmap_<csn>.rbm enumeration in
getFileNamesWithoutChecksums so BACKUP and replication-fetch carry
bitmap sidecars.
Why the API is shaped this way:
- Bitmaps are immutable by version. readBitmap returns
shared_ptr<const DeleteBitmap> so a writer building a new version
must copy, never mutate the cached older snapshot.
- The store is concrete (no IUniqueKeyBitmapStore interface). Future
SharedMergeTree variant runs a fundamentally different protocol
(Keeper-coordinated S3 with (bn, uuid) refs); a shared base wouldn't
fit at this layer.
- Integration with active reads/writes lands in later slices (PR 4
read filter, PR 5 DELETE, PR 13 INSERT/UPSERT commit, PR 14a merge
reconcile).
Tests: 39 / 39 PASS in DeleteBitmap*:*BitmapFileOps*:*MergeTreeBitmapStore*
filter:
- 20 DeleteBitmapTest (primitive)
- 1 DeleteBitmapCacheTest (cache wrapper)
- 11 DeleteBitmapFileOpsTest (file I/O)
- 7 MergeTreeBitmapStoreTest: read empty / max-≤-snapshot pick / cache
reuse / GC removal / GC committed-filter / dropPart / corrupt-file
surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Address review: a positive max_server_memory_usage_to_ram_ratio used to
opt the server into runtime hard-limit shrinking with no way to keep the
previous static cap, while setting the ratio to 0 removed the cap
entirely. The new memory_worker_dynamic_hard_limit server setting
(enabled by default) gates only the runtime adjustment: when disabled,
max_server_memory_usage_to_ram_ratio caps the hard memory limit
statically at startup and on configuration reload, exactly as in
previous versions.
Co-Authored-By: Claude Fable 5 <[email protected]>
