The runtime reload guard validated after `config().replace`, so a rejected
`SYSTEM RELOAD CONFIG` had already swapped in the new configuration:
`system.server_settings` would then report an unsupported legacy value even
though the reload failed, and `ConfigReloader` has no rollback hook. Validate
the incoming configuration before `config().replace`, so a rejected reload
leaves the previous valid config untouched. The integration test now asserts
`system.server_settings` still reports `new_unified_hash` after the rejected
reload.
The fail-closed guard validated only the first loadSettingsFromConfig, so a legacy
`insert_deduplication_version` arriving via a ZooKeeper config include (reloaded during startup) or a
runtime `SYSTEM RELOAD CONFIG` could slip past it, leaving the server running unified-only while the
setting still named a legacy mode. Factor the check into a helper and run it after every
server-settings load: the initial load, the ZooKeeper-include reload, and the config reloader.
Throwing in the reloader rejects the offending reload and keeps the previous valid config.
Extend test_insert_deduplication_version_guard with a reload case asserting that SYSTEM RELOAD CONFIG
to a legacy value is rejected and the server stays up on the valid value.
This build writes only the unified insert deduplication hash. An operator left
on a legacy `insert_deduplication_version` (`old_separate_hashes` or
`compatible_double_hashes`) would otherwise silently lose the mixed-version
deduplication contract during a rolling upgrade. Validate the server setting at
startup and refuse to start, with a migration message, unless it is
`new_unified_hash`.
The setting is therefore no longer obsolete: restore it as a normal server
setting, drop it from the obsolete-settings reference, and stop the
`casa_del_dolor` fuzzer from assigning it a value that would abort startup.
Rework `test_parallel_replicas_protocol` to populate the shared table from a
single replica and propagate by replication; the new build no longer shares an
insert deduplication hash with the old replicas, so inserting identical data
from every replica can no longer rely on cross-version deduplication. Add an
integration test asserting the guard fails closed.
The merge with master brought in the multistage distributed queries
feature, whose `ExchangeServer` startup code in `Server.cpp` throws
`Exception(ErrorCodes::NETWORK_ERROR, ...)`. This branch had removed the
`extern const int NETWORK_ERROR;` declaration from `Server.cpp` because
its only previous use (in `Server::createServer`) was extracted into
`src/Server/createServer.cpp`. Restore the declaration so the new use
compiles.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Third slice of the UNIQUE KEY stack. Lands the per-part delete-bitmap
storage layer on top of PR-2's DeleteBitmap + DeleteBitmapFileOps
primitives, aligned with the CSN-based transaction model documented
in design/txn/transaction_consistency.md.
Why a separate bitmap store. UNIQUE KEY needs to mark logically-deleted
rows without rewriting the part. Each commit publishes one cumulative
bitmap per touched part, tagged by a monotonic per-partition version.
Readers pin a snapshot version and consult the bitmap with
`max version ≤ snapshot` for each part — that's the snapshot-isolation
primitive for UK reads, INSERT-time probe, and merge-time row
resolution. This PR delivers the storage and read/write API for that
bitmap; the transaction-control plane (version allocation, partition
mutex, query snapshot, manifest, recovery) lands in later PRs.
What's in this PR:
- MergeTreeBitmapStore — one per table. Atomic install at a version
(strictly monotonic; throws LOGICAL_ERROR otherwise), snapshot-pinned
read returning (shared_ptr<const DeleteBitmap>, chosen_version),
(V_b, V_next)-rule GC, and dropPart for parts leaving the active
set. Keeps an in-memory version index per part (rebuilt from disk on
first access) so the hot read path is map lookup + binary search,
not a syscall.
- DeleteBitmapCache — process-wide cache for deserialized bitmaps,
keyed by stable part identity + version. Configurable via
unique_key_bitmap_cache_* settings, wired through Context startup/
reload, observable via DeleteBitmapCache* metrics and
UniqueKeyBitmap* events.
- IMergeTreeDataPart — cache-identity accessor for callers building
cache keys; delete_bitmap_<csn>.rbm enumeration in
getFileNamesWithoutChecksums so BACKUP and replication-fetch carry
bitmap sidecars.
Why the API is shaped this way:
- Bitmaps are immutable by version. readBitmap returns
shared_ptr<const DeleteBitmap> so a writer building a new version
must copy, never mutate the cached older snapshot.
- The store is concrete (no IUniqueKeyBitmapStore interface). Future
SharedMergeTree variant runs a fundamentally different protocol
(Keeper-coordinated S3 with (bn, uuid) refs); a shared base wouldn't
fit at this layer.
- Integration with active reads/writes lands in later slices (PR 4
read filter, PR 5 DELETE, PR 13 INSERT/UPSERT commit, PR 14a merge
reconcile).
Tests: 39 / 39 PASS in DeleteBitmap*:*BitmapFileOps*:*MergeTreeBitmapStore*
filter:
- 20 DeleteBitmapTest (primitive)
- 1 DeleteBitmapCacheTest (cache wrapper)
- 11 DeleteBitmapFileOpsTest (file I/O)
- 7 MergeTreeBitmapStoreTest: read empty / max-≤-snapshot pick / cache
reuse / GC removal / GC committed-filter / dropPart / corrupt-file
surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Address review: a positive max_server_memory_usage_to_ram_ratio used to
opt the server into runtime hard-limit shrinking with no way to keep the
previous static cap, while setting the ratio to 0 removed the cap
entirely. The new memory_worker_dynamic_hard_limit server setting
(enabled by default) gates only the runtime adjustment: when disabled,
max_server_memory_usage_to_ram_ratio caps the hard memory limit
statically at startup and on configuration reload, exactly as in
previous versions.
Co-Authored-By: Claude Fable 5 <[email protected]>
The previous fix protected `PipelineExecutor::allocateCPU` and the
query-execution callers, but the lazy initialization of the resource
manager still observed the same shutdown race.
`Context::getWorkloadClassifier` initializes the resource manager lazily
through `getResourceManager`, which calls `createResourceManager`. That
passed `getWorkloadEntityStorage()` (a raw reference whose owning
`shared_ptr` temporary is destroyed at the end of the full expression),
so no owner was held across `WorkloadResourceManager`'s constructor while
it runs `getAllEntitiesAndSubscribe`. If this first lazy init overlaps
shutdown's reset of `workload_entity_storage`, the storage can be
destroyed while the constructor is still using it — the same
`recursive_mutex` use-after-free this PR set out to fix.
Make `WorkloadResourceManager` take and retain a
`shared_ptr<IWorkloadEntityStorage>`, and have `createResourceManager`
pass `getWorkloadEntityStoragePtr`. The storage is only touched in the
constructor, so holding the owner for the manager's lifetime is more than
sufficient.
Also remove the now-unused reference accessor
`Context::getWorkloadEntityStorage` (the last caller in `Server.cpp`'s
startup `loadEntities` is migrated to `getWorkloadEntityStoragePtr`), so
the shutdown-unsafe accessor can no longer be reintroduced.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
New keeper-only system table with one row per on-disk Raft log file
(changelog_<from>_<to>.bin[.zstd]) tracked by the in-process Keeper.
Mirrors the plumbing of system.keeper_snapshots.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Sema Checherinda