Wires the native ZAP query transport into the server, end to end, on the green unify foundation:
- src/Server/ZapServer.{h,cpp}: the transport owner (port + lifecycle thread). Minimal compiling
skeleton; the KJ event loop + zap-rpc TwoPartyServer (promise pipelining) serving the .zap query
interface is the next increment (kept out of the header behind an opaque Runtime).
- ProtocolServerAdapter: IServer/ZapServer ctor + ZapServerAdapterImpl (#if USE_ZAP), mirroring the
old gRPC adapter.
- ServerType::ZAP (+ default-start list); Server.cpp createServer for `zap_port` (#if USE_ZAP);
config.xml <zap_port> (commented, opt-in like grpc_port was).
- USE_ZAP cmakedefine (set when ch_contrib::zap exists); dbms links ch_contrib::zap.
This verifies the whole transport wiring + USE_ZAP build config compiles with zap linked. Next:
implement ZapServer::run() (KJ setupAsyncIo -> listen -> TwoPartyServer) + the .zap query schema +
executeQuery dispatch + result streaming.
The correct, safe broad rebrand: renames "ClickHouse" only as a whole word (boundary on
BOTH sides), so it rewrites display strings/messages/comments but leaves every compound
C++ symbol intact — `ClickHouseDictionarySource`, `getInfoIfClickHouseDictionarySource`,
`mainEntryClickHouseServer`, `ClickHouseRevision`, `ClickHouse_SOURCE_DIR` are untouched
(an earlier one-sided heuristic mangled suffix symbols like `manageClickHouse`; reverted).
1677 occurrences across 482 files. Compile-safe (string/comment content; symbols preserved).
Still NOT renamed (would be a build-coupled symbol rename, needs build-host iteration): the
199 compound identifiers containing ClickHouse, the `DB::` namespace, the `clickhouse_*`
CMake targets/macros, the lowercase `clickhouse` SQL dictionary-source keyword. Needs build
verification.
No more gRPC. Datastore's native RPC transport will be zap (Cap'n Proto / KJ, ~/work/zap);
the HTTP (8123) and native TCP (9000) protocols remain.
- ENABLE_GRPC forced OFF (contrib/grpc-cmake) -> USE_GRPC=0, and since Arrow Flight gates on
ch_contrib::arrow_flight (which needs grpc), USE_ARROWFLIGHT=0 too. All #if USE_GRPC /
#if USE_ARROWFLIGHT code compiles out; the grpc + grpc_protos + arrow_flight CMake is
`if (TARGET ch_contrib::grpc)`-guarded, so nothing builds.
- deleted src/Server/{GRPCServer.cpp,GRPCServer.h,IGRPCServer.h,grpc_protos/,ArrowFlight/}.
- removed the one unconditional ArrowFlight include in Server.cpp; removed the <grpc> config.
Needs build verification. Dead #if USE_GRPC stubs remain in ProtocolServerAdapter + Server.cpp
(guarded, compiled-out) — cleanup follow-up.
The merge with master brought in the multistage distributed queries
feature, whose `ExchangeServer` startup code in `Server.cpp` throws
`Exception(ErrorCodes::NETWORK_ERROR, ...)`. This branch had removed the
`extern const int NETWORK_ERROR;` declaration from `Server.cpp` because
its only previous use (in `Server::createServer`) was extracted into
`src/Server/createServer.cpp`. Restore the declaration so the new use
compiles.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Third slice of the UNIQUE KEY stack. Lands the per-part delete-bitmap
storage layer on top of PR-2's DeleteBitmap + DeleteBitmapFileOps
primitives, aligned with the CSN-based transaction model documented
in design/txn/transaction_consistency.md.
Why a separate bitmap store. UNIQUE KEY needs to mark logically-deleted
rows without rewriting the part. Each commit publishes one cumulative
bitmap per touched part, tagged by a monotonic per-partition version.
Readers pin a snapshot version and consult the bitmap with
`max version ≤ snapshot` for each part — that's the snapshot-isolation
primitive for UK reads, INSERT-time probe, and merge-time row
resolution. This PR delivers the storage and read/write API for that
bitmap; the transaction-control plane (version allocation, partition
mutex, query snapshot, manifest, recovery) lands in later PRs.
What's in this PR:
- MergeTreeBitmapStore — one per table. Atomic install at a version
(strictly monotonic; throws LOGICAL_ERROR otherwise), snapshot-pinned
read returning (shared_ptr<const DeleteBitmap>, chosen_version),
(V_b, V_next)-rule GC, and dropPart for parts leaving the active
set. Keeps an in-memory version index per part (rebuilt from disk on
first access) so the hot read path is map lookup + binary search,
not a syscall.
- DeleteBitmapCache — process-wide cache for deserialized bitmaps,
keyed by stable part identity + version. Configurable via
unique_key_bitmap_cache_* settings, wired through Context startup/
reload, observable via DeleteBitmapCache* metrics and
UniqueKeyBitmap* events.
- IMergeTreeDataPart — cache-identity accessor for callers building
cache keys; delete_bitmap_<csn>.rbm enumeration in
getFileNamesWithoutChecksums so BACKUP and replication-fetch carry
bitmap sidecars.
Why the API is shaped this way:
- Bitmaps are immutable by version. readBitmap returns
shared_ptr<const DeleteBitmap> so a writer building a new version
must copy, never mutate the cached older snapshot.
- The store is concrete (no IUniqueKeyBitmapStore interface). Future
SharedMergeTree variant runs a fundamentally different protocol
(Keeper-coordinated S3 with (bn, uuid) refs); a shared base wouldn't
fit at this layer.
- Integration with active reads/writes lands in later slices (PR 4
read filter, PR 5 DELETE, PR 13 INSERT/UPSERT commit, PR 14a merge
reconcile).
Tests: 39 / 39 PASS in DeleteBitmap*:*BitmapFileOps*:*MergeTreeBitmapStore*
filter:
- 20 DeleteBitmapTest (primitive)
- 1 DeleteBitmapCacheTest (cache wrapper)
- 11 DeleteBitmapFileOpsTest (file I/O)
- 7 MergeTreeBitmapStoreTest: read empty / max-≤-snapshot pick / cache
reuse / GC removal / GC committed-filter / dropPart / corrupt-file
surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Address review: a positive max_server_memory_usage_to_ram_ratio used to
opt the server into runtime hard-limit shrinking with no way to keep the
previous static cap, while setting the ratio to 0 removed the cap
entirely. The new memory_worker_dynamic_hard_limit server setting
(enabled by default) gates only the runtime adjustment: when disabled,
max_server_memory_usage_to_ram_ratio caps the hard memory limit
statically at startup and on configuration reload, exactly as in
previous versions.
Co-Authored-By: Claude Fable 5 <[email protected]>
The previous fix protected `PipelineExecutor::allocateCPU` and the
query-execution callers, but the lazy initialization of the resource
manager still observed the same shutdown race.
`Context::getWorkloadClassifier` initializes the resource manager lazily
through `getResourceManager`, which calls `createResourceManager`. That
passed `getWorkloadEntityStorage()` (a raw reference whose owning
`shared_ptr` temporary is destroyed at the end of the full expression),
so no owner was held across `WorkloadResourceManager`'s constructor while
it runs `getAllEntitiesAndSubscribe`. If this first lazy init overlaps
shutdown's reset of `workload_entity_storage`, the storage can be
destroyed while the constructor is still using it — the same
`recursive_mutex` use-after-free this PR set out to fix.
Make `WorkloadResourceManager` take and retain a
`shared_ptr<IWorkloadEntityStorage>`, and have `createResourceManager`
pass `getWorkloadEntityStoragePtr`. The storage is only touched in the
constructor, so holding the owner for the manager's lifetime is more than
sufficient.
Also remove the now-unused reference accessor
`Context::getWorkloadEntityStorage` (the last caller in `Server.cpp`'s
startup `loadEntities` is migrated to `getWorkloadEntityStoragePtr`), so
the shutdown-unsafe accessor can no longer be reintroduced.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
zooqueen