13,838 upstream commits, forward from the 26.6.1.1 fork point. 129 conflicts:
37 resolved mechanically (take upstream's file, re-apply the brand sweep) and
92 by hand where our fork and upstream had both changed the same code.
The rule throughout: upstream owns the engine, we own the naming and a small set
of deliberate deviations. Never resolve by discarding an upstream change — a
dropped rename or a missed new parameter becomes a compile error a thousand
files away.
Deviations verified intact after the merge:
ENABLE_XRAY still defaults OFF (CMakeLists.txt:105)
X-Datastore-User read by the server (authenticateUserByHTTP.cpp)
X-ClickHouse-* in src/ 0
hanzo/research present
system.*_log retention bound present
ArrowFlight rip held against upstream re-adding it
Two things worth naming, because neither shows up as a conflict:
src/CMakeLists.txt gained an UNGUARDED
`add_object_library(datastore_storages_arrowflight Storages/ArrowFlight)` from
upstream, for a directory our gRPC rip deleted. It sat outside every conflict
hunk, so a resolver reading only the markers would sail straight past it into
a CMake configure error. Removed; the Server/ArrowFlight sibling at line 474
stays because it is inside `if (TARGET ch_contrib::arrow_flight)`.
The brand sweep had rewritten github.com/ClickHouse/ClickHouse issue links to
github.com/Datastore/Datastore — a dead host. Those point at upstream's own
tracker and are restored, alongside the contributors list, sql.clickhouse.com
and the ClickHouse Inc copyright: attribution is theirs, not ours to rename.
base/glibc-compatibility/memcpy/memcpy.h is deleted, not kept: upstream dropped
it for LLVM-libc's, and our only change to it was a comment rebrand.
Step 1 of decoupling the Coordination gtests from KeeperMemoryStorage
internals: replace the typed-test fixture (TestParam<TStorage, compress>)
with a value-parametrized one. TestParam is now a plain struct with a
non-constexpr enable_compression field, CoordinationTest derives from
testing::TestWithParam<TestParam>, all TYPED_TEST become TEST_P, and a
single INSTANTIATE_TEST_SUITE_P lives in gtest_coordination_common.cpp.
A makeKeeperContext helper centralizes KeeperContext creation so a future
TestParam field selecting the KeeperStorage implementation can be applied
in one place.
Storage usages temporarily point at KeeperMemoryStorage; subsequent
commits replace them with KeeperStorage::create and virtual-method access.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The correct, safe broad rebrand: renames "ClickHouse" only as a whole word (boundary on
BOTH sides), so it rewrites display strings/messages/comments but leaves every compound
C++ symbol intact — `ClickHouseDictionarySource`, `getInfoIfClickHouseDictionarySource`,
`mainEntryClickHouseServer`, `ClickHouseRevision`, `ClickHouse_SOURCE_DIR` are untouched
(an earlier one-sided heuristic mangled suffix symbols like `manageClickHouse`; reverted).
1677 occurrences across 482 files. Compile-safe (string/comment content; symbols preserved).
Still NOT renamed (would be a build-coupled symbol rename, needs build-host iteration): the
199 compound identifiers containing ClickHouse, the `DB::` namespace, the `clickhouse_*`
CMake targets/macros, the lowercase `clickhouse` SQL dictionary-source keyword. Needs build
verification.
Review-fix pass over the keeper-snapshot-locking branch (post master merge).
- G-F1: startup-retained duplicate same-index snapshot files were only logged, never
tracked, so detachOutdatedSnapshotsIfNeeded could never reclaim them — they leaked on
disk until the next restart. Track them in retained_duplicate_snapshots (kept
retained-window duplicates and re-point orphans) and retire them together with their
index. Aged out without a restart; new test RetainedDuplicateAgesOutWithoutRestart.
- detachSnapshotForRemoval now returns the detached pins (registered entry + tracked
duplicates) instead of unlinking in place, so the create path's unlink happens in
Phase 4 outside snapshots_lock (no disk I/O under the lock).
- C-F1: snapshots_queue.push can throw bad_alloc (emplaceImpl allocates before the
noexcept move); wrap it and route both the false-return and throw outcomes to the same
inline cleanup so when_done(false) fires exactly once.
- G-F2: keeper-utils snapshot-analyzer deserialized by parsed index, so with same-index
duplicates it could analyze a different file than the one named; deserialize the exact
named path instead.
- C-F4: corrected the create skip-branch log wording (condition is >=, not "bigger").
- retained_duplicate_snapshots is an unordered_map (point lookups only).
C-T1: the SameIndexCreateFailsClosedOnMissingRegisteredFile test was deliberately removed
(it asserted a fail-closed existsFile probe the birth-unique-names design no longer has — a
registered same-index entry is pinned by protected_pending_snapshot_log_idx so retention
cannot unlink it before apply); the adopt site documents this in a comment.
Known untested corners (pre-accepted, not regressions): the cross-disk re-point orphan path
and the create-path Phase-4 deferred unlink (gtest covers the synchronous removeSnapshot
path), the keeper-utils analyzer (no gtest harness), the push bad_alloc leg, and the
deferred receive-path I/O under snapshots_lock (multi-disk only).
Co-Authored-By: Claude Fable 5 <[email protected]>
Snapshot files are now created with birth-unique names
(`snapshot_<idx>_<uuid>.bin[.zstd]`), which removes same-path write races entirely:
local snapshot creation writes outside `snapshots_lock` and publishes metadata under
it (adopting a concurrently published snapshot instead of overwriting it), so
`read_logical_snp_obj` no longer waits behind a full snapshot write.
`apply_snapshot` no longer blocks on in-flight create tasks
(`KeeperStateMachine::storage` is a `shared_ptr` captured by the task), so no
claim/deferral machinery is needed. Startup keeps duplicate files for retained
indexes as redundant recovery points and deletes duplicates outside the retention
window. S3 keys stay legacy-shaped via a canonical name mapping.
Also fix failed snapshot-queue-push cleanup so `when_done(false)` is invoked exactly
once, and make the `KeeperStorageSnapshot` constructor exception-safe.
Co-Authored-By: Claude Fable 5 <[email protected]>
Split request processing (batching, session management, request/response
threads) into a separate `KeeperRequestDispatcher` class, leaving
`KeeperDispatcher` as a thin shell for snapshots, session ID assignment,
session cleanup, and cluster configuration management.
Key interface changes:
- `KeeperStateMachine` now takes a `KeeperResponseCallback` instead of
a `ResponsesQueue &`, decoupling the state machine from the queue type.
- `KeeperServer` constructor takes `KeeperResponseCallback` instead of
`ResponsesQueue &`.
- `KeeperServer::KeeperRaftServer` moved from .cpp anonymous definition
to the header for visibility.
- `ZooKeeperResponseCallback` moved to `KeeperCommon.h`, returns `bool`
for future flow control.
- `shutdown` now takes a `closed_all_connections` parameter.
- Consolidated `finishSession` calls into a single `SCOPE_EXIT` in
`KeeperTCPHandler`.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Add `static` to file-local functions and forward declarations
for cross-TU functions to satisfy `-Wmissing-prototypes`.
This ensures all functions have visible prototypes before their
definitions, improving code quality and catching accidental
linkage issues.
Reference: https://github.com/ClickHouse/ClickHouse/issues/76944
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
zeekay
Michael Kolupaev