13,838 upstream commits, forward from the 26.6.1.1 fork point. 129 conflicts:
37 resolved mechanically (take upstream's file, re-apply the brand sweep) and
92 by hand where our fork and upstream had both changed the same code.
The rule throughout: upstream owns the engine, we own the naming and a small set
of deliberate deviations. Never resolve by discarding an upstream change — a
dropped rename or a missed new parameter becomes a compile error a thousand
files away.
Deviations verified intact after the merge:
ENABLE_XRAY still defaults OFF (CMakeLists.txt:105)
X-Datastore-User read by the server (authenticateUserByHTTP.cpp)
X-ClickHouse-* in src/ 0
hanzo/research present
system.*_log retention bound present
ArrowFlight rip held against upstream re-adding it
Two things worth naming, because neither shows up as a conflict:
src/CMakeLists.txt gained an UNGUARDED
`add_object_library(datastore_storages_arrowflight Storages/ArrowFlight)` from
upstream, for a directory our gRPC rip deleted. It sat outside every conflict
hunk, so a resolver reading only the markers would sail straight past it into
a CMake configure error. Removed; the Server/ArrowFlight sibling at line 474
stays because it is inside `if (TARGET ch_contrib::arrow_flight)`.
The brand sweep had rewritten github.com/ClickHouse/ClickHouse issue links to
github.com/Datastore/Datastore — a dead host. Those point at upstream's own
tracker and are restored, alongside the contributors list, sql.clickhouse.com
and the ClickHouse Inc copyright: attribution is theirs, not ours to rename.
base/glibc-compatibility/memcpy/memcpy.h is deleted, not kept: upstream dropped
it for LLVM-libc's, and our only change to it was a comment rebrand.
The client-side AST fuzzer loop swallowed every exception thrown during a
fuzzing step: the `catch (...)` handler recorded the error, the loop bottom
reset it, and fuzzing continued. When the failure mode is persistent - most
importantly a dead server, where the reconnection attempt inside
`processParsedSingleQuery` throws instantly - the client sprints through the
remaining corpus at hundreds of steps per second without contacting the
server and exits with code 0, which the job reports as "Fuzzer exited with
success".
Evidence from `AST fuzzer (amd_debug, targeted)` on PR #109061
(https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=109061&sha=1419f6b7e95c81c812f945c06333998b92d56ad3&name_0=PR&name_1=AST+fuzzer+%28amd_debug%2C+targeted%29):
`fuzzer.log` records 68972 fuzzing steps, of which 52837 produced no outcome
(silent exception), while `server.log` shows the server received only 209
queries within a 47-second window before dying. On PR #106250
(https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=106250&sha=latest&name_0=PR&name_1=AST+fuzzer+%28amd_debug%2C+targeted%29)
a 5-file corpus with ~20 SELECT statements (about 20000 fuzzing steps, over
an hour of work at the measured debug-build rate of 2-6 steps/s) "completed"
in 74 seconds and the job passed. Overall, 13-33% of targeted AST fuzzer
runs per month finish in under 8 minutes of their 30-minute budget.
Client fixes (`programs/client/FuzzLoop.cpp`):
- Print thrown exceptions in the fuzzing-step catch handlers instead of
swallowing them silently, so `fuzzer.log` shows what actually happened.
- After a thrown exception, check the connection via `tryToReconnect` (the
same check `processASTFuzzerStep` does for errors received without
throwing) and stop fuzzing when the server is gone instead of uselessly
sprinting through the rest of the corpus.
- Register query parameters generated by `QueryFuzzer` (from
`getLastQueryParameters`) for client-side substitution, mirroring the
server-side fuzzer in `executeASTFuzzerQueries`; previously any fuzzed
query containing a substituted `{name:type}` parameter failed with
"Substitution ... is not set" before reaching the server.
Job fix (`ci/jobs/scripts/fuzzer/run-fuzzer.sh`):
- When the AST fuzzer walks the whole corpus and exits 0 with a meaningful
part of the FUZZ_TIME_LIMIT budget remaining, restart it (each pass uses a
fresh random seed) instead of ending the job after a couple of minutes.
This matters for the targeted variant, whose corpus is a handful of test
files. The restart is gated on a server liveness probe so a dead server
still terminates the job for classification.
The correct, safe broad rebrand: renames "ClickHouse" only as a whole word (boundary on
BOTH sides), so it rewrites display strings/messages/comments but leaves every compound
C++ symbol intact — `ClickHouseDictionarySource`, `getInfoIfClickHouseDictionarySource`,
`mainEntryClickHouseServer`, `ClickHouseRevision`, `ClickHouse_SOURCE_DIR` are untouched
(an earlier one-sided heuristic mangled suffix symbols like `manageClickHouse`; reverted).
1677 occurrences across 482 files. Compile-safe (string/comment content; symbols preserved).
Still NOT renamed (would be a build-coupled symbol rename, needs build-host iteration): the
199 compound identifiers containing ClickHouse, the `DB::` namespace, the `clickhouse_*`
CMake targets/macros, the lowercase `clickhouse` SQL dictionary-source keyword. Needs build
verification.
The AST fuzzer was reporting false failures when the server hit its
memory limit. `MEMORY_LIMIT_EXCEEDED` means the server correctly
rejected an expensive fuzzed query, not that it died.
Two fixes:
- In `FuzzLoop.cpp`, treat `MEMORY_LIMIT_EXCEEDED` like
`TOO_DEEP_RECURSION`: clear the error and continue fuzzing instead of
attempting reconnection, failing, and exiting with code 241.
- In `run-fuzzer.sh`, treat `MEMORY_LIMIT_EXCEEDED` during the
post-fuzzing liveness check as a transient condition (like
`TOO_MANY_SIMULTANEOUS_QUERIES`) instead of declaring the server dead.
https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=102000&sha=latest&name_0=PR&name_1=AST+fuzzer+%28arm_asan_ubsan%29
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Resolve merge conflicts:
- `base/base/coverage.cpp`: take master's version which removed the old
sanitizer coverage guard/trace functions (replaced by LLVM profile-based
coverage)
- `src/Client/BuzzHouse/AST/SQLProtoStr.cpp`: take master's unified
`CONV_FN(SQLIdentifier, ident)` replacing the individual per-type
converter functions
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Fix `io_destroy` prototype mismatch in `AIO.h` for FreeBSD (declared as
`int io_destroy(void)` but defined as `int io_destroy(int ctx)`)
- Add forward declarations for `LLVMFuzzerInitialize` and
`LLVMFuzzerTestOneInput` in fuzzer files, and make file-local helper
functions `static` to fix `-Wmissing-prototypes` in arm_fuzzers build
- Add `extern "C"` forward declaration for `reallocarray` in `malloc.cpp`
to fix `-Wmissing-prototypes` in ppc64le build
- Suppress `-Wdisabled-macro-expansion` in `FuzzLoop.cpp` because musl
defines `stderr` as `(stderr)` which is a recursive macro
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Remove `CurrentThread.h` from high-fan-out headers: `ProcessList.h`,
`IQueryPlanStep.h`, `DDLWorker.h`, `TCPHandler.h`, `AggregatingTransform.h`,
`ParallelParsingInputFormat.h`, `StorageTableFunction.h`, `parseSyslogLevel.h`,
`EmbeddedRocksDBBulkSink.h`, `BlockIO.h`, `LocalConnection.h`,
`StorageMaterializedView.h`, `HTTPHandler.h`
- Extract `QueryScope` from `CurrentThread` into standalone `Common/QueryScope.h`
with its own `.cpp`, replacing the nested `CurrentThread::QueryScope` class
- Extract `getCurrentThreadGroup` free function into `Common/ThreadGroupSwitcher.h`,
replacing `CurrentThread::getGroup()` in template headers:
`threadPoolCallbackRunner.h`, `ParallelFormattingOutputFormat.h`,
`HashedDictionaryParallelLoader.h`, `UniqExactSet.h`, `HashedDictionary.h`,
`StatusRequestsPool.h`
- De-inline `MemoryTrackerSwitcher` ctor/dtor into `MemoryTrackerSwitcher.cpp`
- De-inline `ResourceGuard::Request::wait` into `ResourceGuard.cpp`
- De-inline `ManyAggregatedData` destructor into `AggregatingTransform.cpp`
- De-inline `QueryStatus::updateProgressIn/Out` into `ProcessList.cpp`
- Add explicit `CurrentThread.h` includes to `.cpp` files that relied on
transitive includes through the trimmed headers
Co-Authored-By: Claude Opus 4.6 <[email protected]>
zeekay
Pedro Ferreira