13,838 upstream commits, forward from the 26.6.1.1 fork point. 129 conflicts:
37 resolved mechanically (take upstream's file, re-apply the brand sweep) and
92 by hand where our fork and upstream had both changed the same code.
The rule throughout: upstream owns the engine, we own the naming and a small set
of deliberate deviations. Never resolve by discarding an upstream change — a
dropped rename or a missed new parameter becomes a compile error a thousand
files away.
Deviations verified intact after the merge:
ENABLE_XRAY still defaults OFF (CMakeLists.txt:105)
X-Datastore-User read by the server (authenticateUserByHTTP.cpp)
X-ClickHouse-* in src/ 0
hanzo/research present
system.*_log retention bound present
ArrowFlight rip held against upstream re-adding it
Two things worth naming, because neither shows up as a conflict:
src/CMakeLists.txt gained an UNGUARDED
`add_object_library(datastore_storages_arrowflight Storages/ArrowFlight)` from
upstream, for a directory our gRPC rip deleted. It sat outside every conflict
hunk, so a resolver reading only the markers would sail straight past it into
a CMake configure error. Removed; the Server/ArrowFlight sibling at line 474
stays because it is inside `if (TARGET ch_contrib::arrow_flight)`.
The brand sweep had rewritten github.com/ClickHouse/ClickHouse issue links to
github.com/Datastore/Datastore — a dead host. Those point at upstream's own
tracker and are restored, alongside the contributors list, sql.clickhouse.com
and the ClickHouse Inc copyright: attribution is theirs, not ours to rename.
base/glibc-compatibility/memcpy/memcpy.h is deleted, not kept: upstream dropped
it for LLVM-libc's, and our only change to it was a comment rebrand.
`Client::login` read `hosts_and_ports.front()`, but `hosts_and_ports` is only
filled from explicit `--host` arguments; the default host is added later, in
`Client::connect`. Running `clickhouse client --login` without `--host` (e.g.
with the host coming from `--connection`, the config file, or the
`CLICKHOUSE_HOST` environment variable) failed a libc++ hardening assertion:
`front() called on an empty vector`.
Resolve the host from the configuration when `hosts_and_ports` is empty, the
same way `connect` does.
Also fix the client docs, which listed the option as `login` instead of
`--login`.
Closes: https://github.com/ClickHouse/ClickHouse/issues/103603
Co-Authored-By: Claude Fable 5 <[email protected]>
The client-side AST fuzzer loop swallowed every exception thrown during a
fuzzing step: the `catch (...)` handler recorded the error, the loop bottom
reset it, and fuzzing continued. When the failure mode is persistent - most
importantly a dead server, where the reconnection attempt inside
`processParsedSingleQuery` throws instantly - the client sprints through the
remaining corpus at hundreds of steps per second without contacting the
server and exits with code 0, which the job reports as "Fuzzer exited with
success".
Evidence from `AST fuzzer (amd_debug, targeted)` on PR #109061
(https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=109061&sha=1419f6b7e95c81c812f945c06333998b92d56ad3&name_0=PR&name_1=AST+fuzzer+%28amd_debug%2C+targeted%29):
`fuzzer.log` records 68972 fuzzing steps, of which 52837 produced no outcome
(silent exception), while `server.log` shows the server received only 209
queries within a 47-second window before dying. On PR #106250
(https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=106250&sha=latest&name_0=PR&name_1=AST+fuzzer+%28amd_debug%2C+targeted%29)
a 5-file corpus with ~20 SELECT statements (about 20000 fuzzing steps, over
an hour of work at the measured debug-build rate of 2-6 steps/s) "completed"
in 74 seconds and the job passed. Overall, 13-33% of targeted AST fuzzer
runs per month finish in under 8 minutes of their 30-minute budget.
Client fixes (`programs/client/FuzzLoop.cpp`):
- Print thrown exceptions in the fuzzing-step catch handlers instead of
swallowing them silently, so `fuzzer.log` shows what actually happened.
- After a thrown exception, check the connection via `tryToReconnect` (the
same check `processASTFuzzerStep` does for errors received without
throwing) and stop fuzzing when the server is gone instead of uselessly
sprinting through the rest of the corpus.
- Register query parameters generated by `QueryFuzzer` (from
`getLastQueryParameters`) for client-side substitution, mirroring the
server-side fuzzer in `executeASTFuzzerQueries`; previously any fuzzed
query containing a substituted `{name:type}` parameter failed with
"Substitution ... is not set" before reaching the server.
Job fix (`ci/jobs/scripts/fuzzer/run-fuzzer.sh`):
- When the AST fuzzer walks the whole corpus and exits 0 with a meaningful
part of the FUZZ_TIME_LIMIT budget remaining, restart it (each pass uses a
fresh random seed) instead of ending the job after a couple of minutes.
This matters for the targeted variant, whose corpus is a handful of test
files. The restart is gated on a server liveness probe so a dead server
still terminates the job for classification.
The per-query session_timezone seed reads client_local_timezone, captured
once in Client::connect(). It was captured via DateLUT::instance(), which
folds in session_timezone: starting the client with an explicit
--session_timezone (on a host whose real zone differs) cached that override
value instead of the machine-local zone. A later SET session_timezone =
DEFAULT then reseeded future queries from the cached override, so server-side
literal parsing (async INSERT, SELECT) never returned to true
use_client_time_zone semantics.
Use DateLUT::serverTimezoneInstance().getTimeZone() instead, which reads the
process default directly and ignores session_timezone. On the client with
use_client_time_zone=1 setDefaultTimezone() is never called, so the process
default stays the machine-local zone captured at DateLUT construction.
Extends 04401_async_insert_use_client_time_zone to cover the
--session_timezone=UTC start followed by SET session_timezone = DEFAULT.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The previous change seeded session_timezone once in Client::connect(), so it
only reflected the use_client_time_zone value at connect time. On an already
open connection, SET use_client_time_zone / per-query SETTINGS change the
setting without reconnecting, leaving two holes:
- turning use_client_time_zone on mid-session did not seed session_timezone,
so server-side literal parsing kept using the server time zone;
- turning it off left the auto-seeded session_timezone in place, so
server-side parsing never switched back.
Capture the client local time zone once on the first connect (before the
branch that may switch the process default to the server time zone), and seed
session_timezone from it in processParsedSingleQuery for every query whose
effective use_client_time_zone is set and that does not set session_timezone
explicitly. The seed is applied inside the transient per-query settings scope,
so it is recomputed for each query and reverted afterwards, tracking
use_client_time_zone changes in both directions without leaking state.
Extend the regression test to cover the command line flag, a mid-session SET,
a per-query SETTINGS, and turning use_client_time_zone back off.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
With --use_client_time_zone=1 the client parses DateTime string literals in its
local time zone, but literals interpreted server-side (asynchronous INSERT, SELECT
literals) use the session_timezone setting, which the client left empty. The server
then fell back to its own default time zone, so an async INSERT of a string literal
stored a different instant than the equivalent synchronous INSERT.
Set session_timezone to the client local time zone in Client::connect() when
use_client_time_zone is enabled and the user has not set session_timezone explicitly,
so server-side interpretation matches the client-side (synchronous) path. An explicit
--session_timezone still takes priority.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The lowercase sweep renamed source-file references inside programs/*/CMakeLists.txt
(clickhouse-su.cpp -> datastore-su.cpp etc.) but not the physical files -> 'Cannot find source
file datastore-su.cpp'. Renamed all 19 program entry-point sources, the client config xml, and
bash-completion scripts to match. Completes the program-file rebrand.
The correct, safe broad rebrand: renames "ClickHouse" only as a whole word (boundary on
BOTH sides), so it rewrites display strings/messages/comments but leaves every compound
C++ symbol intact — `ClickHouseDictionarySource`, `getInfoIfClickHouseDictionarySource`,
`mainEntryClickHouseServer`, `ClickHouseRevision`, `ClickHouse_SOURCE_DIR` are untouched
(an earlier one-sided heuristic mangled suffix symbols like `manageClickHouse`; reverted).
1677 occurrences across 482 files. Compile-safe (string/comment content; symbols preserved).
Still NOT renamed (would be a build-coupled symbol rename, needs build-host iteration): the
199 compound identifiers containing ClickHouse, the `DB::` namespace, the `clickhouse_*`
CMake targets/macros, the lowercase `clickhouse` SQL dictionary-source keyword. Needs build
verification.
Implement https://github.com/ClickHouse/ClickHouse/issues/95479.
Previously, the client echoed queries differently depending on the mode
with no way to control it granularly: in interactive mode it always
printed a formatted and highlighted version of the query and the
`query_id`, while in batch mode `--echo` printed the raw query text only.
This change makes the behavior configurable while keeping the defaults
identical to before:
- `--echo` now takes an optional boolean value and controls echoing of
queries in both interactive and batch mode. It defaults to on in
interactive mode and off in batch mode.
- `--echo-formatted` is a new boolean option controlling whether the
echoed query is formatted. It defaults to on in interactive mode and
off in batch mode, so highlighting non-formatted queries is now
possible as well.
- `--echo-query-id` is a new boolean option controlling whether the
`query_id` is printed before execution. It defaults to on in
interactive mode and off in batch mode. When enabled in batch mode, a
fresh `query_id` is generated per query so that the printed value
matches the one used for execution.
- `--hilite` (`--highlight`) now controls highlighting of both the
command prompt and the echoed queries. Highlighting is only applied
when writing to a terminal.
The echoing logic for all modes is unified in the new `echoQuery`
method, and the per-client defaults are resolved in
`setupEchoAndHighlightSettings`.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A positional file argument is converted to --queries-file inside
ClientApplicationBaseParser via options.emplace, which populates the
boost options map (and the queries_files member) but never the Poco
config. The conflict check in processConfig used config.has("queries-file"),
so the positional case slipped through and the -q query was silently
dropped instead of erroring.
Check queries_files directly so both forms are rejected consistently.
Closes#104735
* Make client rainbow parentheses optional
* Make client rainbow parentheses optional
* Make client rainbow parentheses optional
* Make client rainbow parentheses optional
* Fix style and clang-tidy CI failures
- In `programs/format/Format.cpp`, replace `options.count(...)` with
`options.contains(...)` to satisfy the `readability-container-contains`
clang-tidy check that fires in the `arm_tidy` build.
- In `src/Client/ClientBaseHelpers.cpp`, drop redundant braces around
single-statement `if (!rainbow_parentheses) ...` blocks. One of them
had the opening brace on the same line, which the C++ style check
rejects (Allman braces are required).
CI report: https://github.com/ClickHouse/ClickHouse/pull/103851
---------
Co-authored-by: Alexey Milovidov <[email protected]>
zeekay
Alexey Milovidov