Embed xterm.js 5.5.0, addon-fit 0.10.0, and addon-web-links 0.11.0
(minified JS and CSS) into the server binary, served at /js/*.
This removes the dependency on CDN and makes the web terminal
page fully self-contained, like the dashboard.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Erase the "Connecting..." line before PTY output starts, so the
terminal content renders from a clean line. Also erase it on
auth failure before showing the error prompt.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The C++ `#embed` directive includes file contents at compile time,
but the build system does not automatically track these as
dependencies. When an embedded file (e.g., `play.html`) changes,
the .cpp file that embeds it is not recompiled.
Fix this by adding `OBJECT_DEPENDS` properties to all source files
that use `#embed`, covering: HTML pages, JavaScript files, XML
configs, and NLP data files.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Remove the overlay login form; the terminal fills the entire page
- Auto-connect with default credentials (or credentials from URL params)
- On auth failure, prompt for username/password directly in the terminal
- If user was provided in the URL, only prompt for password
- After session ends, prompt for credentials again to reconnect
- Hide the scrollbar while keeping the terminal scrollable
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Add a WebSocket-based terminal protocol (webterminal) that runs
a pseudoterminal with embedded clickhouse-client functionality,
accessible via a browser at /webterminal.
The implementation reuses the same `PtyClientDescriptorSet` and
`ClientEmbeddedRunner` infrastructure as the SSH protocol, but
connects via WebSocket instead of SSH. Authentication works the
same way as in the HTTP protocol (URL params, Basic auth,
X-ClickHouse headers).
The HTML page uses xterm.js for terminal emulation and connects
to the server via WebSocket, bridging data bidirectionally
between the browser and the PTY.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Two issues remained after the previous fix:
1. The ASan stack-size warning block (4 lines) was only partially
filtered in `clickhouse_proc.py`. The last line:
"For details see https://github.com/google/sanitizers/issues/189"
was not excluded, leaving `sanitizer_hits` non-empty and triggering
a spurious "Unknown error / Lost connection to server" result for
jobs that had no real sanitizer errors. Add the missing exclusion.
2. OpenSSL's EVP method cache (`evp_md_new`, `construct_evp_method`,
`CRYPTO_THREAD_lock_new`) is never freed at process exit. These
objects are allocated when the S3 client initializes HMAC via
`AWS SDK -> HMAC_Init_ex -> EVP_MD_fetch`. LSan reports them as
leaks, causing the server to abort with SIGABRT (`halt_on_error=1`),
which `clickhouse-test` records as "Unknown error / Lost connection
to server". Add suppressions to `__lsan_default_suppressions`.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
`clickhouse local` (used for post-test log collection) was aborting at
exit under ASan+LSan with 45 KB of leaks from OpenSSL provider
initialization (`ossl_provider_new`, `OSSL_PROVIDER_try_load_ex`, etc.).
OpenSSL 3.x intentionally does not free all global state at exit, so
these are false positives.
Added `__lsan_default_suppressions` alongside the existing
`__lsan_default_options` in `programs/main.cpp` to suppress the five
known OpenSSL leak patterns. This fixes the "Unknown error / Lost
connection to server" failures in:
- Stateless tests (amd_asan_ubsan, distributed plan, parallel, 1/2)
- Stateless tests (amd_asan_ubsan, distributed plan, parallel, 2/2)
- Stateless tests (arm_asan_ubsan, azure, sequential)
CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=99657&sha=24b9445c3f9c8dfc420b55f1a2fb369ae17220be&name_0=PR
Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
shell-like completion support in clickhouse keeper-client (handle
completions of arguments in quotes, i.e. 'foo ba', handle escaped
arguments, i.e. foo\ ba, make ls print quoted nodes, if they have
whitespaces)
Also remove on_complete_modify_callback hack, replace it with pure
getCompletions implementation
Co-Authored-By: Claude <[email protected]>
The userspace page cache (`page_cache_max_size`) was never initialized
in `clickhouse-local`. The `setPageCache` call only existed in
`programs/server/Server.cpp` but was missing from
`programs/local/LocalServer.cpp`.
This adds the same page cache initialization (including validation
of `page_cache_min_size` vs `page_cache_max_size`) to `clickhouse-local`,
enabling the page cache for S3/object storage queries.
Closes https://github.com/ClickHouse/ClickHouse/issues/99499
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Reset elapsed/total timers after warmup/setup so setup time is not
counted toward throughput or time limit
- Rewrite `removeRecursive` to collect deletes in a single post-order
batch (flushed every 1000 ops) instead of removing children then parent
in separate rounds, reducing the number of multi requests
- Add proper error handling for list and multi responses
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Simplify jemalloc integration by replacing the three-mechanism approach
(`JEMALLOC_IS_MALLOC` + `JEMALLOC_OVERRIDE___LIBC_*` + linker `--wrap`)
with a cleaner two-step design:
1. Compile jemalloc with `je_` prefix so it only exports `je_malloc`,
`je_free`, etc. (matching what Darwin already does).
2. Define our own `malloc`/`free` in `malloc.cpp` that perform memory
tracking and delegate to `je_malloc`/`je_free`. Provide `__libc_*`
aliases on glibc to intercept internal calls.
Changes:
- Enable `JEMALLOC_PREFIX "je_"` in all non-Darwin `jemalloc_internal_defs.h.in`
- Undef `JEMALLOC_IS_MALLOC` and all `JEMALLOC_OVERRIDE___LIBC_*` defines
- Remove `JEMALLOC_OVERRIDE_MEMALIGN`/`VALLOC`/`PVALLOC` from `jemalloc_defs.h`
- Add `JEMALLOC_NO_RENAME` interface define so consumer code sees `je_*` names
- Prefix all jemalloc API calls (`mallctl` -> `je_mallctl`, etc.) in ClickHouse
- Rewrite `memory.h` to call `je_malloc`/`je_free` directly instead of `__real_*`
- Rewrite `malloc.cpp` with tracking wrappers calling `je_*` underneath
- Rewrite `AllocationInterceptors.h` to map `__real_*` -> `je_*` or `::*`
- Remove `__wrap_*` functions from `AllocationInterceptors.cpp`
- Remove `--wrap` linker flags from `CMakeLists.txt`
- Add jemalloc + clickhouse_common_io deps to `clickhouse_malloc` OBJECT library
- And also fix JEMALLOC_CPREFIX for Darwin, that was the only platform that uses JE_MALLOC_CONF (over MALLOC_CONF)
Co-Authored-By: Claude <[email protected]>
- Remove `CurrentThread.h` from high-fan-out headers: `ProcessList.h`,
`IQueryPlanStep.h`, `DDLWorker.h`, `TCPHandler.h`, `AggregatingTransform.h`,
`ParallelParsingInputFormat.h`, `StorageTableFunction.h`, `parseSyslogLevel.h`,
`EmbeddedRocksDBBulkSink.h`, `BlockIO.h`, `LocalConnection.h`,
`StorageMaterializedView.h`, `HTTPHandler.h`
- Extract `QueryScope` from `CurrentThread` into standalone `Common/QueryScope.h`
with its own `.cpp`, replacing the nested `CurrentThread::QueryScope` class
- Extract `getCurrentThreadGroup` free function into `Common/ThreadGroupSwitcher.h`,
replacing `CurrentThread::getGroup()` in template headers:
`threadPoolCallbackRunner.h`, `ParallelFormattingOutputFormat.h`,
`HashedDictionaryParallelLoader.h`, `UniqExactSet.h`, `HashedDictionary.h`,
`StatusRequestsPool.h`
- De-inline `MemoryTrackerSwitcher` ctor/dtor into `MemoryTrackerSwitcher.cpp`
- De-inline `ResourceGuard::Request::wait` into `ResourceGuard.cpp`
- De-inline `ManyAggregatedData` destructor into `AggregatingTransform.cpp`
- De-inline `QueryStatus::updateProgressIn/Out` into `ProcessList.cpp`
- Add explicit `CurrentThread.h` includes to `.cpp` files that relied on
transitive includes through the trimmed headers
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Dictionary layout types (`complex_key_hashed`, `range_hashed`,
`ssd_cache`, etc.), source types, and source/layout parameter names
(`PORT`, `PASSWORD`, `DB`, `SIZE_IN_CELLS`, `BLOCK_SIZE`, `SHARDS`,
etc.) were being obfuscated, producing invalid SQL output.
Add `getAllRegisteredNames` to `DictionaryFactory` and
`DictionarySourceFactory`, register their names as known identifiers,
add dictionary parameter keywords to the obfuscator keyword list,
and remove the now-redundant hardcoded dictionary layout/source
keywords that are covered by the factory-based approach.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add `DatabaseFactory::getAllRegisteredNames` to the known identifiers
set so database engine names like `Atomic`, `Memory`,
`MaterializedPostgreSQL`, etc. are not obfuscated.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Skip index type names (`minmax`, `set`, `bloom_filter`, `ngrambf_v1`,
`tokenbf_v1`, etc.) and compression codec names (`LZ4`, `ZSTD`, `Delta`,
etc.) were being obfuscated, producing invalid SQL output.
Add `getAllRegisteredNames` to `MergeTreeIndexFactory` and
`CompressionCodecFactory`, and register their names as known identifiers
in the obfuscator.
Closes#99248
Co-Authored-By: Claude Opus 4.6 <[email protected]>
`Stats::report` already returns early when `elapsed.elapsedSeconds()` is
zero, but `Stats::writeJSON` did not, causing `getThroughput(0)` to
produce infinity values that render invalid JSON.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Two fixes:
1. Measure request latency in the response callback instead of when
`collect_request` drains the in-flight deque. Previously the
`Stopwatch` ran from submission until collection, which at high
`pipeline_depth` included time the request sat completed in the
deque waiting to be collected. Now a shared `Stopwatch` is captured
by the callback and read when the response actually arrives.
2. Compute RPS using wall-clock elapsed time (`Stopwatch`) instead of
deriving it from `sum(individual_latencies) / concurrency`. The old
formula did not account for pipelining: with `pipeline_depth=64` it
reported ~2k RPS while actual wall-clock throughput was ~140k RPS.
`getThroughput` now takes elapsed seconds directly, and the
`concurrency` parameter is removed from `report` and `writeJSON`.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Add `// Ok` comments to `catch (...)` blocks in `Runner.cpp` that handle
exceptions via `handle_request_exception` (which logs and counts errors)
but were flagged by the `catch_all` style check.
Fix seed collision in `MultiRequestGenerator::setSeedImpl`: the inner
`RequestGetter::setSeed` distributed sub-seeds as `seed + i + 1`, which
collided with siblings assigned by the outer `RequestGetter` using the
same scheme. Use a large prime offset (500009) to separate the inner
seed namespace.
https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=98906&sha=latest&name_0=PR
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Restore batched child removal in `removeRecursive` using `multi` requests
in chunks of 1000, reducing network round trips from O(N) to O(N/1000).
Add `std::lock_guard` in `Stats::report` and `Stats::writeJSON` so that
collector fields are consistently protected by `Stats::mutex` regardless
of caller context.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Move the title-setting code above the error branch so the title
text is populated even when the query returns an empty result.
Previously, the title div was shown but empty because the title
text was only set after the early return on error.
Previously, when a dashboard query returned an empty result or
encountered an error, both the chart title and the error message
were shown in the same space with the title hidden. This made it
impossible to identify which chart had the issue.
Keep the chart title visible in error state and position the error
message below it to avoid overlap.
Fixes#85225
`addOp` was modifying `op_collectors` without acquiring `Stats::mutex`,
unlike `addRead` and `addWrite` which both lock it. Currently safe
because callers hold `Runner::mutex`, but would race if used from the
log-replay path where only `Stats::mutex` is relied upon.
Also update the comment to reflect the actual invariant: access is
protected by `Stats::mutex`, not `Runner::mutex`.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
