Introduce "drivers" for executable user-defined functions. A driver is a
small adapter program declared in a server config under
`<user_defined_executable_function_drivers_config>` that knows how to take a
user-supplied code snippet and produce a runnable executable user-defined
function configuration.
New CREATE FUNCTION syntax:
CREATE [OR REPLACE] FUNCTION [IF NOT EXISTS] name [ON CLUSTER c]
ARGUMENTS (a UInt8, b String) RETURNS UInt64
ENGINE = DriverName(key1 = 'value1', key2 = 42)
AS '...code body...'
On CREATE FUNCTION, the corresponding `create_command` of the driver is
invoked: arguments are passed via `--name`, `--return`, `--args` and the
declared engine arguments; the body is sent on stdin; the driver writes the
resulting executable UDF configuration (XML or YAML) to stdout, which is
saved into the new `<dynamic_user_defined_executable_functions_path>` and
picked up by the existing executable UDF loader. The corresponding SQL is
persisted as `ATTACH FUNCTION ...` in `<user_defined>` and re-evaluated on
restart: if the dynamic configuration is still present, the driver is not
invoked; otherwise the driver is re-run to materialize it.
On DROP FUNCTION, the `drop_command` (if any) is invoked, then the dynamic
configuration file and per-function working directory are removed.
Includes a proof-of-concept driver `c_function_body`, which compiles a user
C body into a binary (inside Docker by default for build-time isolation) and
runs it inside Docker at query time with strict sandboxing (`--network=none`,
`--read-only`, dropped capabilities, no-new-privileges, unprivileged user,
resource limits) in `executable_pool` mode for batched per-row evaluation.
See https://github.com/ClickHouse/ClickHouse/issues/71172
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
A default `clickhouse-local` invocation has no system log sections in the config.
Initializing `SystemLogs` regardless meant the unique pointer in `ContextSharedPart`
was written during construction and again during teardown, racing under TSan with a
concurrent read from a `BackgroundSchedulePool` worker that calls
`getBackgroundSchedulePoolLog` after a task finishes. The race surfaced as a
"Scraping system tables" failure in the `Stateless tests (amd_tsan, parallel, 1/2)`
job, since CI scrapes system tables via short-lived `clickhouse-local` processes.
Gate the call on the config actually containing at least one entry from
`LIST_OF_ALL_SYSTEM_LOGS`. The custom config used by `04093_system_log_alter_column`
declares `query_log` and `processors_profile_log`, so the regression test continues
to exercise the original fix; default invocations stop touching `system_logs`.
CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=102316&sha=665b5c743abc6d9c6ac9946eb60d70df95ce08c8&name_0=PR&name_1=Stateless%20tests%20%28amd_tsan%2C%20parallel%2C%201%2F2%29
PR: https://github.com/ClickHouse/ClickHouse/pull/102316
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
When `--no-system-tables` is set, the system database is not attached,
so initializing system logs would fail or violate the flag semantics.
Skip the call in that case, consistent with how system table attachment
is gated.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Add `initializeSystemLogs` call to `LocalServer` so that
`clickhouse-local` creates system log tables (like `query_log`,
`processors_profile_log`) when they are configured via `--config-file`.
Without a config, this is a no-op.
Rewrite the test to run entirely in `clickhouse-local` with a custom
config, avoiding ALTER on the shared server where materialized views
prevent column modifications.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The original PR #96025 was reverted (#98038) because:
1. The install completion message showed wrong relative paths like
`var/run/clickhouse-server` instead of absolute `/var/run/clickhouse-server`.
This happened because `options.contains()` returns true even for options
with default values in boost::program_options. Fix: use `defaulted()`
to only include options that the user explicitly specified.
2. The install completion message did not suggest `sudo` for the
start/restart command. This happened because `getuid() != 0` is false
when install runs as root via sudo, but the user returns to their
unprivileged shell afterward. Fix: always suggest sudo (matching the
original behavior).
Also address review feedback:
- Fix `clickhouse-client` help: remove misleading "initial table definition"
(a `clickhouse-local` concept) from usage line
- Fix typo: "as command line tool" → "as a command line tool"
- Fix `clickhouse-local` help grammar: "After you can execute" → "After that,
you can execute", "in first query" → "in the first query"
- Make keeper `--help` test robust for builds without `ENABLE_CLICKHOUSE_KEEPER`
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
`MemoryWorker` was only started in `clickhouse-server` but not in
`clickhouse-local`. This caused three problems:
1. The userspace page cache stayed stuck at `page_cache_min_size`
(100 MiB by default) and never grew to `page_cache_max_size`,
because `PageCache::autoResize` — which adjusts the cache size
based on available memory — was never called. The result was severe
cache thrashing: blocks evicted immediately after fetching, turning
every "hot" run into a cold run with full S3 re-downloads.
2. RSS was never reported to the global `MemoryTracker`, so memory
tracking in `clickhouse-local` was less accurate.
3. Jemalloc dirty pages were never purged under memory pressure.
Start `MemoryWorker` unconditionally (not only when page cache is
enabled), matching `clickhouse-server` behavior.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The userspace page cache (`page_cache_max_size`) was never initialized
in `clickhouse-local`. The `setPageCache` call only existed in
`programs/server/Server.cpp` but was missing from
`programs/local/LocalServer.cpp`.
This adds the same page cache initialization (including validation
of `page_cache_min_size` vs `page_cache_max_size`) to `clickhouse-local`,
enabling the page cache for S3/object storage queries.
Closes https://github.com/ClickHouse/ClickHouse/issues/99499
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- `Intrinsic::getDeclaration` renamed to `Intrinsic::getOrInsertDeclaration`
in LLVM 22 (abs.cpp, bitCount.cpp)
- `createTargetMachine` first argument changed from `StringRef` to
`const Triple&` — construct `llvm::Triple` explicitly (CHJIT.cpp)
- `std::__gcd` internal removed from libc++ — use `std::gcd` from
`<numeric>` (AggregateFunctionKolmogorovSmirnovTest.cpp)
- Added `(void)` casts for new `[[nodiscard]]` on `std::stod`,
`std::filesystem::is_directory`, and `future.get()`
(KQLGeneralFunctions.cpp, LocalServer.cpp,
ZooKeeperRemoveByList.cpp, EphemeralLockInZooKeeper.cpp)
Assisted-by: Claude Opus 4.6 via GitHub Copilot
- Add missing `extern` for `jemalloc_profiler_sampling_rate` in
`Keeper.cpp`, `LocalServer.cpp`, and `Server.cpp`
- Pass `jemalloc_profiler_sampling_rate` as 5th argument to
`Jemalloc::setup` in `Keeper.cpp` and `LocalServer.cpp`
- Expose `prof.lg_sample` via `system.asynchronous_metrics` as
`jemalloc.prof.lg_sample`
- Add integration test `test_jemalloc_profiler_sampling_rate`
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Extract `ErrnoException` into `Common/ErrnoException.h` to reduce the
include footprint of `Exception.h`. This removes `<base/errnoToString.h>`
and `<cerrno>` from `Exception.h`, which is included by ~95% of all
translation units.
All files referencing `ErrnoException` now include the new header
directly. Files that used `errnoToString` without `ErrnoException`
got an explicit `<base/errnoToString.h>` include.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Allow overriding the default access control improvements via an
explicitly provided config file, instead of unconditionally hardcoding
them to true.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Wrap long lines in `getHelpHeader` and `getHelpFooter` for `clickhouse client`
and `clickhouse local` to keep lines within reasonable width,
fixing the 00910_client_window_size_detection test
- Use `createOptionsDescription` instead of raw `po::options_description` constructor
in `git-import` and `static-files-disk-uploader` to handle zero terminal width
properly, fixing the `git-import --help` abort in debug builds
(03823_clickhouse_help_return_codes test)
Co-Authored-By: Claude Opus 4.6 <[email protected]>
This commit improves the help output for all ClickHouse applications:
- Add `clickhouse help` command that lists all available subcommands
- Return exit code 0 for all help messages (previously some returned 1)
- Output help messages to stdout instead of stderr
- Use "clickhouse <cmd>" format instead of argv[0] in help messages
- Add dynamic `app_name` detection for `clickhouse client` and `clickhouse local`
to show correct invocation name based on how the binary was called
- Add `--no-sudo` option for `clickhouse start` and `clickhouse restart`
for environments without sudo (e.g., Docker containers)
- Add detailed help headers and footers for client and local with usage examples
Closes#58244
Co-Authored-By: Claude Opus 4.5 <[email protected]>
Set all `access_control_improvements` options to true for stricter
access control behavior in the embedded server config and clickhouse-local.
Co-Authored-By: Claude Opus 4.5 <[email protected]>
The default user in clickhouse-local was missing the access_management
privilege, which caused operations like DROP ROW POLICY IF EXISTS to
fail with ACCESS_DENIED error, even though the user should be unrestricted.
Co-Authored-By: Claude Opus 4.5 <[email protected]>
