Call `loadUserDefinedExecutableFunctionDrivers` during initial
`clickhouse-server` startup before persisted user-defined functions are loaded.
This makes source-tree driver configs such as `c_function_body` available
without waiting for a later config reload.
Resolve driver config directories to absolute paths before applying relative
`create_command` and `drop_command` entries. Driver commands run from a
per-function work directory, so a relative config directory would otherwise make
source-tree example commands relative to the wrong directory.
Improve the unknown-driver exception so it distinguishes an empty driver
registry from a typo in the requested driver name.
Tests:
`ninja -C build programs/clickhouse`
`CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
`CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
`clickhouse-server` default-config smoke with `c_function_body`
Refs: https://github.com/ClickHouse/ClickHouse/pull/105131
Introduce "drivers" for executable user-defined functions. A driver is a
small adapter program declared in a server config under
`<user_defined_executable_function_drivers_config>` that knows how to take a
user-supplied code snippet and produce a runnable executable user-defined
function configuration.
New CREATE FUNCTION syntax:
CREATE [OR REPLACE] FUNCTION [IF NOT EXISTS] name [ON CLUSTER c]
ARGUMENTS (a UInt8, b String) RETURNS UInt64
ENGINE = DriverName(key1 = 'value1', key2 = 42)
AS '...code body...'
On CREATE FUNCTION, the corresponding `create_command` of the driver is
invoked: arguments are passed via `--name`, `--return`, `--args` and the
declared engine arguments; the body is sent on stdin; the driver writes the
resulting executable UDF configuration (XML or YAML) to stdout, which is
saved into the new `<dynamic_user_defined_executable_functions_path>` and
picked up by the existing executable UDF loader. The corresponding SQL is
persisted as `ATTACH FUNCTION ...` in `<user_defined>` and re-evaluated on
restart: if the dynamic configuration is still present, the driver is not
invoked; otherwise the driver is re-run to materialize it.
On DROP FUNCTION, the `drop_command` (if any) is invoked, then the dynamic
configuration file and per-function working directory are removed.
Includes a proof-of-concept driver `c_function_body`, which compiles a user
C body into a binary (inside Docker by default for build-time isolation) and
runs it inside Docker at query time with strict sandboxing (`--network=none`,
`--read-only`, dropped capabilities, no-new-privileges, unprivileged user,
resource limits) in `executable_pool` mode for batched per-row evaluation.
See https://github.com/ClickHouse/ClickHouse/issues/71172
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
`DNSResolver::Impl::Impl()` was eagerly running
`Poco::Net::DNS::hostName()` followed by `hostByName` on the local
hostname. The cached result is consumed only by
`updateHostNameAndAddresses` (a server-only periodic task) and by
`getHostName`, which has its own lazy fallback. Yet the work happened
inside a process-wide Meyers singleton, so every program that calls
`DNSResolver::instance` paid the cost — including `clickhouse-client`,
`-local`, `-keeper`, and `-disks`.
When the local hostname could not be resolved (a normal occurrence on
fresh CI/EC2 runners where the auto-generated `ip-X-X-X-X` name is
neither in `/etc/hosts` nor served by the VPC resolver), the catch
block in the constructor logged a stack trace at error level. The
fast-test runner treats any stderr output as a test failure, so this
infrastructure flake masqueraded as a per-test failure.
Move the eager priming out of the singleton constructor:
* `Impl::Impl` is now defaulted; non-server programs no longer touch
the local hostname during startup.
* `programs/server/Server.cpp` calls `updateHostNameAndAddresses`
explicitly before starting `DNSCacheUpdater`, so the first cache
tick on a healthy server still sees a populated value and does not
spuriously emit `IPs of host name X have been changed`.
Behaviour from PR #93726 (cluster config reload on host IP change) is
preserved — verified by `test_reload_cluster_config_if_host_address_change`,
which still passes.
Failing fast tests in https://github.com/ClickHouse/ClickHouse/pull/103689
(report:
https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=103689&sha=3a119a57269adba3557204ce44e58a65054e31df&name_0=PR&name_1=Fast%20test
)
include `Tests/02001_hostname_test`, `Tests/03213_deep_json`,
`Tests/03638_merge_max_dynamic_subcolumns_in_compact_part`, and
`Tests/02433_default_expression_operator_in`, all hitting the same
`Cannot resolve host (ip-172-31-33-233)` startup path.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Drop two heuristics added during PR development that override explicit
user-configured ratios on small systems:
- Forced uplift of `max_server_memory_usage_to_ram_ratio` to `0.95` when
RAM < 4 GiB.
- Forced cap of `merges_mutations_memory_usage_to_ram_ratio` to `0.1`
when RAM < 4 GiB.
Per reviewer feedback: too ad-hoc, removing for now. The remaining
auto-tuning (Parquet watermark, INSERT block size, background pool,
jemalloc dirty decay) is sufficient for the low-memory case and does not
silently override explicit operator settings.
After merging master, two new call sites in `programs/server/Server.cpp`
(lines 2847-2848) pass only three arguments to `MergeTreeSettings::sanityCheck`,
which this branch expanded to take four arguments (adding the
`background_pool_auto_lowered` flag).
Pass `global_context->wasBackgroundPoolAutoLowered()` at both call sites so
the startup-time sanity check agrees with the per-table checks in
`MergeTreeData.cpp`.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Store `Poco::Timespan::TimeDiff` (int64_t) instead of `Poco::Timespan`
in `SettingFieldTimespan`. Poco::Timespan has user-defined copy/dtor
that prevent trivial copyability despite wrapping a single int64_t.
This makes 46 settings (22 Seconds + 28 Milliseconds fields across
Settings alone) trivially copyable, enabling the compiler to include
them in the memcpy prefix of the Data copy constructor.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Set a 30-second `setReceiveTimeout` in the main WebSocket I/O loop to
prevent a malicious client from blocking the handler thread by trickling
bytes after `poll` indicates readability. On timeout, the connection is
terminated (not continued), avoiding protocol desync from partial frame
reads.
- Fix the SSH `#else` branch error message to also mention non-Linux
platforms, since the guard is `USE_SSH && defined(OS_LINUX)`.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Alexey Milovidov