Add an optional `command_pipe_capacity` setting for executable commands and
pass it through `ShellCommandSource` to `ShellCommand`, where it uses the
existing `PipeFDs::tryIncreaseSize` helper on the command pipes before `exec`.
The generated C-body drivers request a 1 MiB pipe capacity and the wrapper also
tries to increase its own `stdin` and `stdout` pipe capacity. This can reduce
short-write wakeups in the `DockerC` path, while preserving the previous default
for other executable command users.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `ninja -C build programs/clickhouse`
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Start one idle Docker container per `DockerC` function at create time and
generate `docker exec -i <container> /user_func` for the `executable_pool`
command. This avoids creating and removing a Docker container for every pool
worker while preserving the container-level network, filesystem, capability,
memory, CPU, and pids restrictions.
The `drop` command removes the persistent container, and the stateless test
checks the generated `docker exec` command and container-name metadata without
requiring Docker.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Add `--log-driver=none` to Docker invocations generated by `DockerC`. The UDF
protocol streams binary `Buffers` data over stdout, and Docker's default log
driver can otherwise duplicate that stream through container logging while it is
also attached back to ClickHouse.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `docker run --rm --log-driver=none alpine:3.20 sh -c 'printf hello'`
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_C_DRIVER_COMPILE_LOCAL=1 docker_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Use `runsc --rootless --network=none do` for the `GVisorC` runtime command
instead of Docker's `--runtime=runsc`. The Docker runtime flag requires daemon
configuration and fails with `unknown or invalid runtime name: runsc` when
`runsc` is installed but not registered in Docker.
Keep `CLICKHOUSE_C_DRIVER_GVISOR_RUNTIME` as a compatibility fallback, and add
`CLICKHOUSE_C_DRIVER_GVISOR_BINARY` for the direct `runsc` executable path.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_C_DRIVER_COMPILE_LOCAL=1 gvisor_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Rename the example `c_function_body` driver to `DockerC` and move the shared
C-body implementation into `c_driver_common.py`. Add `GVisorC` and `UnsafeC`
wrappers/configs that reuse the same generator.
`GVisorC` keeps the Docker command shape but adds `--runtime=runsc` for running
the compiled function. `UnsafeC` compiles locally and emits the compiled
`user_func` path directly without Docker or gVisor.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_C_DRIVER_COMPILE_LOCAL=1 docker_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `CLICKHOUSE_C_DRIVER_COMPILE_LOCAL=1 gvisor_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `unsafe_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Generated wrappers now read `Buffers` payloads directly into column buffers
and process numeric columns through typed `__restrict` views. The numeric
result path writes into the result buffer as a typed array and flushes it
directly, assuming little-endian host layout.
Move per-chunk processing into `ch_process_chunk` and mark it plus
`user_function` inline, so simple bodies such as `UInt64` addition can be
inlined and vectorized by the compiler.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_function_body_driver.py`
- `CLICKHOUSE_C_DRIVER_FORCE_LOCAL=1 c_function_body_driver.py create --return String --args 'x String, y String'`
- `CLICKHOUSE_C_DRIVER_FORCE_LOCAL=1 c_function_body_driver.py create --return UInt64 --args 'x UInt64, y UInt64'`
- direct `Buffers` protocol checks for `String` and `UInt64`
- `cc -O3 -march=native -fopt-info-vec-optimized -c tmp/pr105131_zero_copy_check/wrapper.c`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Rewrite the `c_function_body` example driver from https://github.com/ClickHouse/ClickHouse/pull/105131 to use the `Buffers` format instead of row-wise `RowBinary`.
The generated wrapper now reads one `Buffers` block per `send_chunk_header` chunk, validates the column count and row count, exposes arguments row-by-row to the user body, and writes the result as a single-column `Buffers` block. `String` columns are parsed once per chunk into `struct buf` views over the serialized column buffer, while returned `String` values are serialized into the result column buffer after calls to `alloc`.
Tests:
* `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_function_body_driver.py`
* direct `Buffers` protocol check in `build/test_c_function_body_driver_buffers_protocol.log`
* `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
* `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
* `git diff --check`
Extend the `c_function_body` example driver from https://github.com/ClickHouse/ClickHouse/pull/105131 to support the `String` type in `RowBinary`.
`String` arguments and return values are exposed to the C body as `struct buf { const char * data; size_t size; }`. The generated wrapper also provides `alloc`, a simple chunk-scoped bump allocator for returning strings. The allocator keeps buffers stable for the whole chunk and resets after the chunk is flushed.
Also compile generated functions with `-O3 -march=native`, while keeping the Docker build static.
Tests:
* `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_function_body_driver.py`
* direct `RowBinary` protocol check in `build/test_c_function_body_driver_string_protocol.log`
* `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
* `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
* `git diff --check`
Replace the generated wrapper's `stdio`-based `RowBinary` IO with a small
buffered `read` and `write` layer. Parse the text `send_chunk_header` row count
manually and serialize primitive values by copying fixed-width `RowBinary`
values directly to and from the in-memory buffers.
This keeps the example protocol-compatible while avoiding per-value C library
IO and formatting overhead in the generated executable.
Tests:
- generated-wrapper protocol check with one 3-row chunk
- generated-wrapper benchmark: 200000 rows, about 25M rows/sec in this run
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Refs: https://github.com/ClickHouse/ClickHouse/pull/105131
Update the source-tree `c_function_body` example driver to generate an
`executable_pool` function using `RowBinary` with `send_chunk_header` enabled.
The generated wrapper now reads the text chunk row count, consumes that many
`RowBinary` argument rows, writes the same number of `RowBinary` result rows,
and flushes once per chunk.
Remove the generated `setvbuf` call; chunk-level flushing is enough and matches
the executable UDF protocol better than per-row line buffering.
Tests:
- direct generated-wrapper protocol check with one 3-row chunk
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Refs: https://github.com/ClickHouse/ClickHouse/pull/105131
The source-tree `c_function_body` example driver used Docker with
`--security-opt=no-new-privileges`, but this option prevents containers from
executing on some developer hosts. Keep Docker for the example, but drop that
option from both the build container and the generated runtime command.
Run the Docker containers as the server OS user so they can read and write the
server-created per-function work directory without changing directory
permissions.
Tests:
`python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_function_body_driver.py`
`git diff --check`
`CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
`CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
`clickhouse-server` default-config smoke with Docker-backed `c_function_body`
Refs: https://github.com/ClickHouse/ClickHouse/pull/105131
Introduce "drivers" for executable user-defined functions. A driver is a
small adapter program declared in a server config under
`<user_defined_executable_function_drivers_config>` that knows how to take a
user-supplied code snippet and produce a runnable executable user-defined
function configuration.
New CREATE FUNCTION syntax:
CREATE [OR REPLACE] FUNCTION [IF NOT EXISTS] name [ON CLUSTER c]
ARGUMENTS (a UInt8, b String) RETURNS UInt64
ENGINE = DriverName(key1 = 'value1', key2 = 42)
AS '...code body...'
On CREATE FUNCTION, the corresponding `create_command` of the driver is
invoked: arguments are passed via `--name`, `--return`, `--args` and the
declared engine arguments; the body is sent on stdin; the driver writes the
resulting executable UDF configuration (XML or YAML) to stdout, which is
saved into the new `<dynamic_user_defined_executable_functions_path>` and
picked up by the existing executable UDF loader. The corresponding SQL is
persisted as `ATTACH FUNCTION ...` in `<user_defined>` and re-evaluated on
restart: if the dynamic configuration is still present, the driver is not
invoked; otherwise the driver is re-run to materialize it.
On DROP FUNCTION, the `drop_command` (if any) is invoked, then the dynamic
configuration file and per-function working directory are removed.
Includes a proof-of-concept driver `c_function_body`, which compiles a user
C body into a binary (inside Docker by default for build-time isolation) and
runs it inside Docker at query time with strict sandboxing (`--network=none`,
`--read-only`, dropped capabilities, no-new-privileges, unprivileged user,
resource limits) in `executable_pool` mode for batched per-row evaluation.
See https://github.com/ClickHouse/ClickHouse/issues/71172
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Alexey Milovidov