The client-side AST fuzzer loop swallowed every exception thrown during a
fuzzing step: the `catch (...)` handler recorded the error, the loop bottom
reset it, and fuzzing continued. When the failure mode is persistent - most
importantly a dead server, where the reconnection attempt inside
`processParsedSingleQuery` throws instantly - the client sprints through the
remaining corpus at hundreds of steps per second without contacting the
server and exits with code 0, which the job reports as "Fuzzer exited with
success".
Evidence from `AST fuzzer (amd_debug, targeted)` on PR #109061
(https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=109061&sha=1419f6b7e95c81c812f945c06333998b92d56ad3&name_0=PR&name_1=AST+fuzzer+%28amd_debug%2C+targeted%29):
`fuzzer.log` records 68972 fuzzing steps, of which 52837 produced no outcome
(silent exception), while `server.log` shows the server received only 209
queries within a 47-second window before dying. On PR #106250
(https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=106250&sha=latest&name_0=PR&name_1=AST+fuzzer+%28amd_debug%2C+targeted%29)
a 5-file corpus with ~20 SELECT statements (about 20000 fuzzing steps, over
an hour of work at the measured debug-build rate of 2-6 steps/s) "completed"
in 74 seconds and the job passed. Overall, 13-33% of targeted AST fuzzer
runs per month finish in under 8 minutes of their 30-minute budget.
Client fixes (`programs/client/FuzzLoop.cpp`):
- Print thrown exceptions in the fuzzing-step catch handlers instead of
swallowing them silently, so `fuzzer.log` shows what actually happened.
- After a thrown exception, check the connection via `tryToReconnect` (the
same check `processASTFuzzerStep` does for errors received without
throwing) and stop fuzzing when the server is gone instead of uselessly
sprinting through the rest of the corpus.
- Register query parameters generated by `QueryFuzzer` (from
`getLastQueryParameters`) for client-side substitution, mirroring the
server-side fuzzer in `executeASTFuzzerQueries`; previously any fuzzed
query containing a substituted `{name:type}` parameter failed with
"Substitution ... is not set" before reaching the server.
Job fix (`ci/jobs/scripts/fuzzer/run-fuzzer.sh`):
- When the AST fuzzer walks the whole corpus and exits 0 with a meaningful
part of the FUZZ_TIME_LIMIT budget remaining, restart it (each pass uses a
fresh random seed) instead of ending the job after a couple of minutes.
This matters for the targeted variant, whose corpus is a handful of test
files. The restart is gated on a server liveness probe so a dead server
still terminates the job for classification.
The AST fuzzer was reporting false failures when the server hit its
memory limit. `MEMORY_LIMIT_EXCEEDED` means the server correctly
rejected an expensive fuzzed query, not that it died.
Two fixes:
- In `FuzzLoop.cpp`, treat `MEMORY_LIMIT_EXCEEDED` like
`TOO_DEEP_RECURSION`: clear the error and continue fuzzing instead of
attempting reconnection, failing, and exiting with code 241.
- In `run-fuzzer.sh`, treat `MEMORY_LIMIT_EXCEEDED` during the
post-fuzzing liveness check as a transient condition (like
`TOO_MANY_SIMULTANEOUS_QUERIES`) instead of declaring the server dead.
https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=102000&sha=latest&name_0=PR&name_1=AST+fuzzer+%28arm_asan_ubsan%29
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Resolve merge conflicts:
- `base/base/coverage.cpp`: take master's version which removed the old
sanitizer coverage guard/trace functions (replaced by LLVM profile-based
coverage)
- `src/Client/BuzzHouse/AST/SQLProtoStr.cpp`: take master's unified
`CONV_FN(SQLIdentifier, ident)` replacing the individual per-type
converter functions
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Fix `io_destroy` prototype mismatch in `AIO.h` for FreeBSD (declared as
`int io_destroy(void)` but defined as `int io_destroy(int ctx)`)
- Add forward declarations for `LLVMFuzzerInitialize` and
`LLVMFuzzerTestOneInput` in fuzzer files, and make file-local helper
functions `static` to fix `-Wmissing-prototypes` in arm_fuzzers build
- Add `extern "C"` forward declaration for `reallocarray` in `malloc.cpp`
to fix `-Wmissing-prototypes` in ppc64le build
- Suppress `-Wdisabled-macro-expansion` in `FuzzLoop.cpp` because musl
defines `stderr` as `(stderr)` which is a recursive macro
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Remove `CurrentThread.h` from high-fan-out headers: `ProcessList.h`,
`IQueryPlanStep.h`, `DDLWorker.h`, `TCPHandler.h`, `AggregatingTransform.h`,
`ParallelParsingInputFormat.h`, `StorageTableFunction.h`, `parseSyslogLevel.h`,
`EmbeddedRocksDBBulkSink.h`, `BlockIO.h`, `LocalConnection.h`,
`StorageMaterializedView.h`, `HTTPHandler.h`
- Extract `QueryScope` from `CurrentThread` into standalone `Common/QueryScope.h`
with its own `.cpp`, replacing the nested `CurrentThread::QueryScope` class
- Extract `getCurrentThreadGroup` free function into `Common/ThreadGroupSwitcher.h`,
replacing `CurrentThread::getGroup()` in template headers:
`threadPoolCallbackRunner.h`, `ParallelFormattingOutputFormat.h`,
`HashedDictionaryParallelLoader.h`, `UniqExactSet.h`, `HashedDictionary.h`,
`StatusRequestsPool.h`
- De-inline `MemoryTrackerSwitcher` ctor/dtor into `MemoryTrackerSwitcher.cpp`
- De-inline `ResourceGuard::Request::wait` into `ResourceGuard.cpp`
- De-inline `ManyAggregatedData` destructor into `AggregatingTransform.cpp`
- De-inline `QueryStatus::updateProgressIn/Out` into `ProcessList.cpp`
- Add explicit `CurrentThread.h` includes to `.cpp` files that relied on
transitive includes through the trimmed headers
Co-Authored-By: Claude Opus 4.6 <[email protected]>
