Follow-up to #105171, which added `ruff` with a deliberately narrow,
bug-class-only rule set and left the unused-name and star-import rules
off pending a dedicated cleanup pass. This enables that second batch in
`pyproject.toml` and fixes every violation across the tree (the bulk in
integration tests):
- `F401` unused-import and `F841` unused-variable: removed, preserving
any side-effecting right-hand side (e.g. `result = node.query(...)`
becomes `node.query(...)`). Two `try: import ...` feature probes are
kept with `# noqa: F401`.
- `F541` f-string-missing-placeholders: dropped the redundant `f` prefix.
- `F403`/`F405` star imports: every `from module import *` is expanded
into explicit imports of the names actually used, taken from each
name's real origin, so stdlib/third-party names import from their real
source rather than through a test "convenience bundle". The now-empty
`tests/integration/helpers/kafka/common_direct.py` bundle is removed and
its three remaining consumers import `_VarintBytes`/`KafkaAdminClient`
directly.
Eliminating the star imports surfaced a real bug the wildcard had masked
from `F821`: `generate_presets.py` in `test_prometheus_protocols` called
an undefined `execute_instant_query_with_http_api`; fixed to the existing
`execute_query_via_http_api`.
No `black`/`isort` reformatting was applied (neither is enforced in the
style check), so the diff is limited to the lint fixes themselves.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The `--profiler` CLI option still said "Path to parser_memory_profiler
binary", but after consolidation the script now executes the
`clickhouse-examples` multi-call binary. Update the help text to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The `parser_memory_profiler` is now a subcommand of `clickhouse-examples`.
Update the `run_profiler.sh` help text for `--profiler` and the `README.md`
description of `run_profiler.sh` to reflect the multi-call invocation.
Update build instructions and troubleshooting commands to reference
`clickhouse-examples` instead of the old standalone `parser_memory_profiler`
target, matching the new multi-call binary layout.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The `run_profiler.sh` and `rebuild_reports.py` scripts still referenced the old
standalone `parser_memory_profiler` binary path at
`build/src/Parsers/examples/parser_memory_profiler`. Update them to use the new
`clickhouse-examples` multi-call binary at `build/src/Examples/clickhouse-examples`.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Simplify jemalloc integration by replacing the three-mechanism approach
(`JEMALLOC_IS_MALLOC` + `JEMALLOC_OVERRIDE___LIBC_*` + linker `--wrap`)
with a cleaner two-step design:
1. Compile jemalloc with `je_` prefix so it only exports `je_malloc`,
`je_free`, etc. (matching what Darwin already does).
2. Define our own `malloc`/`free` in `malloc.cpp` that perform memory
tracking and delegate to `je_malloc`/`je_free`. Provide `__libc_*`
aliases on glibc to intercept internal calls.
Changes:
- Enable `JEMALLOC_PREFIX "je_"` in all non-Darwin `jemalloc_internal_defs.h.in`
- Undef `JEMALLOC_IS_MALLOC` and all `JEMALLOC_OVERRIDE___LIBC_*` defines
- Remove `JEMALLOC_OVERRIDE_MEMALIGN`/`VALLOC`/`PVALLOC` from `jemalloc_defs.h`
- Add `JEMALLOC_NO_RENAME` interface define so consumer code sees `je_*` names
- Prefix all jemalloc API calls (`mallctl` -> `je_mallctl`, etc.) in ClickHouse
- Rewrite `memory.h` to call `je_malloc`/`je_free` directly instead of `__real_*`
- Rewrite `malloc.cpp` with tracking wrappers calling `je_*` underneath
- Rewrite `AllocationInterceptors.h` to map `__real_*` -> `je_*` or `::*`
- Remove `__wrap_*` functions from `AllocationInterceptors.cpp`
- Remove `--wrap` linker flags from `CMakeLists.txt`
- Add jemalloc + clickhouse_common_io deps to `clickhouse_malloc` OBJECT library
- And also fix JEMALLOC_CPREFIX for Darwin, that was the only platform that uses JE_MALLOC_CONF (over MALLOC_CONF)
Co-Authored-By: Claude <[email protected]>
Add a set of scripts for profiling memory allocations in the ClickHouse SQL parser
using jemalloc's heap profiling capabilities.
The C++ profiler binary source is at src/Parsers/examples/parser_memory_profiler.cpp
Scripts:
- run_profiler.sh: Batch processing script for multiple queries
- generate_report.py: Generate interactive HTML report for single result set
- generate_comparison_report.py: Compare two result sets (before/after optimization)
- rebuild_reports.py: Regenerate reports from existing heap profile files
- test_queries.txt: Sample set of 100 diverse SQL queries for testing
Features:
- Detailed memory allocation analysis using jeprof
- Interactive flame graphs for visualization
- Call graph SVG generation
- Side-by-side comparison of optimization results
- Self-contained HTML reports
Prerequisites:
- jemalloc with profiling support
- jeprof (from jemalloc)
- FlameGraph (for flame graphs)
- graphviz (for call graph SVGs)
Usage:
./run_profiler.sh -b ../../build
python3 generate_report.py -i profiler_output/results.json -o report.html
