mirror of
https://github.com/hanzoai/datastore.git
synced 2026-09-26 13:55:07 +00:00
Alexei Fedotov
8fe7524811
commit 8fa9d49b6b3d0a79b649d5e6be7b7f46560cbbb6 Merge: 98d2741b6c2 80eea0c32ea Author: Alexei Fedotov <[email protected]> Date: Tue Jul 14 03:49:26 2026 +0000 Merge remote-tracking branch 'upstream/master' into ci/jobs-use-clickhouse-service # Conflicts: # ci/jobs/scripts/clickhouse_proc.py # ci/jobs/scripts/clickhouse_service.py commit 98d2741b6c2321cef5edf86b644d3a9c2b95aa81 Author: Alexei Fedotov <[email protected]> Date: Tue Jul 14 02:56:10 2026 +0000 CI: raise max_query_size for remote *_log CREATE in log replication setup setup_log_cluster.sh ships each system.*_log schema to the remote CI logs host. The metric_log CREATE expands past the remote server's default max_query_size (256 KiB) and is rejected with `Code: 62. Max query size exceeded`, so the `|| continue` skips its _sender/_watcher and metric_log never replicates. The remote parses the statement before the IF NOT EXISTS existence check, so this fires regardless of whether the destination table already exists. Pass --max_query_size=4194304 on that CREATE. Job: https://github.com/ClickHouse/ClickHouse/actions/runs/29244362167/job/86822949088 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> commit 24e2f1c2b2d75ab966dbc86a55f9f7399bb5d7d2 Author: Alexei Fedotov <[email protected]> Date: Mon Jul 13 09:31:58 2026 +0000 CI: make ClickBench log export actually work ClickBench export of `system.*_log` tables to the CI logs cluster was silently broken: `setup_log_cluster.sh --setup-logs-replication` failed but the failures were swallowed (per-table `|| continue`, and the job only prints a WARNING), so the job stayed green while replicating nothing. Two independent defects, one per prerequisite that must live in the config dir the server actually reads: - The `system_logs_export` cluster config was written to `ClickHouseProc`'s default `ch_config_dir` (`/etc/clickhouse-server`) while the ClickBench server runs from `ci/tmp/etc/clickhouse-server`, so the server started without the cluster and every `*_sender` failed with `Code: 701. Requested cluster 'system_logs_export' not found`. `create_log_export_config` now takes an explicit `config_dir` and ClickBench passes the `ClickHouseService` config dir, so the cluster config lands where the server reads it. - The `*_watcher` materialized views are created with `DEFINER = ci_logs_sender`, but that user was never installed, so replication setup failed with `Code: 192. There is no user 'ci_logs_sender'`. Install the functional-test `ci_logs_sender.yaml` as a config hook before start, mirroring `sqlstorm_test.py`. Also track `tests/config/users.d/ci_logs_sender.yaml` in the ClickBench digest as a runtime dependency. Master (Code 701): https://github.com/ClickHouse/ClickHouse/actions/runs/29229353556/job/86765371667 PR (Code 192): https://github.com/ClickHouse/ClickHouse/actions/runs/29097405351/job/86401789490 commit 07f0eb0d49c1840189770137c895bce7cfffbfef Author: Alexei Fedotov <[email protected]> Date: Fri Jul 10 13:02:30 2026 +0000 CI: give config hooks install_ verb names and pass them the dirs Rename the config hooks to install_ verbs and drop the redundant _config suffix, since a config hook is already a config hook: install_base, install_introspection, install_clickbench_cache, install_vector_search. Pass ch_config_dir and ch_var_lib_dir to every hook instead of hardcoding f"{temp_dir}/etc/clickhouse-server", so a hook honors a custom config dir and can build config paths into the data dir (as ClickHouseProc.install_configs does with storage_conf_backups.xml and filesystem_caches_path.xml). create_log_export_config stays a no-arg ClickHouseProc method (it is also called directly elsewhere) and is adapted to the hook contract with a wrapper where ClickBench registers it. commit 0058e6c4d9b30930422f6ffa5cecf2f2b765185a Author: Alexei Fedotov <[email protected]> Date: Fri Jul 10 12:53:43 2026 +0000 CI: rename install_clickbench_config hook to introspection The hook only installs the allow_introspection_functions user profile and is shared by the ClickBench and vector search jobs, so its clickbench-specific name was misleading. Rename it to introspection to reflect what it does; the ClickBench-only cache tweaks stay in install_clickbench_cache_config. commit a938e6a3bdfb1f0c649c8424b92c0699d18ef633 Author: Alexei Fedotov <[email protected]> Date: Fri Jul 10 12:42:57 2026 +0000 CI: make base config install an explicit ClickHouseService hook The base config setup was nested under if not (config_dir / "config.xml").exists() in ClickHouseService.__enter__, so config_hooks, extra_config_files, and config_ignore_patterns stopped applying after the first start in a workspace. Because the config dir is shared across jobs, a later run silently reused stale config. Turn the base config setup into a standard hook that ships with the class, ClickHouseService.install_base_config, and run all config_hooks on every __enter__ with no existence guard. Callers pass install_base_config explicitly as the first hook, followed by their per-job tune-ups, so config is rebuilt from scratch on every start and each job stays isolated. Drop the config_ignore_patterns and extra_config_files parameters: the only user was ClickBench, whose storage_conf_local.xml skip and filesystem_caches_path.xml override now live in a dedicated install_clickbench_cache_config hook rather than leaking into the shared install_clickbench_config used by the vector search jobs. commit ee9e0d863cf1c0b2ca0207af857cb8801a046d23 Author: Alexei Fedotov <[email protected]> Date: Fri Jul 10 11:59:56 2026 +0000 Revert "CI: rebuild ClickHouse config dir on every ClickHouseService start" This reverts commit ccd3da9d7addeedc53df6dcc95ca49def3e80906. commit ccd3da9d7addeedc53df6dcc95ca49def3e80906 Author: Alexei Fedotov <[email protected]> Date: Fri Jul 10 11:40:44 2026 +0000 CI: rebuild ClickHouse config dir on every ClickHouseService start The per-job customizations (config_hooks, extra_config_files, config_ignore_patterns) were nested under if not (config_dir / "config.xml").exists(), so they stopped applying after the first ClickHouseService start in a workspace. Since the config dir is shared across jobs, a later run silently reused stale config: a clickbench.py run could leave config that prevents a subsequent vector_search_stress_tests.py run from installing vector_search.xml, and the reverse order could skip ClickBench cache-path customizations. Wipe and recreate the config dir on every __enter__ so ignore patterns, extra files, and hooks (including files a hook drops into users.d) always apply and never inherit stale config from a prior run. commit 6b6a42760633d9bc4a938c34b0ad726f76c1e74b Merge: a678f7ca292 bf544939cc4 Author: Alexei Fedotov <[email protected]> Date: Fri Jul 10 02:45:24 2026 +0000 Merge remote-tracking branch 'origin/master' into ci/jobs-use-clickhouse-service # Conflicts: # .github/workflows/pull_request.yml commit a678f7ca29293b5f610d7fc2480551871be72439 Author: Alexei Fedotov <[email protected]> Date: Fri Jul 10 02:16:24 2026 +0000 CI: address review comments on ClickHouseService - Remove `Result.raise_if_failed`; callers now raise explicitly. - Remove the light-install path deprecated by `ClickHouseService`: `_install_light`, `start_light`, `install_clickbench_config`, `install_fuzzer_config`, `install_vector_search_config` in `ClickHouseProc`. - Add `config_hooks` to `ClickHouseService` so jobs pass per-job config tune-ups (e.g. `install_clickbench_config`) as callbacks run under the hood before the server starts. commit 205fdec8e8ac3ec4c1e973396ef0a3b84b17854e Author: Alexei Fedotov <[email protected]> Date: Thu Jul 9 12:45:25 2026 +0200 CI: track clickhouse_proc.py in ClickBench digest clickbench.py imports ClickHouseProc and calls create_log_export_config, start_log_exports, and prepare_logs, but the clickbench_jobs digest did not list ./ci/jobs/scripts/clickhouse_proc.py in include_paths. Because the job is change-filtered and cache-keyed from include_paths, a PR touching only clickhouse_proc.py would skip ClickBench in PR CI and could reuse a stale cached success despite changed runtime behavior. Add the path to the digest. commit 9f3951950046b6809588eeaca01635357dd1a7c3 Author: Alexei Fedotov <[email protected]> Date: Thu Jul 9 06:27:06 2026 +0000 CI: replace `_install_light`/`start_light` with `ClickHouseService` in CI jobs Introduce the `ClickHouseService` context manager that installs and starts a local ClickHouse server for CI jobs, replacing the ad-hoc `_install_light`/`start_light` helpers for the ClickBench and vector-search stress jobs. Jobs that still rely on the old helpers (functional tests, fast test, `sqlstorm`, etc.) keep using `clickhouse_proc`. Also: - Run ClickBench on pull requests, gated by its digest so the cost stays bounded (see `clickbench_jobs.digest_config`). - Restore the ClickBench filesystem-cache path override so the `/dev/shm/clickhouse/` cached disk is accepted by the server (otherwise it fails with `Filesystem cache absolute path must lie inside /tmp/filesystem_caches/`). - Record a `Job error` result on exceptions instead of swallowing them, and use `raise_if_failed` instead of a boolean accumulator. - Drop the unused `ci/jobs/fuzzers_job.py`. Rebased onto latest master and regenerated `pull_request.yml` via `praktika yaml`.
60 lines
1.7 KiB
Python
60 lines
1.7 KiB
Python
import argparse
|
|
import shlex
|
|
|
|
from ci.jobs.integration_test_job import start_docker_in_docker
|
|
from ci.jobs.scripts.clickhouse_service import ClickHouseService
|
|
from ci.praktika.result import Result
|
|
from ci.praktika.utils import Utils
|
|
|
|
temp_path = f"{Utils.cwd()}/ci/tmp"
|
|
|
|
|
|
def parse_args():
|
|
parser = argparse.ArgumentParser(description="CI Tests job")
|
|
parser.add_argument(
|
|
"--test",
|
|
help="Optional. Test name patterns passed to pytest -k (space-separated)",
|
|
default=[],
|
|
nargs="+",
|
|
action="extend",
|
|
)
|
|
parser.add_argument(
|
|
"--skip",
|
|
help="Optional. Test name patterns to exclude, passed to pytest -k as 'not' (space-separated)",
|
|
default=[],
|
|
nargs="+",
|
|
action="extend",
|
|
)
|
|
return parser.parse_args()
|
|
|
|
|
|
if __name__ == "__main__":
|
|
args = parse_args()
|
|
pytest_command = "ci/tests/"
|
|
k_expr = ""
|
|
if args.test:
|
|
k_expr = " or ".join(args.test)
|
|
if args.skip:
|
|
k_expr = f"({k_expr})"
|
|
if args.skip:
|
|
k_expr += (" and " if k_expr else "") + " and ".join(
|
|
f"not {p}" for p in args.skip
|
|
)
|
|
if k_expr:
|
|
pytest_command += " -k " + shlex.quote(k_expr)
|
|
|
|
start_docker_in_docker()
|
|
with ClickHouseService(
|
|
config_hooks=[ClickHouseService.install_base]
|
|
) as service:
|
|
test_result = Result.from_pytest_run(
|
|
name="CI Tests",
|
|
command=pytest_command,
|
|
pytest_report_file=f"{temp_path}/pytest_ci_tests.jsonl",
|
|
pytest_logfile=f"{temp_path}/pytest_ci_tests.log",
|
|
logfile=f"{temp_path}/ci_tests.log",
|
|
timeout=600,
|
|
)
|
|
|
|
test_result.complete_job()
|