13,838 upstream commits, forward from the 26.6.1.1 fork point. 129 conflicts:
37 resolved mechanically (take upstream's file, re-apply the brand sweep) and
92 by hand where our fork and upstream had both changed the same code.
The rule throughout: upstream owns the engine, we own the naming and a small set
of deliberate deviations. Never resolve by discarding an upstream change — a
dropped rename or a missed new parameter becomes a compile error a thousand
files away.
Deviations verified intact after the merge:
ENABLE_XRAY still defaults OFF (CMakeLists.txt:105)
X-Datastore-User read by the server (authenticateUserByHTTP.cpp)
X-ClickHouse-* in src/ 0
hanzo/research present
system.*_log retention bound present
ArrowFlight rip held against upstream re-adding it
Two things worth naming, because neither shows up as a conflict:
src/CMakeLists.txt gained an UNGUARDED
`add_object_library(datastore_storages_arrowflight Storages/ArrowFlight)` from
upstream, for a directory our gRPC rip deleted. It sat outside every conflict
hunk, so a resolver reading only the markers would sail straight past it into
a CMake configure error. Removed; the Server/ArrowFlight sibling at line 474
stays because it is inside `if (TARGET ch_contrib::arrow_flight)`.
The brand sweep had rewritten github.com/ClickHouse/ClickHouse issue links to
github.com/Datastore/Datastore — a dead host. Those point at upstream's own
tracker and are restored, alongside the contributors list, sql.clickhouse.com
and the ClickHouse Inc copyright: attribution is theirs, not ours to rename.
base/glibc-compatibility/memcpy/memcpy.h is deleted, not kept: upstream dropped
it for LLVM-libc's, and our only change to it was a comment rebrand.
Keep the published images (`docker/server`, `docker/keeper`) as close to
upstream as possible: the in-region mirror override is enough to fix the
CI reachability failure, so the build-time `Acquire::Retries` is dropped
there. The CI-only images keep it.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Docker builds intermittently fail because Canonical's package mirrors
are unreachable from the CI runners: apt tries `ports.ubuntu.com` (arm64)
or `archive.ubuntu.com` (amd64) and gets `Could not connect ... connect
(101: Network is unreachable)` over IPv6 and `connection timed out` over
IPv4, aborting the build.
Point apt at the in-region AWS mirror (`<region>.ec2.archive.ubuntu.com`
/ `<region>.ec2.ports.ubuntu.com`), which is reachable and fast from the
runners:
- CI-only images (`test-base`, `fasttest`): default the mirror in the
Dockerfile. These are never pulled by end users. The existing mirror
swap only rewrote `archive.ubuntu.com`, so it silently no-oped on arm64
(which uses `ports.ubuntu.com`) and on Ubuntu 24.04 (which keeps its
sources in the deb822 `*.sources` file); the rewrite now covers both
hosts and both source layouts.
- Published images (`docker/server`, `docker/keeper`): keep the canonical
mirror as the Dockerfile default so images still build normally outside
CI, and inject the in-region mirror only in CI via a new
`--apt-mirror-region` argument to `docker_server.py`, passed from the
job definitions. The `sed` there now also rewrites `ports.ubuntu.com`
so the override reaches arm64.
Also add `Acquire::Retries` to ride out transient apt failures.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-07 incident: `system.text_log` grew to ~115 GB / 2.5B rows with no
`TTL` and filled the datastore data disk, taking down api.hanzo.ai.
Bake a `config.d` overlay (`datastore-log-retention.xml`) into the server
image giving every internal `system.*_log` table a 3-day `TTL` and capping
`text_log` to `warning` level. Root element is `<clickhouse>` to match the
shipped main `config.xml`; `opentelemetry_span_log` (the one table the main
config gives an explicit `<engine>`, keyed on `finish_date`) carries its
`TTL` inside `<engine>` because a separate `<ttl>` beside an `<engine>` is
rejected at startup (Code 36 BAD_ARGUMENTS).
Validated by booting ghcr.io/hanzoai/datastore:26.2.3.2 with the overlay:
server starts clean and all 12 tables report the `TTL`.
datastore.com is a dead domain (the debrand pass sed'd clickhouse.com into a
domain we do not own). Point every reference at the Hanzo brand, keeping paths:
datastore.com/cloud -> hanzo.ai/datastore (Datastore cloud product page)
datastore.com/docs/* -> docs.hanzo.ai/* (Hanzo docs site)
datastore.com/blog/* -> hanzo.ai/blog/*
*.datastore.com -> *.hanzo.ai (packages, presentations)
[email protected] -> [email protected]
datastore.com -> hanzo.ai (homepage, company, slack)
26 files, 80 references. config.xml still parses; no double-rewrites.
Co-authored-by: Hanzo Dev <[email protected]>
Runtime regression from the final lowercase pass: it renamed <clickhouse> opening tags to
<datastore> but the / exclusion (protecting /clickhouse paths) left </clickhouse> closing tags,
so embedded + shipped configs became <datastore>...</clickhouse> -> SAXParseException 'Tag
mismatch' at runtime (compile was fine; XML is data). Renamed all <clickhouse/</clickhouse XML
tags -> datastore (101 closes across configs, embedded strings, keeper configs). Path strings
(/var/lib/clickhouse) have no angle bracket and are untouched. Build compiled; this fixes runtime
config parsing.
The white-label name is just `datastore`, not `hanzo-datastore` — one name, DRY.
- binary / packages / paths / user / config: hanzo-datastore → datastore
(binary `datastore`, packages `datastore-{client,server,common-static,keeper}`,
`/var/lib/datastore`, `/etc/datastore-server`, `/var/log/datastore-server`, user `datastore`)
- env vars: HANZO_* → DATASTORE_* across the white-label scripts (entrypoint, init,
service, postinstall, docs) and compose — the docker image env is now consistently
`DATASTORE_*` and compose matches what the entrypoint reads.
- renamed 13 spec/config files: `packages/hanzo-datastore-*` → `packages/datastore-*`,
`docker/server/hanzo-paths.xml` → `docker/server/datastore-paths.xml`.
Left intentionally: GitHub Actions secrets `ROBOT_HANZO_{SSH_KEY,COMMIT_TOKEN}` (renaming
needs rotating org secrets). The C++ source still reads `CLICKHOUSE_*` env and expects a
`<clickhouse>` config root — that's the deeper clickhouse→datastore source debrand, tracked
separately. The Hanzo brand stays in human prose (maintainer, homepage `hanzo.ai`, image
org `hanzoai/datastore`).
A proper fork builds the image from our own source, not by wrapping the upstream
binary image. The root `Dockerfile` did `FROM clickhouse/clickhouse-server:26.5`
— a kludge that redistributed ClickHouse's prebuilt image — and it papered over a
deleted `docker/server/Dockerfile.ubuntu` (whose symlink was left dangling).
- restore `docker/server/Dockerfile.ubuntu`, white-labeled: `FROM ubuntu:22.04`,
installs our `hanzo-datastore-{client,server,common-static}` debs (built from
source by CI) via DIRECT_DOWNLOAD_URLS / deb_location_url / single_binary, with
our paths, config overlays and entrypoint, env `HANZO_CONFIG`. No upstream-repo
fallback — this ships our binary, never a redistributed clickhouse package.
- point the CI "Docker server image" job (`ci/jobs/docker_server.py`) at
`ghcr.io/hanzoai/datastore`. This is the one canonical image build; the
release/master Praktika workflows already run it.
- remove the bolted-on kludge: root `Dockerfile` and the wrapper
`.github/workflows/deploy.yml`.
- retire the stale "Docker keeper image" job: `docker/keeper/` was deleted
(coordination is embedded) but the job, its JobNames entry and 4 workflow
references were left dangling. Removed all of them.
The image builds purely on upstream `clickhouse/clickhouse-server:26.5`: the
binary is renamed to `hanzo-datastore` with `clickhouse-*` symlinks so the
upstream entrypoint keeps working, and `datastore <app>` as the product
surface. Nothing else is braided in.
A ZAP sidecar was grafted into the database fork — that is a separate concern
(its own image, composed at the pod level), not Go/C++ source jammed into a
ClickHouse fork with a hacked entrypoint. Removed:
- `cmd/zap-bridge/` Go duplex listener
- `src/Server/ZAP/` dead C++ handler (no CMakeLists ever linked it)
- `docker/server/entrypoint.sh` zap-bridge startup block
- `docker/server/users.d/insights-writer.xml` the bridge's DB account
- ZAP references in `deploy.yml`, `LLM.md`, `hanzo/compose.yml`
`hanzo/compose.yml` now pulls `ghcr.io/hanzoai/datastore` and drops the
non-existent `datastore-keeper` service (keeper runs embedded in-process).
Real fork of ClickHouse/ClickHouse with the Hanzo overlay applied on top of the
26.5 release lineage (parent is upstream/26.5, so future upstream syncs are clean
3-way merges). Overlay: cmd/zap-bridge (:9999 ZAP duplex), Dockerfile.hanzo +
hanzoai/debian13 bases, hanzo/ config + schema, white-label hanzo-datastore
branding, arcd CI, hanzo-* packaging.
FOLLOW-UP: re-apply Phase-1 CMake decisions (single-binary ENABLE_DATASTORE_KEEPER=OFF;
disabled external engines) + full build on hanzo-build-linux-amd64.
ClickHouse statically links its own openssl, libstdc++, zlib, zstd, gomp.
The binary's DT_NEEDED is only 5 glibc libs (librt, libpthread, libc,
libm, libdl), so the 7 extra packages the cc-debian13 base ships are
dead weight that each turn into a recurring CVE chase (libssl3t64 was
the latest, fixed in #107677).
Swap server + keeper Dockerfile.distroless to base-nossl-debian13:nonroot
which keeps glibc and drops gcc-14-base, libgcc-s1, libgomp1, libssl3t64,
libstdc++6, libzstd1, zlib1g. Base package count goes 13 -> 6.
Smoke-tested on amd64 with VERSION=26.5.2.39:
- clickhouse --version / server --version / keeper --version all clean
- server starts, MergeTree+INSERT 10k rows, ZSTD codec round-trip OK
- outbound HTTPS via url() to api.github.com and S3 works (TLS handshake
succeeds — confirms statically-linked openssl is self-sufficient)
- keeper 4LW (ruok->imok, mntr, srvr) responds, no errors in logs
- Trivy --severity HIGH,CRITICAL --ignore-unfixed: 0 on both prod images
Follow-up to #107677.
The previous pin (2026-06-03, sha256:e1fd250c...) shipped
`libssl3t64 3.5.6-1~deb13u1`. Trivy flags this as a fixable HIGH
(CVE-2026-45447) on the keeper image — the server happens to scan
clean against the same base, but the underlying package is identical,
so bump both for consistency.
New digest (2026-06-16, sha256:d3cda6e9...) ships
`libssl3t64 3.5.6-1~deb13u2` which clears the finding. Confirmed via
`trivy image --severity HIGH,CRITICAL --ignore-unfixed` against the
new base alone — zero fixable findings.
Same bump applied to the debug-nonroot pin to keep production and
debug aligned.
After this lands, run `BackfillDistroless` against the 6 currently-
tracked versions to publish the fresh keeper images.
Refreshes the `gcr.io/distroless/cc-debian13:{nonroot,debug-nonroot}`
pins in `docker/server/Dockerfile.distroless` and
`docker/keeper/Dockerfile.distroless`. The current pins (dated
2026-04-26) ship `libssl3t64 3.5.5-1~deb13u1` which has 1 CRITICAL and
4 HIGH fixable CVEs (CVE-2026-31789, CVE-2026-28387, CVE-2026-28388,
CVE-2026-28389, CVE-2026-28390 — all fixed in 3.5.5-1~deb13u2).
Detected by the distroless CVE scanner cron (2026-06-03 report) — the
26.4 server+keeper images classified as `source-digest-stale`, meaning
the live base image already has the patched libssl3t64 but the
Dockerfile is still pinning the older debian13 layer.
New digests verified with:
docker pull gcr.io/distroless/cc-debian13:{nonroot,debug-nonroot}
docker inspect --format='{{index .RepoDigests 0}}' <image>
The `Docker server image` job intermittently fails when fetching the
prebuilt `.deb` packages from `clickhouse-builds.s3.amazonaws.com`:
```
ERROR 500: Internal Server Error.
ERROR 503: Service Unavailable.
```
Three unrelated PRs hit this S3 download flake over three days
(`#100173` on 2026-05-15 with `500`, `#104694` on 2026-05-14 with `503`,
`#104853` on 2026-05-13 with `503`). On master in the last 14 days the
master `Docker server image` job has 1 failure of the same shape.
`docker/server/Dockerfile.alpine` already wraps `wget` with a retry
helper (added in #100380 to absorb the same kind of transient errors
during Alpine cross-architecture builds via QEMU). The Ubuntu and
distroless flavours still call `wget ... || exit 1` on the very first
attempt, so a single `500`/`503` from S3 kills the whole build.
This change ports the same `wget_with_retry` wrapper to
`docker/server/Dockerfile.ubuntu` and `docker/server/Dockerfile.distroless`
for the `DIRECT_DOWNLOAD_URLS` block (the path used by CI). `WGET_RETRIES`
(default `5`) and `WGET_RETRY_DELAY` (default `1s`) match the Alpine
version and remain overridable via `--build-arg`.
CI report (PR #100173, sha 3ceac71): https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=100173&sha=3ceac71a43a5fd4fc1a7859937b23e71b8fe42ae&name_0=PR&name_1=Docker%20server%20image
Co-Authored-By: Claude Opus 4.7 <[email protected]>
This reverts PR #90043 (merge commit e21a84e759) due to performance
regressions observed in the MSan build.
Note: a post-merge follow-up commit (9df07df5a0, "Pass nm/ar/objcopy
paths from CMake to localize_rust_c_symbols.sh") added an `NM_PATH`
lookup in `cmake/tools.cmake` for the now-removed
`cmake/localize_rust_c_symbols.sh` helper. The lookup is left in place
by this revert; it is unused but harmless and can be cleaned up
separately.
Mend.io scan found 1 CRITICAL + 4 HIGH CVEs in libssl3t64 3.5.5-1~deb13u1
(CVE-2026-31789, CVE-2026-28387..28390). All fixed in 3.5.5-1~deb13u2.
Pin digests from 2026-04-26.
zeekay
