datastore/LLM.md

Hanzo Datastore

Hanzo's analytics database — a fork of ClickHouse with Hanzo-side overlays.

• Repo: https://github.com/hanzoai/datastore
• Upstream: https://github.com/ClickHouse/ClickHouse (upstream remote)
• Image: ghcr.io/hanzoai/datastore

Architecture — S-Chain & the two-engine platform

Datastore is the OLAP half of Hanzo's data platform, and it separates storage from compute. Its MergeTree parts are stored as objects on Hanzo S3 / "S-Chain" — the SeaweedFS-fork storage substrate (hanzoai/s3) — so stateless compute replicas share one zero-copy copy of the data. This is a proven PoC: a 5M-row table whose storage policy targets S-Chain — system.parts reports disk seaweed — and two compute nodes scale out over a single physical copy via zero-copy replication. The gRPC and Arrow Flight RPC surfaces are removed; the live query path is the native TCP protocol (port 9000). ZAP (the Cap'n-Proto-derived RPC) is the intended replacement transport — the server stub has landed, but it is not yet the query path. The planned OLTP half is a decentralized SQL engine (SQLite/Base) — designed, not yet built. The direction for both halves: ride S-Chain storage and move coordination from Raft to Quasar (post-quantum, leaderless).

Design paper: hanzo-datastore/ in hanzoai/papers.

Benchmark vs official ClickHouse (measured, honest)

ClickBench (43 queries, 100M-row hits dataset), spark (aarch64), 4 runs/query, warm. opt = our PGO+ThinLTO+BOLT datastore.bolt 26.6.1.1; stock = the official clickhouse binary.

Metric	Official ClickHouse	Hanzo Datastore (opt)	Delta
Peak RSS (under load)	14.52 GB	13.96 GB	−3.8%
Total query time (Σ per-query medians)	16.83 s	16.82 s	tied (+0.1%)
Per-query	—	faster on 8/43; median 0.95×	~neutral
Binary, stripped	—	398 MB	—

Honest verdict: under production-scale load the optimization buys ~4% less peak memory and is statistically tied on speed. The scale matters and must be quoted: an earlier "~half the memory / much faster" figure was real but measured at 10M rows, where the removed XRay instrumentation + leaner binary are a large fraction of a small working set. At 100M rows the data dominates RSS, so the same fixed savings shrink to ~4% and the speed win vanishes. Both are true at their scale — never cite "half the memory" without "at 10M rows." Caveats on the 100M run: only 4 runs/query (per-query noise ±25%, so "tied" is within noise), one workload, one machine, warm cache. The fork's real value is architectural (S-Chain storage separation, consensus2 coordination, gRPC removed), not raw OLAP speed over stock.

Stack

• Server: C++ (the ClickHouse codebase under programs/, src/, base/)
• Build: CMake → datastore binary

Ports

Port	Protocol	Purpose
8123	HTTP	ClickHouse HTTP interface
9000	TCP	ClickHouse native binary protocol
9009	TCP	ClickHouse interserver replication
9181	TCP	embedded keeper client port (when <keeper_server> is configured)

Layout

datastore/
├── programs/, src/, base/, contrib/  ← upstream ClickHouse C++
├── hanzo/                             ← Hanzo overlay: compose, config.xml, schema.sql
├── packages/, pkg/                    ← RPM/deb packaging (datastore-*.yaml)
├── docker/server/                     ← Hanzo image build (Dockerfile.ubuntu, white-labeled)
└── ci/                                ← upstream + Hanzo CI (Praktika)

Build

# Local dev (compose):
cd hanzo && docker compose up

# CI: .hanzo/workflows/cicd.yml calls hanzoai/ci, which reads the root hanzo.yml
# and builds the repo-root Dockerfile from source, publishing
# ghcr.io/hanzoai/datastore (linux/amd64). See hanzo.yml for why it is the root
# Dockerfile and not docker/server/, and why there is no deploy block.

Upstream Sync

Automated via .hanzo/workflows/upstream-sync.yml (weekly, Mon 06:00 UTC, plus workflow_dispatch). The workflow fetches upstream/master, merges it onto an upstream-sync/<upstream-commit> branch, and opens a WIP: pull request on git.hanzo.ai — it never auto-merges. On conflict the merge commit is pushed with the markers intact and the pull request lists the conflicting files. Naming the branch after the upstream commit makes a re-run on an unchanged upstream head a no-op rather than a duplicate proposal.

There is exactly one upstream sync workflow. Do not add a second.

Net-new product files live under hanzo/ (disjoint from upstream, so most syncs land clean). White-labeling that edits upstream files in place — docker/server/, packages/, programs/server/config.xml, top-level CMakeLists.txt — are the watch areas for merge conflicts.

Phase 1 — Single Binary (Embedded Coordination)

datastore ships exactly one server binary. The standalone clickhouse-keeper/datastore-keeper entry-point is no longer a default build target. Coordination still works — it runs in-process inside datastore-server whenever <keeper_server> is present in the runtime config.

What changed

• programs/CMakeLists.txt — the option defaults flipped to OFF:
  • ENABLE_DATASTORE_KEEPER (was ${ENABLE_DATASTORE_ALL}) — drops the standalone keeper entry-point and the datastore-keeper symlink from the multipurpose binary.
  • ENABLE_DATASTORE_KEEPER_CONVERTER (was ${ENABLE_DATASTORE_ALL}) — drops the ZooKeeper→Keeper snapshot converter tool.
  • ENABLE_DATASTORE_KEEPER_CLIENT (was ${ENABLE_DATASTORE_ALL}) — drops the standalone keeper CLI client.
• docker/server/Dockerfile.ubuntu — PACKAGES installs only datastore-{client,server,common-static}; the keeper package is not installed. The standalone "Docker keeper image" CI job (which built docker/keeper/) was retired — coordination is embedded.
• BUILD_STANDALONE_KEEPER already defaulted OFF upstream; left untouched.

What did NOT change (and why)

• src/Coordination/ is kept and is still compiled into the server library unconditionally (src/CMakeLists.txt:61 add_subdirectory(Coordination), :360 add_object_library(datastore_coordination Coordination)). This is the embedded-Raft logic.
• Server.cpp:2468-2620 (the if (config().has("keeper_server.server_id")) block, gated #if USE_NURAFT) is the embedded-coordination boot path. It uses getKeeperDispatcher and KeeperTCPHandlerFactory from src/Coordination/ and binds keeper_server.tcp_port (default 9181), keeper_server.tcp_port_secure, and keeper_server.http_control.port inside the server process. No separate binary required.
• programs/keeper/keeper_embedded.xml is kept — it's the canonical config shape for the embedded path. keeper_config.xml (the standalone-binary config) stays in the source tree but is no longer installed by the default build.
• programs/keeper/, contrib/NuRaft/, contrib/nuraft-cmake/ — kept. Removing them is Phase C and gated on the lux/consensus → Quasar replacement landing (see Phase B doc above).

Packaging

packages/datastore-keeper.yaml only ships content when BUILD_STANDALONE_KEEPER=ON (which we don't set). packages/datastore-server.yaml is unchanged and now provides the only binary. The datastore-keeper deb/rpm package can be deprecated after one release cycle — leave the spec in tree until then for upstream-sync stability.

programs/install/Install.cpp:440-441

Install.cpp (the datastore install system-installer subcommand, separate from the install/ packaging recipes) still lists datastore-keeper and datastore-keeper-converter in its tools symlink array. Those fs::create_symlink calls are no-ops on a binary that no longer claims those modes — the symlinks would point at the multipurpose datastore binary which would dispatch by argv[0] and fail. We don't ship datastore install from Dockerfile, so this only matters for the deb/rpm path. Cleanup is queued for Phase 1.1; not load-bearing for this image.

Validation

cmake configure on this macOS workstation aborts at CMakeLists.txt:59 — Submodules are not initialized (same gate the Phase B agent hit). The Phase-1 changes are option-default flips and one Dockerfile edit; they do not introduce new files, new sources, or new libraries. The diff is safe to validate on the Linux build host. Per ~/.claude/CLAUDE.md ("NEVER fucking build images on my computer"), CI runs the configure+build.

Disabled External Engines (Phase A landed)

Hanzo Datastore is OLAP-only. External system integrations (stream brokers, foreign-DB read engines, foreign-DB CDC) are out of scope and disabled at build time. Each engine retains its source tree (gated by a USE_* macro) so upstream merges stay clean; the corresponding ENABLE_* CMake option defaults OFF. No runtime config re-enables them.

Engine / Source	CMake option	Source tree	Contrib subtree
Kafka stream	ENABLE_KAFKA	src/Storages/Kafka/	contrib/librdkafka{,-cmake}/, contrib/cppkafka{,-cmake}/, contrib/libgsasl{,-cmake}/
RabbitMQ	ENABLE_AMQPCPP	src/Storages/RabbitMQ/	contrib/AMQP-CPP/, contrib/amqpcpp-cmake/
MySQL engine + dict + DB	ENABLE_MYSQL	src/Storages/MySQL/, src/Databases/MySQL/, src/Core/MySQL/, src/Interpreters/MySQL/, src/Dictionaries/MySQLDictionarySource.{cpp,h}, src/Storages/StorageMySQL.{cpp,h}	contrib/mariadb-connector-c{,-cmake}/
PostgreSQL engine + MaterializedPostgreSQL CDC + dict	ENABLE_LIBPQXX	src/Storages/PostgreSQL/, src/Databases/PostgreSQL/, src/Core/PostgreSQL/, src/Dictionaries/PostgreSQLDictionarySource.{cpp,h}, src/Storages/StoragePostgreSQL.{cpp,h}	contrib/libpqxx{,-cmake}/, contrib/postgres{,-cmake}/
MongoDB engine + dict	USE_MONGODB	src/Storages/StorageMongoDB.{cpp,h}, src/Storages/StorageMongoDBPocoLegacy.cpp, src/Dictionaries/MongoDBDictionarySource.{cpp,h}	contrib/mongo-c-driver{,-cmake}/, contrib/mongo-cxx-driver{,-cmake}/
HDFS storage	ENABLE_HDFS	src/Storages/ObjectStorage/HDFS/, src/Disks/DiskObjectStorage/ObjectStorages/HDFS/	contrib/libhdfs3{,-cmake}/
Hive metastore	ENABLE_HIVE	src/Storages/Hive/, src/Processors/Formats/Impl/HiveTextRowInputFormat.{cpp,h}	contrib/hive-metastore{,-cmake}/
Cassandra dict	ENABLE_CASSANDRA	src/Dictionaries/CassandraDictionarySource.{cpp,h}, src/Dictionaries/CassandraHelpers.{cpp,h}, src/Dictionaries/CassandraSource.{cpp,h}	contrib/cassandra{,-cmake}/
YTsaurus storage + dict + table fn	ENABLE_YTSAURUS	src/Storages/YTsaurus/, src/Core/YTsaurus/, src/Dictionaries/YTsaurusDictionarySource.{cpp,h}, src/Processors/Sources/YTsaurusSource.{cpp,h}	(no dedicated contrib)

Engines that are kept because they belong to an OLAP/HTTP/object-storage shape: MergeTree family, Replicated*, Distributed, File, URL, S3, ObjectStorage, ObjectStorageQueue (S3 SQS-style queue, in-tree), Memory, Log, View, MaterializedView, RocksDB, Hanzo DocDB-shaped engines, ZooKeeper/Keeper coordination, Dictionary engine (sans foreign-DB sources), NATS (Hanzo PubSub transport), FileLog (local-file, USE_FILELOG=1 on Linux). Kerberos client auth (Access/GSSAcceptor, Access/KerberosInit) is server-side and unrelated to disabled engines — contrib/krb5{,-cmake}/ and contrib/cyrus-sasl{,-cmake}/ MUST be kept.

Phase B — Source Deletion Plan (NOT executed)

Phase A flipped defaults to OFF; Phase B physically deletes the disabled source/contrib trees. This conflicts with the Phase A note "Source trees stay … to keep upstream merges clean" — accept the conflict cost in exchange for a smaller compile surface and faster upstream sync (less code to merge).

Cross-reference (src/CMakeLists.txt): some engine source trees are added unconditionally and require a CMake gate edit before deletion. Those rows are flagged risky below.

# disabled: kafka  (gated: if (TARGET ch_contrib::rdkafka) at src/CMakeLists.txt:129)
git rm -r src/Storages/Kafka
git rm -r contrib/librdkafka contrib/librdkafka-cmake
git rm -r contrib/cppkafka contrib/cppkafka-cmake
git rm -r contrib/libgsasl contrib/libgsasl-cmake   # only consumers were rdkafka + libhdfs3

# disabled: rabbitmq  (gated: if (TARGET ch_contrib::amqp_cpp) at src/CMakeLists.txt:154)
git rm -r src/Storages/RabbitMQ
git rm -r contrib/AMQP-CPP contrib/amqpcpp-cmake

# disabled: hive  (gated: if (TARGET ch_contrib::hivemetastore) at src/CMakeLists.txt:189)
git rm -r src/Storages/Hive
git rm -r src/Processors/Formats/Impl/HiveTextRowInputFormat.cpp src/Processors/Formats/Impl/HiveTextRowInputFormat.h
git rm -r contrib/hive-metastore contrib/hive-metastore-cmake

# disabled: postgresql  (gated: if (USE_LIBPQXX) at src/CMakeLists.txt:158)
git rm -r src/Storages/PostgreSQL src/Databases/PostgreSQL src/Core/PostgreSQL
git rm    src/Dictionaries/PostgreSQLDictionarySource.cpp src/Dictionaries/PostgreSQLDictionarySource.h
git rm    src/Storages/StoragePostgreSQL.cpp src/Storages/StoragePostgreSQL.h
git rm    src/Processors/Sources/PostgreSQLSource.cpp src/Processors/Sources/PostgreSQLSource.h
git rm -r contrib/libpqxx contrib/libpqxx-cmake contrib/postgres contrib/postgres-cmake

# disabled: mongodb  (gated: if (USE_MONGODB) at contrib/CMakeLists.txt:172; src files use #if USE_MONGODB)
git rm    src/Storages/StorageMongoDB.cpp src/Storages/StorageMongoDB.h src/Storages/StorageMongoDBPocoLegacy.cpp
git rm    src/Dictionaries/MongoDBDictionarySource.cpp src/Dictionaries/MongoDBDictionarySource.h
git rm    src/Processors/Sources/MongoDBSource.cpp src/Processors/Sources/MongoDBSource.h
git rm    src/TableFunctions/TableFunctionMongoDB.cpp
git rm -r contrib/mongo-c-driver contrib/mongo-c-driver-cmake contrib/mongo-cxx-driver contrib/mongo-cxx-driver-cmake

# disabled: cassandra  (dict source uses #if USE_CASSANDRA; contrib has its own gate)
git rm    src/Dictionaries/CassandraDictionarySource.cpp src/Dictionaries/CassandraDictionarySource.h
git rm    src/Dictionaries/CassandraHelpers.cpp src/Dictionaries/CassandraHelpers.h
git rm    src/Dictionaries/CassandraSource.cpp src/Dictionaries/CassandraSource.h
git rm -r contrib/cassandra contrib/cassandra-cmake

# disabled: hdfs  (RISKY: Storages/ObjectStorage/HDFS added unconditionally at src/CMakeLists.txt:140)
# Requires gating add_headers_and_sources(dbms Storages/ObjectStorage/HDFS) under TARGET ch_contrib::hdfs first.
git rm -r src/Storages/ObjectStorage/HDFS
git rm -r src/Disks/DiskObjectStorage/ObjectStorages/HDFS
git rm -r contrib/libhdfs3 contrib/libhdfs3-cmake

# disabled: ytsaurus  (RISKY: Core/YTsaurus and Storages/YTsaurus added unconditionally at src/CMakeLists.txt:151-152)
# Requires gating both lines under if (ENABLE_YTSAURUS) first.
git rm -r src/Core/YTsaurus src/Storages/YTsaurus
git rm    src/Dictionaries/YTsaurusDictionarySource.cpp src/Dictionaries/YTsaurusDictionarySource.h
git rm    src/Processors/Sources/YTsaurusSource.cpp src/Processors/Sources/YTsaurusSource.h

# disabled: mysql  (RISKY: 4 add_object_library() calls unconditional at src/CMakeLists.txt:273,278,288,294;
# also blocked by ch::mysqlxx hard-linking ch_contrib::mariadbclient — used by Dictionaries/Embedded
# and other always-on call sites. Requires either dropping mysqlxx contract or gating it on ENABLE_MYSQL.)
# DEFER until ch::mysqlxx is detached from mariadbclient.
git rm -r src/Storages/MySQL src/Databases/MySQL src/Core/MySQL src/Interpreters/MySQL
git rm    src/Storages/StorageMySQL.cpp src/Storages/StorageMySQL.h
git rm    src/Dictionaries/MySQLDictionarySource.cpp src/Dictionaries/MySQLDictionarySource.h
git rm    src/Processors/Sources/MySQLSource.cpp
git rm -r contrib/mariadb-connector-c contrib/mariadb-connector-c-cmake

Phase C — deferred deletions (do NOT do in Phase B)

• programs/keeper/, src/Coordination/, contrib/NuRaft/, contrib/nuraft-cmake/ — wait for the keeper-replace Blue (lux/consensus → Quasar) to land. Until then, ZooKeeper coordination is load-bearing for Replicated* engines. Status: in progress — see "Coordination: Raft → Quasar" below.
• HDFS, YTsaurus, MySQL — see RISKY notes above; they require src/CMakeLists.txt edits to gate currently-unconditional add_headers_and_sources / add_object_library calls. That edit is itself a Phase C task because it touches code on the upstream merge path.

Coordination: Raft → Quasar (the NuRaft rip)

The fork is replacing the NuRaft consensus engine under Keeper with native Lux Quasar (libluxconsensus, from luxfi/consensus pkg/c), linked into the C++ server. The ZooKeeper API is kept — KeeperStateMachine over KeeperStorage is the contract Replicated* speaks; only the engine (raft_server, leader election, asio peer transport) is replaced.

Why it's tractable: NuRaft coupling is contained in src/Coordination/ (25 files); the only references elsewhere are src/CMakeLists.txt + configure_config.cmake (build config, not code). Nothing in the query/storage/replication layers touches nuraft:: types. The join point is KeeperStateMachine::commit(log_idx, buf): NuRaft calls it today, the Quasar engine calls the identical method.

Staged so each step lands under green tests (never break it all at once):

1. Engine proven + reusable ✅ — src/Coordination/QuasarKeeperConsensus.{h,cpp} orders + commits batches through the real KeeperStateMachine (single-node: always-leader, 1-of-1 finality). Tested by src/Coordination/examples/ (keeper_quasar_poc, keeper_quasar_engine_test), opt-in behind ENABLE_EXAMPLES AND LUXCONSENSUS_DIR, build-verified on a 26.6.1.1 host.
2. Engine into dbms + KeeperServer cutover — wire libluxconsensus as a contrib (ch_contrib::luxconsensus), move the engine .cpp into dbms, and make KeeperServer construct QuasarKeeperConsensus instead of nuraft::raft_server for single-node. The dispatcher contract it must satisfy (putRequestBatch async result, isLeader/isLeaderAlive/getLeaderID, createSnapshot, applyConfigUpdate, recovery flags) is mapped in the engine README. Gate on the full gtest_coordination suite green.
3. Multi-node Quasar over ZAP — replace NuRaft's asio peer transport with ZAP (the canonical Hanzo transport); leadership/membership derive from the Quasar validator set. This is what makes a real keeper ensemble run on Quasar.
4. Excise residual nuraft:: data types (buffer/log_entry/snapshot/ptr — trivial containers, ~300 sites) and delete contrib/NuRaft + contrib/nuraft-cmake
   • the src/CMakeLists.txt/configure_config.cmake refs. End state: no libnuraft in the fork; coordination is Quasar-native end to end.

Stage 1 is merged. Stages 2–4 are the production cutover and are large (KeeperServer is ~1350 lines welded to raft internals, incl. a KeeperRaftServer : nuraft::raft_server subclass reaching protected members) — do them on a Linux build host with the Coordination test suite as the gate, not on a laptop.

Phase B — execution status

Branch cleanup/phase-b was created but no deletions were committed. Local cmake configure could not be validated:

• macOS AppleClang is rejected by cmake/tools.cmake; switching to Homebrew clang reaches the next gate.
• After installing GNU findutils, GNU grep, and pointing OBJCOPY_PATH at llvm-objcopy, configure reaches CMakeLists.txt:59 and aborts: Submodules are not initialized. The repo's submodule set is multi-GB; initialization was not attempted under the rule "do not patch around configure failures".

Per the rule "If cmake configure fails after deletion, revert that group and report it as risky — don't try to patch around it", baseline configure failure means no group can be safely validated locally on this host. Deletion was therefore not executed; the plan above is the deliverable. Run the deletion + configure check on a Linux build host (CI runner with submodules initialized) before committing any of these git rm -r blocks.

MinIO: purged from the docs, deliberately kept in the test harness

Hanzo S3 is SeaweedFS-derived and Apache-2.0 — never a MinIO fork. The user-facing S3 documentation said otherwise and handed out minioadmin / minioadminpassword as example credentials, so docs/en/** and the src/** doc-strings that mirror it now use placeholders and a Hanzo S3 endpoint.

The integration-test harness still runs MinIO, and that is the correct state until someone ports it properly. Do not sed it. What holds it:

• minio==7.2.20 in ci/docker/integration/runner/requirements.txt is a real dependency: helpers/cluster.py does from minio import Minio, and 60 test files call 16 distinct methods on cluster.minio_client (make_bucket, fput_object, set_bucket_policy, get_object_tags, …). boto3 is already in the same requirements file, but its API is not method-for-method compatible, so this is a rewrite of the harness plus every call site.
• 136 tests pass with_minio=True; ~120 storage-config XMLs hardcode the fixture credential ClickHouse_Minio_P@ssw0rd.
• compose/docker_compose_minio.yml leans on MinIO-only surface: --certs-dir TLS (test_s3_with_https), MINIO_PROMETHEUS_AUTH_TYPE, a console address, and the warehouse.minio / warehouse-rest.minio aliases that exercise virtual-host-style bucket addressing. Hanzo S3 has equivalents (-s3.domainName, /healthz, Hanzo_s3_* metrics) but not the same flags.
• proxy1/proxy2 come from clickhouse/s3-proxy, built in-repo at ci/docker/integration/s3_proxy but pulled from a registry at test time — a migration needs that image rebuilt and republished, which is a CI change, not a source change.

A port is a real project with a real acceptance test (the S3 integration suite green), and it cannot be validated on an arm64 workstation: ghcr.io/hanzoai/s3 publishes linux/amd64 only, and the suite needs a built datastore binary. Run it on a Linux CI host or not at all.

Also left: the MinIO-quirk comments in src/IO/S3/** and src/IO/S3Common.h (the transient InvalidPart / NO_SUCH_KEY retries). They explain why live retry logic exists, and they describe the server CI actually runs — deleting them would delete the rationale and make the comment less true, not more.

src/Parsers/obfuscateQueries.cpp matches minio only as a prefix of the word minion in a word list. Grep case-sensitively and with \bminio before touching anything.

Related

• Used by: hanzo/insights (insights-datastore deployment in ~/work/hanzo/universe/infra/k8s/insights/)

Operating this thing (2026-08-09, learned the hard way)

There is no client binary. Ours is named ours, so clickhouse-client does not exist in the pod. Query it over HTTP from an insights-web pod:

http://datastore:8123/?query=<urlencoded>&user=$DATASTORE_USER&password=$DATASTORE_PASSWORD

Logs are NOT on the PVC. Only /var/lib/hanzo-datastore is persisted; the server logs to /var/log/hanzo-datastore-server/ on the container filesystem, so a crashing container takes its error log with it and kubectl logs shows only the pre-logger startup lines. To see a startup failure, run a throwaway pod with the same image and configmap plus <logger><console>1</console></logger> — that is the only way to read the real exception.

Two config faults that make it refuse to start, both found in production (universe/charts/app/values/hanzo/datastore.yaml, Helm-managed — a kubectl edit to the ConfigMap IS reverted by GitOps within about a minute):

• -- is illegal inside an XML comment. Prose explaining a previous outage used it six times across log-retention.xml and memory.xml; the server dies with SAXParseException: Invalid token. The running process holds its config in memory, so this stays invisible until a restart. Validate every embedded document parses before pushing.
• background_pool_size below the mutation threshold. Setting it to 4 makes the server exit 36: number_of_free_entries_in_pool_to_execute_mutation defaults to 20 and must be LOWER than the pool. It also starves the merges you need most.

The memory death spiral. datastore-0 wedged above its own max_server_memory_usage (10.80 GiB of a 12Gi limit) and refused EVERY query including SELECT 1, so all analytics failed while the site still served 200. Mechanism: merges need memory → memory exhausted → merges stall → parts accumulate → per-part metadata consumes more memory. event.metric_30m reached ~126k of ~126.7k total active parts. The tiny parts were the SYMPTOM, not a bad writer — recent parts were ~4,600 rows each. A restart clears it and it drains on its own (~2.4 parts/sec, memory falling). Reducing the merge pool makes it worse, not better. The node has no headroom: memory limits at 161% overcommit, 12Gi limit against ~13.6Gi allocatable, so raising it in place is not available.

Migrations. migrate_datastore --check failing is not always about migrations — it fails the same way when the server is unreachable or wedged, so read the error before concluding. As of 2026-08-09 there were 83 genuinely unapplied migrations (last applied 0223_event_name_backmap), several of them materializations over a 15.5 GiB table; drain the parts backlog first and classify additive-vs-destructive before applying.

The headers are translated by nginx, not emitted by the engine

There is an nginx header-translation proxy inside the container (Server: nginx, master + 2 workers on 8123, engine demoted to 8124). It rewrites response headers from the engine's spelling to ours. So seeing X-Datastore-* on the wire is NOT evidence that a renamed binary shipped — it translated five and missed three, which is where x-clickhouse-exception-tag came from. The translation is complete now (zero x-clickhouse-*, error path included), but Server: nginx remains because stock nginx cannot rename that header. It goes away when the proxy does.

No published image contains the rename. The newest revision-labelled build is 2026-03-11; the rename commits are 2026-06-21. Every March revision is unreachable — a later re-import of the overlay orphaned that history. 26.6.1.1/26.6 carry no revision label and are a laptop hand-build (arm64, /etc/datastore-server paths, single_binary_location_url=localhost:8899); do not pin them. The fast rebuild path is dead (pkg.hanzo.ai is now an npm proxy, tarball 404s), so a from-source build is the only route: ~245 min against a runner ceiling that was 3h and is now 6h. Two earlier runs died at 190.5 and 193.5 min and were misread as OOM — they were the ceiling. 26.2 → 26.7 is a one-way door (data path AND part format); treat it as a reviewed migration.

subPath mounts do not refresh — and a missing key blocks startup

A ConfigMap key deleted while a subPath mount still names it makes the next pod stop at ContainerCreating. It is invisible until something restarts the pod, because subPath mounts never refresh. This nearly bit us: f2ec4616c removed the paths.xml key and left the mount. If you delete a config key, delete its mount in the same change.

Two real bugs the rename left, worth knowing the shape of

• src/Client/BuzzHouse/Generator/ExternalIntegrations.cpp read is_clickhouse, a member that exists nowhere (it is is_datastore three lines away). It compiles only because ENABLE_BUZZHOUSE is off — turning it on does not build.
• src/Common/FileChecker.cpp wrote {"datastore": …} but read only datastore/yandex. A directory written by a build that said clickhouse reads back as an empty file list — no error, just a checker that believes the table has no data.

The rename crossed the contract line in places, and those are breaking now

Renaming a name is safe; renaming a contract is not. Already throwing or broken: SET dialect='clickhouse'; the dictionary source registers only as datastore, so SOURCE(CLICKHOUSE(…)) and <source><clickhouse> throw (48 examples in FunctionDocumentation are now invalid SQL and are served through system.functions); Prometheus metric prefixes are Datastore*, so any dashboard matching ClickHouse* is broken; the SSH-signature namespace is datastore, so SSH-key auth will not interop with upstream clients. Decide these deliberately — aliases are cheap where interop matters.

ZAP is present but is a port-opener, not a transport

contrib/zap (debranded Cap'n Proto) and src/Server/ZapServer.{h,cpp} (181 lines) exist, but the server serves a null bootstrap capability — no schema, no method, no SQL — and is compiled out (USE_ZAP=0, submodule empty). The ProtocolServerAdapter plumbing is done (~70 lines); that was never the expensive half. For scale: the gRPC server deleted to make room was 2,023 lines plus a 227-line schema.

The decisive constraint: this stub speaks capnp-family RPC, while hanzoai/s3 and the Hanzo Go services speak the envelope wire, and there is no C++ envelope runtime (zapgen emits Go only). Honest cost: demo-grade first cut 8-15 engineer-days; parity with the deleted gRPC transport +30-60; actually wire-compatible with the Go services +20-40 and needs a C++ runtime nobody has written yet.

The from-source build fails on a macro the rename half-renamed

CMakeLists.txt sets DATASTORE_CLOUD and src/Common/config.h.in emits it as #cmakedefine01 DATASTORE_CLOUD, so that is the only spelling that reaches the preprocessor. Eighteen #if sites in src/ still read CLICKHOUSE_CLOUD, which is defined nowhere. This tree builds with -Werror,-Wundef, so an undefined macro in #if is a hard error, not a false branch.

The trap is when it lands. Ninja reaches Storages/ about two hours in, so the failure looks like the build died of exhaustion — and it was twice answered as memory (compile jobs 6 → 3) and once as a timeout. It is neither. Read the log for is not defined, evaluates to 0 before you believe any resource theory.

The definition side of the rename was already complete — CMake variable, config template and the contrib guards all say DATASTORE_CLOUD. Only the reads were missed.

Check the whole class in one pass, because each miss costs a two-hour round trip:

git grep -hoE '#\s*(if|ifdef|ifndef|elif)[^/]*\b(CLICKHOUSE|HANZO)_[A-Z_]+' -- src/ programs/ base/ utils/
comm -23 <(git grep -hoE '#\s*(if|elif)[^/]*\bDATASTORE_[A-Z_]+' -- src/ | grep -oE 'DATASTORE_[A-Z_]+' | sort -u) \
         <(grep -oE 'DATASTORE_[A-Z_]+' src/Common/config.h.in | sort -u)

Both must come back empty.

The published 26.2.3.2 image was never built from this source

Dockerfile.hanzo at the revision the live image carries (8adc31826d42) downloads upstream release tarballs and then runs mv /usr/bin/clickhouse /usr/bin/hanzo-datastore, copies hanzo-paths.xml (which is what resolves path to /var/lib/hanzo-datastore/), a port override that moves the engine to 8124, and nginx-header-proxy.conf. The binary is upstream ClickHouse.

Two consequences worth holding onto. The engine emits X-ClickHouse-* because it is upstream — nginx was baked into the image from the beginning, not bolted on later. And there is no source-level rename on the 26.2 line to port forward: "put the rename on the same 26.2 base" is not a cherry-pick, it is a new from-source image. origin/26.2 and v26.2.3.2-stable are pristine upstream, and main is not a descendant of either (43,008 commits of divergence).

That build is feasible — 26.2 wants clang 21, the same floor main compiles under, and the from-source Dockerfile already takes the Rust nightly as an ARG (26.2 wants nightly-2025-07-07). What it needs decided first is the tag: imgver publishes max(declared, published), which assumes one monotonic lineage and cannot express a 26.2-lineage image once 26.7 is published.

Config that lives in git is not config the cluster has

The runner job ceiling is timeout: in infra/k8s/git-runner/config.yaml in hanzoai/universe. Two independent gaps sit between committing it and a build benefiting from it.

The universe CD Application is deliberately non-enforcing — no syncPolicy.automated — so a commit changes nothing on its own; it reports OutOfSync and waits. Reconcile the one directory with kubectl diff -k infra/k8s/git-runner then apply -k, which moves the cluster toward git rather than away from it.

Then act_runner reads its config once at boot. The ConfigMap is a volume mount, so the file updates inside every running pod within about a minute — checking the file in a pod therefore proves nothing. Only a pod that booted after the apply is actually running the new ceiling. Cycle the StatefulSet and verify against .status.updatedReplicas, not pod age.

Talking to the forge

git.hanzo.ai serves only the git wire from outside; its API is not exposed there. Port-forward svc/hanzo-git and use /v1/ — not /api/v1/, which 404s. Basic auth with the same credentials that push works. Useful paths: /v1/repos/hanzoai/datastore/actions/tasks?limit=N, .../actions/runs/<run>/jobs, and /v1/repos/hanzoai/datastore/actions/jobs/<job>/logs — note the log endpoint takes a job id and only one of a run's two job rows serves logs; the other 404s.

The repo is not a mirror any more ("mirror": false), whatever .hanzo/workflows/cicd.yml says about mirror-sync. Push to it directly to trigger CI. Because the workflow sets cancel-in-progress: true on main, a second push cancels the build the first one started — batch your commits, and do not push docs while a build you want is running.

An image-only tag swap starts green and reads nothing

The live StatefulSet sets no command and no args, so the image's own ENTRYPOINT decides everything. That makes the image, not the manifest, the thing holding the contract — and the from-source image does not hold the same one the running image does.

live (Dockerfile.hanzo)        from-source (Dockerfile)
/usr/bin/hanzo-datastore       /usr/bin/datastore
/etc/hanzo-datastore-server/   /etc/datastore-server/
/var/lib/hanzo-datastore/      /var/lib/datastore/   (config.xml:556)

Every one of those is where the cluster has something mounted: the 200Gi PVC at /var/lib/hanzo-datastore, and the whole universe config.d set — including paths.xml, which is what actually resolves path, and which no image bakes — at /etc/hanzo-datastore-server/config.d/.

So a tag swap does not fail. entrypoint.sh defaults DATASTORE_CONFIG to /etc/datastore-server/config.xml, a directory where none of those overlays are mounted, reads path from the compiled default, and initializes an empty database on the container's ephemeral layer. /ping answers 200, both probes pass, and the pod reports 1/1. The parts are still on the PVC, untouched — the engine simply never looks at them. Reverting the tag brings them back; writes taken during the window are gone.

Worth being precise about the order of the two doors. Until the contract is aligned the new engine never touches the existing data directory, so it cannot rewrite a part — the part-format door is not even open yet. Align the paths and it opens on the first merge, and from there the way back is a volume snapshot rather than a tag revert. A migration therefore needs both, in one reviewed change: the contract (binary, config dir, path) and a snapshot taken before the engine is first pointed at real data.