The forge reads .github/workflows natively and github.com reads nothing else,
so one directory serves both planes instead of two conventions that disagree
about which one is live. The reusable in hanzoai/ci already lives there, so a
caller naming that path now resolves to a file that exists.
Co-authored-by: Hanzo Dev <[email protected]>
hanzo-build-linux-amd64, lux-build-linux-amd64, zoo-build-linux-amd64 and
every other org-prefixed spelling all meant the same machine. Which org a
job belongs to does not change which host can build it, so the org bought
one pool name per org per arch and each could go stale on its own -- the
evo host alone answered to fourteen names for one label.
One vocabulary now: linux-amd64, linux-arm64, macos-arm64, win-amd64. The
generic labels every plain workflow uses (ubuntu-latest, self-hosted) are
untouched, so naive CI keeps working.
Co-authored-by: Hanzo Dev <[email protected]>
hanzoai/ci keeps the reusable at .github/workflows/build.yml — the one path
both planes resolve, since github.com reads a reusable only from there and the
forge reads it too. The v1 tag moved with it, so a caller still naming
.hanzo/workflows/build.yml asks for a blob that is not in the tree:
InsertRun: read hanzoai/ci@v1:.hanzo/workflows/build.yml: object does not exist
The caller itself stays at .hanzo/workflows/cicd.yml on purpose: github.com has
no runner for these labels, so a caller placed there could never be scheduled.
Co-authored-by: Hanzo Dev <[email protected]>
One family. Font stacks name Zen and fall back to the native faces, tokens
name the role rather than a face, and the packages that used to supply a
face now point at @hanzo/font.
Self-hosted @font-face blocks are removed rather than renamed: rewriting the
family in one would leave a src pointing at a file this repo no longer ships.
@hanzo/font carries the binaries and declares the faces, so the local copies
go with the declaration that used them.
Comments naming a face we no longer use are deleted, not reworded.
Co-authored-by: Hanzo Dev <[email protected]>
runs-on was a bare scalar. Jobs written that way queue and are never
picked up — link and quest have waited since 10 August, painter since the
5th, none of them with a single success — while repos writing the same
label as a sequence drain within seconds of a push on the same fleet.
Same label, same runners, different shape. It fails silently: nothing errors,
the job simply never starts, so the lane looks configured and does nothing.
The fork already renamed its internals - the built binary is datastore, the
packages are datastore-server and friends, and the shipped config.xml already
says Datastore - but the surfaces a human reads still said ClickHouse. Issue
templates asked which ClickHouse version was affected, the PR template told
contributors to close upstream's issues, and CI job names, log lines, error
messages and argparse descriptions named a product this repo does not ship.
Only prose moved: display names, log and error messages, comments, docstrings
and form labels. Everything a machine resolves stayed exactly as it was,
because renaming those breaks a build rather than a sentence - binary and path
tokens such as clickhouse-client and /etc/clickhouse-server, python and shell
identifiers, CLICKHOUSE_* environment variables, the /ClickHouse container
mount, docker images in a namespace we do not own, S3 buckets, maven and pypi
coordinates, and every URL pointing at upstream. The attribution in NOTICE and
README is untouched.
Two links were wrong rather than merely stale, so they are corrected here. The
PR template's Closes and Related placeholders sent contributors to upstream's
tracker instead of this one, and an earlier pass had rewritten a source
reference to github.com/Datastore/Datastore, which does not resolve.
Co-authored-by: Hanzo Dev <[email protected]>
docs/ was 502M and 23,841 files: upstream's entire documentation site, in ten
languages, with their branding throughout. Nothing in our pipeline touches it —
the image builds from the root Dockerfile and cmake compiles no documentation,
and our CI is one image build rather than upstream's Praktika jobs, so the doc
jobs and the style check that reads this tree never run for us. docs.hanzo.ai
serves our documentation and 404s /datastore, so none of it reached a reader
either.
The licence is the part worth knowing. docs/LICENSE is CC BY-NC-SA 4.0 —
NONCOMMERCIAL — which is not this repository's Apache-2.0. So that tree was
never ours to ship in a commercial product, independent of whose name is on it.
That exposure did not stop at the tree: programs/server/docs.html inlined four
of those snippets (a JSON-type guide, two settings tables, a data-type mapping)
and that page IS served by the binary. Deleting the tree would have left the
copies behind, so DOC_SNIPPETS is emptied — the resolver still iterates it, so
an entry importing one renders without it rather than failing, and writing our
own is a matter of adding entries back.
The root LICENSE and NOTICE are untouched and still carry the upstream Apache
copyright. The eleven legal files that went with the tree were docs/LICENSE
itself and ten translations of one system-table reference page.
Co-authored-by: Hanzo Dev <[email protected]>
play.html's SQL highlighting was dead. The C API had been renamed to
datastore_lexer_* in Lexer.cpp and play.html's calls renamed with it, but the
base64 payload embedded in the page was never rebuilt — it still exported
clickhouse_lexer_*. So play.html asked WebAssembly for six functions the module
does not have. docs.html kept the old call names, which is the only reason its
highlighting still worked.
This is the failure mode worth naming: rename either side of a binary contract
alone and nothing errors loudly — the highlighter just stops highlighting.
Rebuilt from src/Parsers/Lexer.cpp with the flags src/Parsers/CMakeLists.txt
uses (-Os -fno-exceptions -fno-rtti -DLEXER_STANDALONE_BUILD --target=wasm32
-flto -nostdlib -Wl,--no-entry -Wl,--export-all, plus -std=c++20 for the fold
expressions). 6720 bytes, exporting datastore_lexer_{size,create,next_token,
token_is_error,token_is_end,token_is_significant}. Embedded in BOTH pages and
both now call those names — verified by instantiating the payload and checking
every symbol each page uses is present, which is the check that would have
caught this when the rename happened.
Co-authored-by: Hanzo Dev <[email protected]>
PrometheusMetricsWriter emits DatastoreProfileEvents_, DatastoreMetrics_,
DatastoreAsyncMetrics_, DatastoreErrorMetric_ and Datastore_Info, while four
consumers still asked for the ClickHouse spelling, so each was reading a string
its own server had stopped writing. The three integration tests failed outright,
the prometheus endpoint one twice over, since its allowed-prefix assertion
rejects every metric line before the query counter is ever looked up. The keeper
stress harness was quieter and worse: an allowlist matching nothing reads as a
run that produced no metrics rather than as an error.
The keeper-stress-analysis skill keeps the ClickHouse spelling on purpose. It
queries keeper_stress_tests.keeper_metrics_ts on play.clickhouse.com, where
those names are stored data written by upstream nightlies rather than anything
this server emits, and that table holds no Datastore rows at all. Renaming there
would turn a query that matches into one that matches nothing.
Co-authored-by: Hanzo Dev <[email protected]>
Renames the remaining button id (btnQueryClickhouse → btnQueryDatastore, one
definition and ten references across the two visualizers) and the test harness's
prose about which server it drives.
What stays, and it is not branding: `clickhouse/mysql-js-client` is an upstream
image the integration tests still PULL (ci/defs/defs.py, and the compose file
that runs it). A pull coordinate names someone else's published artifact, so
changing it is publishing our own test image rather than editing a string — the
harness comment now says that where a reader will be standing.
Also still standing, for the reason recorded at its embed site: the
`clickhouse_lexer_*` exports baked into docs.html's WebAssembly payload.
Co-authored-by: Hanzo Dev <[email protected]>
Apache-2.0 grants no trademark licence, so the upstream name and its links do
not belong on a surface a user of this fork sees. The four links that pointed
at the upstream repository — the source-code link, the two marks beside it,
and the About link in the map's attribution strip — point here.
What stays is not branding: the dashboard parses metric names off the wire,
and that prefix is what this server actually emits. The regex, the comment
describing the format and the example in the placeholder are one identifier,
not a name. Changing them means changing the metric the server publishes,
which is a decision about a wire format and every scrape config already
reading it.
The copyright and the attribution are untouched in NOTICE, which is where a
redistributor is asked to keep them.
Co-authored-by: Hanzo Dev <[email protected]>
These 99 lines were written while this repo's two remotes had diverged, on a
commit that only one of them carried, so reconciling them left the content off
main. It stands on its own and is worth having back: the half-renamed -Wundef
macro behind a two-hour build failure and the one-pass check that finds the
whole class of it, what the live 26.2.3.2 image actually is (upstream tarballs
plus a mv), and that the forge API lives at /v1.
Co-authored-by: Hanzo Dev <[email protected]>
The 31 workflows under .github/workflows were upstream ClickHouse's own Praktika
pipeline. Every one targeted a runner pool that exists only in ClickHouse's fleet
-- style-checker-aarch64, arm-large, amd-medium, release-maker and the pr-*
variants -- so none could ever be scheduled here. On git.hanzo.ai they were
already unreachable, because WORKFLOW_DIRS takes the first directory that exists
and .hanzo/workflows does. GitHub kept executing them anyway: revert_broken_prs
ran on a schedule and failed every attempt. Deleting the directory is what stops
it.
What goes with them, stated plainly: the upstream integration, functional,
stress, fuzzer, jepsen and sqlancer matrices; nightly coverage, statistics,
changelog and docs autogeneration; distroless rebuilds; and ClickHouse's release
automation -- auto_releases, create_release, backport_branches, cherry_pick,
merge_queue. None of it ran here. What we actually consume is gated by
.hanzo/workflows/cicd.yml, which builds this tree from source and makes the
binary answer a query before the image is pushed.
retry_infra_failures is gone for good. It reran failed runs of pull_request.yml
through the GitHub Actions API and drove GitHub auto-merge; both its subject and
its mechanism are GitHub's, and neither has a counterpart here.
upstream-sync returns, at .hanzo/workflows. It already ran on our own label and
was only collateral to an earlier sweep of .github/workflows, and LLM.md has
described it as live throughout. It now opens its pull request through the forge
API rather than the GitHub CLI, and names the branch after the upstream commit
instead of the day it ran, so a re-run on an unchanged upstream head does nothing
rather than proposing the same merge twice.
This pipeline builds a 135-submodule C++ tree from source -- hours per run,
against minutes for the rest of the estate -- and the file already explains that
the runner pool is the scarce thing, which is why there is no pull_request
trigger and why concurrency cancels in progress.
The same argument applies to paths and was missing: a push touching only
markdown queued a full build. Measured just now, run 64026, from a SECURITY.md
change. paths-ignore covers the files that cannot change the artifact; anything
that can is deliberately not in the list.
This repo is a fork, and its SECURITY.md still carried the upstream's contact —
so a researcher who found a hole in what WE ship was told to mail a vendor who
cannot fix it, and who would then know about a live vulnerability in our
production before we did.
Fourteen repos were in that state: datastore -> [email protected], git and
its two worktrees -> [email protected], auto -> [email protected],
calendar -> [email protected], dns -> [email protected], python-sdk ->
[email protected], plus bot, gui-gates, gui-geist, playground and
playground-oauthcb pointing at brand domains with no mailbox behind them.
All of them now carry one policy and one address, [email protected], matching
what cloud and gateway already used. The policy says out loud to report here
rather than upstream, because the next fork will inherit the same file and the
same instinct.
The two-hour failure is a half-renamed `-Wundef` macro, not memory and not the
job ceiling, and it is worth writing down with the one-pass check that finds the
whole class — every miss costs another two-hour round trip.
Also records what the live 26.2.3.2 image actually is (upstream tarballs plus a
`mv`, with nginx baked in from the start), which is what makes "rebase the rename
onto 26.2" a new from-source image rather than a cherry-pick; that config
committed to universe does not reach a cluster whose Application is
non-enforcing, and that act_runner reads its ceiling only at boot; and how to
reach the forge API, which lives at /v1/ and no longer mirrors.
Co-authored-by: Hanzo Dev <[email protected]>
CMakeLists.txt sets DATASTORE_CLOUD and src/Common/config.h.in emits it as
`#cmakedefine01 DATASTORE_CLOUD`, so DATASTORE_CLOUD is the only spelling that
ever reaches the preprocessor. Eighteen `#if` sites still read CLICKHOUSE_CLOUD,
which is defined nowhere — and this tree builds with -Werror,-Wundef, so each one
is a hard error rather than a silently-false branch.
That is what has been failing the from-source build, roughly two hours in, once
ninja reaches Storages/: MergeTreeData.h and the two Iceberg Compaction files
report `'CLICKHOUSE_CLOUD' is not defined, evaluates to 0`. It reads like the
build died of exhaustion because of when it lands, which is how it was twice
answered as memory and once as a timeout.
The definition side of the rename was already complete — the CMake variable, the
config template and the contrib guards all say DATASTORE_CLOUD. Only the reads
were missed, so this renames the reads and nothing else.
Co-authored-by: Hanzo Dev <[email protected]>
Written down because it was rediscovered from scratch: there is no client binary, the logs are not
on the PVC so a crashing container takes them with it, the x-datastore-* headers come from an
nginx proxy rather than the engine, and two config faults brick startup -- a `--` inside an XML
comment, and a merge pool below the mutation free-entries threshold.
Also the parts death spiral (tiny parts are the symptom, not the cause), the subPath-mount
landmine, the two bugs the rename left, the contract-line renames that are already throwing, and
an honest cost for ZAP -- which is present as a port-opener serving a null capability, not a
transport.
The broad pass renamed prose and strings but stopped short of a handful of
places where the old name was load-bearing, and in two of them it moved a
declaration without its use. What is left here is the correction, not more
renaming.
Two are bugs, not cosmetics:
BuzzHouse `MySQLIntegration::truncateSuffix` still reads `is_clickhouse`,
a name that exists nowhere in the tree — the member is `is_datastore`
three lines above and three lines below. It compiles today only because
`ENABLE_BUZZHOUSE` is off; turning it on does not build.
`FileChecker` writes `{"datastore":...}` into sizes.json but reads only
`datastore` or `yandex`. A directory written by a build that said
`clickhouse` reads back as an empty file list — no error, no missing
file, just a checker that silently believes the table has no data.
`ConfigProcessor` already accepts all three names for the config root;
this makes the on-disk reader agree with it.
Two gtests assert `ClickHouseHistogramMetrics_` / `ClickHouseDimensionalMetrics_`
against a writer that emits `Datastore*`, so they cannot pass.
`SOURCE(CLICKHOUSE(...))` is no longer valid SQL — the dictionary source
registers as `datastore` — but 48 examples in `FunctionDocumentation` still
teach it, and `system.functions` serves those to users. They now say
`DATASTORE`, matching the 12 that already did.
The rest identify the product at runtime: the `DATASTORE_RHT_KERNEL` env
var, the `datastore_disk_checker_` probe file, the Arrow type-mapping
error, the two window-function docs, the Kafka key set, and a fatal-error
log line that sent our operators to upstream's issue tracker.
Deliberately NOT renamed, because renaming them breaks something real:
the 162 `github.com/ClickHouse/ClickHouse` citations (attribution), the
contributor handles in the generated contributors table (people's names),
and `ClickhouseClientOptions` / `GetClickhouseOptions` in the Azure
blob-storage layer (vendored SDK API — ours to call, not to rename).
Co-authored-by: Hanzo Dev <[email protected]>
Run #8 falsified the previous commit. It ran with 3 compile jobs and died the
same silent way as run #7 — log ending mid-line, no error, no FAILED:, no
"ninja: build stopped" — but EARLIER, at [9688/15284] instead of [11637/15284],
and in rocksdb translation units, which are not memory-hungry. Less parallelism
producing an earlier death is not what a memory ceiling does.
What the two runs actually share is a clock. Job durations were 190.5 and 193.5
minutes. act_runner's job timeout is 3h (git-runner-config says so in as many
words: "runner.timeout allows 3h"), and 3h plus teardown is what both runs spent.
A timeout tears down the container and truncates the log identically to an OOM,
which is why the two are indistinguishable from a single run's evidence — the
distinguishing signal only appears when you line up wall time across runs with
different settings.
So the memory story was wrong. There is no evidence any of this ever OOMed: the
runner pod showed restarts=0 and an empty lastState both times, and no OOM or
eviction event was recorded. Six jobs never exhausted anything; it simply got
further before the same clock ran out. Restored to 6, the pod's CPU limit.
The build does not fit the budget, and that is the finding rather than a tuning
problem. At 6 jobs ninja reached 76% of 15284 targets in 173.6 minutes, so a
complete build needs ~228 minutes of compiling plus ~17 minutes of preamble:
~245 against a 180 minute cap, 1.36x over, short by about an hour, on a pod
limited to 6 CPU. Nothing on the cmake line closes that. Resuming across runs via
a BuildKit cache mount does not either — runs #7 and #8 landed on git-runner-6
and git-runner-18, and each runner has its own dockerd, so a warm /build is a
1-in-20 accident rather than a mechanism.
Closing this needs a longer job timeout for this workload or a larger dedicated
runner. Both are platform decisions, so they are recorded in the Dockerfile and
here instead of being hidden under progressively smaller job counts.
Run #7 compiled for 174 minutes, reached 11637 of 15284 targets, and stopped. Not
failed: stopped. The log's last line is a Building CXX for
Functions/FunctionsConversion_impl27.cpp.o and then nothing — no error, no
FAILED:, no "ninja: build stopped", no signal reported anywhere in 2 MB of log.
git-runner-6 showed restarts=0 and lastState={}, and the namespace recorded no
OOM or eviction. That combination is the signature this Dockerfile already
described: the kill lands on a process inside the pod's own dockerd, so the
kubelet never sees it and the only symptom is a log that ends mid-sentence.
It was our own budget, as far as the evidence reaches. No sibling build finished
anywhere near 00:31Z, and the node now idles at 10 GB of 62.8, so a neighbour
peaking at the same moment is possible but unevidenced. What is evidenced is
where it died. limit_jobs.cmake budgets MAX_COMPILER_MEMORY=2500, which put 6
jobs near 15 GB and looked comfortable inside 26 GiB — but 2500 is an average
over ~15k targets, and FunctionsConversion is the tail that average was never
describing. Upstream split it into impl00..impl27 because a single TU would not
build. Six of them at once exhausted the cgroup, which puts each above ~3.9 GB.
So: 3 compile jobs, holding even if the real peak is 7 GB apiece, rather than 4
which is only safe if it is under ~5.8. Half the cores sit idle through that
stretch and the build gets slower; a run that dies at 75% is slower still,
having spent three hours to produce nothing. Link stays at 1.
Raising this pod's 26Gi limit would be the wrong repair. git-runner runs 20
replicas packed 10 to a node, each requesting 4Gi against a 26Gi limit — 260Gi of
limit on a 62.8 GiB box. A bigger limit just relocates the same invisible kill to
the node. Right-sizing that fleet is a platform decision and is noted here rather
than worked around.
Configure now COMPLETES ("Configuring done (19.3s)") and the build died in the
generate step instead, on "No SOURCES given to target" for _zxc, _zxc_*_avx2/512,
_simdutf and libdeflate.
The targets are not empty; the paths were. CMake defines <PROJECT>_SOURCE_DIR,
and CMakeLists.txt:18 declares project(Datastore), so ClickHouse_SOURCE_DIR is
simply an undefined variable that expands to the empty string. Every contrib
wrapper still spelling it that way built its source path as "/contrib/zxc" or
"/contrib/simdutf/src" — rooted at /, not at the source tree — so the globs
matched nothing and add_library was handed an empty list. An undefined variable
is not an error in CMake, which is why this surfaced as a missing-sources
complaint about the target rather than a bad path.
This is the same unfinished rename as the previous commit wearing different
clothes. Swept both spellings across every in-repo cmake file: exactly two tokens,
ClickHouse_SOURCE_DIR and ClickHouse_BINARY_DIR, 23 occurrences in 13 files, all
of them contrib/*-cmake wrappers plus src/CMakeLists.txt. The submodules
themselves are untouched — the wrappers are ours, contrib/<name>-cmake is not a
.gitmodules path.
Renamed rather than aliased. A one-line set(ClickHouse_SOURCE_DIR
${Datastore_SOURCE_DIR}) in the root would also have worked and would cost less
on the next upstream merge, but the tree already carries hundreds of
Datastore_SOURCE_DIR uses in these same directories: the choice to rename was
made when the project was renamed, and these 13 files are the remainder of that
job, not a new decision. Two spellings for one path is the thing that produced
this bug in the first place.
configure died at src/CMakeLists.txt:127, "Cannot specify link libraries for
target clickhouse_compression which is not built by this project". Two lines
above, add_library creates datastore_compression. The rename reached the
definition and not the reference, and the miss survived because line 127 sits
behind `if (TARGET ch_contrib::sz3)` — it is only evaluated once SZ3 configures,
which nothing in this pipeline had ever got far enough to do.
Fixing that one line would only have surfaced the next, so this is the whole
class rather than the instance. Two sweeps, because one of them is not enough:
diffing every name referenced as the first argument of target_link_libraries /
target_include_directories / target_compile_* / add_dependencies /
set_target_properties against every name created by add_library / add_executable
/ add_custom_target finds 15 undefined, of which 13 are legitimate — corrosion
builds _cargo, _ch_rust_*, delta_kernel_ffi and chdig via corrosion_import_crate,
and google_cloud_cpp_* comes from functions in its own contrib cmake. That leaves
clickhouse_compression and clickhouse_common_io. But a first-argument diff cannot
see a stale name in a dependency LIST or a call to a command that no longer
exists, so the occurrence-level pass matters too and finds both.
Ten occurrences, all of them the same missed rename:
- undefined target, first arg, fatal at configure: src/CMakeLists.txt 127, 570, 578
- undefined names inside unit_tests_dbms's link list: 939, 940, 941. These fail
at LINK, not configure, and only when ENABLE_TESTS=1, so this pipeline would
never have reported them.
- calls to clickhouse_add_executable, which does not exist — the macro is
datastore_add_executable (CMakeLists.txt:668): benchmarks:26, fuzzers:22.
- inert: src/CMakeLists.txt:498 passes a name to add_object_library, whose body
ignores it entirely and only calls add_headers_and_sources(dbms ...); and a
comment at CMakeLists.txt:379 naming the old macro. Neither can break a build;
both are renamed so the next reader cannot mistake them for the live kind.
Renamed on evidence, not on the prefix: ch_contrib::* aliases and upstream contrib
targets keep their names, and every name changed here was confirmed undefined in
the tree with a defined datastore_* counterpart.
Two configure-time defects, both found only because the submodule and imgver
fixes let cmake run at all for the first time.
COMPILER_CACHE=disabled. A missing ccache is fatal here rather than advisory:
FAIL_ON_UNSUPPORTED_OPTIONS_COMBINATION defaults ON, which sets
RECONFIGURE_MESSAGE_LEVEL to FATAL_ERROR, and cmake/ccache.cmake takes that path
when it finds neither ccache nor sccache. The builder image has neither, so
configure stopped at ccache.cmake:48. `disabled` rather than apt-installing one,
because nothing in this Dockerfile or in hanzoai/ci mounts a cache: a compiler
cache here would start cold on every run and leave nothing behind for the next,
which is the same result as disabled plus an apt round trip. Installing one is
worth it only alongside a cache mount that outlives the layer.
PARALLEL_COMPILE_JOBS / PARALLEL_LINK_JOBS. cmake/limit_jobs.cmake derives both
from cmake_host_system_information, which reports the machine and not this
container's cgroup. On the runner fleet that reads 64311 MB / 8 cores while the
pod is capped at 26 GiB / 6 CPU, so it computes 64311/MAX_LINKER_MEMORY(5000) =
12 link jobs, clamps that to the host's 8 cores, and schedules 8 concurrent links
budgeted at 40 GB into a 26 GiB pod. The failure mode is the reason to fix it
before it happens rather than after: the kill lands on a process inside the pod's
own dockerd, so Kubernetes records no OOM event and the build log just stops,
which reads like a compile error and is not one. 6 compiles (the CPU limit; past
that only earns CFS throttling) and 1 link is 6*2500 + 5000 = 20 GB, with room to
spare. Both land as ninja JOB_POOLS.
The build never reached the C++ compile: imgver runs before docker buildx, and
it exits non-zero rather than let a build fall back to sha-<short>. datastore
declared a version nowhere imgver looks — no package.json, no Cargo.toml, no
VERSION, and the root pyproject.toml is pylint configuration with no `version =`
key — so it died with "no version for ghcr.io/hanzoai/datastore" about a minute
in. Run #2 did not show this because it ran an older build.yml that never called
imgver; the ci v1 that main resolves to now does.
26.7.1 is read off the tree, not chosen: cmake/autogenerated_versions.txt carries
VERSION_MAJOR 26, VERSION_MINOR 7, VERSION_PATCH 1. Upstream's VERSION_STRING is
the four-component 26.7.1.1 and imgver's semver is three, hence 26.7.1.
Declaring it is safe against an unknown registry floor, which matters because the
floor could not be read from here (the available credential lacks read:packages).
imgver publishes max(declared, published): verified against the real script at
three floors — none -> 26.7.1, 26.7.1 -> 26.7.2, 26.7.9 -> 26.7.10. The series
cannot be walked backwards by naming a lower number, and no tag can ever cover
two digests.
Same shape and same reason as hanzoai/postgres, which declares version: 18.0.0.
.gitmodules had 135 [submodule] stanzas and the tree had 134 gitlinks. The odd
one out was contrib/zap: config committed, gitlink never was. update-submodules.sh
feeds every .gitmodules path to `git submodule update`, so the path git had never
heard of died with `pathspec 'contrib/zap' did not match any file(s) known to git`,
xargs turned that into exit 123, and the image stopped at Dockerfile:95 ~90s in.
The other 134 clones succeeded around it, which is why the line is easy to miss.
It is load-bearing, not orphaned. contrib/CMakeLists.txt does
add_contrib (zap-cmake zap); zap-cmake reads 67 sources out of contrib/zap/c++/src;
src/CMakeLists.txt links ch_contrib::zap into dbms for src/Server/ZapServer.cpp.
Dropping the stanza instead would have traded a clone error for add_contrib's
FATAL_ERROR on the missing base folder.
Pinned to zap-proto/cpp-core main tip b444363 — the tip specifically, because
update-submodules.sh fetches --depth=1 --single-branch against .gitmodules'
branch = main, and any commit behind the tip costs a second, deeper fetch. All 67
paths zap-cmake names exist there, and the generated .zap.h/.zap.c++ are checked
in, so no bootstrap compiler is needed.
Also pins the caller to hanzoai/ci@v1, the tag the rest of the fleet uses. v1 and
v2 resolve to the same forge commit today, so behaviour is unchanged; this just
stops the one repo on an unmaintained alias from drifting when v1 moves. And the
header comment claiming this file cannot run until the repo is un-mirrored is
struck: runs #1 and #2 both fired from mirror-sync push events.
The S3 engine and table-function docs handed readers `'minioadmin',
'minioadminpassword'` five times as the way to authenticate, plus the
integration fixture's own `ClickHouse_Minio_P@ssw0rd` in the s3Cluster
examples. Copy-pasteable weak credentials in the reference docs are a worse
problem than the branding: they get pasted. Those are placeholders now, and the
endpoints point at Hanzo S3 (`hanzoai/s3` — SeaweedFS-derived, Apache-2.0, and
never a MinIO fork).
`registerStorageObjectStorage.cpp` also linked a docker-compose recipe at
github.com/Datastore/examples, a URL the ClickHouse->Datastore rebrand invented
and that has never resolved. The sentence no longer depends on it.
Scope is deliberately docs only, both copies of them: `docs/en/**` — the one
doc tree `docs/AGENTS.md` says to edit — and the `src/**` `R"DOCS_MD(...)"`
blocks that mirror it into the binary. Every changed line in a .cpp is inside a
documentation raw-string (verified by delimiter-span check, 21/21); the only
header change is a `///` comment. No executable code, no test, no fixture, no
build config.
The integration-test harness keeps MinIO on purpose, and LLM.md now records
why so this is not re-litigated or sed'd: `minio==7.2.20` is a real dependency
behind `from minio import Minio` in helpers/cluster.py, 60 test files call 16
distinct client methods, 136 tests pass with_minio=True, ~120 storage XMLs
hardcode the fixture credential, and the compose file leans on MinIO-only
surface (--certs-dir TLS, prometheus auth mode, console address, virtual-host
bucket aliases) fronted by a registry-pulled clickhouse/s3-proxy image. That
port is a real project whose acceptance test is the S3 suite going green, and
it cannot be run here: ghcr.io/hanzoai/s3 is linux/amd64 only and this is
arm64.
Co-authored-by: Hanzo Dev <[email protected]>
I moved this to .github/workflows on the reasoning that the forge copy is a
read-only mirror and GitHub only resolves reusables out of .github/workflows.
The first half is true; the conclusion was wrong. The GitHub org has ZERO
self-hosted runners registered — /orgs/hanzoai/actions/runners returns
total_count 0, because the estate moved to native CI — so a GitHub-side job
queues against a pool that does not exist. Verified on run 30409963733: accepted,
queued, never scheduled, cancelled. Left in place it would have done that on
every push to main forever, which is worse than not having CI at all.
So the caller belongs at the canonical native path after all, and the real
blocker is the repo, not the file: git.hanzo.ai/hanzoai/datastore is
mirror=true, and a mirror runs no CI. Converting it to a normal repo — the shape
hanzoai/iam and hanzoai/universe already have in that same namespace — is the
one change that makes this build. That precondition is documented in the
workflow rather than left for the next person to rediscover from a silent
no-run.
Also fix a configure-time failure in the Dockerfile: cmake/tools.cmake resolves
llvm-ar/ranlib/objcopy/nm/strip by versioned name and raises FATAL_ERROR when
one is missing. Those ship in llvm-N, not llvm-N-dev, which was the only one
installed — so the build would have died at cmake time.
Co-authored-by: Hanzo Dev <[email protected]>
The forge copy of this repo is a PULL MIRROR — mirror=true, 8h interval,
original_url github.com/hanzoai/datastore — so it is read-only, it syncs
GitHub -> forge rather than the other way, and a mirror runs no CI. GitHub is
the only forge that can build this repo, and GitHub resolves reusable workflows
only out of .github/workflows. The caller I put in .hanzo/workflows would have
been valid YAML that never ran: no error, no run, just an image that never
appears.
@v1 is not a downgrade. hanzoai/ci v1 and v2 differ by a file move and nothing
else — v1 holds .github/workflows/build.yml, v2 holds .hanzo/workflows/build.yml,
byte-identical — and the path is part of the version. Same pipeline, addressed
where GitHub can see it.
If the forge repo is ever de-mirrored to canonical, this moves back to
.hanzo/workflows/cicd.yml @v2 and nothing else changes.
Co-authored-by: Hanzo Dev <[email protected]>
This repo had no native CI, so nothing built it. Add the estate-standard pair —
a ~7-line caller into hanzoai/ci plus a root hanzo.yml holding the real config —
and the from-source Dockerfile they drive, publishing ghcr.io/hanzoai/datastore.
From source because the existing docker/server/Dockerfile installs .deb packages
from packages.hanzo.ai, which does not exist: the global rebrand rewrote the
upstream package host into that name, so the file reads as if we run an apt repo
and only fails at apt-get install. Source is the only path that yields OUR 26.7
rather than somebody else's binary. docker/server/* is left untouched — it is
machinery for a distribution channel we do not operate, and editing it would
conflict on the next upstream merge.
Two things the shared reusable does not do, handled in the Dockerfile: it runs a
bare actions/checkout with no submodule support, so the builder runs upstream's
own contrib/update-submodules.sh (which also strips vendored CMake files); and it
offers no post-build hook, so the artifact gate is the last layer instead —
SELECT version() and system.build_options through the binary just linked, which
fails the build rather than shipping an image that cannot answer a query.
Toolchain is pinned to what ci/docker/fasttest provisions, because that is what
this tree is known to compile under: cmake/tools.cmake hard-fails below clang 21
and the Rust contribs want the pinned nightly. ENABLE_TESTS=0 is the one
deviation from upstream defaults; ENABLE_DATASTORE_ALL stays on so the image
keeps keeper rather than losing it silently.
No deploy block. The live StatefulSet resolves its data directory to
/var/lib/hanzo-datastore while this tree compiles in /var/lib/datastore, so an
image-only swap would miss the PVC and every config mount, initialize an empty
database on the ephemeral layer, and still pass its /ping probes. That migration
needs a snapshot and a mount-path change in the same commit, not a rollout.
Co-authored-by: Hanzo Dev <[email protected]>
This workflow ran on a self-hosted actions-runner-controller pool. ARC is
decommissioned: CI executes inside git.hanzo.ai (Gitea Actions, .hanzo/workflows)
and delivery is cd.hanzo.ai reconciling the reviewed image pin in
hanzoai/universe. GitHub is a mirror and runs nothing.
Removed rather than repointed — there is no GitHub-side pool to move to, and
leaving it would keep ARC load-bearing.
This workflow ran on a self-hosted actions-runner-controller pool. ARC is
decommissioned: CI executes inside git.hanzo.ai (Gitea Actions, .hanzo/workflows)
and delivery is cd.hanzo.ai reconciling the reviewed image pin in
hanzoai/universe. GitHub is a mirror and runs nothing.
Removed rather than repointed — there is no GitHub-side pool to move to, and
leaving it would keep ARC load-bearing.
The config processor treats yandex, clickhouse and datastore as equivalent
root elements so a server started against an existing config file keeps
resolving it. A rename had left the clickhouse branch reading "datastore"
twice, so the third alias tested nothing and a clickhouse-rooted file was
rejected with "Root element doesn't have the corresponding root element".
Deployed config files under o11y and universe use that root, so this is the
difference between a server that starts and one that does not.
Co-authored-by: Hanzo Dev <[email protected]>
The README was upstream's community page with a find-replace applied. It
advertised ClickHouse Inc's meetups, release calls and careers page as ours,
pointed at a `datastore.cloud` we do not run, and carried four dead links plus
an install line that piped an HTML page into `sh`.
Replaced with what this fork actually is: separated storage and compute over
Hanzo S3, one binary with embedded coordination, OLAP-only engine set, no gRPC,
and the Raft to Quasar migration. Benchmark numbers keep their scale caveat.
Ports table and auth headers now match what the server reads.
Co-authored-by: Hanzo Dev <[email protected]>
A prior ClickHouse to Datastore rename rewrote `github.com/ClickHouse/ClickHouse`
to `github.com/Datastore/Datastore`, a repository that does not exist. That
turned 162 references across 131 files into dead links, and turned captured
stack traces and historical changelogs into false statements.
A citation of upstream is a fact, not branding. Product surfaces keep the Hanzo
Datastore name; references to the upstream project point at the upstream project.
Also untracks `.claude/`, agent scratch that was never meant to be committed.
Co-authored-by: Hanzo Dev <[email protected]>
13,838 upstream commits, forward from the 26.6.1.1 fork point. 129 conflicts:
37 resolved mechanically (take upstream's file, re-apply the brand sweep) and
92 by hand where our fork and upstream had both changed the same code.
The rule throughout: upstream owns the engine, we own the naming and a small set
of deliberate deviations. Never resolve by discarding an upstream change — a
dropped rename or a missed new parameter becomes a compile error a thousand
files away.
Deviations verified intact after the merge:
ENABLE_XRAY still defaults OFF (CMakeLists.txt:105)
X-Datastore-User read by the server (authenticateUserByHTTP.cpp)
X-ClickHouse-* in src/ 0
hanzo/research present
system.*_log retention bound present
ArrowFlight rip held against upstream re-adding it
Two things worth naming, because neither shows up as a conflict:
src/CMakeLists.txt gained an UNGUARDED
`add_object_library(datastore_storages_arrowflight Storages/ArrowFlight)` from
upstream, for a directory our gRPC rip deleted. It sat outside every conflict
hunk, so a resolver reading only the markers would sail straight past it into
a CMake configure error. Removed; the Server/ArrowFlight sibling at line 474
stays because it is inside `if (TARGET ch_contrib::arrow_flight)`.
The brand sweep had rewritten github.com/ClickHouse/ClickHouse issue links to
github.com/Datastore/Datastore — a dead host. Those point at upstream's own
tracker and are restored, alongside the contributors list, sql.clickhouse.com
and the ClickHouse Inc copyright: attribution is theirs, not ours to rename.
base/glibc-compatibility/memcpy/memcpy.h is deleted, not kept: upstream dropped
it for LLVM-libc's, and our only change to it was a comment rebrand.
- json::Parser bounds nesting at 256 levels: past the cap value() returns Null
(a parse-fail) instead of recursing, so a deeply-nested response degrades
rather than overflowing the stack. Every server response is parsed here; a
200k-nested body previously SIGSEGV'd the producer.
- Curl transport caps the response body at 8 MiB (a short write aborts the
transfer), so a hostile server cannot OOM the client. Redirects were already
disabled (FOLLOWLOCATION=0). Parity with the Go/Rust ports.
- writeReal emits CPython repr(float) notation: fixed for decimal exponent in
[-4, 16), scientific otherwise, so 1e15/0.0001 match json.dumps. The realistic
value domain is unchanged (ingest byte-identity to Python preserved).
- Docs state the accurate contract: cross-language identity is semantic (server
keys on project + id); this serializer additionally matches json.dumps bytes.
Verified: 200k-nest parse returns without crashing; SHA-256 self-test PASS;
ingest body still 1375 bytes / sha256 1914bc1e… (byte-identical to Python).
Mirror the Python hanzo-research SDK verb-for-verb so native producers (kernels,
benchmarks, the datastore/luxcpp stack) record R&D evidence on /v1/research.
Wire output is byte-identical to Python json.dumps (key order, ensure_ascii,
float repr). Verdict{Proven,Refuted,Inconclusive} is type-safe; kind is an open
string. Zero-config provenance (git/commits/host/libs) auto-captured; the
since-narrative sha is hex-validated before it reaches the shell. Dependency-free
core (ordered JSON + SHA-256 + base64, stdlib only); libcurl is the one swappable
transport behind -DHANZO_RESEARCH_CURL. Self-checking example + CTest.
zeekayandHanzo Dev