datastore

mirror of https://github.com/hanzoai/datastore.git synced 2026-09-26 13:55:07 +00:00

Files

T

History

zooqueen fce0ee72e1 datastore: fix XML config tag mismatch (</clickhouse> closing tags)

Runtime regression from the final lowercase pass: it renamed <clickhouse> opening tags to
<datastore> but the / exclusion (protecting /clickhouse paths) left </clickhouse> closing tags,
so embedded + shipped configs became <datastore>...</clickhouse> -> SAXParseException 'Tag
mismatch' at runtime (compile was fine; XML is data). Renamed all <clickhouse/</clickhouse XML
tags -> datastore (101 closes across configs, embedded strings, keeper configs). Path strings
(/var/lib/clickhouse) have no angle bracket and are untouched. Build compiled; this fixes runtime
config parsing.

2026-06-22 00:25:01 -07:00

configs

datastore: fix XML config tag mismatch (</clickhouse> closing tags)

2026-06-22 00:25:01 -07:00

__init__.py

…

README.md

Remove large file.

2026-04-10 14:19:50 +08:00

test.py

Remove large file.

2026-04-10 14:19:50 +08:00

README.md

Paimon Incremental Read Integration Test

This test validates Paimon incremental read with:

A dedicated writer jar (paimon-incremental-writer) built inside the clickhouse/integration-test-with-hms Docker image
A ClickHouse table created by ENGINE = PaimonLocal(...)
Incremental reads with Keeper state

Test entry

test.py

Writer project

Source: ci/docker/integration/clickhouse_with_hms_catalog/paimon-incremental-data/
Main class: org.apache.paimon.rest.PaimonIncrementalWriter
Jar path inside the Docker image: /opt/paimon/paimon-incremental-writer.jar

How it works

The writer jar is built from source during the Docker image build (clickhouse/integration-test-with-hms) via a multi-stage Dockerfile. At test time, a sidecar container running the same image shares a Docker named volume with the ClickHouse node. The test invokes the writer via docker exec into the sidecar container.

Assertions covered

Warm-up snapshot (1 row) is written and consumed first, to ensure schema can be inferred when creating PaimonLocal table
First business incremental read after snapshot-2 returns 10
Next read without new snapshot returns 0
First business incremental read after snapshot-3 returns 10
Next read without new snapshot returns 0
paimon_target_snapshot_id = 2 is deterministic and returns 10 repeatedly
With max_consume_snapshots = 2, after warm-up consumption and then 3 commits (10 rows per snapshot), reads are capped per query: 20, then 10, then 0
Refreshable MV pipeline: Paimon (incremental read) -> MV (APPEND) -> MergeTree accumulates rows correctly