hanzoai/datastore
2
0
Fork 0
mirror of https://github.com/hanzoai/datastore.git synced 2026-09-26 13:55:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'chore/purge-minio'

Browse Source 
Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
zeekay
2026-09-07 21:52:28 -07:00
co-authored by Hanzo Dev
parent 2cfd692346 8ca48ae27a
commit 1ae727a689
13 changed files with 86 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
LLM.md
+42
View File
@@ -271,6 +271,48 @@ Branch `cleanup/phase-b` was created but **no deletions were committed**. Local
Per the rule "If cmake configure fails after deletion, revert that group and report it as risky — don't try to patch around it", baseline configure failure means *no* group can be safely validated locally on this host. Deletion was therefore not executed; the plan above is the deliverable. Run the deletion + configure check on a Linux build host (CI runner with submodules initialized) before committing any of these `git rm -r` blocks.
## MinIO: purged from the docs, deliberately kept in the test harness
Hanzo S3 is SeaweedFS-derived and Apache-2.0 — never a MinIO fork. The
user-facing S3 documentation said otherwise and handed out `minioadmin` /
`minioadminpassword` as example credentials, so `docs/en/**` and the `src/**`
doc-strings that mirror it now use placeholders and a Hanzo S3 endpoint.
**The integration-test harness still runs MinIO, and that is the correct state
until someone ports it properly.** Do not `sed` it. What holds it:
- `minio==7.2.20` in `ci/docker/integration/runner/requirements.txt` is a real
dependency: `helpers/cluster.py` does `from minio import Minio`, and 60 test
files call 16 distinct methods on `cluster.minio_client` (`make_bucket`,
`fput_object`, `set_bucket_policy`, `get_object_tags`, …). boto3 is already
in the same requirements file, but its API is not method-for-method
compatible, so this is a rewrite of the harness plus every call site.
- 136 tests pass `with_minio=True`; ~120 storage-config XMLs hardcode the
fixture credential `ClickHouse_Minio_P@ssw0rd`.
- `compose/docker_compose_minio.yml` leans on MinIO-only surface: `--certs-dir`
TLS (`test_s3_with_https`), `MINIO_PROMETHEUS_AUTH_TYPE`, a console address,
and the `warehouse.minio` / `warehouse-rest.minio` aliases that exercise
virtual-host-style bucket addressing. Hanzo S3 has equivalents
(`-s3.domainName`, `/healthz`, `Hanzo_s3_*` metrics) but not the same flags.
- `proxy1`/`proxy2` come from `clickhouse/s3-proxy`, built in-repo at
`ci/docker/integration/s3_proxy` but **pulled from a registry** at test time —
a migration needs that image rebuilt and republished, which is a CI change,
not a source change.
A port is a real project with a real acceptance test (the S3 integration suite
green), and it cannot be validated on an arm64 workstation: `ghcr.io/hanzoai/s3`
publishes linux/amd64 only, and the suite needs a built `datastore` binary. Run
it on a Linux CI host or not at all.
Also left: the MinIO-quirk comments in `src/IO/S3/**` and `src/IO/S3Common.h`
(the transient `InvalidPart` / `NO_SUCH_KEY` retries). They explain why live
retry logic exists, and they describe the server CI actually runs — deleting
them would delete the rationale and make the comment less true, not more.
`src/Parsers/obfuscateQueries.cpp` matches `minio` only as a prefix of the word
`minion` in a word list. Grep case-sensitively **and** with `\bminio` before
touching anything.
## Related
- Used by: `hanzo/insights` (insights-datastore deployment in `~/work/hanzo/universe/infra/k8s/insights/`)
docs/en/engines/table-engines/integrations/paimon.md
+1 -1
View File
@@ -159,7 +159,7 @@ SETTINGS
-- S3 storage (the `Paimon` engine defaults to the S3 implementation when no `disk` is specified)
CREATE TABLE paimon_mv_source
ENGINE = Paimon('http://minio:9000/bucket/path/to/table', 'access_key', 'secret_key')
ENGINE = Paimon('http://s3.hanzo.svc:9000/bucket/path/to/table', 'access_key', 'secret_key')
SETTINGS
paimon_incremental_read = 1,
paimon_keeper_path = '/clickhouse/tables/{uuid}',
docs/en/engines/table-engines/integrations/s3.md
+9 -8
View File
@@ -60,7 +60,7 @@ To enable caching use a setting `filesystem_cache_name = '<name>'` and `enable_f
```sql
SELECT *
FROM s3('http://minio:10000/clickhouse//test_3.csv', 'minioadmin', 'minioadminpassword', 'CSV')
FROM s3('http://s3.hanzo.svc:9000/clickhouse//test_3.csv', '<access_key_id>', '<secret_access_key>', 'CSV')
SETTINGS filesystem_cache_name = 'cache_for_s3', enable_filesystem_cache = 1;
```
@@ -135,7 +135,8 @@ arthur :) select _path, * from t_03363_parquet;
### Querying partitioned data {#querying-partitioned-data}
This example uses the [docker compose recipe](https://github.com/ClickHouse/examples/tree/5fdc6ff72f4e5137e23ea075c88d3f44b0202490/docker-compose-recipes/recipes/ch-and-minio-S3), which integrates ClickHouse and MinIO. You should be able to reproduce the same queries using S3 by replacing the endpoint and authentication values.
This example uses a local S3-compatible endpoint (Hanzo S3, `hanzoai/s3`); any
S3-compatible store works by replacing the endpoint and authentication values.
Notice that the S3 endpoint in the `ENGINE` configuration uses the parameter token `{_partition_id}` as part of the S3 object (filename), and that the SELECT queries select against those resulting object names (e.g., `test_3.csv`).
@@ -162,9 +163,9 @@ CREATE TABLE p
)
ENGINE = S3(
-- highlight-next-line
'http://minio:10000/clickhouse//test_{_partition_id}.csv',
'minioadmin',
'minioadminpassword',
'http://s3.hanzo.svc:9000/clickhouse//test_{_partition_id}.csv',
'<access_key_id>',
'<secret_access_key>',
'CSV',
partition_strategy='wildcard')
PARTITION BY column3
@@ -183,7 +184,7 @@ This query uses the s3 table function
```sql
SELECT *
FROM s3('http://minio:10000/clickhouse//test_3.csv', 'minioadmin', 'minioadminpassword', 'CSV')
FROM s3('http://s3.hanzo.svc:9000/clickhouse//test_3.csv', '<access_key_id>', '<secret_access_key>', 'CSV')
```
```response
┌─c1─┬─c2─┬─c3─┐
@@ -194,7 +195,7 @@ FROM s3('http://minio:10000/clickhouse//test_3.csv', 'minioadmin', 'minioadminpa
#### Select from partition 1 {#select-from-partition-1}
```sql
SELECT *
FROM s3('http://minio:10000/clickhouse//test_1.csv', 'minioadmin', 'minioadminpassword', 'CSV')
FROM s3('http://s3.hanzo.svc:9000/clickhouse//test_1.csv', '<access_key_id>', '<secret_access_key>', 'CSV')
```
```response
┌─c1─┬─c2─┬─c3─┐
@@ -205,7 +206,7 @@ FROM s3('http://minio:10000/clickhouse//test_1.csv', 'minioadmin', 'minioadminpa
#### Select from partition 45 {#select-from-partition-45}
```sql
SELECT *
FROM s3('http://minio:10000/clickhouse//test_45.csv', 'minioadmin', 'minioadminpassword', 'CSV')
FROM s3('http://s3.hanzo.svc:9000/clickhouse//test_45.csv', '<access_key_id>', '<secret_access_key>', 'CSV')
```
```response
┌─c1─┬─c2─┬─c3─┐
docs/en/engines/table-engines/mergetree-family/mergetree.md
+1 -1
View File
@@ -872,7 +872,7 @@ If you perform the `SELECT` query between merges, you may get expired data. To a
## Disk types {#disk-types}
In addition to local block devices, ClickHouse supports these storage types:
- [`s3` for S3 and MinIO](#table_engine-mergetree-s3)
- [`s3` for S3-compatible object storage](#table_engine-mergetree-s3)
- [`gcs` for GCS](/integrations/data-ingestion/gcs/index.md/#creating-a-disk)
- [`blob_storage_disk` for Azure Blob Storage](/operations/storing-data#azure-blob-storage)
- [`hdfs` for HDFS](/engines/table-engines/integrations/hdfs)
docs/en/sql-reference/table-functions/file.md
+2 -2
View File
@@ -2,7 +2,7 @@
description: 'A table engine which provides a table-like interface to SELECT from
and INSERT into files, similar to the s3 table function. Use `file` when working
with local files, and `s3` when working with buckets in object storage such as
S3, GCS, or MinIO.'
S3, GCS, or any S3-compatible store.'
sidebar_label: 'file'
sidebar_position: 60
slug: /sql-reference/table-functions/file
@@ -15,7 +15,7 @@ import CloudNotSupportedBadge from '@theme/badges/CloudNotSupportedBadge';
# file Table Function
A table engine which provides a table-like interface to SELECT from and INSERT into files, similar to the [s3](/sql-reference/table-functions/s3.md) table function. Use `file` when working with local files, and `s3` when working with buckets in object storage such as S3, GCS, or MinIO.
A table engine which provides a table-like interface to SELECT from and INSERT into files, similar to the [s3](/sql-reference/table-functions/s3.md) table function. Use `file` when working with local files, and `s3` when working with buckets in object storage such as S3, GCS, or any S3-compatible store.
The `file` function can be used in `SELECT` and `INSERT` queries to read from or write to files.
docs/en/sql-reference/table-functions/iceberg.md
+4 -4
View File
@@ -80,13 +80,13 @@ When using a catalog, most users will want to use the `DataLakeCatalog` database
To use them, create a table with the `IcebergS3` engine and provide the necessary settings.
For example, using REST Catalog with MinIO storage:
For example, using REST Catalog with S3 storage:
```sql
CREATE TABLE `database_name.table_name`
ENGINE = IcebergS3(
'http://minio:9000/warehouse-rest/table_name/',
'minio_access_key',
'minio_secret_key'
'http://s3.hanzo.svc:9000/warehouse-rest/table_name/',
'<access_key_id>',
'<secret_access_key>'
)
```
docs/en/sql-reference/table-functions/s3Cluster.md
+5 -5
View File
@@ -55,9 +55,9 @@ Select the data from all the files in the `/root/data/clickhouse` and `/root/dat
```sql
SELECT * FROM s3Cluster(
'cluster_simple',
'http://minio1:9001/root/data/{clickhouse,database}/*',
'minio',
'ClickHouse_Minio_P@ssw0rd',
'http://s3.hanzo.svc:9000/root/data/{clickhouse,database}/*',
'<access_key_id>',
'<secret_access_key>',
'CSV',
'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))'
) ORDER BY (name, value, polygon);
@@ -73,8 +73,8 @@ For production use cases, it is recommended to use [named collections](/operatio
```sql
CREATE NAMED COLLECTION creds AS
access_key_id = 'minio',
secret_access_key = 'ClickHouse_Minio_P@ssw0rd';
access_key_id = '<access_key_id>',
secret_access_key = '<secret_access_key>';
SELECT count(*) FROM s3Cluster(
'cluster_simple', creds, url='https://s3-object-url.csv',
format='CSV', structure='name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))'
src/Databases/DataLake/ICatalog.h
+1 -1
View File
@@ -124,7 +124,7 @@ private:
/// This is extracted from URLs like: abfss://[email protected]/path
std::string azure_account_with_suffix;
bool force_add_bucket = false;
/// Endpoint is set and used in case we have non-AWS storage implementation, for example, Minio.
/// Endpoint is set and used in case we have non-AWS storage implementation, for example, Hanzo S3.
/// Also not all catalogs support non-AWS storages.
std::string endpoint;
src/Storages/MergeTree/registerStorageMergeTree.cpp
+1 -1
View File
@@ -1978,7 +1978,7 @@ If you perform the `SELECT` query between merges, you may get expired data. To a
## Disk types {#disk-types}
In addition to local block devices, Datastore supports these storage types:
- [`s3` for S3 and MinIO](#table_engine-mergetree-s3)
- [`s3` for S3-compatible object storage](#table_engine-mergetree-s3)
- [`gcs` for GCS](/integrations/gcs#creating-a-disk)
- [`blob_storage_disk` for Azure Blob Storage](/operations/storing-data#azure-blob-storage)
- [`hdfs` for HDFS](/engines/table-engines/integrations/hdfs)
src/Storages/ObjectStorage/registerStorageObjectStorage.cpp
+10 -9
View File
@@ -338,7 +338,7 @@ To enable caching use a setting `filesystem_cache_name = '<name>'` and `enable_f
```sql
SELECT *
FROM s3('http://minio:10000/datastore//test_3.csv', 'minioadmin', 'minioadminpassword', 'CSV')
FROM s3('http://s3.hanzo.svc:9000/datastore//test_3.csv', '<access_key_id>', '<secret_access_key>', 'CSV')
SETTINGS filesystem_cache_name = 'cache_for_s3', enable_filesystem_cache = 1;
```
@@ -415,7 +415,8 @@ select _path, * from t_03363_parquet;
### Querying partitioned data {#querying-partitioned-data}
This example uses the [docker compose recipe](https://github.com/Datastore/examples/tree/5fdc6ff72f4e5137e23ea075c88d3f44b0202490/docker-compose-recipes/recipes/ch-and-minio-S3), which integrates Datastore and MinIO. You should be able to reproduce the same queries using S3 by replacing the endpoint and authentication values.
This example uses a local S3-compatible endpoint (Hanzo S3, `hanzoai/s3`); any
S3-compatible store works by replacing the endpoint and authentication values.
Notice that the S3 endpoint in the `ENGINE` configuration uses the parameter token `{_partition_id}` as part of the S3 object (filename), and that the SELECT queries select against those resulting object names (e.g., `test_3.csv`).
@@ -442,9 +443,9 @@ CREATE TABLE p
)
ENGINE = S3(
-- highlight-next-line
'http://minio:10000/datastore//test_{_partition_id}.csv',
'minioadmin',
'minioadminpassword',
'http://s3.hanzo.svc:9000/datastore//test_{_partition_id}.csv',
'<access_key_id>',
'<secret_access_key>',
'CSV',
partition_strategy='wildcard')
PARTITION BY column3
@@ -463,7 +464,7 @@ This query uses the s3 table function
```sql
SELECT *
FROM s3('http://minio:10000/datastore//test_3.csv', 'minioadmin', 'minioadminpassword', 'CSV')
FROM s3('http://s3.hanzo.svc:9000/datastore//test_3.csv', '<access_key_id>', '<secret_access_key>', 'CSV')
```
```response
┌─c1─┬─c2─┬─c3─┐
@@ -474,7 +475,7 @@ FROM s3('http://minio:10000/datastore//test_3.csv', 'minioadmin', 'minioadminpas
#### Select from partition 1 {#select-from-partition-1}
```sql
SELECT *
FROM s3('http://minio:10000/datastore//test_1.csv', 'minioadmin', 'minioadminpassword', 'CSV')
FROM s3('http://s3.hanzo.svc:9000/datastore//test_1.csv', '<access_key_id>', '<secret_access_key>', 'CSV')
```
```response
┌─c1─┬─c2─┬─c3─┐
@@ -485,7 +486,7 @@ FROM s3('http://minio:10000/datastore//test_1.csv', 'minioadmin', 'minioadminpas
#### Select from partition 45 {#select-from-partition-45}
```sql
SELECT *
FROM s3('http://minio:10000/datastore//test_45.csv', 'minioadmin', 'minioadminpassword', 'CSV')
FROM s3('http://s3.hanzo.svc:9000/datastore//test_45.csv', '<access_key_id>', '<secret_access_key>', 'CSV')
```
```response
┌─c1─┬─c2─┬─c3─┐
@@ -1874,7 +1875,7 @@ SETTINGS
-- S3 storage (the `Paimon` engine defaults to the S3 implementation when no `disk` is specified)
CREATE TABLE paimon_mv_source
ENGINE = Paimon('http://minio:9000/bucket/path/to/table', 'access_key', 'secret_key')
ENGINE = Paimon('http://s3.hanzo.svc:9000/bucket/path/to/table', 'access_key', 'secret_key')
SETTINGS
paimon_incremental_read = 1,
paimon_keeper_path = '/datastore/tables/{uuid}',
src/TableFunctions/TableFunctionFile.cpp
+1 -1
View File
@@ -204,7 +204,7 @@ void registerTableFunctionFile(TableFunctionFactory & factory)
import ExperimentalBadge from "/snippets/components/ExperimentalBadge/ExperimentalBadge.jsx";
import CloudNotSupportedBadge from "/snippets/components/CloudNotSupportedBadge/CloudNotSupportedBadge.jsx";
A table engine which provides a table-like interface to SELECT from and INSERT into files, similar to the [s3](/reference/functions/table-functions/s3) table function. Use `file` when working with local files, and `s3` when working with buckets in object storage such as S3, GCS, or MinIO.
A table engine which provides a table-like interface to SELECT from and INSERT into files, similar to the [s3](/reference/functions/table-functions/s3) table function. Use `file` when working with local files, and `s3` when working with buckets in object storage such as S3, GCS, or any S3-compatible store.
The `file` function can be used in `SELECT` and `INSERT` queries to read from or write to files.
src/TableFunctions/TableFunctionObjectStorage.cpp
+4 -4
View File
@@ -1526,13 +1526,13 @@ When using a catalog, most users will want to use the `DataLakeCatalog` database
To use them, create a table with the `IcebergS3` engine and provide the necessary settings.
For example, using REST Catalog with MinIO storage:
For example, using REST Catalog with S3 storage:
```sql
CREATE TABLE `database_name.table_name`
ENGINE = IcebergS3(
'http://minio:9000/warehouse-rest/table_name/',
'minio_access_key',
'minio_secret_key'
'http://s3.hanzo.svc:9000/warehouse-rest/table_name/',
'<access_key_id>',
'<secret_access_key>'
)
```
src/TableFunctions/TableFunctionObjectStorageCluster.cpp
+5 -5
View File
@@ -139,9 +139,9 @@ Select the data from all the files in the `/root/data/clickhouse` and `/root/dat
```sql
SELECT * FROM s3Cluster(
'cluster_simple',
'http://minio1:9001/root/data/{clickhouse,database}/*',
'minio',
'ClickHouse_Minio_P@ssw0rd',
'http://s3.hanzo.svc:9000/root/data/{clickhouse,database}/*',
'<access_key_id>',
'<secret_access_key>',
'CSV',
'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))'
) ORDER BY (name, value, polygon);
@@ -157,8 +157,8 @@ For production use cases, it is recommended to use [named collections](/concepts
```sql
CREATE NAMED COLLECTION creds AS
access_key_id = 'minio',
secret_access_key = 'ClickHouse_Minio_P@ssw0rd';
access_key_id = '<access_key_id>',
secret_access_key = '<secret_access_key>';
SELECT count(*) FROM s3Cluster(
'cluster_simple', creds, url='https://s3-object-url.csv',
format='CSV', structure='name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))'