mirror of
https://github.com/hanzoai/datastore.git
synced 2026-09-26 13:55:07 +00:00
Max KainovandClaude Opus 4.8
7b03b477c3
CI runners pull Docker Hub images through the `dockerhub-proxy` cache (`registry:2` + nginx backed by S3). That proxy only fronts Docker Hub, so images hosted on other registries (`mcr.microsoft.com`, `ghcr.io`) bypass it and are pulled directly, exposing CI to those registries' anonymous rate limits (e.g. `mcr.microsoft.com` returns HTTP 429 `toomanyrequests` under load). Re-host the integration-test images that were on foreign registries under the `clickhouse/` Docker Hub namespace so the pulls route back through the proxy: - `mcr.microsoft.com/azure-storage/azurite` -> `clickhouse/azure-storage-azurite` - `ghcr.io/projectnessie/nessie` -> `clickhouse/projectnessie-nessie` - `ghcr.io/ytsaurus/local` -> `clickhouse/ytsaurus-local` - `ghcr.io/letsencrypt/pebble` -> `clickhouse/letsencrypt-pebble` - `ghcr.io/letsencrypt/pebble-challtestsrv` -> `clickhouse/letsencrypt-pebble-challtestsrv` `tests/integration/compose/mirror-images.sh` copies these multi-arch manifests into the `clickhouse/` namespace and documents how to add or bump images. Add a `compose_images_from_dockerhub` sub-check to the style check job that scans `tests/integration/compose/*.yml` and fails if any `image:` references an explicit non-Docker-Hub registry host, to prevent reintroducing off-proxy images. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
34 lines
1.9 KiB
Bash
Executable File
34 lines
1.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Mirror third-party integration-test images into the clickhouse/ Docker Hub namespace.
|
|
#
|
|
# Why: CI runners pull Docker Hub images through the dockerhub-proxy cache
|
|
# (registry:2 + nginx, backed by S3 — see tests/ci/terraform/dockerhub-proxy.md).
|
|
# That proxy only fronts Docker Hub. Images hosted on other registries
|
|
# (mcr.microsoft.com, ghcr.io) bypass the proxy and are pulled directly, so CI is
|
|
# exposed to those registries' anonymous rate limits (e.g. mcr.microsoft.com
|
|
# returns HTTP 429 "toomanyrequests" under load). Re-hosting them under
|
|
# clickhouse/ routes the pulls back through the proxy + S3 cache.
|
|
#
|
|
# Usage: log in to Docker Hub with an account that can push to the clickhouse org,
|
|
# then run this script. It is idempotent — re-run it to add images or bump versions.
|
|
# Keep the mapping below in sync with the image references in compose/*.yml.
|
|
|
|
set -xeuo pipefail
|
|
|
|
# upstream-reference clickhouse-reference (slash flattened to dash; Docker Hub repos are org/name only)
|
|
IMAGES=(
|
|
"mcr.microsoft.com/azure-storage/azurite:3.35.0 clickhouse/azure-storage-azurite:3.35.0"
|
|
"ghcr.io/projectnessie/nessie:0.107.2 clickhouse/projectnessie-nessie:0.107.2"
|
|
"ghcr.io/ytsaurus/local:stable-24.2 clickhouse/ytsaurus-local:stable-24.2"
|
|
"ghcr.io/letsencrypt/pebble:2.9.0 clickhouse/letsencrypt-pebble:2.9.0"
|
|
"ghcr.io/letsencrypt/pebble-challtestsrv:2.9.0 clickhouse/letsencrypt-pebble-challtestsrv:2.9.0"
|
|
)
|
|
|
|
for entry in "${IMAGES[@]}"; do
|
|
read -r src dst <<< "$entry"
|
|
# Copy the full manifest list (all architectures) registry-to-registry. This
|
|
# preserves multi-arch so both amd64 and arm64 runners are served; a plain
|
|
# `docker pull && docker push` would only re-push the host's single arch.
|
|
docker buildx imagetools create --tag "$dst" "$src"
|
|
done
|