mirror of
https://github.com/hanzoai/datastore.git
synced 2026-09-26 13:55:07 +00:00
Alexey MilovidovandClaude Fable 5
64eed702a5
S3 sporadically returns transient errors (e.g. `503 Service Unavailable`) for the `clickhouse-datasets` bucket, and `wget` retries only network-level errors, not HTTP errors, so a single 503 failed the whole `Performance Comparison` job at the `Download datasets` step. The step result also carried no `info`, so the CI report showed a bare `ERROR` and the cause was only findable in `job.log`. Move the download into a shared helper that retries each `wget | tar` pipeline with backoff (the pipe cannot be resumed mid-stream, and extraction into the target directory is idempotent, so rerun the whole pipeline) and returns per-dataset error descriptions, which now go into `Result.info`. Use it in both the performance comparison job and the profile collection job. Also drop the `-c` (continue) wget flag, which is a no-op with `-O-`. CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=107415&sha=c0ba65e6aa6a1785e890e54d4ea742802c1da7b9&name_0=PR&name_1=Performance%20Comparison%20%28amd_release%2C%20master_head%2C%202%2F6%29 PR: https://github.com/ClickHouse/ClickHouse/pull/107415 Co-Authored-By: Claude Fable 5 <[email protected]>
41 lines
1.7 KiB
Python
41 lines
1.7 KiB
Python
import time
|
|
from concurrent.futures import ThreadPoolExecutor
|
|
|
|
from ci.praktika.utils import Shell
|
|
|
|
|
|
def download_and_extract_datasets(dataset_urls, target_dir, retries=5):
|
|
"""Download dataset tarballs in parallel and extract them into target_dir.
|
|
|
|
Returns a list of error descriptions, one per dataset that could not be
|
|
downloaded and extracted; an empty list means success. The caller can put
|
|
the errors into Result.info so the failure is visible in the CI report
|
|
without digging through the job log.
|
|
|
|
S3 sporadically returns transient errors (e.g. 503 Service Unavailable),
|
|
and wget retries only network-level errors, not HTTP errors, so a single
|
|
503 would otherwise fail the whole job. A failed attempt cannot be resumed
|
|
mid-stream because the body is piped into tar, so rerun the whole pipeline
|
|
on any failure - extraction into target_dir is idempotent.
|
|
"""
|
|
|
|
def download(url):
|
|
error = ""
|
|
for attempt in range(1, retries + 1):
|
|
exit_code, _, error = Shell.get_res_stdout_stderr(
|
|
f'wget -nv -nd "{url}" -O- | tar --extract -C {target_dir}',
|
|
verbose=True,
|
|
)
|
|
if exit_code == 0:
|
|
return ""
|
|
print(
|
|
f"WARNING: Attempt [{attempt}/{retries}] to download [{url}] failed, err: [{error}]"
|
|
)
|
|
if attempt < retries:
|
|
time.sleep(attempt * 30)
|
|
return f"Failed to download [{url}] after [{retries}] attempts, last error: [{error}]"
|
|
|
|
print(f"Download datasets in parallel: [{len(dataset_urls)}]")
|
|
with ThreadPoolExecutor(max_workers=len(dataset_urls)) as executor:
|
|
return [error for error in executor.map(download, dataset_urls) if error]
|