generate: each language downloads the jar into its own file

`main` maps the languages over a thread pool, so a cold cache calls `jar()`
once per language at the same moment. Every one of them opened the SAME
`<jar>.part`, wrote the same 30 MB into it, and then raced on the rename: the
first `os.replace` moved the file away and the second died with
FileNotFoundError on a path it had just finished writing.

Measured on `python3 generate.py java kotlin` with an empty cache — java
emitted 2274 files and kotlin never ran at all. The traceback names
`os.replace`, so it reads like a corrupt cache rather than two writers.

It needs an empty cache to happen, which is why it is not a known annoyance:
every re-run on a warm cache skips the download entirely, and the failure lands
on CI and on somebody's first clone, where it costs the most and where the
person hitting it has the least context.

`tempfile.mkstemp` in the cache directory gives each caller its own name;
`os.replace` is atomic on POSIX and on Windows, so the losers overwrite an
identical file with an identical file. The partial file is removed if the
download raises, so a failed fetch cannot leave a 30 MB fragment behind that a
later run has to reason about. Verified with eight concurrent callers against a
local URL: no errors, bytes intact, no `.part` left in the cache.

Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
zooqueen
2026-08-04 21:57:18 -07:00
co-authored by Hanzo Dev
parent 370b12c4a4
commit f588e2147c
+24 -4
View File
@@ -75,15 +75,35 @@ MAVEN = "https://repo1.maven.org/maven2/org/openapitools/openapi-generator-cli"
def jar(version):
"""The pinned generator, fetched once. A version is a version everywhere."""
"""The pinned generator, fetched once. A version is a version everywhere.
Every language downloads into its OWN temp file, because `main` maps the
languages over a thread pool and a cold cache therefore calls this once per
language at the same moment. A shared `<jar>.part` made those writes one
file: both threads opened it, both wrote the same 30 MB into it, the first
to finish renamed it away, and the second's `os.replace` died with
FileNotFoundError on a path it had just written. Measured on `generate.py
java kotlin` — java emitted 2274 files and kotlin never ran. The failure
needs an empty cache, so it misses every re-run and lands on CI and on the
first clone, which is where it costs the most.
`os.replace` is atomic on POSIX and on Windows, so the losers of the race
simply overwrite an identical file with an identical file.
"""
path = os.path.join(CACHE, f"openapi-generator-cli-{version}.jar")
if not os.path.exists(path):
os.makedirs(CACHE, exist_ok=True)
url = f"{MAVEN}/{version}/openapi-generator-cli-{version}.jar"
print(f"fetching {url}", flush=True)
with urllib.request.urlopen(url) as r, open(path + ".part", "wb") as f:
shutil.copyfileobj(r, f)
os.replace(path + ".part", path)
fd, part = tempfile.mkstemp(dir=CACHE, prefix=f"{version}.", suffix=".part")
try:
with urllib.request.urlopen(url) as r, os.fdopen(fd, "wb") as f:
shutil.copyfileobj(r, f)
os.replace(part, path)
except BaseException:
if os.path.exists(part):
os.unlink(part)
raise
return path