generate: each language downloads the jar into its own file
`main` maps the languages over a thread pool, so a cold cache calls `jar()` once per language at the same moment. Every one of them opened the SAME `<jar>.part`, wrote the same 30 MB into it, and then raced on the rename: the first `os.replace` moved the file away and the second died with FileNotFoundError on a path it had just finished writing. Measured on `python3 generate.py java kotlin` with an empty cache — java emitted 2274 files and kotlin never ran at all. The traceback names `os.replace`, so it reads like a corrupt cache rather than two writers. It needs an empty cache to happen, which is why it is not a known annoyance: every re-run on a warm cache skips the download entirely, and the failure lands on CI and on somebody's first clone, where it costs the most and where the person hitting it has the least context. `tempfile.mkstemp` in the cache directory gives each caller its own name; `os.replace` is atomic on POSIX and on Windows, so the losers overwrite an identical file with an identical file. The partial file is removed if the download raises, so a failed fetch cannot leave a 30 MB fragment behind that a later run has to reason about. Verified with eight concurrent callers against a local URL: no errors, bytes intact, no `.part` left in the cache. Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
+24
-4
@@ -75,15 +75,35 @@ MAVEN = "https://repo1.maven.org/maven2/org/openapitools/openapi-generator-cli"
|
||||
|
||||
|
||||
def jar(version):
|
||||
"""The pinned generator, fetched once. A version is a version everywhere."""
|
||||
"""The pinned generator, fetched once. A version is a version everywhere.
|
||||
|
||||
Every language downloads into its OWN temp file, because `main` maps the
|
||||
languages over a thread pool and a cold cache therefore calls this once per
|
||||
language at the same moment. A shared `<jar>.part` made those writes one
|
||||
file: both threads opened it, both wrote the same 30 MB into it, the first
|
||||
to finish renamed it away, and the second's `os.replace` died with
|
||||
FileNotFoundError on a path it had just written. Measured on `generate.py
|
||||
java kotlin` — java emitted 2274 files and kotlin never ran. The failure
|
||||
needs an empty cache, so it misses every re-run and lands on CI and on the
|
||||
first clone, which is where it costs the most.
|
||||
|
||||
`os.replace` is atomic on POSIX and on Windows, so the losers of the race
|
||||
simply overwrite an identical file with an identical file.
|
||||
"""
|
||||
path = os.path.join(CACHE, f"openapi-generator-cli-{version}.jar")
|
||||
if not os.path.exists(path):
|
||||
os.makedirs(CACHE, exist_ok=True)
|
||||
url = f"{MAVEN}/{version}/openapi-generator-cli-{version}.jar"
|
||||
print(f"fetching {url}", flush=True)
|
||||
with urllib.request.urlopen(url) as r, open(path + ".part", "wb") as f:
|
||||
shutil.copyfileobj(r, f)
|
||||
os.replace(path + ".part", path)
|
||||
fd, part = tempfile.mkstemp(dir=CACHE, prefix=f"{version}.", suffix=".part")
|
||||
try:
|
||||
with urllib.request.urlopen(url) as r, os.fdopen(fd, "wb") as f:
|
||||
shutil.copyfileobj(r, f)
|
||||
os.replace(part, path)
|
||||
except BaseException:
|
||||
if os.path.exists(part):
|
||||
os.unlink(part)
|
||||
raise
|
||||
return path
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user