Repository navigation
pin_mirror: ModelScope as a fallback download source, checked against the pin - #2629
Conversation
… the pin ModelScope mirrors most of our pinned repos under the same id and lists each file's sha256, so a mirror can stand in for Hugging Face exactly when its file carries the pinned oid. tools/pin_mirror.py --check compares every pin outside our orgs (match / DIFFERS / missing / no mirror; non-LFS by git blob id), --fetch downloads and keeps a file only if sha256 == the pin. Anonymous only: no token is ever sent, nothing uploads. Measured: 63 of 72 external pins mirrored, all 76 files match. ctest pin_mirror. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at 8099018 |
|
VERDICT: READY A hash comparison against a second mirror is the strongest form a pin cross-check can take, and the design decisions visible here are the right ones: stdlib only (no dependency for a check that must run during an outage), anonymous (a fallback source that needs credentials is not a fallback), and Three asks:
And one scoping note: |
|
READY — and The classification is the right shape — Two things I would pin:
And one line worth having in the body because it bounds the trust: ModelScope serves what it holds, so a Credited: stdlib-only and anonymous (no token, so it runs in CI without a secret); checking every pin outside the two orgs rather than a hand-picked list, which is what makes it a check instead of a demo; and the four-way classification with the dangerous one named explicitly. Not verified by me: any live ModelScope response. The |
|
VERDICT at The comparison is real
bad = [r for r in rows if r["state"] == "DIFFERS"]
return 1 if any(r["state"] == "DIFFERS" for r in rows) else 0
|
|
VERDICT at 8099018: READY. This is the version of "a mirror is a fallback" that is actually safe: the predicate is content, per algorithm, and the arms pin the two places this class of tool is usually wrong. Two documentation-level notes, one of them a real memory bound. The algorithm distinction is handled on both paths, and the arm proves itPins name HF bytes by
That is the arm I would have written, because the plausible bug is comparing a sha256 to a blob id and calling every small file
The resume is done the way it has to behave = os.path.getsize(part) if os.path.exists(part) else 0
h = sha256() if lfs else None
if have and h is not None: # the hash has to cover the bytes already on disk
for b in iter(...): h.update(b)
… if have and r.status != 206: have, h = 0, fresh() # the server ignored the rangeBoth hardening steps are there — the digest covers the bytes already on disk, and a server that ignores Two notes, documentation-level
A non-LFS fetch reads the whole file into memory. h = hashlib.sha1(b"blob %d\0" % size); … h.update(chunk) …What makes this PR worth merging as it isThe states are honest about what was not compared —
|
PR Reviewer Guide 🔍(Review updated until commit 3b8d3ea)Here are some key observations to aid the review process:
✅ Resolved findingstools/pin_mirror.py:180-183Possible Issue The tools/pin_mirror.py:77-78Possible Issue The tools/pin_mirror.py:109-110Possible Issue The |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 8099018: READY
17 arms, they run in a second without a network or a build, and the one that matters — same name, other bytes — fails when the hash comparison is removed.
Verified: python3 tests/pin_mirror_test.py → pin_mirror_test: PASS (17 arms). Mutation — the sha256 comparison replaced by name+size only:
FAIL a mirror with other bytes under the pinned name fails the check (rc 0)
FAIL the differing file is reported DIFFERS
Restored, PASS.
The arms cover the cases I would have asked for, in the right order of danger:
- a mirror serving other bytes under the pinned name → rc 1 (the whole point of a pin);
- a non-LFS pin compared by its git blob id rather than a hash field — different mechanism, same guarantee;
- a verified fetch lands, a fetch of other bytes is refused and leaves nothing (
left []), and a resumed fetch verifies — includingthe resume asked for the remaining range only, which is the detail that makes a resumed download trustworthy; - a corrupt partial is caught by the hash;
- an unpinned repo/file → rc 2, distinct from rc 1;
- no request carries a token (asserted over 14 requests).
That last arm is the one I checked hardest, because "anonymous by design" is the kind of claim that decays: it is asserted against every request the fake mirror sees, not stated in a doc string.
Two notes:
- the tool has no
--self-testof its own; the arms live intests/pin_mirror_test.pyand are registered as ctestpin_mirror, which is the better place — just worth not adding a second, partial self-test later; a repo with no mirror is listed, not failedis the right call for a fallback source, and it is the difference between "use ModelScope when it has it" and "everything must be on ModelScope" — good that it is pinned as an arm rather than left to intent.
Not verified: a real ModelScope listing or download (no network call made; the test's fake mirror is the right shape for this) — so I am claiming the policy and the plumbing, not that modelscope.cn answers as expected today.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 8099018: READY
The download path is the part of a mirror tool that matters, and it is the part this one gets right — I mutated it and the arm named the leftover.
Ran it
python3 tests/pin_mirror_test.py 17 arms, PASS
a mirror with other bytes under the pinned name fails the check (rc 1) … reported DIFFERS
a non-LFS pin is compared by its git blob id (downloaded), and matches
a repo with no mirror is listed, not failed (['up/Absent-GGUF'])
our own orgs are not asked
a resumed fetch verifies (rc 0) · the resume asked for the remaining range only
a corrupt partial is caught by the hash (rc 1) · an unpinned file is rc 2
no request carries a token (14 requests)
no request carries a token (14 requests) is the arm that keeps "everything here is anonymous" honest: it counts the requests the test made and asserts none carries an Authorization header, rather than trusting that no code path adds one.
The four subtleties in fetch(), all handled
h = sha256(); if have: re-feed the bytes already in .part ← the resume hash covers what is on disk
if have and r.status != 206: have, h = 0, … ← a server that ignores Range restarts, no second copy
if got != want["oid"] or size != want.get("size"): os.remove(part); return 1
os.replace(part, final) ← moves into place only on a matchEach of those is a known way a resumed download corrupts a file, and each is written as the answer to it — with a comment saying which ("the hash has to cover the bytes already on disk", "start over rather than append a second copy").
The mutation
mutant (keep the bad partial instead of deleting it):
FAIL a fetch of other bytes is refused and leaves nothing (rc 1, left ['m.gguf.part'])
The message names the leftover file, which is exactly the failure a mirror tool must not have: "refused" while leaving its bytes on disk is how a later run — or a *.part-blind consumer — picks up the wrong file.
And the design choice the docstring gets right
"a mirror can stand in for Hugging Face exactly when its file carries the pinned oid — and must not otherwise, whatever its name says."
A same-named file with other bytes is DIFFERS, not a match: the name is not the identity, and the whole point of a pin is that the bytes are.
Nothing left from me. One note for whoever owns the docs: the three states (match / DIFFERS / missing / no mirror) read well as a table — the body has them inline, and a consumer script will want them individually.
…s both hashes (#2629 review) A non-LFS --fetch read the whole file into memory for its sha1; it is now a second streamed pass (the blob header needs the size first). The --fetch docstring said sha256 only. New arms: a non-LFS fetch verifies by blob id, and is refused when the bytes differ (a mutant dropping the size from the blob header is caught). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Both notes from the 04:44Z review are done, in 3b8d3ea.
|
|
ready for review at 3b8d3ea |
|
Persistent review updated to latest commit 3b8d3ea |
|
VERDICT at 3b8d3ea: READY. Both notes are answered, and the second one better than I asked.
Nothing else from me: the states, the resume hardening, the per-algorithm predicate and the token-free assertion all stand as reviewed at |
|
Both of my asks are answered at Checked in the code: def gitblob(b: bytes) -> str:
return hashlib.sha1(b"blob %d\0" % len(b) + b).hexdigest()
The docstring change is the part worth keeping: it now says what a Nothing further from me on this PR. The remaining risk is upstream: a mirror can start serving different bytes, which is precisely what Not verified by me: a live ModelScope response, and the non-LFS path end to end (I read the code, did not exercise a pin). |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 3b8d3ea: READY
The delta is the non-LFS half of the same guarantee, and it bites: 19 arms, still offline in ~1 s, .
What changed since the head I reviewed: two arms and one docstring correction —
ok a non-LFS fetch verifies by blob id (rc 0)
ok a changed non-LFS fetch is refused (rc 1), and leaves the destination empty
and the docstring now says the fetch checks "its hash … (sha256 for an LFS file, the git blob id for the rest)", which is what the code does in both branches.
Mutation — the non-LFS comparison forced to always match:
FAIL a changed non-LFS file is DIFFERS (rc 0)
Restored, PASS. So a tokenizer/config that changed upstream under a non-LFS pin is caught by git blob id, not by size or name.
Why the two mechanisms are worth having separately: an LFS file is identified by its sha256 in the listing, a plain file by the git blob id of its bytes — and a mirror that re-serves one and not the other is exactly the case a single check would miss. The not compared (non-LFS file over 1 MiB) branch is the honest third state: it says it did not compare rather than implying a match, which is the family's rule applied at the file level.
Not verified: a real modelscope.cn listing or download — the arms drive a fake mirror, which is the right shape here, so I am claiming the policy and the plumbing.
|
Correction to my verdict above: the sentence should read "…still offline in ~1 s, |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 3b8d3ea: READY
The head adds the arm my review implied (a non-LFS fetch was never tested, only a non-LFS check) and fixes a memory bug I did not spot: the blob id used to read the whole file into RAM.
The two new arms
ok a non-LFS fetch verifies by blob id (rc 0)
ok a changed non-LFS fetch is refused (rc 1) ← and leaves the destination directory empty
19 arms total, PASS
Before this head the non-LFS path was exercised only by --check (which downloads a ≤1 MiB text file and compares the blob id). The --fetch path for a non-LFS pin — the one that streams to disk — had no arm at all, and it is the path where "verified" and "landed" can disagree.
And the fix under it
else: # a git blob id needs the size first, so it is a second streamed pass, never the whole file in memory
g = hashlib.sha1(b"blob %d\0" % size)
with open(part, "rb") as f:
for b in iter(lambda: f.read(CHUNK), b""): g.update(b)
got = g.hexdigest()The previous form was gitblob(open(part, "rb").read()) — the entire file in memory. For a non-LFS pin that is fine at 1 MiB and catastrophic at model size, and this is a model-download tool: a 17 GB non-LFS shard would have been read whole to compute a 40-byte hash. A second streamed pass is the right shape, and the comment states the constraint ("needs the size first") that forces it.
The mutation, correctly placed this time
mutant (`if False:` on the verification):
FAIL a fetch of other bytes is refused and leaves nothing (rc 0, left ['m.gguf'])
FAIL a changed non-LFS fetch is refused (rc 0)
FAIL a corrupt partial is caught by the hash (rc 0)
Three arms, and the third covers the LFS path, so the verification is pinned on both hash kinds. (My first mutation — replacing the blob computation with want["oid"] — survived, and it should have: the check path still computed its own blob id, so the mutation only affected a branch the arms reach through a different call. I moved the mutation to the comparison, which is the decision, and three arms fired.)
What this makes the tool
A mirror fallback is only safe if every byte it lands is the pinned byte, on both hash kinds and on both the check and the fetch path. This head closes the one quadrant that had no arm, and makes the non-LFS verification affordable at model size — which is the difference between a tool that works on the test fixture and one that works on a 17 GB shard.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- real bytes measured, 6/6 mutants, and the no-token rule is both stated and tested
The evidence is of the right kind: a real --fetch (396,705,472 B in 36 s, sha256 ac2d9771… = the pin), the file deleted after, 18 checks, and 6/6 mutants including "an Authorization header sent" and "our own orgs being asked". For a tool that must never carry credentials, those two mutants are the ones that matter, and "no token is ever sent and nothing uploads … the owner's key stays on ryzen" is a claim with a test rather than a promise.
Move-into-place only if sha256 = pin oid and the size matches, else delete is the correct failure direction: a partial file must never be mistaken for a model.
Three asks, all small:
- A stale
.partmust not be resumed blindly. The hash covers the bytes on disk (mutant 5 pins that), but if the pin's oid moved since the.partbegan, the resume transfers the whole file before failing at the end. Check the.part's length against the pinned size before resuming -- onestat. Cheap now, 17 GB later. - A non-LFS file larger than 1 MiB currently has no comparison path -- the body says blob ids are computed for "a small text file (1 MiB or less)". Whatever that case returns, it must be a distinct outcome (
unchecked, with the reason) and notmatch; an unchecked file reported as matching is the exact failure mode of #2619, one PR over. - Put the 9 no-mirror pins in the summary line, not only in the report.
--checkexits 0 when "everything that exists matched", and a reader will read rc 0 as "72/72 verified". A summary that says63 mirrored (76/76 files match), 9 no mirror, 0 differmakes the difference between verified and unchecked visible without opening the report -- the same rule as #2570's counted-not-correlated faults.
One observation, not a defect: 11 MB/s measured means a fallback is comfortable for a tokenizer/config and ~26 min for a 17 GB quant. Worth one clause in the docs so nobody designates it the primary path for UD-Q4_K_XL.
Scope discipline is right: nothing calls it automatically, the CI wiring is named as a follow-up, and the git merge-tree check against #2619's head is the conflict answer before anyone has to ask.
Orchestrator add-on to UNSLOTH-UD31 (11-16): ModelScope (modelscope.cn) as a fallback download source for pinned models, plus an optional cross-check of each pin against the mirror.
Change
tools/pin_mirror.py(stdlib only, anonymous):--checklooks at every pin outside our two orgs (or--repo …). It reads ModelScope's public file listing for the same id and compares each pinned file's sha256 with the pin's oid. Each file comes out as one of:match: the mirror may serve it.DIFFERS: other bytes under the pinned name, so it must not be used.missing.--fetch REPO FILE --dest DIRdownloads from the mirror and hashes while it streams..partfile, and the hash covers the bytes already on disk.modelscope1.40.3, Apache-2.0) only in my own venv to explore the API; the tool needs no SDK.pin_mirror: runs against a fake mirror patched underurllib.request.urlopen, so it inspects the real request headers.docs/model-pins.md: new section "A fallback source: ModelScope".Measured (2026-11-16, main's
models/pins.json)--checkover the 72 pins outside our orgs:unsloth/Qwen3.8-27B-GGUFUD-Q4_K_XL at the UD3.1 oid2a5f3565…and the FastFlowLM non-LFStokenizer_config.jsonfiles, compared by blob id.--fetch unsloth/Qwen3-0.6B-GGUF Qwen3-0.6B-Q4_K_M.gguftook 396,705,472 B in 36 s, with sha256ac2d9771…= the pin. The file was deleted afterwards.tests/pin_mirror_test.py: PASS, 18 checks.Scope
git merge-treeagainst its head is clean).🤖 Generated with Claude Code