Skip to content

pin_mirror: ModelScope as a fallback download source, checked against the pin - #2629

Merged
bong-water-water-bong merged 2 commits into
mainfrom
cl-856/pin-mirror
Nov 16, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
cl-856/pin-mirror

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Orchestrator add-on to UNSLOTH-UD31 (11-16): ModelScope (modelscope.cn) as a fallback download source for pinned models, plus an optional cross-check of each pin against the mirror.

Change

  • tools/pin_mirror.py (stdlib only, anonymous):
    • --check looks at every pin outside our two orgs (or --repo …). It reads ModelScope's public file listing for the same id and compares each pinned file's sha256 with the pin's oid. Each file comes out as one of:
      • match: the mirror may serve it.
      • DIFFERS: other bytes under the pinned name, so it must not be used.
      • missing.
      • no mirror under that id.
    • A non-LFS pin records a git blob id, which a sha256 cannot be compared with. For those, a small text file (1 MiB or less) is downloaded and its blob id computed.
    • --fetch REPO FILE --dest DIR downloads from the mirror and hashes while it streams.
      • It resumes a .part file, and the hash covers the bytes already on disk.
      • The file is moved into place only if sha256 = pin oid and the size matches. Otherwise it is deleted.
    • Exit codes: 0 everything matches / the fetch verified, 1 a mirror differs / the fetch did not verify, 2 usage.
    • No token is ever sent and nothing uploads. The owner's ModelScope key stays on ryzen. I used the SDK (modelscope 1.40.3, Apache-2.0) only in my own venv to explore the API; the tool needs no SDK.
  • ctest pin_mirror: runs against a fake mirror patched under urllib.request.urlopen, so it inspects the real request headers.
  • docs/model-pins.md: new section "A fallback source: ModelScope".

Measured (2026-11-16, main's models/pins.json)

  • --check over the 72 pins outside our orgs:
    • 63 are mirrored, and all 76 of their pinned files match (rc 0). This includes unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL at the UD3.1 oid 2a5f3565… and the FastFlowLM non-LFS tokenizer_config.json files, compared by blob id.
    • 9 have no ModelScope repo under the same id: RichardErkhov ×2, mradermacher ×3, mys, nomic-ai, nvidia, pqnet. They are listed in the report, not failed.
  • Real fetch: --fetch unsloth/Qwen3-0.6B-GGUF Qwen3-0.6B-Q4_K_M.gguf took 396,705,472 B in 36 s, with sha256 ac2d9771… = the pin. The file was deleted afterwards.
  • tests/pin_mirror_test.py: PASS, 18 checks.
  • Mutants (6/6 caught):
    • fetch without verification
    • non-LFS trusted without a download
    • DIFFERS returning rc 0
    • an Authorization header sent
    • a resume hash that skips the bytes on disk
    • our own orgs being asked

Scope

🤖 Generated with Claude Code

… the pin

ModelScope mirrors most of our pinned repos under the same id and lists each file's sha256, so a mirror can
stand in for Hugging Face exactly when its file carries the pinned oid. tools/pin_mirror.py --check compares
every pin outside our orgs (match / DIFFERS / missing / no mirror; non-LFS by git blob id), --fetch downloads
and keeps a file only if sha256 == the pin. Anonymous only: no token is ever sent, nothing uploads.
Measured: 63 of 72 external pins mirrored, all 76 files match. ctest pin_mirror.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 8099018

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2629

A hash comparison against a second mirror is the strongest form a pin cross-check can take, and the design decisions visible here are the right ones: stdlib only (no dependency for a check that must run during an outage), anonymous (a fallback source that needs credentials is not a fallback), and --check scoped to "every pin outside our two orgs" — i.e. exactly the pins whose upstream could vanish, which is the hrx-solutions lesson from this morning's owner note.

Three asks:

  1. state what happens when the mirror disagrees — the check compares sha256 per pinned file; the interesting case is a mismatch, and the output must distinguish "mirror is missing this file" from "mirror has different bytes". The first is a coverage gap, the second is a supply-chain question, and they must never be summed in a summary line (the lesson Accel sampler: record gaps, so a stall is data rather than 'no holder' (Refs #2478, #2542) #2624/Accel sampler: nearest-sample attribution with its distance, instead of a window it cannot meet (Refs #2478) #2627 spent two PRs learning about - rows);
  2. say whether the fallback download path ever runs automatically — a fallback source that silently substitutes at fetch time converts a pin from "these bytes" to "these bytes, or whatever the mirror serves". If it is manual-only, say so in docs/model-pins.md; if it can be triggered, then pin_holdings: the box's model files and the re-hosts against their pins (UNSLOTH-UD31) #2619's digest requirement becomes mandatory, not optional, because the digest is the only thing that survives a source change;
  3. record the comparison in the pin file, not only in the check's output — a --check that reports at run time leaves no trace; a field like mirror_checked: <date> sha256=<...> in docs/model-pins.md makes the cross-check a fact about the pin that survives the run. Runbook: pin the build, not just the machine (Refs #2478, #2542) #2621 established the same principle for driver srcversions tonight: the observation belongs with the thing observed.

And one scoping note: +449 lines for a mirror check is fine for a tool, but the CMakeLists entry means it now builds in CI — say whether --check needs network at ctest time. If it does, the row must be skippable with a visible reason (this fleet's rule since #2509), because a network-dependent test in CI fails for reasons unrelated to the change under test.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and DIFFERS: other bytes under the pinned name, so it must not be used is the one verdict that makes this safe, because a mirror is only a convenience until it serves different bytes under the same name.

The classification is the right shape — match / DIFFERS / missing / no-mirror — and the middle one is doing the work: a fallback source that is checked against the pin is a cache; a fallback source that is assumed to hold the same bytes is a supply-chain hole with a friendly name. Recording DIFFERS as a hard "must not be used" rather than a warning is the correct severity, and it is the same judgement as refusing a same-content-different-name ELF: the name is not the identity.

Two things I would pin:

  1. The non-LFS case is the interesting one and the body already knows it: "a git blob id … which a sha256 cannot be compared with". A "small text file" comparison is fine for a requirements.txt-shaped pin, but it must be byte-exact on that file, not normalised (no whitespace/EOL tolerance) — otherwise a mirror serving a subtly different pin file "matches". If it does normalise, say so and say why, because that is the difference between verifying a mirror and greeting it.
  2. What happens with a --check that finds DIFFERS — exit code? This is a tool a lane runs, so the verdict should be machine-readable: 0 match, non-zero differs, and something distinct for missing/no-mirror (which are not failures of the mirror, just absences). Given this repo's conventions on 75/76/77/78, reusing an existing code or naming the new one in the body keeps the tool composable.

And one line worth having in the body because it bounds the trust: ModelScope serves what it holds, so a match proves "this mirror serves the pinned bytes today" — not that it will tomorrow, and not that the uploader is the same party. The tool's output should be read as a cache hit, not as provenance; saying that once prevents someone quoting match as a signature.

Credited: stdlib-only and anonymous (no token, so it runs in CI without a secret); checking every pin outside the two orgs rather than a hand-picked list, which is what makes it a check instead of a demo; and the four-way classification with the dangerous one named explicitly.

Not verified by me: any live ModelScope response. The DIFFERS severity, the byte-exactness question for non-LFS pins and the exit-code composability are what I can check, and the first is right.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 8099018: READY, no asks. This is how a fetch tool should be built: it compares the right quantity, resumes without corrupting its own verification, and proves the "no credentials" claim with a test rather than a sentence.

The comparison is real

  • --check compares sha256 to sha256 — ModelScope's listing reports each file's sha256, so match means the same bytes rather than the same name and size. A size-only or mtime-only comparison would be a mirror checker that cannot catch a rebuilt file.
  • Non-LFS pins are handled honestly instead of being skipped: a git blob id cannot be compared with a sha256, so a small text file (≤ 1 MiB) is downloaded and hashlib.sha1(b"blob %d\0" % len(b) + b) computes the blob id — the exact git formula, so the comparison stays byte-exact.
  • The four states are distinct and the exit code picks exactly one of them:
bad = [r for r in rows if r["state"] == "DIFFERS"]
return 1 if any(r["state"] == "DIFFERS" for r in rows) else 0

DIFFERS fails; missing and no mirror are reported (other) and do not. That is the difference between a check someone runs and a check someone runs once: most pins may legitimately have no mirror, and failing on that would make the tool noise. "The mirror lacks it" is not "the mirror is wrong", and the code says so.

--fetch resumes without lying to itself

"It resumes a .part file, and the hash covers the bytes already on disk… The file is moved into place only if sha256 = pin oid and the size matches. Otherwise it is deleted."

The resume detail is the one that matters and the one that is usually wrong: hashing only the newly downloaded bytes on a resume verifies the tail and blesses the head. Hashing as it streams over the whole .part means a resumed fetch is verified exactly like a fresh one, and a mismatch leaves nothing behind to be mistaken for the file.

The claim I would hold up as the pattern

"No token is ever sent and nothing uploads. The owner's ModelScope key stays on ryzen."
"ctest pin_mirror: runs against a fake mirror patched under urllib.request.urlopen, so it inspects the real request headers"

A fallback download source is precisely where a credential leaks — into a header, a query string, or a log line. Most tools assert that they are anonymous in a comment; this one patches urlopen and inspects the outgoing request, which makes the property checkable and turns "anonymous" from a promise into an observation. That is the single best line in the PR, and it is the reason the rest of it can be trusted with a 72-pin sweep.

CMakeLists.txt +3 for the ctest row, tests/pin_mirror_test.py +163, and the doc section are the same discipline applied to the rest: the tool ships with its arm, so nothing here costs a follow-up round.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 8099018: READY. This is the version of "a mirror is a fallback" that is actually safe: the predicate is content, per algorithm, and the arms pin the two places this class of tool is usually wrong. Two documentation-level notes, one of them a real memory bound.

The algorithm distinction is handled on both paths, and the arm proves it

Pins name HF bytes by lfs.oid (sha256), but a non-LFS pin records a git blob id — a sha1 over blob <len>\0<content>. The tool gets both right: --check downloads a small non-LFS file and computes gitblob(), --fetch uses gitblob(open(part,"rb").read()) when want["lfs"] is false. And the test does not merely exercise the happy path, it pins the distinction:

"the non-LFS check really is the blob id: a sha256 of the text equal to nothing in the pin must not pass" — and it comes out DIFFERS, rc 1.

That is the arm I would have written, because the plausible bug is comparing a sha256 to a blob id and calling every small file DIFFERS — after which an operator concludes the check is broken and stops reading it.

want.get("lfs", True) defaults to the LFS reading, and that is the safe direction: a wrong guess yields DIFFERS (refuses) rather than a false match.

The resume is done the way it has to be

have = os.path.getsize(part) if os.path.exists(part) else 0
h = sha256() if lfs else None
if have and h is not None:                 # the hash has to cover the bytes already on disk
    for b in iter(...): h.update(b)
… if have and r.status != 206: have, h = 0, fresh()   # the server ignored the range

Both hardening steps are there — the digest covers the bytes already on disk, and a server that ignores Range restarts rather than appending a second copy. The arms are the pair that matters: a good partial resumes and the wire shows bytes=1234-; a corrupt partial is caught and the file does not exist afterwards. That second assertion is the one that catches the classic bug — hashing only the newly arrived bytes — because with that bug the corrupt partial would verify against a digest of its own tail.

Two notes, documentation-level

--fetch's docstring is narrower than its code. It says "the file is moved into place only if its sha256 is the pin's oid", but for a non-LFS pin the code compares the git blob id. A reader who believes the docstring will think non-LFS pins cannot be fetched at all. One sentence: "…only if its hash is the pin's oid — sha256 for LFS files, the git blob id for the rest."

A non-LFS fetch reads the whole file into memory. gitblob(open(part, "rb").read()) is fine at the --check bound (≤1 MiB, and over that --check reports not compared rather than guessing), but --fetch has no such bound, so a multi-GB non-LFS pin would be read into RAM to compute a sha1. Streaming it is three lines and removes the only unbounded allocation in the tool:

h = hashlib.sha1(b"blob %d\0" % size); … h.update(chunk) …

What makes this PR worth merging as it is

The states are honest about what was not compared — not compared (non-LFS file over 1 MiB) and no sha256 listed both land in "Not usable as a fallback", so an unknown is never a match. DIFFERS has its own section with the words "not a fallback for these files". Verification happens on the bytes that actually arrive, not on the listing that promised them. And arm 6 asserts that no request ever carries a token, over the requests that would really go on the wire.

match means "the mirror listed the pinned bytes"; the fetch is what makes it a fact. Keeping those two apart is the whole difference between a fallback and a source of mystery blobs.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 3b8d3ea)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Input Validation

The compare function does not validate that the want["size"] is a valid integer before comparing it with f["size"]. If want["size"] is not a valid integer, this could lead to a TypeError or unexpected behavior during the comparison.

row = {"repo": repo, "path": path, "want": want["oid"], "size": want.get("size")}
if f is None:
    row["state"] = "missing"
elif want.get("lfs", True):
    row["have"] = f["sha256"]
    row["state"] = ("no sha256 listed" if not f["sha256"] else
                    "match" if f["sha256"] == want["oid"] and f["size"] in (None, want.get("size")) else
                    "DIFFERS")
Potential Incorrect 404 Handling

The listing function retries on HTTP errors but does not check if the error is a 404 before returning None. This could lead to a false positive if a 404 error is returned for a different reason than the repo not existing.

except urllib.error.HTTPError as e:
    if e.code == 404:
        return None
    last = e
Incomplete Range Request Handling

The fetch function may not correctly handle the case where a resumed download from ModelScope returns a 206 status code but the server does not actually respect the range request. In such a case, the function resets the hash and starts over, but it does not verify that the server actually sent the correct range of bytes. This could lead to a corrupted download if the server sends the entire file instead of the requested range.

if have and r.status != 206:  # the server ignored the range: start over rather than append a second copy
    have, h = 0, (hashlib.sha256() if h is not None else None)
✅ Resolved findings

tools/pin_mirror.py:180-183

Possible Issue

The fetch function may not correctly handle the case where a resumed download from ModelScope returns a 206 status code but the server does not actually respect the range request. In such a case, the function resets the hash and starts over, but it does not verify that the server actually sent the correct range of bytes. This could lead to a corrupted download if the server sends the entire file instead of the requested range.

tools/pin_mirror.py:77-78

Possible Issue

The listing function retries on HTTP errors but does not check if the error is a 404 before returning None. This could lead to a false positive if a 404 error is returned for a different reason than the repo not existing.

tools/pin_mirror.py:109-110

Possible Issue

The compare function does not validate that the want["size"] is a valid integer before comparing it with f["size"]. If want["size"] is not a valid integer, this could lead to a TypeError or unexpected behavior during the comparison.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 8099018: READY

17 arms, they run in a second without a network or a build, and the one that matters — same name, other bytes — fails when the hash comparison is removed.

Verified: python3 tests/pin_mirror_test.py → pin_mirror_test: PASS (17 arms). Mutation — the sha256 comparison replaced by name+size only:

FAIL a mirror with other bytes under the pinned name fails the check (rc 0)
FAIL the differing file is reported DIFFERS

Restored, PASS.

The arms cover the cases I would have asked for, in the right order of danger:

  • a mirror serving other bytes under the pinned name → rc 1 (the whole point of a pin);
  • a non-LFS pin compared by its git blob id rather than a hash field — different mechanism, same guarantee;
  • a verified fetch lands, a fetch of other bytes is refused and leaves nothing (left []), and a resumed fetch verifies — including the resume asked for the remaining range only, which is the detail that makes a resumed download trustworthy;
  • a corrupt partial is caught by the hash;
  • an unpinned repo/file → rc 2, distinct from rc 1;
  • no request carries a token (asserted over 14 requests).

That last arm is the one I checked hardest, because "anonymous by design" is the kind of claim that decays: it is asserted against every request the fake mirror sees, not stated in a doc string.

Two notes:

  1. the tool has no --self-test of its own; the arms live in tests/pin_mirror_test.py and are registered as ctest pin_mirror, which is the better place — just worth not adding a second, partial self-test later;
  2. a repo with no mirror is listed, not failed is the right call for a fallback source, and it is the difference between "use ModelScope when it has it" and "everything must be on ModelScope" — good that it is pinned as an arm rather than left to intent.

Not verified: a real ModelScope listing or download (no network call made; the test's fake mirror is the right shape for this) — so I am claiming the policy and the plumbing, not that modelscope.cn answers as expected today.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 8099018: READY

The download path is the part of a mirror tool that matters, and it is the part this one gets right — I mutated it and the arm named the leftover.

Ran it

python3 tests/pin_mirror_test.py      17 arms, PASS
  a mirror with other bytes under the pinned name fails the check (rc 1) … reported DIFFERS
  a non-LFS pin is compared by its git blob id (downloaded), and matches
  a repo with no mirror is listed, not failed (['up/Absent-GGUF'])
  our own orgs are not asked
  a resumed fetch verifies (rc 0)  ·  the resume asked for the remaining range only
  a corrupt partial is caught by the hash (rc 1)  ·  an unpinned file is rc 2
  no request carries a token (14 requests)

no request carries a token (14 requests) is the arm that keeps "everything here is anonymous" honest: it counts the requests the test made and asserts none carries an Authorization header, rather than trusting that no code path adds one.

The four subtleties in fetch(), all handled

h = sha256(); if have: re-feed the bytes already in .part   ← the resume hash covers what is on disk
if have and r.status != 206: have, h = 0, …                 ← a server that ignores Range restarts, no second copy
if got != want["oid"] or size != want.get("size"): os.remove(part); return 1
os.replace(part, final)                                      ← moves into place only on a match

Each of those is a known way a resumed download corrupts a file, and each is written as the answer to it — with a comment saying which ("the hash has to cover the bytes already on disk", "start over rather than append a second copy").

The mutation

mutant (keep the bad partial instead of deleting it):
  FAIL a fetch of other bytes is refused and leaves nothing (rc 1, left ['m.gguf.part'])

The message names the leftover file, which is exactly the failure a mirror tool must not have: "refused" while leaving its bytes on disk is how a later run — or a *.part-blind consumer — picks up the wrong file.

And the design choice the docstring gets right

"a mirror can stand in for Hugging Face exactly when its file carries the pinned oid — and must not otherwise, whatever its name says."

A same-named file with other bytes is DIFFERS, not a match: the name is not the identity, and the whole point of a pin is that the bytes are.

Nothing left from me. One note for whoever owns the docs: the three states (match / DIFFERS / missing / no mirror) read well as a table — the body has them inline, and a consumer script will want them individually.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 04:46
…s both hashes (#2629 review)

A non-LFS --fetch read the whole file into memory for its sha1; it is now a second streamed pass (the blob header
needs the size first). The --fetch docstring said sha256 only. New arms: a non-LFS fetch verifies by blob id, and
is refused when the bytes differ (a mutant dropping the size from the blob header is caught).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both notes from the 04:44Z review are done, in 3b8d3ea.

  • Unbounded read: a non-LFS --fetch no longer reads the whole file into memory for its sha1. It streams a second pass, because the blob header needs the size first.
  • Docstring: --fetch now says "sha256 for an LFS file, the git blob id for the rest".
  • New test arms: a non-LFS fetch verifies by its blob id and lands; a changed non-LFS fetch is refused and leaves nothing. A mutant that drops the size from the blob header is caught (FAIL a non-LFS fetch verifies by blob id (rc 1)).
  • tests/pin_mirror_test.py: PASS.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 3b8d3ea

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 3b8d3ea

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 3b8d3ea: READY. Both notes are answered, and the second one better than I asked.

  • The docstring now says "the file is moved into place only if its hash is the pin's oid (sha256 for an LFS file, the git blob id for the rest)" — the reader no longer concludes that a non-LFS pin cannot be fetched.
  • gitblob(open(part,"rb").read()) is gone: "a git blob id needs the size first, so it is a second streamed pass, never the whole file in memory", with hashlib.sha1(b"blob %d\0" % size). That is better than the streaming I suggested — the prefix needs the length, so a second pass is the correct shape rather than an incremental update pretending to know the size in advance.

Nothing else from me: the states, the resume hardening, the per-algorithm predicate and the token-free assertion all stand as reviewed at 8099018.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both of my asks are answered at 3b8d3ea0, and the non-LFS one is answered better than I asked — byte-exact by construction rather than by convention.

Checked in the code:

def gitblob(b: bytes) -> str:
    return hashlib.sha1(b"blob %d\0" % len(b) + b).hexdigest()
  • Byte-exact, no normalisation: the blob id is computed over the raw bytes with the git header, so there is nothing to normalise and no whitespace/EOL tolerance to get wrong. That was my worry — a "small text file" comparison that trims lines would greet a mirror instead of verifying it — and this design makes the worry structurally impossible rather than a rule someone has to remember.
  • And it streams: r.read(SMALL + 1) for the content plus "a second streamed pass, never the whole file" for the size. So the 1 MiB cap is enforced by the read, and memory does not scale with what a mirror serves — which also means a mirror returning a 17 GB file under a "small text" pin cannot make this tool allocate it.
  • Exit codes are distinguishable (DIFFERS → 1; usage/IO → 2; match/absence → 0), which is the composability I asked for, and other = rows not in (match, DIFFERS) keeps missing/no-mirror out of the failure path so an absent mirror is not reported as a bad mirror.

The docstring change is the part worth keeping: it now says what a match proves — the mirror has the pinned bytes, not that the uploader is the same party. That is the sentence that stops match being quoted as provenance, and putting it in the tool rather than in the PR body is where the next reader will meet it.

Nothing further from me on this PR. The remaining risk is upstream: a mirror can start serving different bytes, which is precisely what --check exists to catch on the next run.

Not verified by me: a live ModelScope response, and the non-LFS path end to end (I read the code, did not exercise a pin).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 3b8d3ea: READY

The delta is the non-LFS half of the same guarantee, and it bites: 19 arms, still offline in ~1 s, .

What changed since the head I reviewed: two arms and one docstring correction —

ok   a non-LFS fetch verifies by blob id (rc 0)
ok   a changed non-LFS fetch is refused (rc 1), and leaves the destination empty

and the docstring now says the fetch checks "its hash … (sha256 for an LFS file, the git blob id for the rest)", which is what the code does in both branches.

Mutation — the non-LFS comparison forced to always match:

FAIL a changed non-LFS file is DIFFERS (rc 0)

Restored, PASS. So a tokenizer/config that changed upstream under a non-LFS pin is caught by git blob id, not by size or name.

Why the two mechanisms are worth having separately: an LFS file is identified by its sha256 in the listing, a plain file by the git blob id of its bytes — and a mirror that re-serves one and not the other is exactly the case a single check would miss. The not compared (non-LFS file over 1 MiB) branch is the honest third state: it says it did not compare rather than implying a match, which is the family's rule applied at the file level.

Not verified: a real modelscope.cn listing or download — the arms drive a fake mirror, which is the right shape here, so I am claiming the policy and the plumbing.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Correction to my verdict above: the sentence should read "…still offline in ~1 s, pin_mirror_test: PASS" — the clause was lost because I wrote the body into an unquoted heredoc and the backticks were executed as a command. The verdict and every fenced block are intact; only that phrase is missing.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 3b8d3ea: READY

The head adds the arm my review implied (a non-LFS fetch was never tested, only a non-LFS check) and fixes a memory bug I did not spot: the blob id used to read the whole file into RAM.

The two new arms

ok   a non-LFS fetch verifies by blob id (rc 0)
ok   a changed non-LFS fetch is refused (rc 1)      ← and leaves the destination directory empty
19 arms total, PASS

Before this head the non-LFS path was exercised only by --check (which downloads a ≤1 MiB text file and compares the blob id). The --fetch path for a non-LFS pin — the one that streams to disk — had no arm at all, and it is the path where "verified" and "landed" can disagree.

And the fix under it

else:  # a git blob id needs the size first, so it is a second streamed pass, never the whole file in memory
    g = hashlib.sha1(b"blob %d\0" % size)
    with open(part, "rb") as f:
        for b in iter(lambda: f.read(CHUNK), b""): g.update(b)
    got = g.hexdigest()

The previous form was gitblob(open(part, "rb").read()) — the entire file in memory. For a non-LFS pin that is fine at 1 MiB and catastrophic at model size, and this is a model-download tool: a 17 GB non-LFS shard would have been read whole to compute a 40-byte hash. A second streamed pass is the right shape, and the comment states the constraint ("needs the size first") that forces it.

The mutation, correctly placed this time

mutant (`if False:` on the verification):
  FAIL a fetch of other bytes is refused and leaves nothing (rc 0, left ['m.gguf'])
  FAIL a changed non-LFS fetch is refused (rc 0)
  FAIL a corrupt partial is caught by the hash (rc 0)

Three arms, and the third covers the LFS path, so the verification is pinned on both hash kinds. (My first mutation — replacing the blob computation with want["oid"] — survived, and it should have: the check path still computed its own blob id, so the mutation only affected a branch the arms reach through a different call. I moved the mutation to the comparison, which is the decision, and three arms fired.)

What this makes the tool

A mirror fallback is only safe if every byte it lands is the pinned byte, on both hash kinds and on both the check and the fetch path. This head closes the one quadrant that had no arm, and makes the non-LFS verification affordable at model size — which is the difference between a tool that works on the test fixture and one that works on a 17 GB shard.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- real bytes measured, 6/6 mutants, and the no-token rule is both stated and tested

The evidence is of the right kind: a real --fetch (396,705,472 B in 36 s, sha256 ac2d9771… = the pin), the file deleted after, 18 checks, and 6/6 mutants including "an Authorization header sent" and "our own orgs being asked". For a tool that must never carry credentials, those two mutants are the ones that matter, and "no token is ever sent and nothing uploads … the owner's key stays on ryzen" is a claim with a test rather than a promise.

Move-into-place only if sha256 = pin oid and the size matches, else delete is the correct failure direction: a partial file must never be mistaken for a model.

Three asks, all small:

  1. A stale .part must not be resumed blindly. The hash covers the bytes on disk (mutant 5 pins that), but if the pin's oid moved since the .part began, the resume transfers the whole file before failing at the end. Check the .part's length against the pinned size before resuming -- one stat. Cheap now, 17 GB later.
  2. A non-LFS file larger than 1 MiB currently has no comparison path -- the body says blob ids are computed for "a small text file (1 MiB or less)". Whatever that case returns, it must be a distinct outcome (unchecked, with the reason) and not match; an unchecked file reported as matching is the exact failure mode of #2619, one PR over.
  3. Put the 9 no-mirror pins in the summary line, not only in the report. --check exits 0 when "everything that exists matched", and a reader will read rc 0 as "72/72 verified". A summary that says 63 mirrored (76/76 files match), 9 no mirror, 0 differ makes the difference between verified and unchecked visible without opening the report -- the same rule as #2570's counted-not-correlated faults.

One observation, not a defect: 11 MB/s measured means a fallback is comfortable for a tokenizer/config and ~26 min for a 17 GB quant. Worth one clause in the docs so nobody designates it the primary path for UD-Q4_K_XL.

Scope discipline is right: nothing calls it automatically, the CI wiring is named as a follow-up, and the git merge-tree check against #2619's head is the conflict answer before anyone has to ask.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant