Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1294,6 +1294,9 @@ if(Python3_Interpreter_FOUND)
# the same check where a lane actually runs it: with the bump staged and HEAD still behind, where a HEAD-only
# read of the gitlink passed a registry recording the pin being replaced (engine#1738's post-commit failure)
add_test(NAME registry_pins_staged COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/tests/registry_pin_staged_test.py)
# ModelScope as a fallback source for pinned files, against a fake mirror: a same-named file with other bytes is
# refused, an unverified fetch never lands, a non-LFS pin is compared by its git blob id, and no request carries a token
add_test(NAME pin_mirror COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/tests/pin_mirror_test.py)
# the 1.9 measurement tool's own paths, on synthetic packs: the "trying next" line it shipped broken (its
# only real pair verified on the first candidate, so the line never ran) and the residual tolerance it
# accepts and refuses at. Fixtures are ~200 KB each and built from the rule, so they can be wrong on purpose.
Expand Down
37 changes: 37 additions & 0 deletions docs/model-pins.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,43 @@ makes the run's first line say so, and adding it is `--pin` plus a `used_for` th
A repo of `.1bp` packs is not pinned here, and a private repo never is (next section) — both are filtered
out, so the census cannot go permanently red over something this file must not contain.

## A fallback source: ModelScope

ModelScope (modelscope.cn) mirrors many of the repos we pin, under the same id, and its public file listing
gives each file's sha256. So a mirror can stand in for Hugging Face exactly when its file carries the pinned oid,
whatever the file is called. `tools/pin_mirror.py` checks that and fetches through it. It is anonymous:
the public listing and download need no token, and the tool never sends one. The owner's ModelScope key stays
off this box, and nothing here uploads.

```bash
tools/pin_mirror.py --check # every pin outside our own orgs: is the mirror the pinned file?
tools/pin_mirror.py --check --repo unsloth/Qwen3.8-27B-GGUF
tools/pin_mirror.py --fetch REPO FILE --dest DIR # download from the mirror; kept only if sha256 == the pin's oid
```

`--check` reports each pinned file as one of four states:

- `match`: the mirror may serve it.
- `DIFFERS`: other bytes under the pinned name, so do not use it.
- `missing`.
- no mirror under that id.

A non-LFS pin records a git blob id, which a sha256 cannot be compared with. For those, a small text file
(≤ 1 MiB) is downloaded and its blob id computed. `--fetch` hashes while it streams, resumes a `.part`
(the hash covers the bytes already on disk), and deletes a file that is not the pinned one rather than moving it
into place. Exit codes:

- **0** everything compared matches, or the fetch verified.
- **1** a mirror differs, or the fetch did not verify.
- **2** usage, or no listing read.

Measured 2026-11-16 against `models/pins.json` (72 pins outside our orgs):

- 63 are mirrored under the same id, and all 76 of their pinned files match, including
`unsloth/Qwen3.8-27B-GGUF`'s UD3.1 `Qwen3.8-27B-UD-Q4_K_XL.gguf` (`2a5f3565…`).
- 9 have no mirror under that id: two RichardErkhov repos, three mradermacher repos, mys, nomic-ai, nvidia and pqnet.
- One real fetch, `unsloth/Qwen3-0.6B-GGUF` `Qwen3-0.6B-Q4_K_M.gguf` (397 MB, 36 s), verified against its pin.

## Build inputs

The model pins above are one kind of input. A build or CI run also takes inputs written inline in workflows and
Expand Down
172 changes: 172 additions & 0 deletions tests/pin_mirror_test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
#!/usr/bin/env python3
# Copyright 2026 bong-water-water-bong
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""tools/pin_mirror.py against a fake ModelScope (no network, ~0.1 s).

The decisive cases: a mirror file under the pinned name with other bytes must be refused as a source, a fetched file
that is not the pinned one must never be moved into place, a non-LFS pin (a git blob id) is compared by its blob id
and not waved through on a sha256, and no request carries a token.
"""
import hashlib
import io
import json
import os
import sys
import tempfile
import urllib.error

HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.join(HERE, "..", "tools"))
import pin_mirror as M # noqa: E402

fails = 0


def check(ok, what):
global fails
print(("ok " if ok else "FAIL ") + what)
if not ok:
fails += 1


def sha(b):
return hashlib.sha256(b).hexdigest()


GOOD, OTHER, TEXT = b"W" * 5000, b"V" * 5000, b'{"chat_template": "t"}\n'
SERVED = {} # (repo, path) -> bytes the fake mirror serves
LISTED = {} # repo -> listing rows
SEEN = [] # every request's headers


class Resp(io.BytesIO):
def __init__(self, b, status=200):
super().__init__(b)
self.status = status

def __enter__(self):
return self

def __exit__(self, *a):
self.close()


def fake_urlopen(req, timeout=0):
# patched under the tool's own _open, so SEEN is what would really go on the wire
url, headers = req.full_url, {k.lower(): v for k, v in req.header_items()}
SEEN.append(headers)
if "/api/v1/models/" in url:
repo = url.split("/api/v1/models/")[1].split("/repo/files")[0]
if repo not in LISTED:
raise urllib.error.HTTPError(url, 404, "nf", {}, None)
return Resp(json.dumps({"Code": 200, "Data": {"Files": LISTED[repo]}}).encode())
repo, path = url.split("/models/")[1].split("/resolve/master/")
b = SERVED[(repo, path)]
rng = headers.get("range")
if rng:
start = int(rng.split("=")[1].rstrip("-"))
return Resp(b[start:], 206)
return Resp(b)


M.urllib.request.urlopen = fake_urlopen


def files(repo, rows):
LISTED[repo] = [{"Path": p, "Size": len(b), "Sha256": sha(b), "IsLFS": lfs, "Type": "blob"}
for p, b, lfs in rows]
for p, b, _ in rows:
SERVED[(repo, p)] = b


with tempfile.TemporaryDirectory() as tmp:
pins = {"pins": {
"up/Good-GGUF": {"files": {"m.gguf": {"oid": sha(GOOD), "size": len(GOOD), "lfs": True},
"tokenizer_config.json": {"oid": M.gitblob(TEXT), "size": len(TEXT), "lfs": False}}},
"up/Moved-GGUF": {"files": {"m.gguf": {"oid": sha(GOOD), "size": len(GOOD), "lfs": True}}},
"up/Partial-GGUF": {"files": {"a.gguf": {"oid": sha(GOOD), "size": len(GOOD), "lfs": True},
"b.gguf": {"oid": sha(GOOD), "size": len(GOOD), "lfs": True}}},
"up/Absent-GGUF": {"files": {"m.gguf": {"oid": sha(GOOD), "size": len(GOOD), "lfs": True}}},
"1bit-MONSTER/Ours": {"files": {"m.gguf": {"oid": sha(GOOD), "size": len(GOOD), "lfs": True}}},
}}
pf = os.path.join(tmp, "pins.json")
json.dump(pins, open(pf, "w"))
files("up/Good-GGUF", [("m.gguf", GOOD, True), ("tokenizer_config.json", TEXT, False)])
files("up/Moved-GGUF", [("m.gguf", OTHER, True)])
files("up/Partial-GGUF", [("a.gguf", GOOD, True)])
out = os.path.join(tmp, "r.json")

# 1. one mirror serves other bytes -> rc 1, and that file is named.
rc = M.main(["--pins", pf, "--check", "--json", out])
r = json.load(open(out))
st = {(x["repo"], x["path"]): x["state"] for x in r["rows"]}
check(rc == 1, f"a mirror with other bytes under the pinned name fails the check (rc {rc})")
check(st.get(("up/Moved-GGUF", "m.gguf")) == "DIFFERS", "the differing file is reported DIFFERS")
check(st.get(("up/Good-GGUF", "m.gguf")) == "match", "a matching LFS file is a match")
check(st.get(("up/Good-GGUF", "tokenizer_config.json")) == "match",
"a non-LFS pin is compared by its git blob id (downloaded), and matches")
check(st.get(("up/Partial-GGUF", "b.gguf")) == "missing", "a pinned file the mirror lacks is missing")
check(r["unmirrored"] == ["up/Absent-GGUF"], f"a repo with no mirror is listed, not failed ({r['unmirrored']})")
check(not any(x["repo"].startswith("1bit-MONSTER/") for x in r["rows"]) and "1bit-MONSTER/Ours" not in r["unmirrored"],
"our own orgs are not asked")

# 2. the non-LFS check really is the blob id: a sha256 of the text equal to nothing in the pin must not pass.
SERVED[("up/Good-GGUF", "tokenizer_config.json")] = TEXT + b" "
rc = M.main(["--pins", pf, "--check", "--repo", "up/Good-GGUF", "--json", out])
st = {x["path"]: x["state"] for x in json.load(open(out))["rows"]}
check(rc == 1 and st["tokenizer_config.json"] == "DIFFERS", f"a changed non-LFS file is DIFFERS (rc {rc})")
SERVED[("up/Good-GGUF", "tokenizer_config.json")] = TEXT

# 3. --repo limited to clean repos -> rc 0.
rc = M.main(["--pins", pf, "--check", "--repo", "up/Good-GGUF", "up/Absent-GGUF"])
check(rc == 0, f"every compared file matches -> rc 0 (rc {rc})")

# 4. fetch: the pinned bytes land in place; other bytes are deleted, never moved into place.
dest = os.path.join(tmp, "dl")
rc = M.main(["--pins", pf, "--fetch", "up/Good-GGUF", "m.gguf", "--dest", dest])
check(rc == 0 and open(os.path.join(dest, "m.gguf"), "rb").read() == GOOD, f"a verified fetch lands (rc {rc})")
rc = M.main(["--pins", pf, "--fetch", "up/Moved-GGUF", "m.gguf", "--dest", os.path.join(tmp, "dl2")])
left = os.listdir(os.path.join(tmp, "dl2"))
check(rc == 1 and left == [], f"a fetch of other bytes is refused and leaves nothing (rc {rc}, left {left})")

# 4b. a non-LFS pin is fetched and verified by its git blob id (streamed), and refused when the bytes differ.
rc = M.main(["--pins", pf, "--fetch", "up/Good-GGUF", "tokenizer_config.json", "--dest", os.path.join(tmp, "dlt")])
check(rc == 0 and open(os.path.join(tmp, "dlt", "tokenizer_config.json"), "rb").read() == TEXT,
f"a non-LFS fetch verifies by blob id (rc {rc})")
SERVED[("up/Good-GGUF", "tokenizer_config.json")] = TEXT + b" "
rc = M.main(["--pins", pf, "--fetch", "up/Good-GGUF", "tokenizer_config.json", "--dest", os.path.join(tmp, "dlt2")])
check(rc == 1 and os.listdir(os.path.join(tmp, "dlt2")) == [], f"a changed non-LFS fetch is refused (rc {rc})")
SERVED[("up/Good-GGUF", "tokenizer_config.json")] = TEXT

# 5. a partial download resumes and the hash covers the bytes already on disk.
d3 = os.path.join(tmp, "dl3")
os.makedirs(d3)
open(os.path.join(d3, "m.gguf.part"), "wb").write(GOOD[:1234])
rc = M.main(["--pins", pf, "--fetch", "up/Good-GGUF", "m.gguf", "--dest", d3])
check(rc == 0 and open(os.path.join(d3, "m.gguf"), "rb").read() == GOOD, f"a resumed fetch verifies (rc {rc})")
check(any(h.get("range") == "bytes=1234-" for h in SEEN), "the resume asked for the remaining range only")
open(os.path.join(d3, "m.gguf.part"), "wb").write(OTHER[:1234]) # a corrupt partial must not pass as resumed
os.remove(os.path.join(d3, "m.gguf"))
rc = M.main(["--pins", pf, "--fetch", "up/Good-GGUF", "m.gguf", "--dest", d3])
check(rc == 1 and not os.path.exists(os.path.join(d3, "m.gguf")), f"a corrupt partial is caught by the hash (rc {rc})")

# 6. unknown inputs are rc 2; no request ever carries a token.
check(M.main(["--pins", pf, "--fetch", "up/Good-GGUF", "nope.gguf", "--dest", dest]) == 2, "an unpinned file is rc 2")
check(M.main(["--pins", pf, "--check", "--repo", "no/Such"]) == 2, "an unpinned repo is rc 2")
check(SEEN and not any("authorization" in h for h in SEEN) and all(h.get("user-agent") == M.UA for h in SEEN),
f"no request carries a token ({len(SEEN)} requests)")

print("pin_mirror_test: " + ("PASS" if fails == 0 else f"{fails} FAILED"))
sys.exit(1 if fails else 0)
Loading
Loading