Skip to content

pin_holdings: the box's model files and the re-hosts against their pins (UNSLOTH-UD31) - #2619

Open
bong-water-water-bong wants to merge 1 commit into
mainfrom
cl-856/pin-holdings
Open

bong-water-water-bong wants to merge 1 commit into
mainfrom
cl-856/pin-holdings

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

UNSLOTH-UD31 step 1 (orchestrator 11-16, owner GO): the "stay updated" gap. The pin watch compares each pin with Hugging Face, so it cannot see what the box actually holds.

What was green while wrong

  • unsloth/Qwen3.8-27B-GGUF has been pinned since 2026-11-12 at the UD3.1 Qwen3.8-27B-UD-Q4_K_XL.gguf: 2a5f3565…, 17,659,407,264 B.
  • ~/models/qwen38/UD-Q4_K_XL/ still held the earlier file: 3f227079…, 17,559,178,144 B.
  • So did our re-host 1bit-MONSTER/Qwen3.8-27B-UD-Q4_K_XL-MTP.
  • Every daily pin-watch run stayed green through this.

Change

  • tools/pin_holdings.py runs two checks:
    • Holdings. models/holdings.json maps each pinned file to its copy on the box. Each copy's sha256 is compared with its pin's oid. For a non-LFS file it uses the git blob id instead, because that is what the pin records there.
      • Hashes are cached by path, size, mtime and inode, so a day with no changes reads no model file.
      • A size mismatch is reported without reading the file.
    • Re-hosts. A pin whose used_for says upstream of R is compared with R's own pin, file by file of the same name. This half reads only pins.json, so it needs neither the box nor the network (--rehost-only).
    • Exit codes: 0 clean, 1 drift, 2 usage or a holding that pins.json cannot resolve.
  • scripts/1bit-pin-holdings.{service,timer} run it daily on the box at 05:00 UTC. Exit 1 marks the unit failed, and the report goes to ~/.1bit/logs/pin-holdings-latest.md.
  • ctest pin_holdings is hermetic.
  • docs/model-pins.md gets a new section, "What the box holds".

Not in this PR: the CI step

The workflow step that runs --rehost-only and warns on the run could not be pushed: this token has no workflow scope. The 12-line step is ready in the lane's scratch. An owner or a workflow-scope lane would need to add it after Compare every pin with its upstream:

      - name: Re-hosts against their upstream pins
        run: |
          set -uo pipefail
          rc=0
          python3 tools/pin_holdings.py --rehost-only --out rehost-report.md > /dev/null || rc=$?
          if [ "$rc" -eq 1 ]; then echo "::warning::$(sed -n 4p rehost-report.md) — see the summary"; fi
          if [ "$rc" -gt 1 ]; then exit "$rc"; fi
          cat rehost-report.md >> "$GITHUB_STEP_SUMMARY"

Evidence

  • Real box run. 10 holdings were checked: the 9 in this map plus the freshly downloaded UD3.1 copy.
    • 9 match their pin's oid:
      • DFlash2 q4k and q8_0
      • flint UD-Q4_K_XL
      • Qwen3-0.6B Q4_K_M and Qwen3-4B Q4_K_M
      • ZAYA1-8B
      • Qwen2.5-0.5B
      • ZR1-1.5B
      • the new download UD-Q4_K_XL-ud31/, whose sha256 = 2a5f3565f39c… = the pin
    • 1 is drift: ~/models/qwen38/UD-Q4_K_XL/ is size differs (17.56 GB held against 17.66 GB pinned).
    • Re-hosts: one is reported, the 27B MTP re-host. It needs a re-upload, which is owner-only and has not been done.
    • The run exits 1, which is correct.
  • --rehost-only on main's pins.json: rc 1, with the same single re-host row. Three upstream files have no same-named file in their re-host (GLM-4.7-Flash, Qwen1.5-MoE, OpenELM, which ship a different quant), so they are not compared.
  • tests/pin_holdings_test.py: PASS, 17 checks.
  • Mutants (5/5 caught):
    • cache keyed on size only: caught by the same-size rewrite check
    • no hash when the size matches
    • re-host check off
    • missing copy treated as ok
    • map errors treated as clean

The 27B default path stays red in this check until the UD3.1 A/B decides the swap (UNSLOTH-UD31 step 3). That is intended.

🤖 Generated with Claude Code

tools/pin_watch.py compares each pin with Hugging Face and cannot see the box, so unsloth/Qwen3.8-27B-GGUF
was pinned at the UD3.1 Qwen3.8-27B-UD-Q4_K_XL.gguf (2a5f3565..., 2026-11-12) while the box and the re-host
1bit-MONSTER/Qwen3.8-27B-UD-Q4_K_XL-MTP kept the earlier file (3f227079...) and every daily run was green.

tools/pin_holdings.py: models/holdings.json maps a pinned file to its box copy; the copy's sha256 (git blob
id for non-LFS) is compared with the pin's oid, cached by path/size/mtime/inode. A pin's "upstream of R" is
compared with R's pin for same-named files (pins.json only, --rehost-only; the CI step needs a workflow-scope push).
scripts/1bit-pin-holdings.{service,timer}: daily on the box, 05:00 UTC. ctest pin_holdings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 2e3ea9e

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Cache Invalidation

The cache invalidation logic in Cache.get() relies on os.stat() to check if a file has changed based on size, mtime, and inode. However, on some filesystems or under certain conditions (e.g., when files are moved or hard-linked), the inode might not change even if the file content is modified. This could lead to stale cached hashes being used, potentially causing false negatives in the pin validation. The check should be more robust to ensure that the cached hash is invalidated when the actual file content changes.

def get(self, path: str, lfs: bool) -> str:
    real = os.path.realpath(path)
    st = os.stat(real)
    key = f"{real}|{'sha256' if lfs else 'gitblob'}"
    row = self.rows.get(key)
    stamp = [st.st_size, st.st_mtime_ns, st.st_ino]
    if row and row.get("stat") == stamp:
        return row["id"]
    got = file_id(real, lfs)
    self.hashed += 1
    self.rows[key] = {"stat": stamp, "id": got}
    return got
Re-host Comparison Logic

The logic in check_rehosts() compares files by name only, using os.path.basename(path) to match files between upstream and re-host pins. If a re-host has a different file structure (e.g., files in subdirectories or renamed files), the comparison may miss drifts. This could lead to a false sense of correctness if the re-host is not actually serving the correct files from the upstream pin.

        theirs = {os.path.basename(k): (k, v) for k, v in mine["files"].items()}
        for path, want in sorted(pin["files"].items()):
            hit = theirs.get(os.path.basename(path))
            if hit is None:
                continue  # a renamed or different quant in the re-host is not the same file to compare
            if hit[1]["oid"] != want["oid"]:
                out.append({"upstream": up, "rehost": rehost, "file": path, "rehost_file": hit[0],
                            "state": "re-host holds a different file",
                            "upstream_oid": want["oid"], "upstream_size": want.get("size"),
                            "rehost_oid": hit[1]["oid"], "rehost_size": hit[1].get("size")})
return out

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 03:07

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 2e3ea9e: CHANGES

Your own self-test caught this, and it named both lists exactly. The build fails in 51 seconds on the Model pins step with exit code 3:

FAIL the header and docs/model-pins.md list the same exit codes: ['0', '1', '2', '3', '4', '0', '1', '2']

Reading that list as the check reads it: the two sites are concatenated, so the header documents five codes — 0, 1, 2, 3, 4 — and docs/model-pins.md documents three — 0, 1, 2. The doc is missing 3 and 4.

The fix

Add 3 and 4 to docs/model-pins.md's list, with their meanings taken from the header. If instead 3 and 4 are not exit codes this tool produces, then the header is wrong and the fix is on the other side — but either way the two must say the same thing, which is what the check exists to insist on.

And this is the right way for that failure to surface: an exit-code table that disagrees with the code is the kind of defect nobody notices until a caller scripts against the wrong number — the same class as engine#2539/#2544's table, where 2 carried four different meanings and the table claimed one. Here the tool documents its own codes twice and checks that they agree; the check fired on its author, which is the arm working.

What is good and should not change in the fix

  • ok --bump refuses a vanished file (a human decides, not an e[rror]) and ok --bump refuses a gone repo — both passed, and both say the refusal is deliberate. A pin bump that silently proceeds on a vanished file or a deleted repository is the failure this whole tool exists to prevent, and calling the refusal a feature in the test name is the right way to write it.
  • ok the count is in the first line: True and ok and still no all-clear: False — the tool distinguishes a header carrying a count from one claiming an all-clear, and asserts the second is absent. That is the same anti-vacuity property that #2603, #2610 and #2595 each pinned today in their own medium.
  • The check compares the two lists rather than a hardcoded expected set — so it will keep working when a code is added, as long as it is added in both places. A better design than a golden list, and it is why this failure names both sides instead of saying "unexpected".

One thing to state in the body when it is fixed

Which codes are 3 and 4, in one line each. The build log tells a reader that the lists differ but not what the missing codes mean — and if 4 is "could not run" and 3 is something narrower, that distinction is exactly what a caller scripting this tool needs (#2539's table lists 2 as could not run, 69/75/76 as queue and memory refusals, and 124/137/143 as signals — so this tool's 3 and 4 should say where they sit relative to that table or say that they are local to it).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 2e3ea9e: CHANGES

The tool is right, and it finds the real gap on this box. On git merge-tree origin/main 2e3ea9e:

  • tests/pin_holdings_test.py gives PASS.

  • Run against the real holdings (read-only, under thermal-run, with the hash cache in my scratch), it reports:

    Holdings: 9 checked, 1 differ from their pin (8 hashed this run, the rest from the cache).
    Re-hosts: 1 differ from their upstream pin.
    - …/qwen38/UD-Q4_K_XL/Qwen3.8-27B-UD-Q4_K_XL.gguf (size differs): pin … 2a5f3565f39c… 17.66 GB; the box holds - 17.56 GB
    - 1bit-MONSTER/Qwen3.8-27B-UD-Q4_K_XL-MTP … 3f227079003a… 17.56 GB; upstream … pinned at 2a5f3565f39c…
    

    That gives rc 1, as the body says. A second run hashes 0 files and takes 0.02 s, so the cache keys work.

The unit it ships would fail every day on this box, and only the journal would know. 1bit-pin-holdings.service runs:

WorkingDirectory=%h/1bit-MONSTER
ExecStart=/usr/bin/python3 -I %h/1bit-MONSTER/tools/pin_holdings.py --out %h/.1bit/logs/pin-holdings-latest.md

On strixhalo, ~/1bit-MONSTER is a checkout of 1bit-MONSTER/1bit-MONSTER, not of this repo. It has no tools/pin_holdings.py and no models/pins.json (ls → No such file). So the timer's run is python3: can't open file, exit 2. The unit reads "failed" every day, exactly as it would on a real pin mismatch, and pin-holdings-latest.md is never written. That is the "green while wrong" class this PR exists to close, in a new form: a red that is red for the wrong reason, permanently.

Fix:

  • point the unit at an engine checkout that exists and stays on main;
  • or state, as 1bit-census-watch.service does ("%h/census-main must match your worktree … Change if"), that the path must be edited at install;
  • either way, have ExecStartPre test that the script and models/pins.json exist, and fail with a message saying so. Then "cannot run" is distinguishable from "a pin differs" (exit 1).

Smaller: the first run hashes about 70 GB of GGUFs on one core. Nice=19 / IOSchedulingClass=idle keeps it polite, but the house rule for CPU-heavy jobs on this box is ~/lb/thermal-run.sh (a hot die starves the GPU queue). Wrapping ExecStart in it costs nothing on the cached days.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2619

A pin watch that compares against Hugging Face is the right instrument for a "stay updated" gap, and the description names the gap precisely rather than proposing a policy: the pins are what the fleet builds against, so a pin that silently ages is a measurement that silently ages with it.

Three asks:

  1. the watch must state what it does not cover — HF metadata says a repo moved; it does not say whether the new revision is better, different, or broken. A watch that reports "pin is behind" invites a bump, and tonight's evidence (npu-pack: a final norm at a pseudo layer is aliased to model.norm.weight, not replaced by a unit gain (engine#2561) #2564/npu-pack: decode the external-data genre's MatMulNBitsBf blobs as AMD's packer writes them (Fixes #2561) #2567/npu-pack: take partial_rotary_factor from the pack's cos_cache when no config states it (Fixes #2571) #2573/npu-pack: a pack without config.json takes its rotary span from the cos_cache (engine#2571) #2574/npu-pack: read the 1.8 all-block-32 Token .fconst: k before q, no sink field (Fixes #2594) #2597) is that a revision change can alter a pack layout silently. So the output should distinguish "a newer revision exists" from "this revision is equivalent", and only the second justifies a bump;
  2. record the digest, not just the revision id — a repo can rewrite a revision (force-push) or change files under a floating branch. docs/model-pins.md should hold the content digest of the tensors/files that matter, so "the pin is unchanged" is a claim about bytes rather than about a name;
  3. +526 lines for a watch needs its failure mode stated — what happens when HF is unreachable, rate-limited (tonight saw a 429 in another lane's CI), or returns a repo that has been deleted? A watch that fails open (reports nothing) is worse than no watch, because its silence looks like a pass.

And one scoping note: this is UNSUDLOTH-UD31 step 1, and the number of model-pin questions tonight (config/laya.json's unchanged revision in #2554, config/gpu-kernels.config in #2572, the engine's llama pin in #2526) suggests docs/model-pins.md should own the whole picture: which pins exist, what each is pinned to (revision or digest), and who bumps it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant