Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@ jobs:
python3 tests/check_pins_test.py
python3 tests/hrx_long_prompt_test.py
python3 tools/template_bom.py --self-test # the chat-template BOM component
- name: Model pins
# fixtures only: models/pins.json itself is compared with the Hub by pin-watch.yml
run: python3 tools/pin_watch.py --self-test
- name: Configure
run: cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc-16 -DCMAKE_CXX_COMPILER=g++-16
- name: Build
Expand Down
99 changes: 99 additions & 0 deletions .github/workflows/pin-watch.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Copyright 2026 bong-water-water-bong
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# The daily model-pin watch (docs/model-pins.md): compare every pin in models/pins.json with the
# upstream repo's current main, re-read the chat template of anything that moved, and open ONE PR per
# changed repo bumping that repo's pin, with the report as the body. Adoption is a reviewed merge; this
# workflow never touches the catalogue, the stick image or a scoreboard row.
#
# Uses the same HRX_BUMP_TOKEN as the bump and census workflows: a PR opened with GITHUB_TOKEN would not
# run CI. That token also lifts Hugging Face's anonymous rate limit where the range reads need it.

name: pin-watch

on:
schedule:
- cron: "43 5 * * *"
workflow_dispatch:

permissions:
contents: read

concurrency:
group: pin-watch
cancel-in-progress: false

jobs:
pin-watch:
runs-on: ubuntu-latest
timeout-minutes: 90
steps:
- name: Require the token
env:
HRX_BUMP_TOKEN: ${{ secrets.HRX_BUMP_TOKEN }}
run: |
if [ -z "$HRX_BUMP_TOKEN" ]; then
echo "::error::secret HRX_BUMP_TOKEN is not set (see the header of this workflow)"
exit 1
fi

- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
token: ${{ secrets.HRX_BUMP_TOKEN }}

- name: Compare every pin with its upstream
id: check
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
run: |
set -uo pipefail
# exit 1 means "a pin moved", which is the normal case this workflow exists for
rc=0
python3 tools/pin_watch.py --check --out pin-report.md || rc=$?
if [ "$rc" -ne 0 ] && [ "$rc" -ne 1 ]; then exit "$rc"; fi
echo "moved=$(grep -c '^## `' pin-report.md || true)" >> "$GITHUB_OUTPUT"
cat pin-report.md >> "$GITHUB_STEP_SUMMARY"

- name: Open one pin bump per changed repo
if: steps.check.outputs.moved != '0'
env:
GH_TOKEN: ${{ secrets.HRX_BUMP_TOKEN }}
HF_TOKEN: ${{ secrets.HF_TOKEN }}
run: |
set -euo pipefail
day=$(date -u +%F)
git config user.name "pin-watch"
git config user.email "pin-watch@users.noreply.github.com"
grep '^## `' pin-report.md | sed 's/^## `//; s/`$//' | while read -r repo; do
slug=$(echo "$repo" | tr '/.' '--')
branch="pin-watch/$slug-$day"
if git ls-remote --exit-code origin "refs/heads/$branch" > /dev/null 2>&1; then
echo "$branch already exists"; continue
fi
git switch -c "$branch" main --quiet
if ! python3 tools/pin_watch.py --bump "$repo"; then
# a tracked file vanished upstream: the report says so, and a human decides
echo "::warning::$repo could not be bumped automatically (see the report)"
git switch main --quiet
continue
fi
python3 tools/pin_watch.py --pr-body "$repo" > body.md || true
newsha=$(python3 -c 'import json,sys; print(json.load(open("models/pins.json"))["pins"][sys.argv[1]]["commit"][:8])' "$repo")
git add models/pins.json
git commit -q -m "pin: $repo at $newsha""
git push -q origin "$branch"
gh pr create --base main --head "$branch" --title "Pin: $repo" --body-file body.md
git switch main --quiet
done
128 changes: 128 additions & 0 deletions docs/model-pins.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
<!--
Copyright 2026 bong-water-water-bong
SPDX-License-Identifier: Apache-2.0

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
# Model pins: the revision of every upstream model we depend on

The census ([registry.md](registry.md)) counts what the engine *can* run. It does not record **which
revision of which upstream model** we actually run, ship, or were built from — so a vendor can re-publish
the quants we re-host and nothing in the repository notices. That happened on 2026-11-04: unsloth
re-quantized `Qwen3.8-Flash-Next-GGUF` ("Better UD 3.1 quants", "mix n match engram"; UD-IQ4_XS went from
three shards to four) and the files we re-hosted were still the 2026-11-02 ones, at the same names.

`models/pins.json` is that record. One entry per Hugging Face repo we depend on:

```json
"unsloth/Qwen3.8-Flash-Next-GGUF": {
"commit": "d96201194f103c906994f283f504209dc8d2a10c",
"pinned": "2026-11-05",
"used_for": ["upstream: our copy predates the pinned revision"],
"files": {
"UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf": {
"oid": "4448186216b3af4cc558bbce2c3213f01608f8f8b2e5267a9767971dd3ec8082",
"size": 10946624, "lfs": true,
"template_sha256": "12827f24b742ea4e80cdc12dbcf9622227056b9f797252a3149263d4f9aaadce"}}}
```

- `commit` — the upstream revision the pin was taken at.
- `files` — the files **we hold**, not the files that happen to be there: `oid` is Hugging Face's content
identity (`lfs.oid`, the sha256 of the bytes; the git blob id for a small text file that is not in LFS),
`size`, and `template_sha256`, the chat template that file carries at that revision.
- `used_for` — why the repo is pinned: a `laya-first` serve class, a stick/Lemonade catalogue row, a
published `1bit-MONSTER` re-host, or the upstream of one.

A model **baked into a stick image** is pinned from the image's own Hugging Face cache, which records the
snapshot commit the image was built from (`~/stage-c25/hf/models--<org>--<name>/snapshots/<commit>`). That
is provenance rather than an assumption: the two chat models the c25 image bakes (`Qwen3-4B-Instruct-2507`,
the FastFlowLM NPU pack) pin to exactly the commits the image's cache names, and `stabilityai/sd-turbo` —
the image model, which carries no chat template — was added with its 5,214,561,328-byte safetensors hashed
locally to confirm it is the file upstream serves at that commit.

Of the 154 model files staged on this box, the 11 whose names match a pinned file are pinned at file
level; the rest are test vocabularies (`ggml-vocab-*`), conversion intermediates, or additional quant
variants of repos that are pinned at revision level — the revision check still catches a vendor re-push for
those, which is the case that matters.

`files` records what we hold rather than only what upstream said, because a pin taken *after* a vendor
re-push would otherwise store our older copies under the newer commit and look clean. Both facts are
therefore reconciled on every check: the commit for "did upstream move", and the per-file identity for
"are the bytes we hold still the bytes it serves".

## The daily check

`tools/pin_watch.py --check` (`.github/workflows/pin-watch.yml`, daily) compares every pin with the repo's
current `main` and writes a Markdown report. It exits non-zero when anything moved, so the run is visible
even if the PR is not opened.

There are three outcomes, and the exit code keeps them apart:

| exit | meaning | what the workflow does |
|---|---|---|
| 0 | every pin was read and is at the revision we record | nothing |
| 1 | something moved, or a pinned repo is **gone upstream** (`404` — a delete, a rename and a privatisation look the same) | opens one bump PR per moved repo; a gone repo is reported and refused for bumping, a human decides |
| 2 | **no pin could be read at all** — a Hub outage or rate-limit is not a quiet day | the job fails |

The report's first line counts what was read (`N of M checked, K not checked`), and the all-clear sentence
only appears when nothing was left unread: a watcher whose "clean" and "could not check" are the same
outcome is blind exactly when it matters.

For a file that moved it also re-reads the **chat template** of the new revision — a GGUF's
`tokenizer.chat_template` over an HTTP range request of the metadata header only, never a tensor; a
container's `tokenizer_config.json` parsed and its `chat_template` **string** hashed, which is the byte
sequence the engine's serve line names, not the JSON file's own digest; a `chat_template.jinja` hashed
verbatim — and reports one of: unchanged, `TEMPLATE CHANGED`, template added, template removed, or not
checked (`--no-template`). That is the gate a re-quantization has to pass before anything may adopt it
([template-publishing.md](template-publishing.md)): weights moving under a stable file name is ordinary,
the instructions the model obeys on every request moving under it is not.

## Adopting a new revision is never automatic

The workflow opens **one PR per changed repo**, bumping only that repo's entry, with the report as the
body. Merging it says *this is the revision we point at*; it does not move the catalogue, the stick image
or a scoreboard row, each of which is its own reviewed change. So:

1. the template column must be clean (unchanged, or the change audited and recorded), and
2. the row that uses the model is re-measured, because a re-quantization is a different file even when
its name is the same.

`--bump` refuses a pin whose tracked file vanished upstream instead of dropping it: a file that is gone is
a decision (re-take it under its new name, or drop the pin), not a diff.

## Commands

```bash
tools/pin_watch.py --check # the daily comparison; writes pin-report.md, exit 0/1/2 (above)
tools/pin_watch.py --check --no-template # same, without the template range reads
tools/pin_watch.py --pr-body REPO # the bump PR body for one moved repo
tools/pin_watch.py --pin REPO [REPO...] # print an entry for a repo at its current main (adding a pin)
tools/pin_watch.py --bump REPO # rewrite one entry in models/pins.json (a reviewed change)
tools/pin_watch.py --self-test # fixtures, no network
```

Adding a repo is `--pin` followed by a hand edit of `used_for`: a pin exists because something serves,
ships or was converted from it, and that reason is what makes the entry reviewable. The template digest is
per **file**, never per model — `Qwen3-4B`'s GGUF in our org and the quant the box serves carry two
different templates, both legitimate.

## What is not pinned, and why

- `base_model` on a card is provenance, not a dependency: what we took from a vendor repo is the file we
re-hosted, and that is the file pinned (`upstream_gguf`). A base model whose weights never reached us is
not a pin.
- Private repositories are not pinned here: this repository is public. A private model is tracked in the
lane state that owns it.
- Third-party git pins (`third_party/*`) are a different mechanism and stay where they are —
[../tools/check_pins.py](../tools/check_pins.py) for a PR that moves one backwards, and
[releases.md](releases.md) for the weekly "what moved" summary.
6 changes: 6 additions & 0 deletions docs/registry.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,10 @@ counts and the coverage read against the registry and the checked results. The
`census.yml` workflow runs daily at 05:17 UTC. It rebuilds the registry from the pins,
sweeps HF, and opens a PR when anything changed.

The census counts what the engine *can* run; which revision of each upstream model we actually run,
ship or were built from is `models/pins.json`, checked daily by `tools/pin_watch.py`
([model-pins.md](model-pins.md)).

The census of 2026-10-17 (the first full sweep ran 2026-09-25), with `registry/census.json` holding the latest:

| | models | share of those with an architecture |
Expand Down Expand Up @@ -185,5 +189,7 @@ tools/census.py --report # recompute coverage from the saved counts
tools/census_watch.py # newest models vs the registry, plus FastFlowLM's org and
# releases (--full to list all); exit 1 on an uncovered class
# or an unmapped FastFlowLM family
tools/pin_watch.py --check # every pinned upstream revision vs its current main; exit 1 when
# one moved, with the chat template of anything that moved re-read
scripts/census-watch.sh # the same, with the log file and the origin/main refresh
```
Loading
Loading