Repository navigation
census: two classes reviewed as not-servable, with the evidence in the entry (engine#2528) - #2548
Conversation
…e entry (engine#2528) The daily new-model watcher reported two classes as UNCOVERED, which reads as "needs engine support" when the honest answer for both is "not a conversion target", and left them to be re-reported every day. MegatronGPTMoeForCausalLM: a training-framework checkpoint format, not a serving topology. HF tags carry megatron_gpt_moe and custom_code (checked 2026-11-17, nagohachi/tiny-lm-v3-japanese-1.5b-a500m-sft). A gguf conversion lands on whichever arch its exporter targets, so there is no single arch to map it to and aliasing it to qwen*moe / llama / deepseek2 would serve the model with the wrong tensor names. RusMindAIForCausalLM: HF tags carry rusmind_ai, from-scratch and custom_code (checked 2026-11-17, DICPlay/RusMind-Zernyshko-2.0.1). from-scratch is the load-bearing tag: it rules out the "renamed finetune" case in which a mapping would otherwise be plausible, and a class string is not evidence of a tensor layout. Both go into registry/significant.json with watch: report (report only, not a mapping) and a section each in docs/arch-gaps.md, which is where the registry's own `about` says these decisions live. Checked: tools/registry_build.py --check-gaps reports "none of the 44 reviewed significant classes is mapped without a reason", and tests/census_watch_test.py passes all checks -- including the one that failed first, "every one of the 44 reviewed classes is cited in docs/arch-gaps.md", which is the guard that made this a two-file change rather than a one-file one. Not addressed here: the UNVERIFIABLE (gated, no config) rows, which need a terminal state of their own so they stop being re-reported; that is a separate change to the watcher, not a registry entry.
PR Reviewer Guide 🔍(Review updated until commit 19f2818)Here are some key observations to aid the review process:
|
|
VERDICT: READY "A reviewed decision rather than a mapping, with the evidence in the entry" is the correct answer to an Three asks:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "a class string is never evidence of a tensor layout" is the right principle, and the date in the table is wrong
The distinction is the one that matters and it is stated better than most rejections I have read: "a gguf conversion of such a checkpoint lands on whichever architecture its exporter targets, so there is no single engine arch to map the class to. Aliasing it to qwen*moe / llama / deepseek2 would serve the model with the wrong tensor names -- a load error at best, wrong numbers at worst." That is the failure an eager mapping produces, and it is silent in the worst case, which is why a reviewed non-mapping beats a plausible alias.
And from-scratch as the load-bearing tag for the second class is the correct use of tags: it rules out the "renamed finetune" case, which is exactly when mapping would be right. "a class string is never evidence of a tensor layout" belongs in docs/arch-gaps.md as a standing rule, not only in these two entries.
--check-gaps catching the requirement, and the test failing first on "every one of the 44 reviewed classes is cited", is the right ordering: the guard existed and demanded evidence before the evidence existed.
The date is wrong, and it is worth fixing because it is the kind of detail a reviewer uses to judge the rest: the table says "HF tags (checked 2026-11-17)" — today is 2026-11-15. Two days in the future is almost certainly a typo, but in a PR whose whole argument is "I looked, and here is what I saw", a wrong observation date is the one field a reader checks to decide whether to trust the rest.
One ask, on the substance: name what a correct support path would look like for MegatronGPTMoeForCausalLM in docs/arch-gaps.md, in one line each for the two halves of the problem -- the exporter that decides the layout, and the tensor-name mapping the engine would need if that exporter ever emitted a fixed target. The entry already says what real support would need; make that sentence name the observable (which exporter, which target), because the next person who sees 2882 models behind a class will want to know whether they are looking at a missing mapping or at a missing exporter convention.
Nothing else: watch: "report" rather than a mapping is the correct use of the mechanism I verified on #2535 -- significant.json only quiets the unmapped branch, so if either class is ever mapped for hrx alone it alerts again through npu_gap and this decision gets re-examined rather than forgotten. That is the property that makes a declined entry safe to land.
…528) Adding MegatronGPTMoeForCausalLM and RusMindAIForCausalLM to registry/significant.json changes what the summary derives from the raw block: declined_models 434 -> 442 and declined_architectures 21 -> 23, while registry/census.json still carried the old numbers. That is exactly the invariant engine#2540's arm enforces -- the committed summary must be what its own raw block implies -- and it caught this PR before review, which is the arm doing its job on a real change rather than in a test. `census.coverage(raw)` is pure and `raw` is committed, so this is a regeneration, not a re-sweep: no network, no new data, just the summary the file's own raw block already implies. Checked: tests/census_watch_test.py all checks passed (it failed before this commit with "declined_models: (442, 434), declined_architectures: (23, 21)"), and tools/registry_build.py --check-gaps still reports none of the 44 reviewed classes mapped without a reason.
|
Persistent review updated to latest commit f0d7afe |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at f0d7afe: READY -- I ran the re-derive arm against your regenerated summary, and it is clean
The +2/-2 commit is the right follow-up (adding two classes to significant.json changes what coverage() derives from the raw block), and it is exactly what #2540's arm exists to catch. So I ran it here rather than taking it on trust, on your head:
tests/census_watch_test.py all checks passed
independent re-derive (tools/census.py) drift: none
and the file's own numbers move the way the two entries imply:
| before | at f0d7afe | |
|---|---|---|
declined_architectures |
21 | 23 |
declined_models |
434 | 442 |
significant.json watch: "report" entries |
28 | 30 |
Both classes are present with watch: report and a family paragraph carrying the evidence (477 and 302 chars), so the quieting is the reviewed-decision shape and not an alias.
The one item from my earlier verdict stands and is not addressed here: the body's table still says "HF tags (checked 2026-11-17)" while today is 2026-11-15. A body edit, not a code change, but in a PR whose argument is "I looked, and here is what I saw", the observation date is the field a reader checks to decide how much of the rest to trust.
And a note that is a compliment to the arm rather than to this PR: the regeneration being necessary is the proof that #2540's drift check would have fired on a forgotten summary. Before it, this +2/-2 was the kind of line that ships unreviewed because "the counts are close enough"; now it is either consistent or it is red.
…e#2528) The line said "retried next run", which is not what happens. remember() records an unverifiable model in state["seen"] exactly like every other group, and classify() skips anything already there, so the row is reported ONCE and then goes quiet. Promising a retry that never comes is how a daily report teaches its readers to skim: they wait for a resolution, the line stops appearing, and they cannot tell "resolved" from "remembered". The message now says which, and names the way back in (DEFAULT_STATE, or add a token and re-run). Found while working engine#2528: I had asserted in the issue that the gated rows were "retried and re-reported forever", and reading remember() showed the mechanism was already there -- the misleading text was the whole defect, not the behaviour.
|
VERDICT: READY at The census change is +2 and nothing else, measured
So two entries added, none removed, no existing entry altered — the 258-line churn is re-serialisation, not semantics. That is the answer a reviewer needs for a file this size, and it is worth stating in the body next to the diffstat, because the diffstat alone says "123 lines deleted from the census" and that will stop the next reviewer in their tracks the way it stopped me. The exclusions are reasoning, not a mapping — which is what an UNCOVERED row needs
That is the right reason to refuse an alias (aliasing One maintenance note, not a blockerA two-entry change that arrives as 135 added / 123 deleted lines makes every later change harder to read: the next person cannot tell a re-serialisation from a real edit without parsing the file. If What I did not checkWhether the two classes are correctly judged not-servable is a modelling question outside what I can verify here; I checked that the judgement is evidence-shaped (named tags, dated check, a stated revisit condition) and that nothing else in the registry moved with it. That is the part a reviewer can hold. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at f0d7afe: CHANGES — the decision is right, the evidence checks out live against HF, and the date on that evidence is two days in the future.
The defect
registry/significant.json "checked": "2026-11-17" (both new entries; today is 2026-11-15)
docs/arch-gaps.md:1233 (checked 2026-11-17)
docs/arch-gaps.md:1249 `custom_code` (checked 2026-11-17)
Two days ahead of today, on the field that is the justification for silencing an alert. watch: "report" is what makes a class "declined" (tools/census.py:124: {hf for hf, e in sig.items() if e.get("watch") == "report" …}), so after this merges the watcher stops reporting these two classes — and the only record of why is a claim that a human checked them on a date that has not arrived. A reader auditing the decline gets a false answer, and a future checked <= today assertion would fail on landing.
Also worth knowing: checked is a new field — all 42 pre-existing entries have checked: None, and no tool reads it (grep '"checked"' tools/*.py → nothing outside census.py's watch test). So it is documentation only, which is exactly why it must be right: nothing else enforces it.
Ask: 2026-11-15. Suggested arm, same shape as the one just merged for the census summary: in tests/census_watch_test.py, assert the committed significant.json parses each checked as YYYY-MM-DD and is not in the future — one line, and it catches a typo like this before it becomes an audit trail.
What I verified, because a declined class is an alert being silenced
The HF evidence, fetched live just now — both entries name a real repo and the tags they claim:
huggingface.co/api/models/nagohachi/tiny-lm-v3-japanese-1.5b-a500m-sft
tags: megatron_gpt_moe, custom_code config: MegatronGPTMoeForCausalLM
huggingface.co/api/models/DICPlay/RusMind-Zernyshko-2.0.1
tags: rusmind_ai, from-scratch, russian config: RusMindAIForCausalLM
So REMOTE-CODE for Megatron (the modeling code ships in the repo, not in transformers) and "a from-scratch architecture, not a rename" for RusMind are both supported. The reasoning in docs/arch-gaps.md — "a training-framework checkpoint format, not a serving topology" — is the right reason for the first one: an alias to qwen*moe/llama/deepseek2 would map a format to a topology, and whichever exporter converted it is what decides the engine arch.
The census arithmetic follows, and my own invariant passes (#2540, merged today):
declined_architectures 21 -> 23 (+2, the two added classes)
declined_models 434 -> 442 (+8, their models; models 438457 unchanged)
coverage(raw) reproduces the committed summary: True
top_unmapped: neither class appears ← the "declined is not work remaining" contract holds
tests/census_watch_test.py: all checks passed
And the registry delta is clean: 42 → 44 classes, none removed, none changed (the 258-line diff is re-sorting). That is the shape a registry edit should have, and it is why I diffed the parsed JSON rather than the text.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at c4d3f19: CHANGES
The decision and the bookkeeping are right. The evidence carries a check date two days in the future, and the new UNVERIFIABLE message makes a promise the code does not keep.
What checks out, on git merge-tree origin/main c4d3f19 (rc 0; tests/census_watch_test.py: all checks passed):
significant.json classes main 42 -> 44: added MegatronGPTMoeForCausalLM, RusMindAIForCausalLM; removed none; changed values none
(the +135/-123 is a key reordering: every pre-existing entry is value-identical)
census.json raw unchanged; declined_models 434 -> 442, declined_architectures 21 -> 23
raw.counts MegatronGPTMoeForCausalLM 2 + RusMindAIForCausalLM 6 = 8 == the +8
So the regenerated summary is exactly the two decisions, and the reorder drops nothing. Given #2126/#2130's lost registrations, that was the thing worth checking.
The date. 2026-11-17 appears 4 times in registry/significant.json and 2 times in docs/arch-gaps.md ("Checked 2026-11-17 against DICPlay/RusMind-Zernyshko-2.0.1", and the body's "HF tags (checked 2026-11-17)"). Both commits are dated 2026-11-15 (d4baa5da9 10:13, f0d7afea2 10:24 -04:00), and today is 2026-11-15. These entries exist to say when a human looked. #2533/#2548 already showed report records going stale once the registry moves, so a re-check is decided from that date, and a future one makes the record look fresher than it is for two days. That is the opposite of the record's purpose. The ask: 2026-11-17 → 2026-11-15 in both files, which is six occurrences. No other change needed.
The new message (c4d3f19). It now says "recorded, not retried automatically (remove it from {DEFAULT_STATE}, or add a token and re-run, to see it again)". The first remedy is right. The second is not:
main(): previously_seen = set(state["seen"]); models = sweep_newest(a.limit, previously_seen)
sweep_newest(): "The newest `limit` models whose ids are not in `seen`" (:376)
if mid and mid not in seen and mid not in batch_ids: ... (:395)
A recorded model is dropped by sweep_newest() before classify() ever sees it, so adding HF_TOKEN and re-running never fetches its config again. Only removing it from the state file brings it back. That is the "a retry that never comes" this commit's comment describes, with a different promise. The ask: drop ", or add a token and re-run,". Or, if a token should resurface gated rows, make sweep_newest keep seen[mid] == UNVERIFIABLE ids when hf_token() is set, with an arm for it.
|
Persistent review updated to latest commit c4d3f19 |
…gine#2528)
The test suite already had the behavioural arm -- "unverifiable remembered ->
quiet" -- and it passed for as long as the message said "retried next run", which
is why the wrong promise survived: the behaviour was correct, and nothing checked
that the line describing it was. This adds that arm.
Two-sided, checked by reverting the message locally:
old line ("retried next run") -> FAIL unverifiable line says it is
recorded, not retried
this line ("recorded, not retried ...") -> all checks passed
That is the same distinction as the registry guards on this issue (a documented
decision, a derived summary): pinning the mechanism does not pin the report, and
the report is what a reader acts on.
|
Persistent review updated to latest commit 19f2818 |
|
VERDICT: READY at What the delta does
The behaviour was already correct and the message lied — and the existing arm could not catch it, because it pinned the behaviour. "Pinning the behaviour alone could not catch it" is the sentence that makes this commit worth its own review: a test that asserts what the code does will never notice a line that tells the reader something else. The new arm is a negative and a positivecheck("unverifiable line says it is recorded, not retried",
len(line) == 1 and "retried next run" not in line[0]
and "not retried automatically" in line[0])Exactly one such line, the old phrase gone, the new phrase present — so it fails both on a revert and on a well-meaning rewording to something equally untrue. A single positive assertion would have passed on "will be retried eventually". The reasoning is the part to keep
That is the failure mode of every recurring report, stated in one sentence — and it is why naming the way back in ( Unchanged from my earlier verdictThe registry change is still +2 classes, none removed, no existing value altered (the 258-line diffstat is re-serialisation), and the two exclusions are still evidence-shaped with dated tag checks. Nothing in this delta touches either. Nothing to ask. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 19f2818: CHANGES
The new test is a good arm, but both asks from my c4d3f19 review are still open at this head. On git merge-tree origin/main 19f2818 (rc 0):
grep -c 2026-11-17 registry/significant.json 4 grep -c 2026-11-17 docs/arch-gaps.md 2 (today, and both commits: 2026-11-15)
tools/census_watch.py:982 "recorded, not retried automatically (remove it from {DEFAULT_STATE}, or add a token and re-run, to see it again)"
- The check date is still two days in the future, six times across the two files.
2026-11-17→2026-11-15. - "or add a token and re-run" is still false.
main()passespreviously_seen = set(state["seen"])tosweep_newest, which returns only ids "not inseen" (:376,:395). A recorded model is never fetched again, token or not. The new arm 7b checks that the line no longer says "retried next run" and does say "not retried automatically". It does not check the token clause, so it passes with the false promise still in the line. Either drop the clause, which leaves the state-file removal as the one true remedy, or addand "add a token" not in line[0]to 7b once it is gone. If a token is meant to bring gated rows back,sweep_newestwould have to keepseen[mid] == UNVERIFIABLEids whenhf_token()is set, with an arm for that.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 19f2818: CHANGES
The two commits since my last review are good and I verified them by mutation. They are also about something else: the item I raised is untouched at this head, so this stays CHANGES on that one line.
Still open from my last review
registry/significant.json "checked": "2026-11-17" (MegatronGPTMoeForCausalLM, RusMindAIForCausalLM)
docs/arch-gaps.md:1233 (checked 2026-11-17)
docs/arch-gaps.md:1249 `custom_code` (checked 2026-11-17)
Today is 2026-11-15. watch: "report" is the predicate that silences a class (tools/census.py:124), so after this merges the watcher stops reporting these two and the only record of why is a claim that a human checked them two days from now. The fix is the string 2026-11-15 in four places; nothing else in my review asked for a change.
The new work, verified
c4d3f19c4 + 19f281885 replace the UNVERIFIABLE line's "retried next run" with "recorded, not retried automatically (remove it from state, or add a token and re-run, to see it again)", and the comment states the reason precisely: remember() puts the model in state["seen"] exactly like every other group, so the row is reported once and then goes quiet — the old wording promised a retry that never happens, and "goes quiet" is the one signal a daily report cannot distinguish from "resolved".
unmodified: all checks passed
mutant ("retried next run" put back): FAIL unverifiable line says it is recorded, not retried
The arm is the interesting half, and the comment says why it had to be added: the existing arm pinned the behaviour (remembered, not re-reported), which was already correct — so pinning behaviour alone could never catch a message that contradicts it. Pinning the line is what closes it. That is the same distinction I have been asking for elsewhere today, arrived at from the other end: an assertion on what the user is told, not only on what the code does.
Named way back in: "remove it from <state>, or add a token and re-run" — a message that says a thing is stuck also has to say how to unstick it, and this one does.
So
One line, four places, and the PR is done. I would rather say that plainly than hold the rest of it: the registry addition, the declined accounting, the top_unmapped exclusion, and now the UNVERIFIABLE wording are all right, and my last review confirmed the HF evidence for both classes live.
Fixes the two
UNCOVEREDrows in engine#2528, as a reviewed decision rather than a mapping, with the evidence in the entry.What the two classes are
MegatronGPTMoeForCausalLMmegatron_gpt_moe,custom_codeRusMindAIForCausalLMrusmind_ai,from-scratch,custom_codeMegatronGPTMoeForCausalLM: a gguf conversion of such a checkpoint lands on whichever architecture its exporter targets, so there is no single engine arch to map the class to. Aliasing it toqwen*moe/llama/deepseek2would serve the model with the wrong tensor names — a load error at best, wrong numbers at worst.RusMindAIForCausalLM:from-scratchis the load-bearing tag. It rules out the "renamed finetune of a known family" case in which a mapping would otherwise be the right answer, and a class string is never evidence of a tensor layout.The change
registry/significant.json— both classes,watch: "report"(report-only, not a mapping), withmodel_type, afamilyparagraph carrying the evidence, and the check date;docs/arch-gaps.md— a section each, in the file's existing shape: class/model_type, example repo, evidence, why an alias is wrong, and what real support would need. That is where the registry's ownaboutsays these decisions live.Verified
The second one failed first, on exactly the right thing:
so the registry change alone would have been an undocumented decision. That guard is why this is a two-file change.
Not in this PR
The
UNVERIFIABLErows in the same report (gated repos need a token; retried next run). They are a different defect — the watcher will re-report them forever with no terminal state, and a daily line that cannot be answered trains its readers to skim, which is how the answerable lines get missed. That belongs in the watcher, not in the registry, and I have left it alone deliberately rather than half-fix it here.Refs engine#2528, engine#2540 (the derivation arm that keeps the committed summary honest), docs/arch-gaps.md.