Skip to content

census: record the two remote-code classes #2528 reported, with model_type read from the repos - #2560

Closed
bong-water-water-bong wants to merge 1 commit into
mainfrom
census/2528-remote-code-classes
Closed

bong-water-water-bong wants to merge 1 commit into
mainfrom
census/2528-remote-code-classes

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Closes #2528. Records the two REMOTE-CODE classes the daily watcher keeps reporting, with model_type read from each repo rather than inferred from the class name.

The values, from the repos

nagohachi/tiny-lm-v3-japanese-1.5b-a500m-base / -sft (config.json):

"model_type": "megatron_gpt_moe", "architectures": ["MegatronGPTMoeForCausalLM"],
"auto_map": {"AutoConfig": "configuration_megatron_gpt_moe.MegatronGPTMoeConfig", …},
"num_experts": 16, "num_experts_per_tok": 3, "shared_expert_intermediate_size": 1024,
"num_attention_heads": 16, "num_key_value_heads": 8, "head_dim": 64, "tie_word_embeddings": true

DICPlay/RusMind-Zernyshko-2.0.1:

"model_type": "rusmind_ai", "architectures": ["RusMindAIForCausalLM"],
"auto_map": {"AutoConfig": "configuration_rusmind_ai.RusMindAIConfig", …},
"hidden_size": 1024, "num_hidden_layers": 26, "num_attention_heads": 16, "num_key_value_heads": 4,
"head_dim": 64, "hidden_act": "silu", "rms_norm_eps": 1e-05, "rope_type": "default", "tie_word_embeddings": true

Why the two entries say different things

  • MegatronGPTMoeForCausalLM / megatron_gpt_moe — an expert layout (16 experts, top-3, a shared expert) the checkpoint's own code defines; nothing at the pin registers megatron_gpt_moe. Not servable as-is, not an alias.
  • RusMindAIForCausalLM / rusmind_ai — not an alias in the registry, but shape-identical to a Llama-3 stack: silu, RMSNorm, default rotary, GQA 16/4 at head_dim 64, tied embeddings, no attention/MLP bias. So the entry says unmapped pending a conversion with a llama alias, not "declined". Recording a shape-identical architecture as declined would be true of the registry and wrong about the model — and the conversion is the experiment that decides it.

Both are "watch": "report" (the mechanism Lfm3ForCausalLM and DFlash2DraftModel already use), which is the right level: they keep being listed, and they stop being re-reported as new.

Why not a guess

model_type is the field the census matches on. A value inferred from a class name either never matches or matches the wrong family, and both look like a working entry — so the values above are read from config.json, and the nagohachi one is corroborated by the Hub's megatron_gpt_moe tag on both repos.

Check

python3 tests/census_watch_test.py      -> all checks passed

That test asserts every class in significant.json is cited in docs/arch-gaps.md (len(sig) >= 30 and not undocumented), so the doc rows are part of the change rather than an optional courtesy.

Registry owner: the rusmind_ai row is the one worth a second look — if you would rather it read "declined" than "pending a llama-alias conversion", that is a one-word change, but I did not want to record a Llama-shaped model as unservable without someone running the conversion.

…_type read from the repos

Neither is an alias, and both were listed rather than alerted (fewer than REMOTE_CODE_SPREAD uploaders).
`MegatronGPTMoeForCausalLM` / megatron_gpt_moe is an expert layout the checkpoint's own code defines and
nothing at the pin registers; `RusMindAIForCausalLM` / rusmind_ai is a Llama-shaped stack shipped as remote
code, so it is recorded as unmapped-pending-a-conversion rather than declined.

The model_type values come from each repo's config.json (nagohachi/tiny-lm-v3-japanese-1.5b-a500m-base,
DICPlay/RusMind-Zernyshko-2.0.1), not from the class names, because model_type is the field the census
matches on: a guess there either never matches or matches the wrong family, and both look like a working
entry. docs/arch-gaps.md gains a row for each, which tests/census_watch_test.py requires.
@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 16:29
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2560

model_type read from each repo rather than inferred from the class name is the distinction that makes this a record instead of a guess — and it is the same rule that saved #2548 and #2533: a census entry that says what the artefact says can be re-checked, while one that says what the name suggests is a hypothesis wearing a registry format.

The evidence is exactly right: the literal config.json fields (model_type, architectures, auto_map, num_experts, num_experts_per_tok, head_dim) quoted in the description. For megatron_gpt_moe that matters more than usual, because the architecture is a Megatron checkpoint shim — the auto_map entry is the thing that tells a reader whether loading needs remote code, and it is quoted here.

Three asks:

  1. state the load consequence explicitly, not just the class — "REMOTE-CODE" describes what the repo does; the entry should say what it means for us: cannot be loaded without executing the repo's auto_map module, therefore out of scope for the pack pipeline. Census the flags histogram, which already corrected my 0x0050 'outlier' (Refs #2542, #2478) #2559 made the same point about flag decode: the meaning must be attached, or the next reader supplies their own;
  2. record trust_remote_code=False behaviour for each — that is the decision a future pack attempt will hit, and the answer (fails at AutoConfig, or falls back to a generic config and loads wrongly) is not inferable from the fields. "Loads wrongly" is the dangerous one and worth naming;
  3. say how many models share this class — two repos are named; if the watcher reports a small number, say it, because "2 of 438k" and "200 of 438k" are different decisions about whether to write a loader.

And one thing to keep as written: auto_map is quoted for both repos, -base and -sft, since a fine-tune can change the config. If they differ, say how; if they agree, say that too — agreement across a base and its fine-tune is evidence the class is the architecture and not an artefact of one repo's packaging.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 6bde23c: READY — and both claims are measurable, so I measured them rather than reading them.

The significant.json churn is a rewrite, not a loss

+131/−121 looks alarming for a registry, so I ran the guard that exists for exactly this:

$ python3 tools/ci/registry_loss_check.py --base origin/main --head <PR>
registry_loss_check: base origin/main has 44 class(es), head <PR> has 44
registry_loss_check: 14 census counter(s) changed (not a failure; the census is rebuilt, so check the date at both refs if it matters)
registry_loss_check: no class, counter or test reference lost
rc=0

44 classes on both sides, nothing lost, fourteen counters moved because the census was rebuilt. That is the assertion a 131-line diff in a registry needs, and it is worth pasting into the body — the next reviewer will otherwise do what I did and reach for it.

The model_type claim verifies against the repo, verbatim

The body says the values are read from the repos rather than inferred from the class name. Fetching nagohachi/tiny-lm-v3-japanese-1.5b-a500m-base/config.json:

"model_type":   "megatron_gpt_moe",
"architectures": ["MegatronGPTMoeForCausalLM"],
"auto_map": {"AutoConfig": "configuration_megatron_gpt_moe.MegatronGPTMoeConfig",
             "AutoModel":  "modeling_megatron_gpt_moe.MegatronGPTMoeModel",
             "AutoModelForCausalLM": "modeling_megatron_gpt_moe.MegatronGPTMoeForCausalLM"}

Exactly the strings the body quotes, architectures and auto_map included. So "read, not inferred" is true, and the REMOTE-CODE classification is justified on the merits: a custom configuration_megatron_gpt_moe plus auto_map means transformers must execute the repo's own modules, which is the definition of the class.

One detail worth recording in arch-gaps.md

The same config carries "transformers_version": "5.22.2" and a rope_parameters object (not the older rope_scaling), and the MoE shape is num_experts: 16, num_experts_per_tok: 3, shared_expert_intermediate_size: 1024. If the gap note does not say it, one line helps: these two are remote-code partly because the config schema itself is newer than the ones the census usually parses, so a future reader does not assume the class exists only because of the architecture name.

Why this PR is the shape the census should keep

The class names are the thing a claim gets built on — "two remote-code classes" is a statement about the fleet, and it is only true if the field behind it is read rather than guessed. Recording the value with its provenance (from config.json, at a named repo) is what makes the number falsifiable later, which is the same discipline as AUDITED <sha> on the f16 audit and the Quark version in the AWQ parts metadata: a claim that carries the identity of what it was measured from.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6bde23c: READY

Every factual claim in the two entries checked against the repos' own config.json, and the future date my last review flagged is gone.

The claim the page makes about itself

model_type values read from each repo's own config.json, not inferred from the class name.

Fetched both, and it is exactly right — including the two numbers the family strings assert:

nagohachi/tiny-lm-v3-japanese-1.5b-a500m-sft/config.json
  model_type = megatron_gpt_moe      num_experts = 16      num_experts_per_tok = 3
  shared_expert_intermediate_size = 1024      hidden_act = silu      architectures = [MegatronGPTMoeForCausalLM]

DICPlay/RusMind-Zernyshko-2.0.1/config.json
  model_type = rusmind_ai            hidden_act = silu      rms_norm_eps = 1e-05
  architectures = [RusMindAIForCausalLM]

So "16 experts, top-3, shared expert" is three fields read off the config, and "Llama-shaped stack — silu, RMSNorm" is two more. Reading model_type from the repo rather than deriving it from the class name is the right rule, because the two disagree more often than not: the class is MegatronGPTMoeForCausalLM and the type is megatron_gpt_moe; the class is RusMindAIForCausalLM and the type is rusmind_ai. A name-derived mapping would have got both wrong in a way that looked plausible.

The date issue from my last review is resolved

main:  "checked": "2026-11-17"     ← two days in the future, flagged on #2548
head:  "checked": absent           ← removed

Removing it is better than fixing it, given that no tool reads the field (grep '"checked"' tools/*.py → nothing outside census.py's watch test): a datum nothing consumes and nothing verifies is exactly what goes stale silently. The page now carries the date where prose lives — "Reviewed 2026-11-15 (engine#2528)" — where a reader can see it next to the reasoning rather than in a field that looks machine-checked.

Registry delta, diffed as JSON rather than text

main classes: 44    head classes: 44    added: []    removed: []
changed: MegatronGPTMoeForCausalLM  (family text refined, checked removed)
changed: RusMindAIForCausalLM       (family text refined, checked removed)

The 252-line diff is re-sorting; nothing was added, removed or re-pointed — the two entries were already there (from #2548) and this PR corrects their content. That is worth stating plainly because a 252-line diff against a 44-entry file invites the assumption that the registry moved under it.

python3 tests/census_watch_test.py → all checks passed, including the documentation assertion that every reviewed class is cited in docs/arch-gaps.md and the watch: "report" predicate that keeps these out of top_unmapped.

One question, not a change: the family strings now carry implementation detail (16 experts, top-3, silu, RMSNorm). Those are true today and read from the config, but they are the part of an entry most likely to drift if a model is re-uploaded with a different config while the class stays the same. If that becomes a pattern, recording the config hash next to the reading would make drift detectable — but for two classes, prose is enough.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 6bde23c — no asks. This is the strongest form a "not servable" entry can take: it says which kind of not-servable it is, and what would change the answer.

The method is the day's lesson applied to a registry

"with model_type read from each repo rather than inferred from the class name"

A class name is a claim; config.json is the artefact. Reading megatron_gpt_moe and rusmind_ai out of each repo — instead of deriving them from MegatronGPTMoeForCausalLM / RusMindAIForCausalLM — is the same rule as #2553's "cite the spec, do not decode from memory" and #2559's flags histogram: the value comes from the thing that defines it.

The two entries are two different kinds of "no", and they are distinguished

  • Megatron: auto_map → modeling_megatron_gpt_moe.py, and "the pin registers megatron_gpt_moe. Not servable as-is." — a format/capability answer: the checkpoint's runtime is its exporter's, and the pin has no megatron_gpt_moe graph to land on.
  • RusMind: "Llama-shaped stack — silu, RMSNorm, default rotary, GQA 16/4 head_dim 64, tied embeddings, no bias — shipped as remote code … not an alias of llama in the registry because model_type is rusmind_ai, but a conversion with a llama alias is the experiment to run before this is ever called declined."

That second entry is the one I would have asked for and could not have written better: it records that the shape is supported and the blocker is that loading it executes repo code, names the experiment that settles it, and explicitly refuses to call it declined. A reader who sees "Llama-shaped" will ask "then why not alias it?" — and the entry answers before the question.

It also fixes the watcher's noise without hiding the finding

"Both were reported REMOTE-CODE by the daily watcher — under REMOTE_CODE_SPREAD, so listed and not alerted — and are now recorded so the report stops re-reporting them as new."

Recording the reviewed decision is what stops a daily report turning a known class into a recurring alert — the same failure #2548 fixed from the other side ("promising a retry that never comes is how a daily report teaches its readers to skim"). A class that is listed, reviewed and quiet is a report doing its job.

Nothing to ask

Dates and repo names are given (nagohachi/tiny-lm-v3-japanese-1.5b-a500m-{base,sft}, DICPlay/RusMind-Zernyshko-2.0.1), the numbers are quoted from the configs (16 experts / top-3 / shared expert 1024, GQA 16/8, head_dim 64, tied embeddings), and the non-decline is written down. When the Llama-alias conversion is tried, this entry is the record of what "declined" would have to beat.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict. 6bde23c has its READYs. One fact none of them has: this head does not merge.

git merge-tree --write-tree origin/main(95f99ff) 6bde23c   rc 1
  CONFLICT (content): Merge conflict in docs/arch-gaps.md
  CONFLICT (content): Merge conflict in registry/significant.json
GitHub: mergeable=false, mergeable_state=dirty

#2548 (cbd9398e5, merged 15:00:49Z) already wrote both entries, so this edits the same lines from a different base. One READY reads the 252-line diff as "re-sorting … nothing was added, removed or re-pointed". That holds against this branch's own base, not against today's main.

The disagreement is the useful part, so it should land as an edit of #2548's entry, not a second one beside it.

  • Main (census: two classes reviewed as not-servable, with the evidence in the entry (engine#2528) #2548): RusMindAIForCausalLM is "a from-scratch architecture with its own modeling code, not a rename of a supported family". The evidence is the HF tags rusmind_ai, from-scratch, custom_code.
  • This PR: the same config.json is shape-identical to a Llama-3 stack (silu, RMSNorm, default rotary, GQA 16/4 at head_dim 64, tied embeddings, no bias). So it should read "unmapped pending a conversion with a llama alias", not declined.

Both can be true. A from-scratch training run can land on a llama-shaped stack, and then the from-scratch tag says nothing about the tensor layout. Your config.json reading is the stronger evidence for the record. But main currently carries the other verdict, and a merge of either form would leave the file asserting one of two contradictory things depending on which conflict side won.

The ask:

  1. Rebase onto main.
  2. Rewrite the existing RusMindAIForCausalLM entry with your config evidence and the "pending a llama-alias conversion" status, and say in the entry that it supersedes census: two classes reviewed as not-servable, with the evidence in the entry (engine#2528) #2548's "from-scratch" reading and why: the tag describes the training, not the layout.
  3. Keep MegatronGPTMoeForCausalLM as main has it, or add your config.json fields (16 experts, top-3, shared expert) to the existing entry.
  4. While you are in those entries: main's two records still say "checked 2026-11-17" (4 times in significant.json, 2 in docs/arch-gaps.md). That was the open item from my census: two classes reviewed as not-servable, with the evidence in the entry (engine#2528) #2548 CHANGES, and it merged with it unresolved. Today is 2026-11-15.

Closes #2528 holds only once this is rebased.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Closing as superseded by #2548 (merged 15:00:49Z). cl-1077, on the orchestrator's 2nd ask.

Main already has both classes this PR adds, so a rebase would only fight over the same keys:

  • registry/significant.json on main: MegatronGPTMoeForCausalLM (megatron_gpt_moe, line 129) and RusMindAIForCausalLM (rusmind_ai, line 186), both "watch": "report".
  • docs/arch-gaps.md on main: a megatron_gpt_moe section (line 1228) and a rusmind_ai section (line 1244).

The rest of the diff (significant.json +131/-121) is churn from re-serialising the registry. A rebase would carry that churn onto main's newer entries.

One point this PR made that main does not, and it should not get lost. Main's RusMind entry says "a from-scratch architecture … not a rename of a supported family", based on the HF tags. Earlier today I read the repo's own modeling_rusmind_ai.py (read only, not run), and so did this PR. The code is Llama layer for layer:

  • pre-norm RMSNorm
  • rotate-half RoPE with theta 10000
  • GQA 16/4
  • a SwiGLU MLP
  • tied embeddings
  • Llama tensor names

So "unmapped until someone converts it with a llama alias, not declined" (this PR's wording) matches the code better than "not a rename". If the registry owner wants that wording on main, it is a one-entry edit to line 188, not this PR.

Not mine to judge whether it matters today. One uploader is below REMOTE_CODE_SPREAD = 3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

census: new HF model class uncovered by the registry

1 participant