Skip to content

Census 2026-11-14 - #2430

Merged
bong-water-water-bong merged 1 commit into
mainfrom
census/2026-11-14
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
census/2026-11-14

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

437950 text-generation models on HF, 345055 with an architecture (3064 architectures).

models share
mapped (a backend's code accepts the architecture) 326464 94.61%
checked (passed serve_e2e on Strix Halo) 219492 63.61%
backend mapped checked
hrx 94.61% 63.61%
npu 55.14% 9.45%

Unmapped architectures with the most models:

  • Step1MoEForCausalLM: 2882
  • ParlerTTSForConditionalGeneration: 1583
  • ChessForCausalLM: 970
  • LlavaLlamaForCausalLM: 453
  • OpenLMForCausalLM: 266
  • DynamicAlibiForCausalLM: 245
  • DynamicForgettingForCausalLM: 240
  • BertLMHeadModel: 212
  • BartForConditionalGeneration: 176
  • MobileLLMForCausalLM: 176

…rchitectures): mapped 326464 (94.61%), checked 219492 (63.61%)
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Data Consistency

The census data includes new architectures and model counts that may not have been fully validated against the hardware backend capabilities. Specifically, the increase in checked_models for the npu backend from 32594 to 32621 suggests a potential discrepancy in the serve_e2e testing coverage, which should be confirmed with actual hardware testing results.

"checked": 32621,
"mapped_pct": 55.14,
Architecture Mapping

The top_unmapped list has been updated, removing some architectures that were previously listed. It's important to verify that these changes accurately reflect the current state of architecture support and that no architectures are being incorrectly marked as unmapped due to incomplete or outdated data.

{
 "architecture": "MixFormerSequentialForCausalLM",
 "models": 156
},
{
 "architecture": "K2HorizonForCausalLM",
 "models": 155
},
{
 "architecture": "RobertaForCausalLM",
 "models": 139
},
{
 "architecture": "NanochatGPTForCausalLM",
 "models": 126
},
{
 "architecture": "BartForCausalLM",
 "models": 100
},
{
 "architecture": "LlavaQwen2ForCausalLM",
 "models": 97
},
{
 "architecture": "BioGptForCausalLM",
 "models": 86
},

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and the one number worth the body is the gap between the two columns

94.61% mapped vs 63.61% checked is the census's most useful output, and it is not the one the table leads with. Mapped means a backend's code accepts the architecture; checked means it passed serve_e2e on Strix Halo. 31% of the surveyed models are claimed but never run -- and that is precisely the gap engine#2427 is about: a registry entry is a claim that the engine runs an architecture, and a census row is a claim that a model exists. The two columns measure the two claims separately, which is the right pair to publish.

Two small asks, both about the delta rather than the snapshot:

  1. Say what moved since yesterday, not just what the total is. +73/-69 suggests the generated tables were rewritten, but if mapped rose by a percentage point, the reason matters more than the number: a backend started accepting a class (good), a class was removed from a backend (a refusal regression -- and exactly the kind of thing guard_loss_check in #2429 exists to find in a diff), or the upstream pin moved and the registry was regenerated without anyone deciding anything. One line naming which.
  2. Name the models behind a shrinking checked share. If checked is a percentage, an unchanged absolute number with a growing denominator reads as a regression that is not one. Percentages and counts together, or the share alone with the count in a footnote -- either is honest, but the reader should not have to guess which moved.

Nothing else: a regenerated daily table does not need prose, and the backend breakdown (hrx 94.61/63.61, npu 55.14/9.x) is the part that will actually be quoted.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 38f1959 — with one number that needs a sentence before merge

The data moves in a direction worth reading carefully:

before after change
models 437,432 437,950 +518
with_architecture 344,820 345,055 +235
mapped_models 325,383 326,464 +1,081
declined_models 828 430 −398 (−48 %)
declined_architectures 26 ? ?

Most of the movement is fine and self-consistent: mapped rose by more than declines fell, so the declines were resolved into mappings rather than dropped, and the percentages track (326,464 / 345,055 = 94.62 %, matching the stated 94.61 %).

But a 48 % drop in declined_models in one day is not ordinary drift, and it is exactly the class this registry has been burned by before — engine#2369 lost models, with_architecture, declined_models and architectures_seen from census.json, which is why registry_loss_check exists and why it enumerated that case. Here the keys are present with new values, so this is not that failure; but a reader cannot tell from the diff whether

  • 398 models that were declined are now mapped (good — say so, and say what changed: a new backend, a fixed mapping, an uploader re-tagging), or
  • 398 models stopped being declined for a weaker reason — an architecture list that no longer trips the decline rule, a narrower definition, a census that stopped attempting them.

The ask is one line in the description: what caused the decline count to halve. A census PR is a claim about the world, and the interesting part of it is never the totals — it is the 398 models that changed category. If they were mapped, that is a win worth recording; if the rule changed, that is a measurement change and the totals are not comparable to yesterday's.

Two smaller things: state whether the file is a clean regeneration of census_watch.py (in which case the diff is reproducible and a reviewer can re-run it), and check that declined_architectures and architectures_seen both still exist — the diff removes 23 key lines and re-adds them, so a key that failed to come back would hide among the churn.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY as data, with one thing missing that would make it reviewable instead of merely plausible.

The diff is confined to registry/census.json — no code, no schema change — and the numbers move coherently (437,950 models; 94.61 % mapped; 63.61 % checked; hrx vs npu per backend). That is what a regenerated census should look like.

What is missing is the provenance. A data PR is judged on one question — was this produced by the generator, or typed by a human? — and nothing in the body answers it. +73/−69 in one JSON file is exactly the shape a hand-edit and a regeneration both take, and the two are indistinguishable by inspection. One line closes it:

python3 tools/…/census.py --org … --out registry/census.json
census written … (n models, m architectures)

with the generator's own summary line pasted, plus the commit/date of the HF snapshot it read. Three reasons that is worth more than it looks:

  1. The mapped/checked percentages are claims about someone else's data. HF's model count changes daily, so 94.61 % is only meaningful next to the snapshot it was measured on. Without that, the next reader cannot tell a real 0.3 % drift from a different day's catalog.
  2. A census that cannot be reproduced is a census you have to trust. The whole value of the mapped-vs-checked split is that it is mechanical — a backend's code accepts the architecture, or serve_e2e passed. If the file can only be refreshed by whoever last refreshed it, that property is lost quietly.
  3. It makes the delta reviewable. With the generator line, a reviewer can re-run it and diff; without it, the only check available is "do these numbers look plausible", which is how a wrong census survives.

One ask beyond that: if there is a scheduled job or a lane that owns the census, say so in the body (name it). Tonight several rows were refreshed by hand because nobody knew whether a timer existed — and a data file with no owner is one that goes stale without anyone noticing, which is the same false-green this repo has spent the day removing from its checks.

Nothing else objects: the per-backend table (hrx 94.61/63.61 vs npu 55.14/9.45) and the top-10 unmapped list read as generated rather than curated, and the ordering by count is what a generator would emit.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 38f1959 — the arithmetic checks; one counter moves 48 % with its inputs unchanged and the body does not mention it

Recomputed from the file, not the prose:

mapped   326464 / 345055 = 94.61 %    body says 94.61 %   ✓
checked  219492 / 345055 = 63.61 %    body says 63.61 %   ✓
npu      190268 / 345055 = 55.14 %    32621 / 345055 = 9.45 %   ✓
hrx row  = the headline row (326464 / 219492), the same identity I noted at engine#2370

and registry_loss_check --base origin/main --head pr2430 is quiet: 41 classes on both sides, no class, counter or test reference lost (16 counters changed, which is the rebuilt-census case and not a failure).

The thing worth one line in the body: declined_models halves while its input does not change.

main this PR change
models 437432 437950 +518
with_architecture 344820 345055 +235
mapped_models 325383 326464 +1081
checked_models 219379 219492 +113
declined_models 828 430 −398 (−48 %)
declined_architectures 26 20 −6

and registry/significant.json — the file that defines what "declined" means (27 report-marked classes, 7 mapped_ok, identical on both sides) — is not in this diff (git diff --name-only origin/main pr2430 = registry/census.json alone).

So a census refresh moved every coverage number slightly up and moved the declined population by half, with the decision set untouched. The two readings are "six declined architectures no longer appear in the HF population this census sampled" and "the census changed how it matches them", and they lead to different follow-ups: the first is nothing, the second is a silent change in what the number means.

This is the second time the counter has moved unseen — at engine#2370 I noted declined_models 828 invisible in the body, and it is now 430. One clause naming the declined population and why it moved would make the pair of censuses readable as a series rather than as two documents that disagree.

Nothing else: the file's own definitions block explains mapped/checked/declined correctly, and the per_backend block is internally consistent with the headline (hrx repeats it because hrx is the pinned backend, which the definitions say).

@bong-water-water-bong
bong-water-water-bong merged commit 49ae618 into main Nov 14, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the census/2026-11-14 branch November 14, 2026 12:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant