Skip to content

Census 2026-11-15 - #2533

Merged
bong-water-water-bong merged 1 commit into
mainfrom
census/2026-11-15
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
census/2026-11-15

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

438457 text-generation models on HF, 345360 with an architecture (3071 architectures).

models share
mapped (a backend's code accepts the architecture) 326746 94.61%
checked (passed serve_e2e on Strix Halo) 219662 63.6%
backend mapped checked
hrx 94.61% 63.6%
npu 55.15% 9.45%

Unmapped architectures with the most models:

  • Step1MoEForCausalLM: 2882
  • ParlerTTSForConditionalGeneration: 1583
  • ChessForCausalLM: 970
  • LlavaLlamaForCausalLM: 453
  • OpenLMForCausalLM: 266
  • DynamicAlibiForCausalLM: 245
  • DynamicForgettingForCausalLM: 240
  • BertLMHeadModel: 212
  • BartForConditionalGeneration: 176
  • MobileLLMForCausalLM: 176

…rchitectures): mapped 326746 (94.61%), checked 219662 (63.6%)
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 1 🔵⚪⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Data Consistency

The census data includes new architectures and model counts that may not be fully validated against the actual backend support. The increase in checked_models for the NPU backend from 32621 to 32645 suggests a potential discrepancy in the test coverage or a miscount in the serve_e2e validation process. This needs verification to ensure that the reported numbers accurately reflect the hardware's capabilities and test results.

"mapped": 190458,
"checked": 32645,
"mapped_pct": 55.15,

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY-if-reproducible 2533

The headline number is the useful one and it is not the total: "438,457 text-generation models on HF, 345,360 with an architecture (3071 architectures)" — the 79 % with an architecture is what decides how much of the corpus the census can classify at all, and the 3071 architectures is the scale the mapping work has to cover. A census whose value is a ratio needs the ratio stated, not just the counts.

The ask, and it is the same one I make of every measurement that will be quoted later: record how the numbers were produced — the HF API query or dataset revision, the date, and whether text-generation is a pipeline tag filter or a search. A number in a registry file becomes a citation, and a citation whose query is not written down cannot be re-checked when someone disputes it six weeks later. One comment block at the top of registry/census.json, or a line in the PR description pointing at the script.

Three more:

  1. the delta matters more than the snapshot — say what changed since the previous census (models added, architectures added/removed). A registry that only ever holds the current number cannot show whether the gap is closing, which is the only question the census exists to answer;
  2. an architecture count of 3071 needs a tail statement — how many of those are covered by the existing mapping, and what the long tail looks like (top-10 share). The number alone invites the wrong conclusion, that 3071 mappings are needed;
  3. census.json should carry the generation date as data, so a stale census is visible without reading the file's git history.

And one caution specific to this repo: a census is an allowlist input. #2529 just established that registry/flm_answers.json answers a class, and the failure mode of such a registry is that an entry outlives its truth. If this file feeds that decision, the same staleness rule applies.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the numbers reconcile, and two of them will be misread without a denominator

I diffed registry/census.json at this head against main key by key rather than reading the table, because a data refresh is only reviewable that way:

base head
coverage.models 437950 438457
coverage.with_architecture 345055 345360
coverage.mapped_models 326464 326746
coverage.checked_models 219492 219662
coverage.architectures_seen 3064 3071
coverage.declined_architectures 20 21
coverage.declined_models 430 434

Every printed figure matches the file, architectures_seen moved by exactly the 9 new / 2 gone keys in raw.counts (the two gone are Inkling2ForConditionalGeneration, ZenithForCausalLM), and nothing was hand-edited that I can see: the diff is counters plus those keys.

Two things a reader will get wrong, both cheap to prevent:

  1. The two share columns have different denominators and the table does not say so. 94.61% is mapped_models / with_architecture = 326746/345360; over models it is 74.55%. Same for 63.6% (checked / with_architecture; over all models it is 50.1%). I ran the other division first and got a number that looks like a 20-point census bug — the row header says models, so the honest fix is a column header (share of models with an architecture) rather than a footnote.
  2. checked_pct went down (63.61 -> 63.60) while checked_models went up (219492 -> 219662) -- because the denominator grew faster. Next to an increasing count, a decreasing percentage reads as a regression, and one clause ("the denominator grew by 407 while the checked count grew by 170") turns it into the good news it is.

And one ask that is not cosmetic: name the +1 declined architecture / +4 declined models. declined_* is the only column in this file that reflects a human decision (registry/significant.json, watch: "report") rather than a crawl, so it should be attributable by class in the body. Here no decl file changed, so the rise must be models of an already-declined class appearing -- a fine outcome, stated in one line instead of left as a moving counter that nobody can trace.

Nothing else: raw is only reordered, top_unmapped keeps its shape, the coverage.definitions prose is unchanged, and Step1MoEForCausalLM at 2882 models is correctly the largest unmapped class rather than a smaller one that merely grew fastest.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 2fc9fc9: READY

Verified offline that this is a regeneration and not a hand-edit — and it incidentally repairs a real drift already on main.

The check that matters

tools/census.py has coverage(raw), the pure function that derives the summary block from the committed raw block. Recomputed from each file's own raw:

origin/main: coverage(raw) -> models=437950 mapped=326464 checked=219492   matches file: False
HEAD:        coverage(raw) -> models=438457 mapped=326746 checked=219662   matches file: True

So on this head the summary is exactly what the committed raw data implies — the file cannot have been edited by hand, because a hand edit would have to recompute two percentages and eight counters consistently. Arithmetic also holds both sides (326746/345360 = 94.61, 219662/345360 = 63.60), and every count moves up together (models +507, architectures_seen +7, mapped +282, checked +170), which is what a day of new HF uploads looks like.

main has a drift this PR fixes

main mismatch: declined_models  file=430   raw-recompute=434

main's census.json shipped a summary that disagrees with its own raw block by four declined models. Nobody would ever see that by reading the diff — it took a recompute to find. This PR's regeneration corrects it as a side effect, and it is the reason I am reading coverage(raw) rather than eyeballing the numbers: the summary is a derived artifact, and the only proof that a derived artifact is right is re-deriving it.

Suggestion, not a request: coverage(raw) == coverage is a one-line invariant, and it would have caught the four-model drift the day it landed. tests/census_watch_test.py already loads the committed file and asserts four things about it; this would be a fifth.

python3 tests/census_watch_test.py → all checks passed.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict. 2fc9fc9 has its two. This answers the review's ask to name the +1 declined architecture / +4 declined models, and adds two checks on the file:

The +1 / +4 is SegaForCausalLM, 4 models. It was reviewed not-an-alias in registry/significant.json by engine#2435 (3ac4a5e19), after the 11-14 census was generated. Today's declined set, watch: "report" and unmapped in architectures.json, summed over the old file's raw.counts, already gives 434. Every declined class has the same count in both files. So no crawl moved: the decision changed, and the census caught up.

Recomputed from the file itself (merge tree):

mapped 326746 / with_architecture 345360 = 94.61   checked 219662 / 345360 = 63.60   npu 55.15 / 9.45   -- all match
top_unmapped (25): none mapped in architectures.json, none a significant.json class

Two small things the numbers show:

  • declined_architectures 21 counts the 21 unmapped report classes, but only 19 have models. DiacritizerModel and BlindForCausalLM are at 0 in raw.counts. That is consistent with declined_models, but the definition string ("a human reviewed the class …") reads as "seen".
  • 7 report classes are mapped now: DeepseekV41ForCausalLM, HrmTextForCausalLM, DFlash2DraftModel, MuseSparkForConditionalGeneration, Glm5NextForConditionalGeneration, GraniteSwitchForCausalLM, Lfm3ForCausalLM. The census already ignores them because mapping wins, but their "deliberately not mapped" records are false today. That is the staleness census: new HF model class uncovered by the registry #2528 hit in census_watch, so worth a follow-up that drops or re-labels them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant