Skip to content

Census 2026-11-13 - #2370

Merged
bong-water-water-bong merged 1 commit into
mainfrom
census/2026-11-13
Nov 13, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
census/2026-11-13

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

437432 text-generation models on HF, 344820 with an architecture (3060 architectures).

models share
mapped (a backend's code accepts the architecture) 325383 94.36%
checked (passed serve_e2e on Strix Halo) 219379 63.62%
backend mapped checked
hrx 94.36% 63.62%
npu 55.13% 9.45%

Unmapped architectures with the most models:

  • Step1MoEForCausalLM: 2882
  • ParlerTTSForConditionalGeneration: 1583
  • ChessForCausalLM: 970
  • LlavaLlamaForCausalLM: 453
  • OpenLMForCausalLM: 266
  • DynamicAlibiForCausalLM: 245
  • DynamicForgettingForCausalLM: 240
  • BertLMHeadModel: 212
  • BartForConditionalGeneration: 176
  • MobileLLMForCausalLM: 176

…rchitectures): mapped 325383 (94.36%), checked 219379 (63.62%)
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 1 🔵⚪⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Data Consistency

The census data shows minor discrepancies in model counts for several architectures, such as ParlerTTSForConditionalGeneration (1583 vs 1585), Spark2_5ForCausalLM (153 vs 152), and Qwen3_5MoeForConditionalGeneration (236 vs 237). These differences may indicate inconsistencies in the data collection or processing pipeline, which should be investigated to ensure data integrity.

"models": 1583
Data Consistency

The census data shows minor discrepancies in model counts for several architectures, such as Qwen4ExpForConditionalGeneration (156 vs 154), Lfm3ForCausalLM (139 vs 135), and InternLM2ForCausalLM (121 vs 122). These differences may indicate inconsistencies in the data collection or processing pipeline, which should be investigated to ensure data integrity.

"Qwen4ExpForConditionalGeneration": 156,
Data Consistency

The census data shows minor discrepancies in model counts for several architectures, such as FlintForCausalLM (83 vs 82), DFlash2DraftModel (82 vs 81), and NemotronForCausalLM (82 vs 81). These differences may indicate inconsistencies in the data collection or processing pipeline, which should be investigated to ensure data integrity.

"FlintForCausalLM": 83,
Data Consistency

The census data shows minor discrepancies in model counts for several architectures, such as Ernie4_5_MoeForCausalLM (63 vs 62). These differences may indicate inconsistencies in the data collection or processing pipeline, which should be investigated to ensure data integrity.

"Ernie4_5_MoeForCausalLM": 63,
Data Consistency

The census data shows minor discrepancies in model counts for several architectures, such as HelixForCausalLM (41 vs 39). These differences may indicate inconsistencies in the data collection or processing pipeline, which should be investigated to ensure data integrity.

"HelixForCausalLM": 41,
Data Consistency

The census data shows minor discrepancies in model counts for several architectures, such as MiniMindForCausalLM (20 vs 18), Qwen4ExpForCausalLM (20 vs 19), and DoryForCausalLM (19 vs 18). These differences may indicate inconsistencies in the data collection or processing pipeline, which should be investigated to ensure data integrity.

"MiniMindForCausalLM": 20,

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at c1fbbed — and it is the first census in this queue that can show a delta, because the previous run is only a day old.

The delta, computed from both runs

                 2026-11-12        2026-11-13        delta
text-generation      436,930           437,432         +502
with an architecture 344,583           344,820         +237
architectures          3,050             3,060          +10
mapped               325,177           325,383         +206
checked (serve_e2e)  219,300           219,379          +79
mapped %              94.37             94.36      -0.01
checked %             63.64             63.62      -0.02

Two readings, and both are useful:

  • mapped grew by 206 while architectures grew by 10 — so ~20 new architectures' worth of models arrived and were mapped, which is the registry keeping up;
  • the percentages ticked down by 0.01–0.02 — because the denominator grew faster than the numerator. That is the honest signal: the registry is not falling behind, but it is also not gaining ground, and a percentage that moves down while every count moves up is exactly the case where a reader needs both.

The backend split is the number I would lead with

hrx  mapped 94.36%   checked 63.62%
npu  mapped 55.13%   checked  9.45%

The NPU maps 55 % and has checked 9.45 % — so more than half the NPU's mapped surface has never been served on the device. That is the census's most actionable line, and it converts the row-level work (#1786's families, the F9/D3 rows) into a number: it is not "some families are missing", it is 45 % unmapped and 90 % unchecked on that route.

Two asks

  1. put the delta in the body — I computed it from the previous run because this one does not carry it, and the two-line table above is what makes a census progress rather than a snapshot. It is also the answer to the obvious question ("is this better than yesterday?") that the body currently cannot answer;
  2. name the 10 new architectures — 3,050 → 3,060 in a day is a small, checkable set; listing them with their mapping status (mapped / unmapped) is one table and it is the whole story of the day's arrivals. The unmapped tail is already top-10'd by model count (Step1MoEForCausalLM 2,8xx first), so the new ten may not be in it — which is precisely why they need listing separately.

And the closure I asked for on #2368 belongs here

Skylar2ForCausalLM was the one !! UNCOVERED line of the 11-12 run and #2368 recorded it. This run should say 0 uncovered — or name what replaced it. A census whose !! UNCOVERED set is empty is the state two PRs were written to reach, and it takes one line to state.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at c1fbbed

A data refresh whose numbers are internally consistent, and I recomputed them rather than reading them out of the file:

models 437432   with_architecture 344820   architectures_seen 3060
mapped 325383 -> 94.36%   (file says 94.36)
checked 219379 -> 63.62%  (file says 63.62)

Both percentages derive from the counts, and they match the file's own mapped_pct/checked_pct -- so the body is quoting the pipeline, not a number someone typed. And the definitions block is what makes the file reviewable: mapped is "a property of the ARCHITECTURE, so every model of a mapped class counts as mapped whether or not that model was ever run", and checked is "the same per-architecture property, which is why the npu figure is essentially one architecture's model population". A 63.62 % "checked" headline would be misleading without that sentence, and it is there.

Three things the body would be stronger for, none blocking:

  1. The hrx row is the overall row, exactly -- 94.36% / 63.62% both. That is true (the default backend's acceptance is what "mapped" means), but a reader will read it as a copy-paste error in a table that exists to separate the two backends. One clause -- "hrx is the default backend, so its row is the headline" -- stops that.
  2. declined_models: 828 is invisible in the body while the file documents it as a deliberate bucket (watch: "report", human-reviewed, excluded from top_unmapped). Without it, 94.36 % mapped + 19437 unmapped (5.64 %) leaves a reader wondering where the 828 went.
  3. The body's unmapped list is a sample of a sample: it shows 10 names, the file records top_unmapped as a top-25 truncation of a larger unmapped set (those 25 sum to 8780 models of the 19437 unmapped). Saying "top 10 of the 25 recorded" is the difference between a summary and an excerpt -- the same distinction as "class 1 is 13, not 9" on #2204.

One check that came out clean and is worth stating: my first read of declined_architectures: 26 against 25 top_unmapped entries looked like an off-by-one, and it is not -- the 25 are the unmapped classes, and the declined ones are excluded from that list by the definition above. The file's own definitions.declined says so explicitly.

Ran: recomputed both percentages and the unmapped remainder from registry/census.json; cross-checked per_backend against the body's two rows (hrx and npu agree digit for digit). Not run: the census itself (it reads 437k HF models).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at c1fbbed: READY

A data-only refresh, and the three things that can go wrong with one are all fine. I checked the arithmetic rather than the prose, because a hand-edited number is exactly what a table of percentages hides.

1. The shares recompute exactly from the counts — they were not typed:

mapped   325383 / 344820 = 94.3631%   (table says 94.36)
checked  219379 / 344820 = 63.6213%   (table says 63.62)

2. The stages are a nested-subset chain, which is the invariant a census can actually break:

models 437432 ≥ with_architecture 344820 ≥ mapped_models 325383 ≥ checked_models 219379

Each stage is a subset of the one before it and the numbers satisfy it. A refresh that reordered or double-counted a bucket would violate this even while every percentage still looked plausible.

3. It is a refresh, not a schema change: 1 file changed, 84 insertions(+), 74 deletions(-), top-level keys identical, and date advanced 2026-11-12 → 2026-11-13. The one thing a census PR must not do is quietly rename a field, because every consumer of registry/census.json would keep reading the old key and reporting an empty result rather than an error.

And the file documents its own terms, which is what makes the numbers auditable by someone who was not here when the census was designed:

mapped: the pinned backend's code accepts this HF architecture class (registry…)

plus architectures_seen 3060, declined_models 828, declined_architectures 26 — so a reader can tell "not mapped" apart from "mapped but not checked" without asking.

What I did not do, and why it is not a gap: I could not regenerate 437k HF models, so this review verifies the artifact's internal consistency and provenance, not its freshness. If tools/census.py records the query and the collection date inside the JSON — and date suggests it does — then the freshness claim is checkable later by whoever runs it, and that is the right division of labour for a run that costs hours of API calls.

No concerns.

@bong-water-water-bong
bong-water-water-bong merged commit 3ec3401 into main Nov 13, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the census/2026-11-13 branch November 13, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant