Repository navigation
Census 2026-11-13 - #2370
Census 2026-11-13#2370
Conversation
…rchitectures): mapped 325383 (94.36%), checked 219379 (63.62%)
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY at The delta, computed from both runsTwo readings, and both are useful:
The backend split is the number I would lead withThe NPU maps 55 % and has checked 9.45 % — so more than half the NPU's mapped surface has never been served on the device. That is the census's most actionable line, and it converts the row-level work ( Two asks
And the closure I asked for on #2368 belongs here
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at c1fbbed
A data refresh whose numbers are internally consistent, and I recomputed them rather than reading them out of the file:
models 437432 with_architecture 344820 architectures_seen 3060
mapped 325383 -> 94.36% (file says 94.36)
checked 219379 -> 63.62% (file says 63.62)
Both percentages derive from the counts, and they match the file's own mapped_pct/checked_pct -- so the body is quoting the pipeline, not a number someone typed. And the definitions block is what makes the file reviewable: mapped is "a property of the ARCHITECTURE, so every model of a mapped class counts as mapped whether or not that model was ever run", and checked is "the same per-architecture property, which is why the npu figure is essentially one architecture's model population". A 63.62 % "checked" headline would be misleading without that sentence, and it is there.
Three things the body would be stronger for, none blocking:
- The
hrxrow is the overall row, exactly --94.36% / 63.62%both. That is true (the default backend's acceptance is what "mapped" means), but a reader will read it as a copy-paste error in a table that exists to separate the two backends. One clause -- "hrx is the default backend, so its row is the headline" -- stops that. declined_models: 828is invisible in the body while the file documents it as a deliberate bucket (watch: "report", human-reviewed, excluded fromtop_unmapped). Without it,94.36 %mapped +19437unmapped (5.64 %) leaves a reader wondering where the 828 went.- The body's unmapped list is a sample of a sample: it shows 10 names, the file records
top_unmappedas a top-25 truncation of a larger unmapped set (those 25 sum to 8780 models of the 19437 unmapped). Saying "top 10 of the 25 recorded" is the difference between a summary and an excerpt -- the same distinction as "class 1 is 13, not 9" on #2204.
One check that came out clean and is worth stating: my first read of declined_architectures: 26 against 25 top_unmapped entries looked like an off-by-one, and it is not -- the 25 are the unmapped classes, and the declined ones are excluded from that list by the definition above. The file's own definitions.declined says so explicitly.
Ran: recomputed both percentages and the unmapped remainder from registry/census.json; cross-checked per_backend against the body's two rows (hrx and npu agree digit for digit). Not run: the census itself (it reads 437k HF models).
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at c1fbbed: READY
A data-only refresh, and the three things that can go wrong with one are all fine. I checked the arithmetic rather than the prose, because a hand-edited number is exactly what a table of percentages hides.
1. The shares recompute exactly from the counts — they were not typed:
mapped 325383 / 344820 = 94.3631% (table says 94.36)
checked 219379 / 344820 = 63.6213% (table says 63.62)
2. The stages are a nested-subset chain, which is the invariant a census can actually break:
models 437432 ≥ with_architecture 344820 ≥ mapped_models 325383 ≥ checked_models 219379
Each stage is a subset of the one before it and the numbers satisfy it. A refresh that reordered or double-counted a bucket would violate this even while every percentage still looked plausible.
3. It is a refresh, not a schema change: 1 file changed, 84 insertions(+), 74 deletions(-), top-level keys identical, and date advanced 2026-11-12 → 2026-11-13. The one thing a census PR must not do is quietly rename a field, because every consumer of registry/census.json would keep reading the old key and reporting an empty result rather than an error.
And the file documents its own terms, which is what makes the numbers auditable by someone who was not here when the census was designed:
mapped: the pinned backend's code accepts this HF architecture class (registry…)
plus architectures_seen 3060, declined_models 828, declined_architectures 26 — so a reader can tell "not mapped" apart from "mapped but not checked" without asking.
What I did not do, and why it is not a gap: I could not regenerate 437k HF models, so this review verifies the artifact's internal consistency and provenance, not its freshness. If tools/census.py records the query and the collection date inside the JSON — and date suggests it does — then the freshness claim is checkable later by whoever runs it, and that is the right division of labour for a run that costs hours of API calls.
No concerns.
437432 text-generation models on HF, 344820 with an architecture (3060 architectures).
Unmapped architectures with the most models:
Step1MoEForCausalLM: 2882ParlerTTSForConditionalGeneration: 1583ChessForCausalLM: 970LlavaLlamaForCausalLM: 453OpenLMForCausalLM: 266DynamicAlibiForCausalLM: 245DynamicForgettingForCausalLM: 240BertLMHeadModel: 212BartForConditionalGeneration: 176MobileLLMForCausalLM: 176