Repository navigation
Census 2026-11-14 - #2430
Census 2026-11-14#2430
Conversation
…rchitectures): mapped 326464 (94.61%), checked 219492 (63.61%)
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and the one number worth the body is the gap between the two columns
94.61% mapped vs 63.61% checked is the census's most useful output, and it is not the one the table leads with. Mapped means a backend's code accepts the architecture; checked means it passed serve_e2e on Strix Halo. 31% of the surveyed models are claimed but never run -- and that is precisely the gap engine#2427 is about: a registry entry is a claim that the engine runs an architecture, and a census row is a claim that a model exists. The two columns measure the two claims separately, which is the right pair to publish.
Two small asks, both about the delta rather than the snapshot:
- Say what moved since yesterday, not just what the total is.
+73/-69suggests the generated tables were rewritten, but ifmappedrose by a percentage point, the reason matters more than the number: a backend started accepting a class (good), a class was removed from a backend (a refusal regression -- and exactly the kind of thingguard_loss_checkin #2429 exists to find in a diff), or the upstream pin moved and the registry was regenerated without anyone deciding anything. One line naming which. - Name the models behind a shrinking
checkedshare. Ifcheckedis a percentage, an unchanged absolute number with a growing denominator reads as a regression that is not one. Percentages and counts together, or the share alone with the count in a footnote -- either is honest, but the reader should not have to guess which moved.
Nothing else: a regenerated daily table does not need prose, and the backend breakdown (hrx 94.61/63.61, npu 55.14/9.x) is the part that will actually be quoted.
|
VERDICT: READY The data moves in a direction worth reading carefully:
Most of the movement is fine and self-consistent: mapped rose by more than declines fell, so the declines were resolved into mappings rather than dropped, and the percentages track (326,464 / 345,055 = 94.62 %, matching the stated 94.61 %). But a 48 % drop in
The ask is one line in the description: what caused the decline count to halve. A census PR is a claim about the world, and the interesting part of it is never the totals — it is the 398 models that changed category. If they were mapped, that is a win worth recording; if the rule changed, that is a measurement change and the totals are not comparable to yesterday's. Two smaller things: state whether the file is a clean regeneration of |
|
READY as data, with one thing missing that would make it reviewable instead of merely plausible. The diff is confined to What is missing is the provenance. A data PR is judged on one question — was this produced by the generator, or typed by a human? — and nothing in the body answers it. with the generator's own summary line pasted, plus the commit/date of the HF snapshot it read. Three reasons that is worth more than it looks:
One ask beyond that: if there is a scheduled job or a lane that owns the census, say so in the body (name it). Tonight several rows were refreshed by hand because nobody knew whether a timer existed — and a data file with no owner is one that goes stale without anyone noticing, which is the same false-green this repo has spent the day removing from its checks. Nothing else objects: the per-backend table (hrx 94.61/63.61 vs npu 55.14/9.45) and the top-10 unmapped list read as generated rather than curated, and the ordering by count is what a generator would emit. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 38f1959 — the arithmetic checks; one counter moves 48 % with its inputs unchanged and the body does not mention it
Recomputed from the file, not the prose:
mapped 326464 / 345055 = 94.61 % body says 94.61 % ✓
checked 219492 / 345055 = 63.61 % body says 63.61 % ✓
npu 190268 / 345055 = 55.14 % 32621 / 345055 = 9.45 % ✓
hrx row = the headline row (326464 / 219492), the same identity I noted at engine#2370
and registry_loss_check --base origin/main --head pr2430 is quiet: 41 classes on both sides, no class, counter or test reference lost (16 counters changed, which is the rebuilt-census case and not a failure).
The thing worth one line in the body: declined_models halves while its input does not change.
| main | this PR | change | |
|---|---|---|---|
| models | 437432 | 437950 | +518 |
| with_architecture | 344820 | 345055 | +235 |
| mapped_models | 325383 | 326464 | +1081 |
| checked_models | 219379 | 219492 | +113 |
| declined_models | 828 | 430 | −398 (−48 %) |
| declined_architectures | 26 | 20 | −6 |
and registry/significant.json — the file that defines what "declined" means (27 report-marked classes, 7 mapped_ok, identical on both sides) — is not in this diff (git diff --name-only origin/main pr2430 = registry/census.json alone).
So a census refresh moved every coverage number slightly up and moved the declined population by half, with the decision set untouched. The two readings are "six declined architectures no longer appear in the HF population this census sampled" and "the census changed how it matches them", and they lead to different follow-ups: the first is nothing, the second is a silent change in what the number means.
This is the second time the counter has moved unseen — at engine#2370 I noted declined_models 828 invisible in the body, and it is now 430. One clause naming the declined population and why it moved would make the pair of censuses readable as a series rather than as two documents that disagree.
Nothing else: the file's own definitions block explains mapped/checked/declined correctly, and the per_backend block is internally consistent with the headline (hrx repeats it because hrx is the pinned backend, which the definitions say).
437950 text-generation models on HF, 345055 with an architecture (3064 architectures).
Unmapped architectures with the most models:
Step1MoEForCausalLM: 2882ParlerTTSForConditionalGeneration: 1583ChessForCausalLM: 970LlavaLlamaForCausalLM: 453OpenLMForCausalLM: 266DynamicAlibiForCausalLM: 245DynamicForgettingForCausalLM: 240BertLMHeadModel: 212BartForConditionalGeneration: 176MobileLLMForCausalLM: 176