Repository navigation
Census 2026-11-15 - #2533
Census 2026-11-15#2533
Conversation
…rchitectures): mapped 326746 (94.61%), checked 219662 (63.6%)
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY-if-reproducible The headline number is the useful one and it is not the total: "438,457 text-generation models on HF, 345,360 with an architecture (3071 architectures)" — the 79 % with an architecture is what decides how much of the corpus the census can classify at all, and the 3071 architectures is the scale the mapping work has to cover. A census whose value is a ratio needs the ratio stated, not just the counts. The ask, and it is the same one I make of every measurement that will be quoted later: record how the numbers were produced — the HF API query or dataset revision, the date, and whether Three more:
And one caution specific to this repo: a census is an allowlist input. #2529 just established that |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the numbers reconcile, and two of them will be misread without a denominator
I diffed registry/census.json at this head against main key by key rather than reading the table, because a data refresh is only reviewable that way:
| base | head | |
|---|---|---|
coverage.models |
437950 | 438457 |
coverage.with_architecture |
345055 | 345360 |
coverage.mapped_models |
326464 | 326746 |
coverage.checked_models |
219492 | 219662 |
coverage.architectures_seen |
3064 | 3071 |
coverage.declined_architectures |
20 | 21 |
coverage.declined_models |
430 | 434 |
Every printed figure matches the file, architectures_seen moved by exactly the 9 new / 2 gone keys in raw.counts (the two gone are Inkling2ForConditionalGeneration, ZenithForCausalLM), and nothing was hand-edited that I can see: the diff is counters plus those keys.
Two things a reader will get wrong, both cheap to prevent:
- The two share columns have different denominators and the table does not say so.
94.61%ismapped_models / with_architecture= 326746/345360; overmodelsit is 74.55%. Same for63.6%(checked / with_architecture; over all models it is 50.1%). I ran the other division first and got a number that looks like a 20-point census bug — the row header saysmodels, so the honest fix is a column header (share of models with an architecture) rather than a footnote. checked_pctwent down (63.61 -> 63.60) whilechecked_modelswent up (219492 -> 219662) -- because the denominator grew faster. Next to an increasing count, a decreasing percentage reads as a regression, and one clause ("the denominator grew by 407 while the checked count grew by 170") turns it into the good news it is.
And one ask that is not cosmetic: name the +1 declined architecture / +4 declined models. declined_* is the only column in this file that reflects a human decision (registry/significant.json, watch: "report") rather than a crawl, so it should be attributable by class in the body. Here no decl file changed, so the rise must be models of an already-declined class appearing -- a fine outcome, stated in one line instead of left as a moving counter that nobody can trace.
Nothing else: raw is only reordered, top_unmapped keeps its shape, the coverage.definitions prose is unchanged, and Step1MoEForCausalLM at 2882 models is correctly the largest unmapped class rather than a smaller one that merely grew fastest.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 2fc9fc9: READY
Verified offline that this is a regeneration and not a hand-edit — and it incidentally repairs a real drift already on main.
The check that matters
tools/census.py has coverage(raw), the pure function that derives the summary block from the committed raw block. Recomputed from each file's own raw:
origin/main: coverage(raw) -> models=437950 mapped=326464 checked=219492 matches file: False
HEAD: coverage(raw) -> models=438457 mapped=326746 checked=219662 matches file: True
So on this head the summary is exactly what the committed raw data implies — the file cannot have been edited by hand, because a hand edit would have to recompute two percentages and eight counters consistently. Arithmetic also holds both sides (326746/345360 = 94.61, 219662/345360 = 63.60), and every count moves up together (models +507, architectures_seen +7, mapped +282, checked +170), which is what a day of new HF uploads looks like.
main has a drift this PR fixes
main mismatch: declined_models file=430 raw-recompute=434
main's census.json shipped a summary that disagrees with its own raw block by four declined models. Nobody would ever see that by reading the diff — it took a recompute to find. This PR's regeneration corrects it as a side effect, and it is the reason I am reading coverage(raw) rather than eyeballing the numbers: the summary is a derived artifact, and the only proof that a derived artifact is right is re-deriving it.
Suggestion, not a request: coverage(raw) == coverage is a one-line invariant, and it would have caught the four-model drift the day it landed. tests/census_watch_test.py already loads the committed file and asserts four things about it; this would be a fifth.
python3 tests/census_watch_test.py → all checks passed.
|
Not a verdict. 2fc9fc9 has its two. This answers the review's ask to name the The Recomputed from the file itself (merge tree): Two small things the numbers show:
|
438457 text-generation models on HF, 345360 with an architecture (3071 architectures).
Unmapped architectures with the most models:
Step1MoEForCausalLM: 2882ParlerTTSForConditionalGeneration: 1583ChessForCausalLM: 970LlavaLlamaForCausalLM: 453OpenLMForCausalLM: 266DynamicAlibiForCausalLM: 245DynamicForgettingForCausalLM: 240BertLMHeadModel: 212BartForConditionalGeneration: 176MobileLLMForCausalLM: 176