Treat systems as their own class of entrant

[LennartPurucker]

TabArena benchmarks models and carries AutoGluon along as a "Reference Pipeline". This makes systems their own class of entrant: a system is a whole pipeline that picks, tunes and ensembles models inside its own budget, whether that is an AutoML framework, an agent, a hosted API, or a model run through a heavier self-managing interface.

Two problems motivated it.

The reference-pipeline handling was already inconsistent. AutoGluon rows sit in website_leaderboard.csv and therefore in the Elo pool, so every model's Elo was computed against them. But the Pareto figure, the cross-subset overview and the win-rate matrix all excluded or hid them. The site showed a "models only" view built on numbers that were not computed models-only.

A participant filter cannot be client-side. Elo is pairwise over the participants, improvability is 1 - best_error_in_pool / error, and the ranks are positions in the field. Change who competes and everyone's numbers change.

What changes

Metadata. MethodMetadata gains method_class (model / system) and tags (with-llm, closed-source-api). method_type stays what it was, a result-shape discriminator deciding which parquet load_results reads and whether HPO simulation applies. Systems record as baselines like any other non-config run, so overloading it would have touched about 30 branch points for nothing.

A systems/ package mirroring models/: one folder per system with system.py, hpo.py and info.py, walked into SYSTEM_REGISTRY by discover_systems(). TabFM+ moves out of models/tabfm/, AutoGluon's metadata is consolidated from four scattered modules into systems/autogluon/info.py, and the example's AutoGluonSystemModel is promoted into a real wrapper. TabFM+ is registered in the collection, so it reaches the leaderboard for the first time.

Entrant pools as a subset axis. Three independently selectable system categories (open, llm, api), so every combination is published as its own evaluation: eight pools (models, open, llm, api, open_llm, open_api, llm_api, open_llm_api) over a 480-cell grid. eval_all.py leaves nips2025_utils on the way past and splits into evaluation/{entrants,subset_grid,eval_all}.py, next to the LeaderboardReporter it drives.

Reference Pipeline becomes System, typed from method_class rather than name prefixes. That retires the ❓ Other trap the update-leaderboard skill has a pre-check for: previously AutoGluon and PORTFOLIO were hardcoded prefixes and every new system landed in Other until someone remembered to extend the table.

Static PNGs stop shipping. They were 80.1 of the 105 MB in the Space's data/ and the only reason it needed Git LFS, while every figure already has an explorer that renders by default and exports its own SVG/PDF/PNG. A pool drops from ~105 MB to ~24.5 MB, so four pools land under today's single-pool footprint. BeyondArena is untouched and keeps its PNGs.

Docs: AGENTS.md sections on systems and entrant pools, a new add-system skill, and refreshes to upload-method and update-leaderboard.

Decisions worth a second opinion

• Eight pools, independent checkboxes. A cumulative four-stop ladder cannot express "LLM-based systems but not the plain open-source ones", which is a question people actually have. The cost is that each new category doubles the artifact count.
• Portfolio-N200-4h is typed as a system. It is a multi-model pipeline like AutoGluon but a TabArena-internal construct rather than an external one. Easy to flip.
• Dropping the static figures from the site. They are still rendered into raw_website_artifacts/ for paper use; only the copy into the published layout goes away.

New systems, and two bugs they surfaced

The AutoGluon 1.6 presets from #466 land here in the systems/ layout rather than in contexts/: both are MethodMetadata.system(...) in systems/autogluon/info.py, the dated module keeps only its list, and autogluon_info points at 1.6 extreme. TabFM+ is processed and uploaded to r2 (suite tabarena-2026-07-07, 816 tasks over 51 datasets). It stays verified=False: the run predates the 2026-07-13 rerun that stopped reloading models from disk around inference, so its time_infer reads high next to the models beside it on the Pareto front.

Registering them exposed two defects in this branch's own work.

Systems without a pinned reference-line color never reached the leaderboard. get_pool_reference_lines intersected the pool's admitted methods with a two-entry color map, and that list is not only what the figures draw: LeaderboardReporter.eval keeps a non-config row only when its method maps to a config framework type or is named in baselines. AutoGluon 1.6 and TabFM+ were absent from all 454 CSVs of the first generation pass, with nothing logged. It now returns every admitted system and cycles a color list whose first two entries keep AutoGluon 1.4 and 1.5 in their existing colors.

Seven of the eight pools had no pool name in their figure headlines. The generator's label map still listed the four keys from the earlier design, and _subset_label drops any segment it cannot name, so the one part of the label that says which field the numbers came from went missing. It reads ENTRANT_POOLS now, and an unknown entrants_* segment raises.

Not in this PR

The companion changes to the leaderboard-testing Space (the control band at the top of the page, the "Who's competing?" and "I care about" selectors, tag chips, the entrants API axis) live in that repo.

Wrappers for LightAutoML, FLAML, TabPFN-3-API and TabPFN-Thinking are deferred; the add-system skill covers them. Until an open LLM-based system or a hosted API exists, the llm and api categories have no entrant, so those four pools are identical to models-only and the Space renders their toggles as "coming soon".

Testing

1417 passed, 2 skipped. ruff check and ruff format --check clean on every touched file. New tests cover the metadata fields and their two serialization forms, the systems registry, pool admission for each of the eight pools, the subset grid and its folder layout, and system typing in website_format.

The artifacts are regenerated at the official 200 bootstrap rounds. Verified across all eight pools: the four admitting open carry 84 rows with all five systems, the other four carry 79 with none, and TabM (tuned + ensembled) moves 1422 to 1420 between them, which is the recompute doing its job.

Also verified from a clean git worktree checkout of HEAD that both systems discover and all three appear in the collection. That check exists because a bare autogluon/ pattern in .gitignore matched at any depth and silently swallowed the whole systems/autogluon/ package: git status never showed it and git add on the parent skipped it. Fixed by anchoring the pattern; without that catch the branch would only have worked in the tree it was written in.

🤖 Generated with Claude Code

[LennartPurucker]

@Innixma fyi here is the PR for the system API re-design. A lot of the leaderboard and plotting code is still in flux from my side right now but the way we would store and treat them inside of TabArena is open for discussion, also if we want to keep the hpo.py style etc

[Innixma]

Generally seems good to me! We can always adjust after, so I'm fine with merging.