feat(recipes)!: split recipe params into hardware and capability layers

[ethan-scitix]

Type

• feature — new benchmark, task, or capability

Summary

Implements Mechanism A of RFC #27. The RFC's scope was split first (see its comment thread): Mechanism B is superseded by #47, and the two-way case neither can express is now #58. This PR is only the recipe schema half.

• The problem is live, not hypothetical. profiles[hardware][precision][framework] mixed perf params with behavior params and matched on family + size only, so a base checkpoint resolved to the instruct profile and inherited its parsers. qwen2.5-72b (size_range: [60, 80]) carries enable_auto_tool_choice: true + tool_call_parser: hermes on every leaf — and Qwen2.5-72B-Base is the named comparison target of the ARC and HellaSwag ppl/clp tasks, which resolve to that same bucket.
• Two layers now: hardware[gpu][precision][framework] (dtype, memory, parallelism, context) and capabilities[instruct|base][framework] (parsers, tool choice). resolve_profile → resolve_hardware_profile + resolve_capability_profile, merged capabilities-last. Collapsing the duplicated capability params takes them from 352 lines to 78, and the three recipe files from 1328 to 1203.
• The model type is derived, not read. No config in the repo sets type: — it is left to inference from the tasks pointing at the model. Reading the explicit field alone (as the original plan did) would have resolved every base checkpoint to instruct capabilities and made this change a no-op on real configs. The derivation the eval session already performed is promoted to a shared derive_model_type, since both sites must agree for one model.
• gpt-oss no_enable_prefix_caching deliberately stays on the hardware layer. It is model-intrinsic rather than hardware, but prefix caching is being modelled as an engine-level constraint in [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47; moving it now would migrate it twice. Noted in the YAML.

Related Issues

Refs #27. Related: #47 (supersedes Mechanism B), #58 (the two-way case).

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check) — the one remaining hit is sieval/_version.py, build-generated and gitignored
• Type check clean (ty check) — "All checks passed!"
• Unit tests pass — 2691 passed. Also integration (52), acceptance (10), check_layer_imports.py (exit 0), and check_preflight.py (all PASS)

New coverage:

• tests/unit/infer/recipes/test_golden.py — characterization against a committed fixture of all 168 pre-split triples: content preserved, order preserved, base path omits capability params
• tests/unit/infer/test_recipes.py — legacy-key rejection, capability_model_type mapping, capability resolution, and a sweep asserting no shipped hardware leaf still carries a parser and every entry declares both model types
• tests/unit/cli/infer/test_resolve.py — end-to-end through resolve_infer_config, including the load-bearing case: no type: in config + a gen task ⇒ base capabilities
• tests/unit/cli/leaderboard/test_session.py — derive_model_type directly, since sieval run calls the function rather than the session method

Manual

• No GPU-backed launch performed. Resolution is exercised through the mocked-GPU path used by the existing suite; nothing here touches deployment.
• Unverified, and it sets this change's priority: whether a base checkpoint launched with enable_auto_tool_choice: true actually fails to start on vLLM or merely carries a dead flag. RFC [RFC]: Recipe capability layer — split model-type capabilities out of hardware profiles #27 says "meaningless", never "breaks". If it breaks, this is a bug fix rather than hygiene.

Checklist

Required (all PRs)

• PR title follows conventional format
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/ — core/ untouched; infer/ still imports nothing above it
• Deleted code verified — resolve_profile had exactly two production call sites (cli/infer/recipe.py, infer/topology/resolver.py), both migrated; no references remain

If: Breaking Change

• Described what breaks and migration path in Summary
• Existing tests updated to reflect new behavior

What breaks. The profiles key is rejected with a migration error naming both replacement layers, rather than read for compatibility. Recipe files ship in-repo so there is no legacy on-disk state to support, and silently accepting profiles would keep handing base checkpoints the params this split exists to withhold. User customization via overrides is unchanged. resolve_profile is removed from the sieval.infer.recipes public surface.

This must ship as a MINOR bump (0.8.0), not a patch. Instruct resolution is content-identical for all 168 triples, but key order changes for the two gpt-oss recipes, whose capability params sat between hardware params pre-split. infer_plans.yaml is dumped sort_keys=False and compared byte-for-byte under --resume, so a 0.7.x run resumed on this code would abort. Under 1.0 the resume version gate's break axis is (major, minor), so as 0.8.0 the gate rejects that resume with a clear version message; as 0.7.1 the user would instead get a confusing plan diff. The golden test pins the reordered set to exactly those two recipes so no further reordering can slip in unnoticed. Staged in the pending-CHANGELOG notes with this reasoning.

Note for reviewers

Two things I did not do, deliberately:

1. Did not commit the design doc. [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 asks for docs/superpowers/specs/2026-07-01-recipe-capability-layer-design.md to be committed, and I initially agreed in the [RFC]: Recipe capability layer — split model-type capabilities out of hardware profiles #27 comment. That is wrong for this repo: docs/designs/ is in .gitignore (line 185) and only docs/guide/ is tracked — in the public and internal repo. Design specs are local-only by convention here, so tracking one would need an explicit convention change, which is not this PR's call. The doc now carries a scope-reconciliation header locally. The underlying concern is still valid but narrower than [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 implies: the sglang truncation evidence is already in sglang_gen_model.py:302-328 and the --max-logprobs evidence in 9 task docstrings; what is genuinely unrecorded anywhere tracked is the vLLM V1 prompt_logprobs × prefix-caching mutual exclusion.
2. Did not touch the 9 task files' prose. [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 states its Milestone 1 must land first — converting executable prose into a declaration nothing acts on is a net loss.

🤖 Generated with Claude Code