[RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints

[ethan-scitix]

Motivation.

Some eval requirements are not request parameters — they are relationships between capabilities, and violating them produces silently wrong scores rather than an error. Today those relationships live only in task docstrings.

Concretely, in sieval/tasks/:

files carrying --disable-radix-cache / --max-logprobs prose:  9
total mentions:                                              28
lines of code that read or set them:                          0

_arc.py, arc_{easy,challenge}_kshot_{ppl,clp}.py, c_eval_kshot_clp.py, cmmlu_kshot_clp.py, mmlu_kshot_clp.py, mmmlu_kshot_clp.py all instruct an operator to launch the server a particular way. Nothing verifies they did. Two known failure modes:

1. PPL tasks × prefix cache. On a cache hit sglang truncates input_token_logprobs to prompt_tokens − cached_tokens and does not report it; vLLM V1 errors instead. Requires --disable-radix-cache / --no-enable-prefix-caching. SglangTransport._guard_radix_cache catches the sglang symptom at request time — but only on the native /generate path, and only after a deployment has already been spun up.
2. CMMLU / C-Eval / MMLU top-k breadth. vLLM defaults to --max-logprobs 20; A/B/C/D are not guaranteed to appear. Needs --max-logprobs 100. Currently, per docs/superpowers/specs/2026-07-01-recipe-capability-layer-design.md, this "relies on an operator reading the task docstring."

RFC #25 introduces a capability model, but its Capability set can only express "this feature is supported". It cannot express "these two must not be on together", "this field needs a value ≥ N", or "this one is mandatory" — so neither failure above is declarable, let alone enforced. This RFC proposes the missing layer.

This is deliberately separate from and downstream of RFC #25. The constraint table has to be filled from probe results against real servers, and those probes have not run yet; three of the six constraint claims currently encoded in comments are unverified (see "Any Other Things"). Writing the table now would repeat the mistake RFC #25 made: modelling from vendor docs instead of from observed behaviour.

Proposed Change.

1. Three axes — where a constraint is declared

Capability is f(frontend, engine, instance), not a property of one object. vLLM and sglang both serve the openai_completions wire protocol; sglang additionally serves its native /generate. So:

Axis	Owns	Example
Frontend (wire protocol)	protocol-shaped constraints	anthropic_messages requires max_tokens
Engine (the serving process type)	engine-behaviour constraints	input_logprobs ⊥ prefix_cache; top-k ceiling; max_new_tokens ≥ 1
Instance (the running process)	not declared — probed	this vLLM was launched with --max-logprobs 20

4 of the 5 constraints below are engine-level, which is why this depends on RFC #25 keeping frontend and engine as separate axes. It also relocates _guard_radix_cache correctly: declared once on the sglang engine, it then covers both the native and openai_completions paths — today it protects only the former.

2. Five closed predicates — not a DSL, not a solver

Frozen dataclasses in sieval/core/models/constraints.py. CapRef is "<group>.<field>" from the RFC #25 vocabulary, so there is one symbol space, not two.

Conflict(a, b, on_violation, remedy, source)      # a and b cannot both hold
Ceiling(field, default_limit, raise_with, on_violation, source)
Floor(when, field, at_least, source)              # when `when` holds, field >= N
Mandatory(field, source)                          # protocol-required
Pinned(when, field, to, source)                   # when `when` holds, field is fixed

Each derived from a real case, none invented. A closed set can be exhaustively matched, mechanically turned into launch flags and into human-readable errors, and serialised into the run record as reproducibility evidence. Adding a sixth kind is an explicit RFC-able act rather than something that accretes.

on_violation: Literal["silent", "error"] is the load-bearing field — it decides whether a constraint must be prevented or may be left for the server to reject.

3. Three checkpoints — only the first can fix anything

Checkpoint	Home	Capability
plan (pre-launch)	sieval/cli/validation.py — a _validate_capabilities beside the existing _validate_models / _validate_tasks; surfaced by sieval eval --dry-run	computes launch flags; hard-fails on a silent conflict with no remedy
setup (post-launch, pre-samples)	frontend verify_instance()	probes the live server and asserts the requested capability is actually present — this is where top-20-vs-100 gets caught
request	frontend lower()	Mandatory / Floor / Pinned, plus RFC #25's unconsumed-field gate

Any constraint carrying a remedy must be consumed at plan time — otherwise a deployment is spun up before the problem is found. This is also why the layer cannot live entirely inside the frontend: plan time precedes frontend construction.

4. Scope red line

Model a constraint only if (a) violating it is silent, or (b) the remedy must be applied at launch time.

If the server rejects it loudly at request time, do not model it — let it reject.

This is what keeps the table from becoming a shadow copy of every provider's API reference. "temperature must be in 0–2" is not modelled; sglang's silent truncation is.

Second red line, per CLAUDE.md ("Safety guards ship strict-only. No --force-* flags, no bypass env vars"): no auto-downgrade. A conflict yields an error plus its remedy, never a silent flip of a requested capability off. There is no --allow-prefix-cache-with-scoring.

5. Provenance — every constraint must be re-falsifiable

source accepts exactly two forms:

• probe:<name> — an executable check that can be re-run against a real server
• spec:<url#anchor> — a protocol definition, commit-pinned

Free-form prose is not an accepted source. This closes the "modelled from the docs" path structurally, and it means an engine upgrade that invalidates a constraint turns a probe red instead of leaving us needlessly disabling prefix cache for a year.

6. Relationship to anomaly detection — link, don't merge

A constraint and an anomaly rule are frequently the same fact at different times: _guard_radix_cache checks len(input_token_logprobs) != prompt_tokens, which is exactly the post-hoc symptom of constraint 1. So Conflict gains an optional detector that registers as a @sieval_detection_rule:

Conflict(
    a="scoring.input_logprobs", b="caching.prefix_cache",
    on_violation="silent",
    remedy="--disable-radix-cache",
    source="probe:sglang_radix_truncation",
    detector="detect_truncated_input_logprobs",
)

One declaration, three roles: verify (probe), prevent (remedy, plan time), detect (post-hoc, anomalies.json). Defense in depth — if prevention is bypassed or the constraint goes stale, the symptom still lands in the report.

They should not become one subpackage. anomaly.py is per-sample, post-hoc, (TaskContext) -> set[int], and advisory with a severity: info|warning|error dial. Constraints are per-run, pre-execution, and must be strict-only. Putting a strict gate inside a framework that ships a downgrade dial invites exactly the bypass the reproducibility contract forbids. The repo already has three detection surfaces at three timings — scripts/check_preflight.py (static/CI), cli/validation.py (plan), core/tasks/anomaly.py (post-hoc) — and this layer belongs in the second, with its detectors in the third.

7. Milestone 1 — mechanically verifiable

1. core/models/constraints.py — the 5 predicates
2. Per-engine declarations for vllm and sglang — 5 constraint instances total
3. All three checkpoints wired
4. The 9 task files' prose becomes one requires = {...} line each; --disable-radix-cache and --max-logprobs occurrence count in sieval/tasks/ goes 28 → 0
5. Two probes in CI: sglang_radix_truncation, vllm_max_logprobs_default

Item 4 is the convergence signal. If that count does not reach zero, the layer added a declaration surface without taking on the responsibility, and Milestone 1 has not landed.

Feedback Period.

One week.

CC List.

@jack-scitix-ai

Any Other Things.

This RFC is blocked on probes, not on design. Of the constraint claims currently encoded in code comments:

• Verified: sglang radix-cache truncation; vLLM V1 prompt_logprobs/echo × prefix caching (both in docs/superpowers/specs/2026-07-01-recipe-capability-layer-design.md); the vLLM --max-logprobs default, validated on an 11,582-sample Qwen2.5-72B run.
• Unverified: "sglang's /v1/completions rejects echo=True together with logprobs" — the sole stated justification for SglangGenModel's existence (sglang_gen_model.py:3), with no supporting record anywhere in docs/designs/. It also looks like a possible mis-attribution of the prefix-cache constraint to the protocol layer.
• Wrong: "top_k is a vLLM extension; upstream OpenAI rejects it" as grounds for a first-class IR field — leaderboards/qwen3_proxy_5min_202606.yaml:78 already passes it through extra_body.
• Needs a probe: Anthropic extended thinking pinning temperature. Not in the Milestone-1 table; it lands with the anthropic_messages frontend.

Two probes gate the scope of both this RFC and the RFC #25 revision, because each can delete an entire code path:

1. Does sglang's /v1/completions actually accept echo=True + logprobs? If yes, the native /generate frontend may be removable — along with token-text normalisation, --skip-tokenizer-init handling and the max_new_tokens=0 clamp.
2. Is vLLM's prompt_logprobs available on the deployed version? If yes, the echo workaround and its usage.input_tokens boundary split are unnecessary. openai_completions.py:6 currently calls echo a workaround for a "missing native scoring endpoint", which appears to be incorrect.

Also note docs/superpowers/specs/2026-07-01-recipe-capability-layer-design.md — the only place these verified backend behaviours are recorded — is not yet tracked by git. It should be committed before this RFC is implemented; it is the ancestor of this design and covers the launch-flag half.

Dependency on RFC #25. Three seams must exist in the RFC #25 revision or this RFC forces it to be reopened:

1. frontend / engine as separate axes — 4 of 5 constraints are engine-level and have nowhere to live if the two are fused (the current SglangGenModel shape).
2. CapRef symbol space, including serving-axis entries (scoring.topk_breadth, caching.prefix_cache). This RFC adds constraints only; it must not have to change the vocabulary.
3. An empty verify_instance() seam on the frontend — it may be a no-op in RFC [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25, but the interface must exist, or every frontend gets touched again.

And a guard against repeating RFC #25's own failure: RFC #25 should explicitly reject serving-axis CapRefs in Task.requires with "declared but not enforced until this RFC", rather than accepting and ignoring them. Better that nobody can write the declaration than that someone writes one which silently does nothing. For the same reason, the 9 task files' prose should not be touched before Milestone 1 lands — converting executable prose that a human can read into a declaration that nothing acts on is a net loss.

Before submitting a new issue...

• Make sure you already searched for relevant issues and documentation.