[RFC]: Observed-serving facts must flow back into dataset packing — the RULER case

[ethan-scitix]

Motivation.

RFC #27's Mechanism B and RFC #47 both model a one-way channel: the eval declares what it needs, the launcher turns it into flags. RULER (#11, landed 2026-07-30) is the first case where the dependency runs both ways, and today it is resolved by a hand-written knob plus a fallback heuristic that its own docstring calls wrong.

The fact in question is the real served context window, and how it was reached (native window vs YaRN-scaled, and to what size). Three places need it and none of them share it:

1. The dataset needs it, and says so. sieval/datasets/ruler/_shared.py:135-139:

"Which case applies is not derivable from context_length alone — it depends on serving topology (native window vs YaRN-scaled, and to what size), which this function doesn't have visibility into. Callers must say so explicitly via reserve_think_budget."

The fallback when the caller doesn't say so is context_length == 131072, documented in the same file (_shared.py:157-162) as "wrong for e.g. a length served natively at that exact size (no YaRN, no larger native window)". The decision changes how many tokens each sample is packed with — a wrong answer does not error, it silently changes the sample set.

2. The launcher already has it, in a different vocabulary. examples/ruler-qwen3-8b-nonthinking.yaml:81-85 expresses the serving side as hand-written user overrides, duplicated per model entry:

overrides:
  context_length: 131072
  disable_cuda_graph: true
  json_model_override_args: '{"rope_scaling": {"rope_type": "yarn", "factor": 4,
    "original_max_position_embeddings": 32768}}'

Every shipped recipe caps max_model_len / context_length at <= 40960 (qwen3: 8192/16384/32768/40960; qwen2.5: 16384/32768). So RULER's 64k and 128k cells are only reachable by overriding the recipe, and nothing connects that override to the dataset's packing decision.

3. The task reads a per-sample stamp of the assumption. ruler.py:302,328 stamps think_budget_reserved onto every sample and ruler_0shot_gen.py:181-188 sizes max_tokens from it. The assumption is already serialized per sample — nothing verifies it against the server that actually answers.

So one fact is written twice, in two layers, in two vocabularies, with a self-declared-wrong default in between. Neither #27's ServingRequirements (bool + int -> launch flags) nor #47's five predicates can express it: both describe what the eval demands of serving, and this needs what serving observed to reach the dataset.

Proposed Change.

Direction, not a final table — the shape depends on #25 and #47 landing their seams first.

1. Name the fact. A plain-data record derived from the resolved plan (recipe hardware + overrides + rope_scaling), not from dataset config:

@dataclass(frozen=True, slots=True)
class ServedWindow:
    native_max_position: int      # from the checkpoint config
    effective_max_len: int        # what the engine was launched with
    scaling: Literal["none", "yarn"]
    factor: float | None

2. Make the verification seam return facts, not a verdict. #47 puts a verify_instance() on the frontend at setup time; #25 owns its signature. For this case it must return observed capability values, not bool. reserve_think_budget then becomes derived (effective_max_len <= max_seq_length -> no headroom -> reserve) instead of user-supplied, and the context_length == 131072 heuristic is deleted.

3. Resolve the ordering problem — two options, and (b) is probably right.

In sieval run the launch loop finishes and populates endpoint_map (cli/run.py:176-180) before arun_session (cli/run.py:195), so the resolved plan is available before any dataset loads. That makes (a) "dataset packing consumes the resolved plan" technically possible there.

But sieval eval against an external endpoint has no plan at all, and that is the more common path. So (b) is the general answer: samples record the window they were packed under; setup-time verification rejects a mismatch. This is cheap because the per-sample stamp already exists — think_budget_reserved becomes a ServedWindow fingerprint, and a mismatch is a hard error, not a silent re-pack. Under the reproducibility contract this must be strict-only: no auto-repack, no downgrade.

4. Fold in the missing one-way half too. min_context_length as an eval->serving requirement — Floor(when="scoring.long_context", field="context.max_model_len", at_least=131072) in #47's vocabulary — so RULER 128k against a <=40960 recipe fails at plan time rather than at request time or, worse, by silent truncation.

Feedback Period.

Two weeks — it should not be locked before #25 fixes the verify_instance() signature.

Any Other Things.

Dependencies. Blocked on two seams that do not exist yet:

1. [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25 — verify_instance() must be able to return observed capability values. If it lands as -> bool / -> None, this RFC forces it reopened.
2. [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 — the CapRef symbol space needs context/window entries, and Floor covers item 4 as-is. [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 explicitly scopes itself to constraints (relationships that must hold); this issue is about facts (what was actually observed) travelling downward. Same seam, opposite direction.

Relationship to #27. #27's scope was split in its comment thread: Mechanism A (recipe hardware/capabilities split) stays there, Mechanism B is superseded by #47. This issue takes the part of Mechanism B that a one-way flag channel structurally cannot express — it is not a third copy of the same idea.

Why now. RULER landed 2026-07-30 and is the first eval in the repo whose correctness depends on a serving detail the eval layer cannot see. The workaround shipped with it (reserve_think_budget plus a heuristic annotated as wrong) is the honest local fix; it should not become the pattern the next long-context benchmark copies.

Out of scope. Auto-launching or reconfiguring a server to satisfy a dataset; choosing a YaRN policy; anything that re-packs samples to match a server (that is a silent-rewrite path the reproducibility contract forbids).

Before submitting a new issue...

• Make sure you already searched for relevant issues and documentation.