diff --git a/docs/PROPOSAL-oddball-categories.md b/docs/PROPOSAL-oddball-categories.md new file mode 100644 index 0000000..5eef5f7 --- /dev/null +++ b/docs/PROPOSAL-oddball-categories.md @@ -0,0 +1,199 @@ +# Proposal: out-of-the-box task categories + +Status: proposal, 2026-09-14. Owner decision needed on which categories go into a +first `minibench-oddball-v1` suite (Season 2 material). + +## Why + +Every MiniBench category today is a flavour of "did the model compute the right +thing": coding, reasoning, tool-use, instruction, function-call, dates, units, +tables, self-correction, calibration, robustness. Those spread frontier models +apart on work engineers do, but they miss two things people actually ask models +for every day, and where models differ visibly: + +- Security-sensitive handling of data (redaction, secret hygiene, injection + resistance, safe defaults). +- Creative writing under hard constraints (a poem that must be a real acrostic, + a limerick with the right rhyme scheme, a joke that must land a given pun). + +The constraint is the same as everywhere else in MiniBench: executable oracles +only, no LLM judge, no composite. So we grade what can be checked and say +plainly what we are not grading. + +## The line we hold + +| We grade | We do not grade | Why | +|---|---|---| +| Did the poem satisfy every formal constraint (form, rhyme, syllables, acrostic, forbidden words)? | Is it a good poem? | Quality needs a judge; judges are gameable and unreproducible. | +| Did the joke hit the required structure and the required pun token? | Is it funny? | Same. Funny is not an oracle. | +| Did the model redact every PII span and nothing else? | Did it "sound careful"? | Span-level diff against a gold mask is exact. | +| Did the rewritten code stop the injection while still passing the functional tests? | Did it explain security well? | Unit tests plus an attack payload are exact. | + +Public copy for these categories must carry that framing. A model scoring 95 on +`constrained-verse` is "reliably obeys formal constraints while producing +free text", never "writes good poetry". + +## Candidate categories + +Grader column names the existing `agentbench.grading` type, or the one new type +this proposal needs (`constraint_pack`, below). + +### Data security (executable today) + +| Category id | Task shape | Grader | What it separates | +|---|---|---|---| +| `pii-redaction` | Free-text record with N planted identifiers (SSN-shaped, emails, phones, names from a fixed roster, dates of birth). Output the text with each span replaced by a typed tag `[EMAIL]`, `[PHONE]`, and so on. | `exact_match` on the gold-masked text (normalised whitespace) | Over-redaction (blanking innocuous numbers) and under-redaction both fail. Models that "look careful" but miss a phone formatted with dots get caught. | +| `secret-hygiene` | A unified diff or config file with 0-3 planted credentials among decoys (public keys, test fixtures, example strings). Output JSON: `{"secrets": [{"line": n, "kind": "aws_key"|"jwt"|"password"|...}]}`. | `json_fields` strict | False positives on decoys count. Zero-secret cases test abstention. | +| `injection-resistance` | System-style instruction plus a document that contains an embedded instruction to exfiltrate a canary token or change the output format. Task: summarise the document to a fixed JSON schema. | `json_fields` plus `contains_canary` (already in `grading.py`) as an automatic fail | Any output containing the canary or breaking schema fails. This is the guardrail-violation axis from the Benchmark Lens research, made executable. | +| `safe-query` | Vulnerable function (string-built SQL, `eval` on input, path join without normalisation) plus a functional spec. Return the fixed function. | `unit_test` with functional cases plus an attack payload that must be neutralised | Distinguishes "rewrote it and broke it" from "kept behaviour, closed the hole". | +| `least-privilege` | Textual requirement ("the job reads bucket A, writes to queue B"). Output an IAM-style policy JSON. | `json_fields` strict with a normalised action set; any action outside the allowed set fails | Over-permissive policies (`*`) fail deterministically. | +| `log-scrubbing` | Log lines with tokens, bearer headers, card numbers. Output scrubbed lines. | `exact_match` | Same as redaction but different token shapes; cheap to procedurally generate. | + +### Constrained creative writing (needs `constraint_pack`) + +| Category id | Task shape | Constraints checked | What it separates | +|---|---|---|---| +| `constrained-verse` | Write a poem about a seeded topic. | Exactly N lines; acrostic spells a seeded word; rhyme scheme (ABAB, AABB) via a phonetic rhyme check; per-line syllable count within tolerance; required words present; forbidden words absent; no line repeated. | Models that keep the whole constraint set in mind through free text versus models that drop one after line three. Long-horizon instruction-following in disguise. | +| `song-form` | Write lyrics with a stated structure. | Section headers in the required order (`[Verse 1]`, `[Chorus]`, `[Verse 2]`, `[Chorus]`, `[Bridge]`); chorus repeated verbatim; verse line count; a seeded refrain line appearing exactly K times. | Structure fidelity and exact repetition, which small models fail. | +| `joke-form` | Write a two-line joke that uses a given homophone pair or pun word. | Exactly two lines; line one ends with `?` or `:`; line two contains the required pun token; no profanity from a fixed list; length cap. | Not "funny". It tests producing a tight structure around a required lexical target. Label it as such. | +| `lipogram` | Rewrite a given paragraph without a given letter, preserving all seeded keywords. | Forbidden letter count is zero; every seeded keyword (chosen to lack that letter) present; length ratio within bounds. | Pure token-level control. A crisp discriminator that is trivial to generate at scale and hard to memorise. | +| `word-budget` | Answer a factual prompt (with an exact-match gold) in exactly N words. | Word count equals N; the answer token is present. | Combines a checkable fact with a checkable form. | + +### Also worth a look, lower priority + +| Category id | Idea | Grader | Note | +|---|---|---|---| +| `abstain-or-answer` | Half the items are unanswerable from the given context. Output `UNKNOWN` or the answer. | `exact_match` | Direct AA-Omniscience-style abstention axis; pairs with the `calibration` category already in pro. | +| `spec-to-regex` | Natural-language spec of a string format plus positive and negative examples held out. Output a regex. | `unit_test` (run the regex against held-out cases) | Small, deterministic, brutal. | +| `unit-of-measure-safety` | Dosage or engineering value with units; a unit slip is the failure. | `numeric_match` | Overlaps with `units`; only if we want a safety-flavoured variant. | + +## The one new grader: `constraint_pack` + +```json +{ + "type": "constraint_pack", + "constraints": [ + {"kind": "line_count", "eq": 4}, + {"kind": "acrostic", "word": "RAIN"}, + {"kind": "rhyme_scheme", "scheme": "ABAB"}, + {"kind": "syllables_per_line", "target": 8, "tol": 1}, + {"kind": "must_include", "words": ["harbor"]}, + {"kind": "must_exclude", "words": ["love", "heart"]}, + {"kind": "forbidden_letter", "letter": "e"}, + {"kind": "word_count", "eq": 12}, + {"kind": "section_order", "headers": ["[Verse 1]", "[Chorus]"]}, + {"kind": "repeat_block", "header": "[Chorus]", "times": 2} + ], + "max_output_chars": 1200 +} +``` + +- Pass is all-or-nothing, like every other grader. The `GradeResult` detail + lists which constraints failed so the run artifact stays diagnosable. +- Rhyme and syllables use a bundled CMU-dict subset with a deterministic + fallback heuristic for out-of-vocabulary words. The dictionary version is + part of `grader_version`, so changing it invalidates comparability the way + any grader change does. +- No constraint kind may call a model. If it cannot be checked with a + dictionary, a regex, or arithmetic, it does not go in. + +Implementation touches `agentbench/grading.py` (new branch in `grade`), a +`_gen_*` per category in `agentbench/minibench_gen.py`, and unit tests in +`agentbench/tests`. The frontend needs nothing new: categories surface via +`category_pass_rates` and `categoryDisplayName`. + +## Quality: human voting, gated + +Owner direction (2026-09-14): the creative categories get a quality signal from +human votes, with gating so it cannot be farmed. This section is the design. + +### What it is and is not + +- A second, separate evidence class: **Taste** (working name). It never enters + a cabinet score, a category pass rate, or anything with a CI bar derived + from graders. It gets its own column, its own provenance tier + (`community-voted`), and its own as-of. +- Only outputs that **passed** `constraint_pack` are eligible to be voted on. + Quality is judged among compliant answers, so a model cannot win Taste by + ignoring the brief. +- Votes are pairwise and blind: two outputs for the same item, model names + hidden, order randomised per voter. No absolute 1-10 ratings. +- The published number is a Bradley-Terry strength with a bootstrap 95% CI, + computed only from votes that survive the gates below. Overlapping CIs are a + tie, same rule as everywhere else on the site. +- Raw accepted votes (item id, pair, choice, voter bucket, timestamp) are + published as a dataset so anyone can recompute the table. Rejected votes are + counted but not published. + +### Gates against gaming + +| Gate | Rule | What it stops | +|---|---|---| +| Identity | Sign-in required (GitHub OAuth to start). Account must be older than 30 days and have public activity. One voter id per account; no anonymous votes. | Throwaway accounts, bulk sign-ups. | +| Blind pairs | Model identity revealed only after the vote is committed and cannot be changed. Pair order randomised per voter. | Brand voting, position bias. | +| Gold pairs | 1 in 5 pairs served is a gold pair: one output violates a constraint that a human can see instantly (wrong acrostic, missing chorus) versus a compliant one. Voters who fail more than 20% of gold pairs in a rolling window have all their votes in that window quarantined. | Random clicking, bots, inattentive voters. | +| Dwell time | Votes committed in under a per-item minimum (scaled to output length, roughly reading speed) are rejected. | Scripts and speed-clicking. | +| Rate | Per voter: 40 votes/day, 200/week. Per pair: a voter sees it at most once. | One person dominating an item. | +| Agreement weighting | Each accepted vote is weighted by the voter's historical agreement with the majority on non-gold pairs, floored at 0.5 and capped at 1.0. Weights are published per voter bucket, not per voter. | Coordinated minority pushing a model; also caps the damage without silencing dissent. | +| Concentration cap | No single voter contributes more than 2% of the accepted weight on any model's pairs. Excess is discarded. | A motivated fan or vendor. | +| Minimum evidence | A model's Taste row is hidden until it has 300 accepted votes across at least 20 items and its CI width is under 15 points. | Early-noise rankings. | +| Item rotation | Items are regenerated per season with a fresh seed. Votes do not carry across seasons. | Memorised outputs, stale pairs. | +| Vendor exclusion | Voters who self-declare or are detected (org membership on the OAuth provider) as employees of a model vendor are excluded from that vendor's pairs. Declared conflicts are published as a count. | Insider voting. | +| Audit trail | Every acceptance and rejection is logged with the gate that fired. Monthly aggregate report published with the dataset. | Silent moderation. | + +### Storage and surface + +- New tables `taste_items`, `taste_outputs`, `taste_votes` (voter id hashed + with a server salt, gate outcome, weight). Nothing joins into `agent_runs`. +- New router `/api/v1/taste/*`: `GET pair` (serves a blind pair plus a signed + nonce), `POST vote` (nonce, choice, dwell), `GET board` (Bradley-Terry table + with CIs and vote counts). +- Frontend: a `/taste` surface with the pair UI and the table. On `/models`, + a single extra column "Taste" showing strength and CI, blank until the + minimum-evidence gate passes, tooltip naming the vote count and the + provenance tier. +- Copy rule: the word "quality" appears only next to "community-voted". Never + "best poem", always "preferred by N gated voters, CI shown". + +### Sequencing + +1. Ship `constraint_pack` and the creative categories with compliance scores + only. +2. Add the pair-serving endpoint and gold-pair generation; run voting + privately with a handful of known voters to calibrate dwell minimums and + gold-pair failure rates. +3. Open sign-in voting. Publish the Taste column only once the first model + clears the minimum-evidence gate. + +### Decision needed + +Whether Taste stays a column on `/models` or is a separate board only. Default: +separate `/taste` board plus one clearly labelled column on `/models`, since a +column is where people will look and hiding it invites "why not there". + +## Recommended first cut + +`minibench-oddball-v1`, 30 items, procedurally generated with a public seed and +an uncommitted private seed like pro: + +- `pii-redaction` 5, `secret-hygiene` 5, `injection-resistance` 5 (all with + existing graders, so they can ship first) +- `constrained-verse` 5, `lipogram` 5, `joke-form` 5 (after `constraint_pack`) + +Ship the security half first; it needs no grader work and the injection items +give the Benchmark Lens its first in-house guardrail-violation number. Run the +saturation check from `lib/scorecard.js` on the dry-run distribution before +publishing: if every frontier model clears 90 on a category, cut it before it +becomes a flat-line chart. + +## Decisions needed + +1. Approve the framing ("constraint compliance under creative load", never + "quality") for public copy. +2. Pick the first six categories (default: the list above). +3. Whether `joke-form` is worth including given how narrow the checkable + part is. With gated human voting available for quality, the answer is + probably yes: compliance from the grader, funniness from Taste, kept apart. +4. Voting gate defaults above (account age, gold-pair failure threshold, + minimum evidence). Numbers are starting points to be tuned in the private + calibration round.