diff --git a/AGENTS.md b/AGENTS.md index 7c3eeef..68e90f1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,6 +8,7 @@ These instructions apply to every agent working in this repository. - Use the **AgentMemory MCP** at `http://localhost:3114` at the start of each task to retrieve relevant project context before inspecting or changing code. - If AgentMemory is unavailable, start `agentmemory --port 3114` as a background process, wait for port 3114 to accept connections, and retry the MCP call. If the global command is unavailable, use `npx -y @agentmemory/agentmemory --port 3114`. If port 3114 is unavailable or occupied by another service, choose a free port, start AgentMemory with `--port `, set `AGENTMEMORY_URL=http://localhost:` for the MCP process, and use that endpoint for the rest of the task. - Store durable project knowledge in AgentMemory after discovering important architecture, conventions, decisions, or non-obvious fixes. Do not store secrets, credentials, transient command output, or guesses. +- Every AgentMemory `project` field (on `memory_save` and anywhere else it appears) MUST be a path-derived slug, never a bare display name like `"tokenfold"` — AgentMemory is a shared local service (`localhost:3114`) that other unrelated projects on this machine also write to, and a bare name is not collision-resistant. Derive the slug from the repo's absolute path: lowercase the drive letter, then `--`, then the rest of the path with every `\` (or `/`) replaced by `-`. For this repo's default clone location (`E:\Github\tokenfold`) that slug is `e--Github-tokenfold` — the same convention already used to name this project's local file-based memory directory (`~/.claude/projects/e--Github-tokenfold/memory/`), so the two memory systems stay identifiable by the same key. If the repo is cloned somewhere else, recompute the slug from that path instead of reusing this example verbatim. - If Ponytail remains unavailable, say so explicitly before continuing and use the closest available fallback. ## Git attribution diff --git a/eval/run_baselines.py b/eval/run_baselines.py index 2801c7e..b5eee62 100644 --- a/eval/run_baselines.py +++ b/eval/run_baselines.py @@ -13,9 +13,12 @@ otherwise a byte/4 heuristic (identical to `tokenfold-core`'s fallback) is used and the report labels itself accordingly. Nothing here touches the shipped Rust/npm/Python runtime. +Selectors: keep_all, forced_only, recency, frequency, bm25, llmlingua_style (a perplexity-free +self-information proxy). Compressor baselines: deterministic-tokenfold (Rust CLI). + Deliberately deferred (documented, not hidden — see `eval/tasks/v04/README.md`): - - RTK, RTK+tokenfold, deterministic-tokenfold (Rust CLI subprocess), LLMLingua-style, and the - unmodified Headroom Kompress-v2 achieved-token sweep are additional `SELECTORS` entries. + - RTK and RTK+tokenfold (external tool) and the unmodified Headroom Kompress-v2 achieved-token + sweep (needs the ML checkpoint) as additional baselines. - Real Tier-B public-repo corpora and project-disjoint train/test splits. - Structural (diff-hunk / JSON-container / AST) segmentation; v0.4-alpha segments by line. - An LLM judge for task success (the current scorer is a deterministic containment proxy). @@ -146,12 +149,40 @@ def sel_bm25(units: list[str], query: str) -> list[float]: return scores +def sel_llmlingua_style(units: list[str], query: str) -> list[float]: + """Perplexity-free LLMLingua-style proxy: keep high-information units, drop predictable / + redundant ones. Scores each unit by mean per-token self-information (surprisal, + `-log2 P(token)`) under a unigram model estimated from the document itself — a deterministic + stand-in for LLMLingua's small-LM token perplexity (the real method needs an LM at inference, + deferred: model-research.md keeps ML off the default path). Query-independent like `frequency`, + but an information-theoretic surprisal rather than a `1/df` heuristic, so boilerplate lines of + common tokens rank low and lines carrying rare/surprising content rank high.""" + counts: dict[str, int] = {} + total = 0 + for unit in units: + for tok in _tokens(unit): + counts[tok] = counts.get(tok, 0) + 1 + total += 1 + if total == 0: + return [0.0] * len(units) + scores = [] + for unit in units: + toks = _tokens(unit) + if not toks: + scores.append(0.0) + continue + surprisal = sum(-math.log2(counts[t] / total) for t in toks) / len(toks) + scores.append(surprisal) + return scores + + SELECTORS = { "keep_all": sel_keep_all, "forced_only": sel_forced_only, "recency": sel_recency, "frequency": sel_frequency, "bm25": sel_bm25, + "llmlingua_style": sel_llmlingua_style, } diff --git a/eval/tasks/v04/README.md b/eval/tasks/v04/README.md index d8d95c5..405d74c 100644 --- a/eval/tasks/v04/README.md +++ b/eval/tasks/v04/README.md @@ -5,6 +5,17 @@ research (`docs/solution-design/model-research.md`). Everything here is **shadow measures deterministic keep/drop selectors against downstream tasks; no model is involved and nothing reaches a served path. +## Coverage (77 Tier-A fixtures across 11 families) + +Spanning the required slices from model-research.md §Prerequisites: `log_qa`, `log_multi_service` +(logs/tool QA), `diff_review`, `code_patch` (diff review / change localization), `code_build_error` +(build/test failures), `json_schema`, `tool_call_json` (JSON/schema + tool calls), +`long_context_needle` (long mixed context with an id/hash/path needle), `ccr_marker` (CCR +reconstruction), and `rust_holdout` + `typescript_holdout` (the project-disjoint Rust/TS hard +slices). Each family has 7 fixtures (`_001`, `_010`-`_015`). Every fixture is gate-validated and +confirmed to *differentiate* selectors (at least one selector fails the task at the 25% ceiling, +so the report is discriminating rather than trivially 1.0 everywhere). + ## Fixture schema ```json @@ -15,7 +26,8 @@ nothing reaches a served path. "source": "the raw captured text to compress", "query": "the downstream question the compressed context must still answer", "gold_answer": "substring that must survive for the task to be answerable", - "critical_atoms": ["ids/hashes/paths that must survive regardless of the selector"] + "critical_atoms": ["ids/hashes/paths that must survive regardless of the selector"], + "notes": "optional: why this fixture discriminates selectors (which ones fail and why)" } ``` @@ -26,6 +38,9 @@ nothing reaches a served path. - **`gold_answer`** should live in a unit that is *not* a critical atom, so whether the task is answerable genuinely depends on the selector + token budget. That is what differentiates the baselines (and, later, a learned selector) instead of every policy trivially scoring 1.0. +- **`notes`** (optional, added from `_010` onward) documents the discrimination design intent per + fixture — which selectors are expected to fail the task at tight ceilings and why. The harness + ignores it; it exists for humans auditing/extending the corpus. ## Governance tiers (model-research.md §Prerequisites and Data) @@ -53,9 +68,12 @@ the harness falls back to the same byte/4 heuristic as `tokenfold-core` and labe ## Baseline kinds: selectors vs. compressors -- **Selectors** (`keep_all`, `forced_only`, `recency`, `frequency`, `bm25`) rank atomic units. - The harness force-keeps critical-atom units and enforces the exact token ceiling on them, so - 100% critical-atom survival and the ceiling are guarantees. +- **Selectors** (`keep_all`, `forced_only`, `recency`, `frequency`, `bm25`, `llmlingua_style`) + rank atomic units. `llmlingua_style` is a perplexity-free proxy — it ranks units by mean + per-token self-information (surprisal) under a document-derived unigram model, a deterministic + stand-in for LLMLingua's small-LM perplexity. The harness force-keeps critical-atom units and + enforces the exact token ceiling on them, so 100% critical-atom survival and the ceiling are + guarantees. - **Compressors** (`deterministic-tokenfold`) run a whole-pipeline best-effort compressor over the source — the harness does *not* force atoms through them, so their critical-atom survival and achieved ratio are **measured, not asserted**. `deterministic-tokenfold` shells out to the @@ -67,9 +85,9 @@ the harness falls back to the same byte/4 heuristic as `tokenfold-core` and labe ## Deferred to later v0.4-alpha work (not hidden) -- Remaining baselines: RTK and RTK+tokenfold (external tool), an LLMLingua-style selector, and - the unmodified Headroom Kompress-v2 achieved-token sweep. (`deterministic-tokenfold` is now - implemented as a compressor baseline — see above.) +- Remaining baselines: RTK and RTK+tokenfold (external tool) and the unmodified Headroom + Kompress-v2 achieved-token sweep (needs the ML checkpoint). (`deterministic-tokenfold` and + `llmlingua_style` are now implemented — see above.) - Tier-B public-repo corpora with license/revision manifests; project-disjoint train/test splits and near-dedup across splits. - Structural segmentation (diff hunks, JSON containers, AST/code blocks) — v0.4-alpha segments by diff --git a/eval/tasks/v04/ccr_marker_001.json b/eval/tasks/v04/ccr_marker_001.json new file mode 100644 index 0000000..189ddf4 --- /dev/null +++ b/eval/tasks/v04/ccr_marker_001.json @@ -0,0 +1,12 @@ +{ + "id": "ccr_marker_001", + "family": "ccr_marker", + "tier": "A", + "source": "$ artifactctl publish --repo prod ./build/app-2.14.0.tar.gz\n[10:22:01] connecting to registry registry.internal.example.com:8443\n[10:22:01] authenticated as ci-bot correlation-id=cid-4b8e2d1a\n[10:22:02] uploading layer 1/3 base-runtime size=41.2MB\n[10:22:03] uploading layer 2/3 dependencies size=88.7MB\n[10:22:05] uploading layer 3/3 application size=12.4MB\n[10:22:06] computing manifest digest from 3 layers\n[10:22:06] published digest sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08\n[10:22:06] assigned immutable tag prod-2.14.0-b317\n[10:22:07] registry quota remaining 214.5GB of 500GB\n[10:22:07] replication started to region eu-west-1\n[10:22:08] replication started to region ap-south-1\n[10:22:08] replication started to region us-east-2\n[10:22:09] audit event recorded status=OK\n[10:22:09] metrics flushed to telemetry pipeline\n[10:22:09] publish complete in 6.8s status=OK\n", + "query": "what sha256 manifest digest was published to the registry for this release?", + "gold_answer": "sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", + "critical_atoms": [ + "cid-4b8e2d1a" + ], + "notes": "CCR: the published digest must be reconstructable. bm25 keeps the digest line (shares 'sha256/digest/published/registry/manifest' with the query) so it reconstructs the value even at the 0.25 ceiling, while recency spends the budget on the later replication/audit/metrics lines and forced_only keeps only the marker line, so both drop the value. The audit marker cid-4b8e2d1a sits on the authentication line and is force-kept independent of the selector." +} diff --git a/eval/tasks/v04/ccr_marker_010.json b/eval/tasks/v04/ccr_marker_010.json new file mode 100644 index 0000000..516c5e3 --- /dev/null +++ b/eval/tasks/v04/ccr_marker_010.json @@ -0,0 +1,13 @@ +{ + "id": "ccr_marker_010", + "family": "ccr_marker", + "tier": "A", + "source": "$ flagctl audit export --window 24h --service edge-gateway\n[2026-03-11T09:01:00Z] flag-rollout adjust flag=checkout-v2 region=us-east percentage=5% actor=svc-flagbot\n[2026-03-11T09:02:00Z] flag-rollout adjust flag=dark-mode region=us-west percentage=62% actor=r-patel\n[2026-03-11T09:03:00Z] flag-rollout adjust flag=payments-retry region=eu-west percentage=18% actor=svc-flagbot\n[2026-03-11T09:04:00Z] change-request opened id=CR-88213 approver=ops-lead scope=flag-rollout\n[2026-03-11T09:05:00Z] flag-rollout adjust flag=onboarding-tour region=ap-south percentage=90% actor=m-santos\n[2026-03-11T09:06:00Z] flag-rollout adjust flag=comment-threads region=sa-east percentage=33% actor=svc-flagbot\n[2026-03-11T09:07:00Z] flag-rollout adjust flag=video-transcode region=us-east percentage=71% actor=i-choudhury\n[2026-03-11T09:08:00Z] flag-rollout adjust flag=price-cache region=us-west percentage=40% actor=svc-flagbot\n[2026-03-11T09:09:00Z] flag-rollout adjust flag=geo-routing region=eu-west percentage=15% actor=r-patel\n[2026-03-11T09:10:00Z] flag-rollout adjust flag=mobile-push region=ap-south percentage=55% actor=svc-flagbot\n[2026-03-11T09:11:00Z] flag-rollout adjust flag=email-digest region=sa-east percentage=27% actor=m-santos\n[2026-03-11T09:12:00Z] flag-rollout adjust flag=chat-widget region=us-east percentage=83% actor=svc-flagbot\n[2026-03-11T09:13:00Z] flag-rollout adjust flag=ab-test-banner region=us-west percentage=9% actor=i-choudhury\n[2026-03-11T09:14:00Z] flag-rollout adjust flag=inventory-sync region=eu-west percentage=46% actor=svc-flagbot\n[2026-03-11T09:15:00Z] flag-rollout adjust flag=refund-flow region=ap-south percentage=61% actor=r-patel\n[2026-03-11T09:16:00Z] flag-rollout adjust flag=loyalty-points region=sa-east percentage=24% actor=svc-flagbot\n[2026-03-11T09:17:00Z] flag-rollout adjust flag=address-autocomplete region=us-east percentage=77% actor=m-santos\n[2026-03-11T09:18:00Z] flag-rollout adjust flag=tax-calc-v3 region=us-west percentage=13% actor=svc-flagbot\n[2026-03-11T09:19:00Z] flag-rollout adjust flag=shipping-estimate region=eu-west percentage=59% actor=i-choudhury\n[2026-03-11T09:20:00Z] flag-rollout adjust flag=search-rerank region=us-east percentage=37% actor=svc-flagbot\n[2026-03-11T09:21:00Z] flag-rollout adjust flag=fraud-score-v2 region=ap-south percentage=68% actor=r-patel\n[2026-03-11T09:22:00Z] flag-rollout adjust flag=recs-carousel region=sa-east percentage=21% actor=svc-flagbot\n[2026-03-11T09:23:00Z] flag-rollout adjust flag=session-replay region=us-east percentage=44% actor=m-santos\n[2026-03-11T09:24:00Z] flag-rollout adjust flag=dark-launch-api region=us-west percentage=87% actor=svc-flagbot\n[2026-03-11T09:25:00Z] flag-rollout adjust flag=feature-gate-x region=eu-west percentage=30% actor=i-choudhury\n[2026-03-11T09:26:00Z] flag-rollout adjust flag=quota-throttle region=ap-south percentage=52% actor=svc-flagbot\n[2026-03-11T09:27:00Z] flag-rollout adjust flag=cdn-failover region=sa-east percentage=11% actor=r-patel\n[2026-03-11T09:28:00Z] flag-rollout adjust flag=image-lazyload region=us-east percentage=65% actor=svc-flagbot\n[2026-03-11T09:29:00Z] deployment batch verified batch-sha=1f3c9a7e2b804d5f status=signed\n[2026-03-11T09:30:00Z] flag-rollout adjust flag=coupon-stack region=us-west percentage=38% actor=m-santos\n[2026-03-11T09:31:00Z] flag-rollout adjust flag=subscription-pause region=eu-west percentage=74% actor=svc-flagbot\n[2026-03-11T09:32:00Z] flag-rollout adjust flag=wishlist-share region=ap-south percentage=20% actor=i-choudhury\n[2026-03-11T09:33:00Z] flag-rollout adjust flag=review-prompt region=sa-east percentage=48% actor=svc-flagbot\n[2026-03-11T09:34:00Z] flag-rollout adjust flag=live-chat-v2 region=us-east percentage=8% actor=r-patel\n[2026-03-11T09:35:00Z] flag-rollout adjust flag=map-cluster region=us-west percentage=56% actor=svc-flagbot\n[2026-03-11T09:36:00Z] flag-rollout adjust flag=search-typo-tolerant region=eu-west percentage=29% actor=m-santos\n[2026-03-11T09:37:00Z] flag-rollout adjust flag=checkout-express region=ap-south percentage=81% actor=svc-flagbot\n[2026-03-11T09:38:00Z] flag-rollout adjust flag=notif-batching region=sa-east percentage=43% actor=i-choudhury\n[2026-03-11T09:39:00Z] flag-rollout audit export complete rows=36\n", + "query": "what rollout percentage was set for the search-rerank flag?", + "gold_answer": "flag=search-rerank region=us-east percentage=37% actor=svc-flagbot", + "critical_atoms": [ + "CR-88213", + "batch-sha=1f3c9a7e2b804d5f" + ], + "notes": "CCR: the flag's rollout percentage must be reconstructable at a tight ~10% budget. All 36 flag-adjustment rows share the identical 'flag-rollout adjust flag=... region=... percentage=...% actor=...' template, so recency (the row sits roughly mid-log, not near the tail), frequency and llmlingua_style (every row carries an equally rare flag-name/percentage/actor combination, so rare-token density does not single out this one) all fail to retain the search-rerank row even at the loosest 0.5 ceiling, and forced_only never carries any row past the two marker lines. bm25 is the only selector that retains the row at every ceiling down to 0.1, because 'search' and 'rerank' are the only tokens it shares with the query, and the one decoy row that could tie on 'search' ('search-typo-tolerant') does not also match 'rerank'. The change-request id=CR-88213 and deployment batch-sha=1f3c9a7e2b804d5f sit on separate audit lines and are force-kept independent of the selector." +} diff --git a/eval/tasks/v04/ccr_marker_011.json b/eval/tasks/v04/ccr_marker_011.json new file mode 100644 index 0000000..24b1a1b --- /dev/null +++ b/eval/tasks/v04/ccr_marker_011.json @@ -0,0 +1,13 @@ +{ + "id": "ccr_marker_011", + "family": "ccr_marker", + "tier": "A", + "source": "$ certinv scan --scope internal-services --format ndjson\n[2026-04-02T11:01:00Z] cert-scan record cn=api-gateway.internal issuer=internal-ca-01 serial=0x11A2C3 notAfter=2026-11-02T00:00:00Z\n[2026-04-02T11:02:00Z] cert-scan record cn=auth-svc.internal issuer=internal-ca-02 serial=0x2B4F09 notAfter=2027-05-19T00:00:00Z\n[2026-04-02T11:03:12Z] cert-scan job started scan-job=8f2e19c4 operator=svc-certbot scope=internal-services\n[2026-04-02T11:04:00Z] cert-scan record cn=billing-worker.internal issuer=internal-ca-01 serial=0x3C1D77 notAfter=2026-09-30T00:00:00Z\n[2026-04-02T11:05:00Z] cert-scan record cn=search-index.internal issuer=internal-ca-03 serial=0x4D8E12 notAfter=2027-02-11T00:00:00Z\n[2026-04-02T11:06:00Z] cert-scan record cn=notification-hub.internal issuer=internal-ca-02 serial=0x5E9F44 notAfter=2026-12-25T00:00:00Z\n[2026-04-02T11:07:00Z] cert-scan record cn=media-cdn.internal issuer=internal-ca-01 serial=0x6FA061 notAfter=2027-07-04T00:00:00Z\n[2026-04-02T11:08:00Z] cert-scan record cn=admin-console.internal issuer=internal-ca-03 serial=0x70B188 notAfter=2026-10-08T00:00:00Z\n[2026-04-02T11:09:00Z] cert-scan record cn=partner-bridge.internal issuer=internal-ca-02 serial=0x81C2AA notAfter=2027-01-30T00:00:00Z\n[2026-04-02T11:10:00Z] cert-scan record cn=webhook-relay.internal issuer=internal-ca-01 serial=0x92D3CC notAfter=2026-08-17T00:00:00Z\n[2026-04-02T11:11:00Z] cert-scan record cn=payments-worker.internal issuer=internal-ca-02 serial=0xA3E4EE notAfter=2027-03-22T00:00:00Z\n[2026-04-02T11:12:00Z] cert-scan record cn=ledger-sync.internal issuer=internal-ca-03 serial=0xB4F510 notAfter=2026-11-29T00:00:00Z\n[2026-04-02T11:13:00Z] cert-scan record cn=fraud-engine.internal issuer=internal-ca-01 serial=0xC50632 notAfter=2027-04-15T00:00:00Z\n[2026-04-02T11:14:00Z] cert-scan record cn=payments-api.internal issuer=internal-ca-02 serial=0x4Ab2F9 notAfter=2027-01-14T00:00:00Z\n[2026-04-02T11:15:00Z] cert-scan record cn=mail-relay.internal issuer=internal-ca-03 serial=0xD61754 notAfter=2026-09-06T00:00:00Z\n[2026-04-02T11:16:00Z] cert-scan record cn=session-store.internal issuer=internal-ca-01 serial=0xE72876 notAfter=2027-06-01T00:00:00Z\n[2026-04-02T11:17:00Z] cert-scan record cn=config-service.internal issuer=internal-ca-02 serial=0xF83998 notAfter=2026-12-11T00:00:00Z\n[2026-04-02T11:18:00Z] cert-scan record cn=telemetry-collector.internal issuer=internal-ca-03 serial=0x094ABA notAfter=2027-02-27T00:00:00Z\n[2026-04-02T11:19:00Z] cert-scan record cn=image-resize.internal issuer=internal-ca-01 serial=0x1A5BDC notAfter=2026-10-19T00:00:00Z\n[2026-04-02T11:20:00Z] cert-scan record cn=export-worker.internal issuer=internal-ca-02 serial=0x2B6CFE notAfter=2027-05-05T00:00:00Z\n[2026-04-02T11:21:00Z] cert-scan record cn=payments-api-canary.internal issuer=internal-ca-02 serial=0x3C7D10 notAfter=2026-08-29T00:00:00Z\n[2026-04-02T11:22:00Z] cert-scan record cn=feature-flags.internal issuer=internal-ca-03 serial=0x4D8E32 notAfter=2027-03-08T00:00:00Z\n[2026-04-02T11:23:00Z] cert-scan record cn=rate-limiter.internal issuer=internal-ca-01 serial=0x5E9F54 notAfter=2026-11-16T00:00:00Z\n[2026-04-02T11:24:00Z] cert-scan record cn=graphql-gateway.internal issuer=internal-ca-02 serial=0x6FA076 notAfter=2027-07-21T00:00:00Z\n[2026-04-02T11:25:00Z] cert-scan record cn=batch-scheduler.internal issuer=internal-ca-03 serial=0x70B198 notAfter=2026-09-13T00:00:00Z\n[2026-04-02T11:26:00Z] cert-scan record cn=audit-log.internal issuer=internal-ca-01 serial=0x81C2BA notAfter=2027-01-02T00:00:00Z\n[2026-04-02T11:27:47Z] cert-scan compliance review ticket=SEC-5521 status=acknowledged\n[2026-04-02T11:28:00Z] cert-scan record cn=secrets-broker.internal issuer=internal-ca-02 serial=0x92D3DC notAfter=2026-12-04T00:00:00Z\n[2026-04-02T11:29:00Z] cert-scan record cn=recs-engine.internal issuer=internal-ca-03 serial=0xA3E4FE notAfter=2027-04-28T00:00:00Z\n[2026-04-02T11:30:00Z] cert-scan record cn=chat-gateway.internal issuer=internal-ca-01 serial=0xB4F520 notAfter=2026-10-25T00:00:00Z\n[2026-04-02T11:31:00Z] cert-scan record cn=geo-router.internal issuer=internal-ca-02 serial=0xC50642 notAfter=2027-06-14T00:00:00Z\n[2026-04-02T11:32:00Z] cert-scan record cn=quota-service.internal issuer=internal-ca-03 serial=0xD61764 notAfter=2026-08-08T00:00:00Z\n[2026-04-02T11:33:00Z] cert-scan record cn=wishlist-svc.internal issuer=internal-ca-01 serial=0xE72886 notAfter=2027-02-19T00:00:00Z\n[2026-04-02T11:34:00Z] cert-scan record cn=review-svc.internal issuer=internal-ca-02 serial=0xF839A8 notAfter=2026-11-07T00:00:00Z\n[2026-04-02T11:35:00Z] cert-scan job complete records=32 status=OK\n", + "query": "When does the TLS certificate for the payments-api service stop being valid for clients?", + "gold_answer": "cn=payments-api.internal issuer=internal-ca-02 serial=0x4Ab2F9 notAfter=2027-01-14T00:00:00Z", + "critical_atoms": [ + "scan-job=8f2e19c4", + "SEC-5521" + ], + "notes": "CCR: the certificate's notAfter date must be reconstructable at a tight ~10% budget. The 32 cert-scan rows never use 'expire'/'valid'/'trust' anywhere, only a bare notAfter= field, so the query's descriptive wording gives frequency, recency and llmlingua_style no purchase and none of them retain the payments-api row even at the loosest 0.5 ceiling; forced_only never gets past the two marker lines. bm25 retains it at every ceiling down to 0.1 by matching the 'payments' and 'api' identifier tokens, despite a same-family decoy row ('payments-api-canary.internal') that shares both tokens and two more ('payments-worker.internal', 'api-gateway.internal') that each share one. The scan-job=8f2e19c4 and ticket=SEC-5521 ids sit on separate audit lines and are force-kept independent of the selector." +} diff --git a/eval/tasks/v04/ccr_marker_012.json b/eval/tasks/v04/ccr_marker_012.json new file mode 100644 index 0000000..d2d087c --- /dev/null +++ b/eval/tasks/v04/ccr_marker_012.json @@ -0,0 +1,13 @@ +{ + "id": "ccr_marker_012", + "family": "ccr_marker", + "tier": "A", + "source": "$ backupctl report --window 24h --tier nightly\n[2026-05-06T02:01:00Z] backup-agent job=nightly-full-orders-db status=completed duration=233s size=41GB checksum=a1e7c930\n[2026-05-06T02:02:00Z] backup-agent job=nightly-incr-sessions status=completed duration=58s size=6GB checksum=b2f8da41\n[2026-05-06T02:03:00Z] backup-agent job=nightly-full-catalog-db status=completed duration=812s size=118GB checksum=c309eb52\n[2026-05-06T02:04:00Z] backup-agent job=nightly-incr-audit-log status=completed duration=44s size=3GB checksum=d41afc63\n[2026-05-06T02:05:00Z] backup-audit manifest opened manifest=BKM-77410 operator=svc-backupbot\n[2026-05-06T02:06:00Z] backup-agent job=nightly-full-billing-db status=completed duration=601s size=97GB checksum=e52b0d74\n[2026-05-06T02:07:00Z] backup-agent job=nightly-incr-search-index status=completed duration=122s size=14GB checksum=f63c1e85\n[2026-05-06T02:08:00Z] backup-agent job=nightly-full-analytics-db status=completed duration=947s size=203GB checksum=074d2f96\n[2026-05-06T02:09:00Z] backup-agent job=nightly-incr-notifications status=completed duration=39s size=2GB checksum=185e30a7\n[2026-05-06T02:10:00Z] backup-agent job=nightly-full-inventory-db status=completed duration=355s size=62GB checksum=296f41b8\n[2026-05-06T02:11:00Z] backup-agent job=nightly-incr-fraud-log status=completed duration=67s size=5GB checksum=3a7052c9\n[2026-05-06T02:12:00Z] backup-agent job=nightly-full-config-db status=completed duration=91s size=9GB checksum=4b8163da\n[2026-05-06T02:13:00Z] backup-agent job=nightly-incr-recs-cache status=completed duration=51s size=4GB checksum=5c9274eb\n[2026-05-06T02:14:00Z] backup-agent job=nightly-full-ledger-db status=completed duration=1188s size=240GB checksum=6da385fc\n[2026-05-06T02:15:00Z] backup-agent job=nightly-incr-webhook-log status=completed duration=46s size=3GB checksum=7eb4960d\n[2026-05-06T02:16:00Z] backup-agent job=nightly-full-media-meta-db status=completed duration=288s size=53GB checksum=8fc5a71e\n[2026-05-06T02:17:00Z] backup-agent job=nightly-incr-loyalty-log status=completed duration=62s size=5GB checksum=90d6b82f\n[2026-05-06T02:18:00Z] backup-agent job=nightly-full-partner-db status=completed duration=402s size=71GB checksum=a1f7c931\n[2026-05-06T02:19:00Z] backup-agent job=nightly-incr-quota-log status=completed duration=33s size=2GB checksum=b2f8da42\n[2026-05-06T02:20:00Z] backup-agent job=nightly-full-mail-db status=completed duration=176s size=22GB checksum=c309eb53\n[2026-05-06T02:21:00Z] backup-agent job=nightly-incr-session-replay status=completed duration=88s size=11GB checksum=d41afc64\n[2026-05-06T02:22:00Z] backup-agent job=nightly-full-secrets-db status=completed duration=64s size=6GB checksum=e52b0d75\n[2026-05-06T02:23:00Z] backup-agent job=nightly-incr-chat-log status=completed duration=55s size=4GB checksum=f63c1e86\n[2026-05-06T02:24:00Z] backup-agent job=nightly-full-geo-db status=completed duration=219s size=34GB checksum=074d2f97\n[2026-05-06T02:25:00Z] backup-agent job=nightly-incr-tax-log status=completed duration=41s size=3GB checksum=185e30a8\n[2026-05-06T02:26:00Z] backup-audit retention policy ref=RET-2299 status=applied\n[2026-05-06T02:27:00Z] backup-agent job=nightly-full-wishlist-db status=completed duration=97s size=8GB checksum=296f41b9\n[2026-05-06T02:28:00Z] backup-agent job=nightly-incr-review-log status=completed duration=60s size=5GB checksum=3a7052ca\n[2026-05-06T02:29:00Z] backup-agent job=nightly-full-address-db status=completed duration=143s size=17GB checksum=4b8163db\n[2026-05-06T02:30:00Z] backup-agent job=nightly-incr-image-meta status=completed duration=76s size=9GB checksum=5c9274ec\n[2026-05-06T02:31:00Z] backup-agent job=nightly-full-export-db status=completed duration=331s size=55GB checksum=6da385fd\n[2026-05-06T02:32:00Z] backup-agent job=nightly-incr-graphql-log status=completed duration=48s size=3GB checksum=7eb4960e\n[2026-05-06T02:33:00Z] backup-agent job=nightly-full-batch-db status=completed duration=205s size=29GB checksum=8fc5a71f\n[2026-05-06T02:34:00Z] backup-agent job=nightly-incr-rate-log status=completed duration=37s size=2GB checksum=90d6b830\n[2026-05-06T02:35:00Z] backup-agent report complete jobs=32\n", + "query": "what was the duration of the nightly-full-analytics-db backup job?", + "gold_answer": "job=nightly-full-analytics-db status=completed duration=947s size=203GB checksum=074d2f96", + "critical_atoms": [ + "BKM-77410", + "RET-2299" + ], + "notes": "CCR: the nightly-full-analytics-db job's duration must be reconstructable at a tight ~10% budget. All 32 rows share the 'backup-agent job=... status=completed duration=...s size=...GB checksum=...' template, each with its own unique-looking checksum. At the loosest 0.5 ceiling frequency and llmlingua_style still retain the row (its job-name and duration tokens are narrowly the rarest), but tightening to 0.25 and then 0.1 drops it from both, because the shrinking budget now only fits a handful of rows and every row's own checksum makes it look equally rare. recency fails at every ceiling since the row sits mid-log. bm25 is the only selector that survives all three ceilings, because it directly matches the query's 'duration' and 'nightly-full-analytics-db' tokens. The manifest=BKM-77410 and ref=RET-2299 ids sit on separate audit lines and are force-kept independent of the selector." +} diff --git a/eval/tasks/v04/ccr_marker_013.json b/eval/tasks/v04/ccr_marker_013.json new file mode 100644 index 0000000..304750d --- /dev/null +++ b/eval/tasks/v04/ccr_marker_013.json @@ -0,0 +1,13 @@ +{ + "id": "ccr_marker_013", + "family": "ccr_marker", + "tier": "A", + "source": "$ incidentctl timeline export --incident INC-40217 --window 2h\n[2026-02-19T14:00:00Z] event component=db-primary action=healthcheck-pass detail=latency-nominal-4ms-window\n[2026-02-19T14:01:00Z] event component=queue-worker action=retry-scheduled detail=backoff-window-2s-applied\n[2026-02-19T14:02:00Z] event component=cdn-node action=cache-evict detail=lru-sweep-window-applied\n[2026-02-19T14:03:00Z] event component=auth-gateway action=scale-up-triggered detail=replica-count-set-to-6\n[2026-02-19T14:04:00Z] event component=session-cache action=connection-reset detail=pool-recycled-window-applied\n[2026-02-19T14:05:00Z] event component=payment-processor action=alert-triggered detail=queue-depth-set-high\n[2026-02-19T14:06:00Z] incident-audit escalation opened ref=ESC-30841 severity=SEV1\n[2026-02-19T14:07:00Z] event component=search-index action=failover-triggered detail=replica-promoted-window\n[2026-02-19T14:08:00Z] event component=billing-worker action=threshold-exceeded detail=cpu-usage-set-82pct\n[2026-02-19T14:09:00Z] event component=notification-svc action=page-triggered detail=oncall-paged-window-applied\n[2026-02-19T14:10:00Z] event component=db-replica-2 action=latency-spike-triggered detail=read-path-set-slow\n[2026-02-19T14:11:00Z] event component=resolver-7 action=config-applied detail=upstream-record-ttl-set-low\n[2026-02-19T14:12:00Z] event component=cdn-node-2 action=canary-rollback-triggered detail=bad-build-reverted-window\n[2026-02-19T14:13:00Z] event component=queue-worker-2 action=circuit-breaker-triggered detail=downstream-set-503\n[2026-02-19T14:14:00Z] event component=db-primary action=auto-heal-triggered detail=replica-resynced-window-applied\n[2026-02-19T14:15:00Z] event component=deploy-agent action=deployment-triggered detail=version-set-to-4-7-2\n[2026-02-19T14:16:00Z] event component=search-index-2 action=gc-pause-triggered detail=pause-window-320ms\n[2026-02-19T14:17:00Z] event component=billing-worker-2 action=cpu-throttle-triggered detail=cgroup-limit-window-applied\n[2026-02-19T14:18:00Z] event component=queue-worker-3 action=queue-backlog-triggered detail=depth-set-to-14000\n[2026-02-19T14:19:00Z] event component=session-cache-2 action=healthcheck-pass detail=latency-nominal-6ms-window\n[2026-02-19T14:20:00Z] event component=auth-gateway-2 action=retry-scheduled detail=backoff-window-1s-applied\n[2026-02-19T14:21:00Z] event component=cdn-node-3 action=cache-evict detail=ttl-sweep-window-applied\n[2026-02-19T14:22:00Z] event component=db-replica-3 action=scale-up-triggered detail=replica-count-set-to-4\n[2026-02-19T14:23:00Z] event component=payment-processor-2 action=alert-triggered detail=error-rate-set-high\n[2026-02-19T14:24:00Z] event component=notification-svc-2 action=failover-triggered detail=queue-promoted-window\n[2026-02-19T14:25:00Z] event component=resolver-8 action=threshold-exceeded detail=conn-usage-set-88pct\n[2026-02-19T14:26:00Z] event component=search-index-3 action=page-triggered detail=oncall-paged-window-applied\n[2026-02-19T14:27:00Z] event component=billing-worker-3 action=latency-spike-triggered detail=write-path-set-slow\n[2026-02-19T14:28:00Z] event component=queue-worker-4 action=canary-rollback-triggered detail=bad-config-reverted-window\n[2026-02-19T14:29:00Z] incident-audit postmortem-doc filed ref=PM-91027 status=draft\n[2026-02-19T14:30:00Z] event component=db-primary-2 action=circuit-breaker-triggered detail=downstream-set-500\n[2026-02-19T14:31:00Z] event component=cdn-node-4 action=auto-heal-triggered detail=node-restarted-window-applied\n[2026-02-19T14:32:00Z] event component=auth-gateway-3 action=deployment-triggered detail=version-set-to-4-7-3\n[2026-02-19T14:33:00Z] event component=session-cache-3 action=gc-pause-triggered detail=pause-window-210ms\n[2026-02-19T14:34:00Z] incident-audit timeline export complete events=32\n", + "query": "What ultimately caused the production outage last night?", + "gold_answer": "component=resolver-7 action=config-applied detail=upstream-record-ttl-set-low", + "critical_atoms": [ + "ESC-30841", + "PM-91027" + ], + "notes": "CCR: the actual root cause of the outage must be reconstructable at a tight ~10% budget. The row ('component=resolver-7 action=config-applied detail=upstream-record-ttl-set-low') shares no token at all with the query ('what ultimately caused the production outage last night'), unlike most of the other 32 event rows, which use an '-triggered' verb that echoes the query's framing without being the actual cause. With zero query-token overlap bm25 falls back to its stable earliest-first tie-break, which happens to still include the row at the loose 0.5 ceiling (it is an early-ish row) but drops it as soon as the ceiling tightens to 0.25. frequency and llmlingua_style hold onto the row through 0.5 and 0.25 on rare-token density, then also lose it once the ceiling tightens further to 0.1, where the budget only admits a handful of rows. recency fails at every ceiling since the row sits mid-log. Only keep_all reconstructs it at the 0.1 ceiling. The escalation ref=ESC-30841 and postmortem ref=PM-91027 ids sit on separate audit lines and are force-kept independent of the selector." +} diff --git a/eval/tasks/v04/ccr_marker_014.json b/eval/tasks/v04/ccr_marker_014.json new file mode 100644 index 0000000..d86cc94 --- /dev/null +++ b/eval/tasks/v04/ccr_marker_014.json @@ -0,0 +1,13 @@ +{ + "id": "ccr_marker_014", + "family": "ccr_marker", + "tier": "A", + "source": "$ iamctl audit export --window 24h --realm prod\n[2026-01-08T16:01:00Z] access-grant granted role=editor to user=j-alvarez region=us-east\n[2026-01-08T16:02:00Z] access-grant granted role=viewer to user=t-nakamura region=us-west\n[2026-01-08T16:03:00Z] access-grant granted role=billing-viewer to user=s-oyelaran region=eu-west\n[2026-01-08T16:04:00Z] access-grant granted role=support-agent to user=k-lindqvist region=ap-south\n[2026-01-08T16:05:00Z] access-grant granted role=editor to user=d-farouk region=sa-east\n[2026-01-08T16:06:00Z] access-audit review opened id=IAM-60215 approver=sec-lead\n[2026-01-08T16:07:00Z] access-grant granted role=viewer to user=p-costa region=us-east\n[2026-01-08T16:08:00Z] access-grant granted role=release-manager to user=w-huang region=us-west\n[2026-01-08T16:09:00Z] access-grant granted role=viewer to user=n-okafor region=eu-west\n[2026-01-08T16:10:00Z] access-grant granted role=billing-viewer to user=l-marchetti region=ap-south\n[2026-01-08T16:11:00Z] access-grant granted role=support-agent to user=a-ivanova region=sa-east\n[2026-01-08T16:12:00Z] access-grant granted role=editor to user=r-benali region=us-east\n[2026-01-08T16:13:00Z] access-grant granted role=on-call to user=m-kowalski region=us-west\n[2026-01-08T16:14:00Z] access-grant granted role=viewer to user=g-tanaka region=eu-west\n[2026-01-08T16:15:00Z] access-grant granted role=editor to user=f-dubois region=ap-south\n[2026-01-08T16:16:00Z] access-grant granted role=admin to user=c-whitfield region=us-east\n[2026-01-08T16:17:00Z] access-grant granted role=viewer to user=h-solberg region=sa-east\n[2026-01-08T16:18:00Z] access-grant granted role=support-agent to user=b-nazarov region=us-west\n[2026-01-08T16:19:00Z] access-grant granted role=editor to user=y-castillo region=eu-west\n[2026-01-08T16:20:00Z] access-grant granted role=release-manager to user=e-abramov region=ap-south\n[2026-01-08T16:21:00Z] access-grant granted role=viewer to user=o-adeyemi region=sa-east\n[2026-01-08T16:22:00Z] access-grant granted role=on-call to user=z-brennan region=us-east\n[2026-01-08T16:23:00Z] access-grant granted role=billing-viewer to user=q-fenwick region=us-west\n[2026-01-08T16:24:00Z] access-grant granted role=editor to user=v-petrova region=eu-west\n[2026-01-08T16:25:00Z] access-audit change-batch verified batch=IAMB-50934 status=signed\n[2026-01-08T16:26:00Z] access-grant granted role=support-agent to user=u-mbeki region=ap-south\n[2026-01-08T16:27:00Z] access-grant granted role=viewer to user=x-delacroix region=sa-east\n[2026-01-08T16:28:00Z] access-grant granted role=editor to user=j-hollander region=us-east\n[2026-01-08T16:29:00Z] access-grant granted role=on-call to user=s-vukovic region=us-west\n[2026-01-08T16:30:00Z] access-grant granted role=viewer to user=k-marino region=eu-west\n[2026-01-08T16:31:00Z] access-grant granted role=billing-viewer to user=d-nkemelu region=ap-south\n[2026-01-08T16:32:00Z] access-grant granted role=release-manager to user=m-ostrowski region=sa-east\n[2026-01-08T16:33:00Z] access-grant granted role=editor to user=t-birkeland region=us-east\n[2026-01-08T16:34:00Z] access-grant audit export complete rows=31\n", + "query": "which user was granted the admin role?", + "gold_answer": "granted role=admin to user=c-whitfield region=us-east", + "critical_atoms": [ + "IAM-60215", + "batch=IAMB-50934" + ], + "notes": "CCR: which user was granted the admin role must be reconstructable at a tight ~10% budget. All 31 rows share the 'access-grant granted role=... to user=... region=...' template, so bm25 (matching the rare 'admin' role token, which appears on exactly one row) and frequency (the row's rare user-handle and role tokens) both retain it at every ceiling down to 0.1. recency fails at every ceiling since the row sits mid-log. llmlingua_style retains the row at the loose 0.5 ceiling but loses it once the budget tightens to 0.25 and 0.1, when its surprisal ranking favors a different subset of the equally rare user-handle rows instead. The access-audit id=IAM-60215 and batch=IAMB-50934 ids sit on separate audit lines and are force-kept independent of the selector." +} diff --git a/eval/tasks/v04/ccr_marker_015.json b/eval/tasks/v04/ccr_marker_015.json new file mode 100644 index 0000000..cc4d01a --- /dev/null +++ b/eval/tasks/v04/ccr_marker_015.json @@ -0,0 +1,13 @@ +{ + "id": "ccr_marker_015", + "family": "ccr_marker", + "tier": "A", + "source": "$ migratectl history export --db prod --window full\n[2026-06-14T03:05:00Z] migrator changeset=00449_add_index_customers_email applied=true duration=1.2s\n[2026-06-14T03:06:00Z] migrator changeset=00450_backfill_null_country applied=true duration=1.2s\n[2026-06-14T03:07:00Z] migration-audit window opened id=MIG-70318 operator=svc-migratebot\n[2026-06-14T03:08:00Z] migrator changeset=00451_add_fk_carts_customer applied=true duration=1.2s\n[2026-06-14T03:09:00Z] migrator changeset=00452_drop_legacy_column_notes applied=true duration=1.2s\n[2026-06-14T03:10:00Z] migrator changeset=00453_add_index_payments_status applied=true duration=1.2s\n[2026-06-14T03:11:00Z] migrator changeset=00454_widen_varchar_phone applied=true duration=1.2s\n[2026-06-14T03:12:00Z] migrator changeset=00455_add_check_constraint_age applied=true duration=1.2s\n[2026-06-14T03:13:00Z] migrator changeset=00456_backfill_default_locale applied=true duration=1.2s\n[2026-06-14T03:14:00Z] migrator changeset=00457_add_index_carts_updated_at applied=true duration=1.2s\n[2026-06-14T03:15:00Z] migrator changeset=00458_rename_column_addr2_to_line2 applied=true duration=1.2s\n[2026-06-14T03:16:00Z] migrator changeset=00459_add_fk_disputes_order applied=true duration=1.2s\n[2026-06-14T03:17:00Z] migrator changeset=00460_drop_index_old_state applied=true duration=1.2s\n[2026-06-14T03:18:00Z] migrator changeset=00461_add_index_orders_created_at applied=true duration=1.2s\n[2026-06-14T03:19:00Z] migrator changeset=00462_backfill_null_region applied=true duration=1.2s\n[2026-06-14T03:20:00Z] migrator changeset=00463_add_fk_invoices_customer applied=true duration=1.2s\n[2026-06-14T03:21:00Z] migrator changeset=00464_drop_legacy_column_flags applied=true duration=1.2s\n[2026-06-14T03:22:00Z] migrator changeset=00465_add_index_sessions_user_id applied=true duration=1.2s\n[2026-06-14T03:23:00Z] migrator changeset=00466_widen_varchar_email applied=true duration=1.2s\n[2026-06-14T03:24:00Z] migrator changeset=00467_add_check_constraint_qty applied=true duration=1.2s\n[2026-06-14T03:25:00Z] migrator changeset=00468_backfill_default_currency applied=true duration=1.2s\n[2026-06-14T03:26:00Z] migrator changeset=00469_add_index_events_ts applied=true duration=1.2s\n[2026-06-14T03:27:00Z] migrator changeset=00470_rename_column_addr_to_address applied=true duration=1.2s\n[2026-06-14T03:28:00Z] migrator changeset=00471_add_fk_refunds_order applied=true duration=1.2s\n[2026-06-14T03:29:00Z] migrator changeset=00472_drop_index_old_status applied=true duration=1.2s\n[2026-06-14T03:30:00Z] migrator changeset=00473_add_index_users_last_login applied=true duration=1.2s\n[2026-06-14T03:31:00Z] migrator changeset=00474_backfill_timezone_default applied=true duration=1.2s\n[2026-06-14T03:32:00Z] migrator changeset=00475_add_partition_events_2026 applied=true duration=1.2s\n[2026-06-14T03:33:00Z] migrator changeset=00476_add_fk_shipments_carrier applied=true duration=1.2s\n[2026-06-14T03:34:00Z] migrator changeset=00477_widen_int_to_bigint_counts applied=true duration=1.2s\n[2026-06-14T03:35:00Z] migrator changeset=00478_add_index_audit_log_actor applied=true duration=1.2s\n[2026-06-14T03:36:00Z] migrator changeset=00479_backfill_null_locale applied=true duration=1.2s\n[2026-06-14T03:37:00Z] migrator changeset=00480_drop_unused_table_staging applied=true duration=1.2s\n[2026-06-14T03:38:00Z] migrator changeset=00481_add_check_constraint_price applied=true duration=1.2s\n[2026-06-14T03:39:00Z] migrator changeset=00482_rename_index_orders_status applied=true duration=1.2s\n[2026-06-14T03:40:00Z] migration-audit changeset checksum recorded manifest-sha=b7c4e19f0a2d\n[2026-06-14T03:41:00Z] migration-audit post-run snapshot taken snapshot=prod-schema-post\n[2026-06-14T03:42:00Z] migration-audit notified subscribers channel=#db-migrations\n[2026-06-14T03:43:00Z] migrator run complete changesets=34 status=idle\n", + "query": "which changeset finalized the schema cutover, i.e. what is the schema currently on?", + "gold_answer": "changeset=00482_rename_index_orders_status applied=true duration=1.2s", + "critical_atoms": [ + "MIG-70318", + "manifest-sha=b7c4e19f0a2d" + ], + "notes": "CCR: the changeset that finalized the schema cutover must be reconstructable at a tight ~10% budget. The word 'changeset' appears on nearly every one of the 34 migration rows, so it contributes no discriminating bm25 signal, and nothing in the log actually names the field 'schema' or 'version' -- the row is identifiable only by being the last-applied entry in the sequence. bm25, frequency and llmlingua_style all fail to retain it at every ceiling, since none of them privileges trailing position over the sea of near-identical 'migrator changeset=... applied=true' rows. recency finds it at the 0.5 and 0.25 ceilings by favoring the tail of the log, but at the tight 0.1 ceiling the budget is too small to reach past the trailing snapshot/notify/manifest bookkeeping lines to the changeset row itself, so recency fails there too and only keep_all reconstructs it. The migration-audit id=MIG-70318 and manifest-sha=b7c4e19f0a2d ids sit on separate lines and are force-kept independent of the selector." +} diff --git a/eval/tasks/v04/code_build_error_001.json b/eval/tasks/v04/code_build_error_001.json new file mode 100644 index 0000000..8836d13 --- /dev/null +++ b/eval/tasks/v04/code_build_error_001.json @@ -0,0 +1,12 @@ +{ + "id": "code_build_error_001", + "family": "code_build_error", + "tier": "A", + "source": "============================= test session starts ==============================\nplatform linux -- Python 3.11.6, pytest-8.1.1, pluggy-1.4.0\nrootdir: /home/ci/checkout-service\ncollected 14 items\n\ntests/test_cart.py ...... [ 42%]\ntests/test_pricing.py ...F.. [ 85%]\ntests/test_orders.py .. [100%]\n\n=================================== FAILURES ===================================\n_________________ test_discount_never_exceeds_subtotal _________________________\n\n def test_discount_never_exceeds_subtotal():\n cart = Cart(items=[Item(\"sku-88\", 12.50)])\n pricing = Pricing(rules=load_rules(\"rules/promo.yaml\"))\n> assert pricing.apply(cart).discount <= cart.subtotal\nE AssertionError: computed discount 15.00 exceeds cart subtotal 12.50\nE + where 15.00 = Pricing.apply(...).discount\n\ntests/test_pricing.py:47: AssertionError\n=========================== short test summary info ============================\nFAILED tests/test_pricing.py::test_discount_never_exceeds_subtotal\n================== 1 failed, 13 passed in 0.42s =================================\n", + "query": "which test failed and what was the assertion where the discount exceeds the cart subtotal?", + "gold_answer": "AssertionError: computed discount 15.00 exceeds cart subtotal 12.50", + "critical_atoms": [ + "tests/test_pricing.py:47" + ], + "notes": "pytest run with 13 passing lines; the assertion-detail line carries the failure answer and uniquely shares 'discount/exceeds/subtotal' with the query, so bm25 keeps it at a tight budget while forced_only (path-only), recency, and llmlingua_style drop it. The file:line traceback path (audit-critical for fault localization) is on a separate line and is force-kept." +} diff --git a/eval/tasks/v04/code_build_error_010.json b/eval/tasks/v04/code_build_error_010.json new file mode 100644 index 0000000..f587025 --- /dev/null +++ b/eval/tasks/v04/code_build_error_010.json @@ -0,0 +1,13 @@ +{ + "id": "code_build_error_010", + "family": "code_build_error", + "tier": "A", + "source": "Starting local Bazel server and connecting to it...\nInvocation ID: 7f3a9c21-9e4d-4a1b-8f2e-0c5d7b3a9e11\nINFO: Analyzed target //services/payments:payments_lib (214 packages loaded, 3891 targets configured).\n[ 1/214] Compiling src/auth/handler.cc\n[ 2/214] Compiling src/billing/client.cc\n[ 3/214] Compiling src/catalog/server.cc\n[ 4/214] Compiling src/cart/store.cc\n[ 5/214] Compiling src/checkout/queue.cc\n[ 6/214] Compiling src/fraud/cache.cc\n[ 7/214] Compiling src/identity/codec.cc\n[ 8/214] Compiling src/inventory/router.cc\n[ 9/214] Compiling src/ledger/pool.cc\n[ 10/214] Compiling src/notify/worker.cc\n[ 11/214] Compiling src/orders/handler.cc\n[ 12/214] Compiling src/pricing/client.cc\n[ 13/214] Compiling src/promo/server.cc\n[ 14/214] Compiling src/refunds/store.cc\n[ 15/214] Compiling src/reports/queue.cc\n[ 16/214] Compiling src/routing/cache.cc\n[ 17/214] Compiling src/search/codec.cc\n[ 18/214] Compiling src/session/router.cc\n[ 19/214] Compiling src/shipping/pool.cc\n[ 20/214] Compiling src/tax/worker.cc\n[ 21/214] Compiling src/users/handler.cc\n[ 22/214] Compiling src/vault/client.cc\n[ 23/214] Compiling src/wallet/server.cc\n[ 24/214] Compiling src/webhook/store.cc\n[ 25/214] Compiling src/gateway/queue.cc\n[ 26/214] Compiling src/queue/cache.cc\n[ 27/214] Compiling src/cache/codec.cc\n[ 28/214] Compiling src/config/router.cc\n[ 29/214] Compiling src/onboarding/pool.cc\n[ 30/214] Compiling src/settlement/worker.cc\n[ 31/214] Compiling src/recs/handler.cc\n[ 32/214] Compiling src/loyalty/client.cc\n[ 33/214] Compiling src/returns/server.cc\n[ 34/214] Compiling src/addresses/store.cc\n[ 35/214] Compiling src/currency/queue.cc\n[ 36/214] Compiling src/exports/cache.cc\n[ 37/214] Compiling src/otp/codec.cc\nERROR: Linking services/payments/payments_lib failed (Exit 1)\n --> services/payments/BUILD.bazel:44:11\nld.lld: error: undefined reference to `crypto_verify_ed25519'\n>>> referenced by payments_signer.cc\n>>> services/payments/payments_signer.o:(PaymentsSigner::verify(std::string const&))\ncollect2: error: ld returned 1 exit status\nTarget //services/payments:payments_lib failed to build\nINFO: Elapsed time: 42.318s, Critical Path: 18.02s\nINFO: 213 processes: 187 local, 26 remote cache hit.\nFAILED: Build did NOT complete successfully\nINFO: Build completed, 1 target failed to build\n", + "query": "During the payments library build, what did the linker report as an undefined reference?", + "gold_answer": "undefined reference to `crypto_verify_ed25519'", + "critical_atoms": [ + "7f3a9c21-9e4d-4a1b-8f2e-0c5d7b3a9e11", + "services/payments/BUILD.bazel:44:11" + ], + "notes": "At a 10% budget, bm25 is the only deterministic selector that recovers the failure line, because the query shares 'undefined'/'reference' with the ld.lld text verbatim. recency keeps the trailing elapsed-time/process-count footer instead, and frequency and llmlingua_style spend the budget on the 37 boilerplate 'Compiling ...' lines, each carrying its own step number and source path that scores just as 'rare' as the real line. The invocation id and the BUILD.bazel file:line (audit-critical for fault localization) sit on separate lines and are force-kept regardless." +} diff --git a/eval/tasks/v04/code_build_error_011.json b/eval/tasks/v04/code_build_error_011.json new file mode 100644 index 0000000..47712ab --- /dev/null +++ b/eval/tasks/v04/code_build_error_011.json @@ -0,0 +1,13 @@ +{ + "id": "code_build_error_011", + "family": "code_build_error", + "tier": "A", + "source": "$ pnpm -w test --ci --maxWorkers=4\ncommit 4b9a2f1e7c3d5608a91f4e2b7c6d9a0f31e8b4c2\nCI run: gh-run-88213004\nDetermining test suites to run across 46 workspace packages...\nPASS packages/auth-service (842 ms)\nPASS packages/billing-core (1204 ms)\nPASS packages/catalog-api (611 ms)\nPASS packages/cart-service (933 ms)\nPASS packages/checkout-flow (1502 ms)\nPASS packages/fraud-rules (288 ms)\nPASS packages/identity-sso (1877 ms)\nPASS packages/inventory-sync (455 ms)\nPASS packages/ledger-core (690 ms)\nPASS packages/notify-hub (1023 ms)\nPASS packages/orders-api (372 ms)\nPASS packages/pricing-engine (1440 ms)\nPASS packages/promo-codes (219 ms)\nPASS packages/refunds-svc (987 ms)\nPASS packages/reporting (1611 ms)\nPASS packages/routing-gw (534 ms)\nPASS packages/search-index (802 ms)\nPASS packages/session-store (1299 ms)\nPASS packages/shipping-calc (456 ms)\nPASS packages/tax-rules (1108 ms)\nPASS packages/user-profile (733 ms)\nPASS packages/vault-secrets (622 ms)\nPASS packages/wallet-core (1567 ms)\nPASS packages/webhook-relay (899 ms)\nPASS packages/gateway-edge (305 ms)\nPASS packages/queue-consumer (1782 ms)\nPASS packages/cache-warmer (468 ms)\nPASS packages/config-loader (1039 ms)\nPASS packages/onboarding-flow (721 ms)\nPASS packages/settlement (856 ms)\nPASS packages/audit-trail (1955 ms)\nPASS packages/feature-flags (341 ms)\nPASS packages/media-pipeline (1204 ms)\nPASS packages/geo-lookup (668 ms)\nPASS packages/recs-engine (913 ms)\nPASS packages/loyalty-points (277 ms)\nPASS packages/returns-flow (1466 ms)\nPASS packages/address-book (588 ms)\nPASS packages/tax-forms (1091 ms)\nPASS packages/currency-fx (744 ms)\nPASS packages/export-jobs (1633 ms)\nPASS packages/sms-gateway (402 ms)\nPASS packages/email-templates (1177 ms)\nPASS packages/push-notify (826 ms)\nPASS packages/otp-verify (559 ms)\nPASS packages/invoice-gen (1350 ms)\nFAIL packages/image-resize\n ● Test suite failed to run\n Jest worker encountered 4 child process exceptions, exceeding retry limit of 3\n at ChildProcessWorker.initialize (jest-worker/build/workers/ChildProcessWorker.js:181:21)\nWorker 3: SIGABRT (core dumped)\nTest Suites: 1 failed, 45 passed, 46 total\nTests: 1211 passed, 1211 total\nSnapshots: 0 total\nTime: 38.7 s, estimated 40 s\nRan all test suites.\n", + "query": "What fault caused the image-resize test worker to die outright rather than report a normal assertion failure?", + "gold_answer": "SIGABRT (core dumped)", + "critical_atoms": [ + "4b9a2f1e7c3d5608a91f4e2b7c6d9a0f31e8b4c2", + "gh-run-88213004" + ], + "notes": "At a 10% budget every deterministic selector here misses the signal line: recency keeps the trailing summary counts, frequency and llmlingua_style spend the budget on the 46 PASS lines' unique per-package durations, and bm25 -- despite the query naming the package, which only appears in the FAIL/stack-trace lines, not the terse summary line carrying the actual answer -- has no reason to prefer that line over the other 'worker' mentions nearby. The commit sha and CI run id are audit-critical and sit on their own header lines, force-kept independent of the failure detail." +} diff --git a/eval/tasks/v04/code_build_error_012.json b/eval/tasks/v04/code_build_error_012.json new file mode 100644 index 0000000..a068897 --- /dev/null +++ b/eval/tasks/v04/code_build_error_012.json @@ -0,0 +1,13 @@ +{ + "id": "code_build_error_012", + "family": "code_build_error", + "tier": "A", + "source": "Terraform v1.7.5 on linux_amd64\nApply run ID: run-8f2c41ab9d3e4f56a1b2c3d4e5f6a7b8\nPlan: 33 to add, 0 to change, 0 to destroy.\ngoogle_storage_bucket.artifacts_01: Creation complete after 13s [id=projects/acme-prod/us-central1/100733]\ngoogle_sql_database_instance.replica_02: Creation complete after 20s [id=projects/acme-prod/us-central1/101466]\ngoogle_compute_subnetwork.svc_03: Creation complete after 27s [id=projects/acme-prod/us-central1/102199]\ngoogle_pubsub_topic.events_04: Creation complete after 34s [id=projects/acme-prod/us-central1/102932]\ngoogle_compute_firewall.allow_internal_05: Creation complete after 41s [id=projects/acme-prod/us-central1/103665]\ngoogle_compute_instance.worker_06: Creation complete after 48s [id=projects/acme-prod/us-central1/104398]\ngoogle_storage_bucket.artifacts_07: Creation complete after 55s [id=projects/acme-prod/us-central1/105131]\ngoogle_sql_database_instance.replica_08: Creation complete after 9s [id=projects/acme-prod/us-central1/105864]\ngoogle_compute_subnetwork.svc_09: Creation complete after 16s [id=projects/acme-prod/us-central1/106597]\ngoogle_pubsub_topic.events_10: Creation complete after 23s [id=projects/acme-prod/us-central1/107330]\ngoogle_compute_firewall.allow_internal_11: Creation complete after 30s [id=projects/acme-prod/us-central1/108063]\ngoogle_compute_instance.worker_12: Creation complete after 37s [id=projects/acme-prod/us-central1/108796]\ngoogle_storage_bucket.artifacts_13: Creation complete after 44s [id=projects/acme-prod/us-central1/109529]\ngoogle_sql_database_instance.replica_14: Creation complete after 51s [id=projects/acme-prod/us-central1/110262]\ngoogle_compute_subnetwork.svc_15: Creation complete after 58s [id=projects/acme-prod/us-central1/110995]\ngoogle_pubsub_topic.events_16: Creation complete after 12s [id=projects/acme-prod/us-central1/111728]\ngoogle_compute_firewall.allow_internal_17: Creation complete after 19s [id=projects/acme-prod/us-central1/112461]\ngoogle_compute_instance.worker_18: Creation complete after 26s [id=projects/acme-prod/us-central1/113194]\ngoogle_storage_bucket.artifacts_19: Creation complete after 33s [id=projects/acme-prod/us-central1/113927]\ngoogle_sql_database_instance.replica_20: Creation complete after 40s [id=projects/acme-prod/us-central1/114660]\ngoogle_compute_subnetwork.svc_21: Creation complete after 47s [id=projects/acme-prod/us-central1/115393]\ngoogle_pubsub_topic.events_22: Creation complete after 54s [id=projects/acme-prod/us-central1/116126]\ngoogle_compute_firewall.allow_internal_23: Creation complete after 8s [id=projects/acme-prod/us-central1/116859]\ngoogle_compute_instance.worker_24: Creation complete after 15s [id=projects/acme-prod/us-central1/117592]\ngoogle_storage_bucket.artifacts_25: Creation complete after 22s [id=projects/acme-prod/us-central1/118325]\ngoogle_sql_database_instance.replica_26: Creation complete after 29s [id=projects/acme-prod/us-central1/119058]\ngoogle_compute_subnetwork.svc_27: Creation complete after 36s [id=projects/acme-prod/us-central1/119791]\ngoogle_pubsub_topic.events_28: Creation complete after 43s [id=projects/acme-prod/us-central1/120524]\ngoogle_compute_firewall.allow_internal_29: Creation complete after 50s [id=projects/acme-prod/us-central1/121257]\ngoogle_compute_instance.worker_30: Creation complete after 57s [id=projects/acme-prod/us-central1/121990]\ngoogle_storage_bucket.artifacts_31: Creation complete after 11s [id=projects/acme-prod/us-central1/122723]\ngoogle_sql_database_instance.replica_32: Creation complete after 18s [id=projects/acme-prod/us-central1/123456]\ngoogle_compute_instance.worker_17: Still creating... [40s elapsed]\n╷\n│ Error: Error creating instance: googleapi: Error 403: Quota 'CPUS' exceeded. Limit: 24.0 in region us-central1.\n│\n│ with google_compute_instance.worker_17,\n│ on main.tf line 118, in resource \"google_compute_instance\" \"worker_17\":\n│ 118: resource \"google_compute_instance\" \"worker_17\" {\n╵\nApply failed after 32 resources created, 1 errored.\nTerraform will not modify infrastructure that is not in its state file.\n", + "query": "When Terraform tried to create worker_17, which cloud quota was exceeded and what was the stated limit?", + "gold_answer": "Quota 'CPUS' exceeded. Limit: 24.0 in region us-central1.", + "critical_atoms": [ + "run-8f2c41ab9d3e4f56a1b2c3d4e5f6a7b8", + "google_compute_instance.worker_17" + ], + "notes": "At a 10% budget, bm25 and frequency both recover the quota line -- bm25 because the query repeats 'quota'/'exceeded'/'limit' verbatim, frequency because the quota message's tokens are rarer document-wide than the many unique-but-mundane resource ids across the 32 completed-resource lines. recency (trailing 'Apply failed'/state-file footer) and llmlingua_style (drawn to the many short, individually high-surprisal id/duration tokens in the completed-resource lines) both miss it. The apply run id and the failing resource address (audit-critical for identifying which apply run and resource to retry) sit on separate lines and are force-kept." +} diff --git a/eval/tasks/v04/code_build_error_013.json b/eval/tasks/v04/code_build_error_013.json new file mode 100644 index 0000000..3607d9a --- /dev/null +++ b/eval/tasks/v04/code_build_error_013.json @@ -0,0 +1,13 @@ +{ + "id": "code_build_error_013", + "family": "code_build_error", + "tier": "A", + "source": "#0 building with \"default\" instance using docker driver\nBuild session: buildx-sess-7f3a2b9c1d8e\n#1 [internal] load build definition from Dockerfile\n#1 DONE 0.0s\n#2 [internal] load metadata for docker.io/library/golang:1.22-bookworm\n#2 DONE 0.4s\n#3 FROM docker.io/library/golang:1.22-bookworm\n base digest: sha256:2f4b1c9d8e7a6f5c4b3a2d1e0f9c8b7a\n#3 DONE 0.6s\n#4 [stage 4/23] load .dockerignore\n#4 DONE 1.7s\n#5 [stage 5/23] load metadata for docker.io/library/golang:1.22-bookworm\n#5 DONE 2.1s\n#6 [stage 6/23] apt-get update && apt-get install -y ca-certificates git\n#6 DONE 2.4s\n#7 [stage 7/23] mkdir -p /build\n#7 DONE 2.8s\n#8 [stage 8/23] copy go.mod go.sum ./\n#8 DONE 3.2s\n#9 [stage 9/23] go mod download\n#9 DONE 0.4s\n#10 [stage 10/23] copy internal/ ./internal/\n#10 DONE 0.8s\n#11 [stage 11/23] copy cmd/ ./cmd/\n#11 DONE 1.2s\n#12 [stage 12/23] go vet ./...\n#12 DONE 1.5s\n#13 [stage 13/23] go build -o /out/tokenfold-gw ./cmd/gw\n#13 DONE 1.9s\n#14 [stage 14/23] copy scripts/entrypoint.sh /entrypoint.sh\n#14 DONE 2.3s\n#15 [stage 15/23] chmod +x /entrypoint.sh\n#15 DONE 2.6s\n#16 [stage 16/23] adduser --disabled-password appuser\n#16 DONE 3.0s\n#17 [stage 17/23] copy --from=build /out/tokenfold-gw /usr/local/bin/\n#17 DONE 0.3s\n#18 [stage 18/23] set workdir /app\n#18 DONE 0.7s\n#19 [stage 19/23] run apk add --no-cache tzdata\n#19 DONE 1.0s\n#20 [stage 20/23] copy configs/prod.yaml /etc/tokenfold/config.yaml\n#20 DONE 1.4s\n#21 [stage 21/23] copy --from=build /out/tokenfold-migrate /usr/local/bin/\n#21 DONE 1.8s\n#22 [stage 22/23] healthcheck --interval=30s CMD tokenfold-gw healthz\n#22 DONE 2.1s\n#23 [stage 23/23] RUN curl -fsSL https://mirror.internal/pkgs/libfoo-2.3.1.tar.gz -o /tmp/libfoo.tar.gz && sha256sum -c libfoo.sha256\n#23 0.812 libfoo.tar.gz: FAILED\n#23 0.842 sha256sum: WARNING: 1 computed checksum did NOT match\n0.840 expected: 9f86d081884c7d659a2feaa0c55ad015\n0.842 got: 3a7bd3e2360a3d29eea436fcfb7e44c7\n#23 ERROR: process \"/bin/sh -c curl -fsSL https://mirror.internal/pkgs/libfoo-2.3.1.tar.gz -o /tmp/libfoo.tar.gz && sha256sum -c libfoo.sha256\" did not complete successfully: exit code: 1\n------\n > [stage 23/23] RUN curl -fsSL https://mirror.internal/pkgs/libfoo-2.3.1.tar.gz ...:\n------\nDockerfile:34\nERROR: failed to solve: process \"/bin/sh -c curl ... && sha256sum -c libfoo.sha256\" did not complete successfully: exit code: 1\nmake: *** [Makefile:12: image] Error 1\n", + "query": "Which step's downloaded artifact failed verification, and what value did the check actually compute instead of the expected one?", + "gold_answer": "got: 3a7bd3e2360a3d29eea436fcfb7e44c7", + "critical_atoms": [ + "buildx-sess-7f3a2b9c1d8e", + "sha256:2f4b1c9d8e7a6f5c4b3a2d1e0f9c8b7a" + ], + "notes": "At a 10% budget every deterministic selector here misses the mismatch line: it is reported as a terse 'got: ' line sharing no vocabulary with the query ('checksum'/'verification' never appear near the hash itself), so bm25 has nothing to match; frequency and llmlingua_style are drawn to the equally rare per-step durations across the 22 cached build steps; and recency keeps the trailing Dockerfile-line/make-error footer instead. The buildx session id and the pinned base-image digest (a different hash, audit-critical for provenance) sit on their own lines well before the failure and are force-kept." +} diff --git a/eval/tasks/v04/code_build_error_014.json b/eval/tasks/v04/code_build_error_014.json new file mode 100644 index 0000000..61398e3 --- /dev/null +++ b/eval/tasks/v04/code_build_error_014.json @@ -0,0 +1,13 @@ +{ + "id": "code_build_error_014", + "family": "code_build_error", + "tier": "A", + "source": " Compiling tokenfold-core v0.4.0-alpha (/home/ci/tokenfold/crates/tokenfold-core)\nCargo build session: cb-4e9a1f2d6c8b\n Compiling tokenfold-cli v0.4.0-alpha\n Compiling tokenfold-io v0.4.0-alpha\n Compiling tokenfold-selectors v0.4.0-alpha\n Compiling serde v1.0.198\n Compiling serde_json v1.0.116\n Compiling clap v4.5.4\n Compiling thiserror v1.0.58\n Compiling anyhow v1.0.82\n Compiling tokio v1.37.0\n Compiling rayon v1.10.0\n Compiling regex v1.10.4\n Compiling bytes v1.6.0\n Compiling indexmap v2.2.6\n Compiling smallvec v1.13.2\n Compiling ahash v0.8.11\n Compiling crossbeam-channel v0.5.12\n Compiling memchr v2.7.2\n Compiling itertools v0.12.1\n Compiling once_cell v1.19.0\n Compiling log v0.4.21\n Compiling env_logger v0.11.3\n Compiling proptest v1.4.0\n Compiling criterion v0.5.1\n Compiling tempfile v3.10.1\n Compiling walkdir v2.5.0\n Compiling unicode-segmentation v1.11.0\n Compiling bstr v1.9.1\n Compiling tokenfold-bench v0.4.0-alpha\n Compiling tokenfold-fixtures v0.4.0-alpha\n Compiling rand v0.8.5\n Compiling rand_chacha v0.3.1\n Compiling zerocopy v0.7.34\n Compiling half v2.4.1\n Compiling num-traits v0.2.19\n Compiling hashbrown v0.14.3\nerror[E0277]: the trait bound `SelectorScore: Ord` is not satisfied\n --> crates/tokenfold-selectors/src/rank.rs:212:18\n |\n212 | scores.sort_by_key(|s| s.value);\n | ^^^^^^^^^^^^^^^^ the trait `Ord` is not implemented for `SelectorScore`\n |\n = help: the trait `Ord` is implemented for `f64`\n = note: required for `Vec` to implement `sort_by_key`\nerror: could not compile `tokenfold-selectors` (lib) due to 1 previous error\nwarning: build failed, waiting for other jobs to finish...\nerror: could not compile `tokenfold-selectors`\nSome errors have detailed explanations: E0277.\nFor more information about an error, try `rustc --explain E0277`.\n", + "query": "Which trait bound was not satisfied when compiling tokenfold-selectors, and for what type?", + "gold_answer": "the trait bound `SelectorScore: Ord` is not satisfied", + "critical_atoms": [ + "cb-4e9a1f2d6c8b", + "crates/tokenfold-selectors/src/rank.rs:212:18" + ], + "notes": "At a 10% budget, bm25 recovers the error line because the query repeats 'trait bound' and 'not satisfied' verbatim from it; recency, frequency, and llmlingua_style all miss it -- frequency and llmlingua_style spend the budget on the 36 boilerplate 'Compiling vX.Y.Z' lines, each with its own equally rare version string, and recency keeps the trailing 'rustc --explain' footer instead. The build session id and the rank.rs file:line (audit-critical for fault localization) sit on separate lines and are force-kept regardless." +} diff --git a/eval/tasks/v04/code_build_error_015.json b/eval/tasks/v04/code_build_error_015.json new file mode 100644 index 0000000..a3b1f1d --- /dev/null +++ b/eval/tasks/v04/code_build_error_015.json @@ -0,0 +1,13 @@ +{ + "id": "code_build_error_015", + "family": "code_build_error", + "tier": "A", + "source": "Running e2e suite against cluster gke_acme-prod_us-central1_ci-e2e-7\nNamespace: e2e-run-9f21 (created)\nCI run: gh-run-55019213-attempt1\ndeployment/auth-gw pod auth-gw-141123fb Running (ready 1/1)\ndeployment/billing-worker pod billing-worker-149a24ce Running (ready 1/1)\ndeployment/catalog-sync pod catalog-sync-152325a1 Running (ready 1/1)\ndeployment/cart-api pod cart-api-15ac2674 Running (ready 1/1)\ndeployment/checkout-web pod checkout-web-16352747 Running (ready 1/1)\ndeployment/fraud-scorer pod fraud-scorer-16be281a Running (ready 1/1)\ndeployment/identity-sso pod identity-sso-174728ed Running (ready 1/1)\ndeployment/inventory-sync pod inventory-sync-17d029c0 Running (ready 1/1)\ndeployment/notify-hub pod notify-hub-18592a93 Running (ready 1/1)\ndeployment/orders-api pod orders-api-18e22b66 Running (ready 1/1)\ndeployment/pricing-engine pod pricing-engine-196b2c39 Running (ready 1/1)\ndeployment/promo-codes pod promo-codes-19f42d0c Running (ready 1/1)\ndeployment/refunds-svc pod refunds-svc-1a7d2ddf Running (ready 1/1)\ndeployment/reporting-job pod reporting-job-1b062eb2 Running (ready 1/1)\ndeployment/routing-gw pod routing-gw-1b8f2f85 Running (ready 1/1)\ndeployment/search-index pod search-index-1c183058 Running (ready 1/1)\ndeployment/session-store pod session-store-1ca1312b Running (ready 1/1)\ndeployment/shipping-calc pod shipping-calc-1d2a31fe Running (ready 1/1)\ndeployment/tax-rules pod tax-rules-1db332d1 Running (ready 1/1)\ndeployment/user-profile pod user-profile-1e3c33a4 Running (ready 1/1)\ndeployment/vault-secrets pod vault-secrets-1ec53477 Running (ready 1/1)\ndeployment/wallet-core pod wallet-core-1f4e354a Running (ready 1/1)\ndeployment/webhook-relay pod webhook-relay-1fd7361d Running (ready 1/1)\ndeployment/queue-consumer pod queue-consumer-206036f0 Running (ready 1/1)\ndeployment/cache-warmer pod cache-warmer-20e937c3 Running (ready 1/1)\ndeployment/config-loader pod config-loader-21723896 Running (ready 1/1)\ndeployment/onboarding-flow pod onboarding-flow-21fb3969 Running (ready 1/1)\ndeployment/audit-trail pod audit-trail-22843a3c Running (ready 1/1)\ndeployment/feature-flags pod feature-flags-230d3b0f Running (ready 1/1)\ndeployment/geo-lookup pod geo-lookup-23963be2 Running (ready 1/1)\ndeployment/ledger-writer pod ledger-writer-7c9b8f6d45-x2k9p Running (ready 0/1)\nPod UID: 3f9a2e7c-8b41-4d5a-9c22-7f1e6b8a4d90\nWarning Unhealthy pod/ledger-writer-7c9b8f6d45-x2k9p Readiness probe failed: HTTP probe failed with statuscode: 503\nWarning Unhealthy pod/ledger-writer-7c9b8f6d45-x2k9p Readiness probe failed: HTTP probe failed with statuscode: 503 (x14 over 3m)\nWarning BackOff pod/report-exporter-58f7d9c6b8-m4qwe Back-off restarting failed container (exit code 2)\nWarning FailedMount pod/audit-relay-6b8d5f9c47-p8vzq Unable to attach or mount volumes: secret \"audit-relay-tls\" not found\n--- container logs: ledger-writer ---\nledger-writer: FATAL config error: missing required env DATABASE_URL, refusing to start listener\ne2e suite aborted: namespace e2e-run-9f21 did not reach Ready within 300s\nCleaning up namespace e2e-run-9f21...\n", + "query": "Why did the readiness probe keep returning an error for the ledger-writer pod during the e2e run?", + "gold_answer": "FATAL config error: missing required env DATABASE_URL, refusing to start listener", + "critical_atoms": [ + "gh-run-55019213-attempt1", + "3f9a2e7c-8b41-4d5a-9c22-7f1e6b8a4d90" + ], + "notes": "At a 10% budget, llmlingua_style is the one deterministic selector that recovers the root-cause line: its vocabulary ('FATAL', 'DATABASE_URL', 'refusing', 'listener') is rarer document-wide than anything else in the log, even though it shares no words with the query. bm25 and frequency instead favor the repeated 'Readiness probe failed...503' event lines (which share 'readiness'/'probe' with the query but are only the symptom), and recency keeps the trailing namespace-cleanup lines. The CI run id and the pod UID (audit-critical for tracing the failing run/pod) sit on their own lines and are force-kept." +} diff --git a/eval/tasks/v04/code_patch_001.json b/eval/tasks/v04/code_patch_001.json new file mode 100644 index 0000000..a6ef6fc --- /dev/null +++ b/eval/tasks/v04/code_patch_001.json @@ -0,0 +1,12 @@ +{ + "id": "code_patch_001", + "family": "code_patch", + "tier": "A", + "source": "diff --git a/src/billing/refund.rs b/src/billing/refund.rs\nindex 4a1c9e2..b7f0d31 100644\n--- a/src/billing/refund.rs\n+++ b/src/billing/refund.rs\n@@ -12,10 +12,13 @@ impl RefundProcessor {\n pub fn process(&self, order: &Order, refund: Money) -> Result {\n let account = self.ledger.account_for(order.customer_id);\n let original_charge = order.total_charged();\n self.audit.record(\"refund_requested\", order.id);\n- let receipt = self.ledger.post_credit(account, refund)?;\n+ if refund > original_charge {\n+ return Err(RefundError::ExceedsOriginalCharge);\n+ }\n+ let receipt = self.ledger.post_credit(account, refund)?;\n self.notify.send_refund_email(order.customer_id, &receipt);\n Ok(receipt)\n }\n }\n", + "query": "what condition did the fix add comparing the refund to the original_charge?", + "gold_answer": "if refund > original_charge", + "critical_atoms": [ + "src/billing/refund.rs" + ], + "notes": "The added guard is the only line carrying both query tokens (refund AND original_charge), so bm25 ranks it top and keeps it, while recency (prefers trailing lines) and forced_only drop it; the audit-critical changed file path lives only on the diff/---/+++ headers and is force-kept." +} diff --git a/eval/tasks/v04/code_patch_010.json b/eval/tasks/v04/code_patch_010.json new file mode 100644 index 0000000..1cff8f5 --- /dev/null +++ b/eval/tasks/v04/code_patch_010.json @@ -0,0 +1,13 @@ +{ + "id": "code_patch_010", + "family": "code_patch", + "tier": "A", + "source": "diff --git a/src/auth/session_config.py b/src/auth/session_config.py\nindex 8e2f4a1..c930bde 100644\n--- a/src/auth/session_config.py\n+++ b/src/auth/session_config.py\n@@ -1,39 +1,40 @@ config for auth timeouts\n # audit: change reviewed under CHG-9931\n PASSWORD_RESET_TIMEOUT_SECONDS = 3600\n MFA_CHALLENGE_TIMEOUT_SECONDS = 300\n EMAIL_VERIFICATION_TIMEOUT_SECONDS = 86400\n OAUTH_STATE_TIMEOUT_SECONDS = 600\n REMEMBER_ME_TIMEOUT_SECONDS = 2592000\n CSRF_TOKEN_TIMEOUT_SECONDS = 7200\n DEVICE_TRUST_TIMEOUT_SECONDS = 604800\n API_KEY_ROTATION_TIMEOUT_SECONDS = 7776000\n LOGIN_LOCKOUT_TIMEOUT_SECONDS = 900\n RATE_LIMIT_WINDOW_TIMEOUT_SECONDS = 60\n REFRESH_TOKEN_TIMEOUT_SECONDS = 1209600\n ACCESS_TOKEN_TIMEOUT_SECONDS = 900\n WEBAUTHN_CHALLENGE_TIMEOUT_SECONDS = 120\n SSO_ASSERTION_TIMEOUT_SECONDS = 300\n INVITE_LINK_TIMEOUT_SECONDS = 259200\n MAGIC_LINK_TIMEOUT_SECONDS = 900\n SESSION_IDLE_TIMEOUT_SECONDS = 1800\n PASSKEY_REGISTRATION_TIMEOUT_SECONDS = 300\n TRUSTED_DEVICE_TIMEOUT_SECONDS = 2592000\n+BREAK_GLASS_TIMEOUT_SECONDS = 120\n ACCOUNT_RECOVERY_TIMEOUT_SECONDS = 3600\n IMPERSONATION_GRANT_TIMEOUT_SECONDS = 1800\n SUPPORT_ESCALATION_TIMEOUT_SECONDS = 300\n BACKUP_CODE_TIMEOUT_SECONDS = 600\n PHONE_VERIFICATION_TIMEOUT_SECONDS = 300\n EMAIL_CHANGE_CONFIRM_TIMEOUT_SECONDS = 3600\n ADMIN_SESSION_TIMEOUT_SECONDS = 900\n SERVICE_TOKEN_TIMEOUT_SECONDS = 3600\n WEBHOOK_SIGNATURE_TIMEOUT_SECONDS = 300\n DOWNLOAD_LINK_TIMEOUT_SECONDS = 600\n SHARE_LINK_TIMEOUT_SECONDS = 604800\n EXPORT_JOB_TIMEOUT_SECONDS = 3600\n ONE_TIME_PASSWORD_TIMEOUT_SECONDS = 300\n SOCIAL_LOGIN_STATE_TIMEOUT_SECONDS = 600\n TENANT_SWITCH_TIMEOUT_SECONDS = 1800\n BILLING_PORTAL_TIMEOUT_SECONDS = 900\n SIGNED_URL_TIMEOUT_SECONDS = 300\n TEMP_PASSWORD_TIMEOUT_SECONDS = 900\n GRACE_PERIOD_TIMEOUT_SECONDS = 86400\n", + "query": "What is the value of BREAK_GLASS_TIMEOUT_SECONDS added by this patch?", + "gold_answer": "BREAK_GLASS_TIMEOUT_SECONDS = 120", + "critical_atoms": [ + "8e2f4a1..c930bde", + "CHG-9931" + ], + "notes": "The added constant sits mid-file among 38 other *_TIMEOUT_SECONDS assignments that are structurally identical and each lexically unique (no repeated variable names). At a ~10% budget only bm25 keeps the answer line, since the query names the added identifier verbatim and it is the sole line carrying that exact token; recency (favors the 19 lines that follow it), frequency and llmlingua_style (near-tied per-unit scores across 38 equally-unique names, so the ascending-index tie-break favors earlier lines), and forced_only all drop it. The index hash and the audit-ticket comment each sit on their own single line, keeping the forced floor small relative to the budget." +} diff --git a/eval/tasks/v04/code_patch_011.json b/eval/tasks/v04/code_patch_011.json new file mode 100644 index 0000000..6772e4f --- /dev/null +++ b/eval/tasks/v04/code_patch_011.json @@ -0,0 +1,12 @@ +{ + "id": "code_patch_011", + "family": "code_patch", + "tier": "A", + "source": "diff --git a/internal/gateway/breaker_policy.go b/internal/gateway/breaker_policy.go\nindex 5c8a3f0..1e4d9b7 100644\n--- a/internal/gateway/breaker_policy.go\n+++ b/internal/gateway/breaker_policy.go\n@@ -1,40 +1,41 @@\n var retryPolicy = map[string]RetryConfig{\n \t\"checkout-api\": {MaxRetries: 3, BackoffMs: 200},\n \t\"inventory-sync\": {MaxRetries: 5, BackoffMs: 400},\n \t\"pricing-cache\": {MaxRetries: 2, BackoffMs: 150},\n \t\"cart-merge\": {MaxRetries: 4, BackoffMs: 300},\n \t\"shipping-quote\": {MaxRetries: 3, BackoffMs: 250},\n \t\"tax-calc\": {MaxRetries: 2, BackoffMs: 100},\n \t\"address-verify\": {MaxRetries: 3, BackoffMs: 180},\n \t\"payment-capture\": {MaxRetries: 6, BackoffMs: 500},\n \t\"payment-void\": {MaxRetries: 4, BackoffMs: 350},\n \t\"fraud-score\": {MaxRetries: 2, BackoffMs: 120},\n \t\"loyalty-award\": {MaxRetries: 3, BackoffMs: 220},\n \t\"coupon-apply\": {MaxRetries: 2, BackoffMs: 90},\n \t\"catalog-index\": {MaxRetries: 5, BackoffMs: 450},\n \t\"search-suggest\": {MaxRetries: 1, BackoffMs: 0},\n \t\"review-submit\": {MaxRetries: 3, BackoffMs: 210},\n \t\"wishlist-sync\": {MaxRetries: 4, BackoffMs: 330},\n \t\"notification-push\": {MaxRetries: 2, BackoffMs: 140},\n \t\"notification-email\": {MaxRetries: 3, BackoffMs: 260},\n \t\"notification-sms\": {MaxRetries: 2, BackoffMs: 110},\n+\t\"ledger-reconcile\": {MaxRetries: 0, BackoffMs: 0},\n \t\"order-create\": {MaxRetries: 5, BackoffMs: 420},\n \t\"order-cancel\": {MaxRetries: 3, BackoffMs: 230},\n \t\"order-fulfill\": {MaxRetries: 4, BackoffMs: 310},\n \t\"refund-issue\": {MaxRetries: 3, BackoffMs: 190},\n \t\"invoice-generate\": {MaxRetries: 2, BackoffMs: 130},\n \t\"warehouse-pick\": {MaxRetries: 1, BackoffMs: 0},\n \t\"warehouse-pack\": {MaxRetries: 3, BackoffMs: 240},\n \t\"carrier-label\": {MaxRetries: 4, BackoffMs: 340},\n \t\"carrier-track\": {MaxRetries: 2, BackoffMs: 160},\n \t\"returns-process\": {MaxRetries: 3, BackoffMs: 200},\n \t\"exchange-process\": {MaxRetries: 2, BackoffMs: 170},\n \t\"subscription-renew\": {MaxRetries: 5, BackoffMs: 410},\n \t\"subscription-cancel\": {MaxRetries: 3, BackoffMs: 250},\n \t\"vendor-sync\": {MaxRetries: 4, BackoffMs: 320},\n \t\"vendor-payout\": {MaxRetries: 6, BackoffMs: 480},\n \t\"analytics-ingest\": {MaxRetries: 1, BackoffMs: 0},\n \t\"session-heartbeat\": {MaxRetries: 2, BackoffMs: 100},\n \t\"cache-warm\": {MaxRetries: 3, BackoffMs: 210},\n \t\"feed-export\": {MaxRetries: 2, BackoffMs: 150},\n }\n", + "query": "Which service's entry was patched so a failed call is never retried and fails immediately instead of backing off?", + "gold_answer": "\"ledger-reconcile\": {MaxRetries: 0, BackoffMs: 0}", + "critical_atoms": [ + "5c8a3f0..1e4d9b7" + ], + "notes": "The added map entry is the only one with both MaxRetries and BackoffMs at zero (meaning 'do not retry'), but the query never names the service or repeats the field tokens, so bm25 has no lexical signal to rank on and, like every other selector here, drops it at both a ~10% and a 25% budget; a few unrelated entries also carry BackoffMs: 0 for latency reasons, diluting that value's apparent rarity for frequency/llmlingua_style, and recency favors the 19 entries that follow it in the map. No deterministic selector in this set recovers the answer at either ceiling, since finding it requires weighing the combination of two fields rather than matching a term or a position -- headroom this corpus leaves open among 38 structurally identical siblings." +} diff --git a/eval/tasks/v04/code_patch_012.json b/eval/tasks/v04/code_patch_012.json new file mode 100644 index 0000000..653f268 --- /dev/null +++ b/eval/tasks/v04/code_patch_012.json @@ -0,0 +1,13 @@ +{ + "id": "code_patch_012", + "family": "code_patch", + "tier": "A", + "source": "diff --git a/crates/ledger-core/src/config/limits.rs b/crates/ledger-core/src/config/limits.rs\nindex a44f1c8..d92b6ea 100644\n--- a/crates/ledger-core/src/config/limits.rs\n+++ b/crates/ledger-core/src/config/limits.rs\n@@ -1,43 +1,44 @@ impl Default for EngineLimits {\n // audit: ops review ticket OPS-7710\n fn default() -> Self {\n Self {\n max_open_orders: 500,\n max_pending_transfers: 200,\n max_batch_size: 1000,\n max_concurrent_jobs: 32,\n max_retry_attempts: 5,\n max_webhook_fanout: 50,\n max_ledger_depth: 128,\n max_audit_entries: 10000,\n max_snapshot_age_secs: 3600,\n max_lock_wait_ms: 250,\n max_query_rows: 5000,\n max_export_rows: 250000,\n max_import_rows: 100000,\n max_upload_bytes: 26214400,\n max_download_bytes: 104857600,\n max_session_tokens: 20,\n max_api_keys_per_org: 10,\n+ max_reconciliation_window_days: 7,\n max_report_schedules: 25,\n max_dashboard_widgets: 40,\n max_alert_rules: 100,\n max_custom_fields: 60,\n max_tag_length: 64,\n max_note_length: 2000,\n max_attachment_count: 20,\n max_attachment_bytes: 20971520,\n max_bulk_actions: 500,\n max_saved_filters: 30,\n max_scheduled_exports: 15,\n max_webhook_retries: 8,\n max_idempotency_keys: 100000,\n max_concurrent_sessions: 10,\n max_login_attempts: 5,\n max_password_history: 10,\n max_mfa_backup_codes: 8,\n max_delegate_accounts: 5,\n max_org_members: 2000,\n max_billing_seats: 500,\n }\n }\n }\n", + "query": "What default value did the patch set for max_reconciliation_window_days?", + "gold_answer": "max_reconciliation_window_days: 7", + "critical_atoms": [ + "a44f1c8..d92b6ea", + "OPS-7710" + ], + "notes": "The new field sits mid-struct among 37 other max_* defaults that are all uniquely named one-liners. At a ~10% budget only bm25 keeps the answer line, since the query repeats the added field's exact identifier and no other field shares that token; recency (favors the 20 fields that follow it), frequency and llmlingua_style (near-tied scores across 37 equally-unique names, so the ascending-index tie-break favors the 17 that precede it), and forced_only all drop it. The index hash and the ops-ticket comment each sit on their own single line, keeping the forced floor small relative to the budget." +} diff --git a/eval/tasks/v04/code_patch_013.json b/eval/tasks/v04/code_patch_013.json new file mode 100644 index 0000000..67392c2 --- /dev/null +++ b/eval/tasks/v04/code_patch_013.json @@ -0,0 +1,12 @@ +{ + "id": "code_patch_013", + "family": "code_patch", + "tier": "A", + "source": "diff --git a/deploy/manifests/analytics-fleet.yaml b/deploy/manifests/analytics-fleet.yaml\nindex 7b3e9a4..2f68c15 100644\n--- a/deploy/manifests/analytics-fleet.yaml\n+++ b/deploy/manifests/analytics-fleet.yaml\n@@ -1,36 +1,37 @@\n containers:\n - {name: event-collector, cpu: \"500m\", memory: \"1Gi\"}\n - {name: schema-validator, cpu: \"250m\", memory: \"512Mi\"}\n - {name: dedup-filter, cpu: \"500m\", memory: \"768Mi\"}\n - {name: geo-enricher, cpu: \"250m\", memory: \"512Mi\"}\n - {name: session-stitcher, cpu: \"500m\", memory: \"1Gi\"}\n - {name: funnel-builder, cpu: \"1000m\", memory: \"2Gi\"}\n - {name: cohort-aggregator, cpu: \"1000m\", memory: \"2Gi\"}\n - {name: retention-calc, cpu: \"500m\", memory: \"1Gi\"}\n - {name: ab-test-scorer, cpu: \"250m\", memory: \"512Mi\"}\n - {name: anomaly-detector, cpu: \"500m\", memory: \"1Gi\"}\n - {name: alert-dispatcher, cpu: \"250m\", memory: \"256Mi\"}\n - {name: export-writer, cpu: \"500m\", memory: \"768Mi\"}\n - {name: parquet-compactor, cpu: \"1000m\", memory: \"2Gi\"}\n - {name: index-builder, cpu: \"1000m\", memory: \"2Gi\"}\n - {name: query-cache-warmer, cpu: \"500m\", memory: \"1Gi\"}\n - {name: dashboard-renderer, cpu: \"250m\", memory: \"512Mi\"}\n - {name: report-scheduler, cpu: \"250m\", memory: \"256Mi\"}\n+ - {name: in-memory-dedup-cache, cpu: \"250m\", memory: \"64Mi\"}\n - {name: snapshot-archiver, cpu: \"500m\", memory: \"1Gi\"}\n - {name: backfill-runner, cpu: \"1000m\", memory: \"2Gi\"}\n - {name: replay-consumer, cpu: \"500m\", memory: \"1Gi\"}\n - {name: dead-letter-drainer, cpu: \"250m\", memory: \"512Mi\"}\n - {name: rate-limiter-sidecar, cpu: \"100m\", memory: \"128Mi\"}\n - {name: auth-sidecar, cpu: \"100m\", memory: \"128Mi\"}\n - {name: log-shipper, cpu: \"100m\", memory: \"64Mi\"}\n - {name: metrics-scraper, cpu: \"100m\", memory: \"128Mi\"}\n - {name: trace-collector, cpu: \"250m\", memory: \"256Mi\"}\n - {name: config-reloader, cpu: \"50m\", memory: \"64Mi\"}\n - {name: secret-rotator, cpu: \"50m\", memory: \"64Mi\"}\n - {name: healthcheck-proxy, cpu: \"50m\", memory: \"64Mi\"}\n - {name: tls-terminator, cpu: \"100m\", memory: \"128Mi\"}\n - {name: connection-pooler, cpu: \"250m\", memory: \"512Mi\"}\n - {name: cache-invalidator, cpu: \"250m\", memory: \"256Mi\"}\n - {name: partition-rebalancer, cpu: \"500m\", memory: \"1Gi\"}\n - {name: offset-committer, cpu: \"250m\", memory: \"512Mi\"}\n - {name: checkpoint-writer, cpu: \"500m\", memory: \"768Mi\"}\n", + "query": "Which newly added container's declared resource footprint looks too small for the kind of workload its name implies, risking crashes under load?", + "gold_answer": "name: in-memory-dedup-cache, cpu: \"250m\", memory: \"64Mi\"", + "critical_atoms": [ + "7b3e9a4..2f68c15" + ], + "notes": "The undersized entry is identifiable only by comparing its declared memory against what its own name implies (an in-memory cache given less headroom than several lightweight sidecars) -- the query shares no vocabulary with the line itself (no 'memory', 'limit', or container name), so bm25 has nothing to match on and, like every other selector here, drops it at both a ~10% and a 25% budget. Several genuinely small sidecars (log-shipper, secret-rotator, healthcheck-proxy) also carry low memory values, so raw-value rarity does not isolate it for frequency/llmlingua_style either, and recency favors the 18 entries that follow it. No deterministic selector in this set recovers the answer at either ceiling, since flagging it requires comparing a declared value against what the entry's own name implies rather than matching a term, a value, or a position." +} diff --git a/eval/tasks/v04/code_patch_014.json b/eval/tasks/v04/code_patch_014.json new file mode 100644 index 0000000..e7171e7 --- /dev/null +++ b/eval/tasks/v04/code_patch_014.json @@ -0,0 +1,13 @@ +{ + "id": "code_patch_014", + "family": "code_patch", + "tier": "A", + "source": "diff --git a/src/experiments/rollout_flags.js b/src/experiments/rollout_flags.js\nindex 9f2c4d1..6a7e0b3 100644\n--- a/src/experiments/rollout_flags.js\n+++ b/src/experiments/rollout_flags.js\n@@ -1,41 +1,42 @@\n // audit: experiment review ticket EXP-5561\n const ROLLOUT_PERCENT = {\n newCheckoutFlow: 50,\n darkModeDefault: 100,\n aiSearchSuggestions: 10,\n socialLoginGoogle: 100,\n socialLoginApple: 75,\n expressShippingBadge: 25,\n giftWrapOption: 40,\n oneClickReorder: 15,\n savedPaymentMethods: 100,\n biometricLogin: 30,\n pushNotificationsOptIn: 60,\n emailDigestWeekly: 80,\n smsAlertsEnabled: 20,\n inAppSurveyPrompt: 5,\n referralProgramBanner: 35,\n loyaltyTierBadge: 90,\n wishlistSharing: 45,\n productVideoPreview: 20,\n arTryOnPreview: 8,\n+ quantumCheckoutPreload: 5,\n priceDropAlerts: 55,\n backInStockAlerts: 65,\n bundleDiscountEngine: 30,\n dynamicPricingTest: 12,\n guestCheckoutSkip: 70,\n addressAutocomplete: 100,\n taxEstimateInline: 85,\n deliveryEtaBanner: 40,\n returnsSelfService: 25,\n chatSupportWidget: 50,\n orderTrackingMap: 60,\n subscriptionPause: 20,\n giftCardBalanceCheck: 90,\n storeCreditPrompt: 15,\n multiCurrencyDisplay: 10,\n localizedReviews: 35,\n accessibilityHighContrast: 100,\n reducedMotionMode: 100,\n cartAbandonmentEmail: 75,\n };\n", + "query": "What rollout percentage was set for quantumCheckoutPreload in this patch?", + "gold_answer": "quantumCheckoutPreload: 5", + "critical_atoms": [ + "9f2c4d1..6a7e0b3", + "EXP-5561" + ], + "notes": "The added flag sits mid-object among 38 other rollout-percentage entries that are all uniquely named one-liners. At a ~10% budget only bm25 keeps the answer line, since the query repeats the added flag's exact camelCase identifier and no other flag shares that token; recency (favors the 19 flags that follow it), frequency and llmlingua_style (near-tied scores across 38 equally-unique names, so the ascending-index tie-break favors the 19 that precede it), and forced_only all drop it. The index hash and the experiment-ticket comment each sit on their own single line, keeping the forced floor small relative to the budget." +} diff --git a/eval/tasks/v04/code_patch_015.json b/eval/tasks/v04/code_patch_015.json new file mode 100644 index 0000000..2ecfcd9 --- /dev/null +++ b/eval/tasks/v04/code_patch_015.json @@ -0,0 +1,12 @@ +{ + "id": "code_patch_015", + "family": "code_patch", + "tier": "A", + "source": "diff --git a/include/ledger_errors.h b/include/ledger_errors.h\nindex 3d9f7c2..0e5b4a8 100644\n--- a/include/ledger_errors.h\n+++ b/include/ledger_errors.h\n@@ -1,38 +1,39 @@\n #define ERR_INVALID_TOKEN 4101\n #define ERR_EXPIRED_SESSION 4102\n #define ERR_RATE_LIMITED 4103\n #define ERR_DUPLICATE_ENTRY 4104\n #define ERR_NOT_FOUND 4105\n #define ERR_PERMISSION_DENIED 4106\n #define ERR_INVALID_SIGNATURE 4107\n #define ERR_CLOCK_SKEW 4108\n #define ERR_MALFORMED_REQUEST 4109\n #define ERR_UNSUPPORTED_VERSION 4110\n #define ERR_PAYLOAD_TOO_LARGE 4111\n #define ERR_CONNECTION_RESET 4112\n #define ERR_TIMEOUT_UPSTREAM 4113\n #define ERR_CIRCUIT_OPEN 4114\n #define ERR_STALE_CACHE 4115\n #define ERR_LOCK_CONTENTION 4116\n #define ERR_QUEUE_FULL 4117\n #define ERR_SCHEMA_MISMATCH 4118\n #define ERR_CHECKSUM_MISMATCH 4119\n+#define ERR_QUOTA_EXCEEDED_STORAGE 4173\n #define ERR_PARTIAL_WRITE 4120\n #define ERR_READ_ONLY_MODE 4121\n #define ERR_MAINTENANCE_WINDOW 4122\n #define ERR_FEATURE_DISABLED 4123\n #define ERR_QUOTA_EXCEEDED_API 4124\n #define ERR_QUOTA_EXCEEDED_BANDWIDTH 4125\n #define ERR_ACCOUNT_SUSPENDED 4126\n #define ERR_ACCOUNT_LOCKED 4127\n #define ERR_BILLING_HOLD 4128\n #define ERR_REGION_BLOCKED 4129\n #define ERR_UNSUPPORTED_CURRENCY 4130\n #define ERR_INVALID_ADDRESS 4131\n #define ERR_DUPLICATE_SHIPMENT 4132\n #define ERR_CARRIER_UNAVAILABLE 4133\n #define ERR_LABEL_VOID_FAILED 4134\n #define ERR_RETURN_WINDOW_CLOSED 4135\n #define ERR_REFUND_ALREADY_ISSUED 4136\n #define ERR_CURRENCY_MISMATCH 4137\n #define ERR_EXPORT_IN_PROGRESS 4138\n", + "query": "Which new error code was introduced for the case where a user has no space left to upload additional files?", + "gold_answer": "ERR_QUOTA_EXCEEDED_STORAGE 4173", + "critical_atoms": [ + "3d9f7c2..0e5b4a8" + ], + "notes": "The added #define is paraphrased in the query ('no space left to upload files') without reusing any of its literal macro token ('err_quota_exceeded_storage'), so bm25 scores it no higher than the 37 other ERR_* lines and, like every other selector here, drops it at both a ~10% and a 25% budget. Two existing macros (ERR_QUOTA_EXCEEDED_API, ERR_QUOTA_EXCEEDED_BANDWIDTH) are near-identical decoys sharing its prefix, and recency favors the 19 macros that follow it in the header. No deterministic selector in this set recovers the answer at either ceiling, since matching the paraphrased failure condition to the right macro requires more than term overlap or position." +} diff --git a/eval/tasks/v04/diff_review_010.json b/eval/tasks/v04/diff_review_010.json new file mode 100644 index 0000000..9e58c83 --- /dev/null +++ b/eval/tasks/v04/diff_review_010.json @@ -0,0 +1,13 @@ +{ + "id": "diff_review_010", + "family": "diff_review", + "tier": "A", + "source": "diff --git a/config/services/replica-overrides.properties b/config/services/replica-overrides.properties\nindex 8f3a1c2d9e0b..4b7e9d0d1a2c 100644\n--- a/config/services/replica-overrides.properties\n+++ b/config/services/replica-overrides.properties\n@@ -1,32 +1,32 @@\n # replica overrides applied per quarterly capacity review\n # change-ticket: CHG-88214\n-auth-service.replicas=4\n+auth-service.replicas=5\n-billing-service.replicas=6\n+billing-service.replicas=7\n-catalog-service.replicas=5\n+catalog-service.replicas=4\n-cart-service.replicas=3\n+cart-service.replicas=4\n-checkout-service.replicas=6\n+checkout-service.replicas=9\n-inventory-service.replicas=4\n+inventory-service.replicas=5\n-notification-service.replicas=2\n+notification-service.replicas=3\n-payments-service.replicas=5\n+payments-service.replicas=6\n-pricing-service.replicas=3\n+pricing-service.replicas=2\n-recommendation-service.replicas=4\n+recommendation-service.replicas=5\n-search-service.replicas=6\n+search-service.replicas=5\n-session-service.replicas=3\n+session-service.replicas=4\n-shipping-service.replicas=4\n+shipping-service.replicas=5\n-tax-service.replicas=2\n+tax-service.replicas=3\n-user-service.replicas=5\n+user-service.replicas=6\n", + "query": "What did the diff change checkout-service's replica count to?", + "gold_answer": "checkout-service.replicas=9", + "critical_atoms": [ + "4b7e9d0d1a2c", + "CHG-88214" + ], + "notes": "35-line properties diff where every one of 15 services gets a near-identical replicas=N change; only checkout-service's new value answers the query, and it shares the query's vocabulary directly (lexical case). At a ~10% budget, recency (favors the last-listed service) and raw frequency (every line has the same token shape 'X.replicas=N') rank the 15 change lines almost identically, so a selector must actually weigh the query term rather than position or repetition. The hash and ticket id each appear on a single header/comment line, separate from the answer line, keeping the forced floor small relative to the haystack." +} diff --git a/eval/tasks/v04/diff_review_011.json b/eval/tasks/v04/diff_review_011.json new file mode 100644 index 0000000..2153870 --- /dev/null +++ b/eval/tasks/v04/diff_review_011.json @@ -0,0 +1,13 @@ +{ + "id": "diff_review_011", + "family": "diff_review", + "tier": "A", + "source": "diff --git a/infra/nginx/upstream-pool-checkout.conf b/infra/nginx/upstream-pool-checkout.conf\nindex a91f4e2b7c31..d02c9a4f1e88 100644\n--- a/infra/nginx/upstream-pool-checkout.conf\n+++ b/infra/nginx/upstream-pool-checkout.conf\n@@ -2,30 +2,30 @@ upstream checkout_pool {\n # incident-ref: INC-51309\n- server 10.4.2.10:8443 weight=6 max_fails=3;\n+ server 10.4.2.10:8443 weight=5 max_fails=3;\n- server 10.4.2.11:8443 weight=6 max_fails=3;\n+ server 10.4.2.11:8443 weight=4 max_fails=3;\n- server 10.4.2.12:8443 weight=7 max_fails=3;\n+ server 10.4.2.12:8443 weight=6 max_fails=3;\n- server 10.4.2.13:8443 weight=5 max_fails=3;\n+ server 10.4.2.13:8443 weight=3 max_fails=3;\n- server 10.4.2.14:8443 weight=6 max_fails=3;\n+ server 10.4.2.14:8443 weight=4 max_fails=3;\n- server 10.4.2.15:8443 weight=7 max_fails=3;\n+ server 10.4.2.15:8443 weight=5 max_fails=3;\n- server 10.4.2.16:8443 weight=5 max_fails=3;\n+ server 10.4.2.16:8443 weight=3 max_fails=3;\n- server 10.4.2.17:8443 weight=6 max_fails=3;\n+ server 10.4.2.17:8443 weight=6 max_fails=3;\n- server 10.4.2.18:8443 weight=7 max_fails=3;\n+ server 10.4.2.18:8443 weight=0 max_fails=3;\n- server 10.4.2.19:8443 weight=6 max_fails=3;\n+ server 10.4.2.19:8443 weight=4 max_fails=3;\n- server 10.4.2.20:8443 weight=7 max_fails=3;\n+ server 10.4.2.20:8443 weight=5 max_fails=3;\n- server 10.4.2.21:8443 weight=5 max_fails=3;\n+ server 10.4.2.21:8443 weight=3 max_fails=3;\n- server 10.4.2.22:8443 weight=6 max_fails=3;\n+ server 10.4.2.22:8443 weight=6 max_fails=3;\n- server 10.4.2.23:8443 weight=6 max_fails=3;\n+ server 10.4.2.23:8443 weight=4 max_fails=3;\n- server 10.4.2.24:8443 weight=7 max_fails=3;\n+ server 10.4.2.24:8443 weight=5 max_fails=3;\n", + "query": "Which backend server in the pool was drained (set to zero weight) during this rebalance?", + "gold_answer": "10.4.2.18:8443 weight=0", + "critical_atoms": [ + "d02c9a4f1e88", + "INC-51309" + ], + "notes": "36-line nginx upstream-pool diff: 15 backend servers each get a small weight change, formatted identically ('server IP:PORT weight=N max_fails=3;'). Only 10.4.2.18 is drained to weight=0, but the query never names the IP or writes 'weight=0' -- it says 'drained', a semantic paraphrase (structural/lexically distant case). Every line shares the tokens 'server', 'weight', and 'max_fails', so term-overlap and frequency selectors see 15 equally-scored candidates; only picking the one structurally distinct value (0 vs everyone else's 3-6) answers the query. The index hash and incident ref sit on their own single lines, apart from the answer line." +} diff --git a/eval/tasks/v04/diff_review_012.json b/eval/tasks/v04/diff_review_012.json new file mode 100644 index 0000000..a37d0a7 --- /dev/null +++ b/eval/tasks/v04/diff_review_012.json @@ -0,0 +1,13 @@ +{ + "id": "diff_review_012", + "family": "diff_review", + "tier": "A", + "source": "diff --git a/migrations/2026_07_expand_user_fields.sql b/migrations/2026_07_expand_user_fields.sql\nindex 3c7e91a0f452..9d1b6e2a7c04 100644\n--- a/migrations/2026_07_expand_user_fields.sql\n+++ b/migrations/2026_07_expand_user_fields.sql\n@@ -1,34 +1,34 @@\n -- migration-id: MIG-20260714-users\n+ALTER TABLE accounts ADD COLUMN referral_code varchar(32);\n+ALTER TABLE accounts ADD COLUMN locale varchar(8);\n+ALTER TABLE addresses ADD COLUMN unit_number varchar(16);\n+ALTER TABLE addresses ADD COLUMN delivery_notes text;\n+ALTER TABLE carts ADD COLUMN abandoned_at timestamp;\n+ALTER TABLE carts ADD COLUMN currency varchar(8);\n+ALTER TABLE catalog_items ADD COLUMN brand_code varchar(16);\n+ALTER TABLE catalog_items ADD COLUMN is_discontinued boolean;\n+ALTER TABLE invoices ADD COLUMN tax_id varchar(24);\n+ALTER TABLE invoices ADD COLUMN po_number varchar(24);\n+ALTER TABLE orders ADD COLUMN gift_wrap boolean;\n+ALTER TABLE orders ADD COLUMN priority_flag boolean;\n+ALTER TABLE payments ADD COLUMN processor_ref varchar(40);\n+ALTER TABLE payments ADD COLUMN retry_count int;\n+ALTER TABLE users ALTER COLUMN email SET NOT NULL;\n+ALTER TABLE users ADD COLUMN marketing_opt_in boolean;\n+ALTER TABLE pricing_rules ADD COLUMN region_code varchar(8);\n+ALTER TABLE pricing_rules ADD COLUMN rounding_mode varchar(16);\n+ALTER TABLE sessions ADD COLUMN device_type varchar(16);\n+ALTER TABLE sessions ADD COLUMN ip_hint varchar(45);\n+ALTER TABLE shipments ADD COLUMN carrier varchar(24);\n+ALTER TABLE shipments ADD COLUMN tracking_url text;\n+ALTER TABLE subscriptions ADD COLUMN renewal_date date;\n+ALTER TABLE subscriptions ADD COLUMN auto_renew boolean;\n+ALTER TABLE warehouses ADD COLUMN capacity_units int;\n+ALTER TABLE warehouses ADD COLUMN region_code varchar(8);\n+ALTER TABLE wishlists ADD COLUMN shared boolean;\n+ALTER TABLE wishlists ADD COLUMN visibility varchar(16);\n+ALTER TABLE reviews ADD COLUMN helpful_count int;\n+ALTER TABLE reviews ADD COLUMN flagged boolean;\n", + "query": "Which migration statement makes the users.email column NOT NULL?", + "gold_answer": "ALTER TABLE users ALTER COLUMN email SET NOT NULL;", + "critical_atoms": [ + "9d1b6e2a7c04", + "MIG-20260714-users" + ], + "notes": "35-line SQL migration diff adding two columns to each of 14 tables (28 near-identical 'ALTER TABLE X ADD COLUMN ...;' lines); one line in the middle instead tightens a constraint on users.email, and the query names that table and column directly (lexical case). Every line starts with 'ALTER TABLE', so raw term frequency and position heuristics can't separate the 29 additions from the one constraint change at a tight budget; the migration id and new blob hash each live on a single line separate from the answer line." +} diff --git a/eval/tasks/v04/diff_review_013.json b/eval/tasks/v04/diff_review_013.json new file mode 100644 index 0000000..9fb8cd7 --- /dev/null +++ b/eval/tasks/v04/diff_review_013.json @@ -0,0 +1,13 @@ +{ + "id": "diff_review_013", + "family": "diff_review", + "tier": "A", + "source": "diff --git a/ops/cron/nightly-jobs.crontab b/ops/cron/nightly-jobs.crontab\nindex 5e0a83c1f9d4..1a6f4b9e3c72 100644\n--- a/ops/cron/nightly-jobs.crontab\n+++ b/ops/cron/nightly-jobs.crontab\n@@ -1,32 +1,32 @@\n # schedule-owner: platform-ops\n # audit-id: AUD-70742\n-0 2 * * * /opt/jobs/run-backup-accounts.sh\n+0 3 * * * /opt/jobs/run-backup-accounts.sh\n-0 2 * * * /opt/jobs/run-backup-billing.sh\n+0 1 * * * /opt/jobs/run-backup-billing.sh\n-0 2 * * * /opt/jobs/run-backup-carts.sh\n+0 4 * * * /opt/jobs/run-backup-carts.sh\n-0 2 * * * /opt/jobs/run-backup-catalog.sh\n+0 3 * * * /opt/jobs/run-backup-catalog.sh\n-0 2 * * * /opt/jobs/run-backup-inventory.sh\n+0 1 * * * /opt/jobs/run-backup-inventory.sh\n-0 2 * * * /opt/jobs/run-backup-invoices.sh\n+0 3 * * * /opt/jobs/run-backup-invoices.sh\n-0 2 * * * /opt/jobs/run-backup-metrics.sh\n+0 * * * * /opt/jobs/run-backup-metrics.sh\n-0 2 * * * /opt/jobs/run-backup-notifications.sh\n+0 4 * * * /opt/jobs/run-backup-notifications.sh\n-0 2 * * * /opt/jobs/run-backup-orders.sh\n+0 1 * * * /opt/jobs/run-backup-orders.sh\n-0 2 * * * /opt/jobs/run-backup-payments.sh\n+0 3 * * * /opt/jobs/run-backup-payments.sh\n-0 2 * * * /opt/jobs/run-backup-pricing.sh\n+0 4 * * * /opt/jobs/run-backup-pricing.sh\n-0 2 * * * /opt/jobs/run-backup-reviews.sh\n+0 1 * * * /opt/jobs/run-backup-reviews.sh\n-0 2 * * * /opt/jobs/run-backup-search.sh\n+0 3 * * * /opt/jobs/run-backup-search.sh\n-0 2 * * * /opt/jobs/run-backup-sessions.sh\n+0 4 * * * /opt/jobs/run-backup-sessions.sh\n-0 2 * * * /opt/jobs/run-backup-shipping.sh\n+0 1 * * * /opt/jobs/run-backup-shipping.sh\n", + "query": "Which nightly job was changed to run every hour instead of once a day?", + "gold_answer": "0 * * * * /opt/jobs/run-backup-metrics.sh", + "critical_atoms": [ + "1a6f4b9e3c72", + "AUD-70742" + ], + "notes": "36-line crontab diff: 15 jobs each shift their nightly hour by one (formatted identically as '0 N * * * /opt/jobs/run-backup-X.sh'); only the metrics job's schedule changes shape entirely (to '0 * * * *'), which the query paraphrases as 'every hour' without naming the job or the cron syntax (semantic/structural case). Every line shares the tokens 'run-backup' and 'jobs', so term-overlap and frequency selectors can't discriminate; only recognizing the one structurally different schedule works at a tight budget. Audit id and index hash each sit on one comment line, apart from the answer line." +} diff --git a/eval/tasks/v04/diff_review_014.json b/eval/tasks/v04/diff_review_014.json new file mode 100644 index 0000000..91f2884 --- /dev/null +++ b/eval/tasks/v04/diff_review_014.json @@ -0,0 +1,13 @@ +{ + "id": "diff_review_014", + "family": "diff_review", + "tier": "A", + "source": "diff --git a/config/flags/rollout-percentages.conf b/config/flags/rollout-percentages.conf\nindex c2a9f047e8b1..7e4d0a9c2f56 100644\n--- a/config/flags/rollout-percentages.conf\n+++ b/config/flags/rollout-percentages.conf\n@@ -1,34 +1,34 @@\n # rollout-owner: growth-eng\n # tracking-id: EXP-33920\n-flag.checkout_v2_ui.rollout=45\n+flag.checkout_v2_ui.rollout=55\n-flag.dark_mode.rollout=20\n+flag.dark_mode.rollout=25\n-flag.one_click_reorder.rollout=10\n+flag.one_click_reorder.rollout=15\n-flag.saved_payment_methods.rollout=30\n+flag.saved_payment_methods.rollout=35\n-flag.guest_checkout.rollout=60\n+flag.guest_checkout.rollout=65\n-flag.express_shipping_badge.rollout=15\n+flag.express_shipping_badge.rollout=20\n-flag.wishlist_sharing.rollout=25\n+flag.wishlist_sharing.rollout=30\n-flag.price_drop_alerts.rollout=40\n+flag.price_drop_alerts.rollout=45\n-flag.new_search_ranking.rollout=90\n+flag.new_search_ranking.rollout=100\n-flag.recommendation_carousel.rollout=35\n+flag.recommendation_carousel.rollout=40\n-flag.loyalty_points_v2.rollout=5\n+flag.loyalty_points_v2.rollout=10\n-flag.split_shipment_ui.rollout=50\n+flag.split_shipment_ui.rollout=55\n-flag.address_autocomplete.rollout=70\n+flag.address_autocomplete.rollout=75\n-flag.review_photos.rollout=15\n+flag.review_photos.rollout=20\n-flag.subscription_pause.rollout=20\n+flag.subscription_pause.rollout=25\n", + "query": "Which feature flag reached 100% rollout in this change?", + "gold_answer": "new_search_ranking.rollout=100", + "critical_atoms": [ + "7e4d0a9c2f56", + "EXP-33920" + ], + "notes": "36-line feature-flag diff: 15 flags each get a plausible +5 rollout bump (identically formatted 'flag.X.rollout=N'); only new_search_ranking's new value is 100, and the query's '100%' and 'rollout' terms match that line directly (lexical case). Recency (favors later flags) and raw frequency (every line has the same shape) rank the 15 change lines near-identically at a ~10% budget, so a selector must actually notice the query-relevant value rather than position or repetition. Tracking id and blob hash each live on one comment/header line, separate from the answer line." +} diff --git a/eval/tasks/v04/diff_review_015.json b/eval/tasks/v04/diff_review_015.json new file mode 100644 index 0000000..8c1854e --- /dev/null +++ b/eval/tasks/v04/diff_review_015.json @@ -0,0 +1,13 @@ +{ + "id": "diff_review_015", + "family": "diff_review", + "tier": "A", + "source": "diff --git a/.ci/pipelines/build-matrix.yml b/.ci/pipelines/build-matrix.yml\nindex 6b1d84f2a0c7..2f9e5a1d8b34 100644\n--- a/.ci/pipelines/build-matrix.yml\n+++ b/.ci/pipelines/build-matrix.yml\n@@ -3,32 +3,32 @@ jobs:\n # pipeline-owner: build-infra\n # audit-id: BLD-60417\n- lint: { timeout_minutes: 6 }\n+ lint: { timeout_minutes: 7 }\n- unit_tests: { timeout_minutes: 9 }\n+ unit_tests: { timeout_minutes: 8 }\n- integration_tests: { timeout_minutes: 12 }\n+ integration_tests: { timeout_minutes: 11 }\n- build_frontend: { timeout_minutes: 7 }\n+ build_frontend: { timeout_minutes: 8 }\n- build_backend: { timeout_minutes: 8 }\n+ build_backend: { timeout_minutes: 19 }\n- e2e_tests: { timeout_minutes: 14 }\n+ e2e_tests: { timeout_minutes: 13 }\n- docker_build: { timeout_minutes: 10 }\n+ docker_build: { timeout_minutes: 11 }\n- security_scan: { timeout_minutes: 9 }\n+ security_scan: { timeout_minutes: 8 }\n- dependency_audit: { timeout_minutes: 6 }\n+ dependency_audit: { timeout_minutes: 7 }\n- contract_tests: { timeout_minutes: 8 }\n+ contract_tests: { timeout_minutes: 9 }\n- perf_smoke: { timeout_minutes: 11 }\n+ perf_smoke: { timeout_minutes: 10 }\n- deploy_staging: { timeout_minutes: 5 }\n+ deploy_staging: { timeout_minutes: 6 }\n- migration_check: { timeout_minutes: 9 }\n+ migration_check: { timeout_minutes: 8 }\n- asset_upload: { timeout_minutes: 4 }\n+ asset_upload: { timeout_minutes: 5 }\n- notify_slack: { timeout_minutes: 3 }\n+ notify_slack: { timeout_minutes: 4 }\n", + "query": "Which CI job's timeout increased the most in this change?", + "gold_answer": "build_backend: { timeout_minutes: 19 }", + "critical_atoms": [ + "2f9e5a1d8b34", + "BLD-60417" + ], + "notes": "36-line CI pipeline diff: 15 jobs each get a plausible +/-1 minute timeout adjustment (identically formatted ' X: { timeout_minutes: N }'); only build_backend jumps by 11 minutes, and the query never names that job -- it asks for the largest increase, which requires comparing magnitudes across lines rather than matching query vocabulary (semantic/structural case). Every line shares the token 'timeout_minutes', so term-overlap selectors see 15 equally 'relevant' lines; only the one numeric outlier answers the query. Audit id and index hash each sit on one comment line, apart from the answer line." +} diff --git a/eval/tasks/v04/json_schema_010.json b/eval/tasks/v04/json_schema_010.json new file mode 100644 index 0000000..3db8d86 --- /dev/null +++ b/eval/tasks/v04/json_schema_010.json @@ -0,0 +1,13 @@ +{ + "id": "json_schema_010", + "family": "json_schema", + "tier": "A", + "source": "{\n \"registry\": \"service-catalog-snapshot\",\n \"generated_at\": \"2026-07-20T03:14:00Z\",\n \"snapshot_id\": \"reg-snap-88213f\",\n \"region\": \"us-east-1\",\n \"entries\": [\n {\"service\": \"auth-gateway\", \"port\": 8443, \"tier\": \"critical\", \"owner\": \"team-iam\"},\n {\"service\": \"session-cache\", \"port\": 6390, \"tier\": \"critical\", \"owner\": \"team-iam\"},\n {\"service\": \"user-profile-api\", \"port\": 9021, \"tier\": \"standard\", \"owner\": \"team-identity\"},\n {\"service\": \"notification-relay\", \"port\": 9102, \"tier\": \"standard\", \"owner\": \"team-comms\"},\n {\"service\": \"billing-worker\", \"port\": 7443, \"tier\": \"standard\", \"owner\": \"team-finance\"},\n {\"service\": \"invoice-renderer\", \"port\": 7521, \"tier\": \"standard\", \"owner\": \"team-finance\"},\n {\"service\": \"tax-calculator\", \"port\": 7602, \"tier\": \"standard\", \"owner\": \"team-finance\"},\n {\"service\": \"ledger-sync\", \"port\": 7710, \"tier\": \"standard\", \"owner\": \"team-finance\"},\n {\"service\": \"search-indexer\", \"port\": 9310, \"tier\": \"standard\", \"owner\": \"team-search\"},\n {\"service\": \"search-query-api\", \"port\": 9311, \"tier\": \"standard\", \"owner\": \"team-search\"},\n {\"service\": \"recommendation-engine\", \"port\": 9450, \"tier\": \"standard\", \"owner\": \"team-ml\"},\n {\"service\": \"feature-store\", \"port\": 9451, \"tier\": \"standard\", \"owner\": \"team-ml\"},\n {\"service\": \"image-resizer\", \"port\": 8810, \"tier\": \"standard\", \"owner\": \"team-media\"},\n {\"service\": \"video-transcoder\", \"port\": 8811, \"tier\": \"standard\", \"owner\": \"team-media\"},\n {\"service\": \"cdn-purger\", \"port\": 8812, \"tier\": \"standard\", \"owner\": \"team-media\"},\n {\"service\": \"email-sender\", \"port\": 9520, \"tier\": \"standard\", \"owner\": \"team-comms\"},\n {\"service\": \"sms-sender\", \"port\": 9521, \"tier\": \"standard\", \"owner\": \"team-comms\"},\n {\"service\": \"push-notifier\", \"port\": 9522, \"tier\": \"standard\", \"owner\": \"team-comms\"},\n {\"service\": \"audit-logger\", \"port\": 9601, \"tier\": \"critical\", \"owner\": \"team-security\"},\n {\"service\": \"secrets-broker\", \"port\": 9602, \"tier\": \"critical\", \"owner\": \"team-security\"},\n {\"service\": \"rate-limiter\", \"port\": 9603, \"tier\": \"critical\", \"owner\": \"team-security\"},\n {\"service\": \"webhook-dispatcher\", \"port\": 9710, \"tier\": \"standard\", \"owner\": \"team-platform\"},\n {\"service\": \"job-scheduler\", \"port\": 9711, \"tier\": \"standard\", \"owner\": \"team-platform\"},\n {\"service\": \"config-server\", \"port\": 9712, \"tier\": \"standard\", \"owner\": \"team-platform\"},\n {\"service\": \"health-aggregator\", \"port\": 9713, \"tier\": \"standard\", \"owner\": \"team-platform\"},\n {\"service\": \"metrics-forwarder\", \"port\": 9714, \"tier\": \"standard\", \"owner\": \"team-platform\"}\n ],\n \"checksum\": \"sha256:5c9e41af3b7d2c88\",\n \"entry_count\": 26\n}\n", + "query": "what port does the billing-worker service listen on?", + "gold_answer": "\"billing-worker\", \"port\": 7443", + "critical_atoms": [ + "reg-snap-88213f", + "sha256:5c9e41af3b7d2c88" + ], + "notes": "26-entry service registry (36 lines); every entry shares the identical {\"service\", \"port\", \"tier\", \"owner\"} shape so recency, frequency, and llmlingua_style (all query-independent, and frequency/llmlingua see near-uniform per-line token rarity since every entry has one unique service name) cannot distinguish the billing-worker line from the other 25 look-alike entries at a ~10% budget. bm25 has a fighting chance since 'billing-worker' and 'port' echo the query, but the ~3-4 line ceiling still forces it to drop most of the list. snapshot_id and checksum sit on separate header/footer lines and are force-kept regardless of selector." +} diff --git a/eval/tasks/v04/json_schema_011.json b/eval/tasks/v04/json_schema_011.json new file mode 100644 index 0000000..deb1d7c --- /dev/null +++ b/eval/tasks/v04/json_schema_011.json @@ -0,0 +1,13 @@ +{ + "id": "json_schema_011", + "family": "json_schema", + "tier": "A", + "source": "{\n \"policy_set\": \"resiliency-defaults-v9\",\n \"policy_id\": \"res-pol-6f1c0a3d\",\n \"owner_team\": \"platform-reliability\",\n \"reviewed_on\": \"2026-06-30\",\n \"circuit_breakers\": [\n {\"target\": \"checkout-api\", \"window_ms\": 5000, \"error_rate_pct\": 50, \"requeue_ceiling\": 6},\n {\"target\": \"cart-service\", \"window_ms\": 4200, \"error_rate_pct\": 55, \"requeue_ceiling\": 5},\n {\"target\": \"pricing-engine\", \"window_ms\": 4500, \"error_rate_pct\": 45, \"requeue_ceiling\": 5},\n {\"target\": \"promo-engine\", \"window_ms\": 4800, \"error_rate_pct\": 47, \"requeue_ceiling\": 6},\n {\"target\": \"shipping-calculator\", \"window_ms\": 5100, \"error_rate_pct\": 44, \"requeue_ceiling\": 5},\n {\"target\": \"tax-service\", \"window_ms\": 4700, \"error_rate_pct\": 46, \"requeue_ceiling\": 6},\n {\"target\": \"fraud-scorer\", \"window_ms\": 3900, \"error_rate_pct\": 60, \"requeue_ceiling\": 4},\n {\"target\": \"payment-gateway\", \"window_ms\": 5300, \"error_rate_pct\": 38, \"requeue_ceiling\": 5},\n {\"target\": \"billing-worker\", \"window_ms\": 6000, \"error_rate_pct\": 40, \"requeue_ceiling\": 3},\n {\"target\": \"invoice-renderer\", \"window_ms\": 5800, \"error_rate_pct\": 42, \"requeue_ceiling\": 5},\n {\"target\": \"refund-processor\", \"window_ms\": 5600, \"error_rate_pct\": 41, \"requeue_ceiling\": 6},\n {\"target\": \"subscription-manager\", \"window_ms\": 4600, \"error_rate_pct\": 49, \"requeue_ceiling\": 5},\n {\"target\": \"loyalty-points\", \"window_ms\": 4300, \"error_rate_pct\": 52, \"requeue_ceiling\": 6},\n {\"target\": \"gift-card-service\", \"window_ms\": 4100, \"error_rate_pct\": 53, \"requeue_ceiling\": 5},\n {\"target\": \"wallet-service\", \"window_ms\": 4900, \"error_rate_pct\": 48, \"requeue_ceiling\": 6},\n {\"target\": \"currency-converter\", \"window_ms\": 4400, \"error_rate_pct\": 51, \"requeue_ceiling\": 5},\n {\"target\": \"address-validator\", \"window_ms\": 4000, \"error_rate_pct\": 54, \"requeue_ceiling\": 6},\n {\"target\": \"geo-locator\", \"window_ms\": 3800, \"error_rate_pct\": 56, \"requeue_ceiling\": 5},\n {\"target\": \"email-templater\", \"window_ms\": 5200, \"error_rate_pct\": 43, \"requeue_ceiling\": 6},\n {\"target\": \"sms-gateway\", \"window_ms\": 5400, \"error_rate_pct\": 39, \"requeue_ceiling\": 5},\n {\"target\": \"push-gateway\", \"window_ms\": 5500, \"error_rate_pct\": 37, \"requeue_ceiling\": 6},\n {\"target\": \"webhook-relay\", \"window_ms\": 5700, \"error_rate_pct\": 36, \"requeue_ceiling\": 5},\n {\"target\": \"audit-trail\", \"window_ms\": 5900, \"error_rate_pct\": 35, \"requeue_ceiling\": 6},\n {\"target\": \"session-store\", \"window_ms\": 4550, \"error_rate_pct\": 47, \"requeue_ceiling\": 5}\n ],\n \"config_hash\": \"sha256:7be2f0c19a44\",\n \"applies_to_env\": \"production\"\n}\n", + "query": "how many times will billing-worker keep re-sending a failed payment authorization call before it gives up and opens the breaker?", + "gold_answer": "\"requeue_ceiling\": 3", + "critical_atoms": [ + "res-pol-6f1c0a3d", + "sha256:7be2f0c19a44" + ], + "notes": "24-entry circuit-breaker policy (34 lines); the answer field is named 'requeue_ceiling', which shares no tokens with the query's 're-sending', 'gives up', or 'opens the breaker' phrasing, so bm25's match on this line comes only from the 'billing'/'worker' tokens in the target name itself, not from any overlap with the requested attribute's wording. frequency and llmlingua_style are query-independent and every entry is equally 'rare' (one target name each), so they cannot single out billing-worker either. This is the semantically-identifiable-but-lexically-distant case: the target name grounds which entry is correct, but the requested attribute's wording does not echo the query. policy_id and config_hash sit on separate lines and are force-kept." +} diff --git a/eval/tasks/v04/json_schema_012.json b/eval/tasks/v04/json_schema_012.json new file mode 100644 index 0000000..6b5230c --- /dev/null +++ b/eval/tasks/v04/json_schema_012.json @@ -0,0 +1,13 @@ +{ + "id": "json_schema_012", + "family": "json_schema", + "tier": "A", + "source": "{\n \"flag_matrix\": \"web-frontend-flags\",\n \"audit_id\": \"flagaudit-3d91c7\",\n \"exported_at\": \"2026-07-22T11:02:44Z\",\n \"environment\": \"production\",\n \"flags\": [\n {\"key\": \"dark-mode-toggle\", \"enabled\": true, \"rollout_percent\": 100, \"owner\": \"team-design\"},\n {\"key\": \"checkout-v2-redesign\", \"enabled\": true, \"rollout_percent\": 35, \"owner\": \"team-checkout\"},\n {\"key\": \"new-search-ranking\", \"enabled\": true, \"rollout_percent\": 20, \"owner\": \"team-search\"},\n {\"key\": \"guest-checkout\", \"enabled\": false, \"rollout_percent\": 0, \"owner\": \"team-checkout\"},\n {\"key\": \"ai-product-summaries\", \"enabled\": true, \"rollout_percent\": 15, \"owner\": \"team-ml\"},\n {\"key\": \"bulk-export-csv\", \"enabled\": true, \"rollout_percent\": 80, \"owner\": \"team-platform\"},\n {\"key\": \"multi-currency-cart\", \"enabled\": false, \"rollout_percent\": 5, \"owner\": \"team-finance\"},\n {\"key\": \"sticky-header-nav\", \"enabled\": true, \"rollout_percent\": 100, \"owner\": \"team-design\"},\n {\"key\": \"lazy-load-images\", \"enabled\": true, \"rollout_percent\": 90, \"owner\": \"team-media\"},\n {\"key\": \"one-click-reorder\", \"enabled\": true, \"rollout_percent\": 25, \"owner\": \"team-checkout\"},\n {\"key\": \"subscription-pause\", \"enabled\": true, \"rollout_percent\": 60, \"owner\": \"team-finance\"},\n {\"key\": \"gift-wrap-option\", \"enabled\": false, \"rollout_percent\": 10, \"owner\": \"team-checkout\"},\n {\"key\": \"dark-launch-recs-v3\", \"enabled\": true, \"rollout_percent\": 8, \"owner\": \"team-ml\"},\n {\"key\": \"email-digest-weekly\", \"enabled\": true, \"rollout_percent\": 70, \"owner\": \"team-comms\"},\n {\"key\": \"push-opt-in-prompt\", \"enabled\": false, \"rollout_percent\": 12, \"owner\": \"team-comms\"},\n {\"key\": \"referral-credits\", \"enabled\": true, \"rollout_percent\": 45, \"owner\": \"team-loyalty\"},\n {\"key\": \"wishlist-sharing\", \"enabled\": true, \"rollout_percent\": 55, \"owner\": \"team-design\"},\n {\"key\": \"address-autocomplete\", \"enabled\": true, \"rollout_percent\": 95, \"owner\": \"team-checkout\"},\n {\"key\": \"fraud-review-queue-v2\", \"enabled\": true, \"rollout_percent\": 30, \"owner\": \"team-security\"},\n {\"key\": \"session-timeout-warn\", \"enabled\": true, \"rollout_percent\": 100, \"owner\": \"team-security\"},\n {\"key\": \"beta-storefront-theme\", \"enabled\": false, \"rollout_percent\": 3, \"owner\": \"team-design\"},\n {\"key\": \"cart-abandon-nudge\", \"enabled\": true, \"rollout_percent\": 65, \"owner\": \"team-comms\"},\n {\"key\": \"inventory-low-badge\", \"enabled\": true, \"rollout_percent\": 85, \"owner\": \"team-platform\"},\n {\"key\": \"gift-card-balance-check\", \"enabled\": true, \"rollout_percent\": 75, \"owner\": \"team-finance\"},\n {\"key\": \"apple-pay-express\", \"enabled\": true, \"rollout_percent\": 50, \"owner\": \"team-checkout\"},\n {\"key\": \"region-price-override\", \"enabled\": false, \"rollout_percent\": 7, \"owner\": \"team-finance\"},\n {\"key\": \"new-returns-flow\", \"enabled\": true, \"rollout_percent\": 22, \"owner\": \"team-checkout\"},\n {\"key\": \"chat-support-widget\", \"enabled\": true, \"rollout_percent\": 40, \"owner\": \"team-comms\"}\n ],\n \"manifest_sha\": \"sha256:0fb6e9a2d417\",\n \"flag_count\": 28\n}\n", + "query": "what rollout_percent is configured for the checkout-v2-redesign flag?", + "gold_answer": "\"checkout-v2-redesign\", \"enabled\": true, \"rollout_percent\": 35", + "critical_atoms": [ + "flagaudit-3d91c7", + "sha256:0fb6e9a2d417" + ], + "notes": "28-flag rollout matrix (36 lines) where every line is the same {\"key\", \"enabled\", \"rollout_percent\", \"owner\"} shape and rollout_percent values are densely packed across the 0-100 range, so recency/frequency/llmlingua_style have no structural signal to prefer the checkout-v2-redesign line over any other flag at a ~10% budget. bm25 gets direct lexical help since 'checkout-v2-redesign' and 'rollout_percent' both echo the query verbatim. audit_id and manifest_sha are on separate header/footer lines and force-kept." +} diff --git a/eval/tasks/v04/json_schema_013.json b/eval/tasks/v04/json_schema_013.json new file mode 100644 index 0000000..23d3328 --- /dev/null +++ b/eval/tasks/v04/json_schema_013.json @@ -0,0 +1,13 @@ +{ + "id": "json_schema_013", + "family": "json_schema", + "tier": "A", + "source": "{\n \"cluster\": \"prod-use1-a\",\n \"spec_id\": \"podspec-77c2e0\",\n \"namespace\": \"media-pipeline\",\n \"generated_at\": \"2026-07-21T08:40:00Z\",\n \"containers\": [\n {\"name\": \"thumbnail-gen\", \"cpu_millis\": 500, \"mem_limit_mb\": 512, \"mem_slack_mb\": 64},\n {\"name\": \"image-resizer\", \"cpu_millis\": 750, \"mem_limit_mb\": 768, \"mem_slack_mb\": 96},\n {\"name\": \"watermark-applier\", \"cpu_millis\": 400, \"mem_limit_mb\": 384, \"mem_slack_mb\": 48},\n {\"name\": \"format-transcoder\", \"cpu_millis\": 900, \"mem_limit_mb\": 1024, \"mem_slack_mb\": 128},\n {\"name\": \"exif-scrubber\", \"cpu_millis\": 300, \"mem_limit_mb\": 256, \"mem_slack_mb\": 32},\n {\"name\": \"video-transcoder-sidecar\", \"cpu_millis\": 1200, \"mem_limit_mb\": 2048, \"mem_slack_mb\": 256},\n {\"name\": \"audio-normalizer\", \"cpu_millis\": 600, \"mem_limit_mb\": 640, \"mem_slack_mb\": 80},\n {\"name\": \"subtitle-extractor\", \"cpu_millis\": 350, \"mem_limit_mb\": 320, \"mem_slack_mb\": 40},\n {\"name\": \"thumbnail-cache-writer\", \"cpu_millis\": 450, \"mem_limit_mb\": 448, \"mem_slack_mb\": 56},\n {\"name\": \"preview-renderer\", \"cpu_millis\": 800, \"mem_limit_mb\": 896, \"mem_slack_mb\": 112},\n {\"name\": \"color-profile-fixer\", \"cpu_millis\": 380, \"mem_limit_mb\": 352, \"mem_slack_mb\": 44},\n {\"name\": \"hls-packager\", \"cpu_millis\": 1100, \"mem_limit_mb\": 1536, \"mem_slack_mb\": 192},\n {\"name\": \"dash-packager\", \"cpu_millis\": 1050, \"mem_limit_mb\": 1408, \"mem_slack_mb\": 176},\n {\"name\": \"scene-detector\", \"cpu_millis\": 700, \"mem_limit_mb\": 768, \"mem_slack_mb\": 96},\n {\"name\": \"upload-scanner\", \"cpu_millis\": 320, \"mem_limit_mb\": 288, \"mem_slack_mb\": 36},\n {\"name\": \"virus-scan-sidecar\", \"cpu_millis\": 260, \"mem_limit_mb\": 224, \"mem_slack_mb\": 28},\n {\"name\": \"metadata-extractor\", \"cpu_millis\": 340, \"mem_limit_mb\": 300, \"mem_slack_mb\": 38},\n {\"name\": \"cdn-warmer\", \"cpu_millis\": 480, \"mem_limit_mb\": 416, \"mem_slack_mb\": 52},\n {\"name\": \"thumbnail-resizer-v2\", \"cpu_millis\": 520, \"mem_limit_mb\": 480, \"mem_slack_mb\": 60},\n {\"name\": \"gif-optimizer\", \"cpu_millis\": 420, \"mem_limit_mb\": 400, \"mem_slack_mb\": 50},\n {\"name\": \"logo-overlay\", \"cpu_millis\": 300, \"mem_limit_mb\": 288, \"mem_slack_mb\": 36},\n {\"name\": \"frame-sampler\", \"cpu_millis\": 560, \"mem_limit_mb\": 544, \"mem_slack_mb\": 68}\n ],\n \"config_digest\": \"sha256:c48a17ef902b\",\n \"container_count\": 22\n}\n", + "query": "how much memory does the video-transcoder-sidecar container hold in reserve above its working set before it risks getting OOM-killed?", + "gold_answer": "\"mem_slack_mb\": 256", + "critical_atoms": [ + "podspec-77c2e0", + "sha256:c48a17ef902b" + ], + "notes": "22-container resource spec (33 lines); the field 'mem_slack_mb' is one underscore-joined token (the harness's word tokenizer keeps underscores), so it never matches the query's 'memory', 'reserve', or 'OOM-killed' wording -- bm25's only overlap is the container name itself, split into 'video/transcoder/sidecar'. frequency and llmlingua_style are query-independent and every container line is equally distinctive (one unique name each), so neither heuristic can single out this line from the other 21 look-alikes at a ~10% budget. spec_id and config_digest are force-kept on separate lines." +} diff --git a/eval/tasks/v04/json_schema_014.json b/eval/tasks/v04/json_schema_014.json new file mode 100644 index 0000000..db70583 --- /dev/null +++ b/eval/tasks/v04/json_schema_014.json @@ -0,0 +1,13 @@ +{ + "id": "json_schema_014", + "family": "json_schema", + "tier": "A", + "source": "{\n \"webhook_registry\": \"partner-integrations\",\n \"registry_id\": \"whreg-52af9d\",\n \"last_synced\": \"2026-07-19T22:15:03Z\",\n \"environment\": \"production\",\n \"subscriptions\": [\n {\"subscriber\": \"order-events-archiver\", \"event\": \"order.created\", \"retry_backoff_ms\": 800, \"active\": true},\n {\"subscriber\": \"fraud-review-hook\", \"event\": \"order.flagged\", \"retry_backoff_ms\": 500, \"active\": true},\n {\"subscriber\": \"inventory-sync\", \"event\": \"order.created\", \"retry_backoff_ms\": 1500, \"active\": true},\n {\"subscriber\": \"warehouse-router\", \"event\": \"order.created\", \"retry_backoff_ms\": 700, \"active\": true},\n {\"subscriber\": \"crm-contact-sync\", \"event\": \"customer.updated\", \"retry_backoff_ms\": 2000, \"active\": true},\n {\"subscriber\": \"loyalty-points-hook\", \"event\": \"order.completed\", \"retry_backoff_ms\": 900, \"active\": true},\n {\"subscriber\": \"email-receipt-sender\", \"event\": \"order.completed\", \"retry_backoff_ms\": 600, \"active\": true},\n {\"subscriber\": \"sms-shipping-alert\", \"event\": \"shipment.created\", \"retry_backoff_ms\": 650, \"active\": false},\n {\"subscriber\": \"analytics-collector\", \"event\": \"order.created\", \"retry_backoff_ms\": 300, \"active\": true},\n {\"subscriber\": \"tax-remit-notifier\", \"event\": \"order.completed\", \"retry_backoff_ms\": 1200, \"active\": true},\n {\"subscriber\": \"returns-desk-hook\", \"event\": \"return.requested\", \"retry_backoff_ms\": 1000, \"active\": true},\n {\"subscriber\": \"subscription-renewer\", \"event\": \"subscription.renewed\", \"retry_backoff_ms\": 1800, \"active\": true},\n {\"subscriber\": \"gift-card-issuer\", \"event\": \"order.completed\", \"retry_backoff_ms\": 750, \"active\": true},\n {\"subscriber\": \"chargeback-alert\", \"event\": \"payment.disputed\", \"retry_backoff_ms\": 400, \"active\": true},\n {\"subscriber\": \"partner-feed-export\", \"event\": \"order.created\", \"retry_backoff_ms\": 2200, \"active\": false},\n {\"subscriber\": \"marketing-audience-sync\", \"event\": \"customer.updated\", \"retry_backoff_ms\": 1600, \"active\": true},\n {\"subscriber\": \"erp-ledger-bridge\", \"event\": \"order.completed\", \"retry_backoff_ms\": 1900, \"active\": true},\n {\"subscriber\": \"supplier-po-notifier\", \"event\": \"inventory.low\", \"retry_backoff_ms\": 1100, \"active\": true},\n {\"subscriber\": \"review-request-hook\", \"event\": \"order.completed\", \"retry_backoff_ms\": 850, \"active\": true},\n {\"subscriber\": \"fraud-hold-release\", \"event\": \"order.flagged\", \"retry_backoff_ms\": 550, \"active\": false},\n {\"subscriber\": \"shipment-tracking-sync\", \"event\": \"shipment.created\", \"retry_backoff_ms\": 720, \"active\": true},\n {\"subscriber\": \"customer-support-hook\", \"event\": \"ticket.created\", \"retry_backoff_ms\": 950, \"active\": true},\n {\"subscriber\": \"refund-ledger-writer\", \"event\": \"refund.issued\", \"retry_backoff_ms\": 1050, \"active\": true}\n ],\n \"integrity_hash\": \"sha256:9de301f5b8c2\",\n \"subscription_count\": 23\n}\n", + "query": "what retry_backoff_ms is set for the inventory-sync webhook subscription?", + "gold_answer": "\"inventory-sync\", \"event\": \"order.created\", \"retry_backoff_ms\": 1500", + "critical_atoms": [ + "whreg-52af9d", + "sha256:9de301f5b8c2" + ], + "notes": "23-subscription webhook registry (33 lines); most lines share the same 'order.created' or 'order.completed' event value and the identical {\"subscriber\", \"event\", \"retry_backoff_ms\", \"active\"} shape, so recency/frequency/llmlingua_style cannot prefer the inventory-sync line over the many same-event decoys at a ~10% budget. bm25 gets direct lexical help since 'inventory-sync' and 'retry_backoff_ms' both echo the query verbatim. registry_id and integrity_hash are force-kept on separate header/footer lines." +} diff --git a/eval/tasks/v04/json_schema_015.json b/eval/tasks/v04/json_schema_015.json new file mode 100644 index 0000000..bf573a1 --- /dev/null +++ b/eval/tasks/v04/json_schema_015.json @@ -0,0 +1,13 @@ +{ + "id": "json_schema_015", + "family": "json_schema", + "tier": "A", + "source": "{\n \"topology\": \"orders-db-shard-map\",\n \"topology_id\": \"topo-4e9b21\",\n \"captured_at\": \"2026-07-23T05:30:11Z\",\n \"cluster\": \"orders-prod\",\n \"shards\": [\n {\"name\": \"shard-01\", \"host\": \"db-shard-01.internal\", \"pool_cap\": 40, \"read_replica\": true},\n {\"name\": \"shard-02\", \"host\": \"db-shard-02.internal\", \"pool_cap\": 42, \"read_replica\": true},\n {\"name\": \"shard-03\", \"host\": \"db-shard-03.internal\", \"pool_cap\": 38, \"read_replica\": false},\n {\"name\": \"shard-04\", \"host\": \"db-shard-04.internal\", \"pool_cap\": 45, \"read_replica\": true},\n {\"name\": \"shard-05\", \"host\": \"db-shard-05.internal\", \"pool_cap\": 36, \"read_replica\": true},\n {\"name\": \"shard-06\", \"host\": \"db-shard-06.internal\", \"pool_cap\": 44, \"read_replica\": false},\n {\"name\": \"shard-07\", \"host\": \"db-shard-07.internal\", \"pool_cap\": 18, \"read_replica\": true},\n {\"name\": \"shard-08\", \"host\": \"db-shard-08.internal\", \"pool_cap\": 41, \"read_replica\": true},\n {\"name\": \"shard-09\", \"host\": \"db-shard-09.internal\", \"pool_cap\": 39, \"read_replica\": true},\n {\"name\": \"shard-10\", \"host\": \"db-shard-10.internal\", \"pool_cap\": 43, \"read_replica\": false},\n {\"name\": \"shard-11\", \"host\": \"db-shard-11.internal\", \"pool_cap\": 37, \"read_replica\": true},\n {\"name\": \"shard-12\", \"host\": \"db-shard-12.internal\", \"pool_cap\": 46, \"read_replica\": true},\n {\"name\": \"shard-13\", \"host\": \"db-shard-13.internal\", \"pool_cap\": 40, \"read_replica\": false},\n {\"name\": \"shard-14\", \"host\": \"db-shard-14.internal\", \"pool_cap\": 35, \"read_replica\": true},\n {\"name\": \"shard-15\", \"host\": \"db-shard-15.internal\", \"pool_cap\": 42, \"read_replica\": true},\n {\"name\": \"shard-16\", \"host\": \"db-shard-16.internal\", \"pool_cap\": 39, \"read_replica\": true},\n {\"name\": \"shard-17\", \"host\": \"db-shard-17.internal\", \"pool_cap\": 44, \"read_replica\": false},\n {\"name\": \"shard-18\", \"host\": \"db-shard-18.internal\", \"pool_cap\": 38, \"read_replica\": true},\n {\"name\": \"shard-19\", \"host\": \"db-shard-19.internal\", \"pool_cap\": 41, \"read_replica\": true},\n {\"name\": \"shard-20\", \"host\": \"db-shard-20.internal\", \"pool_cap\": 43, \"read_replica\": true}\n ],\n \"manifest_hash\": \"sha256:2a6f0d9be174\",\n \"shard_count\": 20\n}\n", + "query": "what is the maximum number of simultaneous connections permitted to shard-07 before new requests start queueing?", + "gold_answer": "\"pool_cap\": 18", + "critical_atoms": [ + "topo-4e9b21", + "sha256:2a6f0d9be174" + ], + "notes": "20-shard topology (32 lines); the field 'pool_cap' shares no tokens with the query's 'maximum', 'simultaneous connections', 'permitted', or 'queueing' phrasing, so bm25's only overlap is the 'shard-07' identifier itself. pool_cap values cluster tightly (18-46) except shard-07's outlier low value, but frequency and llmlingua_style are query-independent and every shard line is equally 'rare' (one unique name/host each), so neither heuristic can single out shard-07 from the other 19 look-alike shards at a ~10% budget. topology_id and manifest_hash are force-kept on separate lines." +} diff --git a/eval/tasks/v04/log_multi_service_001.json b/eval/tasks/v04/log_multi_service_001.json new file mode 100644 index 0000000..4c23d7d --- /dev/null +++ b/eval/tasks/v04/log_multi_service_001.json @@ -0,0 +1,12 @@ +{ + "id": "log_multi_service_001", + "family": "log_multi_service", + "tier": "A", + "source": "2026-07-14T09:12:01Z INFO service=gateway request received route=/checkout method=POST trace=tr-8821\n2026-07-14T09:12:01Z INFO service=orders creating order for cart=cart-3390 items=3\n2026-07-14T09:12:02Z INFO service=inventory reserving stock for order sku=WIDGET-42 qty=3\n2026-07-14T09:12:02Z DEBUG service=inventory opening connection to inventory-db replica=r2\n2026-07-14T09:12:03Z ERROR service=inventory connection pool exhausted max_connections=50 reached code=POOL_EXHAUSTED\n2026-07-14T09:12:03Z WARN service=inventory falling back to primary replica after the failure\n2026-07-14T09:12:04Z INFO service=orders received 503 from inventory, marking order pending\n2026-07-14T09:12:04Z INFO service=payments authorization deferred pending stock confirmation\n2026-07-14T09:12:05Z INFO service=gateway responded status=503 latency=3200ms\n2026-07-14T09:12:06Z INFO service=gateway healthcheck ok upstreams=4\n2026-07-14T09:12:07Z INFO service=orders retry queued for order cart=cart-3390\n2026-07-14T09:12:08Z INFO service=notifications email queued template=order-delayed user=carol\n2026-07-14T09:12:09Z INFO service=gateway request received route=/status method=GET\n2026-07-14T09:12:10Z INFO service=metrics flushed 128 series to sink=prometheus\n2026-07-14T09:12:11Z INFO service=orders reconciliation sweep completed trace=tr-8821 req-id=req-4d7e\n2026-07-14T09:12:12Z INFO service=gateway shutting down drain=graceful\n", + "query": "what caused the inventory service connection pool to become exhausted during checkout?", + "gold_answer": "POOL_EXHAUSTED", + "critical_atoms": [ + "req-4d7e" + ], + "notes": "Root cause is an early ERROR line (POOL_EXHAUSTED) that shares inventory/connection/pool/exhausted with the query, so bm25 keeps it but recency and forced_only drop it at 0.25; the audit-critical req-id sits on a later line and is force-kept regardless of selector." +} diff --git a/eval/tasks/v04/log_multi_service_010.json b/eval/tasks/v04/log_multi_service_010.json new file mode 100644 index 0000000..727ac16 --- /dev/null +++ b/eval/tasks/v04/log_multi_service_010.json @@ -0,0 +1,13 @@ +{ + "id": "log_multi_service_010", + "family": "log_multi_service", + "tier": "A", + "source": "2026-07-18T03:10:01Z INFO service=gateway request received route=/api/v2/orders method=POST trace=tr-2b91f4\n2026-07-18T03:10:01Z INFO service=authsvc token validated user=alice scope=orders:write\n2026-07-18T03:10:02Z INFO service=sessions session touched sid=sess-91a2 ttl=1800\n2026-07-18T03:10:02Z INFO service=ratelimit tier=standard config applied rpm_limit=600 burst=100\n2026-07-18T03:10:03Z INFO service=metrics flushed 142 series to sink=prometheus\n2026-07-18T03:10:03Z DEBUG service=audit heartbeat ok uptime=41213s\n2026-07-18T03:10:04Z INFO service=gateway request received route=/api/v2/status method=GET trace=tr-2b91f5\n2026-07-18T03:10:04Z INFO service=ratelimit tier=basic config applied rpm_limit=300 burst=50\n2026-07-18T03:10:05Z INFO service=orders order created cart=cart-5521 items=2\n2026-07-18T03:10:05Z DEBUG service=authsvc refreshing signing keys keyset=primary\n2026-07-18T03:10:06Z INFO service=sessions session touched sid=sess-91a7 ttl=1800\n2026-07-18T03:10:06Z WARN service=gateway elevated latency route=/api/v2/orders latency=812ms\n2026-07-18T03:10:07Z INFO service=ratelimit tier=premium config applied rpm_limit=900 burst=150\n2026-07-18T03:10:07Z INFO service=metrics flushed 138 series to sink=prometheus\n2026-07-18T03:10:08Z DEBUG service=audit heartbeat ok uptime=41273s\n2026-07-18T03:10:08Z INFO service=gateway request received route=/api/v2/cart method=PUT trace=tr-2b91f6\n2026-07-18T03:10:09Z INFO service=ratelimit tier=enterprise config applied rpm_limit=1200 burst=200\n2026-07-18T03:10:09Z INFO service=orders order updated cart=cart-5521 items=3\n2026-07-18T03:10:10Z INFO service=authsvc token validated user=carol scope=orders:read\n2026-07-18T03:10:10Z INFO service=sessions session touched sid=sess-91b0 ttl=1800\n2026-07-18T03:10:11Z INFO service=gateway request received route=/api/v2/orders method=POST trace=tr-2b91f7\n2026-07-18T03:10:11Z INFO service=ratelimit tier=trial config applied rpm_limit=100 burst=20\n2026-07-18T03:10:12Z INFO service=metrics flushed 151 series to sink=prometheus\n2026-07-18T03:10:12Z DEBUG service=audit heartbeat ok uptime=41333s\n2026-07-18T03:10:13Z WARN service=gateway throttled response=429 route=/api/v2/orders retry_after=5s\n2026-07-18T03:10:13Z INFO service=ratelimit tier=partner config applied rpm_limit=750 burst=125\n2026-07-18T03:10:14Z INFO service=orders reconciliation sweep completed cart=cart-5521\n2026-07-18T03:10:14Z DEBUG service=authsvc refreshing signing keys keyset=secondary\n2026-07-18T03:10:15Z INFO service=sessions session expired sid=sess-90f4\n2026-07-18T03:10:15Z INFO service=gateway request received route=/api/v2/status method=GET trace=tr-2b91f8\n2026-07-18T03:10:16Z DEBUG service=ratelimit tier lookup cache hit tier=enterprise\n2026-07-18T03:10:16Z INFO service=metrics flushed 145 series to sink=prometheus\n2026-07-18T03:10:17Z DEBUG service=audit heartbeat ok uptime=41393s\n2026-07-18T03:10:17Z INFO service=gateway deploy marker applied build=dpl-9c31e0 environment=prod\n2026-07-18T03:10:18Z INFO service=orders order shipped cart=cart-5521 carrier=ups\n2026-07-18T03:10:18Z INFO service=authsvc token validated user=dave scope=orders:write\n2026-07-18T03:10:19Z INFO service=sessions session touched sid=sess-91c2 ttl=1800\n2026-07-18T03:10:19Z INFO service=gateway request received route=/api/v2/health method=GET trace=tr-2b91f9\n2026-07-18T03:10:20Z INFO service=ratelimit tier=standard config applied rpm_limit=600 burst=100\n2026-07-18T03:10:20Z INFO service=metrics flushed 133 series to sink=prometheus\n2026-07-18T03:10:21Z INFO service=audit reconciliation sweep completed req-id=req-7a1c9e\n", + "query": "what request-per-minute threshold triggered rate limiting for the enterprise tier client?", + "gold_answer": "rpm_limit=1200", + "critical_atoms": [ + "req-7a1c9e", + "dpl-9c31e0" + ], + "notes": "rpm_limit=1200 sits on one of seven near-identical `service=ratelimit ... config applied rpm_limit=N` lines (six other tiers, one repeated value) plus a same-vocabulary `tier=enterprise` cache-hit decoy that carries no rpm_limit at all; at a ~10% budget only bm25 (query shares 'tier'/'enterprise') keeps the right line, while recency, frequency, and llmlingua_style are misled by the six other tier lines' identical shape and drop it. The req-id and deploy-hash atoms sit on their own lines and are force-kept regardless of selector." +} diff --git a/eval/tasks/v04/log_multi_service_011.json b/eval/tasks/v04/log_multi_service_011.json new file mode 100644 index 0000000..a1888b2 --- /dev/null +++ b/eval/tasks/v04/log_multi_service_011.json @@ -0,0 +1,13 @@ +{ + "id": "log_multi_service_011", + "family": "log_multi_service", + "tier": "A", + "source": "2026-07-19T11:00:01Z INFO service=heartbeat node=sched-node-01 alive interval=5s\n2026-07-19T11:00:01Z INFO service=heartbeat node=sched-node-03 alive interval=5s\n2026-07-19T11:00:02Z INFO service=workerpool task dispatched job=job-6612 node=sched-node-03\n2026-07-19T11:00:02Z DEBUG service=coordinator candidate registered node=sched-node-01 priority=3\n2026-07-19T11:00:03Z INFO service=gateway request received route=/api/v2/jobs method=GET\n2026-07-19T11:00:03Z INFO service=heartbeat node=sched-node-05 alive interval=5s\n2026-07-19T11:00:04Z DEBUG service=coordinator candidate registered node=sched-node-05 priority=1\n2026-07-19T11:00:04Z INFO service=metrics flushed 96 series to sink=prometheus\n2026-07-19T11:00:05Z INFO service=workerpool task dispatched job=job-6613 node=sched-node-01\n2026-07-19T11:00:05Z DEBUG service=coordinator candidate registered node=sched-node-07 priority=9\n2026-07-19T11:00:06Z INFO service=heartbeat node=sched-node-07 alive interval=5s\n2026-07-19T11:00:06Z INFO service=audit checkpoint saved term=term-00417\n2026-07-19T11:00:07Z INFO service=gateway request received route=/api/v2/status method=GET\n2026-07-19T11:00:07Z INFO service=heartbeat node=sched-node-01 alive interval=5s\n2026-07-19T11:00:08Z DEBUG service=coordinator quorum check members=7 responding=7\n2026-07-19T11:00:08Z INFO service=workerpool task dispatched job=job-6614 node=sched-node-05\n2026-07-19T11:00:09Z INFO service=metrics flushed 101 series to sink=prometheus\n2026-07-19T11:00:09Z INFO service=heartbeat node=sched-node-03 alive interval=5s\n2026-07-19T11:00:10Z INFO service=coordinator election resolved winner=sched-node-07 quorum=5/7\n2026-07-19T11:00:10Z INFO service=workerpool task dispatched job=job-6615 node=sched-node-01\n2026-07-19T11:00:11Z INFO service=gateway request received route=/api/v2/jobs method=POST\n2026-07-19T11:00:11Z INFO service=heartbeat node=sched-node-05 alive interval=5s\n2026-07-19T11:00:12Z DEBUG service=coordinator candidate registered node=sched-node-03 priority=4\n2026-07-19T11:00:12Z INFO service=metrics flushed 108 series to sink=prometheus\n2026-07-19T11:00:13Z INFO service=workerpool task dispatched job=job-6616 node=sched-node-07\n2026-07-19T11:00:13Z INFO service=heartbeat node=sched-node-07 alive interval=5s\n2026-07-19T11:00:14Z INFO service=audit reconciliation sweep completed ref=aud-51de9b\n2026-07-19T11:00:14Z DEBUG service=coordinator candidate registered node=sched-node-01 priority=2\n2026-07-19T11:00:15Z INFO service=gateway request received route=/api/v2/status method=GET\n2026-07-19T11:00:15Z INFO service=workerpool task dispatched job=job-6617 node=sched-node-03\n2026-07-19T11:00:16Z INFO service=heartbeat node=sched-node-01 alive interval=5s\n2026-07-19T11:00:16Z INFO service=metrics flushed 112 series to sink=prometheus\n2026-07-19T11:00:17Z DEBUG service=coordinator candidate registered node=sched-node-05 priority=6\n2026-07-19T11:00:17Z INFO service=workerpool task dispatched job=job-6618 node=sched-node-07\n2026-07-19T11:00:18Z INFO service=heartbeat node=sched-node-03 alive interval=5s\n2026-07-19T11:00:18Z INFO service=gateway request received route=/api/v2/jobs method=GET\n2026-07-19T11:00:19Z INFO service=metrics flushed 119 series to sink=prometheus\n2026-07-19T11:00:19Z INFO service=heartbeat node=sched-node-05 alive interval=5s\n2026-07-19T11:00:20Z INFO service=workerpool task dispatched job=job-6619 node=sched-node-01\n2026-07-19T11:00:20Z INFO service=coordinator shutting down election listener drain=graceful\n", + "query": "which node is currently holding cluster leadership?", + "gold_answer": "winner=sched-node-07", + "critical_atoms": [ + "term-00417", + "aud-51de9b" + ], + "notes": "Leadership resolves to sched-node-07 on a single 'election resolved winner=' line that shares no wording with the query ('holding cluster leadership'), buried among ~30 near-identical heartbeat/candidate/dispatch lines naming the same node ids. At a ~10% budget bm25 has zero query-term overlap to exploit and drops it, as do recency and llmlingua_style; only frequency's rare-token weighting happens to surface the line. The term and audit-ref atoms sit on separate lines and are force-kept regardless of selector." +} diff --git a/eval/tasks/v04/log_multi_service_012.json b/eval/tasks/v04/log_multi_service_012.json new file mode 100644 index 0000000..08909a4 --- /dev/null +++ b/eval/tasks/v04/log_multi_service_012.json @@ -0,0 +1,13 @@ +{ + "id": "log_multi_service_012", + "family": "log_multi_service", + "tier": "A", + "source": "2026-07-20T14:50:01Z INFO service=gateway request received route=/api/v2/checkout method=POST trace=tr-6f10ab\n2026-07-20T14:50:01Z INFO service=orders order created cart=cart-7734 items=1\n2026-07-20T14:50:02Z INFO service=payments authorization requested amount=48.00 currency=USD\n2026-07-20T14:50:02Z INFO service=fraud risk score computed score=12 verdict=allow\n2026-07-20T14:50:03Z INFO service=ledger entry posted account=merchant-4471 amount=48.00\n2026-07-20T14:50:03Z INFO service=metrics flushed 87 series to sink=prometheus\n2026-07-20T14:50:04Z INFO service=payments charge accepted idempotency_key=idem-1a4d02f7 status=CAPTURED\n2026-07-20T14:50:04Z DEBUG service=gateway retry policy evaluated attempt=1 backoff=200ms\n2026-07-20T14:50:05Z INFO service=orders order created cart=cart-7735 items=4\n2026-07-20T14:50:05Z INFO service=payments charge accepted idempotency_key=idem-77bb3e91 status=CAPTURED\n2026-07-20T14:50:05Z WARN service=payments duplicate charge blocked idempotency_key=idem-3c7710aa reason=INSUFFICIENT_FUNDS\n2026-07-20T14:50:06Z INFO service=fraud risk score computed score=8 verdict=allow\n2026-07-20T14:50:06Z INFO service=ledger entry posted account=merchant-4471 amount=91.50\n2026-07-20T14:50:07Z INFO service=metrics flushed 90 series to sink=prometheus\n2026-07-20T14:50:07Z INFO service=gateway request received route=/api/v2/checkout method=POST trace=tr-6f10ac\n2026-07-20T14:50:08Z INFO service=orders order created cart=cart-7736 items=2\n2026-07-20T14:50:08Z INFO service=payments charge accepted idempotency_key=idem-52c9d004 status=CAPTURED\n2026-07-20T14:50:09Z INFO service=fraud risk score computed score=5 verdict=allow\n2026-07-20T14:50:09Z INFO service=ledger entry posted account=merchant-4471 amount=22.00\n2026-07-20T14:50:10Z INFO service=metrics flushed 93 series to sink=prometheus\n2026-07-20T14:50:10Z DEBUG service=gateway retry policy evaluated attempt=2 backoff=400ms\n2026-07-20T14:50:10Z WARN service=payments duplicate charge blocked idempotency_key=idem-84f2b6dd reason=EXPIRED_CARD\n2026-07-20T14:50:11Z WARN service=payments duplicate charge blocked idempotency_key=idem-6f0a19c2 reason=REPLAYED\n2026-07-20T14:50:11Z INFO service=orders order created cart=cart-7737 items=1\n2026-07-20T14:50:12Z INFO service=payments charge accepted idempotency_key=idem-e40b7a15 status=CAPTURED\n2026-07-20T14:50:12Z INFO service=fraud risk score computed score=11 verdict=allow\n2026-07-20T14:50:13Z INFO service=ledger entry posted account=merchant-4471 amount=63.25\n2026-07-20T14:50:13Z INFO service=metrics flushed 97 series to sink=prometheus\n2026-07-20T14:50:14Z INFO service=audit transaction reference recorded txn=txn-88231f\n2026-07-20T14:50:14Z INFO service=gateway request received route=/api/v2/checkout method=POST trace=tr-6f10ad\n2026-07-20T14:50:15Z INFO service=orders order created cart=cart-7738 items=3\n2026-07-20T14:50:15Z INFO service=payments charge accepted idempotency_key=idem-0d88c3f1 status=CAPTURED\n2026-07-20T14:50:16Z INFO service=fraud risk score computed score=7 verdict=allow\n2026-07-20T14:50:16Z INFO service=ledger entry posted account=merchant-4471 amount=15.75\n2026-07-20T14:50:17Z INFO service=metrics flushed 99 series to sink=prometheus\n2026-07-20T14:50:17Z DEBUG service=gateway retry policy evaluated attempt=3 backoff=800ms\n2026-07-20T14:50:18Z INFO service=payments charge accepted idempotency_key=idem-9931fa22 status=CAPTURED\n2026-07-20T14:50:18Z INFO service=orders order created cart=cart-7739 items=2\n2026-07-20T14:50:19Z INFO service=fraud risk score computed score=9 verdict=allow\n2026-07-20T14:50:19Z INFO service=metrics flushed 101 series to sink=prometheus\n2026-07-20T14:50:20Z INFO service=audit reconciliation sweep completed ref=aud-2b77c4\n", + "query": "what idempotency key was recorded when a payment was blocked as a replayed duplicate charge?", + "gold_answer": "idempotency_key=idem-6f0a19c2", + "critical_atoms": [ + "txn-88231f", + "aud-2b77c4" + ], + "notes": "Three visually identical `duplicate charge blocked idempotency_key=... reason=...` lines exist (insufficient-funds and expired-card decoys plus the real replay), interleaved among a dozen ordinary `charge accepted` lines. At a ~10% budget only bm25 -- driven by the query's 'replayed' term, unique to the real line -- keeps it; recency, frequency, and llmlingua_style each keep one of the *other* two near-identical blocked-charge decoys instead. The txn and audit-ref atoms sit on separate lines and are force-kept regardless of selector." +} diff --git a/eval/tasks/v04/log_multi_service_013.json b/eval/tasks/v04/log_multi_service_013.json new file mode 100644 index 0000000..5f323fa --- /dev/null +++ b/eval/tasks/v04/log_multi_service_013.json @@ -0,0 +1,13 @@ +{ + "id": "log_multi_service_013", + "family": "log_multi_service", + "tier": "A", + "source": "2026-07-21T08:25:01Z INFO service=dns record refreshed zone=edge.internal ttl=300\n2026-07-21T08:25:01Z INFO service=gateway request received route=/api/v2/status method=GET\n2026-07-21T08:25:02Z INFO service=certmanager scan started scope=edge-proxy fleet=8\n2026-07-21T08:25:02Z DEBUG service=edge-proxy config reloaded listeners=4\n2026-07-21T08:25:03Z INFO service=certmanager rotation complete for authsvc not_after=2026-08-01T00:00:00Z\n2026-07-21T08:25:03Z INFO service=metrics flushed 76 series to sink=prometheus\n2026-07-21T08:25:04Z INFO service=dns record refreshed zone=api.internal ttl=300\n2026-07-21T08:25:04Z INFO service=certmanager rotation complete for payments not_after=2026-08-05T00:00:00Z\n2026-07-21T08:25:05Z DEBUG service=edge-proxy config reloaded listeners=4\n2026-07-21T08:25:05Z INFO service=gateway request received route=/api/v2/health method=GET\n2026-07-21T08:25:06Z INFO service=certmanager rotation complete for orders not_after=2026-08-11T00:00:00Z\n2026-07-21T08:25:06Z INFO service=metrics flushed 79 series to sink=prometheus\n2026-07-21T08:25:07Z INFO service=dns record refreshed zone=payments.internal ttl=300\n2026-07-21T08:25:07Z INFO service=certmanager rotation complete for edge-proxy not_after=2026-09-02T00:00:00Z\n2026-07-21T08:25:08Z DEBUG service=edge-proxy config reloaded listeners=4\n2026-07-21T08:25:08Z INFO service=gateway request received route=/api/v2/status method=GET\n2026-07-21T08:25:09Z INFO service=certmanager rotation complete for notifications not_after=2026-08-14T00:00:00Z\n2026-07-21T08:25:09Z INFO service=metrics flushed 81 series to sink=prometheus\n2026-07-21T08:25:10Z INFO service=dns record refreshed zone=orders.internal ttl=300\n2026-07-21T08:25:10Z INFO service=audit change recorded ticket=chg-40291\n2026-07-21T08:25:11Z INFO service=certmanager rotation complete for sessions not_after=2026-08-19T00:00:00Z\n2026-07-21T08:25:11Z DEBUG service=edge-proxy config reloaded listeners=4\n2026-07-21T08:25:12Z INFO service=gateway request received route=/api/v2/health method=GET\n2026-07-21T08:25:12Z INFO service=metrics flushed 84 series to sink=prometheus\n2026-07-21T08:25:13Z INFO service=certmanager rotation complete for ratelimit not_after=2026-08-22T00:00:00Z\n2026-07-21T08:25:13Z INFO service=dns record refreshed zone=metrics.internal ttl=300\n2026-07-21T08:25:14Z DEBUG service=edge-proxy config reloaded listeners=4\n2026-07-21T08:25:14Z INFO service=gateway request received route=/api/v2/status method=GET\n2026-07-21T08:25:15Z INFO service=certmanager rotation complete for audit not_after=2026-08-27T00:00:00Z\n2026-07-21T08:25:15Z INFO service=metrics flushed 88 series to sink=prometheus\n2026-07-21T08:25:16Z INFO service=certmanager serial recorded serial=5f3a9c11e0\n2026-07-21T08:25:16Z INFO service=dns record refreshed zone=gateway.internal ttl=300\n2026-07-21T08:25:17Z DEBUG service=edge-proxy config reloaded listeners=4\n2026-07-21T08:25:17Z INFO service=gateway request received route=/api/v2/health method=GET\n2026-07-21T08:25:18Z INFO service=metrics flushed 90 series to sink=prometheus\n2026-07-21T08:25:18Z INFO service=certmanager scan completed scope=edge-proxy fleet=8 issues=0\n", + "query": "when does the edge proxy's current TLS material stop being valid?", + "gold_answer": "not_after=2026-09-02T00:00:00Z", + "critical_atoms": [ + "chg-40291", + "5f3a9c11e0" + ], + "notes": "The edge-proxy's not_after date is one of nine near-identical `certmanager rotation complete for not_after=...` lines for other services, and the query avoids 'rotation'/'not_after' entirely ('stop being valid'). At a ~10% budget every selector -- including bm25, which has no query-term overlap to exploit here -- keeps a different service's expiry line instead of the edge-proxy one; this is the hardest fixture in the batch for every deterministic baseline. The change-ticket and serial atoms sit on separate lines and are force-kept regardless of selector." +} diff --git a/eval/tasks/v04/log_multi_service_014.json b/eval/tasks/v04/log_multi_service_014.json new file mode 100644 index 0000000..767d56b --- /dev/null +++ b/eval/tasks/v04/log_multi_service_014.json @@ -0,0 +1,13 @@ +{ + "id": "log_multi_service_014", + "family": "log_multi_service", + "tier": "A", + "source": "2026-07-22T19:40:01Z INFO service=queue notifications-worker depth queue_depth=1120 threshold=20000\n2026-07-22T19:40:01Z INFO service=gateway request received route=/api/v2/invoices method=GET\n2026-07-22T19:40:02Z INFO service=queue exports-worker depth queue_depth=3402 threshold=25000\n2026-07-22T19:40:02Z DEBUG service=billing-worker polling next batch batch_size=200\n2026-07-22T19:40:03Z INFO service=metrics flushed 64 series to sink=prometheus\n2026-07-22T19:40:03Z INFO service=queue analytics-worker depth queue_depth=7715 threshold=30000\n2026-07-22T19:40:04Z INFO service=gateway request received route=/api/v2/invoices method=POST\n2026-07-22T19:40:04Z DEBUG service=billing-worker processing batch batch_id=bwb-2291\n2026-07-22T19:40:05Z INFO service=queue notifications-worker depth queue_depth=1188 threshold=20000\n2026-07-22T19:40:05Z INFO service=metrics flushed 66 series to sink=prometheus\n2026-07-22T19:40:06Z INFO service=queue exports-worker depth queue_depth=3550 threshold=25000\n2026-07-22T19:40:06Z INFO service=audit worker registered wrk=wrk-38f0c1\n2026-07-22T19:40:07Z DEBUG service=billing-worker processing batch batch_id=bwb-2292\n2026-07-22T19:40:07Z INFO service=gateway request received route=/api/v2/status method=GET\n2026-07-22T19:40:08Z INFO service=queue analytics-worker depth queue_depth=8021 threshold=30000\n2026-07-22T19:40:08Z INFO service=metrics flushed 69 series to sink=prometheus\n2026-07-22T19:40:09Z INFO service=queue notifications-worker depth queue_depth=1240 threshold=20000\n2026-07-22T19:40:09Z DEBUG service=billing-worker processing batch batch_id=bwb-2293\n2026-07-22T19:40:10Z INFO service=queue exports-worker depth queue_depth=3699 threshold=25000\n2026-07-22T19:40:10Z INFO service=gateway request received route=/api/v2/invoices method=GET\n2026-07-22T19:40:11Z INFO service=metrics flushed 71 series to sink=prometheus\n2026-07-22T19:40:11Z WARN service=queue billing-worker backlog queue_depth=48211 threshold=40000\n2026-07-22T19:40:12Z INFO service=queue analytics-worker depth queue_depth=8390 threshold=30000\n2026-07-22T19:40:12Z DEBUG service=billing-worker processing batch batch_id=bwb-2294\n2026-07-22T19:40:13Z INFO service=gateway request received route=/api/v2/status method=GET\n2026-07-22T19:40:13Z INFO service=metrics flushed 73 series to sink=prometheus\n2026-07-22T19:40:14Z INFO service=queue notifications-worker depth queue_depth=1301 threshold=20000\n2026-07-22T19:40:14Z INFO service=audit incident opened inc=inc-77021\n2026-07-22T19:40:15Z INFO service=queue exports-worker depth queue_depth=3812 threshold=25000\n2026-07-22T19:40:15Z DEBUG service=billing-worker processing batch batch_id=bwb-2295\n2026-07-22T19:40:16Z INFO service=gateway request received route=/api/v2/invoices method=POST\n2026-07-22T19:40:16Z INFO service=metrics flushed 75 series to sink=prometheus\n2026-07-22T19:40:17Z INFO service=queue analytics-worker depth queue_depth=8654 threshold=30000\n2026-07-22T19:40:17Z WARN service=queue notifications-worker backlog queue_depth=21908 threshold=20000\n2026-07-22T19:40:18Z DEBUG service=billing-worker processing batch batch_id=bwb-2296\n2026-07-22T19:40:18Z INFO service=metrics flushed 77 series to sink=prometheus\n2026-07-22T19:40:19Z INFO service=gateway request received route=/api/v2/status method=GET\n", + "query": "what was the queue depth for the billing worker just before the backlog alert fired?", + "gold_answer": "queue_depth=48211", + "critical_atoms": [ + "wrk-38f0c1", + "inc-77021" + ], + "notes": "queue_depth=48211 sits on one WARN line among a dozen near-identical `queue depth queue_depth=N threshold=M` lines for three other workers, plus a second, later backlog decoy for a different worker. At a ~10% budget bm25 (query shares 'queue depth'/'billing worker'/'backlog') is the only selector that keeps the right line -- recency prefers the later notifications-worker backlog decoy, and frequency/llmlingua_style spread weight across the many similarly-shaped depth lines and miss it. The worker-id and incident atoms sit on separate lines and are force-kept regardless of selector." +} diff --git a/eval/tasks/v04/log_multi_service_015.json b/eval/tasks/v04/log_multi_service_015.json new file mode 100644 index 0000000..f62922b --- /dev/null +++ b/eval/tasks/v04/log_multi_service_015.json @@ -0,0 +1,13 @@ +{ + "id": "log_multi_service_015", + "family": "log_multi_service", + "tier": "A", + "source": "2026-07-23T05:10:01Z INFO service=etl extract started source=orders-db rows=88213\n2026-07-23T05:10:01Z INFO service=gateway request received route=/api/v2/reports method=GET\n2026-07-23T05:10:02Z INFO service=migrator applied schema change target=orders-db version=0112\n2026-07-23T05:10:02Z DEBUG service=reporting-db connection opened pool=r1\n2026-07-23T05:10:03Z INFO service=metrics flushed 58 series to sink=prometheus\n2026-07-23T05:10:03Z INFO service=migrator applied schema change target=sessions-db version=0009\n2026-07-23T05:10:04Z INFO service=etl extract started source=payments-db rows=41220\n2026-07-23T05:10:04Z DEBUG service=reporting-db connection opened pool=r2\n2026-07-23T05:10:05Z INFO service=migrator applied schema change target=ledger-db version=0031\n2026-07-23T05:10:05Z INFO service=metrics flushed 60 series to sink=prometheus\n2026-07-23T05:10:06Z INFO service=gateway request received route=/api/v2/reports method=GET\n2026-07-23T05:10:06Z INFO service=audit lock acquired lock=lock-9931ac\n2026-07-23T05:10:07Z INFO service=migrator applied schema change target=reporting-db version=0047\n2026-07-23T05:10:07Z DEBUG service=reporting-db connection opened pool=r3\n2026-07-23T05:10:08Z INFO service=etl extract started source=fraud-db rows=15044\n2026-07-23T05:10:08Z INFO service=metrics flushed 63 series to sink=prometheus\n2026-07-23T05:10:09Z INFO service=migrator applied schema change target=notifications-db version=0004\n2026-07-23T05:10:09Z INFO service=gateway request received route=/api/v2/status method=GET\n2026-07-23T05:10:10Z DEBUG service=reporting-db connection opened pool=r4\n2026-07-23T05:10:10Z INFO service=migrator applied schema change target=audit-db version=0018\n2026-07-23T05:10:11Z INFO service=metrics flushed 65 series to sink=prometheus\n2026-07-23T05:10:11Z INFO service=etl extract started source=metrics-db rows=9902\n2026-07-23T05:10:12Z INFO service=migrator applied schema change target=ratelimit-db version=0002\n2026-07-23T05:10:12Z DEBUG service=reporting-db connection opened pool=r5\n2026-07-23T05:10:13Z INFO service=gateway request received route=/api/v2/reports method=GET\n2026-07-23T05:10:13Z INFO service=metrics flushed 67 series to sink=prometheus\n2026-07-23T05:10:14Z INFO service=migrator applied schema change target=orders-db version=0113\n2026-07-23T05:10:14Z INFO service=audit reconciliation sweep completed ref=aud-60fd2e\n2026-07-23T05:10:15Z DEBUG service=reporting-db connection opened pool=r6\n2026-07-23T05:10:15Z INFO service=etl extract completed source=orders-db rows=88213 duration=812ms\n2026-07-23T05:10:16Z INFO service=metrics flushed 69 series to sink=prometheus\n2026-07-23T05:10:16Z INFO service=migrator applied schema change target=sessions-db version=0010\n2026-07-23T05:10:17Z INFO service=gateway request received route=/api/v2/status method=GET\n2026-07-23T05:10:17Z DEBUG service=reporting-db connection opened pool=r7\n2026-07-23T05:10:18Z INFO service=migrator applied schema change target=ledger-db version=0032\n2026-07-23T05:10:18Z INFO service=metrics flushed 71 series to sink=prometheus\n2026-07-23T05:10:19Z INFO service=etl extract started source=audit-db rows=2210\n", + "query": "what revision is the reporting database's schema currently at?", + "gold_answer": "target=reporting-db version=0047", + "critical_atoms": [ + "lock-9931ac", + "aud-60fd2e" + ], + "notes": "version=0047 for reporting-db is one of ten near-identical `migrator applied schema change target= version=N` lines, and the query avoids 'version'/'applied' ('what revision ... currently at'). The reporting-db migration line is the only line carrying both matched query terms ('schema' and 'reporting') at once -- every other migrator line has 'schema' without 'reporting', and every 'reporting-db connection opened' line has 'reporting' without 'schema' -- so at a ~10% budget bm25's compositional match keeps it, while recency, frequency, and llmlingua_style (which do not use the query) keep other migration lines instead. The lock and audit-ref atoms sit on separate lines and are force-kept regardless of selector." +} diff --git a/eval/tasks/v04/log_qa_010.json b/eval/tasks/v04/log_qa_010.json new file mode 100644 index 0000000..63aa5f6 --- /dev/null +++ b/eval/tasks/v04/log_qa_010.json @@ -0,0 +1,12 @@ +{ + "id": "log_qa_010", + "family": "log_qa", + "tier": "A", + "source": "2026-07-24T09:00:00Z INFO service=payment starting up region=us-west-2\n2026-07-24T09:00:01Z INFO service=payment config loaded pool_size=64\n2026-07-24T09:00:02Z INFO service=payment audit trail id txn-8f21ab9c archived\n2026-07-24T09:00:03Z INFO service=payment healthcheck ok\n2026-07-24T09:00:04Z INFO service=payment worker=W1 processing job jobid=J1000 status=ok\n2026-07-24T09:00:05Z INFO service=payment worker=W2 processing job jobid=J1001 status=ok\n2026-07-24T09:00:06Z INFO service=payment worker=W3 processing job jobid=J1002 status=ok\n2026-07-24T09:00:07Z INFO service=payment worker=W4 processing job jobid=J1003 status=ok\n2026-07-24T09:00:08Z INFO service=payment worker=W5 processing job jobid=J1004 status=ok\n2026-07-24T09:00:09Z INFO service=payment worker=W6 processing job jobid=J1005 status=ok\n2026-07-24T09:00:10Z INFO service=payment worker=W1 processing job jobid=J1006 status=ok\n2026-07-24T09:00:11Z INFO service=payment worker=W2 processing job jobid=J1007 status=ok\n2026-07-24T09:00:12Z INFO service=payment worker=W3 processing job jobid=J1008 status=ok\n2026-07-24T09:00:13Z ERROR service=payment job declined jobid=J1009 reason=CARD_EXPIRED\n2026-07-24T09:00:14Z INFO service=payment worker=W1 processing job jobid=J1010 status=ok\n2026-07-24T09:00:15Z INFO service=payment worker=W2 processing job jobid=J1011 status=ok\n2026-07-24T09:00:16Z INFO service=payment worker=W3 processing job jobid=J1012 status=ok\n2026-07-24T09:00:17Z INFO service=payment worker=W4 processing job jobid=J1013 status=ok\n2026-07-24T09:00:18Z INFO service=payment worker=W5 processing job jobid=J1014 status=ok\n2026-07-24T09:00:19Z INFO service=payment worker=W6 processing job jobid=J1015 status=ok\n2026-07-24T09:00:20Z ERROR service=payment job aborted jobid=J1020 reason=INSUFFICIENT_FUNDS\n2026-07-24T09:00:21Z INFO service=payment worker=W1 processing job jobid=J1021 status=ok\n2026-07-24T09:00:22Z INFO service=payment worker=W2 processing job jobid=J1022 status=ok\n2026-07-24T09:00:23Z INFO service=payment worker=W3 processing job jobid=J1023 status=ok\n2026-07-24T09:00:24Z INFO service=payment worker=W4 processing job jobid=J1024 status=ok\n2026-07-24T09:00:25Z INFO service=payment worker=W5 processing job jobid=J1025 status=ok\n2026-07-24T09:00:26Z INFO service=payment worker=W6 processing job jobid=J1026 status=ok\n2026-07-24T09:00:27Z INFO service=payment worker=W1 processing job jobid=J1027 status=ok\n2026-07-24T09:00:28Z INFO service=payment worker=W2 processing job jobid=J1028 status=ok\n2026-07-24T09:00:29Z INFO service=payment worker=W3 processing job jobid=J1029 status=ok\n2026-07-24T09:00:30Z INFO service=payment worker=W4 processing job jobid=J1030 status=ok\n2026-07-24T09:00:31Z INFO service=payment worker=W5 processing job jobid=J1031 status=ok\n2026-07-24T09:00:32Z WARN service=payment retry scheduled jobid=J1032 attempt=2\n2026-07-24T09:00:33Z ERROR service=payment job rejected jobid=J1033 reason=CVV_MISMATCH\n2026-07-24T09:00:34Z INFO service=payment worker=W1 processing job jobid=J1034 status=ok\n2026-07-24T09:00:35Z INFO service=payment worker=W2 processing job jobid=J1035 status=ok\n2026-07-24T09:00:36Z INFO service=payment worker=W3 processing job jobid=J1036 status=ok\n2026-07-24T09:00:37Z INFO service=payment worker=W4 processing job jobid=J1037 status=ok\n2026-07-24T09:00:38Z INFO service=payment worker=W5 processing job jobid=J1038 status=ok\n2026-07-24T09:00:39Z INFO service=payment worker=W6 processing job jobid=J1039 status=ok\n2026-07-24T09:00:59Z INFO service=payment shutting down region=us-west-2\n", + "query": "why was job J1020 aborted?", + "gold_answer": "INSUFFICIENT_FUNDS", + "critical_atoms": [ + "txn-8f21ab9c" + ], + "notes": "Long payment worker-pool log (query terms 'aborted'/'J1020' appear verbatim on the gold line, so this is the lexical case). recency and frequency are challenged: the gold ERROR line sits mid-file with a dozen near-identical 'processing job ... status=ok' lines and two other decoy ERROR lines (CARD_EXPIRED, CVV_MISMATCH) after it, so recency fills the tight budget with later boilerplate/decoys instead; every line carries its own unique job id and timestamp suffix so raw term frequency does not spike on the gold line either. The audit id is on its own force-kept line, separate from the gold line." +} diff --git a/eval/tasks/v04/log_qa_011.json b/eval/tasks/v04/log_qa_011.json new file mode 100644 index 0000000..aa30ee4 --- /dev/null +++ b/eval/tasks/v04/log_qa_011.json @@ -0,0 +1,12 @@ +{ + "id": "log_qa_011", + "family": "log_qa", + "tier": "A", + "source": "2026-07-24T02:00:00Z INFO service=ingest pipeline run started run_id=run-4477c1\n2026-07-24T02:00:01Z INFO service=ingest config loaded pipeline=nightly-etl\n2026-07-24T02:00:04Z INFO service=ingest batch chunk=100 rows=4100 status=ok\n2026-07-24T02:00:07Z INFO service=ingest batch chunk=101 rows=4101 status=ok\n2026-07-24T02:00:10Z INFO service=ingest batch chunk=102 rows=4102 status=ok\n2026-07-24T02:00:13Z INFO service=ingest batch chunk=103 rows=4103 status=ok\n2026-07-24T02:00:16Z INFO service=ingest batch chunk=104 rows=4104 status=ok\n2026-07-24T02:00:19Z INFO service=ingest batch chunk=105 rows=4105 status=ok\n2026-07-24T02:00:22Z INFO service=ingest batch chunk=106 rows=4106 status=ok\n2026-07-24T02:00:25Z INFO service=ingest batch chunk=107 rows=4107 status=ok\n2026-07-24T02:00:28Z INFO service=ingest batch chunk=108 rows=4108 status=ok\n2026-07-24T02:00:31Z INFO service=ingest batch chunk=109 rows=4109 status=ok\n2026-07-24T02:00:34Z INFO service=ingest batch chunk=110 status=completing rows=48210\n2026-07-24T02:00:37Z INFO service=ingest batch chunk=111 rows=4111 status=ok\n2026-07-24T02:00:40Z INFO service=ingest batch chunk=112 rows=4112 status=ok\n2026-07-24T02:00:43Z INFO service=ingest batch chunk=113 rows=4113 status=ok\n2026-07-24T02:00:46Z INFO service=ingest batch chunk=114 rows=4114 status=ok\n2026-07-24T02:00:49Z INFO service=ingest batch chunk=115 rows=4115 status=ok\n2026-07-24T02:00:52Z INFO service=ingest batch chunk=116 rows=4116 status=ok\n2026-07-24T02:00:55Z INFO service=ingest batch chunk=117 rows=4117 status=ok\n2026-07-24T02:00:58Z INFO service=ingest batch chunk=118 rows=4118 status=ok\n2026-07-24T02:01:03Z WARN service=ingest batch deferred pipeline=nightly-etl flag=RATE_LIMITED\n2026-07-24T02:01:06Z INFO service=ingest batch chunk=119 rows=4119 status=ok\n2026-07-24T02:01:09Z INFO service=ingest batch chunk=120 rows=4120 status=ok\n2026-07-24T02:01:12Z INFO service=ingest batch chunk=121 rows=4121 status=ok\n2026-07-24T02:01:15Z INFO service=ingest batch chunk=122 rows=4122 status=ok\n2026-07-24T02:01:18Z INFO service=ingest batch chunk=123 rows=4123 status=ok\n2026-07-24T02:01:21Z INFO service=ingest batch chunk=124 rows=4124 status=ok\n2026-07-24T02:01:24Z INFO service=ingest batch chunk=125 rows=4125 status=ok\n2026-07-24T02:01:27Z INFO service=ingest batch chunk=126 rows=4126 status=ok\n2026-07-24T02:01:30Z INFO service=ingest batch chunk=127 rows=4127 status=ok\n2026-07-24T02:01:33Z INFO service=ingest batch chunk=128 rows=4128 status=ok\n2026-07-24T02:01:36Z INFO service=ingest batch chunk=129 rows=4129 status=ok\n2026-07-24T02:01:39Z INFO service=ingest batch chunk=130 rows=4130 status=ok\n2026-07-24T02:01:49Z INFO service=ingest checkpoint saved offset=982104\n2026-07-24T02:01:54Z INFO service=ingest pipeline run finished pipeline=nightly-etl\n", + "query": "what condition prevented the nightly batch from completing on schedule?", + "gold_answer": "RATE_LIMITED", + "critical_atoms": [ + "run-4477c1" + ], + "notes": "Semantic case: the query's wording ('prevented', 'completing', 'schedule') does not appear on the gold WARN line at all, while an early decoy INFO line literally contains the word 'completing' (a false lexical match) — this baits bm25 toward the wrong line at a tight budget. recency is also challenged: the gold line sits mid-file with a dozen near-identical 'batch chunk=... status=ok' lines both before and after it. The pipeline run id is on its own force-kept startup line, separate from the gold line." +} diff --git a/eval/tasks/v04/log_qa_012.json b/eval/tasks/v04/log_qa_012.json new file mode 100644 index 0000000..94dc712 --- /dev/null +++ b/eval/tasks/v04/log_qa_012.json @@ -0,0 +1,12 @@ +{ + "id": "log_qa_012", + "family": "log_qa", + "tier": "A", + "source": "2026-07-24T14:00:00Z INFO service=scheduler starting up cluster=batch-a\n2026-07-24T14:00:01Z INFO service=scheduler manifest loaded manifest=manifest-a13f9e0\n2026-07-24T14:00:02Z INFO service=scheduler job completed jobid=B2270 exit_code=0\n2026-07-24T14:00:03Z INFO service=scheduler job completed jobid=B2271 exit_code=0\n2026-07-24T14:00:04Z INFO service=scheduler job completed jobid=B2272 exit_code=0\n2026-07-24T14:00:05Z INFO service=scheduler job completed jobid=B2273 exit_code=0\n2026-07-24T14:00:06Z INFO service=scheduler job completed jobid=B2274 exit_code=0\n2026-07-24T14:00:07Z INFO service=scheduler job completed jobid=B2275 exit_code=0\n2026-07-24T14:00:08Z INFO service=scheduler job completed jobid=B2276 exit_code=0\n2026-07-24T14:00:09Z INFO service=scheduler job completed jobid=B2277 exit_code=0\n2026-07-24T14:00:10Z INFO service=scheduler job completed jobid=B2278 exit_code=0\n2026-07-24T14:00:11Z ERROR service=scheduler job terminated jobid=B2279 exit_code=1 reason=CONFIG_INVALID\n2026-07-24T14:00:12Z INFO service=scheduler job completed jobid=B2280 exit_code=0\n2026-07-24T14:00:13Z INFO service=scheduler job completed jobid=B2281 exit_code=0\n2026-07-24T14:00:14Z INFO service=scheduler job completed jobid=B2282 exit_code=0\n2026-07-24T14:00:15Z INFO service=scheduler job completed jobid=B2283 exit_code=0\n2026-07-24T14:00:16Z INFO service=scheduler job completed jobid=B2284 exit_code=0\n2026-07-24T14:00:17Z INFO service=scheduler job completed jobid=B2285 exit_code=0\n2026-07-24T14:00:18Z ERROR service=scheduler job terminated jobid=B2291 exit_code=137 reason=OOM_KILLED\n2026-07-24T14:00:19Z INFO service=scheduler job completed jobid=B2292 exit_code=0\n2026-07-24T14:00:20Z INFO service=scheduler job completed jobid=B2293 exit_code=0\n2026-07-24T14:00:21Z INFO service=scheduler job completed jobid=B2294 exit_code=0\n2026-07-24T14:00:22Z INFO service=scheduler job completed jobid=B2295 exit_code=0\n2026-07-24T14:00:23Z INFO service=scheduler job completed jobid=B2296 exit_code=0\n2026-07-24T14:00:24Z INFO service=scheduler job completed jobid=B2297 exit_code=0\n2026-07-24T14:00:25Z INFO service=scheduler job completed jobid=B2298 exit_code=0\n2026-07-24T14:00:26Z INFO service=scheduler job completed jobid=B2299 exit_code=0\n2026-07-24T14:00:27Z INFO service=scheduler job completed jobid=B2300 exit_code=0\n2026-07-24T14:00:28Z INFO service=scheduler job completed jobid=B2301 exit_code=0\n2026-07-24T14:00:29Z INFO service=scheduler job completed jobid=B2302 exit_code=0\n2026-07-24T14:00:30Z ERROR service=scheduler job terminated jobid=B2303 exit_code=143 reason=SIGTERM\n2026-07-24T14:00:31Z INFO service=scheduler job completed jobid=B2304 exit_code=0\n2026-07-24T14:00:32Z INFO service=scheduler job completed jobid=B2305 exit_code=0\n2026-07-24T14:00:33Z INFO service=scheduler job completed jobid=B2306 exit_code=0\n2026-07-24T14:00:34Z INFO service=scheduler job completed jobid=B2307 exit_code=0\n2026-07-24T14:00:35Z INFO service=scheduler job completed jobid=B2308 exit_code=0\n2026-07-24T14:00:36Z INFO service=scheduler job completed jobid=B2309 exit_code=0\n2026-07-24T14:00:37Z INFO service=scheduler job completed jobid=B2310 exit_code=0\n2026-07-24T14:00:57Z INFO service=scheduler cluster idle cluster=batch-a\n", + "query": "what exit code did job B2291 return when it was OOM killed?", + "gold_answer": "137", + "critical_atoms": [ + "manifest-a13f9e0" + ], + "notes": "Lexical case: query terms 'B2291' and 'OOM killed' appear verbatim on the gold ERROR line, but the exit code itself sits among ~30 near-identical 'job completed exit_code=0' lines plus two other terminated-job decoys (exit_code=1, exit_code=143) that also look like plausible answers to a shallow 'find the ERROR line' or recency heuristic; the SIGTERM decoy is placed after the gold line so recency prefers it instead. Every job id is unique per line so raw term frequency stays flat across the boilerplate. The manifest hash is on its own force-kept startup line." +} diff --git a/eval/tasks/v04/log_qa_013.json b/eval/tasks/v04/log_qa_013.json new file mode 100644 index 0000000..40235e0 --- /dev/null +++ b/eval/tasks/v04/log_qa_013.json @@ -0,0 +1,12 @@ +{ + "id": "log_qa_013", + "family": "log_qa", + "tier": "A", + "source": "2026-07-24T18:00:00Z INFO service=gateway starting up incident_ticket=INC-58213\n2026-07-24T18:00:01Z INFO service=gateway routing table loaded nodes=6\n2026-07-24T18:00:04Z INFO service=gateway health probe ok node=node-alpha-01 latency_ms=20\n2026-07-24T18:00:07Z INFO service=gateway health probe ok node=node-beta-02 latency_ms=21\n2026-07-24T18:00:10Z INFO service=gateway health probe ok node=node-delta-05 latency_ms=22\n2026-07-24T18:00:13Z INFO service=gateway health probe ok node=node-epsilon-06 latency_ms=23\n2026-07-24T18:00:16Z INFO service=gateway health probe ok node=node-alpha-01 latency_ms=24\n2026-07-24T18:00:19Z INFO service=gateway health probe ok node=node-beta-02 latency_ms=25\n2026-07-24T18:00:22Z INFO service=gateway health probe ok node=node-delta-05 latency_ms=26\n2026-07-24T18:00:25Z INFO service=gateway health probe ok node=node-epsilon-06 latency_ms=27\n2026-07-24T18:00:28Z INFO service=gateway health probe ok node=node-alpha-01 latency_ms=28\n2026-07-24T18:00:31Z INFO service=gateway health probe ok node=node-beta-02 latency_ms=29\n2026-07-24T18:00:36Z INFO service=gateway health probe responding latency_ms=42 node=node-beta-02\n2026-07-24T18:00:39Z INFO service=gateway health probe ok node=node-alpha-01 latency_ms=18\n2026-07-24T18:00:42Z INFO service=gateway health probe ok node=node-beta-02 latency_ms=19\n2026-07-24T18:00:45Z INFO service=gateway health probe ok node=node-delta-05 latency_ms=20\n2026-07-24T18:00:48Z INFO service=gateway health probe ok node=node-epsilon-06 latency_ms=21\n2026-07-24T18:00:51Z INFO service=gateway health probe ok node=node-alpha-01 latency_ms=22\n2026-07-24T18:00:54Z INFO service=gateway health probe ok node=node-beta-02 latency_ms=23\n2026-07-24T18:00:57Z INFO service=gateway health probe ok node=node-delta-05 latency_ms=24\n2026-07-24T18:01:00Z INFO service=gateway health probe ok node=node-epsilon-06 latency_ms=25\n2026-07-24T18:01:06Z WARN service=gateway upstream failed liveness handoff=node-gamma-07\n2026-07-24T18:01:09Z INFO service=gateway health probe ok node=node-alpha-01 latency_ms=22\n2026-07-24T18:01:12Z INFO service=gateway health probe ok node=node-beta-02 latency_ms=23\n2026-07-24T18:01:15Z INFO service=gateway health probe ok node=node-delta-05 latency_ms=24\n2026-07-24T18:01:18Z INFO service=gateway health probe ok node=node-epsilon-06 latency_ms=25\n2026-07-24T18:01:21Z INFO service=gateway health probe ok node=node-alpha-01 latency_ms=26\n2026-07-24T18:01:24Z INFO service=gateway health probe ok node=node-beta-02 latency_ms=27\n2026-07-24T18:01:27Z INFO service=gateway health probe ok node=node-delta-05 latency_ms=28\n2026-07-24T18:01:30Z INFO service=gateway health probe ok node=node-epsilon-06 latency_ms=29\n2026-07-24T18:01:33Z INFO service=gateway health probe ok node=node-alpha-01 latency_ms=30\n2026-07-24T18:01:48Z INFO service=gateway routing table stable nodes=6\n", + "query": "after the primary instance stopped responding, where did requests get routed?", + "gold_answer": "node-gamma-07", + "critical_atoms": [ + "INC-58213" + ], + "notes": "Semantic case: the query is phrased around 'stopped responding' / 'routed' but the gold WARN line uses 'upstream failed liveness' / 'handoff' instead, so lexical overlap is near zero. A decoy INFO line literally contains the word 'responding', which baits bm25 toward the wrong (irrelevant) line at a tight budget. recency is also challenged: the gold line sits mid-file surrounded by a couple dozen near-identical 'health probe ok node=...' lines both before and after it, each carrying its own node id, so the WARN line has to be found by more than raw position or line shape. The incident ticket id is on its own force-kept startup line." +} diff --git a/eval/tasks/v04/log_qa_014.json b/eval/tasks/v04/log_qa_014.json new file mode 100644 index 0000000..ac7a04e --- /dev/null +++ b/eval/tasks/v04/log_qa_014.json @@ -0,0 +1,12 @@ +{ + "id": "log_qa_014", + "family": "log_qa", + "tier": "A", + "source": "2026-07-24T05:00:00Z INFO service=cache starting up shard=shard-7\n2026-07-24T05:00:01Z INFO service=cache correlation id corr-55e21f9a assigned\n2026-07-24T05:00:02Z INFO service=cache checksum verified key=inventory:sku:10021 status=ok\n2026-07-24T05:00:03Z INFO service=cache checksum verified key=inventory:sku:44211 status=ok\n2026-07-24T05:00:04Z INFO service=cache checksum verified key=inventory:sku:55894 status=ok\n2026-07-24T05:00:05Z INFO service=cache checksum verified key=inventory:sku:61233 status=ok\n2026-07-24T05:00:06Z INFO service=cache checksum verified key=inventory:sku:77414 status=ok\n2026-07-24T05:00:07Z INFO service=cache checksum verified key=inventory:sku:88904 status=ok\n2026-07-24T05:00:08Z INFO service=cache checksum verified key=inventory:sku:10027 status=ok\n2026-07-24T05:00:09Z INFO service=cache checksum verified key=inventory:sku:44217 status=ok\n2026-07-24T05:00:10Z INFO service=cache checksum verified key=inventory:sku:55900 status=ok\n2026-07-24T05:00:11Z ERROR service=cache checksum mismatch key=inventory:sku:99120 expected=0x1122a3 actual=0x33440b\n2026-07-24T05:00:12Z INFO service=cache checksum verified key=inventory:sku:10121 status=ok\n2026-07-24T05:00:13Z INFO service=cache checksum verified key=inventory:sku:44311 status=ok\n2026-07-24T05:00:14Z INFO service=cache checksum verified key=inventory:sku:55994 status=ok\n2026-07-24T05:00:15Z INFO service=cache checksum verified key=inventory:sku:61333 status=ok\n2026-07-24T05:00:16Z INFO service=cache checksum verified key=inventory:sku:77514 status=ok\n2026-07-24T05:00:17Z INFO service=cache checksum verified key=inventory:sku:89004 status=ok\n2026-07-24T05:00:18Z INFO service=cache checksum verified key=inventory:sku:10127 status=ok\n2026-07-24T05:00:19Z ERROR service=cache checksum mismatch key=inventory:sku:88421 expected=0x9f2c3d actual=0xbad9f2c1\n2026-07-24T05:00:20Z INFO service=cache checksum verified key=inventory:sku:10221 status=ok\n2026-07-24T05:00:21Z INFO service=cache checksum verified key=inventory:sku:44411 status=ok\n2026-07-24T05:00:22Z INFO service=cache checksum verified key=inventory:sku:56094 status=ok\n2026-07-24T05:00:23Z INFO service=cache checksum verified key=inventory:sku:61433 status=ok\n2026-07-24T05:00:24Z INFO service=cache checksum verified key=inventory:sku:77614 status=ok\n2026-07-24T05:00:25Z INFO service=cache checksum verified key=inventory:sku:89104 status=ok\n2026-07-24T05:00:26Z INFO service=cache checksum verified key=inventory:sku:10227 status=ok\n2026-07-24T05:00:27Z INFO service=cache checksum verified key=inventory:sku:44417 status=ok\n2026-07-24T05:00:28Z INFO service=cache checksum verified key=inventory:sku:56100 status=ok\n2026-07-24T05:00:29Z INFO service=cache checksum verified key=inventory:sku:61439 status=ok\n2026-07-24T05:00:30Z INFO service=cache checksum verified key=inventory:sku:77620 status=ok\n2026-07-24T05:00:31Z INFO service=cache checksum verified key=inventory:sku:89110 status=ok\n2026-07-24T05:00:32Z ERROR service=cache checksum mismatch key=inventory:sku:31005 expected=0x0abc11 actual=0x0abc12\n2026-07-24T05:00:33Z INFO service=cache checksum verified key=inventory:sku:10321 status=ok\n2026-07-24T05:00:34Z INFO service=cache checksum verified key=inventory:sku:44511 status=ok\n2026-07-24T05:00:35Z INFO service=cache checksum verified key=inventory:sku:56194 status=ok\n2026-07-24T05:00:36Z INFO service=cache checksum verified key=inventory:sku:61533 status=ok\n2026-07-24T05:00:37Z INFO service=cache checksum verified key=inventory:sku:77714 status=ok\n2026-07-24T05:00:38Z INFO service=cache checksum verified key=inventory:sku:89204 status=ok\n2026-07-24T05:00:58Z INFO service=cache shard stable shard=shard-7\n", + "query": "what actual checksum value was recorded for key inventory:sku:88421?", + "gold_answer": "0xbad9f2c1", + "critical_atoms": [ + "corr-55e21f9a" + ], + "notes": "Lexical case: query terms 'actual checksum' and 'inventory:sku:88421' appear verbatim on the gold ERROR line, but it sits among ~30 near-identical 'checksum verified ... status=ok' lines plus two other checksum-mismatch decoys (different keys/values) that share the same 'expected=.../actual=...' shape and would rank equally under recency. Every line carries a distinct key/value pair, so no single line gets an unearned rarity advantage over the others. The correlation id is on its own force-kept startup line, separate from the gold line." +} diff --git a/eval/tasks/v04/log_qa_015.json b/eval/tasks/v04/log_qa_015.json new file mode 100644 index 0000000..d094281 --- /dev/null +++ b/eval/tasks/v04/log_qa_015.json @@ -0,0 +1,12 @@ +{ + "id": "log_qa_015", + "family": "log_qa", + "tier": "A", + "source": "2026-07-24T11:00:00Z INFO service=notify starting up dedupe_key=dedupe-3f88c2\n2026-07-24T11:00:01Z INFO service=notify config loaded channel=push\n2026-07-24T11:00:02Z INFO service=notify batch sent count=120 channel=push status=ok\n2026-07-24T11:00:03Z INFO service=notify batch sent count=121 channel=push status=ok\n2026-07-24T11:00:04Z INFO service=notify batch sent count=122 channel=push status=ok\n2026-07-24T11:00:05Z INFO service=notify batch sent count=123 channel=push status=ok\n2026-07-24T11:00:06Z INFO service=notify batch sent count=124 channel=push status=ok\n2026-07-24T11:00:07Z INFO service=notify batch sent count=125 channel=push status=ok\n2026-07-24T11:00:08Z INFO service=notify batch sent count=126 channel=push status=ok\n2026-07-24T11:00:09Z INFO service=notify batch sent count=127 channel=push status=ok\n2026-07-24T11:00:10Z INFO service=notify batch sent count=128 channel=push status=ok\n2026-07-24T11:00:11Z INFO service=notify batch sent count=129 channel=push status=ok\n2026-07-24T11:00:16Z INFO service=notify alerts dispatched count=3 channel=pagerduty\n2026-07-24T11:00:19Z INFO service=notify batch sent count=130 channel=push status=ok\n2026-07-24T11:00:22Z INFO service=notify batch sent count=131 channel=push status=ok\n2026-07-24T11:00:25Z INFO service=notify batch sent count=132 channel=push status=ok\n2026-07-24T11:00:28Z INFO service=notify batch sent count=133 channel=push status=ok\n2026-07-24T11:00:31Z INFO service=notify batch sent count=134 channel=push status=ok\n2026-07-24T11:00:34Z INFO service=notify batch sent count=135 channel=push status=ok\n2026-07-24T11:00:37Z INFO service=notify batch sent count=136 channel=push status=ok\n2026-07-24T11:00:40Z INFO service=notify batch sent count=137 channel=push status=ok\n2026-07-24T11:00:43Z INFO service=notify batch sent count=138 channel=push status=ok\n2026-07-24T11:00:51Z WARN service=notify backlog_depth=48210 queue=push-retry\n2026-07-24T11:00:54Z INFO service=notify batch sent count=139 channel=push status=ok\n2026-07-24T11:00:57Z INFO service=notify batch sent count=140 channel=push status=ok\n2026-07-24T11:01:00Z INFO service=notify batch sent count=141 channel=push status=ok\n2026-07-24T11:01:03Z INFO service=notify batch sent count=142 channel=push status=ok\n2026-07-24T11:01:06Z INFO service=notify batch sent count=143 channel=push status=ok\n2026-07-24T11:01:09Z INFO service=notify batch sent count=144 channel=push status=ok\n2026-07-24T11:01:12Z INFO service=notify batch sent count=145 channel=push status=ok\n2026-07-24T11:01:15Z INFO service=notify batch sent count=146 channel=push status=ok\n2026-07-24T11:01:18Z INFO service=notify batch sent count=147 channel=push status=ok\n2026-07-24T11:01:21Z INFO service=notify batch sent count=148 channel=push status=ok\n2026-07-24T11:01:24Z INFO service=notify batch sent count=149 channel=push status=ok\n2026-07-24T11:01:27Z INFO service=notify batch sent count=150 channel=push status=ok\n2026-07-24T11:01:47Z INFO service=notify queue drained queue=push-retry\n", + "query": "how deep had the backlog grown before the alerts fired?", + "gold_answer": "48210", + "critical_atoms": [ + "dedupe-3f88c2" + ], + "notes": "Semantic case: the query's wording ('backlog', 'grown', 'alerts fired') does not appear on the gold WARN line at all (it reads 'backlog_depth=48210', a single underscored token distinct from 'backlog'), while an early decoy INFO line literally contains the word 'alerts', baiting bm25 toward the wrong line at a tight budget. recency is also challenged: the gold line sits mid-file among ~30 near-identical 'batch sent count=... channel=push status=ok' lines both before and after it, each with its own count value so no single boilerplate line stands out by position alone. The dedupe key is on its own force-kept startup line, separate from the gold line." +} diff --git a/eval/tasks/v04/long_context_needle_001.json b/eval/tasks/v04/long_context_needle_001.json new file mode 100644 index 0000000..c242e44 --- /dev/null +++ b/eval/tasks/v04/long_context_needle_001.json @@ -0,0 +1,13 @@ +{ + "id": "long_context_needle_001", + "family": "long_context_needle", + "tier": "A", + "source": "# deploy-runbook: payments-service release pipeline notes\nservice: payments-api\nregion: us-west-2\nreplicas: 6\ncanary_percent: 10\nhealth_check_path: /healthz\ntimeout_seconds: 30\nOn-call opened incident INC-4471 when p99 latency spiked during the rollout.\nRollback stays gated behind the signed manifest /etc/deploy/ring.yaml on bastion-3.\n2026-07-02T14:21:55Z INFO scheduler drained connections from the old replica set\n2026-07-02T14:22:03Z INFO probe reported green across all six pods in us-west-2\n2026-07-02T14:22:09Z INFO deploy promoted artifact digest sha256:9c1f4e2b to the production ring\n2026-07-02T14:22:15Z INFO scheduler scaled the canary pool back to zero after cutover\n2026-07-02T14:22:40Z DEBUG metrics exporter flushed 1284 samples to the collector\nThe release captain verified dashboards looked nominal before closing the change window.\nA follow-up chore is filed to rotate the staging credentials before the next quarterly audit.\nMost pipeline runs are uneventful and simply repeat the same green health probes each minute.\nNothing else in this window required a page or a manual intervention from the operators.\n", + "query": "which artifact digest was promoted to the production ring?", + "gold_answer": "sha256:9c1f4e2b", + "critical_atoms": [ + "INC-4471", + "/etc/deploy/ring.yaml" + ], + "notes": "Needle (artifact digest on a mid-context log line) shares 'artifact/digest/promoted/production/ring' with the query so bm25 keeps it, but it is neither a critical atom nor a late line, so recency/forced_only drop it at 0.25; the incident id and manifest path are force-kept on separate lines." +} diff --git a/eval/tasks/v04/long_context_needle_010.json b/eval/tasks/v04/long_context_needle_010.json new file mode 100644 index 0000000..b093957 --- /dev/null +++ b/eval/tasks/v04/long_context_needle_010.json @@ -0,0 +1,13 @@ +{ + "id": "long_context_needle_010", + "family": "long_context_needle", + "tier": "A", + "source": "# config-audit: quarterly feature-flag rollout review\naudit_cycle: 2026-Q3\nreviewer_board: platform-config-review\nAudit ticket AUD-8823 tracks this rollout review cycle end to end.\nflag=checkout-retry-v2 env=us-east-1 rollout_percent=5 reviewer=wei status=rejected\nflag=cart-price-cache env=us-east-1 rollout_percent=53 reviewer=wei status=paused\nflag=shipping-eta-v3 env=us-east-1 rollout_percent=29 reviewer=wei status=approved\nflag=loyalty-badge-ui env=us-east-1 rollout_percent=41 reviewer=wei status=paused\nflag=search-rerank-b env=us-east-1 rollout_percent=35 reviewer=priya status=rejected\nflag=refund-fastpath env=us-east-1 rollout_percent=83 reviewer=priya status=rejected\nflag=gift-card-split env=us-east-1 rollout_percent=59 reviewer=priya status=rejected\nflag=address-autofill env=us-east-1 rollout_percent=71 reviewer=priya status=approved\nflag=cart-price-cache env=eu-central-1 rollout_percent=23 reviewer=leila status=approved\nflag=shipping-eta-v3 env=eu-central-1 rollout_percent=89 reviewer=tomas status=approved\nflag=loyalty-badge-ui env=eu-central-1 rollout_percent=11 reviewer=leila status=approved\nflag=search-rerank-b env=eu-central-1 rollout_percent=5 reviewer=tomas status=approved\nflag=refund-fastpath env=eu-central-1 rollout_percent=53 reviewer=tomas status=approved\nflag=checkout-retry-v2 env=eu-central-1 rollout_percent=37 reviewer=priya status=approved\nflag=gift-card-split env=eu-central-1 rollout_percent=29 reviewer=tomas status=approved\nflag=address-autofill env=eu-central-1 rollout_percent=41 reviewer=tomas status=paused\nflag=checkout-retry-v2 env=ap-south-1 rollout_percent=35 reviewer=tomas status=approved\nflag=cart-price-cache env=ap-south-1 rollout_percent=83 reviewer=tomas status=approved\nflag=shipping-eta-v3 env=ap-south-1 rollout_percent=59 reviewer=tomas status=rejected\nflag=loyalty-badge-ui env=ap-south-1 rollout_percent=71 reviewer=tomas status=approved\nflag=search-rerank-b env=ap-south-1 rollout_percent=65 reviewer=leila status=approved\nflag=refund-fastpath env=ap-south-1 rollout_percent=23 reviewer=tomas status=rejected\nflag=gift-card-split env=ap-south-1 rollout_percent=89 reviewer=leila status=approved\nflag=address-autofill env=ap-south-1 rollout_percent=11 reviewer=tomas status=approved\nflag=checkout-retry-v2 env=sa-east-1 rollout_percent=5 reviewer=wei status=rejected\nflag=cart-price-cache env=sa-east-1 rollout_percent=53 reviewer=wei status=paused\nflag=shipping-eta-v3 env=sa-east-1 rollout_percent=29 reviewer=wei status=approved\nflag=loyalty-badge-ui env=sa-east-1 rollout_percent=41 reviewer=wei status=paused\nflag=search-rerank-b env=sa-east-1 rollout_percent=35 reviewer=priya status=rejected\nflag=refund-fastpath env=sa-east-1 rollout_percent=83 reviewer=priya status=rejected\nflag=gift-card-split env=sa-east-1 rollout_percent=59 reviewer=priya status=rejected\nflag=address-autofill env=sa-east-1 rollout_percent=71 reviewer=priya status=approved\nflag=checkout-retry-v2 env=us-west-2 rollout_percent=65 reviewer=wei status=rejected\nflag=cart-price-cache env=us-west-2 rollout_percent=23 reviewer=priya status=approved\nflag=shipping-eta-v3 env=us-west-2 rollout_percent=89 reviewer=wei status=approved\nflag=loyalty-badge-ui env=us-west-2 rollout_percent=11 reviewer=priya status=approved\nflag=search-rerank-b env=us-west-2 rollout_percent=5 reviewer=wei status=approved\nflag=refund-fastpath env=us-west-2 rollout_percent=53 reviewer=wei status=approved\nflag=gift-card-split env=us-west-2 rollout_percent=29 reviewer=wei status=approved\nflag=address-autofill env=us-west-2 rollout_percent=41 reviewer=wei status=paused\nSign-off for this cycle is recorded in the manifest at /etc/flags/manifest-19.lock.\nRows above reflect the last poll of the rollout-control-plane snapshot table.\nNo flag in this cycle was rolled back mid-review by an on-call engineer.\n", + "query": "what rollout_percent was approved for the checkout-retry-v2 flag in eu-central-1?", + "gold_answer": "rollout_percent=37", + "critical_atoms": [ + "AUD-8823", + "/etc/flags/manifest-19.lock" + ], + "notes": "The (flag, env) pair the query asks about recurs with a different rollout_percent for every other env of the same flag and every other flag in the same env across ~40 similarly formatted rows, so the needle row shares its own vocabulary query-wide; bm25 combines the flag-name and env tokens and keeps it at both a 25% and a 10% ceiling, but recency (the row sits well before the tail), frequency (every row carries its own equally 'rare' numeric value, so token rarity alone doesn't single it out), and llmlingua_style (uniformly formatted rows carry similar per-token surprisal) all drop it at the 10% ceiling, and forced_only never ranks past the floor. The audit ticket and manifest path are force-kept on separate lines." +} diff --git a/eval/tasks/v04/long_context_needle_011.json b/eval/tasks/v04/long_context_needle_011.json new file mode 100644 index 0000000..fb20ad6 --- /dev/null +++ b/eval/tasks/v04/long_context_needle_011.json @@ -0,0 +1,13 @@ +{ + "id": "long_context_needle_011", + "family": "long_context_needle", + "tier": "A", + "source": "# cache-fleet: eviction telemetry sample (5-minute window)\nfleet: web-cache-fleet\nsample_window: 2026-07-19T03:10:00Z..2026-07-19T03:15:00Z\nIncident ticket CACHE-5502 tracks this window's eviction-rate anomaly.\n2026-07-19T03:10:03Z TTL_EXPIRE node=cache-01 key_count=40 reclaimed_mb=2\n2026-07-19T03:11:06Z TTL_EXPIRE node=cache-02 key_count=57 reclaimed_mb=7\n2026-07-19T03:12:09Z TTL_EXPIRE node=cache-03 key_count=74 reclaimed_mb=12\n2026-07-19T03:13:12Z TTL_EXPIRE node=cache-04 key_count=91 reclaimed_mb=17\n2026-07-19T03:14:15Z TTL_EXPIRE node=cache-05 key_count=108 reclaimed_mb=22\n2026-07-19T03:10:18Z TTL_EXPIRE node=cache-06 key_count=125 reclaimed_mb=27\n2026-07-19T03:11:21Z TTL_EXPIRE node=cache-07 key_count=142 reclaimed_mb=32\n2026-07-19T03:12:24Z TTL_EXPIRE node=cache-08 key_count=159 reclaimed_mb=37\n2026-07-19T03:13:27Z TTL_EXPIRE node=cache-09 key_count=176 reclaimed_mb=2\n2026-07-19T03:14:30Z TTL_EXPIRE node=cache-10 key_count=193 reclaimed_mb=7\n2026-07-19T03:10:33Z TTL_EXPIRE node=cache-11 key_count=210 reclaimed_mb=12\n2026-07-19T03:11:36Z TTL_EXPIRE node=cache-12 key_count=227 reclaimed_mb=17\n2026-07-19T03:12:39Z TTL_EXPIRE node=cache-01 key_count=244 reclaimed_mb=22\n2026-07-19T03:13:42Z TTL_EXPIRE node=cache-02 key_count=261 reclaimed_mb=27\n2026-07-19T03:14:45Z TTL_EXPIRE node=cache-03 key_count=278 reclaimed_mb=32\n2026-07-19T03:10:48Z TTL_EXPIRE node=cache-04 key_count=295 reclaimed_mb=37\n2026-07-19T03:11:51Z TTL_EXPIRE node=cache-05 key_count=312 reclaimed_mb=2\n2026-07-19T03:12:54Z TTL_EXPIRE node=cache-06 key_count=329 reclaimed_mb=7\n2026-07-19T03:13:57Z TTL_EXPIRE node=cache-07 key_count=46 reclaimed_mb=12\n2026-07-19T03:14:00Z TTL_EXPIRE node=cache-08 key_count=63 reclaimed_mb=17\n2026-07-19T03:10:03Z TTL_EXPIRE node=cache-09 key_count=80 reclaimed_mb=22\n2026-07-19T03:11:06Z TTL_EXPIRE node=cache-10 key_count=97 reclaimed_mb=27\n2026-07-19T03:12:09Z TTL_EXPIRE node=cache-11 key_count=114 reclaimed_mb=32\n2026-07-19T03:13:12Z TTL_EXPIRE node=cache-12 key_count=131 reclaimed_mb=37\n2026-07-19T03:14:02Z OOM_EVICT node=cache-07 key_count=812 reclaimed_mb=0\n2026-07-19T03:14:15Z TTL_EXPIRE node=cache-01 key_count=148 reclaimed_mb=2\n2026-07-19T03:10:18Z TTL_EXPIRE node=cache-02 key_count=165 reclaimed_mb=7\n2026-07-19T03:11:21Z TTL_EXPIRE node=cache-03 key_count=182 reclaimed_mb=12\n2026-07-19T03:12:24Z TTL_EXPIRE node=cache-04 key_count=199 reclaimed_mb=17\n2026-07-19T03:13:27Z TTL_EXPIRE node=cache-05 key_count=216 reclaimed_mb=22\n2026-07-19T03:14:30Z TTL_EXPIRE node=cache-06 key_count=233 reclaimed_mb=27\n2026-07-19T03:10:33Z TTL_EXPIRE node=cache-07 key_count=250 reclaimed_mb=32\n2026-07-19T03:11:36Z TTL_EXPIRE node=cache-08 key_count=267 reclaimed_mb=37\n2026-07-19T03:12:39Z TTL_EXPIRE node=cache-09 key_count=284 reclaimed_mb=2\n2026-07-19T03:13:42Z TTL_EXPIRE node=cache-10 key_count=301 reclaimed_mb=7\n2026-07-19T03:14:45Z TTL_EXPIRE node=cache-11 key_count=318 reclaimed_mb=12\n2026-07-19T03:10:48Z TTL_EXPIRE node=cache-12 key_count=335 reclaimed_mb=17\n2026-07-19T03:11:51Z TTL_EXPIRE node=cache-01 key_count=52 reclaimed_mb=22\n2026-07-19T03:12:54Z TTL_EXPIRE node=cache-02 key_count=69 reclaimed_mb=27\n2026-07-19T03:13:57Z TTL_EXPIRE node=cache-03 key_count=86 reclaimed_mb=32\n2026-07-19T03:14:00Z TTL_EXPIRE node=cache-04 key_count=103 reclaimed_mb=37\nEviction policy config for this fleet lives at /var/cache/policy/evict-9.conf.\nMost TTL_EXPIRE events in this window are routine and self-clear within seconds.\nNo manual flush was issued against any node during this sample window.\n", + "query": "which cache node forced an eviction because it ran out of memory instead of a normal expiry?", + "gold_answer": "OOM_EVICT node=cache-07", + "critical_atoms": [ + "CACHE-5502", + "/var/cache/policy/evict-9.conf" + ], + "notes": "The needle is identified only by its OOM_EVICT event tag, never by the query's 'out of memory' wording, while the 'node'/'cache' tokens the query does share are present on all ~40 routine TTL_EXPIRE rows and so carry almost no discriminating weight; bm25 has no reliable query-term signal to prefer this row over the many other node=cache-NN rows and drops it at both a 25% and a 10% ceiling, and recency drops it too since it is not among the last rows. frequency and llmlingua_style do keep it (the OOM_EVICT tag is the one genuinely rare token in the document), showing that a rarity-aware selector, not a term-matching or position-based one, is what survives here. The incident ticket and policy path are force-kept on separate lines." +} diff --git a/eval/tasks/v04/long_context_needle_012.json b/eval/tasks/v04/long_context_needle_012.json new file mode 100644 index 0000000..6c212c7 --- /dev/null +++ b/eval/tasks/v04/long_context_needle_012.json @@ -0,0 +1,13 @@ +{ + "id": "long_context_needle_012", + "family": "long_context_needle", + "tier": "A", + "source": "# replica-topology: cluster role snapshot export\nexport_source: topology-coordinator\nChange ticket TOPO-6140 tracks this failover-review cycle.\ncluster=billing-db host=db-write-43a.internal role=primary election_term=43\ncluster=billing-db host=db-read-44a.internal role=replica election_term=44\ncluster=billing-db host=db-read-44b.internal role=replica election_term=44\ncluster=billing-db host=db-read-44c.internal role=replica election_term=44\ncluster=catalog-db host=db-write-47a.internal role=primary election_term=47\ncluster=catalog-db host=db-read-48a.internal role=replica election_term=48\ncluster=catalog-db host=db-read-48b.internal role=replica election_term=48\ncluster=catalog-db host=db-read-48c.internal role=replica election_term=48\ncluster=auth-db host=db-write-51a.internal role=primary election_term=51\ncluster=auth-db host=db-read-52a.internal role=replica election_term=52\ncluster=auth-db host=db-read-52b.internal role=replica election_term=52\ncluster=auth-db host=db-read-52c.internal role=replica election_term=52\ncluster=sessions-db host=db-write-55a.internal role=primary election_term=55\ncluster=sessions-db host=db-read-56a.internal role=replica election_term=56\ncluster=sessions-db host=db-read-56b.internal role=replica election_term=56\ncluster=sessions-db host=db-read-56c.internal role=replica election_term=56\ncluster=orders-db host=db-write-59a.internal role=primary election_term=59\ncluster=orders-db host=db-read-60a.internal role=replica election_term=60\ncluster=orders-db host=db-read-60b.internal role=replica election_term=60\ncluster=orders-db host=db-read-60c.internal role=replica election_term=60\ncluster=ledger-db host=db-write-63a.internal role=primary election_term=63\ncluster=ledger-db host=db-read-64a.internal role=replica election_term=64\ncluster=ledger-db host=db-read-64b.internal role=replica election_term=64\ncluster=ledger-db host=db-read-64c.internal role=replica election_term=64\ncluster=billing-db host=db-write-14.internal role=primary election_term=58\nCoordinator config and locks are recorded under /etc/topo/coordinator-3.state.\nOnly the highest election_term per cluster is authoritative for routing.\nReplica rows above are refreshed every heartbeat and are not authoritative for writes.\n", + "query": "which host is currently the primary write endpoint for the billing-db cluster?", + "gold_answer": "host=db-write-14.internal", + "critical_atoms": [ + "TOPO-6140", + "/etc/topo/coordinator-3.state" + ], + "notes": "An earlier, superseded role=primary row for the same billing-db cluster (a prior election term) sits before the current row and shares the exact same query-relevant tokens, so a term-overlap score cannot tell the two apart; at the 10% ceiling budget only has room for one of the pair, and every deterministic selector here (forced_only, recency, frequency, bm25, llmlingua_style) fails to surface the current row. At 25% the wider budget happens to include both tied rows so recency and bm25 pass, but frequency and llmlingua_style still miss it since every row also carries its own locally distinct host suffix and term number that dilutes rarity-based ranking. Change ticket and coordinator-state path are force-kept on separate lines." +} diff --git a/eval/tasks/v04/long_context_needle_013.json b/eval/tasks/v04/long_context_needle_013.json new file mode 100644 index 0000000..496ece8 --- /dev/null +++ b/eval/tasks/v04/long_context_needle_013.json @@ -0,0 +1,13 @@ +{ + "id": "long_context_needle_013", + "family": "long_context_needle", + "tier": "A", + "source": "# ci-timing: nightly test-shard duration report\npipeline: integration-suite-nightly\nrun_id: run-2026-07-24-03\nReport artifact is signed and stored at /var/ci/reports/shard-report-77.sig.\nBuild reference RUN-90142 identifies this nightly execution in the CI ledger.\nshard=01 duration_ms=4137 status=ok verdict=within_limits\nshard=02 duration_ms=4274 status=ok verdict=within_limits\nshard=03 duration_ms=4411 status=ok verdict=within_limits\nshard=04 duration_ms=4548 status=ok verdict=within_limits\nshard=05 duration_ms=4685 status=ok verdict=within_limits\nshard=06 duration_ms=4822 status=ok verdict=within_limits\nshard=07 duration_ms=4959 status=ok verdict=within_limits\nshard=08 duration_ms=5096 status=ok verdict=within_limits\nshard=09 duration_ms=5233 status=ok verdict=within_limits\nshard=10 duration_ms=5370 status=ok verdict=within_limits\nshard=11 duration_ms=5507 status=ok verdict=within_limits\nshard=12 duration_ms=5644 status=ok verdict=within_limits\nshard=13 duration_ms=5781 status=ok verdict=within_limits\nshard=14 duration_ms=5918 status=ok verdict=within_limits\nshard=15 duration_ms=6055 status=ok verdict=within_limits\nshard=16 duration_ms=6192 status=ok verdict=within_limits\nshard=18 duration_ms=6466 status=ok verdict=within_limits\nshard=19 duration_ms=6603 status=ok verdict=within_limits\nshard=20 duration_ms=6740 status=ok verdict=within_limits\nshard=21 duration_ms=6877 status=ok verdict=within_limits\nshard=22 duration_ms=7014 status=ok verdict=within_limits\nshard=23 duration_ms=7151 status=ok verdict=within_limits\nshard=24 duration_ms=7288 status=ok verdict=within_limits\nshard=25 duration_ms=7425 status=ok verdict=within_limits\nshard=26 duration_ms=7562 status=ok verdict=within_limits\nshard=27 duration_ms=7699 status=ok verdict=within_limits\nshard=28 duration_ms=7836 status=ok verdict=within_limits\nshard=29 duration_ms=7973 status=ok verdict=within_limits\nshard=30 duration_ms=8110 status=ok verdict=within_limits\nshard=17 duration_ms=13480 status=ok verdict=BUDGET_EXCEEDED\nshard=31 duration_ms=8247 status=ok verdict=within_limits\nshard=32 duration_ms=8384 status=ok verdict=within_limits\nshard=33 duration_ms=8521 status=ok verdict=within_limits\nshard=34 duration_ms=8658 status=ok verdict=within_limits\nshard=35 duration_ms=8795 status=ok verdict=within_limits\nshard=36 duration_ms=8932 status=ok verdict=within_limits\nshard=37 duration_ms=9069 status=ok verdict=within_limits\nshard=38 duration_ms=9206 status=ok verdict=within_limits\nShard duration limits are configured centrally and reviewed each quarter.\nMost shards finish well inside their allotted window without any flags.\n", + "query": "which test shard actually blew through its allotted time budget last night?", + "gold_answer": "shard=17", + "critical_atoms": [ + "RUN-90142", + "/var/ci/reports/shard-report-77.sig" + ], + "notes": "The needle is identified only by its BUDGET_EXCEEDED verdict tag, never by the query's 'blew through'/'time budget' wording; the tokens the query does share ('shard', 'duration_ms') are present on all ~44 rows, so bm25 has no discriminating signal and drops it at both a 25% and a 10% ceiling, and recency drops it too since the row is not near the tail. At the 10% ceiling frequency and llmlingua_style also drop it, since every row's own duration_ms value looks locally unique and dilutes rarity-based ranking; only at 25% do they have enough budget headroom to include it despite that dilution. Build reference and report-signature path are force-kept on separate lines." +} diff --git a/eval/tasks/v04/long_context_needle_014.json b/eval/tasks/v04/long_context_needle_014.json new file mode 100644 index 0000000..7a00d14 --- /dev/null +++ b/eval/tasks/v04/long_context_needle_014.json @@ -0,0 +1,12 @@ +{ + "id": "long_context_needle_014", + "family": "long_context_needle", + "tier": "A", + "source": "# dependency-audit: pinned version drift report\naudit_tool: pin-drift-scanner v4\nWaiver record WAIV-3391 covers any intentional pin exceptions in this report.\nlockfile=gateway-svc package=libcrypt-shim pinned_version=5.10.2\nlockfile=gateway-svc package=http-multiplex pinned_version=2.1.17\nlockfile=gateway-svc package=json-fastpath pinned_version=3.8.12\nlockfile=gateway-svc package=retry-backoff pinned_version=2.7.7\nlockfile=gateway-svc package=trace-context pinned_version=3.2.2\nlockfile=gateway-svc package=queue-fanout pinned_version=6.5.17\nlockfile=gateway-svc package=config-loader pinned_version=1.0.12\nlockfile=gateway-svc package=sig-verify pinned_version=4.3.7\nlockfile=billing-svc package=libcrypt-shim pinned_version=5.10.2\nlockfile=billing-svc package=http-multiplex pinned_version=2.1.17\nlockfile=billing-svc package=json-fastpath pinned_version=3.8.12\nlockfile=billing-svc package=trace-context pinned_version=3.2.2\nlockfile=billing-svc package=queue-fanout pinned_version=6.5.17\nlockfile=billing-svc package=config-loader pinned_version=1.0.12\nlockfile=billing-svc package=retry-backoff pinned_version=2.9.4\nlockfile=billing-svc package=sig-verify pinned_version=4.3.7\nlockfile=search-svc package=libcrypt-shim pinned_version=5.10.2\nlockfile=search-svc package=http-multiplex pinned_version=2.1.17\nlockfile=search-svc package=json-fastpath pinned_version=3.8.12\nlockfile=search-svc package=retry-backoff pinned_version=2.7.7\nlockfile=search-svc package=trace-context pinned_version=3.2.2\nlockfile=search-svc package=queue-fanout pinned_version=6.5.17\nlockfile=search-svc package=config-loader pinned_version=1.0.12\nlockfile=search-svc package=sig-verify pinned_version=4.3.7\nlockfile=notify-svc package=libcrypt-shim pinned_version=5.10.2\nlockfile=notify-svc package=http-multiplex pinned_version=2.1.17\nlockfile=notify-svc package=json-fastpath pinned_version=3.8.12\nlockfile=notify-svc package=retry-backoff pinned_version=2.7.7\nlockfile=notify-svc package=trace-context pinned_version=3.2.2\nlockfile=notify-svc package=queue-fanout pinned_version=6.5.17\nlockfile=notify-svc package=config-loader pinned_version=1.0.12\nlockfile=notify-svc package=sig-verify pinned_version=4.3.7\nlockfile=worker-svc package=libcrypt-shim pinned_version=5.10.2\nlockfile=worker-svc package=http-multiplex pinned_version=2.1.17\nlockfile=worker-svc package=json-fastpath pinned_version=3.8.12\nlockfile=worker-svc package=retry-backoff pinned_version=2.7.7\nlockfile=worker-svc package=trace-context pinned_version=3.2.2\nlockfile=worker-svc package=queue-fanout pinned_version=6.5.17\nlockfile=worker-svc package=config-loader pinned_version=1.0.12\nlockfile=worker-svc package=sig-verify pinned_version=4.3.7\nPin exceptions must reference an active waiver before merge, per policy.\nThis report reflects the lockfile state as of the last scheduled scan.\nPackages without an explicit pin inherit the workspace default constraint.\n", + "query": "what version is retry-backoff pinned to in the billing-svc lockfile?", + "gold_answer": "pinned_version=2.9.4", + "critical_atoms": [ + "WAIV-3391" + ], + "notes": "The (lockfile, package) pair the query asks about recurs with a different pinned_version for every other lockfile pinning the same package and every other package in the same lockfile across ~39 rows, so the needle row shares its own vocabulary query-wide; bm25 combines the lockfile and package tokens and keeps it at both a 25% and a 10% ceiling, but recency (the row sits well before the tail), frequency (every row carries its own equally 'rare' version string), and llmlingua_style all drop it at the 10% ceiling, and forced_only never ranks past the floor. The waiver ticket is force-kept on its own line." +} diff --git a/eval/tasks/v04/long_context_needle_015.json b/eval/tasks/v04/long_context_needle_015.json new file mode 100644 index 0000000..ba0f406 --- /dev/null +++ b/eval/tasks/v04/long_context_needle_015.json @@ -0,0 +1,13 @@ +{ + "id": "long_context_needle_015", + "family": "long_context_needle", + "tier": "A", + "source": "# access-review: quarterly grant attestation export\nreview_cycle: 2026-Q3-access-attestation\nAttestation record ATT-2247 is the system of record for this review cycle.\ngrant_id=GR-1000 user=r.alvarez scope=read:catalog approver=access-bot\ngrant_id=GR-1001 user=j.nakamura scope=read:orders approver=access-bot\ngrant_id=GR-1002 user=s.osei scope=write:staging approver=access-bot\ngrant_id=GR-1003 user=m.kowalski scope=read:metrics approver=access-bot\ngrant_id=GR-1004 user=t.nguyen scope=write:reports approver=access-bot\ngrant_id=GR-1005 user=c.dubois scope=read:catalog approver=access-bot\ngrant_id=GR-1006 user=p.singh scope=read:orders approver=access-bot\ngrant_id=GR-1007 user=a.ferreira scope=write:staging approver=access-bot\ngrant_id=GR-1008 user=d.kim scope=read:metrics approver=access-bot\ngrant_id=GR-1009 user=l.moreau scope=write:reports approver=access-bot\ngrant_id=GR-1010 user=r.alvarez scope=read:catalog approver=access-bot\ngrant_id=GR-1011 user=j.nakamura scope=read:orders approver=access-bot\ngrant_id=GR-1012 user=s.osei scope=write:staging approver=access-bot\ngrant_id=GR-1013 user=m.kowalski scope=read:metrics approver=access-bot\ngrant_id=GR-1014 user=t.nguyen scope=write:reports approver=access-bot\ngrant_id=GR-1015 user=c.dubois scope=read:catalog approver=access-bot\ngrant_id=GR-1016 user=p.singh scope=read:orders approver=access-bot\ngrant_id=GR-1017 user=a.ferreira scope=write:staging approver=access-bot\ngrant_id=GR-1018 user=d.kim scope=read:metrics approver=access-bot\ngrant_id=GR-1500 user=t.nguyen scope=admin:prod-write approver=access-bot\ngrant_id=GR-1019 user=l.moreau scope=write:reports approver=access-bot\ngrant_id=GR-1020 user=r.alvarez scope=read:catalog approver=access-bot\ngrant_id=GR-1021 user=j.nakamura scope=read:orders approver=access-bot\ngrant_id=GR-1022 user=s.osei scope=write:staging approver=access-bot\ngrant_id=GR-1023 user=m.kowalski scope=read:metrics approver=access-bot\ngrant_id=GR-1024 user=t.nguyen scope=write:reports approver=access-bot\ngrant_id=GR-1025 user=c.dubois scope=read:catalog approver=access-bot\ngrant_id=GR-1026 user=p.singh scope=read:orders approver=access-bot\ngrant_id=GR-1027 user=a.ferreira scope=write:staging approver=access-bot\ngrant_id=GR-1028 user=d.kim scope=read:metrics approver=access-bot\ngrant_id=GR-1029 user=l.moreau scope=write:reports approver=access-bot\ngrant_id=GR-1030 user=r.alvarez scope=read:catalog approver=access-bot\ngrant_id=GR-1031 user=j.nakamura scope=read:orders approver=access-bot\ngrant_id=GR-1032 user=s.osei scope=write:staging approver=access-bot\ngrant_id=GR-1033 user=m.kowalski scope=read:metrics approver=access-bot\ngrant_id=GR-1034 user=t.nguyen scope=write:reports approver=access-bot\ngrant_id=GR-1035 user=c.dubois scope=read:catalog approver=access-bot\ngrant_id=GR-1036 user=p.singh scope=read:orders approver=access-bot\ngrant_id=GR-1037 user=a.ferreira scope=write:staging approver=access-bot\ngrant_id=GR-1038 user=d.kim scope=read:metrics approver=access-bot\ngrant_id=GR-1039 user=l.moreau scope=write:reports approver=access-bot\ngrant_id=GR-1040 user=r.alvarez scope=read:catalog approver=access-bot\ngrant_id=GR-1041 user=j.nakamura scope=read:orders approver=access-bot\ngrant_id=GR-1042 user=s.osei scope=write:staging approver=access-bot\ngrant_id=GR-1043 user=m.kowalski scope=read:metrics approver=access-bot\ngrant_id=GR-1044 user=t.nguyen scope=write:reports approver=access-bot\ngrant_id=GR-1045 user=c.dubois scope=read:catalog approver=access-bot\nGrant policy config for this cycle is defined at /etc/iam/policy/review-q3.rego.\nGrants left unreviewed for 90 days are auto-revoked by the attestation job.\nMost grants below are routine read-scope entitlements carried over unchanged.\n", + "query": "which user received elevated production write access in this review cycle?", + "gold_answer": "user=t.nguyen scope=admin:prod-write", + "critical_atoms": [ + "ATT-2247", + "/etc/iam/policy/review-q3.rego" + ], + "notes": "The needle's scope token 'admin:prod-write' never echoes the query's 'elevated'/'production'/'access' wording; only 'write' overlaps at all, and that same token also appears on two other routine write-scope categories spread across ~45 rows, so bm25's weak, shared signal fails to prefer this row over the other write-scope rows and drops it at both a 25% and a 10% ceiling, and recency drops it too since it is not among the last rows. frequency and llmlingua_style do keep it (the admin:prod-write pairing is the one genuinely rare token combination in the document). Attestation record and policy path are force-kept on separate lines." +} diff --git a/eval/tasks/v04/rust_holdout_001.json b/eval/tasks/v04/rust_holdout_001.json new file mode 100644 index 0000000..e2d3a5c --- /dev/null +++ b/eval/tasks/v04/rust_holdout_001.json @@ -0,0 +1,12 @@ +{ + "id": "rust_holdout_001", + "family": "rust_holdout", + "tier": "A", + "source": "//! Segment compaction for the tokenfold on-disk index layer (F-041 holdout slice).\nuse crate::index::segment::SegmentId;\nuse std::sync::atomic::Ordering;\n\n/// Per-segment liveness stats sampled at the start of a compaction cycle.\npub struct SegmentStats {\n pub live_bytes: u64,\n pub tombstones: u32,\n}\n\nimpl Compactor {\n /// True when a segment has accumulated enough tombstones to be rewritten.\n pub fn should_compact(&self, stats: &SegmentStats) -> bool {\n stats.tombstones > self.tombstone_threshold\n }\n\n /// Merge the chosen segments and hand back the id of the newly written one.\n pub fn merge_segments(&mut self, ids: &[SegmentId]) -> Result {\n let target = self.allocate_segment()?;\n self.write_merged(target, ids)?;\n Ok(target)\n }\n\n fn allocate_segment(&mut self) -> SegmentId {\n SegmentId(self.next_id.fetch_add(1, Ordering::SeqCst) + 1)\n }\n}\n", + "query": "what Result type does the merge_segments function return on success?", + "gold_answer": "merge_segments(&mut self, ids: &[SegmentId]) -> Result", + "critical_atoms": [ + "crate::index::segment" + ], + "notes": "Answer is the merge_segments signature line (query-relevant via 'merge_segments'/'Result'), not a critical atom, so recency/forced_only drop it at a 25% budget while bm25 keeps it; the audit-critical module path crate::index::segment sits on the use line and is force-kept." +} diff --git a/eval/tasks/v04/rust_holdout_010.json b/eval/tasks/v04/rust_holdout_010.json new file mode 100644 index 0000000..d277706 --- /dev/null +++ b/eval/tasks/v04/rust_holdout_010.json @@ -0,0 +1,12 @@ +{ + "id": "rust_holdout_010", + "family": "rust_holdout", + "tier": "A", + "source": "//! Subsystem build-tag registry for the tokenfold plugin loader (F-063 holdout slice).\nuse crate::plugin::registry::PluginId;\nuse std::collections::HashMap;\n\n/// Build metadata recorded for each subsystem at compile time.\npub struct SubsystemTag {\n pub name: &'static str,\n pub build_tag: &'static str,\n}\n\npub const SUBSYSTEM_TAGS: &[SubsystemTag] = &[\n SubsystemTag { name: \"index-writer\", build_tag: \"bt-2f19-stable\" },\n SubsystemTag { name: \"index-reader\", build_tag: \"bt-7a3c-stable\" },\n SubsystemTag { name: \"query-planner\", build_tag: \"bt-11d4-stable\" },\n SubsystemTag { name: \"query-executor\", build_tag: \"bt-88ab-stable\" },\n SubsystemTag { name: \"cache-evictor\", build_tag: \"bt-33f0-stable\" },\n SubsystemTag { name: \"cache-warmup\", build_tag: \"bt-90c2-stable\" },\n SubsystemTag { name: \"wal-writer\", build_tag: \"bt-5566-stable\" },\n SubsystemTag { name: \"wal-flush\", build_tag: \"bt-6e21-canary\" },\n SubsystemTag { name: \"wal-replay\", build_tag: \"bt-a10f-stable\" },\n SubsystemTag { name: \"segment-merge\", build_tag: \"bt-9e40-stable\" },\n SubsystemTag { name: \"segment-split\", build_tag: \"bt-c701-stable\" },\n SubsystemTag { name: \"snapshot-export\", build_tag: \"bt-4b8d-stable\" },\n SubsystemTag { name: \"snapshot-import\", build_tag: \"bt-fe22-stable\" },\n SubsystemTag { name: \"replica-sync\", build_tag: \"bt-0a91-stable\" },\n SubsystemTag { name: \"replica-catchup\", build_tag: \"bt-77cd-stable\" },\n SubsystemTag { name: \"gc-sweeper\", build_tag: \"bt-e3f5-stable\" },\n SubsystemTag { name: \"gc-compactor\", build_tag: \"bt-1290-stable\" },\n SubsystemTag { name: \"metrics-emitter\", build_tag: \"bt-b64a-stable\" },\n SubsystemTag { name: \"metrics-rollup\", build_tag: \"bt-d817-stable\" },\n SubsystemTag { name: \"auth-gateway\", build_tag: \"bt-5c3e-stable\" },\n SubsystemTag { name: \"auth-refresh\", build_tag: \"bt-2299-stable\" },\n SubsystemTag { name: \"rate-limiter\", build_tag: \"bt-8f61-stable\" },\n SubsystemTag { name: \"rate-reporter\", build_tag: \"bt-46ab-stable\" },\n SubsystemTag { name: \"schema-loader\", build_tag: \"bt-cc02-stable\" },\n SubsystemTag { name: \"schema-migrator\", build_tag: \"bt-7710-stable\" },\n SubsystemTag { name: \"batch-scheduler\", build_tag: \"bt-af33-stable\" },\n SubsystemTag { name: \"batch-runner\", build_tag: \"bt-1d4e-stable\" },\n SubsystemTag { name: \"config-loader\", build_tag: \"bt-9902-stable\" },\n SubsystemTag { name: \"config-watcher\", build_tag: \"bt-3e6f-stable\" },\n];\n\nimpl SubsystemTag {\n /// Look up a subsystem's build tag by name.\n pub fn find(name: &str) -> Option<&'static str> {\n SUBSYSTEM_TAGS.iter().find(|t| t.name == name).map(|t| t.build_tag)\n }\n}\n", + "query": "In the plugin loader's subsystem build-tag registry, what build_tag is recorded for the wal-flush subsystem?", + "gold_answer": "bt-6e21-canary", + "critical_atoms": [ + "crate::plugin::registry" + ], + "notes": "Lexical case: the query repeats 'wal-flush'/'build_tag' verbatim, which appear on the gold entry. At a tight budget the entry sits among 28 near-identical `SubsystemTag { name, build_tag }` rows with unique per-row names/tags, so raw term frequency stays flat and recency instead keeps only the last few rows (config-loader, config-watcher), missing the mid-array wal-flush row entirely; llmlingua_style also has no strong signal since every row shares the same struct-literal token shape except the rare 'canary' channel marker. The plugin-registry module path is on its own force-kept use line, separate from the gold row." +} diff --git a/eval/tasks/v04/rust_holdout_011.json b/eval/tasks/v04/rust_holdout_011.json new file mode 100644 index 0000000..e282ef6 --- /dev/null +++ b/eval/tasks/v04/rust_holdout_011.json @@ -0,0 +1,12 @@ +{ + "id": "rust_holdout_011", + "family": "rust_holdout", + "tier": "A", + "source": "// rustc diagnostic capture: cargo +nightly clippy --workspace (F-071 holdout slice)\n// build-id: rc-20260318-fe881a\n\nwarning: unused variable: `tmp` --> src/index/reader.rs:14:9\nwarning: unused variable: `buf` --> src/index/reader.rs:22:13\nwarning: unused variable: `n` --> src/index/reader.rs:31:5\nwarning: unused variable: `acc` --> src/index/merge.rs:9:9\nerror[E0502]: cannot borrow `builder` as immutable because it is also borrowed as mutable --> src/index/merge.rs:15:21\nwarning: unused variable: `idx` --> src/index/merge.rs:17:13\nwarning: unused variable: `guard` --> src/index/merge.rs:26:9\nwarning: unused variable: `ok` --> src/index/merge.rs:40:5\nwarning: unused variable: `res` --> src/index/writer.rs:12:9\nwarning: unused variable: `len` --> src/index/writer.rs:19:13\nwarning: unused variable: `path` --> src/index/writer.rs:33:9\nwarning: unused variable: `flag` --> src/index/writer.rs:47:5\nwarning: unused variable: `tag` --> src/index/writer.rs:55:9\nwarning: unused variable: `slot` --> src/index/writer.rs:63:13\nwarning: unused variable: `state` --> src/index/writer.rs:71:9\nerror[E0499]: cannot borrow `state` as mutable more than once at a time --> src/index/writer.rs:88:17\nwarning: unused variable: `epoch` --> src/index/writer.rs:96:9\nwarning: unused variable: `cursor` --> src/index/writer.rs:104:13\nwarning: unused variable: `hint` --> src/index/writer.rs:112:5\nwarning: unused variable: `mode` --> src/index/planner.rs:8:9\nwarning: unused variable: `plan` --> src/index/planner.rs:16:13\nwarning: unused variable: `cost` --> src/index/planner.rs:24:9\nwarning: unused variable: `rank` --> src/index/planner.rs:32:5\nwarning: unused variable: `key` --> src/index/planner.rs:40:9\nwarning: unused variable: `val` --> src/index/planner.rs:48:13\nwarning: unused variable: `depth` --> src/index/planner.rs:56:9\nerror[E0716]: temporary value dropped while borrowed --> src/index/planner.rs:60:19\nwarning: unused variable: `budget` --> src/index/planner.rs:64:5\nwarning: unused variable: `seed` --> src/index/planner.rs:72:9\nwarning: unused variable: `total` --> src/index/planner.rs:80:13\nwarning: unused variable: `retry` --> src/index/planner.rs:88:9\nwarning: unused variable: `limit` --> src/index/planner.rs:96:5\nwarning: unused mut: `builder` --> src/index/planner.rs:104:9\nwarning: unused mut: `stats` --> src/index/planner.rs:112:13\n", + "query": "Scanning this diagnostic capture, which single finding is severe enough to block compilation rather than just being advisory, and what does it report?", + "gold_answer": "cannot borrow `state` as mutable more than once at a time", + "critical_atoms": [ + "rc-20260318-fe881a" + ], + "notes": "Semantic case: the query never names 'borrow', 'mutable', or 'state', so bm25 has no term overlap with any line and falls back to a positional tie-break; two decoy error[E0502]/error[E0716] lines (equally severe-looking, equally rare wording) sit before and after the true error[E0499] line among 29 near-identical `warning: unused variable/mut ... --> path:line:col` entries, so a frequency/surprisal ranking that keys on rare tokens has three similarly-anomalous candidates to choose from, not one. recency is also misled: the target error sits roughly the middle third of the capture, well before the last several warnings a recency-only policy would keep at a tight budget. The build-id is on its own force-kept header line, separate from the error line." +} diff --git a/eval/tasks/v04/rust_holdout_012.json b/eval/tasks/v04/rust_holdout_012.json new file mode 100644 index 0000000..74ed32e --- /dev/null +++ b/eval/tasks/v04/rust_holdout_012.json @@ -0,0 +1,12 @@ +{ + "id": "rust_holdout_012", + "family": "rust_holdout", + "tier": "A", + "source": "//! Wire-protocol error code table for the tokenfold RPC holdout crate (F-058 holdout slice).\nuse crate::rpc::wire::ErrorKind;\n\n/// Numeric codes exchanged on the wire; must stay stable across minor releases.\npub const ERROR_CODES: &[(ErrorKind, u32)] = &[\n (ErrorKind::Ok, 0),\n (ErrorKind::Unknown, 1),\n (ErrorKind::Timeout, 2),\n (ErrorKind::Cancelled, 3),\n (ErrorKind::InvalidArgument, 4),\n (ErrorKind::DeadlineExceeded, 5),\n (ErrorKind::NotFound, 6),\n (ErrorKind::AlreadyExists, 7),\n (ErrorKind::PermissionDenied, 8),\n (ErrorKind::ResourceExhausted, 9),\n (ErrorKind::FailedPrecondition, 10),\n (ErrorKind::Aborted, 11),\n (ErrorKind::OutOfRange, 12),\n (ErrorKind::Unimplemented, 13),\n (ErrorKind::Internal, 14),\n (ErrorKind::Unavailable, 15),\n (ErrorKind::DataLoss, 16),\n (ErrorKind::Unauthenticated, 17),\n (ErrorKind::TimeoutExceeded, 18),\n (ErrorKind::QuotaExceeded, 19),\n (ErrorKind::RetryLater, 20),\n (ErrorKind::StaleRead, 21),\n (ErrorKind::LeaseExpired, 22),\n (ErrorKind::ShardMoved, 23),\n (ErrorKind::WriteConflict, 24),\n (ErrorKind::ReadOnlyMode, 25),\n (ErrorKind::SnapshotTooOld, 26),\n (ErrorKind::CompactionInProgress, 27),\n (ErrorKind::ReplicaLagging, 28),\n];\n\nimpl ErrorKind {\n pub fn code(self) -> u32 {\n ERROR_CODES.iter().find(|(k, _)| *k == self).map(|(_, c)| *c).unwrap_or(1)\n }\n}\n", + "query": "What numeric error code is assigned to ErrorKind::TimeoutExceeded in the wire protocol table?", + "gold_answer": "ErrorKind::TimeoutExceeded, 18", + "critical_atoms": [ + "crate::rpc::wire" + ], + "notes": "Lexical case: the query repeats 'ErrorKind::TimeoutExceeded' verbatim, which appears on the gold row. At a tight budget that row sits among 28 near-identical `(ErrorKind::Variant, N)` tuple rows, each a unique variant/code pair, so raw term frequency stays flat and recency instead keeps only the last handful of variants (ReadOnlyMode..ReplicaLagging), missing the mid-table TimeoutExceeded row; frequency and llmlingua_style also have little to key on since every row shares the same tuple-literal shape. The rpc::wire module path is on its own force-kept use line, separate from the gold row." +} diff --git a/eval/tasks/v04/rust_holdout_013.json b/eval/tasks/v04/rust_holdout_013.json new file mode 100644 index 0000000..7dda09a --- /dev/null +++ b/eval/tasks/v04/rust_holdout_013.json @@ -0,0 +1,12 @@ +{ + "id": "rust_holdout_013", + "family": "rust_holdout", + "tier": "A", + "source": "//! Criterion micro-benchmark capture for the tokenfold segment engine (F-066 holdout slice).\n// run-id: cr-2026-0714-77b2\nuse crate::bench::segment::harness;\n\nbench_index_lookup_hot 312 ns/iter (+/- 18)\nbench_index_lookup_warm 298 ns/iter (+/- 21)\nbench_index_insert_hot 455 ns/iter (+/- 27)\nbench_index_insert_warm 441 ns/iter (+/- 24)\nbench_index_delete_hot 389 ns/iter (+/- 19)\nbench_index_delete_warm 402 ns/iter (+/- 22)\nbench_query_plan_simple 211 ns/iter (+/- 14)\nbench_query_plan_join 678 ns/iter (+/- 33)\nbench_query_exec_scan 512 ns/iter (+/- 29)\nbench_query_exec_filter 334 ns/iter (+/- 20)\nbench_cache_get_hit 98 ns/iter (+/- 6)\nbench_cache_get_miss 187 ns/iter (+/- 11)\nbench_cache_put 223 ns/iter (+/- 15)\nbench_cache_evict 276 ns/iter (+/- 17)\nbench_wal_append 601 ns/iter (+/- 31)\nbench_wal_fsync 892 ns/iter (+/- 41)\nbench_wal_replay_small 744 ns/iter (+/- 38)\nbench_segment_merge_hot 567 ns/iter (+/- 30)\nbench_segment_merge_warm 589 ns/iter (+/- 32)\nbench_segment_split_hot 498 ns/iter (+/- 26)\nbench_segment_compact_hot 612 ns/iter (+/- 29)\nbench_segment_compact_warm 634 ns/iter (+/- 30)\nbench_segment_compact_cold 9187 ns/iter (+/- 402)\nbench_segment_flush_hot 523 ns/iter (+/- 28)\nbench_snapshot_export 701 ns/iter (+/- 35)\nbench_snapshot_import 733 ns/iter (+/- 37)\nbench_replica_sync_hot 445 ns/iter (+/- 23)\nbench_replica_catchup 667 ns/iter (+/- 34)\nbench_gc_sweep_hot 356 ns/iter (+/- 19)\nbench_gc_compact_hot 589 ns/iter (+/- 27)\nbench_metrics_emit 145 ns/iter (+/- 9)\nbench_metrics_rollup 267 ns/iter (+/- 16)\n", + "query": "Scanning this criterion run, which single benchmark's measured latency is roughly an order of magnitude above every other result, and what is that measured value?", + "gold_answer": "bench_segment_compact_cold 9187 ns/iter", + "critical_atoms": [ + "cr-2026-0714-77b2" + ], + "notes": "Semantic case: the query names no bench, module, or number, so bm25 (which needs query term overlap) has nothing to key on and treats all 31 `bench_name N ns/iter (+/- M)` rows as equally irrelevant; identifying the outlier requires comparing magnitudes across rows, not matching vocabulary. recency is also misled: the outlier sits roughly two thirds through the capture, well before the last several rows a recency-only policy would keep at a tight budget. The run id is on its own force-kept header line, separate from the outlier row." +} diff --git a/eval/tasks/v04/rust_holdout_014.json b/eval/tasks/v04/rust_holdout_014.json new file mode 100644 index 0000000..3d85aaf --- /dev/null +++ b/eval/tasks/v04/rust_holdout_014.json @@ -0,0 +1,12 @@ +{ + "id": "rust_holdout_014", + "family": "rust_holdout", + "tier": "A", + "source": "//! Panic backtrace capture from the shard rebalancer holdout crate (F-074 holdout slice).\n// build-id: bt-7c410e-shard\n\nthread 'shard-worker-3' panicked at src/shard/rebalance.rs:233:21:\n #0 core::panicking::panic_fmt at library/core/src/panicking.rs:72:14\n #1 core::result::unwrap_failed at library/core/src/result.rs:1651:5\n #2 std::rt::lang_start_internal at library/std/src/rt.rs:174:18\n #3 tokio::runtime::park::CachedParkThread::block_on at tokio-1.38.0/src/runtime/park.rs:283:31\n #4 tokio::runtime::scheduler::current_thread::Context::enter at tokio-1.38.0/src/runtime/scheduler/current_thread/mod.rs:471:19\n #5 tokio::runtime::scheduler::current_thread::CurrentThread::block_on at tokio-1.38.0/src/runtime/scheduler/current_thread/mod.rs:162:24\n #6 tokio::task::local::LocalSet::block_on at tokio-1.38.0/src/task/local.rs:399:12\n #7 tokio::runtime::runtime::Runtime::block_on at tokio-1.38.0/src/runtime/runtime.rs:351:47\n #8 futures_util::future::future::FutureExt::poll_unpin at futures-util-0.3.30/src/future/future/mod.rs:322:9\n #9 tower::util::oneshot::Oneshot::poll at tower-0.4.13/src/util/oneshot.rs:78:24\n #10 hyper::server::conn::http1::Connection::poll at hyper-1.3.1/src/proto/h1/conn.rs:210:16\n #11 hyper::service::service::Service::call at hyper-1.3.1/src/service/service.rs:44:9\n #12 tower::service_fn::ServiceFn::call at tower-0.4.13/src/util/service_fn.rs:58:13\n #13 crate::shard::rebalance::plan_transfer at src/shard/rebalance.rs:233:21\n #14 crate::shard::dispatch::route_request at src/shard/dispatch.rs:91:17\n #15 crate::server::handler::handle_request at src/server/handler.rs:120:9\n #16 hyper::server::service::HyperService::call at hyper-1.3.1/src/service/service.rs:60:18\n #17 hyper::proto::h1::dispatch::Dispatcher::poll_catch at hyper-1.3.1/src/proto/h1/dispatch.rs:150:22\n #18 tower::buffer::worker::Worker::poll at tower-0.4.13/src/buffer/worker.rs:203:14\n #19 futures_util::stream::stream::StreamExt::poll_next_unpin at futures-util-0.3.30/src/stream/stream/mod.rs:1652:9\n #20 tokio::io::poll_evented::PollEvented::poll_read at tokio-1.38.0/src/io/poll_evented.rs:133:20\n #21 tokio::net::tcp::stream::TcpStream::poll_read_priv at tokio-1.38.0/src/net/tcp/stream.rs:941:9\n #22 mio::poll::Poll::poll at mio-0.8.11/src/poll.rs:422:18\n #23 tokio::runtime::io::Driver::turn at tokio-1.38.0/src/runtime/io/mod.rs:225:13\n #24 tokio::runtime::driver::Driver::park at tokio-1.38.0/src/runtime/driver.rs:61:9\n #25 tokio::runtime::park::ParkThread::park at tokio-1.38.0/src/runtime/park.rs:128:22\n #26 serde_json::de::Deserializer::parse_object at serde_json-1.0.120/src/de.rs:1204:17\n #27 serde::de::Deserialize::deserialize at serde-1.0.204/src/de/mod.rs:560:9\n #28 tonic::codec::decode::Decoder::decode at tonic-0.12.0/src/codec/decode.rs:88:19\n #29 h2::proto::streams::recv::Recv::poll_data at h2-0.4.5/src/proto/streams/recv.rs:301:14\n", + "query": "In this panic backtrace, which application frame (inside our own shard rebalancer code, not a library) shows the file and line where the panic originated?", + "gold_answer": "crate::shard::rebalance::plan_transfer at src/shard/rebalance.rs:233:21", + "critical_atoms": [ + "bt-7c410e-shard" + ], + "notes": "Lexical case: the query's 'shard rebalancer' echoes the gold frame's 'crate::shard::rebalance::plan_transfer' symbol, but that frame is #13 of 30 near-identical `#N symbol at path:line:col` rows drawn from tokio/hyper/tower/serde library internals, each with its own unique symbol/path so raw term frequency stays flat; recency instead keeps only the last several frames (serde_json/tonic/h2 internals), missing frame #13. The build-id is on its own force-kept header line, separate from the application frame." +} diff --git a/eval/tasks/v04/rust_holdout_015.json b/eval/tasks/v04/rust_holdout_015.json new file mode 100644 index 0000000..44f5eae --- /dev/null +++ b/eval/tasks/v04/rust_holdout_015.json @@ -0,0 +1,12 @@ +{ + "id": "rust_holdout_015", + "family": "rust_holdout", + "tier": "A", + "source": "//! Workspace dependency pin manifest for the tokenfold ingest holdout crate (F-081 holdout slice).\nuse crate::workspace::manifest::PinTable;\n\npub const PINNED_DEPS: &[(&str, &str)] = &[\n (\"serde\", \"1.0.204\"),\n (\"serde_json\", \"1.0.120\"),\n (\"tokio\", \"1.38.0\"),\n (\"tokio-util\", \"0.7.11\"),\n (\"bytes\", \"1.6.0\"),\n (\"thiserror\", \"1.0.61\"),\n (\"anyhow\", \"1.0.86\"),\n (\"tracing\", \"0.1.40\"),\n (\"tracing-subscriber\", \"0.3.18\"),\n (\"clap\", \"4.5.7\"),\n (\"regex\", \"1.10.5\"),\n (\"once_cell\", \"1.19.0\"),\n (\"parking_lot\", \"0.12.3\"),\n (\"crossbeam\", \"0.8.4\"),\n (\"rayon\", \"1.10.0\"),\n (\"itertools\", \"0.13.0\"),\n (\"smallvec\", \"1.13.2\"),\n (\"indexmap\", \"2.2.6\"),\n (\"bincode\", \"1.3.3\"),\n (\"prost\", \"0.13.1\"),\n (\"tonic\", \"0.12.0\"),\n (\"hyper\", \"1.4.1\"),\n (\"http\", \"1.1.0\"),\n (\"url\", \"2.5.2\"),\n (\"uuid\", \"1.9.1\"),\n (\"chrono\", \"0.4.38\"),\n (\"rand\", \"0.8.5\"),\n (\"zstd\", \"0.13.1\"),\n (\"lz4_flex\", \"0.11.3\"),\n (\"memmap2\", \"0.9.4\"),\n (\"rustls\", \"2.0.0-rc.3\"),\n (\"webpki-roots\", \"0.26.3\"),\n (\"h2\", \"0.4.5\"),\n (\"tower\", \"0.4.13\"),\n];\n", + "query": "Scanning the workspace's pinned dependency versions, which single entry uses a pre-release version string instead of a stable semver, and what is that exact pin?", + "gold_answer": "(\"rustls\", \"2.0.0-rc.3\")", + "critical_atoms": [ + "crate::workspace::manifest" + ], + "notes": "Semantic case: the query names no crate or version, so bm25 has no term overlap with any of the 34 `(\"crate\", \"version\")` tuple rows; spotting the one pre-release pin requires recognizing the '-rc.' pattern structurally against otherwise plain X.Y.Z strings, not matching vocabulary. The pin sits two thirds through the table, so recency keeps only the last few rows (webpki-roots, h2, tower) and misses it; frequency and llmlingua_style have little to key on since every row shares the same tuple-literal shape. The workspace::manifest module path is on its own force-kept use line, separate from the pin row." +} diff --git a/eval/tasks/v04/tool_call_json_001.json b/eval/tasks/v04/tool_call_json_001.json new file mode 100644 index 0000000..a295d67 --- /dev/null +++ b/eval/tasks/v04/tool_call_json_001.json @@ -0,0 +1,12 @@ +{ + "id": "tool_call_json_001", + "family": "tool_call_json", + "tier": "A", + "source": "{\n \"tool\": \"http_request\",\n \"request_id\": \"req-8c14fa9e2b\",\n \"params\": {\n \"method\": \"POST\",\n \"url\": \"https://api.internal/v2/orders\",\n \"headers\": {\n \"content_type\": \"application/json\",\n \"authorization\": \"Bearer scrubbed\"\n },\n \"retry\": {\n \"max_attempts\": 4,\n \"backoff_ms\": 250,\n \"timeout_ms\": 8000\n },\n \"pagination\": {\n \"page_size\": 50,\n \"cursor\": null\n }\n },\n \"trace\": \"otel-4a2f9b\"\n}\n", + "query": "what backoff_ms delay is set in the retry policy?", + "gold_answer": "\"backoff_ms\": 250", + "critical_atoms": [ + "req-8c14fa9e2b" + ], + "notes": "Nested tool-call JSON (structural segmentation deferred; line-segmented). The retry backoff line shares 'backoff_ms' with the query so bm25 keeps it at a tight budget, while recency/frequency/llmlingua/forced_only drop it for later or forced-only lines; the audit-critical request_id sits on a separate line and is force-kept. Gold is the full key:value so containment can't false-match a stray 250." +} diff --git a/eval/tasks/v04/tool_call_json_010.json b/eval/tasks/v04/tool_call_json_010.json new file mode 100644 index 0000000..457a487 --- /dev/null +++ b/eval/tasks/v04/tool_call_json_010.json @@ -0,0 +1,13 @@ +{ + "id": "tool_call_json_010", + "family": "tool_call_json", + "tier": "A", + "source": "{\n \"tool\": \"webhook_dispatch_batch\",\n \"batch\": \"wh-6f2a19c0\",\n \"sha256\": \"2eaf80c1\",\n \"deliveries\": [\n {\n \"endpoint\": \"svc-checkout/order-updated\",\n \"region\": \"us-east-1\",\n \"http_status\": 200\n },\n {\n \"endpoint\": \"svc-shipping/label-created\",\n \"region\": \"us-east-1\",\n \"http_status\": 200\n },\n {\n \"endpoint\": \"svc-notify/email-queued\",\n \"region\": \"us-east-1\",\n \"http_status\": 200\n },\n {\n \"endpoint\": \"svc-loyalty/points-accrued\",\n \"region\": \"us-east-1\",\n \"http_status\": 200\n },\n {\n \"endpoint\": \"svc-tax/invoice-finalized\",\n \"region\": \"us-east-1\",\n \"http_status\": 200\n },\n {\n \"endpoint\": \"svc-ledger/txn-posted\",\n \"region\": \"us-east-1\",\n \"http_status\": 500,\n \"error_reason\": \"UPSTREAM_TIMEOUT\"\n },\n {\n \"endpoint\": \"svc-search/index-refreshed\",\n \"region\": \"us-east-1\",\n \"http_status\": 200\n },\n {\n \"endpoint\": \"svc-catalog/price-updated\",\n \"region\": \"us-east-1\",\n \"http_status\": 200\n },\n {\n \"endpoint\": \"svc-returns/rma-opened\",\n \"region\": \"us-east-1\",\n \"http_status\": 200\n },\n {\n \"endpoint\": \"svc-crm/contact-synced\",\n \"region\": \"us-east-1\",\n \"http_status\": 200\n }\n ]\n}\n", + "query": "what error_reason was recorded for the failed webhook delivery attempt?", + "gold_answer": "\"error_reason\": \"UPSTREAM_TIMEOUT\"", + "critical_atoms": [ + "wh-6f2a19c0", + "2eaf80c1" + ], + "notes": "Ten near-identical delivery records; only the svc-ledger entry carries an error_reason line (every other delivery just reports http_status 200). At a ~10% budget bm25 exactly matches 'error_reason' from the query and keeps the line, but frequency instead favors other endpoint path segments that happen to score rarer, and llmlingua_style spends its non-forced budget on the tool-name line instead; recency keeps only the trailing brackets of the list. batch and sha256 are force-kept on their own header lines, separate from the gold line." +} diff --git a/eval/tasks/v04/tool_call_json_011.json b/eval/tasks/v04/tool_call_json_011.json new file mode 100644 index 0000000..b7c6393 --- /dev/null +++ b/eval/tasks/v04/tool_call_json_011.json @@ -0,0 +1,13 @@ +{ + "id": "tool_call_json_011", + "family": "tool_call_json", + "tier": "A", + "source": "{\n \"tool\": \"traffic_shaping_policy_apply\",\n \"apply\": \"tsa-914b2c\",\n \"sha256\": \"7d0e4fa1\",\n \"policies\": [\n {\n \"service\": \"checkout-api\",\n \"region\": \"us-east-1\",\n \"bandwidth_kbps\": 20000\n },\n {\n \"service\": \"search-api\",\n \"region\": \"us-east-1\",\n \"bandwidth_kbps\": 20000\n },\n {\n \"service\": \"recs-api\",\n \"region\": \"us-east-1\",\n \"bandwidth_kbps\": 20000\n },\n {\n \"service\": \"media-transcoder\",\n \"region\": \"us-east-1\",\n \"bandwidth_kbps\": 20000\n },\n {\n \"service\": \"billing-api\",\n \"region\": \"us-east-1\",\n \"bandwidth_kbps\": 20000\n },\n {\n \"service\": \"session-store-proxy\",\n \"region\": \"us-east-1\",\n \"bandwidth_kbps\": 20000\n },\n {\n \"service\": \"image-resizer\",\n \"region\": \"us-east-1\",\n \"bandwidth_kbps\": 20000\n },\n {\n \"service\": \"export-worker\",\n \"region\": \"us-east-1\",\n \"bandwidth_kbps\": 640,\n \"throttled\": true\n },\n {\n \"service\": \"geo-lookup\",\n \"region\": \"us-east-1\",\n \"bandwidth_kbps\": 20000\n },\n {\n \"service\": \"pricing-engine\",\n \"region\": \"us-east-1\",\n \"bandwidth_kbps\": 20000\n }\n ]\n}\n", + "query": "which service had its outbound bandwidth capped below its normal ceiling?", + "gold_answer": "\"throttled\": true", + "critical_atoms": [ + "tsa-914b2c", + "7d0e4fa1" + ], + "notes": "Ten service policy records; only export-worker carries a throttled line, and the throttled value itself shares no vocabulary with the query, so bm25 scores it 0 -- its only real overlap is the generic word 'service', tied identically across all ten entries, so it defaults to the first one (checkout-api). At a ~10% budget frequency and llmlingua_style are drawn to other service names and the header lines instead, and recency keeps only the tail of the list. apply and sha256 are force-kept on separate header lines from the gold line." +} diff --git a/eval/tasks/v04/tool_call_json_012.json b/eval/tasks/v04/tool_call_json_012.json new file mode 100644 index 0000000..22ef2aa --- /dev/null +++ b/eval/tasks/v04/tool_call_json_012.json @@ -0,0 +1,13 @@ +{ + "id": "tool_call_json_012", + "family": "tool_call_json", + "tier": "A", + "source": "{\n \"tool\": \"database_migration_runner\",\n \"run\": \"mig-3a7c081f\",\n \"rollback\": \"/mig/3a7c081f.sql\",\n \"steps\": [\n {\n \"step_id\": \"0041_add_orders_index\",\n \"applied_by\": \"migrations-bot\",\n \"status\": \"applied\"\n },\n {\n \"step_id\": \"0042_backfill_customer_region\",\n \"applied_by\": \"migrations-bot\",\n \"status\": \"applied\"\n },\n {\n \"step_id\": \"0043_drop_legacy_view\",\n \"applied_by\": \"migrations-bot\",\n \"status\": \"applied\"\n },\n {\n \"step_id\": \"0044_add_ledger_partition\",\n \"applied_by\": \"migrations-bot\",\n \"status\": \"applied\"\n },\n {\n \"step_id\": \"0045_rename_invoice_column\",\n \"applied_by\": \"migrations-bot\",\n \"status\": \"applied\"\n },\n {\n \"step_id\": \"0046_add_fk_shipments_orders\",\n \"applied_by\": \"migrations-bot\",\n \"status\": \"failed\",\n \"error_code\": \"23503\"\n },\n {\n \"step_id\": \"0047_backfill_tax_rates\",\n \"applied_by\": \"migrations-bot\",\n \"status\": \"applied\"\n },\n {\n \"step_id\": \"0048_add_returns_table\",\n \"applied_by\": \"migrations-bot\",\n \"status\": \"applied\"\n },\n {\n \"step_id\": \"0049_widen_amount_precision\",\n \"applied_by\": \"migrations-bot\",\n \"status\": \"applied\"\n },\n {\n \"step_id\": \"0050_add_audit_trigger\",\n \"applied_by\": \"migrations-bot\",\n \"status\": \"applied\"\n }\n ]\n}\n", + "query": "which error_code was raised by the failed migration step?", + "gold_answer": "\"error_code\": \"23503\"", + "critical_atoms": [ + "mig-3a7c081f", + "/mig/3a7c081f.sql" + ], + "notes": "Ten migration steps; only step 0046 fails, splitting the outcome across two adjacent lines -- 'status': 'failed' and the error_code line beneath it. Both carry one exact query term each and score almost identically under bm25, so the tie is broken by document order and the status line is kept first; at a ~10% budget there is only room for one of the two, and the error_code line loses. frequency and llmlingua_style rank the header path/id lines above it instead, and recency keeps only the trailing steps. run and rollback are force-kept on separate header lines." +} diff --git a/eval/tasks/v04/tool_call_json_013.json b/eval/tasks/v04/tool_call_json_013.json new file mode 100644 index 0000000..ce87f38 --- /dev/null +++ b/eval/tasks/v04/tool_call_json_013.json @@ -0,0 +1,13 @@ +{ + "id": "tool_call_json_013", + "family": "tool_call_json", + "tier": "A", + "source": "{\n \"tool\": \"cluster_health_probe\",\n \"probe\": \"chp-5b0f92da\",\n \"sha256\": \"c41a90fe\",\n \"nodes\": [\n {\n \"node_id\": \"node-a01\",\n \"zone\": \"us-east-1a\",\n \"status\": \"ok\"\n },\n {\n \"node_id\": \"node-a02\",\n \"zone\": \"us-east-1a\",\n \"status\": \"ok\"\n },\n {\n \"node_id\": \"node-a03\",\n \"zone\": \"us-east-1a\",\n \"status\": \"ok\"\n },\n {\n \"node_id\": \"node-a04\",\n \"zone\": \"us-east-1a\",\n \"status\": \"ok\"\n },\n {\n \"node_id\": \"node-b01\",\n \"zone\": \"us-east-1a\",\n \"status\": \"ok\"\n },\n {\n \"node_id\": \"node-b02\",\n \"zone\": \"us-east-1a\",\n \"status\": \"ok\"\n },\n {\n \"node_id\": \"node-b03\",\n \"zone\": \"us-east-1a\",\n \"status\": \"ok\"\n },\n {\n \"node_id\": \"node-c03\",\n \"zone\": \"us-east-1a\",\n \"status\": \"degraded\",\n \"reason\": \"disk_io_saturation\"\n },\n {\n \"node_id\": \"node-d01\",\n \"zone\": \"us-east-1a\",\n \"status\": \"ok\"\n },\n {\n \"node_id\": \"node-d02\",\n \"zone\": \"us-east-1a\",\n \"status\": \"ok\"\n }\n ]\n}\n", + "query": "which cluster node reported a storage bottleneck during the probe, and why?", + "gold_answer": "\"reason\": \"disk_io_saturation\"", + "critical_atoms": [ + "chp-5b0f92da", + "c41a90fe" + ], + "notes": "Ten cluster-probe node records; only node-c03 is degraded and carries a reason line ('disk_io_saturation'), while the query's 'storage bottleneck' shares no vocabulary with that value. bm25's only real overlap is the generic word 'node' inside every node-NNN id, so it defaults to the earliest node rather than node-c03. At a ~10% budget frequency and llmlingua_style favor the header lines instead, and recency keeps only the last couple of nodes. probe and sha256 are force-kept on separate header lines." +} diff --git a/eval/tasks/v04/tool_call_json_014.json b/eval/tasks/v04/tool_call_json_014.json new file mode 100644 index 0000000..514d778 --- /dev/null +++ b/eval/tasks/v04/tool_call_json_014.json @@ -0,0 +1,13 @@ +{ + "id": "tool_call_json_014", + "family": "tool_call_json", + "tier": "A", + "source": "{\n \"tool\": \"cache_invalidation_batch\",\n \"batch\": \"cib-4e9a13\",\n \"lock\": \"/cache/4e9a13.lock\",\n \"targets\": [\n {\n \"cache_key\": \"checkout:cart:summary:v2\",\n \"owner\": \"checkout-team-cache-pool\",\n \"status\": \"conflicted\",\n \"conflict\": \"cart:totals:v2\"\n },\n {\n \"cache_key\": \"profile:avatar:thumb:v1\",\n \"owner\": \"checkout-team-cache-pool\",\n \"status\": \"invalidated\"\n },\n {\n \"cache_key\": \"search:suggest:top:v3\",\n \"owner\": \"checkout-team-cache-pool\",\n \"status\": \"invalidated\"\n },\n {\n \"cache_key\": \"pricing:region:emea:v1\",\n \"owner\": \"checkout-team-cache-pool\",\n \"status\": \"invalidated\"\n },\n {\n \"cache_key\": \"session:idle:timeout:v2\",\n \"owner\": \"checkout-team-cache-pool\",\n \"status\": \"invalidated\"\n },\n {\n \"cache_key\": \"recs:homepage:feed:v4\",\n \"owner\": \"checkout-team-cache-pool\",\n \"status\": \"invalidated\"\n },\n {\n \"cache_key\": \"inventory:sku:count:v1\",\n \"owner\": \"checkout-team-cache-pool\",\n \"status\": \"invalidated\"\n },\n {\n \"cache_key\": \"notif:digest:daily:v2\",\n \"owner\": \"checkout-team-cache-pool\",\n \"status\": \"invalidated\"\n },\n {\n \"cache_key\": \"billing:invoice:pdf:v1\",\n \"owner\": \"checkout-team-cache-pool\",\n \"status\": \"invalidated\"\n },\n {\n \"cache_key\": \"catalog:facet:filters:v3\",\n \"owner\": \"checkout-team-cache-pool\",\n \"status\": \"invalidated\"\n }\n ]\n}\n", + "query": "which cache key did checkout:cart:summary:v2 conflict with during invalidation?", + "gold_answer": "\"conflict\": \"cart:totals:v2\"", + "critical_atoms": [ + "cib-4e9a13", + "/cache/4e9a13.lock" + ], + "notes": "Nine cache-invalidation targets; only checkout:cart:summary:v2 conflicts, and its conflict value sits one line below the cache_key line that shares the query's literal wording. bm25 ranks both lines near the top, but the cache_key line scores slightly higher and is kept first; at a ~10% budget there is no room left for the conflict line itself. frequency favors other, rarer cache keys, llmlingua_style favors the header lines, and recency keeps only the tail of the list. batch and lock are force-kept on separate header lines." +} diff --git a/eval/tasks/v04/tool_call_json_015.json b/eval/tasks/v04/tool_call_json_015.json new file mode 100644 index 0000000..62ae2b7 --- /dev/null +++ b/eval/tasks/v04/tool_call_json_015.json @@ -0,0 +1,13 @@ +{ + "id": "tool_call_json_015", + "family": "tool_call_json", + "tier": "A", + "source": "{\n \"tool\": \"container_restart_sweep\",\n \"sweep\": \"crs-b02f7e91\",\n \"sha256\": \"a68f0d3c\",\n \"containers\": [\n {\n \"container_id\": \"ctr-8a01f2\",\n \"restarted_at\": \"2026-07-24T09:15:00Z\",\n \"exit_code\": 0\n },\n {\n \"container_id\": \"ctr-8a01f3\",\n \"restarted_at\": \"2026-07-24T09:15:00Z\",\n \"exit_code\": 0\n },\n {\n \"container_id\": \"ctr-8a01f4\",\n \"restarted_at\": \"2026-07-24T09:15:00Z\",\n \"exit_code\": 0\n },\n {\n \"container_id\": \"ctr-8a01f5\",\n \"restarted_at\": \"2026-07-24T09:15:00Z\",\n \"exit_code\": 0\n },\n {\n \"container_id\": \"ctr-8a01f6\",\n \"restarted_at\": \"2026-07-24T09:15:00Z\",\n \"exit_code\": 0\n },\n {\n \"container_id\": \"ctr-8a01f7\",\n \"restarted_at\": \"2026-07-24T09:15:00Z\",\n \"exit_code\": 0\n },\n {\n \"container_id\": \"ctr-8a01f8\",\n \"restarted_at\": \"2026-07-24T09:15:00Z\",\n \"exit_code\": 0\n },\n {\n \"container_id\": \"ctr-8a01f9\",\n \"restarted_at\": \"2026-07-24T09:15:00Z\",\n \"exit_code\": 137\n },\n {\n \"container_id\": \"ctr-8a01fa\",\n \"restarted_at\": \"2026-07-24T09:15:00Z\",\n \"exit_code\": 0\n },\n {\n \"container_id\": \"ctr-8a01fb\",\n \"restarted_at\": \"2026-07-24T09:15:00Z\",\n \"exit_code\": 0\n }\n ]\n}\n", + "query": "which container was terminated because it ran out of memory?", + "gold_answer": "\"exit_code\": 137", + "critical_atoms": [ + "crs-b02f7e91", + "a68f0d3c" + ], + "notes": "Ten container-restart records; only ctr-8a01f9 exited with code 137 (every other container exited 0), while the query's 'ran out of memory' shares no vocabulary with 'exit_code' or '137', so bm25 scores every line 0 and falls back to keeping the first few containers by position. At a ~10% budget llmlingua_style's surprisal does pick out the rare '137' token among the repeated '0' exit codes, but recency keeps only the later containers and frequency is drawn to the header lines instead. sweep and sha256 are force-kept on separate header lines." +} diff --git a/eval/tasks/v04/typescript_holdout_001.json b/eval/tasks/v04/typescript_holdout_001.json new file mode 100644 index 0000000..6d2f1ba --- /dev/null +++ b/eval/tasks/v04/typescript_holdout_001.json @@ -0,0 +1,12 @@ +{ + "id": "typescript_holdout_001", + "family": "typescript_holdout", + "tier": "A", + "source": "// src/pricing/discount.ts\nimport { Money } from \"../money\";\n\nexport interface DiscountRule {\n code: string;\n percentOff: number;\n minSubtotal: number;\n expiresAt: Date;\n}\n\nexport function applyDiscount(subtotal: Money, rule: DiscountRule): Money {\n if (subtotal.amount < rule.minSubtotal) {\n return subtotal;\n }\n const reduction = subtotal.amount * (rule.percentOff / 100);\n return subtotal.minus(reduction);\n}\n\nexport function isExpired(rule: DiscountRule, now: Date): boolean {\n return rule.expiresAt.getTime() < now.getTime();\n}\n", + "query": "what numeric type is declared for the percentOff property?", + "gold_answer": "percentOff: number;", + "critical_atoms": [ + "src/pricing/discount.ts" + ], + "notes": "The interface field line carrying `percentOff: number;` is the answer and shares the rare token 'percentoff' with the query, so query-aware bm25 keeps it while recency/frequency (spend the tight 25% budget on later function bodies) and forced_only drop it; the file path is on line 1 and is force-kept." +} diff --git a/eval/tasks/v04/typescript_holdout_010.json b/eval/tasks/v04/typescript_holdout_010.json new file mode 100644 index 0000000..48bf226 --- /dev/null +++ b/eval/tasks/v04/typescript_holdout_010.json @@ -0,0 +1,13 @@ +{ + "id": "typescript_holdout_010", + "family": "typescript_holdout", + "tier": "A", + "source": "// src/errors/codes.ts\n// Central error-code registry for the platform gateway.\n// audit-hash: sha256:4b9c1e2a7f306d58\n\nexport enum ErrorCode {\n E_AUTH_MISSING = 4001,\n E_AUTH_EXPIRED = 4002,\n E_AUTH_INVALID = 4003,\n E_AUTH_REVOKED = 4004,\n E_SESSION_STALE = 4010,\n E_SESSION_LOCKED = 4011,\n E_SESSION_MISSING = 4012,\n E_TOKEN_MALFORMED = 4020,\n E_TOKEN_EXPIRED = 4021,\n E_TOKEN_REUSED = 4022,\n E_SCOPE_DENIED = 4030,\n E_SCOPE_UNKNOWN = 4031,\n E_QUOTA_EXCEEDED = 4040,\n E_QUOTA_RESET_PENDING = 4041,\n E_RATE_LIMIT_EXCEEDED = 4291,\n E_RATE_WINDOW_INVALID = 4292,\n E_PAYLOAD_TOO_LARGE = 4050,\n E_PAYLOAD_MALFORMED = 4051,\n E_PAYLOAD_EMPTY = 4052,\n E_HEADER_MISSING = 4060,\n E_HEADER_INVALID = 4061,\n E_ROUTE_NOT_FOUND = 4070,\n E_ROUTE_DEPRECATED = 4071,\n E_METHOD_NOT_ALLOWED = 4080,\n E_CONTENT_TYPE_UNSUPPORTED = 4090,\n E_VERSION_UNSUPPORTED = 4100,\n E_UPSTREAM_TIMEOUT = 5010,\n E_UPSTREAM_UNAVAILABLE = 5011,\n E_UPSTREAM_RESET = 5012,\n E_CACHE_MISS_FATAL = 5020,\n E_CACHE_CORRUPT = 5021,\n E_DB_CONN_LOST = 5030,\n E_DB_DEADLOCK = 5031,\n E_DB_CONSTRAINT = 5032,\n E_QUEUE_FULL = 5040,\n E_QUEUE_STALLED = 5041,\n E_CONFIG_MISSING = 5050,\n E_CONFIG_INVALID = 5051,\n E_INTERNAL_UNKNOWN = 5999,\n}\n\nexport function isRetryable(code: ErrorCode): boolean {\n return code >= 5000 && code !== ErrorCode.E_INTERNAL_UNKNOWN;\n}\n", + "query": "What numeric value is assigned to the E_RATE_LIMIT_EXCEEDED error code?", + "gold_answer": "E_RATE_LIMIT_EXCEEDED = 4291,", + "critical_atoms": [ + "src/errors/codes.ts", + "sha256:4b9c1e2a7f306d58" + ], + "notes": "A 44-line ErrorCode enum where the query names the target member (E_RATE_LIMIT_EXCEEDED) directly -- a lexical case -- but the header comment's 'error-code registry' phrase gives bm25 an unrelated line with more literal term overlap ('error' + 'code') than the buried enum member itself, so bm25 keeps the wrong line. Every member also carries two effectively unique tokens (its SCREAMING_SNAKE name and its number), so frequency and llmlingua_style tie across almost every member and fall back to file order, landing on the earliest entries rather than the one about a third of the way down. Recency favors the trailing members and the closing function instead. All four heuristic selectors miss the value at the ~10% ceiling; the file path and audit hash sit on their own header lines and are force-kept regardless." +} diff --git a/eval/tasks/v04/typescript_holdout_011.json b/eval/tasks/v04/typescript_holdout_011.json new file mode 100644 index 0000000..6040c3c --- /dev/null +++ b/eval/tasks/v04/typescript_holdout_011.json @@ -0,0 +1,13 @@ +{ + "id": "typescript_holdout_011", + "family": "typescript_holdout", + "tier": "A", + "source": "// src/config/serviceTimeouts.ts\n// generated: do not hand-edit apart from the registry below\n// manifest-id: cfg-2026-04-11-timeouts\n\nexport const SERVICE_TIMEOUTS_MS: Record = {\n authService: 2500,\n sessionService: 2000,\n profileService: 3000,\n searchService: 3500,\n notificationService: 4000,\n emailService: 4500,\n smsService: 4500,\n inventoryService: 5000,\n catalogService: 5000,\n pricingService: 5500,\n cartService: 3000,\n checkoutService: 6000,\n cardNetworkRelay: 12000,\n shippingService: 5000,\n taxService: 4000,\n fraudService: 6500,\n loyaltyService: 3000,\n reviewService: 3500,\n recommendationService: 4000,\n analyticsService: 2000,\n auditService: 3000,\n webhookService: 5000,\n reportingService: 7000,\n exportService: 8000,\n importService: 8000,\n backupService: 9000,\n syncService: 4000,\n metricsService: 2000,\n loggingService: 2000,\n healthCheckService: 1000,\n};\n\nexport function timeoutFor(service: keyof typeof SERVICE_TIMEOUTS_MS): number {\n return SERVICE_TIMEOUTS_MS[service] ?? 5000;\n}\n", + "query": "How many milliseconds does the platform wait on the card-network settlement relay before giving up?", + "gold_answer": "cardNetworkRelay: 12000,", + "critical_atoms": [ + "src/config/serviceTimeouts.ts", + "cfg-2026-04-11-timeouts" + ], + "notes": "A 30-entry service-timeout map (SERVICE_TIMEOUTS_MS) where the query describes the payment relay by function ('card-network settlement relay ... giving up') instead of citing its camelCase key cardNetworkRelay, so bm25 finds no literal term overlap and falls back to file order. Recency favors the map's last few entries (metrics/logging/health-check) and frequency/llmlingua_style tie across most rows, since every key+value pair is locally unique, landing on earlier tied rows instead of the mid-file relay entry. All four heuristic selectors miss it at the ~10% ceiling; the file path and manifest id sit on their own header lines and are force-kept." +} diff --git a/eval/tasks/v04/typescript_holdout_012.json b/eval/tasks/v04/typescript_holdout_012.json new file mode 100644 index 0000000..718244c --- /dev/null +++ b/eval/tasks/v04/typescript_holdout_012.json @@ -0,0 +1,13 @@ +{ + "id": "typescript_holdout_012", + "family": "typescript_holdout", + "tier": "A", + "source": "tsc build report -- CI run rpt-88214-lin\ntsconfig fingerprint: sha1:c93f0ae1d4b2\nstrict mode: true\ntarget: ES2022\n-- diagnostics (33) --\nsrc/components/Avatar.tsx(12,5): error TS2322: Type 'string | undefined' is not assignable to type 'string'.\nsrc/components/Badge.tsx(8,14): error TS2554: Expected 1 arguments, but got 2.\nsrc/components/Banner.tsx(19,3): error TS2739: Type '{}' is missing properties from type 'BannerProps'.\nsrc/components/Breadcrumb.tsx(5,22): error TS2304: Cannot find name 'Crumb'.\nsrc/components/Card.tsx(31,9): error TS2345: Argument of type 'number' is not assignable to parameter of type 'string'.\nsrc/components/Checkbox.tsx(14,17): error TS2531: Object is possibly 'null'.\nsrc/components/Chip.tsx(22,6): error TS2551: Property 'lable' does not exist on type 'ChipProps'.\nsrc/components/Dialog.tsx(40,11): error TS2345: Argument of type 'HTMLElement | null' is not assignable to parameter of type 'HTMLElement'.\nsrc/components/Drawer.tsx(9,4): error TS2322: Type 'number' is not assignable to type 'string'.\nsrc/components/Dropdown.tsx(27,13): error TS2769: No overload matches this call.\nsrc/components/Field.tsx(6,8): error TS2322: Type 'boolean' is not assignable to type 'string'.\nsrc/components/Footer.tsx(3,1): error TS2307: Cannot find module './styles'.\nsrc/components/Grid.tsx(18,20): error TS2339: Property 'span' does not exist on type 'GridProps'.\nsrc/components/Header.tsx(11,7): error TS2345: Argument of type 'string' is not assignable to parameter of type 'number'.\nsrc/components/Icon.tsx(2,15): error TS2304: Cannot find name 'IconName'.\nsrc/components/Input.tsx(35,9): error TS2322: Type 'string' is not assignable to type 'never'.\nsrc/components/Label.tsx(4,12): error TS2551: Property 'htmlFor' does not exist on type 'LabelProps'.\nsrc/components/List.tsx(29,6): error TS2322: Type 'string[]' is not assignable to type 'ReactNode'.\nsrc/components/Menu.tsx(16,10): error TS2739: Type '{}' is missing properties from type 'MenuProps'.\nsrc/components/Modal.tsx(24,3): error TS2531: Object is possibly 'null'.\nsrc/components/Nav.tsx(7,19): error TS2345: Argument of type 'null' is not assignable to parameter of type 'string'.\nsrc/components/Pagination.tsx(13,8): error TS2322: Type 'number' is not assignable to type 'string'.\nsrc/components/Popover.tsx(21,14): error TS2554: Expected 2 arguments, but got 1.\nsrc/components/Progress.tsx(10,5): error TS2322: Type 'string' is not assignable to type 'number'.\nsrc/components/Radio.tsx(15,9): error TS2531: Object is possibly 'null'.\nsrc/components/Select.tsx(33,6): error TS2345: Argument of type 'string' is not assignable to parameter of type 'Option'.\nsrc/components/Skeleton.tsx(1,1): error TS2307: Cannot find module './placeholder'.\nsrc/components/Slider.tsx(26,17): error TS2322: Type 'number[]' is not assignable to type 'number'.\nsrc/components/Spinner.tsx(5,3): error TS2304: Cannot find name 'SpinnerSize'.\nsrc/components/Switch.tsx(17,11): error TS2322: Type 'boolean' is not assignable to type 'string'.\nsrc/components/Table.tsx(38,4): error TS2339: Property 'rowKey' does not exist on type 'TableProps'.\nsrc/components/Tabs.tsx(20,8): error TS2345: Argument of type 'string' is not assignable to parameter of type 'TabKey'.\nsrc/components/Toast.tsx(9,13): error TS2531: Object is possibly 'null'.\n", + "query": "What compiler error does src/components/Card.tsx report?", + "gold_answer": "src/components/Card.tsx(31,9): error TS2345: Argument of type 'number' is not assignable to parameter of type 'string'.", + "critical_atoms": [ + "rpt-88214-lin", + "sha1:c93f0ae1d4b2" + ], + "notes": "A 33-line tsc diagnostic batch dominated by the repeated \"Type X is not assignable to type Y\" pattern; the query asks specifically about Card.tsx, whose diagnostic sits near the top of the list. bm25's idf on the rare 'card' token keeps it comfortably inside the ~10% ceiling. Recency (which favors the last diagnostics in the batch) and frequency/llmlingua_style (whose per-token score is no higher than a dozen other decoys sharing the same assignability wording) all miss it. The CI run id and tsconfig fingerprint sit on the header lines and are force-kept." +} diff --git a/eval/tasks/v04/typescript_holdout_013.json b/eval/tasks/v04/typescript_holdout_013.json new file mode 100644 index 0000000..63c2cd7 --- /dev/null +++ b/eval/tasks/v04/typescript_holdout_013.json @@ -0,0 +1,13 @@ +{ + "id": "typescript_holdout_013", + "family": "typescript_holdout", + "tier": "A", + "source": "// src/net/retryPolicies.ts\n// policy-manifest: sha1:7c4e9b21fa06\n// last-reviewed: sre-oncall-2026w17\n\nexport type RetryPolicy = readonly [service: string, maxAttempts: number, backoffMs: number];\n\nexport const RETRY_POLICIES: RetryPolicy[] = [\n [\"auth\", 3, 150],\n [\"session\", 3, 150],\n [\"profile\", 3, 200],\n [\"search\", 2, 250],\n [\"catalog\", 3, 200],\n [\"pricing\", 3, 200],\n [\"inventory\", 2, 300],\n [\"cart\", 3, 150],\n [\"checkout\", 4, 400],\n [\"ledger-settlement\", 7, 4000],\n [\"shipping\", 3, 300],\n [\"tax\", 3, 200],\n [\"fraud-check\", 4, 500],\n [\"loyalty\", 2, 250],\n [\"review\", 2, 250],\n [\"recommendation\", 2, 300],\n [\"notification\", 3, 200],\n [\"webhook\", 3, 300],\n [\"export\", 3, 500],\n [\"import\", 3, 500],\n [\"backup\", 2, 1000],\n [\"sync\", 3, 400],\n [\"metrics\", 2, 200],\n [\"logging\", 2, 200],\n [\"health-check\", 1, 100],\n [\"config\", 2, 150],\n [\"feature-flag\", 2, 150],\n [\"ab-experiment\", 2, 150],\n [\"support-ticket\", 3, 250],\n];\n\nexport function policyFor(service: string): RetryPolicy | undefined {\n return RETRY_POLICIES.find(([name]) => name === service);\n}\n", + "query": "Before the money-movement step gives up and surfaces an error to the customer, how many attempts and what backoff does it use?", + "gold_answer": "[\"ledger-settlement\", 7, 4000],", + "critical_atoms": [ + "src/net/retryPolicies.ts", + "sha1:7c4e9b21fa06" + ], + "notes": "A 30-entry retry-policy tuple array where the query describes the ledger-settlement row by function ('money-movement step ... gives up') rather than by its literal key, so bm25 finds no term overlap and falls back to file order; recency favors the array's last few entries and misses the mid-file settlement row. frequency ties most rows (every service name plus its attempt/backoff pair is locally unique) and lands on an earlier tied row instead. llmlingua_style's surprisal weighting happens to favor the settlement row's unusually large attempt/backoff values, recovering it where the other three selectors do not. The file path and policy-manifest hash sit on their own header lines and are force-kept." +} diff --git a/eval/tasks/v04/typescript_holdout_014.json b/eval/tasks/v04/typescript_holdout_014.json new file mode 100644 index 0000000..1915a4c --- /dev/null +++ b/eval/tasks/v04/typescript_holdout_014.json @@ -0,0 +1,13 @@ +{ + "id": "typescript_holdout_014", + "family": "typescript_holdout", + "tier": "A", + "source": "// src/build/versionPins.ts\n// pin-manifest: pins-rev-2026-07-r3\n// generated-from: package-lock.json\n\nexport const VERSION_PINS: Record = {\n \"react\": \"18.3.1\",\n \"react-dom\": \"18.3.1\",\n \"typescript\": \"5.5.4\",\n \"eslint\": \"9.7.0\",\n \"prettier\": \"3.3.3\",\n \"vite\": \"5.3.5\",\n \"vitest\": \"2.0.5\",\n \"zod\": \"3.23.8\",\n \"axios\": \"1.7.2\",\n \"lodash\": \"4.17.21\",\n \"date-fns\": \"3.6.0\",\n \"@acme/img-resize\": \"6.2.9\",\n \"@acme/ui-kit\": \"4.1.0\",\n \"@acme/auth-client\": \"2.9.3\",\n \"@acme/logger\": \"1.4.2\",\n \"@acme/config-loader\": \"1.0.7\",\n \"@acme/feature-flags\": \"2.2.1\",\n \"@acme/cache-client\": \"3.0.5\",\n \"@acme/queue-client\": \"1.6.0\",\n \"@acme/db-client\": \"5.3.2\",\n \"webpack\": \"5.93.0\",\n \"babel-loader\": \"9.1.3\",\n \"postcss\": \"8.4.40\",\n \"tailwindcss\": \"3.4.7\",\n \"jest\": \"29.7.0\",\n \"ts-node\": \"10.9.2\",\n \"husky\": \"9.1.4\",\n \"commitlint\": \"19.3.0\",\n \"storybook\": \"8.2.6\",\n \"rollup\": \"4.19.1\",\n};\n\nexport function isPinned(pkg: string): boolean {\n return pkg in VERSION_PINS;\n}\n", + "query": "What version is @acme/img-resize pinned to?", + "gold_answer": "\"@acme/img-resize\": \"6.2.9\",", + "critical_atoms": [ + "src/build/versionPins.ts", + "pins-rev-2026-07-r3" + ], + "notes": "A 30-entry semver pin registry where the query names the package directly (@acme/img-resize), so bm25's idf on the rare 'img'/'resize' tokens finds it comfortably inside the ~10% ceiling. Recency favors the registry's last few pins and misses the mid-file entry; frequency and llmlingua_style tie across most rows, since every package name and version string is locally unique, and land on earlier tied rows instead. The file path and pin-manifest id sit on their own header lines and are force-kept." +} diff --git a/eval/tasks/v04/typescript_holdout_015.json b/eval/tasks/v04/typescript_holdout_015.json new file mode 100644 index 0000000..c8b26c6 --- /dev/null +++ b/eval/tasks/v04/typescript_holdout_015.json @@ -0,0 +1,13 @@ +{ + "id": "typescript_holdout_015", + "family": "typescript_holdout", + "tier": "A", + "source": "tsc --watch log, host build-agent-09\nsession: watch-sess-6f1a02d9\n[10:02:01] File change detected. Starting incremental compilation...\n[10:02:01] Found 0 errors. Watching for file changes.\n[10:04:15] File change detected. Starting incremental compilation...\n[10:04:15] Found 0 errors. Watching for file changes.\n[10:06:30] File change detected. Starting incremental compilation...\n[10:06:30] Found 0 errors. Watching for file changes.\n[10:08:44] File change detected. Starting incremental compilation...\n[10:08:44] Found 0 errors. Watching for file changes.\n[10:11:02] File change detected. Starting incremental compilation...\n[10:11:02] Found 0 errors. Watching for file changes.\n[10:13:19] File change detected. Starting incremental compilation...\n[10:13:19] Found 0 errors. Watching for file changes.\n[10:15:37] File change detected. Starting incremental compilation...\n[10:15:37] Found 0 errors. Watching for file changes.\n[10:17:52] File change detected. Starting incremental compilation...\nsrc/payments/settleInvoice.ts(88,21): error TS2345: Draft is not assignable, want Invoice.\n[10:17:52] Found 1 error. Watching for file changes.\n[10:20:08] File change detected. Starting incremental compilation...\n[10:20:08] Found 1 error. Watching for file changes.\n[10:22:26] File change detected. Starting incremental compilation...\n[10:22:26] Found 1 error. Watching for file changes.\n[10:24:41] File change detected. Starting incremental compilation...\n[10:24:41] Found 1 error. Watching for file changes.\n[10:26:59] File change detected. Starting incremental compilation...\n[10:26:59] Found 1 error. Watching for file changes.\n[10:29:14] File change detected. Starting incremental compilation...\n[10:29:14] Found 1 error. Watching for file changes.\n[10:31:32] File change detected. Starting incremental compilation...\n[10:31:32] Found 1 error. Watching for file changes.\n[10:33:47] File change detected. Starting incremental compilation...\n[10:33:47] Found 1 error. Watching for file changes.\n[10:36:05] File change detected. Starting incremental compilation...\n[10:36:05] Found 1 error. Watching for file changes.\n", + "query": "Which module's change caused the previously clean incremental build to start failing, and what did the compiler say was wrong?", + "gold_answer": "src/payments/settleInvoice.ts(88,21): error TS2345: Draft is not assignable, want Invoice.", + "critical_atoms": [ + "build-agent-09", + "watch-sess-6f1a02d9" + ], + "notes": "A tsc --watch session log dominated by repeated 'File change detected...' / 'Found N error(s). Watching for file changes.' lines; the one substantive diagnostic (a payments-module type error) is sandwiched between them, and the query describes it functionally ('caused the ... build to start failing') without citing the file, error code, or message text, so bm25 finds no term overlap and falls back to file order. Recency is misled even harder: it favors the many identical post-break 'Found 1 error' lines that all sit later in the log, rather than the one line that actually explains the break. frequency and llmlingua_style, by contrast, both recover it -- being the log's only non-boilerplate line, it is also the most locally rare, which is exactly what those two selectors reward. The build-agent host and session id sit on the header line and are force-kept." +}