feat(v0.4-alpha): llmlingua_style selector + 11-family paired-task corpus

AGENTS.md

@@ -8,6 +8,7 @@ These instructions apply to every agent working in this repository.
 - Use the **AgentMemory MCP** at `http://localhost:3114` at the start of each task to retrieve relevant project context before inspecting or changing code.
 - If AgentMemory is unavailable, start `agentmemory --port 3114` as a background process, wait for port 3114 to accept connections, and retry the MCP call. If the global command is unavailable, use `npx -y @agentmemory/agentmemory --port 3114`. If port 3114 is unavailable or occupied by another service, choose a free port, start AgentMemory with `--port <port>`, set `AGENTMEMORY_URL=http://localhost:<port>` for the MCP process, and use that endpoint for the rest of the task.
 - Store durable project knowledge in AgentMemory after discovering important architecture, conventions, decisions, or non-obvious fixes. Do not store secrets, credentials, transient command output, or guesses.
+- Every AgentMemory `project` field (on `memory_save` and anywhere else it appears) MUST be a path-derived slug, never a bare display name like `"tokenfold"` — AgentMemory is a shared local service (`localhost:3114`) that other unrelated projects on this machine also write to, and a bare name is not collision-resistant. Derive the slug from the repo's absolute path: lowercase the drive letter, then `--`, then the rest of the path with every `\` (or `/`) replaced by `-`. For this repo's default clone location (`E:\Github\tokenfold`) that slug is `e--Github-tokenfold` — the same convention already used to name this project's local file-based memory directory (`~/.claude/projects/e--Github-tokenfold/memory/`), so the two memory systems stay identifiable by the same key. If the repo is cloned somewhere else, recompute the slug from that path instead of reusing this example verbatim.
 - If Ponytail remains unavailable, say so explicitly before continuing and use the closest available fallback.
 
 ## Git attribution

eval/run_baselines.py

@@ -13,9 +13,12 @@
 otherwise a byte/4 heuristic (identical to `tokenfold-core`'s fallback) is used and the report
 labels itself accordingly. Nothing here touches the shipped Rust/npm/Python runtime.
 
+Selectors: keep_all, forced_only, recency, frequency, bm25, llmlingua_style (a perplexity-free
+self-information proxy). Compressor baselines: deterministic-tokenfold (Rust CLI).
+
 Deliberately deferred (documented, not hidden — see `eval/tasks/v04/README.md`):
-  - RTK, RTK+tokenfold, deterministic-tokenfold (Rust CLI subprocess), LLMLingua-style, and the
-    unmodified Headroom Kompress-v2 achieved-token sweep are additional `SELECTORS` entries.
+  - RTK and RTK+tokenfold (external tool) and the unmodified Headroom Kompress-v2 achieved-token
+    sweep (needs the ML checkpoint) as additional baselines.
   - Real Tier-B public-repo corpora and project-disjoint train/test splits.
   - Structural (diff-hunk / JSON-container / AST) segmentation; v0.4-alpha segments by line.
   - An LLM judge for task success (the current scorer is a deterministic containment proxy).
@@ -146,12 +149,40 @@ def sel_bm25(units: list[str], query: str) -> list[float]:
     return scores
 
 
+def sel_llmlingua_style(units: list[str], query: str) -> list[float]:
+    """Perplexity-free LLMLingua-style proxy: keep high-information units, drop predictable /
+    redundant ones. Scores each unit by mean per-token self-information (surprisal,
+    `-log2 P(token)`) under a unigram model estimated from the document itself — a deterministic
+    stand-in for LLMLingua's small-LM token perplexity (the real method needs an LM at inference,
+    deferred: model-research.md keeps ML off the default path). Query-independent like `frequency`,
+    but an information-theoretic surprisal rather than a `1/df` heuristic, so boilerplate lines of
+    common tokens rank low and lines carrying rare/surprising content rank high."""
+    counts: dict[str, int] = {}
+    total = 0
+    for unit in units:
+        for tok in _tokens(unit):
+            counts[tok] = counts.get(tok, 0) + 1
+            total += 1
+    if total == 0:
+        return [0.0] * len(units)
+    scores = []
+    for unit in units:
+        toks = _tokens(unit)
+        if not toks:
+            scores.append(0.0)
+            continue
+        surprisal = sum(-math.log2(counts[t] / total) for t in toks) / len(toks)
+        scores.append(surprisal)
+    return scores
+
+
 SELECTORS = {
     "keep_all": sel_keep_all,
     "forced_only": sel_forced_only,
     "recency": sel_recency,
     "frequency": sel_frequency,
     "bm25": sel_bm25,
+    "llmlingua_style": sel_llmlingua_style,
 }
 
 

eval/tasks/v04/README.md

@@ -5,6 +5,17 @@ research (`docs/solution-design/model-research.md`). Everything here is **shadow
 measures deterministic keep/drop selectors against downstream tasks; no model is involved and
 nothing reaches a served path.
 
+## Coverage (77 Tier-A fixtures across 11 families)
+
+Spanning the required slices from model-research.md §Prerequisites: `log_qa`, `log_multi_service`
+(logs/tool QA), `diff_review`, `code_patch` (diff review / change localization), `code_build_error`
+(build/test failures), `json_schema`, `tool_call_json` (JSON/schema + tool calls),
+`long_context_needle` (long mixed context with an id/hash/path needle), `ccr_marker` (CCR
+reconstruction), and `rust_holdout` + `typescript_holdout` (the project-disjoint Rust/TS hard
+slices). Each family has 7 fixtures (`_001`, `_010`-`_015`). Every fixture is gate-validated and
+confirmed to *differentiate* selectors (at least one selector fails the task at the 25% ceiling,
+so the report is discriminating rather than trivially 1.0 everywhere).
+
 ## Fixture schema
 
 ```json
@@ -15,7 +26,8 @@ nothing reaches a served path.
   "source": "the raw captured text to compress",
   "query": "the downstream question the compressed context must still answer",
   "gold_answer": "substring that must survive for the task to be answerable",
-  "critical_atoms": ["ids/hashes/paths that must survive regardless of the selector"]
+  "critical_atoms": ["ids/hashes/paths that must survive regardless of the selector"],
+  "notes": "optional: why this fixture discriminates selectors (which ones fail and why)"
 }
 ```
 
@@ -26,6 +38,9 @@ nothing reaches a served path.
 - **`gold_answer`** should live in a unit that is *not* a critical atom, so whether the task is
   answerable genuinely depends on the selector + token budget. That is what differentiates the
   baselines (and, later, a learned selector) instead of every policy trivially scoring 1.0.
+- **`notes`** (optional, added from `_010` onward) documents the discrimination design intent per
+  fixture — which selectors are expected to fail the task at tight ceilings and why. The harness
+  ignores it; it exists for humans auditing/extending the corpus.
 
 ## Governance tiers (model-research.md §Prerequisites and Data)
 
@@ -53,9 +68,12 @@ the harness falls back to the same byte/4 heuristic as `tokenfold-core` and labe
 
 ## Baseline kinds: selectors vs. compressors
 
-- **Selectors** (`keep_all`, `forced_only`, `recency`, `frequency`, `bm25`) rank atomic units.
-  The harness force-keeps critical-atom units and enforces the exact token ceiling on them, so
-  100% critical-atom survival and the ceiling are guarantees.
+- **Selectors** (`keep_all`, `forced_only`, `recency`, `frequency`, `bm25`, `llmlingua_style`)
+  rank atomic units. `llmlingua_style` is a perplexity-free proxy — it ranks units by mean
+  per-token self-information (surprisal) under a document-derived unigram model, a deterministic
+  stand-in for LLMLingua's small-LM perplexity. The harness force-keeps critical-atom units and
+  enforces the exact token ceiling on them, so 100% critical-atom survival and the ceiling are
+  guarantees.
 - **Compressors** (`deterministic-tokenfold`) run a whole-pipeline best-effort compressor over
   the source — the harness does *not* force atoms through them, so their critical-atom survival
   and achieved ratio are **measured, not asserted**. `deterministic-tokenfold` shells out to the
@@ -67,9 +85,9 @@ the harness falls back to the same byte/4 heuristic as `tokenfold-core` and labe
 
 ## Deferred to later v0.4-alpha work (not hidden)
 
-- Remaining baselines: RTK and RTK+tokenfold (external tool), an LLMLingua-style selector, and
-  the unmodified Headroom Kompress-v2 achieved-token sweep. (`deterministic-tokenfold` is now
-  implemented as a compressor baseline — see above.)
+- Remaining baselines: RTK and RTK+tokenfold (external tool) and the unmodified Headroom
+  Kompress-v2 achieved-token sweep (needs the ML checkpoint). (`deterministic-tokenfold` and
+  `llmlingua_style` are now implemented — see above.)
 - Tier-B public-repo corpora with license/revision manifests; project-disjoint train/test splits
   and near-dedup across splits.
 - Structural segmentation (diff hunks, JSON containers, AST/code blocks) — v0.4-alpha segments by

eval/tasks/v04/ccr_marker_001.json

@@ -0,0 +1,12 @@
+{
+  "id": "ccr_marker_001",
+  "family": "ccr_marker",
+  "tier": "A",
+  "source": "$ artifactctl publish --repo prod ./build/app-2.14.0.tar.gz\n[10:22:01] connecting to registry registry.internal.example.com:8443\n[10:22:01] authenticated as ci-bot correlation-id=cid-4b8e2d1a\n[10:22:02] uploading layer 1/3 base-runtime size=41.2MB\n[10:22:03] uploading layer 2/3 dependencies size=88.7MB\n[10:22:05] uploading layer 3/3 application size=12.4MB\n[10:22:06] computing manifest digest from 3 layers\n[10:22:06] published digest sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08\n[10:22:06] assigned immutable tag prod-2.14.0-b317\n[10:22:07] registry quota remaining 214.5GB of 500GB\n[10:22:07] replication started to region eu-west-1\n[10:22:08] replication started to region ap-south-1\n[10:22:08] replication started to region us-east-2\n[10:22:09] audit event recorded status=OK\n[10:22:09] metrics flushed to telemetry pipeline\n[10:22:09] publish complete in 6.8s status=OK\n",
+  "query": "what sha256 manifest digest was published to the registry for this release?",
+  "gold_answer": "sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
+  "critical_atoms": [
+    "cid-4b8e2d1a"
+  ],
+  "notes": "CCR: the published digest must be reconstructable. bm25 keeps the digest line (shares 'sha256/digest/published/registry/manifest' with the query) so it reconstructs the value even at the 0.25 ceiling, while recency spends the budget on the later replication/audit/metrics lines and forced_only keeps only the marker line, so both drop the value. The audit marker cid-4b8e2d1a sits on the authentication line and is force-kept independent of the selector."
+}

eval/tasks/v04/ccr_marker_010.json

@@ -0,0 +1,13 @@
+{
+  "id": "ccr_marker_010",
+  "family": "ccr_marker",
+  "tier": "A",
+  "source": "$ flagctl audit export --window 24h --service edge-gateway\n[2026-03-11T09:01:00Z] flag-rollout adjust flag=checkout-v2 region=us-east percentage=5% actor=svc-flagbot\n[2026-03-11T09:02:00Z] flag-rollout adjust flag=dark-mode region=us-west percentage=62% actor=r-patel\n[2026-03-11T09:03:00Z] flag-rollout adjust flag=payments-retry region=eu-west percentage=18% actor=svc-flagbot\n[2026-03-11T09:04:00Z] change-request opened id=CR-88213 approver=ops-lead scope=flag-rollout\n[2026-03-11T09:05:00Z] flag-rollout adjust flag=onboarding-tour region=ap-south percentage=90% actor=m-santos\n[2026-03-11T09:06:00Z] flag-rollout adjust flag=comment-threads region=sa-east percentage=33% actor=svc-flagbot\n[2026-03-11T09:07:00Z] flag-rollout adjust flag=video-transcode region=us-east percentage=71% actor=i-choudhury\n[2026-03-11T09:08:00Z] flag-rollout adjust flag=price-cache region=us-west percentage=40% actor=svc-flagbot\n[2026-03-11T09:09:00Z] flag-rollout adjust flag=geo-routing region=eu-west percentage=15% actor=r-patel\n[2026-03-11T09:10:00Z] flag-rollout adjust flag=mobile-push region=ap-south percentage=55% actor=svc-flagbot\n[2026-03-11T09:11:00Z] flag-rollout adjust flag=email-digest region=sa-east percentage=27% actor=m-santos\n[2026-03-11T09:12:00Z] flag-rollout adjust flag=chat-widget region=us-east percentage=83% actor=svc-flagbot\n[2026-03-11T09:13:00Z] flag-rollout adjust flag=ab-test-banner region=us-west percentage=9% actor=i-choudhury\n[2026-03-11T09:14:00Z] flag-rollout adjust flag=inventory-sync region=eu-west percentage=46% actor=svc-flagbot\n[2026-03-11T09:15:00Z] flag-rollout adjust flag=refund-flow region=ap-south percentage=61% actor=r-patel\n[2026-03-11T09:16:00Z] flag-rollout adjust flag=loyalty-points region=sa-east percentage=24% actor=svc-flagbot\n[2026-03-11T09:17:00Z] flag-rollout adjust flag=address-autocomplete region=us-east percentage=77% actor=m-santos\n[2026-03-11T09:18:00Z] flag-rollout adjust flag=tax-calc-v3 region=us-west percentage=13% actor=svc-flagbot\n[2026-03-11T09:19:00Z] flag-rollout adjust flag=shipping-estimate region=eu-west percentage=59% actor=i-choudhury\n[2026-03-11T09:20:00Z] flag-rollout adjust flag=search-rerank region=us-east percentage=37% actor=svc-flagbot\n[2026-03-11T09:21:00Z] flag-rollout adjust flag=fraud-score-v2 region=ap-south percentage=68% actor=r-patel\n[2026-03-11T09:22:00Z] flag-rollout adjust flag=recs-carousel region=sa-east percentage=21% actor=svc-flagbot\n[2026-03-11T09:23:00Z] flag-rollout adjust flag=session-replay region=us-east percentage=44% actor=m-santos\n[2026-03-11T09:24:00Z] flag-rollout adjust flag=dark-launch-api region=us-west percentage=87% actor=svc-flagbot\n[2026-03-11T09:25:00Z] flag-rollout adjust flag=feature-gate-x region=eu-west percentage=30% actor=i-choudhury\n[2026-03-11T09:26:00Z] flag-rollout adjust flag=quota-throttle region=ap-south percentage=52% actor=svc-flagbot\n[2026-03-11T09:27:00Z] flag-rollout adjust flag=cdn-failover region=sa-east percentage=11% actor=r-patel\n[2026-03-11T09:28:00Z] flag-rollout adjust flag=image-lazyload region=us-east percentage=65% actor=svc-flagbot\n[2026-03-11T09:29:00Z] deployment batch verified batch-sha=1f3c9a7e2b804d5f status=signed\n[2026-03-11T09:30:00Z] flag-rollout adjust flag=coupon-stack region=us-west percentage=38% actor=m-santos\n[2026-03-11T09:31:00Z] flag-rollout adjust flag=subscription-pause region=eu-west percentage=74% actor=svc-flagbot\n[2026-03-11T09:32:00Z] flag-rollout adjust flag=wishlist-share region=ap-south percentage=20% actor=i-choudhury\n[2026-03-11T09:33:00Z] flag-rollout adjust flag=review-prompt region=sa-east percentage=48% actor=svc-flagbot\n[2026-03-11T09:34:00Z] flag-rollout adjust flag=live-chat-v2 region=us-east percentage=8% actor=r-patel\n[2026-03-11T09:35:00Z] flag-rollout adjust flag=map-cluster region=us-west percentage=56% actor=svc-flagbot\n[2026-03-11T09:36:00Z] flag-rollout adjust flag=search-typo-tolerant region=eu-west percentage=29% actor=m-santos\n[2026-03-11T09:37:00Z] flag-rollout adjust flag=checkout-express region=ap-south percentage=81% actor=svc-flagbot\n[2026-03-11T09:38:00Z] flag-rollout adjust flag=notif-batching region=sa-east percentage=43% actor=i-choudhury\n[2026-03-11T09:39:00Z] flag-rollout audit export complete rows=36\n",
+  "query": "what rollout percentage was set for the search-rerank flag?",
+  "gold_answer": "flag=search-rerank region=us-east percentage=37% actor=svc-flagbot",
+  "critical_atoms": [
+    "CR-88213",
+    "batch-sha=1f3c9a7e2b804d5f"
+  ],
+  "notes": "CCR: the flag's rollout percentage must be reconstructable at a tight ~10% budget. All 36 flag-adjustment rows share the identical 'flag-rollout adjust flag=... region=... percentage=...% actor=...' template, so recency (the row sits roughly mid-log, not near the tail), frequency and llmlingua_style (every row carries an equally rare flag-name/percentage/actor combination, so rare-token density does not single out this one) all fail to retain the search-rerank row even at the loosest 0.5 ceiling, and forced_only never carries any row past the two marker lines. bm25 is the only selector that retains the row at every ceiling down to 0.1, because 'search' and 'rerank' are the only tokens it shares with the query, and the one decoy row that could tie on 'search' ('search-typo-tolerant') does not also match 'rerank'. The change-request id=CR-88213 and deployment batch-sha=1f3c9a7e2b804d5f sit on separate audit lines and are force-kept independent of the selector."
+}