Conversation
…only) Adds eval/run_baselines.py — the 'baselines first' stage of the v0.4 learned-selector research (model-research.md). Runs deterministic keep/drop selectors (keep_all, forced_only, recency, frequency, bm25) over paired downstream tasks at exact provider-token ceilings and reports an achieved-token / task-score curve. No ML, nothing served — a learned selector will later plug into SELECTORS and be measured against these baselines. - Deterministic critical-atom forcing makes 100% critical-atom survival a structural guarantee (a hard gate), independent of the selector; the allocator re-tokenizes the full candidate each step (spec-faithful, non-additive) to respect the ceiling. - Optional tiktoken (o200k_base) for exact ceilings; byte/4 heuristic fallback otherwise, labeled in the report. Nothing touches the shipped Rust/npm/Python runtime. - 3 synthetic Tier-A fixtures (log_qa, diff_review, json_schema) with the gold answer in a non-forced, query-relevant unit so task success genuinely varies by selector. - --gate self-check (30 checks green): universal critical-atom survival, ceiling respected unless the forced floor exceeds budget, keep_all upper bound. - README documents the fixture schema, governance tiers (A/B/C), and deferred work (RTK/ LLMLingua/Kompress-v2 baselines, Tier-B public corpora, structural segmentation, LLM judge).
feat(v0.4-alpha): deterministic baseline benchmark harness (F-057, shadow-only)
The primary 'baseline to beat': runs the real tokenfold Rust CLI (compress --target-tokens)
as a whole-pipeline best-effort compressor, distinct from the unit-ranking selectors. It does
NOT get the harness's critical-atom forcing or hard ceiling, so its survival/ratio are measured
not asserted. Discovered via TOKENFOLD_BIN -> target/{release,debug} -> PATH; cleanly skipped
(n/a) when absent so the harness still runs build-less. On the current Tier-A fixtures it shows
the key gap the learned selector must close: lossless-safe (task+crit survival = 1.0) but
achieves only ~0.87 and can't reach the 0.5/0.25 budgets on low-repetition inputs.
Also makes task-answer containment whitespace-insensitive so a lossless reformat (JSON minify)
still counts as surviving (selector results unchanged), and gitignores Python __pycache__.
feat(v0.4-alpha): deterministic-tokenfold compressor baseline
A perplexity-free LLMLingua-style keep/drop selector: ranks units by mean per-token self-information (surprisal, -log2 P(token)) under a document-derived unigram model — a deterministic stand-in for LLMLingua's small-LM perplexity (the real method needs an LM at inference, deferred per model-research.md). Query-independent but information-theoretic, so boilerplate ranks low and rare/surprising content ranks high. Gate green (36 checks). Also refreshes the deferred lists: deterministic-tokenfold (PR #3) and llmlingua_style are no longer deferred; RTK and the Headroom Kompress-v2 sweep remain.
Adds 8 Tier-A fixtures spanning the model-research.md required slices: long_context_needle, code_build_error, code_patch, tool_call_json, log_multi_service, ccr_marker, and the project-disjoint rust_holdout + typescript_holdout hard slices. Authored + self-validated against the live gate and adversarially verified (8/8 confirmed) so each passes the gate AND genuinely differentiates selectors (>=1 selector fails at the 25% ceiling). Full 11-fixture gate: pass (132 checks). The richer report now discriminates: bm25 is the strongest deterministic selector (0.82 task @ 0.25), query-independent selectors are weak on query-driven tasks, and deterministic-tokenfold stays safe (1.0 task) but can't hit aggressive budgets (0.937) — the two gaps a learned selector must close.
…nly) Optional-dep seam so a locally-trained keep/drop selector plugs into the baseline harness as just another SELECTORS entry, with no model code in git. - run_baselines.py: _load_learned_selectors() discovers a gitignored local module (TOKENFOLD_LEARNED_MODULE, default learned.selector) and registers its LEARNED_SELECTORS; ML/module/weights absent -> cleanly skipped, exactly like tiktoken and the tokenfold CLI. Fails loud if a learned name would shadow a deterministic baseline. Report + summary label learned vs deterministic. - test_learned_selector_wiring.py: model-free test — skip-when-absent never raises, the real register path flows through build_report/run_gate, a baseline-name collision is rejected, and any learned score fn still passes the hard gates (critical-atom survival + ceiling stay structural). - .gitignore: /eval/learned/ so model/training code + weights never enter git. Contract per model-research.md: the learned selector emits scores only; the deterministic forcing + ceiling allocator + byte-copy assembly own the output. Bar to beat = bm25's 0.82 task @ 0.25 ratio.
Adds `log_field_fold` (v1.0.0), the log-line analogue of `json_field_fold`. Templated log lines (a shared skeleton varying only in digit/hex fields) are folded to a columnar form that emits each distinct line template once plus a compact per-line row of captured values, instead of repeating the skeleton on every line. Losslessly reversible: fold_log/unfold_log with a round_trips gate the pipeline enforces (validate_safety) before adopting the fold, so a fold that would ever lose data is rolled back rather than emitted. A proptest verifies the fold/unfold identity on arbitrary log-shaped input (CRLF, blank lines, missing final newline all preserved byte-for-byte). Unlike log_compaction (which only collapses identical *adjacent* lines), this compresses distinct-but-templated lines: `tokenfold compress --experimental` on a 20k-token templated service log reduces ~34% losslessly. Ships behind --experimental until its fidelity gate is green (same path json_field_fold and log_compaction took), out of Conservative, on plaintext/command-output, ordered before the lossy log_compaction. Wired into modes.rs (TransformId::LogFieldFold + ModeEntry), the pipeline dispatch + round-trip safety gate, and the mode_matrix.toml fixture. Also re-adds the /eval/learned/ gitignore rule for the local research scratch dir.
Add fixtures 010-015 for all 11 Tier-A families, each with a notes field documenting why the fixture discriminates selectors. Gate-validated (eval/run_baselines.py --gate passes, 924 rows checked). Also documents the AgentMemory project-slug convention already used by this repo's local memory directory.
feat(core): log_field_fold — reversible log-line columnar transform
feat(v0.4-alpha): llmlingua_style selector + 11-family paired-task corpus
…seam feat(v0.4-beta): learned-selector seam (model-free, shadow-only)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Promotes
developtomainahead of the v0.3.3 patch release. Carries:develop, never promoted): v0.4-alpha deterministic baseline harness +deterministic-tokenfoldcompressor baseline (shadow-only, F-057).log_field_fold— a losslessly reversible columnar transform for templated log lines (ships behind--experimental).llmlingua_styledeterministic selector baseline + growth of the v0.4-alpha paired-task corpus to 11 families / 77 fixtures (shadow-only research harness, nothing served).ci.ymlnow also runs ondevelop) and a couple of small housekeeping commits (.gitignoreentries).All individual PRs passed real CI (lint/test/security/coverage/node-api/golden-cross-platform/fidelity-smoke) before merging into
develop. Full local validation (fmt, clippy,cargo test --workspace,cargo audit,cargo deny, npm test, fidelity gate, v0.4 baselines gate) also passes ondevelopHEAD as of this PR.This is the first
develop→mainpromotion PR in this repo's history (PR #1 was a direct feature-branch→mainPR) — reviewed the same way as any other PR targetingmain.Test plan
pull_requestevents by this repo's own design) — logs opened directly, not just the checkmark