feat(core): log_field_fold — reversible log-line columnar transform - #6
Merged
Conversation
Adds `log_field_fold` (v1.0.0), the log-line analogue of `json_field_fold`. Templated log lines (a shared skeleton varying only in digit/hex fields) are folded to a columnar form that emits each distinct line template once plus a compact per-line row of captured values, instead of repeating the skeleton on every line. Losslessly reversible: fold_log/unfold_log with a round_trips gate the pipeline enforces (validate_safety) before adopting the fold, so a fold that would ever lose data is rolled back rather than emitted. A proptest verifies the fold/unfold identity on arbitrary log-shaped input (CRLF, blank lines, missing final newline all preserved byte-for-byte). Unlike log_compaction (which only collapses identical *adjacent* lines), this compresses distinct-but-templated lines: `tokenfold compress --experimental` on a 20k-token templated service log reduces ~34% losslessly. Ships behind --experimental until its fidelity gate is green (same path json_field_fold and log_compaction took), out of Conservative, on plaintext/command-output, ordered before the lossy log_compaction. Wired into modes.rs (TransformId::LogFieldFold + ModeEntry), the pipeline dispatch + round-trip safety gate, and the mode_matrix.toml fixture. Also re-adds the /eval/learned/ gitignore rule for the local research scratch dir.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
log_field_fold, a losslessly-reversible columnar transform for templated log lines — the log-line analogue ofjson_field_fold.What & why
Most log lines share a fixed skeleton and vary only in a few fields (timestamps, ids, counts). This replaces each variable token (a digit run or
0x…hex literal) with a placeholder to derive a per-line template, then emits each distinct template once plus a compact per-line row of captured values — so the skeleton text is paid for once instead of on every line.Unlike
log_compaction(which only collapses identical adjacent lines, lossy-with-evidence), this compresses distinct-but-templated lines and is fully lossless.Lossless & safe
fold_log/unfold_log/round_trips; the pipeline (validate_safety) adopts the fold only whenround_tripsholds, so it can never lose data — a fold that wouldn't reverse is rolled back rather than emitted.proptestverifies the fold→unfold identity on arbitrary log-shaped input (CRLF, blank lines, and a missing final newline are all preserved byte-for-byte).Registration
modes.rs:TransformId::LogFieldFold+ModeEntry— experimental, out of Conservative,plain_text/command_output, ordered before the lossylog_compaction(lossless-before-lossy).pipeline.rs: forward dispatch + round-trip safety gate.tests/fixtures/mode_matrix.toml: mirror entry.Ships behind
--experimentaluntil its fidelity gate is green (the same pathjson_field_foldandlog_compactiontook).Testing
tokenfold compress --experimentalon a ~20k-token templated service log → ~34% reduction, losslessly.Also re-adds the
/eval/learned/gitignore rule for the local research scratch dir.