Skip to content

feat(core): log_field_fold — reversible log-line columnar transform - #6

Merged
snchimata merged 6 commits into
developfrom
feat/log-field-fold
Jul 27, 2026
Merged

feat(core): log_field_fold — reversible log-line columnar transform#6
snchimata merged 6 commits into
developfrom
feat/log-field-fold

Conversation

@snchimata

Copy link
Copy Markdown
Owner

Summary

Adds log_field_fold, a losslessly-reversible columnar transform for templated log lines — the log-line analogue of json_field_fold.

What & why

Most log lines share a fixed skeleton and vary only in a few fields (timestamps, ids, counts). This replaces each variable token (a digit run or 0x… hex literal) with a placeholder to derive a per-line template, then emits each distinct template once plus a compact per-line row of captured values — so the skeleton text is paid for once instead of on every line.

Unlike log_compaction (which only collapses identical adjacent lines, lossy-with-evidence), this compresses distinct-but-templated lines and is fully lossless.

Lossless & safe

  • fold_log / unfold_log / round_trips; the pipeline (validate_safety) adopts the fold only when round_trips holds, so it can never lose data — a fold that wouldn't reverse is rolled back rather than emitted.
  • A proptest verifies the fold→unfold identity on arbitrary log-shaped input (CRLF, blank lines, and a missing final newline are all preserved byte-for-byte).

Registration

  • modes.rs: TransformId::LogFieldFold + ModeEntry — experimental, out of Conservative, plain_text/command_output, ordered before the lossy log_compaction (lossless-before-lossy).
  • pipeline.rs: forward dispatch + round-trip safety gate.
  • tests/fixtures/mode_matrix.toml: mirror entry.

Ships behind --experimental until its fidelity gate is green (the same path json_field_fold and log_compaction took).

Testing

  • All 225 core tests + the full workspace (25 suites) pass, including the proptest round-trip.
  • CLI end-to-end: tokenfold compress --experimental on a ~20k-token templated service log → ~34% reduction, losslessly.

Also re-adds the /eval/learned/ gitignore rule for the local research scratch dir.

Adds `log_field_fold` (v1.0.0), the log-line analogue of `json_field_fold`.
Templated log lines (a shared skeleton varying only in digit/hex fields) are
folded to a columnar form that emits each distinct line template once plus a
compact per-line row of captured values, instead of repeating the skeleton on
every line.

Losslessly reversible: fold_log/unfold_log with a round_trips gate the pipeline
enforces (validate_safety) before adopting the fold, so a fold that would ever
lose data is rolled back rather than emitted. A proptest verifies the fold/unfold
identity on arbitrary log-shaped input (CRLF, blank lines, missing final newline
all preserved byte-for-byte).

Unlike log_compaction (which only collapses identical *adjacent* lines), this
compresses distinct-but-templated lines: `tokenfold compress --experimental` on
a 20k-token templated service log reduces ~34% losslessly. Ships behind
--experimental until its fidelity gate is green (same path json_field_fold and
log_compaction took), out of Conservative, on plaintext/command-output, ordered
before the lossy log_compaction.

Wired into modes.rs (TransformId::LogFieldFold + ModeEntry), the pipeline
dispatch + round-trip safety gate, and the mode_matrix.toml fixture. Also
re-adds the /eval/learned/ gitignore rule for the local research scratch dir.
@snchimata
snchimata merged commit 720924c into develop Jul 27, 2026
11 checks passed
@snchimata
snchimata deleted the feat/log-field-fold branch July 27, 2026 00:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant