feat(v0.3): external RTK composition with staged receipts and secure CCR capture - #1
Merged
Merged
Conversation
Adds PipelineReport + PipelineStageReport and the additive CompressionReport.pipeline field (serde default, backward-compatible) per INTERFACES.md §1.9/§2. Separates savings and recoverability across composed stages so RTK's savings are never credited to tokenfold. F-055.
…CR capture F-054/F-055: new tokenfold-cli/src/rtk.rs resolves an external, optional rtk executable (TOKENFOLD_RTK_BIN override or PATH), version-checks it before the child runs, and falls open to the tokenfold-only path when missing/incompatible. wrap --rtk routes command output through RTK, skips tokenfold's overlapping command filter (no double-filtering), keeps mandatory redaction/generic-compress/never_worse, and never reruns a started child. --rtk-capture-raw opts into a permission-restricted per-run RTK_TEE_DIR tee, ingests the pre-RTK raw through the redaction/persistence gate (secret-matched bytes never stored), and deletes the transient capture. doctor --json/human gains an additive rtk object. Staged raw->RTK->tokenfold receipt via PipelineReport. 10 integration tests with a fake rtk cover the ENGINEERING v0.3 gate list.
…input Adversarial review found a receipt-honesty bug: on the wrap --rtk fail-open path (rtk missing/incompatible), the command filter pack still runs, so pipeline_input is smaller than raw. build_pipeline_report was passed raw.len() as the tokenfold stage's input, making its byte savings over-claim the filter's reduction while its token savings (from report.original_tokens, counted post-filter) did not — internally inconsistent. Rename post_rtk_bytes -> compress_input_bytes and pass pipeline_input.len() (equal to raw.len() on the RTK-ran path). CommandReport.raw_output_bytes still reports the true raw byte count. Adds a regression test driving fail-open + a trusted git-version filter that removes a deterministic 5 bytes; verified it fails without the fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the RTK half of v0.3 Phase 7 (F-054 + F-055): compose an external, optional RTK as a command-specific filtering stage before tokenfold's generic pipeline, with honest staged receipts and secure raw capture. RTK is never bundled and never mandatory.
Companion doc updates (Phase 7 status, npm/TS shipped in v0.3.1) live in
docs/solution-design/*, which is gitignored, so they are not part of this diff.What's in it
wrap --rtk— spawnsrtk <child argv...>(no shell) and treats RTK's output as the input tocompress(). Resolved viaTOKENFOLD_RTK_BINor a PATH scan;TOKENFOLD_RTK_DISABLEDturns it off.unavailable/incompatiblestage. Once RTK/the child is spawned, its output and exit code are authoritative — the command is never rerun.never_worsestill run (they live insidecompress()).raw → RTK → tokenfoldreceipt — newPipelineReport/PipelineStageReport(additive,#[serde(default)]). RTK savings live only in the RTK stage andtotal_saved_tokens, never in the top-levelsaved_tokens.--rtk-capture-raw(CCR) — a per-run, owner-onlyRTK_TEE_DIRtee (RAII-deleted after its bytes are read), ingested through the redaction/persistence gate. Secret-matched bytes are structurally never persisted (RetrievalStore::storerefuses them); missing/truncated captures are labeled unrecoverable, never falselyfull.doctor— additivertkobject (status/path/version/compatible/raw_capture/notes);missingis a warning, doctor still exits 0.Testing
crates/tokenfold-cli/tests/rtk.rs) driving a fakertk, covering the ENGINEERING v0.3 gate list: missing/incompatible fail-open, staged receipts, double-filter avoidance, nonzero-child propagation, and CCR complete/secret/absent capture.cargo fmt --check,cargo clippy --workspace -D warnings,cargo test --workspaceall green.Adversarial review
A 4-lens review (security / receipt-honesty / fail-open / contract) found one real bug: on the fail-open path the filter pack still runs, so the tokenfold stage was measuring bytes over
rawinstead of the post-filter compress input, over-crediting the filter's reduction. Fixed (compress_input_bytes = pipeline_input.len()) with a regression test verified to fail without the fix.