-
Notifications
You must be signed in to change notification settings - Fork 35
fix: Fix Redaction Prompt Corruption #178
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -17,11 +17,13 @@ Four public emit orchestrators — `writeRunInputs`, `writeTestRecord`, `writeTe | |||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| ## Secret Redaction | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| Every value that reaches a file crosses `redactResolvedValue(value, bindings)` (exported from `@finalrun/common`, defined in its `repoPlaceholders.ts`), which rewrites a resolved secret value back to its `${secrets.<key>}` placeholder: runner-log lines, a test record's `message` and `analysis`, step JSON (action text and url, analysis, thought, trace and timing detail strings), the synthetic failure message, and the copied device log. | ||||||||||||||||||||||||||||||||||
| Every value that reaches a file crosses `redactResolvedValue(value, bindings)` (exported from `@finalrun/common`, defined in its `repoPlaceholders.ts` — [/common/repo-placeholders.md](/common/repo-placeholders.md)), which rewrites a resolved secret value back to its `${secrets.<key>}` placeholder: runner-log lines, a test record's `message` and `analysis`, step JSON (action text and url, analysis, thought, trace and timing detail strings), the synthetic failure message, and the copied device log. | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| The environment snapshots carry reference forms only — `env.snapshot.yaml` mirrors the *config* `secrets` map (the `${ENV_VAR}` placeholder form) and `env.json` carries `secretReferences` (key plus env-var name). No resolved secret value is written anywhere beneath the run directory, and `reportWriter.test.ts` holds that as a whole-`runDir` sweep over every emitted file after a full lifecycle rather than as per-site assertions. | ||||||||||||||||||||||||||||||||||
| The environment snapshots carry reference forms only — `env.snapshot.yaml` mirrors the *config* `secrets` map (the `${ENV_VAR}` placeholder form) and `env.json` carries `secretReferences` (key plus env-var name). No **redactable** secret value is written anywhere beneath the run directory, and `reportWriter.test.ts` holds that as a whole-`runDir` sweep over every emitted file after a full lifecycle rather than as per-site assertions. | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| The contract is bounded in two ways worth naming, because the sweep's breadth invites reading it as global. It covers **write paths only** — the LLM prompt path is a separate seam, redacted per input inside `AIAgent` when runtime bindings are wired, and stdout INFO output crosses neither ([/cli/test-compiler.md](/cli/test-compiler.md)). And the sweep is a **byte-level** check on emitted files, so `tests/<testId>/screenshots/NNN.jpg` passes it while still showing a typed secret in its pixels: the report carries secret-bearing images even though it carries no secret strings. | ||||||||||||||||||||||||||||||||||
| "Redactable" is the operative word, and the sweep cannot show more than it: its fixture secret is a 17-character value, so what the sweep proves is bounded by the shared function's own bounds. A secret value shorter than `MIN_REDACTABLE_SECRET_LENGTH` (3) is never redacted, so a 1–2 character secret value **does** reach files beneath the run directory raw — an accepted residual, since substituting such a value would rewrite every incidental occurrence of that character sequence in a log line or step record instead. Conversely, an authored `${secrets.KEY}` token already present in report text — test steps like `Enter ${secrets.email}` carry them routinely — is consumed whole by the redaction alternation's last branch and emitted verbatim, never rewritten into a nested token. | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| The contract is bounded in two further ways worth naming, because the sweep's breadth invites reading it as global. It covers **write paths only** — the LLM prompt path is a separate seam, redacted per input inside `AIAgent` when runtime bindings are wired, and stdout INFO output crosses neither ([/cli/test-compiler.md](/cli/test-compiler.md)). And the sweep is a **byte-level** check on emitted files, so `tests/<testId>/screenshots/NNN.jpg` passes it while still showing a typed secret in its pixels: the report carries secret-bearing images even though it carries no secret strings. | ||||||||||||||||||||||||||||||||||
|
Comment on lines
+20
to
+26
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win Limit the guarantee to textual values. Line 20 says every value that reaches a file crosses Proposed wording-Every value that reaches a file crosses `redactResolvedValue(value, bindings)` ...
+Every textual value that reaches a file crosses `redactResolvedValue(value, bindings)` ...
-The sweep is a byte-level check on emitted files ...
+This is a string-redaction guarantee. The sweep is a byte-level check on emitted files ...📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| ## Device Log Artifact Flow | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Qualify the authored-token pass-through claim.
Line 24 promises verbatim output for every authored token.
docs/memory/common/repo-placeholders.mdLines 88-92 document an exact-token secret-value case where the value branch rewrites the token to another placeholder. Restrict “emitted verbatim” to cases where no eligible value branch matches at the token start.Proposed wording
📝 Committable suggestion
🤖 Prompt for AI Agents