Found while debugging a real failure, 2026-08-05, using the answers #190 now stores.
What happened
An rdpapp item failed with the SEARCH text does not occur in the file as whole lines. #190 stores the implementer's answer, so the obvious move is to diff the SEARCH text against the file. The stored answer contained:
serde_json::from_str(r#"{"password":"[redacted]"}"#).unwrap();
assert_eq!(input.password.as_deref(), Some("not-for-logs"));The file really contains "not-for-logs" on both lines. [redacted] is this repository's own marker: redaction.PATTERNS's assignment rule matched "password":"not-for-logs" — source code in a test fixture — and rewrote it on the way into the store.
Redaction is applied only in store.append and audit.append, never to prompts, so the model was given the real file and the answer it sent was almost certainly correct at that point. But the record no longer says so.
The defect
Two things #190 promised are in tension, and nothing says which won:
Today the second silently wins, and the first is quietly false. The stored answer is not the answer. It is a redacted rendering of it, and there is no way to tell from the record which parts were rewritten or how many.
That is worse than it sounds for exactly the case where the record is most wanted: an exact-match edit failure is a question about characters, and the record has had characters changed.
Why it bites here specifically
Credential-shaped text is ordinary in source code. password:, api_key=, token: appear in fixtures, tests, config parsers and documentation. The assignment pattern cannot distinguish a real secret from a test fixture asserting how secrets are handled — and rdpapp, a credential vault, is full of the latter.
So the redactor is most aggressive precisely on the codebase where the answers matter most.
What is wanted
Not weakening #186 — the store must still never carry a credential. Options, roughly in increasing cost:
- Say that it happened, per answer.
answer_redacted already exists as a boolean. Make it a count, or record which patterns fired. A reader would then know the text is not verbatim, instead of reading a rewritten answer as if it were the model's. Cheapest, and removes the silent part of the failure, which is the worst part.
- Record a digest of the unredacted text alongside the redacted body. Anyone holding the original can then prove a match without the store ever holding the secret.
- Distinguish a value from an assertion about a value. Almost certainly not worth it — deciding whether
"password":"x" in a Rust test is a real credential is exactly the judgement a pattern cannot make, and guessing wrong in the permissive direction is unrecoverable.
I would do (1) now and argue about (2).
Blind spots
🤖 Generated with Claude Code
Found while debugging a real failure, 2026-08-05, using the answers #190 now stores.
What happened
An rdpapp item failed with
the SEARCH text does not occur in the file as whole lines. #190 stores the implementer's answer, so the obvious move is to diff the SEARCH text against the file. The stored answer contained:The file really contains
"not-for-logs"on both lines.[redacted]is this repository's own marker:redaction.PATTERNS's assignment rule matched"password":"not-for-logs"— source code in a test fixture — and rewrote it on the way into the store.Redaction is applied only in
store.appendandaudit.append, never to prompts, so the model was given the real file and the answer it sent was almost certainly correct at that point. But the record no longer says so.The defect
Two things #190 promised are in tension, and nothing says which won:
Today the second silently wins, and the first is quietly false. The stored answer is not the answer. It is a redacted rendering of it, and there is no way to tell from the record which parts were rewritten or how many.
That is worse than it sounds for exactly the case where the record is most wanted: an exact-match edit failure is a question about characters, and the record has had characters changed.
Why it bites here specifically
Credential-shaped text is ordinary in source code.
password:,api_key=,token:appear in fixtures, tests, config parsers and documentation. The assignment pattern cannot distinguish a real secret from a test fixture asserting how secrets are handled — and rdpapp, a credential vault, is full of the latter.So the redactor is most aggressive precisely on the codebase where the answers matter most.
What is wanted
Not weakening #186 — the store must still never carry a credential. Options, roughly in increasing cost:
answer_redactedalready exists as a boolean. Make it a count, or record which patterns fired. A reader would then know the text is not verbatim, instead of reading a rewritten answer as if it were the model's. Cheapest, and removes the silent part of the failure, which is the worst part."password":"x"in a Rust test is a real credential is exactly the judgement a pattern cannot make, and guessing wrong in the permissive direction is unrecoverable.I would do (1) now and argue about (2).
Blind spots
answer_redactedflag is stored, so the number is available and nobody has looked.🤖 Generated with Claude Code