fix(redact): mask credential values in harvest and synthesize saves - #114
Merged
Conversation
A sweep of the live store found a Firebase web API key in two harvested observations and an nsk- key in a third, both inherited by synthesized learnings. All five rows were masked in place and re-embedded; this stops the next one from being written. redact_secrets() matches the shape of a VALUE, not a mention: the sweep's fifteen 'password' hits were all prose, and one was the AWS documentation example key. Harvest redacts the summary before the dedup check so the stored and deduped strings match; synthesize redacts the learning before save. save_memory is deliberately untouched - agent-written memories are intentional, and rewriting them would corrupt real content. Mirrors the harvest-only TTL precedent from #36. Part of #113
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #113.
What
Credential values no longer reach the two unattended memory write paths.
neurostack.redact.redact_secrets(text) -> (text, kinds). Masks Google/Firebase API keys,nsk-keys, Stripe secret and restricted keys, GitHub, Slack, OpenAI, JWTs, PEM private-key blocks,Bearertokens, AWS access keys, andpassword=assignments. Prefixes are kept (AIza***REDACTED***) so the text still says which kind of credential was removed.harvest_sessionsredacts the summary before the dedup check, so the stored form and the deduped form are the same string.synthesize_observationsredacts the generated learning beforesave_memory— a learning inherits its members' text, which is exactly how the two live leaks propagated.record["redacted"]/plan["redacted"]).save_memoryis deliberately NOT touched. Agent-written memories are intentional; silently rewriting them would corrupt real content. Mirrors the harvest-only TTL precedent from #36.Precision over recall
False positives here permanently damage stored knowledge, so every pattern matches the shape of a value, not a mention. Cases taken verbatim from the live store that MUST survive:
No Firebase API key in the build at all.apiKey: import.meta.env.VITE_FIREBASE_API_KEY,The commit reverted to hardcoding a password, an unnecessary regression.AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE(AWS's documented example, allowlisted)DB_PASSWORD=${OPERATIONS_APP_PASSWORD}(env reference, not a value)An already-masked string is a no-op: the marker contains no character any pattern accepts.
Live data cleaned
Sweep of
/root/.local/share/neurostack/neurostack.dbon LXC 122 found two real secrets across five rows: a Firebase web key in memories 98, 163 and learning 1841, and a fullnsk-key in memory 217 and learning 1830. All masked in place with a***REDACTED***marker, thenneurostack backfill memoriesre-embedded all five from the masked content. Post-check: 0 hits inmemories,memories_archive, andmemories_fts.Gate
ruff check src/ tests/clean.pytest -q: 791 passed (778 + 13 new intests/test_redact.py, plus one added for the env-reference guard found in self-review). LSP diagnostics clean on all three touched files.