Skip to content

fix: close PII gaps in anonymized responses (STABLE360-13580) + perf for large responses - #5

Merged
anttiviljami merged 1 commit into
mainfrom
fix/STABLE360-13580-pii-gaps
Nov 4, 2026
Merged

anttiviljami merged 1 commit into
mainfrom
fix/STABLE360-13580-pii-gaps

Conversation

@anttiviljami

@anttiviljami anttiviljami commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Summary

Fixes PII that reached anonymized callers (MCP read profile, CLI/SDK with anonymize tokens) through entity-api responses — STABLE360-13580. Also makes anonymizeResponse 2.7–5.6× faster on large responses.

PII fixes

Leak Cause Fix
Activity caller.EpilotAuth (email, cognito:username, claims) caller handling only masked top-level string keys; caller is the request security context, i.e. nested apply embedded-user rules (user strategy) at every depth; add username / cognito:username / preferred_username
getThreadTimeline actor.email message-api creates activities via POST /v1/activity without caller → not isActivityLike → payload walked unscrubbed caller no longer required (payload suffices); plus catch-all: emails/phones/IBANs in strings outside recognized shapes are scrubbed (not inside entities, so data_classification: 'public' fields stay untouched)
Opportunity delivery_address / billing_address type relation_address not classified relation_address: 'address'; $relation_ref kept in address items
journey_data names/street fields not in schema only got pattern scrub apply anonymizeUnknown field heuristics to non-schema object fields (only when schema is known)
message/thread _title (= subject) no curated rule message:_title, thread:_title → redact (also covers thread.latest_message)
cc[].name MESSAGE_PARTY_KEYS was from/to add cc, bcc, reply_to
file filename no rule file:filename, file:_title → value pseudonym (same input → same pseudonym; shape already recognized by epilot-mcp's mask detector)

City / postal code stay visible by existing policy (ADDRESS_KEEP_KEYS).

Performance

Profiled on 0.2.3: ~30% in mapEntries + GC (entries/fromEntries arrays per node), ~20% in scrubText (4 regex passes on every string), 13% in loadSchemas (an await per node), ~12% re-classifying every attribute of every entity.

  • loadSchemas: one synchronous pass collects distinct (org, slug), resolved concurrently once each
  • mapEntries as a plain loop; walker / leaf mappers return unchanged subtrees as-is instead of copying (result may share objects with the input — read-only; input still never mutated)
  • per-call classification cache per (schema, attribute); Set lookups for key lists
  • scrubText prefilter: @ for emails, an exact-superset guard for the IBAN/phone patterns (same lookbehinds) — timestamps, numbers, labels skip all regexes. Fuzz test (20k strings) asserts the guard never skips a match
  • anonymizeUnknown memoizes per-key rules
Workload (median of 7) 0.2.3 this PR
searchEntities, 1000 contacts (2.4 MB) 210 ms 77 ms
activity feed, 100 × opportunity snapshot + 5 ops (6 MB) 507 ms 102 ms
entity with 200k-item json attribute (12.7 MB) 862 ms 153 ms

Output is byte-identical to the unoptimized build of this branch on all three workloads.

Release as 0.3.0 (curated defaults changed); entity-api bump: https://gitlab.com/e-pilot/product/entities/entity-api/-/merge_requests/1830 (^0.2.2 won't pick it up).

Test plan

  • npm test — 45 passed (stable360-13580.test.ts: ticket cases + public fields untouched; performance.test.ts: prefilter fuzz, schema resolved once, input not mutated)
  • tsc --noEmit
  • entity-api src/anonymization tests (20) pass against this build; new ticket tests fail on 0.2.2
  • After publish: lockfile bump in entity-api, re-run ticket repro via MCP read profile

🤖 Generated with Claude Code

https://claude.ai/code/session_01CRtqyNKiBCCdu3JSwr2zGp

- activity caller: apply embedded-user rules at every depth (EpilotAuth email, username claims, token)
- activities stored without a caller (e.g. message API thread activities) are recognized and scrubbed
- catch-all: emails/phones/IBANs in strings outside recognized shapes are scrubbed
- relation_address attributes are anonymized as addresses ($relation_ref kept)
- entity fields not in the schema (journey_data) get the non-schema field heuristics
- message cc/bcc/reply_to names pseudonymized; message/thread _title redacted
- file filename/_title pseudonymized

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRtqyNKiBCCdu3JSwr2zGp
@anttiviljami
anttiviljami merged commit 5216fba into main Nov 4, 2026
5 checks passed
@anttiviljami anttiviljami changed the title fix: close PII gaps in anonymized responses (STABLE360-13580) fix: close PII gaps in anonymized responses (STABLE360-13580) + perf for large responses Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant