Skip to content

feat: value-level PII detection for unclassified fields (STABLE360-13580) - #6

Merged
anttiviljami merged 1 commit into
mainfrom
feat/value-level-pii-detection
Nov 4, 2026
Merged

anttiviljami merged 1 commit into
mainfrom
feat/value-level-pii-detection

Conversation

@anttiviljami

Copy link
Copy Markdown
Member

Summary

Field-name and type rules cannot see personal data inside values of unclassified custom attributes, e.g. opportunity link_gebaeudeansicht holding a Google Maps link with the customer's street, house number and coordinates (found while verifying STABLE360-13580). The catch-all scrub of kept strings (scrubText) now also detects, replacing only the matched span:

  • URLs: components (path segments, query values, hash, userinfo) are decoded and scrubbed. Plain URLs without PII are returned byte-identical.
    • Map links (Google incl. maps.google.*, Apple, Bing, OpenStreetMap, goo.gl/maps / maps.app.goo.gl): place name / address query → street_xxxx pseudonym + postal code and city; place ids (cid, ftid, place_id, short-link ids) pseudonymized; Google's data= segment (feature / knowledge-graph ids, precise pins) dropped; coordinates rounded to 2 decimals (~1 km).
    • Phones are not matched in path segments (/030-1234567 is a slug far more often), but they are in query values.
  • German street addresses: capitalized street word ending in straße/strasse/str./weg/platz/allee/gasse/ring/damm (compound or standalone after up to two capitalized words) + 1-3 digit house number (12a, 12-14, 10/1). Postal code and city stay, consistent with address items.
  • IBAN checksum: IBAN matches are mod-97 validated (shorter prefixes ending at a space are tried, so DE89 … 00 BIC still masks the IBAN).
  • Gate: strings without @, / or a digit skip all detectors; street matching is behind a cheap suffix+digit prefilter.
  • data_classification: 'public' and system keys stay untouched, as before.

Classification fixes:

  • internal_user → user strategy (template created_by / updated_by embed user objects).
  • Curated _title defaults (text) for billing_account (= contact name), calendar_event, contract, lead, request, ticket.
  • data_classification: 'pii' now honours curated and type strategies (pii first_name → person, salutation → redact) instead of falling back to value, and a pii attribute is never kept via a keep override. This unblocks marking names pii in the entity-api blueprints (entity-api MR !1733).

False-positive tradeoffs

  • Compound nouns ending in a street suffix followed by a small number are masked (Arbeitsplatz 2, Lösungsweg 3 → street_…). Years don't match (house numbers ≤ 3 digits).
  • Streets without a suffix (An der Kirche 3, Platz der Republik 1) and names in free text are still not detected.
  • Map links become non-functional for the exact place (by design).

Test plan

  • npm test: 99 passed (53 new in value-detection.test.ts: real link_gebaeudeansicht value on an entity and in an activity payload/operations, all map providers, encoded URLs, streets, IBAN checksum, idempotency, false positives incl. product codes, OBIS, dates, prices, UUIDs, plain URLs)
  • perf case: 2,000 entities × 8 mixed strings well under budget (whole suite 76 ms)
  • npm run lint (tsc) and npm run build
  • Release: version bump commit after merge, as for 0.3.0. Not published.

🤖 Generated with Claude Code

…580)

The catch-all scrub of kept string values now also detects:
- PII inside URLs: components are decoded and scrubbed; map links (Google,
  Apple, Bing, OpenStreetMap, short links) lose the place name, place ids and
  Google's data= blob, and coordinates are rounded to 2 decimals (~1 km).
  Covers custom attributes like opportunity link_gebaeudeansicht.
- German street addresses (street word + 1-3 digit house number); postal
  code and city stay visible, as on address items.
- IBANs are now mod-97 checked (shorter valid prefixes tried), cutting false
  positives on IBAN-shaped codes.

Classification fixes:
- internal_user attributes use the user strategy.
- curated _title defaults for billing_account, calendar_event, contract,
  lead, request and ticket.
- data_classification 'pii' honours curated / type strategies (a pii
  first_name stays a person pseudonym) and is never kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anttiviljami
anttiviljami merged commit 306f227 into main Nov 4, 2026
5 checks passed
@pavish pavish mentioned this pull request Nov 9, 2026
1 task done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant