diff --git a/README.md b/README.md index 2c417cb..40f81a5 100644 --- a/README.md +++ b/README.md @@ -132,6 +132,7 @@ Explore our comprehensive Jupyter notebooks in the [`notebooks/`](./notebooks) d |----------|-------------| | [**Unstructured Data Classification**](https://github.com/IBM/READI/blob/main/notebooks/example-unstructured-data-classification.ipynb) | General overview of READI API for free-text processing | | [**Structured Data Classification**](https://github.com/IBM/READI/blob/main/notebooks/example-structured-data-classification.ipynb) | Working with tabular and structured datasets | +| [**Unstructured Data Masking**](https://github.com/IBM/READI/blob/main/notebooks/example-unstructured-masking.ipynb) | Applying masking actions (redaction, tagging, hash) after PII classification | --- @@ -208,7 +209,7 @@ READI is built on years of privacy research. Key publications: ## πŸ™ Acknowledgment -This project is partly supported by the Innovative Health Initiative Joint Undertaking (IHI JU) under grant agreement No. 101172997 – SEARCH. +This project is partly supported by the Innovative Health Initiative Joint Undertaking (IHI JU) under Grant Agreement No. 101172997 – SEARCH, and by the European Union’s Horizon research and innovation programme under Grant Agreement No. 101298664 - RegulAIze. --- diff --git a/notebooks/example-unstructured-masking.ipynb b/notebooks/example-unstructured-masking.ipynb new file mode 100644 index 0000000..e91462c --- /dev/null +++ b/notebooks/example-unstructured-masking.ipynb @@ -0,0 +1,235 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Masking Actions on Unstructured Data\n", + "\n", + "This notebook demonstrates how to apply **masking actions** to PII entities detected in unstructured text.\n", + "\n", + "The workflow has two steps:\n", + "1. **Classify** β€” detect PII spans in the text using `READIAnalyzer` with the `PII_NO_MODEL` detection type, which relies exclusively on regex-based and dictionary-based identifiers (no ML models required).\n", + "2. **Mask** β€” apply a masking action to each detected entity span using the functions from `risk_assessment.masking.actions`.\n", + "\n", + "The available masking actions are:\n", + "\n", + "| Action | Description |\n", + "|---|---|\n", + "| `tagging_factory()` | Replaces each unique value with a stable label, e.g. `EMAIL-1`, `EMAIL-2` |\n", + "| `tagging_with_hash` | Replaces each value with a type-prefixed hash suffix, e.g. `EMAIL-a3f2c` |\n", + "| `redact_factory()` | Replaces any entity with a fixed-width placeholder, e.g. `XXX` |\n", + "| `redact_size_preserving` | Replaces each character of the value with `X`, preserving length |\n", + "| `format_preserving_redact` | Replaces alphanumeric characters with `X`, preserving separators |\n", + "| `no_action` | Leaves the entity value unchanged |\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Detect PII entities\n", + "\n", + "We use `READIAnalyzer` with `DetectionType.PII_NO_MODEL`, which uses only regex and dictionary lookups β€” no spaCy or transformer models are loaded." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from risk_assessment.readi.analyzer import READIAnalyzer\n", + "\n", + "analyzer = READIAnalyzer(detection_type=READIAnalyzer.DetectionType.PII_NO_MODEL)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "text = \"\"\"\n", + "Dear support team,\n", + "\n", + "My name is Alice Johnson and I need help with my account.\n", + "You can reach me at alice.johnson@example.com or call me on +1-800-555-0199.\n", + "My SSN is 123-45-6789 and my credit card number is 4111 1111 1111 1111.\n", + "My IP address is 192.168.0.42 and my IBAN is GB29 NWBK 6016 1331 9268 19.\n", + "\n", + "Best regards,\n", + "Alice\n", + "\"\"\"\n", + "\n", + "entities = analyzer.detect(text)\n", + "\n", + "print(f\"Detected {len(entities)} entities:\\n\")\n", + "for entity in sorted(entities, key=lambda e: e.start):\n", + " span = text[entity.start : entity.end]\n", + " print(f\" [{entity.start}:{entity.end}] {entity.entity_type:20s} '{span}'\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Available masking actions\n", + "\n", + "All masking actions share the same signature: `(entity_type: str, entity_text: str) -> str`. \n", + "This makes it straightforward to compose them in a policy dictionary." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from risk_assessment.masking.actions import (\n", + " format_preserving_redact,\n", + " no_action,\n", + " redact_factory,\n", + " redact_size_preserving,\n", + " tagging_factory,\n", + " tagging_with_hash,\n", + ")\n", + "\n", + "# Show each action on a sample email address\n", + "sample_type = \"Email\"\n", + "sample_value = \"alice.johnson@example.com\"\n", + "\n", + "print(f\"Original value: '{sample_value}'\\n\")\n", + "print(f\" tagging_factory() -> '{tagging_factory()(sample_type, sample_value)}'\")\n", + "print(f\" tagging_with_hash -> '{tagging_with_hash(sample_type, sample_value)}'\")\n", + "print(f\" redact_factory() -> '{redact_factory()(sample_type, sample_value)}'\")\n", + "print(f\" redact_size_preserving -> '{redact_size_preserving(sample_type, sample_value)}'\")\n", + "print(f\" format_preserving_redact-> '{format_preserving_redact(sample_type, sample_value)}'\")\n", + "print(f\" no_action -> '{no_action(sample_type, sample_value)}'\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. Apply a masking policy\n", + "\n", + "A **masking policy** is a dictionary mapping entity types to masking actions. \n", + "Types not listed in the policy fall back to a `default` action.\n", + "\n", + "Here we apply the detected entities to the original text, processing spans from right to left so that earlier positions are not shifted by replacements." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from collections.abc import Callable\n", + "\n", + "\n", + "def apply_masking(\n", + " text: str,\n", + " entities: list,\n", + " policy: dict[str, Callable[[str, str], str]],\n", + " default: Callable[[str, str], str] = tagging_factory(),\n", + ") -> str:\n", + " \"\"\"Apply masking actions to detected entities in a text.\n", + "\n", + " Processes entities from right to left to preserve character positions.\n", + "\n", + " Args:\n", + " text: The original text.\n", + " entities: List of Entity objects returned by READIAnalyzer.detect().\n", + " policy: Mapping of entity_type -> masking action callable.\n", + " default: Fallback action for types not listed in the policy.\n", + "\n", + " Returns:\n", + " The masked text.\n", + " \"\"\"\n", + " for entity in sorted(entities, key=lambda e: e.start, reverse=True):\n", + " span = text[entity.start : entity.end]\n", + " action = policy.get(entity.entity_type, default)\n", + " replacement = action(entity.entity_type, span)\n", + " text = text[: entity.start] + replacement + text[entity.end :]\n", + " return text" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Define the masking policy\n", + "# - Emails and phone numbers get a stable tag (EMAIL-1, PHONE-1, …)\n", + "# - Credit cards are format-preserving redacted (separators kept)\n", + "# - IBANs are fully redacted to a fixed placeholder\n", + "# - IP addresses are size-preserving redacted\n", + "# - Everything else (SSN, names, etc.) falls back to the default: tagging_with_hash\n", + "\n", + "policy: dict[str, Callable[[str, str], str]] = {\n", + " \"Email\": tagging_factory(),\n", + " \"Phone\": tagging_factory(),\n", + " \"CreditCard\": format_preserving_redact,\n", + " \"IBAN\": redact_factory(symbol=\"*\", size=5),\n", + " \"IP\": redact_size_preserving,\n", + "}\n", + "\n", + "masked_text = apply_masking(text, entities, policy=policy, default=tagging_with_hash)\n", + "\n", + "print(\"=== Masked text ===\")\n", + "print(masked_text)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. Comparing masking strategies side by side\n", + "\n", + "Let's run the same text through three different default strategies to compare the results." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "strategies = {\n", + " \"Stable tagging \": tagging_factory(),\n", + " \"Hash tagging \": tagging_with_hash,\n", + " \"Format preserving\": format_preserving_redact,\n", + "}\n", + "\n", + "for label, action in strategies.items():\n", + " result = apply_masking(text, entities, policy={}, default=action)\n", + " print(f\"--- {label} ---\")\n", + " print(result)\n", + " print()" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "READI (3.12.12)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbformat_minor": 4, + "pygments_lexer": "ipython3" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/src/risk_assessment/masking/__init__.py b/src/risk_assessment/masking/__init__.py new file mode 100644 index 0000000..269d661 --- /dev/null +++ b/src/risk_assessment/masking/__init__.py @@ -0,0 +1,70 @@ +import json +import logging +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from pandas import DataFrame + +from risk_assessment.masking.actions import format_preserving_redact, no_action, tagging_factory + +logger = logging.getLogger(__file__) + + +@dataclass +class NLPReport: + extracted_text: str + entities: list[list[Any]] + + +def _load_entities(path: str | Path) -> NLPReport: + with open(path) as input: + return NLPReport(**json.load(input)) + + +def _default_transformation_policy() -> dict[str, Callable[[str, str], str]]: + return { + "IpAddress": format_preserving_redact, + "Port": format_preserving_redact, + "DateOfBirth": tagging_factory(), + "CreditCardNumber": format_preserving_redact, + "URL": tagging_factory(), + "ANP": no_action, + } + + +def cleanse_dataframe_field( + data: DataFrame, + field_name: Any, + nlp_report_directory: str | Path, + nlp_report_file_pattern: str = r"mytext-{}.json", + transformations: dict[str, Callable[[str, str], str]] = _default_transformation_policy(), + default: Callable[[str, str], str] = tagging_factory(), +) -> DataFrame: + for index in data.index: + logger.info("Processing %s", index) + report = _load_entities(Path(nlp_report_directory) / Path(nlp_report_file_pattern.format(index + 1))) + + row = data.iloc[[index]] + field = row[field_name] + text = field[index] + + if report.extracted_text != text: + raise ValueError(f"{report.extracted_text} -> {text}") + + for [annotation_text, begin, end, entity_type] in sorted(report.entities, key=lambda t: t[1], reverse=True): + entity_text = text[begin:end] + + if entity_text != annotation_text: + raise ValueError("Entity and text span strings do not match") + + transformed_text = (transformations[entity_type] if entity_type in transformations else default)( + entity_type, entity_text + ) + + text = text[0:begin] + transformed_text + text[end:] + + data.at[index, field_name] = text + + return data diff --git a/src/risk_assessment/masking/actions.py b/src/risk_assessment/masking/actions.py new file mode 100644 index 0000000..19b5a4f --- /dev/null +++ b/src/risk_assessment/masking/actions.py @@ -0,0 +1,49 @@ +from abc import ABC, abstractmethod +from collections import defaultdict +from collections.abc import Callable +from hashlib import sha256 + + +class MappingStorage(ABC): + @abstractmethod + def get_or_create(self, type_name: str, value: str) -> str: + pass + + +class InMemoryMappingStorage(MappingStorage): + _known_mapping: dict[str, dict[str, str]] = defaultdict(dict) + + def get_or_create(self, type_name: str, value: str) -> str: + type_dict: dict[str, str] = self._known_mapping[type_name] + + if value not in type_dict: + type_dict[value] = f"{type_name.upper()}-{len(type_dict) + 1}" + + return type_dict[value] + + +def tagging_factory(storage: MappingStorage = InMemoryMappingStorage()) -> Callable[[str, str], str]: + def _tagging_operator(entity_type: str, entity_text: str) -> str: + return storage.get_or_create(entity_type, entity_text) + + return _tagging_operator + + +def redact_factory(symbol: str = "X", size: int = 3) -> Callable[[str, str], str]: + return lambda x, y: symbol * size + + +def tagging_with_hash(entity_type: str, entity_text: str) -> str: + return f"{entity_type.upper()}-{sha256(entity_text.encode()).hexdigest()[-5:]}" + + +def redact_size_preserving(_: str, entity_text: str) -> str: + return "X" * len(entity_text) + + +def format_preserving_redact(_: str, enity_text: str) -> str: + return "".join(["X" if c.isalnum() else c for c in enity_text]) + + +def no_action(value: str, _: str) -> str: + return value diff --git a/tests/masking/__init__.py b/tests/masking/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/masking/data/data.csv b/tests/masking/data/data.csv new file mode 100644 index 0000000..be762fe --- /dev/null +++ b/tests/masking/data/data.csv @@ -0,0 +1,3 @@ +1,2,3,THIS IS WHAT I NEED TO PROCESS JOHN DOE +1,2,3,THIS IS ANOTHER ONE JAMES DOE +1,2,3,AND ANOTHER ONE JIM DOE AND JANE DOE diff --git a/tests/masking/data/data_with_headers.csv b/tests/masking/data/data_with_headers.csv new file mode 100644 index 0000000..642caa8 --- /dev/null +++ b/tests/masking/data/data_with_headers.csv @@ -0,0 +1,4 @@ +feature1,feature2,feature3,text +1,2,3,THIS IS WHAT I NEED TO PROCESS JOHN DOE +1,2,3,THIS IS ANOTHER ONE JAMES DOE +1,2,3,AND ANOTHER ONE JIM DOE AND JANE DOE diff --git a/tests/masking/data/markers/markers-1.json b/tests/masking/data/markers/markers-1.json new file mode 100644 index 0000000..3859372 --- /dev/null +++ b/tests/masking/data/markers/markers-1.json @@ -0,0 +1,6 @@ +{ + "extracted_text": "THIS IS WHAT I NEED TO PROCESS JOHN DOE", + "entities": [ + ["JOHN DOE", 31, 39, "PERSON"] + ] +} diff --git a/tests/masking/data/markers/markers-2.json b/tests/masking/data/markers/markers-2.json new file mode 100644 index 0000000..68eca03 --- /dev/null +++ b/tests/masking/data/markers/markers-2.json @@ -0,0 +1,6 @@ +{ + "extracted_text": "THIS IS ANOTHER ONE JAMES DOE", + "entities": [ + ["JAMES DOE", 20, 29, "PERSON"] + ] +} diff --git a/tests/masking/data/markers/markers-3.json b/tests/masking/data/markers/markers-3.json new file mode 100644 index 0000000..d5b4930 --- /dev/null +++ b/tests/masking/data/markers/markers-3.json @@ -0,0 +1,7 @@ +{ + "extracted_text": "AND ANOTHER ONE JIM DOE AND JANE DOE", + "entities": [ + ["JANE DOE", 28, 36, "PERSON"], + ["JIM DOE", 16, 23, "PERSON"] + ] +} diff --git a/tests/masking/test_actions.py b/tests/masking/test_actions.py new file mode 100644 index 0000000..a42759c --- /dev/null +++ b/tests/masking/test_actions.py @@ -0,0 +1,36 @@ +from risk_assessment.masking.actions import ( + format_preserving_redact, + redact_factory, + redact_size_preserving, + tagging_factory, + tagging_with_hash, +) + + +def test_tagging(): + assert tagging_with_hash("person", "FOO") != "FOO" + + +def test_format_preserving_redact(): + assert format_preserving_redact("FOO", "BAR") != "BAR" + + assert len(format_preserving_redact("FOO", "BAR")) == len("BAR") + assert format_preserving_redact("WHATEVER", "192.168.1.1") == "XXX.XXX.X.X" + + +def test_redact_size_preserving(): + assert redact_size_preserving("FOO", "THIS IS LONG") == "X" * len("THIS IS LONG") + + +def test_tagging_sequential(): + tagging = tagging_factory() + + assert tagging("foo", "BAR") == tagging("foo", "BAR") + assert tagging("fooooo", "BAR") == tagging("fooooo", "BAR") + assert tagging("foo", "BAR") != tagging("fooooo", "BAR") + + +def test_redaction(): + assert redact_factory()("", "VALUE") == "XXX" + assert redact_factory(size=1)("", "VALUE") == "X" + assert redact_factory(symbol="Y", size=5)("", "VALUE") == "YYYYY" diff --git a/tests/masking/test_nlp_masker.py b/tests/masking/test_nlp_masker.py new file mode 100644 index 0000000..9161c98 --- /dev/null +++ b/tests/masking/test_nlp_masker.py @@ -0,0 +1,32 @@ +from importlib.resources import files +from pathlib import Path + +import pandas as pd # type: ignore + +from risk_assessment.masking import cleanse_dataframe_field + + +def test_cleanse(): + res = files(__package__) / "data" / "data.csv" + with res.open() as iostream: + data = pd.read_csv(iostream, header=None) + + data_folder = Path(__file__).parent / "data" / "markers" + + cleansed = cleanse_dataframe_field(data, 3, data_folder, r"markers-{}.json") + + assert cleansed is not None + assert len(cleansed) == len(data) + + +def test_cleanse_with_headers(): + res = files(__package__).joinpath("data/data_with_headers.csv") + with res.open() as iostream: + data = pd.read_csv(iostream) + + data_folder = str(Path(Path(Path(__file__).parent, "data"), "markers").absolute()) + + cleansed = cleanse_dataframe_field(data, "text", data_folder, r"markers-{}.json") + + assert cleansed is not None + assert len(cleansed) == len(data)