Add basic masking feature

README.md

@@ -132,6 +132,7 @@ Explore our comprehensive Jupyter notebooks in the [`notebooks/`](./notebooks) d
 |----------|-------------|
 | [**Unstructured Data Classification**](https://github.com/IBM/READI/blob/main/notebooks/example-unstructured-data-classification.ipynb) | General overview of READI API for free-text processing |
 | [**Structured Data Classification**](https://github.com/IBM/READI/blob/main/notebooks/example-structured-data-classification.ipynb) | Working with tabular and structured datasets |
+| [**Unstructured Data Masking**](https://github.com/IBM/READI/blob/main/notebooks/example-unstructured-masking.ipynb) | Applying masking actions (redaction, tagging, hash) after PII classification |
 
 ---
 
@@ -208,7 +209,7 @@ READI is built on years of privacy research. Key publications:
 
 ## 🙏 Acknowledgment
 
-This project is partly supported by the Innovative Health Initiative Joint Undertaking (IHI JU) under grant agreement No. 101172997 – SEARCH.
+This project is partly supported by the Innovative Health Initiative Joint Undertaking (IHI JU) under Grant Agreement No. 101172997 – SEARCH, and by the European Union’s Horizon research and innovation programme under Grant Agreement No. 101298664 - RegulAIze.
 
 ---
 

notebooks/example-unstructured-masking.ipynb

@@ -0,0 +1,235 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Masking Actions on Unstructured Data\n",
+    "\n",
+    "This notebook demonstrates how to apply **masking actions** to PII entities detected in unstructured text.\n",
+    "\n",
+    "The workflow has two steps:\n",
+    "1. **Classify** — detect PII spans in the text using `READIAnalyzer` with the `PII_NO_MODEL` detection type, which relies exclusively on regex-based and dictionary-based identifiers (no ML models required).\n",
+    "2. **Mask** — apply a masking action to each detected entity span using the functions from `risk_assessment.masking.actions`.\n",
+    "\n",
+    "The available masking actions are:\n",
+    "\n",
+    "| Action | Description |\n",
+    "|---|---|\n",
+    "| `tagging_factory()` | Replaces each unique value with a stable label, e.g. `EMAIL-1`, `EMAIL-2` |\n",
+    "| `tagging_with_hash` | Replaces each value with a type-prefixed hash suffix, e.g. `EMAIL-a3f2c` |\n",
+    "| `redact_factory()` | Replaces any entity with a fixed-width placeholder, e.g. `XXX` |\n",
+    "| `redact_size_preserving` | Replaces each character of the value with `X`, preserving length |\n",
+    "| `format_preserving_redact` | Replaces alphanumeric characters with `X`, preserving separators |\n",
+    "| `no_action` | Leaves the entity value unchanged |\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Detect PII entities\n",
+    "\n",
+    "We use `READIAnalyzer` with `DetectionType.PII_NO_MODEL`, which uses only regex and dictionary lookups — no spaCy or transformer models are loaded."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from risk_assessment.readi.analyzer import READIAnalyzer\n",
+    "\n",
+    "analyzer = READIAnalyzer(detection_type=READIAnalyzer.DetectionType.PII_NO_MODEL)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = \"\"\"\n",
+    "Dear support team,\n",
+    "\n",
+    "My name is Alice Johnson and I need help with my account.\n",
+    "You can reach me at alice.johnson@example.com or call me on +1-800-555-0199.\n",
+    "My SSN is 123-45-6789 and my credit card number is 4111 1111 1111 1111.\n",
+    "My IP address is 192.168.0.42 and my IBAN is GB29 NWBK 6016 1331 9268 19.\n",
+    "\n",
+    "Best regards,\n",
+    "Alice\n",
+    "\"\"\"\n",
+    "\n",
+    "entities = analyzer.detect(text)\n",
+    "\n",
+    "print(f\"Detected {len(entities)} entities:\\n\")\n",
+    "for entity in sorted(entities, key=lambda e: e.start):\n",
+    "    span = text[entity.start : entity.end]\n",
+    "    print(f\"  [{entity.start}:{entity.end}]  {entity.entity_type:20s}  '{span}'\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Available masking actions\n",
+    "\n",
+    "All masking actions share the same signature: `(entity_type: str, entity_text: str) -> str`.  \n",
+    "This makes it straightforward to compose them in a policy dictionary."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from risk_assessment.masking.actions import (\n",
+    "    format_preserving_redact,\n",
+    "    no_action,\n",
+    "    redact_factory,\n",
+    "    redact_size_preserving,\n",
+    "    tagging_factory,\n",
+    "    tagging_with_hash,\n",
+    ")\n",
+    "\n",
+    "# Show each action on a sample email address\n",
+    "sample_type = \"Email\"\n",
+    "sample_value = \"alice.johnson@example.com\"\n",
+    "\n",
+    "print(f\"Original value: '{sample_value}'\\n\")\n",
+    "print(f\"  tagging_factory()       -> '{tagging_factory()(sample_type, sample_value)}'\")\n",
+    "print(f\"  tagging_with_hash       -> '{tagging_with_hash(sample_type, sample_value)}'\")\n",
+    "print(f\"  redact_factory()        -> '{redact_factory()(sample_type, sample_value)}'\")\n",
+    "print(f\"  redact_size_preserving  -> '{redact_size_preserving(sample_type, sample_value)}'\")\n",
+    "print(f\"  format_preserving_redact-> '{format_preserving_redact(sample_type, sample_value)}'\")\n",
+    "print(f\"  no_action               -> '{no_action(sample_type, sample_value)}'\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Apply a masking policy\n",
+    "\n",
+    "A **masking policy** is a dictionary mapping entity types to masking actions.  \n",
+    "Types not listed in the policy fall back to a `default` action.\n",
+    "\n",
+    "Here we apply the detected entities to the original text, processing spans from right to left so that earlier positions are not shifted by replacements."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from collections.abc import Callable\n",
+    "\n",
+    "\n",
+    "def apply_masking(\n",
+    "    text: str,\n",
+    "    entities: list,\n",
+    "    policy: dict[str, Callable[[str, str], str]],\n",
+    "    default: Callable[[str, str], str] = tagging_factory(),\n",
+    ") -> str:\n",
+    "    \"\"\"Apply masking actions to detected entities in a text.\n",
+    "\n",
+    "    Processes entities from right to left to preserve character positions.\n",
+    "\n",
+    "    Args:\n",
+    "        text: The original text.\n",
+    "        entities: List of Entity objects returned by READIAnalyzer.detect().\n",
+    "        policy: Mapping of entity_type -> masking action callable.\n",
+    "        default: Fallback action for types not listed in the policy.\n",
+    "\n",
+    "    Returns:\n",
+    "        The masked text.\n",
+    "    \"\"\"\n",
+    "    for entity in sorted(entities, key=lambda e: e.start, reverse=True):\n",
+    "        span = text[entity.start : entity.end]\n",
+    "        action = policy.get(entity.entity_type, default)\n",
+    "        replacement = action(entity.entity_type, span)\n",
+    "        text = text[: entity.start] + replacement + text[entity.end :]\n",
+    "    return text"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Define the masking policy\n",
+    "# - Emails and phone numbers get a stable tag (EMAIL-1, PHONE-1, …)\n",
+    "# - Credit cards are format-preserving redacted (separators kept)\n",
+    "# - IBANs are fully redacted to a fixed placeholder\n",
+    "# - IP addresses are size-preserving redacted\n",
+    "# - Everything else (SSN, names, etc.) falls back to the default: tagging_with_hash\n",
+    "\n",
+    "policy: dict[str, Callable[[str, str], str]] = {\n",
+    "    \"Email\": tagging_factory(),\n",
+    "    \"Phone\": tagging_factory(),\n",
+    "    \"CreditCard\": format_preserving_redact,\n",
+    "    \"IBAN\": redact_factory(symbol=\"*\", size=5),\n",
+    "    \"IP\": redact_size_preserving,\n",
+    "}\n",
+    "\n",
+    "masked_text = apply_masking(text, entities, policy=policy, default=tagging_with_hash)\n",
+    "\n",
+    "print(\"=== Masked text ===\")\n",
+    "print(masked_text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Comparing masking strategies side by side\n",
+    "\n",
+    "Let's run the same text through three different default strategies to compare the results."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "strategies = {\n",
+    "    \"Stable tagging   \": tagging_factory(),\n",
+    "    \"Hash tagging     \": tagging_with_hash,\n",
+    "    \"Format preserving\": format_preserving_redact,\n",
+    "}\n",
+    "\n",
+    "for label, action in strategies.items():\n",
+    "    result = apply_masking(text, entities, policy={}, default=action)\n",
+    "    print(f\"--- {label} ---\")\n",
+    "    print(result)\n",
+    "    print()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "READI (3.12.12)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbformat_minor": 4,
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

src/risk_assessment/masking/__init__.py

@@ -0,0 +1,70 @@
+import json
+import logging
+from collections.abc import Callable
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+from pandas import DataFrame
+
+from risk_assessment.masking.actions import format_preserving_redact, no_action, tagging_factory
+
+logger = logging.getLogger(__file__)
+
+
+@dataclass
+class NLPReport:
+    extracted_text: str
+    entities: list[list[Any]]
+
+
+def _load_entities(path: str | Path) -> NLPReport:
+    with open(path) as input:
+        return NLPReport(**json.load(input))
+
+
+def _default_transformation_policy() -> dict[str, Callable[[str, str], str]]:
+    return {
+        "IpAddress": format_preserving_redact,
+        "Port": format_preserving_redact,
+        "DateOfBirth": tagging_factory(),
+        "CreditCardNumber": format_preserving_redact,
+        "URL": tagging_factory(),
+        "ANP": no_action,
+    }
+
+
+def cleanse_dataframe_field(
+    data: DataFrame,
+    field_name: Any,
+    nlp_report_directory: str | Path,
+    nlp_report_file_pattern: str = r"mytext-{}.json",
+    transformations: dict[str, Callable[[str, str], str]] = _default_transformation_policy(),
+    default: Callable[[str, str], str] = tagging_factory(),
+) -> DataFrame:
+    for index in data.index:
+        logger.info("Processing %s", index)
+        report = _load_entities(Path(nlp_report_directory) / Path(nlp_report_file_pattern.format(index + 1)))
+
+        row = data.iloc[[index]]
+        field = row[field_name]
+        text = field[index]
+
+        if report.extracted_text != text:
+            raise ValueError(f"{report.extracted_text} -> {text}")
+
+        for [annotation_text, begin, end, entity_type] in sorted(report.entities, key=lambda t: t[1], reverse=True):
+            entity_text = text[begin:end]
+
+            if entity_text != annotation_text:
+                raise ValueError("Entity and text span strings do not match")
+
+            transformed_text = (transformations[entity_type] if entity_type in transformations else default)(
+                entity_type, entity_text
+            )
+
+            text = text[0:begin] + transformed_text + text[end:]
+
+        data.at[index, field_name] = text
+
+    return data

src/risk_assessment/masking/actions.py

@@ -0,0 +1,49 @@
+from abc import ABC, abstractmethod
+from collections import defaultdict
+from collections.abc import Callable
+from hashlib import sha256
+
+
+class MappingStorage(ABC):
+    @abstractmethod
+    def get_or_create(self, type_name: str, value: str) -> str:
+        pass
+
+
+class InMemoryMappingStorage(MappingStorage):
+    _known_mapping: dict[str, dict[str, str]] = defaultdict(dict)
+
+    def get_or_create(self, type_name: str, value: str) -> str:
+        type_dict: dict[str, str] = self._known_mapping[type_name]
+
+        if value not in type_dict:
+            type_dict[value] = f"{type_name.upper()}-{len(type_dict) + 1}"
+
+        return type_dict[value]
+
+
+def tagging_factory(storage: MappingStorage = InMemoryMappingStorage()) -> Callable[[str, str], str]:
+    def _tagging_operator(entity_type: str, entity_text: str) -> str:
+        return storage.get_or_create(entity_type, entity_text)
+
+    return _tagging_operator
+
+
+def redact_factory(symbol: str = "X", size: int = 3) -> Callable[[str, str], str]:
+    return lambda x, y: symbol * size
+
+
+def tagging_with_hash(entity_type: str, entity_text: str) -> str:
+    return f"{entity_type.upper()}-{sha256(entity_text.encode()).hexdigest()[-5:]}"
+
+
+def redact_size_preserving(_: str, entity_text: str) -> str:
+    return "X" * len(entity_text)
+
+
+def format_preserving_redact(_: str, enity_text: str) -> str:
+    return "".join(["X" if c.isalnum() else c for c in enity_text])
+
+
+def no_action(value: str, _: str) -> str:
+    return value

tests/masking/data/data.csv

@@ -0,0 +1,3 @@
+1,2,3,THIS IS WHAT I NEED TO PROCESS JOHN DOE
+1,2,3,THIS IS ANOTHER ONE JAMES DOE
+1,2,3,AND ANOTHER ONE JIM DOE AND JANE DOE

tests/masking/data/data_with_headers.csv

@@ -0,0 +1,4 @@
+feature1,feature2,feature3,text
+1,2,3,THIS IS WHAT I NEED TO PROCESS JOHN DOE
+1,2,3,THIS IS ANOTHER ONE JAMES DOE
+1,2,3,AND ANOTHER ONE JIM DOE AND JANE DOE

tests/masking/data/markers/markers-1.json

@@ -0,0 +1,6 @@
+{
+    "extracted_text": "THIS IS WHAT I NEED TO PROCESS JOHN DOE",
+    "entities": [
+        ["JOHN DOE", 31, 39, "PERSON"]
+    ]
+}

tests/masking/data/markers/markers-2.json

@@ -0,0 +1,6 @@
+{
+    "extracted_text": "THIS IS ANOTHER ONE JAMES DOE",
+    "entities": [
+        ["JAMES DOE", 20, 29, "PERSON"]
+    ]
+}