feat(detection): add regex entity detection

[lipikaramaswamy]

Related Issue

Fixes #262.

Plan Document

Hybrid regex entity detection plan

Summary

Adds regex-based entity detection as a first-class detection source alongside GLiNER and LLM detection.

• Enables built-in rules by default for requested credit/debit card, email, IPv4, IPv6, MAC address, and URL labels.
• Introduces one unified Detect.regex_rules configuration surface for built-in customization and user-defined rules.
• Supports optional local validators and per-rule LLM validation controls, defaulting LLM validation to enabled.
• Allows user-defined rules to replace built-in behavior for the same label without a separate overrides API.
• Adds safe bounded matching, stable content-derived rule IDs, deterministic source-aware merging, and DataDesigner serialization/plugin support.
• Documents the user experience and updates the bundled Anonymizer skill.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Refactoring
• CI, release, or contributor workflow update

Contributor Checklist

• PR title follows Conventional Commits (type(scope): description).
• Related issue is linked.
• Plan document is linked.
• Tests cover the new behavior.
• Public API and documentation changes are included.
• Commits include a DCO sign-off.

Validation

• Commands run:
  • TMPDIR=/tmp make test — 1312 passed, 1 warning.
  • make format-check — passed.
  • Targeted ty check for changed files — passed.
  • Full commit hooks in a clean uv environment — passed, including format, typecheck, lock, and copyright checks.
  • make docs-build from a clean worktree at this commit — passed.
• Skipped checks or known failures: None.

Documentation and Artifacts

• Docs updated, or not needed.
• If docs changed: make docs-build passes locally.
• If tutorial sources changed: notebooks regenerated and checked.
• If end-to-end behavior changed: relevant e2e checks completed, or not applicable.

[greptile-apps]

Confidence Score: 4/5

The PR is not yet safe to merge because the outstanding URL validator defect can miss valid fully qualified URLs ending in an explicit DNS root dot.

Findings

1.  Built-in Overrides User Rule ▶
2.  Overlapping Candidates Lose Tags ▶
3.  Root-Dotted URLs Are Rejected ▶
4.  Valid URLs Are Truncated ▶

Summary

Adds regex-based entity detection as a first-class source alongside GLiNER and LLM detection.

• Introduces configurable built-in and custom regex rules with bounded matching and local validators.
• Integrates regex candidates into validation, overlap resolution, workflow serialization, and DataDesigner plugins.
• Documents the public API and adds configuration, matching, merging, workflow, and serialization coverage.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    T[Input text] --> G[GLiNER candidates]
    T --> R[Regex matching and local validation]
    R --> RV[Candidates requiring LLM validation]
    R --> RA[Locally accepted candidates]
    G --> S[Seed candidate fan-in]
    RV --> S
    S --> V[Chunked LLM validation]
    V --> M[Source-aware merge]
    RA --> M
    M --> A[LLM augmentation]
    A --> F[Final overlap resolution]
    F --> E[Detected entities]

Loading

_{Reviews (20) · Last reviewed commit: "test(detection): cover excluded custom r..."}

[greptile-apps]

Built-in Overrides User Rule

When an overlapping built-in rule bypasses LLM validation while a higher-priority user rule takes the default LLM route, this final merge always prioritizes the accepted-route list. For identical spans with different labels, it therefore discards the validated user match in favor of the built-in match, violating the documented regex_user > regex_builtin > detector precedence and producing the wrong final label. Merge the routes using each candidate's source priority instead of argument order.

[lipikaramaswamy]

Fixed in 46907f8. Validation routes are now recombined by candidate provenance, preserving regex_user > regex_builtin > other detector sources regardless of validate_with_llm. Regression tests cover both route permutations and the finalization merge.

[greptile-apps]

Valid URLs Are Truncated

Blindly stripping every trailing closing delimiter truncates valid URLs containing balanced parentheses, such as https://en.wikipedia.org/wiki/Foo_(bar). The shortened value still passes URL validation and is emitted with an end offset before the legitimate ), so replacement operates on a malformed partial URL. Make punctuation trimming balance-aware rather than applying rstrip unconditionally.

[lipikaramaswamy]

Fixed in 46907f8. URL suffix trimming is now delimiter-balance-aware: balanced delimiters remain part of the URL, while unmatched prose closers are removed. Tests cover nested ASCII delimiters, CJK pairs, unmatched closers, IPv6 host brackets, and exact offsets.

[lipikaramaswamy]

/nvskills-ci

[andreatnvidia]

Some valid international email domains are missed because these character classes exclude Unicode combining marks. I reproduced this with x@उदाहरण.भारत and decomposed x@éxample.com; both domains encode successfully with IDNA but produce no match here. Could we allow IDNA-valid marks, possibly after normalization, and add a non-CJK IDN test?

[lipikaramaswamy]

Fixed in f0c3781. Domain labels now allow Unicode combining marks and are NFC-normalized before IDNA validation, while detected values and offsets remain unchanged. Added Devanagari and decomposed-Latin regression cases.

[andreatnvidia]

IPv4-embedded IPv6 addresses get split unexpectedly here. For example, ::ffff:192.0.2.128 is detected as ::ffff:192, which IPv6Address accepts as a different valid address. With IPv4 detection enabled, the dotted portion may be replaced while ::ffff: remains visible. Could we extend the pattern to consume dotted IPv4 tails and add an exact-span test?

[lipikaramaswamy]

Fixed in f0c3781. The IPv6 rule now consumes dotted IPv4 tails before considering the hexadecimal-only form. Added an exact value and offset regression test for ::ffff:192.0.2.128 with IPv4 detection enabled.

[andreatnvidia]

I think contextual zero-width rules can slip through this check. Patterns such as (?=CASE) and (?<=A) construct successfully, but detection later skips all their zero-length matches, so the rule silently does nothing. It would be safer to reject these during configuration validation, with a couple of lookahead and lookbehind tests.

[lipikaramaswamy]

Fixed in 52cdee3. RegexRule validation now rejects contextual zero-width matches, including the reported lookahead and lookbehind forms, while allowing lookarounds that consume a non-empty match. Runtime detection now fails explicitly with the rule ID and offset if an unprobed context-dependent zero-width match is encountered instead of silently skipping it. Tests cover both configuration cases, consuming lookarounds, and the runtime backstop.

[andreatnvidia]

The reported lookahead and lookbehind cases look good now. There is still a similar gap for \K: RegexRule(label="case_id", pattern=r"CASE\K") passes validation, then produces a (4, 4) span and fails the row at runtime. Could the sequence-width calculation reset when it encounters core.Keep, with a test covering this form?

[lipikaramaswamy]

Fixed in dd5bf2e. Sequence width analysis now resets accumulated width when it encounters regex._regex_core.Keep, so CASE\K is rejected during RegexRule configuration. The runtime zero-width guard remains in place, and the reported form is covered by the configuration regression test.

[andreatnvidia]

Small case-sensitivity mismatch here: the regex accepts WWW.example.com, but this prefix check only recognizes lowercase www.. That leaves the value without a scheme, and urlsplit rejects it. A case-insensitive check such as candidate.value.lower().startswith("www.") should cover it.

[lipikaramaswamy]

Fixed in f0c3781. The www prefix check is now case-insensitive, with a regression test for WWW.example.com/path.

[andreatnvidia]

The copyright check currently flags this file and skills/anonymizer/skill-card.md because they are missing the repository's SPDX headers. Adding the standard headers to both should clear the aggregate CI failure.

[lipikaramaswamy]

Fixed in f0c3781. Added the repository standard SPDX headers to BENCHMARK.md and skill-card.md. The full copyright check now passes.

[andreatnvidia]

f0c37811 added the headers, but the regeneration in 49ea1652 removed them again. The current Check job is failing on BENCHMARK.md and skill-card.md as a result. It looks like the generation step needs to preserve the SPDX headers when rewriting these files.

[andreatnvidia]

This link points to plans/detection-poles/multi-pole-detection.md, which isn't present in the PR. Since the broader framework is larger than this change, could we replace it with a follow-up issue or design doc? A short scope covering common detector output, configurable fan-in, and execution/failure contracts would make the intended direction clear without expanding this implementation.

[lipikaramaswamy]

Updated in 18f7e9e. Removed the missing multi-pole design reference so this document stands on its own as the implementation plan for issue #262.

[andreatnvidia]

One edge case to consider before resolving overlaps here: the winning candidate hasn't necessarily been contextually validated yet. A longer candidate can remove an overlapping fallback, then get dropped by the LLM, leaving neither candidate. Coalescing only exact same-label/span matches before validation, while retaining all their origins, would let final overlap resolution happen after the decisions are known.

[lipikaramaswamy]

Implemented in 6bd28e0. Pre-validation fan-in now coalesces only exact same-label/span duplicates, retains a deterministic origin chain, and preserves partial overlaps and same-span label alternatives as independently identified validation candidates. Final source-aware overlap resolution runs after keep/drop/reclass decisions. Regression coverage confirms that a shorter fallback survives when the longer candidate is dropped, while the longer candidate wins when both survive.

[andreatnvidia]

This feels like the right extension seam. It may be worth noting that GLiNER and regex are the first two producers of a future common detector-output contract, with fan-in eventually accepting a configured set of producer columns. The current source-specific implementation can stay, but the note would help keep the next detector from adding another parallel set of constants and merge paths.

[lipikaramaswamy]

Thanks, Andre. We decided to keep this plan focused on the implemented regex and GLiNER graph and removed the forward-looking multi-pole references, including the missing design-doc link. The current plan now documents the concrete two-producer contract: regex and GLiNER run as independent columns, COL_SEED_ENTITIES is their deterministic fan-in barrier, and final overlap resolution occurs after validation.

[andreatnvidia]

The overall direction makes sense, and regex plus GLiNER as independent Data Designer columns feels like a solid first step toward layered detection. I don't think this PR needs to build the full multi-detector framework. Keeping the public regex API and focusing here on correct two-source behavior seems like the right scope.

Could we capture a follow-up issue or design for the general framework? I'd include a shared candidate/outcome shape with multiple origins, one independently schedulable column per detector, generic fan-in, overlap resolution after validation, and per-detector preparation, resource, timeout, and failure contracts. A useful acceptance test would be adding a dummy third detector without changing fan-in, while producing identical results regardless of completion order. That follow-up could also replace the currently missing multi-pole design referenced by this plan.

[greptile-apps]

Overlapping Candidates Lose Tags

When regex and detector candidates overlap, this change preserves both in the validation candidate list. However, build_tagged_text skips any span that starts before the current cursor, so the tagged prompt omits some same-boundary or partially overlapping candidates even though they remain in the validation skeleton. The LLM may therefore make an incorrect keep, drop, or reclassification decision without seeing the corresponding tagged occurrence. The validation prompt needs a representation that shows every candidate unambiguously.

[lipikaramaswamy]

Fixed in ce66d82 without adding crossing markers to the readable tagged text. Every validation manifest entry now carries its existing context_before and context_after fields into the prompt, and the validator instructions treat that manifest as authoritative when an overlapping candidate cannot be rendered inline. Regression coverage confirms tagged_text stays clean while both overlapping candidates retain independently identifiable context.

[lipikaramaswamy]

Superseded the context-window approach in 6c8a877 to avoid per-candidate token duplication. Validation text now renders each connected overlap region exactly once as a neutral CANDIDATE_GROUP, and the prompt manifest adds only a compact group-id to candidate-id mapping. Non-overlapping entities retain their normal readable tags, and final overlap decisions remain independent. Tests cover nested/crossing spans, overlaps with deterministic accepted candidates, and overlap groups split across validation chunks.

[lipikaramaswamy]

/nvskills-ci

[binaryaaron]

agent (review-pr): Request changes — correctness fixes and integration work remain

local human-driven agent review found a few more bits, in addition to the outstanding comments:

1. Medium: URL validation accepts malformed hosts.
   regex_detection.py:_validate_url() accepts any IDNA-encodable host containing a dot, including http://-bad.com and http://999.999.999.999/path, with validate_with_llm=False. These false positives reduce detection precision and can cause unnecessary replacement of text. Define the intended hostname policy, validate IP-shaped hosts with ipaddress, and validate DNS-style labels before accepting them.

2. Low: callable serialization succeeds but produces an unusable configuration.
   RegexRule serializes a callable as module:qualname, but restored configurations cannot resolve that representation. Direct callables are documented as in-process only; serialization should fail clearly at that boundary instead of emitting a value that later fails as an unknown validator.

3. Previously reported: CASE\K still permits a zero-width match.
   Configuration accepts the pattern, then runtime matching produces (4, 4) and drops the row. Width analysis needs to account for core.Keep.

4. Previously reported: generated files still lack SPDX headers.
   Signing regeneration removed headers from BENCHMARK.md and skill-card.md, and the copyright check still fails. Fix the generator before restoring the headers. The restored files will have different hashes, so regenerate and sign skill.oms.sig after their final contents are settled.

The PR also conflicts with current main across detection code, configuration, tests, interface code, and skill artifacts. This is an integration gate. Conflict resolution must preserve both regex detection and the newer entity-exclusion behavior.

Local validation at 49ea1652: 1341 passed; format and type checks passed; copyright check failed on the two generated Markdown files.

Prompts for managed agents

Agents 1–3 own the fixes below; Agent 4 owns final integration. Each agent should preserve other agents’ changes and report the files changed, validation results, and any unresolved issues.

Agent 1 — URL validation

Fix built-in URL host validation in src/anonymizer/engine/detection/regex_detection.py. Reproduce acceptance of http://-bad.com and http://999.999.999.999/path, and establish the current underscore-host behavior, using BuiltinRegex(label="url", validate_with_llm=False). Define the intended hostname policy, validate IP-shaped hosts with ipaddress, and validate DNS-style labels consistently with IDNA support. Preserve valid Unicode domains, bracketed IPv6 hosts, ports, balanced delimiters, and exact offsets. Add focused positive and negative tests. Keep changes scoped to URL validation.

Agent 2 — regex configuration boundaries

Fix two configuration issues without enabling arbitrary imports. First, reject patterns such as r"CASE\K" during RegexRule validation by accounting for regex._regex_core.Keep semantics; retain the runtime zero-width backstop. Second, make JSON serialization of direct callable validators fail clearly, preserving the documented rule that only registered entry-point names are portable. Add tests covering RegexRule and Detect JSON round trips with registered names, serialization failures for top-level callables, and \K validation. Preserve validator trust boundaries.

Agent 3 — generated skill artifacts

Identify why NVSkills regeneration removes repository SPDX headers from skills/anonymizer/BENCHMARK.md and skills/anonymizer/skill-card.md. Fix the producing template or generation step so future runs retain the headers, then regenerate both files. Renew skills/anonymizer/skill.oms.sig only after the final bytes are stable, and run the copyright and skill-package checks. Coordinate with Agent 4 so any artifact changes during integration are followed by regeneration and signing.

Agent 4 — rebase and integration

Integrate the fixes and rebase PR 265 onto current main, resolving conflicts semantically. Preserve both regex detection and the newer entity-exclusion/denylist behavior through AnonymizerConfig, interface plumbing, detection workflow columns, postprocessing, and tests. Regenerate signed skill artifacts after source conflicts are resolved, and coordinate final signing with Agent 3. Run make test, make format-check, make typecheck, and make copyright-check. Report any remaining conflicts or failed checks without discarding either feature to complete the merge.

[lipikaramaswamy]

Addressed review items 1–3 in dd5bf2e:

• URL validation now distinguishes valid IP literals from invalid IP-shaped hosts and validates every IDNA-normalized DNS label, rejecting leading/trailing hyphens, underscores, empty labels, and malformed numeric addresses. Positive coverage preserves Unicode domains, IPv4, IPv6, and direct deterministic acceptance.
• Regex width validation now handles \\K by resetting sequence width and rejects CASE\\K at configuration time.
• Direct callable validators remain available in Python-mode configuration but now fail clearly on JSON serialization; registered validator names round-trip through both RegexRule and Detect.

Focused tests and the full 1,352-test suite pass; formatting, lint, and type checks pass.

[greptile-apps]

Root-Dotted URLs Are Rejected

Valid fully qualified URLs can include an explicit DNS root dot, such as https://example.com./path. The candidate regex passes this URL to the local validator, and urlsplit preserves the trailing dot. Splitting that hostname creates an empty final component, which fails _DNS_LABEL_RE, so the URL is discarded before contextual validation and sensitive URLs can be missed. Permit the terminal root label while retaining the malformed-label checks.

[lipikaramaswamy]

Merge-resolution note on excluded_entity_labels + regex detection:

• Exclusions are applied to the effective label set before GLiNER, built-in regex rule selection, and LLM prompts are constructed. Custom regex labels are added to the effective scope first, then exclusions are applied.
• Exclusions are applied again after locally accepted regex entities and LLM-validated entities are merged. This second boundary is a safety net for regex rules using validate_with_llm=False, so that route cannot bypass excluded_entity_labels.
• The same final filtering remains in place during finalization to protect against reclassification or any later detection route producing an excluded label.

Added integration coverage for excluded custom-regex labels and locally accepted regex matches at both the pre-augmentation and finalization boundaries. The merged suite passes locally (1,467 tests), and the Python 3.11/3.12/3.13 CI jobs pass.

@memadi tagging you for visibility since you authored the excluded-labels change. The merge keeps its original early filtering behavior and extends the safety-net filtering to the new regex routes.

[lipikaramaswamy]

/nvskills-ci

[lipikaramaswamy]

/nvskills-ci