feat(detection): add entity_label_examples for positive detection guidance

[memadi-nv]

Summary

Adds an implementation plan for issue #259, covering per-run entity-label examples for built-in and custom labels.

• Adds Detect.entity_label_examples: dict[str, list[str]] — per-run, per-label positive examples that steer GLiNER/LLM detection toward domain-specific formats (e.g. vendor-prefixed API keys) without mutating the process-global ENTITY_LABEL_EXAMPLES mapping.
• For a default label, configured examples are appended to its built-in examples and are active without any other config change. For a non-default label, the example key must also appear in an explicit entity_labels set (entity_labels=[*DEFAULT_ENTITY_LABELS, "vendor_api_key"] to keep all defaults and add one) — examples never implicitly activate a label, matching the explicit requirement in Expose per-label detection examples in the public configuration API #259.
• Validator prompts receive the full resolved example set (built-in + configured); augmenter prompts receive only configured examples, to avoid repeating the full built-in list on every row.
• Validation/normalization (merge, stable-dedupe, exclusion precedence, empty-effective-set detection) is centralized in resolve_entity_ontology() / normalize_entity_label_examples() (engine/detection/entity_label_examples.py), used by both Detect's pydantic validation and the engine's direct/distributed export callers, so the two layers can't drift.
• Adds effective_entity_labels / effective_entity_label_count to run telemetry (post-exclusion default-or-explicit label set), alongside the existing pre-exclusion entity_label_count; downstream wandb reporting prefers the new field and falls back to the old one for historical records. Configured example values are never recorded in telemetry, logs, or warning text.
• No change to substitution, evaluation, or AnonymizerResult/PreviewResult — this is detection-only, as scoped in the issue.

Test plan

• make test — full suite green (only a pre-existing, unrelated AF_UNIX path too long failure in a wandb ingress test on this machine)
• make format-check
• make typecheck
• New/updated coverage in tests/config/test_anonymizer_config.py (normalization, isolation, strict-membership validation, exclusion precedence, empty-set errors), tests/engine/test_detection_workflow.py and tests/engine/test_detection_config_serialization.py (validator/augmenter prompt content, GLiNER label propagation, cross-run isolation, export-path parity), tests/interface/test_anonymizer_interface.py, and tests/test_measurement.py / tests/tools/test_measurement_wandb_logging.py (telemetry fields, no example-value leakage, historical-record fallback)
• Docs updated: docs/concepts/detection.md, docs/concepts/choosing-a-strategy.md, docs/troubleshooting.md, docs/development/observability.md, and skills/anonymizer/SKILL.md (including its Detect(...) template)

Validation

• Commands run:
• Skipped checks or known failures:

Documentation and Artifacts

• Docs updated, or not needed
• If docs changed: make docs-build passes locally
• If tutorial sources changed: notebooks regenerated with make convert-notebooks
• If e2e, benchmark, or model-provider behavior changed: relevant validation is listed above

[asteier2026]

Looks good to me

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding correctness, security, or repository-rule issues identified.

Findings

1.  Detection limits are omitted ▶

Summary

This PR adds per-run positive examples for entity labels and consistently propagates the resolved detection ontology through runtime and exported workflows.

• Merges configured examples with built-in validator guidance without mutating global defaults.
• Sends only configured examples to augmentation prompts and filters strict-mode output to the effective label set.
• Validates label membership, normalization, exclusions, and empty effective scopes at the configuration boundary.
• Adds post-exclusion label telemetry while keeping configured example values out of normal measurements and logs.
• Documents provider exposure, evaluation boundaries, prompt-growth costs, and strict versus permissive detection behavior.
• The previous documentation finding is fully addressed: the current documentation explains both non-persistence for evaluation and per-chunk validator prompt costs.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    C[Detect configuration] --> R[Resolve effective ontology]
    B[Built-in examples] --> R
    R -->|Effective labels| G[GLiNER]
    R -->|Built-in plus configured examples| V[Validator]
    R -->|Configured examples only| A[Augmenter]
    G --> V
    V --> A
    A --> F[Filter exclusions and strict scope]
    F --> O[Detection results]
    R -->|Labels and counts only| T[Telemetry]

Loading

_{Reviews (6) · Last reviewed commit: "nit"}

[binaryaaron]

agent (review-pr): Two small observability/documentation suggestions and one nonblocking logging follow-up.

The shared resolver and execution/export paths look sound. I have two small changes to suggest:

1. Report the effective detection scope. With entity_labels=None, configuring examples for a new label activates it alongside the defaults, but measurement metadata and the scope log still report only the default set. For one added label, I reproduced 66 resolved labels versus 65 reported. Please report the effective count using the existing resolution logic, preserving existing metadata field meanings if needed.

2. Document the repeated prompt overhead. Each validator chunk receives the full resolved ontology, while the augmenter receives only configured examples. Please mention this alongside the examples documentation: reducing candidates per chunk does not bound example-related prompt overhead or total request cost. This overlaps Greptile’s comment.

One nonblocking follow-up: existing sync/async validation handlers and outer dependency paths log raw exception messages, which can contain sensitive response content. The new metadata-omission test doesn’t cover those paths. Please distinguish that test’s coverage from an end-to-end no-content-logging guarantee; broader sanitation can be tracked separately.

I also reviewed this against the proposed graph-native SDK design and found no architectural reason to delay the feature or add graph-specific abstractions here. The current result/evaluation behavior and run-local resolver can remain unchanged.

Validation: 251 targeted tests passed. No live provider tests were run.

[lipikaramaswamy]

What do we mean here? Is it that users should be discouraged from passing real values? Or something else? The telemetry note below makes sense to me

[memadi-nv]

Yes, since the examples are sent to model providers. More explained as a warning in the docs here.

[memadi-nv]

Thanks for the great reviews. I believe I addressed them all. Next steps after PR approval:

• Remove the plan from the PR to make the CI/ Check pass.
• Make NVSkills CI tests pass.