Skip to content

Feature: Implement Generic Contact & Address Extractor #66

Description

@lauxenz

📋 User Story
As a Data Engineer
I want to programmatically crawl a foundation's website to find its legal or contact page and extract its address and generic email
So that we can automatically enrich our organization profiles with official contact data without violating GDPR or collecting personal information.
🔍 Proposed Workflow
Base URL Request: Fetch the landing page of the target website.
Link Extraction: Extract all internal links from the HTML.
Keyword Filtering: Filter the extracted links to find high-probability pages using a specific keyword matrix (e.g., impressum, legal, about, kontakt, contact).
Targeted Scraping: Request the filtered pages and scan the content for:
Physical Addresses (Organization location).
Generic Emails (Role-based addresses only).
Data Dropping (Guardrails): Actively ignore and drop phone numbers, individual names, and personalized email addresses.
🛠️ Technical Specifications & Guardrails

  1. Keyword Matrix for Subpage Discovery
    The script should match the href or link text against these variations (case-insensitive):
    impressum, legal-notice, legal
    about, about-us, ueber-uns
    kontakt, contact, contact-us
  2. Privacy Guardrail: Generic Emails Only
    To ensure compliance and prevent the ingestion of personal data, the email parser must validate the local part of the email against a strict whitelist of generic role prefixes.
    Allowed Prefixes: info@, kontakt@, contact@, office@, support@, mail@, sekretariat@, hello@
    Forbidden: Any email containing dots, specific names, or patterns resembling individual people (e.g., john.doe@stiftung.de, m.mueller@...).
  3. Exclusions
    NO Phone Numbers: Do not extract or parse regex patterns for telephone/fax numbers.
    NO Personal Data: Do not store names of board members, managing directors, or employees found in the Impressum.
    ✅ Acceptance Criteria
    [ ] AC 1: The script fetches a base URL and extracts internal links successfully.
    [ ] AC 2: It correctly prioritizes and visits relevant subpages based on the keyword list.
    [ ] AC 3: It parses and returns the physical address of the organization.
    [ ] AC 4: It extracts generic emails matching the whitelist while safely ignoring personalized emails (e.g., vorname.nachname@...).
    [ ] AC 5: No phone numbers or individual staff names are written to the final output object.
    💡 Expected Output Structure
    The function inside extract_impressum.py should return a clean dictionary like this:
{
    "organization_url": "https://example-foundation.org",
    "extracted_at": "2026-07-03T12:00:00Z",
    "source_page_used": "https://example-foundation.org/impressum",
    "generic_email": "info@example-foundation.org",
    "address": "Musterstraße 42, 80333 München, Germany"
}

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions