📋 User Story
As a Data Engineer
I want to programmatically crawl a foundation's website to find its legal or contact page and extract its address and generic email
So that we can automatically enrich our organization profiles with official contact data without violating GDPR or collecting personal information.
🔍 Proposed Workflow
Base URL Request: Fetch the landing page of the target website.
Link Extraction: Extract all internal links from the HTML.
Keyword Filtering: Filter the extracted links to find high-probability pages using a specific keyword matrix (e.g., impressum, legal, about, kontakt, contact).
Targeted Scraping: Request the filtered pages and scan the content for:
Physical Addresses (Organization location).
Generic Emails (Role-based addresses only).
Data Dropping (Guardrails): Actively ignore and drop phone numbers, individual names, and personalized email addresses.
🛠️ Technical Specifications & Guardrails
- Keyword Matrix for Subpage Discovery
The script should match the href or link text against these variations (case-insensitive):
impressum, legal-notice, legal
about, about-us, ueber-uns
kontakt, contact, contact-us
- Privacy Guardrail: Generic Emails Only
To ensure compliance and prevent the ingestion of personal data, the email parser must validate the local part of the email against a strict whitelist of generic role prefixes.
Allowed Prefixes: info@, kontakt@, contact@, office@, support@, mail@, sekretariat@, hello@
Forbidden: Any email containing dots, specific names, or patterns resembling individual people (e.g., john.doe@stiftung.de, m.mueller@...).
- Exclusions
NO Phone Numbers: Do not extract or parse regex patterns for telephone/fax numbers.
NO Personal Data: Do not store names of board members, managing directors, or employees found in the Impressum.
✅ Acceptance Criteria
[ ] AC 1: The script fetches a base URL and extracts internal links successfully.
[ ] AC 2: It correctly prioritizes and visits relevant subpages based on the keyword list.
[ ] AC 3: It parses and returns the physical address of the organization.
[ ] AC 4: It extracts generic emails matching the whitelist while safely ignoring personalized emails (e.g., vorname.nachname@...).
[ ] AC 5: No phone numbers or individual staff names are written to the final output object.
💡 Expected Output Structure
The function inside extract_impressum.py should return a clean dictionary like this:
{
"organization_url": "https://example-foundation.org",
"extracted_at": "2026-07-03T12:00:00Z",
"source_page_used": "https://example-foundation.org/impressum",
"generic_email": "info@example-foundation.org",
"address": "Musterstraße 42, 80333 München, Germany"
}
📋 User Story
As a Data Engineer
I want to programmatically crawl a foundation's website to find its legal or contact page and extract its address and generic email
So that we can automatically enrich our organization profiles with official contact data without violating GDPR or collecting personal information.
🔍 Proposed Workflow
Base URL Request: Fetch the landing page of the target website.
Link Extraction: Extract all internal links from the HTML.
Keyword Filtering: Filter the extracted links to find high-probability pages using a specific keyword matrix (e.g., impressum, legal, about, kontakt, contact).
Targeted Scraping: Request the filtered pages and scan the content for:
Physical Addresses (Organization location).
Generic Emails (Role-based addresses only).
Data Dropping (Guardrails): Actively ignore and drop phone numbers, individual names, and personalized email addresses.
🛠️ Technical Specifications & Guardrails
The script should match the href or link text against these variations (case-insensitive):
impressum, legal-notice, legal
about, about-us, ueber-uns
kontakt, contact, contact-us
To ensure compliance and prevent the ingestion of personal data, the email parser must validate the local part of the email against a strict whitelist of generic role prefixes.
Allowed Prefixes: info@, kontakt@, contact@, office@, support@, mail@, sekretariat@, hello@
Forbidden: Any email containing dots, specific names, or patterns resembling individual people (e.g., john.doe@stiftung.de, m.mueller@...).
NO Phone Numbers: Do not extract or parse regex patterns for telephone/fax numbers.
NO Personal Data: Do not store names of board members, managing directors, or employees found in the Impressum.
✅ Acceptance Criteria
[ ] AC 1: The script fetches a base URL and extracts internal links successfully.
[ ] AC 2: It correctly prioritizes and visits relevant subpages based on the keyword list.
[ ] AC 3: It parses and returns the physical address of the organization.
[ ] AC 4: It extracts generic emails matching the whitelist while safely ignoring personalized emails (e.g., vorname.nachname@...).
[ ] AC 5: No phone numbers or individual staff names are written to the final output object.
💡 Expected Output Structure
The function inside extract_impressum.py should return a clean dictionary like this:
{ "organization_url": "https://example-foundation.org", "extracted_at": "2026-07-03T12:00:00Z", "source_page_used": "https://example-foundation.org/impressum", "generic_email": "info@example-foundation.org", "address": "Musterstraße 42, 80333 München, Germany" }