Engineering internship · confidential social-impact media & technology organisation
Built an AI-powered intelligence aggregation platform for a social-impact media and technology organisation. The system ingests content from public APIs, processes it through a tiered Claude AI pipeline, clusters it into themes using HDBSCAN, and surfaces it via an admin dashboard and a community-facing feed. I implemented a full security model including Row-Level Security, PII detection using Microsoft Presidio, and a compliance audit trail, and documented the system across five technical documents spanning architecture, a user guide, and a reference implementation for future developers.
- Ingestion: Python pipelines that pull and de-duplicate content from public news/data APIs, with retries and graceful error handling.
- AI intelligence layer: sentence-embedding + HDBSCAN clustering into themes; an LLM step for labelling and summarisation with fallbacks when the model is unavailable; sentiment and source-credibility scoring.
- Data & security: PostgreSQL with vector search; a production Row-Level Security model verified by an automated "anonymous surface" leak check; PII screening on user-submitted text before storage.
- Admin dashboard: an operator UI to configure topics/keywords, manage sources, review flagged items, and read the intelligence feed.
Python · PostgreSQL + vector search · sentence embeddings · HDBSCAN clustering · LLM/API integration with fallbacks · Row-Level Security · Microsoft Presidio (PII) · data-pipeline design · QA / verification scripting.
code_samples/ holds generic, self-contained illustrations of techniques I
implemented (theme clustering, PII detection, a Row-Level-Security leak check, and
source-credibility tiering). They demonstrate the approach and my coding style
without reproducing the organisation's system design, data model, or vendor
configuration. The production system, its documentation, and its data are confidential.
| File | Illustrates |
|---|---|
clustering.py |
Embedding-based theme clustering with HDBSCAN plus keyword labelling |
pii_detection.py |
Presidio and regex PII detection with redaction |
rls_qa.py |
Anonymous-surface leak check for a Row-Level-Security policy |
credibility.py |
Rule-based source-credibility tiering from domain signals |
Each file guards its optional imports, so the modules load with nothing extra installed. To run them against your own data:
pip install -r requirements.txt
Then import the function named in the module docstring and call it.
Note: once the platform is live at its public URL, this can be updated to name the company and product and to describe the lead-engineer role, per the organisation's guidance.
The code samples in this repository are released under the MIT License; see LICENSE. The production platform and its documentation are not covered and remain confidential.