Skip to content

About

AI-powered intelligence aggregation platform (internship) - ingestion pipelines, LLM clustering with HDBSCAN, PostgreSQL vector search, and a production Row-Level Security model with PII detection.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

AI Intelligence Aggregation Platform

Engineering internship · confidential social-impact media & technology organisation

Overview

Built an AI-powered intelligence aggregation platform for a social-impact media and technology organisation. The system ingests content from public APIs, processes it through a tiered Claude AI pipeline, clusters it into themes using HDBSCAN, and surfaces it via an admin dashboard and a community-facing feed. I implemented a full security model including Row-Level Security, PII detection using Microsoft Presidio, and a compliance audit trail, and documented the system across five technical documents spanning architecture, a user guide, and a reference implementation for future developers.

What I worked on

  • Ingestion: Python pipelines that pull and de-duplicate content from public news/data APIs, with retries and graceful error handling.
  • AI intelligence layer: sentence-embedding + HDBSCAN clustering into themes; an LLM step for labelling and summarisation with fallbacks when the model is unavailable; sentiment and source-credibility scoring.
  • Data & security: PostgreSQL with vector search; a production Row-Level Security model verified by an automated "anonymous surface" leak check; PII screening on user-submitted text before storage.
  • Admin dashboard: an operator UI to configure topics/keywords, manage sources, review flagged items, and read the intelligence feed.

Skills demonstrated

Python · PostgreSQL + vector search · sentence embeddings · HDBSCAN clustering · LLM/API integration with fallbacks · Row-Level Security · Microsoft Presidio (PII) · data-pipeline design · QA / verification scripting.

About the code samples

code_samples/ holds generic, self-contained illustrations of techniques I implemented (theme clustering, PII detection, a Row-Level-Security leak check, and source-credibility tiering). They demonstrate the approach and my coding style without reproducing the organisation's system design, data model, or vendor configuration. The production system, its documentation, and its data are confidential.

File Illustrates
clustering.py Embedding-based theme clustering with HDBSCAN plus keyword labelling
pii_detection.py Presidio and regex PII detection with redaction
rls_qa.py Anonymous-surface leak check for a Row-Level-Security policy
credibility.py Rule-based source-credibility tiering from domain signals

Each file guards its optional imports, so the modules load with nothing extra installed. To run them against your own data:

pip install -r requirements.txt

Then import the function named in the module docstring and call it.

Note: once the platform is live at its public URL, this can be updated to name the company and product and to describe the lead-engineer role, per the organisation's guidance.

License

The code samples in this repository are released under the MIT License; see LICENSE. The production platform and its documentation are not covered and remain confidential.

About

AI-powered intelligence aggregation platform (internship) - ingestion pipelines, LLM clustering with HDBSCAN, PostgreSQL vector search, and a production Row-Level Security model with PII detection.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages