🏥 Pharma Vigilance — Healthcare & Pharmaceutical Monitoring Framework

A comprehensive monitoring framework for drug safety, adverse events, clinical trials, regulatory actions, and healthcare policy — built for researchers, pharmacovigilance teams, and public health analysts.

---

⚕️ Features

🕷️ 11 Specialized Data Spiders

#	Spider	Source
1	FDA FAERS	FDA Adverse Event Reporting System
2	ClinicalTrials.gov	ClinicalTrials.gov
3	PubMed	PubMed / NCBI
4	EMA	European Medicines Agency
5	MedWatch	FDA MedWatch Safety Alerts
6	WHO VigiBase	WHO VigiBase
7	FDA Recalls	FDA Enforcement Reports
8	Drug Prices	Medicare / Medicaid / GoodRx
9	Healthcare Policy	Federal Register / HHS / CMS
10	Medical Devices	FDA MAUDE Database
11	CDC MMWR	Morbidity & Mortality Weekly Report

📦 Data Models

Structured Pydantic models for every domain:

• AdverseEvent — Patient reactions, seriousness, outcomes, suspect drugs
• DrugRecall — Recall classification (I/II/III), reason, distribution scope
• ClinicalTrial — Phase, enrollment, status, interventions, endpoints
• PubMedArticle — Abstracts, MeSH terms, author affiliations, citations
• SafetyAlert — Boxed warnings, safety communications, Dear Doctor letters
• DrugInfo — Indications, contraindications, pricing, approval history

🏷️ Medical Code Enrichment

Every item is enriched with standard medical ontologies:

• ICD-10 — International Classification of Diseases
• MedDRA — Medical Dictionary for Regulatory Activities (preferred terms, SOCs)
• ATC — Anatomical Therapeutic Chemical Classification System

📊 Classification & Scoring

• Severity classification — Automatically tiers events as Mild / Moderate / Serious / Life-Threatening
• Risk scoring — Composite score based on frequency, severity, regulatory action, and novelty
• Duplicate detection — Cross-source deduplication using fuzzy matching on drug names, event descriptions, and report IDs

💾 Storage Backends

Store wherever makes sense for your pipeline:

Backend	Use Case
CSV	Quick exports, spreadsheet handoffs
JSON	API integrations, data interchange
SQLite	Local querying, single-node analysis
Redis	High-throughput pipelines, real-time feeds

---

🚀 Quick Start

Installation

pip install pharma-vigilance

Run Your First Spider

from pharma_vigilance import Pipeline

pipeline = Pipeline()
pipeline.add_spider("fda_faers", drug="ibuprofen", start_date="2024-01-01")
pipeline.add_spider("pubmed", query="ibuprofen adverse events", max_results=100)
results = pipeline.run(storage="sqlite", output="vigilance.db")
print(f"Collected {results.total} records across {results.spiders_run} spiders")

---

📋 Spider Reference

1. FDA FAERS Spider

Crawls the FDA Adverse Event Reporting System for individual case safety reports (ICSRs).

• Data: Patient demographics, drug dosage, reaction terms, outcomes, reporter type
• Source: https://fis.fda.gov/

2. ClinicalTrials.gov Spider

Monitors clinical trial registrations, status changes, and results postings.

• Data: Trial phases, enrollment counts, primary/secondary endpoints, sponsor info
• Source: https://clinicaltrials.gov/api/v2/

3. PubMed Spider

Retrieves biomedical literature on drug safety, pharmacovigilance, and adverse effects.

• Data: Title, abstract, MeSH terms, publication date, journal, DOI
• Source: https://eutils.ncbi.nlm.nih.gov/entrez/eutils/

4. EMA Spider

Tracks European regulatory decisions, CHMP opinions, and safety referrals.

• Data: Marketing authorizations, variation procedures, PSUR assessments
• Source: https://www.ema.europa.eu/

5. MedWatch Spider

Captures FDA safety alerts, drug safety communications, and label changes.

• Data: Alert type, affected products, risk summary, recommended actions
• Source: https://www.fda.gov/safety/medwatch-safety-alerts

6. WHO VigiBase Spider

Indexes global individual case safety reports from the WHO Programme for International Drug Monitoring.

• Data: Country, reaction group, seriousness criteria, suspect drug
• Source: https://vigibase.who.int/

7. FDA Recalls Spider

Monitors FDA enforcement reports for drug, biologic, and device recalls.

• Data: Recall class, product description, distribution area, recalling firm
• Source: https://www.fda.gov/safety/recalls-market-withdrawals-safety-alerts

8. Drug Prices Spider

Tracks pharmaceutical pricing data from public and commercial sources.

• Data: Brand/generic name, NDC, wholesale acquisition cost, Medicare price
• Source: https://data.cms.gov/, https://query.prices.why/

9. Healthcare Policy Spider

Follows regulatory and legislative changes affecting pharmaceuticals and healthcare.

• Data: Federal Register notices, CMS rules, HHS guidance documents
• Source: https://www.federalregister.gov/api/

10. Medical Devices Spider

Tracks device adverse events, recalls, and 510(k)/PMA approvals via FDA MAUDE.

• Data: Device name, event type, manufacturer, patient outcome, report number
• Source: https://www.fda.gov/medical-devices/

11. CDC MMWR Spider

Indexes the CDC Morbidity and Mortality Weekly Report for outbreak and surveillance data.

• Data: Article title, key findings, disease burden, policy implications
• Source: https://www.cdc.gov/mmwr/

---

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                        Pharma Vigilance Core                        │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐                 │
│  │  FDA FAERS  │  │ ClinicalTrl │  │   PubMed    │                 │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘                 │
│         │                │                │                         │
│  ┌──────┴──────┐  ┌──────┴──────┐  ┌──────┴──────┐                 │
│  │     EMA     │  │  MedWatch   │  │ WHO VigiBase│                 │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘                 │
│         │                │                │                         │
│  ┌──────┴──────┐  ┌──────┴──────┐  ┌──────┴──────┐                 │
│  │ FDA Recalls │  │ Drug Prices │  │  HC Policy  │                 │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘                 │
│         │                │                │                         │
│  ┌──────┴──────┐  ┌──────┴──────┐                                  │
│  │ Med Devices │  │  CDC MMWR   │                                  │
│  └──────┬──────┘  └──────┬──────┘                                  │
│         │                │                                         │
│         └────────┬───────┘                                         │
│                  ▼                                                  │
│  ┌───────────────────────────────────────────────────────────────┐ │
│  │                   Pipeline Engine                              │ │
│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐    │ │
│  │  │  Rate    │ │  Dedup   │ │ Severity │ │  Med Code    │    │ │
│  │  │ Limiter  │ │ Engine   │ │ Classify │ │  Enrichment  │    │ │
│  │  └──────────┘ └──────────┘ └──────────┘ └──────────────┘    │ │
│  └───────────────────────────────────────────────────────────────┘ │
│                  │                                                  │
│                  ▼                                                  │
│  ┌───────────────────────────────────────────────────────────────┐ │
│  │                  Storage Backends                              │ │
│  │    ┌─────┐    ┌──────┐    ┌────────┐    ┌───────┐            │ │
│  │    │ CSV │    │ JSON │    │ SQLite │    │ Redis │            │ │
│  │    └─────┘    └──────┘    └────────┘    └───────┘            │ │
│  └───────────────────────────────────────────────────────────────┘ │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

---

⚖️ Ethics & Compliance

This framework is designed with responsible data practices at its core.

Respecting Data Sources

• robots.txt compliance — Every spider respects robots.txt directives before crawling
• Rate limiting — Configurable per-spider rate limits with exponential backoff; defaults are conservative
• API key management — Keys stored in environment variables, never committed to source control
• Terms of Service — Each spider documents the ToS of its source and flags any restrictions

Data Privacy

• No patient PII — Spiders ingest only publicly available, de-identified aggregate data
• HIPAA awareness — No tools in this framework handle or store Protected Health Information
• Data retention — Configurable TTL on cached data; automatic purge of stale records
• Audit logging — Every crawl is logged with timestamp, source, record count, and any errors

Responsible Use

⚠️ Disclaimer: This tool is for research, monitoring, and analysis purposes only. It does not provide medical advice, diagnosis, or treatment recommendations. Always consult qualified healthcare professionals for clinical decisions.

• Do not use scraped data to make automated clinical decisions
• Do not attempt to re-identify de-identified patient data
• Do not redistribute proprietary data beyond fair-use boundaries
• Report any discovered security vulnerabilities responsibly

---

🤝 Contributing

We welcome contributions from the pharmacovigilance, public health, and open-source communities!

Getting Started

# Clone the repo
git clone https://github.com/pharma-vigilance/pharma-vigilance.git
cd pharma-vigilance

# Create a virtual environment
python -m venv venv && source venv/bin/activate

# Install in dev mode
pip install -e ".[dev]"

# Run tests
pytest tests/ -v

Adding a New Spider

1. Create a new file in pharma_vigilance/spiders/
2. Implement the BaseSpider interface
3. Add your data model to pharma_vigilance/models/
4. Register the spider in the pipeline config
5. Add tests and documentation

Guidelines

• Follow PEP 8 and use type hints
• Write tests for all new features
• Document public APIs with docstrings
• Keep spiders stateless and idempotent
• Open an issue before large architectural changes

---

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.

MIT License

Copyright (c) 2024 Pharma Vigilance Contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

---

_{Built with care for public health 🏥 · healthcare · medical · pharma · vigilance · drug-safety · clinical-trials · pharmacovigilance · FDA · EMA · WHO · democracy}