Add Drugs.com scraper and tests

[NaitikOza]

---

What this does

Adds a Drugs.com scraper to the datasets scraping pipeline. Like the existing source scrapers, discovery and extraction are separate, and every page is normalized into a ScrapedDocument.

Drugs.com does not expose an API for drug monographs, so discovery crawls the public alphabetical index (/alpha) and extracts links to supported drug-information pages. Extraction then downloads each page, removes site navigation and boilerplate, converts the article to normalized markdown, and records structured metadata for downstream retrieval.

The scraper supports both full-corpus crawling and scraping a single document by URL.

---

Discovery

Discovery walks the public alphabetical index:

/alpha/{letter}.html
    -> /alpha/{two-letter}.html
        -> drug pages

Some indexes (such as 0-9) skip the two-letter layer and directly contain drug pages, so both layouts are handled automatically.

Supported page namespaces include:

• consumer pages (/drug-name.html)
• mtm
• cons
• pro
• monograph
• cdi

URLs are normalized into a unique page reference before scraping, and non-document pages (drug information landing pages, interaction checkers, image galleries, etc.) are filtered during discovery.

---

Extraction

Each page is downloaded once and converted into normalized markdown.

During extraction the scraper:

• removes navigation, advertisements, footers, scripts, and other page chrome

• converts HTML into clean markdown

• preserves absolute links

• preserves images as absolute markdown image URLs

• extracts structured metadata including:

  • namespace
  • medically reviewed by
  • last updated
  • copyright holder

• removes repeated site boilerplate

• removes citation markers

• truncates "More about..." sections

• removes unrelated sections such as "Related drugs" and interaction pages

• normalizes whitespace and markdown formatting

Pages that cannot be parsed or contain no usable content are logged and skipped without aborting the scrape.

---

CLI

The scraper supports both corpus mode and single-page mode.

uv run amfv-scrape --source drugscom --documents 1

uv run amfv-scrape --source drugscom --url https://www.drugs.com/ibuprofen.html

uv run amfv-scrape --source drugscom --documents 10 -f markdown -o ./drugs-out/

--source all also includes Drugs.com alongside the other supported scrapers.

---

Licensing

Drugs.com content is proprietary.

Each document records this in its metadata rather than assigning an open license, allowing downstream users to distinguish proprietary content from open datasets.

---

Shared scraper improvements

While implementing the Drugs.com scraper, several shared scraping utilities were improved:

cli.py

• added support for scraping a single document by URL
• improved output handling for JSONL, Markdown, and text formats
• better validation of CLI arguments
• fixed --source all behavior
• improved progress reporting

base.py

• scrape_listing_documents() now supports scrapers that return None for invalid or skipped pages
• document limits now count successfully scraped documents rather than attempted pages
• delay handling was updated so skipped pages still respect crawl timing

html.py

• relative image URLs are now converted to absolute URLs during HTML normalization
• markdown image links are preserved correctly
• existing link normalization behavior remains unchanged

---

Tests

Added comprehensive offline tests covering:

• discovery from alphabetical indexes
• URL parsing
• page extraction
• markdown cleanup
• metadata extraction
• HTML helper functions
• shared scraping primitives
• CLI behavior

All tests use mocked HTTP responses (httpx.MockTransport) and run completely offline.

uv run ruff check datasets --fix

uv run ruff format datasets

uv run ruff check datasets

uv run pytest datasets/test

All scraper, HTML helper, base utility, and CLI tests pass.

---

Notes for review

• Discovery is resilient to both alphabetical index layouts used by Drugs.com.
• Invalid pages are skipped without stopping the overall crawl.
• Single-page scraping was added to simplify debugging and targeted dataset generation.
• Shared HTML utilities now correctly preserve absolute image URLs for all scrapers, not just Drugs.com.
• Shared listing utilities now support sources that may skip invalid documents while still respecting requested document counts.

[CLAassistant]

All committers have signed the CLA.