Skip to content

Add IDSA scraper - #8

Open
KouateMuhamed wants to merge 2 commits into
MedARC-AI:mainfrom
KouateMuhamed:feat/idsa-scraper
Open

Add IDSA scraper#8
KouateMuhamed wants to merge 2 commits into
MedARC-AI:mainfrom
KouateMuhamed:feat/idsa-scraper

Conversation

@KouateMuhamed

Copy link
Copy Markdown

Summary

  • Add an AMFV-native scraper for IDSA practice guidelines.
  • Add IDSA CLI support with archived and in-development inclusion flags.
  • Add quality metadata for short guideline pages and external/PDF link metadata.

Tests

  • env UV_CACHE_DIR=.uv-cache uv run ruff check datasets
  • env UV_CACHE_DIR=.uv-cache uv run ruff format --check datasets
  • env UV_CACHE_DIR=.uv-cache uv run pytest datasets/test -v

QA

  • Scraped 55 default IDSA documents from the live site.
  • Removed Table of Contents navigation from scraped content.
  • Retained short documents with quality_flags=["short_content"].

@CLAassistant

CLAassistant commented Jul 8, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@warner-benjamin

Copy link
Copy Markdown
Collaborator

Thanks for the PR. This looks good, but it also grabs out of date guidelines. If we process those we'll want to add a metadata flag that they are out of date.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants