add Royal Children's Hospital guideline scraper - #10
Open
m-krastev wants to merge 1 commit into
Open
Conversation
|
|
zndr27
added a commit
to zndr27/amfv
that referenced
this pull request
Jul 26, 2026
WikiDoc is a MediaWiki instance, so discovery and extraction go through
api.php rather than the rendered pages: list=allpages is a documented
contract for enumerating articles, and action=parse returns the body,
revision id, categories and section list in one request.
Content is CC BY-SA 3.0, reported by the API itself via
action=query&meta=siteinfo&siprop=rightsinfo.
Notes on the source:
- The API does not remove WikiDoc's navigation templates ("WikiDoc
Resources for X", "X Microchapters", "Resident Survival Guide") because
they are transcluded into the article wikitext. They are consistently
marked class="infobox" while real content tables are "wikitable" or
unclassed, so they are stripped on that selector.
- "[edit | edit source]" markers are dropped with disableeditsection=1
rather than post-processing.
- Mainspace contains ~1.5% non-articles (user sandboxes, titles with
leading quote characters), filtered on title.
- Pagination is by opaque apcontinue token rather than page number, so
list_page keeps the token in a closure and ignores the page argument.
Also cherry-picks the scrape_item -> ScrapedDocument | None change from
PR MedARC-AI#10 (RCH) verbatim, so identical changes merge cleanly whichever lands
first. WikiDoc needs it: a handful of pages are corrupt (Hypertelorism
reports "There is no revision with ID 388595") and one of them would
otherwise abort a run over 141k articles.
zndr27
added a commit
to zndr27/amfv
that referenced
this pull request
Jul 26, 2026
Scrapes the intersection of PubMed's Practice Guideline publication type with the PMC Open Access subset, 2,829 guidelines. That is the only slice where the full text is both retrievable and openly licensed; the other 34k Practice Guideline records expose an abstract only, under publisher copyright. Discovery and extraction use NCBI's E-utilities rather than the rendered pages. esearch paginates by numeric offset, so the page number maps straight onto retstart and an empty page past the end terminates the run. esummary resolves a whole batch of PMIDs to PMCIDs and citation metadata in one request. efetch returns JATS XML carrying body, section structure and license. JATS is close enough to HTML that renaming tags and reusing html_to_markdown is cheaper and less error-prone than a second serializer: table-wrap already contains genuine XHTML tables, and inline markup maps one to one. Licensing is recorded per document rather than claimed for the source, because the Open Access Subset is not uniformly Creative Commons licensed. Censused over all 2,829 records: CC BY 42.8% publisher terms, no CC license 13.3% CC BY-NC 20.5% of which: Elsevier COVID grant 207 CC BY-NC-ND 20.2% PMC OA "unrestricted" 51 CC BY-NC-SA 1.9% no <license> element 101 CC0 1.3% So 44.1% is CC BY or CC0 and carries no restriction on derivatives, 22.4% is non-commercial only, 20.2% asserts NoDerivatives, and 13.3% needs reading case by case. Presence in the subset is not itself a grant to redistribute: 101 records carry only a copyright line such as "(c) Springer-Verlag Tokyo 2007", and Elsevier's pandemic-era deposits grant free access while still reserving all rights. The license name is parsed from the Creative Commons URL rather than the license-type attribute, which the corpus spells 16 different ways. The copyright statement is captured separately because it is a sibling of <license>, not a child, and it holds the reservation of rights. Figures are recorded in metadata rather than linked. Unlike the HTML sources in MedARC-AI#9 and MedARC-AI#10, which absolutize a real <img src>, JATS carries only a bare filename; the served URL inserts a CDN shard and content hash that appear nowhere in the API response, so a constructed link 404s. The filename, label and caption are kept so a later pass can resolve them without re-scraping, and the caption stays in the text. E-utilities calls retry with backoff on 429 and 5xx. One document makes up to two calls back to back and the rate limit is per source address, so NCBI does answer with 429 in practice; without a retry that propagates past the skip handler and kills the whole run. Records PMC holds without a deposited body, 1.1% of the corpus, are logged and skipped rather than aborting the run.
zndr27
added a commit
to zndr27/amfv
that referenced
this pull request
Jul 29, 2026
Scrapes the intersection of PubMed's Guideline publication type with the PMC Open Access subset, 3,184 guidelines. That is the only slice where the full text is both retrievable and openly licensed; the other 40k Guideline records expose an abstract only, under publisher copyright. `Guideline[pt]` rather than `Practice Guideline[pt]`: the former is a strict superset, and the 352 records it adds are clinical rather than administrative (the 2025 Korean CPR guidelines and similar), so the narrower tag would drop 11% of the corpus for nothing. Discovery and extraction use NCBI's E-utilities rather than the rendered pages. esearch paginates by numeric offset, so the page number maps straight onto retstart and an empty page past the end terminates the run. esummary resolves a whole batch of PMIDs to PMCIDs and citation metadata in one request. efetch returns JATS XML carrying body, section structure and license. JATS is close enough to HTML that renaming tags and reusing html_to_markdown is cheaper and less error-prone than a second serializer: table-wrap already contains genuine XHTML tables, and inline markup maps one to one. Licensing is recorded per document rather than claimed for the source, because the Open Access Subset is not uniformly Creative Commons licensed. Censused over all 3,184 records: CC BY 43.9% publisher terms, no CC license 12.6% CC BY-NC 21.1% of which: Elsevier COVID grant 211 CC BY-NC-ND 19.2% no <license> element 112 CC BY-NC-SA 2.1% PMC OA "unrestricted" 58 CC0 1.2% other publisher terms 19 By what that permits: 45.1% unrestricted for derivative works, 23.2% non-commercial only, 19.2% asserting NoDerivatives, 12.6% needing a case-by-case reading. Presence in the subset is not itself a grant to redistribute: 112 records carry only a copyright line such as "(c) Springer-Verlag Tokyo 2007", and Elsevier's pandemic-era deposits grant free access while still reserving all rights. The license name is parsed from the Creative Commons URL rather than the license-type attribute, which the corpus spells 18 different ways. The copyright statement is captured separately because it is a sibling of <license>, not a child, and it holds the reservation of rights. Figures and supplementary files are recorded in metadata rather than linked. Unlike the HTML sources in MedARC-AI#9 and MedARC-AI#10, which absolutize a real <img src>, JATS carries only a bare filename; the served URL inserts a CDN shard and content hash that appear nowhere in the API response, so a constructed link 404s. Supplementary blocks are pointers too: across 40 sampled guidelines every one referenced an external .docx or .tif rather than inline content, totalling 0.18% of body text. Recording name, label and caption keeps the evidence tables findable without re-scraping. E-utilities calls retry with backoff on 429 and 5xx. One document makes up to two calls back to back and the rate limit is per source address, so NCBI does answer with 429 in practice; without a retry that propagates past the skip handler and kills the whole run. external_id prefers the PMID from the record itself, so an article reached from a PMC URL gets the same identifier as one reached from the listing. Records PMC holds without a deposited body, 1.0% of the corpus, are logged and skipped rather than aborting the run.
zndr27
added a commit
to zndr27/amfv
that referenced
this pull request
Jul 29, 2026
Scrapes the intersection of PubMed's Guideline publication type with the PMC Open Access subset, restricted to English: about 3,000 guidelines. That is the only slice where the full text is both retrievable and openly licensed; the other 40k Guideline records expose an abstract only, under publisher copyright. Two scoping choices, both measured rather than assumed: `Guideline[pt]` rather than `Practice Guideline[pt]`. The former is a strict superset and the 352 records it adds are clinical, not administrative (the 2025 Korean CPR guidelines and similar), so the narrower tag would drop 11% of the corpus for nothing. `English[la]`, which drops 185 records. Every other source in this package is already English-only as a side effect of its entry URL: CPS is a bilingual site scraped through its /en/ routes, WHO publishes in six languages and is scraped through its English listing. PubMed's API returns every language, so the filter has to be explicit to match. The excluded records are largely French CMAJ translations of guidelines already in the corpus. Discovery and extraction use NCBI's E-utilities rather than the rendered pages. esearch paginates by numeric offset, so the page number maps straight onto retstart and an empty page past the end terminates the run. esummary resolves a whole batch of PMIDs to PMCIDs and citation metadata in one request. efetch returns JATS XML carrying body, section structure and license. JATS is close enough to HTML that renaming tags and reusing html_to_markdown is cheaper and less error-prone than a second serializer: table-wrap already contains genuine XHTML tables, and inline markup maps one to one. Licensing is recorded per document rather than claimed for the source, because the Open Access Subset is not uniformly Creative Commons licensed. Censused over all 2,999 English records present on 2026-07-29: CC BY 44.5% publisher terms, no CC license 11.7% CC BY-NC 22.1% of which: Elsevier COVID grant, no CC BY-NC-ND 18.4% <license> element at all (112), CC BY-NC-SA 2.1% PMC OA "unrestricted re-use" CC0 1.3% By what that permits: 45.8% unrestricted for derivative works, 24.1% non-commercial only, 18.4% asserting NoDerivatives, 11.7% needing a case-by-case reading. Presence in the subset is not itself a grant to redistribute: 112 records carry only a copyright line such as "(c) Springer-Verlag Tokyo 2007", and Elsevier's pandemic-era deposits grant free access while still reserving all rights. The license name is parsed from the Creative Commons URL rather than the license-type attribute, which the corpus spells 18 different ways. The copyright statement is captured separately because it is a sibling of <license>, not a child, and it holds the reservation of rights. Figures and supplementary files are recorded in metadata rather than linked. Unlike the HTML sources in MedARC-AI#9 and MedARC-AI#10, which absolutize a real <img src>, JATS carries only a bare filename; the served URL inserts a CDN shard and content hash that appear nowhere in the API response, so a constructed link 404s. Supplementary blocks are pointers too: across 40 sampled guidelines every one referenced an external .docx or .tif rather than inline content, totalling 0.18% of body text. Recording name, label and caption keeps the evidence tables findable without re-scraping. E-utilities calls retry with backoff on 429 and 5xx. One document makes up to two calls back to back and the rate limit is per source address, so NCBI does answer with 429 in practice; without a retry that propagates past the skip handler and kills the whole run. external_id prefers the PMID from the record itself, so an article reached from a PMC URL gets the same identifier as one reached from the listing. Records PMC holds without a deposited body, 0.6% of the corpus, are logged and skipped rather than aborting the run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Adds a Royal Children's Hospital Melbourne (RCH) scraper to the datasets scraping pipeline. Same shape as the NICE scraper — discovery and extraction are separate and everything outputs a normalized
ScrapedDocument.The scraper extracts the complete HTML clinical practice guideline, including tables, references, links, and remote figures. It also handles several generations of RCH page layouts and stale index entries.
How discovery works
Discovery starts from the RCH clinical guideline A-Z index. The index contains duplicate labels, synonyms, legacy
.aspxroutes, nested paths, wrapper pages, and non-guideline administrative entries. Links are normalized and deduplicated by canonical URL, while alternate index labels are retained inmetadata.aliasesfor retrieval.How extraction works
Each guideline page is fetched and qualifying
widgetBodyblocks under#rch-primaryare converted through the sharedhtml_to_markdownhelper.The extraction preserves:
CLI
--source allnow includes RCH alongside NICE.--resumeappends to an existing JSONL file and skips document URLs already present. It is intended for recovering long, polite scraping runs. A new output path without--resumestill performs a fresh scrape.Licensing
RCH guidelines are not published under an identified open dataset redistribution license. Automated access and reuse should be treated as permission-gated until the project maintainers confirm policy or obtain permission.
QA
Test plan
uv run pytest datasets/test/test_scraping_rch.py— discovery, aliases, legacy layouts,wrappers, and failure handling
uv run pytest— no regressions (41 passed)uv run ruff check .uv run amfv-scrape --source rch --documents 3 -f markdown -o /tmp/rch-out/