Add WikiDoc scraper - #12
Open
zndr27 wants to merge 3 commits into
Open
Conversation
|
|
WikiDoc is a MediaWiki instance, so discovery and extraction go through
api.php rather than the rendered pages: list=allpages is a documented
contract for enumerating articles, and action=parse returns the body,
revision id, categories and section list in one request.
Content is CC BY-SA 3.0, reported by the API itself via
action=query&meta=siteinfo&siprop=rightsinfo.
Notes on the source:
- The API does not remove WikiDoc's navigation templates ("WikiDoc
Resources for X", "X Microchapters", "Resident Survival Guide") because
they are transcluded into the article wikitext. They are consistently
marked class="infobox" while real content tables are "wikitable" or
unclassed, so they are stripped on that selector.
- "[edit | edit source]" markers are dropped with disableeditsection=1
rather than post-processing.
- Mainspace contains ~1.5% non-articles (user sandboxes, titles with
leading quote characters), filtered on title.
- Pagination is by opaque apcontinue token rather than page number, so
list_page keeps the token in a closure and ignores the page argument.
Also cherry-picks the scrape_item -> ScrapedDocument | None change from
PR MedARC-AI#10 (RCH) verbatim, so identical changes merge cleanly whichever lands
first. WikiDoc needs it: a handful of pages are corrupt (Hypertelorism
reports "There is no revision with ID 388595") and one of them would
otherwise abort a run over 141k articles.
Every article opens with an "Editor-In-Chief: ...; Associate Editor(s)-In-Chief: ..." paragraph. It is authorship rather than clinical content, and it sits at the top of the document where retrieval weights most heavily. Keep it as metadata.editors so CC BY-SA attribution survives, but out of the content body.
zndr27
marked this pull request as ready for review
July 26, 2026 02:41
zndr27
changed the title
WikiDoc articles are frequently a decade old - a random sample of 369 mainspace pages shows a median last edit around 2017 and only 4.6% edited 2021 or later - and the recorded revid is an opaque integer that cannot tell you which. Recording the revision timestamp lets a corpus build filter on recency later without re-scraping, the same reason content_length_chars is already recorded. Discovery switches from list=allpages to the same query as a generator with prop=revisions, so the timestamp arrives in the request the listing already makes. Pages whose latest revision is unreadable come back without one; those are the corrupt pages _scrape_or_skip already drops. A generator returns its pages unordered, so each batch is resorted by title to keep --documents N reproducible. The --url path builds a ref with no timestamp and looks one up; a full listing run adds no requests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Adds a WikiDoc scraper to the datasets scraping pipeline. Same shape as the NICE scraper: discovery and extraction are separate, and everything outputs a normalized
ScrapedDocument. WikiDoc is a MediaWiki instance, roughly 141k mainspace articles, so both discovery and extraction go throughapi.phprather than the rendered pages.list=allpagesandaction=parseare documented contracts that will not break when the site restyles.Open question: should a document be a page or a topic?
WikiDoc splits some topics across a hub page plus roughly 30 sub-pages it calls microchapters. It is opt-in per topic, and the content sits in different places depending:
Sepsis: nav-only hub, content in the sub-pagesHypertension: whole article on the hub (119k chars), stub sub-pagesOne document per page, which is what this PR does, pulls in a lot of noise. About 17% of pages are under 500 characters, including placeholders like
Meningitis MRI, whose entire body is "Please help WikiDoc by adding content here."nice.pyalready handles the equivalent problem by merging chapters inbuild_guideline_text, so one document per page is arguably the deviation here.I prototyped merging. WikiDoc declares the grouping itself: every page of a topic transcludes the same
<Topic> Microchaptersnav table, the one we already strip.Suggest merging to match
nice.py, accepting that this source wants chunking at retrieval either way. Can implement it here. Did not want to decide the corpus shape unilaterally.How discovery works
action=query&list=allpages, namespace 0,apfilterredir=nonredirects, 500 titles per request. MediaWiki paginates by an opaqueapcontinuetoken rather than a page number, solist_pagekeeps the token in a closure and ignores the page argument it is handed.About 1.5% of mainspace is user sandboxes and malformed titles, filtered on title. A few percent of pages are corrupt on WikiDoc's side and return HTTP 200 with
{"error": {"info": "There is no revision with ID 388595"}}(also fails bypageid, so not a client issue). These are logged and skipped rather than aborting the run.How extraction works
action=parse&prop=text|revid|categories|sectionsreturns body, revision id, categories and sections in one request.disableeditsection=1drops the[edit | edit source]markers.WikiDoc's navigation templates ("WikiDoc Resources for X", "X Microchapters") are transcluded into the article, so the API returns them. All carry
class="infobox"while content tables arewikitableor unclassed, so they are removed on that selector. Hypertension has 20 tables and exactly one infobox.Editor bylines move to
metadata.editorsrather than heading every document. Metadata also recordsrevid,categoriesandcontent_length_chars.CLI
--source allnow includes WikiDoc alongside NICE.Licensing
CC BY-SA 3.0, confirmed from the API via
meta=siteinfo&siprop=rightsinforather than a page footer. Recorded per document inmetadata.licenseandmetadata.editors.Tests
37 tests, offline via
httpx.MockTransport. Also scraped and read output for 10 pages across Sepsis, Hypertension, Endocarditis and Tuberculosis.Notes for review
base.pyis taken verbatim from add Royal Children's Hospital guideline scraper #10 so the two merge cleanly in either order. WikiDoc needsscrape_item -> ScrapedDocument | Nonefor the corrupt pages above. Drop it if add Royal Children's Hospital guideline scraper #10 lands first.--source allotherwise untouched: it returns inside the loop onmainso only NICE runs, and add Canadian Paediatric Society statement scraper #9 and add Royal Children's Hospital guideline scraper #10 already fix that two different ways.