Release 1.9.0: IDC v25, idc-index 0.13.0, and data provenance

[fedorov]

idc-index 0.13.0 is a data-release bump with no API change, so the version pin is the small part of this. The substance of v25 is data provenance, which the skill documented nowhere.

Provenance

v25 records who contributed, de-identified, and DICOM-converted each part of the archive. Two things a reader gets wrong unaided, so both are documented explicitly:

It is per source, not per collection, keyed on source_DOI. Collections mix lineage — tcga_brca radiology arrives as DICOM from TCIA (dicom_conversion_by = NOT_APPLICABLE) while its pathology is contributed by IDC, de-identified by GDC, and converted by IDC under a separate Zenodo DOI. collections_index.sources is a list of structs (unnest it) with a nested provenance; analysis_results_index.provenance is a new column.

A NULL struct is a pointer, not missing data. All 92 of the 308 collection sources without provenance are analysis-result DOIs, whose provenance sits in analysis_results_index — populated for all 26 rows. Across the pair, coverage is complete, so reporting "unavailable" would be wrong. NOT_APPLICABLE / UNKNOWN / NOT_DOCUMENTED are values, not nulls.

Documented in index_tables_guide.md (fields, joins, traps), sql_patterns.md (worked queries), and SKILL.md, with TestProvenance pinning the invariants.

Also found while verifying

• Four index columns the guides never carried: sop_class_name, transfer_syntax_name, PhotometricInterpretation, PixelRepresentation (the latter two shipped in 24.2.2 and were missed by 1.8.3). All four are comma-separated when instances in a series disagree (~72 000 series) and the latter two are NULL for SEG/SR/RTSTRUCT, so = silently drops mixed-encoding series
• The new Hugging Face mirror of the Parquet artifacts, documented as a fallback since revision tags are incomplete
• Four Parquet artifacts missing from the file table, including gdc_idc_mapping (the imaging/genomics join)

Version bumps

Pinned minimum idc-index 0.12.5 → 0.13.0 and documented data version v24 → v25 across the frontmatter, the setup guard, check_version.py, and every Tested with: header. Archive totals 176 → 179 collections and ~99 → ~100 TB; mixed-license collections still 39, re-measured.

The DICOMweb store deliberately stays at idc-store-v24 — stores lag a release by 1-2 weeks and idc-store-v25 is not populated yet. dicomweb_guide.md now warns about this at the point of use, since the guide's own "replace {VERSION} with the IDC release number" plus get_idc_version() leads straight to a store name that 404s. This will need a follow-up bump once v25 is populated.

Verification

174 tests pass against live services, plus 33 BigQuery dry-run tests. All snippets were re-run against 0.13.0 / v25 before the headers moved: 91 of 92 passed unchanged, the one failure being the hardcoded version assertion below. The MCP tool inventory, the 19 filterable attributes, and all REST limits were re-checked against 3.0.0b5 and are unchanged — both contract suites pass unedited.

test_snippets.py hardcoded "v24", the one test needing a hand edit every data release. It now reads idc-data-version from the frontmatter, so a frontmatter disagreeing with the pinned idc-index fails CI instead of leaving the guides describing an older release.

🤖 Generated with Claude Code