Skip to content

Add bundle_gas: Group A Strep across Epic Cosmos, NNDSS, and ABCs - #187

Closed
MadiBal wants to merge 2 commits into
ingest-cosmos-gasfrom
bundle-gas
Closed

Add bundle_gas: Group A Strep across Epic Cosmos, NNDSS, and ABCs#187
MadiBal wants to merge 2 commits into
ingest-cosmos-gasfrom
bundle-gas

Conversation

@MadiBal

@MadiBal MadiBal commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Combines Group A Streptococcus surveillance from three sources into six long-format parquets.

Stacked on #186. The base of this PR is ingest-cosmos-gas, so the diff here shows only the abcs_gas source and the bundle. Merge #186 first; this PR's base will retarget to main automatically.

Initialized with dcf::dcf_add_bundle("bundle_gas", source_files = ...), so process.json carries type: "bundle" and scripts: build.R.

Outputs

All six share the bundle conventions: geography holds state names (or "United States"), time is ISO YYYY-mm-dd period-end, value is the plotting column, keyed by a measure identifier column.

Parquet Source Rows Grain
epic_gas.parquet epic_gas 39,312 state + national, quarterly, by age — n_strep_throat / pct_strep_throat / n_patients, plus a per-measure suppressed flag
nnds_stss.parquet nnds 24,752 state + national, weekly — stss_cases_weekly / stss_cases_cumulative
abcs_gas.parquet abcs_gas 674 national, annual, by age/sex/race — case and death rates and counts
abcs_gas_syndromes.parquet abcs_gas 135 national, annual — 5 clinical syndromes
abcs_gas_resistance.parquet abcs_gas 180 national, annual — 6 antibiotics + isolate count
abcs_gas_emm.parquet abcs_gas 421 national, annual — 16 emm types + other

Two things that most need a second pair of eyes

1. NNDSS is published cumulatively, and I de-accumulate it. The raw streptococcal_toxic_shock_syndrome column is a year-to-date running total that resets each MMWR year — national 2024 runs 5 → 647 across weeks 1–52. Plotted as-is it would show a sawtooth, not incidence. build.R differences it within geography-year into stss_cases_weekly and keeps the published cumulative series alongside; the two are not additive. The earlier bundle_gas attempt on the stale gas branch did the same, and the nnds source measure_info documents the column as "Cumulative year-to-date", so this looks right — but it is the one substantive transformation in this PR and worth confirming.

Relatedly, 27 of 12,376 weekly increments come out negative, because NNDSS revises earlier weeks downward. I left them as reported rather than clamping to zero, and build.R logs the count. Clamping is a one-line change if preferred.

2. Which "ABCS" this is. data/abcs on main is invasive pneumococcal disease by serotype — a different pathogen, with no GAS measures. The Group A Strep ABCs data (CDC dataset 9y49-tura) existed only on the stale gas branch and was never merged, so the first commit here brings abcs_gas onto main-line development.

Known gap, pre-existing: that source's own measure_info.json uses generic keys (case_rate, syndrome, pct_resistant, emm_type) that do not match its actual column names, so dcf_check reports 36 undocumented columns on the source. The bundle documents those columns itself via levels inheritance, so the dist files and generated docs are complete — but the source-level check still fails. Happy to fix those 36 entries in a follow-up or here.

Verification

  • dcf::dcf_process("bundle_gas") runs clean and is deterministic — re-running produces no diff.
  • measure_info.json parses; 0 undocumented columns across all 6 parquets, and all 34 levels source_id references resolve against the contributing sources' measure_info.json.
  • _bundle block covers every parquet, its sources equals the union of the per-file sources, and every listed source_file exists.
  • scripts/build_docs.R regenerated (14 bundles, 46 sources); bundle_gas renders a short name plus a description or value list for all 29 columns.

Smaller decisions

  • Four separate ABCs parquets, per the upstream split, but each melted to long format to match the value-column convention rather than kept wide as the earlier attempt did.
  • Epic suppression is mapped per measure: the upstream numerator flag covers n_strep_throat and pct_strep_throat (the percent derives from that same cell), the denominator flag covers n_patients.
  • Territories and non-state NNDSS jurisdictions (e.g. New York City) are dropped, following bundle_enteric_diseases.
  • abcs_gas is national only (a catchment area, not the whole US), and the three sources' time spans differ (Epic 2017→2025, NNDSS 2022→present, ABCs 1997→2023), so cross-source comparison is limited to the overlap. Both are documented in the bundle README.

🤖 Generated with Claude Code

MadiBal and others added 2 commits August 5, 2026 16:10
Brings the abcs_gas source onto main-line development from the stale `gas`
branch, where it was created but never merged. CDC dataset 9y49-tura: national,
annual (1997-2023) invasive Group A Strep case/death rates and counts, clinical
syndrome distribution, antibiotic resistance, and emm type distribution.

Needed by bundle_gas. Note that data/abcs is a different pathogen entirely
(invasive pneumococcal disease by serotype) and has no GAS measures.

Known gap, pre-existing and not addressed here: this source's measure_info.json
uses generic keys (case_rate, syndrome, pct_resistant, emm_type) that do not
match its actual column names, so dcf_check reports 36 undocumented columns.
The bundle documents these columns itself via `levels` inheritance.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Initialized with dcf::dcf_add_bundle("bundle_gas", source_files = ...) so
process.json carries type: "bundle" and scripts: build.R.

build.R writes six long-format parquets to dist/, each with geography as a
state name (or "United States"), ISO time, a `measure` identifier, and `value`:

  epic_gas.parquet            n_strep_throat / pct_strep_throat / n_patients,
                              quarterly by state and age, with a per-measure
                              `suppressed` flag
  nnds_stss.parquet           streptococcal_toxic_shock_syndrome only, weekly
  abcs_gas.parquet            case/death rates and counts, annual national
  abcs_gas_syndromes.parquet  clinical syndrome distribution
  abcs_gas_resistance.parquet antibiotic resistance
  abcs_gas_emm.parquet        emm type distribution

NNDSS publishes counts as a year-to-date running total that resets each MMWR
year (national 2024 runs 5 -> 647 over weeks 1-52), so build.R de-accumulates
it into stss_cases_weekly and keeps the published cumulative series alongside.
27 of 12,376 weekly increments are negative, from NNDSS revising earlier weeks
downward; these are left as reported rather than clamped, and build.R logs the
count.

Epic suppression is mapped per measure: the upstream numerator flag covers
n_strep_throat and pct_strep_throat (the percent derives from that same cell),
the denominator flag covers n_patients.

measure_info.json documents every column of every parquet -- verified 0
undocumented columns and all 34 `levels` source_id references resolving against
the contributing sources -- plus a `_bundle` block mapping each parquet to the
source files it is built from.

docs/ and resources/ regenerated with scripts/build_docs.R (14 bundles,
46 sources).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@MadiBal

MadiBal commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by #189, which rebuilds this bundle on top of #188. Changes since: Epic GAS is read from epic_resp_infections rather than a separate source, geography_fips is carried alongside the geography name, and ABCs Group B Streptococcus is included (it was missing entirely). The abcs_gas ingest also needed a substantial rewrite — CDC restructured the emm and syndrome topics in their 2026 release, and re-running the version here produced an empty emm file and 100x-too-small resistance values.

@MadiBal MadiBal closed this Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant