Add bundle_gas: Group A Strep across Epic Cosmos, NNDSS, and ABCs - #187
Closed
MadiBal wants to merge 2 commits into
Closed
Add bundle_gas: Group A Strep across Epic Cosmos, NNDSS, and ABCs#187MadiBal wants to merge 2 commits into
MadiBal wants to merge 2 commits into
Conversation
Brings the abcs_gas source onto main-line development from the stale `gas` branch, where it was created but never merged. CDC dataset 9y49-tura: national, annual (1997-2023) invasive Group A Strep case/death rates and counts, clinical syndrome distribution, antibiotic resistance, and emm type distribution. Needed by bundle_gas. Note that data/abcs is a different pathogen entirely (invasive pneumococcal disease by serotype) and has no GAS measures. Known gap, pre-existing and not addressed here: this source's measure_info.json uses generic keys (case_rate, syndrome, pct_resistant, emm_type) that do not match its actual column names, so dcf_check reports 36 undocumented columns. The bundle documents these columns itself via `levels` inheritance. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Initialized with dcf::dcf_add_bundle("bundle_gas", source_files = ...) so
process.json carries type: "bundle" and scripts: build.R.
build.R writes six long-format parquets to dist/, each with geography as a
state name (or "United States"), ISO time, a `measure` identifier, and `value`:
epic_gas.parquet n_strep_throat / pct_strep_throat / n_patients,
quarterly by state and age, with a per-measure
`suppressed` flag
nnds_stss.parquet streptococcal_toxic_shock_syndrome only, weekly
abcs_gas.parquet case/death rates and counts, annual national
abcs_gas_syndromes.parquet clinical syndrome distribution
abcs_gas_resistance.parquet antibiotic resistance
abcs_gas_emm.parquet emm type distribution
NNDSS publishes counts as a year-to-date running total that resets each MMWR
year (national 2024 runs 5 -> 647 over weeks 1-52), so build.R de-accumulates
it into stss_cases_weekly and keeps the published cumulative series alongside.
27 of 12,376 weekly increments are negative, from NNDSS revising earlier weeks
downward; these are left as reported rather than clamped, and build.R logs the
count.
Epic suppression is mapped per measure: the upstream numerator flag covers
n_strep_throat and pct_strep_throat (the percent derives from that same cell),
the denominator flag covers n_patients.
measure_info.json documents every column of every parquet -- verified 0
undocumented columns and all 34 `levels` source_id references resolving against
the contributing sources -- plus a `_bundle` block mapping each parquet to the
source files it is built from.
docs/ and resources/ regenerated with scripts/build_docs.R (14 bundles,
46 sources).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Superseded by #189, which rebuilds this bundle on top of #188. Changes since: Epic GAS is read from |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Combines Group A Streptococcus surveillance from three sources into six long-format parquets.
Initialized with
dcf::dcf_add_bundle("bundle_gas", source_files = ...), soprocess.jsoncarriestype: "bundle"andscripts: build.R.Outputs
All six share the bundle conventions:
geographyholds state names (or"United States"),timeis ISOYYYY-mm-ddperiod-end,valueis the plotting column, keyed by ameasureidentifier column.epic_gas.parquetepic_gasn_strep_throat/pct_strep_throat/n_patients, plus a per-measuresuppressedflagnnds_stss.parquetnndsstss_cases_weekly/stss_cases_cumulativeabcs_gas.parquetabcs_gasabcs_gas_syndromes.parquetabcs_gasabcs_gas_resistance.parquetabcs_gasabcs_gas_emm.parquetabcs_gasTwo things that most need a second pair of eyes
1. NNDSS is published cumulatively, and I de-accumulate it. The raw
streptococcal_toxic_shock_syndromecolumn is a year-to-date running total that resets each MMWR year — national 2024 runs 5 → 647 across weeks 1–52. Plotted as-is it would show a sawtooth, not incidence.build.Rdifferences it within geography-year intostss_cases_weeklyand keeps the published cumulative series alongside; the two are not additive. The earlierbundle_gasattempt on the stalegasbranch did the same, and thenndssourcemeasure_infodocuments the column as "Cumulative year-to-date", so this looks right — but it is the one substantive transformation in this PR and worth confirming.Relatedly, 27 of 12,376 weekly increments come out negative, because NNDSS revises earlier weeks downward. I left them as reported rather than clamping to zero, and
build.Rlogs the count. Clamping is a one-line change if preferred.2. Which "ABCS" this is.
data/abcsonmainis invasive pneumococcal disease by serotype — a different pathogen, with no GAS measures. The Group A Strep ABCs data (CDC dataset9y49-tura) existed only on the stalegasbranch and was never merged, so the first commit here bringsabcs_gasonto main-line development.Known gap, pre-existing: that source's own
measure_info.jsonuses generic keys (case_rate,syndrome,pct_resistant,emm_type) that do not match its actual column names, sodcf_checkreports 36 undocumented columns on the source. The bundle documents those columns itself vialevelsinheritance, so the dist files and generated docs are complete — but the source-level check still fails. Happy to fix those 36 entries in a follow-up or here.Verification
dcf::dcf_process("bundle_gas")runs clean and is deterministic — re-running produces no diff.measure_info.jsonparses; 0 undocumented columns across all 6 parquets, and all 34levelssource_idreferences resolve against the contributing sources'measure_info.json._bundleblock covers every parquet, itssourcesequals the union of the per-file sources, and every listedsource_fileexists.scripts/build_docs.Rregenerated (14 bundles, 46 sources);bundle_gasrenders a short name plus a description or value list for all 29 columns.Smaller decisions
value-column convention rather than kept wide as the earlier attempt did.n_strep_throatandpct_strep_throat(the percent derives from that same cell), the denominator flag coversn_patients.bundle_enteric_diseases.abcs_gasis national only (a catchment area, not the whole US), and the three sources' time spans differ (Epic 2017→2025, NNDSS 2022→present, ABCs 1997→2023), so cross-source comparison is limited to the overlap. Both are documented in the bundle README.🤖 Generated with Claude Code