Add a multi-reference version of the existing one-to-one indicator comparison report

[jingjtang]

The new report evaluates one candidate indicator against multiple reference indicators in a single render. Each diagnostic remains pairwise.

Preview

Rendered HTML preview

Main Changes

• Added multi-reference parameters for reference sources, signals, names, and optional CSV inputs.
• Reused the core logic from indicator_comparison.qmd for:
  • exploratory data checks
  • coverage and missingness
  • coefficient of variation comparison
  • spatial correlation over locations
  • temporal correlation within locations
  • lagged correlation analysis
  • geographic agreement at optimal lag
• Added matrix-based Spearman correlation helpers to compute candidate-vs-multiple-reference correlations more efficiently than repeatedly rendering one-to-one reports.
• Added a reference-agreement context check to show agreement among selected reference indicators.
• Standardized table formatting and plot labels for automated reports with variable numbers of references.

Validation

Rendered successfully with:

#!/usr/bin/env Rscript

options(repos = c(
  delphi = "https://cmu-delphi.r-universe.dev",
  CRAN = "https://cloud.r-project.org"
))

if (!require("pacman")) install.packages("pacman")
pacman::p_load(quarto, here, fs)

data_dir <- here::here("indicator_analysis", "data", "multi_comparison_render")

candidate_csv <- fs::path_abs(fs::path(data_dir, "candidate_doctor_visits.csv"))
reference_csvs <- fs::path_abs(c(
  fs::path(data_dir, "ref_quidel.csv"),
  fs::path(data_dir, "ref_hospital_admissions.csv"),
  fs::path(data_dir, "ref_chng_inpatient.csv"),
  fs::path(data_dir, "ref_chng_outpatient.csv")
))

missing_files <- c(candidate_csv, reference_csvs)[!file.exists(c(candidate_csv, reference_csvs))]
if (length(missing_files) > 0) {
  stop(sprintf("Missing CSV snapshots:\n%s", paste(missing_files, collapse = "\n")))
}

old_wd <- setwd(here::here("indicator_analysis"))
on.exit(setwd(old_wd), add = TRUE)

quarto::quarto_render(
  input = "indicator_multi_comparison.qmd",
  output_file = "indicator_multi_comparison.html",
  cache_refresh = TRUE,
  execute_params = list(
    candidate_source = "doctor-visits",
    candidate_indicator = "smoothed_adj_cli",
    candidate_name = "Doctor Visits: Smoothed Adj CLI",
    candidate_csv = candidate_csv,
    reference_sources = c("quidel", "hospital-admissions", "chng", "chng"),
    reference_indicators = c(
      "covid_ag_smoothed_pct_positive",
      "smoothed_covid19_from_claims",
      "7dav_inpatient_covid",
      "7dav_outpatient_covid"
    ),
    reference_names = c(
      "Quidel: Smoothed % Positive COVID Antigen",
      "Hospital Admissions: Smoothed COVID-19 from Claims",
      "CHNG: 7-day Avg Inpatient COVID",
      "CHNG: 7-day Avg Outpatient COVID"
    ),
    reference_csvs = reference_csvs,
    geo_type = "state",
    time_type = "day",
    start_day = "2020-09-01",
    end_day = "2023-03-01",
    max_locations_plot = 12
  )
)

[JavierMtzRdz]

Thanks for this! It improves almost all sections. Since it doesn't conflict with anything, we can merge once you add some instructions on how to use this to the READMEs and address the suggestions and issues. Those marked as minor suggestions (most of my comments) are more open recommendations. Then, eventually, we could replace indicator_correlation.qmdwith this.

If you also include the processing that uses only the necessary parameter depending on the indicator source and handle edge cases with either one location or many candidates, I'm happy to replace indicator_correlation.qmd with this, but that is not necessary. We can leave the full replacement for later.

[JavierMtzRdz]

Minor suggestion: it is worth including an "extra_key" parameter to add extra key columns when creating the epi_df. This works as is, but it may be useful for indicators with age_group.

[JavierMtzRdz]

Minor suggestion: Some of these functions seem to be used only once. In those cases, we apply the theme layers directly for easier debugging.

[JavierMtzRdz]

Praise: Thanks for this function. At some point, I may port it to all other notebooks.

[JavierMtzRdz]

suggestion: refer to the actual repos of the supporting packages to make sure they are up to date.

Suggested change

	options(repos = c(
	delphi = "https://cmu-delphi.r-universe.dev",
	CRAN = "https://cloud.r-project.org"
	))
	if (!require("pak")) install.packages("pak")
	if (!requireNamespace("epiprocess", quietly = TRUE)) pak::pkg_install("cmu-delphi/epiprocess")
	if (!requireNamespace("epidatr", quietly = TRUE)) pak::pkg_install("cmu-delphi/epidatr")

[JavierMtzRdz]

suggestion: I'm inclined to leave these parameters empty to ensure the user provides all the information required for the indicator they are interested in. However, the required indicators vary depending on whether the user is calling an indicator from the V5 API, the new API, or providing a CSV file. The last version of the correlation notebook has some of this.

[JavierMtzRdz]

This appears empty in the rendered version.

[JavierMtzRdz]

issue: Most epidata indicators do not return NA when there is no data for a location or time. Instead, they omit the row. Therefore, summarizing missingness involves mapping all relevant time points and locations that could potentially contain each candidate. For example, the plot above shows some locations with temporal gaps.

[JavierMtzRdz]

question: Is there a reason to prefer a plot over a table for this information?

[JavierMtzRdz]

I'm aiming to keep the rendered files separate from the notebooks and code. Alternatively, you could include the rendering codes in the script/{post/pre}_render.R scripts to generate the output and then add it to the gh branch.

[JavierMtzRdz]

Thanks for this. It looks great!

[JavierMtzRdz]

I've also added you to the repo, so there's no need to create forks.