Skip to content

fix(cso): normalize incidentDate/submittedDate to a uniform string on write - #114

Merged
nesanders merged 1 commit into
mainfrom
fix/cso-date-format-mixed
Aug 22, 2026
Merged

fix(cso): normalize incidentDate/submittedDate to a uniform string on write#114
nesanders merged 1 commit into
mainfrom
fix/cso-date-format-mixed

Conversation

@nesanders

Copy link
Copy Markdown
Owner

Problem

The 2026-08-22 dispatched Weekly Update run (32592642457) got past the lobbying-fetch timeout (fixed by #111/#112 — confirmed, no longer an issue) but then crashed in "Generate dashboard charts":

ValueError: time data "2026-08-08" doesn't match format "%Y-%m-%d %H:%M:%S"

in EEA_DP_CSO_map.py's load_data_cso(), the first parse of MAEEADP_CSO.incidentDate read back from AMEND.db.

Root cause

Reproduced locally against the live EEA CSOAPI. incidentDate is always midnight in the API — time-of-day lives in a separate incidentTime field. This week's incremental batch of new incidents happened to be entirely midnight values; pandas' to_csv formats that batch as bare "YYYY-MM-DD" while older rows in the same file keep "YYYY-MM-DD HH:MM:SS" — producing a genuinely mixed-format CSV column that a plain pd.to_datetime() re-parse can't handle.

A second, related gap: get_data() parsed incidentDate on load but never submittedDate, so pd.concat([existing, new_df]) silently produced a mixed object column (raw CSV strings next to real Timestamps) for submittedDate too — already present in the committed CSV, just never triggered because nothing downstream re-parses that column directly.

Fix

  • get_eea_dp_cso.py get_data(): also parse existing['submittedDate'] on load (matching incidentDate), so both columns are uniformly typed before concat.
  • get_eea_dp_cso.py write_data(): explicitly normalize both date columns to a canonical "%Y-%m-%d %H:%M:%S" string (format='mixed' before formatting) before writing, instead of relying on pandas' to_csv default formatting.
  • EEA_DP_CSO_map.py load_data_cso(): format='mixed' on the read-side parse as a defensive fallback, so a future anomaly degrades gracefully instead of crashing the whole dashboard build.

Verification

  • incidentDate values preserved exactly (0 mismatches vs. previously-committed data).
  • submittedDate differs only by sub-second truncation (max 997ms, zero full-second+ diffs) — intentional; that field is never used for precision-sensitive analysis.
  • The exact crash-reproducing call (pd.to_datetime(), no format, on the written column) now succeeds.
  • Includes the freshly re-fetched, now-uniform CSO data (through 2026-08-22) since the crashed run never reached the commit step.

🤖 Generated with Claude Code

… write

The 2026-08-22 Weekly Update run got past the lobbying-fetch timeout (fixed by
#111/#112) but then crashed in "Generate dashboard charts":

  ValueError: time data "2026-08-08" doesn't match format "%Y-%m-%d %H:%M:%S"

in EEA_DP_CSO_map.py's load_data_cso(), the first parse of MAEEADP_CSO.incidentDate
read back from AMEND.db.

Root cause (reproduced locally against the live EEA CSOAPI): incidentDate is
always midnight in the API — time-of-day lives in a separate incidentTime field.
This week's incremental batch of new incidents happened to be entirely midnight
values; pandas' to_csv formats that batch as bare "YYYY-MM-DD" while older rows
in the same file keep "YYYY-MM-DD HH:MM:SS", producing a genuinely mixed-format
CSV column that a plain pd.to_datetime() re-parse can't handle.

A second, related gap: get_data() parsed incidentDate on load but never
submittedDate, so pd.concat([existing, new_df]) silently produced a mixed
object column (raw CSV strings next to real Timestamps) for submittedDate too
— already present in the committed CSV, just never triggered because nothing
downstream re-parses that column directly.

Fixes:
- get_eea_dp_cso.py get_data(): also parse existing['submittedDate'] on load
  (matching incidentDate), so both columns are uniformly typed before concat.
- get_eea_dp_cso.py write_data(): explicitly normalize both date columns to a
  canonical "%Y-%m-%d %H:%M:%S" string (format='mixed' before formatting, to
  handle whatever mix reaches it) before writing, instead of relying on
  pandas' to_csv default formatting.
- EEA_DP_CSO_map.py load_data_cso(): format='mixed' on the read-side parse as a
  defensive fallback, so a future anomaly degrades gracefully instead of
  crashing the whole dashboard build.

Verified against the real EEA CSOAPI: incidentDate values are preserved exactly
(0 mismatches vs. the previously-committed data); submittedDate differs only by
sub-second truncation (max 997ms, zero full-second+ diffs) — an intentional
trade-off for a metadata timestamp that's never used for precision-sensitive
analysis. The exact crash-reproducing call (plain pd.to_datetime on the written
column) now succeeds. Includes the freshly re-fetched, now-uniform CSO data
(through 2026-08-22) since the crashed run never reached the commit step.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@nesanders
nesanders merged commit 48a34a7 into main Aug 22, 2026
6 checks passed
@nesanders
nesanders deleted the fix/cso-date-format-mixed branch August 22, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant