fix(cso): normalize incidentDate/submittedDate to a uniform string on write - #114
Merged
Conversation
… write The 2026-08-22 Weekly Update run got past the lobbying-fetch timeout (fixed by #111/#112) but then crashed in "Generate dashboard charts": ValueError: time data "2026-08-08" doesn't match format "%Y-%m-%d %H:%M:%S" in EEA_DP_CSO_map.py's load_data_cso(), the first parse of MAEEADP_CSO.incidentDate read back from AMEND.db. Root cause (reproduced locally against the live EEA CSOAPI): incidentDate is always midnight in the API — time-of-day lives in a separate incidentTime field. This week's incremental batch of new incidents happened to be entirely midnight values; pandas' to_csv formats that batch as bare "YYYY-MM-DD" while older rows in the same file keep "YYYY-MM-DD HH:MM:SS", producing a genuinely mixed-format CSV column that a plain pd.to_datetime() re-parse can't handle. A second, related gap: get_data() parsed incidentDate on load but never submittedDate, so pd.concat([existing, new_df]) silently produced a mixed object column (raw CSV strings next to real Timestamps) for submittedDate too — already present in the committed CSV, just never triggered because nothing downstream re-parses that column directly. Fixes: - get_eea_dp_cso.py get_data(): also parse existing['submittedDate'] on load (matching incidentDate), so both columns are uniformly typed before concat. - get_eea_dp_cso.py write_data(): explicitly normalize both date columns to a canonical "%Y-%m-%d %H:%M:%S" string (format='mixed' before formatting, to handle whatever mix reaches it) before writing, instead of relying on pandas' to_csv default formatting. - EEA_DP_CSO_map.py load_data_cso(): format='mixed' on the read-side parse as a defensive fallback, so a future anomaly degrades gracefully instead of crashing the whole dashboard build. Verified against the real EEA CSOAPI: incidentDate values are preserved exactly (0 mismatches vs. the previously-committed data); submittedDate differs only by sub-second truncation (max 997ms, zero full-second+ diffs) — an intentional trade-off for a metadata timestamp that's never used for precision-sensitive analysis. The exact crash-reproducing call (plain pd.to_datetime on the written column) now succeeds. Includes the freshly re-fetched, now-uniform CSO data (through 2026-08-22) since the crashed run never reached the commit step. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The 2026-08-22 dispatched
Weekly Updaterun (32592642457) got past the lobbying-fetch timeout (fixed by #111/#112 — confirmed, no longer an issue) but then crashed in "Generate dashboard charts":in
EEA_DP_CSO_map.py'sload_data_cso(), the first parse ofMAEEADP_CSO.incidentDateread back fromAMEND.db.Root cause
Reproduced locally against the live EEA CSOAPI.
incidentDateis always midnight in the API — time-of-day lives in a separateincidentTimefield. This week's incremental batch of new incidents happened to be entirely midnight values; pandas'to_csvformats that batch as bare"YYYY-MM-DD"while older rows in the same file keep"YYYY-MM-DD HH:MM:SS"— producing a genuinely mixed-format CSV column that a plainpd.to_datetime()re-parse can't handle.A second, related gap:
get_data()parsedincidentDateon load but neversubmittedDate, sopd.concat([existing, new_df])silently produced a mixed object column (raw CSV strings next to realTimestamps) forsubmittedDatetoo — already present in the committed CSV, just never triggered because nothing downstream re-parses that column directly.Fix
get_eea_dp_cso.pyget_data(): also parseexisting['submittedDate']on load (matchingincidentDate), so both columns are uniformly typed before concat.get_eea_dp_cso.pywrite_data(): explicitly normalize both date columns to a canonical"%Y-%m-%d %H:%M:%S"string (format='mixed'before formatting) before writing, instead of relying on pandas'to_csvdefault formatting.EEA_DP_CSO_map.pyload_data_cso():format='mixed'on the read-side parse as a defensive fallback, so a future anomaly degrades gracefully instead of crashing the whole dashboard build.Verification
incidentDatevalues preserved exactly (0 mismatches vs. previously-committed data).submittedDatediffers only by sub-second truncation (max 997ms, zero full-second+ diffs) — intentional; that field is never used for precision-sensitive analysis.pd.to_datetime(), no format, on the written column) now succeeds.🤖 Generated with Claude Code