fix(csv): recover refused rows instead of re-importing the file - #34
Merged
Merged
Conversation
A strict date format refused rows the file expressed differently, and the only way to rescue them was re-running the whole import — which duplicated everything that had already landed, because dedup keys on (source, external_id) and NULL never equals NULL in Postgres. - `read_csv_rows` reads refused rows by number so they can be logged individually; refused rows now carry a row number and reason. - `validate_csv_import` parses the whole file and writes nothing; the import refuses a plan it has not seen, fingerprinted over every argument. A separate tool, not a flag, because approval is keyed by tool name. - A declared format with a time absorbs rows without one (counted in `datesWithoutTime`) rather than refusing them. - Duplicate detection is reported as active/unavailable, never as `skippedDuplicates: 0`. Docs in docs/CSV_IMPORT_FLOW.md. Prompt, capabilities, renderers and eval cases follow the four-step flow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The prompt restated the runtime hints almost verbatim — most-refused vs few-refused, re-import duplicates, revalidate after changing an argument. That guidance belongs in the payload, where it appears at the moment it applies. Steps 3 and 4 now state the flow and defer to the result's `hint`. Verified: both CSV cases still 3/3 at full policy, unchanged trajectories, no validation_required bounces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
Importing a real 451-row export, 9 rows were refused: the file mixed
06/09/2026 22:41:15with abare
07/09/2026, and the declareddd/MM/yyyy HH:mm:ssmatched only the first shape. The onlyway to rescue them was to re-run the import with a different format — which refused the other 442
and imported a second copy of the 9. Three passes later the ledger held triplicates, and
skippedDuplicates: 0said nothing was wrong: dedup keys on(source, external_id), and inPostgres
NULLnever equalsNULL, so for a file with no id column nothing is deduplicated atall. The wrong totals fed several charts before anyone noticed.
Root cause is not the date parser. The importer gave the agent no way to recover a single row,
so its only recovery move was to re-run the whole file — and the system prompt explicitly told it
to.
Changes
read_csv_rows— read refused rows by number, raw. Refused rows now carry a row number and areason (
badDateRows,unparsableRows) under one convention: 1-based over data rows. Recovery isper row via
log_expense, never a re-import.validate_csv_import— parses the whole file, writes nothing. The import returnsvalidation_requiredfor a plan it has not seen, fingerprinted over every argument, so changingonly
dateFormatre-triggers it. A separate tool rather than adryRunflag because approval iskeyed by tool name — a flag would make auto-approval depend on a model-authored boolean.
00:00:00and are counted indatesWithoutTime.duplicateDetection: "active" | "unavailable"plus a warning, replacingskippedDuplicates: 0.Prompt, capabilities page, tool renderers and eval cases follow the four-step flow. Docs in
docs/CSV_IMPORT_FLOW.md.Verification
csv-import-handles-rows-without-a-time3/3 (strict),csv-import-validates-before-writing3/3 (majority). Across nine runs validation alwayspreceded the write, always auto-approved, and the write always paused for approval.
Not included
export) still duplicates.
🤖 Generated with Claude Code