Skip to content

fix(csv): recover refused rows instead of re-importing the file - #34

Merged
IBJunior merged 2 commits into
mainfrom
fix/csv-import-recovery-and-validation
Sep 14, 2026
Merged

IBJunior merged 2 commits into
mainfrom
fix/csv-import-recovery-and-validation

Conversation

@IBJunior

Copy link
Copy Markdown
Member

What happened

Importing a real 451-row export, 9 rows were refused: the file mixed 06/09/2026 22:41:15 with a
bare 07/09/2026, and the declared dd/MM/yyyy HH:mm:ss matched only the first shape. The only
way to rescue them was to re-run the import with a different format — which refused the other 442
and imported a second copy of the 9. Three passes later the ledger held triplicates, and
skippedDuplicates: 0 said nothing was wrong: dedup keys on (source, external_id), and in
Postgres NULL never equals NULL, so for a file with no id column nothing is deduplicated at
all. The wrong totals fed several charts before anyone noticed.

Root cause is not the date parser. The importer gave the agent no way to recover a single row,
so its only recovery move was to re-run the whole file
— and the system prompt explicitly told it
to.

Changes

  • read_csv_rows — read refused rows by number, raw. Refused rows now carry a row number and a
    reason (badDateRows, unparsableRows) under one convention: 1-based over data rows. Recovery is
    per row via log_expense, never a re-import.
  • validate_csv_import — parses the whole file, writes nothing. The import returns
    validation_required for a plan it has not seen, fingerprinted over every argument, so changing
    only dateFormat re-triggers it. A separate tool rather than a dryRun flag because approval is
    keyed by tool name — a flag would make auto-approval depend on a model-authored boolean.
  • Optional time — a declared format carrying a time absorbs rows without one; those land at
    00:00:00 and are counted in datesWithoutTime.
  • Honest duplicatesduplicateDetection: "active" | "unavailable" plus a warning, replacing
    skippedDuplicates: 0.

Prompt, capabilities page, tool renderers and eval cases follow the four-step flow. Docs in
docs/CSV_IMPORT_FLOW.md.

Verification

  • 246 unit tests, root + eval typecheck, Prettier — all clean.
  • Full eval suite: 20/20, 0 inconclusive, no regressions outside CSV.
  • Both new cases at full policy: csv-import-handles-rows-without-a-time 3/3 (strict),
    csv-import-validates-before-writing 3/3 (majority). Across nine runs validation always
    preceded the write, always auto-approved, and the write always paused for approval.

Not included

  • Content-hash dedup. The retry loop is closed, but a deliberate re-import (new month, corrected
    export) still duplicates.
  • The 24 duplicate rows still in the production ledger — real financial data, needs a backup first.

🤖 Generated with Claude Code

IBJunior and others added 2 commits September 14, 2026 12:40
A strict date format refused rows the file expressed differently, and the
only way to rescue them was re-running the whole import — which duplicated
everything that had already landed, because dedup keys on (source,
external_id) and NULL never equals NULL in Postgres.

- `read_csv_rows` reads refused rows by number so they can be logged
  individually; refused rows now carry a row number and reason.
- `validate_csv_import` parses the whole file and writes nothing; the
  import refuses a plan it has not seen, fingerprinted over every argument.
  A separate tool, not a flag, because approval is keyed by tool name.
- A declared format with a time absorbs rows without one (counted in
  `datesWithoutTime`) rather than refusing them.
- Duplicate detection is reported as active/unavailable, never as
  `skippedDuplicates: 0`.

Docs in docs/CSV_IMPORT_FLOW.md. Prompt, capabilities, renderers and eval
cases follow the four-step flow.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The prompt restated the runtime hints almost verbatim — most-refused vs
few-refused, re-import duplicates, revalidate after changing an argument.
That guidance belongs in the payload, where it appears at the moment it
applies. Steps 3 and 4 now state the flow and defer to the result's `hint`.

Verified: both CSV cases still 3/3 at full policy, unchanged trajectories,
no validation_required bounces.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@IBJunior
IBJunior merged commit f3c4753 into main Sep 14, 2026
1 check passed
@IBJunior
IBJunior deleted the fix/csv-import-recovery-and-validation branch September 14, 2026 11:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant