Skip to content

index deadline interrupts nested dedupe savepoint and masks IndexDeadlineExceeded #625

Description

@EtanHey

Problem

On an M1 production index run with the default 1,800-second deadline, APSW's progress handler interrupted a dedupe merge inside the outer batch transaction. merge_duplicate_chunk() then attempted ROLLBACK TO dedupe_merge; SQLite reported no such savepoint: dedupe_merge, masking the intended deadline exception. The CLI exited 1 as a generic indexing failure.

Observed terminal evidence:

SQLITE_INTERRUPT ... [/* TRIGGER chunk_tags_update */ UPDATE chunks SET ...]
Error: no such savepoint: dedupe_merge
[ROLLBACK TO dedupe_merge]

The run had made durable incremental progress before the deadline, but operationally reported the misleading savepoint error instead of IndexDeadlineExceeded. At the measured rate (~2 files/minute for 342 files), the default 30-minute bound is also structurally too short for this host.

Likely path

  • VectorStore.upsert_chunks_batch() installs the APSW deadline progress handler around an outer transaction.
  • merge_duplicate_chunk() opens SAVEPOINT dedupe_merge and catches a generic exception.
  • When the progress handler interrupts an update, the cleanup path issues ROLLBACK TO dedupe_merge even though the interrupted transaction/savepoint is no longer available, replacing the primary exception.

Acceptance criteria

  • Preserve and surface IndexDeadlineExceeded when the progress handler expires, including through nested dedupe helpers.
  • Never mask the primary exception with no such savepoint during rollback/release cleanup.
  • Add a regression test that forces the deadline while a dedupe merge is active.
  • Make the scheduled runtime bound realistic/configurable for large M1 indexes, or make the command explicitly resumable/incremental with a successful deadline exit contract.
  • Emit enough progress/exit telemetry to distinguish a clean bounded run from a genuine indexing defect.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions