fix: index --full FK crash (#16) + cross-process contextfit ingest lock (#17) - #20
Merged
Merged
Conversation
`index --full` failed with `FOREIGN KEY constraint failed` on any vault with a populated `sections` table. The full-mode wipe loop deleted chunks while `sections.chunk_id_first/last` (REFERENCES chunks(id), no ON DELETE) still pointed at them. The per-note re-index path already ordered these correctly; only the full-wipe path missed it. Add `sections.deleteByNote(n.id)` before `chunks.deleteByNote(n.id)`, mirroring the per-note path and single.ts step 7. Add a regression test that asserts the wrong order trips the FK and the corrected order does not. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Concurrent ContextFit KB ingests were not mutually excluded across processes. A CLI `index` running while a serve process fired its debounced re-ingest (or a second stale serve) had both `contextfit ingest` calls `rm` and rewrite the same KB dir; the loser crashed mid-write (`FileNotFoundError: .../chunks/index.json.tmp`), sometimes leaving a half-written KB. The existing `<vault>.lock` (brief daemon, held for its whole lifetime) can't serialize this — reusing it would make every ingest skip forever. Add a dedicated per-vault mutex `locks/<vault>.ingest.lock` (atomic `wx` create, PID-based steal-on-dead, mirroring brief/lock.ts) wrapped inside `indexVaultWithContextFit`, the single chokepoint for all four call sites (CLI index, serve write-refresh, watcher re-ingest, startup catch-up). Contention policy is SKIP + a persisted `<vault>.ingest.dirty` flag: a second-comer marks the vault dirty and returns immediately (no wait, no wasted double-rebuild) with new status "skipped" (treated as success by callers). The lock holder clears the flag before each pass and does one trailing re-ingest if a write set it mid-run, so the latest change is never lost even across processes; MAX_PASSES bounds churn. Startup catch-up honors a leftover dirty flag so a crash-stranded flag is cleaned up at the next boot. Adds ingest-lock.ts + 14 tests (lock primitive: acquire/steal/release, dirty round-trip; orchestration: skip-and-flag, single ingest, trailing re-ingest, lock-released-on-throw, MAX_PASSES backstop). Callers updated to log "skipped" benignly rather than as an error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes two open bugs in the ContextFit backend.
#16 —
index --fullfails withFOREIGN KEY constraint failedThe full-mode wipe loop deleted chunks while
sections.chunk_id_first/last(REFERENCES chunks(id), no ON DELETE) still pointed at them. The per-note
re-index path already ordered these correctly; only the full-wipe path missed it.
Fix: delete sections before chunks in the wipe loop + regression test.
#17 — concurrent contextfit ingests corrupt the KB
A CLI
indexrunning alongside a serve-side debounced re-ingest (or a secondstale serve) had both
contextfit ingestcallsrm/rewrite the same KB dir;the loser crashed mid-write, sometimes leaving a half-written KB.
Fix: a dedicated per-vault mutex
locks/<vault>.ingest.lock(separate fromthe brief daemon's lifetime-held
<vault>.lock) wrapped around the single ingestchokepoint (
indexVaultWithContextFit), covering all four call sites (CLI index,serve write-refresh, watcher re-ingest, startup catch-up). Contention policy is
skip + persisted
.dirtyflag: a second-comer marks dirty and returnsimmediately (no wait, no wasted double-rebuild, new status
"skipped"); the lockholder does one trailing re-ingest if a write lands mid-run, so the latest change
is never lost across processes. Startup catch-up honors a crash-stranded flag.
Out of scope (not bugs in this repo)
cf#1); repo action is docs/version-pinning once fixed upstream.Verification
tsc --noEmitclean · adapter-seam lint green · prettier clean.Closes #16
Closes #17
🤖 Generated with Claude Code