The NTSB cache is ready when the last table lands, not the first - #43
Merged
Conversation
`ntsb-fatal-accidents` failed its first real run with a bare ENOENT, and the cause is a race I wrote into the reader. Both NTSB sources read the same extract from the same directory, which is deliberate: one 96 MB download and one 558 MB Access database serve every source over that archive. But the readiness check was `exists(events.ndjson)`, and `events` is the FIRST of three tables written. `narratives` is the last and takes seconds longer. So the second source arrived mid-extraction, saw the events file, concluded the cache was warm, skipped the download, and then read an aircraft table that was still being written. The fix is a marker file written after every table, so "ready" means all of them rather than the first of them. The archive is also named per publication date now, since two sources unpacking `avall.zip` into one directory under different dates was the same mistake waiting in a different place. Beyond that, a source that finds a table genuinely missing -- a container that died between two exports leaves exactly that -- now deletes the marker and says which table is absent, so the next run re-fetches instead of raising ENOENT against a path nobody can interpret. Tested by pinning the thing the race depended on: both sources are seeded without their own `cacheDir`, because giving either one its own directory would mean two 96 MB downloads, and it was precisely their sharing one that made the ordering matter. 605 tests pass, biome clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjTtJPEpyJbvPeQcVfYPPs
ralyodio
added a commit
that referenced
this pull request
Sep 9, 2026
…last table (#44) Two fixes since 0.5.0. #42: `startRun` pushes next_run_at a full cadence forward before the adapter is looked up, so a source that met a draining container did not fail and retry, it forfeited its whole slot -- up to 23 hours for the daily register sources, and 24 of 36 new sources were in that state across three deploys. The unknown-adapter path now asks for a two-minute retry, and boot brings forward anything already parked on that error for an adapter the build actually has. #43: both NTSB sources share one extract, and readiness was keyed on the first table written rather than the last, so the second source read a file still being written. A marker is now written after all three. Twenty-seven collections, seventy-nine adapters. Claude-Session: https://claude.ai/code/session_01CjTtJPEpyJbvPeQcVfYPPs Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ntsb-fatal-accidentsfailed its first real run with a bare ENOENT.Both NTSB sources read the same extract from the same directory — deliberately, so one 96 MB download serves both. But the readiness check was
exists(events.ndjson), andeventsis the first of three tables written whilenarrativesis the last. The second source arrived mid-extraction, saw the events file, skipped the download, and read an aircraft table still being written.Now a marker file is written after every table, so "ready" means all of them. The archive is named per publication date too, since two sources unpacking
avall.zipinto one directory was the same mistake waiting elsewhere. And a genuinely missing table deletes the marker and names itself, so the next run re-fetches rather than raising ENOENT against an uninterpretable path.Tested by pinning what the race depended on: neither source may carry its own
cacheDir, because separating them would mean two 96 MB downloads and it was their sharing that made ordering matter.605 tests pass, biome clean.
🤖 Generated with Claude Code
https://claude.ai/code/session_01CjTtJPEpyJbvPeQcVfYPPs