A rollout should cost a source two minutes, not a day - #42
Merged
Conversation
Since 0.4.0: #40 added three keyless sources. NDBC's every-buoy file into water, with wave height, period and direction read out the way a surf report says them; the National Weather Service surf zone forecast from every coastal office beside it; and the ADS-B emergency squawk and military watch lists into aviation, held as episodes rather than as positions. Twenty-seven collections, seventy-nine adapters. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjTtJPEpyJbvPeQcVfYPPs
Three deploys this afternoon each stranded a batch of freshly seeded sources, and the reason is worse than the error text suggests. When a deploy adds adapters, the new container seeds their sources and enqueues the first runs, and the container still draining takes some of those jobs. It does not have the new adapters, so `runSource` writes `unknown adapter`. That part is harmless and self-correcting. What is not harmless is that `startRun` pushes `next_run_at` a full cadence forward BEFORE the adapter is looked up. So the source does not fail and retry. It forfeits its whole slot. Measured on the live database just now: every 5m recovered within minutes (aircraft-emergency, run 2, 1 item) every 60m next attempt 19:50 (~1 hour of nothing) every 180m next attempt 21:50 (3 hours) every 720m next attempt 06:50 TOMORROW (11 hours) every 1440m next attempt 18:50 TOMORROW (23 hours) Twenty-four of thirty-six new sources were sitting in that state, most of a day of silence bought by a rollout that lasted seconds. "It heals on the next run" was true and almost useless. Two changes. The unknown-adapter path now asks to be called back in two minutes instead of letting the forfeited slot stand. Two minutes covers a rollout overlap and costs nothing if the adapter really is gone, because the run after that parks it again. And `ensureDefaults` now brings forward anything already parked on that error for an adapter this build actually has. Boot is exactly the moment the adapter appears, so it is the right place to ask. The query is deliberately narrow: only `unknown adapter`, only adapters now registered, only enabled sources, only ones not already due. A source that failed for any other reason keeps its schedule, and one naming an adapter that genuinely no longer exists stays parked rather than spinning the scheduler every tick. Both are tested against a real Postgres in process rather than mocked, because the entire behaviour lives in an `update ... where` and there is nothing else to check. Eight tests: the repair works, it is idempotent, it ignores other errors, it ignores deleted adapters, it ignores disabled sources, an empty adapter list is not a wildcard, and the full-cadence forfeit is pinned so nobody removes the two-minute retry without a test going red. 604 tests pass, biome clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjTtJPEpyJbvPeQcVfYPPs
ralyodio
added a commit
that referenced
this pull request
Sep 9, 2026
…last table (#44) Two fixes since 0.5.0. #42: `startRun` pushes next_run_at a full cadence forward before the adapter is looked up, so a source that met a draining container did not fail and retry, it forfeited its whole slot -- up to 23 hours for the daily register sources, and 24 of 36 new sources were in that state across three deploys. The unknown-adapter path now asks for a two-minute retry, and boot brings forward anything already parked on that error for an adapter the build actually has. #43: both NTSB sources share one extract, and readiness was keyed on the first table written rather than the last, so the second source read a file still being written. A marker is now written after all three. Twenty-seven collections, seventy-nine adapters. Claude-Session: https://claude.ai/code/session_01CjTtJPEpyJbvPeQcVfYPPs Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three deploys this afternoon each stranded a batch of freshly seeded sources. The
unknown adaptererror is harmless; what is not is thatstartRunpushesnext_run_ata full cadence forward before the adapter is looked up, so the source does not fail and retry, it forfeits its whole slot.Measured on the live database:
24 of 36 new sources were in that state. "It heals on the next run" was true and almost useless.
Two changes. The unknown-adapter path asks to be called back in two minutes instead of letting the forfeited slot stand. And
ensureDefaultsbrings forward anything already parked on that error for an adapter this build actually has — boot is exactly when the adapter appears.The repair query is deliberately narrow: only
unknown adapter, only registered adapters, only enabled sources, only ones not already due. Another error keeps its schedule; a genuinely deleted adapter stays parked rather than spinning the scheduler.Tested against a real Postgres in process, because the whole behaviour is an
update ... where. Eight tests including the idempotency, the non-wildcard empty list, and a pin on the full-cadence forfeit so the two-minute retry cannot be removed silently.604 tests pass, biome clean.
🤖 Generated with Claude Code
https://claude.ai/code/session_01CjTtJPEpyJbvPeQcVfYPPs