fix(harvest): reprobe newly-discovered slugs on the next pass#161
Merged
Conversation
a skip-probe discovery recorded a brand-new slug as transient_failure with last_probed_at set to the discovery moment. the reprobe only revisits rows older than --max-age-days (7), so a never-probed slug looked freshly probed and sat unprobed for a full week — invisible to reprobe (too fresh) and to scrape (not yet live). that is why several 2026-07 connectors show only their hand-seeds live while thousands of real, discoverable tenants wait in limbo. stamp new skip-probe slugs with an epoch last_probed_at sentinel so the next reprobe includes them immediately (first_seen_at stays the real discovery day), matching the sitemap backend's liveness-truth convention. also backfill the existing limbo: reset last_probed_at to epoch on the unprobed transient_failure rows of the connectors onboarded this month so the next reprobe promotes the live ones.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes why the connectors onboarded this month show only a handful of live jobs. Not a parser bug — the connectors work (jibeapply's 3 seeds = 3,352 jobs exactly; live-probing 25 random discovered hireology slugs gives 16×200 / 9×404, so they're real).
Root cause
Discovery's skip-probe path recorded a brand-new, never-probed slug as
transient_failurewithlast_probed_at = now. The reprobe (--max-age-daysdefault 7) only revisits rows older than the cutoff, so a never-probed slug looked freshly probed and sat unprobed for a full week — invisible to reprobe (too fresh) and scrape (only takesstatus==live). Every new connector's reprobe logged0 tenants older than 7 days; nothing to do. The "fat" connectors (careerplug 6,376 jobs, jibeapply 3,352) just have high-volume seeds; the "thin" ones (hireology 12, isolvedhire 11) have small seeds — same underlying state, thousands of real tenants stuck in limbo.Fix
runner.ts): stamp new skip-probe slugs with an epochlast_probed_atsentinel (1970-01-01T00:00:00.000Z, matching the sitemap backend's convention) so the next reprobe includes them immediately.first_seen_atstays the real discovery day. Newrunner.test.tsassertion.last_probed_atto epoch on the 14,234 unprobedtransient_failurerows of the six affected connectors (careerplug/jibeapply/isolvedhire/applitrack/hiringthing/hireology) so the next reprobe promotes the live ones now instead of waiting out the window. Byte-verified: onlylast_probed_atchanged, only ontransient_failurerows, live/dead untouched, all schema-valid.Safety (adversarially verified)
--stale-ttl-daysdrops job rows bylast_seen_at, a different entity — can't drop a tenant.liverows; epoch rows aretransient_failure, so they can't trip/suppress it.now, strictly shrinking the stale set.Gates
bun run test1656 pass · typecheck · lint clean. Adversarial review: SHIP (no Critical/Major/Minor).Follow-up after merge: dispatch
weekly-harvestmode=reprobe-onlyto promote the freed tenants; the thin connectors should climb into the thousands.