Skip to content

fix(harvest): reprobe newly-discovered slugs on the next pass#161

Merged
datascry merged 2 commits into
mainfrom
fix/reprobe-new-slug-epoch
Jul 6, 2026
Merged

fix(harvest): reprobe newly-discovered slugs on the next pass#161
datascry merged 2 commits into
mainfrom
fix/reprobe-new-slug-epoch

Conversation

@datascry

@datascry datascry commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Fixes why the connectors onboarded this month show only a handful of live jobs. Not a parser bug — the connectors work (jibeapply's 3 seeds = 3,352 jobs exactly; live-probing 25 random discovered hireology slugs gives 16×200 / 9×404, so they're real).

Root cause

Discovery's skip-probe path recorded a brand-new, never-probed slug as transient_failure with last_probed_at = now. The reprobe (--max-age-days default 7) only revisits rows older than the cutoff, so a never-probed slug looked freshly probed and sat unprobed for a full week — invisible to reprobe (too fresh) and scrape (only takes status==live). Every new connector's reprobe logged 0 tenants older than 7 days; nothing to do. The "fat" connectors (careerplug 6,376 jobs, jibeapply 3,352) just have high-volume seeds; the "thin" ones (hireology 12, isolvedhire 11) have small seeds — same underlying state, thousands of real tenants stuck in limbo.

Fix

  1. Code (runner.ts): stamp new skip-probe slugs with an epoch last_probed_at sentinel (1970-01-01T00:00:00.000Z, matching the sitemap backend's convention) so the next reprobe includes them immediately. first_seen_at stays the real discovery day. New runner.test.ts assertion.
  2. Backlog backfill: reset last_probed_at to epoch on the 14,234 unprobed transient_failure rows of the six affected connectors (careerplug/jibeapply/isolvedhire/applitrack/hiringthing/hireology) so the next reprobe promotes the live ones now instead of waiting out the window. Byte-verified: only last_probed_at changed, only on transient_failure rows, live/dead untouched, all schema-valid.

Safety (adversarially verified)

  • Reprobe has no age-based drop path — it filters stale → sorts oldest-first → probes → updates; every tenant is preserved. Epoch rows are probed, never dropped.
  • --stale-ttl-days drops job rows by last_seen_at, a different entity — can't drop a tenant.
  • Incident guard only counts previously-live rows; epoch rows are transient_failure, so they can't trip/suppress it.
  • No starvation: each connector's epoch count is under the 5,000 batch cap (drains in one run), and probed rows re-stamp to now, strictly shrinking the stale set.

Gates

bun run test 1656 pass · typecheck · lint clean. Adversarial review: SHIP (no Critical/Major/Minor).

Follow-up after merge: dispatch weekly-harvest mode=reprobe-only to promote the freed tenants; the thin connectors should climb into the thousands.

datascry added 2 commits July 6, 2026 10:08
a skip-probe discovery recorded a brand-new slug as transient_failure
with last_probed_at set to the discovery moment. the reprobe only
revisits rows older than --max-age-days (7), so a never-probed slug
looked freshly probed and sat unprobed for a full week — invisible to
reprobe (too fresh) and to scrape (not yet live). that is why several
2026-07 connectors show only their hand-seeds live while thousands of
real, discoverable tenants wait in limbo.

stamp new skip-probe slugs with an epoch last_probed_at sentinel so
the next reprobe includes them immediately (first_seen_at stays the
real discovery day), matching the sitemap backend's liveness-truth
convention. also backfill the existing limbo: reset last_probed_at to
epoch on the unprobed transient_failure rows of the connectors
onboarded this month so the next reprobe promotes the live ones.
@datascry datascry merged commit a6a619e into main Jul 6, 2026
10 checks passed
@datascry datascry deleted the fix/reprobe-new-slug-epoch branch July 6, 2026 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant