Run the tool probe on a timer so the data actually fills in - #51
Merged
Merged
Conversation
The prober shipped inert: nothing ran it, so the catalogue still declared 134 tools. This puts it on a scheduler, the way the official-registry sync and the discovery batcher already work. - Ask 50 endpoints every five minutes at a concurrency of four, so a first pass over ~21,000 takes about a day and a half - Keep a probed endpoint out of the queue for two weeks, so the steady state is a trickle rather than a sweep - Identify ourselves in the user agent, with a link back - Schedule the next tick even when a batch raises, so one bad run cannot stop the loop - Off in dev and test; PROBE_ENABLED=false stops it in production Slow on purpose: these are other people's servers and there is no deadline. --- Pages affected: - [MCP Registry](https://ai.mcpharbor.dev/) — the Model Context Protocol server directory. - [Browse MCP servers](https://ai.mcpharbor.dev/servers) — catalogue whose tool counts this fills in. - [Context7 MCP server](https://ai.mcpharbor.dev/servers/io.github.upstash/context7) — a remote listing that answers a probe. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #50, which shipped the prober but left it inert — nothing ran it, so the catalogue still declares 134 tools.
This puts it on a scheduler, matching how
OfficialRegistry.SchedulerandDiscovery.Batcheralready work.Pace
Slow deliberately. These are other people's servers; a faster sweep looks like scanning and there is no deadline. Requests carry a user agent naming the registry and linking to it, and a probed endpoint is not asked again for two weeks, so the steady state is a trickle.
Off in dev and test so neither ever touches a third party.
PROBE_ENABLED=falsestops it in production;PROBE_BATCH_SIZEandPROBE_CONCURRENCYtune it.A batch that raises still schedules the next tick, so one bad run cannot silently stop the loop.
Expected effect
At ~27% yield, a full pass should take the catalogue from 134 declared tools to roughly 5,800 servers with verified tool lists — and those pages then say the list was read from the server itself rather than repeating a listing.
133 tests pass.
Pages affected:
🤖 Generated with Claude Code