Pack capability sitemaps to a URL budget, not a listing count - #68
Merged
Merged
Conversation
resources-4.xml was serving 56,798 URLs against a 50,000 limit, so a seventh of the resources silo was being rejected without a message. resources-2.xml sat at 49,674, a whisker under. Chunking by a fixed number of listings cannot work when the counts are not uniform. One listing has 960 resources and most have a handful, so 300 listings a file came out at 30,187 URLs in one place and 56,798 in another depending on who landed in it. - Listings are packed to a 40,000-URL budget, leaving 10,000 of headroom - 13 URLs per item is an upper bound, not an estimate, so a file can come in under budget but never over - Chunk sizes are cached with the other catalogue figures The same failure was fixed once for servers-*.xml. A count that works today breaks the moment the distribution changes; a budget does not. Closes #67 --- Pages affected: - [MCP Harbor](https://ai.mcpharbor.dev/) -- registry home, search and recently added servers. - [MCP server directory](https://ai.mcpharbor.dev/servers) -- the listings these files index. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #67.
/sitemaps/resources-4.xmlwas serving 56,798 URLs against a hard limit of 50,000. A file over the limit is rejected without a message, so a seventh of the resources silo was invisible to search from the moment it shipped.resources-2at 49,674 shows how little the old scheme had left.Why a count could never work
One listing has 960 resources; most have a handful. With 300 listings per file, the URL count depends entirely on who lands in that file — 30,187 in one, 56,798 in the next. No fixed number is safe against that distribution, and picking a smaller one only moves the cliff.
Listings are now packed to a 40,000-URL budget, with 10,000 of headroom. The per-listing estimate uses 13 URLs per item, which is an upper bound (the widest client list), not an average — so a file can come in under budget but never over it.
This is the same failure that was fixed once for
servers-*.xml, in a new place. A count that works today breaks the moment the data shifts; a budget does not.Tests: 168 passing, including one that builds ten listings of 400 resources — past both the budget and the limit — and asserts every generated file stays under 50,000.
Pages affected:
🤖 Generated with Claude Code