fix(import): keep Fern/Mintlify/Archbee nav when llms.txt is larger - #45
Draft
cursor[bot] wants to merge 3 commits into
Draft
fix(import): keep Fern/Mintlify/Archbee nav when llms.txt is larger#45cursor[bot] wants to merge 3 commits into
cursor[bot] wants to merge 3 commits into
Conversation
The 75% coverage gate was written for thin HTML scrapes (one tab of a multi-tab site). After #41 it also ran against canonical sidebars, so a curated Fern/Mintlify/Archbee tree that listed a subset of llms.txt was discarded and the importer fell back to invented llms.txt clusters. Skip that gate for those probes. Orphans still slot by path afterwards. Co-authored-by: Jon Ursenbach <erunion@users.noreply.github.com>
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
The 75% coverage exemption still handed Fern/Mintlify/Archbee trees to the existing 2x-orphan gate, which replaced the authored sidebar with URL-derived categories whenever llms.txt listed enough extra pages. Skip that recluster for canonical navs too. Leftover llms.txt rows still slot by path or bucket into extra categories; the authored spine stays. Co-authored-by: Jon Ursenbach <erunion@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The 75% coverage gate in
produceOrganizedForSourcewas written for thin HTML scrapes (one tab of a multi-tab site). After #41 it also ran against canonical sidebars, so a curated Fern, Mintlify, or Archbee tree that listed a subset ofllms.txtwas discarded and the importer fell back to invented llms.txt clusters — the failure mode those probes exist to prevent.Concrete trigger: import a Fern docs site that publishes
llms.txtwith hidden, legacy, or extra pages beyond the authored sidebar.tryFernNavrecovers the real tree, coverage is ~3/8 (<75%), and that tree is thrown away.Fix
Skip the thin-scrape coverage discard when the nav came from Mintlify, Fern, or Archbee. HTML scrapes still use the 75% gate. Orphans continue to slot by path afterwards, so llms.txt pages that are not in the authored sidebar are still imported.
Verification