Repository navigation
fix(repository): /repos 按回源目录布局有界扫描，不再遍历旧缓存元数据树 - #136
Merged
mangoknight merged 1 commit intoOct 21, 2026
Merged
Conversation
List 原先对 api/ 整树 WalkDir,只在遇到 repository.json 时停止下钻。 标记文件引入之前写入的回源缓存没有标记,扫描会钻进每个仓库的 paths-info/revision 树;大镜像节点上这类目录可达数十万,/repos 与 /api/cache/repos 长时间无响应,且结果只包含已注册的少数仓库。 - 回源仓库 ID 最多两段(与 Discover 的约定一致),Hugging Face 与 ModelScope 目录只读两层;托管仓库名可任意深度,dingo-local 仍按标记遍历。 - List 语义不变(仅返回已注册仓库),磁盘清理、缓存管理等调用方不受影响。 - 新增 ListCatalog,额外报告带缓存元数据但无标记的回源目录(只展示, 不注册、不参与访问判断);/repos 改用它,列表恢复完整并排序。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dongzhenyangofficial-ctrl
approved these changes
Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
GET /repos在缓存量大的镜像节点上长时间无响应（实测超过 120s，连响应头都没有），即使最终返回，列表也只包含少数仓库。原因：自 5bcef49 起，
ReposGenerator调用repository.List，它对repos/api/整树WalkDir，只在遇到repository.json时SkipDir；paths-info/revision等数据子树也只有在祖先带标记时才会被剪掉。引入标记之前写入的回源缓存没有repository.json，扫描会钻进每个仓库的逐文件元数据树。某生产节点上约 1000 个回源仓库里只有 16 个带标记，api/下共约 79 万个目录，单遍find就要 214s。/api/cache/repos同样受影响。（5bcef49 之前
/repos用的是filepath.Glob("api/<type>/*/*")，快且完整。）修改
List改为按目录布局有界扫描：回源仓库 ID 最多两段（与Discover的约定一致），Hugging Face 与 ModelScope 目录只读两层（<owner>及<owner>/<name>，兼容无组织的单段 HF ID）；托管仓库名可任意深度，dingo-local仍用原listIn遍历。List语义不变：仍只返回已注册（带标记）的仓库，磁盘清理、缓存管理等调用方行为不变。标记读取逻辑抽为readMarker，与原实现等价（格式错误、路径不符仍报错）。ListCatalog：在已注册仓库之外，报告带缓存元数据（revision/paths-info）但无标记的回源目录。只用于展示，不注册、不参与任何访问判断。/repos改用ListCatalog，列表恢复完整并按名称排序。测试
pkg/repository/catalog_test.go：TestListDoesNotWalkUnmarkedUpstreamCaches：无标记缓存的元数据树底部放一个格式错误的标记；旧实现会读到并报错（已在旧代码上确认失败），新实现不打开该树。TestListCatalogReportsUnregisteredUpstreamCaches：覆盖 HF 两段 / 单段、ModelScope、datasets 的无标记缓存，空组织目录不上报。TestListMatchesFullWalkOnRegisteredNode：全部已注册的节点上，有界扫描与整树遍历结果一致。go test ./...全部通过。List3.7ms，ListCatalog4.7ms（16 + 1000 个仓库全部列出）。开销只与仓库数相关，不随元数据树大小增长。注：Windows 下
internal/dao的TestReclaimDoesNotRaceWithPublish偶发失败，基线代码同样存在（Windows 文件占用语义），与本改动无关；Linux 下新旧代码各跑 20 次均 0 失败。🤖 Generated with Claude Code