feat: add crates.io search engine adapter (#145)#191
Conversation
|
Warning Review limit reached
Next review available in: 5 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughAdds a crates.io API-backed search engine, registers it as a weighted secondary engine for the code vertical, and adds unit tests for request construction, response parsing, error handling, and edge cases. ChangesCrates.io code search
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant CodeSearch
participant CratesIoEngine
participant CratesIoAPI
CodeSearch->>CratesIoEngine: search query with timeout and maxResults
CratesIoEngine->>CratesIoAPI: fetch crates.io API results
CratesIoAPI-->>CratesIoEngine: return crate JSON
CratesIoEngine-->>CodeSearch: return weighted raw search results
Possibly related issues
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
src/search/engines/crates-io.ts (2)
37-39: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueCap
maxResultsto the API limit.The crates.io API limits
per_pageto a maximum of 100. Consider cappingmaxResultsdefensively so that unexpectedly large requested limits do not result in a 400 Bad Request error from the upstream API.♻️ Proposed refactor
async search(query: string, options: SearchEngineOptions = {}): Promise<RawSearchResult[]> { const timeoutMs = options.timeoutMs ?? 10000; - const maxResults = options.maxResults ?? 10; + const maxResults = Math.min(options.maxResults ?? 10, 100); const params = new URLSearchParams({🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/search/engines/crates-io.ts` around lines 37 - 39, Cap the maxResults value initialized in the crates.io search options to the API-supported maximum of 100, while preserving the existing default of 10 and allowing smaller requested values unchanged.
71-75: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valuePrevent leading space in snippets with missing descriptions.
When a crate lacks a description but has a
max_version, the current logic produces a snippet with an awkward leading space (e.g.," (v0.1.0, 10 downloads)"). Filtering out empty strings prevents this formatting quirk and creates a cleaner search snippet.
src/search/engines/crates-io.ts#L71-L75: Use an array and.filter(Boolean)to construct the snippet without a leading space.tests/unit/search/engines/crates-io.test.ts#L66-L68: Remove the leading space from the test's expected snippet output to match the updated formatting.♻️ Proposed refactors
src/search/engines/crates-io.ts
const description = asString(crate.description) ?? ''; const downloads = asNumber(crate.downloads) ?? 0; const maxVersion = asString(crate.max_version); - const snippet = maxVersion ? `${description} (v${maxVersion}, ${downloads} downloads)` : description; + const suffix = maxVersion ? `(v${maxVersion}, ${downloads} downloads)` : ''; + const snippet = [description, suffix].filter(Boolean).join(' '); results.push({tests/unit/search/engines/crates-io.test.ts
const results = await new CratesIoEngine().search('foo'); - expect(results[0].snippet).toBe(' (v0.1.0, 10 downloads)'); + expect(results[0].snippet).toBe('(v0.1.0, 10 downloads)'); });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/search/engines/crates-io.ts` around lines 71 - 75, Update the snippet construction in crates-io.ts around description, downloads, and maxVersion to build the components with an array and filter out empty values before joining, preventing a leading space when the description is missing. Update the expected snippet in tests/unit/search/engines/crates-io.test.ts at lines 66-68 to remove the leading space; both sites require these changes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/search/engines/crates-io.ts`:
- Around line 37-39: Cap the maxResults value initialized in the crates.io
search options to the API-supported maximum of 100, while preserving the
existing default of 10 and allowing smaller requested values unchanged.
- Around line 71-75: Update the snippet construction in crates-io.ts around
description, downloads, and maxVersion to build the components with an array and
filter out empty values before joining, preventing a leading space when the
description is missing. Update the expected snippet in
tests/unit/search/engines/crates-io.test.ts at lines 66-68 to remove the leading
space; both sites require these changes.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: c6639398-d5d3-4f41-8e71-d746d41822b0
📒 Files selected for processing (3)
src/search/core/verticals/code.tssrc/search/engines/crates-io.tstests/unit/search/engines/crates-io.test.ts
…cal tests for the new secondary engine
a273e1e to
a3620e4
Compare
1 participant
Closes #145
Adds
CratesIoEngineusing the crates.io public JSON API (crates.io/api/v1/crates) — scrape-free adapter for thecodevertical.User-Agentper crates.io crawler policy (matches wikipedia/marginalia/lobsters convention)crates.io/crates/<name>→ url, description + version/downloads → snippetMdnEnginehn-algolia.test.tsmocked-fetch pattern (UA header asserted, no live network)Test plan
npx vitest run tests/unit/search/engines/crates-io.test.ts— 10/10npx tsc --noEmitcleanNote: will conflict trivially with #144's PR in
code.tsregistration — whichever merges second rebases in minutes.Summary by CodeRabbit
New Features
Bug Fixes