chore: update Crawlkit to v0.16.3

[hannesrudolph]

What Problem This Solves

Discrawl is one maintenance release behind its shared Crawlkit dependency.

User Impact

Builds use Crawlkit v0.16.3 while retaining the existing Go requirement, SQLite/libc versions, and configuration.

Why This Change Was Made

Adopt the v0.16.3 maintenance release through the module pin and refreshed checksums. No application code changes are needed. The custom-client embedding timeout fix is not presented as a reproduced Discrawl bug: its provider construction uses the default client.

Evidence

Module verification/tidiness, formatting, lint, vet, staticcheck, gosec, and govulncheck passed (no reachable vulnerabilities). Snapshot builds passed for macOS, Linux, and Windows on amd64 and arm64. Structured autoreview returned no findings. A signed native build reads the existing archive status successfully.

Full race/coverage tests and CLI smoke passed, with 85.9% coverage. The local race run used GIT_CONFIG_NOSYSTEM=1 GIT_CONFIG_GLOBAL=/dev/null: an existing publication fixture fails with macOS system init.defaultBranch=main on both v0.16.2 and v0.16.3; isolating Git settings makes it pass. No fixture or application change was added for that pre-existing condition.

The signed macOS build was deployed locally with the same designated signing identity and executable path. Native capture and live embeddings resumed, startup repair completed, and no failed or pending embedding jobs remained. Configuration and archive schema were unchanged.

Dependency reorganization verification

Read both complete module sources locally to cover the dependency-source access gap in the initial ClawSweeper review. A Go parser/token comparison found all 187 remote declarations and all 91 snapshot declarations unchanged, apart from file placement/comments. TUI retained 355 declarations unchanged; the remaining 47 changes preserve function signatures and reduce to integer min/max builtins plus inlining an always-newest sort helper. Thirteen obsolete unexported helpers were removed.

Additionally, GIT_CONFIG_NOSYSTEM=1 GIT_CONFIG_GLOBAL=/dev/null GOWORK=off GOFLAGS=-mod=readonly go test -count=1 -race ./remote ./snapshot ./tui ./embed ./config passed from the published Crawlkit v0.16.3 module. The source check and package tests cover the reorganized code; this does not claim live cloud integration or interactive TUI acceptance.

The earlier draft/test reminder is also resolved: this PR is ready, the full Discrawl race/coverage suite passed locally, and the GitHub test job passed on this head.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed September 14, 2026, 3:12 PM ET / 19:12 UTC (Revision 2).

ClawSweeper review

What this changes

Updates Discrawl’s shared Crawlkit library to v0.16.3, refreshes dependency checksums, and records the upgrade in the changelog.

Merge readiness

✅ Ready for maintainer review

The upgrade remains useful and has no identified blocking defect. The added verification addresses the earlier review concern; current main still uses v0.16.2.

Priority: P3
Reviewed head: c5b1f78e6275ca2a739ad5301be7cca9a111164c

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A focused maintenance patch with relevant reported runtime validation, passing checks, and no actionable findings.
Proof confidence	🐚 platinum hermit (4/6)	Not applicable: The ordinary external-contributor proof gate does not apply to this MEMBER-authored PR. The supplied body additionally reports successful native capture and live embedding recovery on an existing archive; declaration comparisons and race tests supplement that report without claiming live cloud or interactive TUI coverage.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: The ordinary external-contributor proof gate does not apply to this MEMBER-authored PR. The supplied body additionally reports successful native capture and live embedding recovery on an existing archive; declaration comparisons and race tests supplement that report without claiming live cloud or interactive TUI coverage.
Evidence reviewed	7 items	Verified introduced change: The pinned base-to-head delta changes only the Crawlkit requirement, matching checksum entries, and two changelog lines; Go, SQLite, and libc requirements remain unchanged. Main and release still need the upgrade: The fetched main source, a live GitHub main read, and the v0.15.1 go.mod all retain Crawlkit v0.16.2. The merged #234 delivered that earlier version, not this upgrade. Dependency boundary and default client: Discrawl directly imports Crawlkit embedding, remote, snapshot, configuration, and terminal-browser packages, establishing the dependency review boundary. Its embedding factory supplies either configuration alone or an API-key option, never a custom HTTP client.
Findings	None	None.
Security	None	None.

How this fits together

Discrawl archives Discord messages for local search and terminal browsing. Crawlkit supplies shared archive, configuration, embedding, and remote-access components used by those workflows.

flowchart LR
  A[Discord messages] --> B[Discrawl capture]
  B --> C[Local archive]
  D[Crawlkit library] --> C
  D --> E[Embedding providers]
  C --> E
  C --> F[Search and terminal browsing]
  E --> F

Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

None.

Technical review

Best possible solution:

Consume the published maintenance release while preserving Discrawl’s existing configuration, archive format, and runtime dependency pairing.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is dependency maintenance, and the PR explicitly avoids claiming that Crawlkit’s custom-client timeout defect reproduces in Discrawl.

Is this the best way to solve the issue?

Yes: updating the published module pin and corresponding checksums is the narrowest integration path; application changes are unnecessary.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning medium; reviewed against ae7fa0ff9b50.

Labels

Label justifications:

• P3: This is a bounded dependency maintenance update with no demonstrated urgent Discrawl regression.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The ordinary external-contributor proof gate does not apply to this MEMBER-authored PR. The supplied body additionally reports successful native capture and live embedding recovery on an existing archive; declaration comparisons and race tests supplement that report without claiming live cloud or interactive TUI coverage.

Evidence

What I checked:

• Verified introduced change: The pinned base-to-head delta changes only the Crawlkit requirement, matching checksum entries, and two changelog lines; Go, SQLite, and libc requirements remain unchanged. (go.mod:11, c5b1f78e6275)
• Main and release still need the upgrade: The fetched main source, a live GitHub main read, and the v0.15.1 go.mod all retain Crawlkit v0.16.2. The merged chore: refresh CrawlKit and Chinese tokenizer dependencies #234 delivered that earlier version, not this upgrade. (go.mod:11, ae7fa0ff9b50)
• Dependency boundary and default client: Discrawl directly imports Crawlkit embedding, remote, snapshot, configuration, and terminal-browser packages, establishing the dependency review boundary. Its embedding factory supplies either configuration alone or an API-key option, never a custom HTTP client. (internal/cli/embedding_provider.go:18, c5b1f78e6275)
• Published dependency changes: The upstream comparison identifies v0.16.3 as b33f7c3a313d8a47dabcb3a5b5f057d2af529794. Reviewed embedding changes preserve default-client behavior, copy caller-owned clients before applying timeouts, and consolidate equivalent helpers; configuration changes consolidate path construction. Upstream go.mod additions explain the four refreshed transitive checksum pairs. (embed/provider.go, b33f7c3a313d)
• Dependency compatibility policy: Read the full dependency AGENTS.md. Its relevant guidance preserves downstream APIs and uses the published module with GOWORK=off for compatibility verification. (AGENTS.md, b33f7c3a313d)
• Updated verification and review continuity: The complete supplied PR body reports comparison of both published module sources, unchanged remote/snapshot declarations, bounded TUI helper changes, and passing dependency package race tests. It also reports 85.9% Discrawl coverage and successful signed macOS deployment with capture, embeddings, and startup repair restored. These are contributor-reported results, not reviewer executions. The previous completed review had no findings; its dependency-access concern is addressed by the added evidence. Captured context sourceRevision: 5a714244fa75f722262f24e610ff8416f5a8f372e9473019eb82a46f2c16803d. (c5b1f78e6275)

Likely related people:

• Hannes Rudolph: Raw commit 0f35f71 adds internal/cli/embedding_provider.go:18 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 0f35f710441b; files: internal/cli/embedding_provider.go)
• steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (1 earlier review cycle)

• reviewed 2026-09-14T19:04:53.858Z sha c5b1f78 :: blocked before merge. :: none