chore: refresh CrawlKit and Chinese tokenizer dependencies

[steipete]

What Problem This Solves

Refresh the remaining outdated dependencies in Discrawl and its optional Chinese lexical helper.

User Impact

Updates CrawlKit from v0.16.1 to v0.16.2, gse from v1.0.2 to v1.1.0, cedar from v0.30.0 to v0.50.0, and the helper's transitive test dependency tt from v0.20.1 to v0.40.0. Existing Go minimums and CLI behavior are preserved.

Why This Change Was Made

The root module and all three helper modules were checked, along with toolchains, analysis tools, GitHub Actions, release tooling, and Docker inputs. The remaining pins are current. SQLite v1.58.0 explicitly requires libc v1.75.6, so libc v1.75.7 is held. Charm v2 requires a CrawlKit-owned migration and remains outside this dependency refresh. Modules used only by dependencies' own tests/tools are left to their owners rather than adding unused root requirements.

Evidence

Validated head: 6d268929fa0fb6e5cefeb6fe6db323667d73a582.

• All four modules: go mod download, go mod verify, and go mod tidy; SHA-256 hashes of all eight go.mod/go.sum files stayed unchanged. Builds use GOFLAGS=-mod=readonly.
• Full CI: full Go race suite with the 85% coverage floor, all language helper tests, native macOS/Windows build-and-run checks, lint, vulnerability checks, docs, secret scanning, and six-platform unpublished release snapshots passed.
• Built CLI live proof on macOS: wiretap imported a synthetic cached DM into a disposable SQLite archive; search --dm, status, SQL count, dms, and tui --dm --json returned the expected data. No real Discord archive or credentials were used.
• Built Chinese and Japanese helper processes returned ready responses, expected index tokens, and query groups for Chinese/Japanese sample text. Chinese and Korean helper tests also passed locally; Kiwi used a checksum-verified temporary library.
• Independent autoreview: scoped-clean, no accepted/actionable P0–P2 findings.

No application version bump, release, tag, or manual deployment.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed September 13, 2026, 11:52 AM ET / 15:52 UTC.

ClawSweeper review

What this changes

Updates CrawlKit and the optional Chinese tokenizer dependencies while preserving Discrawl’s Go requirements and SQLite dependency pairing.

Merge readiness

⛔ Blocked before merge - 2 items remain

Keep open: the refresh is not on main or in v0.15.0, and no concrete patch defect was found. Real behavior proof remains the merge blocker.

Priority: P3
Reviewed head: 6d268929fa0fb6e5cefeb6fe6db323667d73a582

Review scores

Measure	Result	What it means
Overall readiness	🦪 silver shellfish (2/6)	The patch is focused and appears sound, but completed runtime evidence is not yet available.
Proof confidence	🦪 silver shellfish (2/6)	Needs real behavior proof before merge: Needs real behavior proof before merge: module checks and helper tests do not show the rebuilt Discrawl CLI using CrawlKit and the gse/cedar helper to index and retrieve Chinese messages; the captured body explicitly leaves that runtime evidence in progress. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Needs proof	Needs real behavior proof before merge: Needs real behavior proof before merge: module checks and helper tests do not show the rebuilt Discrawl CLI using CrawlKit and the gse/cedar helper to index and retrieve Chinese messages; the captured body explicitly leaves that runtime evidence in progress. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed	9 items	Verified introduced scope: The pinned base-to-head diff changes only five files: dependency manifests, checksums, and one changelog entry. Go requirements and SQLite/libc pins are unchanged. Main and release still use older dependencies: The pinned main diff establishes CrawlKit v0.16.1 and gse v1.0.2 as the previous versions; v0.15.0 also contains those older pins. The canonical search found no separate replacement for this refresh. Dependency contracts actually exercised: The Chinese helper imports gse, loads its embedded dictionary, and uses CutSearch for indexing and Cut for queries. The root archive code imports CrawlKit/store. These direct consumers establish the relevant dependency boundaries.
Findings	None	None.
Security	None	None.

How this fits together

Discrawl archives Discord messages locally and exposes search through its CLI. CrawlKit supplies shared archive utilities; the optional Chinese helper converts messages and queries into terms for lexical search.

flowchart LR
  A[Discord messages] --> B[Discrawl archive]
  C[CrawlKit utilities] --> B
  B --> D[Chinese tokenizer helper]
  E[Search query] --> D
  D --> F[Lexical index and search]
  F --> G[CLI results]

Loading

Before merge

• Add real behavior proof - Needs real behavior proof before merge: Needs real behavior proof before merge: module checks and helper tests do not show the rebuilt Discrawl CLI using CrawlKit and the gse/cedar helper to index and retrieve Chinese messages; the captured body explicitly leaves that runtime evidence in progress. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
• Complete next step (P2) - Complete the runtime evidence section with observed CLI and Chinese-helper results. Terminal output or redacted logs are sufficient; screenshots or video are welcome when clearer. Remove private endpoints, IP addresses, credentials, and personal data. Updating the PR body should trigger another review; otherwise ask a maintainer to comment @clawsweeper re-review.

Agent review details

Security

None.

Review metrics

None.

Technical review

Best possible solution:

Retain the focused dependency refresh and existing helper-rebuild procedure, with observed archive and Chinese search results confirming compatibility.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR refreshes dependencies rather than reporting a reproducible bug; no runtime commands were executed during this read-only review.

Is this the best way to solve the issue?

Yes: updating existing pins is a narrow approach, and the consumed APIs, Go requirements, and documented helper upgrade path remain intact.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning medium; reviewed against 0701225990c4.

Labels

Label changes:

• add P3: This is a bounded dependency-maintenance update without a demonstrated urgent user-facing failure.
• add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
• add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: Needs real behavior proof before merge: module checks and helper tests do not show the rebuilt Discrawl CLI using CrawlKit and the gse/cedar helper to index and retrieve Chinese messages; the captured body explicitly leaves that runtime evidence in progress. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Label justifications:

• P3: This is a bounded dependency-maintenance update without a demonstrated urgent user-facing failure.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: Needs real behavior proof before merge: module checks and helper tests do not show the rebuilt Discrawl CLI using CrawlKit and the gse/cedar helper to index and retrieve Chinese messages; the captured body explicitly leaves that runtime evidence in progress. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence

What I checked:

• Verified introduced scope: The pinned base-to-head diff changes only five files: dependency manifests, checksums, and one changelog entry. Go requirements and SQLite/libc pins are unchanged. (go.mod:11, 6d268929fa0f)
• Main and release still use older dependencies: The pinned main diff establishes CrawlKit v0.16.1 and gse v1.0.2 as the previous versions; v0.15.0 also contains those older pins. The canonical search found no separate replacement for this refresh. (tools/discrawl-zh/go.mod:5, 2eb4aae89549)
• Dependency contracts actually exercised: The Chinese helper imports gse, loads its embedded dictionary, and uses CutSearch for indexing and Cut for queries. The root archive code imports CrawlKit/store. These direct consumers establish the relevant dependency boundaries. (tools/discrawl-zh/main.go:29, 6d268929fa0f)
• CrawlKit update preserves consumed source APIs: The v0.16.1-to-v0.16.2 comparison contains no Go implementation changes. Its runtime dependency updates match versions already selected by Discrawl, including the unchanged SQLite/libc pairing. (go.mod, 6b3fad4e16e2)
• Tokenizer implementation changes reviewed: The update optimizes dictionary allocation and parsing and adds an opt-in SkipSubSeg field, which Discrawl does not enable. Embedded dictionary data is unchanged, and the dependency’s Go 1.24 requirement remains below the helper’s Go 1.26 minimum. (dict_1.16.go, dab413be5c91)
• Trie compatibility boundary: Reviewed the narrower integer storage and lookup changes. gse uses dictionary token indexes as values and constructs the trie in memory; Discrawl does not invoke the newly added trie serialization or Aho-Corasick APIs. (cedar.go, 448a751aadc8)

Likely related people:

• Jeffrey (Dongkyu) Kim: Raw commit 6e7e023 adds tools/discrawl-zh/main.go:29 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 6e7e023e6c36; files: tools/discrawl-zh/main.go)
• steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Add a redacted transcript from the built CLI and Chinese helper showing archive access, indexing, and expected search results, including replacement of an existing helper through the documented rebuild procedure.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.