feat(cli): add verify subcommand for library-first adoption (#657)#704
Merged
Conversation
Adds contextweaver verify — a deterministic, network-free confidence command for first-run library adopters. Checks five areas: 1. import path + version readable 2. ContextManager instantiates cleanly 3. minimal context build produces a non-empty pack 4. token counter returns positive counts 5. routing graph builds + routes a query Non-Rich --json output is stable for CI/automation. Each failure maps to a clear fix-hint string. Files added: - src/contextweaver/_verify.py — pure verification helpers Files changed: - src/contextweaver/__main__.py — �erify Typer command + imports - ests/test_cli.py — pass/fail + JSON-mode tests - docs/quickstart.md — verify usage section - CHANGELOG.md — entry under Unreleased
There was a problem hiding this comment.
Pull request overview
Adds a new contextweaver verify CLI mode intended to give library-first adopters a fast, deterministic smoke test of core functionality (import, manager, minimal build, token counting, routing), with both human-readable Rich output and a --json payload for automation.
Changes:
- Introduces
src/contextweaver/_verify.pywith five verification helpers returning a_VerifyCheckresult shape. - Adds a
verifyTyper subcommand tosrc/contextweaver/__main__.pywith table output, JSON output, and pass/fail exit codes. - Extends CLI subprocess tests and updates quickstart docs to document
contextweaver verify.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
src/contextweaver/_verify.py |
New verification helper module implementing the five checks. |
src/contextweaver/__main__.py |
Adds the verify subcommand, JSON/table rendering, and next-step messaging. |
tests/test_cli.py |
Adds subprocess tests for verify (happy path + --json). |
docs/quickstart.md |
Documents the new contextweaver verify usage in the install section. |
Benchmark delta (vs
|
| size | recall@k (head Δ vs base) | MRR (head Δ vs base) | p99 (ms) |
|---|---|---|---|
| 50 | ✅ 0.5649 (+0.0000) | ✅ 0.4978 (+0.0000) | ✅ 0.426 (base 0.759) |
| 83 | ✅ 0.3825 (+0.0000) | ✅ 0.3242 (+0.0000) | ✅ 0.640 (base 1.134) |
| 1000 | ✅ 0.1475 (+0.0000) | ✅ 0.1456 (+0.0000) | ✅ 32.023 (base 41.711) |
Per-backend × per-size matrix
| backend | size | recall@k (Δ) | MRR (Δ) | p99 (ms) |
|---|---|---|---|---|
| bm25 | 100 | ✅ 0.3825 (+0.0000) | ✅ 0.3399 (+0.0000) | ✅ 5.759 (base 8.140) |
| bm25 | 500 | ✅ 0.2250 (+0.0000) | ✅ 0.2165 (+0.0000) | ✅ 26.323 (base 38.989) |
| bm25 | 1000 | ✅ 0.1575 (+0.0000) | ✅ 0.1525 (+0.0000) | ✅ 77.308 (base 111.716) |
| embedding_hashing | 100 | ✅ 0.5175 (+0.0000) | ✅ 0.4360 (+0.0000) | ✅ 7.146 (base 7.225) |
| embedding_hashing | 500 | ✅ 0.2700 (+0.0000) | ✅ 0.2674 (+0.0000) | ✅ 34.745 (base 44.182) |
| embedding_hashing | 1000 | ✅ 0.2000 (+0.0000) | ✅ 0.1931 (+0.0000) | ✅ 81.761 (base 98.277) |
| embedding_st | 100 | skipped (skipped: missing sentence-transformers) | — | — |
| embedding_st | 500 | skipped (skipped: missing sentence-transformers) | — | — |
| embedding_st | 1000 | skipped (skipped: missing sentence-transformers) | — | — |
| fuzzy | 100 | skipped (skipped: missing rapidfuzz) | — | — |
| fuzzy | 500 | skipped (skipped: missing rapidfuzz) | — | — |
| fuzzy | 1000 | skipped (skipped: missing rapidfuzz) | — | — |
| tfidf | 100 | ✅ 0.3825 (+0.0000) | ✅ 0.3220 (+0.0000) | ✅ 0.922 (base 1.102) |
| tfidf | 500 | ✅ 0.2325 (+0.0000) | ✅ 0.2314 (+0.0000) | ✅ 8.344 (base 11.492) |
| tfidf | 1000 | ✅ 0.1475 (+0.0000) | ✅ 0.1456 (+0.0000) | ✅ 30.308 (base 50.755) |
Context pipeline (per scenario)
| scenario | tokens | dropped | dedup |
|---|---|---|---|
| large_catalog | 1480 (base 1514, Δ-34) | 0 (base 0, Δ+0) | 0 (base 0, Δ+0) |
| long_conversation | 2500 (base 2548, Δ-48) | 0 (base 0, Δ+0) | 0 (base 0, Δ+0) |
| mixed_payload | 488 (base 497, Δ-9) | 0 (base 0, Δ+0) | 0 (base 0, Δ+0) |
| short_conversation | 487 (base 496, Δ-9) | 0 (base 0, Δ+0) | 0 (base 0, Δ+0) |
| stress_conversation | 6590 (base 6651, Δ-61) | 11 (base 7, Δ+4) | 4 (base 4, Δ+0) |
| tiny_payload | 256 (base 267, Δ-11) | 0 (base 0, Δ+0) | 0 (base 0, Δ+0) |
Numbers come from make benchmark / make benchmark-matrix.
Latency is hardware-dependent — treat the markers as a rough guide.
See benchmarks/scorecard.md for the full picture.
- Switch _check_tokens() to heuristic_counter() for strictly network-free operation. - Remove hard-coded sub-command count from module docstring. - Update next_step to published docs URL. - Add CHANGELOG entry under [Unreleased]. - Add tests/test__verify.py with 5 happy-path + 5 failure-mode unit tests (monkeypatch/unittest.mock.patch).
This was referenced Jun 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #657 — adds a non-gateway
verifymode so library-first adopters get fast confidence the installation works.Changes
src/contextweaver/_verify.py_check_import,_check_manager,_check_build,_check_tokens,_check_routing)src/contextweaver/__main__.pyverifyTyper subcommand with Rich table output +--jsonfor CI/automation; exit 0 on pass, 1 on failuretests/test_cli.py--jsonmode (validates schema +ok=True)docs/quickstart.mdcontextweaver verifyusage block in the install sectionCHANGELOG.md[Unreleased]Approach
_VerifyCheckdataclass withname,ok,detail,fix_hint--jsonfor automation (stable schema)generate_sample_catalog(n=10)+TreeBuilder+Router.route()— same primitives as the quickstartHow verified
ruff check src/ tests/— all passruff format src/ tests/— reformatted_verify.pymypy src/contextweaver/_verify.py src/contextweaver/__main__.py— cleanpytest tests/test_cli.py -k verify -v— 2 passedpytest tests/test_cli.py -v— 42 passed, 0 failed (~110s)python -m contextweaver verify --json— 5/5 checks pass, valid JSON outputTradeoffs / risks
_verify.pyis 161 lines, well under the 300-line ceiling__main__.pygrows by ~83 lines (exempt module)Scope notes
Strictly limited to issue #657 scope. Gateway health checks (ping MCP server, validate upstream catalog) noted as a follow-up candidate.