diff --git a/README.md b/README.md
index c28cc49..85708a4 100644
--- a/README.md
+++ b/README.md
@@ -70,8 +70,8 @@ lockdocs takes the version question off the table:
- **Exact version, zero config.** It reads `package-lock.json`, `pnpm-lock.yaml`, `yarn.lock`, `bun.lock`, `Cargo.lock`, `uv.lock`, `poetry.lock`, `Pipfile.lock`, `requirements*.txt` and `go.mod`. No library IDs, no "use v14" in the prompt.
- **Docs that ship with the code.** READMEs, changelogs and `docs/` folders, plus the API reference in the package itself: `.d.ts` declarations with JSDoc, Python docstrings and stubs, rustdoc comments, Go doc comments. If it is installed, it is documented, including your private and internal packages.
-- **Offline and unlimited.** Everything is read from `node_modules`, your virtualenv, `~/.cargo/registry` and the Go module cache. Offline after a one-time model download (129 MB, kept as 32 MB); keyword-only mode (`LOCKDOCS_EMBED=0`) needs no network at all. No account, no rate limit, and nothing about your dependencies leaves your machine.
-- **Upstream docs at the exact tag, when you want them.** Packages like Next.js, Django and FastAPI ship no docs. `lockdocs fetch` pulls their docs folders from GitHub at the git tag of your pinned version, once, then stays offline.
+- **Offline and unlimited after caching.** Package files are read from `node_modules`, your virtualenv, `~/.cargo/registry` and the Go module cache. The optional model downloads once (129 MB, kept as 32 MB). Use `--offline` (or `LOCKDOCS_OFFLINE=1`) to prohibit all downloads; `LOCKDOCS_EMBED=0` disables only the model. No account or hosted query quota. First-use upstream requests reveal the public repository and version being fetched, not your question or project files.
+- **Upstream docs on first use.** Packages like Next.js, Django and FastAPI ship no docs. Queries automatically add public GitHub docs at the immutable commit resolved from your pinned release tag, anonymously and with bounded downloads. `--no-fetch` or `LOCKDOCS_FETCH=0` opts out; cached docs still work offline. Failures are reported alongside local answers, never replaced with latest-version docs.
- **Meaning, not just words.** Hybrid retrieval: BM25 fused with a small local embedding model (downloaded once, 32 MB on disk), plus API redirects from deprecation notes ("use `model_validate` instead").
- **Small, cited answers.** Packed into a token budget (1,200 by default), every section cited as `package@version path:line`.
@@ -111,13 +111,13 @@ The same call in a pydantic 1 project answers that pydantic 1.10.18 has no `mode
Legacy copies bundled inside a package (`zod/v3` inside zod 4, `pydantic/v1` inside pydantic 2) rank below the current API.
-### Upstream docs and missing packages (opt-in)
+### Upstream docs on first use
-`lockdocs fetch` adds, once, each direct dependency's upstream docs: it finds the GitHub repository in the package's own metadata and the git tag of your pinned version, and downloads only the docs folders at that tag (Markdown, MDX, reStructuredText, docs examples). Answers then cite `next@15.1.0 upstream:docs/01-app/.../cookies.mdx:12`. See [Upstream docs and fetching](https://sylphxai.github.io/lockdocs/guide/fetch).
+The first `docs` or `api` query adds the selected dependencies' public release-tag docs once: it finds the GitHub repository in the package's own metadata and the git tag of your pinned version, resolves that tag to an immutable commit, and downloads only its docs folders (Markdown, MDX, reStructuredText, docs examples). It does not use ambient GitHub credentials or download major-version website docs by default. Previously opted-in docs-site caches are preserved and their provenance stays visible. `lockdocs fetch` remains available to prewarm docs and explicitly add major-version docs sites. Answers then cite `next@15.1.0 upstream:docs/01-app/.../cookies.mdx:12`. See [Upstream docs and fetching](https://sylphxai.github.io/lockdocs/guide/fetch).
### Not installed? Fetch the exact version (opt-in)
-Out of the box lockdocs reads only your disk (plus the one-time embedding model download). If a pinned package is not installed (a fresh clone, CI, a lockfile you are reviewing), it says so and tells you how to install it. Pass `--fetch` (or set `LOCKDOCS_FETCH=1`, or `lockdocs setup --fetch`) to let it download exactly that version from the registry (npm tarball, PyPI wheel or sdist, crates.io `.crate`, Go module proxy zip) into its cache. Fetched answers say `fetched from registry.npmjs.org`. You can also ask for a version you do not use: `lockdocs npm:zod@4.1.5 "strict object" --fetch`.
+Registry package downloads remain opt-in; default upstream enrichment uses installed package metadata (plus the one-time embedding model download). If a pinned package is not installed (a fresh clone, CI, a lockfile you are reviewing), it says so and tells you how to install it. Pass `--fetch` (or set `LOCKDOCS_FETCH=1`, or `lockdocs setup --fetch`) to let it download exactly that version from the registry (npm tarball, PyPI wheel or sdist, crates.io `.crate`, Go module proxy zip) into its cache. Fetched answers say `fetched from registry.npmjs.org`. You can also ask for a version you do not use: `lockdocs npm:zod@4.1.5 "strict object" --fetch`.
## Benchmarks
@@ -176,14 +176,14 @@ lockdocs cache [clean] Show or delete the cache
lockdocs setup Configure MCP clients (--client a,b --dry-run --remove --fetch)
lockdocs mcp MCP server on stdio
-Options: -C/--root
, --pkg , --tokens , --fetch, --offline, --json
+Options: -C/--root , --pkg , --tokens , --fetch, --no-fetch, --offline, --json
```
Prebuilt binaries for macOS (arm64, x64), Linux glibc (x64, arm64) and Windows x64 ship through npm; each [GitHub release](https://github.com/SylphxAI/lockdocs/releases) has them too. From source: `cargo install --git https://github.com/SylphxAI/lockdocs lockdocs`.
## Privacy
-lockdocs reads files on your machine and answers over stdio. Network use: the embedding model once from huggingface.co (pinned revision, SHA-256 checked; `LOCKDOCS_EMBED=0` or `--offline` skips it), and, only when you run `lockdocs fetch` or enable fetching, public registries and GitHub for the exact package versions requested. Nothing about your project is sent. The cache lives in your OS cache directory (`LOCKDOCS_CACHE` overrides it).
+lockdocs reads files on your machine and answers over stdio. Network use: the embedding model once from huggingface.co (pinned revision, SHA-256 checked; `LOCKDOCS_EMBED=0` or `--offline` skips it), and anonymous GitHub requests for public docs at the resolved release commit on first query. `--fetch` / `lockdocs fetch` also access public registries and major-version docs-site repositories; only these explicit fetches may use `GITHUB_TOKEN` / `GH_TOKEN`. Upstream requests disclose the repository, version and file paths, not your question, lockfile or project files. `--no-fetch` / `LOCKDOCS_FETCH=0` disables query package/docs downloads; `--offline` / `LOCKDOCS_OFFLINE=1` prohibits all downloads. The cache lives in your OS cache directory (`LOCKDOCS_CACHE` overrides it).
## Also from Sylphx
diff --git a/bench/run.py b/bench/run.py
index 1a9cfad..22bc82a 100755
--- a/bench/run.py
+++ b/bench/run.py
@@ -19,7 +19,7 @@
context for the question) on the anonymous tier, trying the next search result
when a library answers HTTP 404; 429s are recorded, not retried.
"""
-import json, os, subprocess, sys, time, urllib.parse, urllib.request
+import json, os, subprocess, sys, tempfile, time, urllib.parse, urllib.request
HERE = os.path.dirname(os.path.abspath(__file__))
@@ -135,15 +135,21 @@ def run_context7(q, version):
VARIANTS = [
- ("keyword", "lockdocs, keyword only (BM25), package files", {"LOCKDOCS_EMBED": "0", "LOCKDOCS_NO_UPSTREAM": "1"}),
- ("hybrid", "lockdocs, hybrid (BM25 + embeddings), package files", {"LOCKDOCS_NO_UPSTREAM": "1"}),
- ("fetched", "lockdocs, hybrid + upstream docs (after `lockdocs fetch`)", {}),
+ ("keyword", "lockdocs, keyword only (BM25), package files", {"LOCKDOCS_EMBED": "0", "LOCKDOCS_NO_UPSTREAM": "1", "LOCKDOCS_FETCH": "0"}),
+ ("hybrid", "lockdocs, hybrid (BM25 + embeddings), package files", {"LOCKDOCS_NO_UPSTREAM": "1", "LOCKDOCS_FETCH": "0"}),
+ ("first-use-default", "lockdocs, real first-use defaults (empty isolated cache, anonymous release-tag fetch)",
+ {"LOCKDOCS_FETCH": None, "LOCKDOCS_NO_UPSTREAM": None, "LOCKDOCS_EMBED": None, "LOCKDOCS_OFFLINE": None, "GITHUB_TOKEN": None, "GH_TOKEN": None}),
+ ("fetched", "lockdocs, hybrid + upstream docs (after `lockdocs fetch`)", {"LOCKDOCS_FETCH": "1"}),
]
def run_lockdocs_env(binary, proj, q, env):
old = {k: os.environ.get(k) for k in env}
- os.environ.update(env)
+ for k, v in env.items():
+ if v is None:
+ os.environ.pop(k, None)
+ else:
+ os.environ[k] = v
try:
return run_lockdocs(binary, proj, q)
finally:
@@ -191,12 +197,17 @@ def main():
for p in projs:
proj = os.path.join(projects, p)
t = time.perf_counter()
- r = subprocess.run([binary, "index", "-C", proj, "--json"], capture_output=True, text=True, env={**os.environ, "LOCKDOCS_NO_UPSTREAM": "1"})
+ r = subprocess.run([binary, "index", "-C", proj, "--json"], capture_output=True, text=True, env={**os.environ, "LOCKDOCS_NO_UPSTREAM": "1", "LOCKDOCS_FETCH": "0"})
index[p] = {"ms": round((time.perf_counter() - t) * 1000), "report": json.loads(r.stdout) if r.returncode == 0 else r.stderr}
- variants = [v for v in VARIANTS if do_fetch or v[0] in ("keyword", "hybrid")]
+ variants = [v for v in VARIANTS if do_fetch or v[0] in ("keyword", "hybrid", "first-use-default")]
results = {}
fetch = {}
+ # A real query run before prefetch, with its own initially empty supported cache.
+ # It never borrows upstream files from the explicit-fetch score.
+ first_use_cache = tempfile.TemporaryDirectory(prefix="lockdocs-first-use-")
for name, _, env in variants:
+ if name == "first-use-default":
+ env = {**env, "LOCKDOCS_CACHE": first_use_cache.name}
if name == "fetched" and not fetch:
for p in projs:
t = time.perf_counter()
@@ -209,6 +220,7 @@ def main():
first = text.splitlines()[0] if text else ""
version = first.split(" · ")[0].rsplit("@", 1)[-1] if "@" in first else ""
results[(name, q["id"])] = ({"pass": ok and code == 0, "missing": missing, "rejected": rej, "tokens": count(text), "ms": round(ms, 1)}, version)
+ first_use_cache.cleanup()
main_variant = "fetched" if do_fetch else variants[-1][0]
rows = []
for q in qs:
@@ -232,6 +244,15 @@ def main():
"context7_calls": dict(c7_state, reused=reused) if with_c7 else None, "runner": {"os": os.uname().sysname, "machine": os.uname().machine}}
json.dump(res, open(out, "w"), indent=1)
print(markdown(res, with_c7))
+ check_floors(summary, len(qs))
+
+
+def check_floors(summary, total):
+ if total != 105:
+ return
+ for variant, floor in [("fetched", 96), ("hybrid", 60)]:
+ if variant in summary and summary[variant]["passed"] < floor:
+ raise RuntimeError(f"{variant} regressed: {summary[variant]['passed']}/105, required >= {floor}/105")
def agg(rs, total):
@@ -247,6 +268,13 @@ def result_of(r, key):
return r.get("context7", {}) if key == "context7" else r["variants"].get(key, {})
+def fetch_file_count(package):
+ """Accept the original flat fetch report and the additive manifest layout."""
+ if "files" in package:
+ return package["files"] or 0
+ return (package.get("upstream") or {}).get("files", 0)
+
+
def markdown(res, with_c7):
s = res["summary"]
cols = [(n, label) for n, label in res.get("variants", [["lockdocs", "lockdocs"]])]
@@ -290,7 +318,7 @@ def markdown(res, with_c7):
for p, v in res["fetch"].items():
rep = v["report"]
if isinstance(rep, dict):
- pk = ", ".join(f"{x['package']} {x.get('files', 0)} files" for x in rep.get("packages", []) if x.get("files"))
+ pk = ", ".join(f"{x['package']} {fetch_file_count(x)} files" for x in rep.get("packages", []) if fetch_file_count(x))
out.append(f"- {p}: {v['ms']} ms ({pk or 'no upstream docs'})")
else:
out.append(f"- {p}: {v['ms']} ms (error)")
diff --git a/bench/test_run.py b/bench/test_run.py
new file mode 100644
index 0000000..2acd819
--- /dev/null
+++ b/bench/test_run.py
@@ -0,0 +1,90 @@
+#!/usr/bin/env python3
+"""Network-free tests of benchmark isolation and regression gates."""
+import contextlib
+import importlib.util
+import io
+import json
+import os
+from pathlib import Path
+import sys
+import tempfile
+import unittest
+from unittest.mock import patch
+
+spec = importlib.util.spec_from_file_location("bench_runner", Path(__file__).with_name("run.py"))
+runner = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(runner)
+
+
+class BenchmarkTests(unittest.TestCase):
+ def test_floors_apply_only_to_full_suite(self):
+ runner.check_floors({"fetched": {"passed": 96}, "hybrid": {"passed": 60}}, 105)
+ runner.check_floors({"fetched": {"passed": 0}}, 1)
+ for variant, score in [("fetched", 95), ("hybrid", 59)]:
+ with self.assertRaises(RuntimeError):
+ runner.check_floors({variant: {"passed": score}}, 105)
+
+ def test_fetch_formatter_preserves_flat_and_nested_file_counts(self):
+ for package in [
+ {"package": "axum@0.7.9", "files": 20},
+ {"package": "axum@0.7.9", "upstream": {"files": 20}},
+ {"package": "axum@0.7.9", "files": 20, "upstream": {"files": 20}},
+ ]:
+ self.assertEqual(runner.fetch_file_count(package), 20)
+ result = {"tokenizer": "test", "runner": {"os": "Test", "machine": "test"},
+ "rows": [], "summary": {}, "variants": [], "index": {},
+ "fetch": {"axum07": {"ms": 1, "report": {"packages": [package]}}}}
+ formatted = runner.markdown(result, False)
+ self.assertIn("axum@0.7.9 20 files", formatted)
+ self.assertNotIn("no upstream docs", formatted)
+ self.assertEqual(runner.fetch_file_count({"files": 0, "upstream": {"files": 20}}), 0)
+
+ def test_first_use_has_own_empty_cache_and_no_opt_in(self):
+ with tempfile.TemporaryDirectory(prefix="lockdocs-bench-test-") as tmp:
+ root = Path(tmp)
+ (root / "questions.json").write_text(json.dumps({"questions": [{
+ "id": "fixture", "project": "project", "package": "fixture",
+ "question": "documented API", "why": "harness test", "expect": [["right_api"]],
+ }]}))
+ (root / "projects" / "project").mkdir(parents=True)
+ default_cache = root / "regular-cache"
+ default_cache.mkdir()
+ (default_cache / "prefetched").write_text("already present")
+ log = root / "calls.jsonl"
+ binary = root / "lockdocs"
+ binary.write_text('''#!/usr/bin/env python3
+import json, os, pathlib, sys
+cache = pathlib.Path(os.environ["LOCKDOCS_CACHE"])
+cmd = sys.argv[1]
+with open(os.environ["TEST_LOG"], "a") as log:
+ log.write(json.dumps({"cmd": cmd, "cache": str(cache), "empty": not any(cache.iterdir()),
+ "fetch": os.environ.get("LOCKDOCS_FETCH"), "token": os.environ.get("GITHUB_TOKEN"),
+ "no_upstream": os.environ.get("LOCKDOCS_NO_UPSTREAM")}) + "\\n")
+if cmd in ("index", "fetch"):
+ print("{}")
+else:
+ (cache / "queried").write_text("cached")
+ print("fixture@1.0.0 · npm · source\\nright_api")
+''')
+ binary.chmod(0o755)
+ out = root / "results.json"
+ with patch.object(runner, "HERE", str(root)), patch.object(sys, "argv", [
+ "run.py", str(binary), str(root / "projects"), str(out), "--fetch",
+ ]), patch.dict(os.environ, {
+ "LOCKDOCS_CACHE": str(default_cache), "TEST_LOG": str(log),
+ "LOCKDOCS_FETCH": "1", "GITHUB_TOKEN": "test-placeholder",
+ }), contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
+ runner.main()
+ calls = [json.loads(line) for line in log.read_text().splitlines()]
+ cold = next(c for c in calls if c["cmd"] == "docs" and c["cache"] != str(default_cache))
+ self.assertTrue(cold["empty"])
+ self.assertIsNone(cold["fetch"])
+ self.assertIsNone(cold["token"])
+ self.assertIsNone(cold["no_upstream"])
+ self.assertLess(calls.index(cold), next(i for i, c in enumerate(calls) if c["cmd"] == "fetch"))
+ self.assertEqual(json.loads(out.read_text())["summary"]["first-use-default"]["passed"], 1)
+ self.assertTrue((default_cache / "prefetched").exists())
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/crates/lockdocs-core/src/fetch.rs b/crates/lockdocs-core/src/fetch.rs
index db63ef3..75d6aa0 100644
--- a/crates/lockdocs-core/src/fetch.rs
+++ b/crates/lockdocs-core/src/fetch.rs
@@ -41,6 +41,9 @@ pub fn fetched_dir(dep: &Dep) -> PathBuf {
/// A previously fetched copy, if any (never touches the network).
pub fn cached(dep: &Dep) -> Option {
+ if is_git(dep) {
+ return None;
+ }
let dir = fetched_dir(dep);
if dir.join(".lockdocs-complete").is_file() {
return Some(source_for(dep, &dir));
@@ -78,13 +81,26 @@ fn source_for(dep: &Dep, dir: &Path) -> Source {
}
/// Download and unpack `dep` if it is not cached yet.
+pub fn is_git(dep: &Dep) -> bool {
+ dep.from.ends_with("(git)")
+}
+
+/// Shared guard: a git checkout is not interchangeable with a registry release tag.
+pub fn require_registry_origin(dep: &Dep) -> Result<()> {
+ if is_git(dep) {
+ bail!(
+ "{} is a git dependency; use its resolved checkout files, not registry/release-tag docs",
+ dep.id()
+ );
+ }
+ Ok(())
+}
+
pub fn fetch(dep: &Dep) -> Result {
+ require_registry_origin(dep)?;
if let Some(s) = cached(dep) {
return Ok(s);
}
- if dep.from.ends_with("(git)") {
- bail!("{} is a git dependency; run `cargo fetch` to check it out", dep.id());
- }
let dir = fetched_dir(dep);
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir)?;
diff --git a/crates/lockdocs-core/src/index.rs b/crates/lockdocs-core/src/index.rs
index dcf926e..ff8fa10 100644
--- a/crates/lockdocs-core/src/index.rs
+++ b/crates/lockdocs-core/src/index.rs
@@ -12,7 +12,7 @@ use std::path::{Path, PathBuf};
use std::time::Instant;
/// Bump when extraction or the on-disk format changes.
-pub const FORMAT: u32 = 9;
+pub const FORMAT: u32 = 11;
#[derive(Serialize, Deserialize)]
pub struct PackageIndex {
@@ -33,6 +33,8 @@ pub struct PackageIndex {
pub build_ms: u64,
/// `github.com/o/r@tag (N files)` when upstream docs are included.
pub upstream: Option,
+ /// Fetch manifest used by this index, including immutable commit and failure notes.
+ pub upstream_provenance: Option,
/// Embedding model id, or empty for keyword-only.
pub embed: String,
/// One embedding per entry (empty without a model).
@@ -154,7 +156,14 @@ pub fn embed_text(e: &Entry) -> String {
}
fn cache_path(dep: &Dep, src: &Source, types: Option<&Path>, up: Option<&(PathBuf, Manifest)>, embed: &str) -> PathBuf {
- let up_key = up.map(|(_, m)| format!("{}@{:?}:{}:{:?}", m.repo, m.tag, m.files, m.site)).unwrap_or_default();
+ let up_key = up
+ .map(|(_, m)| {
+ format!(
+ "{}@{:?}:{}:{:?}:{:?}:{:?}:{}:{}",
+ m.repo, m.tag, m.files, m.site, m.commit, m.note, m.docs_sites_checked, m.format
+ )
+ })
+ .unwrap_or_default();
let key = cache::hash(&[
embed,
&up_key,
@@ -210,9 +219,17 @@ pub fn build(dep: &Dep, src: &Source, root: &Path, up: Option<&(PathBuf, Manifes
head,
head_lens,
build_ms: t.elapsed().as_millis() as u64,
- upstream: up
- .filter(|(_, m)| m.files > 0)
- .map(|(_, m)| format!("{}@{} ({} files)", m.repo, m.tag.as_deref().unwrap_or("?"), m.files)),
+ upstream: up.filter(|(_, m)| m.files > 0).map(|(_, m)| {
+ format!(
+ "{}@{}{}{} ({} files)",
+ m.repo,
+ m.tag.as_deref().unwrap_or("?"),
+ m.commit.as_ref().map(|c| format!(" commit {c}")).unwrap_or_default(),
+ m.site.as_ref().map(|s| format!(" + docs site {s}")).unwrap_or_default(),
+ m.files
+ )
+ }),
+ upstream_provenance: up.map(|(_, m)| m.clone()),
embed: if model.is_some() { embed::MODEL_ID.to_string() } else { String::new() },
vecs,
}
diff --git a/crates/lockdocs-core/src/lib.rs b/crates/lockdocs-core/src/lib.rs
index 70e3883..a3f3e44 100644
--- a/crates/lockdocs-core/src/lib.rs
+++ b/crates/lockdocs-core/src/lib.rs
@@ -1,6 +1,7 @@
//! lockdocs core: read lockfiles, find each dependency's installed sources,
//! extract API reference and prose, and answer queries with BM25 under a
-//! token budget. Everything is local unless fetching is explicitly enabled.
+//! token budget. Public release-tag docs enrich first-use queries by default;
+//! opt-out and offline controls keep installed/cached answers local.
pub mod bm25;
pub mod cache;
diff --git a/crates/lockdocs-core/src/lockfile.rs b/crates/lockdocs-core/src/lockfile.rs
index d9a4946..0ddf716 100644
--- a/crates/lockdocs-core/src/lockfile.rs
+++ b/crates/lockdocs-core/src/lockfile.rs
@@ -21,6 +21,16 @@ pub const LOCKFILES: &[(&str, Eco)] = &[
("go.mod", Eco::Go),
];
+fn git_url(s: &str) -> bool {
+ s.contains("git+")
+ || s.contains("git://")
+ || s.contains("git@")
+ || s.contains("github:")
+ || s.contains(".git#")
+ || s.contains(".git?")
+ || s.ends_with(".git")
+}
+
fn dep(eco: Eco, name: &str, version: &str, direct: bool, from: &str) -> Dep {
Dep {
eco,
@@ -95,13 +105,23 @@ pub fn npm_lock(text: &str, from: &str) -> anyhow::Result> {
}
let Some(ver) = p.get("version").and_then(|v| v.as_str()) else { continue };
let name = p.get("name").and_then(|n| n.as_str()).unwrap_or(name);
- out.push(dep(Eco::Npm, name, ver, direct.contains(name), from));
+ let origin = if p.get("resolved").and_then(Value::as_str).is_some_and(git_url) || git_url(ver) {
+ format!("{from} (git)")
+ } else {
+ from.to_string()
+ };
+ out.push(dep(Eco::Npm, name, ver, direct.contains(name), &origin));
}
} else if let Some(deps) = v.get("dependencies").and_then(|d| d.as_object()) {
// lockfileVersion 1
for (name, p) in deps {
if let Some(ver) = p.get("version").and_then(|v| v.as_str()) {
- out.push(dep(Eco::Npm, name, ver, true, from));
+ let origin = if p.get("resolved").and_then(Value::as_str).is_some_and(git_url) || git_url(ver) {
+ format!("{from} (git)")
+ } else {
+ from.to_string()
+ };
+ out.push(dep(Eco::Npm, name, ver, true, &origin));
}
}
// v1 has no reliable direct marker: every top-level entry counts.
@@ -260,6 +280,7 @@ pub fn yarn_lock(text: &str, direct: &HashSet) -> Vec {
let mut out = Vec::new();
let mut seen = HashSet::new();
let mut names: Vec = Vec::new();
+ let mut git_origin = false;
for line in text.lines() {
if line.is_empty() || line.starts_with('#') {
continue;
@@ -270,6 +291,7 @@ pub fn yarn_lock(text: &str, direct: &HashSet) -> Vec {
continue;
}
let header = line.trim_end_matches(':');
+ git_origin = git_url(header);
for spec in header.split(", ") {
let spec = spec.trim().trim_matches('"');
if let Some((name, _)) = split_at_version(spec) {
@@ -286,7 +308,13 @@ pub fn yarn_lock(text: &str, direct: &HashSet) -> Vec {
let ver = ver.trim().trim_matches('"');
for n in &names {
if seen.insert(format!("{n}@{ver}")) {
- out.push(dep(Eco::Npm, n, ver, direct.contains(n), "yarn.lock"));
+ out.push(dep(
+ Eco::Npm,
+ n,
+ ver,
+ direct.contains(n),
+ if git_origin { "yarn.lock (git)" } else { "yarn.lock" },
+ ));
}
}
names.clear();
@@ -417,7 +445,12 @@ pub fn uv_lock(text: &str) -> anyhow::Result> {
if local.contains(name) {
continue;
}
- out.push(dep(Eco::PyPI, name, ver, direct.contains(&norm_name(Eco::PyPI, name)), "uv.lock"));
+ let from = if p.get("source").is_some_and(|s| s.get("git").is_some()) {
+ "uv.lock (git)"
+ } else {
+ "uv.lock"
+ };
+ out.push(dep(Eco::PyPI, name, ver, direct.contains(&norm_name(Eco::PyPI, name)), from));
}
Ok(out)
}
@@ -430,7 +463,15 @@ pub fn poetry_lock(text: &str, from: &str, direct: &HashSet) -> anyhow::
continue;
};
let is_direct = direct.is_empty() || direct.contains(&norm_name(Eco::PyPI, name));
- out.push(dep(Eco::PyPI, name, ver, is_direct, from));
+ let origin = if p
+ .get("source")
+ .is_some_and(|s| s.get("git").is_some() || s.get("type").and_then(toml::Value::as_str) == Some("git"))
+ {
+ format!("{from} (git)")
+ } else {
+ from.to_string()
+ };
+ out.push(dep(Eco::PyPI, name, ver, is_direct, &origin));
}
Ok(out)
}
@@ -441,7 +482,13 @@ pub fn pipfile_lock(text: &str) -> anyhow::Result> {
for sect in ["default", "develop"] {
for (name, p) in v.get(sect).and_then(|s| s.as_object()).into_iter().flatten() {
if let Some(ver) = p.get("version").and_then(|v| v.as_str()) {
- out.push(dep(Eco::PyPI, name, ver.trim_start_matches("=="), true, "Pipfile.lock"));
+ out.push(dep(
+ Eco::PyPI,
+ name,
+ ver.trim_start_matches("=="),
+ true,
+ if p.get("git").is_some() { "Pipfile.lock (git)" } else { "Pipfile.lock" },
+ ));
}
}
}
@@ -537,6 +584,26 @@ fn parse_replace(line: &str, out: &mut BTreeMap) {
mod tests {
use super::*;
+ #[test]
+ fn python_and_npm_git_origins_are_not_registry_releases() {
+ let uv = r#"[[package]]
+name = "git-lib"
+version = "1.2.3"
+source = { git = "https://github.com/o/r?rev=abc#01234567" }
+"#;
+ assert_eq!(uv_lock(uv).unwrap()[0].from, "uv.lock (git)");
+ let poetry = r#"[[package]]
+name = "git-lib"
+version = "1.2.3"
+[package.source]
+type = "git"
+url = "https://github.com/o/r"
+"#;
+ assert_eq!(poetry_lock(poetry, "poetry.lock", &HashSet::new()).unwrap()[0].from, "poetry.lock (git)");
+ let npm = r#"{"packages":{"node_modules/git-lib":{"version":"1.2.3","resolved":"git+https://github.com/o/r.git#abc"}}}"#;
+ assert_eq!(npm_lock(npm, "package-lock.json").unwrap()[0].from, "package-lock.json (git)");
+ }
+
#[test]
fn npm_v3() {
let t = r#"{"lockfileVersion":3,"packages":{"":{"dependencies":{"zod":"^3.23.0"},"devDependencies":{"@types/node":"^20"}},
diff --git a/crates/lockdocs-core/src/query.rs b/crates/lockdocs-core/src/query.rs
index 9269b7a..99fab8c 100644
--- a/crates/lockdocs-core/src/query.rs
+++ b/crates/lockdocs-core/src/query.rs
@@ -24,19 +24,30 @@ const HEAD_WEIGHT: f32 = 0.25;
/// Cross-dependency searches index at most this many direct dependencies.
const MAX_PACKAGES: usize = 80;
-#[derive(Debug, Clone)]
+#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Options {
/// Allow downloading exact versions that are not installed.
pub fetch: bool,
+ /// Anonymous release-tag docs on first use; independent of registry downloads.
+ pub upstream: bool,
}
impl Default for Options {
fn default() -> Self {
- let fetch = std::env::var("LOCKDOCS_FETCH").is_ok_and(|v| matches!(v.as_str(), "1" | "true" | "yes"));
- Options { fetch }
+ let mut fetch = std::env::var("LOCKDOCS_FETCH").is_ok_and(|v| matches!(v.as_str(), "1" | "true" | "yes"));
+ let mut upstream = automatic_upstream(std::env::var("LOCKDOCS_FETCH").ok().as_deref(), std::env::var("LOCKDOCS_NO_UPSTREAM").is_ok());
+ if std::env::var("LOCKDOCS_OFFLINE").is_ok_and(|v| v != "0") {
+ fetch = false;
+ upstream = false;
+ }
+ Options { fetch, upstream }
}
}
+fn automatic_upstream(fetch: Option<&str>, disabled: bool) -> bool {
+ !disabled && !fetch.is_some_and(|v| matches!(v, "0" | "false" | "no"))
+}
+
pub struct Answer {
pub text: String,
pub json: Value,
@@ -46,6 +57,7 @@ pub struct Engine {
pub project: Project,
opts: Options,
indexes: Mutex>>,
+ upstream_notes: Mutex>,
}
/// An indexed package, the dependency it answers for, and a drift note.
@@ -57,6 +69,68 @@ enum Resolved {
Missing(Dep, String),
}
+/// Keep the original flat fields for CLI consumers; richer provenance is additive.
+fn fetch_report(dep: &Dep, status: &upstream::Manifest) -> Value {
+ json!({"package": dep.id(), "repo": status.repo, "tag": status.tag,
+ "files": status.files, "note": status.note, "upstream": status})
+}
+
+fn provenance(ready: &[ReadyPkg]) -> Vec {
+ ready
+ .iter()
+ .map(|(idx, dep, note)| {
+ json!({
+ "package": idx.id(), "requested_version": dep.version, "source": idx.source,
+ "registry_fetched": idx.fetched, "upstream": idx.upstream_provenance, "upstream_label": idx.upstream, "note": note,
+ })
+ })
+ .collect()
+}
+
+/// Keep network fallbacks visible on ordinary broad MCP answers without listing
+/// every dependency's long header or crowding out the actual answer.
+fn broad_header(ready: &[ReadyPkg], missing: &[(Dep, String)], tokens: usize) -> String {
+ let fetched = ready.iter().filter(|r| r.0.fetched).count();
+ let upstream = ready.iter().filter(|r| r.0.upstream.is_some()).count();
+ let notes = ready.iter().filter(|r| r.2.is_some()).count();
+ let mut header = format!("Searched {} direct dependencies: {} local package sources, {fetched} registry sources, {upstream} upstream sources; {notes} fallback notes, {} unavailable.\n", ready.len(), ready.len() - fetched, missing.len());
+ let max_chars = (tokens / 3).clamp(100, 400) * 3;
+ for (idx, _, note) in ready {
+ if let Some(note) = note {
+ let row = format!("Fallback {}: {}\n", idx.id(), truncate(note, 160));
+ if header.chars().count() + row.chars().count() > max_chars {
+ break;
+ }
+ header.push_str(&row);
+ }
+ }
+ for (dep, error) in missing {
+ let row = format!("Skipped {}: {}\n", dep.id(), truncate(error, 140));
+ if header.chars().count() + row.chars().count() > max_chars {
+ break;
+ }
+ header.push_str(&row);
+ }
+ for (idx, _, _) in ready {
+ if let Some(m) = &idx.upstream_provenance {
+ let row = format!(
+ "Upstream {}: {}@{} commit {}{}\n",
+ idx.id(),
+ m.repo,
+ m.tag.as_deref().unwrap_or("none"),
+ m.commit.as_deref().map(|s| &s[..s.len().min(7)]).unwrap_or("none"),
+ if m.site.is_some() { " + opted-in docs site" } else { "" }
+ );
+ if header.chars().count() + row.chars().count() > max_chars {
+ break;
+ }
+ header.push_str(&row);
+ }
+ }
+ header.push_str("Focus `package` or request JSON for full provenance.\n");
+ header
+}
+
fn lang(eco: Eco) -> &'static str {
match eco {
Eco::Npm => "ts",
@@ -125,6 +199,7 @@ impl Engine {
project: Project::load(root),
opts,
indexes: Mutex::new(HashMap::new()),
+ upstream_notes: Mutex::new(HashMap::new()),
}
}
@@ -151,29 +226,17 @@ impl Engine {
if self.opts.fetch {
match fetch::fetch(dep) {
Ok(s) => return Ok((s, None)),
- Err(e) => {
- if local.is_none() {
- return Err(format!("{} is pinned but not installed, and fetching failed: {e:#}", dep.id()));
- }
- }
+ Err(e) => return Err(format!("{}: exact registry fetching failed: {e:#}; no other version substituted", dep.id())),
}
}
if let Some(s) = local {
- let note = format!(
- "{} pins {}@{}, but {} has {}; showing the installed {}. Reinstall to sync{}.",
+ return Err(format!(
+ "{} pins {}, but {} has {}; refusing to substitute the installed version. Reinstall to sync, or enable exact registry fetching (--fetch).",
dep.from,
- dep.name,
- dep.version,
+ dep.id(),
s.label,
- s.version,
- s.version,
- if self.opts.fetch {
- ""
- } else {
- ", or enable fetching (--fetch / LOCKDOCS_FETCH=1) to read the pinned version"
- }
- );
- return Ok((s, Some(note)));
+ s.version
+ ));
}
let how = match dep.eco {
Eco::Npm => "run your package manager's install",
@@ -190,15 +253,30 @@ impl Engine {
fn index(&self, dep: &Dep) -> Resolved {
match self.source(dep) {
- Ok((src, note)) => {
+ Ok((src, mut note)) => {
let key = format!("{}:{}@{}:{}", dep.eco, dep.name, src.version, src.dir.display());
if let Some(i) = self.indexes.lock().unwrap().get(&key) {
// Rebuild once the embedding model has arrived.
if !i.embed.is_empty() || embed::get().is_none() {
+ if let Some(n) = self.upstream_notes.lock().unwrap().get(&dep.id()) {
+ note = Some(match note {
+ Some(old) => format!("{old} {n}"),
+ None => n.clone(),
+ });
+ }
return Resolved::Ready(i.clone(), dep.clone(), note);
}
}
- let up = self.upstream(dep, &src);
+ let (up, up_note) = self.upstream(dep, &src);
+ if let Some(n) = up_note {
+ self.upstream_notes.lock().unwrap().insert(dep.id(), n.clone());
+ note = Some(match note {
+ Some(old) => format!("{old} {n}"),
+ None => n,
+ });
+ } else {
+ self.upstream_notes.lock().unwrap().remove(&dep.id());
+ }
let idx = Arc::new(index::load_or_build(dep, &src, &self.project.root, up.as_ref()));
self.indexes.lock().unwrap().insert(key, idx.clone());
Resolved::Ready(idx, dep.clone(), note)
@@ -209,19 +287,42 @@ impl Engine {
/// Upstream docs for this exact version: a cached copy, or (when fetching
/// is enabled) a one-time download.
- fn upstream(&self, dep: &Dep, src: &Source) -> Option<(PathBuf, upstream::Manifest)> {
+ fn upstream(&self, dep: &Dep, src: &Source) -> (Option<(PathBuf, upstream::Manifest)>, Option) {
if src.version != dep.version || std::env::var("LOCKDOCS_NO_UPSTREAM").is_ok() {
- return None;
+ return (None, None);
+ }
+ let enabled = self.opts.fetch || self.opts.upstream;
+ if fetch::is_git(dep) {
+ return (
+ None,
+ enabled.then(|| "upstream docs skipped: git dependency's resolved commit is not a release tag; using its checkout files".into()),
+ );
}
let cached = upstream::cached(dep);
- if let Some(c) = cached.as_ref().filter(|(_, m)| !(self.opts.fetch && upstream::stale(m))) {
- return Some(c.clone());
+ if let Some(c) = cached.as_ref().filter(|(_, m)| !enabled || !upstream::needs_refresh(m, self.opts.fetch)) {
+ let note = if upstream::needs_refresh(&c.1, false) {
+ Some(format!(
+ "offline fallback: legacy upstream cache format {} has not been exact-tag revalidated; its old commit candidate may have been a branch{}",
+ c.1.format,
+ c.1.note.as_ref().map(|n| format!("; {n}")).unwrap_or_default()
+ ))
+ } else {
+ c.1.note.clone()
+ };
+ return (Some(c.clone()), note);
}
- if self.opts.fetch && upstream::repo_of(dep, src).is_some() {
- let _ = upstream::fetch(dep, src);
- return upstream::cached(dep);
+ if enabled {
+ let result = if self.opts.fetch {
+ upstream::fetch(dep, src)
+ } else {
+ upstream::fetch_automatic(dep, src)
+ };
+ return match result {
+ Ok(m) => (Some((upstream::dir(dep), m.clone())), m.note.clone()),
+ Err(e) => (None, Some(format!("upstream docs fetch failed: {e:#}; using package files"))),
+ };
}
- None
+ (None, None)
}
/// `lockdocs fetch`: download what makes answers complete, once: missing
@@ -252,12 +353,20 @@ impl Engine {
let rows: Vec<(Dep, String, Value)> = deps
.par_iter()
.map(|d| {
+ if fetch::is_git(d) {
+ let src = locate::locate(d, &self.project.root);
+ return (
+ d.clone(),
+ "upstream skipped: git dependency; use resolved checkout files".into(),
+ json!({"package": d.id(), "status": "git-checkout", "source": src.map(|s| s.label)}),
+ );
+ }
let src = match locate::locate(d, &self.project.root)
.filter(|s| s.version == d.version)
.or_else(|| fetch::cached(d))
{
Some(s) => Some(s),
- None if !d.from.ends_with("(git)") => fetch::fetch(d).ok(),
+ None if !fetch::is_git(d) => fetch::fetch(d).ok(),
None => None,
};
let Some(src) = src else {
@@ -267,18 +376,15 @@ impl Engine {
json!({"package": d.id(), "status": "missing"}),
);
};
- let status = match upstream::cached(d).filter(|(_, m)| !upstream::stale(m)) {
- Some((_, m)) => m,
- None => match upstream::fetch(d, &src) {
- Ok(m) => m,
- Err(e) => {
- return (
- d.clone(),
- format!("upstream docs: {e:#}"),
- json!({"package": d.id(), "status": "no-upstream", "error": format!("{e:#}")}),
- );
- }
- },
+ let status = match upstream::fetch(d, &src) {
+ Ok(m) => m,
+ Err(e) => {
+ return (
+ d.clone(),
+ format!("upstream docs fetch failed: {e:#}"),
+ json!({"package": d.id(), "status": "fetch-failed", "error": format!("{e:#}")}),
+ )
+ }
};
let line = match (&status.tag, status.files) {
(_, n) if n > 0 => format!(
@@ -291,8 +397,8 @@ impl Engine {
};
(
d.clone(),
- line,
- json!({"package": d.id(), "repo": status.repo, "tag": status.tag, "files": status.files, "note": status.note}),
+ if let Some(note) = &status.note { format!("{line}; {note}") } else { line },
+ fetch_report(d, &status),
)
})
.collect();
@@ -453,7 +559,7 @@ impl Engine {
if let Some(n) = note {
text.push_str(&format!(" note: {n}\n"));
}
- items.push(json!({"package": idx.id(), "ecosystem": idx.eco.as_str(), "symbols": symbols, "sections": idx.entries.len() - symbols, "source": idx.source, "build_ms": idx.build_ms}));
+ items.push(json!({"package": idx.id(), "ecosystem": idx.eco.as_str(), "symbols": symbols, "sections": idx.entries.len() - symbols, "source": idx.source, "upstream": idx.upstream, "note": note, "build_ms": idx.build_ms}));
}
for (d, e) in &missing {
text.push_str(&format!(" {:<40} missing: {e}\n", d.id()));
@@ -584,7 +690,7 @@ impl Engine {
}
}
if broad {
- header = format!("Searched {} direct dependencies. Pass `package` to focus.\n", ready.len());
+ header = broad_header(&ready, &missing, tokens);
}
for (d, e) in &missing {
if !broad {
@@ -647,7 +753,7 @@ impl Engine {
}
}
Ok(Answer {
- json: json!({"query": q, "packages": ready.iter().map(|r| r.0.id()).collect::>(), "hits": hits_json, "tokens": est_tokens(&out.text)}),
+ json: json!({"query": q, "packages": ready.iter().map(|r| r.0.id()).collect::>(), "hits": hits_json, "provenance": provenance(&ready), "tokens": est_tokens(&out.text)}),
text: out.text,
})
}
@@ -656,6 +762,9 @@ impl Engine {
let mut out = Pack::new(tokens);
for (idx, dep, note) in ready {
out.push_raw(&format!("{} · {} · {} · pinned in {}\n", idx.id(), idx.eco, idx.source, dep.from));
+ if let Some(u) = &idx.upstream {
+ out.push_raw(&format!("Upstream: {u}\n"));
+ }
if let Some(n) = note {
out.push_raw(&format!("Note: {n}\n"));
}
@@ -689,7 +798,7 @@ impl Engine {
}
}
Answer {
- json: json!({"packages": ready.iter().map(|r| r.0.id()).collect::>(), "tokens": est_tokens(&out.text)}),
+ json: json!({"packages": ready.iter().map(|r| r.0.id()).collect::>(), "provenance": provenance(ready), "tokens": est_tokens(&out.text)}),
text: out.text,
}
}
@@ -770,6 +879,9 @@ impl Engine {
format!(" · pinned in {}", dep.from)
}
));
+ if let Some(u) = &idx.upstream {
+ out.push_raw(&format!("Upstream: {u}\n"));
+ }
if let Some(n) = note {
out.push_raw(&format!("Note: {n}\n"));
}
@@ -839,7 +951,7 @@ impl Engine {
}
Ok(Answer {
json: json!({
- "package": idx.id(), "kind": best.kind.as_str(), "path": best.path, "signature": best.sig, "doc": best.doc,
+ "provenance": provenance(&ready), "package": idx.id(), "kind": best.kind.as_str(), "path": best.path, "signature": best.sig, "doc": best.doc,
"file": best.file, "line": best.line, "members": members_json, "tokens": est_tokens(&out.text),
}),
text: out.text,
@@ -1472,7 +1584,7 @@ impl Workspace {
let stamp = lock_stamp(&root);
let mut m = self.engines.lock().unwrap();
if let Some((s, e)) = m.get(&root) {
- if *s == stamp {
+ if *s == stamp && e.opts == *opts {
return e.clone();
}
}
@@ -1491,6 +1603,50 @@ pub fn dep_key(d: &Dep) -> String {
mod tests {
use super::*;
+ #[test]
+ fn fetch_report_preserves_flat_counts_and_adds_full_provenance() {
+ let dep = Dep {
+ eco: Eco::Cargo,
+ name: "axum".into(),
+ version: "0.7.9".into(),
+ direct: true,
+ from: "Cargo.lock".into(),
+ };
+ let status: upstream::Manifest = serde_json::from_value(json!({
+ "format": upstream::FORMAT, "repo": "github.com/tokio-rs/axum", "tag": "axum-v0.7.9",
+ "commit": "0123456789abcdef0123456789abcdef01234567", "files": 20, "bytes": 100, "note": null,
+ }))
+ .unwrap();
+ let report = fetch_report(&dep, &status);
+ assert_eq!(report["files"], 20);
+ assert_eq!(report["files"], report["upstream"]["files"]);
+ assert_eq!(report["tag"], report["upstream"]["tag"]);
+ assert_eq!(report["repo"], report["upstream"]["repo"]);
+ assert!(report.get("note").is_some());
+ }
+
+ #[test]
+ fn upstream_is_default_with_explicit_opt_out() {
+ assert!(automatic_upstream(None, false));
+ assert!(automatic_upstream(Some("1"), false));
+ for value in ["0", "false", "no"] {
+ assert!(!automatic_upstream(Some(value), false));
+ }
+ assert!(!automatic_upstream(None, true));
+ }
+
+ #[test]
+ fn workspace_does_not_reuse_network_policy() {
+ let ws = Workspace::default();
+ let root = std::env::temp_dir();
+ let online = Options { fetch: false, upstream: true };
+ let offline = Options { fetch: false, upstream: false };
+ let a = ws.engine(&root, &online);
+ let b = ws.engine(&root, &offline);
+ assert!(!Arc::ptr_eq(&a, &b));
+ assert!(!b.opts.upstream);
+ }
+
#[test]
fn old_upgrade_guides_and_deprecated_pages() {
let e = |name: &str, file: &str| Entry {
diff --git a/crates/lockdocs-core/src/upstream.rs b/crates/lockdocs-core/src/upstream.rs
index 5e116c0..04a7195 100644
--- a/crates/lockdocs-core/src/upstream.rs
+++ b/crates/lockdocs-core/src/upstream.rs
@@ -3,7 +3,8 @@
//! This module finds the repository from the package's own metadata, finds
//! the tag for the pinned version, and downloads only the docs folders
//! (Markdown, MDX, reStructuredText) from GitHub into the cache, once.
-//! Network use is opt-in: `lockdocs fetch`, `--fetch` or `LOCKDOCS_FETCH=1`.
+//! First-use queries fetch public docs at an immutable release commit. Explicit
+//! `fetch` also supports major-version docs sites and optional GitHub credentials.
use crate::locate::Source;
use crate::{cache, Dep, Eco};
@@ -28,7 +29,7 @@ pub struct Repo {
/// Bump when what `fetch` downloads changes, so `lockdocs fetch` refreshes
/// older copies (a stale copy is still used until then).
-pub const FORMAT: u32 = 2;
+pub const FORMAT: u32 = 4;
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Manifest {
@@ -37,6 +38,12 @@ pub struct Manifest {
pub format: u32,
pub repo: String,
pub tag: Option,
+ /// Immutable package-repository commit resolved from the release tag.
+ #[serde(default)]
+ pub commit: Option,
+ /// Explicit enrichment completed a docs-site lookup (including a genuine empty result).
+ #[serde(default)]
+ pub docs_sites_checked: bool,
pub files: usize,
pub bytes: u64,
/// Why nothing was downloaded, when files == 0.
@@ -123,30 +130,61 @@ const DOCS_SITES: &[DocsSite] = &[
];
/// The latest stable major of a package, from its registry.
-fn latest_major(agent: &ureq::Agent, dep: &Dep) -> Option {
+fn latest_major(agent: &Client, dep: &Dep) -> Result