Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ All notable changes are documented here. This project follows Semantic Versionin

## [Unreleased]

### Fixed

- Stopped `.pyi` stub files with test-like names from inventing executable pytest or `unittest` checks. Stubs remain available to static Python analysis and relationship discovery, but framework discovery now requires an executable `.py` test file; this prevents `--run-checks` from launching a Python test runner merely because a repository contains test-shaped type stubs.

## [0.5.3] - 2026-08-15

### Fixed
Expand Down
2 changes: 1 addition & 1 deletion dist/checks.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion dist/checks.js.map

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/verification-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Node, pytest, and unittest observer records travel over a separate bounded contr
- Coverage ingestion is currently limited to explicitly supplied LCOV plus a user-declared commit that must resolve to the selected target. ProofDiff verifies that equality but does not independently attest artifact provenance. Changed-line reconstruction is capped at 50,000 current lines per file and otherwise remains unmeasured. ProofDiff does not run coverage tools, merge artifacts from different commits, remap source maps, guess CI workspace prefixes, or claim branch/assertion coverage.
- Compiler resolution is intentionally partial: NodeNext-family extensionless paths, directory package metadata, non-default `moduleSuffixes`, Classic lookup, package or array `extends`, project references, standalone `baseUrl`, `${configDir}`, multiple-wildcard mappings, installed packages, and arbitrary bundler aliases are not resolved.
- Package resolution is intentionally partial: only the importing package's exact self-exports under an explicit export-aware compiler mode are considered. Hidden package boundaries, versioned or unmodeled conditions, export patterns/arrays, package imports, workspace dependencies, third-party packages, and `node_modules` are not resolved. Compiler aliases also require bounded evidence that the selected config includes the importer.
- Python namespace packages and dynamic imports may be missed.
- Python namespace packages and dynamic imports may be missed. `.pyi` stubs remain supported static-analysis and test-like relationship inputs, but they do not establish an executable pytest/`unittest` framework or qualify as Python runtime test targets.
- Root project scripts are discovered; monorepo package scripts are only noted.
- Jest/Vitest exact-target support does not interpret shell substitution or chaining, duplicate or more than four environment assignments, `cross-env-shell`, `dotenv`, `concurrently`, arbitrary wrappers, custom runner options/config shapes, workspace package scripts, or package-manager layouts that do not expose the runner as a local `node_modules/<runner>` package. Those cases remain ordinary opaque checks rather than being guessed. A test-map declaration can name a relationship in these repositories, but it cannot make an unsupported runner command exact-target-capable.
- Deleted symbols are inferred only from recognizable removed declarations.
Expand Down
2 changes: 1 addition & 1 deletion src/checks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ async function detectPythonTests(root: string, limit = 2_000): Promise<PythonTes
if (entry.isDirectory() && !entry.isSymbolicLink() && !["node_modules", ".git", "__pycache__", ".venv", "venv", "dist", "build"].includes(entry.name)) {
queue.push({ absolute: target, directory: current.directory === "." && (entry.name === "tests" || entry.name === "test") ? entry.name : current.directory });
}
if (entry.isFile() && /(?:^test_.*|.*_(?:test|spec))\.pyi?$/.test(entry.name)) {
if (entry.isFile() && /(?:^test_.*|.*_(?:test|spec))\.py$/.test(entry.name)) {
const content = await readUtf8File(target, 200_000);
const framework = content !== null && /(?:^|\n)\s*(?:from\s+unittest\b|import\s+unittest\b)|unittest\.TestCase/.test(content) ? "unittest" : "pytest";
if (framework === "pytest") return { framework, directory: current.directory };
Expand Down
18 changes: 18 additions & 0 deletions tests/analyze.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,24 @@ test("Python unittest repositories receive AST-backed related evidence", async (
assert.equal(report.assessments[0]?.status, "verified");
});

test("Python stubs remain static test relationships without inventing a runnable framework", async (context) => {
const root = await initializeRepository({
"value.py": "def value():\n return 1\n",
"tests/test_value.pyi": "from value import value\ndef test_value() -> None: ...\n",
});
context.after(() => rm(root, { recursive: true, force: true }));
await writeFiles(root, { "value.py": "def value():\n return 2\n" });

const report = await analyzeRepository({ repo: root, runChecks: true, timeoutMs: 20_000 });
const assessment = report.assessments[0];
assert.deepEqual(report.checks, []);
assert.deepEqual(assessment?.relatedTests, ["tests/test_value.pyi"]);
assert.deepEqual(assessment?.executedTests, []);
assert.equal(assessment?.status, "unknown");
assert.equal(assessment?.evidenceBoundary?.stage, "runner-qualification");
assert.equal(assessment?.evidenceBoundary?.reason, "runner-unqualified");
});

test("a passing filtered test script cannot imply that an unexecuted related test passed", async (context) => {
const root = await initializeRepository({
"package.json": JSON.stringify({ name: "filtered", private: true, type: "module", scripts: { test: "node --test test/unrelated.test.js" } }, null, 2),
Expand Down
10 changes: 10 additions & 0 deletions tests/checks.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,16 @@ test("a conventional Python test file discovers pytest", async (context) => {
assert.ok(checks.some((check) => check.id === "python:test:pytest"));
});

test("Python stub files do not invent executable pytest or unittest checks", async (context) => {
const root = await initializeRepository({
"tests/test_contract.pyi": "def test_contract() -> None: ...\n",
"tests/test_legacy.pyi": "import unittest\nclass Contract(unittest.TestCase): ...\n",
});
context.after(() => rm(root, { recursive: true, force: true }));
const { checks } = await discoverChecks(root);
assert.deepEqual(checks.filter((check) => check.kind === "test"), []);
});

test("stdlib unittest projects use unittest without requiring pytest", async (context) => {
const root = await initializeRepository({
"value.py": "def value():\n return 2\n",
Expand Down