test(sage): run the Code Review Sage suite on Windows too - #5335
Conversation
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
1 similar comment
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Blanket collection gate replaced with cause-specific per-test guards — the root-cause shape, using the suite's own established probe and skip patterns, with measured Windows results. [DESIGN-REVIEWED] 6d21cdc |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsI've verified the mechanism. Let me confirm the sibling tests' guard pattern and that The candidate is grounded: This is a real, concrete test-quality regression introduced by the diff (the conftest docstring now claims "Tests that pin inherently POSIX behaviour carry their own skipUnless guards ... unprivileged symlinks" — false for this test). But the harm is false-green test coverage on Windows, not a production security hole, crash, data loss, or corruption. It is advisory, not blocking. One symlink-planting test was missed by the PR's own skip-guard sweep, so it passes vacuously on Windows instead of skipping. FINDING — src/kiro_crew/apps/builtins/code_review_sage/tests/test_consolidate.py:294 — [OPUS-REVIEWED] 6d21cdc |
The suite was collected nowhere on Windows: conftest.py disabled collection for the whole directory. The gate predates the app's own Windows support work and conflated two things -- the app-level refusal in discovery.py's gh_bin (tracked separately) and three harness details that the tests themselves can express. Lift the collection gate and give every platform-dependent test the guard it actually needs: - the planted-symlink tests now share one SYMLINKS_OK probe (moved from test_followup.py into tests/fixtures.py, the suite's shared module) and skip only where creating a symlink needs a privilege the host does not grant; the guards they pin keep running everywhere else; - the two owner-only mode-bit assertions that are unobservable through st_mode on Windows keep their POSIX-only gates, and the report-output privacy test now pins its mode bits on POSIX while still asserting file presence and temp-file cleanup on every platform. Measured on Windows 11: before, 21 failed / 735 passed / 11 skipped with the gate lifted; after, 736 passed / 31 skipped / 0 failed, in both serial and -n4 runs.
Head branch was pushed to by a user without write access
6595d3d to
6d21cdc
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (6 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-harness only -- drops the blanket collect_ignore_glob/skipif Windows gate from the Code Review Sage conftest and replaces it with per-test guards (owner-only mode bits, unprivileged symlink creation), so the suite collects on Windows. No production files touched. CodeQL is not applicable on this fork PR (default-setup emits no check-run); SAST coverage is Semgrep only, latest run success with 0 annotations.
Problem / Motivation
Code Review Sage's test suite is collected nowhere on Windows:
tests/conftest.pysetscollect_ignore_glob = ["*"]for the whole directory, so the platform the app is being brought up on has zero automated coverage of it (#4988). The gate predates the app's Windows support work and conflates the app-level refusal insage_lib/discovery.py'sgh_bin()(its review prompts still namepython3— untouched here, tracked separately) with three harness details the tests themselves can express.Why it matters
The coverage hole sits exactly where recent risk was: a Windows-only silent failure in the review worker had to be found by hand because no Windows shard ran a single sage test. Every PR that touches this app lands blind on the platform it now claims to support.
What changed (motivation → approach → change)
Lift the collection gate and give each platform-dependent test the guard its failure actually needs, instead of one blanket skip:
OSError: [WinError 1314]). The suite already had the right pattern —SYMLINKS_OK, a probe for unprivileged symlink creation, defined privately intest_followup.py. That probe now lives intests/fixtures.py, the suite's shared module, and every test that stages a planted link skips only where creating a symlink needs a privilege the host does not grant (Developer Mode / elevated runners run them again). The no-follow guards they pin keep running everywhere else.0600throughst_mode, which Windows never reports (the lockdown there is an ACL). That assertion is scoped to POSIX while file-presence and temp-file-cleanup checks still run on every platform; the two existingskipUnless(platform_compat.IS_POSIX)sites are unchanged.Alternatives weighed: keeping the blanket gate until #4979 lands (leaves the coverage hole open longer), or making the mode assertions ACL-aware instead of POSIX-gated (no cross-platform "verify owner-only DACL" helper exists yet; that would be a new production-side seam, out of scope for a test-enablement change).
Tests
Test-only change; the diff modifies how tests are gated, not what they assert:
@unittest.skipUnless(SYMLINKS_OK, ...)with the suite's established reason string.test_outputs_are_private_and_leave_no_temp_behindpins its0600assertion behindplatform_compat.IS_POSIXand keeps presence/cleanup assertions unconditional.test_followup.pyimports the shared probe instead of defining its own copy (same semantics, one owner).Manual verification
Measured locally on Windows 11 (Python 3.12, the CI pin set: pytest 9.0.3 / xdist 3.5.0 / pytest-timeout 2.2.0):
-n0)-n4 --dist loadgroup)Gates:
flake8clean,isort --check-onlyclean,mypyclean over the suite's 30 files,git diff --checkclean. The five touched files sit in.github/black-baseline.txt; their formatting state is unchanged and the diff adds no new black findings. N/A for browser/UI checks — nothing user-visible moves.Related Issues
Fixes #4988
Checklist
test: ...)