Skip to content

test: drop platform-dependent exit status assertion in shell test - #15

Merged
SamuelDenani merged 1 commit into
mainfrom
chore/shell-test-platform-agnostic
Oct 7, 2026
Merged

SamuelDenani merged 1 commit into
mainfrom
chore/shell-test-platform-agnostic

Conversation

@SamuelDenani

Copy link
Copy Markdown
Owner

What and why

main's quality gate has been red since 2026-08-31 (the merge of #4), and no
PR in this repo can go green until this lands.
It currently blocks #11, #12,
#13 and anything opened after them.

The cause is one assertion in .loopwright/tests/shell.test.mjs that tests the
host OS rather than the code under test:

const result = runShell('echo aaaa…', process.cwd(), { maxBuffer: 4 });
expect(result.status).toBe(0);          // ← removed
expect(result.stderr).toMatch(/ENOBUFS|maxBuffer/i);

Whether an overflowing maxBuffer leaves the child's exit status at 0 or kills
the process varies by platform and by whether coverage instrumentation is
active. The assertion holds on macOS and fails on ubuntu-latest under
--coverage.

Why the CI log is misleading about this

The workflow runs vitest twice, and only the second run feeds the gate:

Step Command Result
Engine unit tests npx vitest run 152 passed ✅
Collect reports vitest run --coverage (from config.json) tests: exit 1, 1 failing assertion ❌

So a run shows Test Files 14 passed near the top and a red gate at the bottom,
which reads like a gate bug rather than a test failure. The coverage run is the
one collect-metrics.mjs reads, and when it fails, coverage-summary.json is
never written — so the tests collector is recorded as failed and the four
coverage metrics block in cascade. Six blockers, one cause. That cascade is the
gate working correctly: an infrastructure failure blocks, and is not the same as
an unconfigured tool.

What changed

The status assertion is removed and the comment above it rewritten — it
previously stated the false premise ("Output overflowing maxBuffer still exits
0"). The assertion the test exists for, that the spawn error surfaces in
stderr instead of being dropped, is retained unchanged.

This is not weakening a test to make a build green. CLAUDE.md forbids
skipping, focusing, deleting or weakening a failing test, and none of those
happened: no .skip, no .only, no try/catch, no process.platform
conditional, and no change to .loopwright/scripts/lib/shell.mjs. runShell
propagating the error into stderr is correct and is still asserted. What was
removed asserted a property of the operating system.

Verification

Command Result
npx vitest run tests/shell.test.mjs 7 passed
npx vitest run tests/shell.test.mjs --coverage 7 passed
npx vitest run 14 files, 152 passed
run-report.mjs --all && quality-gate.mjs exit 0

Local runs cannot reproduce the Linux failure, so this PR's own CI run is the
real proof.

Left alone deliberately

Two warnings remain and are not in scope here:

  • typecheck collector unconfigured — recorded as such in the baseline.
  • audit.high at 2 — brace-expansion and js-yaml, both with a fix
    available upstream
    . No audit.ignore entry was added; CLAUDE.md forbids
    ignoring an advisory that has a fix, so this wants a dependency upgrade in its
    own PR.

.loopwright/baseline.json and .loopwright/config.json are untouched.

🤖 Generated with Claude Code

The `expect(result.status).toBe(0)` assertion tested the host OS rather than
runShell: whether an overflowing maxBuffer leaves the child at status 0 or
kills it varies by platform and by coverage instrumentation. CI runs the
tests collector under `vitest run --coverage` on ubuntu, where the status
differs, so the test was red there while passing on macOS.

The stderr assertion is retained: it is the behaviour the test protects (the
spawn error surfaces in stderr, signalling that stdout is incomplete). The
misleading comment claiming the status stays 0 is corrected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown

✅ Quality gate passed

commit e2214cc · baseline ece7656 · 2 warning(s) · 6 improvement(s) 📈

⚠️ 2 warning(s) — not blocking
Check Baseline Now Verdict
⚠️ TypeScript errors — n/a collector not configured — see docs/loopwright/quality-gate.md
⚠️ High advisories 0 2 regressed 2 (tolerance 0)

High advisories

  • brace-expansion — high: brace-expansion: Quadratic-time expansion of the {a},b} rewrite causes CPU denial of service · fix available — upgrade it
  • js-yaml — high: js-yaml: maxTotalMergeKeys does not limit CPU use for empty merge sources · fix available — upgrade it
📊 All metrics
Check Baseline Now Verdict
✅ Failing tests 0 0 holding
✅ Failing test suites 0 0 holding
⚠️ TypeScript errors — n/a collector not configured — see docs/loopwright/quality-gate.md
✅ Lint errors 0 0 holding
✅ Lint warnings 0 0 holding
✅ Critical advisories 0 0 holding
⚠️ High advisories 0 2 regressed 2 (tolerance 0)
✅ Suppressed advisories 0 0 holding
📈 Line coverage 84.67% 85.01% improved 0.34%
📈 Branch coverage 78.93% 79.44% improved 0.51%
📈 Function coverage 89.65% 89.71% improved 0.06%
📈 Statement coverage 85.01% 85.41% improved 0.40%
📈 Duplicated code 0.90% 0.88% improved 0.02%
✅ Highest function complexity 14 14 holding
📈 Average function complexity 1.81 1.80 improved 0.01
✅ Oversized files 0 0 holding
✅ Skipped tests 0 0 holding
✅ Focused tests (.only) 0 0 holding
✅ Tests with no assertion 0 0 holding
✅ Coverage-ignore hints 1 1 holding
✅ Type suppressions (@ts-ignore etc.) 2 2 holding
✅ Inline lint suppressions 2 2 holding
✅ Empty catch blocks 0 0 holding

📈 This PR improves 6 metric(s). Run node .loopwright/scripts/quality-gate.mjs --update-baseline and commit .loopwright/baseline.json to lock the gain in.


Generated by .loopwright/scripts/quality-gate.mjs · reproduce locally with node .loopwright/scripts/run-report.mjs --all && node .loopwright/scripts/quality-gate.mjs · full reports are in the workflow artifacts.

@SamuelDenani
SamuelDenani marked this pull request as ready for review October 7, 2026 20:41
@claude

claude Bot commented Oct 7, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

@SamuelDenani
SamuelDenani merged commit 2b510fa into main Oct 7, 2026
4 checks passed
SamuelDenani added a commit that referenced this pull request Oct 7, 2026
Brings in the platform-dependent test fix from #15 so this PR's gate can run
against a green base.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant