Repository navigation
Description
VIP's value is the claim that a deployment was validated. The most dangerous failure mode for a validation tool is not a red run — it is a green run that proved nothing. Today those two outcomes are indistinguishable: a check that was legitimately N/A and a check that could not be executed both land in the same skipped bucket, and neither affects the exit code.
Evidence that this is a class, not a one-off
#596 (configured product never authenticates, every test skips, exit 0), #602 (test_workbench_login skips under SSO, so no lane verifies a Workbench login at all), #606 (test_connect_login_ui can never fail — its only assertion is satisfied before credentials are submitted), #550 and #546 (suites that cannot run in CI because [runtimes] is unpopulated and nothing terminates TLS). Each has been triaged individually; nothing prevents the next one.
Structural evidence
src/vip_tests contains 160 pytest.skip() call sites. ReportData (src/vip/reporting.py:90) counts skips as a single undifferentiated total, and the only classification that exists is na_version. Exit status is pytest's raw exitstatus (src/vip/plugin.py:1266), so a run in which every meaningful check skipped exits 0 and renders a report with zero failures. To someone reading that report as an audit artifact, "we did not check" and "we checked and it is fine" look identical.
Proposal
Introduce a third outcome, unproven, sitting between passed and failed.
- Classify at the skip site.
vip.attest.not_applicable(reason)marks a skip as deliberate and non-fatal (product not configured, tier lacks the feature, IDE not installed) and keeps it out of the exit code;vip.attest.unproven(reason)marks one VIP was asked to run and could not. A barepytest.skip()keeps thenot_applicablemeaning, so sites convert deliberately rather than by flag day — defaulting to unproven would make "product not configured" fatal, which is exactly the case fix(cli): vip verify exits 0 when a configured product never authenticates #596's note on the reverse risk says must stay non-fatal. - Surface it everywhere the outcome is reported: report counters and badges, JUnit (
unprovenas a distinct message), SARIF (warning, betweennoneanderror). - Make it affect the exit code, behind
--allow-unprovenfor anyone who needs the old behaviour. - Split the report's top line three ways — Verified / Failed / Not verified — with "not verified" broken into "N/A for this version" and "could not run here".
The tradeoff, stated plainly
This makes VIP louder on day one. Existing lanes that quietly skipped will start failing, and the first run against a real deployment will surface gaps nobody has had to look at. That is the intended effect, but it is a real behaviour change and the escape hatch exists for that reason.
Why it is worth it
A tool that reports "31 of your 47 configured capabilities were verified, here are the 16 I could not reach and why" is more credible as an IQ/OQ artifact than one that reports 47/47 green with the interesting half silently skipped. #596 already names the required distinction — "separate 'nothing to do here' from 'we could not do what was asked'" — this issue supplies the mechanism, and closes #596 as its first consumer.
Scope note
Landing in slices: this issue covers the unproven outcome, its plumbing through the report formats, and the exit-code contract. Retriaging the existing 160 skip sites is follow-up work, as is the capability-attestation layer (asserting that a declared capability had a check that actually exercised it) and a mutation meta-test that proves each scenario can fail.
Activity
ian-flores commentedon Aug 31, 2026
Slice 1 shipped in #617 (merged): the
unprovenoutcome,vip.attest, plumbing through report/JUnit/SARIF/PDF, and the exit-code contract (6, with--allow-unproven). #596 is closed by it.Two things landed differently from this issue as filed. The polarity is opt-in rather than default, for the reason now in Proposal item 1. And
summary_statusgained anUNPROVENverdict, so a report where nothing was verified no longer headlinesPASS— that fell out of the collision with #618's report refactor and is arguably the most valuable single line of the change.Remaining, and what this issue now tracks: retriaging the ~160 existing
pytest.skip()sites; the capability-attestation layer (asserting a declared capability had a check that actually exercised it); and the mutation meta-test that proves each scenario can fail, which is what would have caught #606.Worth recording that the exit-6 path has not yet run against a real deployment — every smoke lane passes because authentication succeeds there. #263's SAML lane is where it would first be exercised for real.
fix(security): report TLS verify failure as attest.unproven
fix(workbench): stop masking VS Code load failures as a skip
[-]feat(reporting): distinguish "not applicable" from "could not verify" and make unproven checks fail the run[/-][+][Feature] Distinguish "not applicable" from "could not verify" and fail the run on unproven checks[/+]