Skip to content

test: make false-passing negative controls fail on regression (#790) - #797

Draft
padak wants to merge 1 commit into
mainfrom
claude/issue-790-false-passing-tests
Draft

padak wants to merge 1 commit into
mainfrom
claude/issue-790-false-passing-tests

Conversation

@padak

@padak padak commented Sep 26, 2026

Copy link
Copy Markdown
Member

What / why

Part of Batch 1 (correctness first) from #790. Each rewritten test was proven to FAIL by temporarily breaking the guarded production code locally, running the test, and restoring.

  • test_file_locking.py::TestTryFlock::test_flock_skipped_when_no_fcntl - patches config_store.fcntl with a mock and asserts flock.assert_not_called(). Proof: disabling the if not _HAS_FCNTL: return guard now fails the test (before, the real flock on fd 42 raised OSError, which was suppressed, so it passed).
  • test_errors.py::test_no_duplicate_values -> test_no_aliased_members, plus @enum.unique on ErrorCode. The old test iterated the enum, which skips aliases, so it could never fail. The new one checks __members__ for aliases. Proof: removing @unique and adding a duplicate value fails it. With @unique in place, a duplicate fails at import.
  • test_server_permissions.py::...::test_app_without_an_engine_fails_closed - now pytest.raises(PermissionDeniedError, match="not built by create_app") instead of status_code != 200. Proof: removing the fail-closed branch in get_permission_engine now fails with AttributeError (before, the 500 from that crash also passed).
  • test_permissions_cli.py::...::test_set_rejected_without_confirmation - drops the mock of require_random_code_confirmation so the real non-TTY refusal runs, and asserts store.load().permissions is None. Proof: moving the save before the confirmation fails the test.
  • test_billing_cli.py::...::test_per_project_errors_surface_as_warnings_exit_0 - uses multi-letter aliases and asserts the full rendered warning line (Warning: Project 'beta': ...) plus the healthy row. Proof: skipping _emit_errors fails it (before, it only asserted that the letter "b" appeared in the output).
  • test_auto_update.py::...::test_sentinel_is_set_even_when_body_raises - asserts _AUTO_UPDATE_RAN is True after the crash and that a second call never reaches the body again. Proof: not setting the sentinel fails it.
  • test_search_service.py::TestResolveApiTypes::test_multiple_types_deduped - input now contains a repeat and two spellings of the same API type; it asserts the exact ordered result. Proof: removing the seen check fails it. (The issue body files this under "semantic layer", but the test lives in the search service.)
  • test_integration.py error-code guard self-tests (TestCheckErrorCodesGuard, TestErrorCodesDocCompleteness) - removed the integration marker. They are offline, and CI's -m "not integration" was skipping them. Paths are now anchored to the repo root. The planted-literal and enum-usage tests now call the script's real _collect_violations instead of a copy of its logic. Proof: breaking the scanner in scripts/check_error_codes.py now fails test_guard_catches_planted_literal (the old copy-of-logic version would still pass).

Not in this PR

  • The lane-1 bulk deletions and batches 2-6 are left for follow-ups.
  • test_e2e_lineage_deep.py::test_12 is untouched: it is an E2E test that needs live tokens.

How tested

make check is fully green: lint, format, ty, changelog check, error-code guard, 6902 passed / 15 skipped. No version bump and no changelog entry.

Refs #790

Rewrite negative controls that passed even with the guarded behavior
broken, add @enum.unique to ErrorCode, and move the offline
check_error_codes.py self-tests out of the `integration` marker so CI
actually runs them.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant