Audit test value, execution coverage, and brittle maintenance assertions

[shiny-code-bot]

Objective

Before the next broad maintenance or feature cycle, audit this repository's tests and test-related validation for behavioral value, execution coverage, brittleness, duplication, flake risk, and maintenance cost.

The goal is not fewer tests. The goal is a smaller and clearer set of tests only where evidence shows that current coverage is vestigial, redundant, overly specific, unexecuted, or coupled to implementation details instead of meaningful behavior and intentional external contracts.

Finish Line

All tests are intentionally executed and protect meaningful behavior or explicit contracts; brittle, redundant, orphaned, and vestigial coverage is removed, replaced, or justified.

Current Status

State: Parked for a future maintenance pass as of August 12, 2026.

A lightweight local filename inventory found approximately 6 paths whose names suggest tests, specs, fixtures, or test support. This is only a discovery hint, not a reliable test count and not evidence that cleanup is needed. Repositories with little or no meaningful test surface may close this issue as no-action after documenting that evidence.

This plan follows the baseline-first lesson from cbusillo/context-panel#577 and cbusillo/context-panel#581: test cleanup must not begin until every current test file is known, its actual execution path is verified, and previously orphaned tests have been run at least once.

Next action when promoted: record the immutable audit-start SHA; inventory test files, helpers, fixtures, and all local/CI/release invocations; execute every current test lane where safely possible; then classify evidence-backed candidates before changing assertions.

Scope

Included:

• Unit, integration, end-to-end, workflow, release, fixture, snapshot, and validation tests present in this repository.
• Test files that are not executed by any real local, CI, release, or runtime gate.
• Duplicate, tautological, implementation-coupled, exact-count/list/text, timing-sensitive, environment-sensitive, and low-diagnostic tests.
• Tests that pin dependency revisions, generated formatting, runner details, filenames, timestamps, or incidental inventory without protecting a real contract.
• Test runtime, flake behavior, failure diagnostics, fixture ownership, and lane boundaries.
• Dead tracked test support and validation artifacts after reference and history evidence.

Excluded:

• Test-count, line-count, or coverage-percentage reduction targets.
• Deleting tests merely because they are large, old, slow, or inconvenient.
• Weakening security, privacy, compatibility, migration, release, signing, schema, data-integrity, deployment, or recovery contracts.
• Adding tests just to increase coverage metrics.
• Product or architecture refactors not required to make a proven test contract deterministic and maintainable.

Acceptance Criteria

• The audit-start source SHA and relevant runner/environment identities are recorded.
• Every current test file and test-support fixture is inventoried.
• Every test file is assigned to an intentional executable lane, or its special execution requirement is documented.
• Every current test is executed at least once where safely possible; blockers are explicit rather than silently skipped.
• Runtime and outcome evidence is recorded at a useful file, suite, or group granularity without comparing unlike runners.
• Candidates are classified as valuable, replace, consolidate, remove, or accepted maintenance cost.
• Exact values are distinguished between incidental implementation details and intentional external contracts.
• Every removal or relaxation includes a behavioral counterexample, mutation-style proof, or historical regression evidence showing what the test protects and whether remaining coverage catches the defect.
• Timing-sensitive tests use deterministic coordination where practical.
• Source-text, workflow-text, snapshot, and exact-inventory checks are replaced with structured or behavioral checks when the exact representation is not the contract.
• Each implementation PR is limited to one test file or one coherent candidate class and remains independently revertible.
• Repository-defined local, CI, security, and release gates pass after each landing slice.
• The audit stops when every current file is classified and the evidence-backed backlog is disposed, or when two consecutive review slices produce no high-confidence improvement.

Decisions

• Complete execution mapping precedes judgment about test value.
• Prefer behavioral guarantees and true external contracts over implementation-shape assertions.
• Do not presume cleanup is necessary; a documented no-action outcome is valid.
• Do not optimize for test count, line count, or superficial CI-time claims.
• Preserve exact values when exactness is the real compatibility, security, release, or operational contract.
• Record accepted maintenance cost rather than extending the audit indefinitely.

Validation

Use this repository's own documented gates and active CI configuration. The audit should first establish which commands are authoritative rather than importing Context Panel's Swift/Python-specific commands. Add runner-specific timing or completeness evidence only when it is maintainable and useful.

Relationships

• Related cross-repository pattern: cbusillo/context-panel#577.
• Baseline-first reference: cbusillo/context-panel#581.

Open Questions

• Are all current tests actually executed by a maintained gate?
• Which historically noisy failures came from overly specific assertions rather than product regressions?
• Which exact values are true contracts and which merely mirror today's implementation?
• Does this repository have enough test surface to justify implementation work, or should this plan close as no-action?