Skip to content

Add a golden/regression test suite and CI#18

Closed
ssiddhantsharma wants to merge 1 commit into
whitehead:masterfrom
ssiddhantsharma:add-golden-tests-ci
Closed

Add a golden/regression test suite and CI#18
ssiddhantsharma wants to merge 1 commit into
whitehead:masterfrom
ssiddhantsharma:add-golden-tests-ci

Conversation

@ssiddhantsharma

@ssiddhantsharma ssiddhantsharma commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Hi maintainers, I use PLAAC in my routine workflows and wanted to add automated tests for the PLAAC CLI, plus CI to run them.

The scoring logic in cli/src/plaac.java currently has no automated test guarding it, so there is no safety net against an edit unintentionally changing results. PLAAC’s output is deterministic, which makes byte-for-byte regression testing straightforward.

What this adds (pure addition — no changes to plaac.java)

  • cli/tests/run_tests.sh — plain bash + java, no test-framework dependency. Two checks:
    1. golden: runs the bundled cli/example/four_classic_prions.fasta at -a 1 -c 60 -d and compares the full output byte-for-byte against a committed expected file. This locks the scored output and the built-in frequency/column tables (from which the web _plaac_headers.haml is generated).
    2. recovery: asserts the four classic yeast prions (Sup35, Ure2, Rnq1, Mot3) each get a called PrLD (PRDlen > 0), a semantic check that survives a reviewed, intentional regeneration of the golden file.
  • cli/tests/expected/four_classic_prions.a1.c60.txt — the golden file.
  • cli/tests/regenerate_golden.sh — regenerates the golden for reviewed numeric changes.
  • cli/tests/README.md — docs.
  • .github/workflows/ci.yml — builds plaac.jar and runs the suite on push/PR (JDK 17), and uploads the actual output as an artifact.

Reproducibility

PLAAC scores use Math.log/Math.exp (platform intrinsics), so I verified the output is byte-identical across macOS/arm64 and Linux/x64 (Temurin 17) before committing; the harness also pins the JVM locale so number formatting is host-independent. CI is green.

Notes

  • No behavior changes; nothing outside cli/tests/ and .github/ is touched.
  • Happy to follow up separately with CLI input-validation hardening (friendlier errors for non-numeric args, etc.) if that would be useful.

Adds the first automated tests for the PLAAC CLI, which previously had
none. PLAAC's scoring is deterministic (Viterbi parse over fixed
frequency tables; no randomness, timestamps, or host paths in the
output), so its output can be guarded byte-for-byte.

cli/tests/run_tests.sh (plain bash + java, no test-framework dependency):
  1. golden   -- full output of the bundled four_classic_prions.fasta at
                 -a 1 -c 60 -d is compared byte-for-byte against a
                 committed expected file. Locks the scored output and the
                 built-in frequency/column tables (from which the web
                 HAML is generated) against unintended changes to the
                 ~5k-line plaac.java.
  2. recovery -- Sup35p/Ure2p/Rnq1p/Mot3p each get a called PrLD
                 (PRDlen > 0); a semantic check that survives a reviewed
                 regeneration of the golden file.

The harness pins the JVM locale so number formatting is reproducible on
CI hosts with a comma-decimal default locale. tests/regenerate_golden.sh
regenerates the golden file for reviewed, intentional numeric changes.

.github/workflows/ci.yml builds plaac.jar and runs the suite on push/PR,
making 'still builds and still produces correct scores' automatic.
@ssiddhantsharma

ssiddhantsharma commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for maintaining PLAAC! After opening this I saw that the devel branch already has a regression test cli-unit-test.yml builds the jar and diffs MOT3.fasta output against a gold file on a Java 11/17 matrix, so this PR largely duplicates work you have already done. Closing it to avoid the redundancy.

I have opened #19 against devel with the part that is not already there: command-line input validation (a friendly error + non-zero exit for non-numeric numeric args instead of a raw stack trace, unknown-option diagnostics moved off stdout so they no longer land in the TSV, and a warning when a value-taking flag is given no value). It keeps the MOT3 gold output byte-identical and leaves the -V / versioning work untouched. Happy to adjust if you would prefer a different approach.

@ssiddhantsharma
ssiddhantsharma deleted the add-golden-tests-ci branch July 26, 2026 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant