Ability to only run failures (and/or to run them first)

[nickrobinson251]

name tbd (perhaps fails_only, fails_first?)

potentially would be nice to add a short-hand for combining this with failfast #132

[nickrobinson251]

a problem with running only previously-failed tests, is that fixing these doesn't mean all tests now pass, since other tests may have begun to fail... so i'm tempted to think running previously-failed tests first is the more useful option, especially if also aborting the test run on the first failure (#132)

[nickrobinson251]

note: should actually distinguished between failed, passes, and not yet seen

[nickrobinson251]

pytest has these options:

  --lf, --last-failed   rerun only the tests that failed at the last run (or
                        all if none failed)
  --ff, --failed-first  run all tests, but run the last failures first.
                        This may re-order tests and thus lead to repeated
                        fixture setup/teardown.
  --nf, --new-first     run tests from new files first, then the rest of the
                        tests sorted by file mtime

https://docs.pytest.org/en/7.1.x/reference/reference.html#id89

see also: https://docs.pytest.org/en/stable/how-to/cache.html

[nickrobinson251]

AFAICT go test doesn't have this option

To me it looks like the only "ordering of tests" option there is -shuffle

	-shuffle off,on,N
	    Randomize the execution order of tests and benchmarks.
	    It is off by default. If -shuffle is set to on, then it will seed
	    the randomizer using the system clock. If -shuffle is set to an
	    integer N, then N will be used as the seed value. In both cases,
	    the seed will be reported for reproducibility.

https://github.com/golang/go/blob/dd8276657f2244193d399f01941ee1d76a79529d/src/cmd/go/internal/test/test.go#L322-L327

[nickrobinson251]

Likewise Rust's cargo test doesn't have such an option AFAICT
https://doc.rust-lang.org/cargo/commands/cargo-test.html

[nickrobinson251]

Ruby's rspec has --only-failures (and --next-failure)
https://rspec.info/features/3-13/rspec-core/command-line/only-failures/

The `--only-failures` option filters what examples are run so that only those that failed the last time they ran are executed.
There's also a `--next-failure` option, which is shorthand for `--only-failures --fail-fast --order defined`. It allows you to repeatedly focus on just one of the currently failing examples, then move on to the next failure, etc.

https://github.com/rspec/rspec/blob/6719497bf88e3d2dea1a93351baf48c9be654adc/rspec-core/features/command_line/only_failures.feature#L1-L5

[nickrobinson251]

Javascript's jest has this via --onlyFailures

Run tests that failed in the previous execution.

https://jestjs.io/docs/cli#--onlyfailures