Upgrade NUnit to v4; misc test improvements, #1001

[paulirwin]

• You've read the Contributor Guide and Code of Conduct.
• You've included unit or integration tests for your change, where applicable.
• You've included inline docs for your change, where applicable.
• There's an open issue for the PR that you are making. If you'd like to propose a change, please open an issue to discuss the change or find an existing issue.

Upgrade NUnit to v4; misc test improvements.

Fixes #1001

Description

This PR upgrades NUnit to v4. For more details on some of the migration, see the docs here: https://docs.nunit.org/articles/nunit/release-notes/Nunit4.0-MigrationGuide.html

NUnit 4 dropped support for netstandard2.0; thus, this PR drops support for it from the TestFramework project. This only really would affect users who need to create an intermediate unit testing library (note: not a unit test project) that targets netstandard2.0 and references TestFramework. We expect this to not have a wide impact given current relatively low usage of TestFramework on NuGet (~100 downloads/mo). But if this is you, there is a simple workaround: either choose a single supported target (i.e. net8.0 or net462), or multi-target (i.e. <TargetFrameworks>net462;net8.0</TargetFrameworks> in your project). Getting Lucene.NET on NUnit 4 will be more impactful in the long run and less restrictive to users than having to support a limited set of netstandard2.0 runtimes that are still in use (i.e. maybe Mono and some versions of Unity), and again this is just for unit tests.

My first pass at this added a global using alias for Assert to point it to our Lucene.Net.TestFramework.Assert class, but that created way too many files changed in this PR, and it can be added later. So while there are some namespace cleanups here, I tried to minimize the amount of files changed since the other stuff is more important. Also, this upgrade exposed several places where we were not using our Assert class, so that has been fixed in several places.

One notable breaking change in NUnit 4.5 and later is that it now throws at runtime if you try to use the TimeoutAttribute on modern .NET. This was previously documented as not working on .NET 5 and later, but now it is an exception. To solve this at the class/test level, it was replaced with CancelAfterAttribute, which will trigger a CancellationToken parameter if the test runs longer than the configured value. So CancellationToken support was added to most of these tests and threaded through to calls where it made sense to do so. Only one test did not have an obvious seam since ForceMerge is not currently cancelable.

For TimeoutAttribute at the assembly level, there is not a good replacement. So I've removed it, since the existing --blame-hang-timeout likely already covers this for both .NET Framework and modern .NET.

Assert saw many documentation and correctness improvements in this PR; too many to mention. Notably, it switched to XML doc comments. Additionally, many unit tests were added for the NUnit-derived attributes.

The private/internal ArgDisplayNames property in NUnit changed to be an auto-prop from a declared field, so the reflection code that pokes into that has been updated to get the internal property instead of the private field. Unit tests were also added for this functionality to help detect and prevent regressions. Although if this regresses with a newer version of NUnit, the code should safely fall back to just using the ToString representation of the class fixture arguments, if anyone is using that (note that we are not using this functionality in our code).

Finally, a concurrency bug in TestRAMDirectory was fixed (which might have been surfaced by the NUnit 4 upgrade), which has the benefit of better matching the upstream Java code anyways. The root case was two different runs of the test could be writing to the folder at the same time.

AI: This was largely done by hand, but some repetitive parts (like removing usings) and some unit tests were done with the help of Claude Code.

[paulirwin]

Marking as draft until NUnit 4.6 is released with my merged PR (nunit/nunit#5236) included, to remove the reflection for ArgDisplayNames.

However, the rest is still able to be reviewed. cc @NightOwl888

[paulirwin]

NUnit 4.6 is now out with my PR fix included, so I'll pick this up again today.

[paulirwin]

@NightOwl888 Thanks again for the incredibly thorough writeup and discussion above. I went back through the discussion carefully, and I want to acknowledge up front that your concerns were legitimate, and the core argument was right. The parts of our test framework that hook directly into NUnit are the parts most at risk from an upgrade, and we did not have adequate test coverage to protect that.

So, I went and closed those test coverage gaps to protect this and future NUnit upgrades. If these tests ever fail, we'll know that the version of NUnit is not compatible with our current test framework.

Below are your concerns, mapped to test coverage.

---

The seeding hack

This was the real risk area that got the most attention. The hack is to seed every test reproducibly during discovery, and the requirements are: no filters when seeding, stable ordering, a single shared randomizer, and correct error routing. These are now covered by:

• LuceneRandomSeedInitializerTests.GenerateRandomSeedsIsReproducibleFromTheInitialSeed re-derives the exact Randomizer draw sequence from the initial seed and asserts every test matches. If discovery ever stops producing deterministic seeds, this fails.
• LuceneRandomSeedInitializerTests.GenerateRandomSeedsAttachesContextToFixtureAndEveryChild proves a single shared initial seed flows to the fixture and every child, with distinct per-test seeds.
• LuceneTestFixtureFilterSeedingTests.FilteringOutMethodsDoesNotChangeSurvivingTestSeeds is the one covering the "ensure there are no filters on the tests" requirement. It builds the fixture twice through the real IFixtureBuilder2 path, once unfiltered and once with an IPreFilter that drops a method, and proves the surviving tests keep the exact seed they'd have had unfiltered. When I temporarily made filtered methods stop advancing the Randomizer, it fails loudly.
• RandomizedContextTests (6 tests) and RepeatAttributeTests.RepeatRegeneratesTestSeedDeterministicallyFromRandomSeed cover seed reset across iterations.

Exception mapping

This one is about our catch blocks depending on the NUnit exception types being mapped into ExceptionExtensions, and a rename in a future NUnit release would silently null them out. TestLuceneTestFrameworkInitializer now asserts every mapping directly, including the internal InvalidPlatformException resolved by name, so a future upgrade that moves any of them fails a test instead of quietly breaking Java exception semantics.

NOTE: I had to fix a x86-related bug here in my latest commit. The newer NUnit3TestAdapter adds in a type-load handler that can incorrectly load reference assemblies, so it was failing on modern .NET on Windows x86 when trying to try-load those runtime exception types for .NET Framework when other reference assemblies like the Windows desktop pack are present. I adjusted this to handle this case and return null. It does not materially impact any tests or have any impact on the validity of testing production code, since it works on .NET Framework where those types exist. And since those types are unlikely to be reintroduced into modern .NET, the risk is negligible. I filed the bug upstream with NUnit: nunit/nunit-console#1835 and we can remove our workaround if desired once/if that is fixed upstream.

Lifecycle ordering

LuceneTestCaseLifecycleTests.LifecycleMethodsFireInExpectedOrder pins OneTimeSetUp -> SetUp/TearDown per test -> OneTimeTearDown, the ordering our DI/initialization relies on.

NUnit messaging / failure state

LuceneTestCommandMessagingTests.DelegatingCommandRecordsThrownException exercises the RecordException tap, so if a future NUnit changes how results are recorded we'll catch it.

Custom attributes

LuceneAttributeTests, TestFixtureAttributeTests, and the existing RepeatAttributeTests now cover Nightly/Weekly/Slow/AwaitsFix, the TestFixtureAttribute properties, and Repeat.

---

On the other points discussed:

I don't believe a full RandomizedTesting port is a prerequisite here. I agree that it's the right long-term direction, and nothing in this PR works against that later on. But coupling the NUnit 4 upgrade to that port turns it into a major effort that could cause this to miss our release window or delay the release further. So I took "Option 4" and added test coverage instead, to protect against any issues.

I believe I demonstrated earlier in the discussion that --blame-hang-timeout is adequate for our TimeoutAttribute replacement needs, including on .NET Framework. And we can always add back conditional compilation of the TimeoutAttribute if we ever see this being insufficient.

I acknowledge and greatly appreciate the work taken to date (mostly by you) to get our tests working so well and being so stable. I hope my efforts here demonstrate that I take that seriously, and I've added long-desired test coverage to protect this work from future disruption. These changes will not only allow us to upgrade now, and ship an up-to-date dependency for TestFramework to our users, but also will enable us to stay up-to-date (even with Renovate automatically) safely.