Dependencies from a detected nested test project are not loadable

[Krastanov-agent]

Problem

A test item belongs to a package but uses a separate project in a nested test directory:

PackageA/
  Project.toml
  test/Project.toml
  test/special/Project.toml
  test/special/Manifest.toml
  test/special/test_aqua.jl

The nested manifest path-develops PackageA. JuliaWorkspaces detects test/special as the project for the test item. TestItemControllers then activates a temporary environment based on the package test/Project.toml. Dependencies declared only in test/special/Project.toml are not loadable.

For example, a test item containing using Aqua fails with:

ArgumentError: Package Aqua not found in current path.

This reproduces with TestItemControllers 1.11.0.

Expected behavior

After the nested project and manifest are selected, their direct dependencies should be available to the test item.

Current workaround

QuantumClifford PR #820 adds the nested project to JULIA_LOAD_PATH for each test process:

https://github.com/QuantumSavory/QuantumClifford.jl/blob/f74d0b7cc6b46b757733728b462cad57b1ec5e17/.buildkite/pipeline.yml#L29-L37

Suggested solution

When projectUri differs from packageUri, preserve the selected project dependencies when creating the temporary test environment. This could merge the selected project dependencies into the temporary project or otherwise keep the selected project loadable. Callers should not need to overwrite JULIA_LOAD_PATH.

[Krastanov]

This issue is discussed in more detail here: https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/23?u=krastanov

[davidanthoff]

Ok, for a while I also thought this is a bug, and I think my answers in the discourse thread were actually incorrect.

But now I think the behavior that is implemented is actually correct, i.e. it just mirrors how Pkg.test handles things, which is the goal for how we handle test env stuff.

The basic story is that the project/manifest in the special folder can pin down versions that are used in the test env, but it can't add additional deps beyond what the packages test env specifies (be that either via an [extra] section or a test/Project.toml).

I think I kind of lost track why the current design is a problem. Why not just add all test dependencies as regular test deps? I know there were various problems with activation timings and timeouts etc, but I think (hope) they are resolved now, so helpful this is a strategy that could work?

[Krastanov]

If I add JET.jl in test/Project.toml then PkgEval will constantly report my package as broken because JET frequently has issues even instantiating on nightly. Same for a bunch of other tools.

If I add CUDA.jl in test/Project.toml to just test a pkgext then all my test runners will download GB of libraries instead of just the CUDA-enabled runner (same for other GPU backends)

If I add Oscar.jl in test/Project.toml so that I can test the pgkext that depends on Oscar, then I can not run tests on Windows (Oscar does not install on Windows) even though the rest of my library supports Windows.

[Krastanov]

In ParallelTestRunner.jl I can solve that issue because its testsets are not "extracted away" from the surrounding environment setup, so stuff like the following works fine:

https://github.com/QuantumSavory/QuantumSavory.jl/blob/464fffa9932ace7d489fe7223f7fe0e1d5218357/test/runtests.jl#L51

[Krastanov]

Could you elaborate a bit. Say we are talking about Aqua. How should I structure the repo so that the "default" test runner does not try to install Aqua but there is a dedicated test runner that runs an environment in which Aqua is installed?

[davidanthoff]

Sorry, Claude is running amok, I didn't write that response. I'm going to delete it.