Flatten all -L search paths into a single list of files

[Kobzol]

This is a pre-emptive optimization for the new Cargo build dir layout.

Before, Cargo usually used to pass rustc a single -L path to a directory containing potentially a large number of .rlib or other dependency paths. Rustc is optimized for this, and does a preprocessing step, where it preloads all files from this directory into a sorted vector, in which it then performs binary search to look for files having a given prefix and postfix.

With the new build dir layout, this changes, and Cargo will pass potentially many (even hundreds, if your crate has many dependencies) -L directories, where each directory will usually have only 1-2 files. This is not ideal for the current rustc search implementation, because the old preprocessing isn't really worth it when the -L directory typically only has a single file, and because rustc will do a lot of work per each -L directory in the lookup phase, including some quadratic (O(n^2)) work.

With the large-workspace stress test from rustc-perf, where the final binary has ~1000 total and ~500 direct dependencies:

• With the old build dir layout, rustc does 4274 calls to FilesIndex::query
• With the new build dir layout, it does 1938314 calls to FilesIndex::query. Not ideal :)

This PR modifies the logic so that instead of going through all -L directories one by one when we are looking for a specific dependency, we pre-gather all files from all passed -L directories at the start, and sort them all together. Then, in the lookup phase, instead of performing <number of -L dirs> * 4 (.rmeta, .rlib, etc.) searched, we only perform the 4 searches across all files. This gets rid of the O(n^2) behavior.

I thought about pre-filtering the files further, e.g. based on their kind, but that is not worth it in usual situations, where ~all of the deps will be .rlibs or .rmetas anyway. What might work is pre-filtering based on their actual names (we strip known prefixes from them and then search based on the names alone, in a hashmap or something?), but that can be a follow-up.

Below are perf. runs with rustc/cargo using the new build dir layout:

• Without this PR (this is what we would likely see in rustc-perf once we merge the Cargo submodule update that enables the new build dir layout by default)
• With this PR
• Diff

Note that this is not intended to be merged as-is, it is not cleaned up and it is not very relevant until Cargo switches to the new build dir layout anyway. But I wanted to get a vibe-check if you think that something like this might make sense, e.g. from @petrochenkov. The main change is in locator.rs, along with the search_paths2 function.

[rustbot]

r? @mati865

rustbot has assigned @mati865.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: compiler
• compiler expanded to 75 candidates
• Random selection from 21 candidates

[Kobzol]

Sorry, meant to open this as a draft, but missclicked.

r? @ghost

[rust-log-analyzer]

The job pr-check-2 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

[petrochenkov]

This generally makes sense to me.

Cargo will pass potentially many (even hundreds, if your crate has many dependencies) -L directories

Was this considered by cargo team when the new build dir layout was designed? Why wasn't it considered an issue?

[Kobzol]

I'm sure that it was considered, though I don't know if there were any alternatives. CC @ranger-ross @epage if you have more context.

Recently we identified some optimizations that can be made to reduce that effect, they are being landed now (rust-lang/cargo#17168).

[epage]

Splitting rlibs into unique directories is fairly fundamental to the design. The trivial solution from there is hundreds of -Ls. I thought we had talked about performance implications but I can't find the thread and might be getting mixed up. We had acknowledged this will make Cargo's verbose output longer but we've already done that before with [lints] and --check-cfg so it seems less important to be careful about that. Maybe there are things we can further do (e.g. create a one off directory with symlinks to all of the rlibs) but we hadn't seen any concerns found during testing until we got to bootstrap recently and it seemed like those were being addressed.

[Kobzol]

I thought about the mega-symlink-dir approach too :) Might be interesting to try, as that might also have some performance implications.

I hope that with rust-lang/cargo#17168 and this PR landing and rust-lang/cargo#17183 being resolved, it will become workable, but we'll have to see how the new build dir layout works in practice to find out.