std: use readdir on nearly all UNIX platforms

[joboet]

POSIX.1-2024 formalised what was already guaranteed by a lot of implementations and required readdir to be thread-safe as long as an individual DIR* is not accessed concurrently (which ReadDir::next ensures by taking a mutable reference). But since our read_dir implementation predates that standard, we currently only utilise readdir on the platforms that guarantee thread-safety in their documentation. On other implementations – notably macOS – we use the readdir_r function, which was always required to be thread-safe but is problematic because it cannot handle directory entries with names longer than NAME_MAX.

However, even the first POSIX issue, POSIX.1-1994, specified that the data in the returned dirent

is not overwritten by another call to readdir() on a different directory stream.

and that guarantee together with the requirement that the underlying syscalls need to be thread-safe already because of readdir_r result in readdir being thread-safe on nearly all implementations, even if they predate POSIX.1-2024. Given the now formalised guarantee I think it safe to assume that currently thread-safe implementations will not be changed in a way that violates thread-safety.

CC T-libs, do you agree?
@rustbot label +I-libs-nominated

I thus looked at the readdir implementation of all the UNIXes currently utilising readdir_r to check for thread-safety:

• Apple platforms, starting with Mac OS X 10.0. While the original implementation did not use any synchronisation, it only modifies data of the DIR* it acts on.
• Cygwin: same thing.
• Dragonfly uses locks.
• L4Re: only modifies the passed DIR*.
• LynxOS: closed-source, the documentation says that readdir is not reentrant. CC @rfatykhov-lynx
• Managarm: only modifies the passed DIR*.
• NetBSD: either uses locks depending on configuration, or only modifies the passed DIR* .
• OpenBSD: uses locks.
• Unikraft
• VxWorks: closed-source, the documentation does not make any reference to thread-safety. CC @biabbas @hax0kartik
• QNX: the documentation gives the same guarantee as POSIX.1-2024.
• NuttX: only modifies the passed DIR*.
• 3DS: the underlying directory read implementation in libctru is thread-safe
• RTEMS: I'm unable to find the implementation. CC @thesummer
• QuRT: CC @androm3da
• ESP-IDF: readdir is not thread-safe (at least on FAT) due to caching file metadata in the VFS context without locks. opendir does the same thing, too?! And the cache is never invalidated, even when the file is deleted?! Honestly, this is just broken... CC @ivmarkov @MabezDev @SergioGasquez
• Emscripten: derived from musl, only modifies the current DIR*.

On the implementations where I couldn't confirm thread-safety ReadDir will still use readdir_r, but I've changed the code so that this edge-case is limited to ReadDir::next. All other platforms now use readdir.

[rustbot]

miri is developed in its own repository. If possible, consider making this change to rust-lang/miri instead.

cc @rust-lang/miri

[rustbot]

r? @JohnTitor

rustbot has assigned @JohnTitor.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: @ChrisDenton, libs
• @ChrisDenton, libs expanded to 13 candidates
• Random selection from 7 candidates

[jonathanpallant]

@joboet - you've linked to VxWorks docs under the QNX entry?

https://www.qnx.com/developers/docs/8.0/com.qnx.doc.neutrino.lib_ref/topic/r/readdir.html says:

The readdir() function is safe to use in multithreaded environments as long as the directory stream, DIR *dirp, isn't shared between threads.

https://www.qnx.com/developers/docs/7.1/com.qnx.doc.neutrino.lib_ref/topic/r/readdir.html says the same thing

[joboet]

Whoops, I got the RTOSs confused 😉 ... thanks for the link!

[RalfJung]

Miri changes LGTM.

At some point we can probably remove readdir_r from macOS then given that it is generally considered a deprecated API, the only reason we support it is because std used it.

View changes since this review

[ivmarkov]

• ESP-IDF: readdir is not thread-safe (at least on FAT) due to caching file metadata in the VFS context without locks. opendir does the same thing, too?! And the cache is never invalidated, even when the file is deleted?! Honestly, this is just broken... CC @ivmarkov @MabezDev @SergioGasquez

@joboet I completely agree with your analysis that FAT readdir in ESP-IDF V5.4.0+ is completely broken (LittleFS and Spiffs seem OK).

I've filed the issue upstream.
In the meantime, and as per the PR, ESP-IDF should stay on readdir_r.

Thanks a lot for spotting this!

EDIT: I haven't put an effort to actually reproduce it on real hardware yet; only static analysis (confirmed by Fable though). So still a slim chance we might be missing something. Very slim. :-)

[nia-e]

Per today's t-libs meeting, we like this, +1. The possibility was also raised for those other platforms of invoking the underlying syscall directly if they're stable, but that's probably a question for a different PR. Thanks ^^

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.