Portability fixes: guard Linux-only APIs for FreeBSD/non-Linux builds#664
Open
GenericRikka wants to merge 1 commit into
Open
Portability fixes: guard Linux-only APIs for FreeBSD/non-Linux builds#664GenericRikka wants to merge 1 commit into
GenericRikka wants to merge 1 commit into
Conversation
Author
|
Hi, This change is not meant to alter Linux behavior; the goal is to make Linux-specific assumptions explicit by guarding Linux-only APIs/headers and using small fallbacks where needed so the code builds cleanly on non-Linux platforms as well. Linux fast paths remain in place. I’ve retitled the PR to better reflect that broader portability focus. From my side, the patchset is intended to be minimal and upstreamable, and I’d be happy to split it further if that would make review easier. When you have time, I’d appreciate another look. |
freebsd-git
pushed a commit
to freebsd/freebsd-ports
that referenced
this pull request
Apr 15, 2026
… scalability on large filesystems FileUtils provides a library (libmfu) and a suite of MPI-parallel file utilities such as dcp (parallel copy), dcmp (compare), ddup (duplicate finder), dfind, dtar/dbz2, dchmod, and more. Designed for high scalability on large filesystems. https://hpc.github.io/mpifileutils/ https://github.com/hpc/mpifileutils/ Pull request to upstream repo with local patches: hpc/mpifileutils#664 PR: 291679 Sponsored by: UNIS Labs Co-authored-by: Vladimir Druzenko <vvd@FreeBSD.org>
gahr
pushed a commit
to gahr/freebsd-ports
that referenced
this pull request
Apr 17, 2026
… scalability on large filesystems FileUtils provides a library (libmfu) and a suite of MPI-parallel file utilities such as dcp (parallel copy), dcmp (compare), ddup (duplicate finder), dfind, dtar/dbz2, dchmod, and more. Designed for high scalability on large filesystems. https://hpc.github.io/mpifileutils/ https://github.com/hpc/mpifileutils/ Pull request to upstream repo with local patches: hpc/mpifileutils#664 PR: 291679 Sponsored by: UNIS Labs Co-authored-by: Vladimir Druzenko <vvd@FreeBSD.org>
Author
|
I've also filed issue #676 as a tracking issue. |
063ceea to
3d95cb4
Compare
These are the core FreeBSD portability changes extracted from the FreeBSD ports tree work. The port currently is still under review. Port PR: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=291679 Port review: https://reviews.freebsd.org/D54230 The Patches: - Add FreeBSD compatibility for stat64/lstat64 by mapping them to stat/lstat (which are 64 bit by default on FreeBSD) - Include missing system headers in the path/parameter helper so struct stat and related macros are consistently available - Fix MFU_LOG(... "errno=%d %s" ...) callsites in mfu_util.c to actually pass both errno and strerrno(errno) (prevents format/argument mismatch and undefined behavior) - Include <fcntl.h> in mfu_util.c for file/flag-related definitions used by the code on non-Linux systems Signed-off-by: Rikka Göring <rikka.goering@outlook.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mpifileutils currently assumes a Linux build environment throughout: it
includes Linux-specific kernel headers unconditionally, uses
sysinfo(2),truncate64(), andSYS_getdentsdirectly, and guardsxattr code only against the
DCOPY_USE_XATTRSbuild flag rather thanplatform availability.
This PR makes mpifileutils build on FreeBSD by guarding Linux-specific
headers and APIs behind
#ifdef __linux__and providing smallcompatibility shims where needed. All changes are intentionally minimal
— Linux behavior and all Linux-only fast paths (fiemap, xattr,
sysinfo-based compressor) remain completely unchanged.
Background
These changes were developed while packaging mpifileutils for the
FreeBSD ports tree (
sysutils/mpifileutils). The port landed in April2026 (ports commit
5f283cddd7b7,
co-authored with Vladimir Druzenko / vvd@FreeBSD.org) and has been
carrying these as local patches since then. This PR upstreams them.
What the patch actually does
Linux-only kernel headers
<linux/fs.h>,<linux/fiemap.h>,<linux/limits.h>, and<sys/sysinfo.h>do not exist on FreeBSD and cause hard build failuresif included unconditionally. Each is now guarded with
#ifdef __linux__and a portable alternative is provided where one exists:
<linux/limits.h>→<limits.h>(POSIX) indcmp.canddsync.c<sys/vfs.h>→<sys/param.h>+<sys/mount.h>on FreeBSD inmfu_bz2.c,mfu_util.c, anddbcast.csysinfo-dependent bz2 compressor
mfu_compress_bz2_libcircle()usessysinfo(2)(Linux-only) todetermine available memory for wave sizing. On non-Linux, the entire
sysinfo/libcircle-dependent implementation is compiled out and the
function forwards to
mfu_compress_bz2_static(), which is portable.The decompressor side has no sysinfo dependency so only the header guard
is needed there.
stat64 / lstat64
FreeBSD's libc provides only
stat/lstat; 64-bit file sizes areimplicit.
mfu_io.handmfu_io.cmapstat64/lstat64to theirnon-suffixed equivalents under
#ifdef __FreeBSD__.xattr support
xattr APIs differ between Linux and FreeBSD. Rather than attempting a
cross-platform xattr implementation, xattr support is simply disabled on
non-Linux:
mfu_flist.cundefinesDCOPY_USE_XATTRSon non-Linux before anyxattr-dependent code is reached, so existing
#if DCOPY_USE_XATTRSguards in the file continue to work correctly without modification.
mfu_flist.handdcp1/common.hgate<sys/xattr.h>ondefined(__linux__) && DCOPY_USE_XATTRS.mfu_flist_copy.caddsENOTSUP-returning stubs forllistxattr/lgetxattr/lsetxattrunder#if DCOPY_USE_XATTRS && !defined(__linux__), somfu_copy_xattrs()degrades gracefully (it already handles
ENOTSUP).fiemap sparse copy
mfu_fiemap_get_extents()and the fiemap ioctl copy path inmfu_copy_file_extents()are gated on#ifdef __linux__. On non-Linuxthe function falls through to the normal copy path via
goto fail_normal_copy.Directory walking
walk_getdents_process_dir()called the rawSYS_getdentssyscalldirectly. On FreeBSD the libc
getdents(2)wrapper is used instead.Miscellaneous portability fixes
O_LARGEFILEfallback macro (implicit on FreeBSD) inmfu_bz2_static.cHOST_NAME_MAXfallback indbcast.c(FreeBSD usesMAXHOSTNAMELEN)POSIX_FADV_SEQUENTIALfallback inmfu_util.cPATH_MAXfallback indsync.ctruncate64()→mfu_truncate()indcp1/cleanup.c<sys/types.h>/<sys/stat.h>added tomfu_param_path.cfor FreeBSDIncidental bug fixes found during porting
mfu_flist_walk.c:MFU_LOG(MFU_LOG_INFO, "skip %s")missingnameargument (format/argument mismatch, undefined behavior)
mfu_util.c:MFU_LOGcall passingstrerror(errno)for a%dformat specifier;
errnoargument was missingdbcast.c: threeMFU_LOGcalls with missing%sformat specifiersor missing
(unsigned long long)casts on%lluargumentsPlatform impact
enabled. The diff to existing Linux-compiled code is zero.
xattr preservation are disabled pending native implementations.
Testing
poudriere testport,stage-qa,plist checks) on FreeBSD 14.3 and 15.0 amd64.
dcp,dcmp,dsync,dbcast,dbz2on small test trees.
Note on
#ifdef __linux__vs cmake feature detectionI've used
#ifdef __linux__rather than cmakecheck_include_file()feature gates throughout, since the guarded headers (
<linux/fs.h>,<sys/sysinfo.h>, etc.) are Linux kernel interfaces with nocross-platform equivalent — feature detection wouldn't make the intent
clearer. Happy to convert to cmake-style guards in a follow-up if
preferred.
References