Skip to content

Portability fixes: guard Linux-only APIs for FreeBSD/non-Linux builds#664

Open
GenericRikka wants to merge 1 commit into
hpc:mainfrom
GenericRikka:main
Open

Portability fixes: guard Linux-only APIs for FreeBSD/non-Linux builds#664
GenericRikka wants to merge 1 commit into
hpc:mainfrom
GenericRikka:main

Conversation

@GenericRikka
Copy link
Copy Markdown

@GenericRikka GenericRikka commented Dec 22, 2025

Summary

mpifileutils currently assumes a Linux build environment throughout: it
includes Linux-specific kernel headers unconditionally, uses
sysinfo(2), truncate64(), and SYS_getdents directly, and guards
xattr code only against the DCOPY_USE_XATTRS build flag rather than
platform availability.

This PR makes mpifileutils build on FreeBSD by guarding Linux-specific
headers and APIs behind #ifdef __linux__ and providing small
compatibility shims where needed. All changes are intentionally minimal
— Linux behavior and all Linux-only fast paths (fiemap, xattr,
sysinfo-based compressor) remain completely unchanged.

Background

These changes were developed while packaging mpifileutils for the
FreeBSD ports tree (sysutils/mpifileutils). The port landed in April
2026 (ports commit
5f283cddd7b7,
co-authored with Vladimir Druzenko / vvd@FreeBSD.org) and has been
carrying these as local patches since then. This PR upstreams them.

What the patch actually does

Linux-only kernel headers

<linux/fs.h>, <linux/fiemap.h>, <linux/limits.h>, and
<sys/sysinfo.h> do not exist on FreeBSD and cause hard build failures
if included unconditionally. Each is now guarded with #ifdef __linux__
and a portable alternative is provided where one exists:

  • <linux/limits.h><limits.h> (POSIX) in dcmp.c and dsync.c
  • <sys/vfs.h><sys/param.h> + <sys/mount.h> on FreeBSD in
    mfu_bz2.c, mfu_util.c, and dbcast.c

sysinfo-dependent bz2 compressor

mfu_compress_bz2_libcircle() uses sysinfo(2) (Linux-only) to
determine available memory for wave sizing. On non-Linux, the entire
sysinfo/libcircle-dependent implementation is compiled out and the
function forwards to mfu_compress_bz2_static(), which is portable.
The decompressor side has no sysinfo dependency so only the header guard
is needed there.

stat64 / lstat64

FreeBSD's libc provides only stat/lstat; 64-bit file sizes are
implicit. mfu_io.h and mfu_io.c map stat64/lstat64 to their
non-suffixed equivalents under #ifdef __FreeBSD__.

xattr support

xattr APIs differ between Linux and FreeBSD. Rather than attempting a
cross-platform xattr implementation, xattr support is simply disabled on
non-Linux:

  • mfu_flist.c undefines DCOPY_USE_XATTRS on non-Linux before any
    xattr-dependent code is reached, so existing #if DCOPY_USE_XATTRS
    guards in the file continue to work correctly without modification.
  • mfu_flist.h and dcp1/common.h gate <sys/xattr.h> on
    defined(__linux__) && DCOPY_USE_XATTRS.
  • mfu_flist_copy.c adds ENOTSUP-returning stubs for
    llistxattr/lgetxattr/lsetxattr under
    #if DCOPY_USE_XATTRS && !defined(__linux__), so mfu_copy_xattrs()
    degrades gracefully (it already handles ENOTSUP).

fiemap sparse copy

mfu_fiemap_get_extents() and the fiemap ioctl copy path in
mfu_copy_file_extents() are gated on #ifdef __linux__. On non-Linux
the function falls through to the normal copy path via
goto fail_normal_copy.

Directory walking

walk_getdents_process_dir() called the raw SYS_getdents syscall
directly. On FreeBSD the libc getdents(2) wrapper is used instead.

Miscellaneous portability fixes

  • O_LARGEFILE fallback macro (implicit on FreeBSD) in mfu_bz2_static.c
  • HOST_NAME_MAX fallback in dbcast.c (FreeBSD uses MAXHOSTNAMELEN)
  • POSIX_FADV_SEQUENTIAL fallback in mfu_util.c
  • PATH_MAX fallback in dsync.c
  • truncate64()mfu_truncate() in dcp1/cleanup.c
  • <sys/types.h> / <sys/stat.h> added to mfu_param_path.c for FreeBSD

Incidental bug fixes found during porting

  • mfu_flist_walk.c: MFU_LOG(MFU_LOG_INFO, "skip %s") missing name
    argument (format/argument mismatch, undefined behavior)
  • mfu_util.c: MFU_LOG call passing strerror(errno) for a %d
    format specifier; errno argument was missing
  • dbcast.c: three MFU_LOG calls with missing %s format specifiers
    or missing (unsigned long long) casts on %llu arguments

Platform impact

  • Linux: no behavioral change. All Linux-only fast paths remain
    enabled. The diff to existing Linux-compiled code is zero.
  • FreeBSD / non-Linux: builds cleanly; fiemap sparse copy and Linux
    xattr preservation are disabled pending native implementations.

Testing

  • Built via FreeBSD ports tooling (poudriere testport, stage-qa,
    plist checks) on FreeBSD 14.3 and 15.0 amd64.
  • Basic runtime smoke tests: dcp, dcmp, dsync, dbcast, dbz2
    on small test trees.

Note on #ifdef __linux__ vs cmake feature detection

I've used #ifdef __linux__ rather than cmake check_include_file()
feature gates throughout, since the guarded headers (<linux/fs.h>,
<sys/sysinfo.h>, etc.) are Linux kernel interfaces with no
cross-platform equivalent — feature detection wouldn't make the intent
clearer. Happy to convert to cmake-style guards in a follow-up if
preferred.

References

@GenericRikka GenericRikka changed the title Non-Linux portability fixes (FreeBSD): xattr/fiemap/sysinfo guards Portability fixes: guard Linux-only xattr/fiemap/sysinfo usage Mar 14, 2026
@GenericRikka
Copy link
Copy Markdown
Author

Hi,
I wanted to follow up on this PR and clarify the intent a bit.

This change is not meant to alter Linux behavior; the goal is to make Linux-specific assumptions explicit by guarding Linux-only APIs/headers and using small fallbacks where needed so the code builds cleanly on non-Linux platforms as well. Linux fast paths remain in place.

I’ve retitled the PR to better reflect that broader portability focus. From my side, the patchset is intended to be minimal and upstreamable, and I’d be happy to split it further if that would make review easier.

When you have time, I’d appreciate another look.

freebsd-git pushed a commit to freebsd/freebsd-ports that referenced this pull request Apr 15, 2026
… scalability on large filesystems

FileUtils provides a library (libmfu) and a suite of MPI-parallel file
utilities such as dcp (parallel copy), dcmp (compare), ddup (duplicate
finder), dfind, dtar/dbz2, dchmod, and more. Designed for high
scalability on large filesystems.
https://hpc.github.io/mpifileutils/
https://github.com/hpc/mpifileutils/

Pull request to upstream repo with local patches:
hpc/mpifileutils#664

PR:		291679
Sponsored by:	UNIS Labs
Co-authored-by:	Vladimir Druzenko <vvd@FreeBSD.org>
gahr pushed a commit to gahr/freebsd-ports that referenced this pull request Apr 17, 2026
… scalability on large filesystems

FileUtils provides a library (libmfu) and a suite of MPI-parallel file
utilities such as dcp (parallel copy), dcmp (compare), ddup (duplicate
finder), dfind, dtar/dbz2, dchmod, and more. Designed for high
scalability on large filesystems.
https://hpc.github.io/mpifileutils/
https://github.com/hpc/mpifileutils/

Pull request to upstream repo with local patches:
hpc/mpifileutils#664

PR:		291679
Sponsored by:	UNIS Labs
Co-authored-by:	Vladimir Druzenko <vvd@FreeBSD.org>
@GenericRikka GenericRikka changed the title Portability fixes: guard Linux-only xattr/fiemap/sysinfo usage Portability fixes: guard Linux-only APIs for FreeBSD/non-Linux builds Apr 18, 2026
@GenericRikka
Copy link
Copy Markdown
Author

I've also filed issue #676 as a tracking issue.

@GenericRikka GenericRikka force-pushed the main branch 5 times, most recently from 063ceea to 3d95cb4 Compare May 11, 2026 20:57
These are the core FreeBSD portability changes extracted from the FreeBSD ports tree work. The port currently is still under review.
Port PR: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=291679
Port review: https://reviews.freebsd.org/D54230

The Patches:
 - Add FreeBSD compatibility for stat64/lstat64 by mapping them to stat/lstat (which are 64 bit by default on FreeBSD)
 - Include missing system headers in the path/parameter helper so struct stat and related macros are consistently available
 - Fix MFU_LOG(... "errno=%d %s" ...) callsites in mfu_util.c to actually pass both errno and strerrno(errno) (prevents format/argument mismatch and undefined behavior)
 - Include <fcntl.h> in mfu_util.c for file/flag-related definitions used by the code on non-Linux systems

Signed-off-by: Rikka Göring <rikka.goering@outlook.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant