Created attachment 266205 [details] new port sysutils/mpifileutils (v.0.10.1) Hi, I'd like to submit a new port: Port name: sysutils/mpifileutils Upstream: https://github.com/hpc/mpifileutils License: BSD-3-Clause WWW: https://hpc.github.io/mpifileutils/ Description: mpifileutils is a suite of MPI-based tools to perform common file and directory operations in parallel on large HPC systems. It includes utilities such as dcp, dcmp, dsync, dstripe, and others, designed for high-performance parallel filesystems. Reason for adding this port: mpifileutils is an important building block in modern HPC environments, providing scalable file copy, comparison, and verification tools that integrate well with MPI and parallel storage. This port is part of my broader effort to improve the HPC software stack on FreeBSD. Tested configurations: - FreeBSD 14.3-RELEASE amd64 - FreeBSD 15.0-RELEASE amd64 - Poudriere bulk builds on the above Optional dependencies and flavors: - Uses MPI (tested with OpenMPI – see Makefile OPTIONS for details) - BZ2 and zlib compression support - Lustre- and Linux-specific fast paths are disabled on FreeBSD; portable code paths are used instead. Implementation notes: - Many upstream source files assume Linux-only headers and syscalls (e.g., <sys/sysinfo.h>, <linux/fs.h>, SYS_getdents, xattr APIs). - The port adds small, conditional patches to: * guard Linux-specific headers with #ifdef __linux__ * provide FreeBSD-friendly compatibility macros for stat64/lstat64 and O_LARGEFILE where needed * prefer generic POSIX interfaces (getdents, struct stat, etc.) on non-Linux systems * disable Linux xattr / fiemap usage cleanly on FreeBSD while keeping core functionality intact * reuse the existing portable MPI-based bz2 compressor in place of the Linux+libcircle specific implementation where the Linux sysinfo(2) API is required QA: - portlint -AC - poudriere testport for: * 14.3-RELEASE amd64 * 15.0-RELEASE amd64 - Basic runtime tests: * dcp / dcmp / dsync on small test trees * dbcast / dbz2 smoke tests I attached the according patch to this comment.
Created Differential Review, due to the amount of patches required and Linuxisms encountered. Differential: https://reviews.freebsd.org/D54230
Created attachment 266207 [details] new port sysutils/mpifileutils (v.0.10.1) Removed DISTNAME from Makefile, due to redundancy.
Created attachment 267021 [details] new port sysutils/mpifileutils (v.0.12) Updated sysutils/mpifileutils to version 0.12
Created attachment 267718 [details] new port sysutils/mpifileutils (v.0.12) Updated patch according to review feedback.
Replace USE_GITHUB=yes with MASTER_SITES here too.
Created attachment 269649 [details] v1
Created attachment 269650 [details] v2 Attached wrong patch.
(In reply to Vladimir Druzenko from comment #7) Thanks for the improvements. I reviewed the updated patch and tested it locally. The port builds and packages successfully here, and the changes look good to me. From my side, this looks good.
PORTNAME= mpifileutils DISTVERSIONPREFIX= v DISTVERSION= 0.12 CATEGORIES= sysutils parallel MASTER_SITES= https://github.com/hpc/${PORTNAME}/releases/download/v${DISTVERSION}/ MAINTAINER= rikka.goering@outlook.de COMMENT= MPI-based parallel file utilities (dcp, dcmp, ddup, dtar, etc.) WWW= https://hpc.github.io/mpifileutils/ LICENSE= BSD3CLAUSE LICENSE_FILE= ${WRKSRC}/${PORTNAME}/LICENSE LIB_DEPENDS= libcircle.so:devel/libcircle \ libdtcmp.so:devel/dtcmp USES= cmake localbase:ldflags mpi:openmpi pkgconfig ssl tar:tgz USE_LDCONFIG= yes CMAKE_ARGS= -DCMAKE_C_COMPILER=${MPICC} \ -DCMAKE_CXX_COMPILER=${MPICXX} CMAKE_OFF= ENABLE_GPFS ENABLE_LUSTRE ENABLE_XATTRS LDFLAGS+= -lcircle -ldtcmp EXTRACT_AFTER_ARGS= --exclude dtcmp \ --exclude libcircle \ --exclude lwgrp \ --no-same-owner --no-same-permissions PORTDOCS= AUTHORS CONTRIBUTING.md DAOS-Support.md NOTICE README.md OPTIONS_DEFINE= DOCS do-install-DOCS-on: @${MKDIR} ${STAGEDIR}${DOCSDIR} ${INSTALL_DATA} ${PORTDOCS:S|^|${WRKSRC}/${PORTNAME}/|} ${STAGEDIR}${DOCSDIR} .include <bsd.port.mk> --- CMakeLists.txt.orig 2025-04-17 19:28:45 UTC +++ CMakeLists.txt @@ -298,9 +298,6 @@ LIST(APPEND libmfu_srcs mpifileutils/src/common/mfu_progress.c mpifileutils/src/common/mfu_util.c mpifileutils/src/common/strmap.c - ${lwgrp_srcs} - ${dtcmp_srcs} - ${libcircle_srcs} ) IF(ENABLE_LIBARCHIVE) LIST(APPEND libmfu_srcs @@ -328,7 +325,3 @@ INSTALL(TARGETS mfu-static DESTINATION ${CMAKE_INSTALL # some projects require a "make install" command to work, # so define at least a basic INSTALL function -INSTALL(FILES lwgrp/README lwgrp/LICENSE.TXT DESTINATION share/lwgrp) -INSTALL(FILES dtcmp/README.md dtcmp/LICENSE.TXT DESTINATION share/dtcmp) -INSTALL(FILES libcircle/COPYING DESTINATION share/libcircle) -INSTALL(FILES mpifileutils/LICENSE mpifileutils/NOTICE DESTINATION share/mpifileutils)
Created attachment 269792 [details] v3: [NEW PORT] sysutils/mpifileutils: MPI parallel file utilities for high scalability on large filesystems. Adjusted patch to review feedback: - created docs option and wired it up - Extended CMakeLists.txt patching to also remove INSTALL(FILES mpifileutils/LICENSE mpifileutils/NOTICE DESTINATION share/mpifileutils)
Update pull request: https://github.com/hpc/mpifileutils/pull/664
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=5f283cddd7b7c17cd59a5a887bf4ae74566ccbd1 commit 5f283cddd7b7c17cd59a5a887bf4ae74566ccbd1 Author: Generic Rikka <rikka.goering@outlook.de> AuthorDate: 2026-04-15 13:23:56 +0000 Commit: Vladimir Druzenko <vvd@FreeBSD.org> CommitDate: 2026-04-15 13:37:45 +0000 sysutils/mpifileutils: New port: MPI parallel file utilities for high scalability on large filesystems FileUtils provides a library (libmfu) and a suite of MPI-parallel file utilities such as dcp (parallel copy), dcmp (compare), ddup (duplicate finder), dfind, dtar/dbz2, dchmod, and more. Designed for high scalability on large filesystems. https://hpc.github.io/mpifileutils/ https://github.com/hpc/mpifileutils/ Pull request to upstream repo with local patches: https://github.com/hpc/mpifileutils/pull/664 PR: 291679 Sponsored by: UNIS Labs Co-authored-by: Vladimir Druzenko <vvd@FreeBSD.org> sysutils/Makefile | 1 + sysutils/mpifileutils/Makefile (new) | 39 ++++++++++++ sysutils/mpifileutils/distinfo (new) | 3 + .../mpifileutils/files/patch-CMakeLists.txt (new) | 20 ++++++ .../patch-mpifileutils_src_common_mfu__bz2.c (new) | 15 +++++ .../patch-mpifileutils_src_common_mfu__bz2.h (new) | 10 +++ ...pifileutils_src_common_mfu__bz2__static.c (new) | 24 +++++++ ...rc_common_mfu__compress__bz2__libcircle.c (new) | 28 +++++++++ ..._common_mfu__decompress__bz2__libcircle.c (new) | 12 ++++ ...atch-mpifileutils_src_common_mfu__flist.c (new) | 44 +++++++++++++ ...atch-mpifileutils_src_common_mfu__flist.h (new) | 11 ++++ ...ileutils_src_common_mfu__flist__archive.c (new) | 32 ++++++++++ ...pifileutils_src_common_mfu__flist__copy.c (new) | 56 +++++++++++++++++ ...pifileutils_src_common_mfu__flist__walk.c (new) | 41 ++++++++++++ .../patch-mpifileutils_src_common_mfu__io.c (new) | 15 +++++ .../patch-mpifileutils_src_common_mfu__io.h (new) | 20 ++++++ ...pifileutils_src_common_mfu__param__path.c (new) | 40 ++++++++++++ ...patch-mpifileutils_src_common_mfu__util.c (new) | 46 ++++++++++++++ ...patch-mpifileutils_src_common_mfu__util.h (new) | 13 ++++ .../patch-mpifileutils_src_dbcast_dbcast.c (new) | 73 ++++++++++++++++++++++ .../files/patch-mpifileutils_src_dbz2_dbz2.c (new) | 12 ++++ .../files/patch-mpifileutils_src_dcmp_dcmp.c (new) | 15 +++++ .../patch-mpifileutils_src_dcp1_cleanup.c (new) | 13 ++++ .../patch-mpifileutils_src_dcp1_common.h (new) | 11 ++++ .../patch-mpifileutils_src_dsync_dsync.c (new) | 22 +++++++ sysutils/mpifileutils/pkg-descr (new) | 4 ++ sysutils/mpifileutils/pkg-plist (new) | 43 +++++++++++++ 27 files changed, 663 insertions(+)
Thanks.