Created attachment 226285 [details] Git diff of new port. Provide a cross platform library which can parse, modify and abstract ELF, PE and MachO formats. Main features: Parsing: LIEF can parse ELF, PE, MachO, OAT, DEX, VDEX, ART and provides an user-friendly API to access to format internals. Modify: LIEF enables to modify some parts of these formats Abstract: Three formats have common features like sections, symbols, entry point... LIEF factors them. API: LIEF can be used in C, C++ and Python This will be required by an upcoming update to the currently broken cad/py-cadquery.
Review items: - Add python to CATEGORIES - Use MASTER=SITES=CHEESESHOP (PyPI) unless there is a compelling case not to - Match COMMENT to upstream setup.py:description (Library to instrument executable formats), or suggest upstream improve their description. - Use USE_PYTHON=autoplist unless there is a compelling case not to. pkg-plist not requires after this change - Use USE_PYTHON=concurrent for Python packages which install files into common (non python version specific) locations. THis port does (LOCALBASE/bin/*) - Remove do-build: target, shouldn't be necessary with USE_PYTHON=distutils - Strip shared library in a post-install: target See also: https://github.com/lief-project/LIEF/issues/575
(In reply to Kubilay Kocak from comment #1) - I assume that python is a virtual category? This does not bode well for my other port submissions, which are also python and also not tagged as such. - I chose github as this is where the project is based. Is it a policy to use pypi if available even though there may be more up to date version available elsewhere? If so, see above comment about not boding well. - That's a better comment (yours, not mine). It's not shown on github. - Alas autoplist does not work, for the very reason of the github issue you linked: the autoplist is incorrectly generated. It works fine as submitted. - I'm not exactly sure what the ramifications of USE_PYTHON=concurrent are, but without autoplist it may do nothing. - Sorry for the do-build. This was copied over from a port that needed it. - Strip added.
(In reply to Neal Nelson from comment #2) - Yep, its a virtual category. Please add them - Yep, it's policy, but the 'reason' is 'sdists via pypi' tests the end-to-end Python packaging workflows and best-practices of the upstream project. In this case, its highlighted their sdists (and pypi entry) are not up to date. Using sources other than PyPI if there is a "compelling temporary case" [1] is fine. Examples include, desired files (tests, licences) are not packaged in the sdist, or the latest version is not (yet) uploaded to PyPI. Please ask upstream to fix these issues - Mentioned comment is upstreams not mine :) (setup.py:description usually). - autoplist using a pkg-plist is fine until that's resolved. If its possible to fix an issue in a local port patch until the next upstream release is cut, that's preferred. - concurrent handles most common-location filename renaming in both autoplist and static plist cases. You'll want to enable DEVELOPER=yes in make.conf and run tests with portlint -AC and poudriere for correct and complete QA. We're available on #freebsd-ports on Libera IRC if you need help
If you hit any issues and feel like it's hard, ask. It's a complex environment.
Created attachment 226389 [details] Git diff of new port. Here's a new version of the port with all the suggested changes. It's been portlinted and poudriere testported, as before. I have a couple of other new port submission that I will update with the same changes where appropriate. These are all dependent ports on a large change to cad/OCP and cad/cadquery that I am waiting to submit. cad/OCP currently is broken because cad/opencascade changed. The dependent ports are build dependencies and to be frank I have no other interest in them, including this port, other than that they have to exist for the build to work. The other prs are: bug #256924 and bug #256925.
It's unable to generate a correct ${_PYTHONPKGLIST} with USE_PYTHON=autoplist # make check-plist ====> Checking for pkg-plist issues (check-plist) ===> Parsing plist ===> Checking for items in STAGEDIR missing from pkg-plist Error: Orphaned: %%PYTHON_SITELIBDIR%%/lief%%PYTHON_EXT_SUFFIX%%.so ===> Checking for items in pkg-plist which are not in STAGEDIR Error: Missing: %%PYTHON_SITELIBDIR%%/.11.5/lief%%PYTHON_EXT_SUFFIX%%.so ===> Error: Plist issues found. *** Error code 1 Stop. make: stopped in /usr/ports/devel/py-lief There's a weird path in ${_PYTHONPKGLIST}: /usr/local/lib/python3.8/site-packages/.11.5/lief.cpython-38.so The weird path was caused by _mutate_outputs() in distutils/command/install_lib.py: https://github.com/python/cpython/blob/v3.8.10/Lib/distutils/command/install_lib.py#L143-L156 143 def _mutate_outputs(self, has_any, build_cmd, cmd_option, output_dir): 144 if not has_any: 145 return [] 146 147 build_cmd = self.get_finalized_command(build_cmd) 148 build_files = build_cmd.get_outputs() 149 build_dir = getattr(build_cmd, cmd_option) 150 151 prefix_len = len(build_dir) + len(os.sep) 152 outputs = [] 153 for file in build_files: 154 outputs.append(os.path.join(output_dir, file[prefix_len:])) 155 156 return outputs on my 14.0-CURRENT aarch64: build_dir => build/lib.freebsd-14.0-CURRENT-arm64-3.8 file => /usr/ports/devel/py-lief/work-py38/LIEF-0.11.5/lief.cpython-38.so prefix_len => len(build_dir) + len(os.sep) => 40 + 1 => 41 file[prefix_len:] => .11.5/lief.cpython-38.so output_dir => /usr/ports/devel/py-lief/work-py38/stage/usr/local/lib/python3.8/site-packages/ os.path.join(output_dir, file[prefix_len:]) => /usr/ports/devel/py-lief/work-py38/stage/usr/local/lib/python3.8/site-packages/.11.5/lief.cpython-38.so It looks like we are supposed to get a file name build/lib.freebsd-14.0-CURRENT-arm64-3.8/lief.cpython-38.so instead of /usr/ports/devel/py-lief/work-py38/LIEF-0.11.5/lief.cpython-38.so https://github.com/python/cpython/blob/v3.8.10/Lib/distutils/command/build_ext.py#L637-L664 637 # -- Name generators ----------------------------------------------- 638 # (extension names, filenames, whatever) 639 def get_ext_fullpath(self, ext_name): 640 """Returns the path of the filename for a given extension. 641 642 The file is located in `build_lib` or directly in the package 643 (inplace option). 644 """ 645 fullname = self.get_ext_fullname(ext_name) 646 modpath = fullname.split('.') 647 filename = self.get_ext_filename(modpath[-1]) 648 649 if not self.inplace: 650 # no further work needed 651 # returning : 652 # build_dir/package/path/filename 653 filename = os.path.join(*modpath[:-1]+[filename]) 654 return os.path.join(self.build_lib, filename) 655 656 # the inplace option requires to find the package directory 657 # using the build_py command for that 658 package = '.'.join(modpath[0:-1]) 659 build_py = self.get_finalized_command('build_py') 660 package_dir = os.path.abspath(build_py.get_package_dir(package)) 661 662 # returning 663 # package_dir/filename 664 return os.path.join(package_dir, filename) inplace == 0 get_ext_fullpath(ext.name) => build/lib.freebsd-14.0-CURRENT-arm64-3.8/lief.cpython-38.so inplace == 1 get_ext_fullpath(ext.name) => /usr/ports/devel/py-lief/work-py38/LIEF-0.11.5/lief.cpython-38.so We could change the inlpace value to 0 in ${WRKSRC}/setup.cfg, this should make USE_PYTHON=autoplist work: post-patch: @${REINPLACE_CMD} -e 's|^inplace=1|inplace=0|' ${WRKSRC}/setup.cfg
https://github.com/lief-project/LIEF/blob/0.11.5/setup.py#L110 100 jobs = self.parallel if self.parallel else 1 We might need to add a --parallel option to respect MAKE_JOBS_NUMBER PYDISTUTILS_BUILDARGS+= build_ext --parallel=${MAKE_JOBS_NUMBER}
(In reply to Fukang Chen from comment #7) > PYDISTUTILS_BUILDARGS+= build_ext --parallel=${MAKE_JOBS_NUMBER} This should be only --parallel without the build_ext command PYDISTUTILS_BUILDARGS+= --parallel=${MAKE_JOBS_NUMBER}
(In reply to Fukang Chen from comment #6) This must be an artefact of running on either 14.0-CURRENT or aarch64, or maybe both, as it does not exhibit on 13.0-RELEASE-p3 amd64.
(In reply to Neal Nelson from comment #9) comment #6 is not about reporting the port fails to build on 14.0-CURRENT aarch64, I was just trying to explaining why USE_PYTHON=autoplist doesn't work. OS version and architecture are irrelevant to USE_PYTHON=autoplist, I will just use the testport log from dvl@ here https://gist.github.com/dlangille/78902092fd8839bc4514492f949bad16 on my 14.0-CURRENT aarch64: ====> Checking for pkg-plist issues (check-plist) ===> Parsing plist ===> Checking for items in STAGEDIR missing from pkg-plist Error: Orphaned: %%PYTHON_SITELIBDIR%%/lief%%PYTHON_EXT_SUFFIX%%.so ===> Checking for items in pkg-plist which are not in STAGEDIR Error: Missing: %%PYTHON_SITELIBDIR%%/.11.5/lief%%PYTHON_EXT_SUFFIX%%.so ===> Error: Plist issues found. dvl@'s testport log ====> Checking for pkg-plist issues (check-plist) ===> Parsing plist ===> Checking for items in STAGEDIR missing from pkg-plist Error: Orphaned: %%PYTHON_SITELIBDIR%%/lief%%PYTHON_EXT_SUFFIX%%.so ===> Checking for items in pkg-plist which are not in STAGEDIR Error: Missing: %%PYTHON_SITELIBDIR%%/f-0.11.4/lief%%PYTHON_EXT_SUFFIX%%.so ===> Error: Plist issues found. https://github.com/python/cpython/blob/v3.8.10/Lib/distutils/command/install_lib.py#L143-L156 in the function _mutate_outputs(): build_dir => build/lib.freebsd-14.0-CURRENT-arm64-3.8 build_dir => build/lib.freebsd-12.2-RELEASE-p2-amd64-3.8 file => /usr/ports/devel/py-lief/work-py38/LIEF-0.11.5/lief.cpython-38.so file => /wrkdirs/usr/ports/vrt/py-lief/work-py38/lief-0.11.4/lief.cpython-38.so prefix_len => len(build_dir) + len(os.sep) => 40 + 1 => 41 prefix_len => len(build_dir) + len(os.sep) => 43 + 1 => 44 file[prefix_len:] => .11.5/lief.cpython-38.so file[prefix_len:] => f-0.11.4/lief.cpython-38.so os.path.join(output_dir, file[prefix_len:]) => /usr/ports/devel/py-lief/work-py38/stage/usr/local/lib/python3.8/site-packages/.11.5/lief.cpython-38.so os.path.join(output_dir, file[prefix_len:]) => /wrkdirs/usr/ports/vrt/py-lief/work-py38/stage/usr/local/lib/python3.8/site-packages/f-0.11.4/lief.cpython-38.so
(In reply to Fukang Chen from comment #6) Nice isolation Fukang. And yeh, this is a common class of issue for upstreams that modify or subclass the default installation routines, particular for c extensions. Feel free to address this in whatever way you think is most suitable, with preference for fixing upstream source to correct the error (rather than special or custom build/install targets (ideally). We recently fixed one [1] (IRC) by removing 'inplace' overrides from the setup.py which was sufficient to correct the issue. They were also 'double building', which was more problematic. Regarding the parallel builds, go for it, though it would be nice to have a common (python) way to do this, and I've reached out to upstream to find out if there's options in that regard. [1] https://github.com/overviewer/Minecraft-Overviewer
Going to need this for checksec.py (and others). I'll test latest patch plus Fukangs comment 10 and report back post QA
The bundled dependencies (see third-party directory in distfile) are somewhat problematic. We are expected to unbundle all dependencies and use system (ports/package) versions of them. In the short term, we may be able to get away with leaving bundled, any software that has no port/package for it yet, with a long term view to porting them. It would be good to go through the list (I can see at least mbedtls and pybind are in ports) and determine the best method to 'unbundle' them. Nothing stood out with a very quick cursory review of upstream Installation docs, so we may need to use cmake vars or modify the cmake invocations. *Hopefully* there's a simple way given this python package uses cmake, which itself is unorthodox to say the least. I'm running the build now and will run through QA again with additional feedback/changes to come. If someone can clarify this 'how do we unbundle these third-party dependencies' question, it would go a long way to progressing and landing this.
Given that ports seems to be suffering lots of incompatible requirements for conflicting ports at the moment (tbb, onetbb, various opencasecade versions, etc.), maybe a port containing it's own dependencies isn't such a bad thing, as long as it doesn't pollute any other ports. Or to put it another way, I have no idea how to go about removing *some* of the third-party files, which we have in ports, but not others, which we do not. I can see part of the CMakeLists.txt that deals with third-party, but not being a CMake user I could only have a stab at trying to remove the dependencies.
(In reply to Neal Nelson from comment #14) Just mentioned it for the record and future reference. I personally lean toward action unless it sets us backward, and as long as the remaining issues are tracked (and dont fall into a blackhole and forgotten). Clarity on potential paths to unbundle would remain useful, and in the meantime, it would be great to get this port using autoplist given Fukangs isolation, and patch suggestion in comment 6, QA confirmed (once more), such that it can be landed. I'm happy to review that given a QA pass
Any update on getting this over the finish lines? I can help testing and I'd also vote for _not_ unbundling anything, unless it would add to CONFLICTS.
(In reply to Ulrich Spörlein from comment #16) Current status on https://github.com/lief-project/LIEF/issues/575: "To be reopen if there are still the issue with the latest version" Latest release is 0.13.1
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=2cde3efd909901be11505bac4617e88ad1d7a34b commit 2cde3efd909901be11505bac4617e88ad1d7a34b Author: Neal Nelson <ports@nicandneal.net> AuthorDate: 2023-09-01 16:27:03 +0000 Commit: Muhammad Moinur Rahman <bofh@FreeBSD.org> CommitDate: 2023-09-01 17:16:15 +0000 devel/py-lief: New Port The purpose of this project is to provide a cross platform library which can parse, modify and abstract ELF, PE and MachO formats. Main features: - Parsing: LIEF can parse ELF, PE, MachO, OAT, DEX, VDEX, ART and provides a user-friendly API to access to format internals. - Modify: LIEF enables to modify some parts of these formats - Abstract: Three formats have common features like sections, symbols, entry point... LIEF factors them. - API: LIEF can be used in C, C++ and Python WWW: https://github.com/lief-project/lief PR: 257041 devel/Makefile | 1 + devel/py-lief/Makefile (new) | 26 ++++++++++++++++++++++++++ devel/py-lief/distinfo (new) | 3 +++ devel/py-lief/pkg-descr (new) | 11 +++++++++++ devel/py-lief/pkg-plist (new) | 4 ++++ 5 files changed, 45 insertions(+)