Bug 253905

Summary: databases/mysql80-server excessive free storage requirements for building from source
Product: Ports & Packages Reporter: Viktor Štujber <viktor.stujber+freebsd-bugs_v4CCPfay>
Component: Individual Port(s)Assignee: Jochen Neumeister <joneum>
Status: Closed FIXED    
Severity: Affects Only Me CC: joneum
Priority: --- Flags: bugzilla: maintainer-feedback? (joneum)
Version: Latest   
Hardware: Any   
OS: Any   

Description Viktor Štujber 2021-02-28 11:18:46 UTC 
mysql-server at this point requires 1GB of storage to build via portmaster, multiple times more than any other software I'm using. And from what I've observed, it's not due to the inherent complexity of the source code, but due to compounding inefficiencies in the build process.

The distfile 'mysql-boost-8.0.23.tar.gz' is 278 MB.
(mysql-5.5.62.tar.gz size is 20 MB.)
The extracted source tree is 810 MB.
(From that, the mysql-test/ directory is 516 MB.)
The intermediate .build/ directory, without tests, is 947 MB.
The stage/ directory is another 177 MB.

In total, the required free storage is no less than 2212 MB, to produce a 177 MB package - a 12.5 : 1 ratio. Back in the mysql 5.x days, it was possible to fit the whole build, distfile included, onto a 1 GB tmpfs. Perhaps it would be worthwhile to get closer to that kind of number. Here are my observations.

The ports makefile currently has no options. One of them could be to disable tests. I did not find any direct knob in their cmakefile that would do that, but patching it out is easy. That in turn allows having a post-extract step delete the mysql-test subdirectory for immediate savings of ~516 MB.

The build process generates intermediate .o files and resulting executables/libraries for the entire source tree without any cleanup steps inbetween. That is, even though the results are already built, it still keeps the temporary stuff around. And it does in fact require them to be there - they will be recompiled if missing, even though all the outputs have already been built. If the build system instead cleaned its temporary files when they were no longer needed, and did not require them once the corresponding outputs were built, that would save ~200 MB.

When the build completes and the files are staged for installation, the staging is done via copying. It might be possible to do this via symlinks during staging, and then resolving them during install. This would save another ~177 MB.

I understand that these ideas may not be compatible with how freebsd and mysql currently does builds, and changes could require upstream cooperation. I just figured I'd write this to give this issue (or non-issue?) a bit of attention.
Comment 1 Jochen Neumeister freebsd_committer freebsd_triage 2022-01-14 23:31:18 UTC 
Does the problem still exist?
Comment 2 Viktor Štujber 2022-01-15 17:33:25 UTC 
I do not see any related changes in the port makefile, and the issue was not yet raised upstream, so I assume things remain the same.

Regarding the huge distfile, 85% of it is testing files. Removing tests and repacking with lzma2 can reduce it from 278MB to 27MB. But that's an extreme change.
Regarding the unpacked source tree, deleting mysql-test/ when tests are disabled would save 516MB. There is currently no port option to disable tests. Another bugreport suggested CMAKE_ARGS += -DWITH_UNIT_TESTS=OFF, I don't know if it actually works. I am currently using a local port modification that deletes the test directory in post-extract.
Regarding the build forcing retention of intermediate object files even though the targets are already built, and not cleaning them up gradually as they are built, is a structural build system inefficiency that would have to be addressed upstream.
Regarding the staging directory holding copies instead of symlinks/hardlinks, it's just a simple idea that I had when going over the build output. I don't know how the build system picks which files to stage, or how it stages them, but I assume it's done via simple file copy. This is something that Mk\bsd.port.mk would be handling.

The motivation behind this was that in the past, it used to be possible to build mysql-server on tmpfs comfortably on a cheap pc, without consuming all available system memory and/or running out and not being able to complete the build.
Comment 3 Jochen Neumeister freebsd_committer freebsd_triage 2023-09-02 11:44:05 UTC 
Since no one else has posted on this problem, or had the same problem, I'm closing here. If you have further problems with this topic feel free to open it again