Bug 264433
| Summary: | [regression] installkernel produces /boot/kernel directory over 500MB in size | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Eugene Grosbein <eugen> |
| Component: | kern | Assignee: | Ed Maste <emaste> |
| Status: | Closed FIXED | ||
| Severity: | Affects Some People | CC: | CrazyMihey, dim, emaste, grahamperrin, imp, markj |
| Priority: | --- | Keywords: | needs-qa |
| Version: | 13.1-STABLE | ||
| Hardware: | Any | ||
| OS: | Any | ||
|
Description
Eugene Grosbein
2022-06-03 11:56:25 UTC
Forgot to include my /etc/src.conf CPUTYPE?= native WITHOUT_KERNEL_SYMBOLS= WITHOUT_DEBUG_FILES= WITHOUT_TESTS= WITHOUT_LLDB= WITHOUT_LLVM_TARGET_ALL= WITH_LLVM_TARGET_X86= WITHOUT_CLANG_FULL= # EOF Also, I have full builworld and buildkernel output saved. But not installkernel. Also I had filemon.ko loaded and WITH_META_MODE= yes in /etc/src-env.conf and empty obj directory before starting build. Here is zfs.ko.meta from the module obj directory, in case it matters: http://www.grosbein.net/freebsd/zfs.ko.meta I'm afraid the following merge to stable/13 created the problem: https://cgit.freebsd.org/src/commit/?h=stable/13&id=177466dac805df0271511a0d7cd3152a467820c6 I compared zfs.ko.meta for this problematic system and another my stable/13 (earlier) system that does not have the problem and found the difference. "Normal" system has this: CMD ld -m elf_x86_64_fbsd -d -warn-common --build-id=sha1 -T /usr/src/sys/conf/ldscript.kmod.amd64 -r -d -o zfs.ko ... Note second "-d" flag. And newer after-commit problematic one has this: CMD ld -m elf_x86_64_fbsd -d -warn-common --build-id=sha1 -T /home/src/sys/conf/ldscript.kmod.amd64 -r -o zfs.ko ... Note that second -d is missing. (In reply to Eugene Grosbein from comment #5) Why do you think more than one -d option makes any difference? It should not. That said, with lld the -d option is really non-functional, in the sense that it does not do anything. Newer versions warn about this being obsolete, so that's why the option is removed for lld >= 14. (In reply to Dimitry Andric from comment #6) That was pure guess. Indeed, I think another option should not make difference but if we talk about bugs, sometimes it happens. (In reply to Eugene Grosbein from comment #1) > WITHOUT_KERNEL_SYMBOLS= What this actually does is disable splitting of debug info into separate files under /usr/lib/debug, so instead the debug info is kept in /boot/kernel. The name of the option (and its documentation in src.conf.5) is misleading. To omit debug info entirely you need to remove "makeoptions DEBUG=-g" from the kernel configuration (or add some additional configuration to strip debug info at installkernel time). (In reply to Mark Johnston from comment #8) If it is true then it is POLA issue as the option used to work as documented for many major releases including stable/13 of 16 April 2022: no *.symbols files installed and no debug info in the installed *.ko files. Another difference between "old good" and "new bad" stable/13 systems: old one is virtual machine with 1GB RAM only and it has also in /etc/src.conf: WITHOUT_CLANG_BOOTSTRAP= WITHOUT_LLD_BOOTSTRAP= WITHOUT_ELFTOOLCHAIN_BOOTSTRAP= WITHOUT_TOOLCHAIN= And it has older system compiler: FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe) Target: x86_64-unknown-freebsd13.0 Newer is bare metal system with 16GB RAM, it does not have four WITHOUT_* above knobs and used newver system compiler to build: FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303) Target: x86_64-unknown-freebsd13.1 The problem does not reproduce if I use problematic system to build any single kernel module outside of kernel build environment. I've reproduced the problem using "old good" stable/13 system that has MODULES_WITH_WORLD= in its /etc/src.conf. Without this option, it builds kernel modules with debug info, too. So, it seems to be regression comparing to stable/12. Likely this change: commit 9b6edf364eb05aca4709732136e92a6ab739bb36 Author: Conrad Meyer <cem@FreeBSD.org> Date: Tue Jun 23 18:25:31 2020 +0000 kmod.mk: Don't split out debug symbols if requested Ports bsd.kmod.mk explicitly sets MK_KERNEL_SYMBOLS=no to prevent auto- splitting of debuginfo from kernel modules. If that knob is set, don't split out a .ko.debug and .ko from .ko.full; just generate a .ko with debuginfo and leave it be. Otherwise, with DEBUG_FLAGS set and MK_KERNEL_SYMBOLS=no, we would helpfully strip out the debuginfo from the .ko.full and then not install it. That is not the desired result a WITH_DEBUG port kmod build. Reviewed by: emaste, jhb Differential Revision: https://reviews.freebsd.org/D24835 Notes: svn path=/head/; revision=362549 How about this description for WITHOUT_KERNEL_SYMBOLS: Do not build or install standalone kernel debug symbol files. Debug data (if enabled by the build) will be included in the kernel and modules. (In reply to Ed Maste from comment #14) Seems ok. I would go a bit further and note that the kernel configuration file itself is what's typically used to enable generation of debug info. Do not build or install standalone kernel debug symbol files. Debug data (if enabled by the kernel configuration file) will be included in the kernel and modules. A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=92fc5822a8301d0c20d292e97d1661b25f5035bc commit 92fc5822a8301d0c20d292e97d1661b25f5035bc Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2022-06-07 14:20:39 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2022-06-07 15:33:12 +0000 Clarify WITHOUT_KERNEL_SYMBOLS description Debug data is enabled via `makeoptions DEBUG=-g` in the kernel config file (e.g. GENERIC). If debug data is enabled and WITHOUT_KERNEL_SYMBOLS is set then debug data is included in the kernel and module files. PR: 264433 Discussed with: markj MFC after: 1 week Sponsored by: The FreeBSD Foundation share/man/man5/src.conf.5 | 6 ++++-- tools/build/options/WITHOUT_KERNEL_SYMBOLS | 4 +++- 2 files changed, 7 insertions(+), 3 deletions(-) To reduce the size of kernel/modules in /boot, either remove `makeoptions DEBUG=-g` from the kernel configuration file, or remove WITHOUT_KERNEL_SYMBOLS from src.conf. It is a regression still. There should be a way to build GENERIC and not install any debugging symbols as it was supported for all releases before FreeBSD 13. (In reply to Eugene Grosbein from comment #19) This can be done with "make buildkernel KERNCONF=GENERIC DEBUG=". (In reply to Mark Johnston from comment #20) Unexpected overflow of small root partition for legacy UFS servers will not be good for our userbase. Why don't we keep previous meaning of WITHOUT_KERNEL_SYMBOLS for installworld context? Is it possible to strip kernel module while installing it? Originally the intent of this knob was to not *INSTALL* the symbol files, but to still generate them. So you would have them in your build tree, but not blow out the tiny root filesystems of the time in case you needed to debug a kernel core file. I'd support restoring that behavior and/or making it behave that way or something similar. I'm not sure when it changed and am too lazy to do the archeology to see if I screwed up implementing it, or someone changed it along the way. Therefore, I think cem@'s change is a bug and should be reverted. (In reply to Warner Losh from comment #22) > I'm not sure when it changed 12.3-STABLE does not have the problem. I routinely use MODULES_WITH_WORLD= for minor upgrades to speedup kernel build but disable MODULES_WITH_WORLD for major upgrades. > I'd support restoring that behavior Another reason to restore that: now we have inconsistent module builds within or outside kernel build environment. It was consistent before the change. (In reply to Warner Losh from comment #22) > Therefore, I think cem@'s change is a bug and should be reverted. Or maybe not reverted but improved to strip debug_info during installation. (In reply to Warner Losh from comment #22) > I'm not sure when it changed It changed with cem's 9b6edf364eb05 which is in 13 but not 12. > not blow out the tiny root filesystems of the time Note that we do not install the .debug files in /boot (for some time), so this is no longer an issue. We have four possible cases (assuming main as of today, unless otherwise stated): 1) WITH_KERNEL_SYMBOLS DEBUG=-g This is the default. Kernel debug files are installed in /usr/lib/debug/... and do not take up space in the root / boot filesystem. 2) WITHOUT_KERNEL_SYMBOLS DEBUG=-g Debug data is embedded in the kernel and modules (and so require a large enough root/boot filesystem). 3) cem's change (perhaps logically) reverted WITHOUT_KERNEL_SYMBOLS DEBUG=-g Debug data is split during build, but is not installed. 4) DEBUG= (either editing kernel config, or passing DEBUG= on build command line) Debug data is not generated at all. Cases 1, 3, and 4 are compatible with "small" root/boot filesystems. cem's change was prompted by ports: bsd.kmod.mk forces WITHOUT_KERNEL_SYMBOLS, and prior to the change that meant we threw away debug info for ports kmods. With cem's change reverted case 2 is not available to the user - there is no way to install kernel/modules that have built-in debug info. There are a few ways we could address this: A) Add some sort of kernel module debug file support for ports, and revert cem's change. B) Introduce a new knob WITH_/WITHOUT_SPLIT_KERNEL_DEBUG to control whether we do the debug info splitting, and have WITH_/WITHOUT_KERNEL_SYMBOLS control only the installation of standalone debug files. Perhaps rename it as well, WITH_/WITHOUT_INSTALL_KERNEL_DEBUG. This scheme is also available by setting WITH_KERNEL_SYMBOLS during buildkernel and WITHOUT_KERNEL_SYMBOLS during installkernel. This could also be addressed by some sort of general scheme for providing different values for certain knobs at build/install time (e.g. every WITH_FOO would get a corresponding WITH_BUILD_FOO and WITH_INSTALL_FOO). C) Leave it as is (perhaps with more documentation/notes), as all configurations are available. > Another reason to restore that: now we have inconsistent module builds > within or outside kernel build environment. It was consistent before the change. I suspect it only appears so because building modules outside of the kernel environment is omitting the DEBUG=-g from the kernel config. (In reply to Ed Maste from comment #25) > 1) > > WITH_KERNEL_SYMBOLS > DEBUG=-g > > This is the default. Kernel debug files are installed in /usr/lib/debug/... and > do not take up space in the root / boot filesystem. It still takes lots of space on /usr/lib. And it did not before WITHOUT_KERNEL_SYMBOLS. For example, all my UFS-only systems have small 1GB /usr partition mounted r/o and separated from /usr/{local|src|ports|obj}. Two reasons: 1) it does not need fsck to mount in case of non-clean reboot and 2) there are no reasons for /usr to be mounted r/w except of upgrade moments when it is upgraded to r/w temporary. > as all configurations are available. I still do not see an option compatible with 12.x behaviour that does not install debugging info at all except of building with additional DEBUG= that breaks POLA for upgrade from 12.3 (In reply to Ed Maste from comment #25) > Introduce a new knob I'm for this way. This needs to be a new knob. Cem's changes make the installed image larger for a knob whose purpose is to make the image smaller. Basing it on DEBUG=-g is wrong. It needs to be a new knob if people want this odd behavior. When I created this, it was definitely to omit the .debug files that were then at the time new and a huge space burden. I wanted a way to have the debug files in my build tree and not install them in a resource constrained CF card that FreeBSD was running on. So I'm definitely of the opinion this change is incompatible and undesirable. In case it wasn't clear, I did the original for two reasons. First, I didn't want to blow up my root on my laptop. I also didn't want to bump up yhe 64MB CF card to a 1GB CF card. The move to /usr/lib fixes 1, but not 2. I stopped using it for my laptop long ago, but still used it for nanobsd... Approach B in https://reviews.freebsd.org/D35437 A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=e3709cfe6a2a1f7d345db2870be175c9cf2a8979 commit e3709cfe6a2a1f7d345db2870be175c9cf2a8979 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2022-06-08 21:02:24 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2022-06-08 21:14:46 +0000 Add SPLIT_KERNEL_DEBUG knob Prior to 9b6edf364eb0 WITHOUT_KERNEL_SYMBOLS split kernel debug data into standalone debug files at build time, but did not install those files. As of 9b6edf364eb0 it stopped splitting the debug data, leaving it in the kernel and modules (the default kernel configs include DEBUG=-g). Revert 9b6edf364eb0 and introduce a new build-time SPLIT_KERNEL_DEBUG knob, as some people rely on the pre-9b6edf364eb0 WITHOUT_KERNEL_SYMBOLS behaviour and that was imp's original intent. PR: 264433 Reviewed by: eugen, imp MFC after: 3 weeks Relnotes: yes Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D35437 sys/conf/kern.opts.mk | 5 +++++ sys/conf/kmod.mk | 4 ++-- tools/build/options/WITHOUT_KERNEL_SYMBOLS | 5 ++--- tools/build/options/WITHOUT_SPLIT_KERNEL_DEBUG (new) | 3 +++ 4 files changed, 12 insertions(+), 5 deletions(-) A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=0c4d13c521aaa56725e31d5a052760b9effd2dd5 commit 0c4d13c521aaa56725e31d5a052760b9effd2dd5 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2022-06-07 14:20:39 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2022-07-18 15:01:23 +0000 Add SPLIT_KERNEL_DEBUG knob Prior to 9b6edf364eb0 WITHOUT_KERNEL_SYMBOLS split kernel debug data into standalone debug files at build time, but did not install those files. As of 9b6edf364eb0 it stopped splitting the debug data, leaving it in the kernel and modules (the default kernel configs include DEBUG=-g). Revert 9b6edf364eb0 and introduce a new build-time SPLIT_KERNEL_DEBUG knob, as some people rely on the pre-9b6edf364eb0 WITHOUT_KERNEL_SYMBOLS behaviour and that was imp's original intent. PR: 264433 Reviewed by: eugen, imp MFC after: 3 weeks Relnotes: yes Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D35437 (cherry picked from commit 92fc5822a8301d0c20d292e97d1661b25f5035bc) (cherry picked from commit e3709cfe6a2a1f7d345db2870be175c9cf2a8979) (cherry picked from commit a7bd57de86372d834cf9312fcc4c39070653e009) share/man/man5/src.conf.5 | 15 +++++++++++++-- sys/conf/kern.opts.mk | 5 +++++ sys/conf/kmod.mk | 4 ++-- tools/build/options/WITHOUT_KERNEL_SYMBOLS | 3 ++- tools/build/options/WITHOUT_SPLIT_KERNEL_DEBUG (new) | 3 +++ 5 files changed, 25 insertions(+), 5 deletions(-) 9b6edf364eb0 was not merged to stable/12 so I won't merge this either. So, Now If SomeOne wants to Minimize Kernel Size (of UnKnown Version), He must write in «/etc/src.conf» something like this:
WITHOUT_KERNEL_DEBUG_FILES="NoKernelModulesDeBug"
WITHOUT_DEBUG_FILES="NoExecutAbleFilesDeBug"
.if ( ${TargetKernelVersion} < "13.0" ) || ( ${TargetKernelVersion} >= "13.2" ) but ${TargetKernelVersion} is NOT a Correct Variable Name!
WITHOUT_KERNEL_SYMBOLS="WithOut"
WITH_SPLIT_KERNEL_DEBUG="Yes"
. else
. undef WITHOUT_KERNEL_SYMBOLS
.endif # KernelVersion ≈ 13.0 — 13.1
but ${TargetKernelVersion} is NOT a Correct Variable Name. I did not find, how to Get Accurate New (Builded) Kernel Version during «/etc/src.conf» Processing…
|