Bug 257514 - Can not embed ZFS into Kernel on RelEng/13.0 i386
Summary: Can not embed ZFS into Kernel on RelEng/13.0 i386
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.0-RELEASE
Hardware: i386 Any
: --- Affects Only Me
Assignee: freebsd-fs (Nobody)
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2021-07-31 03:14 UTC by CrazyMihey
Modified: 2023-10-01 14:53 UTC (History)
0 users

See Also:


Attachments
Simplyfied Kernel Configuration, Full Building Process Results and System Information (172.84 KB, application/x-xz)
2021-07-31 03:14 UTC, CrazyMihey
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description CrazyMihey 2021-07-31 03:14:26 UTC
Created attachment 226811 [details]
Simplyfied Kernel Configuration, Full Building Process Results and System Information

Hello, Developers! & Sorry, if I do SomeThing wrong.
Some Time ago, on FreeBSD (at least 11.3 & 12.1 & 12.2) on x86 32-Bit I included ZFS Support into Custom Kernel in following Way:
device	zfs	# in «/usr/src/sys/i386/conf/CrazyBook»
But after DownLoading RelEng/13.0 Sources, This Kernel Configuration does not Build any more. :(
Process of Getting 13.0:
	pkg install gitup
	sed -E -i "" "s~(releng/)([\.\,0-9]*)~\113.0~" "/usr/local/etc/gitup.conf"
	$(whereis -b -q time) -l rm -rf /usr/src/* /usr/src/.[!.]* /usr/obj/* /usr/obj/.[!.]* /usr/ports/* /usr/ports/.[!.]* "/var/db/portsnap" "/var/db/gitup"
	$(whereis -b -q time) -l gitup "release" -v "1" && rm -Rf "/var/db/gitup/release" && for ConfigurationPath in $(find "${Src%"/"}"/sys/*/conf -prune -print); do [ -f "${ConfigurationPath}/GENERIC" ] && printf "\"%s\"\n" "${ConfigurationPath}" && ln -f -s -v ~/Kernels/* "${ConfigurationPath}"; done
First Build the Compiler:
	cd "/usr/src" && script -a -q "/var/tmp/$(date -j "+%Y.%m.%d-%H:%M:%S")-BuildWorld-$(sh -c 'set -o "pipefail" && egrep -i -o "^[[:space:]]*#[[:space:]]*DeFine.*BSD.*Version[[:space:]]+[[:digit:]]+" "/usr/src/sys/sys/param.h" | rev | egrep -o "[[:digit:]]+" | head -n "1" | rev | tr -d "[[:space:]]"' || svnlite info "/usr/src" 2>"/dev/null" | grep --ignore-case -- "RevisiOn" | grep --only-matching "[[:digit:]]\+" || cat "/usr/src/.svnversion" 2>"/dev/null" | tr -d "\r\n" | grep "" || cut -f "2" -d ":" "/usr/src/.gituprevision")-$(( "$(sysctl -n "hw.realmem")" >> 30 ))GB-$(($(sysctl -n "kern.sched.topology_spec" 2>"/dev/null" | grep --extended-regexp --ignore-case --quiet "<Flags>.*Thread.*Group.*</Flags>" >"/dev/null" 2>&1 && printf "$(($(sysctl -n "kern.smp.cpus")/2))" || printf "$(sysctl -n "kern.smp.cpus")")+1))Th.Out" $(whereis -b -q time) -l make buildworld KERNCONF="CrazyBook" -j "$(($(sysctl -n "kern.sched.topology_spec" 2>"/dev/null" | grep --extended-regexp --ignore-case --quiet "<Flags>.*Thread.*Group.*</Flags>" >"/dev/null" 2>&1 && printf "$(($(sysctl -n "kern.smp.cpus")/2))" || printf "$(sysctl -n "kern.smp.cpus")")+1))"	# I understand, the Command looks Ugly, but this is just a Part of Future Script, where Long Expressions will be moved into Functions or AliAces…
To make sure about CLang Parameters, I removed Parameter Files «/etc/src.conf» and «/etc/make.conf»:
	rm -Rf "/etc/src.conf" "/etc/make.conf"
Second try to Build a Custom Kernel:
	cd "/usr/src" && script -a -q "/var/tmp/$(date -j "+%Y.%m.%d-%H:%M:%S")-BuildKernel-$(sh -c 'set -o "pipefail" && egrep -i -o "^[[:space:]]*#[[:space:]]*DeFine.*BSD.*Version[[:space:]]+[[:digit:]]+" "/usr/src/sys/sys/param.h" | rev | egrep -o "[[:digit:]]+" | head -n "1" | rev | tr -d "[[:space:]]"' || svnlite info "/usr/src" 2>"/dev/null" | grep --ignore-case -- "RevisiOn" | grep --only-matching "[[:digit:]]\+" || cat "/usr/src/.svnversion" 2>"/dev/null" | tr -d "\r\n" | grep "" || cut -f "2" -d ":" "/usr/src/.gituprevision")-$(( "$(sysctl -n "hw.realmem")" >> 30 ))GB-$(($(sysctl -n "kern.sched.topology_spec" 2>"/dev/null" | grep --extended-regexp --ignore-case --quiet "<Flags>.*Thread.*Group.*</Flags>" >"/dev/null" 2>&1 && printf "$(($(sysctl -n "kern.smp.cpus")/2))" || printf "$(sysctl -n "kern.smp.cpus")")+1))Th.Out" $(whereis -b -q time) -l make buildkernel KERNCONF="CrazyBook" -j "$(($(sysctl -n "kern.sched.topology_spec" 2>"/dev/null" | grep --extended-regexp --ignore-case --quiet "<Flags>.*Thread.*Group.*</Flags>" >"/dev/null" 2>&1 && printf "$(($(sysctl -n "kern.smp.cpus")/2))" || printf "$(sysctl -n "kern.smp.cpus")")+1))" && printf "\007" && sleep "0.500" && printf "\007" && sleep "1.000" && printf "\007" && sleep "2.000" && printf "\007" && sleep "4.000" && printf "\007"
Now I get Following Error:
…	…	…	…
--- opensolaris_atomic.o ---
cc -target i386-unknown-freebsd13.0 --sysroot=/usr/obj/usr/src/i386.i386/tmp -B/usr/obj/usr/src/i386.i386/tmp/usr/bin -c -x assembler-with-cpp -DLOCORE -DFREEBSD_NAMECACHE  -D_SYS_VMEM_H_  -D__KERNEL  -D__KERNEL__  -nostdinc  -include /usr/src/sys/modules/zfs/static_ccompile.h  -I/usr/src/sys/contrib/openzfs/include  -I/usr/src/sys/contrib/openzfs/include/os/freebsd  -I/usr/src/sys/contrib/openzfs/include/os/freebsd/spl  -I/usr/src/sys/contrib/openzfs/include/os/freebsd/zfs   -I/usr/src/sys/modules/zfs  -I/usr/src/sys/contrib/openzfs/module/zstd/include  -O2 -pipe  -fno-strict-aliasing  -g -nostdinc  -I. -I/usr/src/sys -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common    -MD  -MF.depend.opensolaris_atomic.o -MTopensolaris_atomic.o -fdebug-prefix-map=./machine=/usr/src/sys/i386/include -fdebug-prefix-map=./x86=/usr/src/sys/x86/include -mno-mmx -mno-sse -msoft-float -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -Wno-error-unused-function -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-address-of-packed-member -Wno-format-zero-length   -mno-aes -mno-avx  -std=iso9899:1999  -Wno-cast-qual  -Wno-duplicate-decl-specifier  -Wno-inline  -Wno-missing-braces  -Wno-missing-prototypes  -Wno-nested-externs  -Wno-parentheses  -Wno-pointer-arith  -Wno-redundant-decls  -Wno-strict-prototypes  -Wno-switch  -Wno-undef  -Wno-uninitialized  -Wno-unknown-pragmas  -Wno-unused  -include /usr/src/sys/contrib/openzfs/include/os/freebsd/spl/sys/ccompile.h  -I/usr/src/sys/cddl/contrib/opensolaris/uts/common  -I/usr/src/sys -I/usr/src/sys/cddl/compat/opensolaris -DBUILDING_ZFS -DHAVE_UIO_ZEROCOPY  -DWITH_NETDUMP -D__KERNEL__ -D_SYS_CONDVAR_H_ -DSMP  -DIN_FREEBSD_BASE -DHAVE_KSID -DBITS_PER_LONG=32 -Werror /usr/src/sys/cddl/contrib/opensolaris/common/atomic/i386/opensolaris_atomic.S
cc: error: no such file or directory: '/usr/src/sys/cddl/contrib/opensolaris/common/atomic/i386/opensolaris_atomic.S'
cc: error: no input files
*** [opensolaris_atomic.o] Error code 1

make[2]: stopped in /usr/obj/usr/src/i386.i386/sys/CrazyBook
…	…	…	…
When I try to use «options ZFS» instead of «device zfs», or both in kernel Config, I get the same Error.
The very Simplyfied Kernel Configuration, that Produces Described Error is attached to Message. Also attached Full «Make BuildKernel» OutPut (Big File) and Current System «kldstat -v», which shows that Modules «zfs_vdev», «zfs_zvol», «zfsctrl», «zfs», «opensolaris» are in «/boot/kernel/kernel» on 12.1, how to make the same in 13.0?

I saw file «/usr/src/sys/cddl/contrib/opensolaris/common/atomic/i386/opensolaris_atomic.S» in 12.2 Sources, but It was ReMoved in 13.0. I did not try to Copy this File from Previous Distributions, because IMHO this is a «Wrong Way».

Why am I worried about This?
	1. I Read that some Modules (for example, "IPFW", …) work Better (Theoretically, Faster) when they are Embedded into the FreeBSD Kernel, than if they are KLDLoaded during Boot.
	2. SomeTimes (on Some Small-Disk Systems) is useful to make «bzip2 --best --verbose "/boot/kernel/kernel"», but It does not Work on «/boot/kernel/*.ko».
	3. Finally, It worked in Previous Versions of the System — so, Should work in 13.0 too. And the Solution, most likely, is not very Labor-Intensive…

What is the Best Way to Do in Case to Make the Kernel, including «All Needed»? is it safe to use:
	device	zfs
	options	KSTACK_PAGES=4
	makeoptions	WITHOUT_MODULES+="opensolaris zfs"
or better:
	options	ZFS
I did not Find some Detailed Documentation about This Parameters and what is the Difference between «options ZFS» and «device zfs»…
Comment 1 CrazyMihey 2021-11-16 18:57:04 UTC
UnFortunately, NoOne answers My Question ↑, so I dicided to Try to Fix This Annoying Bug MySelf.
It seems, the Reason of Error just on «i386» Architecture is in Next Part of «/usr/src/Makefile.inc1»:
#──────────────────────────────────────────────────────────────────────────────────────────────────#
# 20191009  r353340  removal of opensolaris_atomic.S (also r353381)
.if ${MACHINE} != i386
.for f in opensolaris_atomic
.for m in opensolaris zfs
	@if [ -e "${KRNLOBJDIR}/${KERNCONF}/modules${SRCTOP}/sys/modules/${m}/.depend.${f}.o" ] && \
	    grep -q ${f}.S "${KRNLOBJDIR}/${KERNCONF}/modules${SRCTOP}/sys/modules/${m}/.depend.${f}.o"; then \
		echo "Removing stale dependencies for opensolaris_atomic"; \
		rm -f ${KRNLOBJDIR}/${KERNCONF}/modules${SRCTOP}/sys/modules/${m}/.depend.${f}.*; \
	fi
.endfor
.endfor
.endif
#──────────────────────────────────────────────────────────────────────────────────────────────────#
I will Try to Build a Simpliest Kernel Configuration is «/usr/src/sys/i386/conf/Virtual_ZFS»:
#──────────────────────────────────────────────────────────────────────────────────────────────────#
include	GENERIC			# За Основу берётся Общий Файл Конфигурации «GENERIC» для FreeBSD/AMD64.

ident	Virtual_ZFS		# Уникальное Идентификационное НаЗвание Ядра. Должно совпадать с Именем Этого Файла.

nooptions	ZFS
options		ZFS		# ПодДержка ZettaByte File System в Ядре ОС.
options		KSTACK_PAGES=4	# Количество Страниц Памяти, ВыДеляемых для Стэка Ядра на Каждый Процесс (Поток). Повышает Стабильность Работы ZFS на Архитектуре i386 (x86, 32 Бит). Для Архитектуры AMD64 (x86-64, 64 Бит) Рекомендуемое Значение: «5».
makeoptions	WITHOUT_MODULES+="crypto cryptodev aesni opensolaris zfs"	# Если Код Менеджера Томов/Файловой Системы ZFS ЛинкуетСя в Ядро Системы, то НеОбХодимОСть в ЗаГружаемых Модулях «/boot/kernel/opensolaris.ko» и «/boot/kernel/zfs.ko» ОтПадает.
#──────────────────────────────────────────────────────────────────────────────────────────────────#
Now I see Two Ways to ReEnAble Building i386-Kernel with BuiltIn ZFS Support in 13.0:
1. Get the File «opensolaris_atomic.S» from any Previous Release of FreeBSD Sources:
fetch --output="-" --verbose "HTTPS://DownLoad.FreeBSD.Org/ftp/releases/i386/12.2-RELEASE/src.txz" | tar --extract -f "-" --directory "/" --include "*[Oo][Pp][Ee][Nn][Ss][Oo][Ll][Aa][Rr][Ii][Ss]_[Aa][Tt][Oo][Mm][Ii][Cc].[Ss]" --fast-read --preserve-permissions --verbose
But It does not Compile with the following Error:
#──────────────────────────────────────────────────────────────────────────────────────────────────#
…
--- all_subdir_opensolaris ---
/usr/src/sys/contrib/openzfs/include/os/freebsd/spl/sys/ccompile.h:174:13: error: unexpected token in argument list
--- all_subdir_ossl ---
--- sha1-586.o ---
--- all_subdir_otus ---
--- device_if.h ---
--- all_subdir_opensolaris ---
typedef int bool_t;
            ^
/usr/src/sys/contrib/openzfs/include/os/freebsd/spl/sys/ccompile.h:175:13: error: unexpected token in argument list
typedef int enum_t;
            ^
*** [opensolaris_atomic.o] Error code 1

make[4]: stopped in /usr/src/sys/modules/opensolaris
…
#──────────────────────────────────────────────────────────────────────────────────────────────────#
Does It Really Can Atomic-Operate with 128-Bit Numbers on a 32-Bit CPU without Support of «CmpXChg16B»?
2. Completely throw out «opensolaris_atomic.S» from the i386-Sources Tree:
sed -E -i "~" "s~^([^#]*opensolaris_atomic.S.*)$~#\1~" "/usr/src/sys/conf/files.i386"
After that, Kernel Builds SuccessFully and even Runs, but I am not Sure, that It is a «Good Idea»:
File «opensolaris_atomic.S» looks like must Operate Much Faster, than «/usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_atomic.c», which uses Mutexes on Every Operation.
Can SomeOne from the KnowledgEable Suggest a More Correct Solution?
Comment 2 Graham Perrin 2023-10-01 14:53:13 UTC
Is this reproducible with releng/13.2 or releng/14.0?

(In reply to CrazyMihey from comment #1)

> UnFortunately, NoOne answers My Question ↑, so I dicided to Try to 
> Fix This Annoying Bug MySelf. …

I'm sorry, although (^Triage) please note that use of Bugzilla here is normally reserved for actionable bug reports. Mailing lists and the like are often better for questions and answers; <https://www.freebsd.org/support/>.