Created attachment 248338 [details] list of differing artifacts On stable/13 cperciva discovered a number of artifacts produced by buildworld that differ between native (e.g. i386 buildworld in i386 jail) and cross (e.g. i386 buildworld on amd64 with TARGET/TARGET_ARCH set) builds. PR276960 describes one example. Other cases are different stack layouts or instruction ordering, pointing to nondeterminism in Clang. This example is a disassembly of mkfs.o, responsible for the difference in makefs: - objdump --line-numbers --disassemble --demangle --reloc --no-show-raw-insn --section=.text {} @@ -24590,15 +24590,15 @@ movl $0x0, 0x42ca00 movl $0x0, 0x42cccc movl $0x1, 0x42cec0 movl $0x1, 0x42cec4 xorl %eax, %eax movl $0x2000, %ecx # imm = 0x2000 jmp 0x41e7fa <ffs_mkfs+0x79a> + movl %ebx, 0x2c(%esp) - movl %ebx, 0x30(%esp) movl $0x19540119, 0x42ced0 # imm = 0x19540119 movl $0x0, 0x42cd60 movl $0x10000, 0x42cd5c # imm = 0x10000 movl 0x4(%esp), %eax movl %eax, %ecx shrl $0x3, %ecx movl %ecx, %edi @@ -24611,15 +24611,15 @@ movl $0x10000, %ecx # imm = 0x10000 movl 0x8(%esp), %edx cmpl $0x1, 0x70(%edx) movl %esi, 0x20(%esp) movl %edi, (%esp) jne 0x41e7f6 <ffs_mkfs+0x796> orb $0x2, 0x42ce94 + movl 0x2c(%esp), %ebx - movl 0x30(%esp), %ebx movl 0x14(%esp), %edx addl %ecx, %edx adcl %eax, %ebx addl $0x2000, %edx # imm = 0x2000 adcl $0x0, %ebx movl 0xc(%esp), %esi movl %esi, %eax @@ -24749,57 +24749,57 @@ leal (%ebx,%ecx), %edx leal (%eax,%edx), %ecx decl %ecx movl %ecx, %eax xorl %edx, %edx divl %ebx subl %edx, %ecx + movl %ecx, 0x2c(%esp) - movl %ecx, 0x3c(%esp) movl %ebx, 0x28(%esp) movl 0xc(%esp), %eax addl %ebx, %eax decl %eax movl %eax, 0x38(%esp) movl 0x42ca28, %eax addl %eax, %eax leal (%eax,%eax,2), %eax movl %eax, 0x34(%esp) movl 0x42c9a4, %eax addl $-0x8, %eax + movl %eax, 0x48(%esp) + movl 0x42c9a8, %eax movl %eax, 0x40(%esp) - movl 0x42c9a8, %eax - movl %eax, 0x44(%esp) negl %eax + movl %eax, 0x30(%esp) - movl %eax, 0x2c(%esp) movl %edi, (%esp) leal -0x1(%edi), %eax movl %eax, 0x4(%esp) movl 0x42ce98, %eax + movl %eax, 0x3c(%esp) - movl %eax, 0x48(%esp) leal (,%eax,4), %eax movl %eax, 0x50(%esp) movl %esi, %edx movl 0x14(%esp), %ecx
Looking at the difference in boot1.efi there's an extra instruction in the cross-built case --- i386-on-amd64/boot/boot1.efi +++ i386-on-i386/boot/boot1.efi ├── objdump │ @@ -55,15 +55,15 @@ │ Entry d 0000000000000000 00000000 Delay Import Directory │ Entry e 0000000000000000 00000000 CLR Runtime Header │ Entry f 0000000000000000 00000000 Reserved │ │ │ Sections: │ Idx Name Size VMA Type │ - 0 .text 000121ba 00001000 TEXT │ + 0 .text 000121b6 00001000 TEXT │ 1 .data 0000a538 00014000 DATA │ 2 .sdata 00000024 0001f000 DATA ... │ - 221c: movl %edi, 0x28(%esp) │ - 2220: movl %eax, (%esp) │ - 2223: movl %ecx, 0x4(%esp) │ - 2227: je 0x2252 <.text+0x1252> │ - 2229: movl -0x1ec4(%ebx), %ecx ... │ + 221c: movl %eax, (%esp) │ + 221f: movl %ecx, 0x4(%esp) │ + 2223: je 0x224e <.text+0x124e> │ + 2225: movl -0x1ec4(%ebx), %ecx
An initial analysis seems to indicate that with clang 18, I can get bit-identical output for e.g. mkfs.o and a few other select .o files that I found, when comparing between "cross" and "native" compiler output. With clang 17 I see the same type of difference, making it likely that some stack alignment setting leaks in from the host architecture. It is going to be very time consuming to figure out the exact upstream commit(s) that fixed this. For now I am doing 13.3-RELEASE-p1 buildworlds with clang 18.1.3 (as it is in main right now) as host compiler, both in "cross" and "native" mode. I will report back when I have some results there.
I did the analysis again on 15-CURRENT with clang 18.1.5, and this is now the list, when excluding host binaries (i.e. those that file(1) shows as ELF 64-bit binaries on a amd64 host): usr.cross/src/i386.i386/cddl/lib/libzfs/libzfs.a usr.cross/src/i386.i386/cddl/lib/libzfs/libzfs.so usr.cross/src/i386.i386/cddl/lib/libzfs/libzfs.so.4 usr.cross/src/i386.i386/cddl/lib/libzfs/libzfs.so.4.debug usr.cross/src/i386.i386/cddl/lib/libzfs/libzfs.so.4.full usr.cross/src/i386.i386/cddl/lib/libzfs/zfs_fletcher_sse.o usr.cross/src/i386.i386/cddl/lib/libzfs/zfs_fletcher_sse.pico usr.cross/src/i386.i386/cddl/lib/libzpool/libzpool.a usr.cross/src/i386.i386/cddl/lib/libzpool/libzpool.so usr.cross/src/i386.i386/cddl/lib/libzpool/libzpool.so.2 usr.cross/src/i386.i386/cddl/lib/libzpool/libzpool.so.2.debug usr.cross/src/i386.i386/cddl/lib/libzpool/libzpool.so.2.full usr.cross/src/i386.i386/cddl/lib/libzpool/zfs_fletcher_sse.o usr.cross/src/i386.i386/cddl/lib/libzpool/zfs_fletcher_sse.pico usr.cross/src/i386.i386/lib/clang/liblldb/Plugins/Process/FreeBSD/NativeRegisterContextFreeBSD_x86_64.o usr.cross/src/i386.i386/lib/clang/liblldb/liblldb.a usr.cross/src/i386.i386/lib/libarchive/archive_write_set_format_7zip.pico usr.cross/src/i386.i386/lib/libarchive/libarchive.so usr.cross/src/i386.i386/lib/libarchive/libarchive.so.7 usr.cross/src/i386.i386/lib/libarchive/libarchive.so.7.debug usr.cross/src/i386.i386/lib/libarchive/libarchive.so.7.full usr.cross/src/i386.i386/rescue/rescue/bectl.lo usr.cross/src/i386.i386/rescue/rescue/rescue usr.cross/src/i386.i386/rescue/rescue/zdb.lo usr.cross/src/i386.i386/rescue/rescue/zfs.lo usr.cross/src/i386.i386/rescue/rescue/zfsbootcfg.lo usr.cross/src/i386.i386/rescue/rescue/zpool.lo usr.cross/src/i386.i386/sbin/ipf/ipmon/ipmon usr.cross/src/i386.i386/sbin/ipf/ipmon/ipmon.debug usr.cross/src/i386.i386/sbin/ipf/ipmon/ipmon.full usr.cross/src/i386.i386/sbin/ipf/ipmon/ipmon.o usr.cross/src/i386.i386/stand/ficl/softcore.c usr.cross/src/i386.i386/stand/i386/loader_4th/loader_4th usr.cross/src/i386.i386/stand/i386/loader_4th/loader_4th.bin usr.cross/src/i386.i386/stand/i386/loader_4th/loader_4th.sym usr.cross/src/i386.i386/stand/i386/loader_4th/vers.c usr.cross/src/i386.i386/stand/i386/loader_4th/vers.o usr.cross/src/i386.i386/stand/i386/loader_lua/loader_lua usr.cross/src/i386.i386/stand/i386/loader_lua/loader_lua.bin usr.cross/src/i386.i386/stand/i386/loader_lua/loader_lua.sym usr.cross/src/i386.i386/stand/i386/loader_lua/vers.c usr.cross/src/i386.i386/stand/i386/loader_lua/vers.o usr.cross/src/i386.i386/stand/i386/loader_simp/loader_simp usr.cross/src/i386.i386/stand/i386/loader_simp/loader_simp.bin usr.cross/src/i386.i386/stand/i386/loader_simp/loader_simp.sym usr.cross/src/i386.i386/stand/i386/loader_simp/vers.c usr.cross/src/i386.i386/stand/i386/loader_simp/vers.o usr.cross/src/i386.i386/stand/i386/pxeldr/loader usr.cross/src/i386.i386/stand/i386/pxeldr/pxeboot usr.cross/src/i386.i386/toolchain-metadata.mk usr.cross/src/i386.i386/usr.bin/clang/lld/ELF/SyntheticSections.o usr.cross/src/i386.i386/usr.bin/clang/lld/ld.lld usr.cross/src/i386.i386/usr.bin/clang/lld/ld.lld.debug usr.cross/src/i386.i386/usr.bin/clang/lld/ld.lld.full usr.cross/src/i386.i386/usr.bin/clang/lldb-server/lldb-server usr.cross/src/i386.i386/usr.bin/clang/lldb-server/lldb-server.debug usr.cross/src/i386.i386/usr.bin/clang/lldb-server/lldb-server.full Note that the differences in .a or .so files, or executables, is due to one or more object files differing. Also, the stand/i386/loader files differ because vers.c contains a build date and hostname, but I assume that would go away if you use WITH_REPRODUCIBLE_BUILD? So the list of interesting files can be reduced to (grouped by library/program): usr.cross/src/i386.i386/cddl/lib/libzfs/zfs_fletcher_sse.o usr.cross/src/i386.i386/cddl/lib/libzfs/zfs_fletcher_sse.pico usr.cross/src/i386.i386/cddl/lib/libzpool/zfs_fletcher_sse.o usr.cross/src/i386.i386/cddl/lib/libzpool/zfs_fletcher_sse.pico usr.cross/src/i386.i386/lib/clang/liblldb/Plugins/Process/FreeBSD/NativeRegisterContextFreeBSD_x86_64.o usr.cross/src/i386.i386/lib/libarchive/archive_write_set_format_7zip.pico usr.cross/src/i386.i386/rescue/rescue/bectl.lo usr.cross/src/i386.i386/rescue/rescue/zdb.lo usr.cross/src/i386.i386/rescue/rescue/zfs.lo usr.cross/src/i386.i386/rescue/rescue/zfsbootcfg.lo usr.cross/src/i386.i386/rescue/rescue/zpool.lo usr.cross/src/i386.i386/sbin/ipf/ipmon/ipmon.o usr.cross/src/i386.i386/usr.bin/clang/lld/ELF/SyntheticSections.o Of these, most don't have differences in their assembly, but readelf shows that due to slightly larger or smaller sections, various offsets are different. The left overs are: usr.cross/src/i386.i386/lib/clang/liblldb/Plugins/Process/FreeBSD/NativeRegisterContextFreeBSD_x86_64.o usr.cross/src/i386.i386/lib/libarchive/archive_write_set_format_7zip.pico usr.cross/src/i386.i386/usr.bin/clang/lld/ELF/SyntheticSections.o I'm going to spend some time looking at the ways these get compiled.