Bug 242798 - sysutils/e2fsprogs: Fails to build on FreeBSD 12.1-RELEASE-p1 powerpc 32 bit: Tests failed: f_inlinedata_dirblocks
Summary: sysutils/e2fsprogs: Fails to build on FreeBSD 12.1-RELEASE-p1 powerpc 32 bit:...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: powerpc Any
: --- Affects Some People
Assignee: Matthias Andree
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-12-22 09:54 UTC by canardo
Modified: 2020-02-17 22:18 UTC (History)
1 user (show)

See Also:
mandree: maintainer-feedback+
mandree: merge-quarterly+


Attachments
Log build sysutils/e2fsprogs on powerpc (37.29 KB, text/plain)
2019-12-22 09:55 UTC, canardo
no flags Details
build log file WITH_DEBUG (174.37 KB, text/plain)
2019-12-23 08:23 UTC, canardo
no flags Details
FreeBSD.bug.242798.e2fsprogs_v1.45.5_3-FULL.LOG.WITH.DEBUG.txt.zip (19.73 KB, application/zip)
2020-02-04 22:48 UTC, canardo
no flags Details
FreeBSD.bug.242798.e2fsprogs_v1.45.5_3-STACKTRACE (13.08 KB, text/plain)
2020-02-05 21:34 UTC, canardo
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description canardo 2019-12-22 09:54:17 UTC
Problem found when building sysutils/e2fsprogs on FreeBSD 12.1-RELEASE-p1 powerpc 32 bit, running on Apple Powerbook 17".
(e2fsprogs required to build sysutils/dsbmd)


# cd /usr/ports/sysutils/e2fsprogs 
# make -DBATCH install clean
===>  Building for e2fsprogs-1.45.3_2
.....
.....
gmake[1]: Entering directory '/usr/ports/sysutils/e2fsprogs/work/e2fsprogs-1.45.3/tests'
Running e2fsprogs test suite...
.....
f_messy_inode: bad file and directory acl pointers: ok
f_inlinedata_dirblocks: check inline dir as two dirent blocks: failed
i_qcow: create/convert raw/qcow2 images: ok
.....
d_fallocate_bigalloc: fallocate sparse files and big files with bigalloc: ok
     317 tests succeeded	       1 tests failed
Tests failed: f_inlinedata_dirblocks 
gmake[1]: *** [Makefile:396: test_post] Error 1
.....
.....
-Exit status is 0
+Entry '..' in Signal (11) SIGSEGV si_code=SEGV_MAPERR fault addr=0x2ff0
+Exit status is 8
*** Error code 1


NB: doc https://github.com/tytso/e2fsprogs/tree/master/tests
mentions:
"These images contain various forms of corrupted filesystem which
e2fsck will correct.  They are used as a regression test for e2fsck."

(see full log attached)
Comment 1 canardo 2019-12-22 09:55:29 UTC
Created attachment 210133 [details]
Log build sysutils/e2fsprogs on powerpc
Comment 2 Matthias Andree freebsd_committer 2019-12-22 16:37:35 UTC
Ooops. That's bad.

Unfortunately I don't have any powerpc stuff here, so I will be unable to reproduce things. 

I have updated the port to 1.45.4 yesterday, but am not sure if powerpc32 triggers CVE-2019-5094 here - see <http://e2fsprogs.sourceforge.net/e2fsprogs-release.html#1.45.4>

I know that the self-tests of 1.45.4 passed for i386, amd64 on 11.3/12.0/12.1 and for 12.1 on MIPS64 and ARM64.

So can you first retry with 1.45.4? It should have propagated to the mirrors now (for portsnap, or SVN)  If that still fails, can you try recompiling with WITH_DEBUG=yes and then grab a backtrace from the program that caught SIGSEGV? We may need to send that upstream to Ted Y. Ts'o if 1.45.4 still crashes.

Let me know if you need more detailed instructions.
Comment 3 canardo 2019-12-23 08:22:29 UTC
Via portsnap, I've updated version to e2fsprogs-1.45.4
Error still present.

So I run 
# make clean
# make WITH_DEBUG=yes install clean
(I guess it is the right command)

Please find attached new log WITH_DEBUG
Comment 4 canardo 2019-12-23 08:23:14 UTC
Created attachment 210171 [details]
build log file WITH_DEBUG
Comment 5 Matthias Andree freebsd_committer 2019-12-23 10:28:31 UTC
Comment on attachment 210171 [details]
build log file WITH_DEBUG

it's this part we need debugged, from the f_inlinedata_dirblocks test:
"SIGSEGV si_code=SEGV_MAPERR fault addr=0x2f6c" - can you get a stack backtrace of this crash, is there a core file you could get the backtrace from?
Comment 6 Matthias Andree freebsd_committer 2019-12-27 13:32:38 UTC
Hi Canardo,

as written by Ted in a previous e-mail (invisible to Bugzilla), 
please see if there is a e2fsck.core file under the work directories' e2fsprogs*/tests directory (possibly: find . -name '*core') and if you don't know how to get a stack trace from it, upload the file and the e2fsck/e2fsck program for me somewhere.

If there is a version that used to work, let us know which so we can see where between that version and v1.43.3 it broke.
Comment 7 Matthias Andree freebsd_committer 2019-12-27 19:25:10 UTC
Another approach would be adding USE_GCC=yes to the Makefile and see if a GCC-from-ports fares better than the stale GCC 4.2.1 from the base system, on powerpc.
Comment 8 Matthias Andree freebsd_committer 2019-12-28 17:15:09 UTC
Submitter wrote that he doesn't currently have access to his powerpc hardware and will see to it in 2020 when he regains access.

Instructions for getting a backtrace:
https://stackoverflow.com/questions/5745215/getting-stacktrace-from-core-dump

possibly adapted if you build the port from source, for bash and similar shells:
d=`make -V WRKSRC`
gdb ${d}/e2fsck/e2fsck `find ${d} -name e2fsck.core`

This works assuming that there is exactly one e2fsck.core file from the failing self-test. The compile should have been made WITH_DEBUG=yes (possibly re-build and re-install e2fsprogs-libuuid and e2fsprogs-libblkid also WITH_DEBUG=yes).
Comment 9 commit-hook freebsd_committer 2020-01-30 23:08:06 UTC
A commit references this bug:

Author: mandree
Date: Thu Jan 30 23:07:44 UTC 2020
New revision: 524661
URL: https://svnweb.freebsd.org/changeset/ports/524661

Log:
  sysutils/e2fsprogs: Try and enable backtrace() where possible.

  The configure script doesn't find backtrace() because it does not
  attempt linking against -lexecinfo -lelf.

  Note that currently backtrace() is broken at least on 12.1 amd64, and
  does not return any frames.  Filed separately as
  <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243746>

  PR:		242798 (which prompts this change, but is not fixed)

Changes:
  head/sysutils/e2fsprogs/Makefile
Comment 10 commit-hook freebsd_committer 2020-01-31 02:14:18 UTC
A commit references this bug:

Author: mandree
Date: Fri Jan 31 02:13:25 UTC 2020
New revision: 524662
URL: https://svnweb.freebsd.org/changeset/ports/524662

Log:
  Try to fix powerpc build, add LIBUNWIND option.

  We've had a USES+=compiler:c11 on powerpc64,
  see <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231570>
  extend and use it on all powerpc* in an attempt to fix PR 242798.

  If that doesn't help, add a LIBUNWIND option that makes e2fsck's
  signal catcher use libunwind instead of libexecinfo - FreeBSD's
  stack unwinder doesn't currently work from signal handlers (#243746),
  libunwind does and is supposed to work on powerpc and powerpc64,
  among others.

  PR:		242798
  Reported by:	canardo909@gmx.com

Changes:
  head/sysutils/e2fsprogs/Makefile
  head/sysutils/e2fsprogs/files/extrapatch-e2fsck_sigcatcher.c
Comment 11 Matthias Andree freebsd_committer 2020-01-31 02:15:54 UTC
Looking at a powerpc64 issue that got solved per
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231570

and in particular its attachment 197332 [details] here
https://bz-attachments.freebsd.org/attachment.cgi?id=197332

...this looks familiar.

I've taken a shot in the dark with extending our "use newer compiler on powerpc64" choice to powerpc*.

I have also added a LIBUNWIND option - best combined with WITH_DEBUG=yes to hopefully get a proper backtrace.


@Canardo, can you please update your ports tree, confirm you have e2fsprogs 1.45.5_2 (PORTREVISION=2 in its makefile) and then rebuild that (after make clean)? 

If it still fails self-tests, please enable the LIBUNWIND option (make config), and re-build and re-install the port with added WITH_DEBUG=yes so we get the new backtrace from the log.
Comment 12 Matthias Andree freebsd_committer 2020-02-01 09:38:51 UTC
Uh, possibly libunwind doesn't build on powerpc - possibly it can be made to build by removing the ONLY_FOR_ARCH... line in its makefile, but that's an experiment with uncertain result.
Comment 13 canardo 2020-02-01 19:16:57 UTC
(In reply to Matthias Andree from comment #11)


Tests results are not OK


/usr/ports/sysutils/e2fsprogs # more Makefile | grep "PORT"
PORTNAME=	e2fsprogs
PORTVERSION=	1.45.5
PORTREVISION?=	2
.....
.....
.....
===> Fetching all distfiles required by e2fsprogs-1.45.5_2 for building
===>  Extracting for e2fsprogs-1.45.5_2
=> SHA256 Checksum OK for e2fsprogs-1.45.5.tar.xz.
===>  Patching for e2fsprogs-1.45.5_2
===>  Applying FreeBSD patches for e2fsprogs-1.45.5_2
.....
.....
.....
cd /usr/ports/sysutils/e2fsprogs/work/e2fsprogs-1.45.5/e2fsck && /bin/rm -f -f e2fsck  && gmake e2fsck V=1  LIBS="../lib/libsupport.a ../lib/libext2fs.a ../lib/libcom_err.a   /usr/lib/libexecinfo.a /usr/lib/libelf.a  /usr/local/lib/libblkid.a /usr/local/lib/libuuid.a /usr/local/lib/libintl.a ../lib/libe2p.a"  LIBMAGIC=/usr/lib/libmagic.a\ -lz
gmake[2]: Entering directory '/usr/ports/sysutils/e2fsprogs/work/e2fsprogs-1.45.5/e2fsck'
gcc9   -fstack-protector-strong -Wl,-rpath=/usr/local/lib/gcc9 -L/usr/local/lib/gcc9  -L/usr/local/lib -Wl,--rpath -Wl,/usr/local/lib/e2fsprogs -Wl,-rpath-link,../lib -rdynamic -o e2fsck unix.o e2fsck.o super.o pass1.o pass1b.o pass2.o pass3.o pass4.o pass5.o journal.o badblocks.o util.o dirinfo.o dx_dirinfo.o ehandler.o problem.o message.o quota.o recovery.o region.o revoke.o ea_refcount.o rehash.o logfile.o sigcatcher.o  readahead.o extents.o ../lib/libsupport.a ../lib/libext2fs.a ../lib/libcom_err.a   /usr/lib/libexecinfo.a /usr/lib/libelf.a  /usr/local/lib/libblkid.a /usr/local/lib/libuuid.a /usr/local/lib/libintl.a ../lib/libe2p.a 
gmake[2]: Leaving directory '/usr/ports/sysutils/e2fsprogs/work/e2fsprogs-1.45.5/e2fsck'
===>  checking that e2fsck depends on no shared objects outside /lib: FAIL
===>  e2fsck depends on:
/usr/local/lib/gcc9/libgcc_s.so.1
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/sysutils/e2fsprogs
*** Error code 1




Then, I noticed I can't use libunwind

/usr/ports/devel/libunwind # make fetch-recursive
===> Fetching all distfiles for libunwind-20170615 and dependencies
===>  libunwind-20170615 is only for aarch64 amd64 i386, while you are running
powerpc.
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/devel/libunwind
*** Error code 1



And regarding the "ONLY_FOR_ARCH" parameter, I don't find it in the Makefile

/usr/ports/sysutils/e2fsprogs # more Makefile | grep -i "ARCH"
. if ! empty(ARCH:Mpowerpc*)
.if ${MASTERDIR} == ${.CURDIR} && ${ARCH} == "aarch64"
.if !empty(PORT_OPTIONS:MNOTESTS) && (${OPSYS} == FreeBSD) && (${OSVERSION} >= 1200000 || ((${ARCH} != i386) && (${ARCH} != amd64)))
Comment 14 commit-hook freebsd_committer 2020-02-03 00:35:24 UTC
A commit references this bug:

Author: mandree
Date: Mon Feb  3 00:34:16 UTC 2020
New revision: 525006
URL: https://svnweb.freebsd.org/changeset/ports/525006

Log:
  Be more careful about what and how we link.

  Only link what we need (--as-needed) to get unnecessary dependencies
  out of the executables.

  Be sure to properly link libc dynamically and the rest statically,
  especially libgcc (-static-libgcc, should fix powerpc* builds => [1])

  After two years, drop the "for now" LLD_UNSAFE=yes (r457508 by emaste@).

  As e2fsprogs dlopen()s libmagic these days, our LIBMAGIC override is
  useless.  Instead patch libmagic.so.1 -> libmagic.so (12.1 calls it
  libmagic.so.4; patch to lib/support/plausible.c).

  Refresh files/patch-misc__Makefile.in

  PR:		242798 [1]

Changes:
  head/sysutils/e2fsprogs/Makefile
  head/sysutils/e2fsprogs/files/patch-lib_support_plausible.c
  head/sysutils/e2fsprogs/files/patch-misc__Makefile.in
Comment 15 Matthias Andree freebsd_committer 2020-02-03 00:37:04 UTC
Canardo, could you try 1.45.5_3 (you'll need to "make config" and disable the LIBUNWIND option again)?  It adds -static-libgcc, i. e. should link libgcc statically so the "check[...] that e2fsck depends on no shared objects outside /lib" should now pass.
Comment 16 canardo 2020-02-04 22:47:12 UTC
(In reply to Matthias Andree from comment #15)

Unfortunately test fails again with version 1.45.5_3


/usr/ports/sysutils/e2fsprogs # cat Makefile | grep "PORT"
PORTNAME=	e2fsprogs
PORTVERSION=	1.45.5
PORTREVISION?=	3


# make config
everything left by default, LIBUNWIND is not selected


# make clean
# make WITH_DEBUG=yes install clean
===>  License GPLv2+ accepted by the user
===>   e2fsprogs-1.45.5_3 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by e2fsprogs-1.45.5_3 for building
===>  Extracting for e2fsprogs-1.45.5_3
.....
.....
     317 tests succeeded	       1 tests failed
Tests failed: f_inlinedata_dirblocks 
gmake[2]: *** [Makefile:395: test_post] Error 1
gmake[2]: Leaving directory '/usr/ports/sysutils/e2fsprogs/work/e2fsprogs-1.45.5/tests'
--- f_inlinedata_dirblocks/expect.1	2020-01-06 23:10:17.000000000 +0000
+++ f_inlinedata_dirblocks.1.log	2020-02-04 22:25:24.795749000 +0000
@@ -1,25 +1,4 @@
 Pass 1: Checking inodes, blocks, and sizes
 Pass 2: Checking directory structure
-Entry '..' in ??? (12) has invalid inode #: 1752440867.
-Clear? yes
.....
.....
-test_filesys: ***** FILE SYSTEM WAS MODIFIED *****
-test_filesys: 12/128 files (0.0% non-contiguous), 17/512 blocks
-Exit status is 1
+Entry '..' in Signal (11) SIGSEGV si_code=SEGV_MAPERR fault addr=0x2ed8
+Exit status is 8
--- f_inlinedata_dirblocks/expect.2	2020-01-06 23:10:17.000000000 +0000
+++ f_inlinedata_dirblocks.2.log	2020-02-04 22:25:24.842697000 +0000
@@ -1,7 +1,4 @@
 Pass 1: Checking inodes, blocks, and sizes
 Pass 2: Checking directory structure
-Pass 3: Checking directory connectivity
-Pass 4: Checking reference counts
-Pass 5: Checking group summary information
-test_filesys: 12/128 files (0.0% non-contiguous), 17/512 blocks
-Exit status is 0
+Entry '..' in Signal (11) SIGSEGV si_code=SEGV_MAPERR fault addr=0x2ed8
+Exit status is 8
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/sysutils/e2fsprogs
*** Error code 1



See full log WITH_DEBUG=yes attached
Comment 17 canardo 2020-02-04 22:48:13 UTC
Created attachment 211361 [details]
FreeBSD.bug.242798.e2fsprogs_v1.45.5_3-FULL.LOG.WITH.DEBUG.txt.zip
Comment 18 Matthias Andree freebsd_committer 2020-02-04 23:49:08 UTC
:-(

Then we'll need to do some more debugging along these lines:

    cd /usr/ports/sysutils/e2fsprogs
    make clean NOCLEANDEPENDS=1
    make WITH_DEBUG=yes configure # this stops before building so we can edit source code
    edit work/e2fsprogs-1.45.5/e2fsck/sigcatcher.c
    just above line 400, erase the entire line that reads "sigaction(SIGSEGV, &sa, 0);", this makes sure that the FreeBSD kernel with drop the core on SIGSEGV.
    make WITH_DEBUG=yes # this completes the build
    => now you should get a core file from the tests, work/e2fsprogs-1.45.5/tests/e2fsck.core
    gdb work/e2fsprogs-1.45.5/e2fsck/e2fsck work/e2fsprogs-1.45.5/tests/e2fsck.core
    inside gdb, optionally: set pagination off  (see how to disable pagination prompt in GDB)
    inside gdb: bt full
    inside gdb: bt
Comment 19 canardo 2020-02-05 21:33:36 UTC
(In reply to Matthias Andree from comment #18)

Core file generated successfully !

Please find attach the results of bt* commands

Full log and .core available if needed
Comment 20 canardo 2020-02-05 21:34:14 UTC
Created attachment 211391 [details]
FreeBSD.bug.242798.e2fsprogs_v1.45.5_3-STACKTRACE
Comment 21 commit-hook freebsd_committer 2020-02-17 21:55:19 UTC
A commit references this bug:

Author: mandree
Date: Mon Feb 17 21:54:24 UTC 2020
New revision: 526434
URL: https://svnweb.freebsd.org/changeset/ports/526434

Log:
  sysutils/e2fsprogs: Fix powerpc32 build

  PR:		242798
  Reported by:	canardo909@gmx.com
  Obtained from:	Theodore Y. Ts'o <tytso@mit.edu> (upstream maintainer)
  MFH:		2020Q1 (fixes build on powerpc)

Changes:
  head/sysutils/e2fsprogs/Makefile
  head/sysutils/e2fsprogs/files/patch-lib_ext2fs_swapfs.c
Comment 22 Matthias Andree freebsd_committer 2020-02-17 21:56:29 UTC
Fixed on head, MFH request sent to ports-secteam@ (automatically from the commit) - will invoke blanket approval.
Comment 23 commit-hook freebsd_committer 2020-02-17 22:15:21 UTC
A commit references this bug:

Author: mandree
Date: Mon Feb 17 22:14:33 UTC 2020
New revision: 526435
URL: https://svnweb.freebsd.org/changeset/ports/526435

Log:
  MFH: r526434

  sysutils/e2fsprogs: Fix powerpc32 build/self-tests
  libext2fs: avoid array buffer overruns caused by insane directory blocks

  PR:		242798
  Reported by:	canardo909@gmx.com
  Obtained from:	Theodore Y. Ts'o <tytso@mit.edu> (upstream maintainer)

  Approved by:	ports-secteam (blanket, fixing broken self-test by small patch)

Changes:
_U  branches/2020Q1/
  branches/2020Q1/sysutils/e2fsprogs/Makefile
  branches/2020Q1/sysutils/e2fsprogs/files/patch-lib_ext2fs_swapfs.c