Bug 200001 - devel/gdb: gdb 7.9 crashes on certain core files
Summary: devel/gdb: gdb 7.9 crashes on certain core files
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Tijl Coosemans
URL:
Keywords:
Depends on: 200638
Blocks:
  Show dependency treegraph
 
Reported: 2015-05-06 16:02 UTC by wjenkner
Modified: 2015-06-04 17:11 UTC (History)
5 users (show)

See Also:
linimon: maintainer-feedback? (luca.pizzamiglio)


Attachments
C source file (416 bytes, text/plain)
2015-05-06 16:02 UTC, wjenkner
no flags Details
C source (312 bytes, text/plain)
2015-05-07 13:08 UTC, wjenkner
no flags Details
clear_thread_list.patch (426 bytes, patch)
2015-05-11 20:20 UTC, John Baldwin
no flags Details | Diff
get_current_thread before thread enumeration (1.03 KB, patch)
2015-05-28 15:33 UTC, Eric Badger
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description wjenkner 2015-05-06 16:02:21 UTC
Created attachment 156437 [details]
C source file

Compile and run the attached foo.c, then examine the resulting core dump with gdb79 (please see the transcript below).  This results in gdb79 dumping core.
For comparison purposes, gdb782 has a glitch here (a duplicated backtrace for one thread) but it doesn't crash.

[1 ~]$ uname -a
FreeBSD xxx.yy 10.1-STABLE FreeBSD 10.1-STABLE #0 r280136: Mon Mar 16 18:59:32 CET 2015     adm@xxx.yy:/usr/obj/usr/src/sys/IZNOGOUD  amd64
[2 ~]$ cc -Wall -g -O0 -o foo foo.c -lthr
[3 ~]$ ./foo
Abort trap (core dumped)
[4 ~]$ gdb79 --core=foo.core ./foo
GNU gdb (GDB) 7.9 [GDB v7.9 for FreeBSD]
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd10.1".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./foo...done.
[New process 101209]
[New process 100339]
[New Thread 801406800 (LWP 101209)]
[New Thread 801406400 (LWP 100339)]
Core was generated by `foo'.
Program terminated with signal SIGABRT, Aborted.
Invalid selected thread.
(gdb) thread apply all bt

Thread 4 (Thread 801406400 (LWP 100339)):
#0  _nanosleep () at _nanosleep.S:3
#1  0x0000000800829b8c in __thr_nanosleep (time_to_sleep=0x7fffffffe4f8, 
    time_remaining=0x7fffffffe4e8) at /usr/src/lib/libthr/thread/thr_syscalls.c:269
#2  0x0000000800aa2ffb in __sleep (seconds=2147483647)
    at /usr/src/lib/libc/gen/sleep.c:60
#3  0x0000000000400952 in main () at foo.c:35
Segmentation fault (core dumped)
[5 ~]$ /tmp/usr/local/bin/gdb782 --core=foo.core ./foo
GNU gdb (GDB) 7.8.2 [GDB v7.8.2 for FreeBSD]
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd10.1".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./foo...done.
[New process 101209]
[New process 100339]
[New Thread 801406800 (LWP 101209)]
[New Thread 801406400 (LWP 100339)]
Core was generated by `foo'.
Program terminated with signal SIGABRT, Aborted.
#0  thr_kill () at thr_kill.S:3
3	RSYSCALL(thr_kill)
(gdb) thread apply all bt

Thread 4 (Thread 801406400 (LWP 100339)):
#0  _nanosleep () at _nanosleep.S:3
#1  0x0000000800829b8c in __thr_nanosleep (time_to_sleep=0x7fffffffe4f8, 
    time_remaining=0x7fffffffe4e8) at /usr/src/lib/libthr/thread/thr_syscalls.c:269
#2  0x0000000800aa2ffb in __sleep (seconds=2147483647)
    at /usr/src/lib/libc/gen/sleep.c:60
#3  0x0000000000400952 in main () at foo.c:35

Thread 3 (Thread 801406800 (LWP 101209)):
#0  thr_kill () at thr_kill.S:3
#1  0x0000000800b896c6 in __raise (s=6) at /usr/src/lib/libc/gen/raise.c:51
#2  0x0000000800b87ea9 in abort () at /usr/src/lib/libc/stdlib/abort.c:65
#3  0x00000000004008f0 in foo_thread (arg=0x0) at foo.c:19
#4  0x0000000800827775 in thread_start (curthread=0x801406800)
    at /usr/src/lib/libthr/thread/thr_create.c:288
#5  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdfffe000

Thread 1 (Thread 801406800 (LWP 101209)):
#0  thr_kill () at thr_kill.S:3
#1  0x0000000800b896c6 in __raise (s=6) at /usr/src/lib/libc/gen/raise.c:51
#2  0x0000000800b87ea9 in abort () at /usr/src/lib/libc/stdlib/abort.c:65
#3  0x00000000004008f0 in foo_thread (arg=0x0) at foo.c:19
#4  0x0000000800827775 in thread_start (curthread=0x801406800)
    at /usr/src/lib/libthr/thread/thr_create.c:288
#5  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdfffe000
(gdb) quit
[6 ~]$ gdb --core=foo.core ./foo
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Core was generated by `foo'.
Program terminated with signal 6, Aborted.
Reading symbols from /lib/libthr.so.3...Reading symbols from /usr/lib/debug//lib/libthr.so.3.debug...done.
done.
Loaded symbols for /lib/libthr.so.3
Reading symbols from /lib/libc.so.7...Reading symbols from /usr/lib/debug//lib/libc.so.7.debug...done.
done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /libexec/ld-elf.so.1...Reading symbols from /usr/lib/debug//libexec/ld-elf.so.1.debug...done.
done.
Loaded symbols for /libexec/ld-elf.so.1
#0  thr_kill () at thr_kill.S:3
3	RSYSCALL(thr_kill)
[New Thread 801406800 (LWP 101209/foo)]
[New Thread 801406400 (LWP 100339/foo)]
(gdb) thread apply all bt

Thread 2 (Thread 801406400 (LWP 100339/foo)):
#0  _nanosleep () at _nanosleep.S:3
#1  0x0000000800829b8c in __thr_nanosleep (time_to_sleep=0x7fffffffe4f8, 
    time_remaining=0x7fffffffe4e8) at /usr/src/lib/libthr/thread/thr_syscalls.c:269
#2  0x0000000800aa2ffb in __sleep () at /usr/src/lib/libc/gen/sleep.c:60
#3  0x0000000000400952 in main () at foo.c:35
Current language:  auto; currently asm

Thread 1 (Thread 801406800 (LWP 101209/foo)):
#0  thr_kill () at thr_kill.S:3
#1  0x0000000800b896c6 in __raise (s=<value optimized out>)
    at /usr/src/lib/libc/gen/raise.c:51
#2  0x0000000800b87ea9 in abort () at /usr/src/lib/libc/stdlib/abort.c:65
#3  0x00000000004008f0 in foo_thread (arg=0x0) at foo.c:19
#4  0x0000000800827775 in thread_start (curthread=0x801406800)
    at /usr/src/lib/libthr/thread/thr_create.c:288
#5  0x0000000000000000 in ?? ()
(gdb) [7 ~]$
Comment 1 wjenkner 2015-05-07 13:08:39 UTC
Created attachment 156468 [details]
C source

Simpler test case.
Comment 2 John Baldwin freebsd_committer freebsd_triage 2015-05-07 14:15:39 UTC
I was unable to reproduce on HEAD, will try stable/10 later today.  However, in my case I only got "New Thread" lines when parsing the core, I did not get both "New Process" and "New Thread" and I suspect that is related.  Did you build gdb79 from ports or install it from a package repository?  If the latter, was it from pkg.freebsd.org it from your own build?
Comment 3 luca.pizzamiglio 2015-05-07 16:16:43 UTC
Hi,

The big suspect here is the thread support and how the thread list is created when the libthr.so is loaded. 

There was an issue with gdb78, (solved in the revision 365578) and I guess that the fix is not able to handle the load of libthr.so when the process has already more thread (attach or core dump load of multithread programs)

In other words, if you run the application inside gdb, libthr.so is processed at the beginning, when only 1 thread is running and everything is fine.
gdb79 ./foo
tested, it works.

If you analyze the core dump or you dynamically attach gdb to a running process, gdb has inconsistent information about threads.
Comment 4 wjenkner 2015-05-07 16:49:19 UTC
(In reply to John Baldwin from comment #2)
I compiled it from ports (using portmaster).
Comment 5 wjenkner 2015-05-07 16:54:54 UTC
(In reply to luca.pizzamiglio from comment #3)
Running foo under gdb79 works for me as well.

As the transcript above shows, gdb782 --core=foo.core ./foo does not crash but it seems to produce a duplicate backtrace of one thread, somehow.
Comment 6 John Baldwin freebsd_committer freebsd_triage 2015-05-11 20:20:49 UTC
Created attachment 156667 [details]
clear_thread_list.patch

This is the workaround I've been testing.  It does fix this, but it's not clear to me what the most correct fix for this is.  Other libthread_db targets in gdb for Solaris and Linux just avoid attaching to core dumps entirely.  This patch makes the FreeBSD thread target throw away the threads enumerated by the default core dump target and re-enumerate them using the FreeBSD target directly.
Comment 7 Eric Badger 2015-05-28 15:33:52 UTC
Created attachment 157220 [details]
get_current_thread before thread enumeration
Comment 8 Eric Badger 2015-05-28 15:35:39 UTC
(In reply to Eric Badger from comment #7)

I think Luca's patch can work here, but the setting of ptid happens too late; when enumerating threads, the old thread isn't matched and so a new (duplicate) is created. Attached a possible patch that seems to handle correctly core files and the case of attaching to running processes with multiple threads.
Comment 9 John Baldwin freebsd_committer freebsd_triage 2015-05-28 15:38:06 UTC
I have given Luca a similar patch (albeit a bit larger) that generally does the same thing to avoid adding a double thread.  I think there are still issues with the resume method though that aren't easy to untangle.
Comment 10 luca.pizzamiglio 2015-06-04 13:14:47 UTC
Hi to all

The patch I gave to Eric is extracted from the John's one. I hopefully thought it could solve the attaching issue.
I'm going to integrate the previous submitted patch, that solves this issue, in the next PR I'm preparing.
Comment 11 commit-hook freebsd_committer 2015-06-04 17:09:10 UTC
A commit references this bug:

Author: tijl
Date: Thu Jun  4 17:08:27 UTC 2015
New revision: 388538
URL: https://svnweb.freebsd.org/changeset/ports/388538

Log:
  - Update to 7.9.1
  - Fix crash on some core files
  - Fix plist

  PR:		200001, 200011, 200638
  Submitted by:	luca.pizzamiglio@gmail.com (maintainer)

Changes:
  head/devel/gdb/Makefile
  head/devel/gdb/distinfo
  head/devel/gdb/files/fbsd-threads.c
  head/devel/gdb/pkg-plist