Created attachment 158169 [details] Simple 2 threaded program This was discussed somewhat in Bug 201149, but not fully resolved there, so I'm writing up this PR to keep track of the issue. Attaching to a program with multiple threads results in a duplicate listing like below: (gdb) info threads Id Target Id Frame 3 Thread 801406400 (LWP 101466 main) 0x0000000800b648ba in nanosleep () from /lib/libc.so.7 * 2 Thread 801406800 (LWP 100123 otherthread) 0x0000000800b648ba in nanosleep () from /lib/libc.so.7 * 1 Thread 801406800 (LWP 100123 otherthread) 0x0000000800b648ba in nanosleep () from /lib/libc.so.7 Attached the test program used for the above demonstration. I proposed a patch in Bug 201149, but it sounds like jhb@ has a more complete patch.
Sorry, I copied the wrong bug number; the previous PR I meant to reference is Bug 200001
I just updated my ports tree and built gdb 7.10 and this problem still seems to exist. Why was this closed?
This bug is not yet resolved and there is no easy solution for that. jhb@ is working (long term project) to rewrite gdb thread support for FreeBSD, with a cleaner and better implementation. There is no ETA, so it's to say to say when the new implementation will be available. For now, this bug entry should stay in OPEN status, but I've no right to change the status...
You have to specify a comment when changing the Status of a bug from Closed to Open.
I still have a patch for fbsd-threads.c that tries to make this better. It's not perfect but does seem to help in at least some cases.
Created attachment 163019 [details] gdb_dup_thread.patch
I made some recent progress on my alternate threads target. It has some rough edges still and does not support some of the more esoteric features of fbsd-threads.c such as the 'tsd' or 'signal' commands, but in my limited testing it was able to examine threads both live and in cores ok for both 32-bit and 64-bit processes. The URL below is a patch relative to GDB 7.10. I have not tried to integrate it into the port at all. You might be able to drop it in in place of 'extrapatch-threads' https://github.com/bsdjhb/gdb/compare/gdb-7.10-release...freebsd-threads.diff It could use some more widespread testing. Note that this target uses ptrace() directly so it should support 32-bit binaries (it did in my testing, though you can't run an i386 binary from gdb on amd64, you can only attach to a running one) and possibly Linux binaries in HEAD. It does not use libthread_db so it does not support user threads in libkse or libc_r, but threads in libthr should work fine. It is also less invasive for adding more GDB arch backends as all they need to do is support LWP IDs when fetching / storing registers.
I have now added the new thread target to the port as a new option. I turned the existing THREADS option into a radio button. You can find the diff to the port at https://reviews.freebsd.org/D4807 I have tested this new thread target but would really like to have more wide spread testing. My plan is to commit this thread support to GDB upstream (hopefully before 7.11) at which point the old thread support would probably be removed.
Created attachment 165188 [details] gdb_port_ptrace_threads.patch Adding the port patch directly since phabricator doesn't show the contents of new files apparently.
To be clear, I would like to commit the gdb_port_ptrace_threads.patch soon. I plan to upstream the new thread target for gdb 7.11 if I can make the deadline. That means we will want to switch to the new target for 7.11 in the next month or two. The sooner the new target can get wider testing the better.
Hi John, finally I've time to test the ptrace implementation. In phabricator I added a small comment/bugfix in the Makefile. I guess you're working directly on the git repo of gdb and unfortunately it creates some problems. I found two so far: * the biggest problem is that the binutils directory doesn't exist in the gdb tarball. gdb and binutils were somehow merged in a unique git repository, because they share several libraries, but they follow two separated release process. The conclusion is that no binutils directory is available and patch fail * amd64bsd-nat.c the patch you provide doesn't take count of the the patch present in the port affecting that file and the patch fails to apply. The second problem is trivial I can take care of it, but the first one is more relevant and I don't know how to solve it.
Hi John, could you please re-attach the patch here? In phabricator the patch is still empty... AFAIK, the elf format parser is somehow in the bfd directory (binary file descriptor library), the library shared between gdb and binutils. BTW, the thread name support is at the moment not so important.
Created attachment 165447 [details] gdb_port_ptrace_threads.patch This is the latest version of the port patch. I've built it fine and have run the new thread target this time. I did put back enough of the bits so that thread names in cores now work as well. It is also updated for the local patch in amd64bsd-nat.c.
Created attachment 165834 [details] gdb_port_ptrace_threads.patch I merged the ptrace() based thread support to GDB mainline today. It will be included in GDB 7.11. I've updated the port patch to match what was merged upstream (there were some minor changes made during the review and push upstream). I would really like to get this in and tested soon. Supporting the old thread target on 7.11 is going to be a bit of work (we'd basically have to disable the new one in the stock sources). Ideally the new one would work well enough we can just drop the old one for 7.11.
Apologies for the delay on my part. I've got gdb built and running with your latest patch and will be testing as much as I can over the next few days; anything that seems amiss I'll note here.
I've tested the patch and it works really good. The thread numeration seems more consistent and all previous features (like fork-on-follow and so on) seem to work. There's only one thing that I notice: New thread are called new process: [New process 2641, LWP 101562] [New process 2641, LWP 100881] [Switching to process 2641, LWP 101563] That could be misleading for users, because they are new threads, not processes. Do you think it's something that could be fixed? BTW, the patch could be immediately applied for a broader testing phase.
Created attachment 165924 [details] gdbtest producing unusual SIGSEGV I'm finding a few oddities ostensibly related to dlopen. The first involved gdb breaking at r_debug_state and reporting "ptrace: No such process.". I'll get back about this problem since I'm still trying to produce a simple test case to reproduce. While searching for the source of the above problem, I ran into another. I've attached a test program and test gdbscript. Do: make gdb -x gdbscript to run. This test is SEGVing some percentage of the time with the new gdb patch (I don't see this with current ports gdb). I'll poke around to see if I can figure it out, but I'm posting here in case you want to look in parallel.
(In reply to luca.pizzamiglio from comment #16) Mmm, I had noticed the message but thought I was matching the behavior on Linux. However, Linux's pid_to_str method only mentions the LWP and not the pid at all. I've posted a change to reword it as "LWP x of process y".
Created attachment 165926 [details] gdbtest producing unusual SIGSEGV Oops, my dlopen path was off in the example I posted (sorry). Here's a better one. make cd prog gdb -x gdbscript
Humm, just FYI, while gdb 7.10 crashes with this, gdb master (soon to be 7.11) runs the test fine. (Will work on trying it on 7.10 in a sec.)
Actually, I will need to retest gdb 7.11 as on 7.10 it doesn't crash every time. It is clearly a race of some sort. The sigsegv is "real" in that the kernel really thinks it has faulted. I think the issue might instead be a race with rtld and threads, but I'm still looking. When the sigsegv occurs, the non-faulting thread is buried inside of rtld trying to acquire a read lock on the global rtld_bind_lock: (gdb) r Starting program: /usr/home/john/work/git/gdb/gdbfork2/prog/gdbfork [New LWP 101324 of process 87794] [Switching to LWP 101324 of process 87794] Breakpoint 1, loadlib (arg=0x0) at gdbfork.c:37 37 sleep(1); Program received signal SIGSEGV, Segmentation fault. [Switching to LWP 100236 of process 87794] 0x00007fffffffdca0 in ?? () (gdb) info threads Id Target Id Frame 2 LWP 101324 of process 87794 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 * 1 LWP 100236 of process 87794 0x00007fffffffdca0 in ?? () (gdb) thread 2 [Switching to thread 2 (LWP 101324 of process 87794)] #0 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 37 RSYSCALL_ERR(_umtx_op) (gdb) where #0 _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 #1 0x00000008008293e8 in __thr_rwlock_rdlock (rwlock=0x800a41d80, flags=<optimized out>, tsp=<optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.c:277 #2 0x0000000800830a9c in _thr_rwlock_rdlock (flags=<optimized out>, tsp=<optimized out>, rwlock=<optimized out>, flags=<optimized out>, tsp=<optimized out>) at /usr/src/lib/libthr/thread/thr_umtx.h:196 #3 _thr_rtld_rlock_acquire (lock=0x800a41d80) at /usr/src/lib/libthr/thread/thr_rtld.c:123 #4 0x000000080060b2c2 in rlock_acquire (lock=0x80081e960 <rtld_locks>, lockstate=0x7fffdfffde78) at /usr/src/libexec/rtld-elf/rtld_lock.c:201 #5 0x0000000800604c8d in _rtld_bind (obj=0x800621000, reloff=48) at /usr/src/libexec/rtld-elf/rtld.c:697 #6 0x000000080060246d in _rtld_bind_start () at /usr/src/libexec/rtld-elf/amd64/rtld_start.S:121 #7 0x0000000000400818 in loadlib (arg=0x0) at gdbfork.c:37 #8 0x0000000800827855 in thread_start (curthread=0x801406800) at /usr/src/lib/libthr/thread/thr_create.c:288 #9 0x0000000000000000 in ?? () Backtrace stopped: Cannot access memory at address 0x7fffdfffe000 This is the state of the lock: (gdb) p/x *(struct urwlock *)$l $12 = {rw_state = 0xa0000000, rw_flags = 0x2, rw_blocked_readers = 0x1, rw_blocked_writers = 0x0, rw_spare = {0x0, 0x0, 0x0, 0x0}} This means the write lock must be held by the other thread. The registers for thread 1 (esp. rsp and rip) seem dubious, though %rax == r_debug. (gdb) p r_debug $13 = {r_version = 0, r_map = 0x800621220, r_brk = 0x800604950 <r_debug_state>, r_state = RT_ADD} Also, the test library isn't in the link map yet: (gdb) p *r_debug.r_map $15 = {l_addr = 0x400000 "\177ELF\002\001\001\t", l_name = 0x7fffffffefae "/usr/home/john/work/git/gdb/gdbfork2/prog/gdbfork", l_ld = 0x600a18, l_next = 0x800621620, l_prev = 0x0} (gdb) p *r_debug.r_map->l_next $16 = {l_addr = 0x80081f000 "\177ELF\002\001\001\t", l_name = 0x80061f080 "/lib/libthr.so.3", l_ld = 0x800a37420, l_next = 0x800621a20, l_prev = 0x800621220} (gdb) p *r_debug.r_map->l_next->l_next $17 = {l_addr = 0x800a44000 "\177ELF\002\001\001\t", l_name = 0x80061f0c0 "/lib/libc.so.7", l_ld = 0x800dc0238, l_next = 0x80081de48 <obj_rtld+544>, l_prev = 0x800621620} (gdb) p *r_debug.r_map->l_next->l_next->l_next $18 = {l_addr = 0x800600000 "\177ELF\002\001\001\t", l_name = 0x80061f000 "/libexec/ld-elf.so.1", l_ld = 0x80081d418, l_next = 0x0, l_prev = 0x800621a20} Hmm, in the thread that sigsegv'd: (gdb) x/gx $rsp 0x7fffffffd7a8: 0x0000000800608dfb (gdb) x/i 0x0000000800608dfb 0x800608dfb <dlopen_object+139>: mov 0x215106(%rip),%r12 # 0x80081df08 <obj_tail> ... (gdb) x/20i 0x0000000800608dfb - 20 0x800608de7 <dlopen_object+119>: decl -0x75(%rax) 0x800608dea <dlopen_object+122>: add $0x2147b9,%eax 0x800608def <dlopen_object+127>: movl $0x1,0x18(%rax) 0x800608df6 <dlopen_object+134>: callq 0x800604950 <r_debug_state> 0x800608dfb <dlopen_object+139>: mov 0x215106(%rip),%r12 # 0x80081df08 <obj_tail> So in fact it called r_debug_state, probably stopped in GDB, and then appears to have gone kablooey?
(In reply to John Baldwin from comment #21) Program received signal SIGSEGV, Segmentation fault. [Switching to LWP 100236 of process 87794] 0x00007fffffffdca0 in ?? () Is this a faulted thread ? If 0x00007fffffffdca0 is the faulted instruction address, then the fault is somewhat understandable. It is a stack, and definitely not the shared page. Since typical stack is non-executable, attempt to jump there faulted. Why the thread state was changed into that, I do not know. In fact, set sysctl machdep.uprintf_signal to 1 and see what is the pristine fault address as seen by kernel.
Yes, the signal is real, but yes, I think the question is how the thread ended up in that state: Breakpoint 1, loadlib (arg=0x0) at gdbfork.c:37 37 sleep(1); pid 39777 comm gdbfork: signal 5 err 0 code 2 type 10 addr 0x800604951 rsp 0x7fffffffe090 rip 0x800604951 <48 89 e5 5d c3 66 2e 0f> pid 39777 comm gdbfork: signal 5 err 0 code 2 type 10 addr 0x400813 rsp 0x7fffdfffdf90 rip 0x400813 <e8 4c fd ff ff 48 8d 3c> pid 39777 comm gdbfork: signal 5 err 0 code 2 type 10 addr 0x800604951 rsp 0x7fffffffe088 rip 0x800604951 <48 89 e5 5d c3 66 2e 0f> pid 39777 comm gdbfork: signal 11 err 15 code 2 type 12 addr 0x7fffffffe590 rsp 0x7fffffffe098 rip 0x7fffffffe590 <60 ea ff ff ff 7f 00 00> Program received signal SIGSEGV, Segmentation fault. [Switching to LWP 100105 of process 39777] 0x00007fffffffe590 in ?? ()
I wrote a python wrapper around some DTrace probes to report the registers changed by each call to PT_SETREGS in case gdb was doing something bizarre and trashing the register state: https://github.com/bsdjhb/kdbg/blob/master/dtrace/set_regs_diff.py However, I found no differences in a good vs bad run. :( Here's what I see on each run (it looks like gdb is probably just doing fixups after hitting a software breakpoint by moving %rip back after removing the breakpoint): set_regs() changes: tf_rip: 0x800604951 => 0x800604950 set_regs() changes: tf_rip: 0x800604951 => 0x800604950 set_regs() changes: tf_rip: 0x400812 => 0x400811 set_regs() changes: tf_rip: 0x800604951 => 0x800604950 FYI: (gdb) info break Num Type Disp Enb Address What 1 breakpoint keep y 0x0000000000400811 in loadlib at gdbfork.c:37 breakpoint already hit 1 time c
(In reply to John Baldwin from comment #24) Could you also track ptrace_set_pc(9) ? I would suspect a bug with PT_CONTINUE (or _SCE/SCX/SYSCALL) usage, where the addr argument passed might appear to be a stack pointer instead of 0/1/pc.
Created attachment 166037 [details] DTrace script for ptrace calls I'm wondering if what may be happening is that the push %rbp at the top of r_debug_state is being mistakenly executed twice. I'd also been playing with a DTrace script (attached). In a failing case, it looks like the faulting thread (thread 1) may be, due to interference from thread 2, be rewound twice to execute the actual instruction at the top of r_debug_state. I'll look again tomorrow and see if I can form a coherent explanation of how this could happen.
It seems likely that a double 'push' is at fault. If I break at its companion 'pop', a pass looks like: Breakpoint 1, r_debug_state (rd=0x800a40480, m=0x7fffffffe018) at /usr/src/libexec/rtld-elf/rtld.c:3512 3512 } (gdb) disassemble r_debug_state Dump of assembler code for function r_debug_state: 0x0000000800604570 <+0>: push %rbp 0x0000000800604571 <+1>: mov %rsp,%rbp => 0x0000000800604574 <+4>: pop %rbp 0x0000000800604575 <+5>: retq End of assembler dump. (gdb) p $rsp $1 = (void *) 0x7fffffffdfe0 (gdb) p $rbp $2 = (void *) 0x7fffffffdfe0 (gdb) x/2xg $rsp 0x7fffffffdfe0: 0x00007fffffffe4e0 0x0000000800608782 (gdb) c Continuing. Breakpoint 1, r_debug_state (rd=0x80061e800, m=0x18) at /usr/src/libexec/rtld-elf/rtld.c:3512 3512 } (gdb) c Continuing. [Inferior 1 (process 40137) exited normally] While a fail looks like: Breakpoint 1, r_debug_state (rd=0x800a40480, m=0x7fffffffe018) at /usr/src/libexec/rtld-elf/rtld.c:3512 3512 } (gdb) disassemble Dump of assembler code for function r_debug_state: 0x0000000800604570 <+0>: push %rbp 0x0000000800604571 <+1>: mov %rsp,%rbp => 0x0000000800604574 <+4>: pop %rbp 0x0000000800604575 <+5>: retq End of assembler dump. (gdb) p $rsp $1 = (void *) 0x7fffffffdfd8 (gdb) p $rbp $2 = (void *) 0x7fffffffdfd8 (gdb) x/2xg $rsp 0x7fffffffdfd8: 0x00007fffffffe4e0 0x00007fffffffe4e0 (gdb) c Continuing. Program received signal SIGSEGV, Segmentation fault. 0x00007fffffffe4e0 in ?? () I'm also been unable to get a SEGV with a script that disarms the 'push' and 'pop' like so: file ./gdbfork set pagination off break main commands set $x = &r_debug_state # Change everything except retq to a nop set {char}$x = 0x90 set {char}($x + 1) = 0x90 set {char}($x + 2) = 0x90 set {char}($x + 3) = 0x90 set {char}($x + 4)= 0x90 disassemble r_debug_state c end break loadlib commands c end r
So gdb's master branch triggers an assertion here: (top-gdb) l 5487 if (had_step_over_info) 5488 { 5489 /* If we're stepping over a breakpoint with all threads locked, 5490 then only the thread that was stepped should be reporting 5491 back an event. */ 5492 gdb_assert (ecs->event_thread->control.trap_expected); 5493 5494 if (ecs->event_thread->suspend.stop_signal == GDB_SIGNAL_TRAP) 5495 clear_step_over_info (); 5496 } This seems promising at least.
I don't have a fix yet, but I'm pretty sure this is the issue. GDB expects that the next event reported after a single step is from the thread that was single stepped. In the broken case, the new thread starts and reports its first stop before the stepping thread which confuses GDB. I suspect that the release branch doesn't have invariants enabled and GDB thus moves %rip back incorrectly causing the double push.
A few updates: 1) When compiled with LWP events (PT_LWP_EVENTS recently added to HEAD), this does not break. 2) I tried just disabling schedlock for the !PT_LWP_EVENTS case, but I still got an assertion failure. If this works it would probably be the simplest approach to take. 3) I have a WIP patch to try to suspend and "hide" new LWPs while single stepping until single stepping has finished but it exhibits some odd behavior in other test cases. So, I don't have a fix yet, but am looking still.
I believe that the update to 7.11 in the port (along with the workaround patch for Eric's test case while I work on the fuller solution) should resolve this.
Confirmed, GDB 7.11 fixes this bug.
(In reply to John Baldwin from comment #31) Thanks John, it looks ok for me as well. When I have a chance, I'll try to chase down any other curiosities I find.