Bug 193593 - [dtrace] root@test3:/home/adrian/git/norse/dsniff # dtrace -n 'pid$target:libc*:flockfile:entry { @[execname, ustack()] = count(); } END { trunc(@, 10); }' -p 44592 dtrace: description 'pid$target:libc*:flockfile:entry ' matched 2 probes Segmentation faul
Summary: [dtrace] root@test3:/home/adrian/git/norse/dsniff # dtrace -n 'pid$target:lib...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: Normal Affects Only Me
Assignee: Mark Johnston
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-09-13 02:38 UTC by Adrian Chadd
Modified: 2018-02-19 01:41 UTC (History)
2 users (show)

See Also:


Attachments
dump file (126.15 KB, text/plain)
2014-11-05 12:08 UTC, Danilo Egea Gondolfo
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Adrian Chadd freebsd_committer 2014-09-13 02:38:39 UTC
When dtrac'ing a process that does a lot of fprintf()s:

root@test3:/home/adrian/git/norse/dsniff # dtrace -n 'pid$target:libc*:flockfile:entry { @[execname, ustack()] = count(); } END { trunc(@, 10); }' -p 44592
dtrace: description 'pid$target:libc*:flockfile:entry ' matched 2 probes
Segmentation fault (core dumped)

.. and the original process just vanishes.
Comment 1 Adrian Chadd freebsd_committer 2014-09-13 02:39:12 UTC
Grr!

FreeBSD test3 10.1-PRERELEASE FreeBSD 10.1-PRERELEASE #0 r270990M: Tue Sep  2 10:38:43 PDT 2014     adrian@test3:/home/adrian/work/freebsd/stable/10/obj/home/adrian/work/freebsd/stable/10/src/sys/NETMAP  amd64
Comment 2 Adrian Chadd freebsd_committer 2014-09-13 02:41:39 UTC
And, uselessly:

root@test3:/home/adrian/git/norse/dsniff # gdb762 /usr/sbin/dtrace ./dtrace.core
GNU gdb (GDB) 7.6.2 [GDB v7.6.2 for FreeBSD]
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd10.0".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/dtrace...(no debugging symbols found)...done.
[New process 100302]
[New process 100316]
[New Thread 802808c00 (LWP 100316)]
[New Thread 802806400 (LWP 100302)]
Core was generated by `dtrace'.
Program terminated with signal 11, Segmentation fault.
#0  0x000000080083149b in ?? () from /lib/libthr.so.3
(gdb) bt
#0  0x000000080083149b in ?? () from /lib/libthr.so.3
#1  0x0000000800837145 in pthread_cond_broadcast () from /lib/libthr.so.3
#2  0x0000000800a6c277 in dt_proc_continue () from /lib/libdtrace.so.2
#3  0x00000000004037df in ?? ()
#4  0x000000000040233f in ?? ()
#5  0x0000000800625000 in ?? ()
#6  0x0000000000000000 in ?? ()
(gdb)
Comment 3 Mark Johnston freebsd_committer 2014-09-13 03:56:32 UTC
Could you provide a copy of the core?

Does dtrace consistently crash when run this way? Does it happen immediately, or when detaching?
Comment 4 Adrian Chadd freebsd_committer 2014-09-13 04:01:43 UTC
* it crashes immediately
* it crashes consistently - the target process is doing a _lot_ of fprintf()'s into files.


-a
Comment 5 Mark Johnston freebsd_committer 2014-09-22 18:27:48 UTC
Still waiting on a copy of the core. A test case which reproduces the problem would be ok too. I haven't been able to reproduce it myself.
Comment 6 Danilo Egea Gondolfo freebsd_committer 2014-11-05 12:08:04 UTC
Created attachment 149063 [details]
dump file
Comment 7 Danilo Egea Gondolfo freebsd_committer 2014-11-05 12:08:34 UTC
+1

This command "dtrace -n 'pid$target:libc.so*:: { ustack(); }' -p 1234"
crashes the application 1234, segfault dtrace and if I run it more times the system panics.
Comment 8 Adrian Chadd freebsd_committer 2015-01-24 02:55:47 UTC
Just tried the latest stable/10 from today:

root@test3:/home/adrian/git/norse/dsniff # dtrace -n 'pid$target:libc*:bcopy:entry { @[ustack()] = count(); }' -p 2530
dtrace: description 'pid$target:libc*:bcopy:entry ' matched 1 probe
Segmentation fault (core dumped)
root@test3:/home/adrian/git/norse/dsniff # gdb78  `which dtrace` ./dtrace.core
GNU gdb (GDB) 7.8 [GDB v7.8 for FreeBSD]
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd10.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/dtrace...(no debugging symbols found)...done.
[New process 100188]
[New process 100333]
[New Thread 802808800 (LWP 100333)]
[New Thread 802806400 (LWP 100188)]
Core was generated by `dtrace'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000080083266b in ?? () from /lib/libthr.so.3
(gdb) bt
#0  0x000000080083266b in ?? () from /lib/libthr.so.3
#1  0x00000008008383c5 in pthread_cond_broadcast () from /lib/libthr.so.3
#2  0x0000000800a6d397 in dt_proc_continue () from /lib/libdtrace.so.2
#3  0x00000000004037bf in ?? ()
#4  0x000000000040231f in ?? ()
#5  0x0000000800627000 in ?? ()
#6  0x0000000000000000 in ?? ()
(gdb)
Comment 9 Mark Johnston freebsd_committer 2015-01-24 02:59:47 UTC
Ok, can you give me the backtraces of all the threads in dtrace? i.e. thread apply all bt
Comment 10 Adrian Chadd freebsd_committer 2015-01-24 03:04:15 UTC
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000080083266b in ?? () from /lib/libthr.so.3
(gdb) bt
#0  0x000000080083266b in ?? () from /lib/libthr.so.3
#1  0x00000008008383c5 in pthread_cond_broadcast () from /lib/libthr.so.3
#2  0x0000000800a6d397 in dt_proc_continue () from /lib/libdtrace.so.2
#3  0x00000000004037bf in ?? ()
#4  0x000000000040231f in ?? ()
#5  0x0000000800627000 in ?? ()
#6  0x0000000000000000 in ?? ()
(gdb) thread apply all bt

Thread 4 (Thread 802806400 (LWP 100188)):
#0  0x000000080083266b in ?? () from /lib/libthr.so.3
#1  0x00000008008383c5 in pthread_cond_broadcast () from /lib/libthr.so.3
#2  0x0000000800a6d397 in dt_proc_continue () from /lib/libdtrace.so.2
#3  0x00000000004037bf in ?? ()
#4  0x000000000040231f in ?? ()
#5  0x0000000800627000 in ?? ()
#6  0x0000000000000000 in ?? ()

Thread 3 (Thread 802808800 (LWP 100333)):
#0  0x0000000800839b4c in ?? () from /lib/libthr.so.3
#1  0x00000008008300ce in ?? () from /lib/libthr.so.3
#2  0x000000080083800e in ?? () from /lib/libthr.so.3
#3  0x0000000800a6dd9b in ?? () from /lib/libdtrace.so.2
#4  0x0000000800a6d7ee in ?? () from /lib/libdtrace.so.2
#5  0x000000080082e6e5 in ?? () from /lib/libthr.so.3
#6  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdfffe000

Thread 1 (Thread 802806400 (LWP 100188)):
#0  0x000000080083266b in ?? () from /lib/libthr.so.3
#1  0x00000008008383c5 in pthread_cond_broadcast () from /lib/libthr.so.3
#2  0x0000000800a6d397 in dt_proc_continue () from /lib/libdtrace.so.2
#3  0x00000000004037bf in ?? ()
#4  0x000000000040231f in ?? ()
#5  0x0000000800627000 in ?? ()
#6  0x0000000000000000 in ?? ()
Comment 11 Mark Johnston freebsd_committer 2015-01-24 03:09:05 UTC
These backtraces aren't very illuminating on their own...

Does the crash happen roughly when the target process exits? Or does the target process crash too? If it's the former, I suspect I know what the problem is - if so, can you try removing the ustack() and see if the problem keeps happening?
Comment 12 commit-hook freebsd_committer 2015-01-30 04:52:54 UTC
A commit references this bug:

Author: markj
Date: Fri Jan 30 04:51:59 UTC 2015
New revision: 277914
URL: https://svnweb.freebsd.org/changeset/base/277914

Log:
  In fasttrap_sigtrap(), use tdsendsignal() rather than tdksignal() to send
  SIGTRAP. The latter requires that its thread argument be non-NULL, but
  fasttrap_sigtrap() does not.

  PR:		193593
  MFC after:	1 week
  Reported by:	danilo

Changes:
  head/sys/cddl/contrib/opensolaris/uts/common/dtrace/fasttrap.c
Comment 13 Mark Johnston freebsd_committer 2015-01-30 05:06:54 UTC
(In reply to Danilo Egea Gondolfo from comment #7)

r277914 should address the panic you saw. It does not fix the application crashes though.
Comment 14 Danilo Egea Gondolfo freebsd_committer 2015-01-30 21:01:26 UTC
(In reply to Mark Johnston from comment #13)

I'm testing here, so far so good. And now I can see a lot of stack traces before the process segfault. When I reported the problem I couldn't see even the first one.

Thanks!
Comment 15 Glen Barber freebsd_committer 2015-07-08 18:32:07 UTC
To originators/assignees of this PR:

A commit to the tree references this PR, however the PR is still in a non-closed state.

Please review this PR and close as appropriate, or if closing the PR requires a merge to stable/10, please let re@ know as soon as possible.

Thank you.

Glen
Comment 16 Mark Linimon freebsd_committer freebsd_triage 2018-02-19 01:41:56 UTC
Feedback timeout (several years).