Bug 110655
| Summary: | [threads] 32 bit threaded applications crash on amd64 SMP kernel. | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Martin M. Mladenov <fm> |
| Component: | amd64 | Assignee: | freebsd-amd64 (Nobody) <amd64> |
| Status: | Closed FIXED | ||
| Severity: | Affects Only Me | ||
| Priority: | Normal | ||
| Version: | 6.2-RELEASE | ||
| Hardware: | Any | ||
| OS: | Any | ||
This problem also affects a non-SMP RELENG_6 kernel from 30th January. -- Peter Jeremy I can confirm this problem for 6.2-RELEASE-p2 amd64 UP system, for both libpthread and libthr. i386 simple test programs die with the following symptoms: [libpthread] program dies with sigsegv. What's interesting, when run under ktrace my very short program sometimes completes successfully and sometime crashes with the same symptoms. There is one strange line in ktrace: 66443 thr_test CALL kse_create(0x804b50c,0x1) 66443 thr_test RET kse_create 0 66443 thr_test CALL __sysctl(0xffffd7b8,0x2,0xffffd7c0,0xffffd7b4,0x2809c63c,0x18) 66443 thr_test RET __sysctl 0 66443 thr_test CALL __sysctl(0xffffd7c0,0x3,0xffffd86c,0xffffd870,0,0) 66443 thr_test RET __sysctl 0 66443 thr_test CALL break(0x805c000) 66443 thr_test RET break 0 66443 thr_test CALL mmap(0xffbfc000,0x101000,0x3,0x400,0xffffffff,0,0,0) 66443 thr_test RET mmap -4210688/0xffbfc000 66443 thr_test CALL mprotect(0xffbfc000,0x1000,0) 66443 thr_test RET mprotect 0 66443 thr_test CALL clock_gettime(0,0x8051f5c) 66443 thr_test RET clock_gettime 0 66443 thr_test CALL fstat(0x1,0xffcfc7c0) 66443 thr_test RET fstat 0 ================= 66443 thr_test RET fork 0 ================= 66443 thr_test PSIG SIGSEGV SIG_DFL 66443 thr_test NAMI "thr_test.core" [libthr] the problem here seems to be more obvious, but unexpected (at least for me): 69654 thr_test CALL sysarch(0xa,0xffffd700) 69654 thr_test RET sysarch 0 69654 thr_test CALL sigprocmask(0x3,0xffffd730,0xffffd720) 69654 thr_test RET sigprocmask 0 69654 thr_test CALL sigaction(0x20,0xffffd6d0,0) 69654 thr_test RET sigaction 0 69654 thr_test CALL sigprocmask(0x3,0xffffd720,0) 69654 thr_test RET sigprocmask 0 69654 thr_test CALL sigprocmask(0x1,0x28070a20,0xffffd760) 69654 thr_test RET sigprocmask 0 69654 thr_test CALL sigprocmask(0x3,0x28070a30,0) 69654 thr_test RET sigprocmask 0 69654 thr_test CALL _umtx_op 69654 thr_test RET _umtx_op -1 errno 78 Function not implemented 69654 thr_test PSIG SIGSYS SIG_DFL 69654 thr_test NAMI "thr_test.core" -- Andriy Gapon Andriy Gapon <avg@icyb.net.ua> wrote: > There is one strange line in ktrace: > [...] > 66443 thr_test RET fork 0 http://lists.freebsd.org/pipermail/freebsd-threads/2007-February/ 003858.html According to this post upcalls from KSE are reported as returns from fork. Thank you for the pointer, that explains it. In this view I think that the most probable cause of the crash is binary incompatibility between KSE structures in amd64 kernel and i386 userland, which is not so unexpected, unfortunately. -- Andriy Gapon Would it be appropriate to follow up on this with the KSE folks or the i386-on-amd64 folks? (Or is there some lucky individual around who knows about both?) This bug is causing a lot of problems for us, and we're happy to do whatever we can to help fix it, but both subsystems are pretty much opaque to me, so I am at a loss as to how we can help. Thanks, Jeff We are still having this problem on 6.2-RELEASE-p4. I have confirmed that using the example source code above with the thr library also does not work: $ cc -m32 -B/usr/lib32 -lthr -o crash32 crash32.c $ ./crash32 Bad system call(coredump) $ cc -lthr -o crash32 crash32.c $ ./crash32 Thread. $ Thanks, Jeff -- Jeff Wheelhouse jdw@wheelhouse.org Kris Kennaway wrote:
> -m32 doesn't DTRT on FreeBSD, can you confirm that this problem exists
> with a binary compiled by an i386 compiler?
I can.
$ uname -m
i386
$ cc -o crash32-thr crash32.c -lthr
$ cc -o crash32-pthread crash32.c -pthread
$ md5 crash32-thr crash32-pthread
MD5 (crash32-thr) = 91aea395dd2c5798d7cba9a216c4c227
MD5 (crash32-pthread) = a590f1c97e5ec11ddc94fa3d7f4bf167
$ ./crash32-thr
Thread.
$ ./crash32-pthread
Thread.
$
$ uname -m
amd64
$ md5 crash32-thr crash32-pthread
MD5 (crash32-thr) = 91aea395dd2c5798d7cba9a216c4c227
MD5 (crash32-pthread) = a590f1c97e5ec11ddc94fa3d7f4bf167
$ ./crash32-thr
Bad system call(coredump)
$ ./crash32-pthread
Memory fault(coredump)
$
Thanks,
Jeff
On Wed, Jun 06, 2007 at 05:11:25PM -0400, Jeff Wheelhouse wrote:
> Kris Kennaway wrote:
> >-m32 doesn't DTRT on FreeBSD, can you confirm that this problem exists
> >with a binary compiled by an i386 compiler?
>
> I can.
>
> $ uname -m
> i386
> $ cc -o crash32-thr crash32.c -lthr
> $ cc -o crash32-pthread crash32.c -pthread
> $ md5 crash32-thr crash32-pthread
> MD5 (crash32-thr) = 91aea395dd2c5798d7cba9a216c4c227
> MD5 (crash32-pthread) = a590f1c97e5ec11ddc94fa3d7f4bf167
> $ ./crash32-thr
> Thread.
> $ ./crash32-pthread
> Thread.
> $
>
> $ uname -m
> amd64
> $ md5 crash32-thr crash32-pthread
> MD5 (crash32-thr) = 91aea395dd2c5798d7cba9a216c4c227
> MD5 (crash32-pthread) = a590f1c97e5ec11ddc94fa3d7f4bf167
> $ ./crash32-thr
> Bad system call(coredump)
> $ ./crash32-pthread
> Memory fault(coredump)
> $
OK, that's useful to know. I ran into a problem myself that looked
similar but wanted to check first (I didn't isolate it to libthr yet).
Kris
6.3 should have freebsd32 compat for libthr and 7.0 should have freebsd32 compat for both libthr and libkse. Can you retest on newer OSs? -- John Baldwin It now works for me on FreeBSD 7.1 (GENERIC kernel, which includes options SMP, but single cpu): $ uname -m i386 $ cc -o crash32-thr crash32.c -lthr $ cc -o crash32-pthread crash32.c -pthread $ md5 crash32-thr crash32-pthread MD5 (crash32-thr) = 30b58238379b4c3496413f22863c2e86 MD5 (crash32-pthread) = 734e11117fbfd63efb06376cf74430d5 $ ./crash32-thr Thread. $ ./crash32-pthread Thread. $ uname -m amd64 $ md5 crash32-thr crash32-pthread MD5 (crash32-thr) = 30b58238379b4c3496413f22863c2e86 MD5 (crash32-pthread) = 734e11117fbfd63efb06376cf74430d5 $ ./crash32-thr Thread. $ ./crash32-pthread Thread. Can we close this ticket? State Changed From-To: open->closed This is fixed in 6.3 (libthr only) and 7.0+ (libthr + libkse). |
32 bit applications using the posix threads library crash when calling pthread_create on an SMP amd64 kernel. This happens with existing binaries, built natively on a x86 machine, as well as binaries built with cc -m32 on the amd64 system. A backtrace for the example in the next section yields: Core was generated by `pthread-crash'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/lib32/libpthread.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib32/libpthread.so.2 Reading symbols from /usr/lib32/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib32/libc.so.6 Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x0804d100 in ?? () [New Thread 0x805301408053200 (runnable)] [New Thread 0x2809f1c808053000 (runnable)] Cannot get thread info: generic error (gdb) bt #0 0x0804d100 in ?? () This has been reproduced on two machines with completely different hardware. Fix: Non known. How-To-Repeat: The problem can be reproduced by the following code (pthread_crash.c): #include <stdio.h> #include <pthread.h> void *thread(void *data) { puts("Thread."); return NULL; } int main() { pthread_t pth; void *pv; pthread_create(&pth,NULL,thread,NULL); pthread_join(pth,&pv); return 0; } The code is compiled with: cc -m32 -B/usr/lib32 -pthread -o pthread-crash pthread-crash.c This code runs fine when compiled to 64 bit.