Created attachment 184876 [details] test version https://github.com/xoreaxeaxeax/sandsifter Port undone yet, but it works. cd /usr/ports/security/sandsifter make cd /tmp/ports/usr/ports/security/sandsifter/work/sandsifter-dff63246fed84d90118441b8ba5b5d3bdd094427 (probably you path will different) sysctl security.bsd.map_at_zero=1 ./sifter.py --unk --dis --len --sync --tick -- -P1 -t --save
./sifter.py --unk --dis --len --sync --tick --save -- -P1 -t -j8
Thank you for your new port contribution. New ports should not be created without a maintainer (ports@f.o), could you please update the patch and additionally confirm that this port passes QA (portlint & poudriere in particular) please.
(In reply to rozhuk.im from comment #0) thanks alot for that; had to manually install "gmake" and patch both "sifter.py" and "summarize.py" to use "/usr/bin/env python" in their sehbang lines. Also, because I'm connected via SSH under a "konsole" terminal, I had to manually set "TERM=xterm-256color" as environment variable because sandsifter uses colorful curses. It saves its data to a sub-folder named "~" into ".sandsifter/sync", so I assume that it initially meant my home dir but has troubles resolving that alias. I'm currently running it on my other Ryzen system now...
(In reply to Nils Beyer from comment #3) hey cool, I got a kernel panic: ------------------------------------------------------------------------------ capetown.renzel.net dumped core - see /var/crash/vmcore.0 Tue Aug 1 13:57:11 CEST 2017 FreeBSD capetown.renzel.net 11.1-RELEASE FreeBSD 11.1-RELEASE #7 r321628M: Mon Jul 31 09:13:21 CEST 2017 root@capetown.renzel.net:/usr/obj/usr/src/sys/GENERIC amd64 panic: tdsendsignal(): invalid signal 0 GNU gdb (GDB) 7.12.1 [GDB v7.12.1 for FreeBSD] Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd11.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel...Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...done. done. Unread portion of the kernel message buffer: panic: tdsendsignal(): invalid signal 0 cpuid = 3 KDB: stack backtrace: #0 0xffffffff80aada97 at kdb_backtrace+0x67 #1 0xffffffff80a6bb76 at vpanic+0x186 #2 0xffffffff80a6b9e3 at panic+0x43 #3 0xffffffff80a71bbd at tdsendsignal+0xcbd #4 0xffffffff80a70be4 at trapsignal+0x184 #5 0xffffffff80edf3cd at trap+0x58d #6 0xffffffff80ec3671 at calltrap+0x8 Uptime: 5h3m50s Dumping 903 out of 16282 MB:..2%..11%..22%..31%..41%..52%..61%..71%..82%..91% Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done. done. Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. done. Reading symbols from /boot/kernel/uhid.ko...Reading symbols from /usr/lib/debug//boot/kernel/uhid.ko.debug...done. done. Reading symbols from /boot/kernel/pflog.ko...Reading symbols from /usr/lib/debug//boot/kernel/pflog.ko.debug...done. done. Reading symbols from /boot/kernel/pf.ko...Reading symbols from /usr/lib/debug//boot/kernel/pf.ko.debug...done. done. __curthread () at ./machine/pcpu.h:222 222 __asm("movq %%gs:%1,%0" : "=r" (td) (kgdb) #0 __curthread () at ./machine/pcpu.h:222 #1 doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:298 #2 0xffffffff80a6b6f1 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:366 #3 0xffffffff80a6bbb0 in vpanic (fmt=<optimized out>, ap=0xfffffe0466890780) at /usr/src/sys/kern/kern_shutdown.c:759 #4 0xffffffff80a6b9e3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:690 #5 0xffffffff80a71bbd in tdsendsignal (p=0xfffff80044340000, td=0xfffff8004433b560, sig=<optimized out>, ksi=<unavailable>) at /usr/src/sys/kern/kern_sig.c:2137 #6 0xffffffff80a70be4 in trapsignal (td=<optimized out>, ksi=<optimized out>) at /usr/src/sys/kern/kern_sig.c:2021 #7 0xffffffff80edf3cd in trap (frame=0xfffffe0466890ac0) at /usr/src/sys/amd64/amd64/trap.c:578 #8 <signal handler called> #9 0x000000080121e000 in ?? () Backtrace stopped: Cannot access memory at address 0x866800 (kgdb) ------------------------------------------------------------------------------
(In reply to Nils Beyer from comment #4) I should probably add, that this happened on an AMD Ryzen 1700 system (stock)...
(In reply to Nils Beyer from comment #4) and another kernel panic; same reason "tdsendsignal(): invalid signal 0" - if you want to crash your system, then this tool is absolutely able to do that... ;-)
This crash looks like race condition in kernel. Ask kib to fix it. On my system I got reboot (boot loop die to asrock) after few hours. And now it run this test 5+ hrs.
(In reply to rozhuk.im from comment #7) panicked on an Intel system, too - so you're right and it is a kernel bug; new bugzilla opened for that: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221151 > On my system I got reboot (boot loop die to asrock) after few hours. I don't understand that sentence; it crashed because of "boot loop die to ASRock"? What died there? So no kernel panic?
I suspect that my asrock taichi goes to boot loop (bios init fail in phase '00' until reset/poweroff) but other vendors mobo do reset. I have no coredumps, but I do not check is it correct tuned.
(In reply to Nils Beyer from comment #8) kernel panic seems to be fixed by a patch in the mentioned bug report. But, the "injector" process exits with signal 12 (non-existent system call): pid 1598 (injector), uid 0: exited on signal 12 Last line of the "sandsifter.py"'s output to stdout is: 26cd920dfff70000 1 8 5 2 (26cd920dfff700000000000000000000) Rozhuk, how to you manage to have it running for 5h and longer? What FreeBSD version do you use? I have no idea how to proceed on my side. BTW: that tool is not mentioned to be run over SSH; very, very slow then and too much output after...
Created attachment 184923 [details] coredump I got coredump for process only at the end of scanning: #0 0x0000000800968931 in pthread_sigmask () from /lib/libthr.so.3 (gdb) bt #0 0x0000000800968931 in pthread_sigmask () from /lib/libthr.so.3 #1 0x0000000800967908 in pthread_getspecific () from /lib/libthr.so.3 #2 0x00000008009677c4 in pthread_getspecific () from /lib/libthr.so.3 #3 0x0000000800970029 in pthread_attr_getaffinity_np () from /lib/libthr.so.3 #4 0x000000080096a308 in pthread_mutex_destroy () from /lib/libthr.so.3 #5 0x0000000800969906 in pthread_mutex_lock () from /lib/libthr.so.3 #6 0x0000000000401cb5 in sync_fflush () #7 0x0000000000403a8e in give_result () #8 0x0000000000404492 in main () And same errors I got some times in another apps. Not sure, may be this starts after upgrade to ryzen. FreeBSD rimwks 11.1-STABLE FreeBSD 11.1-STABLE #0 r321518M: Wed Jul 26 17:09:26 MSK 2017 root@rimwks:/usr/obj/usr/src/sys/RIMWKSx64 amd64 I have no idea why and what syscall missed in your system. And why my system not crash without kib patch.
Konstantin, can you please see coredump? May be this related with ryzen instability too, because I got same error some times in other apps, but I have no private patches and I dont remember this errors on intel.
Created attachment 184924 [details] core dump and executable program of "injector" with signal 12 Core dumps is a good idea. GDB's "bt" doesn't say much to me, though: ----------------------------------------------------------------------------- Core was generated by `./injector -P1 -t -j16 -i 26cd920dfff700000000000000000000 -t -R -0 -s 241886859'. Program terminated with signal 12, Bad system call. Reading symbols from /lib/libthr.so.3...Reading symbols from /usr/lib/debug//lib/libthr.so.3.debug...done. done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /lib/libc.so.7...Reading symbols from /usr/lib/debug//lib/libc.so.7.debug...done. done. Loaded symbols for /lib/libc.so.7 Reading symbols from /libexec/ld-elf.so.1...Reading symbols from /usr/lib/debug//libexec/ld-elf.so.1.debug...done. done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x0000000000402efd in resume () (gdb) bt #0 0x0000000000402efd in resume () #1 0x0000000000404540 in main () -----------------------------------------------------------------------------
Sorry, I dont know how to see syscall number, may be it logged into /var/log/messages? I try run via ssh and via su - no difference in speed: 9500-65000 per/sec depend on instructions set. I run with -j 1 - IMHO thats why I got no crash. Now I run with -j 15 ./sifter.py --unk --dis --len --sync --tick -- -P1 -j 15 -l 1 and quickly (few seconds - few minutes) get exit code 10: #0 0x0000000800967db1 in pthread_getspecific () from /lib/libthr.so.3 #1 <signal handler called> #2 0x000000080121e000 in ?? () #3 0x0000000000000000 in ?? () Program terminated with signal 6, Aborted. #0 0x0000000800c4731a in thr_kill () from /lib/libc.so.7 (gdb) bt #0 0x0000000800c4731a in thr_kill () from /lib/libc.so.7 #1 0x0000000800c472e4 in raise () from /lib/libc.so.7 #2 0x0000000800c47259 in abort () from /lib/libc.so.7 #3 0x00000008009703fa in pthread_attr_getaffinity_np () from /lib/libthr.so.3 #4 0x000000080096b225 in pthread_mutex_unlock () from /lib/libthr.so.3 #5 0x000000080096ad3f in pthread_mutex_unlock () from /lib/libthr.so.3 #6 0x0000000000401cea in sync_fflush () #7 0x0000000000403a8e in give_result () #8 0x0000000000404492 in main () on intel machine same errors.
Thank you everyone. Please: - Use bug 221151 for discussion/analysis/resolution of the kernel panic issue. - Use attachments (instead of comments) for long bodies of text like backtraces, configuration/log files, etc. If landing this port depends on bug 221151 please add it to the 'Depends On' field here. Pending updated patch and QA as per comment 2
Created attachment 186552 [details] improved port submission, run-tested
Rozhuk, do you want to maintain this port? If not, I can do it.
Ok, take it. I dont know where files should be installed. :(
The injector program still crashes with signal 12: from signal(3): > 12 SIGSYS create core image non-existent system call invoked So it looks like these "26 cd 92 xx xx xx" instructions, which disassemble to "es int 26", invoke this somehow.
As kib@ explained to me on IRC, interrupt 0x92 is used for DTrace return which is hooked up to the kernel by default. A similar error can happen with interrupt 0x93 if Xen is enabled. I will add those to the blacklist in injector.c and see if the program survives on my laptop.
(In reply to Rene Ladan from comment #20) It does survive this when adding int 0x92 and int 0x93 to the blacklist. The injector does trigger another bug(?) in libthr: r: 64 0924050e000000... or dword ptr fs:[rax + 0xe], esp ( 8) r: ( 8) sigtrap 2 ffffffff 64 0924050e000000 Fatal error 'mutex 0x800797000 own 0x0 is not on list 0x0 0x0' at line 138 in file /usr/src-vanilla/lib/libthr/thread/thr_mutex.c (errno = 0) where the first line is the last instruction tested. I think however it is fine to commit the port, so that other people can test it on their hardware and see what other bugs are lurking.
A commit references this bug: Author: rene Date: Sat Sep 30 15:13:33 UTC 2017 New revision: 450997 URL: https://svnweb.freebsd.org/changeset/ports/450997 Log: The sandsifter audits x86 processors for hidden instructions and hardware bugs, by systematically generating machine code to search through a processor's instruction set, and monitoring execution for anomalies. Sandsifter has uncovered secret processor instructions from every major vendor; ubiquitous software bugs in disassemblers, assemblers, and emulators; flaws in enterprise hypervisors; and both benign and security-critical hardware bugs in x86 chips. WWW: https://github.com/xoreaxeaxeax/sandsifter PR: 221132 Submitted by: rozhuk.im AT gmail.com Reviewed by: swills Changes: head/security/Makefile head/security/sandsifter/ head/security/sandsifter/Makefile head/security/sandsifter/distinfo head/security/sandsifter/files/ head/security/sandsifter/files/patch-Makefile head/security/sandsifter/files/patch-injector.c head/security/sandsifter/files/patch-sifter.py head/security/sandsifter/pkg-descr head/security/sandsifter/pkg-message head/security/sandsifter/pkg-plist