Bug 221132 - [NEW PORT] security/sandsifter: x86 processor fuzzer
Summary: [NEW PORT] security/sandsifter: x86 processor fuzzer
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Rene Ladan
URL: https://github.com/xoreaxeaxeax/sands...
Keywords: feature, needs-patch, needs-qa
Depends on:
Blocks:
 
Reported: 2017-08-01 00:47 UTC by Ivan Rozhuk
Modified: 2017-09-30 15:13 UTC (History)
5 users (show)

See Also:


Attachments
test version (7.43 KB, patch)
2017-08-01 00:47 UTC, Ivan Rozhuk
no flags Details | Diff
coredump (851.63 KB, application/octet-stream)
2017-08-01 18:48 UTC, Ivan Rozhuk
no flags Details
core dump and executable program of "injector" with signal 12 (423.43 KB, application/x-xz)
2017-08-01 19:09 UTC, Nils Beyer
no flags Details
improved port submission, run-tested (9.22 KB, text/plain)
2017-09-19 18:36 UTC, Rene Ladan
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ivan Rozhuk 2017-08-01 00:47:42 UTC
Created attachment 184876 [details]
test version

https://github.com/xoreaxeaxeax/sandsifter

Port undone yet, but it works.

cd /usr/ports/security/sandsifter
make
cd /tmp/ports/usr/ports/security/sandsifter/work/sandsifter-dff63246fed84d90118441b8ba5b5d3bdd094427
(probably you path will different)

sysctl security.bsd.map_at_zero=1

./sifter.py --unk --dis --len --sync --tick -- -P1 -t --save
Comment 1 Ivan Rozhuk 2017-08-01 00:57:31 UTC
./sifter.py --unk --dis --len --sync --tick --save -- -P1 -t -j8
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2017-08-01 10:09:07 UTC
Thank you for your new port contribution. 

New ports should not be created without a maintainer (ports@f.o), could you please update the patch and additionally confirm that this port passes QA (portlint & poudriere in particular) please.
Comment 3 Nils Beyer 2017-08-01 11:25:33 UTC
(In reply to rozhuk.im from comment #0)

thanks alot for that; had to manually install "gmake" and patch both "sifter.py" and "summarize.py" to use "/usr/bin/env python" in their sehbang lines.

Also, because I'm connected via SSH under a "konsole" terminal, I had to manually set "TERM=xterm-256color" as environment variable because sandsifter uses colorful curses.

It saves its data to a sub-folder named "~" into ".sandsifter/sync", so I assume that it initially meant my home dir but has troubles resolving that alias.

I'm currently running it on my other Ryzen system now...
Comment 4 Nils Beyer 2017-08-01 12:07:35 UTC
(In reply to Nils Beyer from comment #3)

hey cool, I got a kernel panic:
------------------------------------------------------------------------------
capetown.renzel.net dumped core - see /var/crash/vmcore.0

Tue Aug  1 13:57:11 CEST 2017

FreeBSD capetown.renzel.net 11.1-RELEASE FreeBSD 11.1-RELEASE #7 r321628M: Mon Jul 31 09:13:21 CEST 2017     root@capetown.renzel.net:/usr/obj/usr/src/sys/GENERIC  amd64

panic: tdsendsignal(): invalid signal 0

GNU gdb (GDB) 7.12.1 [GDB v7.12.1 for FreeBSD]
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd11.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...done.
done.

Unread portion of the kernel message buffer:
panic: tdsendsignal(): invalid signal 0
cpuid = 3
KDB: stack backtrace:
#0 0xffffffff80aada97 at kdb_backtrace+0x67
#1 0xffffffff80a6bb76 at vpanic+0x186
#2 0xffffffff80a6b9e3 at panic+0x43
#3 0xffffffff80a71bbd at tdsendsignal+0xcbd
#4 0xffffffff80a70be4 at trapsignal+0x184
#5 0xffffffff80edf3cd at trap+0x58d
#6 0xffffffff80ec3671 at calltrap+0x8
Uptime: 5h3m50s
Dumping 903 out of 16282 MB:..2%..11%..22%..31%..41%..52%..61%..71%..82%..91%

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Reading symbols from /boot/kernel/uhid.ko...Reading symbols from /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
done.
Reading symbols from /boot/kernel/pflog.ko...Reading symbols from /usr/lib/debug//boot/kernel/pflog.ko.debug...done.
done.
Reading symbols from /boot/kernel/pf.ko...Reading symbols from /usr/lib/debug//boot/kernel/pf.ko.debug...done.
done.
__curthread () at ./machine/pcpu.h:222
222             __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) #0  __curthread () at ./machine/pcpu.h:222
#1  doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:298
#2  0xffffffff80a6b6f1 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:366
#3  0xffffffff80a6bbb0 in vpanic (fmt=<optimized out>, ap=0xfffffe0466890780)
    at /usr/src/sys/kern/kern_shutdown.c:759
#4  0xffffffff80a6b9e3 in panic (fmt=<unavailable>)
    at /usr/src/sys/kern/kern_shutdown.c:690
#5  0xffffffff80a71bbd in tdsendsignal (p=0xfffff80044340000, 
    td=0xfffff8004433b560, sig=<optimized out>, ksi=<unavailable>)
    at /usr/src/sys/kern/kern_sig.c:2137
#6  0xffffffff80a70be4 in trapsignal (td=<optimized out>, ksi=<optimized out>)
    at /usr/src/sys/kern/kern_sig.c:2021
#7  0xffffffff80edf3cd in trap (frame=0xfffffe0466890ac0)
    at /usr/src/sys/amd64/amd64/trap.c:578
#8  <signal handler called>
#9  0x000000080121e000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x866800
(kgdb) 
------------------------------------------------------------------------------
Comment 5 Nils Beyer 2017-08-01 12:08:06 UTC
(In reply to Nils Beyer from comment #4)

I should probably add, that this happened on an AMD Ryzen 1700 system (stock)...
Comment 6 Nils Beyer 2017-08-01 12:54:52 UTC
(In reply to Nils Beyer from comment #4)

and another kernel panic; same reason "tdsendsignal(): invalid signal 0" - if you want to crash your system, then this tool is absolutely able to do that... ;-)
Comment 7 Ivan Rozhuk 2017-08-01 13:17:47 UTC
This crash looks like race condition in kernel.
Ask kib to fix it.

On my system I got reboot (boot loop die to asrock) after few hours.
And now it run this test 5+ hrs.
Comment 8 Nils Beyer 2017-08-01 14:17:13 UTC
(In reply to rozhuk.im from comment #7)

panicked on an Intel system, too - so you're right and it is a kernel bug; new bugzilla opened for that:

    https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221151


> On my system I got reboot (boot loop die to asrock) after few hours.

I don't understand that sentence; it crashed because of "boot loop die to ASRock"? What died there? So no kernel panic?
Comment 9 Ivan Rozhuk 2017-08-01 15:02:43 UTC
I suspect that my asrock taichi goes to boot loop (bios init fail in phase '00' until reset/poweroff) but other vendors mobo do reset.

I have no coredumps, but I do not check is it correct tuned.
Comment 10 Nils Beyer 2017-08-01 16:38:58 UTC
(In reply to Nils Beyer from comment #8)

kernel panic seems to be fixed by a patch in the mentioned bug report. But, the "injector" process exits with signal 12 (non-existent system call):

    pid 1598 (injector), uid 0: exited on signal 12

Last line of the "sandsifter.py"'s output to stdout is:

    26cd920dfff70000  1  8  5  2 (26cd920dfff700000000000000000000)

Rozhuk, how to you manage to have it running for 5h and longer? What FreeBSD version do you use? I have no idea how to proceed on my side.

BTW: that tool is not mentioned to be run over SSH; very, very slow then and too much output after...
Comment 11 Ivan Rozhuk 2017-08-01 18:48:18 UTC
Created attachment 184923 [details]
coredump

I got coredump for process only at the end of scanning:

#0  0x0000000800968931 in pthread_sigmask () from /lib/libthr.so.3
(gdb) bt
#0  0x0000000800968931 in pthread_sigmask () from /lib/libthr.so.3
#1  0x0000000800967908 in pthread_getspecific () from /lib/libthr.so.3
#2  0x00000008009677c4 in pthread_getspecific () from /lib/libthr.so.3
#3  0x0000000800970029 in pthread_attr_getaffinity_np () from /lib/libthr.so.3
#4  0x000000080096a308 in pthread_mutex_destroy () from /lib/libthr.so.3
#5  0x0000000800969906 in pthread_mutex_lock () from /lib/libthr.so.3
#6  0x0000000000401cb5 in sync_fflush ()
#7  0x0000000000403a8e in give_result ()
#8  0x0000000000404492 in main ()

And same errors I got some times in another apps.
Not sure, may be this starts after upgrade to ryzen.

FreeBSD rimwks 11.1-STABLE FreeBSD 11.1-STABLE #0 r321518M: Wed Jul 26 17:09:26 MSK 2017     root@rimwks:/usr/obj/usr/src/sys/RIMWKSx64  amd64

I have no idea why and what syscall missed in your system.
And why my system not crash without kib patch.
Comment 12 Ivan Rozhuk 2017-08-01 18:51:31 UTC
Konstantin, can you please see coredump?
May be this related with ryzen instability too, because I got same error some times in other apps, but I have no private patches and I dont remember this errors on intel.
Comment 13 Nils Beyer 2017-08-01 19:09:41 UTC
Created attachment 184924 [details]
core dump and executable program of "injector" with signal 12

Core dumps is a good idea. GDB's "bt" doesn't say much to me, though:
-----------------------------------------------------------------------------
Core was generated by `./injector -P1 -t -j16 -i 26cd920dfff700000000000000000000 -t -R -0 -s 241886859'.
Program terminated with signal 12, Bad system call.
Reading symbols from /lib/libthr.so.3...Reading symbols from /usr/lib/debug//lib/libthr.so.3.debug...done.
done.
Loaded symbols for /lib/libthr.so.3
Reading symbols from /lib/libc.so.7...Reading symbols from /usr/lib/debug//lib/libc.so.7.debug...done.
done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /libexec/ld-elf.so.1...Reading symbols from /usr/lib/debug//libexec/ld-elf.so.1.debug...done.
done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x0000000000402efd in resume ()
(gdb) bt
#0  0x0000000000402efd in resume ()
#1  0x0000000000404540 in main ()
-----------------------------------------------------------------------------
Comment 14 Ivan Rozhuk 2017-08-02 00:21:54 UTC
Sorry, I dont know how to see syscall number, may be it logged into /var/log/messages?

I try run via ssh and via su - no difference in speed: 9500-65000 per/sec depend on instructions set.

I run with -j 1 - IMHO thats why I got no crash.
Now I run with -j 15
./sifter.py --unk --dis --len --sync --tick -- -P1 -j 15 -l 1
and quickly (few seconds - few minutes) get exit code 10:

#0  0x0000000800967db1 in pthread_getspecific () from /lib/libthr.so.3
#1  <signal handler called>
#2  0x000000080121e000 in ?? ()
#3  0x0000000000000000 in ?? ()

Program terminated with signal 6, Aborted.
#0  0x0000000800c4731a in thr_kill () from /lib/libc.so.7
(gdb) bt
#0  0x0000000800c4731a in thr_kill () from /lib/libc.so.7
#1  0x0000000800c472e4 in raise () from /lib/libc.so.7
#2  0x0000000800c47259 in abort () from /lib/libc.so.7
#3  0x00000008009703fa in pthread_attr_getaffinity_np () from /lib/libthr.so.3
#4  0x000000080096b225 in pthread_mutex_unlock () from /lib/libthr.so.3
#5  0x000000080096ad3f in pthread_mutex_unlock () from /lib/libthr.so.3
#6  0x0000000000401cea in sync_fflush ()
#7  0x0000000000403a8e in give_result ()
#8  0x0000000000404492 in main ()

on intel machine same errors.
Comment 15 Kubilay Kocak freebsd_committer freebsd_triage 2017-08-02 04:42:46 UTC
Thank you everyone. Please:

- Use bug 221151 for discussion/analysis/resolution of the kernel panic issue.
- Use attachments (instead of comments) for long bodies of text like backtraces, configuration/log files, etc.

If landing this port depends on bug 221151 please add it to the 'Depends On' field here.

Pending updated patch and QA as per comment 2
Comment 16 Rene Ladan freebsd_committer freebsd_triage 2017-09-19 18:36:06 UTC
Created attachment 186552 [details]
improved port submission, run-tested
Comment 17 Rene Ladan freebsd_committer freebsd_triage 2017-09-19 18:37:14 UTC
Rozhuk, do you want to maintain this port? If not, I can do it.
Comment 18 Ivan Rozhuk 2017-09-19 19:34:42 UTC
Ok, take it.
I dont know where files should be installed. :(
Comment 19 Rene Ladan freebsd_committer freebsd_triage 2017-09-22 12:40:43 UTC
The injector program still crashes with signal 12:

from signal(3):
> 12    SIGSYS           create core image    non-existent system call invoked

So it looks like these "26 cd 92 xx xx xx" instructions, which disassemble to "es int 26", invoke this somehow.
Comment 20 Rene Ladan freebsd_committer freebsd_triage 2017-09-22 13:30:15 UTC
As kib@ explained to me on IRC, interrupt 0x92 is used for DTrace return which is hooked up to the kernel by default. A similar error can happen with interrupt 0x93 if Xen is enabled. I will add those to the blacklist in injector.c and see if the program survives on my laptop.
Comment 21 Rene Ladan freebsd_committer freebsd_triage 2017-09-30 15:11:49 UTC
(In reply to Rene Ladan from comment #20)

It does survive this when adding int 0x92 and int 0x93 to the blacklist.

The injector does trigger another bug(?) in libthr:

r: 64 0924050e000000...         or dword ptr fs:[rax + 0xe], esp                 ( 8)   r: ( 8) sigtrap   2  ffffffff 64 0924050e000000

Fatal error 'mutex 0x800797000 own 0x0 is not on list 0x0 0x0' at line 138 in file /usr/src-vanilla/lib/libthr/thread/thr_mutex.c (errno = 0)

where the first line is the last instruction tested.

I think however it is fine to commit the port, so that other people can test it on their hardware and see what other bugs are lurking.
Comment 22 commit-hook freebsd_committer freebsd_triage 2017-09-30 15:13:52 UTC
A commit references this bug:

Author: rene
Date: Sat Sep 30 15:13:33 UTC 2017
New revision: 450997
URL: https://svnweb.freebsd.org/changeset/ports/450997

Log:
  The sandsifter audits x86 processors for hidden instructions and
  hardware bugs, by systematically generating machine code to search
  through a processor's instruction set, and monitoring execution for
  anomalies. Sandsifter has uncovered secret processor instructions from
  every major vendor; ubiquitous software bugs in disassemblers,
  assemblers, and emulators; flaws in enterprise hypervisors; and both
  benign and security-critical hardware bugs in x86 chips.

  WWW: https://github.com/xoreaxeaxeax/sandsifter

  PR:		221132
  Submitted by:	rozhuk.im AT gmail.com
  Reviewed by:	swills

Changes:
  head/security/Makefile
  head/security/sandsifter/
  head/security/sandsifter/Makefile
  head/security/sandsifter/distinfo
  head/security/sandsifter/files/
  head/security/sandsifter/files/patch-Makefile
  head/security/sandsifter/files/patch-injector.c
  head/security/sandsifter/files/patch-sifter.py
  head/security/sandsifter/pkg-descr
  head/security/sandsifter/pkg-message
  head/security/sandsifter/pkg-plist