Bug 175091

Summary: Crash: Fatal trap 12: page fault while in kernel mode
Product: Base System Reporter: Rasmus Skaarup <freebsd>
Component: amd64Assignee: freebsd-amd64 (Nobody) <amd64>
Status: Closed FIXED    
Severity: Affects Only Me    
Priority: Normal    
Version: 9.1-RELEASE   
Hardware: Any   
OS: Any   

Description Rasmus Skaarup 2013-01-07 08:10:01 UTC
On of my virtualized FreeBSD machines has been panic'ing two times within the last two weeks. After the first panic I ran freebsd-update and upgraded to 9.1-RELEASE succesfully. Today the machine panic'ed again.

I have another virtualized FreeBSD machine running on the same host, and it does not exhibit this behaviour.

Here is the output from dmesg, after reboot:

****
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address   = 0x48
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80bd5139
stack pointer           = 0x28:0xffffff81625536c0
frame pointer           = 0x28:0xffffff8162553750
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 62083 (httpd)
trap number             = 12
panic: page fault
cpuid = 2
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80bd8240 at trap_fatal+0x290
#3 0xffffffff80bd857d at trap_pfault+0x1ed
#4 0xffffffff80bd8b9e at trap+0x3ce
#5 0xffffffff80bc315f at calltrap+0x8
#6 0xffffffff80b41133 at vm_fault_hold+0x1b13
#7 0xffffffff80b41cc3 at vm_fault+0x73
#8 0xffffffff80bd84b4 at trap_pfault+0x124
#9 0xffffffff80bd8c6c at trap+0x49c
#10 0xffffffff80bc315f at calltrap+0x8
Uptime: 13h6m22s
*********

This machine is running virtualized on a Centos 6.3 host, with 

qemu-kvm-0.12.1.2-2.295.el6_3.2.x86_64

and the following configuration:

*********
<domain type='kvm'>
  <name>FreeBSD3</name>
  <uuid>77777777-8794-11e1-b2d2-003005fb82e6</uuid>
  <memory unit='KiB'>5242880</memory>
  <currentMemory unit='KiB'>5242880</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/home/vms/FreeBSD3-SystemDisk.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/home/vms/FreeBSD3-DataDisk.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='usb' index='0'/>
    <interface type='bridge'>
      <mac address='52:54:00:b2:12:8c'/>
      <source bridge='br0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:b2:12:a2'/>
      <source bridge='br1'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='00:16:8e:1a:f3:a9'/>
      <source bridge='virbr0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='10.0.0.29' keymap='en-us'>
      <listen type='address' address='10.0.0.29'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
</domain>
*******

/boot/loader.conf:
********
virtio_load="YES"
virtio_pci_load="YES"
virtio_blk_load="YES"
if_vtnet_load="YES"
virtio_balloon_load="YES"
*******

pciconf -l
*******
hostb0@pci0:0:0:0:	class=0x060000 card=0x11001af4 chip=0x12378086 rev=0x02 hdr=0x00
isab0@pci0:0:1:0:	class=0x060100 card=0x11001af4 chip=0x70008086 rev=0x00 hdr=0x00
atapci0@pci0:0:1:1:	class=0x010180 card=0x11001af4 chip=0x70108086 rev=0x00 hdr=0x00
uhci0@pci0:0:1:2:	class=0x0c0300 card=0x11001af4 chip=0x70208086 rev=0x01 hdr=0x00
none0@pci0:0:1:3:	class=0x068000 card=0x11001af4 chip=0x71138086 rev=0x03 hdr=0x00
vgapci0@pci0:0:2:0:	class=0x030000 card=0x11001af4 chip=0x00b81013 rev=0x00 hdr=0x00
re0@pci0:0:3:0:	class=0x020000 card=0x11001af4 chip=0x813910ec rev=0x20 hdr=0x00
virtio_pci0@pci0:0:4:0:	class=0x050000 card=0x00051af4 chip=0x10021af4 rev=0x00 hdr=0x00
em0@pci0:0:5:0:	class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00
em1@pci0:0:6:0:	class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00
virtio_pci1@pci0:0:7:0:	class=0x010000 card=0x00021af4 chip=0x10011af4 rev=0x00 hdr=0x00
virtio_pci2@pci0:0:8:0:	class=0x010000 card=0x00021af4 chip=0x10011af4 rev=0x00 hdr=0x00
********

How-To-Repeat: Happens regularly by itself.
Comment 1 John Baldwin freebsd_committer freebsd_triage 2013-01-07 14:32:36 UTC
Can you enable crashdumps by setting 'dumpdev="AUTO"' in /etc/rc.conf?

Also, can you run 'gdb /boot/kernel/kernel' and then at the prompt run
'l *vm_fault_hold+0x1b13' and reply with the output?

-- 
John Baldwin
Comment 2 Rasmus Skaarup 2013-01-07 18:11:48 UTC
Thank you for the quick response. I enabled the setting in rc.conf as =
you mentioned, and the machine has crashed twice since. The two dumps =
are uploaded here:

http://gal.dk/crash0.tar.gz
http://gal.dk/crash1.tar.gz

gdb output for the original error:

(gdb) l *vm_fault_hold+0x1b13
0xffffffff80b41133 is in vm_fault_hold (/usr/src/sys/vm/vm_fault.c:936).
931		 * because pmap_enter() may sleep.  We don't put the =
page
932		 * back on the active queue until later so that the =
pageout daemon
933		 * won't find it (yet).
934		 */
935		pmap_enter(fs.map->pmap, vaddr, fault_type, fs.m, prot, =
wired);
936		if ((fault_flags & VM_FAULT_CHANGE_WIRING) =3D=3D 0 && =
wired =3D=3D 0)
937			vm_fault_prefault(fs.map->pmap, vaddr, =
fs.entry);
938		VM_OBJECT_LOCK(fs.object);
939		vm_page_lock(fs.m);
940=09
(gdb)=20


The two other crashes, had different excuses. Here is the first:

Fatal trap 9: general protection fault while in kernel mode
cpuid =3D 3; apic id =3D 03
instruction pointer     =3D 0x20:0xffffffff81612ace
stack pointer           =3D 0x28:0xffffff816230a4c0
frame pointer           =3D 0x28:0xffffff816230a4e0
code segment            =3D base rx0, limit 0xfffff, type 0x1b
                        =3D DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        =3D interrupt enabled, resume, IOPL =3D 0
current process         =3D 18778 (imapd)
processor eflags        =3D interrupt enabled, resume, IOPL =3D 0
current process         =3D 18778 (imapd)
trap number             =3D 9
panic: general protection fault
cpuid =3D 1
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80bd8240 at trap_fatal+0x290
#3 0xffffffff80bd88d5 at trap+0x105
#4 0xffffffff80bc315f at calltrap+0x8
#5 0xffffffff8164915d at dnode_free_range+0x29d
#6 0xffffffff81639d5f at dmu_free_long_range_impl+0x13f
#7 0xffffffff81639f9c at dmu_free_long_range+0x4c
#8 0xffffffff816a7839 at zfs_rmnode+0x69
#9 0xffffffff816be9b6 at zfs_inactive+0x66
#10 0xffffffff816beb7a at zfs_freebsd_inactive+0x1a
#11 0xffffffff8097f61d at vinactive+0x8d
#12 0xffffffff80982de8 at vputx+0x2d8
#13 0xffffffff80986f4f at kern_unlinkat+0x1df
#14 0xffffffff80bd7ae6 at amd64_syscall+0x546
#15 0xffffffff80bc3447 at Xfast_syscall+0xf7
Uptime: 1h17m3s

(gdb) l *trap_fatal+0x290=20
0xffffffff80bd8240 is in trap_fatal =
(/usr/src/sys/amd64/amd64/trap.c:852).
847			printf("Idle\n");
848		}
849=09
850	#ifdef KDB
851		if (debugger_on_panic || kdb_active)
852			if (kdb_trap(type, 0, frame))
853				return;
854	#endif
855		printf("trap number		=3D %d\n", type);
856		if (type <=3D MAX_TRAP_MSG)
(gdb)=20


The other new crash:

Fatal trap 12: page fault while in kernel mode
cpuid =3D 1; apic id =3D 01
fault virtual address   =3D 0x0
fault code              =3D supervisor read data, page not present
instruction pointer     =3D 0x20:0xffffffff80bcf1fb
stack pointer           =3D 0x28:0xffffff8161fef950
frame pointer           =3D 0x28:0xffffff8161fef990
code segment            =3D base rx0, limit 0xfffff, type 0x1b
                        =3D DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        =3D interrupt enabled, resume, IOPL =3D 0
current process         =3D 75787 (httpd)
trap number             =3D 12
panic: page fault
cpuid =3D 1
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80bd8240 at trap_fatal+0x290
#3 0xffffffff80bd857d at trap_pfault+0x1ed
#4 0xffffffff80bd8b9e at trap+0x3ce
#5 0xffffffff80bc315f at calltrap+0x8
#6 0xffffffff80bcf290 at pmap_is_modified+0x40
#7 0xffffffff80b52f7e at vm_page_dontneed+0x17e
#8 0xffffffff80b4f0cd at vm_object_madvise+0x4dd
#9 0xffffffff80b49beb at vm_map_madvise+0x1bb
#10 0xffffffff80b4bff1 at sys_madvise+0x91
#11 0xffffffff80bd7ae6 at amd64_syscall+0x546
#12 0xffffffff80bc3447 at Xfast_syscall+0xf7
Uptime: 5h5m23s

(gdb) l *pmap_is_modified+0x40
0xffffffff80bcf290 is in pmap_is_modified =
(/usr/src/sys/amd64/amd64/pmap.c:4264).
4259		VM_OBJECT_LOCK_ASSERT(m->object, MA_OWNED);
4260		if ((m->oflags & VPO_BUSY) =3D=3D 0 &&
4261		    (m->aflags & PGA_WRITEABLE) =3D=3D 0)
4262			return (FALSE);
4263		rw_wlock(&pvh_global_lock);
4264		rv =3D pmap_is_modified_pvh(&m->md) ||
4265		    ((m->flags & PG_FICTITIOUS) =3D=3D 0 &&
4266		    =
pmap_is_modified_pvh(pa_to_pvh(VM_PAGE_TO_PHYS(m))));
4267		rw_wunlock(&pvh_global_lock);
4268		return (rv);
(gdb) l *trap_pfault+0x1ed
0xffffffff80bd857d is in trap_pfault =
(/usr/src/sys/amd64/amd64/trap.c:773).
768			if (td->td_intr_nesting_level =3D=3D 0 &&
769			    PCPU_GET(curpcb)->pcb_onfault !=3D NULL) {
770				frame->tf_rip =3D =
(long)PCPU_GET(curpcb)->pcb_onfault;
771				return (0);
772			}
773			trap_fatal(frame, eva);
774			return (-1);
775		}
776=09
777		return((rv =3D=3D KERN_PROTECTION_FAILURE) ? SIGBUS : =
SIGSEGV);
(gdb)=20

(not sure I'm gbd'ing what you need, but let me know).

I am beginning to suspect the hardware, but the strange thing is that =
the host (CentOS 6.3) and the other virtual machine works completely =
fine. And the other virtual machine has plenty of user on it.

Best regards,
Rasmus skaarup
Comment 3 Rasmus Skaarup 2013-03-04 18:53:09 UTC
I have opened at new bug instead of this one, with more crashes. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=176636

Br
Rasmus
Comment 4 Mark Linimon freebsd_committer freebsd_triage 2013-03-05 16:31:26 UTC
State Changed
From-To: open->closed

see kern/176636.