Bug 213055

Summary: bhyve guest crashes when second disk is added
Product: Base System Reporter: Palle Girgensohn <girgen>
Component: binAssignee: freebsd-virtualization (Nobody) <virtualization>
Status: New ---    
Severity: Affects Only Me CC: grehan
Priority: ---    
Version: 11.0-STABLE   
Hardware: Any   
OS: Any   

Description Palle Girgensohn freebsd_committer freebsd_triage 2016-09-28 22:40:18 UTC 
Hi,

I'm trying to get bhyve running with Debian jessie (esp Github Enterprise virtual appliance) on 11.0-RELEASE.

When running with one disk only, converted with qemu-img convert... it works fine. 

But this appliance need a second disk, and with that it crashes. Oddly, I had it running for a day, but now it is imppossible to get it working again.

# cat device.map 
(hd0) /dev/zvol/tank/githubroot
(hd1) /dev/zvol/tank/githubdata

# cat grub.in 
set root=(hd0,msdos1)
linux /vmlinuz root=/dev/vda1
initrd /initrd.img
boot

starting with 
# grub-bhyve -m device.map -r hd0 -M 18000 github < grub.in  
# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0         -s 3:0,virtio-blk,/home/girgen/github.img -s 4:0,virtio-blk,/dev/zvol/tank/githubdata     -l com1,/dev/nmdm0A -c 1 -m 18000 github

...

host's dmesg:
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz (2297.39-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x37ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,NFPUSG>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 103079215104 (98304 MB)
avail memory = 99955458048 (95324 MB)


and here's some typical output: it lives for some brief seconds...

[  OK  ] Started GitHub Enterprise Preflight Checks.
[    7.644335] BUG: unable to handle kernel paging request at 0000000010020000
[    7.645210] IP: [<ffffffff81287dee>] __blk_bios_map_sg+0x1de/0x400
[    7.645959] PGD b9440067 PUD b945e067 PMD 0 
[    7.646526] Oops: 0000 [#1] SMP 
[    7.646944] Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink bluetooth 6lowpan_iphc rfkill tcp_diag inet_diag binfmt_misc ip6t_REJECT xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 xt_comment xt_LOG xt_limit ipt_REJECT xt_tcpudp xt_addrtype xt_owner nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables thermal_sys crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd pcspkr serio_raw evdev button autofs4 ext4 crc16 mbcache jbd2 dm_mod virtio_net virtio_blk crct10dif_pclmul crct10dif_common crc32c_intel psmouse virtio_pci virtio_ring virtio
[    7.655837] CPU: 3 PID: 349 Comm: kworker/3:1H Not tainted 3.16.0-4-amd64 #1 Debian 3.16.36-1+deb8u1
[    7.656903] Hardware name:   BHYVE, BIOS 1.00 03/14/2014
[    7.657563] Workqueue: kblockd blk_mq_run_work_fn
[    7.658131] task: ffff880490af6290 ti: ffff88048df94000 task.ti: ffff88048df94000
[    7.659012] RIP: 0010:[<ffffffff81287dee>]  [<ffffffff81287dee>] __blk_bios_map_sg+0x1de/0x400
[    7.660049] RSP: 0000:ffff88048df97cd8  EFLAGS: 00010206
[    7.660680] RAX: 0000000010020000 RBX: 000000000001d000 RCX: 0000000000000004
[    7.661521] RDX: 0000000010020001 RSI: ffffea000005b738 RDI: 0000000000000000
[    7.662360] RBP: ffff88048df97d58 R08: ffff880481f73620 R09: 0000000000000025
[    7.663202] R10: 0000000000000000 R11: 0000000001a22000 R12: 0000000000000000
[    7.664065] R13: 0000000000001000 R14: 0000000000000004 R15: ffff88048df97d68
[    7.664902] FS:  0000000000000000(0000) GS:ffff8804a4d80000(0000) knlGS:0000000000000000
[    7.665871] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    7.666549] CR2: 0000000010020000 CR3: 00000000b9be8000 CR4: 00000000000406e0
[    7.667395] Stack:
[    7.667649]  ffff88048df97c50 ffff88048db067a0 ffff88048d8fe008 0100000000000004
[    7.668591]  ffff880481f73620 ffff880400000025 0000000000000004 ffffea000005b738
[    7.669542]  0000000000001000 ffffea000005b738 0000000000001000 ffff88048db06600
[    7.670507] Call Trace:
[    7.670802]  [<ffffffff8128803f>] ? blk_rq_map_sg+0x2f/0x170
[    7.671488]  [<ffffffffa005c4db>] ? virtio_queue_rq+0x9b/0x240 [virtio_blk]
[    7.672309]  [<ffffffff8128bc72>] ? __blk_mq_run_hw_queue+0x1c2/0x320
[    7.673079]  [<ffffffff81082b73>] ? process_one_work+0x143/0x430
[    7.673791]  [<ffffffff810832f3>] ? worker_thread+0x113/0x4f0
[    7.674465]  [<ffffffff81514931>] ? __schedule+0x2b1/0x6f0
[    7.675136]  [<ffffffff810831e0>] ? rescuer_thread+0x2d0/0x2d0
[    7.675828]  [<ffffffff810894bd>] ? kthread+0xbd/0xe0
[    7.676429]  [<ffffffff81089400>] ? kthread_create_on_node+0x180/0x180
[    7.677205]  [<ffffffff81518498>] ? ret_from_fork+0x58/0x90
[    7.677880]  [<ffffffff81089400>] ? kthread_create_on_node+0x180/0x180
[    7.678661] Code: 3f 44 89 4c 24 28 48 89 4c 24 30 e8 5d 5a 03 00 8b 7c 24 44 48 8b 74 24 38 4c 8b 44 24 20 44 8b 4c 24 28 48 8b 4c 24 30 49 89 07 <48> 8b 10 83 e2 03 40 f6 c6 03 0f 85 b2 01 00 00 48 09 f2 89 78 
[    7.682068] RIP  [<ffffffff81287dee>] __blk_bios_map_sg+0x1de/0x400
[    7.682840]  RSP <ffff88048df97cd8>
[    7.683269] CR2: 0000000010020000
[    7.683684] ---[ end trace 94aad109fa174203 ]---
[    7.688265] BUG: unable to handle kernel paging request at ffffffffffffffd8
[    7.689152] IP: [<ffffffff81089a8c>] kthread_data+0xc/0x20
[    7.689815] PGD 1816067 PUD 1818067 PMD 0 
[    7.690340] Oops: 0000 [#2] SMP 
[    7.690755] Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink bluetooth 6lowpan_iphc rfkill tcp_diag inet_diag binfmt_misc ip6t_REJECT xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 xt_comment xt_LOG xt_limit ipt_REJECT xt_tcpudp xt_addrtype xt_owner nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables thermal_sys crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd pcspkr serio_raw evdev button autofs4 ext4 crc16 mbcache jbd2 dm_mod virtio_net virtio_blk crct10dif_pclmul crct10dif_common crc32c_intel psmouse virtio_pci virtio_ring virtio
[    7.699044] CPU: 3 PID: 349 Comm: kworker/3:1H Tainted: G      D       3.16.0-4-amd64 #1 Debian 3.16.36-1+deb8u1
[    7.700196] Hardware name:   BHYVE, BIOS 1.00 03/14/2014
[    7.700815] task: ffff880490af6290 ti: ffff88048df94000 task.ti: ffff88048df94000
[    7.701668] RIP: 0010:[<ffffffff81089a8c>]  [<ffffffff81089a8c>] kthread_data+0xc/0x20
[    7.702596] RSP: 0018:ffff88048df979f0  EFLAGS: 00010002
[    7.703206] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 000000000000000f
[    7.704017] RDX: 000000000000000e RSI: 0000000000000003 RDI: ffff880490af6290
[    7.704830] RBP: ffff880490af6290 R08: 0000000000000001 R09: 000000000000b8e1
[    7.705643] R10: ffffffff81ae045c R11: 0000000000000000 R12: ffff8804a4d92f40
[    7.706452] R13: 0000000000000003 R14: 0000000000000000 R15: ffff880490af6290
[    7.707284] FS:  0000000000000000(0000) GS:ffff8804a4d80000(0000) knlGS:0000000000000000
[    7.708201] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    7.708854] CR2: 0000000000000028 CR3: 000000048d8a4000 CR4: 00000000000406e0
[    7.709670] Stack:
[    7.709908]  ffffffff8108373d ffff880490af66f0 ffffffff81514abd 0000000000012f40
[    7.710829]  ffff88048df97fd8 0000000000012f40 ffff880490af6290 ffff880490af6910
[    7.711736]  ffff880490af65e8 ffff880490af6280 ffff8804915d72b0 ffff880490af6280
[    7.712656] Call Trace:
[    7.712952]  [<ffffffff8108373d>] ? wq_worker_sleeping+0xd/0x80
[    7.713632]  [<ffffffff81514abd>] ? __schedule+0x43d/0x6f0
[    7.714266]  [<ffffffff8106b08f>] ? do_exit+0x6df/0xa70
[    7.714882]  [<ffffffff81017277>] ? oops_end+0x97/0xe0
[    7.715485]  [<ffffffff8150ec7c>] ? no_context+0x2b2/0x2be
[    7.716120]  [<ffffffff810581b0>] ? __do_page_fault+0x80/0x4f0
[    7.716786]  [<ffffffffa0016060>] ? virtqueue_add_sgs+0x350/0x5f7 [virtio_ring]
[    7.717605]  [<ffffffff812acf20>] ? cpumask_next_and+0x30/0x40
[    7.718281]  [<ffffffffa00161bc>] ? virtqueue_add_sgs+0x4ac/0x5f7 [virtio_ring]
[    7.719125]  [<ffffffff8151a568>] ? page_fault+0x28/0x30
[    7.719734]  [<ffffffff81287dee>] ? __blk_bios_map_sg+0x1de/0x400
[    7.720432]  [<ffffffff81287dd3>] ? __blk_bios_map_sg+0x1c3/0x400
[    7.721140]  [<ffffffff8128803f>] ? blk_rq_map_sg+0x2f/0x170
[    7.721814]  [<ffffffffa005c4db>] ? virtio_queue_rq+0x9b/0x240 [virtio_blk]
[    7.722620]  [<ffffffff8128bc72>] ? __blk_mq_run_hw_queue+0x1c2/0x320
[    7.723361]  [<ffffffff81082b73>] ? process_one_work+0x143/0x430
[    7.724059]  [<ffffffff810832f3>] ? worker_thread+0x113/0x4f0
[    7.724729]  [<ffffffff81514931>] ? __schedule+0x2b1/0x6f0
[    7.725370]  [<ffffffff810831e0>] ? rescuer_thread+0x2d0/0x2d0
[    7.726060]  [<ffffffff810894bd>] ? kthread+0xbd/0xe0
[    7.726648]  [<ffffffff81089400>] ? kthread_create_on_node+0x180/0x180
[    7.727406]  [<ffffffff81518498>] ? ret_from_fork+0x58/0x90
[    7.728055]  [<ffffffff81089400>] ? kthread_create_on_node+0x180/0x180
[    7.728799] Code: 08 04 00 00 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 87 08 04 00 00 <48> 8b 40 d8 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 
[    7.732132] RIP  [<ffffffff81089a8c>] kthread_data+0xc/0x20
[    7.732788]  RSP <ffff88048df979f0>
[    7.733202] CR2: ffffffffffffffd8
[    7.733591] ---[ end trace 94aad109fa174204 ]---
[    7.734133] Fixing recursive fault but reboot is needed!
Comment 1 Peter Grehan freebsd_committer freebsd_triage 2016-10-01 21:38:32 UTC 
I noticed in your grub device map that two zvols were listed:

(hd0) /dev/zvol/tank/githubroot
(hd1) /dev/zvol/tank/githubdata

but only 1 of these was configured from the bhyve command line:

-s 3:0,virtio-blk,/home/girgen/github.img   <- should be the githubroot zvol?
-s 4:0,virtio-blk,/dev/zvol/tank/githubdata

Not sure if that's intentional, and it may not make a difference.

One thing to try is to use ahci-hd for the disk images e.g.

-s 3:0,ahci-hd,/home/girgen/github.img
-s 4:0,ahci-hd,/dev/zvol/tank/githubdata