Bug 176112 - [jail] [panic] kernel panic when starting jails
Summary: [jail] [panic] kernel panic when starting jails
Status: Closed DUPLICATE of bug 194515
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 9.1-RELEASE
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-jail (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-02-13 14:20 UTC by Moritz Wilhelmy
Modified: 2014-12-13 12:46 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Moritz Wilhelmy 2013-02-13 14:20:01 UTC
When I start jails on the above kernel from releng/9.1 branch which has rctl,
altq, VIMAGE and epair enabled (and SCTP support disabled), it died repeatedly
whenever I tried to start a jail with the following commandline after defining
an epair interface: jail -c vnet name=test host.hostname=test path=/ persist

After rebuilding the kernel with VIMAGE disabled, the system is stable again.

/etc/pf.conf:
-- snip --
# define macros for each network interface
extif = "re0"
intif = "lo0"
tcp_services = "{ 22 }"

# define our networks
intnet   = "{ 127.0.1.0/24 }"
extaddr  = "176.9.47.245"
mailjail = "127.0.1.1"
wwwjail  = "127.0.1.3"
wwwports = "{ 80 443 }"
xmppj    = "127.0.1.4"
xmppp    = "{ 5222 5269 }"
pbulkj   = "127.0.1.5"
pbulkp   = "{ 3322 }"
ircj     = "127.0.1.2"
ircp     = "{ 6667 7776 7005 2222 }"

set loginterface $extif

# Normalizes packets and masks the OS's shortcomings such as SYN/FIN packets 
# [scrub reassemble tcp](BID 10183) and sequence number approximation 
# bugs (BID 7487).
scrub on $extif reassemble tcp no-df random-id

# NAT for certain jails..
nat on $extif from 127.0.1.0/24 to any -> $extaddr

#pass in on $extif inet6 proto icmp6 icmp6-type {routeradv,echoreq,neighbrsol,neighbradv}

rdr pass on $extif proto tcp from any to $extaddr port $wwwports -> $wwwjail
#rdr pass on $extif proto tcp from any to $extaddr port $xmppp    -> $xmppj
rdr pass proto tcp from any to $extaddr port $xmppp    -> $xmppj
rdr pass on $extif proto tcp from any to $extaddr port $pbulkp   -> $pbulkj
rdr pass on { $extif, $intif } proto tcp from any to $extaddr port $ircp   -> $ircj
-- snip --

/pool/crash/core.txt.4 (obtained using crashinfo(8); Wuff --- Wuff is screen's
visual bell before anybody asks):
-- snip --
host.wzff.de dumped core - see /pool/crash/vmcore.4

Wed Feb 13 01:43:51 UTC 2013

FreeBSD host.wzff.de 9.1-RELEASE FreeBSD 9.1-RELEASE #3 r246626: Sun Feb 10 18:26:17 UTC 2013     root@host.wzff.de:/usr/obj/usr/src/sys/RCTL_ALTQ  amd64

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:



Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 03
fault virtual address	= 0x0
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff802cf02f
stack pointer	        = 0x28:0xffffff845d647350
frame pointer	        = 0x28:0xffffff845d647390
code segment		= base rx0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 1598 (jail)
trap number		= 12
panic: page fault
cpuid = 3
KDB: stack backtrace:
#0 0xffffffff8050a226 at kdb_backtrace+0x66
#1 0xffffffff804d3e5e at panic+0x1ce
#2 0xffffffff806fea80 at trap_fatal+0x290
#3 0xffffffff806fedbd at trap_pfault+0x1ed
#4 0xffffffff806ff3de at trap+0x3ce
#5 0xffffffff806e999f at calltrap+0x8
#6 0xffffffff802d0525 at pf_altq_ifnet_event+0x45
#7 0xffffffff802ccdf4 at pfi_attach_ifnet_event+0x64
#8 0xffffffff8058f25b at if_attach_internal+0x21b
#9 0xffffffff8059f573 at lo_clone_create+0xa3
#10 0xffffffff80599d19 at ifc_simple_create+0x89
#11 0xffffffff80599965 at if_clone_createif+0x65
#12 0xffffffff80599ab3 at ifc_simple_attach+0x63
#13 0xffffffff8059a644 at if_clone_attach+0x154
#14 0xffffffff805b08ef at vnet_sysinit+0x3f
#15 0xffffffff805b09f3 at vnet_alloc+0xa3
#16 0xffffffff804ad221 at kern_jail_set+0x3481
#17 0xffffffff804adb52 at sys_jail_set+0x62
Uptime: 3m9s
Dumping 737 out of 16281 MB:..3%..11%..22%..31%..42%..53%..61%..72%..81%..92%

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/geom_label.ko...Reading symbols from /boot/kernel/geom_label.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/geom_label.ko
Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/geom_mirror.ko
Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/linux.ko
#0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
224	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
#1  0xffffffff804d3941 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:448
#2  0xffffffff804d3e37 in panic (fmt=0x1 <Address 0x1 out of bounds>)
    at /usr/src/sys/kern/kern_shutdown.c:636
#3  0xffffffff806fea80 in trap_fatal (frame=0xc, eva=Variable "eva" is not available.
)
    at /usr/src/sys/amd64/amd64/trap.c:857
#4  0xffffffff806fedbd in trap_pfault (frame=0xffffff845d6472a0, usermode=0)
    at /usr/src/sys/amd64/amd64/trap.c:773
#5  0xffffffff806ff3de in trap (frame=dwarf2_read_address: Corrupted DWARF expression.
) at /usr/src/sys/amd64/amd64/trap.c:456
#6  0xffffffff806e999f in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:228
#7  0xffffffff802cf02f in pf_begin_altq (ticket=0xffffff845d6473d4)
    at /usr/src/sys/contrib/pf/net/pf_ioctl.c:866
#8  0xffffffff802d0525 in pf_altq_ifnet_event (ifp=0xfffffe0004cd4000, 
    remove=0) at /usr/src/sys/contrib/pf/net/pf_ioctl.c:1117
#9  0xffffffff802ccdf4 in pfi_attach_ifnet_event (arg=Variable "arg" is not available.
)
    at /usr/src/sys/contrib/pf/net/pf_if.c:1046
#10 0xffffffff8058f25b in if_attach_internal (ifp=0xfffffe0004cd4000, vmove=Variable "vmove" is not available.
)
    at /usr/src/sys/net/if.c:699
#11 0xffffffff8059f573 in lo_clone_create (ifc=0xffffff80043d89e0, unit=0, 
    params=Variable "params" is not available.
) at /usr/src/sys/net/if_loop.c:151
#12 0xffffffff80599d19 in ifc_simple_create (ifc=0xffffff80043d89e0, 
    name=0xffffff845d647560 "lo0", len=16, params=0x0)
    at /usr/src/sys/net/if_clone.c:575
#13 0xffffffff80599965 in if_clone_createif (ifc=0xffffff80043d89e0, 
    name=0xffffff845d647560 "lo0", len=16, params=0x0)
    at /usr/src/sys/net/if_clone.c:176
#14 0xffffffff80599ab3 in ifc_simple_attach (ifc=0xffffff80043d89e0)
    at /usr/src/sys/net/if_clone.c:528
#15 0xffffffff8059a644 in if_clone_attach (ifc=0xffffff80043d89e0)
    at /usr/src/sys/net/if_clone.c:310
#16 0xffffffff805b08ef in vnet_sysinit () at /usr/src/sys/net/vnet.c:616
#17 0xffffffff805b09f3 in vnet_alloc () at /usr/src/sys/net/vnet.c:255
#18 0xffffffff804ad221 in kern_jail_set (td=0xfffffe0009153470, 
    optuio=0xfffffe0009c93c00, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_jail.c:1298
#19 0xffffffff804adb52 in sys_jail_set (td=0xfffffe0009153470, 
    uap=0xffffff845d647a70) at /usr/src/sys/kern/kern_jail.c:516
#20 0xffffffff806fe326 in amd64_syscall (td=0xfffffe0009153470, traced=0)
    at subr_syscall.c:135
#21 0xffffffff806e9c87 in Xfast_syscall ()
    at /usr/src/sys/amd64/amd64/exception.S:387
#22 0x0000000800ea25ac in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) 

------------------------------------------------------------------------
ps -axl

ps: can't read sigacts at 0xfffffe000970b000

------------------------------------------------------------------------
vmstat -s

   162719 cpu context switches
    26798 device interrupts
    15210 software interrupts
   100592 traps
   183470 system calls
       24 kernel threads created
     1570  fork() calls
        4 vfork() calls
        0 rfork() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
      347 vnode pager pageins
     2554 vnode pager pages paged in
        0 vnode pager pageouts
        0 vnode pager pages paged out
        0 page daemon wakeups
        0 pages examined by the page daemon
        0 pages reactivated
    46182 copy-on-write faults
      345 copy-on-write optimized faults
    33921 zero fill pages zeroed
        0 zero fill pages prezeroed
        0 intransit blocking page faults
   116719 total VM faults taken
        0 pages affected by kernel thread creation
   802814 pages affected by  fork()
     2164 pages affected by vfork()
        0 pages affected by rfork()
        0 pages cached
   125815 pages freed
        0 pages freed by daemon
        0 pages freed by exiting processes
     1850 pages active
     2350 pages inactive
        0 pages in VM cache
    53612 pages wired down
  3979390 pages free
     4096 bytes per page
    21260 total name lookups
          cache hits (86% pos + 7% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
        DEVFS    18     1K       -       19  16,32,128
       DEVFSP     2     1K       -        2  64
    sysctltmp     0     0K       -      287  16,32,64,128,256,512,4096
    sysctloid  4305   212K       -     4417  16,32,64,128
       sysctl     0     0K       -      652  16,32,64
      tidhash     1    16K       -        1  
      callout     7  3584K       -        7  
         umtx   846   106K       -      846  128
     p1003.1b     1     1K       -        1  16
         SWAP     4  4377K       -        4  64
       bus-sc    98   175K       -     1670  16,32,64,128,256,512,1024,2048,4096
          bus   896    81K       -     4009  16,32,64,128,256,512,1024
      devstat    16    33K       -       16  32,4096
 eventhandler    90     7K       -       90  64,128
  nullfs_hash     1     1K       -        1  128
         kobj   173   692K       -      283  4096
    pfs_nodes    21     6K       -       21  256
      Per-cpu     1     1K       -        1  32
      entropy  1024    64K       -     1024  64
         GEOM   209    35K       -      961  16,32,64,128,256,512,1024
         rman   208    24K       -      564  16,32,128
      acpidev    40     3K       -       40  64
         sbuf     0     0K       -     1210  16,32,64,128,256,512,1024,2048,4096
     acpitask     1     2K       -        1  2048
      CAM SIM     2     1K       -        2  256
        stack     0     0K       -        2  256
    taskqueue    93     9K       -      123  16,32,64,128,1024
       Unitno    12     1K       -      296  32,64
          iov     1     1K       -     1084  64,128,256,512
       select    11     2K       -       11  128
     ioctlops     0     0K       -     2362  16,32,64,128,256,512,1024,2048,4096
          msg     4    30K       -        4  2048,4096
          sem     4   106K       -        4  2048,4096
          shm     1    20K       -        1  
          tty    18    18K       -       20  1024,2048
          pts     1     1K       -        1  256
     mbuf_tag     0     0K       -    14769  32,64,128
         ksem     1     8K       -        1  
        shmfd     1     8K       -        1  
          pcb    13    25K       -       28  16,32,1024,2048,4096
       soname     3     1K       -      125  16,32,128
          acl     0     0K       -        2  4096
     vfscache     1  4096K       -        1  
     vfs_hash     1  2048K       -        1  
  ata_generic     2     2K       -        2  1024
       vnodes     3     1K       -        3  64,256
        mount   309    12K       -      744  16,32,64,128,256,1024
  vnodemarker     0     0K       -      325  512
          BPF    11    10K       -       11  128,512,4096
  ether_multi   103     6K       -      122  16,32,64
       ifaddr   139    55K       -      140  32,64,128,256,512,2048,4096
        ifnet     9    15K       -        9  128,2048
      ata_pci     1     1K       -        1  64
      acpisem    28     4K       -       28  128
     pci_link    16     2K       -       16  64,128
    ad_driver     2     1K       -        2  32
    ar_driver     0     0K       -       12  512,2048
       kbdmux     8    18K       -        8  16,512,1024,2048
        clone    11    44K       -       11  4096
       arpcom     1     1K       -        1  16
      lltable    50    16K       -       50  256,512
       isadev     7     1K       -        7  128
       USBdev    37     9K       -       37  64,128,512,4096
     routetbl   149  8256K       -      364  32,64,128,256,512
vnet_data_free     1     1K       -        1  32
    vnet_data     2    64K       -        2  
         vnet     2     1K       -        2  64
         igmp     7     2K       -        7  256
         ipid     2    24K       -        2  
          USB    53    85K       -       62  16,32,64,128,256,512,2048,4096
         cdev     8     2K       -        8  256
     in_multi     2     1K       -        2  256
    hostcache     2    56K       -        2  
     syncache     2   192K       -        2  
        sigio     1     1K       -        1  64
    in6_multi    90     8K       -       90  32,256
     filedesc    36    18K       -     1599  512
          mld     7     1K       -        7  128
      NFS FHA     1     2K       -        1  2048
          rpc     2     9K       -        2  256
audit_evclass   179     6K       -      218  32
     savedino     0     0K       -       10  256
     freework     3     1K       -       45  32,128
    newdirblk     0     0K       -        6  64
       dirrem     4     1K       -       60  128
        mkdir     0     0K       -       12  128
       diradd     1     1K       -       52  128
     freefile     0     0K       -       49  64
     freeblks     2     1K       -       43  128
     freefrag     0     0K       -        1  128
       newblk     2   513K       -       39  256
    bmsafemap     1     8K       -       60  256
     inodedep     7  2051K       -      102  512
      pagedep     5   513K       -       25  256
  ufs_dirhash    36     7K       -       36  16,32,64,128,512
    ufs_mount    12    50K       -       13  512,1024,4096
    vm_pgdata     3   129K       -        3  128
         kenv    75    11K       -       80  16,32,64,128
       kqueue     0     0K       -        6  256,2048
    acpi_perf     8     4K       -        8  512
    proc-args    11     1K       -      488  16,32,64,128,256
        hhook     4     1K       -        4  128
   CAM periph     2     1K       -        6  16,32,64,256
      ithread    97    17K       -       97  32,128,256
 prison_racct     1     1K       -        1  512
       prison     1     4K       -        1  4096
      memdesc     1     4K       -        1  4096
       KTRACE   100    13K       -      100  128
     atkbddev     1     1K       -        1  64
CAM dev queue     2     1K       -        2  128
       linker   127   168K       -      147  16,32,64,128,256,512,2048
        lockf    14     2K       -       46  64,128
   loginclass     1     1K       -        1  64
      CAM XPT    12     8K       -       30  32,64,128,256,2048
       ip6ndp    10     1K       -       25  64,128
         temp    72    15K       -     6271  16,32,64,128,256,512,1024,2048,4096
       devbuf  5692 12906K       -     6090  16,32,64,128,256,512,1024,2048,4096
    CAM queue     7     1K       -       17  16
       module   281    36K       -      281  128
       pfsync     1     1K       -        1  512
       apmdev     1     1K       -        1  128
     mtx_pool     2    16K       -        2  
   madt_table     0     0K       -        1  4096
     acpiintr     1     1K       -        1  64
       acpica  4480   456K       -    63001  16,32,64,128,256,512,1024,2048,4096
          osd     3     1K       -        6  16,64,128
       DEVFS1   110    55K       -      116  512
     pmchooks     1     1K       -        1  128
       DEVFS3   133    34K       -      143  256
      subproc   151   202K       -     1714  512,4096
      io_apic     1     2K       -        1  2048
         proc     2    16K       -        2  
      session     9     2K       -       11  128
         pgrp    11     2K       -       39  128
          MCA    18     3K       -       18  64,128
          msi     9     2K       -        9  128
     nexusdev     3     1K       -        3  16
         cred    66    11K       -     3984  64,256
      uidinfo     3     3K       -        4  128,2048
       plimit     2     1K       -       21  256
      solaris 40916 43269K       -   123295  16,32,64,128,256,512,1024,2048,4096
   kstat_data     4     1K       -        4  64
  mirror_data    14     4K       -      189  64,128,512
        linux    15     1K       -       15  64

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     213,       8,     213,   0,   0
UMA Zones:             1408,      0,     213,       1,     213,   0,   0
UMA Slabs:              568,      0,    1922,       3,    3279,   0,   0
UMA RCntSlabs:          568,      0,     742,       0,     742,   0,   0
UMA Hash:               256,      0,      81,       9,      81,   0,   0
16 Bucket:              152,      0,     179,      21,     179,   0,   0
32 Bucket:              280,      0,     248,       4,     248,  13,   0
64 Bucket:              536,      0,     207,       3,     207,  75,   0
128 Bucket:            1048,      0,     203,       1,     203,1145,   0
VM OBJECT:              232,      0,     927,     337,   19788,   0,   0
MAP:                    232,      0,       7,      25,       7,   0,   0
KMAP ENTRY:             120, 529325,      28,     964,    6716,   0,   0
MAP ENTRY:              120,      0,     229,     639,   45040,   0,   0
fakepg:                 120,      0,       0,       0,       0,   0,   0
mt_zone:               4112,      0,     240,      31,     240,   0,   0
16:                      16,      0,    3307,     725,   38287,   0,   0
32:                      32,      0,    2298,    1035,   18410,   0,   0
64:                      64,      0,   43973,     995,  108960,   0,   0
128:                    128,      0,    6889,     535,   14226,   0,   0
256:                    256,      0,     682,     353,    5221,   0,   0
512:                    512,      0,     860,     260,   43029,   0,   0
1024:                  1024,      0,     137,    1047,    5542,   0,   0
2048:                  2048,      0,     253,     197,     845,   0,   0
4096:                  4096,      0,     488,     203,    6674,   0,   0
Files:                   80,      0,      44,     406,    5776,   0,   0
TURNSTILE:              136,      0,     424,      76,     424,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
racct:                  168,      0,      40,     224,    1604,   0,   0
rctl_rule_link:          32,      0,       0,       0,       0,   0,   0
rctl_rule:               80,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1184,      0,      35,      79,    1598,   0,   0
THREAD:                1128,      0,     340,      83,     490,   0,   0
SLEEPQUEUE:              80,      0,     424,     156,     424,   0,   0
VMSPACE:                392,      0,      12,     168,    1576,   0,   0
cpuset:                  72,      0,     251,     199,     401,   0,   0
audit_record:           960,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,     256,    1030,    7641,   0,   0
mbuf:                   256,      0,       1,    1029,    7622,   0,   0
mbuf_cluster:          2048,  25600,    1280,     204,    1371,   0,   0
mbuf_jumbo_page:       4096,  12800,       0,       0,       0,   0,   0
mbuf_jumbo_9k:         9216,  19200,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,  12800,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  232,      0,       0,     432,   13890,   0,   0
ttyinq:                 160,      0,      30,     210,     300,   0,   0
ttyoutq:                256,      0,      16,     104,     160,   0,   0
ata_request:            320,      0,       1,     263,    5282,   0,   0
ata_composite:          336,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,     504,      33,   0,   0
VNODE:                  480,      0,     601,      79,     656,   0,   0
VNODEPOLL:              112,      0,       0,       0,       0,   0,   0
NAMEI:                 1024,      0,       0,      96,    9629,   0,   0
S VFS Cache:            108,      0,     484,     143,    1019,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,       0,       0,       0,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NFSMOUNT:               632,      0,       0,       0,       0,   0,   0
NFSNODE:                656,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,      50,      46,      50,   0,   0
zio_cache:              880,      0,       1,     775,    4505,   0,   0
zio_link_cache:          48,      0,       0,    1584,    4709,   0,   0
zio_buf_512:            512,      0,       0,       0,       0,   0,   0
zio_data_buf_512:       512,      0,       0,       0,       0,   0,   0
zio_buf_1024:          1024,      0,       0,       0,       0,   0,   0
zio_data_buf_1024:     1024,      0,       0,       0,       0,   0,   0
zio_buf_1536:          1536,      0,       0,       0,       0,   0,   0
zio_data_buf_1536:     1536,      0,       0,       0,       0,   0,   0
zio_buf_2048:          2048,      0,       0,       0,       0,   0,   0
zio_data_buf_2048:     2048,      0,       0,       0,       0,   0,   0
zio_buf_2560:          2560,      0,       0,       0,       0,   0,   0
zio_data_buf_2560:     2560,      0,       0,       0,       0,   0,   0
zio_buf_3072:          3072,      0,       0,       0,       0,   0,   0
zio_data_buf_3072:     3072,      0,       0,       0,       0,   0,   0
zio_buf_3584:          3584,      0,       0,       0,       0,   0,   0
zio_data_buf_3584:     3584,      0,       0,       0,       0,   0,   0
zio_buf_4096:          4096,      0,       0,       0,       0,   0,   0
zio_data_buf_4096:     4096,      0,       0,       0,       0,   0,   0
zio_buf_5120:          5120,      0,       0,       0,       0,   0,   0
zio_data_buf_5120:     5120,      0,       0,       0,       0,   0,   0
zio_buf_6144:          6144,      0,       0,       0,       0,   0,   0
zio_data_buf_6144:     6144,      0,       0,       0,       0,   0,   0
zio_buf_7168:          7168,      0,       0,       0,       0,   0,   0
zio_data_buf_7168:     7168,      0,       0,       0,       0,   0,   0
zio_buf_8192:          8192,      0,       0,       0,       0,   0,   0
zio_data_buf_8192:     8192,      0,       0,       0,       0,   0,   0
zio_buf_10240:        10240,      0,       0,       0,       0,   0,   0
zio_data_buf_10240:   10240,      0,       0,       0,       0,   0,   0
zio_buf_12288:        12288,      0,       0,       0,       0,   0,   0
zio_data_buf_12288:   12288,      0,       0,       0,       0,   0,   0
zio_buf_14336:        14336,      0,       0,       0,       0,   0,   0
zio_data_buf_14336:   14336,      0,       0,       0,       0,   0,   0
zio_buf_16384:        16384,      0,       0,       0,       0,   0,   0
zio_data_buf_16384:   16384,      0,       0,       0,       0,   0,   0
zio_buf_20480:        20480,      0,       0,       0,       0,   0,   0
zio_data_buf_20480:   20480,      0,       0,       0,       0,   0,   0
zio_buf_24576:        24576,      0,       0,       0,       0,   0,   0
zio_data_buf_24576:   24576,      0,       0,       0,       0,   0,   0
zio_buf_28672:        28672,      0,       0,       0,       0,   0,   0
zio_data_buf_28672:   28672,      0,       0,       0,       0,   0,   0
zio_buf_32768:        32768,      0,       0,       0,       0,   0,   0
zio_data_buf_32768:   32768,      0,       0,       0,       0,   0,   0
zio_buf_36864:        36864,      0,       0,       0,       0,   0,   0
zio_data_buf_36864:   36864,      0,       0,       0,       0,   0,   0
zio_buf_40960:        40960,      0,       0,       0,       0,   0,   0
zio_data_buf_40960:   40960,      0,       0,       0,       0,   0,   0
zio_buf_45056:        45056,      0,       0,       0,       0,   0,   0
zio_data_buf_45056:   45056,      0,       0,       0,       0,   0,   0
zio_buf_49152:        49152,      0,       0,       0,       0,   0,   0
zio_data_buf_49152:   49152,      0,       0,       0,       0,   0,   0
zio_buf_53248:        53248,      0,       0,       0,       0,   0,   0
zio_data_buf_53248:   53248,      0,       0,       0,       0,   0,   0
zio_buf_57344:        57344,      0,       0,       0,       0,   0,   0
zio_data_buf_57344:   57344,      0,       0,       0,       0,   0,   0
zio_buf_61440:        61440,      0,       0,       0,       0,   0,   0
zio_data_buf_61440:   61440,      0,       0,       0,       0,   0,   0
zio_buf_65536:        65536,      0,       0,       0,       0,   0,   0
zio_data_buf_65536:   65536,      0,       0,       0,       0,   0,   0
zio_buf_69632:        69632,      0,       0,       0,       0,   0,   0
zio_data_buf_69632:   69632,      0,       0,       0,       0,   0,   0
zio_buf_73728:        73728,      0,       0,       0,       0,   0,   0
zio_data_buf_73728:   73728,      0,       0,       0,       0,   0,   0
zio_buf_77824:        77824,      0,       0,       0,       0,   0,   0
zio_data_buf_77824:   77824,      0,       0,       0,       0,   0,   0
zio_buf_81920:        81920,      0,       0,       0,       0,   0,   0
zio_data_buf_81920:   81920,      0,       0,       0,       0,   0,   0
zio_buf_86016:        86016,      0,       0,       0,       0,   0,   0
zio_data_buf_86016:   86016,      0,       0,       0,       0,   0,   0
zio_buf_90112:        90112,      0,       0,       0,       0,   0,   0
zio_data_buf_90112:   90112,      0,       0,       0,       0,   0,   0
zio_buf_94208:        94208,      0,       0,       0,       0,   0,   0
zio_data_buf_94208:   94208,      0,       0,       0,       0,   0,   0
zio_buf_98304:        98304,      0,       0,       0,       0,   0,   0
zio_data_buf_98304:   98304,      0,       0,       0,       0,   0,   0
zio_buf_102400:      102400,      0,       0,       0,       0,   0,   0
zio_data_buf_102400: 102400,      0,       0,       0,       0,   0,   0
zio_buf_106496:      106496,      0,       0,       0,       0,   0,   0
zio_data_buf_106496: 106496,      0,       0,       0,       0,   0,   0
zio_buf_110592:      110592,      0,       0,       0,       0,   0,   0
zio_data_buf_110592: 110592,      0,       0,       0,       0,   0,   0
zio_buf_114688:      114688,      0,       0,       0,       0,   0,   0
zio_data_buf_114688: 114688,      0,       0,       0,       0,   0,   0
zio_buf_118784:      118784,      0,       0,       0,       0,   0,   0
zio_data_buf_118784: 118784,      0,       0,       0,       0,   0,   0
zio_buf_122880:      122880,      0,       0,       0,       0,   0,   0
zio_data_buf_122880: 122880,      0,       0,       0,       0,   0,   0
zio_buf_126976:      126976,      0,       0,       0,       0,   0,   0
zio_data_buf_126976: 126976,      0,       0,       0,       0,   0,   0
zio_buf_131072:      131072,      0,       0,       0,       0,   0,   0
zio_data_buf_131072: 131072,      0,       0,       0,       0,   0,   0
sa_cache:                80,      0,      46,     269,      46,   0,   0
dnode_t:                856,      0,     460,     148,    1293,   0,   0
dmu_buf_impl_t:         224,      0,     675,     158,     690,   0,   0
arc_buf_hdr_t:          216,      0,     492,     210,     544,   0,   0
arc_buf_t:              104,      0,     491,     229,     662,   0,   0
zil_lwb_cache:          192,      0,       0,       0,       0,   0,   0
zfs_znode_cache:        400,      0,      46,      62,      46,   0,   0
Mountpoints:            792,      0,      28,      37,      28,   0,   0
pipe:                   728,      0,       1,      79,    1067,   0,   0
ksiginfo:               112,      0,     106,     950,     122,   0,   0
itimer:                 344,      0,       0,       0,       0,   0,   0
pfsrctrpl:              152,  10000,       0,       0,       0,   0,   0
pfrulepl:               936,      0,      15,       5,      15,   0,   0
pfstatepl:              288,  10010,      72,     110,     236,   0,   0
pfstatekeypl:           288,      0,      67,      76,     264,   0,   0
pfstateitempl:          288,      0,      67,      76,     264,   0,   0
pfaltqpl:               240,      0,       0,       0,       0,   0,   0
pfpooladdrpl:            88,      0,      14,      70,      14,   0,   0
pfrktable:             1296,   1002,       0,       0,       0,   0,   0
pfrkentry:              160, 200016,       0,       0,       0,   0,   0
pfrkcounters:            64,      0,       0,       0,       0,   0,   0
pffrent:                 32,   5050,       0,       0,       0,   0,   0
pffrag:                  80,      0,       0,       0,       0,   0,   0
pffrcache:               80,  10035,       0,       0,       0,   0,   0
pffrcent:                24,  50022,       0,       0,       0,   0,   0
pfstatescrub:            40,      0,     133,     371,     439,   0,   0
pfiaddrpl:              120,      0,       0,       0,       0,   0,   0
pfospfen:               112,      0,     700,      26,     700,   0,   0
pfosfp:                  40,      0,     410,      94,     410,   0,   0
KNOTE:                  128,      0,       0,      58,       6,   0,   0
bridge_rtnode:           64,      0,       0,       0,       0,   0,   0
socket:                 680,  25602,       8,      76,     388,   0,   0
ipq:                     56,    819,       0,       0,       0,   0,   0
udp_inpcb:              392,  25600,       0,     120,     338,   0,   0
udpcb:                   16,  25704,       0,    1176,     338,   0,   0
tcp_inpcb:              392,  25600,       0,      20,       3,   0,   0
tcpcb:                  976,  25600,       0,       8,       3,   0,   0
tcptw:                   72,   5150,       0,       0,       0,   0,   0
syncache:               152,  15375,       0,       0,       0,   0,   0
hostcache:              136,  15372,       0,       0,       0,   0,   0
tcpreass:                40,   1680,       0,       0,       0,   0,   0
sackhole:                32,      0,       0,       0,       0,   0,   0
ripcb:                  392,  25600,       1,      19,       1,   0,   0
unpcb:                  240,  25600,       6,     170,      34,   0,   0
rtentry:                200,      0,      85,     105,      86,   0,   0
pfsync:                  88,      0,       0,       0,       0,   0,   0
selfd:                   56,      0,      23,     544,    5751,   0,   0
SWAPMETA:               288, 116519,       0,       0,       0,   0,   0
ip4flow:                 56, 197694,    1016,    1063,    4772,   0,   0
ip6flow:                 80, 197640,      39,     231,     162,   0,   0
FFS inode:              168,      0,     462,     176,     512,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
FFS2 dinode:            256,      0,     462,     138,     512,   0,   0
ipq:                     56,    819,       0,       0,       0,   0,   0
udp_inpcb:              392,  25600,       0,       0,       0,   0,   0
udpcb:                   16,  25704,       0,       0,       0,   0,   0
tcp_inpcb:              392,  25600,       0,       0,       0,   0,   0
tcpcb:                  976,  25600,       0,       0,       0,   0,   0
tcptw:                   72,   5150,       0,       0,       0,   0,   0
syncache:               152,  15375,       0,       0,       0,   0,   0
hostcache:              136,  15372,       0,       0,       0,   0,   0
tcpreass:                40,   1680,       0,       0,       0,   0,   0
sackhole:                32,      0,       0,       0,       0,   0,   0
ripcb:                  392,  25600,       0,       0,       0,   0,   0
rtentry:                200,      0,       0,       0,       0,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq1: atkbd0                           6          0
irq20: atapci0                      3309          0
irq23: ehci0 ehci1                  1642          0
cpu0:timer                         11945          0
irq264: re0                        21841          0
cpu1:timer                          1410          0
cpu5:timer                          1350          0
cpu7:timer                          1982          0
cpu2:timer                          1496          0
cpu4:timer                          2254          0
cpu3:timer                          1897          0
cpu6:timer                          1384          0
Total                              50516          0

------------------------------------------------------------------------
pstat -T

 44/12328 files
0M/32767M swap space

------------------------------------------------------------------------
pstat -s

Device          1K-blocks     Used    Avail Capacity
/dev/gptid/deaa05f3-fb51-11e0-8  16777084        0 16777084     0%
/dev/label/swap1  16777084        0 16777084     0%
Total            33554168        0 33554168     0%

------------------------------------------------------------------------
iostat

iostat: kvm_read(_tk_nin): invalid address (0x0)
iostat: disabling TTY statistics
             ad4              ad6             cpu
  KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
 19.31   0  0.00  14.97   0  0.00   0  0  0  0 100

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME   

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME   

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME   


------------------------------------------------------------------------
ipcs -T

msginfo:
	msgmax:        16384	(max characters in a message)
	msgmni:           40	(# of message queues)
	msgmnb:         2048	(max characters in a message queue)
	msgtql:           40	(max # of messages in system)
	msgssz:            8	(size of a message segment)
	msgseg:         2048	(# of message segments in system)

shminfo:
	shmmax:    536870912	(max shared memory segment size)
	shmmin:            1	(min shared memory segment size)
	shmmni:          192	(max number of shared memory identifiers)
	shmseg:          128	(max shared memory segments per process)
	shmall:       131072	(max amount of shared memory in pages)

seminfo:
	semmni:           50	(# of semaphore identifiers)
	semmns:          340	(# of semaphores in system)
	semmnu:          150	(# of undo structures in system)
	semmsl:          340	(max # of semaphores per id)
	semopm:          100	(max # of operations per semop call)
	semume:           50	(max # of undo entries per process)
	semusz:          632	(size in bytes of undo structure)
	semvmx:        32767	(semaphore maximum value)
	semaem:        16384	(adjust on exit max value)


------------------------------------------------------------------------
nfsstat

nfsstat: new client/server not loaded

------------------------------------------------------------------------
netstat -s

tcp:
	1 packet sent
		0 data packets (0 bytes)
		0 data packets (0 bytes) retransmitted
		0 data packets unnecessarily retransmitted
		0 resends initiated by MTU discovery
		0 ack-only packets (0 delayed)
		0 URG only packets
		0 window probe packets
		0 window update packets
		1 control packet
	294 packets received
		0 acks (for 0 bytes)
		0 duplicate acks
		0 acks for unsent data
		0 packets (0 bytes) received in-sequence
		0 completely duplicate packets (0 bytes)
		0 old duplicate packets
		0 packets with some dup. data (0 bytes duped)
		0 out-of-order packets (0 bytes)
		0 packets (0 bytes) of data after window
		0 window probes
		0 window update packets
		0 packets received after close
		0 discarded for bad checksums
		0 discarded for bad header offset fields
		0 discarded because packet too short
		0 discarded due to memory problems
	1 connection request
	0 connection accepts
	0 bad connection attempts
	0 listen queue overflows
	0 ignored RSTs in the windows
	0 connections established (including accepts)
	3 connections closed (including 0 drops)
		0 connections updated cached RTT on close
		0 connections updated cached RTT variance on close
		0 connections updated cached ssthresh on close
	1 embryonic connection dropped
	0 segments updated rtt (of 1 attempt)
	0 retransmit timeouts
		0 connections dropped by rexmit timeout
	0 persist timeouts
		0 connections dropped by persist timeout
	0 Connections (fin_wait_2) dropped because of timeout
	0 keepalive timeouts
		0 keepalive probes sent
		0 connections dropped by keepalive
	0 correct ACK header predictions
	0 correct data packet header predictions
	0 syncache entries added
		0 retransmitted
		0 dupsyn
		0 dropped
		0 completed
		0 bucket overflow
		0 cache overflow
		0 reset
		0 stale
		0 aborted
		0 badack
		0 unreach
		0 zone failures
	0 cookies sent
	0 cookies received
	0 hostcache entries added
		0 bucket overflow
	0 SACK recovery episodes
	0 segment rexmits in SACK recovery episodes
	0 byte rexmits in SACK recovery episodes
	0 SACK options (SACK blocks) received
	0 SACK options (SACK blocks) sent
	0 SACK scoreboard overflow
	0 packets with ECN CE bit set
	0 packets with ECN ECT(0) bit set
	0 packets with ECN ECT(1) bit set
	0 successful ECN handshakes
	0 times ECN reduced the congestion window
udp:
	6891 datagrams received
	0 with incomplete header
	0 with bad data length field
	0 with bad checksum
	0 with no checksum
	6888 dropped due to no socket
	0 broadcast/multicast datagrams undelivered
	0 dropped due to full socket buffers
	0 not for hashed pcb
	3 delivered
	3 datagrams output
	0 times multicast source filter matched
ip:
	7008 total packets received
	0 bad header checksums
	0 with size smaller than minimum
	0 with data size < data length
	0 with ip length > max ip packet size
	0 with header length < data size
	0 with data length < header length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 packets reassembled ok
	7003 packets for this host
	1 packet for unknown/unsupported protocol
	0 packets forwarded (0 packets fast forwarded)
	4 packets not forwardable
	0 packets received for unknown multicast group
	0 redirects sent
	7001 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	25 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 tunneling packets that can't find gif
	84 datagrams with bad address in header
icmp:
	6706 calls to icmp_error
	0 errors not generated in response to an icmp message
	Output histogram:
		destination unreachable: 6706
	0 messages with bad code fields
	0 messages less than the minimum length
	0 messages with bad checksum
	0 messages with bad length
	0 multicast echo requests ignored
	0 multicast timestamp requests ignored
	Input histogram:
		destination unreachable: 1
	0 message responses generated
	0 invalid return addresses
	0 no return routes
igmp:
	0 messages received
	0 messages received with too few bytes
	0 messages received with wrong TTL
	0 messages received with bad checksum
	0 V1/V2 membership queries received
	0 V3 membership queries received
	0 membership queries received with invalid field(s)
	0 general queries received
	0 group queries received
	0 group-source queries received
	0 group-source queries dropped
	0 membership reports received
	0 membership reports received with invalid field(s)
	0 membership reports received for groups to which we belong
	0 V3 reports received without Router Alert
	0 membership reports sent
pfsync:
	0 packets received (IPv4)
	0 packets received (IPv6)
		0 packets discarded for bad interface
		0 packets discarded for bad ttl
		0 packets shorter than header
		0 packets discarded for bad version
		0 packets discarded for bad HMAC
		0 packets discarded for bad action
		0 packets discarded for short packet
		0 states discarded for bad values
		0 stale states
		0 failed state lookup/inserts
	0 packets sent (IPv4)
	0 packets sent (IPv6)
		0 send failed due to mbuf memory error
		0 send error
arp:
	16 ARP requests sent
	0 ARP replies sent
	1 ARP request received
	1 ARP reply received
	2 ARP packets received
	0 total packets dropped due to no ARP entry
	0 ARP entrys timed out
	0 Duplicate IPs seen
ip6:
	362 total packets received
	0 with size smaller than minimum
	0 with data size < data length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 fragments that exceeded limit
	0 packets reassembled ok
	182 packets for this host
	0 packets forwarded
	0 packets not forwardable
	0 redirects sent
	365 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 packets that violated scope rules
	0 multicast packets which we don't join
	Input histogram:
		UDP: 182
		ICMP6: 164
		no next header: 16
	Mbuf statistics:
		0 one mbuf
		362 one ext mbuf
		0 two or more ext mbuf
	0 packets whose headers are not contiguous
	0 tunneling packets that can't find gif
	0 packets discarded because of too many headers
	0 failures of source address selection
	Source addresses selection rule applied:
		8 first candidate
		47 appropriate scope
icmp6:
	182 calls to icmp6_error
	0 errors not generated in response to an icmp6 message
	0 errors not generated because of rate limitation
	Output histogram:
		unreach: 182
		echo reply: 155
		neighbor solicitation: 22
		neighbor advertisement: 2
		MLDv2 listener report: 4
	0 messages with bad code fields
	0 messages < minimum length
	0 bad checksums
	0 messages with bad length
	Input histogram:
		echo: 155
		neighbor solicitation: 2
		neighbor advertisement: 7
	Histogram of error messages to be generated:
		0 no route
		0 administratively prohibited
		0 beyond scope
		0 address unreachable
		182 port unreachable
		0 packet too big
		0 time exceed transit
		0 time exceed reassembly
		0 erroneous header field
		0 unrecognized next header
		0 unrecognized option
		0 redirect
		0 unknown
	155 message responses generated
	0 messages with too many ND options
	0 messages with bad ND options
	0 bad neighbor solicitation messages
	0 bad neighbor advertisement messages
	0 bad router solicitation messages
	0 bad router advertisement messages
	0 bad redirect messages
	0 path MTU changes
rip6:
	0 messages received
	0 checksum calculations on inbound
	0 messages with bad checksum
	0 messages dropped due to no socket
	0 multicast messages dropped due to no socket
	0 messages dropped due to full socket buffers
	0 delivered
	0 datagrams output

------------------------------------------------------------------------
netstat -m

257/2059/2316 mbufs in use (current/cache/total)
250/1234/1484/25600 mbuf clusters in use (current/cache/total/max)
256/1030 mbuf+clusters out of packet secondary zone in use (current/cache)
0/0/0/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/19200 9k jumbo clusters in use (current/cache/total/max)
0/0/0/12800 16k jumbo clusters in use (current/cache/total/max)
564K/2982K/3547K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

------------------------------------------------------------------------
netstat -id

netstat: invalid address (0xfffffe00096f6c00)
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
usbus     0 <Link#1>                               0     0     0        0     0     0    0 
re0    1500 <Link#2>      54:04:a6:7e:ef:62     7370     0     0     7272     0     0    0 
re0    1500 2a01:4f8:150: 2a01:4f8:150:63e2        5     -     -        6     -     -    - 
re0    1500 fe80::5604:a6 fe80::5604:a6ff:f        1     -     -        4     -     -    - 
re0    1500 176.9.216.32/ static.33.216.9.1        1     -     -        0     -     -    - 
re0    1500 176.9.216.34/ nils.cc               3280     -     -        0     -     -    - 
re0    1500 176.9.216.35/ s4msung.de               3     -     -        0     -     -    - 
re0    1500 176.9.216.36/ static.36.216.9.1        9     -     -        0     -     -    - 
re0    1500 176.9.216.37/ hektor.lnpls.de          2     -     -        0     -     -    - 
re0    1500 176.9.216.38/ fgenesis                 2     -     -        0     -     -    - 
re0    1500 176.9.216.39/ static.39.216.9.1        3     -     -        0     -     -    - 
re0    1500 176.9.216.40/ furnace                 11     -     -        0     -     -    - 
re0    1500 176.9.216.41/ static.41.216.9.1        2     -     -        0     -     -    - 
re0    1500 176.9.216.42/ static.42.216.9.1        4     -     -        0     -     -    - 
re0    1500 176.9.216.43/ static.43.216.9.1        2     -     -        0     -     -    - 
re0    1500 176.9.216.44/ static.44.216.9.1     3452     -     -        0     -     -    - 
re0    1500 176.9.216.45/ static.45.216.9.1        1     -     -        0     -     -    - 
re0    1500 176.9.216.46/ static.46.216.9.1        5     -     -        0     -     -    - 
re0    1500 nils.cc       2a01:4f8:150:63e2      181     -     -      181     -     -    - 
re0    1500 s4msung.de    2a01:4f8:150:63e2        0     -     -        0     -     -    - 
re0    1500 furnace.wzff. 2a01:4f8:150:63e2        0     -     -        0     -     -    - 
re0    1500 2a01:4f8:150: 2a01:4f8:150:63e2        0     -     -        0     -     -    - 
re0    1500 hektor.lnpls. 2a01:4f8:150:63e2        0     -     -        0     -     -    - 
re0    1500 fgenesis.wzff 2a01:4f8:150:63e2        0     -     -        0     -     -    - 
re0    1500 valhalla.phry 2a01:4f8:150:63e2        0     -     -        0     -     -    - 
re0    1500 2a01:4f8:150: 2a01:4f8:150:63e2        0     -     -        0     -     -    - 
re0    1500 2a01:4f8:150: 2a01:4f8:150:63e2        0     -     -        0     -     -    - 
re0    1500 2a01:4f8:150: 2a01:4f8:150:63e2        0     -     -        0     -     -    - 
re0    1500 2a01:4f8:150: 2a01:4f8:150:63e2      157     -     -      157     -     -    - 
re0    1500 2a01:4f8:150: 2a01:4f8:150:63e2        0     -     -        0     -     -    - 
re0    1500 2a01:4f8:150: 2a01:4f8:150:63e2        0     -     -        0     -     -    - 
re0    1500 176.9.47.224/ host                   140     -     -     6890     -     -    - 
usbus     0 <Link#3>                               0     0     0        0     0     0    0 
pflog 33152 <Link#4>                               0     0     0        0     0     0    0 
pfsyn  1500 <Link#5>                               0     0     0        0     0     0    0 
lo0   16384 <Link#6>                               2     0     0        2     0     0    0 
lo0   16384 localhost     ::1                      0     -     -        0     -     -    - 
lo0   16384 fe80::1%lo0   fe80::1                  0     -     -        0     -     -    - 

------------------------------------------------------------------------
netstat -anr

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            176.9.47.225       UGS      1169     6974    re0
127.0.0.1          link#6             UH          0        0    lo0
127.0.1.1          link#6             UH          0        2    lo0
127.0.1.2          link#6             UH          0        0    lo0
127.0.1.3          link#6             UH          0        0    lo0
127.0.1.4          link#6             UH          0        0    lo0
127.0.1.5          link#6             UH          0        0    lo0
127.0.1.6          link#6             UH          0        0    lo0
127.0.1.7          link#6             UH          0        0    lo0
127.0.1.8          link#6             UH          0        0    lo0
127.0.1.9          link#6             UH          0        0    lo0
127.0.1.10         link#6             UH          0        0    lo0
127.0.1.11         link#6             UH          0        0    lo0
127.0.1.12         link#6             UH          0        0    lo0
127.0.2.1          link#6             UH          0        0    lo0
127.0.2.2          link#6             UH          0        0    lo0
127.0.2.3          link#6             UH          0        0    lo0
127.0.2.4          link#6             UH          0        0    lo0
127.0.2.5          link#6             UH          0        0    lo0
127.0.2.6          link#6             UH          0        0    lo0
127.0.2.7          link#6             UH          0        0    lo0
127.0.2.8          link#6             UH          0        0    lo0
127.0.2.9          link#6             UH          0        0    lo0
127.0.2.10         link#6             UH          0        0    lo0
127.0.2.11         link#6             UH          0        0    lo0
127.0.2.12         link#6             UH          0        0    lo0
176.9.47.224/27    link#2             U           0        0    re0
176.9.47.245       link#2             UHS         0        0    lo0
176.9.216.32/28    link#2             U           0        0    re0
176.9.216.33       link#2             UHS         0        0    lo0
176.9.216.34       link#2             UHS         0        0    lo0 =>
176.9.216.34/32    link#2             U           0        0    re0
176.9.216.35       link#2             UHS         0        0    lo0 =>
176.9.216.35/32    link#2             U           0        0    re0
176.9.216.36       link#2             UHS         0        0    lo0 =>
176.9.216.36/32    link#2             U           0        0    re0
176.9.216.37       link#2             UHS         0        0    lo0 =>
176.9.216.37/32    link#2             U           0        0    re0
176.9.216.38       link#2             UHS         0        0    lo0 =>
176.9.216.38/32    link#2             U           0        0    re0
176.9.216.39       link#2             UHS         0        0    lo0 =>
176.9.216.39/32    link#2             U           0        0    re0
176.9.216.40       link#2             UHS         0        0    lo0 =>
176.9.216.40/32    link#2             U           0        0    re0
176.9.216.41       link#2             UHS         0        0    lo0 =>
176.9.216.41/32    link#2             U           0        0    re0
176.9.216.42       link#2             UHS         0        0    lo0 =>
176.9.216.42/32    link#2             U           0        0    re0
176.9.216.43       link#2             UHS         0        0    lo0 =>
176.9.216.43/32    link#2             U           0        0    re0
176.9.216.44       link#2             UHS         0        0    lo0 =>
176.9.216.44/32    link#2             U           0        0    re0
176.9.216.45       link#2             UHS         0        0    lo0 =>
176.9.216.45/32    link#2             U           0        0    re0
176.9.216.46       link#2             UHS         0        0    lo0 =>
176.9.216.46/32    link#2             U           0        0    re0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0 =>
default                           2a01:4f8:150:63e0::1          UGS         re0
::1                               link#6                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
2a01:4f8:150:63e0::/59            link#2                        U           re0
2a01:4f8:150:63e2::2              link#2                        UHS         lo0
2a01:4f8:150:63e2::3              link#2                        UHS         lo0
2a01:4f8:150:63e2::4              link#2                        UHS         lo0
2a01:4f8:150:63e2::5              link#2                        UHS         lo0
2a01:4f8:150:63e2::6              link#2                        UHS         lo0
2a01:4f8:150:63e2::7              link#2                        UHS         lo0
2a01:4f8:150:63e2::8              link#2                        UHS         lo0
2a01:4f8:150:63e2::9              link#2                        UHS         lo0
2a01:4f8:150:63e2::a              link#2                        UHS         lo0
2a01:4f8:150:63e2::b              link#2                        UHS         lo0
2a01:4f8:150:63e2::10             link#2                        UHS         lo0
2a01:4f8:150:63e2::11             link#2                        UHS         lo0
2a01:4f8:150:63e2::12             link#2                        UHS         lo0
2a01:4f8:150:63e2::13             link#2                        UHS         lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%re0/64                     link#2                        U           re0
fe80::5604:a6ff:fe7e:ef62%re0     link#2                        UHS         lo0
fe80::%lo0/64                     link#6                        U           lo0
fe80::1%lo0                       link#6                        UHS         lo0
ff01::%re0/32                     2a01:4f8:150:63e2::2          U           re0
ff01::%lo0/32                     ::1                           U           lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%re0/32                     2a01:4f8:150:63e2::2          U           re0
ff02::%lo0/32                     ::1                           U           lo0

------------------------------------------------------------------------
netstat -anA

Active UNIX domain sockets
Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
fffffe0009d80e10 stream      0      0        0 fffffe0009c99000        0        0
fffffe0009c99000 stream      0      0        0 fffffe0009d80e10        0        0
fffffe0009d86780 stream      0      0 fffffe00092b31e0        0        0        0 /var/run/devd.pipe
fffffe0009d86690 dgram       0      0        0 fffffe0009d8fe10        0        0
fffffe0009d8fe10 dgram       0      0 fffffe0009c5a780        0 fffffe0009d86690        0 /var/run/logpriv
fffffe0009d80000 dgram       0      0 fffffe0009c5a960        0        0        0 /var/run/log

------------------------------------------------------------------------
netstat -aL

unix  0/0/4          /var/run/devd.pipe

------------------------------------------------------------------------
fstat

Suspended

------------------------------------------------------------------------
dmesg

Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.1-RELEASE #3 r246626: Sun Feb 10 18:26:17 UTC 2013
    root@host.wzff.de:/usr/obj/usr/src/sys/RCTL_ALTQ amd64
can't re-use a leaf (geom_label)!
can't re-use a leaf (geom_mirror)!
module_register: module g_label already exists!
Module g_label failed to register: 17
module_register: module g_mirror already exists!
Module g_mirror failed to register: 17
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz (3411.56-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206a7  Family = 6  Model = 2a  Stepping = 7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x1fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16432263168 (15671 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <ALASKA A M I> on motherboard
ACPI Error: [RAMB] Namespace lookup failure, AE_NOT_FOUND (20110527/psargs-392)
ACPI Exception: AE_NOT_FOUND, Could not execute arguments for [RAMW] (Region) (20110527/nsinit-380)
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xfe000000-0xfe3fffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on pci0
pci0: <simple comms> at device 22.0 (no driver attached)
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfe503000-0xfe5033ff irq 23 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci3: <ACPI PCI bus> on pcib3
pci3: <serial bus, USB> at device 0.0 (no driver attached)
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0
pci4: <ACPI PCI bus> on pcib4
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0xd0004000-0xd0004fff,0xd0000000-0xd0003fff irq 18 at device 0.0 on pci4
re0: Using 1 MSI-X message
re0: Chip rev. 0x2c000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 54:04:a6:7e:ef:62
pcib5: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 0.0 on pci5
pci6: <ACPI PCI bus> on pcib6
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfe502000-0xfe5023ff irq 23 at device 29.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel Cougar Point SATA300 controller> port 0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xfe501000-0xfe5017ff irq 20 at device 31.2 on pci0
atapci0: AHCI called from vendor specific driver
atapci0: AHCI v1.30 controller with 6 6Gbps ports, PM not supported
ata2: <ATA channel> at channel 0 on atapci0
ata3: <ATA channel> at channel 1 on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcd7ff,0xcd800-0xce7ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
ZFS filesystem version 5
ZFS storage pool version 28
Timecounters tick every 1.000 msec
The GEOM class MIRROR is already loaded.
The GEOM class LABEL is already loaded.
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ad4: 2861588MB <Seagate ST33000651AS CC45> at ata2-master UDMA100 SATA 6Gb/s
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ad6: 2861588MB <Seagate ST33000651AS CC45> at ata3-master UDMA100 SATA 6Gb/s
SMP: AP CPU #1 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #6 Launched!
Timecounter "TSC-low" frequency 13326417 Hz quality 1000
GEOM_MIRROR: Device mirror/root launched (2/2).
GEOM_MIRROR: Device mirror/usr launched (2/2).
GEOM_MIRROR: Device mirror/var launched (2/2).
GEOM_MIRROR: Device mirror/tmp launched (2/2).
GEOM_MIRROR: Component ad4p7 (device zfs) broken, skipping.
GEOM_MIRROR: Device mirror/zfs launched (1/2).
GEOM_MIRROR: Component gptid/78c5646b-fb5e-11e0-80cc-5404a67eef62 (device zfs) broken, skipping.
GEOM_MIRROR: Cannot add disk gptid/78c5646b-fb5e-11e0-80cc-5404a67eef62 to zfs (error=22).
Root mount waiting for: usbus1 usbus0
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
Root mount waiting for: usbus1 usbus0
ugen0.2: <vendor 0x8087> at usbus0
uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen1.2: <vendor 0x8087> at usbus1
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
Root mount waiting for: usbus1 usbus0
uhub2: 6 ports with 6 removable, self powered
uhub3: 8 ports with 8 removable, self powered
Root mount waiting for: usbus1
ugen1.3: <Peppercon AG> at usbus1
umass0: <Peppercon AG MultiDevice, class 0/0, rev 1.10/0.01, addr 3> on usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x4100
umass0:0:0:-1: Attached to scbus0
ukbd0: <Peppercon AG MultiDevice, class 0/0, rev 1.10/0.01, addr 3> on usbus1
kbd2 at ukbd0
uhid0: <Peppercon AG MultiDevice, class 0/0, rev 1.10/0.01, addr 3> on usbus1
Root mount waiting for: usbus1
ugen1.4: <CHESEN> at usbus1
ukbd1: <CHESEN PS2 to USB Converter, class 0/0, rev 1.10/0.10, addr 4> on usbus1
kbd3 at ukbd1
ums0: <CHESEN PS2 to USB Converter, class 0/0, rev 1.10/0.10, addr 4> on usbus1
ums0: 5 buttons and [XYZ] coordinates ID=1
Trying to mount root from ufs:/dev/mirror/root [rw]...
WARNING: / was not properly dismounted
Setting hostuuid: 2062ea40-5bcb-11d9-98a3-5404a67eef62.
Setting hostid: 0x832a19a6.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
/dev/mirror/root: DEFER FOR BACKGROUND CHECKING
/dev/mirror/var: DEFER FOR BACKGROUND CHECKING
/dev/mirror/tmp: DEFER FOR BACKGROUND CHECKING
/dev/mirror/usr: DEFER FOR BACKGROUND CHECKING
WARNING: / was not properly dismo
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2013-02-13 15:58:50 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-jail

Fix synopsis and assign.
Comment 2 joeb1 2013-05-15 14:53:33 UTC
I have not debugged your dump.
From your description I see you have the pf firewall altq traffic shaper
compiled into the kernel with vimage.
My current understanding is pf has been fixed to run on the host when vimage
is compiled into the kernel.
But altq has not been made vimage aware yet and there is no pr on this
problem.

I have tested trying to get pf to start in a vnet/vimage jail without
success.
Pf suffers from the rc.d nojail keyword problem. This command sequence is a
work around.

Jexec jailname service netif start
Jexec jailname service routing start
Jexec jailname service pf start

And of course you have to do the reverse before stopping the said jail.

My suggestion is to remove altq from being compiled into the kernel and
don't use altq on a host that has
vimage compiled into its kernel.
Comment 3 moritz 2013-05-15 18:18:46 UTC
Thanks. I don't actually use ALTQ at the moment, so just disabling it should
do the trick for now.
Comment 4 Mark Linimon 2014-05-13 18:53:13 UTC
On Tue, May 13, 2014 at 10:35:18AM -0500, Dustin Wenz wrote:
> Is there any way this PR could be bumped up in priority?

Unfortunately over the years the priority -related fields have been so
abused as to have become meaningless.

mcl
Comment 5 Mark Linimon 2014-05-13 18:53:58 UTC
----- Forwarded message from Dustin Wenz <dustinwenz@ebureau.com> -----

Date: Tue, 13 May 2014 10:35:18 -0500
From: Dustin Wenz <dustinwenz@ebureau.com>
To: bugbusters@FreeBSD.org
Subject: Update request: kernel panic when starting jails
X-Mailer: Apple Mail (2.1874)

I would like to update PR:

	kern/176112: [jail] [panic] kernel panic when starting jails

This problem is still present in FreeBSD 10.0-STABLE #0 r265159, when
built for amd64. Is there any way this PR could be bumped up in priority?
The current description of the bug appears adequate, even if it's over
a year old. Please let me know if any more information would be helpful.

Thanks,

	- .Dustin Wenz

----- End forwarded message -----
Comment 6 Craig Rodrigues freebsd_committer freebsd_triage 2014-12-13 12:46:12 UTC

*** This bug has been marked as a duplicate of bug 194515 ***