Bug 195102 - dummynet_send() may panic the kernel (bad switch -256)
Summary: dummynet_send() may panic the kernel (bad switch -256)
Status: Closed DUPLICATE of bug 220078
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 8.4-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: Eugene Grosbein
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-11-17 13:35 UTC by Eugene Grosbein
Modified: 2018-10-22 17:19 UTC (History)
2 users (show)

See Also:


Attachments
crashdump details (8.00 KB, text/plain)
2014-11-17 13:35 UTC, Eugene Grosbein
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Eugene Grosbein 2014-11-17 13:35:15 UTC
Created attachment 149513 [details]
crashdump details

This panic occured in 8.4-STABLE for me. I know stable/8 approaches it EOL but the HEAD's code in question is the same so I believe the bug is not fixed there too.

Sometimes I have similar kernel panics with my mpd5/PPPoE access server acting as traffic shaper using dummynet and io_fast enabled. This time I've got good crashdump and spent some time reading the code and I believe the culprit is dummynet_send() function from ipfw/ip_dn_io.c

Kgdb backtrace and additional info are attached.

Here is part of kernel log:

Nov 17 17:02:21 m-19-pc-2 kernel: dummynet: bad switch -256!
Nov 17 17:02:21 m-19-pc-2 kernel:
Nov 17 17:02:21 m-19-pc-2 kernel:
Nov 17 17:02:21 m-19-pc-2 kernel: Fatal trap 12: page fault while in kernel mode
Nov 17 17:02:21 m-19-pc-2 kernel: cpuid = 0; apic id = 00
Nov 17 17:02:21 m-19-pc-2 kernel: fault virtual address = 0x1
Nov 17 17:02:21 m-19-pc-2 kernel: fault code            = supervisor read instruction, page not present
Nov 17 17:02:21 m-19-pc-2 kernel: instruction pointer   = 0x20:0x1
Nov 17 17:02:21 m-19-pc-2 kernel: stack pointer         = 0x28:0xffffff8122b0ba20

As one can see from dummynet_send() code, "bad switch" in the log means that tag = m_tag_first(m) was not NULL at the moment of the check. However, kgdb shows (see attachment) that is was NULL at the moment of kernel panic.

It seems for me we have some kind of race here, so the mbuf is processed and freed in between of these two moments and UMA panices due to double free attempt. I see no protection from this kind of race.

The box has 4 CPU cores (hyperthreading disabled) and these tunnables enabled:

net.isr.bindthreads=1
net.isr.maxthreads=4
net.inet.ip.fastforwarding=1
net.inet.ip.dummynet.pipe_slot_limit=1000
net.inet.ip.dummynet.io_fast=1

sysctls net.isr.direct and net.isr.direct_force are 1 by default
Comment 1 Eugene Grosbein 2014-11-17 13:45:50 UTC
kernel.debug and vmcore file are available for download, 189MB total:

http://www.grosbein.net/freebsd/crash/20141117/kernel.debug.xz
http://www.grosbein.net/freebsd/crash/20141117/vmcore.6.xz
Comment 2 Eugene Grosbein 2014-11-17 13:53:26 UTC
Also, the box has systctl kern.ipc.nmbclusters=400000 and about 17% of them was utilized just before panic, as per output of:

vmstat -z | awk 'BEGIN { FS=","; OFMT="%.0f"; } /mbuf_cluster:/ { print $3*100/$2 }'
Comment 3 Eugene Grosbein freebsd_committer 2018-10-21 22:24:20 UTC

*** This bug has been marked as a duplicate of bug 220078 ***