Description Adrian Chadd 2012-01-22 19:50:10 UTC

I saw a crash inside the net80211 stack when either deleting or down'ing a vap.


Unread portion of the kernel message buffer:
KDB: stack backtrace:
#0 0xc0727697 at kdb_backtrace+0x47
#1 0xc073b675 at _witness_debugger+0x25
#2 0xc073cb8e at witness_warn+0x1fe
#3 0xc095e465 at trap+0x195
#4 0xc09478ac at calltrap+0x6
#5 0xc77e2bf1 at ieee80211_free_node_debug+0xb1
#6 0xc77ce017 at ieee80211_input_mimo_all+0xe7
#7 0xc77cdf22 at ieee80211_input_all+0x32
#8 0xc784dcc5 at ath_rx_proc+0xc45
#9 0xc784d071 at ath_rx_tasklet+0x101
#10 0xc073446b at taskqueue_run_locked+0xeb
#11 0xc0734ec7 at taskqueue_thread_loop+0x67
#12 0xc06c76b8 at fork_exit+0xb8
#13 0xc0947924 at fork_trampoline+0x8

The debugging indicated something rather amusing at this point.


ath0: ath_node_alloc: an 0xc7adf000
ieee80211_ref_node: 0xc7adf000: ieee80211_reset_bss /usr/home/adrian/work/freebsd/ath/head/src/sys/modules/wl
an/../../net80211/ieee80211_node.c:434
wlan0: Ethernet address: 00:03:7f:11:a3:f3
ath0: ath_init: if_flags 0x8803
ath0: ath_stop_locked: invalid 0 if_flags 0x8803
ath0: ath_newstate: INIT -> INIT
ath0: ath_newstate: RX filter 0x6497 bssid 00:00:00:00:00:00 aid 0x0
ath0: ath_newstate: INIT -> SCAN
ath0: ath_newstate: RX filter 0x6497 bssid 00:00:00:00:00:00 aid 0x0
ath0: ath_node_alloc: an 0xc7aea000

. now at this point, there are two sets of messages which overlap, indicating that they ran concurrently:

ieee80211_ref_node: 0xc7aea000: ieee80211_create_ibss /usr/home/adrian/work/freebsd/ath/head/src/sys/modules/
wlan/../../net802

ieee80211_ref_node: 0xc7adf000: ieee80211_input_mimo_all /usr/home/adrian/work/freebsd/ath/head/src/sys/modul
es/wlan/../../net

11/ieee80211_node.c:412

80211/ieee80211_input.c:143
ath0: ath_node_free: ni 0xc7adf000

. and bang:

Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex ath0_node_lock (ath0_node_lock) r = 0 (0xc79316c0) locked @ /usr/home/adrian/work/freeb
sd/ath/head/src/sys/modules/wlan/../../net80211/ieee80211_node.c:1702

I'm gathering here that the delete was ongoing whilst traffic was being processed via ath_rx_tasklet() and the underlying vap was either deleted or the vap->iv_bss node was changed.

There seems to be a larger class of bugs where the vap->iv_bss node is changed in parallel with some other process (eg beacon free/alloc) without suitable locking.

Fix: 

I think the RX path should be properly aborted during a a vap down/delete. This doesn't just mean stopping the hardware (which is what ath_stop_locked() currently does) but also waiting for the ath_rx_tasklet() and the TX completion tasklet to complete.
How-To-Repeat: It's difficult to reproduce. I reproduced it in a lab environment with lots of busy air. I guess anything that triggers constant incoming traffic and keeps the RX queue deep is going to make triggering this bug.

What needs to happen:

* ath_rx_tasklet() needs to take a while to run;
* the ifconfig process (and net80211 taskqueue) needs to be scheduled on another CPU, so it can run _in parallel_ with the ath taskqueue (which ath_rx_tasklet() runs in)
* somehow you have to get a vap down/delete in during this RX.