Bug 123892 - [tap] [patch] No buffer space available
Summary: [tap] [patch] No buffer space available
Status: Closed Overcome By Events
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: Unspecified
Hardware: Any Any
: Normal Affects Only Me
Assignee: Kyle Evans
Depends on:
Reported: 2008-05-22 13:10 UTC by Oleg
Modified: 2019-10-06 02:33 UTC (History)
2 users (show)

See Also:

file.txt (3.34 KB, text/plain)
2008-05-22 13:10 UTC, Oleg
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Oleg 2008-05-22 13:10:01 UTC

First, I meet this bug along time ago
and write small program, similar to
but, that work with tap device, to reproduce bug.

And no feedback come.

Later, I make small research while debugging kernel 
and again no help :(

Today I got this bug agan but with another way:

You need to have multicore machine, FreeBSD 7 kernel with 

options SCHED_ULE
options SMP 


# PREEMPTION allows the threads that are in the kernel to be preempted
#     by higher priority threads.  It helps with interactivity and
#     allows interrupt threads to run sooner rather than waiting.
#     WARNING! Only tested on amd64 and i386.

When preempt is possible tap-device have 2 unsafe functions (if_tap.c):
tapifstart and tapread.

static int
tapread(struct cdev *dev, struct uio *uio, int flag)
	// skipped
	// ..

	/* sleep until we get a packet */
	do {
		s = splimp();
		IF_DEQUEUE(&ifp->if_snd, m);

		// (1)
		if (m == NULL) {
			if (flag & O_NONBLOCK)
				return (EWOULDBLOCK);

			tp->tap_flags |= TAP_RWAIT;
			// (2)
			error = tsleep(tp,PCATCH|(PZERO+1),"taprd",0);
			if (error)
				return (error);
	} while (m == NULL);

	// skipped
	// ..

static void
tapifstart(struct ifnet *ifp)
	// skipped
	// ..

	s = splimp();
	ifp->if_drv_flags |= IFF_DRV_OACTIVE;

	if (ifp->if_snd.ifq_len != 0) {
		if (tp->tap_flags & TAP_RWAIT) {
			tp->tap_flags &= ~TAP_RWAIT;
			// (3)

		if ((tp->tap_flags & TAP_ASYNC) && (tp->tap_sigio != NULL)) {
			pgsigio(&tp->tap_sigio, SIGIO, 0);
		} else

		selwakeuppri(&tp->tap_rsel, PZERO+1);
		KNOTE_UNLOCKED(&tp->tap_rsel.si_note, 0);
		ifp->if_opackets ++; /* obytes are counted in ether_output */

	ifp->if_drv_flags &= ~IFF_DRV_OACTIVE;

So, when some process make block read on tap-device, tapread called,
it check  if_snd, and if it empty make sleep until we get a packet for
sending (tapifstart called).

Now imagine, that tapread thread see that queue are empty and going to
sleep, but between (1) and (2) interrupted.
While that thread rest interrupted someplace between (1)-(2)
network stack got or generate many packets for tap-device
(50 e.g. >= if_snd.ifq_maxlen), tapifstart called for each packet,
it make wakeup (3) for tapread thread, but they are lost, because no threads
sleeping. Now really if_snd queue are full.

Next, our tapread thread execution returned and go sleep - never wakeup,
because if_snd queue are full, tcp/ip stack will drop new packets for tap-device
and will not call tapifstart.

p.s. tun device have same problem.

Best regards,
   Oleg Dolgov.

Fix: Here are patch, condition variable with if_snd.ifq_mtx
make necessary synchronization.

Patch attached with submission follows:
Comment 1 Volker Werth freebsd_committer 2008-05-24 00:19:33 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-net

Over to maintainer(s).
Comment 2 Eitan Adler freebsd_committer freebsd_triage 2017-12-31 08:00:40 UTC
For bugs matching the following criteria:

Status: In Progress Changed: (is less than) 2014-06-01

Reset to default assignee and clear in-progress tags.

Mail being skipped
Comment 3 Kyle Evans freebsd_committer 2019-10-06 02:33:37 UTC
In the 11 years since this issue failed to receive any feedback (=( apologies), this looks to be overcome by events.  tunread() (in a tuntap(4) world, mind you- so fixed in both tun/tap) holds the tun lock, sets TUN_RWAIT, then uses mtx_sleep(9). tunstart/tunstart_l2 both grab the tun lock and then wakeup(9) if if_snd is no longer empty.