Comment 1 Hans Petter Selasky 2014-10-24 14:03:38 UTC

Hi,

Here are some more mbuf allocations, after TUN close, which my test program did not mark as stuck:

KDB: stack backtrace:
db_trace_self_wrapper(c0a38777,c0a678ac,c06c87f0,c7bd3838,a,c7bd3880,5,ffffffff,0,0,c326fd18,c13eea14,c13eea10,c7bd3860,c06c824
e,c7bd3880,c7bd3874,c06c82fb,c0a678ac,c7bd3880) at db_trace_self_wrapper+0x26/frame 0xc7bd3818
kdb_backtrace(c0a678ac,b6,20,1,c0ab8680,...) at kdb_backtrace+0x2b/frame 0xc7bd3874
uma_zalloc_arg(c13e45a0,c7bd38ec,1,c0b0a194,c7bd3914,...) at uma_zalloc_arg+0x706/frame 0xc7bd38c4
rt_msg1(c7bd3914,30,c2a54800,0,0,...) at rt_msg1+0x59/frame 0xc7bd3900
rtsock_addrmsg(2,c3250e00,0,c2eee9b4,0,...) at rtsock_addrmsg+0x73/frame 0xc7bd3950
rtinit(c3250e00,2,0,c7bd3aa8,0,...) at rtinit+0x1a4/frame 0xc7bd3a58
tunclose(c3281800,7,2000,c2afc2f0,c7bd3abc,...) at tunclose+0x280/frame 0xc7bd3a80
devfs_close(c7bd3af4,c7bd3af4,c338cd50,7,c7bd3b18,...) at devfs_close+0x17f/frame 0xc7bd3ac4
VOP_CLOSE_APV(c0a99ce0,c7bd3af4,c0a40f74,141,c0ad08a0,...) at VOP_CLOSE_APV+0x4a/frame 0xc7bd3adc
vn_close(c338cd50,7,c2956180,c2afc2f0,0,...) at vn_close+0x99/frame 0xc7bd3b18
vn_closefile(c314b460,c2afc2f0,c314b460,0,c2afc2f0,...) at vn_closefile+0x53/frame 0xc7bd3b74
devfs_close_f(c314b460,c2afc2f0,3000000,0,1,...) at devfs_close_f+0x34/frame 0xc7bd3b90
_fdrop(c314b460,c2afc2f0,0,c7bd3c00,2,0,0,c2b7d2d8,4,2,c7bd3c1c,c09b22e9,c2957760,288df000,2,0,c7bd3c10,c06462a4,1f,c314b460) at _fdrop+0x2d/frame 0xc7bd3bac
closef(c314b460,c2afc2f0,0,c7bd3c38,c09b1d76,...) at closef+0x5b/frame 0xc7bd3c10
kern_close(c2afc2f0,6,c7bd3c98,c09bb3b2,c0aff1c0,...) at kern_close+0x18d/frame 0xc7bd3c48
syscall(c7bd3d08) at syscall+0x535/frame 0xc7bd3cfc
Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xc7bd3cfc
--- syscall (6, FreeBSD ELF32, sys_close), eip = 0x283c2393, esp = 0xbfbfe37c, ebp = 0xbfbfe388 ---

KDB: stack backtrace:
db_trace_self_wrapper(c0a38777,c0a678ac,c06c87f0,c7bd380c,a,c7bd3854,5,ffffffff,0,0,c0ab35e0,c7bd3840,c0676dca,c7bd3834,c06c824
e,c7bd3854,c7bd3848,c06c82fb,c0a678ac,c7bd3854) at db_trace_self_wrapper+0x26/frame 0xc7bd37ec
kdb_backtrace(c0a678ac,b6,20,1,c7bd38c0,...) at kdb_backtrace+0x2b/frame 0xc7bd3848
uma_zalloc_arg(c13e45a0,c7bd38c0,1,936,c7bd38e4,...) at uma_zalloc_arg+0x706/frame 0xc7bd3898
rt_msg1(c7bd38e4,30,0,c32eba00,c32eba1c,...) at rt_msg1+0x59/frame 0xc7bd38d4
rtsock_routemsg(2,c2a54800,0,c2eee9b4,0,...) at rtsock_routemsg+0x4f/frame 0xc7bd3920
rt_newaddrmsg_fib(2,c3250e00,0,c2eee9b4,0,...) at rt_newaddrmsg_fib+0x4e/frame 0xc7bd3950
rtinit(c3250e00,2,0,c7bd3aa8,0,...) at rtinit+0x1a4/frame 0xc7bd3a58
tunclose(c3281800,7,2000,c2afc2f0,c7bd3abc,...) at tunclose+0x280/frame 0xc7bd3a80
devfs_close(c7bd3af4,c7bd3af4,c338cd50,7,c7bd3b18,...) at devfs_close+0x17f/frame 0xc7bd3ac4
VOP_CLOSE_APV(c0a99ce0,c7bd3af4,c0a40f74,141,c0ad08a0,...) at VOP_CLOSE_APV+0x4a/frame 0xc7bd3adc
vn_close(c338cd50,7,c2956180,c2afc2f0,0,...) at vn_close+0x99/frame 0xc7bd3b18
vn_closefile(c314b460,c2afc2f0,c314b460,0,c2afc2f0,...) at vn_closefile+0x53/frame 0xc7bd3b74
devfs_close_f(c314b460,c2afc2f0,3000000,0,1,...) at devfs_close_f+0x34/frame 0xc7bd3b90
_fdrop(c314b460,c2afc2f0,0,c7bd3c00,2,0,0,c2b7d2d8,4,2,c7bd3c1c,c09b22e9,c2957760,288df000,2,0,c7bd3c10,c06462a4,1f,c314b460) a
t _fdrop+0x2d/frame 0xc7bd3bac
closef(c314b460,c2afc2f0,0,c7bd3c38,c09b1d76,...) at closef+0x5b/frame 0xc7bd3c10
kern_close(c2afc2f0,6,c7bd3c98,c09bb3b2,c0aff1c0,...) at kern_close+0x18d/frame 0xc7bd3c48
syscall(c7bd3d08) at syscall+0x535/frame 0xc7bd3cfc
Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xc7bd3cfc
--- syscall (6, FreeBSD ELF32, sys_close), eip = 0x283c2393, esp = 0xbfbfe37c, ebp = 0xbfbfe388 ---

Comment 2 Andrey V. Elsukov 2014-10-27 08:23:44 UTC

Hi, Hans,

If I understand correctly, you are able to track what mbuf were allocated and not freed in some period. Is it possible to modify your patch for printing content of these mbufs? I mean something like this:

struct ip *ip;
struct ip6_hdr *ip6;

if (m->m_len > sizeof(struct ip)) {
  ip = mtod(m, struct ip*);
  printf("IP version: %u\n", ip->ip_v);
  switch(ip->ip_v) {
  case IPVERSION:
  /* print ip_src, ip_dst, ip_p */
  break;
  case (IPV6_VERSION >> 4):
  ip6 = mtod(m, struct ip6_hdr *);
  /* print ip6_src, ip6_dst, ip6_nxt */
  break;
  }
}

Comment 3 Hans Petter Selasky 2014-10-27 08:36:58 UTC

Yes, can you send me a working example code with everything you want to print, and I'll add it to the kernel, taking a "struct mbuf *" as input.

--HPS

Comment 4 Andrey V. Elsukov 2014-10-27 09:06:01 UTC

Created attachment 148695 [details]
kernel function for printing mbuf info

Comment 5 Andrey V. Elsukov 2014-10-27 09:07:17 UTC

the function is untested, so be careful :)

Comment 6 Hans Petter Selasky 2014-10-27 10:00:32 UTC

Testing right now. You will have the result soonish ...

Comment 7 Hans Petter Selasky 2014-10-27 10:15:09 UTC

Hi,

Here are the candidates for lost mbufs:

Stuck MBUF[2] TID=100082 LEN=0
mbuf 0xc3265e00: len 20, flags 0x00000002
mbuf 0xc3265e00: ip_v 0
Stuck MBUF[3] TID=100087 LEN=0
mbuf 0xc2b04e00: len 20, flags 0x00000002
mbuf 0xc2b04e00: ip_v 0
Stuck MBUF[5] TID=100087 LEN=0
mbuf 0xc2b02900: len 20, flags 0x00000002
mbuf 0xc2b02900: ip_v 0
Stuck MBUF[22] TID=100082 LEN=0
mbuf 0xc2b05100: len 20, flags 0x00000002
mbuf 0xc2b05100: ip_v 0
Stuck MBUF[23] TID=100082 LEN=0
mbuf 0xc2b07600: len 20, flags 0x00000002
mbuf 0xc2b07600: ip_v 0
Stuck MBUF[89] TID=100087 LEN=0
mbuf 0xc325da00: len 20, flags 0x00000002
mbuf 0xc325da00: ip_v 0 

Note, this is 9-stable, so the M_PRINT_FLAGS was not there, and I used 0x%08x instead.

--HPS

Comment 8 Andrey V. Elsukov 2014-10-27 19:54:35 UTC

Comment on attachment 148695 [details]
kernel function for printing mbuf info

>	printf("mbuf %p: len %u, flags %b\n", m, m->m_len,
>	    m->m_flags, M_FLAG_PRINTF);

Nothing specific :(
It seems the last what here can be interesting is the following:

if (m->m_flags & M_PKTHDR)
  printf("  hdrlen %u\n", m, m->m_pkthdr.len)

>	if (m->m_len < sizeof(struct ip))
>		return;

Comment 9 Hans Petter Selasky 2014-10-28 11:18:13 UTC

Hi,

Here is how to reproduce:

FreeBSD-9-stable:

1) install openvpn from ports
2) generate an openvpn key
3) start two instances of openvpn like this (no need for a client!)

/usr/local/sbin/openvpn --keepalive 120 240 --float --lport 543 --dev tun3 \
 --ifconfig 10.1.2.6 10.1.2.7 \
 --secret xxxx.key --daemon testlink

/usr/local/sbin/openvpn --proto tcp_server --keepalive 120 240 --float --lport 544 --dev tun4 \
 --ifconfig 10.1.2.8 10.1.2.9 \
 --secret xxxx.key --daemon testlink2

4) watch

vmstat -z | grep -E "LIMIT|mbuf:"

After some minutes, the number of mbufs in use starts growing simply.

Maybe that's simpler for you to reproduce?

--HPS

Comment 10 Andrey V. Elsukov 2014-10-29 21:02:41 UTC

Created attachment 148778 [details]
Proposed patch

Hi, Hans,

can you try this patch?
My investigations led me to the following conclusions.
The leak isn't specific to tun(4) device, it could be reproduced with any device where MLD works.

The backtrace to the allocation that will not be freed is

uma_zalloc_arg
mld_v2_enqueue_group_record+0x678
mld_change_state+0x3b9
in6_mc_join_locked+0x346
in6_mc_join+0x94
in6_joingroup+0x58
in6_update_ifa+0xd2c
in6_ifattach+0x506
ifioctl+0x8e0
kern_ioctl+0x3cd
sys_ioctl+0x13c

Comment 11 Hans Petter Selasky 2014-10-29 21:52:30 UTC

Hi,

I see no more buffer leakages currently. I'll let you know if I find more.

Don't forget to MFC to 9- and 10- stable. Might be possible to get it in before 10.1-R too ...

--HPS

Comment 12 Hans Petter Selasky 2014-10-30 07:28:26 UTC

Created attachment 148792 [details]
Cleanup altq before destroy

Hi,

While at it, I suggest you add the attached patch to the commit aswell. What do you think?

--HPS

Comment 13 commit-hook 2014-10-30 11:00:06 UTC

A commit references this bug:

Author: ae
Date: Thu Oct 30 10:59:58 UTC 2014
New revision: 273855
URL: https://svnweb.freebsd.org/changeset/base/273855

Log:
  Fix mbuf leak in IPv6 multicast code.
  When multicast capable interface goes away, it leaves multicast groups,
  this leads to generate MLD reports, but MLD code does deffered send and
  MLD reports are queued in the in6_multi's in6m_scq ifq. The problem is
  that in6_multi structures are freed when interface leaves multicast groups
  and thread that does deffered send will not take these queued packets.

  PR:		194577
  MFC after:	1 week
  Sponsored by:	Yandex LLC

Changes:
  head/sys/netinet6/in6_mcast.c

Comment 14 Andrey V. Elsukov 2014-10-30 11:01:00 UTC

Patched in head/.

Comment 15 commit-hook 2014-11-06 09:17:08 UTC

A commit references this bug:

Author: ae
Date: Thu Nov  6 09:16:53 UTC 2014
New revision: 274168
URL: https://svnweb.freebsd.org/changeset/base/274168

Log:
  MFC r273855:
    Fix mbuf leak in IPv6 multicast code.
    When multicast capable interface goes away, it leaves multicast groups,
    this leads to generate MLD reports, but MLD code does deffered send and
    MLD reports are queued in the in6_multi's in6m_scq ifq. The problem is
    that in6_multi structures are freed when interface leaves multicast groups
    and thread that does deffered send will not take these queued packets.

    PR:		194577

  MFC r273857:
    Move ifq drain into in6m_purge().

    Suggested by:	bms

  Sponsored by:	Yandex LLC

Changes:
_U  stable/10/
  stable/10/sys/netinet6/in6_mcast.c

Comment 16 commit-hook 2014-11-06 16:32:50 UTC

A commit references this bug:

Author: ae
Date: Thu Nov  6 16:31:49 UTC 2014
New revision: 274181
URL: https://svnweb.freebsd.org/changeset/base/274181

Log:
  MFC r273855:
    Fix mbuf leak in IPv6 multicast code.
    When multicast capable interface goes away, it leaves multicast groups,
    this leads to generate MLD reports, but MLD code does deffered send and
    MLD reports are queued in the in6_multi's in6m_scq ifq. The problem is
    that in6_multi structures are freed when interface leaves multicast groups
    and thread that does deffered send will not take these queued packets.

    PR:		194577

  MFC r273857:
    Move ifq drain into in6m_purge().

    Suggested by:	bms
  Sponsored by:	Yandex LLC

Changes:
_U  stable/9/sys/
  stable/9/sys/netinet6/in6_mcast.c