Bug 164763 - [vnet] Memory leak in VNET
Summary: [vnet] Memory leak in VNET
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 9.0-STABLE
Hardware: Any Any
: Normal Affects Many People
Assignee: Bjoern A. Zeeb
URL:
Keywords: needs-patch, needs-qa, vimage
Depends on:
Blocks:
 
Reported: 2012-02-04 15:10 UTC by Poul-Henning Kamp
Modified: 2016-12-21 21:26 UTC (History)
11 users (show)

See Also:
koobs: mfc-stable10?
koobs: mfc-stable9?


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Poul-Henning Kamp 2012-02-04 15:10:09 UTC
	Destroying a jail with attached vnet interfaces results in UMA
	keg leakage:

	Freed UMA keg was not empty (30 items).  Lost 2 pages of memory.
	Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
	hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
	hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required

How-To-Repeat: 
	ifconfig epair0 create
	jail -i -c vnet name=ext persist
	jexec ext ifconfig
	jail -r ext
Comment 1 Mark van Dijk 2013-01-09 09:20:44 UTC
Our current rule of thumb is "if you need to shutdown one vimage jail
then you need to reboot the host."

So we just shut down the services in each jail, leave the jails
themselves up, and just reboot the host.

Of course this is far from optimal. Is this PR still on the radar of
anyone?
Comment 2 joeb1 2014-05-02 16:18:25 UTC
This is still occurring 1n RELEASE 8.x, RELEASE 9.x, and RELEASE 10.0.
Comment 3 Hiroo Ono 2014-05-22 16:02:36 UTC
I tracked this issue down a little.

I put some printf's and found that this occurs in the function
vnet_destroy() in sys/net/vnet.c.
The memory leak seems to happen when vnet_sysuninit() is called.
So, something needs to be done before or inside vnet_sysuninit(), but
I do not know
what to do.

The messages I saw on head and stable/10 around 2014-04-30 are:
Freed UMA keg (udp_inpcb) was not empty (135 items). Lost 9 pages of memory.
Freed UMA keg (udpcb) was not empty (250 items). Lost 1 pages of memory.
Freed UMA keg (tcp_inpcb) was not empty (75 items). Lost 5 pages of memory.
Freed UMA keg (tcpcb) was not empty (15 items). Lost 3 pages of memory.
Comment 4 Mark Linimon 2014-05-22 21:53:34 UTC
----- Forwarded message from "Bjoern A. Zeeb" <bz@FreeBSD.org> -----

Date: Thu, 22 May 2014 15:16:43 +0000
From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: "Hiroo Ono (å°éå¯ç)" <hiroo.ono+freebsd@gmail.com>
Cc: freebsd-bugs@FreeBSD.org
Subject: Re: kern/164763: [vimage] Memory leak in VNET
X-Mailer: Apple Mail (2.1874)


The fixes for UDP have been in a perforce branch for two years and need
updating and merging into HEAD.

TCP was and is the only thing that could not be completely freed (back
two years ago) synchronously and thus would continue to leak.  Itâs not
not fixable and probably with some tw changes lately got better (or maybe
they didnât happen).

â 
Bjoern A. Zeeb             "Come on. Learn, goddamn it.", WarGames, 1983

----- End forwarded message -----
Comment 5 Craig Rodrigues freebsd_committer 2014-10-17 06:19:00 UTC
Bjoern gave a link to his Perforce repo here:

https://lists.freebsd.org/pipermail/freebsd-net/2014-October/040075.html

It would be good to merge these memory leak fixes into FreeBSD HEAD
Comment 6 Hiren Panchasara freebsd_committer 2015-10-15 17:12:45 UTC
Following appeared on freebsd-bugs@ so adding here:

Hello FreeBSD folks,
I'm Sudarshan, a Software Engineer from a Networking team at NetApp.

While working on a bug, I observed a memory leak in keg_dtor() and looking at the call stack
it looks like the FreeBSD bug 164763, i.e. memory leak during vnet_sysuninit() triggered from the UDP path.
Reference: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=164763

I read that the fix has been in the perforce branch for a while now and not sure if it got merged into HEAD.
If the fix is already available as a patch, can you point me to the location?
If not, can you provide an estimate as when would it be available?

Looking forward to hear.

Thanks,
Sudarshan
Comment 7 sudarshannc 2015-10-28 14:40:42 UTC
(In reply to Hiren Panchasara from comment #6)
Thanks Hiren.

All,
Is this memory leak fixed in FreeBSD 10 or has it been planned to do that
sometime? Would be helpful to know the current status.

Sudarshan
Comment 8 Kubilay Kocak freebsd_committer freebsd_triage 2015-10-28 14:48:06 UTC
Issue cant be In Progress without an Assignee. Resetting.
Comment 9 Marie Helene Kvello-Aune 2016-04-05 11:01:02 UTC
Just here to report that the memory leaks still happen on 10.3-STABLE (r297561). I have not tested on 11-CURRENT as of yet.

I upgraded my jail server to latest 10-stable today and built a custom kernel which includes GENERIC and adds 'options VIMAGE' and 'device epair'. (FreeBSD jupiter 10.3-STABLE FreeBSD 10.3-STABLE #0 r297561: Mon Apr  4 21:39:12 CEST 2016     root@jupiter:/usr/obj/usr/src/sys/JupiterVIMAGE  amd64)

After creating a jail and assigning two epairXb devices to it, and then stopping the jail, I still observe the following dmesg output on host:

Freed UMA keg (udp_inpcb) was not empty (240 items).  Lost 24 pages of memory.
Freed UMA keg (udpcb) was not empty (2171 items).  Lost 13 pages of memory.
Freed UMA keg (tcp_inpcb) was not empty (90 items).  Lost 9 pages of memory.
Freed UMA keg (tcpcb) was not empty (27 items).  Lost 9 pages of memory.
Freed UMA keg (ripcb) was not empty (90 items).  Lost 9 pages of memory.

The interfaces are back on host, which is as expected, since I never executed a command to actually destroy them.

If someone want to guide me through debugging these memory leaks, I'd be happy to provide any information required to get to the bottom of this. I do have a 11-CURRENT system I can use for this purpose as well.
Comment 10 Bjoern A. Zeeb freebsd_committer 2016-04-05 11:06:26 UTC
I have Reviews in PB for most of these and should commit them to HEAD the next days.
If you want to give them a try sooner, I can probably do that this afternoon.

There might be a possible leak still for one of the TCP zones after that but I was not able to reproduce it yet.
Comment 11 Marie Helene Kvello-Aune 2016-04-05 12:56:39 UTC
I'm glad to hear fixes are inbound! :)

Waiting a few days is fine, I'm not in a hurry on this.
Comment 12 commit-hook freebsd_committer 2016-04-09 10:45:24 UTC
A commit references this bug:

Author: bz
Date: Sat Apr  9 10:44:58 UTC 2016
New revision: 297735
URL: https://svnweb.freebsd.org/changeset/base/297735

Log:
  Mfp: r296345

  No need to keep type stability on raw sockets zone.
  We've also been running with a KASSERT since r222488 to make sure the
  ipi_count is 0 on destroy.

  PR:		164763
  Reviewed by:	gnn
  MFC after:	2 weeks
  Sponsored by:	The FreeBSD Foundation
  Differential Revision:	https://reviews.freebsd.org/D5735

Changes:
  head/sys/netinet/raw_ip.c
Comment 13 commit-hook freebsd_committer 2016-04-09 10:58:30 UTC
A commit references this bug:

Author: bz
Date: Sat Apr  9 10:58:08 UTC 2016
New revision: 297738
URL: https://svnweb.freebsd.org/changeset/base/297738

Log:
  Mfp: r296259

  We attach the "counter" to the tcpcbs. Thus don't free the
  TCP Fastopen zone before the tcpcbs are gone, as otherwise
  the zone won't be empty.
  With that it should be safe to destroy the "tfo" zone without
  leaking the memory.

  PR:		164763
  Reviewed by:	gnn
  MFC after:	2 weeks
  Sponsored by:	The FreeBSD Foundation
  Differential Revision:	https://reviews.freebsd.org/D5731

Changes:
  head/sys/netinet/tcp_fastopen.c
  head/sys/netinet/tcp_subr.c
Comment 14 commit-hook freebsd_committer 2016-04-09 11:28:35 UTC
A commit references this bug:

Author: bz
Date: Sat Apr  9 11:27:48 UTC 2016
New revision: 297740
URL: https://svnweb.freebsd.org/changeset/base/297740

Log:
  Mfp: r296260

  The tcp_inpcb (pcbinfo) zone should be safe to destroy.

  PR:		164763
  Reviewed by:	gnn
  MFC after:	2 weeks
  Sponsored by:	The FreeBSD Foundation
  Differential Revision:	https://reviews.freebsd.org/D5732

Changes:
  head/sys/netinet/tcp_subr.c
Comment 15 commit-hook freebsd_committer 2016-04-09 12:05:42 UTC
A commit references this bug:

Author: bz
Date: Sat Apr  9 12:05:24 UTC 2016
New revision: 297742
URL: https://svnweb.freebsd.org/changeset/base/297742

Log:
  Mfp: r296310,r296343

  It looks like as with the safety belt of DELAY() fastened (*) we can
  completely tear down and free all memory for TCP (after r281599).

  (*) in theory a few ticks should be good enough to make sure the timers
  are all really gone. Could we use a better matric here and check a
  tcbcb count as an optimization?

  PR:		164763
  Reviewed by:	gnn, emaste
  MFC after:	2 weeks
  Sponsored by:	The FreeBSD Foundation
  Differential Revision:	https://reviews.freebsd.org/D5734

Changes:
  head/sys/netinet/tcp_subr.c
Comment 16 Marie Helene Kvello-Aune 2016-04-14 12:53:34 UTC
On r297742, -CURRENT, I still seem to get a memory leak when restarting jail, but I'm not able to reliably reproduce it.

Apr 14 14:48:15 venus kernel: ifa_maintain_loopback_route: deletion failed for interface lo0: 48
Apr 14 14:48:15 venus kernel: Freed UMA keg (tcptw) was not empty (765 items).  Lost 17 pages of memory.

uname: FreeBSD venus 11.0-CURRENT FreeBSD 11.0-CURRENT #1 r297742M: Sat Apr  9 14:21:50 CEST 2016     root@venus:/usr/obj/usr/src/sys/Venus  amd64

I'll try to find a reliable way of reproducing it this weekend.
Comment 17 Marie Helene Kvello-Aune 2016-04-20 20:10:01 UTC
I am not entirely sure, but it seems like the memory leak: 
Freed UMA keg (tcptw) was not empty (630 items).  Lost 14 pages of memory.

correlates with large transfers over TCP. I was using nginx (without accf_http, not that it should matter), and had someone download some garbage from me.

Potentially relevant nginx settings: sendfile, tcp_nopush, tcp_nodelay are enabled. Output buffers: 1 512k.

Potentially relevant network stack settings:
net.inet.tcp.cc.algorithm=htcp
net.inet.tcp.cc.htcp.adaptive_backoff=1
net.inet.tcp.cc.htcp.rtt_scaling=1
Comment 18 Bjoern A. Zeeb freebsd_committer 2016-05-06 10:40:24 UTC
Just to confirm; I was able to reproduce the TCP leaks; looking into it.
Comment 19 Bjoern A. Zeeb freebsd_committer 2016-05-26 17:26:04 UTC
In case you have cycles, can you test the projects/vnet branch (I am trying to keep it up to date with HEAD weekly)?

Do not try anything related to pf (or ipfilter) yet however.

Any feedback welcome.
Comment 20 Bjoern A. Zeeb freebsd_committer 2016-07-05 17:08:53 UTC
Can you please try FreeBSD 11.0-ALPHA6 or later?
Comment 21 Corvin Wimmer 2016-07-28 14:19:02 UTC
(In reply to Bjoern A. Zeeb from comment #20)

I've tested 11.0-BETA2 using iperf3 in both directions between two jails with UDP and TCP. I couldn't find any of the usual keg leakage on VNET teardown.
Comment 22 Bjoern A. Zeeb freebsd_committer 2016-07-28 15:54:53 UTC
(In reply to Corvin Wimmer from comment #21)

And I assume no panics either?   Thanks a lot for the feedback!
Comment 23 Corvin Wimmer 2016-07-28 16:02:07 UTC
(In reply to Bjoern A. Zeeb from comment #22)

No panics either. I was able to crash the system using iocage by recreating stopped jails but was not able to reproduce that specific issue using jail(8) manually.
Comment 24 Jack Maco 2016-09-15 23:32:51 UTC
Not sure if this is helpful or not but I thought I would post in;

FreeBSD 11-RC2, test command:

perl -e 'my $count=100000; while($count--) { print "Remaining: $count\n"; `jail -l -u root -c path=/jails/jail1 name=foo persist vnet && jexec foo ifconfig lo0 127.0.0.1/8 && jail -r foo` }'

I was only watching top, no debug tools enabled, as this is a raw base install no other daemons or such. It completed the entire 100K restarts with zero memory problems.

As an aside top looked the same when I started to when it completed but /var/log/messages had 100K 

ifa_maintain_loopback_route: deletion failed for interface lo0: 48

So ... can we assume VIMAGE is finally safe to restart jails in? :)

.. I use /etc/jail.conf instead of iocage
Comment 25 Ed Maste freebsd_committer 2016-12-21 21:26:56 UTC
It appears these issues have been resolved in 11.0. Please submit a new PR if additional memory leaks are found.