Bug 211432 - emulators/virtualbox-ose: wrong free memory calculation
Summary: emulators/virtualbox-ose: wrong free memory calculation
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Jung-uk Kim
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-07-28 23:53 UTC by Miroslav Lachman
Modified: 2017-09-08 01:28 UTC (History)
3 users (show)

See Also:
bugzilla: maintainer-feedback? (vbox)


Attachments
freemem calc fix (2.32 KB, patch)
2017-09-06 13:50 UTC, rozhuk.im
no flags Details | Diff
Fix RTSystemQueryAvailableRam() (2.57 KB, patch)
2017-09-06 18:21 UTC, Jung-uk Kim
no flags Details | Diff
Fix RTSystemQueryAvailableRam() (take 2) (2.50 KB, patch)
2017-09-07 04:40 UTC, Jung-uk Kim
no flags Details | Diff
Fix getHostMemoryUsage() and getMemoryAvailable() (4.51 KB, patch)
2017-09-07 06:01 UTC, Jung-uk Kim
no flags Details | Diff
Fix getHostMemoryUsage() and getMemoryAvailable() (take 2) (4.51 KB, patch)
2017-09-07 16:38 UTC, Jung-uk Kim
no flags Details | Diff
without PagesMemInactive (5.11 KB, patch)
2017-09-08 01:28 UTC, rozhuk.im
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Miroslav Lachman 2016-07-28 23:53:54 UTC
After upgrade from VirtualBox 4.3.38 to 5.0.26 and phpVirtualBox 4.3.2 to 5.0.5 the "Base memory" on the web page shows wrong numbers:

===============
General
OS Type:	FreeBSD (10.3-RELEASE-p6)
VirtualBox:	5.0.26_OSE (108824)
Base Memory:	65454 MB
64420 MB (98%)934 MB
===============

It shows this after reboot when almost all memory is free. (shown as 98% full)


"top" confirms that there is 61GB Free

==============
Mem: 434M Active, 307M Inact, 830M Wired, 6344K Cache, 61G Free
ARC: 190M Total, 26M MFU, 155M MRU, 3344K Anon, 1232K Header, 4833K Other
==============


I am not sure if this numbers are calculated by phpVirtualBox of VirtualBox it self.

We have the same problem on all machines with VirtualBox + phpVirtualBox 5. It was correct before upgrade.
Comment 1 rozhuk.im 2017-09-06 12:30:23 UTC
This is not phpvirtualbox, it from vboxwebserver

tcpdump -e -n -A -i lo0 tcp port 18083

POST / HTTP/1.1
Host: 127.0.0.1:18083
Connection: Keep-Alive
User-Agent: 
Content-Type: text/xml; charset=utf-8
SOAPAction: ""
Content-Length: 317

<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="http://www.virtualbox.org/"><SOAP-ENV:Body><ns1:IHost_getMemoryAvailable><_this>20a3257c497727a8-0000000000000002</_this></ns1:IHost_getMemoryAvailable></SOAP-ENV:Body></SOAP-ENV:Envelope>



HTTP/1.1 200 OK
Server: gSOAP/2.8
X-Frame-Options: SAMEORIGIN
Content-Type: text/xml; charset=utf-8
Content-Length: 474
Keep-Alive: timeout=60, max=98
Connection: keep-alive

<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:vbox="http://www.virtualbox.org/"><SOAP-ENV:Body><vbox:IHost_getMemoryAvailableResponse><returnval>3636</returnval></vbox:IHost_getMemoryAvailableResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>

3636 - wrong, should be ~15Gb
Comment 2 rozhuk.im 2017-09-06 13:50:37 UTC
Created attachment 186120 [details]
freemem calc fix

Please rename bug to emulators/virtualbox-ose


Apply patch, rebuild and reinstall virtualbox-ose.
Next reboot or:
sockstat | grep 18083
kill PID_from_prev_command
/usr/local/etc/rc.d/vboxwebsrv forcerestart
Comment 3 Jung-uk Kim freebsd_committer 2017-09-06 18:21:50 UTC
Created attachment 186123 [details]
Fix RTSystemQueryAvailableRam()

Please try this patch and let me know.  Thanks!
Comment 4 rozhuk.im 2017-09-06 18:53:24 UTC
I dont know about PagesMemCached, but PagesMemInactive - not free mem that can be used by applications or VirtualBox.

u_int -> uint64_t - why not?

+-    u_long cbMemPhys = 0;
++    uint64_t cbMemPhys = 0;
why not?
Comment 5 Jung-uk Kim freebsd_committer 2017-09-06 19:08:28 UTC
(In reply to rozhuk.im from comment #4)
> I dont know about PagesMemCached, but PagesMemInactive - not free mem that can be used by applications or VirtualBox.

It depends on the definition of "free". :-)

> u_int -> uint64_t - why not?

Because it should match the actual variable in kernel.

https://svnweb.freebsd.org/base/head/sys/sys/vmmeter.h?revision=317061&view=markup#l128
Comment 6 rozhuk.im 2017-09-06 19:25:21 UTC
Ok, but uint64_t work ok.
Well, PagesMemInactive is unavailable to apps, at least on FreeBSD 11.
And may be should not extend free to (free+inact+memc). :)

My patch works ok, and your should, I only not agree with free!=free :)
Same my patch for xfce4-taskmanager: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219155
because it is very annoying then you see that half of RAM is "free" but get freezes and OOM, especially if swap disabled.
Comment 7 Jung-uk Kim freebsd_committer 2017-09-06 19:26:20 UTC
(In reply to rozhuk.im from comment #4)
> +-    u_long cbMemPhys = 0;
> ++    uint64_t cbMemPhys = 0;
> why not?

https://svnweb.freebsd.org/base/head/sys/kern/kern_mib.c?revision=317055&view=markup#l190
Comment 8 Jung-uk Kim freebsd_committer 2017-09-06 19:52:30 UTC
(In reply to rozhuk.im from comment #6)
> Ok, but uint64_t work ok.

Only by chance.

> Well, PagesMemInactive is unavailable to apps, at least on FreeBSD 11.
> And may be should not extend free to (free+inact+memc). :)

Different applications have different meanings.  Often times, it means "unused".

https://unix.stackexchange.com/questions/134862/what-do-the-different-memory-counters-in-freebsd-mean

Actually, there are many applications ported from Linux using the formula because there is no exact Linux-equivalent.  In fact, "available physical memory" does not make much sense in some cases.
Comment 9 Jung-uk Kim freebsd_committer 2017-09-07 04:40:35 UTC
Created attachment 186140 [details]
Fix RTSystemQueryAvailableRam() (take 2)

There was a redundant #if/#endif pair in the previous patch.
Comment 10 Jung-uk Kim freebsd_committer 2017-09-07 06:01:40 UTC
Created attachment 186142 [details]
Fix getHostMemoryUsage() and getMemoryAvailable()

This patch fixes another function.
Comment 11 rozhuk.im 2017-09-07 11:52:35 UTC
(In reply to Jung-uk Kim from comment #10)

Thanks!
I will test it in next 24 hours.

But I still disagree with PagesMemInactive as free mem.
I got OOM then free ~0 and PagesMemInactive more than 5-10gb.
So if VBox or xfce4-taskmanager report (free+PagesMemInactive+..) - then they lie and that info is crap and can not be used to make a decision.

https://unix.stackexchange.com/questions/134862/what-do-the-different-memory-counters-in-freebsd-mean
That was in 2014, why you thitk that it is actual?
PagesMemInactive - kernel will free by a bit strange algo, I newer see that kernel give memory from that "poll" to my apps then free=0 and apps need more mem.
Kernel always waiting for something and free small portions only if free is very small.
PagesMemInactive for me (and for user space apps) it is not free, it is unavailable memory.

I see some patches to more agressive free PagesMemInactive mem, but not try yet.

Please, let free be only free, without any unpredictable additions.
Comment 12 rozhuk.im 2017-09-07 12:21:15 UTC
https://wiki.freebsd.org/Memory

Inactive:
- Inactively (not recently used) used by userland
- Not recently used (LRU)
- !!!Cannot be re-allocated to other processes!!!
Comment 13 Jung-uk Kim freebsd_committer 2017-09-07 16:31:06 UTC
(In reply to rozhuk.im from comment #11)
> But I still disagree with PagesMemInactive as free mem.
> I got OOM then free ~0 and PagesMemInactive more than 5-10gb.
> So if VBox or xfce4-taskmanager report (free+PagesMemInactive+..) - then
> they lie and that info is crap and can not be used to make a decision.

Some applications refuse to start at all if "available" memory is too small.

> https://unix.stackexchange.com/questions/134862/what-do-the-different-memory-counters-in-freebsd-mean
> That was in 2014, why you thitk that it is actual?

Because it didn't change much.

> https://wiki.freebsd.org/Memory
> 
> Inactive:
> - Inactively (not recently used) used by userland
> - Not recently used (LRU)
> - !!!Cannot be re-allocated to other processes!!!

That page was written in 2013.

https://wiki.freebsd.org/action/info/Memory?action=info

Why do you think that is actual? :-P
Comment 14 Jung-uk Kim freebsd_committer 2017-09-07 16:38:58 UTC
Created attachment 186149 [details]
Fix getHostMemoryUsage() and getMemoryAvailable() (take 2)

ULONG is not actually unsigned long.  Use ULONG64 instead.
Comment 15 rozhuk.im 2017-09-07 20:39:24 UTC
(In reply to Jung-uk Kim from comment #13)

> Some applications refuse to start at all if "available" memory is too small.

I think vbox and xfce4-taskmanager not affected by this.


> Why do you think that is actual? :-P

> Because it didn't change much.

%)))

My experiences told me that inactive memory still unavailable for applications and we should not count it as free.
No guarantees that inactive became free then we need it.
Comment 16 Jung-uk Kim freebsd_committer 2017-09-07 22:31:52 UTC
(In reply to rozhuk.im from comment #15)
My point is many applications (usually ported from Windows) expect "total = used  + available" for physical memory but there is no such thing in FreeBSD (and many modern OSes).  We just try to make it look "reasonably" good for each use case.
Comment 17 Jung-uk Kim freebsd_committer 2017-09-08 01:14:22 UTC
Committed, thanks!
Comment 18 rozhuk.im 2017-09-08 01:28:59 UTC
Created attachment 186154 [details]
without PagesMemInactive

Thanks!
I prefer to stay on private patch without inactive mem counting.