Bug 248300

Hi.

I had one report about this earlier. I'll paste the mail conversation below. You hit a limit of mlock-able memory.

As mentioned in the man page of mlock you can take a look at the values of:
sysctl vm.stats.vm.v_wire_count vm.max_wired

count should be less than max_wired.

The other person did not have the problem anymore after a reboot.

And if you are running "mongo" in a jail you shouls set "allow.mlock" in the jail.conf.

Ronald.


-------------------------------------------------------------------------

    Do you have any other settings which might influence this? In rc.conf of sysctl.conf or make.conf/src.conf?

I don't think so. It's a brand new install of freeBSD and mongo is one of the first package I installed. The server has 64GB of memory.

Just these settings in `sysctl.conf` :

kern.randompid=1
vfs.zfs.min_auto_ashift=12
vfs.usermount=1

sysctl vm.stats.vm.v_wire_count vm.max_wired security.bsd.unprivileged_mlock
vm.stats.vm.v_wire_count: 13913803
vm.max_wired: 5381304
security.bsd.unprivileged_mlock: 1

I run `service mongod` to start the daemon, and it starts without problem.

Anyway, I restarted my server and the mongo cli started without error…

Strange…

Thanks,

Stan.
On 03/07/2020 10:45, Ronald Klop wrote:

     

    Van: Stanislas SABATIER <***>
    Datum: vrijdag, 3 juli 2020 09:33
    Aan: Ronald Klop <***>
    Onderwerp: Re: bug in mongo 4.2.8 on FreeBSD 12.1-p6 ?

        Hello Ronald,

        Thank you for your quick response.

        I'm not running `mongo` into a jail, neither `mongod`. And I launch `mongo` as root…

        All the mongo binaries are working (mongod, mongodump, etc.), **except** `mongo` client.

        `ulimit -a` :

        ```

        #  ulimit -a
        -t: cpu time (seconds)              unlimited
        -f: file size (blocks)              unlimited
        -d: data seg size (kbytes)          33554432
        -s: stack size (kbytes)             524288
        -c: core file size (blocks)         unlimited
        -m: resident set size (kbytes)      unlimited
        -l: locked-in-memory size (kbytes)  unlimited
        -u: processes                       63493
        -n: file descriptors                1876410
        -b: socket buffer size (bytes)      unlimited
        -v: virtual memory size (kbytes)    unlimited
        -p: pseudo-terminals                unlimited
        -w: swap size (kbytes)              unlimited
        -k: kqueues                         unlimited
        -o: umtx shared locks               unlimited
        ```

        This is weird.

        Stan.


    Interesting indeed. I'm running this on 12.1-p6/aarch64 instead of amd64. I can try amd64 tonight or this weekend.

    Do you have any other settings which might influence this? In rc.conf of sysctl.conf or make.conf/src.conf?

    "man 2 mlock" also has some information about the limits:
    "     Since physical memory is a potentially scarce resource, processes are
         limited in how much they can lock down.  The amount of memory that a
         single process can mlock() is limited by both the per-process
         RLIMIT_MEMLOCK resource limit and the system-wide "wired pages" limit
         vm.max_wired.  vm.max_wired applies to the system as a whole, so the
         amount available to a single process at any given time is the difference
         between vm.max_wired and vm.stats.vm.v_wire_count.

         If security.bsd.unprivileged_mlock is set to 0 these calls are only
         available to the super-user.
    "

    For these values I have (when mongodb is not running):
    # sysctl vm.stats.vm.v_wire_count vm.max_wired security.bsd.unprivileged_mlock
    vm.stats.vm.v_wire_count: 50341
    vm.max_wired: 75456
    security.bsd.unprivileged_mlock: 1

    With mongod + mongo runiing v_wire_count peaked to 53735.

    Can you show the exact commands you use to run mongod and mongo? As mongod runs as user "mongodb" if you use the rc.d/mongod script.

    If you want to have more people involved I advise to cc a freebsd mailinglist also.

    Regards,
    Ronald.

     

        On 02/07/2020 19:19, Ronald Klop wrote:

             

            Van: ***
            Datum: 2 juli 2020 17:02
            Aan: ***
            Onderwerp: bug in mongo 4.2.8 on FreeBSD 12.1-p6 ?

                Hello Ronald,

                Thank you for your work as a maintainer of the mongo42 port.

                I installed mongo42 on a brand new FreeBSD 12.1-p6 server. Installation went well and I'm able to run the `mongod` daemon.

                But, when I try to run the mongo client `mongo`, I get the error below :

                ```

                2020-07-02T16:51:32.389+0200 F - [main] Failed to mlock: Cannot allocate locked memory. For more details see: https://dochub.mongodb.org/core/cannot-allocate-locked-memory: Resource temporarily unavailable
                2020-07-02T16:51:32.389+0200 F - [main] Fatal Assertion 28832 at src/mongo/base/secure_allocator.cpp 255
                2020-07-02T16:51:32.389+0200 F - [main]

                ***aborting after fassert() failure


                2020-07-02T16:51:32.390+0200 F - [main] Got signal: 6 (Abort trap).
                0x28aeea6 0x28ae71b 0x803c843ce
                ----- BEGIN BACKTRACE -----
                {"backtrace":[{"b":"1021000","o":"188DEA6","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"},{"b":"1021000","o":"188D71B","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"803C70000","o":"143CE","s":"_pthread_sigmask"}],"processInfo":{ "mongodbVersion" : "4.2.8", "gitVersion" : "43d25964249164d76d5e04dd6cf38f6111e21f5f", "compiledModules" : [], "uname" : { "sysname" : "FreeBSD", "release" : "12.1-RELEASE-p6", "version" : "FreeBSD 12.1-RELEASE-p6 GENERIC", "machine" : "amd64" } }}
                mongo(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x36) [0x28aeea6]
                mongo(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x15B) [0x28ae71b]
                libthr.so.3(_pthread_sigmask+0x53E) [0x803c843ce]
                ----- END BACKTRACE -----
                [1] 9183 abort (core dumped) mongo

                ```

                Do you have any idea of this issue ?

                Thank you in advance for your response.

                Regards,

                Stan.



            Hi,
             
            is this in a jail? In jails you need to set allow.mlock.
            Outside of a jail I have not seen this yet, but the error message points in the same direction. A setting/limit in the account of mlock-able memory for this user.
             
            Regards,
            Ronald.

Thanks! vm.max_wired=1000000 fixed an issue.

"Thanks! vm.max_wired=1000000 fixed an issue."

Increasing vm.max_wired is the solution I was looking for too!