top on today's STABLE: last pid: 3719; load averages: 23,07, 23,36, 17,75 up 0+00:33:11 17:03:28 40 processes: 1 running, 39 sleeping CPU: 12,4% user, 0,0% nice, 15,2% system, 0,4% interrupt, 72,0% idle Mem: 16G Active, 6059M Inact, 883M Laundry, 14G Wired, 3311M Free ARC: 9583M Total, 1377M MFU, 6691M MRU, 3403K Anon, 237M Header, 21M Other 7543M Compressed, 16G Uncompressed, 2,14:1 Ratio Swap: 32G Total, 13G Used, 19G Free, 41% Inuse [...] dmesg: [root@db0-de:~]# grep memory /var/run/dmesg.boot real memory = 137434759168 (131068 MB) avail memory = 133681565696 (127488 MB) vmstat: [root@db0-de:~]# vmstat -h procs memory page disks faults cpu r b w avm fre flt re pi po fr sr nv0 nv1 in sy cs us sy id 22 0 9 132G 3.0G 22K 2902 591 698 13K 338K 47132 15711 55073 132K 303K 13 16 72 Also reproducible on 13.0. uname: FreeBSD db0-de.findmykids.org 13.1-STABLE FreeBSD 13.1-STABLE #0 stable/13-n252309-f25bd922c97: Wed Sep 7 12:38:08 MSK2022 emz@balancer1-de.findmykids.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
Note: this happens after some slight uptime (dozens on minutes), right after the boot top shows numbers close to real amount.
Adding kib@ to CC. Maybe Konstantin has something to say.
Please attach output of the command: sysctl hw.physmem hw.usermem hw.realmem vm.vmtotal vm.kvm_free vm.kvm_size vm.reserv
[root@db0-de:db/mysql]# sysctl hw.physmem hw.usermem hw.realmem vm.vmtotal vm.kvm_free vm.kvm_size vm.reserv hw.physmem: 137276436480 hw.usermem: 104537735168 hw.realmem: 137434759168 vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) =============================================== Processes: (RUNQ: 12 Disk Wait: 0 Page Wait: 0 Sleep: 6414) Virtual Memory: (Total: 111950456K Active: 111936556K) Real Memory: (Total: 93233204K Active: 93232128K) Shared Virtual Memory: (Total: 114028K Active: 100228K) Shared Real Memory: (Total: 49808K Active: 48808K) Free Memory: 5470588K vm.kvm_free: 2183768567808 vm.kvm_size: 2199023251456 zsh: command not found: vm.reserv
Sorry, pasted along with linefeed: [root@db0-de:db/mysql]# sysctl vm.reserv vm.reserv.reclaimed: 13318 vm.reserv.partpopq: DOMAIN LEVEL SIZE NUMBER 0, -1, 83100K, 71 1, -1, 241968K, 289 2, -1, 196620K, 213 3, -1, 766132K, 1061 vm.reserv.fullpop: 23780 vm.reserv.freed: 12994 vm.reserv.broken: 13335
Also, show: sysctl vfs.bufspace vm.stats.vm
Can you show the top output right after the boot?
I'm afraid - no, at least not in the foreseeable future: this is a production database for a service with hundreds of thousands of users (like literally), so it's rebooted very rarely, probably once per year. However, I can probably reboot it's replica: the bug is reproducible there too, however if in original case top misses like 90 gigs of system memory, on the replica this number is a bit smaller: top: [...] Mem: 1651M Active, 9930M Inact, 2152M Laundry, 60G Wired, 330K Buf, 23G Free [...] reality: real memory = 171793448960 (163835 MB) avail memory = 167167340544 (159423 MB) Please let me know if this is acceptable.
What database do you use? Is it postgres? Did you tweaked kern.ipc.shm_use_phys?
databases/mysql80-server. Nope, didn't touch that sysctl oid.
Possibly related to bug 256507
Could you please try the patch committed for PR 256507? I strongly suspect it's the same problem.
Is this still a problem?
I no longer have the access to the server.
Submitter no longer in position to deal with the problem. Also, the PR seems to be duplicate of PR# 256507. *** This bug has been marked as a duplicate of bug 256507 ***