Bug 190300
| Summary: | vmdaemon kernel thread consumes 100% cpu | ||
|---|---|---|---|
| Product: | Base System | Reporter: | zuborg |
| Component: | kern | Assignee: | freebsd-bugs (Nobody) <bugs> |
| Status: | Closed FIXED | ||
| Severity: | Affects Only Me | CC: | alc, emaste, pi, steve.polyack |
| Priority: | Normal | ||
| Version: | 10.0-RELEASE | ||
| Hardware: | Any | ||
| OS: | Any | ||
|
Description
zuborg
2014-05-27 13:00:00 UTC
FreeBSD 9.x is not affected `swapoff -a` didn't help When I've reduced load on server -' vmdaemon' continues to use 100% cpu, but stop doing it when I've restarted perl worker processes. We're also seeing this on a FreeBSD server which has recently been upgraded from 9.2 to 10.0, and using postgresql-9.3 The server has 256GB of RAM, but only 64GB are dedicated to postgres via shared_buffers. We've also noticed mbuf denied failures and very poor interactive performance when vmdaemon CPU usage is high, which is almost all of the time. # vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 384, 0, 95, 5, 95, 0, 0 UMA Zones: 4736, 0, 95, 0, 95, 0, 0 UMA Slabs: 80, 0, 7694, 6, 22132, 0, 0 UMA RCntSlabs: 88, 0, 11976, 39, 11976, 0, 0 UMA Hash: 256, 0, 2, 13, 8, 0, 0 4 Bucket: 32, 0, 157, 6468,37583750, 0, 0 8 Bucket: 64, 0, 236, 3794, 5426, 0, 0 16 Bucket: 128, 0, 371, 3659, 728, 36, 0 32 Bucket: 256, 0, 593, 6292, 162352,4213, 0 64 Bucket: 512, 0, 1019, 1493, 444970,232031, 0 128 Bucket: 1024, 0, 1428, 1264, 1637010, 277, 0 vmem btag: 56, 0, 3206, 12059, 299550, 215, 0 VM OBJECT: 256, 0, 33864, 2376, 798557, 0, 0 RADIX NODE: 144, 0, 4311308, 12742,15145691, 57, 0 MAP: 240, 0, 3, 61, 3, 0, 0 KMAP ENTRY: 128, 0, 7, 396, 7, 0, 0 MAP ENTRY: 128, 0, 6265, 5112, 1725230, 0, 0 VMSPACE: 448, 0, 145, 1106, 34759, 0, 0 fakepg: 104, 0, 0, 0, 0, 0, 0 mt_zone: 4112, 0, 343, 0, 343, 0, 0 16: 16, 0, 4637, 3144, 1155861, 0, 0 32: 32, 0, 4603, 3897, 821025, 0, 0 64: 64, 0, 9892, 68910,122945074, 0, 0 128: 128, 0, 7760, 48350, 9999121, 0, 0 256: 256, 0, 7126, 48704, 9244069, 0, 0 512: 512, 0, 1053, 32379, 123871, 0, 0 1024: 1024, 0, 123, 673, 40412, 0, 0 2048: 2048, 0, 341, 207, 46366, 0, 0 4096: 4096, 0, 573, 190, 39254, 0, 0 SLEEPQUEUE: 80, 0, 622, 3532, 622, 0, 0 uint64 pcpu: 8, 0, 1552, 112, 1552, 0, 0 Files: 80, 0, 2208, 4542,39412500, 0, 0 TURNSTILE: 136, 0, 622, 1518, 622, 0, 0 rl_entry: 40, 0, 415, 3585, 415, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0, 0 PROC: 1208, 0, 162, 219, 34775, 0, 0 THREAD: 1168, 0, 571, 50, 626, 0, 0 cpuset: 72, 0, 176, 209, 208, 0, 0 audit_record: 1248, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 104592120, 16370, 7380,112650926, 0, 0 mbuf: 256, 104592120, 16, 4494,100391690, 5, 0 mbuf_cluster: 2048, 16342518, 23750, 0, 23750,12060, 0 mbuf_jumbo_page: 4096, 8171259, 0, 101,37590064, 92, 0 mbuf_jumbo_9k: 9216, 2421113, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 1361876, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 g_bio: 248, 0, 0, 7392,12897141, 0, 0 ttyinq: 160, 0, 300, 1125, 1245, 0, 0 ttyoutq: 256, 0, 157, 1193, 655, 0, 0 ata_request: 336, 0, 0, 0, 0, 0, 0 vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0 FPU_save_area: 832, 0, 0, 0, 0, 0, 0 VNODE: 472, 0, 114408, 928, 155200, 0, 0 VNODEPOLL: 112, 0, 0, 770, 13, 0, 0 BUF TRIE: 144, 0, 3714, 102234, 9440298, 0, 0 S VFS Cache: 108, 0, 120884, 3891, 223399, 0, 0 STS VFS Cache: 148, 0, 0, 0, 0, 0, 0 L VFS Cache: 328, 0, 441, 375, 845, 0, 0 LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0 NAMEI: 1024, 0, 0, 520,41296842, 0, 0 DIRHASH: 1024, 0, 934, 210, 1275, 0, 0 NCLNODE: 528, 0, 1, 13, 1, 0, 0 Mountpoints: 816, 0, 10, 70, 11, 0, 0 pipe: 744, 0, 116, 524, 24303, 0, 0 procdesc: 128, 0, 0, 0, 0, 0, 0 ksiginfo: 112, 0, 373, 4002, 16780, 0, 0 itimer: 352, 0, 0, 33, 1, 0, 0 KNOTE: 128, 0, 23, 3728, 2156120, 0, 0 socket: 696, 8386445, 245, 485, 109956, 0, 0 ipq: 56, 510703, 0, 0, 0, 0, 0 udp_inpcb: 392, 8386450, 6, 924, 98789, 0, 0 udpcb: 16, 8386663, 6, 4010, 98789, 0, 0 tcp_inpcb: 392, 8386450, 176, 784, 2934, 0, 0 tcpcb: 1024, 8386448, 176, 372, 2934, 0, 0 tcptw: 88, 27810, 0, 1125, 258, 0, 0 syncache: 160, 15375, 0, 750, 2342, 0, 0 hostcache: 136, 15370, 17, 679, 58, 0, 0 tcpreass: 40, 1021500, 0, 800, 33960, 0, 0 sackhole: 32, 0, 0, 375, 37, 0, 0 sctp_ep: 1408, 8386446, 0, 0, 0, 0, 0 sctp_asoc: 2352, 40000, 0, 0, 0, 0, 0 sctp_laddr: 48, 80012, 0, 664, 6, 0, 0 sctp_raddr: 728, 80000, 0, 0, 0, 0, 0 sctp_chunk: 136, 400026, 0, 0, 0, 0, 0 sctp_readq: 104, 400026, 0, 0, 0, 0, 0 sctp_stream_msg_out: 104, 400026, 0, 0, 0, 0, 0 sctp_asconf: 40, 400000, 0, 0, 0, 0, 0 sctp_asconf_ack: 48, 400060, 0, 0, 0, 0, 0 ripcb: 392, 8386450, 0, 30, 1, 0, 0 unpcb: 240, 8386448, 63, 1889, 8227, 0, 0 rtentry: 200, 0, 25, 475, 25, 0, 0 selfd: 56, 0, 554, 4629,51234943, 0, 0 SWAPMETA: 288, 32685042, 0, 0, 0, 0, 0 FFS inode: 168, 0, 114348, 1871, 155129, 0, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 114348, 1857, 155128, 0, 0 Are you aware of the recent work on that topic ? https://www.freebsd.org/news/status/report-2014-04-2014-06.html#PostgreSQL- Performance-Improvements Maybe kib@ knows more about this ? I can confirm that r265945 fixes this. Unfortunately, it's only available in the 10-STABLE tree. We patched it into 10.0-RELEASE internally. My initial assessment was incorrect, r265945 does not fix the problem. However, r270666 by alc does fix the problem. This has been MFC'd to 10-STABLE in r270995. I think we can mark this as fixed. Patch applied to 10-STABLE. Fix will appear in 10.1-RELEASE. |
