Bug 231794 - Panic in ZFS due to arc related memory exhaustion
Summary: Panic in ZFS due to arc related memory exhaustion
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.2-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-fs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-09-29 00:30 UTC by Dave Robison
Modified: 2018-10-19 17:36 UTC (History)
5 users (show)

See Also:


Attachments
Photo of backtrace (710.11 KB, image/png)
2018-09-29 00:30 UTC, Dave Robison
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dave Robison 2018-09-29 00:30:16 UTC
Created attachment 197582 [details]
Photo of backtrace

We are evaluating two servers based on the HP DL360 G10 (16 gigs RAM) and HP DL380 G10 (32 gigs RAM) motherboards. We can routinely panic these machines by putting them under load while running ZFS. Running six instances of bonnie++ and six instances of memtester (testing 2g) is enough to panic the DL360 in around 15 minutes and the DL380 in 10-13 hours.

Reducing ARC dramatically using vfs.zfs.arc_min and vfs.zfs.arc_max seems to mitigate this problem, at least after a day of testing under 12.0-A7. We are testing now on 11.2-RELEASE which we will use in production.

Daiichi from Japan was here to help diagnose this problem, and has been in contact with core team members who requested this bugzilla submission.

More panic photos available on request.
Comment 1 rainer 2018-10-04 06:59:54 UTC
See also:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231296

There's something seriously wrong with the default settings.
Comment 2 Andriy Gapon freebsd_committer 2018-10-04 14:39:37 UTC
(In reply to Dave Robison from comment #0)
I just want to note that the problem here is not with exhausting the physical memory, but rather with exhausting the kernel virtual address space (KVA).
There could be many reasons for that such as incorrect tuning, bugs, KVA fragmentation, etc.