Bug 254721

Summary:	Swap on ZFS can crash server in high memory usage conditions
Product:	Base System	Reporter:	dmilith <dmilith>
Component:	kern	Assignee:	freebsd-bugs (Nobody) <bugs>
Status:	New ---
Severity:	Affects Some People	CC:	crest, grahamperrin, ota, rob2g2-freebsd
Priority:	---	Keywords:	crash
Version:	12.2-STABLE
Hardware:	Any
OS:	Any
See Also:	https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262595

Description dmilith 2021-04-02 14:37:55 UTC

I have ELK stack on a dedicated i7 server which has 100% memory usage and swap on ZFS (mounted with "zfs create -V 64gb -o org.freebsd:swap=on -o checksum=off -o compression=off -o dedup=off -o sync=disabled -o primarycache=none zroot/swap
swapon /dev/zvol/zroot/swap").

With some extreme load cases, my whole machine crashes with dump like:

```
[109840] Fatal trap 12: page fault while in kernel mode
[109840] cpuid = 2; apic id = 02
[109840] fault virtual address  = 0xde20140520
[109840] fault code             = supervisor write data, page not present
[109840] instruction pointer    = 0x20:0xffffffff811daef7
[109840] stack pointer          = 0x28:0xfffffe00005f1850
[109840] frame pointer          = 0x28:0xfffffe00005f1870
[109840] code segment           = base rx0, limit 0xfffff, type 0x1b
[109840]                        = DPL 0, pres 1, long 1, def32 0, gran 1
[109840] processor eflags       = interrupt enabled, resume, IOPL = 0
[109840] current process                = 16 (solthread 0xfffffff)
[109840] trap number            = 12
[109840] panic: page fault
[109840] cpuid = 2
[109840] time = 1611201271
[109840] version = FreeBSD 12.2-RELEASE-HBSD  VerKnowSys
[109840] KDB: stack backtrace:
[109840] #0 0xffffffff80daf88d at kdb_backtrace+0x7d
[109840] #1 0xffffffff80d3e98e at vpanic+0x23e
[109840] #2 0xffffffff80d3e750 at vpanic+0
[109840] #3 0xffffffff813adce6 at trap_fatal+0x636
[109840] #4 0xffffffff813adf7f at trap_pfault+0x1af
[109840] #5 0xffffffff813acdfa at trap+0x7ba
[109840] #6 0xffffffff813ae478 at trap_check+0x48
[109840] #7 0xffffffff8137a4d8 at calltrap+0x8
[109840] #8 0xffffffff811dfa82 at bucket_cache_drain+0x52
[109840] #9 0xffffffff811d64cf at zone_drain_wait+0x12f
[109840] #10 0xffffffff811d639a at zone_drain+0x1a
[109840] #11 0xffffffff826a9ba8 at arc_kmem_reap_soon+0x88
[109840] #12 0xffffffff826a608b at arc_reap_cb+0xb
[109840] #13 0xffffffff82761eb1 at zthr_procedure+0xc1
[109840] #14 0xffffffff80cd0c69 at fork_exit+0x179
[109840] #15 0xffffffff8137b50e at fork_trampoline+0xe
```

Comment 1 Andriy Gapon freebsd_committer

2021-04-04 14:20:20 UTC

The crash is in UMA code, it does not look directly related to ZFS.
Also, what is 12.2-RELEASE-HBSD ?

Comment 2 Graham Perrin freebsd_committer

2022-03-07 11:21:46 UTC

> on ZFS 

See also: <https://github.com/openzfs/zfs/issues/7734>

>> Swap deadlock in 0.7.9 · Issue #7734 · openzfs/zfs

Comment 3 Graham Perrin freebsd_committer

2022-03-07 11:24:54 UTC

(In reply to Graham Perrin from comment #2)

… sorry, I overlooked comment 1. My bad.

Comment 4 crest 2022-03-07 15:06:09 UTC

The (long and) short of it is: writing to ZFS can require heap allocations making ZVOLs unsuitable as swap devices (swapping to normal files on ZFS mounts would be even worse).

The only reliable solution is to use something simpler as swap space e.g. a gmirror+geli partition. I know this sucks if you have no space left on your disks.