Bug 233377 - [PowerPC64] Panic during high disk I/O activity
Summary: [PowerPC64] Panic during high disk I/O activity
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: powerpc Any
: --- Affects Some People
Assignee: freebsd-ppc mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-11-21 12:50 UTC by Sean Bruno
Modified: 2019-02-20 13:30 UTC (History)
8 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sean Bruno freebsd_committer 2018-11-21 12:50:42 UTC
During substantial ZFS disk activity, the pkg builder in the cluster will fail.  This is a Tyan PowerPC64 Machine.


rm: /usr/local/poudriere/data/.m/head-powerpc64-default/ref/lib: Directory not 

empty
panic: Memory modified after free 0xc000000065ceec40(32) val=0 @ 0xc000000065ceec40

cpuid = 31
time = 1542804498
KDB: stack backtrace:
0xe000000090207020: at .kdb_backtrace+0x5c
0xe000000090207150: at .vpanic+0x1b4
0xe000000090207210: at .panic+0x38
0xe0000000902072a0: at .trash_ctor+0x58
0xe000000090207320: at .trash_fini+0x1c
0xe0000000902073a0: at .uma_zdestroy+0x164
0xe000000090207450: at .uma_zdestroy+0x42c
0xe0000000902074e0: at .sys_swapoff+0x2c4
0xe000000090207570: at .uma_zfree_pcpu_arg+0x2ec
0xe000000090207600: at .zone_drain+0x18
0xe000000090207680: at .uma_avail+0x4c4
0xe000000090207710: at .zone_drain+0x378
0xe0000000902077a0: at .uma_reclaim_worker+0x20c
0xe000000090207850: at .fork_exit+0xd0
0xe0000000902078f0: at .fork_trampoline+0x10
0xe000000090207920: at -0x4
KDB: enter: panic
[ thread pid 15 tid 100219 ]
Stopped at      .kdb_enter+0x60:        ld      r2, r1, 0x28
db> bt
Comment 1 Mark Johnston freebsd_committer 2018-11-21 16:59:32 UTC
The first step will be to figure out which zone this is.  I think it'll be tricky to track this down without vmcores to look at, but this patch will give us a starting point.

diff --git a/sys/vm/uma_core.c b/sys/vm/uma_core.c
index 7d14586a31cd..8087a86584c9 100644
--- a/sys/vm/uma_core.c
+++ b/sys/vm/uma_core.c
@@ -1041,6 +1041,7 @@ void
 zone_drain(uma_zone_t zone)
 {
 
+       printf("draining zone %s\n", zone->uz_name);
        zone_drain_wait(zone, M_NOWAIT);
 }
Comment 2 Sean Bruno freebsd_committer 2018-11-21 19:02:54 UTC
(In reply to Mark Johnston from comment #1)
Oof.  That's dumping a lot of text to the serial console at the moment.  Every drain is emitting a line of text and its happening a lot.  It probably won't hit the bug if its a race-style I suspect.
Comment 3 Sean Bruno freebsd_committer 2018-11-22 17:14:28 UTC
I'm not sure this is helpful, but with the debug printf in the kernel the panic looks like this on the console.  Is there any debugging I can do from the db> prompt to give us a clue here?

draining zone 2048
draining zone 1024
draining zone 512
draining zone 256
draining zone 128
draining zone 64
draining zone 32
draining zone 16
draining zone mt_zone
draining zone mt_stats_zone
draining zone 64 pcpu
draining zone fakepg
draining zone UPVO entry
draining zone VMSPACE
draining zone MAP ENTRY
draining zone KMAP ENTRY
draining zone MAP
draining zone RADIX NODE
draining zone VM OBJECT
draining zone vmem btag
draining zone vmem
draining zone 256 Bucket
draining zone 128 Bucket
draining zone 64 Bucket
draining zone 32 Bucket
draining zone 16 Bucket
draining zone 12 Bucket
panic: Memory modified after free 0xc00000037d3c0c00(96) val=0 @ 0xc00000037d3c0c00

cpuid = 31
time = 1542904594
KDB: stack backtrace:
0xe000000090207010: at .kdb_backtrace+0x5c
0xe000000090207140: at .vpanic+0x1b4
0xe000000090207200: at .panic+0x38
0xe000000090207290: at .trash_ctor+0x58
0xe000000090207310: at .trash_fini+0x1c
0xe000000090207390: at .uma_zdestroy+0x164
0xe000000090207440: at .uma_zdestroy+0x42c
0xe0000000902074d0: at .sys_swapoff+0x2c4
0xe000000090207560: at .uma_zfree_pcpu_arg+0x2ec
0xe0000000902075f0: at .zone_drain+0x34
0xe000000090207680: at .uma_avail+0x4c4
0xe000000090207710: at .zone_drain+0x398
0xe0000000902077a0: at .uma_reclaim_worker+0x20c
0xe000000090207850: at .fork_exit+0xd0
0xe0000000902078f0: at .fork_trampoline+0x10
0xe000000090207920: at -0x4
KDB: enter: panic
[ thread pid 15 tid 100219 ]
Stopped at      .kdb_enter+0x60:        ld      r2, r1, 0x28
Comment 4 Mark Johnston freebsd_committer 2018-11-22 17:33:58 UTC
We will want to repro this a number of times to see if the zone is consistent.  I will provide a patch to embed the name of the zone in the panic message instead.

Could you provide the output of:

db> x/gx 0xc00000037d3c0000,200

assuming you still have the system in ddb?  (There will be quite a bit of output.)
Comment 5 Sean Bruno freebsd_committer 2018-11-22 17:36:16 UTC
(In reply to Mark Johnston from comment #4)
Its *almost* all deadc0de, except for one or two bits:

db> x/gx 0xc00000037d3c0000,200
0xc00000037d3c0000:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0010:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0020:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0030:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0040:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0050:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0060:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0070:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0080:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0090:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c00a0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c00b0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c00c0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c00d0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c00e0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c00f0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0100:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0110:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0120:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0130:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0140:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0150:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0160:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0170:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0180:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0190:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c01a0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c01b0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c01c0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c01d0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c01e0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c01f0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0200:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0210:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0220:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0230:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0240:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0250:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0260:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0270:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0280:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0290:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c02a0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c02b0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c02c0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c02d0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c02e0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c02f0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0300:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0310:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0320:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0330:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0340:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0350:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0360:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0370:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0380:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0390:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c03a0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c03b0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c03c0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c03d0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c03e0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c03f0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0400:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0410:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0420:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0430:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0440:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0450:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0460:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0470:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0480:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0490:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c04a0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c04b0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c04c0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c04d0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c04e0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c04f0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0500:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0510:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0520:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0530:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0540:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0550:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0560:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0570:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0580:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0590:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c05a0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c05b0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c05c0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c05d0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c05e0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c05f0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0600:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0610:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0620:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0630:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0640:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0650:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0660:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0670:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0680:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0690:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c06a0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c06b0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c06c0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c06d0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c06e0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c06f0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0700:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0710:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0720:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0730:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0740:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0750:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0760:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0770:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0780:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0790:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c07a0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c07b0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c07c0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c07d0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c07e0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c07f0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0800:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0810:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0820:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0830:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0840:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0850:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0860:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0870:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0880:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0890:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c08a0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c08b0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c08c0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c08d0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c08e0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c08f0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0900:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0910:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0920:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0930:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0940:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0950:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0960:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0970:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0980:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0990:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c09a0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c09b0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c09c0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c09d0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c09e0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c09f0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0a00:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0a10:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0a20:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0a30:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0a40:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0a50:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0a60:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0a70:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0a80:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0a90:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0aa0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0ab0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0ac0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0ad0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0ae0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0af0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0b00:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0b10:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0b20:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0b30:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0b40:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0b50:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0b60:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0b70:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0b80:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0b90:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0ba0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0bb0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0bc0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0bd0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0be0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0bf0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0c00:     0                               deadc0dedeadc0de
0xc00000037d3c0c10:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0c20:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0c30:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0c40:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0c50:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0c60:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0c70:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0c80:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0c90:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0ca0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0cb0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0cc0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0cd0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0ce0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0cf0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0d00:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0d10:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0d20:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0d30:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0d40:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0d50:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0d60:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0d70:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0d80:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0d90:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0da0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0db0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0dc0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0dd0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0de0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0df0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0e00:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0e10:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0e20:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0e30:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0e40:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0e50:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0e60:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0e70:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0e80:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0e90:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0ea0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0eb0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0ec0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0ed0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0ee0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0ef0:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0f00:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0f10:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0f20:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0f30:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0f40:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0f50:     deadc0dedeadc0de                deadc0dedeadc0de
0xc00000037d3c0f60:     0                               0
0xc00000037d3c0f70:     0                               0
0xc00000037d3c0f80:     0                               0
0xc00000037d3c0f90:     c0000003fd428540                c00000025fa02f90
0xc00000037d3c0fa0:     c0000003fd428610                c000000151073f90
0xc00000037d3c0fb0:     c00000037d3c0000                ffffffffffffffff
0xc00000037d3c0fc0:     ffffffffffffffff                ffffffffffffffff
0xc00000037d3c0fd0:     ffffffffffffffff                0
0xc00000037d3c0fe0:     0                               0
0xc00000037d3c0ff0:     0                               29080000000000
Comment 6 Sean Bruno freebsd_committer 2018-11-25 13:20:22 UTC
This looks a bit more useful.  


panic: Memory modified after free 0xc00000036e8621a0(32) val=0 @ 0xc00000036e8621a0

cpuid = 3
time = 1543128633
KDB: stack backtrace:
0xe00000008ac7cd80: at .kdb_backtrace+0x5c
0xe00000008ac7ceb0: at .vpanic+0x1b4
0xe00000008ac7cf70: at .panic+0x38
0xe00000008ac7d000: at .trash_ctor+0x58
0xe00000008ac7d080: at .uma_zalloc_arg+0x1f0
0xe00000008ac7d140: at .uma_zalloc_pcpu_arg+0x174
0xe00000008ac7d1e0: at .uma_zfree_arg+0x43c
0xe00000008ac7d290: at .free+0xb4
0xe00000008ac7d320: at .zfs_kmem_free+0x18
0xe00000008ac7d3a0: at .zio_data_buf_free+0x90
0xe00000008ac7d430: at .arc_space_return+0x3f0
0xe00000008ac7d4d0: at .arc_loan_compressed_buf+0x590
0xe00000008ac7d570: at .arc_buf_destroy+0x22c
0xe00000008ac7d620: at .dbuf_destroy+0x98
0xe00000008ac7d6d0: at .dbuf_rm_spill+0x260
0xe00000008ac7d760: at .dbuf_rm_spill+0x5b8
0xe00000008ac7d850: at .fork_exit+0xd0
0xe00000008ac7d8f0: at .fork_trampoline+0x10
0xe00000008ac7d920: at -0x4
KDB: enter: panic
[ thread pid 3 tid 100168 ]
Stopped at      .kdb_enter+0x60:        ld      r2, r1, 0x28
Comment 7 Mark Johnston freebsd_committer 2018-11-25 19:20:26 UTC
(In reply to Sean Bruno from comment #6)
Indeed, this points again at the UMA bucket zones.  The stack is somewhat bogus; I believe the only way that we can call uma_zalloc() while freeing something is by allocating a bucket.  I think the best approach will be to figure out why memguard-enabled kernels don't boot, and then use memguard to find the use-after-free.
Comment 8 Leandro Lupori 2018-11-30 11:13:56 UTC
After some hours of package building on ZFS I got this panic:

panic: Memory modified after free 0xc0000006152a4200(128) val=0 @ 0xc0000006152a4200

cpuid = 63
time = 1543556801
KDB: stack backtrace:
0xe0000002c2b69b20: at .kdb_backtrace+0x5c
0xe0000002c2b69c50: at .vpanic+0x1b4
0xe0000002c2b69d10: at .panic+0x38
0xe0000002c2b69da0: at .trash_ctor+0x58
0xe0000002c2b69e20: at .uma_zdestroy+0x824
0xe0000002c2b69ed0: at .uma_zalloc_arg+0x740
0xe0000002c2b69f90: at .uma_zalloc_pcpu_arg+0x174
0xe0000002c2b6a030: at .uma_zalloc_arg+0x4d0
0xe0000002c2b6a0f0: at .uma_zalloc_pcpu_arg+0x174
0xe0000002c2b6a190: at .uma_zfree_arg+0x43c
0xe0000002c2b6a240: at ._fdrop+0xa8
0xe0000002c2b6a2d0: at .closef+0x27c
0xe0000002c2b6a3d0: at .fdsetugidsafety+0x350
0xe0000002c2b6a480: at .kern_close+0x1f0
0xe0000002c2b6a530: at .sys_close+0x18
0xe0000002c2b6a5b0: at .trap+0x664
0xe0000002c2b6a770: at .powerpc_interrupt+0x290
0xe0000002c2b6a810: user SC trap by 0x811a49fe8: srr1=0x900000000000f032
            r1=0x3fffffffffffb740 cr=0x24024828 xer=0 ctr=0x811a49fe0 r2=0x811aa5a90
Comment 9 Leandro Lupori 2019-02-01 18:32:26 UTC
r343616 seems to fix this issue, based on the commit description and in my investigations.

I'll leave a machine building for 1 or 2 days with this change to confirm.
Comment 10 Mark Johnston freebsd_committer 2019-02-01 18:42:40 UTC
(In reply to Leandro Lupori from comment #9)
I was wondering about that, but AFAICS that bug was introduced in r343026, i.e., after the powerpc issue was observed.
Comment 11 Leandro Lupori freebsd_committer 2019-02-05 10:35:24 UTC
(In reply to Mark Johnston from comment #10)

Yes, you are right, r343616 fixes the bug introduced by r343026.

However, I cannot reproduce this issue anymore on my machine.
So, maybe another revision fixed the issue not fixed by r343616?
Or there is something else that needs to happen to reproduce the previous issue?
I'll keep an eye on this issue, if it happens again, but for now I'll stop trying to reproduce it, because maybe it is gone anyway.
Comment 12 Mark Johnston freebsd_committer 2019-02-05 18:52:38 UTC
(In reply to Leandro Lupori from comment #11)
Prior to rXXXXXX, zone_alloc_bucket() did:

max = MIN(bucket->ub_entries, zone->uz_count);
bucket->ub_cnt = zone->uz_import(zone->uz_arg, bucket->ub_bucket,
    max, domain, flags);

However, the zone lock is not held at this point, so uz_count may change.  In particular, since MIN is a macro that evaluates its arguments twice, I believe it's possible for max to end up being larger than bucket->ub_entries, which would result in a use-after-free.

r343026 fixed this problem as part of some refactoring, so that might explain why you don't see it anymore.
Comment 13 Sean Bruno freebsd_committer 2019-02-20 13:30:05 UTC
We just completed a full package set rebuild on pylon.nyi.freebsd.org

Marking this as fixed.  Thank you!