During substantial ZFS disk activity, the pkg builder in the cluster will fail. This is a Tyan PowerPC64 Machine. rm: /usr/local/poudriere/data/.m/head-powerpc64-default/ref/lib: Directory not empty panic: Memory modified after free 0xc000000065ceec40(32) val=0 @ 0xc000000065ceec40 cpuid = 31 time = 1542804498 KDB: stack backtrace: 0xe000000090207020: at .kdb_backtrace+0x5c 0xe000000090207150: at .vpanic+0x1b4 0xe000000090207210: at .panic+0x38 0xe0000000902072a0: at .trash_ctor+0x58 0xe000000090207320: at .trash_fini+0x1c 0xe0000000902073a0: at .uma_zdestroy+0x164 0xe000000090207450: at .uma_zdestroy+0x42c 0xe0000000902074e0: at .sys_swapoff+0x2c4 0xe000000090207570: at .uma_zfree_pcpu_arg+0x2ec 0xe000000090207600: at .zone_drain+0x18 0xe000000090207680: at .uma_avail+0x4c4 0xe000000090207710: at .zone_drain+0x378 0xe0000000902077a0: at .uma_reclaim_worker+0x20c 0xe000000090207850: at .fork_exit+0xd0 0xe0000000902078f0: at .fork_trampoline+0x10 0xe000000090207920: at -0x4 KDB: enter: panic [ thread pid 15 tid 100219 ] Stopped at .kdb_enter+0x60: ld r2, r1, 0x28 db> bt
The first step will be to figure out which zone this is. I think it'll be tricky to track this down without vmcores to look at, but this patch will give us a starting point. diff --git a/sys/vm/uma_core.c b/sys/vm/uma_core.c index 7d14586a31cd..8087a86584c9 100644 --- a/sys/vm/uma_core.c +++ b/sys/vm/uma_core.c @@ -1041,6 +1041,7 @@ void zone_drain(uma_zone_t zone) { + printf("draining zone %s\n", zone->uz_name); zone_drain_wait(zone, M_NOWAIT); }
(In reply to Mark Johnston from comment #1) Oof. That's dumping a lot of text to the serial console at the moment. Every drain is emitting a line of text and its happening a lot. It probably won't hit the bug if its a race-style I suspect.
I'm not sure this is helpful, but with the debug printf in the kernel the panic looks like this on the console. Is there any debugging I can do from the db> prompt to give us a clue here? draining zone 2048 draining zone 1024 draining zone 512 draining zone 256 draining zone 128 draining zone 64 draining zone 32 draining zone 16 draining zone mt_zone draining zone mt_stats_zone draining zone 64 pcpu draining zone fakepg draining zone UPVO entry draining zone VMSPACE draining zone MAP ENTRY draining zone KMAP ENTRY draining zone MAP draining zone RADIX NODE draining zone VM OBJECT draining zone vmem btag draining zone vmem draining zone 256 Bucket draining zone 128 Bucket draining zone 64 Bucket draining zone 32 Bucket draining zone 16 Bucket draining zone 12 Bucket panic: Memory modified after free 0xc00000037d3c0c00(96) val=0 @ 0xc00000037d3c0c00 cpuid = 31 time = 1542904594 KDB: stack backtrace: 0xe000000090207010: at .kdb_backtrace+0x5c 0xe000000090207140: at .vpanic+0x1b4 0xe000000090207200: at .panic+0x38 0xe000000090207290: at .trash_ctor+0x58 0xe000000090207310: at .trash_fini+0x1c 0xe000000090207390: at .uma_zdestroy+0x164 0xe000000090207440: at .uma_zdestroy+0x42c 0xe0000000902074d0: at .sys_swapoff+0x2c4 0xe000000090207560: at .uma_zfree_pcpu_arg+0x2ec 0xe0000000902075f0: at .zone_drain+0x34 0xe000000090207680: at .uma_avail+0x4c4 0xe000000090207710: at .zone_drain+0x398 0xe0000000902077a0: at .uma_reclaim_worker+0x20c 0xe000000090207850: at .fork_exit+0xd0 0xe0000000902078f0: at .fork_trampoline+0x10 0xe000000090207920: at -0x4 KDB: enter: panic [ thread pid 15 tid 100219 ] Stopped at .kdb_enter+0x60: ld r2, r1, 0x28
We will want to repro this a number of times to see if the zone is consistent. I will provide a patch to embed the name of the zone in the panic message instead. Could you provide the output of: db> x/gx 0xc00000037d3c0000,200 assuming you still have the system in ddb? (There will be quite a bit of output.)
(In reply to Mark Johnston from comment #4) Its *almost* all deadc0de, except for one or two bits: db> x/gx 0xc00000037d3c0000,200 0xc00000037d3c0000: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0010: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0020: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0030: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0040: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0050: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0060: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0070: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0080: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0090: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c00a0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c00b0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c00c0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c00d0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c00e0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c00f0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0100: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0110: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0120: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0130: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0140: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0150: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0160: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0170: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0180: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0190: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c01a0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c01b0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c01c0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c01d0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c01e0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c01f0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0200: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0210: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0220: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0230: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0240: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0250: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0260: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0270: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0280: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0290: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c02a0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c02b0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c02c0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c02d0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c02e0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c02f0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0300: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0310: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0320: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0330: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0340: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0350: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0360: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0370: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0380: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0390: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c03a0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c03b0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c03c0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c03d0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c03e0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c03f0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0400: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0410: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0420: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0430: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0440: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0450: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0460: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0470: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0480: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0490: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c04a0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c04b0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c04c0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c04d0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c04e0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c04f0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0500: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0510: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0520: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0530: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0540: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0550: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0560: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0570: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0580: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0590: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c05a0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c05b0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c05c0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c05d0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c05e0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c05f0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0600: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0610: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0620: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0630: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0640: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0650: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0660: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0670: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0680: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0690: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c06a0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c06b0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c06c0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c06d0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c06e0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c06f0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0700: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0710: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0720: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0730: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0740: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0750: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0760: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0770: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0780: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0790: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c07a0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c07b0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c07c0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c07d0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c07e0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c07f0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0800: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0810: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0820: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0830: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0840: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0850: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0860: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0870: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0880: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0890: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c08a0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c08b0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c08c0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c08d0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c08e0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c08f0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0900: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0910: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0920: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0930: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0940: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0950: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0960: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0970: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0980: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0990: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c09a0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c09b0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c09c0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c09d0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c09e0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c09f0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0a00: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0a10: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0a20: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0a30: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0a40: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0a50: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0a60: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0a70: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0a80: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0a90: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0aa0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0ab0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0ac0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0ad0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0ae0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0af0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0b00: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0b10: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0b20: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0b30: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0b40: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0b50: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0b60: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0b70: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0b80: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0b90: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0ba0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0bb0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0bc0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0bd0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0be0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0bf0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0c00: 0 deadc0dedeadc0de 0xc00000037d3c0c10: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0c20: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0c30: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0c40: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0c50: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0c60: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0c70: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0c80: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0c90: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0ca0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0cb0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0cc0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0cd0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0ce0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0cf0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0d00: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0d10: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0d20: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0d30: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0d40: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0d50: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0d60: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0d70: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0d80: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0d90: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0da0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0db0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0dc0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0dd0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0de0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0df0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0e00: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0e10: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0e20: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0e30: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0e40: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0e50: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0e60: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0e70: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0e80: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0e90: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0ea0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0eb0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0ec0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0ed0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0ee0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0ef0: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0f00: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0f10: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0f20: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0f30: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0f40: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0f50: deadc0dedeadc0de deadc0dedeadc0de 0xc00000037d3c0f60: 0 0 0xc00000037d3c0f70: 0 0 0xc00000037d3c0f80: 0 0 0xc00000037d3c0f90: c0000003fd428540 c00000025fa02f90 0xc00000037d3c0fa0: c0000003fd428610 c000000151073f90 0xc00000037d3c0fb0: c00000037d3c0000 ffffffffffffffff 0xc00000037d3c0fc0: ffffffffffffffff ffffffffffffffff 0xc00000037d3c0fd0: ffffffffffffffff 0 0xc00000037d3c0fe0: 0 0 0xc00000037d3c0ff0: 0 29080000000000
This looks a bit more useful. panic: Memory modified after free 0xc00000036e8621a0(32) val=0 @ 0xc00000036e8621a0 cpuid = 3 time = 1543128633 KDB: stack backtrace: 0xe00000008ac7cd80: at .kdb_backtrace+0x5c 0xe00000008ac7ceb0: at .vpanic+0x1b4 0xe00000008ac7cf70: at .panic+0x38 0xe00000008ac7d000: at .trash_ctor+0x58 0xe00000008ac7d080: at .uma_zalloc_arg+0x1f0 0xe00000008ac7d140: at .uma_zalloc_pcpu_arg+0x174 0xe00000008ac7d1e0: at .uma_zfree_arg+0x43c 0xe00000008ac7d290: at .free+0xb4 0xe00000008ac7d320: at .zfs_kmem_free+0x18 0xe00000008ac7d3a0: at .zio_data_buf_free+0x90 0xe00000008ac7d430: at .arc_space_return+0x3f0 0xe00000008ac7d4d0: at .arc_loan_compressed_buf+0x590 0xe00000008ac7d570: at .arc_buf_destroy+0x22c 0xe00000008ac7d620: at .dbuf_destroy+0x98 0xe00000008ac7d6d0: at .dbuf_rm_spill+0x260 0xe00000008ac7d760: at .dbuf_rm_spill+0x5b8 0xe00000008ac7d850: at .fork_exit+0xd0 0xe00000008ac7d8f0: at .fork_trampoline+0x10 0xe00000008ac7d920: at -0x4 KDB: enter: panic [ thread pid 3 tid 100168 ] Stopped at .kdb_enter+0x60: ld r2, r1, 0x28
(In reply to Sean Bruno from comment #6) Indeed, this points again at the UMA bucket zones. The stack is somewhat bogus; I believe the only way that we can call uma_zalloc() while freeing something is by allocating a bucket. I think the best approach will be to figure out why memguard-enabled kernels don't boot, and then use memguard to find the use-after-free.
After some hours of package building on ZFS I got this panic: panic: Memory modified after free 0xc0000006152a4200(128) val=0 @ 0xc0000006152a4200 cpuid = 63 time = 1543556801 KDB: stack backtrace: 0xe0000002c2b69b20: at .kdb_backtrace+0x5c 0xe0000002c2b69c50: at .vpanic+0x1b4 0xe0000002c2b69d10: at .panic+0x38 0xe0000002c2b69da0: at .trash_ctor+0x58 0xe0000002c2b69e20: at .uma_zdestroy+0x824 0xe0000002c2b69ed0: at .uma_zalloc_arg+0x740 0xe0000002c2b69f90: at .uma_zalloc_pcpu_arg+0x174 0xe0000002c2b6a030: at .uma_zalloc_arg+0x4d0 0xe0000002c2b6a0f0: at .uma_zalloc_pcpu_arg+0x174 0xe0000002c2b6a190: at .uma_zfree_arg+0x43c 0xe0000002c2b6a240: at ._fdrop+0xa8 0xe0000002c2b6a2d0: at .closef+0x27c 0xe0000002c2b6a3d0: at .fdsetugidsafety+0x350 0xe0000002c2b6a480: at .kern_close+0x1f0 0xe0000002c2b6a530: at .sys_close+0x18 0xe0000002c2b6a5b0: at .trap+0x664 0xe0000002c2b6a770: at .powerpc_interrupt+0x290 0xe0000002c2b6a810: user SC trap by 0x811a49fe8: srr1=0x900000000000f032 r1=0x3fffffffffffb740 cr=0x24024828 xer=0 ctr=0x811a49fe0 r2=0x811aa5a90
r343616 seems to fix this issue, based on the commit description and in my investigations. I'll leave a machine building for 1 or 2 days with this change to confirm.
(In reply to Leandro Lupori from comment #9) I was wondering about that, but AFAICS that bug was introduced in r343026, i.e., after the powerpc issue was observed.
(In reply to Mark Johnston from comment #10) Yes, you are right, r343616 fixes the bug introduced by r343026. However, I cannot reproduce this issue anymore on my machine. So, maybe another revision fixed the issue not fixed by r343616? Or there is something else that needs to happen to reproduce the previous issue? I'll keep an eye on this issue, if it happens again, but for now I'll stop trying to reproduce it, because maybe it is gone anyway.
(In reply to Leandro Lupori from comment #11) Prior to rXXXXXX, zone_alloc_bucket() did: max = MIN(bucket->ub_entries, zone->uz_count); bucket->ub_cnt = zone->uz_import(zone->uz_arg, bucket->ub_bucket, max, domain, flags); However, the zone lock is not held at this point, so uz_count may change. In particular, since MIN is a macro that evaluates its arguments twice, I believe it's possible for max to end up being larger than bucket->ub_entries, which would result in a use-after-free. r343026 fixed this problem as part of some refactoring, so that might explain why you don't see it anymore.
We just completed a full package set rebuild on pylon.nyi.freebsd.org Marking this as fixed. Thank you!