Bug 257285 - Kernel panic at ip_divert_event_tag on FreeBSD 13.0-RELEASE
Summary: Kernel panic at ip_divert_event_tag on FreeBSD 13.0-RELEASE
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.0-STABLE
Hardware: i386 Any
: --- Affects Only Me
Assignee: Mark Johnston
URL: https://reviews.freebsd.org/D30129
Keywords: crash
Depends on:
Blocks:
 
Reported: 2021-07-19 20:28 UTC by Deyan
Modified: 2021-07-24 14:07 UTC (History)
1 user (show)

See Also:
koobs: mfc-stable13+
markj: mfc-stable12-
markj: mfc-stable11-


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Deyan 2021-07-19 20:28:28 UTC
I get kernel panic on my freebsd 13 system:

panic: vm_fault_lookup: fault on nofault entry, addr: 0
cpuid = 1
time = 1626721843
KDB: stack backtrace:
#0 0x104318f at kdb_backtrace+0x4f
#1 0xffc71a at vpanic+0x11a
#2 0xffc5f4 at panic+0x14
#3 0x1257943 at vm_fault+0x1753
#4 0x1256127 at vm_fault_trap+0x87
#5 0x139f276 at trap_pfault+0x146
#6 0x139e841 at trap+0x381
#7 0xffc0319f at ip_divert_event_tag+0xe29cc977
#8 0x28 at ll+0x7
Uptime: 5h57m28s
Automatic reboot in 15 seconds - press a key on the console to abort
Rebooting...

Intervals between panic events vary from several minutes to several tens of hours. I haven't seen my system running for more than two days.

Version info & HW:
FreeBSD 13.0-RELEASE
CPU: Intel(R) Atom(TM) CPU  330   @ 1.60GHz
MB: INTEL  D945GLF2
Video: agp0: <Intel 82945G (945G GMCH) SVGA controller> on vgapci0
Audio controller: hdac0: <Intel 82801G HDA Controller> mem 0x883a0000-0x883a3fff irq 22 at device 27.0 on pci0
Eth adapters:
        re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0x2000-0x20ff mem 0x88200000-0x88200fff,0x88000000-0x8800ffff irq 16 at device 0.0 on pci1
        rl0: <RealTek 8139 10/100BaseTX> port 0x1000-0x10ff mem 0x88100000 -0x881000ff irq 21 at device 0.0 on pci2
HDD Controller: atapci0: <Intel ICH7 SATA300 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30a0-0x30af irq 19 at device 31.2 on pci0
HDD: ada0: <WDC WD5000AAKX-603CA0 18.01H18> ATA8-ACS SATA 3.x device

Loaded modules:
# kldstat
Id Refs Address        Size Name
 1   10   0x800000  177e46c kernel
 2    1 0x1d200000     6000 ichsmb.ko
 3    1 0x1d206000     4000 smbus.ko
 4    2 0x1d20a000    28000 ipfw.ko
 5    1 0x1d232000     5000 ipdivert.ko
Comment 1 Mark Johnston freebsd_committer 2021-07-19 20:30:40 UTC
Do you have the latest 13.0 patches from freebsd-update?  There was an ipdivert bug fix that could be related.
Comment 2 Deyan 2021-07-19 20:36:32 UTC
(In reply to Mark Johnston from comment #1)

I am not sure if I have these. How can I check?
Today I built a new kernel to test if this will change something - it
had no effect.
Comment 3 Deyan 2021-07-19 20:55:20 UTC
I just managed to apply patches up to p3.
freebsd-update output said:

The following files will be updated as part of updating to
13.0-RELEASE-p3:
/bin/freebsd-version
/boot/kernel/if_vlan.ko
/boot/kernel/ipdivert.ko
/boot/kernel/ipfw.ko
/boot/kernel/kernel
/boot/kernel/linux.ko
/boot/kernel/mpt.ko
/boot/kernel/virtio_pci.ko
/lib/libcasper.so.1
/usr/bin/bc
/usr/bin/dc
/usr/lib/debug/boot/kernel/if_vlan.ko.debug
/usr/lib/debug/boot/kernel/ipdivert.ko.debug
/usr/lib/debug/boot/kernel/ipfw.ko.debug
/usr/lib/debug/boot/kernel/kernel.debug
/usr/lib/debug/boot/kernel/linux.ko.debug
/usr/lib/debug/boot/kernel/mpt.ko.debug
/usr/lib/debug/boot/kernel/virtio_pci.ko.debug
/usr/lib/libradius.a
/usr/lib/libradius.so.4
/usr/lib/libradius_p.a
/usr/src/contrib/bc/src/program.c
/usr/src/lib/libc++/Makefile
/usr/src/lib/libcasper/libcasper/libcasper_impl.c
/usr/src/lib/libcasper/libcasper/libcasper_impl.h
/usr/src/lib/libcasper/libcasper/service.c
/usr/src/lib/libcasper/libcasper/zygote.c
/usr/src/lib/libradius/radlib.c
/usr/src/sys/amd64/amd64/support.S
/usr/src/sys/amd64/linux/linux_support.s
/usr/src/sys/amd64/linux32/linux32_support.s
/usr/src/sys/compat/linux/linux_futex.c
/usr/src/sys/conf/newvers.sh
/usr/src/sys/dev/mpt/mpt_cam.c
/usr/src/sys/dev/pms/freebsd/driver/ini/src/agdef.h
/usr/src/sys/dev/pms/freebsd/driver/ini/src/agtiapi.c
/usr/src/sys/dev/virtio/pci/virtio_pci_legacy.c
/usr/src/sys/net/if_vlan.c
/usr/src/sys/netinet/ip_divert.c
/usr/src/sys/netpfil/ipfw/ip_fw_pfil.c
/usr/src/tools/build/mk/OptionalObsoleteFiles.inc

# uname -r -> 13.0-RELEASE-p3
I will report in a few days if patching did ellimnate my kernel panics.
Comment 4 Mark Johnston freebsd_committer 2021-07-19 21:37:19 UTC
(In reply to Deyan from comment #2)
Which branch did you build the kernel from?
Comment 5 Mark Johnston freebsd_committer 2021-07-19 21:45:42 UTC
Hmm, ipdivert might be a red herring, I didn't notice the large offset before:

#7 0xffc0319f at ip_divert_event_tag+0xe29cc977

Assuming that you are in fact using divert sockets though (e.g., some ipfw rule diverts packets to natd), it is worth ruling out that problem since it can cause random panics.
Comment 6 Deyan 2021-07-19 22:46:19 UTC
(In reply to Mark Johnston from comment #5)
I have such a rule in ipfw:
${fw} 500 divert natd ip from any to any in via ${oif}

my natd.conf has these lines:

interface ${natd_interface}
dynamic yes
unregistered_only yes
redirect_port tcp ...
... multiple lines as the one above follow
Comment 7 Deyan 2021-07-19 22:50:46 UTC
(In reply to Mark Johnston from comment #4)

I suppose it was the 13.0 RELEASE
I followed this manual: https://docs.freebsd.org/doc/6.1-RELEASE/usr/share/doc/handbook/kernelconfig-building.html Procedure 2. Building a Kernel the “New” Way
Comment 8 Mark Johnston freebsd_committer 2021-07-23 13:09:22 UTC
Have there been any crashes on 13.0-p3?
Comment 9 Deyan 2021-07-23 13:47:04 UTC
Hi,
no crashes after my last comment so I assume this issue is no more present on p3.

You may eventually close this ticket if no deep-digging to find the root cause is necessary.
Comment 10 Mark Johnston freebsd_committer 2021-07-23 13:58:51 UTC
(In reply to Deyan from comment #9)
Thanks.  I can't find another PR to dup this to, so I'll just mark it fixed.  I suspect the panic was addressed by https://cgit.freebsd.org/src/commit/?id=a1fadf7de25b973a308b86d04c4ada4fa8be193f

Since that can cause fairly random memory corruption I don't think it's worth digging deeper into this particular instance.
Comment 11 Kubilay Kocak freebsd_committer freebsd_triage 2021-07-24 02:29:17 UTC
^Triage: Assign to committer that (tentatively/probably :)) resolved, tracke merges

(In reply to Mark Johnston from comment #10)

Did stable/12,11 need/receive the merge? I believe epoch stuff was 12, or 12/13 only? Set '-' for the ones that didn't, ta!
Comment 12 Mark Johnston freebsd_committer 2021-07-24 14:07:25 UTC
(In reply to Kubilay Kocak from comment #11)
The bug is only in 13.0.