Hi! Trying to start mpd5, it loads negraph kernel module and I got root@rpi3:/usr/home/pi # service mpd5 onestart Starting mpd5. root@rpi3:/usr/home/pi # WARNING: attempt to domain_add(netgraph) after domainfinalize() Kernel page fault with the following non-sleepable locks held: exclusive sleep mutex ng_node (ng_node) r = 0 (0xfffffd000194b570) locked @ /usr/local/src/CURR/head/sys/netgraph/ng_base.c:2325 stack backtrace: #0 0xffff000000368e0c at witness_debugger+0x64 #1 0xffff00000036a16c at witness_warn+0x43c #2 0xffff0000005f6904 at data_abort+0xe0 #3 0xffff0000005e0074 at handle_el1h_sync+0x74 #4 0xffff000040d887e4 at ng_snd_item+0x36c #5 0xffff000040d45d84 at ngc_send+0x1bc #6 0xffff0000003968e0 at sosend_generic+0x380 #7 0xffff00000039cc20 at kern_sendit+0x28c #8 0xffff00000039cf08 at sendit+0x1a4 #9 0xffff00000039cd54 at sys_sendto+0x50 #10 0xffff0000005f72fc at do_el0_sync+0x884 #11 0xffff0000005e01f4 at handle_el0_sync+0x74 x0: fffffd000194b570 x1: 8 x2: ffff000040d76637 x3: 915 x4: 0 x5: 3 x6: fffffd000f0fc700 x7: 40 x8: 2 x9: 0 x10: 18 x11: 10000 x12: ffff000000a2b218 x13: ffffffff x14: 0 x15: fffffd000f0fc690 x16: ffff000040d967c8 x17: ffff0000002eca4c x18: ffff0000402f36f0 x19: 0 x20: fffffd000194b588 x21: fffffd000194b500 x22: 0 x23: fffffd000194b568 x24: fffffd0001994338 x25: fffffd0001fb24c0 x26: 0 x27: fffffd0001b6c6c8 x28: 0 x29: ffff0000402f3720 sp: ffff0000402f36f0 lr: ffff000040d887e8 elr: ffff000040d887f4 spsr: 80000345 far: 0 esr: 96000007 panic: data abort in critical section or under mutex cpuid = 2 time = 1511451953 KDB: stack backtrace: db_trace_self() at db_trace_self_wrapper+0x28 pc = 0xffff0000005de110 lr = 0xffff000000088a94 sp = 0xffff0000402f3150 fp = 0xffff0000402f3360 db_trace_self_wrapper() at vpanic+0x184 pc = 0xffff000000088a94 lr = 0xffff00000030ba3c sp = 0xffff0000402f3370 fp = 0xffff0000402f33f0 vpanic() at panic+0x48 pc = 0xffff00000030ba3c lr = 0xffff00000030bac8 sp = 0xffff0000402f3400 fp = 0xffff0000402f3480 panic() at data_abort+0x250 pc = 0xffff00000030bac8 lr = 0xffff0000005f6a74 sp = 0xffff0000402f3490 fp = 0xffff0000402f3540 data_abort() at handle_el1h_sync+0x74 pc = 0xffff0000005f6a74 lr = 0xffff0000005e0074 sp = 0xffff0000402f3550 fp = 0xffff0000402f3660 handle_el1h_sync() at ng_snd_item+0x36c pc = 0xffff0000005e0074 lr = 0xffff000040d887e4 sp = 0xffff0000402f3670 fp = 0xffff0000402f3720 ng_snd_item() at ngc_send+0x1bc pc = 0xffff000040d887e4 lr = 0xffff000040d45d84 sp = 0xffff0000402f3730 fp = 0xffff0000402f37b0 ngc_send() at sosend_generic+0x380 pc = 0xffff000040d45d84 lr = 0xffff0000003968e0 sp = 0xffff0000402f37c0 fp = 0xffff0000402f3860 sosend_generic() at kern_sendit+0x28c pc = 0xffff0000003968e0 lr = 0xffff00000039cc20 sp = 0xffff0000402f3870 fp = 0xffff0000402f3900 kern_sendit() at sendit+0x1a4 pc = 0xffff00000039cc20 lr = 0xffff00000039cf08 sp = 0xffff0000402f3910 fp = 0xffff0000402f3950 sendit() at sys_sendto+0x50 pc = 0xffff00000039cf08 lr = 0xffff00000039cd54 sp = 0xffff0000402f3960 fp = 0xffff0000402f39a0 sys_sendto() at do_el0_sync+0x884 pc = 0xffff00000039cd54 lr = 0xffff0000005f72fc sp = 0xffff0000402f39b0 fp = 0xffff0000402f3a70 do_el0_sync() at handle_el0_sync+0x74 pc = 0xffff0000005f72fc lr = 0xffff0000005e01f4 sp = 0xffff0000402f3a80 fp = 0xffff0000402f3b90 handle_el0_sync() at 0x405e4e28 pc = 0xffff0000005e01f4 lr = 0x00000000405e4e28 sp = 0xffff0000402f3ba0 fp = 0x0000ffffffffe770 KDB: enter: panic [ thread pid 680 tid 100093 ] Stopped at kdb_enter+0x40: undefined d4200000 db> Seems like no correct lock procedure in *.c file
FreeBSD SVN revision 320146 Kernel config below # # GENERIC -- Generic kernel configuration file for FreeBSD/arm64 # # For more information on this file, please read the config(5) manual page, # and/or the handbook section on Kernel Configuration Files: # # http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html # # The handbook is also available locally in /usr/share/doc/handbook # if you've installed the doc distribution, otherwise always see the # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the # latest information. # # An exhaustive list of options and more detailed explanations of the # device lines is also present in the ../../conf/NOTES and NOTES files. # If you are in doubt as to the purpose or necessity of a line, check first # in NOTES. # # $FreeBSD: head/sys/arm64/conf/GENERIC 318744 2017-05-23 13:57:55Z kib $ cpu ARM64 ident GENERIC makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols makeoptions WITH_CTF=1 # Run ctfconvert(1) for DTrace support options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options INET6 # IPv6 communications protocols options IPSEC # IP (v4/v6) security options TCP_HHOOK # hhook(9) framework for TCP options TCP_OFFLOAD # TCP offload options SCTP # Stream Control Transmission Protocol options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling options QUOTA # Enable disk quotas for UFS options MD_ROOT # MD is a potential root device options NFSCL # Network Filesystem Client options NFSD # Network Filesystem Server options NFSLOCKD # Network Lock Manager options NFS_ROOT # NFS usable as /, requires NFSCL options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_RAID # Soft RAID functionality. options GEOM_LABEL # Provides labelization options COMPAT_FREEBSD11 # Compatible with FreeBSD11 options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options CAPABILITY_MODE # Capsicum capability mode options CAPABILITIES # Capsicum capabilities options MAC # TrustedBSD MAC Framework options KDTRACE_FRAME # Ensure frames are compiled in options KDTRACE_HOOKS # Kernel DTrace hooks options VFP # Floating-point support options RACCT # Resource accounting framework options RACCT_DEFAULT_TO_DISABLED # Set kern.racct.enable=0 by default options RCTL # Resource limits options SMP options INTRNG # Debugging support. Always need this: options KDB # Enable kernel debugger support. options KDB_TRACE # Print a stack trace for a panic. # For full debugger support use (turn off in stable branch): options DDB # Support DDB. #options GDB # Support remote GDB. options DEADLKRES # Enable the deadlock resolver options INVARIANTS # Enable calls of extra sanity checking options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS options WITNESS # Enable checks to detect deadlocks and cycles options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed options MALLOC_DEBUG_MAXZONES=8 # Separate malloc(9) zones # SoC support options SOC_ALLWINNER_A64 options SOC_CAVM_THUNDERX options SOC_HISI_HI6220 options SOC_BRCM_BCM2837 # Annapurna Alpine drivers device al_ccu # Alpine Cache Coherency Unit device al_nb_service # Alpine North Bridge Service device al_iofic # I/O Fabric Interrupt Controller device al_serdes # Serializer/Deserializer device al_udma # Universal DMA # VirtIO support device virtio device virtio_pci device virtio_mmio device virtio_blk device vtnet # CPU frequency control device cpufreq # Bus drivers device pci device al_pci # Annapurna Alpine PCI-E options PCI_HP # PCI-Express native HotPlug options PCI_IOV # PCI SR-IOV support # Ethernet NICs device mii device miibus # MII bus support device awg # Allwinner EMAC Gigabit Ethernet device axgbe # AMD Opteron A1100 integrated NIC device em # Intel PRO/1000 Gigabit Ethernet Family device ix # Intel 10Gb Ethernet Family device msk # Marvell/SysKonnect Yukon II Gigabit Ethernet device smc # SMSC LAN91C111 device vnic # Cavium ThunderX NIC device al_eth # Annapurna Alpine Ethernet NIC # Block devices device ahci device scbus device da # ATA/SCSI peripherals device pass # Passthrough device (direct ATA/SCSI access) # MMC/SD/SDIO Card slot support device sdhci device aw_mmc # Allwinner SD/MMC controller device mmc # mmc/sd bus device mmcsd # mmc/sd flash cards device dwmmc # Serial (COM) ports device uart # Generic UART driver device uart_ns8250 # ns8250-type UART driver device uart_snps device pl011 # USB support options USB_DEBUG # enable debug msgs device aw_ehci # Allwinner EHCI USB interface (USB 2.0) device aw_usbphy # Allwinner USB PHY device dwcotg # DWC OTG controller device ohci # OHCI USB interface device ehci # EHCI USB interface (USB 2.0) device xhci # XHCI PCI->USB interface (USB 3.0) device usb # USB Bus (required) device ukbd # Keyboard device umass # Disks/Mass storage - Requires scbus and da # USB ethernet support device smcphy device smsc # GPIO device aw_gpio # Allwinner GPIO controller device gpio device gpioled device fdt_pinctrl # I2C device aw_rsb # Allwinner Reduced Serial Bus device bcm2835_bsc # Broadcom BCM283x I2C bus device iicbus device iic # Clock and reset controllers device aw_ccu # Allwinner clock controller # Interrupt controllers device aw_nmi # Allwinner NMI support # Real-time clock support device aw_rtc # Allwinner Real-time Clock # Watchdog controllers device aw_wdog # Allwinner Watchdog # Power management controllers device axp81x # X-Powers AXP81x PMIC # EFUSE device aw_sid # Allwinner Secure ID EFUSE # Thermal sensors device aw_thermal # Allwinner Thermal Sensor Controller # SPI device spibus device bcm2835_spi # Broadcom BCM283x SPI bus # Console device vt device kbdmux # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device vlan # 802.1Q VLAN support device tun # Packet tunnel. device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device firmware # firmware assist module device psci # Support for ARM PSCI # EXT_RESOURCES pseudo devices options EXT_RESOURCES device clk device phy device hwreset device regulator # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter # Chip-specific errata options THUNDERX_PASS_1_1_ERRATA options FDT device acpi # The crypto framework is required by IPSEC device crypto # Required by IPSEC
MPD5 config default: load l2tp_client log +link +auth +chat l2tp_client: create bundle static L2TPB set iface up-script /usr/local/etc/mpd5/if-up.sh set iface down-script /usr/local/etc/mpd5/if-down.sh create link static L2TPL l2tp set link action bundle L2TPB set auth authname "candy" set auth password "kHnMJ%D5G" set link max-redial 0 set link keep-alive 20 75 set l2tp peer 77.37.146.85 open
You have to add NETGRAPH_DEBUG to your kernel configuration to obtain more debug information. Please note that NETGRAPH_DEBUG changes netgraph ABI, so you won't be able to use pre-built netgraph modules with such a kernel, so add all needed options statically: options NETGRAPH_DEBUG options NETGRAPH options NETGRAPH_SOCKET options NETGRAPH_L2TP options NETGRAPH_MPPC_COMPRESSION options NETGRAPH_MPPC_ENCRYPTION options NETGRAPH_IFACE options NETGRAPH_PPP options NETGRAPH_TEE options NETGRAPH_VJC options NETGRAPH_ETHER options NETGRAPH_ASYNC Then retry your test.
Done! Here the output: Multi-link PPP daemon for FreeBSD process 841 started, version 5.8Kernel page fault with the following non-sleepab exclusive sleep mutex ng_node (ng_node) r = 0 (0xfffffd00014e6670) locked @ /usr stack backtrace: #0 0xff (rooff000000368f00 at witness_debugger+0x64 #1 0xffff000000arm64-defaul36a260 at witness_warn+0x43c #2 0xffff000000616104 at data_abot-job-20 03:rt+0xe0 #3 0xffff0000005ff874 at hand38 18-le_el1h_sync+0x74 #4 0xffff00000041d22c at ng_snd_item+017) x3bc #5 0xffff000000433cf0 at ngc_send+0x20c #6 0xffff0000003969d4 at sosend_generic+0x380 #7 0xffff00000039cd14 at kern_sendit+0x28c #8 0xffff00000039cffc at sendit+0x1a4 #9 0xffff00000039ce48 at sys_sendto+0x50 #10 0xffff000000616afc at do_el0_sync+0x884 #11 0xffff0000005ff9f4 at handle_el0_sync+0x74 x0: fffffd00014e6670 x1: 8 x2: ffff0000006ad8a0 x3: 915 x4: 0 x5: 0 x6: fffffd00184b0c40 x7: 40 x8: 2 x9: 0 x10: 18 x11: 10000 x12: ffff000000a52648 x13: ffffffff x14: 0 x15: fffffd00184b0b60 x16: 406002b0 x17: 407924e4 x18: ffff00004034c6f0 x19: 0 x20: fffffd00014e6688 x21: fffffd00014e6600 x22: ffff0000006ad8a0 x23: fffffd00014e6668 x24: ffff00004034c740 x25: fffffd000164a038 x26: fffffd00184014c0 x27: fffffd0001c20370 x28: 0 x29: ffff00004034c720 sp: ffff00004034c6f0 lr: ffff00000041d230 elr: ffff00000041d23c spsr: 80000345 far: 0 esr: 96000007 panic: data abort in critical section or under mutex cpuid = 3 time = 1511729548 KDB: stack backtrace: db_trace_self() at db_trace_self_wrapper+0x28 pc = 0xffff0000005fde50 lr = 0xffff000000088b88 sp = 0xffff00004034c150 fp = 0xffff00004034c360 db_trace_self_wrapper() at vpanic+0x184 pc = 0xffff000000088b88 lr = 0xffff00000030bb30 sp = 0xffff00004034c370 fp = 0xffff00004034c3f0 vpanic() at panic+0x48 pc = 0xffff00000030bb30 lr = 0xffff00000030bbbc sp = 0xffff00004034c400 fp = 0xffff00004034c480 panic() at data_abort+0x250 pc = 0xffff00000030bbbc lr = 0xffff000000616274 sp = 0xffff00004034c490 fp = 0xffff00004034c540 data_abort() at handle_el1h_sync+0x74 pc = 0xffff000000616274 lr = 0xffff0000005ff874 sp = 0xffff00004034c550 fp = 0xffff00004034c660 handle_el1h_sync() at ng_snd_item+0x3bc pc = 0xffff0000005ff874 lr = 0xffff00000041d22c sp = 0xffff00004034c670 fp = 0xffff00004034c720 ng_snd_item() at ngc_send+0x20c pc = 0xffff00000041d22c lr = 0xffff000000433cf0 sp = 0xffff00004034c730 fp = 0xffff00004034c7b0 ngc_send() at sosend_generic+0x380 pc = 0xffff000000433cf0 lr = 0xffff0000003969d4 sp = 0xffff00004034c7c0 fp = 0xffff00004034c860 sosend_generic() at kern_sendit+0x28c pc = 0xffff0000003969d4 lr = 0xffff00000039cd14 sp = 0xffff00004034c870 fp = 0xffff00004034c900 kern_sendit() at sendit+0x1a4 pc = 0xffff00000039cd14 lr = 0xffff00000039cffc sp = 0xffff00004034c910 fp = 0xffff00004034c950 sendit() at sys_sendto+0x50 pc = 0xffff00000039cffc lr = 0xffff00000039ce48 sp = 0xffff00004034c960 fp = 0xffff00004034c9a0 sys_sendto() at do_el0_sync+0x884 pc = 0xffff00000039ce48 lr = 0xffff000000616afc sp = 0xffff00004034c9b0 fp = 0xffff00004034ca70 do_el0_sync() at handle_el0_sync+0x74 pc = 0xffff000000616afc lr = 0xffff0000005ff9f4 sp = 0xffff00004034ca80 fp = 0xffff00004034cb90 handle_el0_sync() at 0x405e4e28 pc = 0xffff0000005ff9f4 lr = 0x00000000405e4e28 sp = 0xffff00004034cba0 fp = 0x0000ffffffffe620 KDB: enter: panic [ thread pid 841 tid 100115 ] Stopped at kdb_enter+0x40: undefined d4200000
Here is more detailed out root@rpi3:/usr/home/pi # service mpd5 onestart Starting mpd5. root@rpi3:/usr/home/pi # Kernel page fault with the following non-sleepable locks held: exclusive sleep mutex ng_node (ng_node) r = 0 (0xfffffd0001b14e70) locked @ /usr/local/src/CURR/head/sys/netgraph/ng_base.c:2325 stack backtrace: #0 0xffff000000368f00 at witness_debugger+0x64 #1 0xffff00000036a260 at witness_warn+0x43c #2 0xffff000000616104 at data_abort+0xe0 #3 0xffff0000005ff874 at handle_el1h_sync+0x74 #4 0xffff00000041d22c at ng_snd_item+0x3bc #5 0xffff000000433cf0 at ngc_send+0x20c #6 0xffff0000003969d4 at sosend_generic+0x380 #7 0xffff00000039cd14 at kern_sendit+0x28c #8 0xffff00000039cffc at sendit+0x1a4 #9 0xffff00000039ce48 at sys_sendto+0x50 #10 0xffff000000616afc at do_el0_sync+0x884 #11 0xffff0000005ff9f4 at handle_el0_sync+0x74 x0: fffffd0001b14e70 x1: 8 x2: ffff0000006ad8a0 x3: 915 x4: 0 x5: 2 x6: fffffd0001fa8d20 x7: 40 x8: 2 x9: 0 x10: 18 x11: 10000 x12: ffff000000a52868 x13: ffffffff x14: 0 x15: fffffd0001fa8c40 x16: 406002b0 x17: 407924e4 x18: ffff00004031f6f0 x19: 0 x20: fffffd0001b14e88 x21: fffffd0001b14e00 x22: ffff0000006ad8a0 x23: fffffd0001b14e68 x24: ffff00004031f740 x25: fffffd0001bcc238 x26: fffffd0001de0200 x27: fffffd0001cf8018 x28: 0 x29: ffff00004031f720 sp: ffff00004031f6f0 lr: ffff00000041d230 elr: ffff00000041d23c spsr: 80000345 far: 0 esr: 96000007 panic: data abort in critical section or under mutex cpuid = 1 time = 1511729631 KDB: stack backtrace: db_trace_self() at db_trace_self_wrapper+0x28 pc = 0xffff0000005fde50 lr = 0xffff000000088b88 sp = 0xffff00004031f150 fp = 0xffff00004031f360 db_trace_self_wrapper() at vpanic+0x184 pc = 0xffff000000088b88 lr = 0xffff00000030bb30 sp = 0xffff00004031f370 fp = 0xffff00004031f3f0 vpanic() at panic+0x48 pc = 0xffff00000030bb30 lr = 0xffff00000030bbbc sp = 0xffff00004031f400 fp = 0xffff00004031f480 panic() at data_abort+0x250 pc = 0xffff00000030bbbc lr = 0xffff000000616274 sp = 0xffff00004031f490 fp = 0xffff00004031f540 data_abort() at handle_el1h_sync+0x74 pc = 0xffff000000616274 lr = 0xffff0000005ff874 sp = 0xffff00004031f550 fp = 0xffff00004031f660 handle_el1h_sync() at ng_snd_item+0x3bc pc = 0xffff0000005ff874 lr = 0xffff00000041d22c sp = 0xffff00004031f670 fp = 0xffff00004031f720 ng_snd_item() at ngc_send+0x20c pc = 0xffff00000041d22c lr = 0xffff000000433cf0 sp = 0xffff00004031f730 fp = 0xffff00004031f7b0 ngc_send() at sosend_generic+0x380 pc = 0xffff000000433cf0 lr = 0xffff0000003969d4 sp = 0xffff00004031f7c0 fp = 0xffff00004031f860 sosend_generic() at kern_sendit+0x28c pc = 0xffff0000003969d4 lr = 0xffff00000039cd14 sp = 0xffff00004031f870 fp = 0xffff00004031f900 kern_sendit() at sendit+0x1a4 pc = 0xffff00000039cd14 lr = 0xffff00000039cffc sp = 0xffff00004031f910 fp = 0xffff00004031f950 sendit() at sys_sendto+0x50 pc = 0xffff00000039cffc lr = 0xffff00000039ce48 sp = 0xffff00004031f960 fp = 0xffff00004031f9a0 sys_sendto() at do_el0_sync+0x884 pc = 0xffff00000039ce48 lr = 0xffff000000616afc sp = 0xffff00004031f9b0 fp = 0xffff00004031fa70 do_el0_sync() at handle_el0_sync+0x74 pc = 0xffff000000616afc lr = 0xffff0000005ff9f4 sp = 0xffff00004031fa80 fp = 0xffff00004031fb90 handle_el0_sync() at 0x405e4e28 pc = 0xffff0000005ff9f4 lr = 0x00000000405e4e28 sp = 0xffff00004031fba0 fp = 0x0000ffffffffe770 KDB: enter: panic [ thread pid 651 tid 100106 ] Stopped at kdb_enter+0x40: undefined d4200000 db>
I did not work with ARM yet, so I may be completely wrong, but it seems to me we have several problems here. First, netgraph's ng_snd_item() function does something wrong (NULL pointer dereference?) at causes exception. Then, control passes to handle_el1h_sync that does something wrong again (?) causing second exception due to ARM64 platform-specific "data abort" exception. Someone with ARM64 knowledge is needed here.
This issue is ARM-specific as it does not manifest for other platforms. ARM is not FreeBSD Tier1 platform yet, so MFC is not important for this.