Bug 244002 - Kernel panic while trying to read multicast stream
Summary: Kernel panic while trying to read multicast stream
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.1-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Hans Petter Selasky
URL:
Keywords: panic
Depends on:
Blocks:
 
Reported: 2020-02-09 13:32 UTC by Vladislav V. Prodan
Modified: 2020-02-25 02:59 UTC (History)
4 users (show)

See Also:
koobs: mfc-stable12+
koobs: mfc-stable11+


Attachments
kgdb.out (3.37 KB, text/plain)
2020-02-09 13:32 UTC, Vladislav V. Prodan
no flags Details
My kernel config (24.66 KB, text/plain)
2020-02-10 09:23 UTC, Vladislav V. Prodan
no flags Details
Debug patch (806 bytes, patch)
2020-02-10 09:30 UTC, Hans Petter Selasky
no flags Details | Diff
Patch to solve problem (1.19 KB, patch)
2020-02-10 11:05 UTC, Hans Petter Selasky
no flags Details | Diff
Patch to solve problem (1.28 KB, patch)
2020-02-12 09:11 UTC, Hans Petter Selasky
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Vladislav V. Prodan 2020-02-09 13:32:29 UTC
Created attachment 211509 [details]
kgdb.out

# uname -a
FreeBSD mary-teresa.XXXXX.ua 12.1-STABLE FreeBSD 12.1-STABLE r356112 SUPPORT-12-1-2  amd64

Kind commands
vlc udp: //@238.39.40.4: 5000
or
ffmpeg -i udp: //@238.39.40.4: 5000
call kernel panic

# kldstat
Id Refs Address                Size Name
 1   19 0xffffffff80200000  25bea10 kernel
 2    1 0xffffffff827c0000     2338 cc_htcp.ko
 3    1 0xffffffff827c3000   3a9738 zfs.ko
 4    2 0xffffffff82b6d000     a3e8 opensolaris.ko
 5    1 0xffffffff82d21000     1880 uhid.ko
 6    1 0xffffffff82d23000     1a80 wmt.ko
Comment 1 Vladislav V. Prodan 2020-02-09 14:04:19 UTC
vlc udp://@238.39.40.4:5000
or
ffmpeg -i udp://@238.39.40.4:5000
Comment 2 Hans Petter Selasky freebsd_committer 2020-02-10 08:32:43 UTC
What is the output from "ifconfig" before running this test?

--HPS
Comment 3 Vladislav V. Prodan 2020-02-10 08:43:52 UTC
(In reply to Hans Petter Selasky from comment #2)

Multicast is available through interface re0 and fxp0


# ifconfig
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 00:e0:4d:a0:f8:46
        inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
        inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
        inet 10.0.0.19 netmask 0xffffff00 broadcast 10.0.0.255
        inet 10.0.0.22 netmask 0xffffff00 broadcast 10.0.0.255
        inet 10.0.0.21 netmask 0xffffff00 broadcast 10.0.0.255
        inet6 fe80::2e0:4dff:fea0:f846%re0 prefixlen 64 scopeid 0x1
        inet6 2001:XXX:XX:XX::5 prefixlen 64
        inet6 2001:XXX:XX::5 prefixlen 48
        inet6 2001:XXX:XX:XX::2 prefixlen 64
        inet6 2001:XXX:XX:XX::7 prefixlen 64
        inet6 2001:XXX:XX:XX::8 prefixlen 64
        inet6 2001:XXX:XX:XX::21:1 prefixlen 64
        inet6 2001:XXX:XX:XX:8693:2a27:a219:6e05 prefixlen 64
        inet6 2001:XXX:XX:XX:80ca:f097:df2e:e343 prefixlen 64
        inet6 2001:XXX:XX:XX:dca9:45be:ca8:f598 prefixlen 64
        inet6 2001:XXX:XX:XX:4dc2:29f4:246a:3954 prefixlen 64
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:0e:0c:62:f9:37
        inet6 fe80::20e:cff:fe62:f937%fxp0 prefixlen 64 scopeid 0x2
        inet XX.XX.XX.236 netmask 0xffffff00 broadcast XX.XX.XX.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 00:07:e9:89:cf:c1
        inet6 fe80::207:e9ff:fe89:cfc1%em0 prefixlen 64 scopeid 0x3
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
        groups: enc
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet6 2001:XXX:XX:XX:2b0:3a1a:6258:1a08 prefixlen 64
        inet6 2001:XXX:XX:XX:70f0:9b24:5e51:ccf9 prefixlen 64
        inet6 2001:XXX:XX:XX:8f4:372d:e9b8:92f6 prefixlen 64
        inet6 2001:XXX:XX:XX:7eda:9ead:bbcb:c82b prefixlen 64
        inet6 2001:XXX:XX:XX:f19c:896b:4ff0:2203 prefixlen 64
        inet6 2001:XXX:XX:XX:43a:3a80:49bd:af9a prefixlen 64
        inet6 2001:XXX:XX:XX:12f5:89a1:e201:1b prefixlen 64
        inet6 2001:XXX:XX:XX:40cc:c1a0:b172:b45a prefixlen 64
        inet6 2001:XXX:XX:XX:bc6d:b65e:2896:7f18 prefixlen 64
        inet6 2001:XXX:XX:XX:755c:414f:da52:3bc1 prefixlen 64
        inet6 2001:XXX:XX:XX:3a85:44a4:f9b4:23ca prefixlen 64
        inet6 ::10.0.0.21 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
        inet 127.0.1.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33672
        groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
        syncpeer: 0.0.0.0 maxupd: 128 defer: off
        groups: pfsync
gif2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1480
        description: He.NET
        options=80000<LINKSTATE>
        tunnel inet XX.XX.XX.236 --> 216.66.80.90
        inet6 fe80::2e0:4dff:fea0:f846%gif2 prefixlen 64 scopeid 0x8
        inet6 2001:XXX:XX:XX::2 --> 2001:XXX:XX:XX::1 prefixlen 128
        groups: gif
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ng0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1494
        inet 10.0.1.1 --> 10.0.2.48 netmask 0xffffffff
        inet6 fe80::2e0:4dff:fea0:f846%ng0 prefixlen 64 scopeid 0x9
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Comment 4 Hans Petter Selasky freebsd_committer 2020-02-10 08:57:13 UTC
It looks like m->m_pkthdr.rcvif == NULL.

Is this kernel built with INVARIANTS enabled?
Comment 5 Vladislav V. Prodan 2020-02-10 09:17:58 UTC
(In reply to Hans Petter Selasky from comment #4)
No.

The INVARIANTS and INVARIANT_SUPPORT options are not included in my kernel.
Comment 6 Vladislav V. Prodan 2020-02-10 09:23:30 UTC
Created attachment 211529 [details]
My kernel config
Comment 7 Hans Petter Selasky freebsd_committer 2020-02-10 09:30:29 UTC
Created attachment 211530 [details]
Debug patch

Can you try the attached debug patch, and send the print it generates in dmesg, if any?
Comment 8 Hans Petter Selasky freebsd_committer 2020-02-10 09:31:55 UTC
Apply patch like this:

cd /usr/src
cat netisr.diff | patch -p1

make buildkernel
make installkernel
reboot
Comment 9 Vladislav V. Prodan 2020-02-10 10:12:04 UTC
I applied this patch and am building my kernel, not GENERIC.
Comment 10 Vladislav V. Prodan 2020-02-10 10:35:16 UTC
NULL RCVIF AT:
KDB: stack backtrace:
#0 0xffffffff80cc68b7 at kdb_backtrace+0x67
#1 0xffffffff80dce6b5 at netisr_queue_src+0x95
#2 0xffffffff80dce989 at netisr_dispatch_src+0x69
#3 0xffffffff80e7ef19 at igmp_fasttimo+0x9e9
#4 0xffffffff80d06904 at pffasttimo+0x54
#5 0xffffffff80c9461e at softclock_call_cc+0x13e
#6 0xffffffff80c94ad9 at softclock+0x79
#7 0xffffffff80c3d6d4 at ithread_loop+0x1d4
#8 0xffffffff80c3a5b2 at fork_exit+0x82
#9 0xffffffff8123bfce at fork_trampoline+0xe
NULL RCVIF AT:
KDB: stack backtrace:
#0 0xffffffff80cc68b7 at kdb_backtrace+0x67
#1 0xffffffff80dce6b5 at netisr_queue_src+0x95
#2 0xffffffff80dce989 at netisr_dispatch_src+0x69
#3 0xffffffff80e7ef19 at igmp_fasttimo+0x9e9
#4 0xffffffff80d06904 at pffasttimo+0x54
#5 0xffffffff80c9461e at softclock_call_cc+0x13e
#6 0xffffffff80c94ad9 at softclock+0x79
#7 0xffffffff80c3d6d4 at ithread_loop+0x1d4
#8 0xffffffff80c3a5b2 at fork_exit+0x82
#9 0xffffffff8123bfce at fork_trampoline+0xe
Comment 11 Vladislav V. Prodan 2020-02-10 10:38:09 UTC
NULL RCVIF AT:
KDB: stack backtrace:
#0 0xffffffff80cc68b7 at kdb_backtrace+0x67
#1 0xffffffff80dce6b5 at netisr_queue_src+0x95
#2 0xffffffff80dce989 at netisr_dispatch_src+0x69
#3 0xffffffff80e80a0c at igmp_v1v2_queue_report+0x15c
#4 0xffffffff80e7f4e8 at igmp_change_state+0x328
#5 0xffffffff80e8c7d0 at in_joingroup_locked+0x180
#6 0xffffffff80e8f137 at inp_setmoptions+0x12a7
#7 0xffffffff80ea45af at ip_ctloutput+0x12f
#8 0xffffffff80f48b1f at udp_ctloutput+0x13f
#9 0xffffffff80d15b56 at sosetopt+0xe6
#10 0xffffffff80d1b200 at kern_setsockopt+0xb0
#11 0xffffffff80d1b144 at sys_setsockopt+0x24
#12 0xffffffff81262df2 at amd64_syscall+0x362
#13 0xffffffff8123b8b0 at fast_syscall_common+0x101
Comment 12 Hans Petter Selasky freebsd_committer 2020-02-10 11:04:53 UTC
Hi Bjorn,

IGMP stores the VNET in "m->m_pkthdr.PH_loc.ptr", but the netisr code expects m->m_pkthdr.rcvif to be valid!

--HPS
Comment 13 Hans Petter Selasky freebsd_committer 2020-02-10 11:05:35 UTC
Created attachment 211532 [details]
Patch to solve problem

Hi,

Can you try this patch for now?

--HPS
Comment 14 Vladislav V. Prodan 2020-02-10 11:12:18 UTC
(In reply to Hans Petter Selasky from comment #13)

I applied this second patch and am building my kernel.
Comment 15 Vladislav V. Prodan 2020-02-10 12:06:39 UTC
Thanks.
all works!

#ffmpeg -i udp://@239.2.147.238:1234
...

Input #0, mpegts, from 'udp://@239.2.147.238:1234':
  Duration: N/A, start: 10866.865244, bitrate: N/A
  Program 1
    Metadata:
      service_name    : Service01
      service_provider: R48
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1280x720 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 124 kb/s
At least one output file must be specified
Comment 16 Bjoern A. Zeeb freebsd_committer 2020-02-12 09:01:29 UTC
Comment on attachment 211532 [details]
Patch to solve problem

Why do we only need to do this in the VIMAGE case and not generally?  Is no one else ever looking at the rcvif otherwise?

Also the KASSERT could at least have a "%s: .. %p", __func__, m  as does the KASSERT before.
Comment 17 Hans Petter Selasky freebsd_committer 2020-02-12 09:11:21 UTC
Created attachment 211583 [details]
Patch to solve problem

Update patch according to bz@ suggestions.

Basically the rcvif is only used when VIMAGE is active. But it doesn't hurt to set it always.

Updated the assert to use VNET_ASSERT().
Comment 18 Vladislav V. Prodan 2020-02-13 17:39:44 UTC
(In reply to Hans Petter Selasky from comment #17)

Does this patch need to be tested too?
Comment 19 Hans Petter Selasky freebsd_committer 2020-02-13 20:45:46 UTC
Feel free to test it while bz@ is doing review.

--HPS
Comment 20 Bjoern A. Zeeb freebsd_committer 2020-02-17 09:04:22 UTC
Comment on attachment 211583 [details]
Patch to solve problem

Looks ok to me (without further context).
Comment 21 commit-hook freebsd_committer 2020-02-17 09:47:18 UTC
A commit references this bug:

Author: hselasky
Date: Mon Feb 17 09:46:33 UTC 2020
New revision: 358013
URL: https://svnweb.freebsd.org/changeset/base/358013

Log:
  Fix kernel panic while trying to read multicast stream.

  When VIMAGE is enabled make sure the "m_pkthdr.rcvif" pointer is set
  for all mbufs being input by the IGMP/MLD6 code. Else there will be a
  NULL-pointer dereference in the netisr code when trying to set the
  VNET based on the incoming mbuf. Add an assert to catch this when
  queueing mbufs on a netisr to make debugging of similar cases easier.

  Found by:	Vladislav V. Prodan
  PR:		244002
  Reviewed by:	bz@
  MFC after:	1 week
  Sponsored by:	Mellanox Technologies

Changes:
  head/sys/net/netisr.c
  head/sys/netinet/igmp.c
  head/sys/netinet6/mld6.c
Comment 22 Hans Petter Selasky freebsd_committer 2020-02-17 09:47:50 UTC
Will be MFC'ed next week.

Thank you for the bug-report!
Comment 23 commit-hook freebsd_committer 2020-02-24 09:58:50 UTC
A commit references this bug:

Author: hselasky
Date: Mon Feb 24 09:57:48 UTC 2020
New revision: 358273
URL: https://svnweb.freebsd.org/changeset/base/358273

Log:
  MFC r358013:
  Fix kernel panic while trying to read multicast stream.

  When VIMAGE is enabled make sure the "m_pkthdr.rcvif" pointer is set
  for all mbufs being input by the IGMP/MLD6 code. Else there will be a
  NULL-pointer dereference in the netisr code when trying to set the
  VNET based on the incoming mbuf. Add an assert to catch this when
  queueing mbufs on a netisr to make debugging of similar cases easier.

  Found by:	Vladislav V. Prodan
  PR:		244002
  Reviewed by:	bz@
  Sponsored by:	Mellanox Technologies

Changes:
_U  stable/12/
  stable/12/sys/net/netisr.c
  stable/12/sys/netinet/igmp.c
  stable/12/sys/netinet6/mld6.c
Comment 24 commit-hook freebsd_committer 2020-02-24 09:59:52 UTC
A commit references this bug:

Author: hselasky
Date: Mon Feb 24 09:58:58 UTC 2020
New revision: 358274
URL: https://svnweb.freebsd.org/changeset/base/358274

Log:
  MFC r358013:
  Fix kernel panic while trying to read multicast stream.

  When VIMAGE is enabled make sure the "m_pkthdr.rcvif" pointer is set
  for all mbufs being input by the IGMP/MLD6 code. Else there will be a
  NULL-pointer dereference in the netisr code when trying to set the
  VNET based on the incoming mbuf. Add an assert to catch this when
  queueing mbufs on a netisr to make debugging of similar cases easier.

  Found by:	Vladislav V. Prodan
  PR:		244002
  Reviewed by:	bz@
  Sponsored by:	Mellanox Technologies

Changes:
_U  stable/11/
  stable/11/sys/net/netisr.c
  stable/11/sys/netinet/igmp.c
  stable/11/sys/netinet6/mld6.c