Bug 259879 - enabling PF blocks multicast/igmp sendto
Summary: enabling PF blocks multicast/igmp sendto
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.2-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-11-16 12:44 UTC by Johan Ström
Modified: 2021-11-21 07:53 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Johan Ström 2021-11-16 12:44:40 UTC
While trying to get multicast routing to work on my FreeBSD 12.2-based router.
I tried with mrouted, then igmpproxy, them pimd.. The latter two actually shows errors, mrouted did not. In any case, I never got the machine to send out any IGMP join packets, nor pick up any IGMP join's from local nodes.
On another machine, where I ran basic socat test (see below), the machine did never produce any IGMP Join packets. Trying the same on a linux machine worked fine.

At least one other person have had these issues with pimd, but probably not related to pimd: https://github.com/troglobit/pimd/issues/171


After doing tests on clean VMs, I've nailed it down to PF. Having pf just enabled, even with blank rules, seems to block outbound multicast/igmp somehow.

Reproducable:

1. Launch blank VM with FreeBSD 12.2 or 13.0 qcow image in KVM:
2. Prepare:
  
  pkg install pimd truss 
  kldload ip_mroute
 
3. Launch pimd, working with no errors:
  
root@freebsd:~ #  pimd -f
^C 

4. Enable pf (blank, no rules):

 
root@freebsd:~ # pfctl -e
root@freebsd:~ # 

5. Now trying to use pimd, gives failures to send:

root@freebsd:~ # pimd -f
pimd: 12:30:03.170 Sendto to 224.0.0.1 on 172.28.6.15: Permission denied


6. Disable pf again and it works fine again...
7. truss output (from socket creation to sendto failure) with pf enabled (but no rules at all):

socket(PF_INET,SOCK_RAW,IPPROTO_IGMP)            = 4 (0x4)                                                             
setsockopt(4,IPPROTO_IP,IP_HDRINCL,0x7fffffffe6dc,4) = 0 (0x0)                                                         
setsockopt(4,SOL_SOCKET,SO_SNDBUF,0x7fffffffe6bc,4) = 0 (0x0)                                                          
setsockopt(4,SOL_SOCKET,SO_RCVBUF,0x7fffffffe6bc,4) = 0 (0x0)                              
setsockopt(4,IPPROTO_IP,IP_MULTICAST_TTL,0x7fffffffe6df,1) = 0 (0x0)                       
setsockopt(4,IPPROTO_IP,IP_MULTICAST_LOOP,0x7fffffffe6d7,1) = 0 (0x0)                                                  
socket(PF_INET,SOCK_RAW,IPPROTO_PIM)             = 5 (0x5)                                                             
setsockopt(5,IPPROTO_IP,IP_HDRINCL,0x7fffffffe6dc,4) = 0 (0x0)                                                         
setsockopt(5,SOL_SOCKET,SO_SNDBUF,0x7fffffffe6bc,4) = 0 (0x0)                                                          
setsockopt(5,SOL_SOCKET,SO_RCVBUF,0x7fffffffe6bc,4) = 0 (0x0)                                                          
setsockopt(5,IPPROTO_IP,IP_MULTICAST_TTL,0x7fffffffe6df,1) = 0 (0x0)                                                   
setsockopt(5,IPPROTO_IP,IP_MULTICAST_LOOP,0x7fffffffe6d7,1) = 0 (0x0)                                                  
mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34366939136 (0x8006de000)
mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367074304 (0x8006ff000)
socket(PF_ROUTE,SOCK_RAW,0)                      = 6 (0x6)                                                             
fcntl(6,F_SETFL,O_RDONLY|O_NONBLOCK)             = 0 (0x0)                                                             
socket(PF_INET,SOCK_DGRAM,0)                     = 7 (0x7)                                                                                                                                                                                     
ioctl(7,SIOCGIFCONF,0x7fffffffe690)              = 0 (0x0)                                                                                                                                                                                     
ioctl(7,SIOCGIFFLAGS,0x7fffffffe6a0)             = 0 (0x0)                                                                                                                                                                                     
ioctl(7,SIOCGIFNETMASK,0x7fffffffe6a0)           = 0 (0x0)                                                                                                                                                                                     
ioctl(7,SIOCGIFMTU,0x7fffffffe6a0)               = 0 (0x0)                                                                                                                                                                                     
ioctl(7,SIOCGIFFLAGS,0x7fffffffe6a0)             = 0 (0x0)                                                             
open("/usr/local/etc//pimd.conf",O_RDONLY,0666)  = 8 (0x8)                                                             
fstat(8,{ mode=-rw-r--r-- ,inode=321234,size=6435,blksize=32768 }) = 0 (0x0)                                           
mmap(0x0,36864,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367209472 (0x800720000)                                                                                                                                                   
read(8,"# Exmaple configuration file for"...,32768) = 6435 (0x1923)                                                    
read(8,0x8007204c0,32768)                        = 0 (0x0)                                                                                                                                                                                     
close(8)                                         = 0 (0x0)                                                                                                                                                                                     
setsockopt(4,IPPROTO_IP,100,0x7fffffffe6dc,4)    = 0 (0x0)                                                             
setsockopt(4,IPPROTO_IP,107,0x7fffffffe6dc,4)    = 0 (0x0)                                                             
getrandom("\M-2\f\M-M\M-@\M-7\^\ \M-jU\v"...,40,0) = 40 (0x28)                    
mmap(0x0,1104,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367246336 (0x800729000)                            
minherit(0x800729000,1104,INHERIT_ZERO)          = 0 (0x0)                                                             
setsockopt(4,IPPROTO_IP,102,0x7fffffffe698,16)   = 0 (0x0)                                                             
setsockopt(5,IPPROTO_IP,IP_ADD_MEMBERSHIP,0x7fffffffe698,8) = 0 (0x0)                                                  
setsockopt(4,IPPROTO_IP,IP_ADD_MEMBERSHIP,0x7fffffffe698,8) = 0 (0x0)                                                  
setsockopt(4,IPPROTO_IP,IP_ADD_MEMBERSHIP,0x7fffffffe698,8) = 0 (0x0)                                                  
setsockopt(4,IPPROTO_IP,IP_MULTICAST_IF,0x7fffffffe5c8,4) = 0 (0x0)                                                    
setsockopt(4,IPPROTO_IP,IP_MULTICAST_LOOP,0x7fffffffe5c7,1) = 0 (0x0)                         
sendto(4,"F\M-@\0$\0\0\0\0\M^?\^B\0\0\M-,"...,36,0,{ AF_INET 224.0.0.1:0 },16) ERR#13 'Permission denied'                                                                                                                                      




Another test case, with socat:
1. Disable pf on FreeBSD machine (172.28.6.15)
2. Start tcpdump on another machine in same network.
3. Start socat on freebsd machine:

   socat -d -d -u UDP4-RECV:5568,ip-add-membership=239.255.0.100:172.28.6.15 /dev/null

4. Check tcpdump output on another machine, you can see the IGMP Joins

   13:40:29.226382 IP 172.28.6.15 > 224.0.0.22: igmp v3 report, 1 group record(s)

5. Enable pf (blank rules), run socat again. No IGMP traffic whatsoever seen on remote machine.
Comment 1 Kristof Provost freebsd_committer freebsd_triage 2021-11-16 22:01:33 UTC
I've been able to replicate part of this on main. It looks like the IGMP packets are dropped because they have IP header options, which appears to be expected behaviour:

     allow-opts
           By default, IPv4 packets with IP options or IPv6 packets with routing
           extension headers are blocked.  When allow-opts is specified for a
           pass rule, packets that pass the filter based on that rule (last
           matching) do so even if they contain IP options or routing extension
           headers.  For packets that match state, the rule that initially
           created the state is used.  The implicit pass rule that is used when
           a packet does not match any rules does not allow IP options.
Comment 2 Johan Ström 2021-11-16 22:13:41 UTC
Ooooh.. Ok, yes indeed, adding the following rule makes pimd seem to work as expected:


   pass on $intf inet proto igmp allow-opts


I had this without allow-opts before, but it "silently" failed (my "block drop log all" did not log it, iirc).

Thank you for super fast reply! I'm not sure if this was just crap behind the keyboard, or an actual bug. Leaving that for you to decide.
Comment 3 Kristof Provost freebsd_committer freebsd_triage 2021-11-16 22:19:21 UTC
It's non-intuitive behaviour. It surprised me too, but it is documented as working this way and it also works this way in OpenBSD pf so I'm not going to change it.
Comment 4 Johan Ström 2021-11-16 22:24:36 UTC
Sounds sane. Perhaps a note in pf.conf man page about effect on multicast/IGMP could be useful, or in other docs (unless I missed it).

Anyhow, thanks!
Comment 5 Johan Ström 2021-11-17 06:56:02 UTC
I verified that logging is not working as expected (pass rule without allow-opts blocks but does not log):


enabling pf with no rules
launch the socat command that tries to join multicast address.
Pf now blocks igmp (as we now know is expected)

adding a rule "block return log on $if all" and then running the socat again yields log entries in pflog:

root@freebsd:~ # tcpdump -i pflog0 igmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 262144 bytes
06:47:55.756617 IP 172.28.6.15 > igmp.mcast.net: igmp v3 report, 1 group record(s)
06:47:57.756382 IP 172.28.6.15 > igmp.mcast.net: igmp v3 report, 1 group record(s)
06:47:58.556249 IP 172.28.6.15 > igmp.mcast.net: igmp v3 report, 1 group record(s)

Adding a incomplete PF rule "pass on $if inet proto igmp" and starting socat. No igmp traffic out, but nothing in pflog eitiher. And that feels more like a bug?


One gotcha while debugging this: adding 'allow-opts' to the above role and rerunning socat does not actually work immediately , you have to flush/wait for the states to expire. Then it works.
Comment 6 Kristof Provost freebsd_committer freebsd_triage 2021-11-18 00:13:36 UTC
(In reply to Johan Ström from comment #5)
I don't think that's a bug. Log rules are only going to log when they match the traffic. These rules do not match it, that's why it's getting dropped.

That final gotcha is also expected. Traffic keeps matching the rule that created its state, even if the ruleset changes later.
Comment 7 Johan Ström 2021-11-18 06:36:56 UTC
The "block return log on $if all" IS matching and IS logging, as long as there isn't a pass rule for igmp. If I add a pass rule *without allow-opts* it stops logging, even if the pass rule does not pass the traffic:


block return log on vtnet0 all

logs to pflog0

06:30:59.154898 rule 0/0(match): block out on vtnet0: (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 40, options (RA))
    172.28.6.15 > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr 239.255.0.100 to_in, 0 source(s)]


but


block return log on vtnet0 all
pass on vtnet0 inet proto icmp

does not pass traffic (since missing allow-opts on pass rule), but neither does it log it in pflog anymore.
Comment 8 Kristof Provost freebsd_committer freebsd_triage 2021-11-20 23:58:33 UTC
(In reply to Johan Ström from comment #7)
Try adding a log to your pass rule.
Comment 9 Johan Ström 2021-11-21 07:52:43 UTC
TEST CASE:


block return log on $if all


flushing state and starting socat.
Logs on pflog0:

07:50:32.756386 rule 0/0(match): block out on vtnet0: (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 40, options (RA))
    172.28.6.15 > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr 239.255.0.100 to_ex, 0 source(s)]

And nothing on vtnet0.


TEST CASE:


block return log on $if all
pass log on $if inet proto igmp allow-opts

flushing state and starting socat, logs:

07:44:57.756384 rule 2/0(match): pass out on vtnet0: (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 40, options (RA))
    172.28.6.15 > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr 239.255.0.100 to_ex, 0 source(s)]

and expected igmp on vnet0.
pf rule Packet counter is incremented.




Tested:

block return log on $if all
pass log on $if inet proto


flushing state and starting socat, logs

07:46:55.356406 rule 2/8(ip-option): pass out on vtnet0: (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 40, options (RA))
    172.28.6.15 > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr 239.255.0.100 to_ex, 0 source(s)]


Nothing on vtnet0.
pf rule Packet counter is NOT incremented (only Evaluated).



So, from a pflog perspective it seems the rule is matched, but from counter and actual traffic perspective, not matched.
Comment 10 Johan Ström 2021-11-21 07:53:37 UTC
Btw, these latest tests have all been running on a KVM vm with FreeBSD 12.2 image.