While trying to get multicast routing to work on my FreeBSD 12.2-based router. I tried with mrouted, then igmpproxy, them pimd.. The latter two actually shows errors, mrouted did not. In any case, I never got the machine to send out any IGMP join packets, nor pick up any IGMP join's from local nodes. On another machine, where I ran basic socat test (see below), the machine did never produce any IGMP Join packets. Trying the same on a linux machine worked fine. At least one other person have had these issues with pimd, but probably not related to pimd: https://github.com/troglobit/pimd/issues/171 After doing tests on clean VMs, I've nailed it down to PF. Having pf just enabled, even with blank rules, seems to block outbound multicast/igmp somehow. Reproducable: 1. Launch blank VM with FreeBSD 12.2 or 13.0 qcow image in KVM: 2. Prepare: pkg install pimd truss kldload ip_mroute 3. Launch pimd, working with no errors: root@freebsd:~ # pimd -f ^C 4. Enable pf (blank, no rules): root@freebsd:~ # pfctl -e root@freebsd:~ # 5. Now trying to use pimd, gives failures to send: root@freebsd:~ # pimd -f pimd: 12:30:03.170 Sendto to 224.0.0.1 on 172.28.6.15: Permission denied 6. Disable pf again and it works fine again... 7. truss output (from socket creation to sendto failure) with pf enabled (but no rules at all): socket(PF_INET,SOCK_RAW,IPPROTO_IGMP) = 4 (0x4) setsockopt(4,IPPROTO_IP,IP_HDRINCL,0x7fffffffe6dc,4) = 0 (0x0) setsockopt(4,SOL_SOCKET,SO_SNDBUF,0x7fffffffe6bc,4) = 0 (0x0) setsockopt(4,SOL_SOCKET,SO_RCVBUF,0x7fffffffe6bc,4) = 0 (0x0) setsockopt(4,IPPROTO_IP,IP_MULTICAST_TTL,0x7fffffffe6df,1) = 0 (0x0) setsockopt(4,IPPROTO_IP,IP_MULTICAST_LOOP,0x7fffffffe6d7,1) = 0 (0x0) socket(PF_INET,SOCK_RAW,IPPROTO_PIM) = 5 (0x5) setsockopt(5,IPPROTO_IP,IP_HDRINCL,0x7fffffffe6dc,4) = 0 (0x0) setsockopt(5,SOL_SOCKET,SO_SNDBUF,0x7fffffffe6bc,4) = 0 (0x0) setsockopt(5,SOL_SOCKET,SO_RCVBUF,0x7fffffffe6bc,4) = 0 (0x0) setsockopt(5,IPPROTO_IP,IP_MULTICAST_TTL,0x7fffffffe6df,1) = 0 (0x0) setsockopt(5,IPPROTO_IP,IP_MULTICAST_LOOP,0x7fffffffe6d7,1) = 0 (0x0) mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34366939136 (0x8006de000) mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367074304 (0x8006ff000) socket(PF_ROUTE,SOCK_RAW,0) = 6 (0x6) fcntl(6,F_SETFL,O_RDONLY|O_NONBLOCK) = 0 (0x0) socket(PF_INET,SOCK_DGRAM,0) = 7 (0x7) ioctl(7,SIOCGIFCONF,0x7fffffffe690) = 0 (0x0) ioctl(7,SIOCGIFFLAGS,0x7fffffffe6a0) = 0 (0x0) ioctl(7,SIOCGIFNETMASK,0x7fffffffe6a0) = 0 (0x0) ioctl(7,SIOCGIFMTU,0x7fffffffe6a0) = 0 (0x0) ioctl(7,SIOCGIFFLAGS,0x7fffffffe6a0) = 0 (0x0) open("/usr/local/etc//pimd.conf",O_RDONLY,0666) = 8 (0x8) fstat(8,{ mode=-rw-r--r-- ,inode=321234,size=6435,blksize=32768 }) = 0 (0x0) mmap(0x0,36864,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367209472 (0x800720000) read(8,"# Exmaple configuration file for"...,32768) = 6435 (0x1923) read(8,0x8007204c0,32768) = 0 (0x0) close(8) = 0 (0x0) setsockopt(4,IPPROTO_IP,100,0x7fffffffe6dc,4) = 0 (0x0) setsockopt(4,IPPROTO_IP,107,0x7fffffffe6dc,4) = 0 (0x0) getrandom("\M-2\f\M-M\M-@\M-7\^\ \M-jU\v"...,40,0) = 40 (0x28) mmap(0x0,1104,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367246336 (0x800729000) minherit(0x800729000,1104,INHERIT_ZERO) = 0 (0x0) setsockopt(4,IPPROTO_IP,102,0x7fffffffe698,16) = 0 (0x0) setsockopt(5,IPPROTO_IP,IP_ADD_MEMBERSHIP,0x7fffffffe698,8) = 0 (0x0) setsockopt(4,IPPROTO_IP,IP_ADD_MEMBERSHIP,0x7fffffffe698,8) = 0 (0x0) setsockopt(4,IPPROTO_IP,IP_ADD_MEMBERSHIP,0x7fffffffe698,8) = 0 (0x0) setsockopt(4,IPPROTO_IP,IP_MULTICAST_IF,0x7fffffffe5c8,4) = 0 (0x0) setsockopt(4,IPPROTO_IP,IP_MULTICAST_LOOP,0x7fffffffe5c7,1) = 0 (0x0) sendto(4,"F\M-@\0$\0\0\0\0\M^?\^B\0\0\M-,"...,36,0,{ AF_INET 224.0.0.1:0 },16) ERR#13 'Permission denied' Another test case, with socat: 1. Disable pf on FreeBSD machine (172.28.6.15) 2. Start tcpdump on another machine in same network. 3. Start socat on freebsd machine: socat -d -d -u UDP4-RECV:5568,ip-add-membership=239.255.0.100:172.28.6.15 /dev/null 4. Check tcpdump output on another machine, you can see the IGMP Joins 13:40:29.226382 IP 172.28.6.15 > 224.0.0.22: igmp v3 report, 1 group record(s) 5. Enable pf (blank rules), run socat again. No IGMP traffic whatsoever seen on remote machine.
I've been able to replicate part of this on main. It looks like the IGMP packets are dropped because they have IP header options, which appears to be expected behaviour: allow-opts By default, IPv4 packets with IP options or IPv6 packets with routing extension headers are blocked. When allow-opts is specified for a pass rule, packets that pass the filter based on that rule (last matching) do so even if they contain IP options or routing extension headers. For packets that match state, the rule that initially created the state is used. The implicit pass rule that is used when a packet does not match any rules does not allow IP options.
Ooooh.. Ok, yes indeed, adding the following rule makes pimd seem to work as expected: pass on $intf inet proto igmp allow-opts I had this without allow-opts before, but it "silently" failed (my "block drop log all" did not log it, iirc). Thank you for super fast reply! I'm not sure if this was just crap behind the keyboard, or an actual bug. Leaving that for you to decide.
It's non-intuitive behaviour. It surprised me too, but it is documented as working this way and it also works this way in OpenBSD pf so I'm not going to change it.
Sounds sane. Perhaps a note in pf.conf man page about effect on multicast/IGMP could be useful, or in other docs (unless I missed it). Anyhow, thanks!
I verified that logging is not working as expected (pass rule without allow-opts blocks but does not log): enabling pf with no rules launch the socat command that tries to join multicast address. Pf now blocks igmp (as we now know is expected) adding a rule "block return log on $if all" and then running the socat again yields log entries in pflog: root@freebsd:~ # tcpdump -i pflog0 igmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 262144 bytes 06:47:55.756617 IP 172.28.6.15 > igmp.mcast.net: igmp v3 report, 1 group record(s) 06:47:57.756382 IP 172.28.6.15 > igmp.mcast.net: igmp v3 report, 1 group record(s) 06:47:58.556249 IP 172.28.6.15 > igmp.mcast.net: igmp v3 report, 1 group record(s) Adding a incomplete PF rule "pass on $if inet proto igmp" and starting socat. No igmp traffic out, but nothing in pflog eitiher. And that feels more like a bug? One gotcha while debugging this: adding 'allow-opts' to the above role and rerunning socat does not actually work immediately , you have to flush/wait for the states to expire. Then it works.
(In reply to Johan Ström from comment #5) I don't think that's a bug. Log rules are only going to log when they match the traffic. These rules do not match it, that's why it's getting dropped. That final gotcha is also expected. Traffic keeps matching the rule that created its state, even if the ruleset changes later.
The "block return log on $if all" IS matching and IS logging, as long as there isn't a pass rule for igmp. If I add a pass rule *without allow-opts* it stops logging, even if the pass rule does not pass the traffic: block return log on vtnet0 all logs to pflog0 06:30:59.154898 rule 0/0(match): block out on vtnet0: (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 40, options (RA)) 172.28.6.15 > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr 239.255.0.100 to_in, 0 source(s)] but block return log on vtnet0 all pass on vtnet0 inet proto icmp does not pass traffic (since missing allow-opts on pass rule), but neither does it log it in pflog anymore.
(In reply to Johan Ström from comment #7) Try adding a log to your pass rule.
TEST CASE: block return log on $if all flushing state and starting socat. Logs on pflog0: 07:50:32.756386 rule 0/0(match): block out on vtnet0: (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 40, options (RA)) 172.28.6.15 > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr 239.255.0.100 to_ex, 0 source(s)] And nothing on vtnet0. TEST CASE: block return log on $if all pass log on $if inet proto igmp allow-opts flushing state and starting socat, logs: 07:44:57.756384 rule 2/0(match): pass out on vtnet0: (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 40, options (RA)) 172.28.6.15 > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr 239.255.0.100 to_ex, 0 source(s)] and expected igmp on vnet0. pf rule Packet counter is incremented. Tested: block return log on $if all pass log on $if inet proto flushing state and starting socat, logs 07:46:55.356406 rule 2/8(ip-option): pass out on vtnet0: (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto IGMP (2), length 40, options (RA)) 172.28.6.15 > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr 239.255.0.100 to_ex, 0 source(s)] Nothing on vtnet0. pf rule Packet counter is NOT incremented (only Evaluated). So, from a pflog perspective it seems the rule is matched, but from counter and actual traffic perspective, not matched.
Btw, these latest tests have all been running on a KVM vm with FreeBSD 12.2 image.