Created attachment 146665 [details] MulticastTest.java .
Previously, I reported some error messages reported by Jenkins on startup: http://lists.freebsd.org/pipermail/freebsd-java/2014-February/010593.html I looked at the Jenkins source code, and isolated the problem. I wrote this simple testcase (see attached MulticastTest.java), which I am also including inline: ================================================================================ /* * To build this test, * (1) Make sure that the OpenJDK is installed from ports: * * pkg install openjdk * * (2) Rename this file to: MulticastTest.java * (3) Build it: * * javac MulticastTest.java * * (4) Run it: * * java MulticastTest.java * */ import java.net.InetAddress; import java.net.MulticastSocket; class MulticastTest { public static void main(String[] args) { try { int PORT = Integer.getInteger("hudson.udp",33848); InetAddress MULTICAST = InetAddress.getByAddress(new byte[]{(byte)239, (byte)77, (byte)124, (byte)213}); MulticastSocket mcs = new MulticastSocket(PORT); mcs.joinGroup(MULTICAST); } catch (Exception e) { e.printStackTrace(); System.exit(-1); } } } ================================================================================ If I run this testcase, I get the same error as what I reported earlier with Jenkins: java.net.SocketException: Invalid argument at java.net.PlainDatagramSocketImpl.join(Native Method) at java.net.AbstractPlainDatagramSocketImpl.join(AbstractPlainDatagramSocketImpl.java:178) at java.net.MulticastSocket.joinGroup(MulticastSocket.java:319) at MulticastTest.main(MulticastTest.java:31) If I run: ktrace java MulticastTest I see that the error here: 13253 java CALL setsockopt(0x4,0x29,0x1b,0x7fffffbfd7dc,0x4) 13253 java RET setsockopt 0 13253 java CALL setsockopt(0x4,SOL_SOCKET,SO_BROADCAST,0x7fffffbfd7d8,0x4) 13253 java RET setsockopt 0 13253 java CALL getsockopt(0x4,SOL_SOCKET,SO_TYPE,0x7fffffbfd77c,0x7fffffbfd778) 13253 java RET getsockopt 0 13253 java CALL setsockopt(0x4,SOL_SOCKET,SO_REUSEPORT,0x7fffffbfd7e0,0x4) 13253 java RET setsockopt 0 13253 java CALL setsockopt(0x4,SOL_SOCKET,SO_REUSEADDR,0x7fffffbfd7e0,0x4) 13253 java RET setsockopt 0 13253 java CALL bind(0x4,0x7fffffbfd7a8,0x1c) 13253 java STRU struct sockaddr { AF_INET6, [::]:33848 } 13253 java RET bind 0 13253 java CALL setsockopt(0x4,0x29,0x9,0x7fffffbfd7f4,0x4) 13253 java RET setsockopt 0 13253 java CALL getsockopt(0x4,0x29,0x9,0x7fffffbfd8ac,0x7fffffbfd864) 13253 java RET getsockopt 0 13253 java CALL setsockopt(0x4,0x29,0xc,0x7fffffbfd8c0,0x14) 13253 java RET setsockopt -1 errno 22 Invalid argument This looks like a bug in the FreeBSD networking code for multicast, or a bug in the FreeBSD code in the OpenJDK. This Java code works under Linux, Solaris, Windows, etc., so it would be good to fix this problem on FreeBSD. Can a networking person help me with this?
I did some more analysis and found that this code: 13253 java CALL bind(0x4,0x7fffffbfd7a8,0x1c) 13253 java STRU struct sockaddr { AF_INET6, [::]:33848 } 13253 java RET bind 0 13253 java CALL setsockopt(0x4,0x29,0x9,0x7fffffbfd7f4,0x4) 13253 java RET setsockopt 0 13253 java CALL getsockopt(0x4,0x29,0x9,0x7fffffbfd8ac,0x7fffffbfd864) 13253 java RET getsockopt 0 13253 java CALL setsockopt(0x4,0x29,0xc,0x7fffffbfd8c0,0x14) 13253 java RET setsockopt -1 errno 22 Invalid argument is happening inside the mcast_join_leave() function inside the JDK here: http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/src/solaris/native/java/net/PlainDatagramSocketImpl.c From looking at the FreeBSD header files, IPPROTO_IPV6 = 41 (0x29) IPV6_MULTICAST_IF = IPV6_JOIN_GROUP = 12 (0xc)_ inside the mcast_join_leave() function what is basically happening is: setsockopt(0x4, IPPROTO_IPV6, IPV6_MULTICAST_IF, ...) getsockopt(0x4, IPPROTO_IPV6, IPV6_MULTICAST_IF, ...) setsockopt(0x4, IPPROTO_IPV6, IPV6_JOIN_GROUP, ...) the second setsockopt() is returning EINVAL
I ran the same test program under truss with: truss java MulticastTest and found this: socket(PF_INET6,SOCK_DGRAM,0) = 4 (0x4) setsockopt(0x4,0x29,0x1b,0x7fffffbfd7dc,0x4,0x0) = 0 (0x0) setsockopt(0x4,0xffff,0x20,0x7fffffbfd7d8,0x4,0x0) = 0 (0x0) getsockopt(0x4,0xffff,0x1008,0x7fffffbfd77c,0x7fffffbfd778,0x83f5292d8) = 0 (0x0) setsockopt(0x4,0xffff,0x200,0x7fffffbfd7e0,0x4,0x83f5292d8) = 0 (0x0) setsockopt(0x4,0xffff,0x4,0x7fffffbfd7e0,0x4,0x83f5292d8) = 0 (0x0) bind(4,{ AF_INET6 [108c:bd00:800:0:4b41:9700:100:0]:36096 },28) = 0 (0x0) setsockopt(0x4,0x29,0x9,0x7fffffbfd7f4,0x4,0x83f54efd0) = 0 (0x0) getsockopt(0x4,0x29,0x9,0x7fffffbfd8ac,0x7fffffbfd864,0x83f54c798) = 0 (0x0) setsockopt(0x4,0x29,0xc,0x7fffffbfd8c0,0x14,0x83f54c798) ERR#22 'Invalid argument'
I tracked this down some more. Inside the JDK, there is this code in http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/src/solaris/native/java/net/PlainDatagramSocketImpl.c =============================================================================== /* * IPv6 join. If it's an IPv4 multicast group then we use an IPv4-mapped * address. */ #ifdef AF_INET6 { struct ipv6_mreq mname6; jbyteArray ipaddress; jbyte caddr[16]; jint family; jint address; family = (*env)->GetIntField(env, iaObj, ia_familyID) == IPv4? AF_INET : AF_INET6; if (family == AF_INET) { /* will convert to IPv4-mapped address */ memset((char *) caddr, 0, 16); address = (*env)->GetIntField(env, iaObj, ia_addressID); caddr[10] = 0xff; caddr[11] = 0xff; caddr[12] = ((address >> 24) & 0xff); caddr[13] = ((address >> 16) & 0xff); caddr[14] = ((address >> 8) & 0xff); caddr[15] = (address & 0xff); =============================================================================== I can confirm that the address created by this code looks something like: 0 0 0 0 0 0 0 0 0 0 ff ff ef 4d 7c d5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 In FreeBSD, in src/sys/netinet6/in6_mcast.c inside in6p_join_group(), there is this: if (!IN6_IS_ADDR_MULTICAST(&gsa->sin6.sin6_addr)) return (EINVAL); Since IN6_IS_ADDR_MULTICAST() only checks if the first octet is 0xff, that is what is returning the EINVAL. So the JDK is creating an IPV4 multicast address mapped inside an IPV6 address. The FreeBSD kernel code is rejecting this as a valid IPV6 multicast address. I'm not sure if it is better to fix this in the kernel or the JDK.
Don't you really want an IPv4 socket here anyway? It seems rather convoluted to create an IPv6 socket so you can listen for IPv4 multicast. My guess is that the in6_mcast code doesn't handle IPv4 multicast groups, but bms@ might know.
I'm looking at this from the perspective that 3rd party multicast code written in Java "just works" under Linux and Solaris, but fails under FreeBSD. Sure, using an IPv4 socket from the beginning would be the way to go, but now that will require pushing patches upstream to all Java software using multicast, in order to accomodate FreeBSD. It's possible, but not practical. It would be nice if the FreeBSD IPv6 stack could be modified to deal with IPv4-mapped multicast addresses for IPv6, similar to how Solaris does it. See comment from Andrey, http://lists.freebsd.org/pipermail/freebsd-net/2014-September/039686.html If that could be made to work, then no changes to upstream code would be required.
In src/sys/netinet6, I see that there is usage of a IN6_IS_ADDR_V4MAPPED() macro in other places in the code, like in udp6 and sctp6, so V4 mapped addresses are supported for other things.
It seems to be a non-trivial amount of work. From bms@ on IRC: <quote> There's no hard and fast reasons why it couldn't be done. The code as it stands will reject that as being an API mixup (you want v4 memberships, use the v4 APIs). The tension points are the Layer 4 ingress filtering for SSM, and actually calling Layer 2 in the right way. The nasty thing about IP6-mapped is that you need to track the memberships in v6 terms, but hand-off all the work to the v4 routines to do the right thing. The easiest way to go about doing it is to deal with the ASM case first, and just punch a hole in ingress filtering if someone tries to use SSM (which is what the stack has to do anyway). I'm not going to stick around to see what happens, though. ;-) </quote> I think what this means is that in_mcast6.c is very much tied to doing MLDv6 (or whatever the v6 equivalent of IGMP is), but for these groups, you need to be somehow calling into in_mcast.c to do IGMP instead. However, I'm not sure at what level in_mcast6.c needs to call into in_mcast.c myself. I do think Bruce is your best bet for someone to ask.
Not a solution, but does it work as a workaround to add this option on the commandline to java? -Djava.net.preferIPv4Stack=true
Ronald, Thanks for the tip. java -Djava.net.preferIPv4Stack=true MulticastTest seems to work around the problem. I would still like to see FreeBSD fixed so that this workaround is not required. Even though it is a lot of work, I would like to see Java on FreeBSD behave out of the box, "just like Linux", without the FreeBSD community needing to push lots of patches upstream to different Java software authors. I want there to be less motivation for people to migrate from FreeBSD to Linux if they are deploying Java applications. That's why I've spent the time to analyze the problem and reporting my findings in this bug report. I find this audit trail is quite interesting. :)
-------- Forwarded Message -------- Subject: Re: [Bug 193246] Bug in IPv6 multicast join(), uncovered by Jenkins Date: Fri, 05 Sep 2014 03:05:14 +0100 From: Bruce Simpson <bms@fastmail.net> To: bugzilla-noreply@freebsd.org On 04/09/2014 19:16, bugzilla-noreply@freebsd.org wrote: > I think what this means is that in_mcast6.c is very much tied to doing MLDv6 > (or whatever the v6 equivalent of IGMP is), but for these groups, you need to > be somehow calling into in_mcast.c to do IGMP instead. However, I'm not sure > at what level in_mcast6.c needs to call into in_mcast.c myself. I do think > Bruce is your best bet for someone to ask. > Unfortunately I'm fully booked. But yes, that is a good summary. If Craig (or someone else) is willing to volunteer to support v4-mapped addresses: one approach would be to extend in6_mship{} to include them. Pushing state down to IGMP will need to be added as a special case. To keep it simple, assume that only the legacy any-source multicast (ASM) model will be supported, i.e. listeners will not specify source filters. Looking at the JDK source, it appears they used to handle the v4/v6 swizzle themselves because of limitations in Linux 2.4/2.6. In other words, we do not support RFC 3493 Sec 3.7 for multicast groups at the moment. A more appropriate errno value to return is EPROTONOOPT. [Interestingly, Sec 5.2 discusses IPv6 multicast options, but does not contain any reference to IPv4 at all.] There now follows a late-night writeup of the rationale behind this code -- and this is as concise as I can make it, I'm afraid. in[6]_mcast.c is split into a top half (i.e. per-socket state) and a bottom half (i.e. stack-wide state, and the IGMPv1/2/3 and MLDv1/2 wire protocols). IPv6 mcast options are processed separately from IPv4. Both implement a transaction scheme to protect global membership state (in the bottom half) from failures at the granularity of a single socket (or PCB) in the top half. Why all the complexity? Well, this is to support source-specific multicast (SSM, aka "inclusive mode"). To cut a long story short: as the size of an internetwork increases, it gets more difficult for routers to track the state of multicast listeners, unless they are aware of where the traffic originates from. The book "Interdomain Multicast Routing" by Brian M. Edwards discusses this in lurid detail. So, SSM was introduced to support inter-domain multicast. In this model, joining a multicast group is no longer a simple matter of registering for a channel -- you must also specify the sources you are interested in. However, the majority of multicast applications are built on the older model: i.e. they do not cross more than one IP network hop, and do not specify sources ("any-source multicast", aka ASM). The network stack must be able to cope with both of these uses. It does so by representing the legacy ASM scheme as "exclusive mode". The RFC 3678 APIs also have the advantage that the application can block unwanted senders, even if ASM is in use. [The main API it specifies, setsourcefilter(), does not explicitly mandate v4-mapped support.] So, in the bottom half of mcast, each group has a shared RB-tree of listener state. This is created from the set union of the filter state on each subscribed socket. If there are no filters for a particular group, i.e. all of the sockets/PCBs in the system are in "exclusive" mode and have no filters, then of course the RB-tree for that group will be empty. Otherwise, if there is a mix of "exclusive" and "inclusive" mode listeners, the tree will need to be recomputed. The shared tree is then used to fill out the IGMP/MLD messages sent to on-link routers to request group subscription. It is also used to filter input traffic from within the relevant transports (e.g. UDP, raw sockets). Previously, this filtering required that the network layer take a socket-layer lock. In closing: this isn't just a simple matter of adding a few defines. The v6 code will need to check that a v4-mapped group was passed, and make sure to perform the transaction pushdown to IGMP.
Thanks for excellent analysis! I ran into this one today at https://discuss.gradle.org/t/messagingservicestest-cannot-join-mcast-group-on-freebsd-invalid-argument/9662
Hi, just to pitch in with additional argumentation for resolution of this bug. I ran into this exact problem with openHAB, the workaround works. But as the original reporter writes, it puts of people using FreeBSD. It was also not trivial to diagnose. Cheers, Marc.
MARKED AS SPAM
Any progress here? I can see Tomcat test failures with multicast.
I don't think anyone is actively working on this. bms@ is no longer working on FreeBSD currently, so it will require finding someone with interest in this area (and multicast is kind of an unusual case relative to most network stack use cases).
(In reply to John Baldwin from comment #16) Unfortunately, I can't. This is out of my C knowledge and as far as I understand this issue, FreeBSD cannot use IPv6-mapped IPv4 addresses.
FreeBSD can use IPv6-mapped IPv4 addresses just fine for unicast traffic. What this bug is about is joining IPv4-addressed multicast groups on an IPv6 socket.
I think the question is not only about joining an IPv4 multicast group on an IPv6 socket, but about whether mapped addresses are supported for multicast in general... That might make the fix more complex than just fixing the join operation.