Created attachment 146665 [details] MulticastTest.java .
Previously, I reported some error messages reported by Jenkins on startup: http://lists.freebsd.org/pipermail/freebsd-java/2014-February/010593.html I looked at the Jenkins source code, and isolated the problem. I wrote this simple testcase (see attached MulticastTest.java), which I am also including inline: ================================================================================ /* * To build this test, * (1) Make sure that the OpenJDK is installed from ports: * * pkg install openjdk * * (2) Rename this file to: MulticastTest.java * (3) Build it: * * javac MulticastTest.java * * (4) Run it: * * java MulticastTest.java * */ import java.net.InetAddress; import java.net.MulticastSocket; class MulticastTest { public static void main(String[] args) { try { int PORT = Integer.getInteger("hudson.udp",33848); InetAddress MULTICAST = InetAddress.getByAddress(new byte[]{(byte)239, (byte)77, (byte)124, (byte)213}); MulticastSocket mcs = new MulticastSocket(PORT); mcs.joinGroup(MULTICAST); } catch (Exception e) { e.printStackTrace(); System.exit(-1); } } } ================================================================================ If I run this testcase, I get the same error as what I reported earlier with Jenkins: java.net.SocketException: Invalid argument at java.net.PlainDatagramSocketImpl.join(Native Method) at java.net.AbstractPlainDatagramSocketImpl.join(AbstractPlainDatagramSocketImpl.java:178) at java.net.MulticastSocket.joinGroup(MulticastSocket.java:319) at MulticastTest.main(MulticastTest.java:31) If I run: ktrace java MulticastTest I see that the error here: 13253 java CALL setsockopt(0x4,0x29,0x1b,0x7fffffbfd7dc,0x4) 13253 java RET setsockopt 0 13253 java CALL setsockopt(0x4,SOL_SOCKET,SO_BROADCAST,0x7fffffbfd7d8,0x4) 13253 java RET setsockopt 0 13253 java CALL getsockopt(0x4,SOL_SOCKET,SO_TYPE,0x7fffffbfd77c,0x7fffffbfd778) 13253 java RET getsockopt 0 13253 java CALL setsockopt(0x4,SOL_SOCKET,SO_REUSEPORT,0x7fffffbfd7e0,0x4) 13253 java RET setsockopt 0 13253 java CALL setsockopt(0x4,SOL_SOCKET,SO_REUSEADDR,0x7fffffbfd7e0,0x4) 13253 java RET setsockopt 0 13253 java CALL bind(0x4,0x7fffffbfd7a8,0x1c) 13253 java STRU struct sockaddr { AF_INET6, [::]:33848 } 13253 java RET bind 0 13253 java CALL setsockopt(0x4,0x29,0x9,0x7fffffbfd7f4,0x4) 13253 java RET setsockopt 0 13253 java CALL getsockopt(0x4,0x29,0x9,0x7fffffbfd8ac,0x7fffffbfd864) 13253 java RET getsockopt 0 13253 java CALL setsockopt(0x4,0x29,0xc,0x7fffffbfd8c0,0x14) 13253 java RET setsockopt -1 errno 22 Invalid argument This looks like a bug in the FreeBSD networking code for multicast, or a bug in the FreeBSD code in the OpenJDK. This Java code works under Linux, Solaris, Windows, etc., so it would be good to fix this problem on FreeBSD. Can a networking person help me with this?
I did some more analysis and found that this code: 13253 java CALL bind(0x4,0x7fffffbfd7a8,0x1c) 13253 java STRU struct sockaddr { AF_INET6, [::]:33848 } 13253 java RET bind 0 13253 java CALL setsockopt(0x4,0x29,0x9,0x7fffffbfd7f4,0x4) 13253 java RET setsockopt 0 13253 java CALL getsockopt(0x4,0x29,0x9,0x7fffffbfd8ac,0x7fffffbfd864) 13253 java RET getsockopt 0 13253 java CALL setsockopt(0x4,0x29,0xc,0x7fffffbfd8c0,0x14) 13253 java RET setsockopt -1 errno 22 Invalid argument is happening inside the mcast_join_leave() function inside the JDK here: http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/src/solaris/native/java/net/PlainDatagramSocketImpl.c From looking at the FreeBSD header files, IPPROTO_IPV6 = 41 (0x29) IPV6_MULTICAST_IF = IPV6_JOIN_GROUP = 12 (0xc)_ inside the mcast_join_leave() function what is basically happening is: setsockopt(0x4, IPPROTO_IPV6, IPV6_MULTICAST_IF, ...) getsockopt(0x4, IPPROTO_IPV6, IPV6_MULTICAST_IF, ...) setsockopt(0x4, IPPROTO_IPV6, IPV6_JOIN_GROUP, ...) the second setsockopt() is returning EINVAL
I ran the same test program under truss with: truss java MulticastTest and found this: socket(PF_INET6,SOCK_DGRAM,0) = 4 (0x4) setsockopt(0x4,0x29,0x1b,0x7fffffbfd7dc,0x4,0x0) = 0 (0x0) setsockopt(0x4,0xffff,0x20,0x7fffffbfd7d8,0x4,0x0) = 0 (0x0) getsockopt(0x4,0xffff,0x1008,0x7fffffbfd77c,0x7fffffbfd778,0x83f5292d8) = 0 (0x0) setsockopt(0x4,0xffff,0x200,0x7fffffbfd7e0,0x4,0x83f5292d8) = 0 (0x0) setsockopt(0x4,0xffff,0x4,0x7fffffbfd7e0,0x4,0x83f5292d8) = 0 (0x0) bind(4,{ AF_INET6 [108c:bd00:800:0:4b41:9700:100:0]:36096 },28) = 0 (0x0) setsockopt(0x4,0x29,0x9,0x7fffffbfd7f4,0x4,0x83f54efd0) = 0 (0x0) getsockopt(0x4,0x29,0x9,0x7fffffbfd8ac,0x7fffffbfd864,0x83f54c798) = 0 (0x0) setsockopt(0x4,0x29,0xc,0x7fffffbfd8c0,0x14,0x83f54c798) ERR#22 'Invalid argument'
I tracked this down some more. Inside the JDK, there is this code in http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/src/solaris/native/java/net/PlainDatagramSocketImpl.c =============================================================================== /* * IPv6 join. If it's an IPv4 multicast group then we use an IPv4-mapped * address. */ #ifdef AF_INET6 { struct ipv6_mreq mname6; jbyteArray ipaddress; jbyte caddr[16]; jint family; jint address; family = (*env)->GetIntField(env, iaObj, ia_familyID) == IPv4? AF_INET : AF_INET6; if (family == AF_INET) { /* will convert to IPv4-mapped address */ memset((char *) caddr, 0, 16); address = (*env)->GetIntField(env, iaObj, ia_addressID); caddr[10] = 0xff; caddr[11] = 0xff; caddr[12] = ((address >> 24) & 0xff); caddr[13] = ((address >> 16) & 0xff); caddr[14] = ((address >> 8) & 0xff); caddr[15] = (address & 0xff); =============================================================================== I can confirm that the address created by this code looks something like: 0 0 0 0 0 0 0 0 0 0 ff ff ef 4d 7c d5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 In FreeBSD, in src/sys/netinet6/in6_mcast.c inside in6p_join_group(), there is this: if (!IN6_IS_ADDR_MULTICAST(&gsa->sin6.sin6_addr)) return (EINVAL); Since IN6_IS_ADDR_MULTICAST() only checks if the first octet is 0xff, that is what is returning the EINVAL. So the JDK is creating an IPV4 multicast address mapped inside an IPV6 address. The FreeBSD kernel code is rejecting this as a valid IPV6 multicast address. I'm not sure if it is better to fix this in the kernel or the JDK.
Don't you really want an IPv4 socket here anyway? It seems rather convoluted to create an IPv6 socket so you can listen for IPv4 multicast. My guess is that the in6_mcast code doesn't handle IPv4 multicast groups, but bms@ might know.
I'm looking at this from the perspective that 3rd party multicast code written in Java "just works" under Linux and Solaris, but fails under FreeBSD. Sure, using an IPv4 socket from the beginning would be the way to go, but now that will require pushing patches upstream to all Java software using multicast, in order to accomodate FreeBSD. It's possible, but not practical. It would be nice if the FreeBSD IPv6 stack could be modified to deal with IPv4-mapped multicast addresses for IPv6, similar to how Solaris does it. See comment from Andrey, http://lists.freebsd.org/pipermail/freebsd-net/2014-September/039686.html If that could be made to work, then no changes to upstream code would be required.
In src/sys/netinet6, I see that there is usage of a IN6_IS_ADDR_V4MAPPED() macro in other places in the code, like in udp6 and sctp6, so V4 mapped addresses are supported for other things.
It seems to be a non-trivial amount of work. From bms@ on IRC: <quote> There's no hard and fast reasons why it couldn't be done. The code as it stands will reject that as being an API mixup (you want v4 memberships, use the v4 APIs). The tension points are the Layer 4 ingress filtering for SSM, and actually calling Layer 2 in the right way. The nasty thing about IP6-mapped is that you need to track the memberships in v6 terms, but hand-off all the work to the v4 routines to do the right thing. The easiest way to go about doing it is to deal with the ASM case first, and just punch a hole in ingress filtering if someone tries to use SSM (which is what the stack has to do anyway). I'm not going to stick around to see what happens, though. ;-) </quote> I think what this means is that in_mcast6.c is very much tied to doing MLDv6 (or whatever the v6 equivalent of IGMP is), but for these groups, you need to be somehow calling into in_mcast.c to do IGMP instead. However, I'm not sure at what level in_mcast6.c needs to call into in_mcast.c myself. I do think Bruce is your best bet for someone to ask.
Not a solution, but does it work as a workaround to add this option on the commandline to java? -Djava.net.preferIPv4Stack=true
Ronald, Thanks for the tip. java -Djava.net.preferIPv4Stack=true MulticastTest seems to work around the problem. I would still like to see FreeBSD fixed so that this workaround is not required. Even though it is a lot of work, I would like to see Java on FreeBSD behave out of the box, "just like Linux", without the FreeBSD community needing to push lots of patches upstream to different Java software authors. I want there to be less motivation for people to migrate from FreeBSD to Linux if they are deploying Java applications. That's why I've spent the time to analyze the problem and reporting my findings in this bug report. I find this audit trail is quite interesting. :)
-------- Forwarded Message -------- Subject: Re: [Bug 193246] Bug in IPv6 multicast join(), uncovered by Jenkins Date: Fri, 05 Sep 2014 03:05:14 +0100 From: Bruce Simpson <bms@fastmail.net> To: bugzilla-noreply@freebsd.org On 04/09/2014 19:16, bugzilla-noreply@freebsd.org wrote: > I think what this means is that in_mcast6.c is very much tied to doing MLDv6 > (or whatever the v6 equivalent of IGMP is), but for these groups, you need to > be somehow calling into in_mcast.c to do IGMP instead. However, I'm not sure > at what level in_mcast6.c needs to call into in_mcast.c myself. I do think > Bruce is your best bet for someone to ask. > Unfortunately I'm fully booked. But yes, that is a good summary. If Craig (or someone else) is willing to volunteer to support v4-mapped addresses: one approach would be to extend in6_mship{} to include them. Pushing state down to IGMP will need to be added as a special case. To keep it simple, assume that only the legacy any-source multicast (ASM) model will be supported, i.e. listeners will not specify source filters. Looking at the JDK source, it appears they used to handle the v4/v6 swizzle themselves because of limitations in Linux 2.4/2.6. In other words, we do not support RFC 3493 Sec 3.7 for multicast groups at the moment. A more appropriate errno value to return is EPROTONOOPT. [Interestingly, Sec 5.2 discusses IPv6 multicast options, but does not contain any reference to IPv4 at all.] There now follows a late-night writeup of the rationale behind this code -- and this is as concise as I can make it, I'm afraid. in[6]_mcast.c is split into a top half (i.e. per-socket state) and a bottom half (i.e. stack-wide state, and the IGMPv1/2/3 and MLDv1/2 wire protocols). IPv6 mcast options are processed separately from IPv4. Both implement a transaction scheme to protect global membership state (in the bottom half) from failures at the granularity of a single socket (or PCB) in the top half. Why all the complexity? Well, this is to support source-specific multicast (SSM, aka "inclusive mode"). To cut a long story short: as the size of an internetwork increases, it gets more difficult for routers to track the state of multicast listeners, unless they are aware of where the traffic originates from. The book "Interdomain Multicast Routing" by Brian M. Edwards discusses this in lurid detail. So, SSM was introduced to support inter-domain multicast. In this model, joining a multicast group is no longer a simple matter of registering for a channel -- you must also specify the sources you are interested in. However, the majority of multicast applications are built on the older model: i.e. they do not cross more than one IP network hop, and do not specify sources ("any-source multicast", aka ASM). The network stack must be able to cope with both of these uses. It does so by representing the legacy ASM scheme as "exclusive mode". The RFC 3678 APIs also have the advantage that the application can block unwanted senders, even if ASM is in use. [The main API it specifies, setsourcefilter(), does not explicitly mandate v4-mapped support.] So, in the bottom half of mcast, each group has a shared RB-tree of listener state. This is created from the set union of the filter state on each subscribed socket. If there are no filters for a particular group, i.e. all of the sockets/PCBs in the system are in "exclusive" mode and have no filters, then of course the RB-tree for that group will be empty. Otherwise, if there is a mix of "exclusive" and "inclusive" mode listeners, the tree will need to be recomputed. The shared tree is then used to fill out the IGMP/MLD messages sent to on-link routers to request group subscription. It is also used to filter input traffic from within the relevant transports (e.g. UDP, raw sockets). Previously, this filtering required that the network layer take a socket-layer lock. In closing: this isn't just a simple matter of adding a few defines. The v6 code will need to check that a v4-mapped group was passed, and make sure to perform the transaction pushdown to IGMP.
Thanks for excellent analysis! I ran into this one today at https://discuss.gradle.org/t/messagingservicestest-cannot-join-mcast-group-on-freebsd-invalid-argument/9662
Hi, just to pitch in with additional argumentation for resolution of this bug. I ran into this exact problem with openHAB, the workaround works. But as the original reporter writes, it puts of people using FreeBSD. It was also not trivial to diagnose. Cheers, Marc.
MARKED AS SPAM
Any progress here? I can see Tomcat test failures with multicast.
I don't think anyone is actively working on this. bms@ is no longer working on FreeBSD currently, so it will require finding someone with interest in this area (and multicast is kind of an unusual case relative to most network stack use cases).
(In reply to John Baldwin from comment #16) Unfortunately, I can't. This is out of my C knowledge and as far as I understand this issue, FreeBSD cannot use IPv6-mapped IPv4 addresses.
FreeBSD can use IPv6-mapped IPv4 addresses just fine for unicast traffic. What this bug is about is joining IPv4-addressed multicast groups on an IPv6 socket.
I think the question is not only about joining an IPv4 multicast group on an IPv6 socket, but about whether mapped addresses are supported for multicast in general... That might make the fix more complex than just fixing the join operation.
Just regained access after nearly a decade. There was some personal tragedy involved. The resolution for this is being tracked at: https://reviews.freebsd.org/D55031 With a minor errno correction at: https://reviews.freebsd.org/D55233 TL;DR This is purely an optional extension. It is not and never was a bug. The OpenJDK and other software started relying on it over the years, but it really doesn't fit with how the socket APIs involved were intended to operate, at all. It largely stems from uncertainty about the extent of IPv4-mapped address support, as tuexen has indicated. The cheap fix is to just do what Apple did and just pass it through to netinet for simple ASM-mode adds/joins, which is the most common multicast use case anyway, and it's usually for local link scoped groups.
(In reply to Bruce M Simpson from comment #20) Please call me stupid but is that review supposed to fix that? I have tried: Applied the one-off patch to 16-CURRENT, installed a new kernel, rebooted and tried the Java code (OpenJDK 25) for the join. Same issue. What did I miss?
Please don't jump the gun, thanks. I only just got reinstated by core@ on Sunday, and I only just regained access to the cluster this morning after the better part of 10 years. You didn't state which "Patch" you tried to use. There are two Differentials. One of those patches only changes the errno value for the non-AF_INET6 address and length case. The other is based on a change from Apple and is most likely incomplete, and Glebius already pointed out it needs adjustment for our stack. You clearly read the review, so you are aware it hasn't actually been tested yet, as there is no Kyua test case for it. And it most likely will only work because inp_moptions is still present even in a NOINET kernel.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=79eff4ad64fc445ea78ed733e16482c45dbd5b50 commit 79eff4ad64fc445ea78ed733e16482c45dbd5b50 Author: Bruce M Simpson <bms@FreeBSD.org> AuthorDate: 2026-02-14 15:36:47 +0000 Commit: Bruce M Simpson <bms@FreeBSD.org> CommitDate: 2026-02-14 16:00:58 +0000 netinet6: Return EAFNOSUPPORT for non-IPv6 addresses in mcast sockopts. This is a non-functional change; it just returns the correct errno value where IPv6 multicast socket options were passed non-AF_INET6 arguments, in preparation for handling PR 193246 with a side-call into netinet as xnu currently does. Reviewed by: glebius Approved by: glebius PR: 193246 (with refinements) Differential revision: https://reviews.freebsd.org/D55233 sys/netinet6/in6_mcast.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-)
Bruce, I would like to say big thank you for coming back here and working on this.
Track dependent refactoring requested by Gleb: https://reviews.freebsd.org/D55344 https://reviews.freebsd.org/D55345
I believe we have a fix: https://reviews.freebsd.org/D58590
Michaelo, can you please retry your test with OpenJDK 25 with most recent revisions from Bruce? If that works we definitely need to augment a patch with a Kyua test that covers this.
(In reply to Gleb Smirnoff from comment #27) I will happily test after I get back next week.
It is all on main now, because of my git rebase mishap earlier today.
I have updated my -current to 1c0d2f0b1a98526e4f1f5a051ce6904b4c3164e1 and played with Jython a bit. When I try on java 11: import java.net g = java.net.InetAddress.getAllByName("228.5.6.7") z = java.net.MulticastSocket(6790) z.joinGroup(g[0]) This gives: > jython -Djava.net.preferIPv4Stack=false join_mcast.py WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by jnr.posix.JavaLibCHelper (file:/usr/local/lib/jython/jython.jar) to method sun.nio.ch.SelChImpl.getFD() WARNING: Please consider reporting this to the maintainers of jnr.posix.JavaLibCHelper WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Traceback (most recent call last): File "join_mcast.py", line 4, in <module> z.joinGroup(g[0]) at java.base/java.net.PlainDatagramSocketImpl.join(Native Method) at java.base/java.net.AbstractPlainDatagramSocketImpl.join(AbstractPlainDatagramSocketImpl.java:235) at java.base/java.net.MulticastSocket.joinGroup(MulticastSocket.java:328) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) java.net.SocketException: java.net.SocketException: Invalid argument I can confirm that udp46 socket gets created before the program exists udp46 0 0 *.6790 *.*
That's going to be a tad harder for me to track down. I'll try to spare cycles for this, we need to get it killed off because the issue is over a decade old now, but there is other stuff to land, and other FreeBSD related projects I must instigate. Java is enough to obscure some of what's going on. truss doesn't really report much that is useful in relation to setsockopt(), and ktrace will show the call, but not the sockopt. And of course that gets further obfuscated by the fact that the need to support this forces a sockopt to be mutated in-kernel post-facto. I'm "on the hook" so to speak for the Kyua test Gleb wanted, the mtest extension will exercise the path manually, so please do try with that, as it's unclear from this output exactly which syscall returned EINVAL. The mtest code is about as simple as one can get. I seem to remember syncthing had issues in this area It is probably expecting too much for the conceptual originator of the change to come forward and upstream that which could have been upstreamed over a decade ago when I was busy with other more pressing matters. It is also disappointing that the JDK backed out the very code preceding the introduction of a "de facto" vs "de jure" feature that the IETF never ratified which would have supported FreeBSD's discrete IPv4/IPv6 stacks in its prior form.
Hm, can dtrace help? You should be able to dump out the full args of syscalls and then dig into the kernel function call side of things to figure it out.