Bug 193246 - Bug in IPv6 multicast join(), uncovered by Jenkins
Summary: Bug in IPv6 multicast join(), uncovered by Jenkins
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 10.0-RELEASE
Hardware: Any Any
: --- Affects Many People
Assignee: freebsd-net mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-09-02 08:06 UTC by Craig Rodrigues
Modified: 2018-08-04 12:36 UTC (History)
12 users (show)

See Also:


Attachments
MulticastTest.java (812 bytes, text/plain)
2014-09-02 08:06 UTC, Craig Rodrigues
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Craig Rodrigues freebsd_committer 2014-09-02 08:06:20 UTC
Created attachment 146665 [details]
MulticastTest.java

.
Comment 1 Craig Rodrigues freebsd_committer 2014-09-02 08:21:31 UTC
Previously, I reported some error messages reported by Jenkins
on startup:

http://lists.freebsd.org/pipermail/freebsd-java/2014-February/010593.html

I looked at the Jenkins source code, and isolated the problem.
I wrote this simple testcase (see attached MulticastTest.java),
which I am also including inline:

================================================================================
/*
 * To build this test,
 * (1) Make sure that the OpenJDK is installed from ports:
 *
 *       pkg install openjdk
 *
 * (2) Rename this file to: MulticastTest.java
 * (3) Build it:
 *     
 *        javac MulticastTest.java
 *
 *  (4) Run it:
 *
 *        java MulticastTest.java
 *
 */

import java.net.InetAddress;
import java.net.MulticastSocket;

class MulticastTest
{

    public static void main(String[] args)
    {
       try {
        int PORT = Integer.getInteger("hudson.udp",33848);
 
        InetAddress MULTICAST = InetAddress.getByAddress(new byte[]{(byte)239, (byte)77, (byte)124, (byte)213});
        MulticastSocket mcs = new MulticastSocket(PORT);
        mcs.joinGroup(MULTICAST);
       } catch (Exception e) {
          e.printStackTrace();
          System.exit(-1);  
       }
    }
}
================================================================================

If I run this testcase, I get the same error as what I reported earlier
with Jenkins:

java.net.SocketException: Invalid argument
        at java.net.PlainDatagramSocketImpl.join(Native Method)
        at java.net.AbstractPlainDatagramSocketImpl.join(AbstractPlainDatagramSocketImpl.java:178)
        at java.net.MulticastSocket.joinGroup(MulticastSocket.java:319)
        at MulticastTest.main(MulticastTest.java:31)


If I run:

ktrace java MulticastTest

I see that the error here:

 13253 java     CALL  setsockopt(0x4,0x29,0x1b,0x7fffffbfd7dc,0x4)
 13253 java     RET   setsockopt 0
 13253 java     CALL  setsockopt(0x4,SOL_SOCKET,SO_BROADCAST,0x7fffffbfd7d8,0x4)
 13253 java     RET   setsockopt 0
 13253 java     CALL  getsockopt(0x4,SOL_SOCKET,SO_TYPE,0x7fffffbfd77c,0x7fffffbfd778)
 13253 java     RET   getsockopt 0
 13253 java     CALL  setsockopt(0x4,SOL_SOCKET,SO_REUSEPORT,0x7fffffbfd7e0,0x4)
 13253 java     RET   setsockopt 0
 13253 java     CALL  setsockopt(0x4,SOL_SOCKET,SO_REUSEADDR,0x7fffffbfd7e0,0x4)
 13253 java     RET   setsockopt 0
 13253 java     CALL  bind(0x4,0x7fffffbfd7a8,0x1c)
 13253 java     STRU  struct sockaddr { AF_INET6, [::]:33848 }
 13253 java     RET   bind 0
 13253 java     CALL  setsockopt(0x4,0x29,0x9,0x7fffffbfd7f4,0x4)
 13253 java     RET   setsockopt 0
 13253 java     CALL  getsockopt(0x4,0x29,0x9,0x7fffffbfd8ac,0x7fffffbfd864)
 13253 java     RET   getsockopt 0
 13253 java     CALL  setsockopt(0x4,0x29,0xc,0x7fffffbfd8c0,0x14)
 13253 java     RET   setsockopt -1 errno 22 Invalid argument


This looks like a bug in the FreeBSD networking code for multicast,
or a bug in the FreeBSD code in the OpenJDK.

This Java code works under Linux, Solaris, Windows, etc., so it
would be good to fix this problem on FreeBSD.

Can a networking person help me with this?
Comment 2 Craig Rodrigues freebsd_committer 2014-09-02 10:08:03 UTC
I did some more analysis and found that this code:


 13253 java     CALL  bind(0x4,0x7fffffbfd7a8,0x1c)
 13253 java     STRU  struct sockaddr { AF_INET6, [::]:33848 }
 13253 java     RET   bind 0
 13253 java     CALL  setsockopt(0x4,0x29,0x9,0x7fffffbfd7f4,0x4)
 13253 java     RET   setsockopt 0
 13253 java     CALL  getsockopt(0x4,0x29,0x9,0x7fffffbfd8ac,0x7fffffbfd864)
 13253 java     RET   getsockopt 0
 13253 java     CALL  setsockopt(0x4,0x29,0xc,0x7fffffbfd8c0,0x14)
 13253 java     RET   setsockopt -1 errno 22 Invalid argument

is happening inside the mcast_join_leave() function inside the JDK here:

http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/src/solaris/native/java/net/PlainDatagramSocketImpl.c

From looking at the FreeBSD header files,
IPPROTO_IPV6 = 41 (0x29)
IPV6_MULTICAST_IF = 
IPV6_JOIN_GROUP = 12 (0xc)_

inside the mcast_join_leave() function what is basically happening is:

setsockopt(0x4, IPPROTO_IPV6, IPV6_MULTICAST_IF, ...)
getsockopt(0x4, IPPROTO_IPV6, IPV6_MULTICAST_IF, ...)
setsockopt(0x4, IPPROTO_IPV6, IPV6_JOIN_GROUP, ...)

the second setsockopt() is returning EINVAL
Comment 3 Craig Rodrigues freebsd_committer 2014-09-02 11:13:53 UTC
I ran the same test program under truss with:

truss java MulticastTest

and found this:
socket(PF_INET6,SOCK_DGRAM,0)                    = 4 (0x4)
setsockopt(0x4,0x29,0x1b,0x7fffffbfd7dc,0x4,0x0) = 0 (0x0)
setsockopt(0x4,0xffff,0x20,0x7fffffbfd7d8,0x4,0x0) = 0 (0x0)
getsockopt(0x4,0xffff,0x1008,0x7fffffbfd77c,0x7fffffbfd778,0x83f5292d8) = 0 (0x0)
setsockopt(0x4,0xffff,0x200,0x7fffffbfd7e0,0x4,0x83f5292d8) = 0 (0x0)
setsockopt(0x4,0xffff,0x4,0x7fffffbfd7e0,0x4,0x83f5292d8) = 0 (0x0)
bind(4,{ AF_INET6 [108c:bd00:800:0:4b41:9700:100:0]:36096 },28) = 0 (0x0)
setsockopt(0x4,0x29,0x9,0x7fffffbfd7f4,0x4,0x83f54efd0) = 0 (0x0)
getsockopt(0x4,0x29,0x9,0x7fffffbfd8ac,0x7fffffbfd864,0x83f54c798) = 0 (0x0)
setsockopt(0x4,0x29,0xc,0x7fffffbfd8c0,0x14,0x83f54c798) ERR#22 'Invalid argument'
Comment 4 Craig Rodrigues freebsd_committer 2014-09-03 10:05:53 UTC
I tracked this down some more.

Inside the JDK, there is this code in http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/src/solaris/native/java/net/PlainDatagramSocketImpl.c

===============================================================================
  /*
     * IPv6 join. If it's an IPv4 multicast group then we use an IPv4-mapped
     * address.
     */
#ifdef AF_INET6
    {
        struct ipv6_mreq mname6;
        jbyteArray ipaddress;
        jbyte caddr[16];
        jint family;
        jint address;
        family = (*env)->GetIntField(env, iaObj, ia_familyID) == IPv4? AF_INET : AF_INET6;
        if (family == AF_INET) { /* will convert to IPv4-mapped address */
            memset((char *) caddr, 0, 16);
            address = (*env)->GetIntField(env, iaObj, ia_addressID);

            caddr[10] = 0xff;
            caddr[11] = 0xff;

            caddr[12] = ((address >> 24) & 0xff);
            caddr[13] = ((address >> 16) & 0xff);
            caddr[14] = ((address >> 8) & 0xff);
            caddr[15] = (address & 0xff);


===============================================================================




I can confirm that the address created by this code looks something like:

0 0 0 0 0 0 0 0 0 0 ff ff ef 4d 7c d5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0


In FreeBSD, in  src/sys/netinet6/in6_mcast.c inside in6p_join_group(), there
is this:

        if (!IN6_IS_ADDR_MULTICAST(&gsa->sin6.sin6_addr))
                return (EINVAL);

Since IN6_IS_ADDR_MULTICAST() only checks if the first octet is 0xff, that is what is returning the EINVAL.  So the JDK is creating
an IPV4 multicast address mapped inside an IPV6 address.  The FreeBSD
kernel code is rejecting this as a valid IPV6 multicast address.

I'm not sure if it is better to fix this in the kernel or the JDK.
Comment 5 John Baldwin freebsd_committer freebsd_triage 2014-09-04 14:27:21 UTC
Don't you really want an IPv4 socket here anyway?  It seems rather convoluted to create an IPv6 socket so you can listen for IPv4 multicast.  My guess is that the in6_mcast code doesn't handle IPv4 multicast groups, but bms@ might know.
Comment 6 Craig Rodrigues freebsd_committer 2014-09-04 16:22:08 UTC
I'm looking at this from the perspective that
3rd party multicast code written in Java "just works" under Linux
and Solaris, but fails under FreeBSD.

Sure, using an IPv4 socket from the beginning would be the way to go,
but now that will require pushing patches upstream to all Java software
using multicast, in order to accomodate FreeBSD.  It's possible, but not practical.

It would be nice if the FreeBSD IPv6 stack could be modified
to deal with IPv4-mapped multicast addresses for IPv6, similar to
how Solaris does it.  See comment from Andrey, http://lists.freebsd.org/pipermail/freebsd-net/2014-September/039686.html

If that could be made to work, then no changes to upstream code would be required.
Comment 7 Craig Rodrigues freebsd_committer 2014-09-04 16:29:59 UTC
In src/sys/netinet6, I see that there is usage of a 
IN6_IS_ADDR_V4MAPPED() macro in other places in the code, like in udp6 and sctp6,
so V4 mapped addresses are supported for other things.
Comment 8 John Baldwin freebsd_committer freebsd_triage 2014-09-04 18:16:41 UTC
It seems to be a non-trivial amount of work.  From bms@ on IRC:

<quote>
There's no hard and fast reasons why it couldn't be done. The code as it stands will reject that as being an API mixup (you want v4 memberships, use the v4 APIs).
The tension points are the Layer 4 ingress filtering for SSM, and actually calling Layer 2 in the right way.

The nasty thing about IP6-mapped is that you need to track the memberships in v6 terms, but hand-off all the work to the v4 routines to do the right thing.

The easiest way to go about doing it is to deal with the ASM case first, and just punch a hole in ingress filtering if someone tries to use SSM (which is what the
stack has to do anyway).

I'm not going to stick around to see what happens, though. ;-)
</quote>

I think what this means is that in_mcast6.c is very much tied to doing MLDv6 (or whatever the v6 equivalent of IGMP is), but for these groups, you need to be somehow calling into in_mcast.c to do IGMP instead.  However, I'm not sure at what level in_mcast6.c needs to call into in_mcast.c myself.  I do think Bruce is your best bet for someone to ask.
Comment 9 Ronald Klop 2014-09-05 09:30:32 UTC
Not a solution, but does it work as a workaround to add this option on the commandline to java?
-Djava.net.preferIPv4Stack=true
Comment 10 Craig Rodrigues freebsd_committer 2014-09-05 15:25:58 UTC
Ronald,

Thanks for the tip.

java -Djava.net.preferIPv4Stack=true MulticastTest

seems to work around the problem.

I would still like to see FreeBSD fixed so that this workaround is not required.

Even though it is a lot of work, I would like to see Java on FreeBSD behave out of the box, "just like Linux", without the FreeBSD community needing to
push lots of patches upstream to different Java software authors.
I want there to be less motivation for people to migrate from FreeBSD to Linux
if they are deploying Java applications.

That's why I've spent the time to analyze the problem and reporting my
findings in this bug report.  I find this audit trail is quite interesting. :)
Comment 11 Craig Rodrigues freebsd_committer 2014-09-07 02:06:14 UTC
-------- Forwarded Message --------
Subject: 	Re: [Bug 193246] Bug in IPv6 multicast join(), uncovered by Jenkins
Date: 	Fri, 05 Sep 2014 03:05:14 +0100
From: 	Bruce Simpson <bms@fastmail.net>
To: 	bugzilla-noreply@freebsd.org


On 04/09/2014 19:16, bugzilla-noreply@freebsd.org wrote:
> I think what this means is that in_mcast6.c is very much tied to doing MLDv6
> (or whatever the v6 equivalent of IGMP is), but for these groups, you need to
> be somehow calling into in_mcast.c to do IGMP instead.  However, I'm not sure
> at what level in_mcast6.c needs to call into in_mcast.c myself.  I do think
> Bruce is your best bet for someone to ask.
>
Unfortunately I'm fully booked. But yes, that is a good summary.

If Craig (or someone else) is willing to volunteer to support v4-mapped 
addresses: one approach would be to extend in6_mship{} to include them. 
Pushing state down to IGMP will need to be added as a special case. To 
keep it simple, assume that only the legacy any-source multicast (ASM) 
model will be supported, i.e. listeners will not specify source filters.

Looking at the JDK source, it appears they used to handle the v4/v6 
swizzle themselves because of limitations in Linux 2.4/2.6.

In other words, we do not support RFC 3493 Sec 3.7 for multicast groups 
at the moment. A more appropriate errno value to return is EPROTONOOPT. 
[Interestingly, Sec 5.2 discusses IPv6 multicast options, but does not 
contain any reference to IPv4 at all.]

There now follows a late-night writeup of the rationale behind this code 
-- and this is as concise as I can make it, I'm afraid.

in[6]_mcast.c is split into a top half (i.e. per-socket state) and a 
bottom half (i.e. stack-wide state, and the IGMPv1/2/3 and MLDv1/2 wire 
protocols). IPv6 mcast options are processed separately from IPv4.

Both implement a transaction scheme to protect global membership state 
(in the bottom half) from failures at the granularity of a single socket 
(or PCB) in the top half.

Why all the complexity? Well, this is to support source-specific 
multicast (SSM, aka "inclusive mode"). To cut a long story short: as the 
size of an internetwork increases, it gets more difficult for routers to 
track the state of multicast listeners, unless they are aware of where 
the traffic originates from. The book "Interdomain Multicast Routing" by 
Brian M. Edwards discusses this in lurid detail.

So, SSM was introduced to support inter-domain multicast. In this model, 
joining a multicast group is no longer a simple matter of registering 
for a channel -- you must also specify the sources you are interested 
in. However, the majority of multicast applications are built on the 
older model: i.e. they do not cross more than one IP network hop, and do 
not specify sources ("any-source multicast", aka ASM).

The network stack must be able to cope with both of these uses. It does 
so by representing the legacy ASM scheme as "exclusive mode". The RFC 
3678 APIs also have the advantage that the application can block 
unwanted senders, even if ASM is in use.  [The main API it specifies, 
setsourcefilter(), does not explicitly mandate v4-mapped support.]

So, in the bottom half of mcast, each group has a shared RB-tree of 
listener state. This is created from the set union of the filter state 
on each subscribed socket.

If there are no filters for a particular group, i.e. all of the 
sockets/PCBs in the system are in "exclusive" mode and have no filters, 
then of course the RB-tree for that group will be empty. Otherwise, if 
there is a mix of "exclusive" and "inclusive" mode listeners, the tree 
will need to be recomputed.

The shared tree is then used to fill out the IGMP/MLD messages sent to 
on-link routers to request group subscription. It is also used to filter 
input traffic from within the relevant transports (e.g. UDP, raw 
sockets). Previously, this filtering required that the network layer 
take a socket-layer lock.

In closing: this isn't just a simple matter of adding a few defines. The 
v6 code will need to check that a v4-mapped group was passed, and make 
sure to perform the transaction pushdown to IGMP.
Comment 12 Marcin Cieślak 2015-05-17 17:15:00 UTC
Thanks for excellent analysis!

I ran into this one today at https://discuss.gradle.org/t/messagingservicestest-cannot-join-mcast-group-on-freebsd-invalid-argument/9662
Comment 13 marc 2016-01-08 09:31:00 UTC
Hi, 

just to pitch in with additional argumentation for resolution of this bug. I ran into this exact problem with openHAB, the workaround works. But as the original reporter writes, it puts of people using FreeBSD. It was also not trivial to diagnose.

Cheers,
Marc.
Comment 14 vali gholami 2017-12-17 07:12:02 UTC
MARKED AS SPAM
Comment 15 Michael Osipov 2018-08-03 22:25:15 UTC
Any progress here? I can see Tomcat test failures with multicast.
Comment 16 John Baldwin freebsd_committer freebsd_triage 2018-08-03 23:30:25 UTC
I don't think anyone is actively working on this.  bms@ is no longer working on FreeBSD currently, so it will require finding someone with interest in this area (and multicast is kind of an unusual case relative to most network stack use cases).
Comment 17 Michael Osipov 2018-08-04 08:12:06 UTC
(In reply to John Baldwin from comment #16)

Unfortunately, I can't. This is out of my C knowledge and as far as I understand this issue, FreeBSD cannot use IPv6-mapped IPv4 addresses.
Comment 18 John Baldwin freebsd_committer freebsd_triage 2018-08-04 12:17:31 UTC
FreeBSD can use IPv6-mapped IPv4 addresses just fine for unicast traffic.  What this bug is about is joining IPv4-addressed multicast groups on an IPv6 socket.
Comment 19 Michael Tuexen freebsd_committer 2018-08-04 12:36:32 UTC
I think the question is not only about joining an IPv4 multicast group on an IPv6 socket, but about whether mapped addresses are supported for multicast in general...
That might make the fix more complex than just fixing the join operation.