Bug 204831 - mld_v2 listener report does not report all active groups to the router
Summary: mld_v2 listener report does not report all active groups to the router
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 9.3-STABLE
Hardware: amd64 Any
: --- Affects Some People
Assignee: Andrey V. Elsukov
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-11-26 20:47 UTC by scheffler
Modified: 2015-12-08 08:13 UTC (History)
3 users (show)

See Also:


Attachments
Wireshark trace (269.21 KB, application/vnd.tcpdump.pcap)
2015-11-26 20:47 UTC, scheffler
no flags Details
Wireshark trace from Linux system (325.33 KB, application/vnd.tcpdump.pcap)
2015-11-30 09:38 UTC, scheffler
no flags Details
Proposed patch for stable/9 (untested) (846 bytes, patch)
2015-11-30 10:35 UTC, Andrey V. Elsukov
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description scheffler 2015-11-26 20:47:52 UTC
Created attachment 163558 [details]
Wireshark trace

When using applications that receive traffic from a rather large number (510) multicast groups, traffic from some groups is lost after time. 

Reception for all groups starts fine, but after a Multicast General Querry has been received from the on-link router, FreeBSD starts to loose lower numbered groups in the Multicast Listener Report Message. 
'ifmcstat' shows all Groups are still active, but they are no longer reported to the router and therefore traffic is no longer forwarded to the receiver.

This behaviour has first been observed on a FreeNAS using vnet, but could be replicated on a FreeBSD 9.3 with the latest patches (FreeBSD 9.3-RELEASE-p24).

I include a Wireshark trace that shows the behaviour.
Starting with packet 35 all 510 groups are reported. However, in packet 42 the reporting does not stop at the last group and all groups are reported again.
This continues until we see in packet 62 that the first multicast groups are lost:
ff15::1001:39 - ff15::1001:1
The loss contiues to grow until we have lost the first 221 groups, where it stays stable.

In order to replicate the behaviour, an active Multicast Router is required on the link (it seems to be triggered by the reception of the General Querry Message from the router). The behaviour can not be seen, when no router is present.

I used mgen and a simple c-programm to verify the behaviour. If needed, I can supply the testprogam.
Comment 1 Andrey V. Elsukov freebsd_committer 2015-11-30 03:58:40 UTC
It looks like the router, that sent general query uses very small "Maximum Response Delay" for such number of groups - 125 milliseconds. Is it possible to increase it? What is the system and software used on this router? 
FreeBSD limits the burst of replies to 4 packets, thereafter not yet sent packets will be send only after ~500 ms delay. Since response delay is very small, probably router just doesn't wait all packets?
Comment 2 scheffler 2015-11-30 09:08:16 UTC
Andrey,

are you sure you are reading the traces correctly?

1.) Maximum Response Code (Delay) is set to 10000 (10s) by the router. Which is the default value given by RFC3810. 
2.) The Query Intervall (QQIC) is set to the (default) value of 125, but the unit of this value is seconds.

The router is a Cisco 2811 running IOS 15.1-4.M10, the latest IOS for this platform supported by Cisco. The router has a basic MC configuration, no timer values have been changed from the default. The behaviour starts as soon as I enable 'IPv6 multicast-routing'. I also reproduced the behaviour on a 2901.

Your description of 4-packet burts makes sense - I was wondering about the 500ms delay between packet groups. The 510 groups need 8 packets to report. 
So, the kernel should stop after the second packet group (having reported all 510 groups to the router for this reporting period). However, it does not!
The trace clearly shows that it keeps on reporting the same groups over and over again until it suddenly starts losing groups from the report.
So to me it looks like 2 bugs:
 1.) Reporting should stop after having reported all 510 groups.
 2.) We should not lose groups from the report which are still active.

In the meantime I found a Linux-Box to run my MC code on and connected it to the very same router. Here the behaviour is very different. The router sends a General Query very 125 seconds. Linux reports the 510 groups (using 8 packets) and stays silent until it receives the next General Query. It also never reports less than the full 510 groups.
If you think it helps, I can also attach the Linux-trace.

Thomas
Comment 3 Andrey V. Elsukov freebsd_committer 2015-11-30 09:17:21 UTC
(In reply to scheffler from comment #2)
> are you sure you are reading the traces correctly?
> 
> 1.) Maximum Response Code (Delay) is set to 10000 (10s) by the router. Which
> is the default value given by RFC3810. 

Ah, yes, you are right. I read it incorrectly.

> 2.) The Query Intervall (QQIC) is set to the (default) value of 125, but the
> unit of this value is seconds.
> 
> The router is a Cisco 2811 running IOS 15.1-4.M10, the latest IOS for this
> platform supported by Cisco. The router has a basic MC configuration, no
> timer values have been changed from the default. The behaviour starts as
> soon as I enable 'IPv6 multicast-routing'. I also reproduced the behaviour
> on a 2901.
> 
> Your description of 4-packet burts makes sense - I was wondering about the
> 500ms delay between packet groups. The 510 groups need 8 packets to report. 
> So, the kernel should stop after the second packet group (having reported
> all 510 groups to the router for this reporting period). However, it does
> not!

Yes it looks strange to me too. But I think it should repeat these packets once again due to QRV equal to 2.

> The trace clearly shows that it keeps on reporting the same groups over and
> over again until it suddenly starts losing groups from the report.
> So to me it looks like 2 bugs:
>  1.) Reporting should stop after having reported all 510 groups.
>  2.) We should not lose groups from the report which are still active.
> 
> In the meantime I found a Linux-Box to run my MC code on and connected it to
> the very same router. Here the behaviour is very different. The router sends
> a General Query very 125 seconds. Linux reports the 510 groups (using 8
> packets) and stays silent until it receives the next General Query. It also
> never reports less than the full 510 groups.
> If you think it helps, I can also attach the Linux-trace.

Yes, it is interesting. Thanks.
Comment 4 Andrey V. Elsukov freebsd_committer 2015-11-30 09:28:31 UTC
It looks I was wrong again:

"Therefore, to enhance protocol robustness, in spite of the possible unreliability of message exchanges, messages are retransmitted several times.  Furthermore, timers are set so as to take into account the possible message losses, and to wait for retransmissions.

Periodical General Queries and Current State Reports do not apply
this rule, in order not to overload the link; it is assumed that in
general these messages do not generate state changes, their main
purpose being to refresh existing state."

So, it shouldn't repeat these packets.
Comment 5 scheffler 2015-11-30 09:32:45 UTC
(In reply to Andrey V. Elsukov from comment #4)


Correct. Repeated transmission is only expected for 'state change reports'.
Otherwise section 9.4 applies and takes care of lost reports: http://tools.ietf.org/html/rfc3810#section-9.4 

I will attach the Linux trace in a minute...
Comment 6 scheffler 2015-11-30 09:38:19 UTC
Created attachment 163679 [details]
Wireshark trace from Linux system

This trace shows the mldv2 traffic between a Linux system (running the same MC code) and a similarily configured Cisco router. 

(It is not exactly the same router as in the FreeBSD case, because I am not in the lab right now. But this should not matter...).
Comment 7 Andrey V. Elsukov freebsd_committer 2015-11-30 10:35:39 UTC
Created attachment 163681 [details]
Proposed patch for stable/9 (untested)

I found the problem.
Since mld_fasttimo_vnet() always calls mld_v2_dispatch_general_query() to handle general query, for each call it queues new replies. So, in your case after 4 packets was sent, mld_v2_dispatch_general_query() called again to send not yet sent packets. And this queues new 8 packets. After each call the number of queued packets grows, then probably queue is overflowed and some of groups start to get lost.

This patch add the check that queue is still have some packets to send and do not queue new packets. Instead it finishes the sending.
Comment 8 commit-hook freebsd_committer 2015-12-01 11:18:31 UTC
A commit references this bug:

Author: ae
Date: Tue Dec  1 11:17:42 UTC 2015
New revision: 291578
URL: https://svnweb.freebsd.org/changeset/base/291578

Log:
  mld_v2_dispatch_general_query() is used by mld_fasttimo_vnet() to send
  a reply to the MLDv2 General Query. In case when router has a lot of
  multicast groups, the reply can take several packets due to MTU limitation.
  Also we have a limit MLD_MAX_RESPONSE_BURST == 4, that limits the number
  of packets we send in one shot. Then we recalculate the timer value and
  schedule the remaining packets for sending.
  The problem is that when we call mld_v2_dispatch_general_query() to send
  remaining packets, we queue new reply in the same mbuf queue. And when
  number of packets is bigger than MLD_MAX_RESPONSE_BURST, we get endless
  reply of MLDv2 reports.
  To fix this, add the check for remaining packets in the queue.

  PR:		204831
  MFC after:	1 week
  Sponsored by:	Yandex LLC

Changes:
  head/sys/netinet6/mld6.c
Comment 9 commit-hook freebsd_committer 2015-12-08 07:26:27 UTC
A commit references this bug:

Author: ae
Date: Tue Dec  8 07:26:16 UTC 2015
New revision: 291986
URL: https://svnweb.freebsd.org/changeset/base/291986

Log:
  MFC r291578:
    mld_v2_dispatch_general_query() is used by mld_fasttimo_vnet() to send
    a reply to the MLDv2 General Query. In case when router has a lot of
    multicast groups, the reply can take several packets due to MTU limitation.
    Also we have a limit MLD_MAX_RESPONSE_BURST == 4, that limits the number
    of packets we send in one shot. Then we recalculate the timer value and
    schedule the remaining packets for sending.
    The problem is that when we call mld_v2_dispatch_general_query() to send
    remaining packets, we queue new reply in the same mbuf queue. And when
    number of packets is bigger than MLD_MAX_RESPONSE_BURST, we get endless
    reply of MLDv2 reports.
    To fix this, add the check for remaining packets in the queue.

    PR:		204831

Changes:
_U  stable/10/
  stable/10/sys/netinet6/mld6.c
Comment 10 commit-hook freebsd_committer 2015-12-08 07:36:31 UTC
A commit references this bug:

Author: ae
Date: Tue Dec  8 07:36:26 UTC 2015
New revision: 291988
URL: https://svnweb.freebsd.org/changeset/base/291988

Log:
  MFC r291578:
    mld_v2_dispatch_general_query() is used by mld_fasttimo_vnet() to send
    a reply to the MLDv2 General Query. In case when router has a lot of
    multicast groups, the reply can take several packets due to MTU limitation.
    Also we have a limit MLD_MAX_RESPONSE_BURST == 4, that limits the number
    of packets we send in one shot. Then we recalculate the timer value and
    schedule the remaining packets for sending.
    The problem is that when we call mld_v2_dispatch_general_query() to send
    remaining packets, we queue new reply in the same mbuf queue. And when
    number of packets is bigger than MLD_MAX_RESPONSE_BURST, we get endless
    reply of MLDv2 reports.
    To fix this, add the check for remaining packets in the queue.

    PR:		204831

Changes:
_U  stable/9/sys/
  stable/9/sys/netinet6/mld6.c
Comment 11 Andrey V. Elsukov freebsd_committer 2015-12-08 08:13:18 UTC
Committed in head/, stable/10 and stable/9. Thanks!