Bug 211872 - IPv6 UDP traffic sometimes sent using wrong mac address
Summary: IPv6 UDP traffic sometimes sent using wrong mac address
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.0-RC1
Hardware: amd64 Any
: --- Affects Some People
Assignee: Mike Karels
URL:
Keywords: patch
: 211926 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-08-15 16:54 UTC by Mike Andrews
Modified: 2016-09-11 11:35 UTC (History)
6 users (show)

See Also:
koobs: maintainer-feedback? (mandrews)
koobs: mfc-stable11+

Attachments
Proposed patch (877 bytes, patch)
2016-08-18 07:53 UTC, Mike Karels 		no flags Details | Diff
Patch to disable UDP/IPv6 L2 caching (474 bytes, patch)
2016-08-19 00:08 UTC, Mike Karels 		no flags Details | Diff
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Mike Andrews 2016-08-15 16:54:29 UTC 
Outbound IPv6 UDP traffic sometimes appears to put packets on the wire with the destination MAC address of a completely different system.  "ndp -a" shows correct MAC addresses.

Example:

Two nameservers, trying to query each other, on IPv6 private addresses (fc00::/10)

fdfa::fafa:d53a, mac 00:25:90:57:21:b3
fdfa::fafa:d53b, mac 00:25:90:38:6f:fa

These are lagg0 interfaces aggregating two Intel em nics.

On each machine, run:  tcpdump -e -n net fdfa::/16 and port 53

On machine 2, run "host -t a www.fark.com fdfa::fafa:d53a" and it will timeout, with its own tcpdump showing:
12:46:22.835604 00:25:90:38:6f:fa > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 92: fdfa::fafa:d53b.15484 > fdfa::fafa:d53a.53: 47157+ A? www.fark.com. (30)
12:46:27.836716 00:25:90:38:6f:fa > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 92: fdfa::fafa:d53b.39323 > fdfa::fafa:d53a.53: 47157+ A? www.fark.com. (30)

and machine 1's tcpdump showing a wrong MAC address used in the replies:
12:46:22.835118 00:25:90:38:6f:fa > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 92: fdfa::fafa:d53b.15484 > fdfa::fafa:d53a.53: 47157+ A? www.fark.com. (30)
12:46:22.835272 00:25:90:57:21:b3 > 00:25:90:23:be:bc, ethertype IPv6 (0x86dd), length 232: fdfa::fafa:d53a.53 > fdfa::fafa:d53b.15484: 47157* 1/2/4 A 64.191.171.200 (170)
12:46:27.836186 00:25:90:38:6f:fa > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 92: fdfa::fafa:d53b.39323 > fdfa::fafa:d53a.53: 47157+ A? www.fark.com. (30)
12:46:27.836340 00:25:90:57:21:b3 > 00:25:90:23:be:bc, ethertype IPv6 (0x86dd), length 232: fdfa::fafa:d53a.53 > fdfa::fafa:d53b.39323: 47157* 1/2/4 A 64.191.171.200 (170)

00:50:90:23:be:bc is a totally different 3rd machine.  That's.... weird. :)

Add the -T flag to the "host" command to force TCP, and it works.

Flip the queries around so that machine 1 queries machine 2, and it works.

I'm seeing similar issues with other similar machines (and similar hardware, all Supermicro machines with Intel em nics of varying vintage) that have been updated to 11.0-RC1.  All ware fine on 10.3-RELEASE.  The only em-specific tweak I've got is "-tso", but turning tso back on doesn't change behavior any.

Another weird and possibly related issue: "ndp -c" fails on machine 1 -- it deletes one or two entries and then stops with "ndp: writing to routing socket: Operation not permitted".  On machine 2, "ndp -c" completes with no problems.

I haven't yet tried dropping the lagg interface and using the em interfaces directly.
Comment 1 Mike Andrews 2016-08-15 17:09:49 UTC 
ok, on a lark I tried "ndp -nc" and it let me clear the table, where "ndp -c" doesn't.  And that clears the wrong-mac problem up temporarily.  But it will reappear later, for example after a reboot, and it's somewhat random and difficult to reproduce on demand.

Even stranger, "ndp -c" sometimes bombs out on FreeBSD 10.3 also, with the same "writing to routing socket: Invalid argument" error.  So maybe that error is an unrelated bug, or a race condition where it's trying to delete a just-expired entry or something.
Comment 2 Andrey V. Elsukov freebsd_committer freebsd_triage 2016-08-16 19:15:34 UTC 
(In reply to Mike Andrews from comment #0)
> Another weird and possibly related issue: "ndp -c" fails on machine 1 -- it
> deletes one or two entries and then stops with "ndp: writing to routing
> socket: Operation not permitted".  On machine 2, "ndp -c" completes with no
> problems.

It is interesting.
When we call `ndp -c` it does double conversion getnameinfo()+getaddrinfo() for each address. Probably for some reason your address has some ambiguous mapping from name to address and vice versa. I will try to fix this.
Comment 3 Mike Andrews 2016-08-16 19:24:10 UTC 
I found and fixed one bad DNS entry I had, and that helps "ndp -c" run correctly.  It shouldn't crash on bad DNS though.
Comment 4 Mike Andrews 2016-08-16 19:38:19 UTC 
removing lagg0 and using em0 directly changes nothing.
Comment 5 Andrey V. Elsukov freebsd_committer freebsd_triage 2016-08-17 12:55:30 UTC 
(In reply to Mike Andrews from comment #0)
> 12:46:22.835272 00:25:90:57:21:b3 > 00:25:90:23:be:bc, ethertype IPv6
> (0x86dd), length 232: fdfa::fafa:d53a.53 > fdfa::fafa:d53b.15484: 47157*
> 1/2/4 A 64.191.171.200 (170)
> 12:46:27.836186 00:25:90:38:6f:fa > 00:25:90:57:21:b3, ethertype IPv6
> (0x86dd), length 92: fdfa::fafa:d53b.39323 > fdfa::fafa:d53a.53: 47157+ A?
> www.fark.com. (30)
> 12:46:27.836340 00:25:90:57:21:b3 > 00:25:90:23:be:bc, ethertype IPv6
> (0x86dd), length 232: fdfa::fafa:d53a.53 > fdfa::fafa:d53b.39323: 47157*
> 1/2/4 A 64.191.171.200 (170)
> 
> 00:50:90:23:be:bc is a totally different 3rd machine.  That's.... weird. :)
> 
> Add the -T flag to the "host" command to force TCP, and it works.
> 
> Flip the queries around so that machine 1 queries machine 2, and it works.
> 
> I'm seeing similar issues with other similar machines (and similar hardware,
> all Supermicro machines with Intel em nics of varying vintage) that have
> been updated to 11.0-RC1.  All ware fine on 10.3-RELEASE.  The only
> em-specific tweak I've got is "-tso", but turning tso back on doesn't change
> behavior any.

Can you check `ndp -an` on the problematic machine at the time when you see wrong addresses?
Comment 6 Mike Andrews 2016-08-17 21:31:00 UTC 
(In reply to Andrey V. Elsukov from comment #5)
short answer: ndp -na shows the correct mac addresses even when UDP fails.  Again, this seems to not impact TCP or ICMP at all.

Here's another tcpdump from fdfa::fafa:d53a, showing a query from a different system...

The other system runs "host -T www.fark.com fdfa::fafa:d53a" first (TCP, which works) followed by "host www.fark.com fdfa::fafa:d53a" (UDP, which doesn't, due to the destination MAC suddenly changing).

ndp -na run immediately after that shows correct info.

# tcpdump -e -n -i lagg0 net fdfa::/16 and port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lagg0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:24:36.486572 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 94: fdfa::fafa:f9.40778 > fdfa::fafa:d53a.53: Flags [S], seq 2781749677, win 65535, options [mss 4940,nop,wscale 6,sackOK,TS val 270410290 ecr 0], length 0
17:24:36.486598 00:25:90:57:21:b3 > 00:30:48:8e:dc:ef, ethertype IPv6 (0x86dd), length 94: fdfa::fafa:d53a.53 > fdfa::fafa:f9.40778: Flags [S.], seq 1464385697, ack 2781749678, win 65535, options [mss 4940,nop,wscale 6,sackOK,TS val 1861713864 ecr 270410290], length 0
17:24:36.486767 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 86: fdfa::fafa:f9.40778 > fdfa::fafa:d53a.53: Flags [.], ack 1, win 1080, options [nop,nop,TS val 270410291 ecr 1861713864], length 0
17:24:36.486794 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 118: fdfa::fafa:f9.40778 > fdfa::fafa:d53a.53: Flags [P.], seq 1:33, ack 1, win 1080, options [nop,nop,TS val 270410291 ecr 1861713864], length 3237420+ A? www.fark.com. (30)
17:24:36.487026 00:25:90:57:21:b3 > 00:30:48:8e:dc:ef, ethertype IPv6 (0x86dd), length 258: fdfa::fafa:d53a.53 > fdfa::fafa:f9.40778: Flags [P.], seq 1:173, ack 33, win 1080, options [nop,nop,TS val 1861713864 ecr 270410291], length 17237420* 1/2/4 A 64.191.171.200 (170)
17:24:36.487243 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 86: fdfa::fafa:f9.40778 > fdfa::fafa:d53a.53: Flags [F.], seq 33, ack 173, win 1080, options [nop,nop,TS val 270410291 ecr 1861713864], length 0
17:24:36.487254 00:25:90:57:21:b3 > 00:30:48:8e:dc:ef, ethertype IPv6 (0x86dd), length 86: fdfa::fafa:d53a.53 > fdfa::fafa:f9.40778: Flags [.], ack 34, win 1080, options [nop,nop,TS val 1861713864 ecr 270410291], length 0
17:24:36.487344 00:25:90:57:21:b3 > 00:30:48:8e:dc:ef, ethertype IPv6 (0x86dd), length 86: fdfa::fafa:d53a.53 > fdfa::fafa:f9.40778: Flags [F.], seq 173, ack 34, win 1080, options [nop,nop,TS val 1861713864 ecr 270410291], length 0
17:24:36.487573 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 86: fdfa::fafa:f9.40778 > fdfa::fafa:d53a.53: Flags [.], ack 174, win 1080, options [nop,nop,TS val 270410291 ecr 1861713864], length 0
17:24:36.554081 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 94: fdfa::fafa:f9.40779 > fdfa::fafa:d53a.53: Flags [S], seq 1782349797, win 65535, options [mss 4940,nop,wscale 6,sackOK,TS val 270410358 ecr 0], length 0
17:24:36.554090 00:25:90:57:21:b3 > 00:30:48:8e:dc:ef, ethertype IPv6 (0x86dd), length 94: fdfa::fafa:d53a.53 > fdfa::fafa:f9.40779: Flags [S.], seq 67161330, ack 1782349798, win 65535, options [mss 4940,nop,wscale 6,sackOK,TS val 334898630 ecr 270410358], length 0
17:24:36.554338 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 86: fdfa::fafa:f9.40779 > fdfa::fafa:d53a.53: Flags [.], ack 1, win 1080, options [nop,nop,TS val 270410358 ecr 334898630], length 0
17:24:36.554350 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 118: fdfa::fafa:f9.40779 > fdfa::fafa:d53a.53: Flags [P.], seq 1:33, ack 1, win 1080, options [nop,nop,TS val 270410358 ecr 334898630], length 3251242+ AAAA? www.fark.com. (30)
17:24:36.554633 00:25:90:57:21:b3 > 00:30:48:8e:dc:ef, ethertype IPv6 (0x86dd), length 270: fdfa::fafa:d53a.53 > fdfa::fafa:f9.40779: Flags [P.], seq 1:185, ack 33, win 1080, options [nop,nop,TS val 334898631 ecr 270410358], length 18451242* 1/2/4 AAAA 2607:f100:3:164:fa12:1c:bee:12 (182)
17:24:36.554864 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 86: fdfa::fafa:f9.40779 > fdfa::fafa:d53a.53: Flags [F.], seq 33, ack 185, win 1080, options [nop,nop,TS val 270410360 ecr 334898631], length 0
17:24:36.554871 00:25:90:57:21:b3 > 00:30:48:8e:dc:ef, ethertype IPv6 (0x86dd), length 86: fdfa::fafa:d53a.53 > fdfa::fafa:f9.40779: Flags [.], ack 34, win 1080, options [nop,nop,TS val 334898631 ecr 270410360], length 0
17:24:36.554968 00:25:90:57:21:b3 > 00:30:48:8e:dc:ef, ethertype IPv6 (0x86dd), length 86: fdfa::fafa:d53a.53 > fdfa::fafa:f9.40779: Flags [F.], seq 185, ack 34, win 1080, options [nop,nop,TS val 334898631 ecr 270410360], length 0
17:24:36.555030 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 94: fdfa::fafa:f9.40780 > fdfa::fafa:d53a.53: Flags [S], seq 248280764, win 65535, options [mss 4940,nop,wscale 6,sackOK,TS val 270410360 ecr 0], length 0
17:24:36.555045 00:25:90:57:21:b3 > 00:30:48:8e:dc:ef, ethertype IPv6 (0x86dd), length 94: fdfa::fafa:d53a.53 > fdfa::fafa:f9.40780: Flags [S.], seq 2766214482, ack 248280765, win 65535, options [mss 4940,nop,wscale 6,sackOK,TS val 1138173303 ecr 270410360], length 0
17:24:36.555273 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 86: fdfa::fafa:f9.40779 > fdfa::fafa:d53a.53: Flags [.], ack 186, win 1080, options [nop,nop,TS val 270410360 ecr 334898631], length 0
17:24:36.555283 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 86: fdfa::fafa:f9.40780 > fdfa::fafa:d53a.53: Flags [.], ack 1, win 1080, options [nop,nop,TS val 270410360 ecr 1138173303], length 0
17:24:36.555293 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 118: fdfa::fafa:f9.40780 > fdfa::fafa:d53a.53: Flags [P.], seq 1:33, ack 1, win 1080, options [nop,nop,TS val 270410360 ecr 1138173303], length 3256258+ MX? www.fark.com. (30)
17:24:36.555462 00:25:90:57:21:b3 > 00:30:48:8e:dc:ef, ethertype IPv6 (0x86dd), length 169: fdfa::fafa:d53a.53 > fdfa::fafa:f9.40780: Flags [P.], seq 1:84, ack 33, win 1080, options [nop,nop,TS val 1138173303 ecr 270410360], length 8356258* 0/1/0 (81)
17:24:36.555751 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 86: fdfa::fafa:f9.40780 > fdfa::fafa:d53a.53: Flags [F.], seq 33, ack 84, win 1080, options [nop,nop,TS val 270410361 ecr 1138173303], length 0
17:24:36.555760 00:25:90:57:21:b3 > 00:30:48:8e:dc:ef, ethertype IPv6 (0x86dd), length 86: fdfa::fafa:d53a.53 > fdfa::fafa:f9.40780: Flags [.], ack 34, win 1080, options [nop,nop,TS val 1138173304 ecr 270410361], length 0
17:24:36.555785 00:25:90:57:21:b3 > 00:30:48:8e:dc:ef, ethertype IPv6 (0x86dd), length 86: fdfa::fafa:d53a.53 > fdfa::fafa:f9.40780: Flags [F.], seq 84, ack 34, win 1080, options [nop,nop,TS val 1138173304 ecr 270410361], length 0
17:24:36.556030 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 86: fdfa::fafa:f9.40780 > fdfa::fafa:d53a.53: Flags [.], ack 85, win 1080, options [nop,nop,TS val 270410361 ecr 1138173304], length 0
17:24:41.155082 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 92: fdfa::fafa:f9.33694 > fdfa::fafa:d53a.53: 6389+ A? www.fark.com. (30)
17:24:41.155170 00:25:90:57:21:b3 > 00:25:90:38:6f:fa, ethertype IPv6 (0x86dd), length 232: fdfa::fafa:d53a.53 > fdfa::fafa:f9.33694: 6389* 1/2/4 A 64.191.171.200 (170)
17:24:46.161268 00:30:48:8e:dc:ef > 00:25:90:57:21:b3, ethertype IPv6 (0x86dd), length 92: fdfa::fafa:f9.11621 > fdfa::fafa:d53a.53: 6389+ A? www.fark.com. (30)
17:24:46.161362 00:25:90:57:21:b3 > 00:25:90:38:6f:fa, ethertype IPv6 (0x86dd), length 232: fdfa::fafa:d53a.53 > fdfa::fafa:f9.11621: 6389* 1/2/4 A 64.191.171.200 (170)
^C
31 packets captured
73930 packets received by filter
0 packets dropped by kernel
# ndp -an | egrep 'fdfa::fafa:(d53a|f9)'
fdfa::fafa:d53a                      00:25:90:57:21:b3  lagg0 permanent R 
fdfa::fafa:f9                        00:30:48:8e:dc:ef  lagg0 39s       R
Comment 7 Andrey V. Elsukov freebsd_committer freebsd_triage 2016-08-17 21:46:21 UTC 
Since you use 11.0-RC1, can you try to revert r303698 and build and test?
You can do this with the following commands:
# cd /usr/src
# svn merge -c -303698 ^/stable/11
Comment 8 Mike Andrews 2016-08-17 23:10:16 UTC 
At first glance, r303698 does seem to be the culprit.  I'll keep running tests over the next 24 hours.
Comment 9 Mike Karels freebsd_committer freebsd_triage 2016-08-18 07:53:08 UTC 
Created attachment 173814 [details]
Proposed patch

I have attached a patch to ip6_output.c, which was missing invalidation code present in ip_output.c for v4.  Peter Wemm indicates that this is working so far.
Comment 10 Mike Karels freebsd_committer freebsd_triage 2016-08-18 07:55:22 UTC 
Assigning to myself.
Comment 11 Mike Andrews 2016-08-18 14:52:24 UTC 
Unfortunately Mike's patch (along with putting r303698 back in) doesn't fix it for me.
Comment 12 Mike Karels freebsd_committer freebsd_triage 2016-08-19 00:08:12 UTC 
Created attachment 173838 [details]
Patch to disable UDP/IPv6 L2 caching
Comment 13 Mike Karels freebsd_committer freebsd_triage 2016-08-19 00:10:10 UTC 
Mike, could you try the new patch I attached for udp6_usrreq.c in place of the previous patch?  It disables L2 caching for UDP/IPv6.
Comment 14 Kubilay Kocak freebsd_committer freebsd_triage 2016-08-19 13:32:48 UTC 
There's no non-maintainer feedback flag, so using maintainer-feedback (as a hack) to make sure Mike (Andrews) gets the message. 11.0-RC2 is blocked/delayed on this currently
Comment 15 Mike Andrews 2016-08-19 14:54:34 UTC 
Fast first glance says the one-liner patch to sys/netinet6/udp6_usrreq.c is a winner.  (Sorry for the delay, couldn't get to test it last night.)

FWIW the previous patch, while it didn't fix the problem for me, didn't appear to break anything either, so maybe both are useful if it helps w/ Peter Wemm's PR 211926.

Random data point re that PR, which I hadn't looked at until now: some of my affected systems have jails and some do not, but there are v6 alias addresses involved on all of them and I'm using a /128 mask instead of /64 in /etc/rc.conf for 'em.  If that's not best practice anymore I'll change to /64.  If you want, I can go to /64's *and* back out the patch too, if it helps narrow down a root cause better.  I suspect disabling caches is good enough for 11.0-RC2 but there may be a root cause further down that has yet to be found?
Comment 16 Mike Andrews 2016-08-19 15:02:41 UTC 
(also if there's a rush on this, since it's holding up 11.0-RC2, I can drop onto IRC at some point today and summarize back to this PR or 211926.  I'm in US/EDT timezone)
Comment 17 Mike Karels freebsd_committer freebsd_triage 2016-08-19 18:55:46 UTC 
Thanks for testing, Mike.  Putting in both patches is an option, but I imagine we'll use the simplest patch for now.
Comment 18 commit-hook freebsd_committer freebsd_triage 2016-08-20 20:47:14 UTC 
A commit references this bug:

Author: karels
Date: Sat Aug 20 20:46:54 UTC 2016
New revision: 304545
URL: https://svnweb.freebsd.org/changeset/base/304545

Log:
  Disable L2 caching for UDP over IPv6

  The ip6_output routine is missing L2 cache invalication as done
  in ip_output.  Even with that code, some problems with UDP over
  IPv6 have been reported.  Diabling L2 cache for that problem works
  around the problem for now.

  PR:		211872 211926
  Reviewed by:	gnn
  Approved by:	gnn (mentor)
  MFC after:	immediate

Changes:
  head/sys/netinet6/udp6_usrreq.c
Comment 19 commit-hook freebsd_committer freebsd_triage 2016-08-20 20:57:20 UTC 
A commit references this bug:

Author: karels
Date: Sat Aug 20 20:56:36 UTC 2016
New revision: 304546
URL: https://svnweb.freebsd.org/changeset/base/304546

Log:
  MFC r304545: Disable L2 caching for UDP over IPv6

  The ip6_output routine is missing L2 cache invalication as done
  in ip_output.  Even with that code, some problems with UDP over
  IPv6 have been reported.  Diabling L2 cache for that problem works
  around the problem for now.

  PR:		211872 211926
  Reviewed by:	gnn
  Approved by:	gnn (mentor)
  Tested by:	peter@, Mike Andrews
  MFC after:	immediate

Changes:
_U  stable/11/
  stable/11/sys/netinet6/udp6_usrreq.c
Comment 20 Mike Andrews 2016-08-22 15:56:57 UTC 
Mike gave me a revised version of his first patch, and 12 hours later it's still working fine.
Comment 21 commit-hook freebsd_committer freebsd_triage 2016-08-22 22:30:39 UTC 
A commit references this bug:

Author: karels
Date: Mon Aug 22 22:29:57 UTC 2016
New revision: 304642
URL: https://svnweb.freebsd.org/changeset/base/304642

Log:
  MFC r304546: Disable L2 caching for UDP over IPv6

  The ip6_output routine is missing L2 cache invalication as done
  in ip_output.  Even with that code, some problems with UDP over
  IPv6 have been reported.  Diabling L2 cache for that problem works
  around the problem for now.

  PR:             211872 211926
  Reviewed by:    gnn
  Approved by:    gnn (mentor)
  Approved by:    re (gjb)
  Tested by:      peter@, Mike Andrews

Changes:
_U  releng/11.0/
  releng/11.0/sys/netinet6/udp6_usrreq.c
Comment 22 Mike Karels freebsd_committer freebsd_triage 2016-08-22 22:33:04 UTC 
*** Bug 211926 has been marked as a duplicate of this bug. ***
Comment 23 Mike Karels freebsd_committer freebsd_triage 2016-08-22 22:36:31 UTC 
Workaround applied to releng/11.0: r304642.  Proper fix being evaluated, but will not be in 11.0.
Comment 24 Kubilay Kocak freebsd_committer freebsd_triage 2016-09-11 11:35:29 UTC 
Correctly record that commit was merged to stable/11