I have only observed this on 11.0-RELEASE on AWS as that is the only platform that I have access to, but I believe this effects all versions. uname: $ uname -ar FreeBSD redis-prod0.skippy.com 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0 r306420: Thu Sep 29 01:43:23 UTC 2016 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 Problem description: If you invoke systat with the "ifstat" flag egress traffic is not updated for non-loopback interfaces in the UI. Ingress updates correctly, and back directions update as expected on lo0. I have verified that I am not seeing this problem on 10.3-RELEASE ec2 instances. Interestingly enough I'm not seeing this issue on 12-CURRENT either.
I see this also on 11.0-RELEASE-p1 with the ixv0 interface: $ uname -a FreeBSD ip-172-30-0-105 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0 r306420: Thu Sep 29 01:43:23 UTC 2016 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 $ systat -ifstat ... ixv0 in 0.000 KB/s 0.000 KB/s 633.016 MB out 0.000 KB/s 0.000 KB/s 0.000 KB ... netstat(1) also shows zero outgoing packets/bytes on the interface: $ netstat -bn -I ixv0 Name Mtu Network Address Ipkts Ierrs Idrop Ibytes Opkts Oerrs Obytes Coll ixv0 1500 <Link#1> 02:f1:96:c7:46:19 0 0 0 663770534 0 0 0 0 ixv0 - 172.30.0.0/24 172.30.0.105 942591 - - 648858505 1050933 - 352207042 -
It was suggested that I test out r308126 to see if this addresses the issue. I have rebuilt the kernel on one of my systems using this patch but am seeing the same issue: $ uname -ar FreeBSD netfront-test.skippy.com 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0 r308150M: Mon Oct 31 21:50:43 UTC 2016 pewright@netfront-test.skippy.com:/usr/obj/usr/home/pewright/svn/11.0.1/sys/GENERIC amd64
Interface Traffic Peak Total wlan0 in 0.063 KB/s 3.513 KB/s 291.085 MB out 0.065 KB/s 1.246 KB/s 17.089 MB lo0 in 0.000 KB/s 0.000 KB/s 48.340 KB out 0.000 KB/s 0.000 KB/s 48.340 KB % uname -v FreeBSD 11.0-RELEASE-p1 I don't have seem to have issues.
(In reply to ota from comment #3) It looks like you are using a wireless interface on 11, which also works for me as well. I believe this issue is isolated to systems running under the Xen hypervisor (on AWS for example).
I am also seeing this problem on AWS It isn't specific to the tools systat -ifstat netstat -I ixv0 1 sysutils/nload all show the same behaviour
This is also consistent on smaller instances such as the t2's. And this appeared in 11.0-RELEASE. It was working perfectly fine in 10.1, 10.2 and 10.3. It happens both with ixv and xn drivers. It's not happening with Linux instances or windows instances. Making me believe this is in the FreeBSD drivers.
There is lack of statistic counters update in the ixv driver. I'm not sure and unable to test, but probably you can add several IXGBE_SET_XXX() macro in the ixv_update_stats() like in the ixgbe_update_stats_counters(). Also, probably if_setgetcounterfn() should be used to set custom if_get_counter() handler and IFCAP_HWSTATS caps added to if_capabilities.
a couple of observations as i've had a few cycles to take a look at ixgbe/if_ix.c and ixgbe/if_ixv.c source. 1) It looks like both drivers are only populating rx_bytes SYSCTL statistic, but rx and tx packets counters are defined (and are updating as per testing on my dev machines). - I've found that other drivers such as ixl do populate rx and tx byte counters. 2) the if_ixv code does *not* have any code to populate the OS statistics structure as per previous comment. There *is* code in the if_ix.c though, so perhaps it is easy to port that to the if_ixv device? My suspicion is that adding this functionality will fix userland tools. Since I'm pretty green when it comes to hacking device drivers I'm going to play with adding tx_bytes to SYSCTl and seeing how that goes. If anyone with more experience hacking on drivers wants to take a stab at getting stats in there I'd be more than happy to test :)
Any news on this? I would be happy to try out patches.
(In reply to a.andersson.thn from comment #9) I have not made much progress on my end unfortunately, like you though I'm keen to test out patches as this is effecting my production systems in AWS. For now I'm using cloudwatch metrics for network utilization metrics as a workaround but would prefer to have the OS report this data correctly.
Re-assign to more appropriate ML (freebsd-net), cc'ing original ML (virtualization).
This is fixed in https://reviews.freebsd.org/D11058
CC release engineering for 11.1-RELEASE
This still does not work.
(In reply to Jeff Pieper from comment #12) The commited fix does *not* work on 11.1-RELEASE on AWS: $ uname -ar FreeBSD snap-prod2.iad0.tribdev.com 11.1-RELEASE-p1 FreeBSD 11.1-RELEASE-p1 #0: Wed Aug 9 11:55:48 UTC 2017 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 I've confirmed on multiple C4 instances running the same official AMI.
Change title back. xn(4) was an unrelated issue that only looked similar.
(In reply to pete from comment #15) I've just encountered this missing TX issue on AWS with the xn0 driver: $ uname -v FreeBSD 11.0-RELEASE-p2 #0: Mon Oct 24 06:55:27 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC $ systat -ifstat 1 Interface Traffic Peak Total xn0 in 1.439 KB/s 2.444 KB/s 20.686 GB out 0.000 KB/s 0.000 KB/s 0.000 KB $ netstat -I xn0 -b Name Mtu Network Address Ipkts Ierrs Idrop Ibytes Opkts Oerrs Obytes Coll xn0 1500 <Link#2> 06:d0:71:b3:c1:47 296475044 0 0 22211709752 582065784 0 0 0 xn0 - 10.10.0.0/20 10.10.10.20 295927017 - - 18077218113 581504705 - 32914263942 - I'm glad to see that other people have reproduced this issue and that it's not only something that I have experienced.
xn(4) seems to be working for me on a RootBSD vm. ixlv(4) seems to be working for others and this should not longer be a problem in 12-CURRENT and the 11.2-RELEASE images. Please retest if you wish at your earliest convenience.