Bug 216523

Summary: FreeBSD router throughput 1/3 of total available
Product: Base System Reporter: Michael Cress <michael.cress>
Component: armAssignee: freebsd-arm (Nobody) <freebsd-arm>
Status: New ---    
Severity: Affects Some People CC: darkfiberiru
Priority: ---    
Version: 11.0-STABLE   
Hardware: arm   
OS: Any   

Description Michael Cress 2017-01-27 10:28:29 UTC
On an original Model B Raspberry pi running the FreeBSD-11.0-STABLE-arm-armv6-RPI-B-20161209-r309771.img with an additional D-Link 10/100 USB NIC, my Internet speed test results are 1/3 of what they are going through my cheap Linksys wireless router via a wired connection through a 10/100 switch. Although IPFW is configured to provided NAT services, no other QoS settings were set.

Test A setup (Results: 8-9 Mb/s down, 2 Mb/s up):
Computer ->  (ue1 interface...DLink USB NIC) -> RPi/FreeBSD router/firewall -> (ue0 interface...RPi native)-> cable modem

Test B setup (Results: 27-29 Mb/s down, 2.7-3 Mb/s up):
Computer -> 10/100 switch -> Linksys wireless router (via wired connection) -> cable modem


The CPU utilization of the RPi in Test A varies between 2-7% during the test.

netstat -i ue1:

Name  Mtu   Network       Address            Ipkts Ierrs    Opkts Oerrs  Coll
lo0   16384 <Link#1>                         44469     0    44469     0     0
lo0   16384 127           localhost          44469     -    44469     -     -
lo0   16384 localhost   ::1                  44469     -    44469     -     -
lo0   16384 fe80::1%lo0 fe80:1::1            44469     -    44469     -     -
gif0* 1280  <Link#2>                             0     0        0     0     0
stf0* 1280  <Link#3>                             0     0        0     0     0
en0   1500  <Link#4>    a8:20:66:4f:e3:7c  3615853     0  1647904     0     0
en0   1500  michaels-im fe80:4::a5:91b4:f  3615853     -  1647904     -     -
en0   1500  192.168.4     192.168.4.20     3615853     -  1647904     -     -
en0   1500  fd1d:61d4:a fd1d:61d4:a56d::c  3615853     -  1647904     -     -
en0   1500  fd1d:61d4:a fd1d:61d4:a56d::8  3615853     -  1647904     -     -
en1   1500  <Link#5>    8c:2d:aa:41:4e:77        0     0        0     0     0
en2   1500  <Link#6>    32:00:14:2c:25:e0        0     0        0     0     0
en3   1500  <Link#7>    32:00:14:2c:25:e1        0     0        0     0     0
p2p0* 2304  <Link#8>    0e:2d:aa:41:4e:77        0     0        0     0     0
awdl0 1484  <Link#9>    82:ab:eb:d9:f7:5b        0     0        2     0     0
bridg 1500  <Link#10>   32:00:14:2c:25:e0        0     0        1     0     0
utun0 2000  <Link#11>                            0     0        3     0     0
utun0 2000  fe80::c960: fe80:b::c960:3443        0     -        3     -     -



netstat -i ue0

Name  Mtu   Network       Address            Ipkts Ierrs    Opkts Oerrs  Coll
lo0   16384 <Link#1>                         44474     0    44474     0     0
lo0   16384 127           localhost          44474     -    44474     -     -
lo0   16384 localhost   ::1                  44474     -    44474     -     -
lo0   16384 fe80::1%lo0 fe80:1::1            44474     -    44474     -     -
gif0* 1280  <Link#2>                             0     0        0     0     0
stf0* 1280  <Link#3>                             0     0        0     0     0
en0   1500  <Link#4>    a8:20:66:4f:e3:7c  3624424     0  1650780     0     0
en0   1500  michaels-im fe80:4::a5:91b4:f  3624424     -  1650780     -     -
en0   1500  192.168.4     192.168.4.20     3624424     -  1650780     -     -
en0   1500  fd1d:61d4:a fd1d:61d4:a56d::c  3624424     -  1650780     -     -
en0   1500  fd1d:61d4:a fd1d:61d4:a56d::8  3624424     -  1650780     -     -
en1   1500  <Link#5>    8c:2d:aa:41:4e:77        0     0        0     0     0
en2   1500  <Link#6>    32:00:14:2c:25:e0        0     0        0     0     0
en3   1500  <Link#7>    32:00:14:2c:25:e1        0     0        0     0     0
p2p0* 2304  <Link#8>    0e:2d:aa:41:4e:77        0     0        0     0     0
awdl0 1484  <Link#9>    82:ab:eb:d9:f7:5b        0     0        2     0     0
bridg 1500  <Link#10>   32:00:14:2c:25:e0        0     0        1     0     0
utun0 2000  <Link#11>                            0     0        3     0     0
utun0 2000  fe80::c960: fe80:b::c960:3443        0     -        3     -     -
Comment 1 Michael Cress 2017-01-27 10:31:25 UTC
Clarification:

The CPU utilization of the RPi in Test A varies between 2-7% during the test as measured by top. In other words: top's CPU usage read 2% user to 7% user (peak value observed).
Comment 2 Michael Cress 2017-01-27 18:56:47 UTC
To verify that the RPi hardware is up to the task, I ran the test using the latest Raspbian (Debian) image and the numbers reflected those seen during the original test using the wireless router. This confirms that the issue lies somewhere in the FreeBSD system code/configuration.
Comment 3 Nick Wolff 2017-01-27 20:57:45 UTC
I can't say a whole lot about your specific case. There maybe some people with much better answers in terms to hardware or other limitations. But if you wanted to do a fairly comprehensive test setup this is where I would start

First off your going to want to run top -S so that you can see system processes. Your network path/fw etc should all be in the kernel. 

But for further test case I would a use something like iperf for end to end testing.
https://www.veritas.com/support/en_US/article.000066021

Doing multiple test runs each time (maybe 5 as a low number) using udp and tcp For a sustained period (5 minutes). Don't forget to set speed of udp tests at 100mbps. Also be aware iperf is usually iperf2 and iperf3 is a similar but not connected but not compatible descendent

I would do the above for the following:
raspbian end to end iperf test
freebsd no ipfw loaded test
freebsd ipfw but open no nat
freebsd ipfw nat on

This would come out to about 40 iperf tests.

I realize at 5 minutes that is 4-5 hours of work depending on test setup time and other restraints. But you should be able to write a set of scripts to cut down on configuration and save the results fairly simply.



FYI: 
Both the onboard ethernet and your usb dongle are going over the same usb root hub. While I get you can push this traffic in debian it's another place to be aware you could be running into issues.