Bug 253261

I am trying to move all my AWS systems back from Amazon Linux to FreeBSD. But I encounter massive slowness.

People have reported about slow ENA network driver before. 
Even worse is the performance of the NVMe nvd disks. This is so bad that my attempt to go with FreeBSD is now completely dead. And it makes me really sad.

FreeBSD was once "the power to serve" and now it feels like I'm running Ultrix on a VAX for the fun of it but for any real work I need Linux. So sad!

The issue can be reproduced by anyone on AWS. Just get a FreeBSD 12 (latest) instance and allocate a gp2 of 1 TB or a gp3 of smaller size for which you specify 3000 IOPS and 250 MB/s IO capacity. Let's say that disk is /dev/nvd2

Now do this:

dd if=/dev/zero of=/dev/nvd2 bs=100M status=progress

and you see that it's writing with the whopping speed of 70 MB/s. 

That used to be good, but it is no longer good. Compare Amazon Linux doing the same thing at 300 MB/s.

Now, when you put a file system over it, zfs or ufs, then instantly the performance gets better:

newfs /dev/nvd2
mount /dev/nvd2 /mnt
dd if=/dev/zero of=/mnt/test bs=100M status=progress

now that works at about 250 MB/s. Decent. So, problem solved?

No! It turns out if I create a PostgreSQL database over this setup, then again there is massive delay on the read and write and throughput will drop to even worse than 70 MB/s. Creating one index takes 10 times as long as that same on the Linux system.

The dd issue is probably the best manifestation for fixing at least part of the problem. 

Is this about tuning? Even if it was, there is a severe problem with how the system is set up by default. But I'm afraid it's not even about tuning so bad is it.