Created attachment 199442 [details]
Contains simple kernel module source and a makefile
I am seeing a FreeBSD 11.x OS poor performance issue on Xeon Gold CPU.
CPU is Intel(R) Xeon(R) Gold 5115 CPU @ 2.40GHz ( dual socket CPU with 10 cores per socket )
I have attached a simple kernel module program ( for FreeBSD) and its Makefile.
This just allocates 32KB contiguous memory using contigmalloc and then frees it using contigfree.
This repeated 1000 times and time taken by contigmalloc and contigfree is computed .
I can see that contigfree takes way too much time .
I have another older server running FreeBSD 11.2 Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (dual socket 6 core per socket and in this I do not see such an issue .
Any idea why contigfree takes so much time on FreeBSD 11.x running on Xeon Gold 5115 CPU ?
It looks like this issue affects only the systems which have more than 16 physical CPU cores.
Because of this issue the driver for our PCIe card takes more than 2 minutes to load and upto 15 minutes to unload.
System and CPU details:
CPU: Intel(R) Xeon(R) Gold 5115 CPU @ 2.40GHz (2400.07-MHz K8-class CPU)
FreeBSD/SMP: Multiprocessor System Detected: 40 CPUs
Product Name: Super Server
Base Board Information
Product Name: X11DPH-Tq
I tried contacting SuperMicro and then Intel tech support. Both suggested that the issue could be specific to FreeBSD OS.
So later I tried FreeBSD 12 Beta 3 version on this server with Xeon Gold 5115 CPU.
All these problems have disappeared. In this FreeBSD 12 Beta3 also contigfree takes much more time compared to contigmalloc.
However when I compare the absolute values to FreeBSD 11.x number I can see huge improvement in FreeBSD 12 Beta 3.
Now in FreeBSD 12 Beta3 my PCIe card driver takes just a few seconds to load and few seconds to unload.
However on FreeBSD 11.x it takes 2+ minutes to load and upto 15 minutes to unload.
Hence it looks like the problem is with FreeBSD 11.x.
Hope the attached simple program in original bug report helps you guys to reproduce this issue.