Created attachment 222878 [details] Change lookup of incoming packets to use both addresses While lookup of outgoing packets uses hash based on both source and destination address, for incoming packets only alias address and port is used. So when multiple connections from different addresses target the same port of redirected address (using redirect_addr or redirect_port in IPFW), the link table must be searched sequentially - tens of thousand of items for every incoming packet. To make it worse, the search is under a lock, so it is forced to run on a single core. Consequently just 1000pps from different addresses are enough to bring down a server with the fastest CPU available in under a minute. The attached patch fixes the problem by using both addresses and ports for lookup of incoming packets. We are running it currently on 11.2, but it applies to CURRENT without changes.
Even though libalias database was mostly rewriten, this bug is still present in 13.1. The problem wasn't in the data structure itself (hash table or splay tree), but in the key used for lookup. If the key contains only alias address and port, all packets from many addresses in DDoS targeting the same local address and port will collide in a single entry. I made a new patch working in a similar way to the previous one. I did some simple tests, but we haven't deployed it in production yet.
Created attachment 234750 [details] A similar fix for rewritten libalias in FreeBSD 13.1
Keyword: patch or patch-ready – in lieu of summary line prefix: [patch] * bulk change for the keyword * summary lines may be edited manually (not in bulk). Keyword descriptions and search interface: <https://bugs.freebsd.org/bugzilla/describekeywords.cgi>
I have got the same problem: BitTorrent client behind the NAT, where thousands of various external addresses come to single 100Mbps forwarded UDP/TCP port. Applied patch on top of 13.2 and it runs smoothly. Before that, libalias-related kernel processes ate 50-70% of my 6-core Xeon CPU. Now all of that (firewall+NAT+FQ-CoDel+gif-tunnel) eat 1% of a single core. Thanks for the patch and solution!