Created attachment 222878 [details]
Change lookup of incoming packets to use both addresses
While lookup of outgoing packets uses hash based on both source and destination address, for incoming packets only alias address and port is used. So when multiple connections from different addresses target the same port of redirected address (using redirect_addr or redirect_port in IPFW), the link table must be searched sequentially - tens of thousand of items for every incoming packet. To make it worse, the search is under a lock, so it is forced to run on a single core. Consequently just 1000pps from different addresses are enough to bring down a server with the fastest CPU available in under a minute.
The attached patch fixes the problem by using both addresses and ports for lookup of incoming packets. We are running it currently on 11.2, but it applies to CURRENT without changes.