Created attachment 148440 [details]
( applies with -p1 on CURRENT ~r273263 )

Log:
==================================8<==================================

Add an intermediate VM_FREELIST "DMA32" between DEFAULT and ISADMA on
AMD64. Physical allocations try freelists ascending, so the effect is to
prefer DEFAULT, then DMA32, then ISADMA.

This leaves low memory available for things that need it (DMA) on
large-memory systems which statically allocate a lot (e.g. bufs).

Sponsored by:   EMC / Isilon storage division

==================================8<==================================

Historical note/context:

So, we have 96-256 GB systems, and scale nbufs up with memory. On these systems we tune for much more than 4GB in bufs. It turns out the 'buf' static array is allocated from physmem fairly early (along with some other things that scale with memory), and without this change, devices starved for 32-bit DMA space.

Internally, we had just bumped the ISADMA region from 16MB (24-bit) to 4GB. I think the attached patch with a seperate 32-bit DMA region is slightly cleaner (and avoids other devices starving out ISA BUS DMA devices, if those still exist). But, either approach fixes the problem for us. So, if people object to adding another freelist, a compromise might be renaming the ISADMA freelist to "DMA" and bumping the region up to 32 bits.

Testing on an AMD64 VM:
# sysctl vm.phys_free
vm.phys_free:
DOMAIN 0:

FREE LIST 0:

  ORDER (SIZE)  |  NUMBER
                |  POOL 0  |  POOL 1  |  POOL 2
--            -- --      -- --      -- --      --
  12 ( 16384K)  |     257  |       0  |       0
  11 (  8192K)  |       0  |       0  |       0
  10 (  4096K)  |       2  |       1  |       0
   9 (  2048K)  |       2  |       0  |       0
   8 (  1024K)  |       1  |       0  |       0
   7 (   512K)  |       1  |       0  |       0
   6 (   256K)  |       1  |       0  |       0
   5 (   128K)  |       0  |       2  |       0
   4 (    64K)  |       0  |       1  |       0
   3 (    32K)  |       4  |       1  |       0
   2 (    16K)  |       3  |       3  |       0
   1 (     8K)  |       0  |       9  |       0
   0 (     4K)  |       1  |       1  |       0

FREE LIST 1:

  ORDER (SIZE)  |  NUMBER
                |  POOL 0  |  POOL 1  |  POOL 2
--            -- --      -- --      -- --      --
  12 ( 16384K)  |     221  |       0  |       0
  11 (  8192K)  |       1  |       0  |       0
  10 (  4096K)  |       0  |       0  |       0
   9 (  2048K)  |       1  |       0  |       0
   8 (  1024K)  |       0  |       0  |       0
   7 (   512K)  |       1  |       0  |       0
   6 (   256K)  |       1  |       0  |       0
   5 (   128K)  |       1  |       0  |       0
   4 (    64K)  |       1  |       0  |       0
   3 (    32K)  |       0  |       0  |       0
   2 (    16K)  |       0  |       0  |       0
   1 (     8K)  |       0  |       0  |       0
   0 (     4K)  |       1  |       0  |       0

FREE LIST 2:

  ORDER (SIZE)  |  NUMBER
                |  POOL 0  |  POOL 1  |  POOL 2
--            -- --      -- --      -- --      --
  12 ( 16384K)  |       0  |       0  |       0
  11 (  8192K)  |       0  |       0  |       0
  10 (  4096K)  |       0  |       0  |       0
   9 (  2048K)  |       0  |       0  |       0
   8 (  1024K)  |       1  |       0  |       0
   7 (   512K)  |       0  |       0  |       0
   6 (   256K)  |       1  |       0  |       0
   5 (   128K)  |       1  |       0  |       0
   4 (    64K)  |       2  |       0  |       0
   3 (    32K)  |       2  |       0  |       0
   2 (    16K)  |       1  |       0  |       0
   1 (     8K)  |       0  |       0  |       0
   0 (     4K)  |       1  |       0  |       0