virtio is allocating the standard descriptors using contigmalloc() but the indirect descriptors using malloc(). This doesn't normally cause an issue because the malloc() happens to get contiguous memory, but I was using a different memory allocator and ran into this issue. In my scenario then the virtio network driver caused QEMU on Linux to exit. Index: sys/dev/virtio/virtqueue.c =================================================================== --- sys/dev/virtio/virtqueue.c (revision 330873) +++ sys/dev/virtio/virtqueue.c (working copy) @@ -249,7 +249,8 @@ for (i = 0; i < vq->vq_nentries; i++) { dxp = &vq->vq_descx[i]; - dxp->indirect = malloc(size, M_DEVBUF, M_NOWAIT); + dxp->indirect = contigmalloc(size, M_DEVBUF, M_NOWAIT, + 0, 0xFFFFFFFFUL, PAGE_SIZE, 0); if (dxp->indirect == NULL) { device_printf(dev, "cannot allocate indirect list\n"); return (ENOMEM); @@ -274,7 +275,7 @@ if (dxp->indirect == NULL) break; - free(dxp->indirect, M_DEVBUF); + contigfree(dxp->indirect, vq->vq_indirect_mem_size, M_DEVBUF); dxp->indirect = NULL; dxp->indirect_paddr = 0; }
See this comment in virtio.h: /* * Each virtqueue indirect descriptor list must be physically contiguous. * To allow us to malloc(9) each list individually, limit the number * supported to what will fit in one page. With 4KB pages, this is a limit * of 256 descriptors. If there is ever a need for more, we can switch to * contigmalloc(9) for the larger allocations, similar to what * bus_dmamem_alloc(9) does. * * Note the sizeof(struct vring_desc) is 16 bytes. */ #define VIRTIO_MAX_INDIRECT ((int) (PAGE_SIZE / 16)) This code is quite old but my recollection is that the minimum allocation returned by contigmalloc() is (was?) a page which ends up wasting a lot of memory. What is the "different memory allocator"? If you're indifferent to VirtIO then disabling indrect descriptors is an option. However, I believe there are a handful of other places in VirtIO where the malloc allocation is capped at one page. That being said, I don't like depending on the behavior of malloc behavior here. I'm working on VirtIO V1 support, and after that, was going to switch VirtIO to bus_dma(9) for later IOMMU/"physical" virtio devices support.
I'm actually taking the FreeBSD kernel networking components and running them in user space on another OS. I've got the kernel malloc(9) macro'd through to the normal system malloc() and on this OS the allocations are not physically contiguous. I get an address that is close to the end of one virtual page and it spans on to the next virtual page, however these virtual pages are not physically contiguous. I don't see anything on the man page for malloc(9) that specifies that it allocates from within a single page for smaller allocations. I agree that it will because of the use of the UMA layer, but I don't see it as required. This isn't a showstopper for me, I can work around it and others are unlikely to run in to it. If you just want to defer this and have it come out in the switch to bus_dma(9) that's OK.
While the malloc(9) behavior may not be documented, VirtIO is not alone in depending on the behavior. In fact, bus_dmamem_alloc(9) uses malloc(9) for <= PAGE_SIZE allocations (given suitable alignment and attributes) so you may still be susceptible to this depending on what kernel interfaces have been reimplemented. Also, note there is a strong assumpting in the network stack and drivers that mbuf clusters (including jumbo) are physically contiguous.
Closing because this is working as intended.