279830 – Using AMD EPYC 9374F 32-Core Processor, PF driver gives error when loading for more than 62 VFs

Bug 279830 - Using AMD EPYC 9374F 32-Core Processor, PF driver gives error when loading for more than 62 VFs

Summary: Using AMD EPYC 9374F 32-Core Processor, PF driver gives error when loading fo...

Status:	New

Alias:	None

Product:	Base System
Classification:	Unclassified
Component:	kern (show other bugs)
Version:	14.0-RELEASE
Hardware:	amd64 Any

Importance:	--- Affects Only Me
Assignee:	freebsd-pf (Nobody)

URL:
Keywords:

Depends on:
Blocks:

Reported:	2024-06-18 13:33 UTC by vdubey
Modified:	2024-06-25 07:00 UTC (History)
CC List:	1 user (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description vdubey 2024-06-18 13:33:12 UTC

I am running on a Supermicro server which has 2xAMD EPYC 9374F 32-Core Processor and amd64 architecture. I have written a driver to use my own PCIe device in SR-IOV environment. This device supports up to 64 VFs. 
I have set num_vfs to 62 in my iovctl.conf is below:
PF {
    device : "dre_drv0";
    num_vfs : 62;
} 
DEFAULT {
    passthrough : true;
}
After this, sudo iovctl -C -f /etc/iovctl.conf will load PF driver successfully.
dmesg is as below:
dre_drv0: DRE_drvIovInit: Called with num_vfs 62.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 0.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 1.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 2.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 3.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 4.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 5.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 6.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 7.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 8.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 9.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 10.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 11.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 12.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 13.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 14.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 15.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 16.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 17.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 18.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 19.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 20.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 21.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 22.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 23.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 24.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 25.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 26.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 27.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 28.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 29.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 30.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 31.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 32.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 33.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 34.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 35.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 36.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 37.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 38.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 39.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 40.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 41.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 42.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 43.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 44.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 45.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 46.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 47.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 48.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 49.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 50.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 51.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 52.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 53.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 54.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 55.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 56.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 57.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 58.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 59.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 60.
dre_drv0: DRE_drvIovAddVf: Called for vfnum 61.
ppt0 at device 0.128 numa-domain 0 on pci9
ppt1 at device 0.129 numa-domain 0 on pci9
ppt2 at device 0.130 numa-domain 0 on pci9
ppt3 at device 0.131 numa-domain 0 on pci9
ppt4 at device 0.132 numa-domain 0 on pci9
ppt5 at device 0.133 numa-domain 0 on pci9
ppt6 at device 0.134 numa-domain 0 on pci9
ppt7 at device 0.135 numa-domain 0 on pci9
ppt8 at device 0.136 numa-domain 0 on pci9
ppt9 at device 0.137 numa-domain 0 on pci9
ppt10 at device 0.138 numa-domain 0 on pci9
ppt11 at device 0.139 numa-domain 0 on pci9
ppt12 at device 0.140 numa-domain 0 on pci9
ppt13 at device 0.141 numa-domain 0 on pci9
ppt14 at device 0.142 numa-domain 0 on pci9
ppt15 at device 0.143 numa-domain 0 on pci9
ppt16 at device 0.144 numa-domain 0 on pci9
ppt17 at device 0.145 numa-domain 0 on pci9
ppt18 at device 0.146 numa-domain 0 on pci9
ppt19 at device 0.147 numa-domain 0 on pci9
ppt20 at device 0.148 numa-domain 0 on pci9
ppt21 at device 0.149 numa-domain 0 on pci9
ppt22 at device 0.150 numa-domain 0 on pci9
ppt23 at device 0.151 numa-domain 0 on pci9
ppt24 at device 0.152 numa-domain 0 on pci9
ppt25 at device 0.153 numa-domain 0 on pci9
ppt26 at device 0.154 numa-domain 0 on pci9
ppt27 at device 0.155 numa-domain 0 on pci9
ppt28 at device 0.156 numa-domain 0 on pci9
ppt29 at device 0.157 numa-domain 0 on pci9
ppt30 at device 0.158 numa-domain 0 on pci9
ppt31 at device 0.159 numa-domain 0 on pci9
ppt32 at device 0.160 numa-domain 0 on pci9
ppt33 at device 0.161 numa-domain 0 on pci9
ppt34 at device 0.162 numa-domain 0 on pci9
ppt35 at device 0.163 numa-domain 0 on pci9
ppt36 at device 0.164 numa-domain 0 on pci9
ppt37 at device 0.165 numa-domain 0 on pci9
ppt38 at device 0.166 numa-domain 0 on pci9
ppt39 at device 0.167 numa-domain 0 on pci9
ppt40 at device 0.168 numa-domain 0 on pci9
ppt41 at device 0.169 numa-domain 0 on pci9
ppt42 at device 0.170 numa-domain 0 on pci9
ppt43 at device 0.171 numa-domain 0 on pci9
ppt44 at device 0.172 numa-domain 0 on pci9
ppt45 at device 0.173 numa-domain 0 on pci9
ppt46 at device 0.174 numa-domain 0 on pci9
ppt47 at device 0.175 numa-domain 0 on pci9
ppt48 at device 0.176 numa-domain 0 on pci9
ppt49 at device 0.177 numa-domain 0 on pci9
ppt50 at device 0.178 numa-domain 0 on pci9
ppt51 at device 0.179 numa-domain 0 on pci9
ppt52 at device 0.180 numa-domain 0 on pci9
ppt53 at device 0.181 numa-domain 0 on pci9
ppt54 at device 0.182 numa-domain 0 on pci9
ppt55 at device 0.183 numa-domain 0 on pci9
ppt56 at device 0.184 numa-domain 0 on pci9
ppt57 at device 0.185 numa-domain 0 on pci9
ppt58 at device 0.186 numa-domain 0 on pci9
ppt59 at device 0.187 numa-domain 0 on pci9
ppt60 at device 0.188 numa-domain 0 on pci9
ppt61 at device 0.189 numa-domain 0 on pci9

But when i change iovctl.conf to use num_vfs to 64. But after i execute sudo iovctl -C -f /etc/iovctl.conf, it is failing with below error message:

dre_drv0: DRE_drvIovInit: Called with num_vfs 64.
dre_drv0: 0x2000000 bytes of rid 0x264 res 3 failed (0, 0xffffffffffffffff).
dre_drv0: DRE_drvIovUnInit: Called.
Seems like some memory is not able to be allocated. 

I am using the driver code on another server which has Intel CPU and everything works fine with PF driver able to load with 64 VFs on FreeBSD 14.

Can you help to check why it is unable to load with 64 VFs on AMD CPU.

Comment 1 Santiago Martinez 2024-06-19 15:46:27 UTC

Hi there, 
I just checked on an AMD based machine and can create 64VF.

sysctl hw.vmm.ppt.devices
hw.vmm.ppt.devices: 64

Now based, on your output, if it starts from ppt0 then ppt 64 should not be called right?

Comment 2 Santiago Martinez 2024-06-19 15:52:23 UTC

Just to add more info, I have created 64VF on ixl1 (before was on ixl0) getting a total of 128VF without any issues.

We aware that pci-pass on AMD EPYC is not stable at all, most probably your machine will just reboot when a vf or pf is passed to a bhyve vm.

Comment 3 vdubey 2024-06-24 08:37:23 UTC

Hi Santiago, Thanks for the response.

I had added more verbose logging for my device.

When num_vfs=62. Below is the log:
dre_drv0: DRE_drvIovInit: Called with num_vfs 62.
pcib15: allocated memory range (0xf9000000-0xf93dffff) for rid 254 of dre_drv0
dre_drv0: Lazy allocation of 0x3e0000 bytes rid 0x254 type 3 at 0xf9000000
pcib15: allocated memory range (0xf93e0000-0xf94d7fff) for rid 260 of dre_drv0
dre_drv0: Lazy allocation of 0xf8000 bytes rid 0x260 type 3 at 0xf93e0000
pcib15: allocated memory range (0xf9500000-0xfb3fffff) for rid 264 of dre_drv0
dre_drv0: Lazy allocation of 0x1f00000 bytes rid 0x264 type 3 at 0xf9500000
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=128
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 0.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=129
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 1.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=130
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 2.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=131
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 3.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=132
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 4.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=133
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 5.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=134
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 6.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=135
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 7.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=136
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 8.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=137
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 9.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=138
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 10.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=139
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 11.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=140
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 12.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=141
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 13.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=142
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 14.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=143
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 15.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=144
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 16.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=145
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 17.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=146
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 18.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=147
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 19.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=148
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 20.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=149
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 21.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=150
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 22.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=151
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 23.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=152
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 24.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=153
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 25.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=154
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 26.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=155
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 27.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=156
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 28.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=157
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 29.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=158
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 30.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=159
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 31.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=160
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 32.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=161
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 33.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=162
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 34.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=163
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 35.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=164
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 36.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=165
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 37.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=166
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 38.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=167
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 39.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=168
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 40.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=169
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 41.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=170
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 42.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=171
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 43.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=172
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 44.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=173
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 45.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=174
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 46.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=175
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 47.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=176
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 48.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=177
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 49.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=178
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 50.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=179
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 51.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=180
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 52.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=181
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 53.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=182
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 54.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=183
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 55.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=184
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 56.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=185
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 57.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=186
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 58.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=187
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 59.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=188
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 60.
found->	vendor=0x1e43, dev=0x8900, revid=0x00
	domain=0, bus=33, slot=0, func=189
	class=12-00-00, hdrtype=0x00, mfdev=0
	cmdreg=0x0000, statreg=0x0010, cachelnsz=0 (dwords)
	lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
	MSI supports 1 message, 64 bit, vector masks
	MSI-X supports 131 messages in map 0x1c
dre_drv0: DRE_drvIovAddVf: Called for vfnum 61.
ppt0 at device 0.128 numa-domain 0 on pci15
ppt0: attached
ppt1 at device 0.129 numa-domain 0 on pci15
ppt1: attached
ppt2 at device 0.130 numa-domain 0 on pci15
ppt2: attached
ppt3 at device 0.131 numa-domain 0 on pci15
ppt3: attached
ppt4 at device 0.132 numa-domain 0 on pci15
ppt4: attached
ppt5 at device 0.133 numa-domain 0 on pci15
ppt5: attached
ppt6 at device 0.134 numa-domain 0 on pci15
ppt6: attached
ppt7 at device 0.135 numa-domain 0 on pci15
ppt7: attached
ppt8 at device 0.136 numa-domain 0 on pci15
ppt8: attached
ppt9 at device 0.137 numa-domain 0 on pci15
ppt9: attached
ppt10 at device 0.138 numa-domain 0 on pci15
ppt10: attached
ppt11 at device 0.139 numa-domain 0 on pci15
ppt11: attached
ppt12 at device 0.140 numa-domain 0 on pci15
ppt12: attached
ppt13 at device 0.141 numa-domain 0 on pci15
ppt13: attached
ppt14 at device 0.142 numa-domain 0 on pci15
ppt14: attached
ppt15 at device 0.143 numa-domain 0 on pci15
ppt15: attached
ppt16 at device 0.144 numa-domain 0 on pci15
ppt16: attached
ppt17 at device 0.145 numa-domain 0 on pci15
ppt17: attached
ppt18 at device 0.146 numa-domain 0 on pci15
ppt18: attached
ppt19 at device 0.147 numa-domain 0 on pci15
ppt19: attached
ppt20 at device 0.148 numa-domain 0 on pci15
ppt20: attached
ppt21 at device 0.149 numa-domain 0 on pci15
ppt21: attached
ppt22 at device 0.150 numa-domain 0 on pci15
ppt22: attached
ppt23 at device 0.151 numa-domain 0 on pci15
ppt23: attached
ppt24 at device 0.152 numa-domain 0 on pci15
ppt24: attached
ppt25 at device 0.153 numa-domain 0 on pci15
ppt25: attached
ppt26 at device 0.154 numa-domain 0 on pci15
ppt26: attached
ppt27 at device 0.155 numa-domain 0 on pci15
ppt27: attached
ppt28 at device 0.156 numa-domain 0 on pci15
ppt28: attached
ppt29 at device 0.157 numa-domain 0 on pci15
ppt29: attached
ppt30 at device 0.158 numa-domain 0 on pci15
ppt30: attached
ppt31 at device 0.159 numa-domain 0 on pci15
ppt31: attached
ppt32 at device 0.160 numa-domain 0 on pci15
ppt32: attached
ppt33 at device 0.161 numa-domain 0 on pci15
ppt33: attached
ppt34 at device 0.162 numa-domain 0 on pci15
ppt34: attached
ppt35 at device 0.163 numa-domain 0 on pci15
ppt35: attached
ppt36 at device 0.164 numa-domain 0 on pci15
ppt36: attached
ppt37 at device 0.165 numa-domain 0 on pci15
ppt37: attached
ppt38 at device 0.166 numa-domain 0 on pci15
ppt38: attached
ppt39 at device 0.167 numa-domain 0 on pci15
ppt39: attached
ppt40 at device 0.168 numa-domain 0 on pci15
ppt40: attached
ppt41 at device 0.169 numa-domain 0 on pci15
ppt41: attached
ppt42 at device 0.170 numa-domain 0 on pci15
ppt42: attached
ppt43 at device 0.171 numa-domain 0 on pci15
ppt43: attached
ppt44 at device 0.172 numa-domain 0 on pci15
ppt44: attached
ppt45 at device 0.173 numa-domain 0 on pci15
ppt45: attached
ppt46 at device 0.174 numa-domain 0 on pci15
ppt46: attached
ppt47 at device 0.175 numa-domain 0 on pci15
ppt47: attached
ppt48 at device 0.176 numa-domain 0 on pci15
ppt48: attached
ppt49 at device 0.177 numa-domain 0 on pci15
ppt49: attached
ppt50 at device 0.178 numa-domain 0 on pci15
ppt50: attached
ppt51 at device 0.179 numa-domain 0 on pci15
ppt51: attached
ppt52 at device 0.180 numa-domain 0 on pci15
ppt52: attached
ppt53 at device 0.181 numa-domain 0 on pci15
ppt53: attached
ppt54 at device 0.182 numa-domain 0 on pci15
ppt54: attached
ppt55 at device 0.183 numa-domain 0 on pci15
ppt55: attached
ppt56 at device 0.184 numa-domain 0 on pci15
ppt56: attached
ppt57 at device 0.185 numa-domain 0 on pci15
ppt57: attached
ppt58 at device 0.186 numa-domain 0 on pci15
ppt58: attached
ppt59 at device 0.187 numa-domain 0 on pci15
ppt59: attached
ppt60 at device 0.188 numa-domain 0 on pci15
ppt60: attached
ppt61 at device 0.189 numa-domain 0 on pci15
ppt61: attached


But, when I set num_vfs=64, we get the following log:
dre_drv0: DRE_drvIovInit: Called with num_vfs 64.
pcib15: allocated memory range (0xf9000000-0xf93fffff) for rid 254 of dre_drv0
dre_drv0: Lazy allocation of 0x400000 bytes rid 0x254 type 3 at 0xf9000000
pcib15: allocated memory range (0xf9400000-0xf94fffff) for rid 260 of dre_drv0
dre_drv0: Lazy allocation of 0x100000 bytes rid 0x260 type 3 at 0xf9400000
pcib15: attempting to grow memory window for (0-0xffffffff,0x2000000)
	front candidate range: 0xf7000000-0xf8ffffff
	back candidate range: 0xfb580000-0xfd57ffff
dre_drv0: 0x2000000 bytes of rid 0x264 res 3 failed (0, 0xffffffffffffffff).
dre_drv0: DRE_drvIovUnInit: Called.

Can you give us an idea on what "attempting to grow memory window for (0-0xffffffff,0x2000000)" is trying to do.

Comment 4 vdubey 2024-06-25 07:00:20 UTC

Also, I had further debugged this issue.

I had run 64 VFs on server with Intel CPUs (Intel 6342) which can successfully load my PF driver. The log is as follows:

dre_drv0: DRE_drvIovInit: Called with num_vfs 64.
pcib7: allocated memory range (0xa3d00000-0xa40fffff) for rid 254 of dre_drv0
dre_drv0: Lazy allocation of 0x400000 bytes rid 0x254 type 3 at 0xa3d00000
pcib7: allocated memory range (0xa4100000-0xa41fffff) for rid 260 of dre_drv0
dre_drv0: Lazy allocation of 0x100000 bytes rid 0x260 type 3 at 0xa4100000
pcib7: attempting to grow memory window for (0-0xffffffff,0x2000000)
	front candidate range: 0xa1d00000-0xa3cfffff
	back candidate range: 0xa6280000-0xa827ffff
pcib7: grew memory window to 0xa1d00000-0xa62fffff
pcib7: allocated memory range (0xa1d00000-0xa3cfffff) for rid 264 of dre_drv0
dre_drv0: Lazy allocation of 0x2000000 bytes rid 0x264 type 3 at 0xa1d00000

In this case, as we can see that it was successfully able to "grew memory window to 0xa1d00000-0xa62fffff". But in the case of AMD CPU, we are not able to grow the memory space and get an error as seen in log below:

dre_drv0: DRE_drvIovInit: Called with num_vfs 64.
pcib15: allocated memory range (0xf9000000-0xf93fffff) for rid 254 of dre_drv0
dre_drv0: Lazy allocation of 0x400000 bytes rid 0x254 type 3 at 0xf9000000
pcib15: allocated memory range (0xf9400000-0xf94fffff) for rid 260 of dre_drv0
dre_drv0: Lazy allocation of 0x100000 bytes rid 0x260 type 3 at 0xf9400000
pcib15: attempting to grow memory window for (0-0xffffffff,0x2000000)
	front candidate range: 0xf7000000-0xf8ffffff
	back candidate range: 0xfb580000-0xfd57ffff
dre_drv0: 0x2000000 bytes of rid 0x264 res 3 failed (0, 0xffffffffffffffff).
dre_drv0: DRE_drvIovUnInit: Called.