Created attachment 251062 [details] dmesg output for isp0 on FreeBSD 13.3 with firmware loading disabled We have a bunch of Dell servers with Qlogic QLE2692 16Gb FC cards, pulling in storage from a switched FC network. Upgrading from FreeBSD 13.2 to 13.3 leaves us with all the paths to our multipath devices and thus zpools missing. Debug output as per hint.isp.0.debug="0x70f" in /boot/device.hints looks like firmware loading is not an issue. However, on 13.3, we get a message isp0: fabric topology, but cannot get info about fabric controller (0x400a) Booting with hint.isp.0.debug="0x71f", this is preceeded by: isp0: Chan 0 handle 0x7fe Port 0xfffffe flags 0x0 curstate 44 laststate 44 as opposed to (13.2): isp0: Chan 0 handle 0x7fe Port 0xfffffe flags 0x0 curstate 4 laststate 4 Rolling back to 13.2 results in all devices/paths showing up properly again. Enabling or disabling firmware loading (hint.isp.0.fwload_disable="0" or "1") on 13.3 does not make any difference. Using even newer firmware than provided by FreeBSD on the card (9.14.0 from Dell's firmware packages) shows the same effect. In addition to the QLE2692 card, we installed an older QLE2662 16Gb into one of the servers, as the updated firmware loading in 13.3 (bug #273263) only affects newer cards. With the older card (isp2 and isp3) everything works as expected. I'll attach the dmesg entries for isp0 (QLE2692) on FreeBSD 13.2 and 13.3 as well as isp3 (QLE2662) on 13.3.
Created attachment 251063 [details] dmesg output for isp0 on FreeBSD 13.2 with firmware loading disabled
Created attachment 251064 [details] dmesg output for isp3 on FreeBSD 13.3
Did you check what has changed in the driver between those versions?
(In reply to Michael Osipov from comment #3) There has been a change in how firmware is loaded for Qlogic cards (https://cgit.freebsd.org/src/commit/?id=10ed63fc06cb9902cc783ce8d0086c9aa97ed1e1), which is included in 13.3 and since last week also 14.1. See also bug #273263 for more context. I now built a custom kernel based on the releng/13.3 branch and reverted the commits introducing revised firmware loading. Rebooting into the kernel with the "old" firmware handling, everything works again and multipath devices show up properly. It's possible that our cards' firmware is in a non-optimal state (i.e. primary image not active). However, we've been running in this hardware/firmware configuration for years without any issues, so this is still a breaking change for us. I will try to have a closer look at the firmware state next week.
Can you please check if this commit https://cgit.freebsd.org/src/commit/sys/dev/isp?id=137b004e2b7ab504abf98c4aad9d52607df47b9a or using 13.4 helps in your case?
(In reply to Joerg Pulz from comment #5) Looks good so far, both 13.3 with the additional patch and 13.4 do the trick on our test server.
We updated all of our servers to 13.4 by now and everything is running fine. I'll close the issue for now. Feel free to reopen in case of ongoing issues in this context.
MARKED AS SPAM