Bug 279381

Summary:

FreeBSD 13.3-RELEASE breaks isp driver with Qlogic QLE2692 16Gb fibre channel cards

Product:

Base System

Reporter:

drkspc <drkspc>

Component:

kern

Assignee:

freebsd-bugs (Nobody) <bugs>

Status:

New ---

Severity:

Affects Only Me

CC:

crest, d.stoye, jens.holzkaemper, michaelo

Priority:

---

Keywords:

regression

Version:

13.3-RELEASE

Hardware:

amd64

OS:

Any

Attachments:

Description	Flags
dmesg output for isp0 on FreeBSD 13.3 with firmware loading disabled	none
dmesg output for isp0 on FreeBSD 13.2 with firmware loading disabled	none
dmesg output for isp3 on FreeBSD 13.3	none

Description drkspc 2024-05-29 13:22:31 UTC

Created attachment 251062 [details]
dmesg output for isp0 on FreeBSD 13.3 with firmware loading disabled

We have a bunch of Dell servers with Qlogic QLE2692 16Gb FC cards, pulling in storage from a switched FC network.
Upgrading from FreeBSD 13.2 to 13.3 leaves us with all the paths to our multipath devices and thus zpools missing.

Debug output as per hint.isp.0.debug="0x70f" in /boot/device.hints looks like firmware loading is not an issue. However, on 13.3, we get a message
isp0: fabric topology, but cannot get info about fabric controller (0x400a)

Booting with hint.isp.0.debug="0x71f", this is preceeded by:
isp0: Chan 0 handle 0x7fe Port 0xfffffe flags 0x0 curstate 44 laststate 44

as opposed to (13.2):
isp0: Chan 0 handle 0x7fe Port 0xfffffe flags 0x0 curstate 4 laststate 4

Rolling back to 13.2 results in all devices/paths showing up properly again. Enabling or disabling firmware loading (hint.isp.0.fwload_disable="0" or "1") on 13.3 does not make any difference. Using even newer firmware than provided by FreeBSD on the card (9.14.0 from Dell's firmware packages) shows the same effect.

In addition to the QLE2692 card, we installed an older QLE2662 16Gb into one of the servers, as the updated firmware loading in 13.3 (bug #273263) only affects newer cards. With the older card (isp2 and isp3) everything works as expected.

I'll attach the dmesg entries for isp0 (QLE2692) on FreeBSD 13.2 and 13.3 as well as isp3 (QLE2662) on 13.3.

Comment 1 drkspc 2024-05-29 13:23:40 UTC

Created attachment 251063 [details]
dmesg output for isp0 on FreeBSD 13.2 with firmware loading disabled

Comment 2 drkspc 2024-05-29 13:24:22 UTC

Created attachment 251064 [details]
dmesg output for isp3 on FreeBSD 13.3

Comment 3 Michael Osipov freebsd_committer

2024-06-12 15:10:44 UTC

Did you check what has changed in the driver between those versions?

Comment 4 drkspc 2024-06-12 15:22:07 UTC

(In reply to Michael Osipov from comment #3)
There has been a change in how firmware is loaded for Qlogic cards (https://cgit.freebsd.org/src/commit/?id=10ed63fc06cb9902cc783ce8d0086c9aa97ed1e1), which is included in 13.3 and since last week also 14.1. See also bug #273263 for more context.

I now built a custom kernel based on the releng/13.3 branch and reverted the commits introducing revised firmware loading. Rebooting into the kernel with the "old" firmware handling, everything works again and multipath devices show up properly.

It's possible that our cards' firmware is in a non-optimal state (i.e. primary image not active). However, we've been running in this hardware/firmware configuration for years without any issues, so this is still a breaking change for us. I will try to have a closer look at the firmware state next week.