Bug 30965
| Summary: | Cyclades Cyclom-Yep causes FreeBSD to hang during boot | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Scott Klement <klemscot> |
| Component: | i386 | Assignee: | freebsd-bugs (Nobody) <bugs> |
| Status: | Closed FIXED | ||
| Severity: | Affects Only Me | ||
| Priority: | Normal | ||
| Version: | 4.4-RELEASE | ||
| Hardware: | Any | ||
| OS: | Any | ||
On Thu, 27 Sep 2001, Scott Klement wrote: > The same problem occurs in 4.2-RELEASE and 4.3-RELEASE as well. > Problem has occurred on Dell OptiPlex GX100 and GX150. > > > > >Description: > System will stop responding ("hard-lock") while probing devices > as part of the booting process whenever a Cyclades Cyclom-Y/PCI > card is installed & configured in the kernel. -current has a small chance of working better (it has an updated pci probe for this driver), but I suspect the problem is with interrupt configuration and the driver doesn't have any significant changes related to that. I have seen hangs like this under Linux but not under FreeBSD. Linux hung on a BP6 motherboard when the YeP interrupt was shared with the second disk controller. This was presumably because the ide disk driver and/or the cy driver under Linux couldn't handle shared interrupts. I spent a lot of time shuffling pci cards to unshare the relevant interrupts (the BP6 BIOS strongly prefered to map the YeP interrupt to a bad place, perhaps because the YeP claims to be a "simple" serial card). Perhaps FreeBSD has the same problem but with a different driver (OTOH, a problem with interrupt sharing in the FreeBSD ata disk driver was fixed very recently). > The same symptom occurs both when "options CY_PCI_FASTINTR" is > enabled in the kernel and also when it is not enabled. This shouldn't matter much, but debug without it anyway. > I believe that this is a software issue because the problem > does NOT occur under RedHat Linux, on the same machines. I've > tried it with 3 different (known working) Cyclades units, > and with 3 different computers (2 Dell Optiplex GX100, 1 GX150) > > The same Cyclades units do work under FreeBSD with a different > (non-Dell) computer system. You will have to debug this further by finding the relevant differences. Apart from interrupt configuration, it's possible that the Dell BIOS maps something in a way that the driver doesn't understand, but this is less likely. Bruce On Thu, 18 Oct 2001, Bruce Evans wrote: > On Thu, 27 Sep 2001, Scott Klement wrote: > > > The same problem occurs in 4.2-RELEASE and 4.3-RELEASE as well. > > Problem has occurred on Dell OptiPlex GX100 and GX150. > > > > >Description: > > System will stop responding ("hard-lock") while probing devices > > as part of the booting process whenever a Cyclades Cyclom-Y/PCI > > card is installed & configured in the kernel. > > -current has a small chance of working better (it has an updated pci > probe for this driver), but I suspect the problem is with interrupt > configuration and the driver doesn't have any significant changes > related to that. I'll have to try upgrading to -current, then. I'll let you know how that turns out. > I have seen hangs like this under Linux but not under FreeBSD. Linux > hung on a BP6 motherboard when the YeP interrupt was shared with the > second disk controller. This was presumably because the ide disk > driver and/or the cy driver under Linux couldn't handle shared interrupts. > I spent a lot of time shuffling pci cards to unshare the relevant > interrupts (the BP6 BIOS strongly prefered to map the YeP interrupt > to a bad place, perhaps because the YeP claims to be a "simple" serial > card). Perhaps FreeBSD has the same problem but with a different driver > (OTOH, a problem with interrupt sharing in the FreeBSD ata disk driver > was fixed very recently). I've set this computer up for dual-boot between RedHat 7.1 and FreeBSD. The setup works, as-is, in RedHat. (I don't know if saying that helps you troubleshoot the issue at all.) The BIOS does not show any IRQ conflicts. The dmesg also does not show any IRQ conflicts. (But perhaps something is responding on IRQs that it shouldnt? I wish I knew more about how this works...) > > The same symptom occurs both when "options CY_PCI_FASTINTR" is > > enabled in the kernel and also when it is not enabled. > > This shouldn't matter much, but debug without it anyway. > > > I believe that this is a software issue because the problem > > does NOT occur under RedHat Linux, on the same machines. I've > > tried it with 3 different (known working) Cyclades units, > > and with 3 different computers (2 Dell Optiplex GX100, 1 GX150) > > > > The same Cyclades units do work under FreeBSD with a different > > (non-Dell) computer system. > > You will have to debug this further by finding the relevant differences. > Apart from interrupt configuration, it's possible that the Dell BIOS > maps something in a way that the driver doesn't understand, but this is > less likely. > > Bruce I'd be happy to search for the relevant differences -- but I've tried everything that I can think of. Troubleshooting down at the driver level is a bit over my head. :( I've already tried removing or disabling all of the PCI devices on the machine except for video and IDE, but it still hangs. I can certainly send you a dmesg if that will help. I could also send the e-mail conversation that I had with the Tech Support person at Cyclades about this.. would that help? On Fri, 19 Oct 2001, Scott Klement wrote: > The BIOS does not show any IRQ conflicts. The dmesg also does not show any > IRQ conflicts. (But perhaps something is responding on IRQs that it > shouldnt? I wish I knew more about how this works...) Do you mean it desn't show any shared IRQs? PCI IRQs can't really conflict. Shareing them is supposed to work. > I can certainly send you a dmesg if that will help. I could also send > the e-mail conversation that I had with the Tech Support person at > Cyclades about this.. would that help? Start with the complete dmesg for booting with -v, and "pciconf -lv" output. Did the Tech Support seem helpful? Bruce On Sun, 21 Oct 2001, Bruce Evans wrote: > On Fri, 19 Oct 2001, Scott Klement wrote: > > > The BIOS does not show any IRQ conflicts. The dmesg also does not show any > > IRQ conflicts. (But perhaps something is responding on IRQs that it > > shouldnt? I wish I knew more about how this works...) > > Do you mean it desn't show any shared IRQs? PCI IRQs can't really conflict. > Shareing them is supposed to work. > Errr.. sorry, you're right. I meant that no two devices are assigned the same IRQ number -- i.e. nothing is shared. I have to disable devices to get it to that point, where nothing is shared, but doing so does not prevent FreeBSD from hanging during boot. > > I can certainly send you a dmesg if that will help. I could also send > > the e-mail conversation that I had with the Tech Support person at > > Cyclades about this.. would that help? > > Start with the complete dmesg for booting with -v, and "pciconf -lv" > output. I can do the dmesg by using a serial console, and recording the output with script(1) or similar... But I can't really do a pciconf, since the FreeBSD hangs before I can get to a shell prompt. Unless, of course, running pciconf without the Cyclades equipment attached would help? Also, the pciconf in 4.4R doesn't appear to have a -v option. Perhaps I should upgrade to -current before trying these things. I'll begin the upgrade process as soon as I can Monday morning... It's a fast machine, so making world shouldn't be too bad. > Did the Tech Support seem helpful? They were courteous, and treated me with respect -- making them the best tech support I've ever worked with :) But, they didn't suggest anything that I hadn't already thought of or tried. After I convinced him that I wasn't sharing the IRQ with another device, and that the unit itself wasn't defective, he told me that I needed to talk to Dell, that something was wrong with their BIOS. Dell, of course, promised to be spectacularly unhelpful, insisting thatt the problem was either with the "3rd party devices or the 3rd party software" and that they couldn't provide support for them. They refused to even consider the idea that it might be their BIOS. > > Bruce > Thanks for your help, so far! I'll try what you've suggested and send you the results. Scott On Thu, 25 Oct 2001, Arjan Knepper wrote: > >From: Scott Klement <klemscot@klements.com> > >Date: Mon, 22 Oct 2001 01:16:27 -0500 (CDT) > > > > On Sun, 21 Oct 2001, Bruce Evans wrote: > > > Do you mean it desn't show any shared IRQs? PCI IRQs can't really conflict. > > > Shareing them is supposed to work. > > > > Errr.. sorry, you're right. I meant that no two devices are assigned the > > same IRQ number -- i.e. nothing is shared. > > > > I have to disable devices to get it to that point, where nothing is > > shared, but doing so does not prevent FreeBSD from hanging during boot. > > > I have the same kind of trouble, are you using 1 cy board? It does seem like the same problem for 1 board. > Already tried the -CURRENT, dind't help me. > Bellow an email to -HACKERS with some test results: > > > Hello, > > > > There are problems with the Cyclades Cyclom YeP driver. (see PR > > i386/30965). I've put printf's in the driver code on several places to > > check where the point is of hard locking and its seems to be on line > > 136 in the /usr/src/sys/pci/cy_pci.c in my situation. > > > > --------<snipped>--------------------------------------------------------------------- > > > > case PLX_9050: > > outw(ioport + CY_PLX_9050_ICS, > > inw(ioport + CY_PLX_9050_ICS) | > > CY_PLX_9050_ICS_IENABLE | > > CY_PLX_9050_ICS_LOCAL_IENABLE); > > --------<snipped>--------------------------------------------------------------------- Sorry I didn't reply to this before. I think it locks up here just because this enables the interrupt, an interrupt occures immediately, and interrupt handling never completes. You could try putting printfs in the interrupt handler (cyintr()). Or using ddb, put breakpoints at interesting places in the interrupt handler and see if they are hit. The initial interesting places are the start of the interrupt handler (cyintr()) and when it returns (get the return address using a trace command). > > Attached my kernel conf file and below the piece of code containg the > > lines. > >... > > case PLX_9050: > > outw(ioport + CY_PLX_9050_ICS, > > inw(ioport + CY_PLX_9050_ICS) | > > CY_PLX_9050_ICS_IENABLE | > > CY_PLX_9050_ICS_LOCAL_IENABLE); > > break; > > case PLX_9060: > > case PLX_9080: > > default: /* Old board, use PLX_9060 values. */ > > outw(ioport + CY_PLX_9060_ICS, > > inw(ioport + CY_PLX_9060_ICS) | > > CY_PLX_9060_ICS_IENABLE | > > CY_PLX_9060_ICS_LOCAL_IENABLE); > > break; > > } > >... > BTW Linux redhat 7.1 runs just fine on this system with 3 Cyclades PCI > boards. The corresponding code in Linux has interesting similarities and differences. % /* enable interrupts in the PCI interface */ % plx_ver = cy_readb(cy_pci_addr2 + CyPLX_VER) & 0x0f; % switch (plx_ver) { % case PLX_9050: % % cy_writeb(cy_pci_addr0+0x4c, 0x43); % break; % % case PLX_9060: % case PLX_9080: % default: /* Old boards, use PLX_9060 */ % % plx_init(cy_pci_addr0, 0x6c); % /* For some yet unknown reason, once the PLX9060 reloads % the EEPROM, the IRQ is lost and, thus, we have to % re-write it to the PCI config. registers. % This will remain here until we find a permanent fix. */ % pci_write_config_byte(pdev, PCI_INTERRUPT_LINE, cy_pci_irq); % % cy_writew(cy_pci_addr0+0x68, % cy_readw(cy_pci_addr0+0x68)|0x0900); % break; Differences: - for the PLX_9050 case, the magic number 0x43 is spelled non-magically in FreeBSD, and has a different value (0x41). - for the other cases, FreeBSD doesn't do the plx_init() or the IRQ reload hack. Thes differences may be related to the following commits in the Linux driver: % * Revision 2.3.2.6 2000/05/05 13:56:05 ivan % ... % * Implemented workaround for PLX9050 bug that would cause a system lockup % * in certain systems, depending on the MMIO addresses allocated to the % * board. % * ... % * Revision 2.2.1.9 1998/12/30 18:18:30 ivan % * Changed access to PLX PCI bridge registers from I/O to MMIO, in % * order to make PLX9050-based boards work with certain motherboards. % ... % * Revision 2.2.2.3 1999/06/28 11:13:29 ivan % ... % * Implemented workaround for IRQ setting loss on the PCI configuration % * registers after a PCI bridge EEPROM reload (affects PLX9060 only); You could try the 0x41 -> 0x43 change easily. Unfortunately, I don't have docs for any of this. I have corresponded with ivan@cyclades.com, but not for 2.5 years. Bruce Bruce Evans wrote: >>>Hello, >>> >>>There are problems with the Cyclades Cyclom YeP driver. (see PR >>>i386/30965). I've put printf's in the driver code on several places to >>>check where the point is of hard locking and its seems to be on line >>>136 in the /usr/src/sys/pci/cy_pci.c in my situation. >>> >>>--------<snipped>--------------------------------------------------------------------- >>> >>> case PLX_9050: >>> outw(ioport + CY_PLX_9050_ICS, >>> inw(ioport + CY_PLX_9050_ICS) | >>>CY_PLX_9050_ICS_IENABLE | >>> CY_PLX_9050_ICS_LOCAL_IENABLE); >>>--------<snipped>--------------------------------------------------------------------- >>> > >Sorry I didn't reply to this before. > >I think it locks up here just because this enables the interrupt, an >interrupt occures immediately, and interrupt handling never completes. >You could try putting printfs in the interrupt handler (cyintr()). > I did that already with result that booting the machine leaves it looping in the interrupt handler. >Or using ddb, put breakpoints at interesting places in the interrupt >handler and see if they are hit. The initial interesting places are >the start of the interrupt handler (cyintr()) and when it returns (get >the return address using a trace command). > Ehhummm never used DDB before, I've never done kernel debugging at all but I will give it a try today. > > >>>Attached my kernel conf file and below the piece of code containg the >>>lines. >>>... >>> case PLX_9050: >>> outw(ioport + CY_PLX_9050_ICS, >>> inw(ioport + CY_PLX_9050_ICS) | >>>CY_PLX_9050_ICS_IENABLE | >>> CY_PLX_9050_ICS_LOCAL_IENABLE); >>> break; >>> case PLX_9060: >>> case PLX_9080: >>> default: /* Old board, use PLX_9060 values. */ >>> outw(ioport + CY_PLX_9060_ICS, >>> inw(ioport + CY_PLX_9060_ICS) | >>>CY_PLX_9060_ICS_IENABLE | >>> CY_PLX_9060_ICS_LOCAL_IENABLE); >>> break; >>> } >>>... >>> >The corresponding code in Linux has interesting similarities and >differences. > >% /* enable interrupts in the PCI interface */ >% plx_ver = cy_readb(cy_pci_addr2 + CyPLX_VER) & 0x0f; >% switch (plx_ver) { >% case PLX_9050: >% >% cy_writeb(cy_pci_addr0+0x4c, 0x43); >% break; >% >% case PLX_9060: >% case PLX_9080: >% default: /* Old boards, use PLX_9060 */ >% >% plx_init(cy_pci_addr0, 0x6c); >% /* For some yet unknown reason, once the PLX9060 reloads >% the EEPROM, the IRQ is lost and, thus, we have to >% re-write it to the PCI config. registers. >% This will remain here until we find a permanent fix. */ >% pci_write_config_byte(pdev, PCI_INTERRUPT_LINE, cy_pci_irq); >% >% cy_writew(cy_pci_addr0+0x68, >% cy_readw(cy_pci_addr0+0x68)|0x0900); >% break; > >Differences: >- for the PLX_9050 case, the magic number 0x43 is spelled non-magically > in FreeBSD, and has a different value (0x41). > Could you explain this a bit more? What does 'magic number' mean? And why or how is it 'non-magically spelled' in FreeBSD? What does that mean? > >- for the other cases, FreeBSD doesn't do the plx_init() or the IRQ reload > hack. > >Thes differences may be related to the following commits in the Linux >driver: > >% * Revision 2.3.2.6 2000/05/05 13:56:05 ivan >% ... >% * Implemented workaround for PLX9050 bug that would cause a system lockup >% * in certain systems, depending on the MMIO addresses allocated to the >% * board > >% * ... >% * Revision 2.2.1.9 1998/12/30 18:18:30 ivan >% * Changed access to PLX PCI bridge registers from I/O to MMIO, in >% * order to make PLX9050-based boards work with certain motherboards. >% ... >% * Revision 2.2.2.3 1999/06/28 11:13:29 ivan >% ... >% * Implemented workaround for IRQ setting loss on the PCI configuration >% * registers after a PCI bridge EEPROM reload (affects PLX9060 only); > >You could try the 0x41 -> 0x43 change easily. Unfortunately, I don't >have docs for any of this. I have corresponded with ivan@cyclades.com, >but not for 2.5 years. > >Bruce > Bruce Evans wrote: >On Thu, 25 Oct 2001, Arjan Knepper wrote: > >>> >>>--------<snipped>--------------------------------------------------------------------- >>> >>> case PLX_9050: >>> outw(ioport + CY_PLX_9050_ICS, >>> inw(ioport + CY_PLX_9050_ICS) | >>>CY_PLX_9050_ICS_IENABLE | >>> CY_PLX_9050_ICS_LOCAL_IENABLE); >>>--------<snipped>--------------------------------------------------------------------- >>> > >Sorry I didn't reply to this before. > >I think it locks up here just because this enables the interrupt, an >interrupt occures immediately, and interrupt handling never completes. >You could try putting printfs in the interrupt handler (cyintr()). >Or using ddb, put breakpoints at interesting places in the interrupt >handler and see if they are hit. The initial interesting places are >the start of the interrupt handler (cyintr()) and when it returns (get >the return address using a trace command). > > >You could try the 0x41 -> 0x43 change easily. > Bruce, I have have just done this and it seems to solve the problems. I have to perform some test to make it sure. Scott Klement, Could you please try this? Change the the lines from line 135-138 in /usr/src/sys/pci/cy_pci.c to: --------<snipped>--------------------------------------------------------------------- case PLX_9050: outw(ioport + CY_PLX_9050_ICS, inw(ioport + CY_PLX_9050_ICS) | CY_PLX_9050_ICS_IENABLE | CY_PLX_9050_ICS_LOCAL_IENABLE | 0x02 ); --------<snipped>--------------------------------------------------------------------- Added '| 0x02' in line 138. Thanks, Arjan Knepper On Fri, 26 Oct 2001, Arjan Knepper wrote: > > Scott Klement, > Could you please try this? > > Change the the lines from line 135-138 in /usr/src/sys/pci/cy_pci.c to: > > --------<snipped>--------------------------------------------------------------------- > > case PLX_9050: > outw(ioport + CY_PLX_9050_ICS, > inw(ioport + CY_PLX_9050_ICS) | > CY_PLX_9050_ICS_IENABLE | > CY_PLX_9050_ICS_LOCAL_IENABLE | 0x02 ); > --------<snipped>--------------------------------------------------------------------- > > > Added '| 0x02' in line 138. > Yes, that works! The Cyclades box works great with that change -- it even shares the IRQ without any problems! Thank you! Hi, I have been running 4 computers with Arjan's patch since October with zero problems. It's really been a lifesaver (thank you, Arjan!) Any chance this will be committed/MFC-ed by the next (4.6) release? On Fri, 26 Oct 2001, Arjan Knepper wrote: > > Bruce, > I have have just done this and it seems to solve the problems. I have to > perform some test to make it sure. > > Scott Klement, > Could you please try this? > > Change the the lines from line 135-138 in /usr/src/sys/pci/cy_pci.c to: > > --------<snipped>--------------------------------------------------------------------- > > case PLX_9050: > outw(ioport + CY_PLX_9050_ICS, > inw(ioport + CY_PLX_9050_ICS) | > CY_PLX_9050_ICS_IENABLE | > CY_PLX_9050_ICS_LOCAL_IENABLE | 0x02 ); > --------<snipped>--------------------------------------------------------------------- > > > Added '| 0x02' in line 138. > > Thanks, > Arjan Knepper > On Wed, 6 Feb 2002, Scott Klement wrote: > I have been running 4 computers with Arjan's patch since October with zero > problems. It's really been a lifesaver (thank you, Arjan!) > > Any chance this will be committed/MFC-ed by the next (4.6) release? Thanks for the feedback. Sorry I haven't asked Cyclades what the magic bit does. I would like to know before putting it in -stable. Bruce Bruce Evans wrote: >On Wed, 6 Feb 2002, Scott Klement wrote: > >>I have been running 4 computers with Arjan's patch since October with zero >>problems. It's really been a lifesaver (thank you, Arjan!) >> >>Any chance this will be committed/MFC-ed by the next (4.6) release? >> > >Thanks for the feedback. Sorry I haven't asked Cyclades what the magic >bit does. I would like to know before putting it in -stable. > Can imagine that. > > >Bruce > I'm still not satisfied with the Cyclades YeP boards. There is some strange behaviour with SMP kernels (panics) and with more than one YeP board in a PC the detection of the CD1400 sometimes failes. Arjan State Changed From-To: open->closed Fixed in rev.1.26 (-current), 1.17.2.1 (RELENG_4) and 1.10.2.3 (RELENG_3). Same fix as in the PR followup but from the vendor and with less magic. On Tue, 19 Feb 2002, Arjan Knepper wrote:
> I'm still not satisfied with the Cyclades YeP boards. There is some
> strange behaviour with SMP kernels (panics) and with more than one YeP
> board in a PC the detection of the CD1400 sometimes failes.
Please open a new PR about these if they are still problems.
Bruce
|
System will stop responding ("hard-lock") while probing devices as part of the booting process whenever a Cyclades Cyclom-Y/PCI card is installed & configured in the kernel. This same symptom occurs when booting both single-user mode and when booting to multi-user mode. The same symptom occurs both when "options CY_PCI_FASTINTR" is enabled in the kernel and also when it is not enabled. I believe that this is a software issue because the problem does NOT occur under RedHat Linux, on the same machines. I've tried it with 3 different (known working) Cyclades units, and with 3 different computers (2 Dell Optiplex GX100, 1 GX150) The same Cyclades units do work under FreeBSD with a different (non-Dell) computer system. Fix: Turn the computer off. Remove the "ports" (the white box with all of the DB25 connectors) from the cable. Turn the computer back on. You won't be able to use Cyclom device, but the system will boot. How-To-Repeat: Enable "device cy" in a kernel on a Dell OptiPlex GX100 or GX150 system, and install the Cyclades card, cable & ports. When you re-boot, it will hard-lock.