Dear community, I was in the process of installing FreeBSD on my old (Mid-2010) MacBook Pro (dual core, 4 GiB memory). While installing a snapshot of 12.0 (same effect with 11.2-RELEASE), I'm asked whether I want to configure IPv4 for the wired interface. This interface is connected to a Gigabit switch, which is connected with a 100MBit cable (don't ask) to another switch, which has the DHCP server. I answer yes, am asked whether I want to use DHCP, I say yes again, and the install then hangs. The console will show dhclient[...]: send_packet: Network is down. This message will repeat, apparently indefinitely. The DHCP server shows me this in its logs: dhcpd: DHCPDISCOVER from c8:bc:c8:xx:xx:xx via eth0 dhcpd: DHCPDOFFER on ... to c8:bc:c8:xx:xx:xx via eth0 The interface, as identified by the FreeBSD installer, is: Broadcom NetXtreme Fast Ethernet Controller, ASIC Revision 0x5784100. The light on the workgroup switch shows a 1GBit link, but will turn off and on again occasionally. (Some form of reset seems to be going on.) The DHCP server works fine for other machines hanging off the same workgroup switch. It seems as if the DHCPOFFER isn't seen by the NIC, but I have no clue why this should be so. Any ideas?
This can be caused by Spanning Tree Protocol (STP) on the port that the mac book is connected to. If the switch is using the old standard slow STP the port can be in a down state for longenough after carrier comes up that all packets are lost during the DHCP process. If you can check the switch to see if you can either disable STP on this port, or set it to the newer standard faststp. If that resolves the problem then it was STP interfering, if the problem remains then something else is occuring.
See also https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229432 Despite of Broadcom chip, you may be having same problem. Can you check if DHCPOFFER has option 26 "Interface MTU"? If so, try to temporary disable it in the DHCP server configuration or use another DHCP server to verify.
Hi Eugene, I checke, and the DHCPOFFER sent by the DHCP server (bog standard ISC by the way) contains the following options, in that order: 53 DHCP message type (offer) 54 DHCP server identifier 51 IP address lease time (1 day, which is correct) 1 Subnet mask (correct) 3 Router (correct) 15 Domain name (correct) 6 Domain name server (correct) 12 Host name (correct) 255 End As you can see, option 16 (interface MTU) is not among them.
Hi Rodney, I don't think it's the STP. If it were the STP, I don't see how the same MacBook could work with Debian (which it does). If you think that it could still be the STP, please say so and I'll try to put the MacBook on the same switch as the DHCP server. Cheers, Stephan
(In reply to Stephan Neuhaus from comment #4) My experience has been that STP interacts very poorly with any clients trying to do DHCP as the normal time for STP to decide a port is NOT connected to another switch is on the order of 30 seconds, which is frequently beyond the timeouts of DHCP request/offer cycles. Some machines well work, some well not, and it is all very flackey until you either turn STP off on the port, or set it to fast mode. Part of the problem is that boot time stuff likes to reset NIC's, and resetting the NIC drops carrier, which causes the STP decision all over. This is especially annoyed when chain loading from a native PXE to iPXE, often leading to iPXE load failures. The only way to remove this variable is to either make sure STP is off on the port, or set to stp fast mode. Using the same switch for both client/server does NOT remove stp from the path, that single switch should still be trying to do stp.
I just noted this in your original report: The light on the workgroup switch shows a 1GBit link, but will turn off and on again occasionally. (Some form of reset seems to be going on.) That is indicative of a NIC reset and re-negotiate, that can cause the switch to be deaf/mute on that port until it redoes STP. Please if you can turn STP off on the port of the machine having issues. I also would like to ask: Are you using the PXE client in the broadcom BIOS? Are you using any VLAN taggging stuff? Is this NIC in use as a shared IPMI device?
Replying to Rodney W. Grimes form comment #5 and comment #6 There are two other machines (one another Debian, one Windows 10) hanging off that (very cheap, consumer-grade Netgear) switch for which DHCP works. Also, if I install Debian on the same laptop on the same port on the same switch, DHCP *also* works. So I don't see how STP can be the root of this issue, but I'll try your suggestions. I cannot turn off STP or change the way it does STP, since it is not managed. If you say that putting it next to the server (which hangs off another switch, this time managed) doesn't solve the problem, I will try fiddling with the STP parameters on *that* switch and see if it solves things. I must admit though, should that make it work, it would be deeply mysterious to me. But I'm willing to concede that my mental model of how these things ought to work might be totally wrong :) As to your other questions: "Are you using the PXE client in the broadcom BIOS?" I don't even know what that means, TBH. I'm using an old MacBook Pro, which happens to have a Broadcom NIC in it. "Are you using any VLAN taggging stuff?" Nope. "Is this NIC in use as a shared IPMI device?" Nope. This NIC runs in a laptop. Cheers Stephan
Another reply to comment #5 I can see the DHCPDISCOVER reaching the DHCP server. (I've verified this with a tcpdump on the server: the DHCPDISCOVER is there, as is the outgoing DHCPOFFER.) If STP were the issue, the port would be dead, and the DHCPDISCOVER wouldn't go anywhere, would it? Cheers Stephan
(In reply to Stephan Neuhaus from comment #3) It is still obvious that Broadcom driver resets the link while dhclient applies received parameters and tries to commit the lease. I do not know if bsdinstall allows you to enter the shell temporary and run "ifconfig" to see interface name and "dhclient -b $ifname" manually, wait for link to re-establish and DHCP transaction to complete then return to bsdinstaller user interface. Try this. Also, please show interface name (bce, bge, other?). And if possible, attach server's DHCPOFFER captured to .pcap file with something like "tcpdump -s0 -w dhcpoffer.pcap -c 1 src udp port 67".
(In reply to Stephan Neuhaus from comment #8) What concerns me is that you can get the dhcp request, the dhcp reply goes out, the machine receives it, then in the process of setting the NIC values it drops carrier on the link, and the switch goes back through a STP retrain process, during which time your port looks dead. Realize any time your port drops link status your going to have about a 50 second dead time for STP to go through the blocking(20s), listening(15s), learning(15s) transition before it properly passes traffic again. This is usually long enough to cause all sorts of heartache.
(In reply to Rodney W. Grimes from comment #10) While it is generally true for an managed switch, is does not apply for unmanaged one we are dealing with here. It seems the NIC driver flaps link for too long itself, without STP.
Created attachment 195139 [details] DCHPOFFER for recalcitrant bge0
I reply to Eugene Grosbein from comment #8 I've attached a pcap containing the DHCPOFFER. I've also dropped into a shell from the installer. The interface name is indeed bge0 and this is what i see when I manually dhclient bge0: DHCPDISCOVER on bge0 to 255.255.255.255 port 67 interval 6 DHCPDISCOVER on bge0 to 255.255.255.255 port 67 interval 15 bge0 link state up -> down bge0 link state down -> up DHCPDISCOVER on bge0 to 255.255.255.255 port 67 interval 3 DHCPDISCOVER on bge0 to 255.255.255.255 port 67 interval 4 DHCPDISCOVER on bge0 to 255.255.255.255 port 67 interval 9 bge0 link state up -> down bge0 link state down -> up And so on ad infinitum. (Sorry, I relaise this is pertinent information, but I didn't see that from the installer and it only now occurred to me to drop into a shell.)
Attribution on comment #13 is wrong. Should be comment #9 instead. Sorry.
(In reply to Stephan Neuhaus from comment #13) It seems your box does not really receive a reply from DHCP server despite it is being sent. Please verify this by entering shell again and running commands: killall dhclient; dhclient -b bge0; tcpdump -npi bge0 udp src port 67 I'm afraid you'll see no packets from server's DHCP port 67. If so, you should digress from DHCP for a while and make sure if your link is stable while using static IP address instead.
Replying to comment #15 Yep, you're right, no DHCPOFFER seen by the NIC. Here is what I see. (Possibility of slight typos as I'm typing this by hand.) One thing I find odd is that the interface has the SIMPLEX flag initially and keeps that flag even after attempting DHCP. # dmesg | grep bge0 bge0: <Broadcom NetXtreme Fast Ethernet Controller, ASIC rev. 0x5784100> mem 0xd310ffff at device 0.0 on pci3 bge0: CHIP ID 0x5784100; ASIC REV 0x5784; CHIP REV 0x57841; PCI-E miibus0: <MII bus> on bge0 bge0: Using defaults for TSO: 65518/35/2048 bge0: Eyjernet address: c8:bc:c8:a4:f3:30 # ifconfig bge0 bge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE> ether c8:bc:c8:a4:f3:30 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> # killall dhclient No matching processes were found # dhclient -b bge0 # tcpdump -npi bge0 udp port 67 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on bge0, link-type EN10MB (Ethernet), capture size 262144 bytes 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from c8:bc:c8:a4:f3:30, length 300 [ this repeats ] ^C 7 packets captured 7 packets received by filter 0 packets dropped by kernel # ifconfig bge0 bge0: flags=8843<UP,BROADCAST,RUNNUNG,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE> inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255 ether c8:bc:c8:a4:f3:30 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active A further "dmesg | grep bge0" yields many repetitions of these lines: bge0: watchdog timeout -- resetting bge0: link state changed to DOWN bge0: link state changed to UP Again, this laptop works fine when I install Debian on it, so it's very unlikely to be the hardware. Do you have any ideas?
SIMPLEX flag is pretty normal and should be. It just says the NIC does not get incoming copies of frames it have just sent and system does not need to filter them out. Almost all modern ethernet interfaces have this flag. So, your real problem are "bge watchdog timeouts" and not DHCP. These point to some kind of driver problem or hardware problem (cables, ethernet switch and/or ports etc.) Are you having no problems with Debian and EXACTLY same set of hardware including cables? If no, try replacing hardware with known working set. If yes, this rules out hardware problem and points to driver problem. There is Korean blog article http://hyogeollee.blogspot.com/2011/11/ of year 2011 describing similar problem with FreeBSD on MacBook Pro (2010). It says: "In case of bge which is Ethernet driver, watchdog timeout occurs and it is not operated. However, if you add the polling option to the kernel configuration, it will work normally. Maybe there is a problem on the driver side. If you build by adding options DEVICE_POLLING to the kernel configuration, and then boot to the new kernel, it works fine." The kernel option DEVICE_POLLING prevents usage of MSI within bge(4) driver that may be source of this problem due to wrong interrupt handling. There are other ways to disable usage of MSI. You should try to escape to bootloader prompt at early stage while booting installation media and do: set hw.pci.enable_msi=0 set hw.pci.enable_msix=0 boot Then check if this works to run using static IP address. If so, re-check with DHCP.
OTOH, you should first try with "set dev.bge.0.msi=0" instead of disabling hw.pci.enable_msi/hw.pci.enable_msix. This way, you only disable MSI for bge and not for all other drivers in the system.
Now I get # killall dhclient No matching processes were found # dhclient bge0 DHCPDISCOVER on bge0 to 255.255.255.255 port 67 interval 6 DHCPDISCOVER on bge0 to 255.255.255.255 port 67 interval 8 DHCPOFFER from 192.168.0.253 DHCPREQUEST on bge0 to 255.255.255.255 port 67 DHCPACK from 192.168.0.253 bound to 192.168.0.10 -- renewal in 43200 seconds So it seems indeed to be some kind of watchdog issue, although I don't understand why the first DHCPDISCOVER did not yield a DHCPOFFER, or at least not one that the NIC could see. Now the question is, how do I make this permanent? But thanks already for the huge help. I would not have been able to get to this point without your help.
(In reply to Stephan Neuhaus from comment #19) After successfull installation, do: echo dev.bge.0.msi=0 >> /boot/loader.conf Note no "set" this time.
(Replying to Eugene Grosbein from comment #20) Yup, that did it. I now have a working install, even though there is a speedbump when bge0 is activated on boot: the switch shows only 100 MBit/s initially, then the link goes DOWN then UP again (this time with 1000 MBit/s), and the DHCP takes a while, probably because the first DHCPOFFER doesn't get through. I doubt that this is expected behaviour, but I can live with it. Since that fixes the issue for me (at least for now), shall we close it?
Replying to my own comment #21 If you can spare the time, could you explain to me what the "bge watchdog" does? But anyway, thanks a lot for walking this newbie through a difficult process, with clear instructions and a lot of patience :-) Cheers, Stephan
Created attachment 195169 [details] proposed fix Attached patch disables MSI for this revision of the NIC. Stephan, could you please test the patch? After installation, make sure you have kernel sources installed, then: 1) fetch the patch 2) cd /usr/src && patch < /path/to/patch 3) rebuild and reinstall the kernel, comment out the workaround in /boot/loader.conf, reboot with new kernel and it should just work.
(In reply to Stephan Neuhaus from comment #22) "watchdog timeout" is an indication of some unexpected interrupt processing problem. I do not know anything about Broadcom's chip internals, sorry.
(In reply to Eugene Grosbein from comment #23) I started the process but during make buldkernel, the MacBook started to act up: the console font changed and there were kernel(?) errors (that I couldn't read properly because, well, the font had changed). The upshot is that apparently the MacBook doesn't tolerate prolonged full load any more. I guess I have to open it up, clean it, remove dust etc, which may take a while. I'll also install Debian and give that a spin to see if this issue can be replicated on another OS. Of course, if this problem persists, this disqualifies the machine for any serious work, and thus FreeBSD (or any other OS, for that matter) will probably not see any production use on it. Furthermore, I can't rule out that the bge problems I had and the weird errors under full load have the same root cause, so I'd be cautious: even if the patch solves my issue (from looking at it, it seems to make "no watchdog" the default for this particular chipset revision, so it absolutely should), I'm not sure if this is the right thing for everyone.
I tried again with a virgin install and from what I can see from the screwed-up font, this time, I landed in the kernel debugger, but with an unresponsive keyboard. Something is definitely not right here, but I doubt it has anything to do with this bug. FYI, I noticed a warning about "lock order" or something similar rather early after booting. This sounds pretty scary to me, but as I've learned from SIMPLEX, this may nothing to be worried about. I'd say we close this issue as FIXED and I go shout at my MacBook. What do you think?
(In reply to Stephan Neuhaus from comment #26) As we know from that old blog post, you are not only one having bge problem with this MacBook Pro model and I'd like not to loose the result we got discussing this problem. Can you try 11.2-RELEASE instead of CURRENT? It could be more stable and the patch will apply cleanly to both of 11.2-RELEASE and 12.0-CURRENT. As for "lock order reversal" messages, you can safely ignore it for a time being.
(In reply to Eugene Grosbein from comment #27) OK, kernel build running now, fingers crossed! :-)
Nope, same effect: font screwed, then apparently AHCI errors. TBH, this looks like some kind of memory corruption to me. Trying with Debian...
(In reply to Stephan Neuhaus from comment #29) FreeBSD has sysutils/memtester port (or package) that allows testing memory using running system. Before start, one should raise the limit sysctl vm.max_wired (in 4K pages) so kernel would allow memtester to lock maximum possible physical memory. For example, if "grep 'avail memory' /var/run/dmesg.boot" shows 3863 MB, you should try set "sysctl vm.max_wired" to 972800 (=3800MB*1024*1024/4096) then run "memtester 3800M"
Replying to my own comment #29 and to Eugene Grosbein's comment #30 I've just installed Debian 9 on the MacBook and compiled gcc as a stress test. Zero problems. Bge came up right away and there were no issues with the compile. So I don't think it's the hardware or the memory.
(In reply to Stephan Neuhaus from comment #31) Then it should be some problem with new console video driver vt(4) using graphics mode. Please try installing FreeBSD 11.2 back and switch it to old stable syscons(4) driver: echo kern.vty=sc >> /boot/loader.conf
Though this may be a broken hardware issue, to me it does make since to at least patch the man page in the section on DIAGNOSTICS watchdog timeout to indicate that this may be caused by non functional MSI interrupts and that one should try the dev.bge.%d.msi sysctl to disable them, with a further note on how to report that one had to do this so we might gather additional data.
(In reply to Rodney W. Grimes from comment #33) Feel free to prepare a change for bge(4) manual page. I won't be doing it as English is not my native language.
(In reply to Eugene Grosbein in comment #34) When I set kern.vty=sc, booting hangs after giving the EFI framebuffer information: EFI framebuffer information: addr, size: 0xc0010000, 0x640000 dimensions: 1280 x 800 stride: 2048 masks 0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000
Wrong attribution in comment #35, should be comment #32, sorry.
(In reply to Stephan Neuhaus from comment #35) If this model does not support old BIOS-style boot mode, try using text mode of vt(4) driver: echo hw.vga.textmode=1 >> /boot/loader.conf instead of "kern.vty=sc" (remove it from the loader.conf).
(In reply to Stephan Neuhaus from comment #35) Also, you can try redirect building output to a file to verify if this is video console driver's problem: make buildkernel >& /var/log/buildkernel.log
Replying to Eugene Grosbein from comment #38 Success! The kernel built, I installed and rebooted, and now have a working bge0. I still have hw.vga.textmode=1 in /boot/loader.conf, but that seems a different issue. Can/should I be using this kernel for production? Cheers and thanks Stephan
(In reply to Stephan Neuhaus from comment #39) Yes, you should. And thank you for your non-stop marathon testing despite of all holes on the way :-)
Fixed. Thanks to Rodney W. Grimes and especially Eugene Grosbein for his unwavering attention.
Please do not close the PR. The case is not closed until fix committed and merged.
I'm going to commit https://bugs.freebsd.org/bugzilla/attachment.cgi?id=195169 soon unless an objection raised.
A commit references this bug: Author: eugen Date: Wed Jul 18 18:28:18 UTC 2018 New revision: 336461 URL: https://svnweb.freebsd.org/changeset/base/336461 Log: bge(4): disable MSI for BGE_ASICREV_BCM5784/BGE_CHIPREV_5784_AX found in some MacBook Pro. PR: 229727 Reported by: Stephan Neuhaus <sten@artdecode.de> and others Tested by: Stephan Neuhaus <sten@artdecode.de> Approved by: mav (mentor) MFC after: 1 month Changes: head/sys/dev/bge/if_bge.c
A commit references this bug: Author: eugen Date: Fri Aug 17 19:18:59 UTC 2018 New revision: 337985 URL: https://svnweb.freebsd.org/changeset/base/337985 Log: MFC r336461: bge(4): disable MSI for BGE_ASICREV_BCM5784/BGE_CHIPREV_5784_AX found in some MacBook Pro. PR: 229727 Reported by: Stephan Neuhaus <sten@artdecode.de> and others Tested by: Stephan Neuhaus <sten@artdecode.de> Approved by: mav (mentor) Changes: _U stable/11/ stable/11/sys/dev/bge/if_bge.c