Hi,

Do the link UP/DOWN messages generally occur after ~15 minutes, or does the time between initializing the adapter and those messages vary?

Do you send IPv4 or IPv6 traffic, specific UDP/TCP traffic like NFS, or does the adapter idle until the link starts flapping?

By default, the adapter should use MSI instead of legacy interrupts. Did you change that setting?

Jack (the maintainer) is on a vacation, so I get to look at these bugs while he's gone. We tried passing TCP/IPv4 traffic using a NUC we have in-house, and we couldn't get the link up/down messages after ~30 minutes, so I'm wondering if we need to do something specific to reproduce this issue.

- Eric

(In reply to Eric Joyner from comment #1)
> Hi,

Thanks for taking the time!

> Do the link UP/DOWN messages generally occur after ~15 minutes, or does the
> time between initializing the adapter and those messages vary?

weird! Now they have stopped just as quick as they started!
Grepping through my logs I can see that I had this problem between Sept. 8 - 13
(my oldest logfile goes back to the 6th).

Here are the first few entries from /var/log/messages:
(IP's & hostnames redacted)

Sep  8 04:04:49 domus afpd[93184]: afp_alarm: reconnect timer expired, goodbye
Sep  8 04:04:49 domus afpd[93184]: Disconnected session terminating
Sep  8 08:18:06 domus kernel: em0: link state changed to DOWN
Sep  8 08:18:09 domus kernel: em0: link state changed to UP
Sep  8 08:18:09 domus devd: Executing '/etc/rc.d/dhclient quietstart em0'
Sep  8 14:07:24 domus ntpd[886]: bind() fd 31, family AF_INET6, port 123, scope 0, addr xxxx::xxxx:xxxx:xxxx:xxxx, mc
Sep  8 14:07:24 domus ntpd[886]: unable to create socket on em0 (176) for xxxx::xxxx:xxxx:xxxx:xxxx#123
Sep  8 15:03:15 domus kernel: em0: link state changed to DOWN
Sep  8 15:03:19 domus kernel: em0: link state changed to UP
Sep  8 15:03:19 domus devd: Executing '/etc/rc.d/dhclient quietstart em0'
Sep  8 18:56:33 domus kernel: em0: link state changed to DOWN
Sep  8 18:56:37 domus kernel: em0: link state changed to UP
Sep  8 18:56:37 domus devd: Executing '/etc/rc.d/dhclient quietstart em0'
Sep  8 21:10:26 domus ntpd[886]: bind() fd 32, family AF_INET6, port 123, scope 0, addr xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx
Sep  8 21:10:26 domus ntpd[886]: unable to create socket on em0 (182) for xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx#123
Sep  8 21:10:26 domus ntpd[886]: bind() fd 32, family AF_INET6, port 123, scope 0, addr xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx
Sep  8 21:10:26 domus ntpd[886]: unable to create socket on em0 (183) for xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx#123
Sep  8 21:11:54 domus kernel: em0: link state changed to DOWN
Sep  8 21:11:57 domus kernel: em0: link state changed to UP
Sep  8 21:11:57 domus devd: Executing '/etc/rc.d/dhclient quietstart em0'

and here the last few entries:

Sep 13 19:32:52 domus kernel: em0: link state changed to DOWN
Sep 13 19:32:56 domus kernel: em0: link state changed to UP
Sep 13 19:32:56 domus devd: Executing '/etc/rc.d/dhclient quietstart em0'
Sep 13 19:34:10 domus kernel: em0: link state changed to DOWN
Sep 13 19:34:14 domus kernel: em0: link state changed to UP
Sep 13 19:34:14 domus devd: Executing '/etc/rc.d/dhclient quietstart em0'
Sep 13 19:37:54 domus kernel: em0: link state changed to DOWN
Sep 13 19:37:57 domus kernel: em0: link state changed to UP
Sep 13 19:37:57 domus devd: Executing '/etc/rc.d/dhclient quietstart em0'
Sep 13 19:39:39 domus kernel: em0: link state changed to DOWN
Sep 13 19:39:43 domus kernel: em0: link state changed to UP
Sep 13 19:39:43 domus devd: Executing '/etc/rc.d/dhclient quietstart em0'
Sep 13 19:40:04 domus rpc.statd: Failed to contact host backups.home: RPC: Port mapper failure - RPC: Timed out
Sep 13 19:40:18 domus kernel: em0: link state changed to DOWN
Sep 13 19:40:49 domus kernel: em0: link state changed to UP
Sep 13 19:40:49 domus devd: Executing '/etc/rc.d/dhclient quietstart em0'
Sep 13 19:42:04 domus rpc.statd: Failed to contact host laptop.home: RPC: Port mapper failure - RPC: Timed out
Sep 13 20:44:04 domus rpc.statd: Failed to contact host backups.home: RPC: Port mapper failure - RPC: Timed out
Sep 13 20:46:05 domus rpc.statd: Failed to contact host laptop.home: RPC: Port mapper failure - RPC: Timed out
Sep 13 21:48:05 domus rpc.statd: Failed to contact host backups.home: RPC: Port mapper failure - RPC: Timed out
Sep 13 21:50:05 domus rpc.statd: Failed to contact host laptop.home: RPC: Port mapper failure - RPC: Timed out
Sep 13 22:52:05 domus rpc.statd: Failed to contact host backups.home: RPC: Port mapper failure - RPC: Timed out
Sep 13 22:54:05 domus rpc.statd: Failed to contact host laptop.home: RPC: Port mapper failure - RPC: Timed out
Sep 13 23:56:05 domus rpc.statd: Failed to contact host backups.home: RPC: Port mapper failure - RPC: Timed out
Sep 13 23:58:05 domus rpc.statd: Failed to contact host laptop.home: RPC: Port mapper failure - RPC: Timed out
Sep 14 00:07:24 domus afpd[80087]: Login by steve (AFP3.3)

Here's a quick summary of how often it occurred:
        drops   max
        / day   / hour
Sep  8:   5      1
Sep  9:  26      6
Sep 10:  58      7
Sep 11: 273     18
Sep 12: 620     38
Sep 13: 453     33

No configuration changes took place at either end - although I did reboot a
few times in the hopes of shaking out some cobwebs. And as the log shows, there
wasn't even a reboot around the time it stopped.

> Do you send IPv4 or IPv6 traffic, specific UDP/TCP traffic like NFS, or does
> the adapter idle until the link starts flapping?

Yes, I use both. I actually tried to set up an IPv6-only network earlier
this year, but failed - too many utils still fall back to IPv4.
I've got NFS and AFP deamons running but their log messages don't seem to
coincide with the link drops. (at least not obviously)

> By default, the adapter should use MSI instead of legacy interrupts. Did you
> change that setting?

Not knowingly. Where / how would I go about changing it? How can I find out
what setting(s) I currently have?
As a stab in the dark, is this what you're talking about?

# sysctl -a | grep -i msi
hw.aac.enable_msi: 1
hw.bce.msi_enable: 1
hw.em.enable_msix: 1
hw.igb.enable_msix: 1
hw.ix.enable_msix: 1
hw.malo.pci.msi_disable: 0
hw.mfi.msi: 1
hw.pci.enable_msi: 1
hw.pci.enable_msix: 1
hw.pci.honor_msi_blacklist: 1

This actually more my problem: I see that something is wrong, but have no idea
how to get more information from the system about the cause.
I've got 20 years of unix experience but only a few months with FreeBSD,
it's all so familiar and yet still seems like black magic at times :-(

Again, thanks for your time!

Steve

...and their back again, this time with up to 80 drops per hour,
every hour, regardless of traffic, load or memory use.
(The machine is mostly idle)

I'd really love to be able to increase the logging level for the
link controller / driver. Is there a way to do this?

Any tips would be greatly appreciated!

Thanks in advance,

Steve

Unfortunately, I don't think there's a way to increase the debug logging level of the driver. You can try setting dev.em.0.debug to 1 and reading the output in /var/log/messages, but that information probably isn't very useful in this case. Have you tried connecting something else to the NUC's link partner instead of the NUC, or connecting something else to the NUC, and seeing if the link flaps occur?

update:

although I was unable to determine any conclusive faults, I replaced the network switch
today and it appears that this solved the problem.

Thanks for helping.

Steve