Bug 239240 - igb: TX(2) desc avail = 1024, pidx = 0 messages appear when the network card (igb/ixgbe/em) loses ethernet link
Summary: igb: TX(2) desc avail = 1024, pidx = 0 messages appear when the network card ...
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-net mailing list
URL:
Keywords: IntelNetworking, needs-patch, needs-qa
Depends on:
Blocks:
 
Reported: 2019-07-16 05:37 UTC by Richard Gallamore
Modified: 2019-10-02 06:08 UTC (History)
10 users (show)

See Also:
koobs: mfc-stable12?


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Richard Gallamore freebsd_committer 2019-07-16 05:37:23 UTC
Currently, I have a router running FreeBSD 12.0-RELEASE on atom c2758. My switch seems to have become unresponsive which required a hard reset.

Sometime around when the switch became unresponsive, (I think it was at least a few minutes after) I started receiving "igb6: TX(2) desc avail = 1024, pidx = 0". The interface no longer works and will continue to print the message until the interface is down. Bringing it back up will start the message again.

Most of the settings are below. There is a lot that were removed as I don't think they have any relevance to this issue. Let me know if there is anything else that may help.

pciconf -lv
igb6@pci0:0:20:3:       class=0x020000 card=0x1f4115d9 chip=0x1f418086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection I354'
    class      = network
    subclass   = ethernet

sysctl -a | grep igb\.0
dev.igb.0.interrupts.rx_overrun: 0
dev.igb.0.interrupts.rx_desc_min_thresh: 0
dev.igb.0.interrupts.tx_queue_min_thresh: 65832053
dev.igb.0.interrupts.tx_queue_empty: 71983081
dev.igb.0.interrupts.tx_abs_timer: 0
dev.igb.0.interrupts.tx_pkt_timer: 0
dev.igb.0.interrupts.rx_abs_timer: 0
dev.igb.0.interrupts.rx_pkt_timer: 65832053
dev.igb.0.interrupts.asserts: 74311168
dev.igb.0.mac_stats.tso_ctx_fail: 0
dev.igb.0.mac_stats.tso_txd: 46548
dev.igb.0.mac_stats.tx_frames_1024_1522: 48022265
dev.igb.0.mac_stats.tx_frames_512_1023: 301529
dev.igb.0.mac_stats.tx_frames_256_511: 900639
dev.igb.0.mac_stats.tx_frames_128_255: 3062062
dev.igb.0.mac_stats.tx_frames_65_127: 16876524
dev.igb.0.mac_stats.tx_frames_64: 2820062
dev.igb.0.mac_stats.mcast_pkts_txd: 27545
dev.igb.0.mac_stats.bcast_pkts_txd: 211
dev.igb.0.mac_stats.good_pkts_txd: 71983081
dev.igb.0.mac_stats.total_pkts_txd: 71983081
dev.igb.0.mac_stats.good_octets_txd: 75469715088
dev.igb.0.mac_stats.good_octets_recvd: 59838583722
dev.igb.0.mac_stats.rx_frames_1024_1522: 37344025
dev.igb.0.mac_stats.rx_frames_512_1023: 5784017
dev.igb.0.mac_stats.rx_frames_256_511: 472870
dev.igb.0.mac_stats.rx_frames_128_255: 2642368
dev.igb.0.mac_stats.rx_frames_65_127: 11918975
dev.igb.0.mac_stats.rx_frames_64: 7669798
dev.igb.0.mac_stats.mcast_pkts_recvd: 43168
dev.igb.0.mac_stats.bcast_pkts_recvd: 1268
dev.igb.0.mac_stats.good_pkts_recvd: 65832053
dev.igb.0.mac_stats.total_pkts_recvd: 65832053
dev.igb.0.mac_stats.xoff_txd: 0
dev.igb.0.mac_stats.xoff_recvd: 0
dev.igb.0.mac_stats.xon_txd: 0
dev.igb.0.mac_stats.xon_recvd: 0
dev.igb.0.mac_stats.coll_ext_errs: 0
dev.igb.0.mac_stats.alignment_errs: 0
dev.igb.0.mac_stats.crc_errs: 0
dev.igb.0.mac_stats.recv_errs: 0
dev.igb.0.mac_stats.recv_jabber: 0
dev.igb.0.mac_stats.recv_oversize: 0
dev.igb.0.mac_stats.recv_fragmented: 0
dev.igb.0.mac_stats.recv_undersize: 0
dev.igb.0.mac_stats.recv_no_buff: 0
dev.igb.0.mac_stats.missed_packets: 0
dev.igb.0.mac_stats.defer_count: 0
dev.igb.0.mac_stats.sequence_errors: 0
dev.igb.0.mac_stats.symbol_errors: 0
dev.igb.0.mac_stats.collision_count: 0
dev.igb.0.mac_stats.late_coll: 0
dev.igb.0.mac_stats.multiple_coll: 0
dev.igb.0.mac_stats.single_coll: 0
dev.igb.0.mac_stats.excess_coll: 0
dev.igb.0.queue_rx_3.rx_irq: 0
dev.igb.0.queue_rx_3.rxd_tail: 942
dev.igb.0.queue_rx_3.rxd_head: 944
dev.igb.0.queue_rx_2.rx_irq: 0
dev.igb.0.queue_rx_2.rxd_tail: 369
dev.igb.0.queue_rx_2.rxd_head: 371
dev.igb.0.queue_rx_1.rx_irq: 0
dev.igb.0.queue_rx_1.rxd_tail: 1010
dev.igb.0.queue_rx_1.rxd_head: 1012
dev.igb.0.queue_rx_0.rx_irq: 0
dev.igb.0.queue_rx_0.rxd_tail: 341
dev.igb.0.queue_rx_0.rxd_head: 343
dev.igb.0.queue_tx_3.tx_irq: 0
dev.igb.0.queue_tx_3.txd_tail: 748
dev.igb.0.queue_tx_3.txd_head: 748
dev.igb.0.queue_tx_2.tx_irq: 0
dev.igb.0.queue_tx_2.txd_tail: 667
dev.igb.0.queue_tx_2.txd_head: 667
dev.igb.0.queue_tx_1.tx_irq: 0
dev.igb.0.queue_tx_1.txd_tail: 48
dev.igb.0.queue_tx_1.txd_head: 48
dev.igb.0.queue_tx_0.tx_irq: 0
dev.igb.0.queue_tx_0.txd_tail: 363
dev.igb.0.queue_tx_0.txd_head: 363
dev.igb.0.fc_low_water: 32752
dev.igb.0.fc_high_water: 32768
dev.igb.0.rx_control: 71335966
dev.igb.0.device_control: 404488769
dev.igb.0.watchdog_timeouts: 0
dev.igb.0.rx_overruns: 0
dev.igb.0.tx_dma_fail: 0
dev.igb.0.mbuf_defrag_fail: 0
dev.igb.0.link_irq: 2
dev.igb.0.dropped: 0
dev.igb.0.eee_control: 1
dev.igb.0.itr: 488
dev.igb.0.tx_abs_int_delay: 66
dev.igb.0.rx_abs_int_delay: 66
dev.igb.0.tx_int_delay: 66
dev.igb.0.rx_int_delay: 0
dev.igb.0.rs_dump: 0
dev.igb.0.reg_dump: General Registers
dev.igb.0.fc: 0
dev.igb.0.debug: -1
dev.igb.0.nvm: -1
dev.igb.0.iflib.rxq3.rxq_fl0.credits: 1023
dev.igb.0.iflib.rxq3.rxq_fl0.cidx: 944
dev.igb.0.iflib.rxq3.rxq_fl0.pidx: 943
dev.igb.0.iflib.rxq2.rxq_fl0.credits: 1023
dev.igb.0.iflib.rxq2.rxq_fl0.cidx: 371
dev.igb.0.iflib.rxq2.rxq_fl0.pidx: 370
dev.igb.0.iflib.rxq1.rxq_fl0.credits: 1023
dev.igb.0.iflib.rxq1.rxq_fl0.cidx: 1012
dev.igb.0.iflib.rxq1.rxq_fl0.pidx: 1011
dev.igb.0.iflib.rxq0.rxq_fl0.credits: 1023
dev.igb.0.iflib.rxq0.rxq_fl0.cidx: 343
dev.igb.0.iflib.rxq0.rxq_fl0.pidx: 342
dev.igb.0.iflib.txq3.r_abdications: 0
dev.igb.0.iflib.txq3.r_restarts: 0
dev.igb.0.iflib.txq3.r_stalls: 0
dev.igb.0.iflib.txq3.r_starts: 15127348
dev.igb.0.iflib.txq3.r_drops: 0
dev.igb.0.iflib.txq3.r_enqueues: 15165787
dev.igb.0.iflib.txq3.ring_state: pidx_head: 0347 pidx_tail: 0347 cidx: 0347 state: IDLE
dev.igb.0.iflib.txq3.txq_cleaned: 15730369
dev.igb.0.iflib.txq3.txq_processed: 15730409
dev.igb.0.iflib.txq3.txq_in_use: 43
dev.igb.0.iflib.txq3.txq_cidx_processed: 745
dev.igb.0.iflib.txq3.txq_cidx: 705
dev.igb.0.iflib.txq3.txq_pidx: 748
dev.igb.0.iflib.txq3.no_tx_dma_setup: 0
dev.igb.0.iflib.txq3.txd_encap_efbig: 0
dev.igb.0.iflib.txq3.tx_map_failed: 0
dev.igb.0.iflib.txq3.no_desc_avail: 0
dev.igb.0.iflib.txq3.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq3.m_pullups: 59
dev.igb.0.iflib.txq3.mbuf_defrag: 0
dev.igb.0.iflib.txq2.r_abdications: 0
dev.igb.0.iflib.txq2.r_restarts: 0
dev.igb.0.iflib.txq2.r_stalls: 0
dev.igb.0.iflib.txq2.r_starts: 29491761
dev.igb.0.iflib.txq2.r_drops: 0
dev.igb.0.iflib.txq2.r_enqueues: 29560202
dev.igb.0.iflib.txq2.ring_state: pidx_head: 1418 pidx_tail: 1418 cidx: 1418 state: IDLE
dev.igb.0.iflib.txq2.txq_cleaned: 43517553
dev.igb.0.iflib.txq2.txq_processed: 43517593
dev.igb.0.iflib.txq2.txq_in_use: 42
dev.igb.0.iflib.txq2.txq_cidx_processed: 665
dev.igb.0.iflib.txq2.txq_cidx: 625
dev.igb.0.iflib.txq2.txq_pidx: 667
dev.igb.0.iflib.txq2.no_tx_dma_setup: 0
dev.igb.0.iflib.txq2.txd_encap_efbig: 0
dev.igb.0.iflib.txq2.tx_map_failed: 0
dev.igb.0.iflib.txq2.no_desc_avail: 0
dev.igb.0.iflib.txq2.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq2.m_pullups: 33
dev.igb.0.iflib.txq2.mbuf_defrag: 0
dev.igb.0.iflib.txq1.r_abdications: 0
dev.igb.0.iflib.txq1.r_restarts: 0
dev.igb.0.iflib.txq1.r_stalls: 0
dev.igb.0.iflib.txq1.r_starts: 16060858
dev.igb.0.iflib.txq1.r_drops: 0
dev.igb.0.iflib.txq1.r_enqueues: 16161222
dev.igb.0.iflib.txq1.ring_state: pidx_head: 0454 pidx_tail: 0454 cidx: 0454 state: IDLE
dev.igb.0.iflib.txq1.txq_cleaned: 22249477
dev.igb.0.iflib.txq1.txq_processed: 22249517
dev.igb.0.iflib.txq1.txq_in_use: 43
dev.igb.0.iflib.txq1.txq_cidx_processed: 45
dev.igb.0.iflib.txq1.txq_cidx: 5
dev.igb.0.iflib.txq1.txq_pidx: 48
dev.igb.0.iflib.txq1.no_tx_dma_setup: 0
dev.igb.0.iflib.txq1.txd_encap_efbig: 0
dev.igb.0.iflib.txq1.tx_map_failed: 0
dev.igb.0.iflib.txq1.no_desc_avail: 0
dev.igb.0.iflib.txq1.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq1.m_pullups: 90
dev.igb.0.iflib.txq1.mbuf_defrag: 0
dev.igb.0.iflib.txq0.r_abdications: 0
dev.igb.0.iflib.txq0.r_restarts: 0
dev.igb.0.iflib.txq0.r_stalls: 0
dev.igb.0.iflib.txq0.r_starts: 11114832
dev.igb.0.iflib.txq0.r_drops: 0
dev.igb.0.iflib.txq0.r_enqueues: 11144763
dev.igb.0.iflib.txq0.ring_state: pidx_head: 1595 pidx_tail: 1595 cidx: 1595 state: IDLE
dev.igb.0.iflib.txq0.txq_cleaned: 11860289
dev.igb.0.iflib.txq0.txq_processed: 11860329
dev.igb.0.iflib.txq0.txq_in_use: 42
dev.igb.0.iflib.txq0.txq_cidx_processed: 361
dev.igb.0.iflib.txq0.txq_cidx: 321
dev.igb.0.iflib.txq0.txq_pidx: 363
dev.igb.0.iflib.txq0.no_tx_dma_setup: 0
dev.igb.0.iflib.txq0.txd_encap_efbig: 0
dev.igb.0.iflib.txq0.tx_map_failed: 0
dev.igb.0.iflib.txq0.no_desc_avail: 0
dev.igb.0.iflib.txq0.mbuf_defrag_failed: 0
dev.igb.0.iflib.txq0.m_pullups: 116
dev.igb.0.iflib.txq0.mbuf_defrag: 0
dev.igb.0.iflib.override_nrxds: 0
dev.igb.0.iflib.override_ntxds: 0
dev.igb.0.iflib.tx_abdicate: 0
dev.igb.0.iflib.rx_budget: 0
dev.igb.0.iflib.disable_msix: 0
dev.igb.0.iflib.override_qs_enable: 0
dev.igb.0.iflib.override_nrxqs: 0
dev.igb.0.iflib.override_ntxqs: 0
dev.igb.0.iflib.driver_version: 7.6.1-k
dev.igb.0.%parent: pci4
dev.igb.0.%pnpinfo: vendor=0x8086 device=0x1533 subvendor=0x15d9 subdevice=0x1533 class=0x020000
dev.igb.0.%location: slot=0 function=0 dbsf=pci0:4:0:0
dev.igb.0.%driver: igb
dev.igb.0.%desc: Intel(R) PRO/1000 PCI-Express Network Driver

rc.conf:
powerd_enable="YES"
powerd_flags="-a min -b min -m 1200/1000 -M 2400/10000"
ifconfig_igb6="up"
ifconfig_bridge1="inet 10.93.232.1/24 addm lagg1 addm igb6 addm igb5"
ifconfig_bridge1_ipv6="inet6 auto_linklocal"
gateway_enable="YES"
ipv6_gateway_enable="YES"
ipv6_cpe_wanif="igb0"
ipv6_privacy="NO"

sysctl.conf:
net.link.bridge.pfil_member=0
net.link.bridge.pfil_onlyip=0
net.link.stf.permit_rfc1918=1
net.inet.tcp.fastopen.server_enable=1
net.inet.tcp.cc.algorithm=cubic
net.inet.tcp.rfc6675_pipe=1
net.inet.tcp.recvbuf_inc=65536
net.inet.tcp.recvspace=131072
net.inet.tcp.sendbuf_inc=65536
net.inet.tcp.sendspace=131072
net.inet.tcp.mssdflt=1460
net.inet.tcp.minmss=536
net.inet.tcp.syncache.rexmtlimit=3
net.inet.tcp.abc_l_var=44
net.inet.tcp.initcwnd_segments=16
net.inet.tcp.syncookies=0
dev.igb.0.fc=0

loader.conf
net.link.bridge.inherit_mac="1"
net.link.ifqmaxlen="2048"
hw.igb.rx_process_limit="-1"
hw.igb.num_queues="2"
net.pf.request_maxcount=1048575
Comment 1 IPTRACE 2019-07-16 17:00:35 UTC
Hello!

I'm affected as well for FreeBSD 12.0-RELEASE-p7.

kernel: igb0: TX(0) desc avail = 1024, pidx = 0
...
kernel: igb2: TX(5) desc avail = 1024, pidx = 0


igb0@pci0:129:0:0:      class=0x020000 card=0x00001458 chip=0x15218086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'I350 Gigabit Network Connection'
    class      = network
    subclass   = ethernet


igb2@pci0:132:0:0:      class=0x020000 card=0xa02c8086 chip=0x10e88086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82576 Gigabit Network Connection'
    class      = network
    subclass   = ethernet


Use case (forced by me but generally it happens randomly):
1. All was working.
2. I detached Ethernet cable from modem side.
3. Connected again and saw "no carrier" on igb0 interface.
4. System logged as follow:
      Jul  5 22:31:35 hpv kernel: igb0: TX(0) desc avail = 1024, pidx = 0
      Jul  5 22:31:38 hpv kernel: igb0: TX(0) desc avail = 1024, pidx = 0
      Jul  5 22:31:40 hpv kernel: igb0: TX(0) desc avail = 1024, pidx = 0
5. I was trying "server:# service netif restart igb0 && service routing restart" without success.
6. Resarted OS to back to normal.

Is there any workaround instead of restart a system?
I was trying below without progress as well.

# ifconfig igb0 down
# ifconfig igb0 up

I think the problem is when the network card loses ethernet link and then errors occurs with non-working interface?!
Comment 2 John Delano 2019-08-27 13:03:33 UTC
I am seeing the problem as well on RELEASE-p10. It is affecting IGB and IX interfaces. The problem occurs when I disconnect an active network cable. After about 10 minutes, the error messages begin. Nothing other than a reboot will restore the interface to working after this occurs.

Connecting the cable back to the interface, ifconfig still reports the cable status as "no carrier".

I am going to try to make a script to down/up all the interfaces according to the interface connection status to mitigate this until a fix is available.
Comment 3 John Delano 2019-08-27 14:24:25 UTC
I have discovered that if I unplug the ethernet cable, ifconfig still shows the cable is "status: active"!

FreeBSD 11 immediately goes to "status: no carrier" when I do the same.
Comment 4 76nemo76 2019-09-01 09:35:48 UTC
I have the same problem with FreeBSD12 patch level 7.

The machine has 4 igb ports running as a bridge. Whe the
physical network switch (where the IGB card and thus 
the bridge is connected) is switched off, I get the messages:
   Jul 29 14:04:20 oche kernel: igb0: TX(0) desc avail = 1024, pidx = 0
   Jul 29 14:04:51 oche syslogd: last message repeated 18 times

and nothing, except a reboot, makes the bridge (and network card) functionnal again.

This parameter is set in the "/boot/loader.conf"
--------------------------/boot/loader.conf (extract)---------------------------
#
#Increase the memory allocated for the network card
#driver. Useful/necessary to take in account
#the 4 network card (instead of one).
#See this link:
#  https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshootil
#for some explanations.
#
kern.ipc.nmbclusters="131072"

here is the dmesg.txt of the machine:

------------------------------dmesg-------------------------------------------
---<<BOOT>>---
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-RELEASE-p7 GENERIC amd64
FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
VT(vga): resolution 640x480
CPU: AMD GX-412TC SOC                                (998.15-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x730f01  Family=0x16  Model=0x30  Stepping=1
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x3ed8220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C>
  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  AMD Features2=0x1d4037ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,Topology,PNXC,DBE,PTSC,PL2I>
  Structured Extended Features=0x8<BMI1>
  XSAVE Features=0x1<XSAVEOPT>
  SVM: NP,NRIP,AFlush,DAssist,NAsids=8
  TSC: P-state invariant, performance statistics
real memory  = 4294967296 (4096 MB)
avail memory = 4092137472 (3902 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <CORE   COREBOOT>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
ioapic1: Changing APIC ID to 5
ioapic0 <Version 2.1> irqs 0-23 on motherboard
ioapic1 <Version 2.1> irqs 24-55 on motherboard
Launching APs: 1 2 3
Timecounter "TSC" frequency 998147250 Hz quality 1000
random: entropy device external interface
netmap: loaded module
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0xffffffff810f9a00, 0) error 19
kbd0 at kbdmux0
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
acpi0: <CORE COREBOOT> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x818-0x81b on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.1 on pci0
pcib1: failed to allocate initial I/O port window: 0x1000-0x1fff
pci1: <ACPI PCI bus> on pcib1
igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> mem 0xfe500000-0xfe51ffff,0xfe520000-0xfe523fff at device 0.0 on pci1
igb0: attach_pre capping queues at 2
igb0: using 1024 tx descriptors and 1024 rx descriptors
igb0: msix_init qsets capped at 2
igb0: pxm cpus: 4 queue msgs: 4 admincnt: 1
igb0: using 2 rx queues 2 tx queues 
igb0: Using MSIX interrupts with 3 vectors
igb0: allocated for 2 tx_queues
igb0: allocated for 2 rx_queues
igb0: Ethernet address: 00:0d:b9:4e:88:10
igb0: netmap queues/slots: TX 2/1024, RX 2/1024
pcib2: <ACPI PCI-PCI bridge> at device 2.2 on pci0
pci2: <ACPI PCI bus> on pcib2
igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x2000-0x201f mem 0xfe600000-0xfe61ffff,0xfe620000-0xfe623fff at device 0.0 on pci2
igb1: attach_pre capping queues at 2
igb1: using 1024 tx descriptors and 1024 rx descriptors
igb1: msix_init qsets capped at 2
igb1: pxm cpus: 4 queue msgs: 4 admincnt: 1
igb1: using 2 rx queues 2 tx queues 
igb1: Using MSIX interrupts with 3 vectors
igb1: allocated for 2 tx_queues
igb1: allocated for 2 rx_queues
igb1: Ethernet address: 00:0d:b9:4e:88:11
igb1: netmap queues/slots: TX 2/1024, RX 2/1024
pcib3: <ACPI PCI-PCI bridge> at device 2.3 on pci0
pci3: <ACPI PCI bus> on pcib3
igb2: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x3000-0x301f mem 0xfe700000-0xfe71ffff,0xfe720000-0xfe723fff at device 0.0 on pci3
igb2: attach_pre capping queues at 2
igb2: using 1024 tx descriptors and 1024 rx descriptors
igb2: msix_init qsets capped at 2
igb2: pxm cpus: 4 queue msgs: 4 admincnt: 1
igb2: using 2 rx queues 2 tx queues 
igb2: Using MSIX interrupts with 3 vectors
igb2: allocated for 2 tx_queues
igb2: allocated for 2 rx_queues
igb2: Ethernet address: 00:0d:b9:4e:88:12
igb2: netmap queues/slots: TX 2/1024, RX 2/1024
pcib4: <ACPI PCI-PCI bridge> at device 2.4 on pci0
pci4: <ACPI PCI bus> on pcib4
igb3: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x4000-0x401f mem 0xfe800000-0xfe81ffff,0xfe820000-0xfe823fff at device 0.0 on pci4
igb3: attach_pre capping queues at 2
igb3: using 1024 tx descriptors and 1024 rx descriptors
igb3: msix_init qsets capped at 2
igb3: pxm cpus: 4 queue msgs: 4 admincnt: 1
igb3: using 2 rx queues 2 tx queues 
igb3: Using MSIX interrupts with 3 vectors
igb3: allocated for 2 tx_queues
igb3: allocated for 2 rx_queues
igb3: Ethernet address: 00:0d:b9:4e:88:13
igb3: netmap queues/slots: TX 2/1024, RX 2/1024
pci0: <encrypt/decrypt> at device 8.0 (no driver attached)
xhci0: <AMD FCH USB 3.0 controller> mem 0xfeb22000-0xfeb23fff at device 16.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Unable to map MSI-X table 
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
ahci0: <AMD Hudson-2 AHCI SATA controller> port 0x5010-0x5017,0x5020-0x5023,0x5018-0x501f,0x5024-0x5027,0x5000-0x500f mem 0xfeb25000-0xfeb253ff at device 17.0 on pci0
ahci0: AHCI v1.30 with 2 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ehci0: <AMD FCH USB 2.0 controller> mem 0xfeb25400-0xfeb254ff at device 19.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
usbus1: 480Mbps High Speed USB v2.0
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
sdhci_pci0: <Generic SD HCI> mem 0xfeb25500-0xfeb255ff at device 20.7 on pci0
sdhci_pci0: 1 slot(s) allocated
mmc0: <MMC/SD bus> on sdhci_pci0
amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb5
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
orm0: <ISA Option ROM> at iomem 0xef000-0xeffff pnpid ORM0000 on isa0
uart1: <16550 or compatible> at port 0x2f8 irq 3 on isa0
uart1: non-PNP ISA device will be removed from GENERIC in FreeBSD 12.
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
fuse-freebsd: version 0.4.4, FUSE ABI 7.8
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ugen0.1: <0x1022 XHCI root HUB> at usbus0
ugen1.1: <AMD EHCI root HUB> at usbus1
uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub1: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
mmcsd0: 64GB <SDHC EC2QT 3.0 SN C54F610D MFG 12/2017 by 27 SM> at mmc0 50.0MHz/4bit/65535-block
uhub0: 4 ports with 4 removable, self powered
uhub1: 2 ports with 2 removable, self powered
ugen1.2: <vendor 0x0438 product 0x7900> at usbus1
uhub2 on uhub1
uhub2: <vendor 0x0438 product 0x7900, class 9/0, rev 2.00/0.18, addr 2> on usbus1
uhub2: 4 ports with 4 removable, self powered
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <Samsung SSD 860 EVO mSATA 500GB RVT41B6Q> ACS-4 ATA SATA 3.x device
ada0: Serial Number S41NNB0K101594V
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 476940MB (976773168 512 byte sectors)
Trying to mount root from zfs:zroot/ROOT/default []...
GEOM_ELI: Device ada0.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device mmcsd0p3.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: hardware
bridge0: Ethernet address: 02:91:bc:80:6d:00
lo0: link state changed to UP
igb0: promiscuous mode enabled
bridge0: link state changed to UP
igb1: promiscuous mode enabled
igb2: promiscuous mode enabled
igb3: promiscuous mode enabled
igb3: link state changed to UP
intsmb0: <AMD FCH SMBus Controller> at device 20.0 on pci0
smbus0: <System Management Bus> on intsmb0
Security policy loaded: MAC/ntpd (mac_ntpd)
Comment 5 Ozkan KIRIK 2019-09-04 09:09:18 UTC
I have tested same problem with;

- Intel i210 (igb)
- Intel i350 (igb)
- Intel x540 (ixgbe)
- Intel x520 (ixgbe)

all cards above. When the active cable uplugged, TX desc avail messages starting to repeat and ifconfig down/up sometimes works sometime not works. Usually a reboot needs to start working again.

But ixl cards (Intel x710 ie) are working stable. There is no such problem about ixl.

In addition to cable unplug, mediaselect configuration change also triggers TX desc avail bug.
Comment 6 Krzysztof Galazka 2019-09-04 17:15:56 UTC
I think it should be fixed by this patch: https://reviews.freebsd.org/D18545. Unfortunately it was discovered too late to push it to 12.0 but it is MFCed to 12-STABLE. Could you try with STABLE?
Comment 7 John Delano 2019-09-06 16:43:21 UTC
I just applied the mentioned patch and the interface (IGB) does not immediately fail, but if unplugged for a few minutes, the error messages begin and the interface goes hard down with no method of recovery other than a reboot.

This makes 12.0 Stable currently unusable as a server.
Comment 8 76nemo76 2019-09-08 11:10:35 UTC
Finally I have compiled and installed FreeBSD Prerelease 12.1. A little bit long and painful (about 14 hours of compilation). 

The results are better now. It seems that if the cable is disconnected from the machine or the network switch is unplugged the connection is no more lost.

However, when passing the command "ifconfig -a" the network card (where the cable is disconnected) is still marked as active. To see the igb0 card no more marked as "active", I have to pass the commands (this card is a member of a bridge)
                   1) ifconfig bridge0 down
                   2) ifconfig igb0 down
                   3) ifconfig -a indicates yet that the connection
                                  is no more "active"
                   3) ifconfig igb0 up
                   4) ifconfig bridge0 up

Since someone has indicated that after a while when the cable is disconnected the probleam is present again, I have to do more check before being certain that this problem is really solved.
Comment 9 76nemo76 2019-09-09 05:31:37 UTC
After compiling the Prerelease 12.1, the problem is a little bit different. As somebody else has noticed, when the network card is disconnected the connection seems to work for quite a long time (about 30 minutes in my longest test).

But if you unplug the network switch for longer (one night for instance), a similar but different message (this time it is about a watchdog) is displayed.

The message produced is:
igb0: Watchdog timeout (TX: 0 desc avail: 1024 pidx: 0) -- resetting

Is it possible to suppress the watchdog?
Comment 10 76nemo76 2019-09-10 09:01:02 UTC
(In reply to 76nemo76 from comment #9)
After further tests, I noticed that if many cards are available in the machine, even if the card where the cable is disconnected is unavaible with the message about the wtachdog, it is still possible to do following things:
            1) ifconfig igb0 down
            2) reconnect the cable to igb1

The card igb1 is functionnal and the display of the messages disappear.

As soon as the command "ifconfig igb0 up" is passed, the error messages are displayed again. A full reboot correct the problem.
Comment 11 Piotr Pietruszewski 2019-09-10 15:58:16 UTC
We were able to partially reproduce this issue in our environment. We will continue trying to fully reproduce it and investigate the root cause.
Comment 12 Kubilay Kocak freebsd_committer freebsd_triage 2019-09-20 04:24:43 UTC
*** Bug 240658 has been marked as a duplicate of this bug. ***
Comment 13 Krzysztof Galazka 2019-09-24 09:05:25 UTC
The watchdog message is cased by an issue in the iflib framework. The queue hang detection code does not check the state of the interface. We are working on a fix in this review: https://reviews.freebsd.org/D21712

And this patch: https://reviews.freebsd.org/D21769 should fix a problem with em/igb not reporting current link state.
Comment 14 Kubilay Kocak freebsd_committer freebsd_triage 2019-09-24 11:44:39 UTC
(In reply to Krzysztof Galazka from comment #13)

If this needs to / should make it into 12.1-RELEASE, please add bug 240700 to this issues Blocks: field

In doing so, it may also be worth adding re@ to the reviews for their approval to merge to releng/12.1
Comment 15 Harald Schmalzbauer 2019-09-24 17:24:49 UTC
(In reply to Krzysztof Galazka from comment #13)

Thanks four your D21769 fix.  Unfortunately it doesn't solve the link-state-change-issue.  Please see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240658
I described why I re-opened the ticket as a separate report.

Thanks,

-harry
Comment 16 John Delano 2019-09-24 20:43:27 UTC
Odd, I applied this (https://reviews.freebsd.org/D21769) manually, recompiled the kernel and it seems to have fixed the state change and the buffer warnings.

I also have https://reviews.freebsd.org/D18545 applied as well.

I have not installed https://reviews.freebsd.org/D21712
Comment 17 Rodney W. Grimes freebsd_committer 2019-09-24 22:32:09 UTC
(In reply to Kubilay Kocak from comment #14)
> If this needs to / should make it into 12.1-RELEASE, please add bug 240700 to this issues Blocks: field

I conncur, this is the correct thing to do.

> In doing so, it may also be worth adding re@ to the reviews for their approval > to merge to releng/12.1

Please do not add re@ to bug reports for the purpose of merge requesting, there is an official process in place that says ONCE you have committed a fix to head and merged to stable you need to send an email request.  Involving RE@ at the bug state is too early, only once an agreed fix has been committed, and merged back to the appropriate branch if needed possible with out authroization is completed should RE@ be involved, they simply do not have the time to read through a bug being evolved and dealt with.

Regards,
Rod (former RE@ member)
Comment 18 Harald Schmalzbauer 2019-09-25 07:25:12 UTC
(In reply to John Delano from comment #16)

Since the issue/fix (https://reviews.freebsd.org/D21712 , queue-detection) of this topic wasn't involved in your latest test, please see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240658#c5 for further discussion.

Thanks,

-harry
Comment 19 John Delano 2019-09-25 13:23:26 UTC
(In reply to Harald Schmalzbauer from comment #18)
I did apply https://reviews.freebsd.org/D21712 to have a complete fix, but the problem seemed solved before applying that fix.
Comment 20 76nemo76 2019-09-27 12:52:53 UTC
I am a little busy this week. I will apply the patches and test the system probably next week.

Against what sources (FreeBSD 12_1?) should the patches be applied?

Thanks
Comment 21 76nemo76 2019-10-01 06:37:34 UTC
(In reply to 76nemo76 from comment #20)
Finally I have applied the patches D21712 and D21769 on FreeBSD 12.1-PRERELEASE.

This was probably not exactly the sources where the patches have been created since I have to apply one patch manually (the other was applied with the "patch" command).

The result is better now but still with some strange errors. 
What I have done is:
          1) ping www.freebsd.org       The result was OK
          2) unplug the network cable   The command ifconfig was OK and
                                        the port marked as down
          3) plug again on an other 
             port the cable             The command ifconfig was OK
                                        and the port marked as up
          4) ping www.freebsd.org       The result was not OK (no network)
          5) pass the command  
             "service netif restart"    The result was OK
          6) ping other machines        The result was OK
          7) ping 127.0.0.1             The result was OK
          8) ping 172.16.0.11 
             (the address set on 
              the bridge)               The result was not OK

If I ping the address of the bridge (172.16.0.11) immediately after a reboot, it works.

-------------------log produced by the above manipulations------------------
root@oche:~ # ping www.freebsd.org
PING wfe0.nyi.freebsd.org (96.47.72.84): 56 data bytes
64 bytes from 96.47.72.84: icmp_seq=0 ttl=47 time=87.104 ms
64 bytes from 96.47.72.84: icmp_seq=1 ttl=47 time=87.136 ms
64 bytes from 96.47.72.84: icmp_seq=2 ttl=47 time=86.914 ms
^C
--- wfe0.nyi.freebsd.org ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 86.914/87.051/87.136/0.098 ms
root@oche:~ # ifconfig -a
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:10
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:11
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:12
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:13
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:91:bc:80:6d:00
        inet 172.16.0.11 netmask 0xfffffe00 broadcast 172.16.1.255
        inet6 2a02:168:c000:0:9937::1 prefixlen 64
        inet6 fd4f:bd4e:6d27:0:9937::1 prefixlen 64
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: igb3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 55
        member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 20000
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 55
        groups: bridge
        nd6 options=1<PERFORMNUD>
root@oche:~ # ifconfig -a
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:10
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:11
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:12
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:13
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:91:bc:80:6d:00
        inet 172.16.0.11 netmask 0xfffffe00 broadcast 172.16.1.255
        inet6 2a02:168:c000:0:9937::1 prefixlen 64
        inet6 fd4f:bd4e:6d27:0:9937::1 prefixlen 64
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: igb3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 55
        member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 20000
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 55
        groups: bridge
        nd6 options=1<PERFORMNUD>
root@oche:~ # ifconfig -a
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:10
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:11
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:12
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:13
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:91:bc:80:6d:00
        inet 172.16.0.11 netmask 0xfffffe00 broadcast 172.16.1.255
        inet6 2a02:168:c000:0:9937::1 prefixlen 64
        inet6 fd4f:bd4e:6d27:0:9937::1 prefixlen 64
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: igb3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 55
        member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 20000
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 55
        groups: bridge
        nd6 options=1<PERFORMNUD>
root@oche:~ # ping www.freebsd.org
ping: cannot resolve www.freebsd.org: Host name lookup failure
root@oche:~ # ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.207 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.236 ms
^C
--- 127.0.0.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.207/0.221/0.236/0.015 ms
root@oche:~ # ping 172.16.0.11
PING 172.16.0.11 (172.16.0.11): 56 data bytes
^C
--- 172.16.0.11 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
root@oche:~ # service netif restart
Stopping Network: lo0 igb0 igb1 igb2 igb3 bridge0.
lo0: flags=8048<LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb0: flags=8d02<BROADCAST,PROMISC,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:10
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1: flags=8d02<BROADCAST,PROMISC,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:11
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb2: flags=8d02<BROADCAST,PROMISC,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:12
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb3: flags=8d02<BROADCAST,PROMISC,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:13
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:91:bc:80:6d:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: igb3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 55
        member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 20000
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 55
        groups: bridge
        nd6 options=1<PERFORMNUD>
Destroyed clone interfaces: bridge0.
Created clone interfaces: bridge0.
ifconfig: BRDGADD igb0: File exists
Starting Network: lo0 igb0 igb1 igb2 igb3 bridge0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:10
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:11
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:12
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0d:b9:4e:88:13
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:91:bc:80:6d:00
        inet 172.16.0.11 netmask 0xfffffe00 broadcast 172.16.1.255
        inet6 2a02:168:c000:0:9937::1 prefixlen 64
        inet6 fd4f:bd4e:6d27:0:9937::1 prefixlen 64
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: igb3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 55
        member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 55
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge
        nd6 options=1<PERFORMNUD>
root@oche:~ # ping 172.16.0.11
PING 172.16.0.11 (172.16.0.11): 56 data bytes
^C
--- 172.16.0.11 ping statistics ---
7 packets transmitted, 0 packets received, 100.0% packet loss
root@oche:~ # ping 172.16.0.10
PING 172.16.0.10 (172.16.0.10): 56 data bytes
64 bytes from 172.16.0.10: icmp_seq=0 ttl=255 time=0.626 ms
64 bytes from 172.16.0.10: icmp_seq=1 ttl=255 time=0.421 ms
64 bytes from 172.16.0.10: icmp_seq=2 ttl=255 time=0.362 ms
64 bytes from 172.16.0.10: icmp_seq=3 ttl=255 time=0.397 ms
64 bytes from 172.16.0.10: icmp_seq=4 ttl=255 time=0.350 ms
64 bytes from 172.16.0.10: icmp_seq=5 ttl=255 time=0.351 ms
64 bytes from 172.16.0.10: icmp_seq=6 ttl=255 time=0.360 ms
64 bytes from 172.16.0.10: icmp_seq=7 ttl=255 time=0.351 ms
64 bytes from 172.16.0.10: icmp_seq=8 ttl=255 time=0.355 ms
64 bytes from 172.16.0.10: icmp_seq=9 ttl=255 time=0.317 ms
64 bytes from 172.16.0.10: icmp_seq=10 ttl=255 time=0.337 ms
^C
--- 172.16.0.10 ping statistics ---
11 packets transmitted, 11 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.317/0.384/0.626/0.081 ms
Comment 22 76nemo76 2019-10-01 08:38:53 UTC
After further tests, I think the problems I still had are no more 
related to the fact that the network card is unplugged.

If I plug it again in the same port, the network is  working
correctly. It should be a problem with the management of the bridge by FreeBSD.
If I plug the cable on an other port, the network is down.

By the way, I have installed the new world (for my former tests I have installed only the new kernel), but I am almost certain that this is not the reason of this better behaviour.

To be complete, I have still to test the disconnection for a very long peiod. In my previous tests, the problem was less serious after a break of some minutes but it becomes more serious after few hours.
Comment 23 76nemo76 2019-10-02 05:50:57 UTC
This morning I have been able to connect to the machine after the network switch
was switched off for the night. This has never worked until now. It seems then that this problem is now solved.