Bug 228302

(In reply to sandreas from comment #0)
Uname -a:
FreeBSD 11.1-RELEASE-p9 FreeBSD 11.1-RELEASE-p9 #0: Tue Apr  3 16:59:16 UTC 2018

Comment 2 Sean Bruno 2018-05-18 14:57:17 UTC

I do not think we enable WOL features by default.

Can you try:

ifconfig igb0 wol
ifconfig em0  wol

Then setting:
dev.igb.0.wake=1
dev.em.0.wake=1

Please post the results of this and I'll debug further.

Thank you for your quick reply... unfortunately that did not fix the problem. Tried both interfaces with multiple reboots and shutdowns. The wake on lan tool i use cannot cause the problem, because it works with other devices on the same switch. Heres the log (xxx-ed out the macs):

root@nas:~ # sysctl dev.igb.0.wake
dev.igb.0.wake: 1
root@nas:~ # sysctl dev.em.0.wake
dev.em.0.wake: 1
root@nas:~ # ifconfig igb0 wol
root@nas:~ # ifconfig em0 wol
root@nas:~ # ifconfig igb0
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether xxx
	hwaddr xxx
	inet 192.168.8.9 netmask 0xffffff00 broadcast 192.168.8.255 
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
root@nas:~ # ifconfig em0
em0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
	ether xxx
	hwaddr xxx
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: no carrier

Comment 4 Sean Bruno 2018-05-21 13:56:37 UTC

(In reply to sandreas from comment #3)
There are no "WOL" flags in the ifconfig output.  It doesn't look like your devices have wake on lan active.

Here are my devices:

root@:~ # ifconfig igb0 wol
root@:~ # ifconfig igb0
igb0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e535bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:db:93:e0
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active


root@:~ # ifconfig em0 wol
root@:~ # ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=85359b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,LRO,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
        ether 7c:05:07:0d:cf:e1
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10baseT/UTP <full-duplex>)
        status: active

I activated WOL in BIOS (i checked twice) and added 

dev.igb.0.wake=1
dev.em.0.wake=1

in /etc/sysctl.conf. Then i ran

ifconfig igb0 wol
ifconfig em0  wol

And tried different things with and without reboots. Nothing worked. Is there something else i could have missed? 


Is it possible, that it is not a hardware / hardware-version specific problem? In https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=208343 WOL-flags do not show up, although everything is configured correctly. 

Perhaps the ASRock Rack C236 WSI has newer Intel NICs and the freebsd driver does not support WOL on these specific device versions. I think i checked all possible settings and as i remember WOL was working on Linux (although i'm not 100% sure here, its been a while...)


If you need any debug output or if i should re-check, wether WOL is working on linux, just ask - should not be too much effort to boot a linux usb stick...

I also could provide a screenshot of BIOS settings, if it helps, but I'm 100% sure, WOL is enabled here.

Comment 6 Sean Bruno 2018-05-21 16:32:46 UTC

(In reply to sandreas from comment #5)
igb0@pci0:5:0:0:        class=0x020000 card=0x00028086 chip=0x15338086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'I210 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base rxf7900000, size 1048576, enabled
    bar   [1c] = type Memory, range 32, base rxf7a00000, size 16384, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks 
    cap 11[70] = MSI-X supports 5 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR NS
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 a0369fffffdb93e0
    ecap 0017[1a0] = TPH Requester 1

I definitely have one of the devices you have.

Let me fire up stable/11 and see if I can reproduce.

Ok, i'll recheck everything again in the meantime.

(In reply to sandreas from comment #7)

Please try this patch for i219. I don't have any idea for i210.

Index: sys/dev/e1000/if_em.c
===================================================================
--- sys/dev/e1000/if_em.c	(revision 334009)
+++ sys/dev/e1000/if_em.c	(working copy)
@@ -5411,7 +5411,8 @@
 	if ((adapter->hw.mac.type == e1000_pchlan)  ||
 	    (adapter->hw.mac.type == e1000_pch2lan) ||
 	    (adapter->hw.mac.type == e1000_pch_lpt) ||
-	    (adapter->hw.mac.type == e1000_pch_spt)) {
+	    (adapter->hw.mac.type == e1000_pch_spt) ||
+	    (adapter->hw.mac.type == e1000_pch_cnp)) {
 		error = em_enable_phy_wakeup(adapter);
 		if (error)
 			goto pme;

Ok, i re-checked every possible setting that i found.

1.) Hardware / Mainboard should support wake on lan according to the manual (http://asrock.pc.cdn.bitgravity.com/Manual/C236%20WSI.pdf), Page 4 (10 in PDF):
	
This motherboard supports Wake from on Board LAN. To use this function, please make
sure that the “Wake on Magic Packet from power off state” is enabled in Device Manager
> Intel® Ethernet Connection > Power Management. And the “PCI Devices Power On” is
enabled in UEFI SETUP UTILITY > Advanced > ACPI Configuration. After that, onboard
LAN1&2 can wake up S5 under OS.

The "Device Manager" Part is for Windows i think, but i attached BIOS Screenshots for the EFI ACPI Configuration.

2.) PCI Devices Power On is enabled as described above
[![pci-power-enabled](https://abload.de/thumb/pci_power_enabledsesj3.jpg)](http://abload.de/image.php?img=pci_power_enabledsesj3.jpg)

3.) Both LAN-Ports are Enabled (See lan_enabled.jpg)
[![lan enabled](https://abload.de/img/lan_enabledtvs5s.jpg)](http://abload.de/image.php?img=lan_enabledtvs5s.jpg)

4.) Sysctl and ifconfig

I configured:

dev.igb.0.wake=1
dev.em.0.wake=1

AND

ifconfig igb0 wol
ifconfig em0  wol

with and without reboots to ensure state is available... none worked. 



If theres something else i should check, feel free to point it out. Last things to check would be:

1.) The patch Kaho Toshikazu send me (thank you very much)
2.) Check for a newer BIOS / EFI Update 

But this could take some days.

(In reply to sandreas from comment #9)
Sorry... Markdown did not work... here are the direct links... 
LAN Enabled: https://abload.de/img/lan_enabledtvs5s.jpg

PCI Power on Enabled: https://abload.de/img/pci_power_enabledsesj3.jpg

(In reply to sandreas from comment #9)

These sysctl don't have relation to WOL. Please don't touch them. 
> dev.igb.0.wake=1
> dev.em.0.wake=1

Try `ifconfig igb0 up wol` to add "up" option.

How about do the LEDs of LAN ports operate at power-down state?

Comment 12 Sean Bruno 2018-05-23 21:20:50 UTC

(In reply to sandreas from comment #9)
When your system boots, do you get an ethernet card option ROM prompt? I've seen controllers that have WOL turned off there.

> Try `ifconfig igb0 up wol` to add "up" option.

No change...

root@nas:~ # ifconfig igb0 up wol
root@nas:~ # ifconfig igb0
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether xxx
	hwaddr xxx
	inet 192.168.8.9 netmask 0xffffff00 broadcast 192.168.8.255 
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active

> How about do the LEDs of LAN ports operate at power-down state?

i use: shutdown -p now
leds are off... see here: https://abload.de/img/dsc06115a7utj.jpg


> When your system boots, do you get an ethernet card option ROM prompt?
No, nothing special i saw but I'll re-check this tomorrow


Another thing: BIOS Update did not change anything, i'm on 2.60 (latest) now.


Things left to try:
- Compile Kernel with patch (take some days to set up everything for this, not much time atm)
- Look for an ethernet card option ROM prompt

If there is something else left, i would appreciate comments.

(In reply to sandreas from comment #13)

Thanks for your reports.

`ifconfig -m igb0` can show "capabilities". Does it have any WOL relational 
capabilities? If it does not have, your system failed to apply the patch.
11.1-RELEASE does not have WOL functionality in its igb driver.
Can you try 11.2-BETA or latest 11-stable ?

Ok, here are the next steps i took:

# IFCONFIG -m:

ifconfig -m igb0
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=7507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether xxx
	hwaddr xxx
	inet 192.168.8.9 netmask 0xffffff00 broadcast 192.168.8.255 
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	supported media:
		media autoselect
		media 1000baseT
		media 1000baseT mediaopt full-duplex
		media 100baseTX mediaopt full-duplex
		media 100baseTX
		media 10baseT/UTP mediaopt full-duplex
		media 10baseT/UTP
		
		
# FREEBSD UPGRADE (LATEST STABLE):
FreeBSD nas.xxx 11.1-RELEASE-p10 FreeBSD 11.1-RELEASE-p10 #0: Tue May  8 05:21:56 UTC 2018     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

# ETHERNET CARD OPTION ROM:
I tried to find any hints for an ethernet card option ROM prompt, but i did not find anything.
There indeed is an Intel ME Prompt (CTRL+P), where i can't do anything but attempt to login (Login tries to set a new password, which leads to an "Error: Cannot set password", i can't do something else)

I found a tool from Intel to change Ethernet Boot Options (https://downloadcenter.intel.com/download/19186),
but it seems to be only for PCI cards, not for onboard devices.


# NEXT STEPS:
- Boot linux or windows to verify, WOL does work with at least one of these systems and it is not a hardware defect or misconfiguration in BIOS
- Compile FreeBSD Kernel with patch or use newer Kernel
- Post a Video of the BIOS settings and boot process, if it helps to find out, what could be wrong

Ok, next step done, checked WOL on linux (ubuntu 18.04), no BIOS or EFI changes, works like a charm:
- Booted Ubuntu 18.04
- Used ethtool to enable WOL on Magic Packet (Supports Wake-on: pumbg, see below)
- shutdown -P now
- system is powered off, orange/left LAN LED is on (flashing)
- WOL powers on the machine like expected

# Next step (and last one;)
- Compile newer freebsd kernel and check, if problem is fixed, otherwise report back

# ubuntu shell input
root@ubuntu:/etc/default# ifconfig enp2s0
enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.8.9  netmask 255.255.255.0  broadcast 192.168.8.255
        inet6 fe80::fdf0:f12:a54e:ffab  prefixlen 64  scopeid 0x20<link>
        ether xxx  txqueuelen 1000  (Ethernet)
        RX packets 4614  bytes 3574528 (3.5 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2310  bytes 217550 (217.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xdf000000-df07ffff 
		
root@ubuntu:~# ethtool enp2s0
Settings for enp2s0:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Supported pause frame use: Symmetric
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Advertised pause frame use: Symmetric
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 1
	Transceiver: internal
	Auto-negotiation: on
	MDI-X: off (auto)
	Supports Wake-on: pumbg
	Wake-on: g
	Current message level: 0x00000007 (7)
			       drv probe link
	Link detected: yes
	
	
root@ubuntu:~# ethtool -s enp2s0 wol g

OMG it finally works... Upgrade to FreeBSD 11.2-BETA3 finally fixed the problem - what a journey. Thank you guys for all the great help...

Comment 18 commit-hook 2018-05-26 20:03:23 UTC

A commit references this bug:

Author: sbruno
Date: Sat May 26 20:02:39 UTC 2018
New revision: 334246
URL: https://svnweb.freebsd.org/changeset/base/334246

Log:
  Activate Wake On Lan features for Ice Lake and Cannon Lake devices.

  This is a direct commit to stable/11 as its not needed in -current.

  PR:		228302
  Submitted by:	Kaho Toshikazu <kaho@elam.kais.kyoto-u.ac.jp>
  Approved by:	re (kib)

Changes:
  stable/11/sys/dev/e1000/if_em.c

Comment 19 commit-hook 2018-05-26 21:59:00 UTC

A commit references this bug:

Author: sbruno
Date: Sat May 26 21:58:43 UTC 2018
New revision: 334251
URL: https://svnweb.freebsd.org/changeset/base/334251

Log:
  Activate Wake On Lan features for Ice Lake and Cannon Lake devices.

  This is a direct commit to stable/10 as its not needed in -current.

  PR:		228302
  Submitted by:	Kaho Toshikazu <kaho@elam.kais.kyoto-u.ac.jp>

Changes:
  stable/10/sys/dev/e1000/if_em.c