Bug 238411

Summary: igb(4): Wake on Lan not working with Intel I210
Product: Base System Reporter: Nikita Druba <nikita>
Component: kernAssignee: freebsd-net (Nobody) <net>
Status: Open ---    
Severity: Affects Some People CC: abraham, dutchman01, freebsd, kaho, net, yyv83
Priority: --- Keywords: IntelNetworking, needs-qa, regression
Version: 12.0-RELEASEFlags: koobs: maintainer-feedback? (freebsd)
koobs: mfc-stable13?
koobs: mfc-stable12?
Hardware: Any   
OS: Any   
Attachments:
Description Flags
dmesg.boot
none
pciconf -lv
none
patch:force to enable wol none

Description Nikita Druba 2019-06-08 03:34:02 UTC
Hi to all! 

After updating to 12.0-RELEASE wake on lan function stoped working in my configuration.
I have Intel I210 Gigabit Network Connection, driver igb.
rc.conf at server that need to wake up:

ifconfig_igb0="-wol_ucast -wol_mcast wol_magic"

powered on at another server by:

wake igb0 11:22:33:44:55:66;

I tryed to change sysctl by
dev.igb.0.wake=0 -> 1
but no result.

Can anybody confirm this bug?
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2019-06-11 05:18:41 UTC
@NIkita Could you please provide additional information, including:

- Exact version (uname -a) the update was 'from'
- Exact system versions now (uname -a)
- /var/run/dmesg.boot output (as an attachment)
- pciconf -lv output (as an attachment
Comment 2 Nikita Druba 2019-06-11 09:39:04 UTC
Created attachment 204981 [details]
dmesg.boot
Comment 3 Nikita Druba 2019-06-11 09:39:49 UTC
Created attachment 204982 [details]
pciconf -lv
Comment 4 Nikita Druba 2019-06-11 09:40:27 UTC
(In reply to Kubilay Kocak from comment #1)
updated from 11.1-RELEASE r318763
to FreeBSD SERVERBAK 12.0-RELEASEp5 FreeBSD 12.0-RELEASE-p5 r348792 MYKERNEL amd64
Comment 5 Catwoolfii 2020-10-31 11:36:07 UTC
A similar problem is reproduced on 12.2-release with Intel I210 hardware
Comment 6 Dutchman01 2021-04-04 01:49:20 UTC
Intel has released new/updated drivers for freebsd
Version: 2.5.16 (Latest) Date: 8/11/2020

don't know why not yet updated on freebsd/src
it does fix some issues for I210 among others
Comment 8 Abraham122x 2021-04-24 22:43:55 UTC
Compiled new Intel driver igb-2.5.18.tar and also with them WOL is not working.

Any idea what could be the issue?
Comment 9 Abraham122x 2021-04-25 08:36:15 UTC
A very unprofessional approach I did yesterday, but just as curiosity.

The original Intel driver I compiled yesterday are not showing the WOL flags anymore for my NICS.  Therefore I commented following "if" source code line in "if_igb.c":

/* APME bit in EEPROM is mapped to WUC.APME */
eeprom_data = E1000_READ_REG(&adapter->hw, E1000_WUC) & E1000_WUC_APME;
/* if (eeprom_data) */
  adapter->wol = E1000_WUFC_MAG;

After that, when I shut down the machine, the LED's of the NICS where still active (as it should be with WOL). When I sent a WOL magic packet to the NIC, the corresponding LED of the NIC was flashing. So the packet was arrived and accepted, but unfortunately the machine was not powered on.

Does this someone help for further analysis?
Comment 10 Catwoolfii 2021-04-25 14:48:34 UTC
I followed your example and commented out the line: https://github.com/freebsd/freebsd-src/blob/releng/13.0/sys/dev/e1000/if_em.c#L3681

After that, I rebuilt and reinstalled the kernel and wol worked with Intel I210 hardware.
Comment 11 Abraham122x 2021-04-25 16:42:44 UTC
Sorry I am no expert in FreeBSD and compiling. I am using an OPNsense installation which is based on FreeBSD 12.1-RELEASE-p15-HBSD. So far I tried various FreeBSD installations and none worked out of-the-box.

Currently I managed to compile the "igb-2.5.18.tar.gz" driver and overrode it using 'if_igb_updated_load="YES"' in my OPNsense installation. The instructions I used are documented here: https://forum.opnsense.org/index.php?topic=21517.msg101148#msg101148

Which driver should I use to compile instead? The "em" from Intel (em-7.7.8.tar.gz) are very old.
Comment 12 Catwoolfii 2021-04-25 17:14:25 UTC
Still using "em" driver.

I would suggest building the module from opnsense sources:
1. Download and unpack the source code:
wget https://github.com/opnsense/src/archive/refs/heads/stable/21.1.tar.gz
tar zxvf 21.1.tar.gz --strip 1 -C /usr/src
2. In these sources, you also need to comment out in file /usr/src/sys/dev/e1000/if_em.c
Look like https://github.com/opnsense/src/blob/stable/21.1/sys/dev/e1000/if_em.c#L3661
3. Now you can build new module:
cd /usr/src/modules/em
make
make install

New kernel module should appear in /boot/modules
Comment 13 Kaho Toshikazu 2021-04-25 19:11:32 UTC
(In reply to Abraham122x from comment #9)
These lines check the enable bit of WOL in the flash ROM.
If you would like to neglect flash ROM setting,
I think your suggestion is reasonable.

If the default of WOL is still disabled when WOL is diabled on the flash ROM,
you should not change these lines.

In the function em_enable_wakeup(), 
	if ((if_getcapenable(ifp) & IFCAP_WOL_MAGIC) == 0)
		adapter->wol &= ~E1000_WUFC_MAG;

after these line, please insert like this,
	else
		adapter->wol |= E1000_WUFC_MAG;
if you don't want to change default settings.
Comment 14 Abraham122x 2021-04-25 21:29:24 UTC
@Catwoolfii, thanks for guiding me how to compile. I managed it and now WOL is working on my machine as well. This is really great!

@Kaho, I tried your suggestion and can confirm that also this source-code change works.

Could we now also find the root cause of this issue resp. make me understand why this is not working properly?

I see that this if statement 
  if ((if_getcapenable(ifp) & IFCAP_WOL_MAGIC) == 0)
seems to be always false in my case, although in the interface capabilities and options the WOL_MAGIC flag is set.

I will try some different source code variants to understand which parameter is causing my problem.
Comment 15 Kaho Toshikazu 2021-04-25 22:27:22 UTC
(In reply to Abraham122x from comment #14)

> I see that this if statement 
>   if ((if_getcapenable(ifp) & IFCAP_WOL_MAGIC) == 0)
> seems to be always false in my case, although in the interface capabilities and  options the WOL_MAGIC flag is set.

It is not correct, "adapter->wol" is always 0,
if the flash ROM disables WOL regardless the device has a WOL functionality.

The definition IGB_CAPS has IFCAP_WOL flag, and the igb device is always
reported WOL capable to system, and ifconfig command can set or unset WOL flags.
But if the flash ROM indicates WOL disabled, the variable adapter->wol is
always 0 regardless indicating WOL enabled in the variable isc_capenable.
The isc_capenable preserves status which flags are enabled or disabled.
Comment 16 Abraham122x 2021-04-25 23:03:18 UTC
Let's ask differently in reference to the non-modified FreeBSD drivers. 

Regardless if I set "ifconfig igb0 wol" or not, I cannot enable the WOL feature on my NICS. So for me it's not clear with which functionality from OS perspective I can influence the field "adapter->wol".

My only workaround I found is to change the variable directly in the source code of the driver and overrule it.

What I see from my theoretical analysis, the variables are stored in ROM at shutdown -> method "em_enable_phy_wakeup":
  E1000_WRITE_REG(hw, E1000_WUC, E1000_WUC_PHY_WAKE | E1000_WUC_PME_EN | E1000_WUC_APME);
  E1000_WRITE_REG(hw, E1000_WUFC, adapter->wol);

At startup the register is read again:
  apme_mask = E1000_WUC_APME;
  eeprom_data = E1000_READ_REG(&adapter->hw, E1000_WUC);
  if (eeprom_data & apme_mask)
    adapter->wol = (E1000_WUFC_MAG | E1000_WUFC_MC);

As the variable "adapter->wol" at startup seems to be zero, it means that the flag E1000_WUC_APME is not set, although it was written at shutdown.

So seen from my hardware perspective, the ROM settings are not properly recognized at startup, hence "adapter->wol" is always zero. The variable cannot be influenced by anything known to me.

Is my analysis correct? Did I miss something?

P.S: I have an E1000_DEV_ID_I210_COPPER_FLASHLESS installed.
Comment 17 Kaho Toshikazu 2021-04-25 23:59:25 UTC
(In reply to Abraham122x from comment #16)
Yes, I think your analysis is correct. i210 flashless is always disabled WOL
at first time on booting form power off. FreeBSD igb driver picks up
the status of WOL, and the variable adapter->wol is initialized to 0.
If adapter->wol is 0, the function "em_enable_phy_wakeup" is not called,
then WOL is not enabled.

comment #9 suggests to initialize adapter->wol to E1000_WUFC_MAG instead of 0.
comment #13 suggests to rewrite adapter->wol if WOL is enabled in the flags.
Comment 18 Kubilay Kocak freebsd_committer freebsd_triage 2021-04-26 01:02:44 UTC
^Triage: Original report for 12.0, set mfc-* accordingly unless positive evidence provided its not an issue or reproducible on stable/12
Comment 19 Kaho Toshikazu 2021-04-26 02:02:41 UTC
(In reply to Kubilay Kocak from comment #18)
See also bug #208343, comment #54 .
Comment 20 Abraham122x 2021-04-26 17:55:45 UTC
Not sure if any action is required from my side. I am not familiar with the mfc-* flags.

For myself I will change the driver to my needs, but I hope that some expert can solve the bug for everybody facing this issue. 

In case I can help, please let me know. Happy to support.

FYI, I tried a lot of versions between FreeBSD 11 and 13. Everywhere the same issue with my hardware.
Comment 21 Kaho Toshikazu 2021-05-01 10:47:17 UTC
(In reply to Abraham122x from comment #20)

FreeBSD 12 and 13 have almost same code but 11 has difference.
I think that wol capability is not set with these type devices on FreeBSD 11,
then you can not toggle enable/disable wol by using ifconfig.
Comment 22 Abraham122x 2021-05-02 08:04:34 UTC
What I can remember of my tests for my flashless NIC:

 - Some FreeBSD 11 releases: possible to set WOL flags with ifconfig (important is to avoid MCAST and UCAST flags, as they are triggered automatically seconds after shutdown and boot the device)

 - FreeBSD 12/13: no chance anymore to influence the driver settings in regard to WOL flags; the flags are set and visible in ifconfig, but they have no effect.

Don't understand the logic that the ifconfig flags have less priority than the ROM settings. My proposal would be to use the ROM settings as default after boot (and set the ifconfig flags accordingly) and at shutdown the ifconfig settings should become priority.
Comment 23 Kaho Toshikazu 2021-05-02 13:32:26 UTC
Created attachment 224609 [details]
patch:force to enable wol

(In reply to Abraham122x from comment #22)

I'm sorry, you are right. Not all but some FreeBSD 11 igb driver can use wol
despite from disabled wol at boot time. FreeBSD 11 em driver can not use wol
if wol is disabled at boot time. The igb driver integrated to em driver
at FreeBSD 12-current, and now FreeBSD 12/13 em driver has almost same default
at FreeBSD 11. At only very short period, FreeBSD 12 integrated em driver
could force to enable wol. If you would like to restore this function,
please try the patch.
Comment 24 Abraham122x 2021-05-02 18:33:50 UTC
(In reply to Kaho Toshikazu from comment #23)

I applied your patch (except sub "em_get_wakeup" because I don't have the new enumerations in my package) and can confirm it does exactly what I need: ifconfig igb1 wol_packet sets the WOL option and the single NIC gets in WOL state when powered down.

Perfect. Thanks.