Bug 231151 - ifconfig: unable to disable rxcsum, rxcsum6 and vlanhwcsum on em driver
Summary: ifconfig: unable to disable rxcsum, rxcsum6 and vlanhwcsum on em driver
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: CURRENT
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-net mailing list
URL:
Keywords: IntelNetworking
Depends on:
Blocks:
 
Reported: 2018-09-04 12:25 UTC by Shirkdog
Modified: 2019-01-17 19:51 UTC (History)
4 users (show)

See Also:


Attachments
vlan_hwcsum flag sets changeable. (1.89 KB, patch)
2018-09-11 12:20 UTC, Kaho Toshikazu
no flags Details | Diff
Allow rxcsum to be disabled, but keep it tied to rxcsum6 (1.29 KB, patch)
2018-09-12 14:19 UTC, Stephen Hurd
no flags Details | Diff
Update to include WoL and vlan hwcsum fixes (2.56 KB, patch)
2018-09-12 17:54 UTC, Stephen Hurd
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Shirkdog 2018-09-04 12:25:56 UTC
In 12-CURRENT, I would like to be able to turn off all NIC offloading features (TSO,LRO,Checksums) to ensure I am reading everything as it appears off the wire for network security (Bro NSM).

Putting all of the flags together, this is what I have as the line that should disable all of the features

/sbin/ifconfig em0 -rxcsum -txcsum -tso4 -tso6 -lro -rxcsum6 -txcsum6 -vlanhwcsum -vlanhwtso

It appears most work with and are enabled and disabled, the exceptions are rxcsum, rxcsum6 and vlanhwcsum. I cannot disable vlanhwcsum at all, and when you disable rxcsum, it enables rxcsum6 and vice versa.

Example:

# ifconfig em0
em0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500                                                                                                                        
        options=810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>                                                                                                                                   
        ether 68:05:ca:xx:xx:xx                                                                                                                                                                                   
        media: Ethernet autoselect (1000baseT <full-duplex>)                                                                                                                                                       
        status: active                                                                                                                                                                                             
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>                                                                                                                                                       
# ifconfig em0 -rxcsum                                                                                                                                                                               
# ifconfig em0  

em0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500                                                                                                                        
        options=a10098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6>                                                                                                                              
        ether 68:05:ca:xx:xx:xx                                                                                                                                                                               
        media: Ethernet autoselect (1000baseT <full-duplex>)                                                                                                                                                       
        status: active                                                                                                                                                                                             
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
# ifconfig em0 -rxcsum6                                                                                                                                                                                                                                                                                                                                   
# ifconfig em0                                                                                                                                                                                       
em0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500                                                                                                                        
        options=810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>                                                                                                                                   
        ther 68:05:ca:xx:xx:xx                                                                                                                                                                                   
        media: Ethernet autoselect (1000baseT <full-duplex>)                                                                                                                                                       
        status: active                                                                                                                                                                                             
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> 

I see the same behavior with this driver on two separate cards.

em0@pci0:1:0:0: class=0x020000 card=0x115e8086 chip=0x105e8086 rev=0x06 hdr=0x00                                                                                                                                   
    vendor     = 'Intel Corporation'                                                                                                                                                                               
    device     = '82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications)'                                                                                                                         
    class      = network                                                                                                                                                                                           
    subclass   = ethernet

em0@pci0:0:31:6:        class=0x020000 card=0x06d91028 chip=0x15b78086 rev=0x31 hdr=0x00                                                                                                                           
    vendor     = 'Intel Corporation'                                                                                                                                                                               
    device     = 'Ethernet Connection (2) I219-LM'                                                                                                                                                                 
    class      = network                                                                                                                                                                                           
    subclass   = ethernet


If this is "by design" in the driver, then this bug can be closed, but I would assume that if you want to disable all receive side offloading, you should be able to disable it for both IPv4 and IPv6. Also, if this is a driver issue (iflib), then I can move the bug to kernel component.
Comment 1 Kaho Toshikazu 2018-09-11 12:20:31 UTC
Created attachment 197023 [details]
vlan_hwcsum flag sets changeable.

(In reply to Shirkdog from comment #0)

I think it is only a iflib bug. See bug #220698 .
Please try the attached patch.
Comment 2 Stephen Hurd freebsd_committer 2018-09-11 18:17:51 UTC
Yeah, for the RX checksum stuff, it's pretty clear that if *either* rxcsum or rxcsum6 is set in the target capabilities, both will be forced on.  Since ifconfig does a separate ioctl for each command-line flag, there's never a set that has both set to zero, so neither ever gets disabled.

I'll see if I can figure out why r307562 made this change, and come up with a Better Way to twi the two flags together.
Comment 3 Kaho Toshikazu 2018-09-12 13:10:54 UTC
(In reply to Stephen Hurd from comment #2)

> I'll see if I can figure out why r307562 made this change, and come up
> with a  Better Way to twi the two flags together.

I don't know exactly but igb devices are not able to enable/disable rxcsum and
rxcsum6 separately by the hardware level limitation. If only one rxcsum or
rxcsum6 flag is changed, another flag would like to be changed forcefully
to hold same state.

I think iflib should not do a special thing, and it is better that
an individual driver has some workarounds if need.
Comment 4 Shirkdog 2018-09-12 13:18:13 UTC
(In reply to Kaho Toshikazu from comment #3)

I think that is it okay for this, if someone wants to disable a NIC feature for IPv4, there is no reason to leave it enabled for IPv6. 

I am also rebuilding a kernel to test the patch attached this bug.
Comment 5 Stephen Hurd freebsd_committer 2018-09-12 14:19:30 UTC
Created attachment 197054 [details]
Allow rxcsum to be disabled, but keep it tied to rxcsum6

This patch allows rxcsum to be disabled, but retains the restriction that both flags be toggled together.

Still looking into the VLAN_HWCSUM issue.
Comment 6 Stephen Hurd freebsd_committer 2018-09-12 14:41:31 UTC
(In reply to Kaho Toshikazu from comment #3)

> I think iflib should not do a special thing, and it is better that
> an individual driver has some workarounds if need.

The driver isn't involved in this ioctl, so it needs to support the lowest common denominator of iflib devices.  This ioctl works by stopping the driver, setting the capabilities, then restarting the driver... which then applies the capabilities.

If a driver can't support a particular combination of flags, there's no way for it to tell iflib at this time.  For rxcsum, the ifconfig specifically mentions that rxcsum and rxcsum6 may not be interdependent.

It looks like there's some brokenness in the WoL stuff too.  I'm digging into that once I verify your vlan_hwcsum changes.
Comment 7 Kaho Toshikazu 2018-09-12 15:31:31 UTC
(In reply to Stephen Hurd from comment #5)

> +		mask &= ctx->ifc_softc_ctx.isc_capabilities;

An em class device doesn't have rxcsum6 capability and this line is useful.
If a device doesn't have rxcsum6 capability, don't touch a flag related rxcsum6
and the variable setmask should be also masked by isc_capabilities, I think. 

> It looks like there's some brokenness in the WoL stuff too. 

No, it is not broken. But the Wol feature doesn't have any effect until shutdown
or suspend time, and set flags only and avoid re-initialization.

> I'm digging into that once I verify your vlan_hwcsum changes.

The vlan_hwcsum is simply forgot in the  definition IFCAP_FLAGS. 
If the IFCAP_FLAGS has the Wol flags, any special masking operation is not need.
Comment 8 Stephen Hurd freebsd_committer 2018-09-12 17:53:44 UTC
(In reply to Kaho Toshikazu from comment #7)

> the variable setmask should be also masked by isc_capabilities, I think. 

Everything going into setmask should be masked by mask already... I've got that fixed in my patch now, will update right away.

> No, it is not broken. 

I meant in the existing code, not in your patch.  I'm trying to figure out why the old code was forcing only IFCAP_WOL_MCAST|IFCAP_WOL_MAGIC if any WoL flag (including IFCAL_WOL_UCAST) was set.

If a driver requires something like this, and the isc_capabilities isn't enough to restrict it, there needs to be a flag, a callback, or at least an update to some manpages regarding it.  It's almost certainly better to set the WoL capabilities as specified and have some not work then it is to not allow some to be specified at all or to change what set is enabled.

> But the Wol feature doesn't have any effect until shutdown or suspend time, and set flags only and avoid re-initialization.

Yeah, that part is good.  We may actually want a larger set of things that don't reinit.

> The vlan_hwcsum is simply forgot in the  definition IFCAP_FLAGS. 

You also moved the if_vlancap(ifp) line to after the reinit.  I'm trying to figure out if it should be set while the interface is stopped.  It currently doesn't really make a difference as far as I can tell, but it may make sense to update while the interface is down, but then the context will be locked.
Comment 9 Stephen Hurd freebsd_committer 2018-09-12 17:54:34 UTC
Created attachment 197061 [details]
Update to include WoL and vlan hwcsum fixes
Comment 10 Kaho Toshikazu 2018-09-12 22:25:45 UTC
(In reply to Stephen Hurd from comment #8)

> I'm trying to figure out why the old code was forcing only
> IFCAP_WOL_MCAST|IFCAP_WOL_MAGIC if any WoL flag (including IFCAL_WOL_UCAST)
> was set.

I suspect that because of a machine is easily woken up by a unintended packet
if wol_ucast is enabled.

> You also moved the if_vlancap(ifp) line to after the reinit.

A vlan interface requires the flags of the parent interface.
Because of very limited flags are passed through, in the existing code,
the vlan interface can usually get correct flags but sometimes get
old incorrect flags.
Comment 11 Shirkdog 2018-09-12 23:33:35 UTC
(In reply to Stephen Hurd from comment #9)

rebuilt and tested using the update and they appear to be honored (running ifconfig em0 promisc up at boot:

                                                                                                                                                                                     
# ifconfig em0
em0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500                                                                                                                        
        options=81049b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,VLAN_HWFILTER>                                                                                                                        
        ether 68:05:ca:33:42:f0                                                                                                                                                                                    
        media: Ethernet autoselect (1000baseT <full-duplex>)                                                                                                                                                       
        status: active                                                                                                                                                                                             
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>                                                                                                                                                       
# ifconfig em0 -rxcsum -txcsum -tso4 -tso6 -lro -rxcsum6 -txcsum6 -vlanhwcsum -vlanhwtso                                              
# ifconfig em0                                                                                                                                                                                       
em0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500                                                                                                                        
        options=810018<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWFILTER>                                                                                                                                                      
        ether 68:05:ca:33:42:f0                                                                                                                                                                                    
        media: Ethernet autoselect (1000baseT <full-duplex>)                                                                                                                                                       
        status: active                                                                                                                                                                                             
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>                                                                                                                                                       

I did notice that I no longer observed the any of the IPv6 flags when I try to enable them, and this interface has no IP configuration. But I am testing sniffing on the interface and will report back if I have any issues.
Comment 12 Kaho Toshikazu 2018-09-13 03:40:51 UTC
(In reply to Shirkdog from comment #11)

> I did notice that I no longer observed the any of the IPv6 flags
> when I try to enable them,

 It is correct because em doesn't have ipv6 offloads. You can see the capabilities of the device with `ifconfig -m em0`. The old code changes the rxcsum6 flag in spite of its absence capability.
Comment 13 commit-hook freebsd_committer 2018-09-20 19:36:33 UTC
A commit references this bug:

Author: shurd
Date: Thu Sep 20 19:35:35 UTC 2018
New revision: 338838
URL: https://svnweb.freebsd.org/changeset/base/338838

Log:
  Fix capabilities handling for iflib drivers

  Various capabilities were not being handled correctly in the
  SIOCSIFCAP handler. Specifically:

  IFCAP_RXCSUM and IFCAP_RXCSUM_IPV6 could be set even if not supported

  It was impossible to disable IFCAP_RXCSUM and/or IFCAP_RXCSUM_IPV6 via
  ifconfig since it does ioctl() per command-line flag rather than combine
  them into a single call.

  IFCAP_VLAN_HWCSUM could not be modified via the ioctl()

  Setting any combination of the three IFCAP_WOL flags would set only
  IFCAP_WOL_MCAST | IFCAP_WOL_MAGIC. For example, setting only
  IFCAP_WOL_UCAST would result in both IFCAP_WOL_MCAST and IFCAP_WOL_MAGIC
  being enabled, but IFCAP_WOL_UCAST would not be enabled.

  Because if_vlancap() was called before if_togglecapenable(), vlan flags
  were sometimes not applied correctly.

  Interfaces were being unnecessarily stopped and restarted for WoL

  PR:		231151
  Submitted by:	Kaho Toshikazu <kaho@elam.kais.kyoto-u.ac.jp>
  Reported by:	Shirkdog <mshirk@daemon-security.com>
  Reviewed by:	galladin
  Approved by:	re (gjb)
  Sponsored by:	Limelight Networks
  Differential Revision:	https://reviews.freebsd.org/D17158

Changes:
  head/sys/net/iflib.c