Bug 210408

Summary:	Problem with outgoing traffic using ipfw and kernel nat originated from local address
Product:	Base System	Reporter:	aigars
Component:	kern	Assignee:	freebsd-ipfw (Nobody) <ipfw>
Status:	New ---
Severity:	Affects Some People	CC:	feld, smithi
Priority:	---
Version:	10.2-RELEASE
Hardware:	amd64
OS:	Any

Description aigars 2016-06-20 13:32:02 UTC

There is a problem with outgoing connections made from local address which is aliased using ipfw kernel nat. Sending out data stops after 20-30 packets. Connections from other hosts using this host as gateway have no problem.
The problem disappears when RXCSUM,TXCSUM,TSO4,TSO6 are switched off.
The same problem is with natd but then only TSO4,TSO6 needs to be switched off.
NIC driver - igb.

Comment 1 Ian Smith 2016-06-22 08:05:21 UTC

From ipfw(8), second last paragraph:

"    Due to the architecture of libalias(3), ipfw nat is not compatible with
     the TCP segmentation offloading (TSO).  Thus, to reliably nat your net-
     work traffic, please disable TSO on your NICs using ifconfig(8)."

I don't know about RXCSUM & TXCSUM in this context, but NAT is only for
IPv4 - make sure your NAT rules only apply to ipv4 packets specifically -
 and you certainly need to turn TSO4 off when using either kernel NAT
 or natd(8), both of which use libalias(3).

Comment 2 Mark Felder freebsd_committer

2016-07-06 17:07:36 UTC

(In reply to smithi from comment #1)

I'm surprised we don't just handle this automatically. I don't see a good reason why the user should have to discover this and need to manually turn off TSO4.

I also know that libalias is being overhauled/rewritten by Bill Yuan of Dflybsd. He might be able to address this deficiency. I'm going to link him to this PR.

Comment 3 Ian Smith 2016-07-07 08:43:06 UTC

(In reply to Mark Felder from comment #2)

Offhand I can think of three places that could be done:

1) /sbin/ipfw (userland)
a) when NAT is configured, eg ipfw nat 123 config [ip address | if iface] ...
b) when NAT would be first invoked, eg ipfw add [ruleno] nat 123 [condition/s]

2) /sys/netpfil/ipfw/ip_fw2.c (kernel)
when NAT is first actually invoked on a packet on the NAT interface.

3) /sys/netinet/libalias (kernel) (or from userland for natd(8))
on first use of an interface, ie (only) on the first packet processed.

(1a) seems unlikely, as 'ip address' may not map to an iface on rule creation, 
and a particular nat config may not even be used, or its rule not encountered.

(1b) perhaps, though its config needs consulting, and unless 'if iface' is 
specified it may not be straightforward to determine which interface - and 
we would only want TSO4 disabled on the NAT interface, not on any others.

(1) is userland, so it might be more appropriate to 'call' /sbin/ifconfig from 
there, though again the address to interface mapping - from routing table/s I 
assume - may not already be in place upon ruleset creation.

(2) and (3) are in-kernel. Perhaps the new libifconfig (ono) can be used from 
there, but I've only seen that go by in freebsd-net in passing.  This would 
 require testing for TSO4 being on, then setting it off (-tso or -tso4).

Separately, /sbin/ipfw should probably insist on (or change to) 'ipv4' rather 
 than 'ip' or 'all' on nat rules, to guard against passing libalias(3) any 
 ipv6 packets, another potential foot-shot.

Just a few thought-bubbles, FWIW ..

Ian