I changed the rule to use 'route-to' instead of 'reply-to' and the
ICMP storm stopped.

----------
if_isp1="em0"
isp1_router="GW1.GW1.GW1.GW1"
if_isp2="em1"
isp2_router="GW2.GW2.GW2.GW2"

pass in all no state
pass out all
pass out route-to ( $if_isp1 $isp1_router ) from $if_isp1
pass out route-to ( $if_isp2 $isp2_router ) from $if_isp2
----------

I'm not sure about the implementation difference of 'reply-to' and
'route-to'.

I'm sorry that my previous follow-up was incorrect.

It seems that I need 'no state' for both 'route-to' lines or they are
ignored.  After that, packets are redirected to correct interfaces,
and ICMP storm also revived.

Comment 3 Mark Linimon 2010-02-26 11:23:27 UTC

Responsible Changed
From-To: freebsd-bugs->freebsd-pf

Over to maintainer(s).

I found a workaround for the problem.

The problem won't happen when I removed TSO support from the interface
which is used for the default route.

About my old server, only msk(4) has TSO support, so the problem only
happend when I used msk(4) for the default route.  (My original post
was a bit confused and incorrect, sorry).

I guess there is something wrong with TSO related code (ip_output.c or
tcp_output.c ?), but it is too much for me to understand them....

Can you please test the attached patch (by Pyun YongHyeon) and let us know if 
this fixes the situation for you?

Thanks,
  Max Laier

I applied the patch on 8-STABLE (fetched today), rebuilt and installed
the kernel and rebooted, but the problem still occured.

-- 
Yoshiaki Kasahara
Research Institute for Information Technology, Kyushu University
kasahara@nc.kyushu-u.ac.jp

I found a blog reporting a similar symptom using
ipfw(4)+divert(4)+natd(8) on 8.0R and 7.2R (less severe on 7.2R).

http://www.bsddiary.net/d/201002.html#09

It is written in Japanese, but I guess you can still read
configuration used for the test in the article.

He also reported later that disabling TSO worked as a workaround.
He used em(4) and fxp(4).

I wonder if it might help locating the problem...

Comment 8 Bjoern A. Zeeb 2010-08-21 13:47:15 UTC

Hey,

have you re-tried with an updated kernel and this patch again?
It seems to help other people. Could you give us an update?

/bz

-- 
Bjoern A. Zeeb                       This signature is about you not me.

I'm sorry for my late reply.

I updated my test machine to 8.1-STABLE on Sep. 2nd and applied the
patch, but the problem persists.

--Y.Kasahara

Comment 10 Bjoern A. Zeeb 2010-09-09 16:29:52 UTC

Responsible Changed
From-To: freebsd-pf->bz

Take to commit the patch, which will be half of the fix. 
Ermal has the other half as well that I'll need to review and 
follow-up with.

Comment 11 dfilter service 2010-09-10 01:00:26 UTC

Author: bz
Date: Fri Sep 10 00:00:06 2010
New Revision: 212403
URL: http://svn.freebsd.org/changeset/base/212403

Log:
  When using pf routing options, properly handle IP fragmentation
  for interfaces with TSO enabled, otherwise one would see an extra
  ICMP unreach, frag needed pre matching packet on lo0.
  This syncs pf code to ip_output.c r162084.
  
  PR:		kern/144311
  Submitted by:	yongari via mlaier
  Reviewed by:	eri
  Tested by:	kib
  MFC after:	8 days

Modified:
  head/sys/contrib/pf/net/pf.c

Modified: head/sys/contrib/pf/net/pf.c
==============================================================================
--- head/sys/contrib/pf/net/pf.c	Thu Sep  9 23:45:59 2010	(r212402)
+++ head/sys/contrib/pf/net/pf.c	Fri Sep 10 00:00:06 2010	(r212403)
@@ -6375,6 +6375,7 @@ pf_route(struct mbuf **m, struct pf_rule
 	m0->m_pkthdr.csum_flags &= ifp->if_hwassist;
 
 	if (ntohs(ip->ip_len) <= ifp->if_mtu ||
+	    (m0->m_pkthdr.csum_flags & ifp->if_hwassist & CSUM_TSO) != 0 ||
 	    (ifp->if_hwassist & CSUM_FRAGMENT &&
 		((ip->ip_off & htons(IP_DF)) == 0))) {
 		/*
@@ -6449,7 +6450,7 @@ pf_route(struct mbuf **m, struct pf_rule
 	 * Too large for interface; fragment if possible.
 	 * Must be able to put at least 8 bytes per fragment.
 	 */
-	if (ip->ip_off & htons(IP_DF)) {
+	if (ip->ip_off & htons(IP_DF) || (m0->m_pkthdr.csum_flags & CSUM_TSO)) {
 		KMOD_IPSTAT_INC(ips_cantfrag);
 		if (r->rt != PF_DUPTO) {
 #ifdef __FreeBSD__
_______________________________________________
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"

Comment 12 Bjoern A. Zeeb 2010-09-10 01:03:42 UTC

Hey,

as stated when taking the ticket, the suggested patch (which was now
comitted to HEAD) is only half of the fix.  I'll try to review Ermal's
patch the next days and follow-up with that for testing.  Just for
your information, so you don't wonder why this is going in though it
doesn't help you.

/bz

-- 
Bjoern A. Zeeb                              Welcome a new stage of life.

Comment 13 dfilter service 2010-09-20 18:03:17 UTC

Author: bz
Date: Mon Sep 20 17:03:10 2010
New Revision: 212905
URL: http://svn.freebsd.org/changeset/base/212905

Log:
  MFC r212403:
  
    When using pf routing options, properly handle IP fragmentation
    for interfaces with TSO enabled, otherwise one would see an extra
    ICMP unreach, frag needed pre matching packet on lo0.
    This syncs pf code to ip_output.c r162084.
  
    Submitted by: yongari via mlaier
    Reviewed by:  eri
    Tested by:    kib
  PR:           kern/144311

Modified:
  stable/8/sys/contrib/pf/net/pf.c
Directory Properties:
  stable/8/sys/   (props changed)
  stable/8/sys/amd64/include/xen/   (props changed)
  stable/8/sys/cddl/contrib/opensolaris/   (props changed)
  stable/8/sys/contrib/dev/acpica/   (props changed)
  stable/8/sys/contrib/pf/   (props changed)
  stable/8/sys/dev/xen/xenpci/   (props changed)

Modified: stable/8/sys/contrib/pf/net/pf.c
==============================================================================
--- stable/8/sys/contrib/pf/net/pf.c	Mon Sep 20 16:43:17 2010	(r212904)
+++ stable/8/sys/contrib/pf/net/pf.c	Mon Sep 20 17:03:10 2010	(r212905)
@@ -6375,6 +6375,7 @@ pf_route(struct mbuf **m, struct pf_rule
 	m0->m_pkthdr.csum_flags &= ifp->if_hwassist;
 
 	if (ntohs(ip->ip_len) <= ifp->if_mtu ||
+	    (m0->m_pkthdr.csum_flags & ifp->if_hwassist & CSUM_TSO) != 0 ||
 	    (ifp->if_hwassist & CSUM_FRAGMENT &&
 		((ip->ip_off & htons(IP_DF)) == 0))) {
 		/*
@@ -6449,7 +6450,7 @@ pf_route(struct mbuf **m, struct pf_rule
 	 * Too large for interface; fragment if possible.
 	 * Must be able to put at least 8 bytes per fragment.
 	 */
-	if (ip->ip_off & htons(IP_DF)) {
+	if (ip->ip_off & htons(IP_DF) || (m0->m_pkthdr.csum_flags & CSUM_TSO)) {
 		KMOD_IPSTAT_INC(ips_cantfrag);
 		if (r->rt != PF_DUPTO) {
 #ifdef __FreeBSD__
_______________________________________________
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"

Comment 14 dfilter service 2010-09-20 19:26:42 UTC

Author: bz
Date: Mon Sep 20 18:26:37 2010
New Revision: 212910
URL: http://svn.freebsd.org/changeset/base/212910

Log:
  MFC r212403:
  
    When using pf routing options, properly handle IP fragmentation
    for interfaces with TSO enabled, otherwise one would see an extra
    ICMP unreach, frag needed pre matching packet on lo0.
    This syncs pf code to ip_output.c r162084.
  
    Submitted by: yongari via mlaier
  
  PR:	kern/144311

Modified:
  stable/7/sys/contrib/pf/net/pf.c
Directory Properties:
  stable/7/sys/   (props changed)
  stable/7/sys/cddl/contrib/opensolaris/   (props changed)
  stable/7/sys/contrib/dev/acpica/   (props changed)
  stable/7/sys/contrib/pf/   (props changed)

Modified: stable/7/sys/contrib/pf/net/pf.c
==============================================================================
--- stable/7/sys/contrib/pf/net/pf.c	Mon Sep 20 18:20:35 2010	(r212909)
+++ stable/7/sys/contrib/pf/net/pf.c	Mon Sep 20 18:26:37 2010	(r212910)
@@ -6376,6 +6376,7 @@ pf_route(struct mbuf **m, struct pf_rule
 	m0->m_pkthdr.csum_flags &= ifp->if_hwassist;
 
 	if (ntohs(ip->ip_len) <= ifp->if_mtu ||
+	    (m0->m_pkthdr.csum_flags & ifp->if_hwassist & CSUM_TSO) != 0 ||
 	    (ifp->if_hwassist & CSUM_FRAGMENT &&
 		((ip->ip_off & htons(IP_DF)) == 0))) {
 		/*
@@ -6450,7 +6451,7 @@ pf_route(struct mbuf **m, struct pf_rule
 	 * Too large for interface; fragment if possible.
 	 * Must be able to put at least 8 bytes per fragment.
 	 */
-	if (ip->ip_off & htons(IP_DF)) {
+	if (ip->ip_off & htons(IP_DF) || (m0->m_pkthdr.csum_flags & CSUM_TSO)) {
 		ipstat.ips_cantfrag++;
 		if (r->rt != PF_DUPTO) {
 #ifdef __FreeBSD__
_______________________________________________
svn-src-all@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"

Just to add that I'm still seeing this issue. 

 9.1-STABLE FreeBSD 9.1-STABLE #0 r252282

Network adapter:

<Intel(R) PRO/1000 Network Connection 7.3.7>

Disabling TSO4, or removing the route-to entry from pf solves the problem.



Disclaimer: http://www.peralex.com/disclaimer.html

Comment 16 Bjoern A. Zeeb 2014-05-18 06:01:19 UTC

Responsible Changed
From-To: bz->gnn

I shall not use bugzilla (at least until we will have a CLI).

I am still seeing this issue on 10.0-RELEASE.

I can test patches, interestingly, this ONLY occurs for me on 'local' IPs.

EG: when the destination IP address is a locally bound address, things break, otherwise they seem to work fine (when the resulting IP is actually routed beyond the firewall)

Hi!

Found the same problem on both 9.3 and 10.1 versions

uname -rp
9.3-RELEASE-p7 i386

wan1 adapter properties
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>

wan2 adapter properties
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500     options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>

pf.conf includes 

set skip on { lo0 }
set state-policy if-bound
scrub in all fragment reassemble

pass in quick on $wan1 reply-to ($wan1 $gw_wan1) inet proto tcp from any to $wan1 port ssh
pass in quick on $wan2 reply-to ($wan2 $gw_wan2) inet proto tcp from any to $wan2 port ssh
pass out quick route-to ($wan1 $gw_wan1) inet proto tcp from $wan1 port ssh to any
pass out quick route-to ($wan2 $gw_wan2) inet proto tcp from $wan2 port ssh to any 


Second machine has
uname -rp
10.1-RELEASE-p8 i386


em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500     options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>

em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500    options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>


and the same rules listsing (I also has the same pf.conf on over 10 machines and they work well, seems like the reason is TSO4 )

Additional information to last post

cat /var/run/dmesg.boot | grep em1
em1: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0xc000-0xc01f mem 0xfe4c0000-0xfe4dffff,0xfe400000-0xfe47ffff,0xfe4e0000-0xfe4e3fff irq 17 at device 0.0 on pci4
em1: Using MSIX interrupts with 3 vectors


cat /var/run/dmesg.boot | grep em0
em0: <Intel(R) PRO/1000 Network Connection 7.4.2> port 0xf020-0xf03f mem 0xfe500000-0xfe51ffff,0xfe527000-0xfe527fff irq 20 at device 25.0 on pci0
em0: Using an MSI interrupt

All of this adapters integrated in motherboard

Comment 20 Eitan Adler 2018-05-28 19:41:12 UTC

batch change:

For bugs that match the following
-  Status Is In progress 
AND
- Untouched since 2018-01-01.
AND
- Affects Base System OR Documentation

DO:

Reset to open status.


Note:
I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.