Bug 239380 - sys.netpfil.pf.forward.{v4,v6} and sys.netpfil.pf.set_tos.v4 fail on i386
Summary: sys.netpfil.pf.forward.{v4,v6} and sys.netpfil.pf.set_tos.v4 fail on i386
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: tests (show other bugs)
Version: CURRENT
Hardware: i386 Any
: --- Affects Many People
Assignee: freebsd-testing mailing list
URL:
Keywords:
: 240085 240086 241493 241739 (view as bug list)
Depends on:
Blocks:
 
Reported: 2019-07-22 14:23 UTC by Li-Wen Hsu
Modified: 2019-11-14 14:01 UTC (History)
4 users (show)

See Also:


Attachments
Fixes for scapy bpf on FreeBSD (i386) (2.29 KB, patch)
2019-11-06 13:33 UTC, Bjoern A. Zeeb
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Li-Wen Hsu freebsd_committer 2019-07-22 14:23:35 UTC
These 3 tests always fail on i386:

sys.netpfil.pf.forward.v4
sys.netpfil.pf.forward.v6
sys.netpfil.pf.set_tos.v4

There are some analysis from kp@:

https://lists.freebsd.org/pipermail/freebsd-testing/2019-June/001933.html
https://lists.freebsd.org/pipermail/freebsd-testing/2019-June/001934.html
Comment 1 commit-hook freebsd_committer 2019-07-22 18:55:23 UTC
A commit references this bug:

Author: lwhsu
Date: Mon Jul 22 18:54:26 UTC 2019
New revision: 350221
URL: https://svnweb.freebsd.org/changeset/base/350221

Log:
  Temporarily skip sys.netpfil.pf.forward.{v4,v6} and sys.netpfil.pf.set_tos.v4
  on i386 as they are flakey on it

  PR:		239380
  Sponsored by:	The FreeBSD Foundation

Changes:
  head/tests/sys/netpfil/pf/forward.sh
  head/tests/sys/netpfil/pf/set_tos.sh
Comment 2 commit-hook freebsd_committer 2019-07-23 08:21:30 UTC
A commit references this bug:

Author: lwhsu
Date: Tue Jul 23 08:20:53 UTC 2019
New revision: 350237
URL: https://svnweb.freebsd.org/changeset/base/350237

Log:
  MFC r350221

  Temporarily skip sys.netpfil.pf.forward.{v4,v6} and sys.netpfil.pf.set_tos.v4
  on i386 as they are flakey on it

  PR:		239380
  Sponsored by:	The FreeBSD Foundation

Changes:
_U  stable/12/
  stable/12/tests/sys/netpfil/pf/forward.sh
  stable/12/tests/sys/netpfil/pf/set_tos.sh
Comment 3 Ed Maste freebsd_committer 2019-08-12 19:29:45 UTC
(In reply to Li-Wen Hsu from comment #0)
kp suggests it appears to be a bpf issue
Comment 4 Kristof Provost freebsd_committer 2019-09-03 15:27:55 UTC
It looks like I was wrong and this isn't actually a problem in bpf, but with scapy itself.
With scapy-2.3.3 the tests pass, with py27-scapy-2.4.2_2 they do not.
Comment 5 commit-hook freebsd_committer 2019-09-05 13:26:57 UTC
A commit references this bug:

Author: bofh
Date: Thu Sep  5 13:26:15 UTC 2019
New revision: 511187
URL: https://svnweb.freebsd.org/changeset/ports/511187

Log:
  net/scapy: Update version 2.4.2=>2.4.3

  - Use net/py-pypcap instead of net/py-pcap as dnet and py-pcap will be
    deprecated in future scapy versions. One other reason is py-pcap is
    failing with python3. [1]

  PR:		239380 [1]
  Submitted by:	lwhsu [1]

Changes:
  head/net/scapy/Makefile
  head/net/scapy/distinfo
  head/net/scapy/files/
Comment 6 commit-hook freebsd_committer 2019-09-11 10:24:52 UTC
A commit references this bug:

Author: bofh
Date: Wed Sep 11 10:24:29 UTC 2019
New revision: 511811
URL: https://svnweb.freebsd.org/changeset/ports/511811

Log:
  inet/scapy: Update version 2.4.2=>2.4.3

  - Use net/py-pypcap instead of net/py-pcap as dnet and py-pcap will be
    deprecated in future scapy versions. One other reason is py-pcap is
    failing with python3. [1]
  - Change MASTER_SITE to PYPI for better availability [2]
  - Update WWW [2]

  PR:		239380 [1] 240493 [2]
  Submitted by:	lwhsu [1] [2]

Changes:
  head/net/scapy/Makefile
  head/net/scapy/distinfo
  head/net/scapy/files/
  head/net/scapy/pkg-descr
Comment 8 Muhammad Moinur Rahman freebsd_committer 2019-09-17 19:02:15 UTC
(In reply to Li-Wen Hsu from comment #7)
Is it possible to comment out the RUN_DEPENDS and rerun the test without committing ?
Comment 9 Li-Wen Hsu freebsd_committer 2019-09-17 19:10:26 UTC
(In reply to Muhammad Moinur Rahman from comment #8)
It's possible by manually building the package and installing to the VM.  Do you mean all the RUN_DEPENDS?  Can you provide a patch?
Comment 10 Li-Wen Hsu freebsd_committer 2019-09-17 21:39:45 UTC
I tried to remove py27-pypcap-1.2.3, libdnet-1.13_3, py27-libdnet-1.13_3 with `pkg -f` to keep py27-scapy-2.4.3, but the tests are still failing.
Comment 11 Li-Wen Hsu freebsd_committer 2019-10-28 14:01:46 UTC
Just tried with py36-scapy-2.4.3_1 again, with and without py36-pypcap-1.2.3 and py36-libdnet-1.13_3, both failed.
Comment 12 commit-hook freebsd_committer 2019-11-05 10:54:43 UTC
A commit references this bug:

Author: bz
Date: Tue Nov  5 10:53:56 UTC 2019
New revision: 354357
URL: https://svnweb.freebsd.org/changeset/base/354357

Log:
  frag6 tests: set end to catch timeout as error

  There are times when we have to wait for reply packets.  There are
  either an ICMPv6 (error) reply or the expiration timeout.
  In these cases synchonous ICMPv6 replies should arrive, always,
  unless the packet is lost.  Due to errors experienced with the
  test software sending an invlaid request on at least i386 (*) these
  packets are not generated.  That means we are waiting for a long time
  for the replies or even timeout the test case.

  Manually set the "End" flag on these test cases as well, so they do
  fail rather than timeout as the sniffer timeout happens.  This improves
  debugging options, reflects the error properly, and saves time on each
  test suit run.

  (*) The real cause for that is still to be found (see the referenced PRs)

  PR:			241493, 239380
  MFC after:		2 weeks
  Sponsored by:		Netflix

Changes:
  head/tests/sys/netinet6/frag6/frag6_02.py
  head/tests/sys/netinet6/frag6/frag6_04.py
Comment 13 Bjoern A. Zeeb freebsd_committer 2019-11-06 13:16:13 UTC
*** Bug 241493 has been marked as a duplicate of this bug. ***
Comment 14 Bjoern A. Zeeb freebsd_committer 2019-11-06 13:16:57 UTC
*** Bug 240085 has been marked as a duplicate of this bug. ***
Comment 15 Bjoern A. Zeeb freebsd_committer 2019-11-06 13:17:11 UTC
*** Bug 240086 has been marked as a duplicate of this bug. ***
Comment 16 Bjoern A. Zeeb freebsd_committer 2019-11-06 13:26:18 UTC
*** Bug 241739 has been marked as a duplicate of this bug. ***
Comment 17 Bjoern A. Zeeb freebsd_committer 2019-11-06 13:33:10 UTC
Created attachment 208911 [details]
Fixes for scapy bpf on FreeBSD (i386)

Hi,

this diff is from the installed version of scapy and not relative to the port.
It fixes both the alignment on (32bit) platforms where sizeof(long) == 4 and
for i386 the offset on the bpf_hdr struct as time_t still is 32bit.

Scapy does not set any more specific BPF_T_* time option, so we are not switching to bpf_xhdr (which would probably be the better long-term solution).  That said, the current situation in FreeBSD bpf is a mess until we BURN_BRIDGES on the old bpf_hdr.

I have not yet submitted a ports PR or the change upstream to scapy.

This work was
Sponsored by:  Netflix
Comment 18 Bjoern A. Zeeb freebsd_committer 2019-11-06 13:51:05 UTC
(In reply to Bjoern A. Zeeb from comment #17)

As a follow-up.

The problem for the test cases was as follows (at least for the IPv6 ones).

To resolve the neighbour's address, scapy sends a NS.
The kernel of the destination/next hop replies with a NA.
Scapy receives the packets but due to the wrong offsets within the bpf header the frame sizes are all way off and way too large.
As a result there is no result packet to the internal AsyncSniffer and neither the original packet nor the NA reply is seen.
With the failed address resolution scapy uses a broadcast Ethernet destination MAC address on the ethernet packets.  FreeBSD will mark this packet with M_BCAST as a result and higher up in the kernel certain functions will check for that and get an invalid result (or not accept the packet, or not trigger and ICMPv6 reply).
With that the expected reply packet from the test cases are missing and the sniffer there will not see any further packets and either timeout or checks will fail and with that the test times out or fails.

Hope that explains some of this.
Comment 19 commit-hook freebsd_committer 2019-11-06 14:44:28 UTC
A commit references this bug:

Author: bz
Date: Wed Nov  6 14:43:36 UTC 2019
New revision: 354395
URL: https://svnweb.freebsd.org/changeset/base/354395

Log:
  IPv6 tests:	test case for scapy bpf parsing bug

  PR:		239380
  Reported by:	lwhsu, CI system
  MFC after:	3 weeks
  Sponsored by:	Netflix

Changes:
  head/tests/sys/netinet6/Makefile
  head/tests/sys/netinet6/scapyi386.py
  head/tests/sys/netinet6/scapyi386.sh
Comment 20 commit-hook freebsd_committer 2019-11-06 17:57:49 UTC
A commit references this bug:

Author: lwhsu
Date: Wed Nov  6 17:56:48 UTC 2019
New revision: 516910
URL: https://svnweb.freebsd.org/changeset/ports/516910

Log:
  Add a patch which fixes both the alignment on (32bit) platforms where
  sizeof(long) == 4 and for i386 the offset on the bpf_hdr struct as time_t
  still is 32bit.

  PR:		239380
  Approved by:	bofh (maintainer)
  Sponsored by:	Netflix (bz)
  Sponsored by:	The FreeBSD Foundation (lwhsu)

Changes:
  head/net/scapy/Makefile
  head/net/scapy/files/
  head/net/scapy/files/patch-i386
Comment 21 Bjoern A. Zeeb freebsd_committer 2019-11-06 22:22:39 UTC
Submitted upstream:
https://github.com/secdev/scapy/pull/2329
Comment 22 commit-hook freebsd_committer 2019-11-08 18:56:50 UTC
A commit references this bug:

Author: lwhsu
Date: Fri Nov  8 18:56:04 UTC 2019
New revision: 354553
URL: https://svnweb.freebsd.org/changeset/base/354553

Log:
  Reenable netinet6 and netpfil tests on i386, net/scapy 2.4.3_2 contains the fix

  PR:		239380
  Sponsored by:	The FreeBSD Foundation

Changes:
  head/tests/sys/netinet6/exthdr.sh
  head/tests/sys/netinet6/scapyi386.sh
  head/tests/sys/netpfil/common/forward.sh
  head/tests/sys/netpfil/common/tos.sh
  head/tests/sys/netpfil/pf/forward.sh
  head/tests/sys/netpfil/pf/set_tos.sh
Comment 23 Kristof Provost freebsd_committer 2019-11-14 14:01:49 UTC
bz@ has fixed this problem, and lwhsu@ has re-activated the tests. This bug can be closed.