Bug 132107 - [carp] carp(4) advskew setting ignored when carp IP used on a gif(4) interface
Summary: [carp] carp(4) advskew setting ignored when carp IP used on a gif(4) interface
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: Unspecified
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-02-25 19:00 UTC by Andrew Daugherity
Modified: 2017-12-31 22:32 UTC (History)
0 users

See Also:


Attachments
rc.common.conf (2.04 KB, application/octet-stream)
2011-12-17 03:29 UTC, Gleb Smirnoff
no flags Details
rc.conf-lb3 (245 bytes, application/octet-stream)
2011-12-17 03:29 UTC, Gleb Smirnoff
no flags Details
rc.conf-lb4 (261 bytes, application/octet-stream)
2011-12-17 03:29 UTC, Gleb Smirnoff
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Daugherity 2009-02-25 19:00:09 UTC
For background, see my post to the freebsd-pf mailing list:
http://docs.freebsd.org/cgi/mid.cgi?38ce25da0901271659m2b6d8a1fg2e425df93781f6f5

When an IP configured on a carp(4) interface is used on a gif(4) tunnel, in order that the endpoint of the tunnel fails over, the advskew setting for this carp interface is ignored and the interface is brought up with advskew 0.  This obviously prevents proper operation of a "preferred master" configuration.

After configuring devd to ignore attach events for carp interfaces (devd automatically runs "/etc/rc.d/netif start carp0" after you do an "ifconfig carp0 create", and does so needlessly after carp0 is already configured at boot time; that's a separate bug I shall file shortly), I have determined that the "start all interfaces" mode of /etc/rc.d/netif (i.e. "rc.d/netif start", without specifying an interface) configures the advskew correctly.  It's the "rc.d/netif start carp0" mode that loses the advskew setting.

In the standard configuration, "rc.d/netif start" is called and carp interfaces are brought up (correctly).  A bit later in the bootup, devd starts, and thinks it needs to configure carpN, so it calls "rc.d/netif start carp0", which resets advskew to 0.

I'm thinking it's a kernel bug since enabling rc_debug and logging the outputs of "sh -x /etc/rc.d/netif start" vs. "ifconfig carp0 create; sh -x /etc/rc.d/netif start carp0" shows that the exact same sequence of "ifconfig carp0 ..." commands is run.  The only difference is that in the global start mode, other interfaces (including gif0) are ifconfig'd between "ifconfig carp0 create" and the rest of the "ifconfig carp0 ..." commands.

I have observed this problem on both 7.1/amd64 and 6.2/i386.

Fix: 

A workaround is to configure devd to ignore attach events for carp interfaces, like so:
====
# don't have devd call network_start on carp0; for some reason this
#   loses the advskew setting (besides being entirely superfluous).

notify 100 {
	match "system" "IFNET";
	match "subsystem" "carp[0-9]+";
	match "type"	"ATTACH";
	action "echo devd called on $subsystem, type $type";
};
====
This overrides the following section of devd.conf:
notify 0 {
        match "system"          "IFNET";
        match "type"            "ATTACH";
        action "/etc/pccard_ether $subsystem start";
};


After doing this, carp interfaces are brought up correctly by "rc.d/netif start" on boot, and devd does not call "rc.d/netif start carp0", so the advskew setting remains correct.  This doesn't fix the underlying problem but it is an acceptable workaround to me.
How-To-Repeat: Assign the same IP to a carp interface and gif interface, e.g.:
cloned_interfaces="carp0 carp1"
gif_interfaces="gif0"
carp0_advskew="advskew 100"
ifconfig_carp0="vhid 16 $carp_advskew pass XXXXXX 10.95.8.1/21"
ifconfig_gif0="10.95.8.1 10.95.0.1 netmask 255.255.255.255"

The name "carp0" is nothing special; I can call it carp3 and the behavior is the same.  Only the 'ifconfig_gif0' setting matters; the gifconfig_gif0 setting doesn't (an IP I used for that is configured on carp1, which has no problems).  Also, the mere presence of gif0 doesn't matter -- when gif0 is configured with IPs that are NOT assigned to a carp interface, carp0 comes up with the correct advskew setting.
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2009-02-25 22:07:24 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-net

Over to maintainer(s).
Comment 2 dd 2009-06-19 07:51:09 UTC
Having the same issue here and the workaround isn't working for me --  
it preserves the advskew but the vpn doesn't seem to work without the  
devd attach event.  Hope someone will fix this soon.
Comment 3 Gleb Smirnoff freebsd_committer freebsd_triage 2011-10-15 10:00:11 UTC
  Andrew,

  can you please share the entire configuration on all involved
boxes: local and remote side of tunnel, the CARP master and slave
boxes?

  I'd like to reproduce your configuration and check whether my
new CARP implementation is subject to your problem or not. For more
info, see: http://people.freebsd.org/~glebius/newcarp/

-- 
Totus tuus, Glebius.
Comment 4 Gleb Smirnoff freebsd_committer freebsd_triage 2011-10-15 10:00:31 UTC
Responsible Changed
From-To: freebsd-net->glebius

I'm now working on major rewrite of CARP for FreeBSD 10, and 
I'd like to take all related PRs.
Comment 5 Gleb Smirnoff freebsd_committer freebsd_triage 2011-12-16 12:59:09 UTC
State Changed
From-To: open->closed

Feedback timeout.
Comment 6 Andrew Daugherity 2011-12-16 17:13:34 UTC
Sorry, I missed your earlier email.  I am sending you my config off-list.
Comment 7 Gleb Smirnoff freebsd_committer freebsd_triage 2011-12-17 03:29:21 UTC
  Add to Audit-Trail.

----- Forwarded message from "Daugherity, Andrew W" <adaugherity@tamu.edu> -----

Date: Fri, 16 Dec 2011 17:37:34 +0000
From: "Daugherity, Andrew W" <adaugherity@tamu.edu>
To: "glebius@FreeBSD.org" <glebius@FreeBSD.org>
Subject: carp config (kern/132107)

Sorry for not responding earlier.  Here's my config for a carp failover pair, which is one end of the gif tunnel.  The other end of the tunnel is another pair configured very similarly.  I abstracted as much as possible into rc.common.conf which I keep synced between the two.

ipsec etc. settings have no effect on the this bug and can be ignored.

I am now running 8.2 all around (this pair is amd64, the other i386); I haven't checked whether it behave any differently than 6.x/7.x did without my workaround (I would presume so, unless there are some carp changes in the kernel I don't know about) but the workaround still works and doesn't cause any problems.

Kernel config:
====
% uname -a
FreeBSD imslb3.tamu.edu 8.2-RELEASE-p4 FreeBSD 8.2-RELEASE-p4 #9: Mon Oct 31 15:27:42 CDT 2011     root@imslb3.tamu.edu:/usr/obj/usr/src/sys/IMS  amd64
% cat /usr/src/sys/amd64/conf/IMS
# include the GENERIC config (as of 7.0, GENERIC has SMP support, and there
#  are no more separate GENERIC and SMP kernels)
include GENERIC

ident           VPR-SMP-GENERIC

device          carp
device          pf
device          pflog
device          pfsync

# In order to enable IPSEC you MUST also add device crypto to
# your kernel configuration
options         IPSEC                   #IP security (requires device crypto)
#options        IPSEC_DEBUG             #debug for IP security
device          crypto
====
rc.confs attached.  It's nearly completely intact for your benefit, but please scrub public IPs and hostnames (including in comments) before sharing publicly.




Thanks,


Andrew Daugherity
Systems Analyst
Division of Research, Texas A&M University
adaugherity@tamu.edu



----- End forwarded message -----

-- 
Totus tuus, Glebius.
Comment 8 Gleb Smirnoff freebsd_committer freebsd_triage 2011-12-17 03:29:29 UTC
State Changed
From-To: closed->open

Feedback received.
Comment 9 Eitan Adler freebsd_committer freebsd_triage 2017-12-31 07:59:16 UTC
For bugs matching the following criteria:

Status: In Progress Changed: (is less than) 2014-06-01

Reset to default assignee and clear in-progress tags.

Mail being skipped