Bug 208001

Summary: After turning off the jail does not remove network routes
Product: Base System Reporter: Vladyslav V. Prodan <admin>
Component: binAssignee: freebsd-jail (Nobody) <jail>
Status: Closed Feedback Timeout    
Severity: Affects Many People CC: jamie
Priority: ---    
Version: 10.3-BETA2   
Hardware: amd64   
OS: Any   

Description Vladyslav V. Prodan 2016-03-14 22:43:30 UTC
After turning off the jail does not remove network routes.
Helps only change the IP address or reboot the server


# service jail start
Starting jails: web.
# service jail status
 JID             IP Address      Hostname                      Path
 web             10.0.0.22       web.otrada.local              /jails/web
# service jail stop
Stopping jails: web.
# service jail status
 JID             IP Address      Hostname                      Path
# service jail start
Starting jails: cannot start jail  "web":
ifconfig: ioctl (SIOCAIFADDR): No such process
jail: web: /sbin/ifconfig re0 inet 10.0.0.22/24 alias: failed
.
# netstat -rn4 | grep 10.0
10.0.0.0/24        link#1             U           re0
10.0.0.0/24        link#1             U           re0
10.0.0.0/24        link#1             U           re0
10.0.0.0/24        link#1             U           re0
10.0.0.0/8         10.0.0.125         UGS         re0
10.0.0.1           link#1             UHS         lo0
10.0.0.19          link#1             UHS         lo0
10.0.1.1           link#10            UHS         lo0
10.0.2.48          link#10            UH          ng1
10.0.3.5           link#11            UH          ng2
# route get 10.0.0.0/24
   route to: 10.0.0.0
destination: 10.0.0.0
       mask: 255.255.255.0
        fib: 0
  interface: re0
      flags: <UP,DONE,PINNED>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0


Example config /etc/jail.conf:

allow.raw_sockets = 1;
exec.clean;
exec.system_user = "root";
exec.jail_user = "root";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_${name}_console.log";
mount.devfs;
mount.fstab = "/etc/fstab.$name";
allow.mount;
allow.set_hostname = 0;
allow.sysvipc = 0;
path = "/jails/${name}";

web {
    host.hostname = "web.otrada.local";
    ip4.addr = "re0|10.0.0.22/24";
    ip4.addr += "lo0|127.0.1.1/8";
    ip6.addr += "re0|2001:470:28:140::121/64";
    allow.sysvipc = 1;
    mount.procfs;
    devfs_ruleset = 7;
#    path = "/jails/billing-test";
}
Comment 1 Vladyslav V. Prodan 2016-03-14 22:58:20 UTC
# ifconfig re0 | grep 10\\.0
        inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
        inet 10.0.0.19 netmask 0xffffff00 broadcast 10.0.0.255
Comment 2 Vladyslav V. Prodan 2016-03-14 23:00:44 UTC
# arp -d 10.0.0.22
10.0.0.22 (10.0.0.22) deleted
# arp -na | grep 10\\.0\\.0\\.22
? (10.0.0.22) at 00:e0:4d:a0:f8:46 on re0 permanent [ethernet]
Comment 3 Jamie Gritton freebsd_committer freebsd_triage 2016-05-28 23:32:38 UTC
I suspect the problem lies in the "/24" of "re0|10.0.0.22/24".  Jail aliases are almost always going to use the /32 netmask, which is the default if one isn't specified.  From what I see of your netstat output, your primary address is on the 10.0.0.0/24 subnet.  The only time you'd want a non-/32 subnet on a jail is if your set aside a separate subnet used only for that jail.

Note this instruction from the "alias" section of ifconfig(8):
"If the address is on the same subnet as the first network address for this interface, a non-conflicting netmask must be given.  Usually 0xffffffff is most appropriate."

In any case, I was unable to reproduce this behavior.  I don't know all the parameters of your setup, but it looks like you should just remove those netmasks.
Comment 4 Vladyslav V. Prodan 2016-05-29 00:16:24 UTC
The problem is not the creation of an IP address, network mask, other than /32.
I can give some examples of network topology, when the IP jail need to assign a subnet mask for the correct operation of the network protocols. For example, a DHCP-server or Samba services (nmbd and winbindd)
The problem is in the removal of a route to this IP when you turn off the jail.
Comment 5 Jamie Gritton freebsd_committer freebsd_triage 2016-05-29 00:30:46 UTC
Yes, of course there are cases where something besides a /32 is appropriate - that is why jail(8) allows that.  However, as I mentioned it did appear that you had violated the specification that an alias should be on a non-conflicting netmask.

The fact remains that I am unable to reproduce your problem.  Perhaps I could if I had your entire configuration - all jails, all other network setup.

jail(8) simply calls ifconfig(8) with "alias" to add IP addresses, and with "-alias" to remove them - see the output of "jail -vc" and "jail -vr".  The jail will not be removed if the "ifconfig ... -alias" command fails, which implies that the command is succeeding.  Unless of course there actually is a bug in the way jail(8) is running this program.  My guess is the command is succeeding, but isn't removing some arp entry because the alias when incorrectly specified when it was created.

If it's clear (from "jail -v") that the correct ifconfig commands are being run, then this might be considered an ifconfig bug.  If the correct commands aren't being run, then it could be a jail bug.