Bug 208001

Summary: After turning off the jail does not remove network routes
Product: Base System Reporter: Vladislav V. Prodan <admin>
Component: binAssignee: freebsd-jail mailing list <jail>
Status: Open ---    
Severity: Affects Many People CC: jamie
Priority: ---    
Version: 10.3-BETA2   
Hardware: amd64   
OS: Any   

Description Vladislav V. Prodan 2016-03-14 22:43:30 UTC
After turning off the jail does not remove network routes.
Helps only change the IP address or reboot the server

# service jail start
Starting jails: web.
# service jail status
 JID             IP Address      Hostname                      Path
 web          web.otrada.local              /jails/web
# service jail stop
Stopping jails: web.
# service jail status
 JID             IP Address      Hostname                      Path
# service jail start
Starting jails: cannot start jail  "web":
ifconfig: ioctl (SIOCAIFADDR): No such process
jail: web: /sbin/ifconfig re0 inet alias: failed
# netstat -rn4 | grep 10.0        link#1             U           re0        link#1             U           re0        link#1             U           re0        link#1             U           re0         UGS         re0           link#1             UHS         lo0          link#1             UHS         lo0           link#10            UHS         lo0          link#10            UH          ng1           link#11            UH          ng2
# route get
   route to:
        fib: 0
  interface: re0
      flags: <UP,DONE,PINNED>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

Example config /etc/jail.conf:

allow.raw_sockets = 1;
exec.system_user = "root";
exec.jail_user = "root";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_${name}_console.log";
mount.fstab = "/etc/fstab.$name";
allow.set_hostname = 0;
allow.sysvipc = 0;
path = "/jails/${name}";

web {
    host.hostname = "web.otrada.local";
    ip4.addr = "re0|";
    ip4.addr += "lo0|";
    ip6.addr += "re0|2001:470:28:140::121/64";
    allow.sysvipc = 1;
    devfs_ruleset = 7;
#    path = "/jails/billing-test";
Comment 1 Vladislav V. Prodan 2016-03-14 22:58:20 UTC
# ifconfig re0 | grep 10\\.0
        inet netmask 0xffffff00 broadcast
        inet netmask 0xffffff00 broadcast
Comment 2 Vladislav V. Prodan 2016-03-14 23:00:44 UTC
# arp -d ( deleted
# arp -na | grep 10\\.0\\.0\\.22
? ( at 00:e0:4d:a0:f8:46 on re0 permanent [ethernet]
Comment 3 Jamie Gritton freebsd_committer 2016-05-28 23:32:38 UTC
I suspect the problem lies in the "/24" of "re0|".  Jail aliases are almost always going to use the /32 netmask, which is the default if one isn't specified.  From what I see of your netstat output, your primary address is on the subnet.  The only time you'd want a non-/32 subnet on a jail is if your set aside a separate subnet used only for that jail.

Note this instruction from the "alias" section of ifconfig(8):
"If the address is on the same subnet as the first network address for this interface, a non-conflicting netmask must be given.  Usually 0xffffffff is most appropriate."

In any case, I was unable to reproduce this behavior.  I don't know all the parameters of your setup, but it looks like you should just remove those netmasks.
Comment 4 Vladislav V. Prodan 2016-05-29 00:16:24 UTC
The problem is not the creation of an IP address, network mask, other than /32.
I can give some examples of network topology, when the IP jail need to assign a subnet mask for the correct operation of the network protocols. For example, a DHCP-server or Samba services (nmbd and winbindd)
The problem is in the removal of a route to this IP when you turn off the jail.
Comment 5 Jamie Gritton freebsd_committer 2016-05-29 00:30:46 UTC
Yes, of course there are cases where something besides a /32 is appropriate - that is why jail(8) allows that.  However, as I mentioned it did appear that you had violated the specification that an alias should be on a non-conflicting netmask.

The fact remains that I am unable to reproduce your problem.  Perhaps I could if I had your entire configuration - all jails, all other network setup.

jail(8) simply calls ifconfig(8) with "alias" to add IP addresses, and with "-alias" to remove them - see the output of "jail -vc" and "jail -vr".  The jail will not be removed if the "ifconfig ... -alias" command fails, which implies that the command is succeeding.  Unless of course there actually is a bug in the way jail(8) is running this program.  My guess is the command is succeeding, but isn't removing some arp entry because the alias when incorrectly specified when it was created.

If it's clear (from "jail -v") that the correct ifconfig commands are being run, then this might be considered an ifconfig bug.  If the correct commands aren't being run, then it could be a jail bug.