Bug 259149

Summary: mac_portacl not in affect when running VNET jail
Product: Base System Reporter: uros
Component: kernAssignee: freebsd-jail (Nobody) <jail>
Status: Closed Works As Intended    
Severity: Affects Many People CC: freebsd, michaelo, tom, zlei
Priority: ---    
Version: 13.0-RELEASE   
Hardware: amd64   
OS: Any   

Description uros 2021-10-13 18:31:57 UTC
I was trying to use mac_portacl and initial test on host works as described.

sysctl is set as follows

security.mac.portacl.rules: uid:1000:tcp:88
security.mac.portacl.port_high: 1023
security.mac.portacl.autoport_exempt: 1
security.mac.portacl.suser_exempt: 1
security.mac.portacl.enabled: 1

While testing with jails I stumble upon weird issue. 

Using jail with either alias IP or host IP everything seems to work just file. Output from `truss nc -l 88` as non root user
sigaction(SIGPIPE,{ SIG_IGN SA_RESTART ss_t },{ SIG_DFL SA_RESTART ss_t }) = 0 (0x0)
socket(PF_INET,SOCK_DGRAM|SOCK_CLOEXEC,0)	 = 3 (0x3)
close(3)					 = 0 (0x0)
socket(PF_INET,SOCK_STREAM,IPPROTO_TCP)		 = 3 (0x3)
setsockopt(3,SOL_SOCKET,SO_REUSEPORT,0x7fffffffa7d4,4) = 0 (0x0)
bind(3,{ AF_INET 0.0.0.0:88 },16)		 = 0 (0x0)
listen(3,1)					 = 0 (0x0)

After this it started to listen on port 88

jls -nj alias2
devfs_ruleset=0 nodying enforce_statfs=2 host=new ip4=disable ip6=disable jid=417 linux=new name=alias2 osreldate=1300139 osrelease=13.0-RELEASE parent=0 path=/jails/alias2/m persist securelevel=-1 sysvmsg=new sysvsem=new sysvshm=new vnet=inherit allow.chflags allow.nomlock allow.nomount allow.mount.nodevfs allow.mount.nofdescfs allow.mount.nolinprocfs allow.mount.nonullfs allow.mount.noprocfs allow.mount.notmpfs allow.mount.nozfs allow.noquotas allow.raw_sockets allow.noread_msgbuf allow.reserved_ports allow.noset_hostname allow.socket_af allow.suser allow.sysvipc allow.unprivileged_proc_debug children.cur=0 children.max=0 cpuset.id=15 host.domainname="" host.hostid=0 host.hostname=alias2.dev host.hostuuid=00000000-0000-0000-0000-000000000000 ip4.addr=172.20.0.14 ip4.saddrsel ip6.addr= ip6.saddrsel linux.osname=Linux linux.osrelease=3.17.0 linux.oss_version=198144


But on jail with VNET enabled Output from `truss nc -l 88` as non root user

sigaction(SIGPIPE,{ SIG_IGN SA_RESTART ss_t },{ SIG_DFL SA_RESTART ss_t }) = 0 (0x0)
socket(PF_INET,SOCK_DGRAM|SOCK_CLOEXEC,0)	 = 3 (0x3)
close(3)					 = 0 (0x0)
socket(PF_INET,SOCK_STREAM,IPPROTO_TCP)		 = 3 (0x3)
setsockopt(3,SOL_SOCKET,SO_REUSEPORT,0x7fffffffa7d4,4) = 0 (0x0)
bind(3,{ AF_INET 0.0.0.0:88 },16)		 ERR#13 'Permission denied'
close(3)					 = 0 (0x0)


jls -nj vnet2
devfs_ruleset=0 nodying enforce_statfs=2 host=new ip4=inherit ip6=inherit jid=453 linux=new name=vnet2 osreldate=1300139 osrelease=13.0-RELEASE parent=0 path=/jails/vnet2/m persist securelevel=-1 sysvmsg=new sysvsem=new sysvshm=new vnet=new allow.chflags allow.nomlock allow.nomount allow.mount.nodevfs allow.mount.nofdescfs allow.mount.nolinprocfs allow.mount.nonullfs allow.mount.noprocfs allow.mount.notmpfs allow.mount.nozfs allow.noquotas allow.raw_sockets allow.noread_msgbuf allow.reserved_ports allow.noset_hostname allow.socket_af allow.suser allow.sysvipc allow.unprivileged_proc_debug children.cur=0 children.max=0 cpuset.id=5 host.domainname="" host.hostid=0 host.hostname=vnet2.dev host.hostuuid=00000000-0000-0000-0000-000000000000 ip4.addr= ip4.saddrsel ip6.addr= ip6.saddrsel linux.osname=Linux linux.osrelease=3.17.0 linux.oss_version=198144


ifconfig for vnet2 jail

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
	groups: pflog
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:aa:dd:12:46:0b
	inet 172.20.0.3 netmask 0xffff0000 broadcast 172.20.255.255
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>


Same thing works on both for root. Looks like that this port <1024 still applied somehow when using VNET even though mac_portacl is doing it's job on other cases.

Expected behaviour for port that is not allowed via mac_portacl throws me this

sigaction(SIGPIPE,{ SIG_IGN SA_RESTART ss_t },{ SIG_DFL SA_RESTART ss_t }) = 0 (0x0)
socket(PF_INET,SOCK_DGRAM|SOCK_CLOEXEC,0)	 = 3 (0x3)
close(3)					 = 0 (0x0)
socket(PF_INET,SOCK_STREAM,IPPROTO_TCP)		 = 3 (0x3)
setsockopt(3,SOL_SOCKET,SO_REUSEPORT,0x7fffffffa7d4,4) = 0 (0x0)
bind(3,{ AF_INET 0.0.0.0:80 },16)		 ERR#1 'Operation not permitted'
close(3)					 = 0 (0x0)

So it's clear that error is not the same as if I try allowed 88 port.

Is all this expected behaviour or what could cause this to work differently.
Comment 1 uros 2022-01-19 11:49:46 UTC
Anyone have any idea what could be checked here?
Comment 2 Mina Galić freebsd_triage 2023-09-14 14:48:17 UTC
assigning to jail@ for now, maybe someone there can make sense of this
Comment 3 Thomas Hurst 2023-09-14 15:29:30 UTC
Investigating this finds 'net.inet.ip.portrange.reservedhigh=1023' within a vnet jail, so mac_portacl doesn't get a chance to do anything.

As on the host this needs to be set to 0 to allow mac_portacl to operate, though to change this I had to drop the initial jail securelevel.
Comment 4 Zhenlei Huang freebsd_committer freebsd_triage 2023-09-16 02:21:23 UTC
(In reply to Thomas Hurst from comment #3)
> Investigating this finds 'net.inet.ip.portrange.reservedhigh=1023' within a
> vnet jail, so mac_portacl doesn't get a chance to do anything.

The sysctl variable `net.inet.ip.portrange.reservedhigh` is a per-vnet one. So it is default to 1023 for vnet jail. That is expected.

```
/*
 * Reserved ports accessible only to root. There are significant
 * security considerations that must be accounted for when changing these,
 * but the security benefits can be great. Please be careful.
 */
VNET_DEFINE(int, ipport_reservedhigh) = IPPORT_RESERVED - 1;    /* 1023 */
VNET_DEFINE(int, ipport_reservedlow);

SYSCTL_INT(_net_inet_ip_portrange, OID_AUTO, reservedhigh,
        CTLFLAG_VNET | CTLFLAG_RW | CTLFLAG_SECURE,
        &VNET_NAME(ipport_reservedhigh), 0, "");
```



> As on the host this needs to be set to 0 to allow mac_portacl to operate,
> though to change this I had to drop the initial jail securelevel.

Tree indeed.
Comment 5 Zhenlei Huang freebsd_committer freebsd_triage 2023-09-16 02:23:01 UTC
(In reply to Zhenlei Huang from comment #4)
> Tree indeed.

Sorry for the typo :( It was `True indeed`.
Comment 6 Zhenlei Huang freebsd_committer freebsd_triage 2023-09-16 02:36:04 UTC
(In reply to uros from comment #1)
> Anyone have any idea what could be checked here?

For vnet jails, adjust vnet sysctl `net.inet.ip.portrange.reservedhigh` appropriately.
Comment 7 Zhenlei Huang freebsd_committer freebsd_triage 2023-09-16 02:43:26 UTC
Close as that is expected behavior.
Comment 8 Thomas Hurst 2023-09-16 11:30:47 UTC
A small note in mac_portacl(4) that the reservedlow/high oids are per-vnet - in contrast with the global mac_portacl policy - would probably have made all the difference here.
Comment 9 Mina Galić freebsd_triage 2023-09-16 13:12:11 UTC
(In reply to Thomas Hurst from comment #8)
you wanna prepare a patch?