Hi,

Rick Macklem was kind to implement the possibility to run NFS server nfsd(8) in a VNET Jail - as described in details in the FreeBSD 2022 Q3 Status Report:

- https://freebsd.org/status/report-2022-10-2022-12/#_enable_the_nfs_server_to_run_in_a_vnet_prison

Relevant commits:
- https://freshbsd.org/freebsd/src/commit/bba7a2e89602e6745bb2ec474f5ab714aef49f42
- https://freshbsd.org/freebsd/src/commit/d4b4f3b9c356938de6140ccea20d502d207b18a7
- https://freshbsd.org/freebsd/src/commit/6a76d35cac8e1549f74bd4cdceccc2ee52c8e556
- https://freshbsd.org/freebsd/src/commit/99187c3a44c2c3e168e462a30d45af075748195f
- https://freshbsd.org/freebsd/src/commit/7926a01ed7ae7cefd81ef4cc2142c35b84d81913
- https://freshbsd.org/freebsd/src/commit/7e44856e3a6deb194c2c376e886854b256360c40
- https://freshbsd.org/freebsd/src/commit/4d68605f31fb536722529dc90f16cc47d964882e
- https://freshbsd.org/freebsd/src/commit/ab0440af75ba0c2dbf263c5441ccbe4058515fff
- https://freshbsd.org/freebsd/src/commit/9d329bbc9aea6b8f47df251072bc65403ac3e43e
- https://freshbsd.org/freebsd/src/commit/f0db2b6022dfa15f375f5fcdd278b9df21cb88f5
- https://freshbsd.org/freebsd/src/commit/6444662a563ba714fed8563645764262c6f5e90f
- https://freshbsd.org/freebsd/src/commit/2894c8c96b9b94f35aaa27ee5ef3ac11c276fe3f
- https://freshbsd.org/freebsd/src/commit/b039ca0776774036a9e33aa45f50296a7f6ba547
- https://freshbsd.org/freebsd/src/commit/cd406ac94d8beae7f184adb14a3c94e058366b9a
- https://freshbsd.org/freebsd/src/commit/ed03776ca7f43de8275da80cfa89a9ecc4732f82
- https://freshbsd.org/freebsd/src/commit/ef6fcc5e2b0714c859d2e4ba23a55b1fd12f8a4e
- https://freshbsd.org/freebsd/src/commit/357492c99597d13bc966441f30bb44f6ef659f08
- https://freshbsd.org/freebsd/src/commit/ef4e8f0cf91f7009745d5a7a90d3bdd2d9e25780
- https://freshbsd.org/freebsd/src/commit/4036fcb8053adf3ac54c8428eef0dd076dfc1718
- https://freshbsd.org/freebsd/src/commit/10dff9da9748b0eadd2d02dded3afd2321d15537
- https://freshbsd.org/freebsd/src/commit/a90b47abcbdbd48c68fbf7c407546293479056de
- https://freshbsd.org/freebsd/src/commit/57ff348804f98d956f2e203b665de5a8989dbf8c
- https://freshbsd.org/freebsd/src/commit/4bbbd5875d32f3cbe76235d90243f713eff9b9d0
- https://freshbsd.org/freebsd/src/commit/0bb08f21cc5c62d0e2dfcea500521fa801058dd3
- https://freshbsd.org/freebsd/src/commit/84eac070494d81d0e0ded098a0275791874e5251
- https://freshbsd.org/freebsd/src/commit/9432e798fc6daaad341a496e9abcf9e3b760a63b
- https://freshbsd.org/freebsd/src/commit/a1254dcaa869bba20e46d966c53c7473bb24d02b

I even created article how to use it::
- https://vermaden.wordpress.com/2023/07/01/nfsv4-server-inside-freebsd-vnet-jail/

Rick also made important setup guide:
- https://people.freebsd.org/%7Ermacklem/nfsd-vnet-prison-setup.txt

I contacted Rick to make sure that I did not do any misconfiguration or something like that - an we came to a conclusion that some newer commit/change must have broke that.

I just tried to run (Again) NFSv4 server inside FreeBSD VNET Jail ... and I can not even mount the NFS share ... this is on 14.1-RELEASE.



These are exported NFS shares.

nfsd # cat /etc/exports 
V4: / -sec=sys -network 10.0.0.0/24
/share -sec=sys -maproot=root -network 10.0.0.0/24



I can not mount on 'other' system:

poudriere root ~ # mount -o nfsv4 10.1.1.99:/share /mnt
mount_nfs: nmount: /mnt: Permission denied



... or even on the nfsd(8) server:

nfsd # mount -o nfsv4 10.1.1.99:/share /mnt
mount_nfs: nmount: /mnt: Operation not permitted



After checking with tcpdump(8) the interesting error message seems to be this one:
- NFS reply xid 2939613111 reply ERR 20: Auth Credentials are too weak



Complete dump below.

nfsd # tcpdump -n port 2049
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on epair99b, link-type EN10MB (Ethernet), snapshot length 262144
bytes
18:27:49.336751 IP 10.1.1.123.619 > 10.1.1.99.2049: Flags [S], seq
1126678540, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val
4116429320 ecr 0], length 0
18:27:49.336788 IP 10.1.1.99.2049 > 10.1.1.123.619: Flags [S.], seq
2606098797, ack 1126678541, win 65535, options [mss 1460,nop,wscale
6,sackOK,TS val 1369673091 ecr 4116429320], length 0
18:27:49.336986 IP 10.1.1.123.619 > 10.1.1.99.2049: Flags [.], ack 1,
win 1027, options [nop,nop,TS val 4116429320 ecr 1369673091], length 0
18:27:49.337118 IP 10.1.1.123.619 > 10.1.1.99.2049: Flags [P.], seq
1:45, ack 1, win 1027, options [nop,nop,TS val 4116429320 ecr 1369673091],
length 44: NFS request xid 3699330492 40 null
18:27:49.337138 IP 10.1.1.99.2049 > 10.1.1.123.619: Flags [.], ack 45,
win 29128, options [nop,nop,TS val 1369673091 ecr 4116429320], length 0
18:27:49.337155 IP 10.1.1.99.2049 > 10.1.1.123.619: Flags [P.], seq
1:29, ack 45, win 29128, options [nop,nop,TS val 1369673091 ecr
4116429320], length 28: NFS reply xid 3699330492 reply ok 24 null
18:27:49.337385 IP 10.1.1.123.619 > 10.1.1.99.2049: Flags [F.], seq 45,
ack 29, win 1027, options [nop,nop,TS val 4116429320 ecr 1369673091],
length 0
18:27:49.337401 IP 10.1.1.99.2049 > 10.1.1.123.619: Flags [.], ack 46,
win 29128, options [nop,nop,TS val 1369673091 ecr 4116429320], length 0
18:27:49.337423 IP 10.1.1.99.2049 > 10.1.1.123.619: Flags [F.], seq 29,
ack 46, win 29128, options [nop,nop,TS val 1369673091 ecr 4116429320],
length 0
18:27:49.337584 IP 10.1.1.123.619 > 10.1.1.99.2049: Flags [.], ack 30,
win 1027, options [nop,nop,TS val 4116429320 ecr 1369673091], length 0
18:27:49.337800 IP 10.1.1.123.620 > 10.1.1.99.2049: Flags [S], seq
42080079, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 106674985
ecr 0], length 0
18:27:49.337829 IP 10.1.1.99.2049 > 10.1.1.123.620: Flags [S.], seq
4274899720, ack 42080080, win 65535, options [mss 1460,nop,wscale
6,sackOK,TS val 2800873825 ecr 106674985], length 0
18:27:49.337996 IP 10.1.1.123.620 > 10.1.1.99.2049: Flags [.], ack 1,
win 1027, options [nop,nop,TS val 106674985 ecr 2800873825], length 0
18:27:49.338055 IP 10.1.1.123.620 > 10.1.1.99.2049: Flags [P.], seq
1:281, ack 1, win 4352, options [nop,nop,TS val 106674985 ecr 2800873825],
length 280: NFS request xid 2939613111 276 getattr fh 0,10/1229193216
18:27:49.338071 IP 10.1.1.99.2049 > 10.1.1.123.620: Flags [.], ack 281,
win 29128, options [nop,nop,TS val 2800873825 ecr 106674985], length 0
18:27:49.338090 IP 10.1.1.99.2049 > 10.1.1.123.620: Flags [P.], seq
1:25, ack 281, win 29128, options [nop,nop,TS val 2800873825 ecr
106674985], length 24: NFS reply xid 2939613111 reply ERR 20: Auth
Credentials are too weak
18:27:49.338341 IP 10.1.1.123.620 > 10.1.1.99.2049: Flags [F.], seq 281,
ack 25, win 4352, options [nop,nop,TS val 106674985 ecr 2800873825], length
0
18:27:49.338356 IP 10.1.1.99.2049 > 10.1.1.123.620: Flags [.], ack 282,
win 29128, options [nop,nop,TS val 2800873825 ecr 106674985], length 0
18:27:49.338363 IP 10.1.1.99.2049 > 10.1.1.123.620: Flags [F.], seq 25,
ack 282, win 29128, options [nop,nop,TS val 2800873825 ecr 106674985],
length 0
18:27:49.338496 IP 10.1.1.123.620 > 10.1.1.99.2049: Flags [.], ack 26,
win 4352, options [nop,nop,TS val 106674985 ecr 2800873825], length 0



This is the VNET Jail config.

% cat /etc/jail.conf.d/nfsd.conf 
nfsd {

 # GLOBAL
 exec.start = "/bin/sh /etc/rc";
 exec.stop = "/bin/sh /etc/rc.shutdown";
 exec.consolelog = "/var/log/jail_console_${name}.log";
 exec.clean;
 mount.devfs;
 host.hostname = ${name};
 path = /jail/${name};

 # PERMISSIONS
 allow.raw_sockets;
 devfs_ruleset = 110;

 # NFSD/VNET
 allow.nfsd;
 enforce_statfs = 1;

 # VNET/VIMAGE
 vnet;
 vnet.interface = "${if}b";

 # NETWORKS/INTERFACES
 $id = "99";
 $ip = "10.1.1.${id}/24";
 $gw = "10.1.1.1";
 $br = "vm-public";
 $if = "epair${id}";

 # ADD TO bridge0 INTERFACE
 exec.prestart += "ifconfig ${if} create up";
 exec.prestart += "ifconfig ${if}a up descr jail:${name}";
 exec.prestart += "ifconfig ${br} addm ${if}a up";
 exec.start += "ifconfig ${if}b ${ip} up";
 exec.start += "route add default ${gw}";
 exec.poststop += "ifconfig ${if}a destroy";
}



As 14.2-RELEASE is month away from release it would be great to fix that before release.

Thanks,
vermaden