I have been running jailed kerberized NFSv4 server for many months on FreeBSD 14.x (latest being 14.3). I have been mounting exports from this server on my linux clients and the server has been working flawlessly. After an attempted upgrade to 15.0-RELEASE, I can no longer mount any of the exports. the mount fails with a "Permission Denied" error. I have replicated the server configuration on a non-jailed host and have no problem mounting the shares. When running gssd in verbose mode on the 15.0 jail, I see no evidence in daemon.log of any upcalls from the kgssapi kernel module in the jail. Conversely, when running the server with 14.3 or on an unjailed host, I see the expected upcalls to get names and credentials in the jail and ultimately authentication is successful. Examining source code for gssd, I noticed a change in IPC (kernel to gssd) strategy from using local unix sockets on 14.x to using Netlink multicast on 15.0. In reading reading man pages, web articles and other literature on the subject, I am unclear whether the boundary/scope of Netlink multicast traffic is the machine or the network stack. If it is the later, it seems that Netlink multicast was not designed to traverse bridges and epairs to another VNET and that would certainly explain why gssd is not working in a VNET jail.
*** Bug 294502 has been marked as a duplicate of this bug. ***
I've added glebius@ as a cc. Hopefully he knows what it takes to make netlink upcalls work within a vnet prison?
It would help to at least know if Netlink Multicast RPC calls will propagate to a VNET jail through a bridge and an epair.
Could be related to this: https://reviews.freebsd.org/D39180
(In reply to Richard Straka from comment #3) Netlink is by definition a local socket to communicate from a process to kernel, just like route(4) did before it. There are no packets on wire generated by netlink. The vnet(4) by definition is an emulation of a separate host that is connected to other vnets by "emulated wires". Thus, the very last question isn't a legit one as the whole idea of netlink traversing bridges and epairs. However, the bug report definitely is legit. Looks like there is a mismatch between the vnet namespace where the NFS server operates and where gssd. Can you please share how you start the server and associated daemons in your jail?
I got email from glebius@ to-day. He thinks the problem is that the name is not vnet'd. I will be setting up a jailed test config in about a week and will see if I can come up with a patch. If you want to look at the netlink code and figure out where to vnet the name stuff, feel free to do so.
With very brief analysis I did today, I don't think that the fact that the name database is not virtualized is the cause of this bug report. The global database could lead to different bugs - e.g. impossible to register a name in vnet A due to vnet B already registered it. I'm afraid there is more than that.
Thanks for setting me straight on the fact that netlink does not use the network stack for transport and is essentially just another IPC mechanism between the kernel and processes. It sounds like this issue is one of vnet isolation and namespace/scope. To answer your question regarding how I start the processes Host's loader.conf kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" zfs_load="YES" kgssapi_load="YES" kgssapi_krb5_load="YES" Jail's rc.conf (NFS relevant portion) nslcd_enable="YES" gssd_enable="YES" gssd_flags="-v -h" nfs_server_enable="YES" nfs_server_flags="-t" nfsv4_server_enable="YES" nfsv4_server_only="YES" nfsuserd_enable="YES" nfs_reserved_port_only="NO" mountd_enable="YES" mountd_flags="-n"
Can you please also share the jail configuration?
I use bastille for jail management. jail.conf: nfstest { enforce_statfs = 1; devfs_ruleset = 13; exec.clean; exec.consolelog = /var/log/bastille/nfstest_console.log; exec.start = '/bin/sh /etc/rc'; exec.stop = '/bin/sh /etc/rc.shutdown'; host.hostname = nfstest.testdom.net; mount.devfs; mount.fstab = /bastille/jails/nfstest/fstab; path = /bastille/jails/nfstest/root; securelevel = 2; osrelease = 15.0-RELEASE; mount.fdescfs; allow.mount; allow.mount.fdescfs; zfs.mount_snapshot = 1; allow.nfsd; vnet; vnet.interface = e0b_nfstest; exec.prestart += "epair0=\$(ifconfig epair create) && ifconfig \${epair0} up name e0a_nfstest && ifconfig \${epair0%a}b up name e0b_nfstest"; exec.prestart += "ifconfig bridge0 addm e0a_nfstest"; exec.prestart += "ifconfig e0a_nfstest description \"vnet0 host interface for Bastille jail nfstest\""; exec.poststop += "ifconfig e0a_nfstest destroy"; }
Ok, I'm gonna ask here, since you're jail guys... I'm trying to set up a vnet jail to test this stuff (which I haven't done in years), but I can't get the network going. Every way I try it, I get.. ifconfig: BRDGADD e0a_myjail: Invalid argument (extended error Invalid argument size) so although the jail looks ok, it can't talk to the rest of the network, because the "ifconfig bridge0 addm e0a_myjail" fails. Is there another way to get the epair to talk to the net than using bridge0?
(In reply to Rick Macklem from comment #11) I worked around it. I don't know why I couldn't get the bridge to work, but if I just use an epair on a separate subnet and then route things, it works. This should be good enough for me to do testing.
(In reply to Gleb Smirnoff from comment #7) Well, the name is a problem, in part because the call to client_nl_create() is done in kgss_load(). Once I moved it to a kgss_vnetinit() function and used a different name for each vnet, it gets further (although not working, as yet). I've looked at netlink_generic.c and making the genl_register_xxx() and genl_unregister_xxx() functions use vnet'd arrays for families and groups isn't too bad. --> However, they get called from various places and CURVNET_SET()/CURVNET_RESTORE() will need to be done around the calls that are not done by vnetinit functions. I see two approaches.. - Continue with vnet'ng the netlink_generic.c functions (and there are probably others). This will take a while to get correct. or - Leave the namespace global and require applications like gssd to use vnet specific names, such as "kgss.jid" instead of just "kgss". This would be easier and a much smaller patch, although I cannot be sure that the upcalls will actually work until I try it. So, which approach do you think is preferred?
Thanks a lot for working on that, Rick! Where was it stuck and where further did you get? Sorry, I recalled my code and I'd probably withdraw my earlier advice on looking at making RB_HEAD rpcnl_clients virtualized :( I should have recalled the code better before making advices. The way I initially designed it is the following. Since generic Netlink is global, a kernel module (e.g. kgss) registers a global clnt_nl. The word "client" is a bit misleading here, as it hints for "ehrm, this should be virtualized". In reality this "client" is more like a "module", it creates linkage between certain class of kernel clients and a certain class of userland helpers. Thus, the vnet context appears only RPC call level. So, all calls and all replies for all vnets are routed through the same clnt_nl client instance. That's why kgss registers itself at gss_load() globally. Then, when kgss actually does a call, it does clnt_nl_call() in a proper vnet context and clnt_nl_call() will broadcast the message only to the helpers in this vnet and go to sleep(9). A kgssd(8) jailed to the same vnet that did clnt_nl_call() shall receive this message. Did you see that in your debugging? When kgssd(8) sends reply on its Netlink socket it is processed again in the same vnet and clnt_nl_reply() shall process it in the correct vnet (see CURVNET_ASSERT() there). Searching through request queue it matches this vnet against request vnet. So, the krpc<->netlink gate is kind of transparent wrt VIMAGE. All you need to do is to call clnt_nl_call in the right vnet.
Okay, I see the problem. Despite the client is created globally, the variable itself is virtualized. That's not correct. So, we need to move in the opposite direction :) Instead of bloating kgssd code with more vnet syntax, we should actually reduce it.
Here is a patch to try. Not that it was only compile tested. https://github.com/glebius/FreeBSD/commit/b40e5624c7cdc9103e42b0f3d7eedc687ccf6b6d
Created attachment 269946 [details] kgssapi: Fix upcalls in vnet jails You can try this patch. It seems to make the upcalls to the gssd daemons work for me. (My kerberos mounts aren't quite working, but I don't think that's because of this.) I've left some printf()s in, which might be useful for further debugging, if it doesn't work for you. I'm sticking this up now, since I won't be able to look at it again until next weekend. Maybe the reporter can test this and report back w.r.t. how it went?
(In reply to Gleb Smirnoff from comment #16) All my patch does is move kgss_gssd_handle out of the vnet, which was what I think you suggested. It shouldn't be necessary to have all those CURVNET_SET()/CURVNET_RESTORE() calls in the gss functions. The assumption is that the caller will set the vnet correctly. (The nfsd threads are all set correctly.) I suppose there should be a check for the vnet being set at the beginning of the functions. Anyhow, once I moved kgss_gssd_handle out of the vnet, the upcalls seem to be working.
Yes, you are right, all these added CURVNET_SET() are unnecessary, as if the curvnet was not set, we would have catched CURVNET_ASSERT_SET() in clnt_nl_call(). But there are three places in patch, where CURVNET_SET() was not added, but rather adjusted. I wonder if they are necessary? Anyway, it is going to look rather neat after all fixups. :)
After applying Rick's patch, I have no problem mounting NFS shares on my linux clients. I see the appropriate upcall activity in the nfs server jail's daemon.log and I also see what looks like kernel side activity (kgss_gssd_client) in the host's message.log. The kerberos credentials are now properly negotiated and authentication is successful. Thank you for fixing this issue. Is there a possibility this patch could make it into 15.1-RELEASE?
Richard, thanks a lot! Any chance you can test another patch later on a kernel compiled with INVARIANTS? I'd like to prepare a more thorough cleanup. As discussed above we have some CURVNET_SET() that are superfluous.
Sure, I'm happy to perform any tests you want. Just let me know what you need. I may need some instructions on configuring and compiling the kernel to get the info you want.
Richard, there are three patches that together make a candidate to fix this issue: https://reviews.freebsd.org/D56560 https://reviews.freebsd.org/D56561 https://reviews.freebsd.org/D56562 Alternatively, you can fetch them from here: https://github.com/freebsd/freebsd-src/compare/main...glebius:FreeBSD:kgss-vnet I have checked that they apply to stable/15 and I'm 99% confident that will apply to release/15.0. Can you please apply them one by one, compile kernel with options INVARIANTS and test the desired functionality?
I incrementally applied you patches to stable/15, recompiled the kernel with INVARIANTS each time and tested the ability to perform kerberos authenticated mounts from the jailed nfs server. Here are the results patch 1: mount NOT SUCCESSFUL - failed with "Permission denied" errors on the client - No evidence of upcalls to the gssd daemon in the log file on the jailed server patch 2: mount NOT SUCCESSFUL - failed with "Permission denied" errors on the client - No evidence of upcalls to the gssd daemon in the log file on the jailed server patch3: mount SUCCESSFUL - expected evidence of upcalls to the gssd daemon in the log file on the jailed server All other aspects of both the host and the jail appeared to be functioning nominally with each incremental patch set Please let me know if you need more info or further testing.
Thanks a lot Richard! I'm sorry that I wasn't clear enough that only the full patchset is worth testing for functionality.
All good. Just got a little more practice with patching and building kernels. A healthy exercise for sure!
(In reply to Richard Straka from comment #24) With the patch I did I got "permission denied" as well. I thought that it was because I had used a pre-release snapshot where the kerberos transition from Heimdal->MIT was in progress, but.. I'll set up a test environment using 15.0 on the weekend and poke at it. I thought I saw stuff in /var/log/daemon.log in the jail, but it was late Sun. night. (The attachment patch does basically what the glebius@ patches do, so I doubt it will behave differently?)
In applying all 3 patches from glebius@ to stable/15, kerberos authentication is working just fine for a jail nfs server. I am able to mount using kerberos authentication without problem on my linux clients. I had to apply all 3 patches in sequence before it worked properly. I now also see proper gssd daemon activity in the jail's daemon.log file.
(In reply to Richard Straka from comment #28) I just redid my test setup so that it works. (Used an up-to-date snapshot and not the broken old iso.) With glebius@'s 3 patches, everything works for me, too. So, once they are committed and MFC'd I think this PR is fixed. Thanks go to Richard and glebius@ for theid help, rick
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=4602d45eb3b1d33e0ea0d97c4d18033af95d7fca commit 4602d45eb3b1d33e0ea0d97c4d18033af95d7fca Author: Gleb Smirnoff <glebius@FreeBSD.org> AuthorDate: 2026-04-25 02:55:55 +0000 Commit: Gleb Smirnoff <glebius@FreeBSD.org> CommitDate: 2026-04-25 02:55:55 +0000 kgss: de-virtualize kgss_gssd_handle The RPC client is more of a class rather than an instance. RPCs from different VNETs are served by the same client. This makes the kgss layer fully transparent to VIMAGE and not even required to be aware of it. It is responsibility of the rpcsec_gss module to have curvnet set on the calling thread when doing RPC calls via kgssapi. This change should enable proper operation of an NFS server with gssd(8) in a VIMAGE jail. PR: 294501 Reviewed by: rmacklem Differential Revision: https://reviews.freebsd.org/D56562 sys/kgssapi/gss_impl.c | 24 +++++------------------- 1 file changed, 5 insertions(+), 19 deletions(-)
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=b5f4d40951c302cf803d3b7c18a17f9bc0f1c3f0 commit b5f4d40951c302cf803d3b7c18a17f9bc0f1c3f0 Author: Gleb Smirnoff <glebius@FreeBSD.org> AuthorDate: 2026-04-25 02:55:55 +0000 Commit: Gleb Smirnoff <glebius@FreeBSD.org> CommitDate: 2026-04-27 16:11:51 +0000 kgss: de-virtualize kgss_gssd_handle The RPC client is more of a class rather than an instance. RPCs from different VNETs are served by the same client. This makes the kgss layer fully transparent to VIMAGE and not even required to be aware of it. It is responsibility of the rpcsec_gss module to have curvnet set on the calling thread when doing RPC calls via kgssapi. This change should enable proper operation of an NFS server with gssd(8) in a VIMAGE jail. PR: 294501 Reviewed by: rmacklem Differential Revision: https://reviews.freebsd.org/D56562 (cherry picked from commit 4602d45eb3b1d33e0ea0d97c4d18033af95d7fca) sys/kgssapi/gss_impl.c | 24 +++++------------------- 1 file changed, 5 insertions(+), 19 deletions(-)
The glebius@ patches have been committed and MFC'd.
Thank you Rick and glebius@ for the rapid resolution to this issue. You guys are awesome!