When using direct attached console (monitor/keyboard) it is not possible to start jail from console different from console 0 (ttyv0) $ tty /dev/ttyv1 $ service jail onestart jailname results in Starting jails: cannot start jail "jailname" ID jail: jailname: jail_attach: Operation not permitted jail: jailname: /bin/sh /etc/rc: failed On FreeBSD 13.3 it works from any virtual console. Network tty (e.g. ssh localhost from ttyv1) works fine too. I use standard non-modified /etc/defaults/devfs.rules It does not contain any unhide for ttyv* on both FreeBSD 14.1 and 13.3, so the problem is probably not here.
Also, it jail was started from /etc/rc startup: it is not possible to jexec from any ttyv* terminal, even from ttyv0 ssh localhost jexec jailname helps
To start jail in a standard way you need to elevate your privileges first. Why do you think it's a bug and begin spamming the project's Bugzilla wasting time and resources?
Anything is, of course, from root.
(In reply to Marek Zarychta from comment #2) This is unprofessional. The reporter has shown enough skills not being that stupid not to be root in this case.
To be precise: 1) This is clean 14.1 install (in VMWare) 2) then updated to 14.1-p5 via freebsd-update 3) PermitRootLogin yes added to sshd_config for ssh tests 4) no other users added to system, everything is from root, no way to change user. 13.3 was tested the same way (clean install to VM, freebsd-update) And, yes, I use FreeBSD since 1994 or so for most my servers.
(In reply to Alex Tutubalin from comment #3) Please let me apologise then, and neither don't get offended. Thank you for reporting the issue. (In reply to Michael Osipov from comment #4) I follow Bugzilla and we have recently been receiving plenty of low-valued reports. The "$" sign at the reporter's prompt misled me this time. Thank you for defending the reporter. I am really sorry I have responded rudely without a deep investigation of the thread.
(In reply to Marek Zarychta from comment #6) Accepted. I know that you would look at '$' instead of being '#', but the user an change the prompt at will. Don't rely on such information.
(In reply to Marek Zarychta from comment #6) Error message was re-typed from VMWare screen, it seemed to me that $ is clearer than # (and I'm too lazy to retype full shell prompt with root@host)
Let me try to restart a Bastille jail from a HPE iLO 4 virtual terminal...
Tried: root ttyv1 28 Sep. 18:31 on FreeBSD deblndw011x.ad001.siemens.net 13.4-STABLE FreeBSD 13.4-STABLE 6bde10b63 GENERIC amd64 and it works... @Alex, how can this be a directly attached monitor/keyboard if it is a VM?
(In reply to Michael Osipov from comment #10) I do not think that from FreeBSD's side VM's ttyv* are different from the physical ones Will try to repeat on a physical PC tomorrow
(In reply to Michael Osipov from comment #10) And, yes, it works on 13.3. On 14.1 it does not.
(In reply to Alex Tutubalin from comment #11) (I forgot that I can connect a monitor to my home mail server/router) Reproduced with PHYSICAL monitor/keyboard on it service jail onestart jailname - starts ok on ttyv0 but not on ttyv1 FreeBSD 14.1-RELEASE-p2 (please do not ask to upgrade: no way to reboot it for a next few days: If I leave my family without internet for a several minutes, it will be the last thing I do.)
Could you do it under ktrace and find which syscall failed?
Created attachment 253874 [details] ktrace.out of service jail onestart example ktrace.out of service jail onestart example on ttyv1 (failed)
(In reply to Konstantin Belousov from comment #14) Sorry for answering with a dump of unknown quality - it's already very late here, tomorrow morning I'll read which particular call failed.
Created attachment 253882 [details] ktrace /usr/sbin/jail -c -f /etc/jail.conf example much smaller ktrace variant, no service jails.... but direct /usr/sbin/jail call Works on ttyv0 but not on ttyv1
(In reply to Konstantin Belousov from comment #14) attached much smaller variant: ktrace /usr/sbin/jail -c -f /etc/jail.conf example works on ttyv0, but not on ttyv1 My skills are not enough to quikly understand what went wrong
Created attachment 253883 [details] ktrace jexec 1 to jail created on startup, ttyv0 jail 1 has started from /etc/rc via pot service (it is bunch of shell scripts that calls /usr/sbin/jail to actual jail creation) jexec 1 from ttyv1 fails, ktrace attached.
Created attachment 253884 [details] ktrace jexec 1 to jail created on startup, pts/0 - success Same jail created on startup ssh root@localhost tty -> /dev/pts/0 ktrace jexec 1 => success
(In reply to Konstantin Belousov from comment #14) Also attached two ktraces to jexec 1 Jail is created on startup and shown via jls ttyv0: jexec 1 fails pts/0: jexec 1 succeeds
Comment on attachment 253883 [details] ktrace jexec 1 to jail created on startup, ttyv0 correction to the attachment message: jexec fails on ttyv0
Use at least the '-i' option for ktrace, to trace the children as well. From the attachment 'ktrace /usr/sbin/jail -c -f /etc/jail.conf example', the error comes from the /etc/rc execution in the jail. But we cannot see what happens there because the trace did not followed forks. BTW, did you looked at the jail startup log?
Created attachment 253886 [details] ktrace -i /usr/sbin/jail -c -f /etc/jail.conf example from ttyv1 ktrace -i /usr/sbin/jail -c -f /etc/jail.conf example from ttyv1
Created attachment 253887 [details] jail started via /etc/rc on boot: ktrace -i jexec 1 on ttyv0 jail started via /etc/rc on boot: ktrace -i jexec 1 on ttyv0
Created attachment 253888 [details] ssh localhost; ktrace -i jexec 1 date on jail started via /etc/rc (success) ssh localhost; ktrace -i jexec 1 date on jail started via /etc/rc (success)
(In reply to Konstantin Belousov from comment #23) Three ktrace -i has been added: 1) ktrace -i /usr/sbin/jail -c -f /etc/jail.conf example from ttyv1 - no jails are running - try to start new one on ttyv1 (it succeeds on ttyv0) 2) jail started via /etc/rc on boot: ktrace -i jexec 1 on ttyv0 - jail started on boot via /etc/rc (service pot start) - try to jexec 1 on ttyv0 3) ssh localhost; ktrace -i jexec 1 date on jail started via /etc/rc (success) successful one: same as #2 but current terminal is /dev/pts/0
The failing op is clear: 1727 jail CALL jail_attach(0x2) 1727 jail RET jail_attach -1 errno 1 Operation not permitted but why is it failing with EPERM is not. First question, do you have any mac modules configured and loaded? Second, could you try to install HEAD kernel (just kernel, not world) temporary and see if the funny failure mode persist?
(In reply to Konstantin Belousov from comment #28) Here is kldstat output: Id Refs Address Size Name 1 43 0xffffffff80200000 1f370e8 kernel 2 1 0xffffffff82138000 77d8 cryptodev.ko 3 1 0xffffffff82141000 5cd608 zfs.ko 4 1 0xffffffff83406000 3220 intpm.ko 5 1 0xffffffff8340a000 2178 smbus.ko 6 1 0xffffffff8340d000 7480 vmci.ko 7 1 0xffffffff83415000 3360 uhid.ko 8 1 0xffffffff83419000 33c0 usbhid.ko 9 1 0xffffffff8341d000 3380 hidbus.ko 10 1 0xffffffff83421000 3360 wmt.ko 11 1 0xffffffff83425000 2a68 mac_ntpd.ko Will try to install kernel This one, right: https://download.freebsd.org/snapshots/amd64/15.0-CURRENT/kernel.txz ?
(In reply to Alex Tutubalin from comment #29) Perhaps. Never installed kernel by any other way than compiling from sources.
(In reply to Konstantin Belousov from comment #30) The FreeBSD jtest14 15.0-CURRENT FreeBSD 15.0-CURRENT #0 main-n272495-6a4f0c063718: Thu Sep 26 03:46:06 UTC 2024 root@releng3.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 kernel does not solve the issue :( Still the same: - unable to /usr/sbin/jail -c jailname from ttyv1 - no problems with pts/0
(In reply to Konstantin Belousov from comment #28) Also, removing mac_ntpd (by not running ntpd) does not change anything.
(In reply to Alex Tutubalin from comment #32) This is very weird- can you try out a dtrace one-liner to try and narrow it down? From https://wiki.freebsd.org/DTrace/One-Liners and adjusted: dtrace -n 'fbt:::return /(int)arg1 == EPERM && pid == $pid/ { stack(); }' -c <command> I don't recall how dtrace -c word-splitting does or doesn't work, might need to toss the jail command into a shell script and execute that.
(In reply to Kyle Evans from comment #33) tried: dtrace -n 'fbt:::return /(int)arg1 == EPERM && pid == $pid/ { stack(); }' -c '/usr/sbin/jail -c example' it says: dtrace: description 'fbt::return ' matched 36022 probes and provides 1.7MB output/ 48k lines. Probably this is not you're looking for. Sorry, I'm FreeBSD user, not developer, know nothing about dtrace, so I can run test command sequences in monkey mode, but I'm unable to modify your one-liner for better match.
@kib, @kevans: Too early to tell for this case, but the only failures of jail_attach(2) with EPERM I've seen so far were caused by an outstanding open directory in the root shell launching jail(8) or jexec(8). This is how I recently discovered the pam_xdg() file descriptor leak, fixed in -CURRENT on 08/30 (06986e899972) and MFCed to 14-STABLE on 09/03 (9e8d504bb5a1) (13-STABLE not concerned). Given the tests performed by Alex, this is not the exact problem here (else he wouldn't be able to reproduce it on a recent snapshot of -CURRENT), but might be the same kind of problem. @Alex I'd suggest a couple of things: 1. Report the output of 'fstat -p $$' from the shells used to launch the jail(8) or jexec(8) commands, both in the succeeding and failing cases. 2. Launch and keep running this dtrace script, then launch a failing 'jexec': dtrace -n 'fbt::mac_vnode_check_chroot:return,fbt::change_dir:return,fbt::pwd_chroot_chdir:return /(int)arg1 == EPERM/ { stack(); }' You shouldn't get much output from it, so you can keep it running for a while. Try to correlate its new outputs with your actions (particular launches), and post all that here. Thanks and regards.
(In reply to Olivier Certner from comment #35) 1) fstat -p on ttyv0 (/usr/sbin/jail -c succeeds): USER CMD PID FD MOUNT INUM MODE SZ|DV R/W root sh 6384 text / 2188 -r-xr-xr-x 168344 r root sh 6384 ctty /dev 74 crw------- ttyv0 rw root sh 6384 wd / 5 drwxr-xr-x 23 r root sh 6384 root / 34 drwxr-xr-x 27 r root sh 6384 0 /dev 74 crw------- ttyv0 rw root sh 6384 1 /dev 74 crw------- ttyv0 rw root sh 6384 2 /dev 74 crw------- ttyv0 rw root sh 6384 10 /dev 74 crw------- ttyv0 rw 2) fstat -p on ttyv1 (jail -c fails): USER CMD PID FD MOUNT INUM MODE SZ|DV R/W root sh 6812 text / 2188 -r-xr-xr-x 168344 r root sh 6812 ctty /dev 75 crw------- ttyv1 rw root sh 6812 wd / 5 drwxr-xr-x 24 r root sh 6812 root / 34 drwxr-xr-x 27 r root sh 6812 0 /dev 75 crw------- ttyv1 rw root sh 6812 1 /dev 75 crw------- ttyv1 rw root sh 6812 2 /dev 75 crw------- ttyv1 rw root sh 6812 5 / 69283 drwx------ 4 r root sh 6812 10 /dev 75 crw------- ttyv1 rw It differs only by inode 69283 so find / -inum 69283 results in /var/run/xdg/root (this is folder with xdg_session.* file) 3) dtrace script you propose: - shows nothing if jail -c on ttyv0 - outputs this if jail -c is started on ttyv1: CPU ID FUNCTION:NAME 7 51518 pwd_chroot_chdir:return kernel`do_jail_attach+0x20b kernel`amd64_syscall+0x158 kernel`0xffffffff8104e7bb Things are more clear now: /var/run/xdg/username is created by pam_xdg, on 1st user login on ttyvN but not on pts/N So, ttyv0 is not magical: - logged out from all virtual consoles (ttyvN) - /var/run/xdg/root disappears - logged in on ttyv1 - folder re-created. - Now I can start jail from ttyv1 but not from ttyv0 or ttyv2 Still do not know what pam_xdg is :)
(In reply to Olivier Certner from comment #35) And, YES, there is no pam_xdg on FreeBSD 13.3 (reinstalled to see it)
(In reply to Olivier Certner from comment #35) And, OF COURSE, disabling pam_xdg in pam.d/session solves the issue.
(In reply to Alex Tutubalin from comment #36) Ok, so all is well and good... You can't reproduce the problem on 13.3, but on 14.1 and 14.1-p5. I was just wondering how you could reproduce on a recent 15-CURRENT (main-n272495-6a4f0c063718), which has the pam_xdg(8) fix, but from comment #28 and comment #29 I infer that you only installed the new kernel but not the associated world, which indeed explains. So, this problem is already fixed on -CURRENT and 14-STABLE, and the fix will appear in 14.2-RELEASE. That said, I will think about some followup commits (such as having jexec(8) automatically close all FDs being directories, or print something) because it is hard for users to diagnose such problems on their own.
(In reply to Olivier Certner from comment #39) This change should be part of the next 14.1-RELEASE patch release.
As for pam_xdg(8), you can lookup the man page (https://man.freebsd.org/cgi/man.cgi?query=pam_xdg), or see the commit that introduced it (https://cgit.freebsd.org/src/commit/?id=6e69612d5df1c1d5bd86990ea4d9a170c030b292). XDG's specification is here: https://specifications.freedesktop.org/basedir-spec/latest/. I don't know anything more on that topic.
(In reply to Olivier Certner from comment #39) Yes, it is 15-CURRENT kernel and 14.1-p5 userland. Great to hear the fix will be included in 14.1 too.
(In reply to Michael Osipov from comment #40) (In reply to Alex Tutubalin from comment #42) Well, decision to include as a patch on top of 14.1 is not up to me, but I've started talking about it, so let's see.
Created attachment 253912 [details] Erratum Notice proposal For secteam@: Attached an erratum notice proposal.
I hit this bug too. Will the fix be included in any 14.1 patch?
(In reply to Peter TKATCHENKO from comment #45) AFAIK, it was queued to be included in the next patch series, yes. As to when precisely it will be out, I don't know, but I expect in no more than a few weeks (else 14.2 will be around the corner anyway). In the meantime, you can workaround the problem, e.g., by disabling pam_xdg(8) (/etc/pam.d/system) or closing the leaked file descriptor from your shells. See the Errata Notice proposal attached to this bug.
I found this issue too. Am learning jails. (FreeBSD Noob) Running freebsd in a VM with amd64 14.1-RELEASE-p5. `freebsd-upgrade` did not fix it for me. Fixed by commenting out all the lines beginning with 'session' in /etc/pam.d/system then reboot. Files in /var/run/xdg/root disappeared after reboot. Thankyou for the helpful info here!
The erratum was published yesterday. Upgrading to 14.1-RELEASE-p6 will fix this problem.