This is one of the most weird bug I have come across since using FreeBSD jails. I have now made an rc.d script to restart my jail a 2nd time every boot… and sysvipc feature will then start working inside of the jail. Without such intervention, a jail is normally started 1 time after system reboot. Then for me it isn't working inside of the jail. This concerns only "allow.sysvipc" in jail.conf file. I have not set any other kinds of "global" sysvipc options for the host or the jail. Sorry I can't provide a test case for this problem. It was found with qjail-3.4, on FreeNAS. But it would be great if anyone else can try to reproduce this situation. To see if it occurs on any other FreeBSD versions - 9.2, 9.3, 10.0, STABLE or CURRENT. My uname -a: FreeBSD freenas.local 9.2-RELEASE-p3 FreeBSD 9.2-RELEASE-p3 #0 r262572+7b72365: Fri Mar 14 15:50:04 PDT 2014 root@build.ixsystems.com:/home/jkh/9.2.1-BRANCH/freenas/os-base/amd64/tank/home/jkh/9.2.1-BRANCH/freenas/FreeBSD/src/sys/FREENAS.amd64 amd64
Over to maintainers.
Dreamcat, Testing on two stables built within last 24 hours # uname -oprUK FreeBSD 9.3-PRERELEASE i386 903500 903500 and FreeBSD 10.0-STABLE amd64 1000710 1000710 Looks like this issue is no reproducible in later versions (there have been a lot of changes in Stable since 9.2R): # sysctl -a|grep sysvi security.jail.param.allow.sysvipc: 0 security.jail.sysvipc_allowed: 1 My jail.conf contains test1 { ip4.addr = "10.0.5.241"; devfs_ruleset = "4"; allow.sysvipc; allow.chflags; } which is the jail that I tested for sysvipc.
Thanks man! It's good news to hear. Hopefully the bug will go away once I'm moved to 10.0.(In reply to dewayne from comment #2) > Dreamcat, Testing on two stables built within last 24 hours > # sysctl -a|grep sysvi > security.jail.param.allow.sysvipc: 0 > security.jail.sysvipc_allowed: 1 > > My jail.conf contains > test1 { ip4.addr = "10.0.5.241"; devfs_ruleset = "4"; allow.sysvipc; > allow.chflags; } > > which is the jail that I tested for sysvipc. Thanks for testing this. You individual jail setting looks good. My global sysvipc setting is different than yours however: freenas ~/ root^> sysctl -a|grep Sylvia security.jail.param.allow.sysvipc: 0 security.jail.sysvipc_allowed: 0 Reason: Don't want to enable it for all the other jails. It is only needed on 1 specific jail. FYI: Another PR relevant to isolating ipc to single-jail: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=48471
I have the identical issue using Freebsd 10.0, qjail 3.4, and the allow.sysvipc flag. It does not matter whether security.jail.sysvipc_allowed is set to 0 or 1 on the host. # uname -oprUK FreeBSD 10.0-RELEASE-p6 amd64 1000510 1000510 Relevant section of qjail log file on first jail start: FATAL: could not create shared memory segment: Function not implemented DETAIL: Failed system call was shmget(key=5432001, size=40, 03600). Like dreamcat4, sysvipc works once the jail has been started a second time after each time the host boots.
When you say the allow.sysvipc parameter has no effect on a jails first start after system boot. Just how are you determining this? Do you see the "allow.sysvipc" listed by the "jls -name -j jailname" command. I installed 10.0 from disc1.iso to a empty hard drive and running qjail-3.4 and after starting the jail "jls -name -j jailname" shows "allow.sysvipc" which means its enabled, and no error messages in the jails console log. Seeing jls showing the "allow.sysvipc" instead of "allow.nosysvipc" is the only indicator I have available to verify its being set correctly. This indicator does not really prove the sysvipc function for the jail is functional. As far as I know you need to run some application in the jail that requires sysvipc access as the only true test. This application may have to be started one time to set some application internal default setting before it knows sysvipc is enabled on its second start. Look for a application configure file to set sysvipc as the default instead of the tcp default setting. What application are you running in the jail and how does that application get started? Almost 99% sure your problem is caused by your jailed application and not qjail or jail(8).
(In reply to joeb1 from comment #5) > When you say the allow.sysvipc parameter has no effect on a jails first > start after system boot. Just how are you determining this? I was previously checking the log file of the program 'zabbix2-server'. Which is unable to start, and logfile gives the reason: zabbix_server [4414]: cannot create Semaphore: [78] Function not implemented zabbix_server [4414]: unable to create mutex for log file However now that someone else has reproduced it too, I will try more things! > Do you see the "allow.sysvipc" listed by the "jls -name -j jailname" command. > I installed 10.0 from disc1.iso to a empty hard drive and running qjail-3.4 > and after starting the jail "jls -name -j jailname" shows "allow.sysvipc" > which means its enabled, and no error messages in the jails console log. This is on my host, after a fresh reboot: freenas // root^> qjail list STA JID NIC IP Jailname --- ---- --- --------------- -------------------------------------------------- DR 1 re0 192.168.1.205 nas4free DR 2 re0 192.168.1.81 nginx-webdav DR 3 re0 192.168.1.206 openvpn lo0|127.0.0.1 DR 4 re0 192.168.1.38 ps3netsrv DR 5 re0 192.168.1.207 tvheadend lo0|127.0.0.207 DR 6 re0 192.168.1.223 ums4 lo0|127.0.0.223 DR 7 re0 192.168.1.41 virtualbox lo0|127.0.0.2 DR 8 re0 192.168.1.214 webcamd lo0|127.0.0.214 DR 9 re0 192.168.1.212 zabbix lo0|127.0.0.212 freenas // root^> jls -h -j zabbix allow.sysvipc allow.sysvipc 0 freenas // root^> qjail restart zabbix Jail successfully stopped zabbix Jail successfully started zabbix freenas // root^> jls -h -j zabbix allow.sysvipc allow.sysvipc 1 Above we can see that jls will indeed report the problem if it occurs. Since I can still reproduce the error, I am investigating more today. Please bear with me... > Seeing jls showing the "allow.sysvipc" instead of "allow.nosysvipc" is the > only indicator I have available to verify its being set correctly. This There is also the command 'ipcs', which can be run inside the jail. Here again is my output after a another system reboot: freenas // root^> qjail console zabbix Last login: Sun Jul 6 08:05:03 on pts/0 FreeBSD 9.2-RELEASE-p3 (FREENAS.amd64) #0 r262572+7b72365: Fri Mar 14 15:50:04 PDT 2014 Welcome to your FreeBSD jail. zabbix ~/ root~# ipcs Message Queues: T ID KEY MODE OWNER GROUP Shared Memory: T ID KEY MODE OWNER GROUP Semaphores: T ID KEY MODE OWNER GROUP zabbix ~/ root~# exit logout freenas // root^> qjail restart zabbix Jail successfully stopped zabbix Jail successfully started zabbix freenas // root^> qjail console zabbix Last login: Sun Jul 6 08:53:45 on pts/0 FreeBSD 9.2-RELEASE-p3 (FREENAS.amd64) #0 r262572+7b72365: Fri Mar 14 15:50:04 PDT 2014 Welcome to your FreeBSD jail. zabbix ~/ root~# ipcs Message Queues: T ID KEY MODE OWNER GROUP Shared Memory: T ID KEY MODE OWNER GROUP m 65536 1745323649 --rw------- zabbix zabbix m 65537 2013759105 --rw------- zabbix zabbix m 65538 1946650241 --rw------- zabbix zabbix m 65539 1728546433 --rw------- zabbix zabbix m 65540 1929873025 --rw------- zabbix zabbix m 65541 1393002113 --rw------- zabbix zabbix m 65542 1980204673 --rw------- zabbix zabbix m 65543 1812431314 --rw------- zabbix zabbix Semaphores: T ID KEY MODE OWNER GROUP s 65536 2047313537 --rw------- zabbix zabbix s 65537 2047312338 --rw------- zabbix zabbix zabbix ~/ root~# > indicator does not really prove the sysvipc function for the jail is > functional. As far as I know you need to run some application in the jail > that requires sysvipc access as the only true test. This application may > have to be started one time to set some application internal default setting > before it knows sysvipc is enabled on its second start. Look for a That would suggest be could just be restarting the zabbix_server application (rather than the jail). However that is not the case here. 2nd, 3rd, 4th, restart etc of zabbix_server rc.d script makes no difference. Wheras restarting the jail once, zabbix did not repeat error message, and all was OK. > application configure file to set sysvipc as the default instead of the tcp > default setting. What application are you running in the jail and how does > that application get started? Unfortunately sysvipc / unix semaphores is always required for this particular program (zabbix). It has no option to switch them off, or use some alternative mechanism instead (such as TCP). Otherwise I would have disabled sysvipc usage in the zabbix application a long time ago. > Almost 99% sure your problem is caused by your jailed application and not > qjail or jail(8). Nah. I would be very surprised, given today's output from jls and ipcs commands, that the problem is anything to do with the zabbix application itself. It just seems some of us could reproduce this issue, and some of us can't. We seem to have 2 reports of success. And equally 2 of fail. What seems to be missing is better instructions to reproduce this (my fault). There must be some other circumstances specific to my host, which is triggering this to occur... I will find out today. For one thing, we know that on startup, qjail is changing the same jail.conf file. Then re-calling jail(8) program again on the next jail in the list. So maybe that's got something to do with it. Please bear with me. I will look into it further.
Created attachment 144450 [details] More comprehensive test cases & results OK. This occur on qjail 3.4. I have tested more thoroughly now. For full details of those testing (to know too what else is eliminated), is documented in the attached .TXT file "testing-details.txt". My new findings: * The problem only appears when jail is started by the 'qjail.bootime' rc.d script. * The problem does not occur if the jail is started from the command line (by typing "qjail start $jailname"). * The 'qjail.bootime' rc.script can be restarted once, twice, tree times. After bootup (of which not matter if occur with rest of system boot). And the reported problem (sysvipc not working) will still occur. Does not 'go away' 2nd time. Recommend: future investigations should focus more on what is happening inside the 'qjail.bootime' rc.d scipt. I will continue to look further. Many thanks.
Found it. The problem occurs in qjail program (not rc.d script). When saving the definition records (the qjail config file_. I think when enable with 'config -y', it save 'allow.sysvipc' correctly into 'qjail.local/$jailname'. But not get saved into 'qjail.global/$jailname'. So hence the rc.d start not work, whereas the command line does work. Not just sysvipc line. But devfs_ruleset line is also missing: ruleset="" sysvipc="" Do not appear in the qjail.global copy of the definition record. It is likely that the variable is set correct inside qjail program, but those extra line were not inserted of both templates (only local), so is missed when writing the 'qjail.global' to disk. See here: freenas // root^> cat /usr/local/etc/qjail.global/webcamd name="webcamd" ip4="192.168.1.214,lo0|127.0.0.214" ip6="" path="/usr/jails/webcamd" interface="re0" fstab="/usr/local/etc/qjail.fstab/webcamd" securelevel="" cpuset="" fib="" vnet="" vinterface="" rsockets="allow.raw_sockets" quotas="" nullfs="" zfs="" poststartssh="" deffile="/usr/local/etc/qjail.local/webcamd" image="" imagetype="" imageblockcount="" imagedevice="" freenas // root^> cat /usr/local/etc/qjail.local/webcamd name="webcamd" ip4="192.168.1.214,lo0|127.0.0.214" ip6="" path="/usr/jails/webcamd" interface="re0" fstab="/usr/local/etc/qjail.fstab/webcamd" securelevel="" cpuset="" fib="" vnet="" vinterface="" rsockets="allow.raw_sockets" ruleset="" sysvipc="allow.sysvipc" quotas="" nullfs="" zfs="" poststartssh="" deffile="/usr/local/etc/qjail.local/webcamd" image="" imagetype="" imageblockcount="" imagedevice="" freenas // root^>
No. Scratch that. This IS the qjial.bootime rc.d script. Who knew that was a place to overwrite jails settings files ??!! Clearly not I. Otherwise would have known to be including such fixes / ammendments in with the original patch. Don't even want to ask why. Or wish to hear any explanations for reasons behind what is going on in there. Let's just make a patch for it (the qjail rc.d script). To be included whenever the next qjail release. (from my point) This is not considered to be terribly urgent (for me / Finch users). Since we have an auto-patching function for qjail. I make a patch soon.
Created attachment 144452 [details] Patch file, in 'diff -ruN' format This patch aught to solve the issue. Not considered urgent. (Finch will auto patch in meantime, until next official qjail release). Also included: -ge 92 compatibility fix for Finch users on FreeNAS and NAS4Free. Who are still on FreeBSD 9.2. (they still can't upgrade yet). Without that will causes some Finch users problems / qjail installation error.
Created attachment 144453 [details] official maintainer qjail.bootime patch
Comment on attachment 144452 [details] Patch file, in 'diff -ruN' format Superseeded by Joe's patch.
(In reply to joeb1 from comment #11) > Created attachment 144453 [details] > official maintainer qjail.bootime patch Joe, thanks for the patch correction. And fast response. Have re-tested with patch applied (it works).
Joe has fixed in qjail 3.5. It's been committed.