Berkeley DB5 (likely other versions as well) running on FreeBSD/arm 11.0 on a Raspberry Pi 2 issues the error BDB1577 Berkeley DB library configured to support only private environments for many operations. Specifically I'm using it with netatalk3 having everything installed from the official pkg repository. When netatalk tries to open a share, it needs to create a DB file to store the necessary mapping data. It always fails, and the db_errlog file is full of the above error message. Causing netatalk3 to use one of its fallback in-memory storage systems, one can connect to the files system using a Mac quite well. However, this is not a recommended configuration. The error can also be emitted by going to a directory with a DB file in it, and running db_checkpoint-5.3 -1 -h . This seems related to the --enable-posixmutexes option that is turned on for the arm build of the db5 port as per bug #197227 According to the DB5 manual for this option: "...configuring to use POSIX mutexes when the implementation does not have inter-process support will only allow the creation of private database environments". It appears that FreeBSD/arm 11.0 does not implement multi process mutexes still. There are other projects that also need to have shared mutexes working and this has been discussed occasionally since FreeBSD 9 at least. For example, sphinxsearch as per <https://lists.freebsd.org/pipermail/freebsd-questions/2012-June/242870.html> which also references a discussion from two years prior <http://freebsd.1045724.x6.nabble.com/What-is-the-status-of-thread-process-shared-synchronization-td4224458.html> which seems to not have a real conclusion. What's the solution here? Turning off the posixthreads seems like it will cause segfaults, and turning it on makes it useless in a multiprocessing context.
I see two approaches: 1. someone extends our pthreads implementation so that it supports PTHREAD_PROCESS_SHARED for pthread_condattr_setpshared(). I can't do that. 2. we add a new *CLEAN ROOM* ARM assembly synchronization for $WRKSRC/src/dbinc/mutex_int.h starting near #ifdef HAVE_MUTEX_ARM_GCC_ASSEMBLY -- that needs to use LDREX and STREX (load/store exclusive) instructions on newer ARM revisions, probably ldrex for the mutex_set and strex for the reset. Starting points are: * http://infocenter.arm.com/help/topic/com.arm.doc.dht0008a/DHT0008A_arm_synchronization_primitives.pdf and * http://www.keil.com/support/man/docs/armasm/armasm_dom1361289875835.htm (and the link to STREX). I am not familiar with ARM architecture and assembly, so I cannot implement that. 3. it appears that db6 has an implementation that works on newer ARM and that uses the newer ldrex/strex instructions, however as db6 is under the stricter Affero GNU General Public License v3 (which also requires the source code be made available for software-as-a-service deployments), and without further inquiry or license from Oracle I think we cannot usefully backport it because that would taint the db5 package with the db6 license - in that situation, it's more useful to use db6 directly.
A commit references this bug: Author: mandree Date: Tue Oct 4 09:32:48 UTC 2016 New revision: 423245 URL: https://svnweb.freebsd.org/changeset/ports/423245 Log: Avoid POSIX mutexes on armv6* and aarch64. db6 switched from the stpb-based mutexes that db5 used to ldrex/strex. PR: 213167 (related) Changes: head/databases/db6/Makefile
After commit r423245, db6 should support shared-process mutexes on ARMv6. Someone let me know if that also works on aarch64. If the stricter db6 license is suitable for the user's application, that should be a workaround.
Thanks so much Matthias! This is a great solution for me. I've configured my local poudriere box to build ARM6 packages so I can get a db6 version of netatalk3 built for my embedded device (also I build my packages without X11 which saves a ton of space). FWIW, I found this ancient bug report in Debian where someone submitted gcc ASM code for db3 mutexes: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=291901;msg=7 I have no clue if that is useful here based on your comments.
I've looked at the Debian stuff, and the implementation for ARM also uses the same deprecated SWPB instruction that db5 uses by default, so it's useless for us.
Created attachment 175438 [details] log showing build failure of db6 port Sad times... build failure. Output from poudriere cross-compile from an amd64 box is attached. The build succeeded until this patch was applied.
Info on ARM barrier instructions, which looks relevant. http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/ka14041.html This is all getting beyond my knowledge, but I'll try to see how to integrate it into the code.
Disregarding the host/jail mismatch warnings because you are cross-compiling, does it help if you add -march=armv6 or similar to the CFLAGS in your build environment, possibly through the poudriere-based etc/make.conf replacement? Not knowing ARM well, chances are that the compiler/assembler refuses to generate the DMB and DSB instructions because they aren't available in v5 and older processors and the build apparently has not permitted code that runs only on v6 or newer. If so, the FreeBSD-ARM guys may want to set that globally in the ports frameworks. Also note that cross-compiles may fail because the configure script uses run-time checks that are unavailable on amd64, and I haven't checked what assumptions these make for cross-compilation, so we may need to override these manually, or add --with-mutex=ARM/gcc-assembly to CONFIGURE_ARGS on ARMv6 and newer. In doubt, please try to compile natively on ARM.
Please let's continue the Berkeley-DB-6-on-ARM discussion in a new bug report, https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213223
The issue "Berkeley DB library configured to support only private environments" when attempting multi-process access to the database affects all architectures that need to use POSIX mutexes on FreeBSD because all FreeBSD releases as of today (comment's date) including 11.0-RELEASE do _not_ support pthread_condattr_getpshared(..., PTHREAD_PROCESS_SHARED).
Note that the databases/db6 changes for ARM/gcc-assembly mutexes are not directly portable backwards because: 1. db5 uses a swpb/eor command sequence, and swpb is not reflected in atomic.h and there is no alternative implementation for newer processors that do not support it. 2. Also note that copying the newer implementation back to db5 would bring the Affero GPL v3 into db5 - in that situation it will usually be easier to go for db6 directly on ARM.
In bug #213223 databases/db6 was patched to use the mutex from the machine/atomic.h header. Given that, should netatalk3 be marked "broken" for DB < 6 or can this same patch to use atomic.h work with db5? I think ideally if db5 can be patched, then the official packages can have a working netatalk3 since they are built against db5 by default.
It appears that https://svnweb.freebsd.org/changeset/base/296162 (OSVERSION > 1100100 according to jbeich@'s research) might make PTHREAD_PROCESS_SHARED workable for ARM; testing on FreeBSD 11.0-RELEASE required though.
Vick, could you do the following on FreeBSD 11.0-RELEASE on ARMv6: # update ports tree (your usual means) cd /usr/ports/databases/db5 make CONFIGURE_ARGS=--with-mutex=POSIX/pthreads cd `make -V WRKSRC` make test_mutex ./test_mutex The expected PASS result is a long line of checks, similar to this: test_mutex: 5 processes, 4 threads/process, 10000 lock requests from 20 locks test_mutex: backing data 984 bytes Locker: ID 008 (PID: 72426; TID: 801616500) Locker: ID 016 (PID: 72428; TID: 801616500) [...] 013: 9000 018: 9000 007: 9000 017: 9000 008: 9000 [...] Per-lock mutex statistics. mutex 0: wait: 5350; no wait 4495 mutex 1: wait: 5350; no wait 4560 [...] mutex 19: wait: 5626; no wait 4467 test_mutex: test succeeded A test FAILURE looks like this: test_mutex: 5 processes, 4 threads/process, 10000 lock requests from 20 locks test_mutex: backing data 984 bytes test_mutex: BDB1577 Berkeley DB library configured to support only private environments test_mutex: environment open: TESTDIR: Invalid argument If things go well, please 1. run "make clean" and 2. repeat the entire test procedure, omitting the CONFIGURE_ARGS=... from the make command.
before "make clean" in the final step, you'll need to cd back to /usr/ports/databases/db5
I've done the tests on my beaglebone running 12-CURRENT: All tests pass with CONFIGURE_ARGS=--with-mutex=POSIX/pthreads and the tests fail without CONFIGURE_ARGS Thanks!
A commit references this bug: Author: mandree Date: Fri Nov 4 00:09:52 UTC 2016 New revision: 425280 URL: https://svnweb.freebsd.org/changeset/ports/425280 Log: Force pthreads mutexes on 11+ and aarch64/ARMV6 This currently leaves ARMv6 and aarch64 on FreeBSD 9.x and 10.x out in the rain because these will still autoconfigure to only permit private mutexes. PR: 213167 Submitted by: jbeich (IRC) Changes: head/databases/db5/Makefile
(In reply to mikael.urankar from comment #16) Mikael, thanks for your tests. I'll leave this bug report open for now since we don't yet have a solution for db5 on FreeBSD 9 or 10 on ARM v6 or newer/AARCH64 - if we find one, we'll consider it, otherwise this PR will be closed when 10.3 goes out of support in May 2018 as "overcome by events".
(In reply to Matthias Andree from comment #18) Since the affected FreeBSD versions have gone out of support, we can close this report.