Bug 213167 - databases/db5 "Berkeley DB library configured to support only private environments" with POSIX mutexes (f. i. ARMv6)
Summary: databases/db5 "Berkeley DB library configured to support only private environ...
Status: Closed Overcome By Events
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: arm Any
: --- Affects Many People
Assignee: Matthias Andree
URL:
Keywords:
Depends on: 213223
Blocks:
  Show dependency treegraph
 
Reported: 2016-10-03 02:15 UTC by Vick Khera
Modified: 2019-01-07 00:22 UTC (History)
2 users (show)

See Also:
mandree: maintainer-feedback+


Attachments
log showing build failure of db6 port (27.76 KB, text/plain)
2016-10-05 01:57 UTC, Vick Khera
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Vick Khera 2016-10-03 02:15:33 UTC
Berkeley DB5 (likely other versions as well) running on FreeBSD/arm 11.0 on a Raspberry Pi 2 issues the error

 BDB1577 Berkeley DB library configured to support only private environments

for many operations.

Specifically I'm using it with netatalk3 having everything installed from the official pkg repository. When netatalk tries to open a share, it needs to create a DB file to store the necessary mapping data. It always fails, and the db_errlog file is full of the above error message. Causing netatalk3 to use one of its fallback in-memory storage systems, one can connect to the files system using a Mac quite well. However, this is not a recommended configuration.

The error can also be emitted by going to a directory with a DB file in it, and running 

 db_checkpoint-5.3 -1 -h .


This seems related to the --enable-posixmutexes option that is turned on for the arm build of the db5 port as per bug #197227

According to the DB5 manual for this option: "...configuring to use POSIX mutexes when the implementation does not have inter-process support will only allow the creation of private database environments". It appears that FreeBSD/arm 11.0 does not implement multi process mutexes still.

There are other projects that also need to have shared mutexes working and this has been discussed occasionally since FreeBSD 9 at least. For example, sphinxsearch as per <https://lists.freebsd.org/pipermail/freebsd-questions/2012-June/242870.html> which also references a discussion from two years prior <http://freebsd.1045724.x6.nabble.com/What-is-the-status-of-thread-process-shared-synchronization-td4224458.html> which seems to not have a real conclusion.

What's the solution here? Turning off the posixthreads seems like it will cause segfaults, and turning it on makes it useless in a multiprocessing context.
Comment 1 Matthias Andree freebsd_committer 2016-10-04 09:29:02 UTC
I see two approaches:

1. someone extends our pthreads implementation so that it supports PTHREAD_PROCESS_SHARED for pthread_condattr_setpshared(). I can't do that.

2. we add a new *CLEAN ROOM* ARM assembly synchronization for  $WRKSRC/src/dbinc/mutex_int.h starting near #ifdef HAVE_MUTEX_ARM_GCC_ASSEMBLY -- that needs to use LDREX and STREX (load/store exclusive) instructions on newer ARM revisions, probably ldrex for the mutex_set and strex for the reset. 

Starting points are:
* http://infocenter.arm.com/help/topic/com.arm.doc.dht0008a/DHT0008A_arm_synchronization_primitives.pdf and 
* http://www.keil.com/support/man/docs/armasm/armasm_dom1361289875835.htm (and the link to STREX).

I am not familiar with ARM architecture and assembly, so I cannot implement that.

3. it appears that db6 has an implementation that works on newer ARM and that uses the newer ldrex/strex instructions, however as db6 is under the stricter Affero GNU General Public License v3 (which also requires the source code be made available for software-as-a-service deployments), and without further inquiry or license from Oracle I think we cannot usefully backport it because that would taint the db5 package with the db6 license - in that situation, it's more useful to use db6 directly.
Comment 2 commit-hook freebsd_committer 2016-10-04 09:33:44 UTC
A commit references this bug:

Author: mandree
Date: Tue Oct  4 09:32:48 UTC 2016
New revision: 423245
URL: https://svnweb.freebsd.org/changeset/ports/423245

Log:
  Avoid POSIX mutexes on armv6* and aarch64.

  db6 switched from the stpb-based mutexes that db5 used to ldrex/strex.

  PR:		213167 (related)

Changes:
  head/databases/db6/Makefile
Comment 3 Matthias Andree freebsd_committer 2016-10-04 09:34:10 UTC
After commit r423245, db6 should support shared-process mutexes on ARMv6. Someone let me know if that also works on aarch64. If the stricter db6 license is suitable for the user's application, that should be a workaround.
Comment 4 Vick Khera 2016-10-04 12:58:51 UTC
Thanks so much Matthias! This is a great solution for me.

I've configured my local poudriere box to build ARM6 packages so I can get a db6 version of netatalk3 built for my embedded device (also I build my packages without X11 which saves a ton of space).

FWIW, I found this ancient bug report in Debian where someone submitted gcc ASM code for db3 mutexes:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=291901;msg=7

I have no clue if that is useful here based on your comments.
Comment 5 Matthias Andree freebsd_committer 2016-10-04 21:03:57 UTC
I've looked at the Debian stuff, and the implementation for ARM also uses the same deprecated SWPB instruction that db5 uses by default, so it's useless for us.
Comment 6 Vick Khera 2016-10-05 01:57:46 UTC
Created attachment 175438 [details]
log showing build failure of db6 port

Sad times... build failure. Output from poudriere cross-compile from an amd64 box is attached. The build succeeded until this patch was applied.
Comment 7 Vick Khera 2016-10-05 02:40:19 UTC
Info on ARM barrier instructions, which looks relevant.

http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/ka14041.html

This is all getting beyond my knowledge, but I'll try to see how to integrate it into the code.
Comment 8 Matthias Andree freebsd_committer 2016-10-05 06:49:34 UTC
Disregarding the host/jail mismatch warnings because you are cross-compiling, does it help if you add -march=armv6 or similar to the CFLAGS in your build environment, possibly through the poudriere-based etc/make.conf replacement?  

Not knowing ARM well, chances are that the compiler/assembler refuses to generate the DMB and DSB instructions because they aren't available in v5 and older processors and the build apparently has not permitted code that runs only on v6 or newer.

If so, the FreeBSD-ARM guys may want to set that globally in the ports frameworks.


Also note that cross-compiles may fail because the configure script uses run-time checks that are unavailable on amd64, and I haven't checked what assumptions these make for cross-compilation, so we may need to override these manually, or add --with-mutex=ARM/gcc-assembly to CONFIGURE_ARGS on ARMv6 and newer.  In doubt, please try to compile natively on ARM.
Comment 9 Matthias Andree freebsd_committer 2016-10-05 06:59:23 UTC
Please let's continue the Berkeley-DB-6-on-ARM discussion in a new bug report, https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213223
Comment 10 Matthias Andree freebsd_committer 2016-10-05 07:07:53 UTC
The issue "Berkeley DB library configured to support only private environments" when attempting multi-process access to the database affects all architectures that need to use POSIX mutexes on FreeBSD because all FreeBSD releases as of today (comment's date) including 11.0-RELEASE do _not_ support pthread_condattr_getpshared(..., PTHREAD_PROCESS_SHARED).
Comment 11 Matthias Andree freebsd_committer 2016-10-06 18:12:46 UTC
Note that the databases/db6 changes for ARM/gcc-assembly mutexes are not directly portable backwards because:

1. db5 uses a swpb/eor command sequence, and swpb is not reflected in atomic.h and there is no alternative implementation for newer processors that do not support it. 

2. Also note that copying the newer implementation back to db5 would bring the Affero GPL v3 into db5 - in that situation it will usually be easier to go for db6 directly on ARM.
Comment 12 Vick Khera 2016-10-06 19:31:18 UTC
In bug #213223 databases/db6 was patched to use the mutex from the machine/atomic.h header.

Given that, should netatalk3 be marked "broken" for DB < 6 or can this same patch to use atomic.h work with db5?

I think ideally if db5 can be patched, then the official packages can have a working netatalk3 since they are built against db5 by default.
Comment 13 Matthias Andree freebsd_committer 2016-10-27 21:16:35 UTC
It appears that https://svnweb.freebsd.org/changeset/base/296162 (OSVERSION  > 1100100 according to jbeich@'s research) might make PTHREAD_PROCESS_SHARED workable for ARM; testing on FreeBSD 11.0-RELEASE required though.
Comment 14 Matthias Andree freebsd_committer 2016-10-28 18:45:29 UTC
Vick,

could you do the following on FreeBSD 11.0-RELEASE on ARMv6:

# update ports tree (your usual means)
cd /usr/ports/databases/db5
make CONFIGURE_ARGS=--with-mutex=POSIX/pthreads
cd `make -V WRKSRC`
make test_mutex
./test_mutex

The expected PASS result is a long line of checks, similar to this:

test_mutex: 5 processes, 4 threads/process, 10000 lock requests from 20 locks
test_mutex: backing data 984 bytes
Locker: ID 008 (PID: 72426; TID: 801616500)
Locker: ID 016 (PID: 72428; TID: 801616500)
[...]
013: 9000
018: 9000
007: 9000
017: 9000
008: 9000
[...]
Per-lock mutex statistics.
mutex  0: wait: 5350; no wait 4495
mutex  1: wait: 5350; no wait 4560
[...]
mutex 19: wait: 5626; no wait 4467
test_mutex: test succeeded


A test FAILURE looks like this:

test_mutex: 5 processes, 4 threads/process, 10000 lock requests from 20 locks
test_mutex: backing data 984 bytes
test_mutex: BDB1577 Berkeley DB library configured to support only private environments
test_mutex: environment open: TESTDIR: Invalid argument


If things go well, please 1. run "make clean" and 2. repeat the entire test procedure, omitting the CONFIGURE_ARGS=... from the make command.
Comment 15 Matthias Andree freebsd_committer 2016-10-28 18:46:45 UTC
before "make clean" in the final step, you'll need to cd back to /usr/ports/databases/db5
Comment 16 mikael.urankar 2016-10-29 14:10:43 UTC
I've done the tests on my beaglebone running 12-CURRENT:
All tests pass with CONFIGURE_ARGS=--with-mutex=POSIX/pthreads and the tests fail without CONFIGURE_ARGS
Thanks!
Comment 17 commit-hook freebsd_committer 2016-11-04 00:10:15 UTC
A commit references this bug:

Author: mandree
Date: Fri Nov  4 00:09:52 UTC 2016
New revision: 425280
URL: https://svnweb.freebsd.org/changeset/ports/425280

Log:
  Force pthreads mutexes on 11+ and aarch64/ARMV6

  This currently leaves ARMv6 and aarch64 on FreeBSD 9.x and 10.x out in
  the rain because these will still autoconfigure to only permit private
  mutexes.

  PR:		213167
  Submitted by:	jbeich (IRC)

Changes:
  head/databases/db5/Makefile
Comment 18 Matthias Andree freebsd_committer 2016-11-04 00:17:00 UTC
(In reply to mikael.urankar from comment #16)
Mikael, thanks for your tests.  

I'll leave this bug report open for now since we don't yet have a solution for db5 on FreeBSD 9 or 10 on ARM v6 or newer/AARCH64 - if we find one, we'll consider it, otherwise this PR will be closed when 10.3 goes out of support in May 2018 as "overcome by events".
Comment 19 Matthias Andree freebsd_committer 2019-01-07 00:22:09 UTC
(In reply to Matthias Andree from comment #18)
Since the affected FreeBSD versions have gone out of support, we can close this report.