Created attachment 212019 [details] Sample program reproducing the issue Hi there, ports r519246 switched LMDB from MDB_USE_POSIX_SEM to MDB_USE_POSIX_MUTEX. Unfortunately, it seems that there are some edge cases in which LMDB does not play nicely with FreeBSD's process-shared mutexes. The particular problem I observed is that when a single process reopens an LMDB environment (that is, an environment is opened, closed, then opened again by the same process), then other processes trying to access the reopened environment fail to grab the read table lock - pthread_mutex_lock() returns EINVAL (22). AFAICT, this happens because libthr is unable to find the shared memory segment with the relevant part of the LMDB lockfile mmap()'d. I attached the simplest test case I could come up with. To reproduce the problem, first compile lmdb-mutex.c: cc -I/usr/local/include -L/usr/local/lib -llmdb lmdb-mutex.c -o lmdb-mutex Then, run the first instance of the program. It should start fine and sleep for 30 seconds. Before the first instance exits, start a second instance of the program. It should fail with: Assertion failed: (mdb_txn_begin(env, 0, MDB_RDONLY, &txn) == MDB_SUCCESS), function main, file lmdb-mutex.c, line 14. The sample program works fine in the above scenario if LMDB is compiled with MDB_USE_POSIX_SEM. It also works fine on other operating systems using MDB_USE_POSIX_MUTEX. One workaround I could come up with is enabling ASLR - it causes mmap() to return different addresses for the LMDB lockfile mapping upon each call to mdb_env_open(), causing libthr to use different off-pages for the read table mutexes for the "old" and "new" environment (IIUC). Note that LMDB never calls pthread_mutex_destroy() for the read table lock when an environment is closed which I believe prevents the shared memory segment for the "old" environment from being released. But please do not take my word for it, I do not understand sys/kern/kern_umtx.c, libthr, and LMDB internals well enough to fully explain what is happening (though I sure would like to find out!) To see an example occurrence of this problem in the wild, install BIND (e.g. dns/bind911), put the following into named.conf: options { allow-new-zones yes; }; and then run "named -g -c named.conf". After it starts up, run "named-nzd2nzf _default.nzd". It will fail with: named-nzd2nzf: mdb_txn_begin: Invalid argument My humble suggestion would be to revert the LMDB port back to MDB_USE_POSIX_SEM for the time being, unless someone can immediately see what the problem is and is able to fix it. Hope this helps, please let me know if I can be of any further assistance with this issue.
A commit references this bug: Author: delphij Date: Tue Jun 16 21:51:56 UTC 2020 New revision: 539379 URL: https://svnweb.freebsd.org/changeset/ports/539379 Log: databases/lmdb: in db_env_close0(), destroy robust mutexes if we are the only remaining user. When closing an lmdb database, all memory and file descriptor resources are released, including the shared memory pages that contained the robust mutex. However, before this commit, prior to unmapping the pages that contained the robust mutexex, lmdb did not destroy the mutexes first. This would create a problem when an application opens and closes a database, then open it again. According to libthr(3), by default, a shared lock backed by a mapped file in memory is automatically destroyed on the last unmap of the corresponding file' page, which is allowed by POSIX. After unmapping the shared pages, the kernel writes off all active robust mutexes associated with these pages. However, the userland threading library still keeps the record (pshared_lookup in thr_pshared.c of libthr) for these objects as they are not really destroyed before, so that it don't have to ask the kernel every time when looking them up. Now, a later re-open of the database might have mapped the lock file to the same memory location. Because the threading library have remembered the robust mutex object, it would just reuse it even though it was already invalid from kernel's point of view. Unfortunately, regular lock operations would still work for this process. Should another lmdb process opens the same database, it would attempt to obtain the robust mutex (no longer recognized by kernel) because it would see another process holding a file lock, but that would fail because the robust mutex is invalid for the kernel. Explicitly destroy the mutex if we are the last remaining user to ensure the mutex is always in a known defined state. OpenLDAP ITS #9278 With debugging help from: kib PR: 244493 MFH: 2020Q2 Changes: head/databases/lmdb/Makefile head/databases/lmdb/files/patch-mdb.c
A commit references this bug: Author: delphij Date: Tue Jun 16 21:54:59 UTC 2020 New revision: 539380 URL: https://svnweb.freebsd.org/changeset/ports/539380 Log: MFH: r539379 databases/lmdb: in db_env_close0(), destroy robust mutexes if we are the only remaining user. When closing an lmdb database, all memory and file descriptor resources are released, including the shared memory pages that contained the robust mutex. However, before this commit, prior to unmapping the pages that contained the robust mutexex, lmdb did not destroy the mutexes first. This would create a problem when an application opens and closes a database, then open it again. According to libthr(3), by default, a shared lock backed by a mapped file in memory is automatically destroyed on the last unmap of the corresponding file' page, which is allowed by POSIX. After unmapping the shared pages, the kernel writes off all active robust mutexes associated with these pages. However, the userland threading library still keeps the record (pshared_lookup in thr_pshared.c of libthr) for these objects as they are not really destroyed before, so that it don't have to ask the kernel every time when looking them up. Now, a later re-open of the database might have mapped the lock file to the same memory location. Because the threading library have remembered the robust mutex object, it would just reuse it even though it was already invalid from kernel's point of view. Unfortunately, regular lock operations would still work for this process. Should another lmdb process opens the same database, it would attempt to obtain the robust mutex (no longer recognized by kernel) because it would see another process holding a file lock, but that would fail because the robust mutex is invalid for the kernel. Explicitly destroy the mutex if we are the last remaining user to ensure the mutex is always in a known defined state. OpenLDAP ITS #9278 With debugging help from: kib PR: 244493 Approved by: ports-secteam Changes: _U branches/2020Q2/ branches/2020Q2/databases/lmdb/Makefile branches/2020Q2/databases/lmdb/files/patch-mdb.c