168317 – [libthr] [patch] posix_mutex_trylock does not work with a PTHREAD_MUTEX_ADAPTIVE_NP thread

no flags

I discovered this running glib-2.32.3 testsuite.

If I feed pthread_mutex_trylock with a PTHREAD_MUTEX_ADAPTIVE_NP thread, it returns EINVAL instead of EBUSY if it fails to lock.

Reduced testcase:

# cat mutex.c 
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <pthread.h>

int main() {
        pthread_mutex_t mutex;
        pthread_mutexattr_t attr;
        int status;

        pthread_mutexattr_init (&attr);
        pthread_mutexattr_settype (&attr, PTHREAD_MUTEX_ADAPTIVE_NP);
        status = pthread_mutex_init (&mutex, &attr);
        printf("status = %i\n", status);

        status = pthread_mutex_trylock(&mutex);
        printf("status = %i\n", status);

        status = pthread_mutex_trylock(&mutex);
        printf("status = %i\n", status);

        printf("%s\n", strerror(status));
}

# gcc -pthread mutex.c -o mutex
# ./mutex 
status = 0
status = 0
status = 22
Invalid argument

However, with a PTHREAD_MUTEX_NORMAL thread it is fine:

# cat mutex2.c 
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <pthread.h>

int main() {
        pthread_mutex_t mutex;
        pthread_mutexattr_t attr;
        int status;

        pthread_mutexattr_init (&attr);
        pthread_mutexattr_settype (&attr, PTHREAD_MUTEX_NORMAL);
        status = pthread_mutex_init (&mutex, &attr);
        printf("status = %i\n", status);

        status = pthread_mutex_trylock(&mutex);
        printf("status = %i\n", status);

        status = pthread_mutex_trylock(&mutex);
        printf("status = %i\n", status);

        printf("%s\n", strerror(status));
}

# gcc -pthread mutex2.c -o mutex2
# ./mutex2 
status = 0
status = 0
status = 16
Device busy

Some investigation showed that svn rev 173154 got that right but then svn rev 173173 removed this.

Fix: I'm attaching a patch against head that sould fix this.

Patch attached with submission follows:
How-To-Repeat: Try the above testcases, or run glib-2.32.3 testsuite, or watch bailing out any glib based program using trylock.