Bug 124860 - flockfile(3) doesn't work when the memory has been exhausted.
Summary: flockfile(3) doesn't work when the memory has been exhausted.
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: standards (show other bugs)
Version: 6.3-RELEASE
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-standards (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-06-22 00:00 UTC by Tomohisa Tanaka
Modified: 2018-01-03 05:13 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tomohisa Tanaka 2008-06-22 00:00:13 UTC
	When the flockfile(3) acquires a lock, and the specified
	stream has never locked, and the memory has been exhausted,
	flockfile(3) returns without the stream locked.

	The flockfile(3) calls _pthread_mutex_lock() according to the
	file /usr/src/lib/libc/stdio/_flock_stub.c, as follows.

/usr/src/lib/libc/stdio/_flock_stub.c:
     69 #define _lock _extra
     ...
     83                 _pthread_mutex_lock(&fp->_lock->fl_mutex);

        The _pthread_mutex_lock() does never fail if
        fp->_extra->fl_mutex has already been dynamically initialized.
        However, if fp->_extra->fl_mutex has been statically
        initialized, the _pthread_mutex_lock() tries to perform the
        dynamic initialization of it by calling the
        pthread_mutex_init(3), as follows.

/usr/src/lib/libpthread/thread/thr_mutex.c:
    302 static int
    303 init_static_private(struct pthread *thread, pthread_mutex_t *mutex)
    304 {
        ...
    310                 ret = pthread_mutex_init(mutex, &static_mattr);
        ...
    317 }
        ...
    849 int
    850 _pthread_mutex_lock(pthread_mutex_t *m)
    851 {
        ...
    866         else if ((*m != NULL) ||
    867             ((ret = init_static_private(curthread, m)) == 0))
    868                 ret = mutex_lock_common(curthread, m, NULL);
        ...
    871 }

    	When the memory has been exhausted, the pthread_mutex_init(3)
    	returns ENOMEM.  All mutexes of the stdio stream are
    	statically initialized, according to the file
    	/usr/src/lib/libc/stdio/findfp.c.

Fix: 

I think that the function __sfp() of
	/usr/src/lib/libc/stdio/findfp.c must perform the dynamic
	initialization of fp->_extra->fl_mutex, if it has been only
	statically initialized.  And if it failed, __sfp() must return
	NULL.
How-To-Repeat: % cat test.c
#include <stdio.h>
#include <stdlib.h>
#include <err.h>
#include <errno.h>
#include <pthread.h>

static pthread_barrier_t barrier;
static FILE *fp;

static void *
run(void *arg)
{
  pthread_barrier_wait(&barrier);
  flockfile(fp); /* [1] */
  return NULL;
}

int
main(void)
{
  pthread_t thread;
  pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
  void *p;
  size_t k;

  if (pthread_barrier_init(&barrier, NULL, 2) != 0)
    err(1, "pthread_barrier_init");
  if (pthread_create(&thread, NULL, run, NULL) != 0)
    err(1, "pthread_create");
  if ((fp = fopen("log", "w")) == NULL)
    err(1, "log");

#if 1
  for (k = 1; k < 64; ++k) {
    while ((p = malloc(k)) != NULL)
      ;
  }
  /*
   * Now, we are in a special situation where the memory has been
   * extremely exhausted.  Here, 'mutex' has not initialized with
   * pthread_mutex_init(3), so pthread_mutex_lock(3) as follows will
   * try to initialize "mutex" using pthread_mutex_init(3) internally.
   * However, malloc(3) within the pthread_mutex_init(3) returns NULL,
   * and then pthread_mutex_lock(3) fails and returns ENOMEM.
   */
  if ((errno = pthread_mutex_lock(&mutex)) != 0) {
    warn("pthread_mutex_lock");
  }
#endif

  flockfile(fp); /* [2] */
  /*
   * Here, this main thread does NOT have the lock of "fp", because
   * pthread_mutex_lock(3) within flockfile(3) at [2] has failed.  We
   * must remember "fp->_extra->fl_mutex" has not been initialized by
   * pthread_mutex_init(3) as well as "mutex".
   */
  pthread_barrier_wait(&barrier);
  /*
   * Another thread will call flockfile(3) at [1], but it returns
   * immediately.  No thread can get the lock of 'fp' in this
   * situation.  If we comment out from "#if 1" to "#endif", a
   * deadlock occurs.
   */
  pthread_join(thread, NULL);
  return 0;
}

% cat test.sh
#!/bin/sh
ulimit -v 10000
./a.out

% gcc -pthread test.c 
% sh test.sh  
a.out: pthread_mutex_lock: Cannot allocate memory 
%
Comment 1 Eitan Adler freebsd_committer freebsd_triage 2017-12-31 08:00:25 UTC
For bugs matching the following criteria:

Status: In Progress Changed: (is less than) 2014-06-01

Reset to default assignee and clear in-progress tags.

Mail being skipped