Bug 241801 - lang/python38: regression: hang in locking in multiprocessing.Pool
Summary: lang/python38: regression: hang in locking in multiprocessing.Pool
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-python mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-11-08 12:46 UTC by Dmitry Marakasov
Modified: 2019-11-08 23:04 UTC (History)
1 user (show)

See Also:
bugzilla: maintainer-feedback? (python)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dmitry Marakasov freebsd_committer 2019-11-08 12:46:54 UTC
This simple program

    from multiprocessing import Pool
    from time import sleep

    Pool().map(sleep, [0.01] * 10)

works fine with python 3.7, but is likely (about 20-50% probability on my 4 core box) to hang with python 3.8. Example backtraces after interruption:

% python3.8 1.py
^CException ignored in: <Finalize object, dead>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/multiprocessing/util.py", line 201, in __call__
Process ForkPoolWorker-2:
Process ForkPoolWorker-4:
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 689, in _terminate_pool
    cls._help_stuff_finish(inqueue, task_handler, len(pool))
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 674, in _help_stuff_finish
    inqueue._rlock.acquire()
KeyboardInterrupt: 
Process ForkPoolWorker-3:
Process ForkPoolWorker-1:

% python3.8 1.py
^CException ignored in: <Finalize object, dead>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/multiprocessing/util.py", line 201, in __call__
Process ForkPoolWorker-3:
Process ForkPoolWorker-4:
Process ForkPoolWorker-1:
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 689, in _terminate_pool
    cls._help_stuff_finish(inqueue, task_handler, len(pool))
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 674, in _help_stuff_finish
    inqueue._rlock.acquire()
KeyboardInterrupt: 
Process ForkPoolWorker-2:

% python3.8 1.py
^CException ignored in: <Finalize object, dead>
Process ForkPoolWorker-2:
Process ForkPoolWorker-3:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/multiprocessing/util.py", line 201, in __call__
Process ForkPoolWorker-1:
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 689, in _terminate_pool
    cls._help_stuff_finish(inqueue, task_handler, len(pool))
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 674, in _help_stuff_finish
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/local/lib/python3.8/multiprocessing/queues.py", line 355, in get
    with self._rlock:
  File "/usr/local/lib/python3.8/multiprocessing/queues.py", line 355, in get
    with self._rlock:
  File "/usr/local/lib/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/local/lib/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
KeyboardInterrupt
    inqueue._rlock.acquire()
KeyboardInterrupt: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/local/lib/python3.8/multiprocessing/queues.py", line 356, in get
    res = self._reader.recv_bytes()
  File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/lib/python3.8/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
Process ForkPoolWorker-4:


Should I submit this upstream?
Comment 1 Dmitry Marakasov freebsd_committer 2019-11-08 15:01:29 UTC
Created upstream issue: https://bugs.python.org/issue38744
Comment 2 Wen Heping freebsd_committer 2019-11-08 23:04:03 UTC
(In reply to Dmitry Marakasov from comment #1)
Nice work !