Bug 222112

Summary: lang/python36 selectors.select() does not block on named pipes / mkfifo
Product: Ports & Packages Reporter: Jeff Kletsky <jeff+freebsd>
Component: Individual Port(s)Assignee: freebsd-python (Nobody) <python>
Status: Closed Works As Intended    
Severity: Affects Some People Flags: bugzilla: maintainer-feedback? (python)
Priority: ---    
Version: Latest   
Hardware: Any   
OS: Any   

Description Jeff Kletsky 2017-09-07 01:18:19 UTC 
An instance of selectors.select() blocks as expected until data is ready on a named pipe or a regular file. On FreeBSD 11.1-RELEASE-p1 it does not block after that data has been read from a named pipe (but does continue to block with a regular file). 

This is both unexpected and inconsistent with at least behavior under Mac OS X, which continues to block as expected with both a named pipe and a regular file.

Observed behavior:
==================
* my_selectors.select() blocks as expected until data is written to the pipe
* the data is read from the pipe
* my_selectors.select() no longer blocks

Expected behavior:
==================
* my_selectors.select() would block until there was more data written to the pipe


Same behavior is seen with the "default" selectors.KqueueSelector as well as with explicitly using a selectors.PollSelector (Mac OS X default).


Impact:
=======
* No clear way to use select from Python to "read when ready" from a named pipe
* Programs that rely on select may have very unexpected behavior


Workarounds:
============
(none at this time)


Environment:
============
FreeBSD 11.1-RELEASE-p1
python3-3_3
python36-3.6.1_4

jailed or host system, same behavior
venv or not, same behavior


To replicate:
=============
(warning, this will spew to the terminal until interrupted if commands-in is a named pipe)

$ rm commands-in
$ mkfifo commands-in
(run the program, below)
$ echo "Something" >> commands-in   # ">" also fails

using 'touch' to obtain a regular file will demonstrate the regular-file behavior

key_event_list is retained for visibility in debugger
timeout=None is the default, made explicit to confirm not the issue
"If timeout is None, the call will block until a monitored file object becomes ready.​"

See also https://forums.freebsd.org/threads/62377/

Example Python 3 code follows:

import logging
import selectors


def test():

    command_pipe = 'commands-in'

    logging.basicConfig(level=logging.INFO)
    logger = logging.getLogger()

    logger.info(f"Opening command pipe: '{command_pipe}'")
    with open(command_pipe, 'r') as cp:
        select_cp = selectors.DefaultSelector()
        select_cp.register(cp, selectors.EVENT_READ)
        while True:
            key_event_list = select_cp.select(timeout=None)
            line = cp.readline()
            logger.info(f"Read: '{line}'")


if __name__ == '__main__':
Comment 1 Jeff Kletsky 2017-09-07 20:40:24 UTC 
Based on tobic@ comment, this does not appear to be a bug, but an outcome of how FIFOs are implemented.

https://forums.freebsd.org/threads/62377/#post-360237

Looking at this on an Ubuntu system and a similar Python version shows similar behavior to that seen under FreeBSD. It appears that Mac OS X is "different" in how it is implemented.

Closing as "works as intended"
Comment 2 Jeff Kletsky 2017-09-07 20:41:16 UTC 
@tobic code, for anyone that finds this through search:

After opening the FIFO for reading, open it again for writing to prevent your program from ever seeing EOF at all (i.e. pretend there is always at least 1 writer). There is no need to use select() here; readline() will already block and wait for new data.

Code:
import logging

def test():
    command_pipe = 'commands-in'

    logging.basicConfig(level=logging.INFO)
    logger = logging.getLogger()

    logger.info(f"Opening command pipe: '{command_pipe}'")
    with open(command_pipe, "r") as cp, open(command_pipe, "w"):
            while True:
                line = cp.readline()
                logger.info(f"Read: '{line}'")


if __name__ == '__main__':
    test()