Bug 197376 - lang/python27: random module leaks file descriptor into child process
Summary: lang/python27: random module leaks file descriptor into child process
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Kubilay Kocak
URL: http://bugs.python.org/issue23458
Keywords: needs-patch
Depends on:
Blocks:
 
Reported: 2015-02-06 16:35 UTC by Ed Maste
Modified: 2020-11-04 17:55 UTC (History)
2 users (show)

See Also:
koobs: maintainer-feedback+


Attachments
leaked fd test program from LLDB test suite (581 bytes, text/x-csrc)
2015-02-06 16:35 UTC, Ed Maste
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ed Maste freebsd_committer 2015-02-06 16:35:46 UTC
Created attachment 152624 [details]
leaked fd test program from LLDB test suite

Version: python27-2.7.9

LLDB recently added a test that the debugger is not leaking file descriptors into child processes. This fails on FreeBSD, and the leaked fd comes from Python's random. I've attached the test program from LLDB.

To reproduce:

- Build test program and confirm it returns 0:
% clang main.c && ./a.out
% echo $?
0

- Invoke it from python without using random and confirm it returns 0:
>>> import os
>>> os.system("./a.out")
0

- Invoke it from python with random:

>>> import os, random
>>> os.system("./a.out")
File descriptor 4 is open.
512
Comment 1 Bugzilla Automation freebsd_committer 2015-02-06 16:35:46 UTC
Auto-assigned to maintainer python@FreeBSD.org
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2015-02-11 08:25:31 UTC
Ed, 

Can you reproduce this in python33, python34 and against python upstream branch default?

Please also provide uname -a output as attachment
Comment 3 Ed Maste freebsd_committer 2015-02-13 04:13:14 UTC
It is fixed in python 3.4.2 - see test result below. Looks like the fix[1] is not in 2.x though. The fix is PEP 446 - issue 18571[2]

[1] https://github.com/python/cpython/commit/621f57c094e7572bb4aa2734fe7264856921fc27
[2] http://bugs.python.org/issue18571

LLDB people do not see this issue on Linux; I do not yet know if they're using Python 3 or if there's some other difference.

Also os.urandom() is sufficient to demonstrate the problem:

Python 2.7.9 (default, Jan  8 2015, 21:47:19) 
[GCC 4.2.1 Compatible FreeBSD Clang 3.3 (tags/RELEASE_33/final 183502)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.urandom(1)
'8'
>>> os.system("./a.out")
File descriptor 3 is open.
512
>>> 
feynman% python3
Python 3.4.2 (default, Jan 28 2015, 05:38:04) 
[GCC 4.2.1 Compatible FreeBSD Clang 3.3 (tags/RELEASE_33/final 183502)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.urandom(1)
b'\x00'
>>> os.system("./a.out")
0
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2015-02-13 09:24:35 UTC
Victor Stinner (haypo) @ Python has graciously created an issue with patch upstream to take care of this.

Once merged I'll backport it to 2.7.9 (and other versions if applicable)
Comment 5 Ed Maste freebsd_committer 2015-02-18 17:27:13 UTC
Python issue is http://bugs.python.org/issue17070
Comment 6 Ed Maste freebsd_committer 2015-02-18 17:40:18 UTC
That was the issue for the original close-on-exec change

I meant to link to the change for Python 2.7, which is: http://bugs.python.org/issue23458
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2015-02-24 13:37:21 UTC
Victor notes that the upstream issue has now been resolved/closed.

Ed, can we resolve this issue, or would you like to request a backport pending the next release. ETA is unknown for 2.7.10 at this point.
Comment 8 Ed Maste freebsd_committer 2015-02-24 13:54:06 UTC
The LLDB test now tests the Python version XFAILs for Python >= 2.7.8. 
http://llvm.org/viewvc/llvm-project?view=revision&revision=230215

Since it's handled in upstream Python I think there's no need to do anything in FreeBSD but wait for 2.7.10 to come out. Note that there is still a small concern wrt threads and setting the close-on-exec flag, which may impact LLDB. But that is not a FreeBSD-specific issue and is being discussed on the Python ticket.
Comment 9 Todd Fiala 2015-09-14 18:13:31 UTC
Hey Ed,

I'm seeing this fixed on Python 2.7.10, at least on OS X.  I'm updating the expectedFailure function to check for versions >= 2.7.8 and < 2.7.10.
Comment 10 Todd Fiala 2015-09-14 18:14:31 UTC
My diff for this currently looks like this:

diff --git a/test/functionalities/avoids-fd-leak/TestFdLeak.py b/test/functionalities/avoids-fd-leak/TestFdLeak.py
index 415530b..a317354 100644
--- a/test/functionalities/avoids-fd-leak/TestFdLeak.py
+++ b/test/functionalities/avoids-fd-leak/TestFdLeak.py
@@ -8,17 +8,24 @@ import lldb
 from lldbtest import *
 import lldbutil

+
+def python_leaky_fd_version(test):
+    import sys
+    # python random leaks fd on some versions.
+    return sys.version_info >= (2, 7, 8) and sys.version_info < (2, 7, 10)
+
+
 class AvoidsFdLeakTestCase(TestBase):

     mydir = TestBase.compute_mydir(__file__)

-    @expectedFailure(lambda x: sys.version_info >= (2, 7, 8), "bugs.freebsd.org/197376") # python random leaks fd
+    @expectedFailure(python_leaky_fd_version, "bugs.freebsd.org/197376")
     @skipIfWindows # The check for descriptor leakage needs to be implemented differently here.
     @skipIfTargetAndroid() # Android have some other file descriptors open by the shell
     def test_fd_leak_basic (self):
         self.do_test([])

-    @expectedFailure(lambda x: sys.version_info >= (2, 7, 8), "bugs.freebsd.org/197376") # python random leaks fd
+    @expectedFailure(python_leaky_fd_version, "bugs.freebsd.org/197376")
     @skipIfWindows # The check for descriptor leakage needs to be implemented differently here.
     @skipIfTargetAndroid() # Android have some other file descriptors open by the shell
     def test_fd_leak_log (self):
@@ -40,7 +47,7 @@ class AvoidsFdLeakTestCase(TestBase):
         self.assertTrue(process.GetExitStatus() == 0,
                 "Process returned non-zero status. Were incorrect file descriptors passed?")

-    @expectedFailure(lambda x: sys.version_info >= (2, 7, 8), "bugs.freebsd.org/197376") # python random leaks fd
+    @expectedFailure(python_leaky_fd_version, "bugs.freebsd.org/197376")
     @expectedFlakeyLinux
     @skipIfWindows # The check for descriptor leakage needs to be implemented differently here.
     @skipIfTargetAndroid() # Android have some other file descriptors open by the shell
Comment 11 Ed Maste freebsd_committer 2015-09-14 18:28:49 UTC
Thanks Todd, that seems correct to me on a quick look.
Comment 12 Todd Fiala 2015-09-14 18:38:12 UTC
That change went in (with a few others) here:

The fix for this went in here (with a couple more XPASS cleanups):

Sending        test/expression_command/radar_9673664/TestExprHelpExamples.py
Sending        test/functionalities/avoids-fd-leak/TestFdLeak.py
Sending        test/lang/c/const_variables/TestConstVariables.py
Transmitting file data ...
Committed revision 247591.

I verified on OS X and Ubuntu 14.04.  OS X (latest beta for El Capitan) has Python 2.7.10 where the issue is resolved and it properly does not expect failure there.  Ubuntu 14.04.3 has Python 2.7.6, where we also don't expect it to fail, and that is working as well.
Comment 13 Ed Maste freebsd_committer 2015-09-14 20:11:21 UTC
The LLDB test is failing for me now, with Python version:
'2.7.10 (default, Aug 27 2015, 01:18:07) \n[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)]'

However the simplified reproduction case does not demonstrate the problem:

Python 2.7.10 (default, Aug 27 2015, 01:18:07) 
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import os,random
>>> os.system("./a.out")
0

so it appears there's another, new leak problem on FreeBSD.
Comment 14 Todd Fiala 2015-09-14 21:01:12 UTC
Drats.

Well, we can make it a slightly more complex predicate.  Perhaps keep the current predicate portion if it is not FreeBSD, and if it is FreeBSD, keep it on 2.7.8+ (with no upper limit)?
Comment 15 Ed Maste freebsd_committer 2020-11-04 17:55:35 UTC
llvm cleared this in https://reviews.llvm.org/D90757 (removed as LLDB no longer supports python < 3)