Created attachment 152624 [details] leaked fd test program from LLDB test suite Version: python27-2.7.9 LLDB recently added a test that the debugger is not leaking file descriptors into child processes. This fails on FreeBSD, and the leaked fd comes from Python's random. I've attached the test program from LLDB. To reproduce: - Build test program and confirm it returns 0: % clang main.c && ./a.out % echo $? 0 - Invoke it from python without using random and confirm it returns 0: >>> import os >>> os.system("./a.out") 0 - Invoke it from python with random: >>> import os, random >>> os.system("./a.out") File descriptor 4 is open. 512
Auto-assigned to maintainer python@FreeBSD.org
Ed, Can you reproduce this in python33, python34 and against python upstream branch default? Please also provide uname -a output as attachment
It is fixed in python 3.4.2 - see test result below. Looks like the fix[1] is not in 2.x though. The fix is PEP 446 - issue 18571[2] [1] https://github.com/python/cpython/commit/621f57c094e7572bb4aa2734fe7264856921fc27 [2] http://bugs.python.org/issue18571 LLDB people do not see this issue on Linux; I do not yet know if they're using Python 3 or if there's some other difference. Also os.urandom() is sufficient to demonstrate the problem: Python 2.7.9 (default, Jan 8 2015, 21:47:19) [GCC 4.2.1 Compatible FreeBSD Clang 3.3 (tags/RELEASE_33/final 183502)] on freebsd10 Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.urandom(1) '8' >>> os.system("./a.out") File descriptor 3 is open. 512 >>> feynman% python3 Python 3.4.2 (default, Jan 28 2015, 05:38:04) [GCC 4.2.1 Compatible FreeBSD Clang 3.3 (tags/RELEASE_33/final 183502)] on freebsd10 Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.urandom(1) b'\x00' >>> os.system("./a.out") 0
Victor Stinner (haypo) @ Python has graciously created an issue with patch upstream to take care of this. Once merged I'll backport it to 2.7.9 (and other versions if applicable)
Python issue is http://bugs.python.org/issue17070
That was the issue for the original close-on-exec change I meant to link to the change for Python 2.7, which is: http://bugs.python.org/issue23458
Victor notes that the upstream issue has now been resolved/closed. Ed, can we resolve this issue, or would you like to request a backport pending the next release. ETA is unknown for 2.7.10 at this point.
The LLDB test now tests the Python version XFAILs for Python >= 2.7.8. http://llvm.org/viewvc/llvm-project?view=revision&revision=230215 Since it's handled in upstream Python I think there's no need to do anything in FreeBSD but wait for 2.7.10 to come out. Note that there is still a small concern wrt threads and setting the close-on-exec flag, which may impact LLDB. But that is not a FreeBSD-specific issue and is being discussed on the Python ticket.
Hey Ed, I'm seeing this fixed on Python 2.7.10, at least on OS X. I'm updating the expectedFailure function to check for versions >= 2.7.8 and < 2.7.10.
My diff for this currently looks like this: diff --git a/test/functionalities/avoids-fd-leak/TestFdLeak.py b/test/functionalities/avoids-fd-leak/TestFdLeak.py index 415530b..a317354 100644 --- a/test/functionalities/avoids-fd-leak/TestFdLeak.py +++ b/test/functionalities/avoids-fd-leak/TestFdLeak.py @@ -8,17 +8,24 @@ import lldb from lldbtest import * import lldbutil + +def python_leaky_fd_version(test): + import sys + # python random leaks fd on some versions. + return sys.version_info >= (2, 7, 8) and sys.version_info < (2, 7, 10) + + class AvoidsFdLeakTestCase(TestBase): mydir = TestBase.compute_mydir(__file__) - @expectedFailure(lambda x: sys.version_info >= (2, 7, 8), "bugs.freebsd.org/197376") # python random leaks fd + @expectedFailure(python_leaky_fd_version, "bugs.freebsd.org/197376") @skipIfWindows # The check for descriptor leakage needs to be implemented differently here. @skipIfTargetAndroid() # Android have some other file descriptors open by the shell def test_fd_leak_basic (self): self.do_test([]) - @expectedFailure(lambda x: sys.version_info >= (2, 7, 8), "bugs.freebsd.org/197376") # python random leaks fd + @expectedFailure(python_leaky_fd_version, "bugs.freebsd.org/197376") @skipIfWindows # The check for descriptor leakage needs to be implemented differently here. @skipIfTargetAndroid() # Android have some other file descriptors open by the shell def test_fd_leak_log (self): @@ -40,7 +47,7 @@ class AvoidsFdLeakTestCase(TestBase): self.assertTrue(process.GetExitStatus() == 0, "Process returned non-zero status. Were incorrect file descriptors passed?") - @expectedFailure(lambda x: sys.version_info >= (2, 7, 8), "bugs.freebsd.org/197376") # python random leaks fd + @expectedFailure(python_leaky_fd_version, "bugs.freebsd.org/197376") @expectedFlakeyLinux @skipIfWindows # The check for descriptor leakage needs to be implemented differently here. @skipIfTargetAndroid() # Android have some other file descriptors open by the shell
Thanks Todd, that seems correct to me on a quick look.
That change went in (with a few others) here: The fix for this went in here (with a couple more XPASS cleanups): Sending test/expression_command/radar_9673664/TestExprHelpExamples.py Sending test/functionalities/avoids-fd-leak/TestFdLeak.py Sending test/lang/c/const_variables/TestConstVariables.py Transmitting file data ... Committed revision 247591. I verified on OS X and Ubuntu 14.04. OS X (latest beta for El Capitan) has Python 2.7.10 where the issue is resolved and it properly does not expect failure there. Ubuntu 14.04.3 has Python 2.7.6, where we also don't expect it to fail, and that is working as well.
The LLDB test is failing for me now, with Python version: '2.7.10 (default, Aug 27 2015, 01:18:07) \n[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)]' However the simplified reproduction case does not demonstrate the problem: Python 2.7.10 (default, Aug 27 2015, 01:18:07) [GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10 Type "help", "copyright", "credits" or "license" for more information. >>> import os,random >>> os.system("./a.out") 0 so it appears there's another, new leak problem on FreeBSD.
Drats. Well, we can make it a slightly more complex predicate. Perhaps keep the current predicate portion if it is not FreeBSD, and if it is FreeBSD, keep it on 2.7.8+ (with no upper limit)?
llvm cleared this in https://reviews.llvm.org/D90757 (removed as LLDB no longer supports python < 3)