Bug 197376

Comment 1 Bugzilla Automation 2015-02-06 16:35:46 UTC

Auto-assigned to maintainer python@FreeBSD.org

Comment 2 Kubilay Kocak 2015-02-11 08:25:31 UTC

Ed, 

Can you reproduce this in python33, python34 and against python upstream branch default?

Please also provide uname -a output as attachment

Comment 3 Ed Maste 2015-02-13 04:13:14 UTC

It is fixed in python 3.4.2 - see test result below. Looks like the fix[1] is not in 2.x though. The fix is PEP 446 - issue 18571[2]

[1] https://github.com/python/cpython/commit/621f57c094e7572bb4aa2734fe7264856921fc27
[2] http://bugs.python.org/issue18571

LLDB people do not see this issue on Linux; I do not yet know if they're using Python 3 or if there's some other difference.

Also os.urandom() is sufficient to demonstrate the problem:

Python 2.7.9 (default, Jan  8 2015, 21:47:19) 
[GCC 4.2.1 Compatible FreeBSD Clang 3.3 (tags/RELEASE_33/final 183502)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.urandom(1)
'8'
>>> os.system("./a.out")
File descriptor 3 is open.
512
>>> 
feynman% python3
Python 3.4.2 (default, Jan 28 2015, 05:38:04) 
[GCC 4.2.1 Compatible FreeBSD Clang 3.3 (tags/RELEASE_33/final 183502)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.urandom(1)
b'\x00'
>>> os.system("./a.out")
0

Comment 4 Kubilay Kocak 2015-02-13 09:24:35 UTC

Victor Stinner (haypo) @ Python has graciously created an issue with patch upstream to take care of this.

Once merged I'll backport it to 2.7.9 (and other versions if applicable)

Comment 5 Ed Maste 2015-02-18 17:27:13 UTC

Python issue is http://bugs.python.org/issue17070

Comment 6 Ed Maste 2015-02-18 17:40:18 UTC

That was the issue for the original close-on-exec change

I meant to link to the change for Python 2.7, which is: http://bugs.python.org/issue23458

Comment 7 Kubilay Kocak 2015-02-24 13:37:21 UTC

Victor notes that the upstream issue has now been resolved/closed.

Ed, can we resolve this issue, or would you like to request a backport pending the next release. ETA is unknown for 2.7.10 at this point.

Comment 8 Ed Maste 2015-02-24 13:54:06 UTC

The LLDB test now tests the Python version XFAILs for Python >= 2.7.8. 
http://llvm.org/viewvc/llvm-project?view=revision&revision=230215

Since it's handled in upstream Python I think there's no need to do anything in FreeBSD but wait for 2.7.10 to come out. Note that there is still a small concern wrt threads and setting the close-on-exec flag, which may impact LLDB. But that is not a FreeBSD-specific issue and is being discussed on the Python ticket.

Hey Ed,

I'm seeing this fixed on Python 2.7.10, at least on OS X.  I'm updating the expectedFailure function to check for versions >= 2.7.8 and < 2.7.10.

My diff for this currently looks like this:

diff --git a/test/functionalities/avoids-fd-leak/TestFdLeak.py b/test/functionalities/avoids-fd-leak/TestFdLeak.py
index 415530b..a317354 100644
--- a/test/functionalities/avoids-fd-leak/TestFdLeak.py
+++ b/test/functionalities/avoids-fd-leak/TestFdLeak.py
@@ -8,17 +8,24 @@ import lldb
 from lldbtest import *
 import lldbutil

+
+def python_leaky_fd_version(test):
+    import sys
+    # python random leaks fd on some versions.
+    return sys.version_info >= (2, 7, 8) and sys.version_info < (2, 7, 10)
+
+
 class AvoidsFdLeakTestCase(TestBase):

     mydir = TestBase.compute_mydir(__file__)

-    @expectedFailure(lambda x: sys.version_info >= (2, 7, 8), "bugs.freebsd.org/197376") # python random leaks fd
+    @expectedFailure(python_leaky_fd_version, "bugs.freebsd.org/197376")
     @skipIfWindows # The check for descriptor leakage needs to be implemented differently here.
     @skipIfTargetAndroid() # Android have some other file descriptors open by the shell
     def test_fd_leak_basic (self):
         self.do_test([])

-    @expectedFailure(lambda x: sys.version_info >= (2, 7, 8), "bugs.freebsd.org/197376") # python random leaks fd
+    @expectedFailure(python_leaky_fd_version, "bugs.freebsd.org/197376")
     @skipIfWindows # The check for descriptor leakage needs to be implemented differently here.
     @skipIfTargetAndroid() # Android have some other file descriptors open by the shell
     def test_fd_leak_log (self):
@@ -40,7 +47,7 @@ class AvoidsFdLeakTestCase(TestBase):
         self.assertTrue(process.GetExitStatus() == 0,
                 "Process returned non-zero status. Were incorrect file descriptors passed?")

-    @expectedFailure(lambda x: sys.version_info >= (2, 7, 8), "bugs.freebsd.org/197376") # python random leaks fd
+    @expectedFailure(python_leaky_fd_version, "bugs.freebsd.org/197376")
     @expectedFlakeyLinux
     @skipIfWindows # The check for descriptor leakage needs to be implemented differently here.
     @skipIfTargetAndroid() # Android have some other file descriptors open by the shell

Comment 11 Ed Maste 2015-09-14 18:28:49 UTC

Thanks Todd, that seems correct to me on a quick look.

That change went in (with a few others) here:

The fix for this went in here (with a couple more XPASS cleanups):

Sending        test/expression_command/radar_9673664/TestExprHelpExamples.py
Sending        test/functionalities/avoids-fd-leak/TestFdLeak.py
Sending        test/lang/c/const_variables/TestConstVariables.py
Transmitting file data ...
Committed revision 247591.

I verified on OS X and Ubuntu 14.04.  OS X (latest beta for El Capitan) has Python 2.7.10 where the issue is resolved and it properly does not expect failure there.  Ubuntu 14.04.3 has Python 2.7.6, where we also don't expect it to fail, and that is working as well.

Comment 13 Ed Maste 2015-09-14 20:11:21 UTC

The LLDB test is failing for me now, with Python version:
'2.7.10 (default, Aug 27 2015, 01:18:07) \n[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)]'

However the simplified reproduction case does not demonstrate the problem:

Python 2.7.10 (default, Aug 27 2015, 01:18:07) 
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import os,random
>>> os.system("./a.out")
0

so it appears there's another, new leak problem on FreeBSD.

Drats.

Well, we can make it a slightly more complex predicate.  Perhaps keep the current predicate portion if it is not FreeBSD, and if it is FreeBSD, keep it on 2.7.8+ (with no upper limit)?

Comment 15 Ed Maste 2020-11-04 17:55:35 UTC

llvm cleared this in https://reviews.llvm.org/D90757 (removed as LLDB no longer supports python < 3)