Bug 263939 - python3.8 os.sched_getaffinity(0); broken?
Summary: python3.8 os.sched_getaffinity(0); broken?
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-python (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-05-12 11:22 UTC by Bjoern A. Zeeb
Modified: 2022-05-13 11:46 UTC (History)
3 users (show)

See Also:
bugzilla: maintainer-feedback? (python)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Bjoern A. Zeeb freebsd_committer 2022-05-12 11:22:55 UTC
As part of a build with yesterday's src and ports tree.

blas_mkl_info:
customize UnixCCompiler
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "setup.py", line 513, in <module>
    setup_package()
  File "setup.py", line 505, in setup_package
    setup(**metadata)
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/core.py", line 135, in setup
    config = configuration()
  File "setup.py", line 173, in configuration
    config.add_subpackage('numpy')
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/misc_util.py", line 1019, in add_subpackage
    config_list = self.get_subpackage(subpackage_name, subpackage_path,
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/misc_util.py", line 985, in get_subpackage
    config = self._get_configuration_from_setup_py(
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/misc_util.py", line 927, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "numpy/setup.py", line 8, in configuration
    config.add_subpackage('core')
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/misc_util.py", line 1019, in add_subpackage
    config_list = self.get_subpackage(subpackage_name, subpackage_path,
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/misc_util.py", line 985, in get_subpackage
    config = self._get_configuration_from_setup_py(
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/misc_util.py", line 927, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "numpy/core/setup.py", line 757, in configuration
    blas_info = get_info('blas_opt', 0)
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/system_info.py", line 586, in get_info
    return cl().get_info(notfound_action)
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/system_info.py", line 846, in get_info
    self.calc_info()
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/system_info.py", line 1991, in calc_info
    if self._calc_info(blas):
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/system_info.py", line 1983, in _calc_info
    return getattr(self, '_calc_info_{}'.format(name))()
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/system_info.py", line 1920, in _calc_info_mkl
    info = get_info('blas_mkl')
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/system_info.py", line 586, in get_info
    return cl().get_info(notfound_action)
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/system_info.py", line 846, in get_info
    self.calc_info()
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/system_info.py", line 1300, in calc_info
    info = self.check_libs2(lib_dirs, mkl_libs)
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/system_info.py", line 1002, in check_libs2
    exts = self.library_extensions()
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/system_info.py", line 961, in library_extensions
    c = customized_ccompiler()
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/system_info.py", line 203, in customized_ccompiler
    global_compiler = _customized_ccompiler()
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/__init__.py", line 50, in customized_ccompiler
    c.customize('')
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/ccompiler.py", line 90, in <lambda>
    m = lambda self, *args, **kw: func(self, *args, **kw)
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/ccompiler.py", line 529, in CCompiler_customize
    self.compile([fn], output_dir=tmpdir,
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/ccompiler.py", line 90, in <lambda>
    m = lambda self, *args, **kw: func(self, *args, **kw)
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/ccompiler.py", line 268, in CCompiler_compile
    jobs = get_num_build_jobs()
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.20.3/numpy/distutils/misc_util.py", line 96, in get_num_build_jobs
    cpu_count = len(os.sched_getaffinity(0))
OSError: [Errno 0] Error
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/math/py-numpy
Comment 1 Bjoern A. Zeeb freebsd_committer 2022-05-12 11:23:47 UTC
Add See also to the update which seems in progress for a while now but wasn't committed.  I wonder if that would improve the situation?
Comment 2 Bjoern A. Zeeb freebsd_committer 2022-05-12 11:49:49 UTC
Trying to build the update to py38-numpy-1.22.3,1 runs into the same error:

...
Processing numpy/random/bit_generator.pyx
Processing numpy/random/_pcg64.pyx
Processing numpy/random/mtrand.pyx
INFO: blas_opt_info:
INFO: blas_armpl_info:
INFO: customize UnixCCompiler
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "setup.py", line 461, in <module>
    setup_package()
  File "setup.py", line 453, in setup_package
    setup(**metadata)
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/core.py", line 135, in setup
    config = configuration()
  File "setup.py", line 131, in configuration
    config.add_subpackage('numpy')
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/misc_util.py", line 1054, in add_subpackage
    config_list = self.get_subpackage(subpackage_name, subpackage_path,
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/misc_util.py", line 1020, in get_subpackage
    config = self._get_configuration_from_setup_py(
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/misc_util.py", line 962, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "numpy/setup.py", line 9, in configuration
    config.add_subpackage('core')
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/misc_util.py", line 1054, in add_subpackage
    config_list = self.get_subpackage(subpackage_name, subpackage_path,
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/misc_util.py", line 1020, in get_subpackage
    config = self._get_configuration_from_setup_py(
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/misc_util.py", line 962, in _get_configuration_from_setup_py
    config = setup_module.configuration(*args)
  File "numpy/core/setup.py", line 813, in configuration
    blas_info = get_info('blas_opt', 0)
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/system_info.py", line 587, in get_info
    return cl().get_info(notfound_action)
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/system_info.py", line 847, in get_info
    self.calc_info()
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/system_info.py", line 2079, in calc_info
    if self._calc_info(blas):
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/system_info.py", line 2065, in _calc_info
    return getattr(self, '_calc_info_{}'.format(name))()
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/system_info.py", line 1981, in _calc_info_armpl
    info = get_info('blas_armpl')
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/system_info.py", line 587, in get_info
    return cl().get_info(notfound_action)
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/system_info.py", line 847, in get_info
    self.calc_info()
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/system_info.py", line 1339, in calc_info
    info = self.check_libs2(lib_dirs, armpl_libs)
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/system_info.py", line 1003, in check_libs2
    exts = self.library_extensions()
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/system_info.py", line 962, in library_extensions
    c = customized_ccompiler()
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/system_info.py", line 216, in customized_ccompiler
    global_compiler = _customized_ccompiler()
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/__init__.py", line 50, in customized_ccompiler
    c.customize('')
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/ccompiler.py", line 89, in <lambda>
    m = lambda self, *args, **kw: func(self, *args, **kw)
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/ccompiler.py", line 539, in CCompiler_customize
    self.compile([fn], output_dir=tmpdir,
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/ccompiler.py", line 89, in <lambda>
    m = lambda self, *args, **kw: func(self, *args, **kw)
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/ccompiler.py", line 272, in CCompiler_compile
    jobs = get_num_build_jobs()
  File "/var/tmp/work-20220224/usr/ports/math/py-numpy/work-py38/numpy-1.22.3/numpy/distutils/misc_util.py", line 93, in get_num_build_jobs
    cpu_count = len(os.sched_getaffinity(0))
OSError: [Errno 0] Error
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/math/py-numpy
Comment 3 Bjoern A. Zeeb freebsd_committer 2022-05-12 13:22:22 UTC
Renaming this as it seems the actual problem is in python itself (also removing the see-also).

On amd64 with python 3.8 installed from pkg I get:

% python3.8
Python 3.8.13 (default, Apr 28 2022, 01:23:51) 
[Clang 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a on freebsd14
Type "help", "copyright", "credits" or "license" for more information.
>>> import os;
>>> os.sched_getaffinity(0);
{0, 1, 2, 3, 4, 5, 6, 7}
>>> quit();
%

On arm64 where I am building I get:

# python3.8
Python 3.8.13 (default, May 11 2022, 17:57:27) 
[Clang 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a on freebsd14
Type "help", "copyright", "credits" or "license" for more information.
>>> import os;
>>> os.sched_getaffinity(0);
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 0] Error
>>> quit();
#


I can see a difference on an older installation where the problem was that the function wasn't present yet to my reading (again here on arm64 but probably in general);  the change is probably driven by @kib's changes to sched.h in src last year so it became supported?

# python3.8
Python 3.8.10 (default, Jul 10 2021, 10:18:40) 
[Clang 12.0.1 (git@github.com:llvm/llvm-project.git llvmorg-12.0.1-rc2-0-ge7dac on freebsd14
Type "help", "copyright", "credits" or "license" for more information.
>>> import os;
>>> os.sched_getaffinity(0);
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'os' has no attribute 'sched_getaffinity'
>>> quit();
#
Comment 4 Antoine Brodin freebsd_committer 2022-05-12 13:56:45 UTC
Could this be related to recent 
 https://cgit.FreeBSD.org/src/commit/?id=f35093f8d6d8155ab2e56c11ee03d474688b16a2
?
numpy started to fail on main-amd64 and main-i386 this night, it was successful during the previous build
Comment 5 Bjoern A. Zeeb freebsd_committer 2022-05-12 14:05:16 UTC
Thanks Antoine; made it less arm64 specific then too..  I haven't even made it to commit emails yet :(
Comment 6 commit-hook freebsd_committer 2022-05-12 17:07:23 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=3e11d3f61ae2b2b8a8ffeff4ef32067d95c065a1

commit 3e11d3f61ae2b2b8a8ffeff4ef32067d95c065a1
Author:     Dmitry Chagin <dchagin@FreeBSD.org>
AuthorDate: 2022-05-12 17:06:13 +0000
Commit:     Dmitry Chagin <dchagin@FreeBSD.org>
CommitDate: 2022-05-12 17:06:13 +0000

    Fixed the value returned by sched_getaffinity().

    On success gnu libc sched_getaffinity() should return 0, unlike underlying
    Linux syscall which returns the size of CPU mask copied to user.

    PR:             263939
    MFC after:      2 weeks

 lib/libc/gen/sched_getaffinity.c | 4 ----
 1 file changed, 4 deletions(-)
Comment 7 Dmitry Chagin freebsd_committer 2022-05-12 20:47:46 UTC
dchagin@mordor:~ % python3.8
Python 3.8.13 (default, Apr  5 2022, 02:04:31) 
[Clang 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a on freebsd14
Type "help", "copyright", "credits" or "license" for more information.
>>> import os;
>>> 
>>> os.sched_getaffinity(0);
{0, 1, 2, 3}
>>> len(os.sched_getaffinity(0));
4
>>>

fixed, thanks. close please
Comment 8 Bjoern A. Zeeb freebsd_committer 2022-05-13 11:46:03 UTC
Thank you Dmitry for looking into it and fixing this so quickly!
Much appreciated.