Bug 204793 - www/mod_python35: Fatal Python error: PyEval_SaveThread: NULL tstate
Summary: www/mod_python35: Fatal Python error: PyEval_SaveThread: NULL tstate
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Olli Hauer
URL: https://github.com/grisha/mod_python/...
Keywords: crash, patch
Depends on:
Blocks:
 
Reported: 2015-11-25 00:18 UTC by SBB
Modified: 2016-03-16 04:18 UTC (History)
4 users (show)

See Also:
koobs: maintainer-feedback-
ohauer: merge-quarterly-


Attachments
Do not import site.py patch (1.06 KB, patch)
2016-01-08 15:24 UTC, SBB
no flags Details | Diff
patch-mod_python35 (15.53 KB, patch)
2016-01-15 18:34 UTC, SBB
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description SBB 2015-11-25 00:18:36 UTC
Not sure what's needed here but I've included everything I can think of. Let me know if I've missed anything or more info is needed.

Problem: Apache runs for about a day but then crashes with this error in the error log (even with little to no use). I have 2 httpd core dumps if that would help.
> [Wed Nov 25 00:00:01.048663 2015] [:notice] [pid 7108] mod_python: Creating 8 session mutexes based on 256 max processes and 0 max threads.
> [Wed Nov 25 00:00:01.048682 2015] [:notice] [pid 7108] mod_python: using mutex_directory /tmp
Fatal Python error: PyEval_SaveThread: NULL tstate
> [Wed Nov 25 00:00:01.048950 2015] [core:notice] [pid 7108] AH00060: seg fault or similar nasty error detected in the parent process

Found a similar issue on http://bugs.python.org/ but the python devs indicate it's not a problem with python but the 3rd party program. Not sure if it applies to this but thought I'd mention: http://bugs.python.org/issue16749

uname -a:
> FreeBSD example.com 10.2-RELEASE-p7 FreeBSD 10.2-RELEASE-p7 #2: Tue Nov 24 22:05:12 UTC 2015     user@example.com:/usr/obj/usr/src/sys/CUSTOM  amd64

Important packages installed:
> ap24-mod_python35-3.5.0_1
> apache24-2.4.17
> mod_php56-5.6.14
> nagios-3.5.1_8
> php56-5.6.14
> pkg-1.6.1_2
> python-2.7_2,2
> python2-2_3
> python27-2.7.10_1

apachectl -M:
> Loaded Modules:
>  core_module (static)
>  so_module (static)
>  http_module (static)
>  mpm_prefork_module (static)
>  authn_file_module (shared)
>  authn_core_module (shared)
>  authz_host_module (shared)
>  authz_groupfile_module (shared)
>  authz_user_module (shared)
>  authz_core_module (shared)
>  access_compat_module (shared)
>  auth_basic_module (shared)
>  reqtimeout_module (shared)
>  filter_module (shared)
>  mime_module (shared)
>  log_config_module (shared)
>  env_module (shared)
>  headers_module (shared)
>  setenvif_module (shared)
>  version_module (shared)
>  unixd_module (shared)
>  status_module (shared)
>  autoindex_module (shared)
>  cgi_module (shared)
>  dir_module (shared)
>  alias_module (shared)
>  php5_module (shared)
>  python_module (shared)
>  auth_digest_module (shared)
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2015-11-25 01:01:25 UTC
Backtraces from the coredumps (as attachments) might prove useful.

If mod_python (and/or Apache) can be built with DEBUG options and the symptom is still reproducible, it might be worth enabling that too.

I also see this issue upstream (CentOS)

https://github.com/grisha/mod_python/issues/46

Please add your information there as well
Comment 2 SBB 2015-11-25 03:52:08 UTC
Ah, yes, I saw that issue as well but forgot to mention it. I've posted my information there as well.

I'm not sure exactly how to get a backtrace, ie what commands to run but simply running "gdb /usr/local/sbin/httpd /root/httpd.core" gives the following:

> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)...
> Core was generated by `httpd'.
> Program terminated with signal 6, Aborted.
> Reading symbols from /usr/local/lib/libpcre.so.1...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/lib/libpcre.so.1
> Reading symbols from /usr/local/lib/libaprutil-1.so.0...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/lib/libaprutil-1.so.0
> Reading symbols from /usr/local/lib/libexpat.so.1...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/lib/libexpat.so.1
> Reading symbols from /usr/local/lib/libapr-1.so.0...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/lib/libapr-1.so.0
> Reading symbols from /lib/libcrypt.so.5...(no debugging symbols found)...done.
> Loaded symbols for /lib/libcrypt.so.5
> Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done.
> Loaded symbols for /lib/libthr.so.3
> Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done.
> Loaded symbols for /lib/libc.so.7
> Reading symbols from /lib/libm.so.5...(no debugging symbols found)...done.
> Loaded symbols for /lib/libm.so.5
> Reading symbols from /usr/local/lib/libpython2.7.so.1...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/lib/libpython2.7.so.1
> Reading symbols from /usr/local/lib/libintl.so.8...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/lib/libintl.so.8
> Reading symbols from /lib/libutil.so.9...(no debugging symbols found)...done.
> Loaded symbols for /lib/libutil.so.9
> Reading symbols from /usr/local/lib/python2.7/lib-dynload/_locale.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/lib/python2.7/lib-dynload/_locale.so
> Reading symbols from /usr/local/libexec/apache24/mod_authn_file.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_authn_file.so
> Reading symbols from /usr/local/libexec/apache24/mod_authn_core.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_authn_core.so
> Reading symbols from /usr/local/libexec/apache24/mod_authz_host.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_authz_host.so
> Reading symbols from /usr/local/libexec/apache24/mod_authz_groupfile.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_authz_groupfile.so
> Reading symbols from /usr/local/libexec/apache24/mod_authz_user.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_authz_user.so
> Reading symbols from /usr/local/libexec/apache24/mod_authz_core.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_authz_core.so
> Reading symbols from /usr/local/libexec/apache24/mod_access_compat.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_access_compat.so
> Reading symbols from /usr/local/libexec/apache24/mod_auth_basic.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_auth_basic.so
> Reading symbols from /usr/local/libexec/apache24/mod_reqtimeout.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_reqtimeout.so
> Reading symbols from /usr/local/libexec/apache24/mod_filter.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_filter.so
> Reading symbols from /usr/local/libexec/apache24/mod_mime.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_mime.so
> Reading symbols from /usr/local/libexec/apache24/mod_log_config.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_log_config.so
> Reading symbols from /usr/local/libexec/apache24/mod_env.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_env.so
> Reading symbols from /usr/local/libexec/apache24/mod_headers.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_headers.so
> Reading symbols from /usr/local/libexec/apache24/mod_setenvif.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_setenvif.so
> Reading symbols from /usr/local/libexec/apache24/mod_version.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_version.so
> Reading symbols from /usr/local/libexec/apache24/mod_unixd.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_unixd.so
> Reading symbols from /usr/local/libexec/apache24/mod_status.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_status.so
> Reading symbols from /usr/local/libexec/apache24/mod_autoindex.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_autoindex.so
> Reading symbols from /usr/local/libexec/apache24/mod_cgi.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_cgi.so
> Reading symbols from /usr/local/libexec/apache24/mod_dir.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_dir.so
> Reading symbols from /usr/local/libexec/apache24/mod_alias.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_alias.so
> Reading symbols from /usr/local/libexec/apache24/libphp5.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/libphp5.so
> Reading symbols from /usr/local/lib/libxml2.so.2...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/lib/libxml2.so.2
> Reading symbols from /lib/libz.so.6...(no debugging symbols found)...done.
> Loaded symbols for /lib/libz.so.6
> Reading symbols from /usr/lib/liblzma.so.5...(no debugging symbols found)...done.
> Loaded symbols for /usr/lib/liblzma.so.5
> Reading symbols from /usr/local/libexec/apache24/mod_python.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_python.so
> Reading symbols from /usr/local/libexec/apache24/mod_auth_digest.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/libexec/apache24/mod_auth_digest.so
> Reading symbols from /usr/local/lib/php/20131226/xml.so...(no debugging symbols found)...done.
> Loaded symbols for /usr/local/lib/php/20131226/xml.so
> Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done.
> Loaded symbols for /libexec/ld-elf.so.1
> #0  0x000000080169d64a in thr_kill () from /lib/libc.so.7
> [New Thread 802006400 (LWP 100948/<unknown>)]

I'm not sure I can rebuild the apache and/or mod_python with debugging on this particular machine but I'll look into it. Otherwise let me know if there's anything else that would help.
Comment 3 SBB 2016-01-06 22:37:18 UTC
The fix committed here seems to have fixed the problem. I've been running a patched version for a month now with out issue.

https://github.com/grisha/mod_python/commit/5bb5d6d0113f6bbd72966a5c1f3e6d40c2e9c8fd
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2016-01-07 03:07:56 UTC
Maintainer timeout, open to take

@SBB could you produce a patch against the port head backporting the commit mentioned on comment 3?
Comment 5 SBB 2016-01-08 15:24:35 UTC
Created attachment 165267 [details]
Do not import site.py patch

I'm not 100% sure if this is what you are asking for as this is my first time posting a patch but this is what I tested with and what is working for me.
Comment 6 Kubilay Kocak freebsd_committer freebsd_triage 2016-01-08 16:10:43 UTC
(In reply to SBB from comment #5)

That's a great effort (and correct, for a patch against the upstream sources), though what is preferable is a patch against the port itself, creating a files/patch-* file.

Here's how you do it:

1) # cd /path/to/category/port
2) # make clean extract
3) # cd work/path/to/file
4) # cp file file.orig
5) edit file
6) go back to port directory
7) # make makepatch

You'll find a brand new file created for you in files/ :)

Then just `svn diff` > patch.diff in the port directory to produce a diff against the port

There is documentation on this, but it's a little ambiguous:

https://www.freebsd.org/doc/en/books/porters-handbook/slow-patch.html

Also, bonus points if you patch in the entire upstream commit (including comment)

You can grab that here:


https://github.com/grisha/mod_python/commit/5bb5d6d0113f6bbd72966a5c1f3e6d40c2e9c8fd.patch
Comment 7 SBB 2016-01-11 15:16:21 UTC
(In reply to Kubilay Kocak from comment #6)

Ok, I will give that a try today or tomorrow. Thanks for explaining this and your patience!
Comment 8 SBB 2016-01-15 18:34:35 UTC
Created attachment 165638 [details]
patch-mod_python35

Followed your instructions and this is what I came up with. Does it look correct?
Comment 9 Olli Hauer freebsd_committer 2016-03-15 14:23:43 UTC
The port was updated to reflects the latest git revision.

Sorry for delay.
Comment 10 commit-hook freebsd_committer 2016-03-15 15:20:29 UTC
A commit references this bug:

Author: ohauer
Date: Tue Mar 15 14:02:10 UTC 2016
New revision: 411163
URL: https://svnweb.freebsd.org/changeset/ports/411163

Log:
  - update patch to reflect latest commit on github
    (there is no new release sinc 2013 ...)
  - do not alter httpd.conf, install a dedicated module file

  PR:		204793
  Submitted by:	SBB

Changes:
  head/www/mod_python35/Makefile
  head/www/mod_python35/files/270_mod_python.conf.sample.in
  head/www/mod_python35/files/patch-mod_python35
  head/www/mod_python35/files/patch-test_test.py
  head/www/mod_python35/files/pkg-message.in
  head/www/mod_python35/pkg-message
  head/www/mod_python35/pkg-plist
Comment 11 Kubilay Kocak freebsd_committer freebsd_triage 2016-03-16 03:55:34 UTC
Assign to committer that resolved

Re-open pending as MFH candidate

@Olli, please set merge-quarterly to "+" after merging, or to "-" with comment if MFH not appropriate
Comment 12 Olli Hauer freebsd_committer 2016-03-16 04:18:48 UTC
(In reply to Kubilay Kocak from comment #11)

Koobs, I know it sounds silly, but would you mind to not always reopen PR's asking for MFH if there is only one report over a longer period?