Bug 211641 - graphics/py-mayavi mayavi2 runtime error
Summary: graphics/py-mayavi mayavi2 runtime error
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-python mailing list
URL:
Keywords: needs-patch, needs-qa, patch
Depends on: 208120
Blocks:
  Show dependency treegraph
 
Reported: 2016-08-07 14:54 UTC by Chen Xie
Modified: 2018-01-17 07:04 UTC (History)
3 users (show)

See Also:
vladimir.chukharev: maintainer-feedback+
koobs: merge-quarterly?


Attachments
Patch (1.22 KB, patch)
2016-08-08 18:42 UTC, vladimir.chukharev
vladimir.chukharev: maintainer-approval+
Details | Diff
Patch (1.87 KB, patch)
2016-08-11 12:54 UTC, vladimir.chukharev
vladimir.chukharev: maintainer-approval+
Details | Diff
Patch adding pkg-message (878 bytes, patch)
2016-08-17 12:27 UTC, vladimir.chukharev
vladimir.chukharev: maintainer-approval+
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Chen Xie 2016-08-07 14:54:59 UTC
First thanks maintainer for porting this software!

There is noting wrong when building package in poudriere and installing. When I run a example.py in terminal, it seems not work.

> mayavi2 -x sample.py
Traceback (most recent call last):
  File "/usr/local/bin/mayavi2", line 9, in <module>
    load_entry_point('mayavi==4.5.0', 'gui_scripts', 'mayavi2')()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 542, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2569, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2229, in load
    return self.resolve()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2235, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python2.7/site-packages/mayavi/scripts/mayavi2.py", line 494, in <module>
    raise ImportError(msg)
ImportError: /lib/libgcc_s.so.1: version GCC_4.6.0 required by /usr/local/lib/gcc48/libgfortran.so.3 not found
________________________________________________________________________________
Could not load envisage. You might have a missing dependency.
Do you have the EnvisageCore and EnvisagePlugins installed?

If you installed Mayavi with easy_install, try 'easy_install <pkg_name>'.
'easy_install Mayavi[app]' will also work.

If you performed a source checkout and installed via 'python setup.py develop',
be sure to run the same command in the EnvisageCore and EnvisagePlugins folders.

If these packages appear to be installed, check that your numpy and
configobj are installed and working. If you need numpy, 'easy_install numpy'
will install numpy. Similarly, 'easy_install configobj' will install
configobj.

I add this line to /etc/libmap.conf:
libgcc_s.so.1   gcc48/libgcc_s.so.1
rebuild and reinstall py27-mayavi py27-envisage py27-configobj, it is solved.
But I do not how to solve this by adapting the Makefile.
USE_GCC=yes? USES=fortran?
Comment 1 vladimir.chukharev 2016-08-07 19:00:20 UTC
You do not even need to rebuild anything. Simply start it with an env variable:
LD_LIBRARY_PATH=/usr/local/lib/gcc48 python2.7 mayavi2 -x sample.py

The variable can be set in ~/.profile or what is your shell initializing script.

The path might change with the gcc version change.

This is a known issue without a known complete solution. It affects or affected quite many ports, e.g. math/py-numpy, and mayavi2 via dependences. I do not know how to work around this within the port. In my opinion, it should be done in gcc* ports...
 
I'll try one idea, though. It should not take more than two days to check if it works.
Comment 2 Chen Xie 2016-08-08 01:23:29 UTC
Thank you. I also noticed "-Wl -rpath=/usr/local/lib/gcc*" everywhere.
Comment 3 vladimir.chukharev 2016-08-08 18:42:41 UTC
Created attachment 173418 [details]
Patch

I failed to set an environment variable from inside of mayavi2 script the way that it affect the libraries as I planed. Therefore I renamed the script as mayavi2.py, and prepared a shell script named mayavi2 just to set the env variable and call the renamed script with all parameters passed along.

Please test the patch with reverted changes to lib configs, if possible.

Builds under poudriere and synth look fine.

DEVELOPER=yes portlint -abcmt
looks fine.
Comment 4 Chen Xie 2016-08-09 07:54:25 UTC
I do manually as you described in the first paragraph, it is ok.
Patch may need to be revised./usr/local/bin/mayavi2 mode is 644. /usr/local/bin/mayavi2.py is not installed. The first line of mayavi2.sh miss "/bin/sh" ? Or I patched incorrectly.
Comment 5 vladimir.chukharev 2016-08-10 10:30:28 UTC
(In reply to Chen Xie from comment #4)
Thanks for testing, it's very helpful.

The exec bits are set in the patch for svn, I do not know how this should be treated locally. I hope a committer will understand this correctly from the patch.

Thanks for catching that he renamed script is not installed. I missed it. It is not also deleted when the port is removed. Now fighting a way to do this correctly with the installer used upstream. Unfortunately, this might take some time.

You are right that it's better to put shebang completely, with the path. I cannot find now that it defaults to sh in modern docs. Will fix in the new version of the patch. Thanks for mentioning it.

The port builds fail on 9.3 systems on pkg builders, I'm going to fix also those problems in the new patch. This will delay the result even more. Sorry.
Comment 6 Chen Xie 2016-08-11 02:39:57 UTC
(In reply to vladimir.chukharev from comment #5)
I was aware of that you used "svn diff" and "svn patch" because I am a newbie. I used patch(1) can not handle files in a subdirectory.
A few people like me always use tcsh rather than bash, and we need the path of interpreter.
Thanks for your work. I can enjoy matplotlib and mayavi both on FreeBSD now!
Comment 7 vladimir.chukharev 2016-08-11 12:54:39 UTC
Created attachment 173546 [details]
Patch

A reworked patch.

1. Add execinfo to USES, this fixes build on older systems without libexecinfo.so [1].
2. Define environment variable LD_LIBRARY_PATH at the beginning of the mayavi2 script. This works around library search errors in the port dependences.
3. Do stripping in post-stage: rather then in post-install: target.
4. Bump portrevision.

[1] Reported by pkg-fallout@freebsd.org. Subject: [package - 93amd64-default][graphics/py-mayavi] Failed for py27-mayavi-4.5.0 in build

Please test. Passed my tests including builds with poudriere in jails 9Samd64 and 10Samd64.
Comment 8 Kubilay Kocak freebsd_committer freebsd_triage 2016-08-12 08:40:36 UTC
This needs to be solved at the root, which (so far is, may be) bug 208120
Comment 9 vladimir.chukharev 2016-08-12 09:20:59 UTC
(In reply to Kubilay Kocak from comment #8)
Please note that the patch also fixes pakaging on 9* systems. I'm on CC list of bug 208120 and will test this port when a patch will be added.

IMHO it's better to commit the patch now (probably not closing this bug). Is it possible? The problem with LD_LIBRARY_PATH is at least a year old...
Comment 10 Kubilay Kocak freebsd_committer freebsd_triage 2016-08-12 09:58:19 UTC
(In reply to vladimir.chukharev from comment #9)

If you'd like this to be a workaround in leui of fixing the root cause that's fine.

A couple of review comments on attachment 173546 [details]

* Use PYTHONPREFIX_SITELIBDIR for ${STAGEDIR}${PREFIX}/lib/python2.7/site-packages/tvtk/array_ext.so
* The post-install -> post-stage switch is unexplained (for stripping). post-install should be fine for both commands executed
* The additional of mayavi2 executable is unexplained. Related, for version suffixed script names, USE_PYTHON=concurrent is preferred.
* os.environ["LD_LIBRARY_PATH"] = "/usr/local/lib/gcc48" should likely use %%LOCALBASE%% in files, and be replaced (REINPLACE_CMD) in post-patch to to be prefix safe. You'll probably need SUB_FILES too.

Finally please reconfirm QA across as many arch/versions as possible, as this will be committed then merged to the quarterly branch.
Comment 11 Chen Xie 2016-08-13 08:20:09 UTC
(In reply to vladimir.chukharev from comment #7)
It seems that "os.environ" does not work on my 10.3-release-amd64. Building has no problem. When mayavi2 runs, it has the same error: ImportError: /lib/libgcc_s.so.1: version GCC_4.6.0 required by /usr/local/lib/gcc48/libgfortran.so.3 not found.
Comment 12 vladimir.chukharev 2016-08-13 08:43:59 UTC
(In reply to Chen Xie from comment #11)
Please reconfirm that you used the patch. It's important that os.environ is assigned before other imports. (My first attempts to use os.environ also did not help, that's why I made the shell wrapper script.)
Comment 13 vladimir.chukharev 2016-08-13 10:38:35 UTC
(In reply to Kubilay Kocak from comment #10)
That's right, I consider setting env as a temporary workaround in a hope that the core problem will be solved in the compiling infrastructure relatively soon. That's why I did not put much effort into cleaning the patch (in this part).

Unfortunately, the patch from bag 208120 does not solve it on my computer.

Now to your comments.
> * Use PYTHONPREFIX_SITELIBDIR for ${STAGEDIR}${PREFIX}/lib/python2.7/site-packages/tvtk/array_ext.so

Thanks, I'll make this change. Now testing.

> * The post-install -> post-stage switch is unexplained (for stripping). post-install should be fine for both commands executed

Yes, they are both fine. If I understand correctly, stripping before packaging is slightly better. Even better, before staging, e.g. in post-build:, but then it cannot be coupled with patching bin/mayavi2 script which is created only in stage.

> * The additional of mayavi2 executable is unexplained. Related, for version suffixed script names, USE_PYTHON=concurrent is preferred.

Actually, this is just adding 'import os; os.environ["LD_LIBRARY_PATH"] = "/usr/local/lib/gcc48"' with proper formatting to the existing script by overwriting the script with a modified one. This is the very workaround. In the previous patch, I tried to rename the original script and call it from a shell wrapper, but did not manage to add the renamed script into the auto generated plist.
As to suffixes, some of the used libraries are not yet python3 ready.

>* os.environ["LD_LIBRARY_PATH"] = "/usr/local/lib/gcc48" should likely use %%LOCALBASE%% in files, and be replaced (REINPLACE_CMD) in post-patch to to be prefix safe. You'll probably need SUB_FILES too.

The unknown part of this is how to find out the version of gcc in the Makefile... Can you help with it, please?

> Finally please reconfirm QA across as many arch/versions as possible, as this will be committed then merged to the quarterly branch.

My notebook is about 6 years old and has 4GB of memory. Adding the second jail to poudriere took more than 24 hours. So, I cannot promise testing on anything more than 10-STABLE and 9-STABLE on amd64. Sorry.
Comment 14 Chen Xie 2016-08-13 16:03:05 UTC
(In reply to vladimir.chukharev from comment #12)
Yes, I used the patch. But I chmod 755 files/mayavi2 myself because I use patch(1) rather than "svn patch". I even add the line 
"print os.environ["LD_LIBRARY_PATH"]" before if block in /usr/local/bin/mayavi2, it prints "/usr/local/lib/gcc48" and then "ImportError....". So weird.
Comment 15 vladimir.chukharev 2016-08-13 16:38:52 UTC
(In reply to Chen Xie from comment #14)
I restored the "solution" from the patch (I've changed it already), it does not work! Thanks for catching it. Now I'm lost... Back to the shell wrapper, I guess.
Comment 16 Chen Xie 2016-08-14 05:17:48 UTC
(In reply to vladimir.chukharev from comment #15)
Yes, you are right at the beginning. Shell wrapper is a good compromise. We should wait some change in /usr/ports/Mk.
Comment 17 vladimir.chukharev 2016-08-17 12:27:06 UTC
Created attachment 173780 [details]
Patch adding pkg-message

I still cannot solve a seemingly trivial and common problem.

The port installs a Python script as /usr/local/bin/mayavi2. I want to rename it as /usr/local/bin/mayavi2.py and install a shell script which is a wrapper for the renamed Python script. I cannot find a way to do it so that both scripts are added correctly to the auto generated plist, registered in the package and in the pkgng database.

The original script is created directly to the stage dir, it does not exist in the build dir. And this process is buried deeply in the setup.py with many libraries, so few hours were not enough for me to find out how this can be changed to do the new task.

My current proposal is to just add a pkg-message informing about the need for the environment variable LD_LIBRARY_PATH=/usr/local/lib/gcc48. The new patch adds pkg-message.

Please comment. Particularly I'm interested to learn how a wrapping script can be made.
Comment 18 vladimir.chukharev 2016-08-17 12:31:23 UTC
Forget to mention that I've separated the patch for package builds on older systems as bug 211937.
Comment 19 w.schwarzenfeld freebsd_triage 2018-01-17 07:04:28 UTC
pkg-message is still not in the port. I guess the error with gfortran still exists.