Bug 211346 - [NEW PORT] devel/py-llvmlite: Lightweight LLVM python binding for writing JIT compilers
Summary: [NEW PORT] devel/py-llvmlite: Lightweight LLVM python binding for writing JIT...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Kurt Jaeger
Keywords: feature, patch-ready
Depends on:
Blocks: 211347
  Show dependency treegraph
Reported: 2016-07-24 20:06 UTC by David Kalliecharan
Modified: 2016-09-05 16:12 UTC (History)
4 users (show)

See Also:
pi: maintainer-feedback+

py-llvmlite shar archive (2.18 KB, text/plain)
2016-07-24 20:06 UTC, David Kalliecharan
no flags Details
[UPDATED] py-llvmlite shell archive (2.47 KB, text/plain)
2016-07-26 00:32 UTC, David Kalliecharan
no flags Details
Poudriere py-llvmlite log (30.51 KB, text/x-log)
2016-07-26 00:33 UTC, David Kalliecharan
no flags Details
[UPDATE] py-llvmlite shar archive (2.43 KB, text/plain)
2016-08-31 16:08 UTC, David Kalliecharan
no flags Details
[UPDATE+] py-llvmlite 0.13.0 shar archive (3.44 KB, text/plain)
2016-09-04 00:45 UTC, David Kalliecharan
dave: maintainer-approval? (pi)

Note You need to log in before you can comment on or make changes to this bug.
Description David Kalliecharan 2016-07-24 20:06:44 UTC
Created attachment 172946 [details]
py-llvmlite shar archive

A port of llvmlite, which replaces llvmpy for numba.
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2016-07-25 09:17:44 UTC
Thank you for your contribution David

Can you please confirm this passes QA (portlint, poudriere) please, I can see at least one issue (leading indefinite article in COMMENT)
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2016-07-25 09:18:08 UTC
More information on testing can be found here: https://www.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/testing.html
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2016-07-25 09:20:56 UTC

- Add LICENSE_FILE if one exists in WRKSRC (I can see that is does: LICENSE)
- Upstream trove classifiers for Python version supported stipulate:

Programming Language :: Python :: 2.7
Programming Language :: Python :: 3.4
Programming Language :: Python :: 3.5 

But USES=python specifies 2.7-3.3

Since python.mk doesn't currently support X,Y-Z or X,Y+, please set this as USES=python with a comment above saying # Python 2.7,3.4+
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2016-07-25 09:22:27 UTC
Also, install_requires in setup.py are RUN_DEPENDS not BUILD_DEPENDS, and the enum34 RUN_DEPENDS needs to be conditional on PYTHON_REL < 3400 (ie; 2.7 only) as per setup.py
Comment 5 Kubilay Kocak freebsd_committer freebsd_triage 2016-07-25 09:23:28 UTC
Also, as per repository README:

"As of version 0.12, llvmlite requires LLVM 3.8"

But the port currently specifies 37
Comment 6 David Kalliecharan 2016-07-25 21:11:09 UTC
Hi Kubilay,

Thanks for getting back so quickly, I have some notes/comments on your suggestions:

1. Comment 4: RUN_DEPENDS
In my test of using RUN_DEPENDS, the port fails to build claiming enum34 needs to be installed (regardless of being in if statements or not). Indicating to me that it is required during the build process needing BUILD_DEPENDS.

2. Comment 7: LLVM 3.8 requirement
I checked the README.rst for the 0.12.1 branch, and it still says LLVM 3.7. However the master branch claims that LLVM 3.8 is required for 12.0+, but this is not true as build claims:

File "/usr/ports/devel/py-llvmlite/work/llvmlite-0.12.1/ffi/build.py", line 119, in main_posix
    raise RuntimeError(msg)
RuntimeError: Building llvmlite requires LLVM 3.7.x.

I believe the implementation and documentation for the version I am currently working on is out of date, which is fixed in the master branch. With that said, I will work with the stable release, and document the Makefile. I will post a message on github for them to raise the issue to them.

Currently just reading up on port-mgmt/poudriere
Comment 7 David Kalliecharan 2016-07-25 21:12:41 UTC
Comment 6 mentions Comment 7, but should reference Comment 5, sorry.
Comment 8 David Kalliecharan 2016-07-25 21:26:50 UTC
The LLVM 3.7 requirement issue is mentioned here for v0.12.0.dev+8.g30bcfc0.dirty
which is still present in 0.12.1.
Comment 9 David Kalliecharan 2016-07-26 00:32:12 UTC
Created attachment 172979 [details]
[UPDATED] py-llvmlite shell archive

Updated the shar file to include the necessary changes requested by Kubilay Kocak. Tested against recent ports-mgmt/poudriere, success.
Comment 10 David Kalliecharan 2016-07-26 00:33:11 UTC
Created attachment 172980 [details]
Poudriere py-llvmlite log

Including ports-mgmt/poudriere success log of py-llvmlite
Comment 11 Kubilay Kocak freebsd_committer freebsd_triage 2016-07-26 05:29:59 UTC
(In reply to David Kalliecharan from comment #6)

Thanks for the feedback David. 

If all of these things have been checked/tested/confirmed then there's no issue, though I've never seen enum34 (or anything else except C extensions and modules providing pkg_resources (like setuptools, pbr) being required at build time (setup_requires). It would be nice to see/understand exactly whats going on (the failure if set to RUN_DEPENDS

Please create issues and work with upstream to correct the dependency declarations in particular, as they are the primary thing downstreams have to go by. 

Also note that you can create poudriere jails for any version of FreeBSD, not just the version of the host. Having said that, it is highly recommended to run a host as late a version as possible, as hosts are more backward compatible with younger jails than they are the reverse

Great work on the port :)
Comment 12 David Kalliecharan 2016-07-26 11:42:51 UTC
(In reply to Kubilay Kocak from comment #11)
Regarding the RUN_DEPENDS

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "setup.py", line 24, in <module>
    from llvmlite.utils import get_library_files
  File "llvmlite/__init__.py", line 6, in <module>
    raise ImportError("could not find the 'enum' module; please install "
ImportError: could not find the 'enum' module; please install it using e.g. 'pip install enum34'
*** Error code 1

If you look at __init__.py one finds

    import enum
except ImportError:
    raise ImportError("could not find the 'enum' module; please install "
"it using e.g. 'pip install enum34'")

which is part of Fix 18
Fix #18: give a better error message when enum34 is missing
Comment 13 Kubilay Kocak freebsd_committer freebsd_triage 2016-07-26 11:54:01 UTC
Or just let *_requires do its thing :)

This looks like a spurious and unnecessary error unless llvmlite.utils is used at build time, but then the install_requires is incorrect :)

Thanks for providing the feedback, until upstream sort it out, this is OK to proceed as is.
Comment 14 David Kalliecharan 2016-07-26 13:48:56 UTC
Aweseome! Thanks for the feedback. I will sort this out by opening up an issue for llvmlite.
Comment 15 Kurt Jaeger freebsd_committer 2016-08-14 10:02:17 UTC
Comment 16 Kurt Jaeger freebsd_committer 2016-08-14 10:07:48 UTC
testbuild on 9.3a fails because:

In file included from assembly.cpp:1:
In file included from /usr/local/llvm37/include/llvm/IR/LLVMContext.h:18:
In file included from /usr/local/llvm37/include/llvm-c/Core.h:18:
In file included from /usr/local/llvm37/include/llvm-c/Support.h:17:
/usr/local/llvm37/include/llvm/Support/DataTypes.h:35:10: fatal error: 'cmath' file not found
#include <cmath>
1 warning and 1 error generated.

Should we mark this port BROKEN on 9 or do you plan to submit a fix for it ?
Comment 17 Kubilay Kocak freebsd_committer freebsd_triage 2016-08-16 10:54:55 UTC
Maybe Dimitry (CC'd) can provide assistance on the LLVM issue cause/fix for 9.3
Comment 18 David Kalliecharan 2016-08-23 16:40:43 UTC
(In reply to Kubilay Kocak from comment #17)

Hi Kubilay, sorry for the late reply, I will try and see whats wrong with the port for 9.3a this weekend. I don't have a 9.3 system at the moment. If that is too long then it may be wise to mark BROKEN on 9.
Comment 19 Kubilay Kocak freebsd_committer freebsd_triage 2016-08-23 17:10:19 UTC
(In reply to David Kalliecharan from comment #18)

Not at all. We can do the broken, but would prefer not to for a new port if we can avoid it without much trouble.

You also have the option of testing with poudriere (which would automate the freebsd (9.3) jail stuff)
Comment 20 David Kalliecharan 2016-08-23 18:09:38 UTC
(In reply to Kubilay Kocak from comment #19)
Okay I will do that this weekend, in the process of moving so a bit distracted at the moment.
Comment 21 David Kalliecharan 2016-08-30 19:40:34 UTC
I looked at version 9.3a with poudriere, and got the same error. I suspected that it wasn't linking to libc++ (an apparent problem with 9.3a in some cases). I tried adding

.if ${_OSVERSION_MAJOR} < 10
CXXFLAGS= -stdlib=libc++ -std=c++11
LDFLAGS= -stdlib=libc++

found at:

But this did *not* work, and the build already did the above mentioned. I would need more time to look at it. At the moment I am okay with it being marked as BROKEN for 9. Do I need to add that into the Makefile?
Comment 22 Kurt Jaeger freebsd_committer 2016-08-31 03:08:07 UTC
Upstream has released 0.13.0, can you update the port and test if it works ?
Comment 23 David Kalliecharan 2016-08-31 16:08:57 UTC
Created attachment 174256 [details]
[UPDATE] py-llvmlite shar archive

Updated py-llvmlite to 0.13.0, builds on FreeBSD 11, 10.x, but still BROKEN on 9.3.
Comment 24 Kurt Jaeger freebsd_committer 2016-09-03 19:57:06 UTC
Committed, thanks!
Comment 25 commit-hook freebsd_committer 2016-09-03 19:57:25 UTC
A commit references this bug:

Author: pi
Date: Sat Sep  3 19:56:56 UTC 2016
New revision: 421299
URL: https://svnweb.freebsd.org/changeset/ports/421299

  New port: devel/py-llvmlite

  A lightweight LLVM python binding for writing JIT compilers

  The old llvmpy binding exposes a lot of LLVM APIs but the mapping
  of C++-style memory management to Python is error prone. Numba and
  many JIT compilers do not need a full LLVM API. Only the IR builder,
  optimizer, and JIT compiler APIs are necessary.

  llvmlite is a project originally tailored for Numba's needs, using
  the following approach:

  - A small C wrapper around the parts of the LLVM C++ API we need
    that are not already exposed by the LLVM C API.
  - A ctypes Python wrapper around the C API.
  - A pure Python implementation of the subset of the LLVM IR builder
    that we need for Numba.

  WWW: https://github.com/numba/llvmlite

  PR:		211346
  Submitted by:	David Kalliecharan <dave@dal.ca>

Comment 26 David Kalliecharan 2016-09-04 00:45:42 UTC
Created attachment 174358 [details]
[UPDATE+] py-llvmlite 0.13.0 shar archive

Fixed the libgcc_s.so.1 error so that libllvmlite.so links to /usr/local/lib/gcc${_GCC_VER}, added a patch file to work around the issue.

Tested with poudriere 10, 11, still BROKEN for 9.3

This satisfies the requirement for the port py-numba to operate as expected
Comment 27 David Kalliecharan 2016-09-04 00:47:14 UTC
Opened, because py-llvmlite was the issue with libgcc_s.so.1 linking issue with py-numba port.

The updated shar file is attached and needs approval. Sorry for being late with this.
Comment 28 Kurt Jaeger freebsd_committer 2016-09-05 16:11:35 UTC
Committed, thanks!
Comment 29 commit-hook freebsd_committer 2016-09-05 16:12:12 UTC
A commit references this bug:

Author: pi
Date: Mon Sep  5 16:11:24 UTC 2016
New revision: 421377
URL: https://svnweb.freebsd.org/changeset/ports/421377

  devel/py-llvmlite: fix linking of libllvmlite.so to the correct libgcc_s.so

  PR:		211346
  Submitted by:	David Kalliecharan <dave@dal.ca> (maintainer)