Bug 216609 - devel/py-ice: fails to build with libc++ 4.0
Summary: devel/py-ice: fails to build with libc++ 4.0
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Michael Gmelin
URL:
Keywords: needs-patch
Depends on:
Blocks: 216008
  Show dependency treegraph
 
Reported: 2017-01-30 16:50 UTC by Jan Beich
Modified: 2017-01-31 03:47 UTC (History)
5 users (show)

See Also:
bugzilla: maintainer-feedback? (freebsd)
jbeich: merge-quarterly?


Attachments
c++ -E output diff for Time.cpp (2.04 KB, patch)
2017-01-30 17:00 UTC, Jan Beich
no flags Details | Diff
workaround (875 bytes, patch)
2017-01-30 18:33 UTC, Jan Beich
grembo: maintainer-approval+
Details | Diff
c++ -E output diff for Time.cpp (2.06 KB, patch)
2017-01-30 18:40 UTC, Jan Beich
no flags Details | Diff
poudriere build log devel/ice using clang 4.0 (592.18 KB, text/plain)
2017-01-31 03:44 UTC, Michael Gmelin
no flags Details
poudriere build log devel/py-ice using clang 4.0 (51.50 KB, text/plain)
2017-01-31 03:45 UTC, Michael Gmelin
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jan Beich freebsd_committer 2017-01-30 16:50:45 UTC
/usr/local/bin/python2.7 ../config/s2py.py --prefix Ice_ --no-package -I../../slice --ice ../../slice/Ice/BuiltinSequences.ice
Traceback (most recent call last):
  File "../config/s2py.py", line 22, in <module>
    import IcePy
ImportError: ../config/../python/IcePy.so: Undefined symbol "_ZN7IceUtil4TimeC1Ex"

build log: http://sprunge.us/LLFM
regressed by: https://github.com/llvm-mirror/libcxx/commit/08fa01095a75
Comment 1 Jan Beich freebsd_committer 2017-01-30 17:00:25 UTC
Created attachment 179431 [details]
c++ -E output diff for Time.cpp
Comment 2 Michael Gmelin freebsd_committer 2017-01-30 17:05:36 UTC
How can I reproduce this output?
(FreeBSD version, make.conf version, portstree version, installed packages etc.)
Comment 3 Jan Beich freebsd_committer 2017-01-30 17:14:41 UTC
$ poudriere jail -cj clang40-amd64 -v projects/clang400-import -m svn
$ poudriere bulk -Ctj clang40-amd64 devel/ice devel/py-ice

or if you want to play with just libc++

$ git clone https://github.com/llvm-mirror/libcxx/ /path/to/libcxx
$ (cd /path/to/libcxx; git checkout origin/release_40)
$ cd /usr/ports/devel/ice
$ echo 'CXXFLAGS += -nostdinc++ -isystem /path/to/libcxx/include' >>Makefile.local
Comment 4 Jan Beich freebsd_committer 2017-01-30 17:17:28 UTC
i386 is probably unaffected since it actually wants Int64 as long long unlike amd64.
Comment 5 Jan Beich freebsd_committer 2017-01-30 17:21:03 UTC
Actually, for bisecting I've used the following command line:

$ cd /usr/ports/devel/ice
$ make
$ cd $(make -V WRKSRC)/cpp/src/IceUtil
$ c++ -nostdinc++ -isystem /path/to/libcxx/include -c -I../../include -I/usr/include -I/usr/include -DICE_UTIL_API_EXPORTS -I.. -MMD Time.cpp -MF .depend/Time.d; nm Time.o | fgrep _ZN7IceUtil4TimeC1Ex
Comment 6 Jan Beich freebsd_committer 2017-01-30 17:43:57 UTC
__WORDSIZE as defined by <stdint.h> is broken for C++ without __STDC_LIMIT_MACROS. I'm going to blame base r228529 for the issue here.

# Test for 10.3 or 11.0 amd64
$ cat >a.cc
#include <stdint.h>
#include <stdio.h>

int main()
{
    printf("%d\n", __WORDSIZE);
    return 0;
}

$ c++ a.cc
$ ./a.out
32
Comment 7 Dimitry Andric freebsd_committer 2017-01-30 17:58:43 UTC
It's pretty weird, I've built devel/ice with default settings, on the clang400-import branch, and my Time.o does contain the symbol:

$ nm /wrkdirs/share/dim/ports/devel/ice/work/ice-3.6.3/cpp/src/IceUtil/Time.o | grep _ZN7IceUtil4TimeC1Ex
00000c80 T _ZN7IceUtil4TimeC1Ex

Consequently, I have no problems linking py-ice.  Also, I don't see any code in Time.cpp that conditionalizes Time::Time(Int64 usec)?  So does this depend on the typedef for Int64?
Comment 8 Dimitry Andric freebsd_committer 2017-01-30 18:01:21 UTC
(In reply to Dimitry Andric from comment #7)
> It's pretty weird, I've built devel/ice with default settings, on the
> clang400-import branch, and my Time.o does contain the symbol:
> 
> $ nm
> /wrkdirs/share/dim/ports/devel/ice/work/ice-3.6.3/cpp/src/IceUtil/Time.o |
> grep _ZN7IceUtil4TimeC1Ex
> 00000c80 T _ZN7IceUtil4TimeC1Ex

Ah, I have built on i386, obviously. I didn't test amd64.
Comment 9 Dimitry Andric freebsd_committer 2017-01-30 18:06:17 UTC
(In reply to Jan Beich (mail not working) from comment #6)
> __WORDSIZE as defined by <stdint.h> is broken for C++ without
> __STDC_LIMIT_MACROS. I'm going to blame base r228529 for the issue here.
> 
> # Test for 10.3 or 11.0 amd64
> $ cat >a.cc
> #include <stdint.h>
> #include <stdio.h>
> 
> int main()
> {
>     printf("%d\n", __WORDSIZE);
>     return 0;
> }
> 
> $ c++ a.cc
> $ ./a.out
> 32

Ugh, for C it works, for C++ it doesn't. :(  E.g that means devel/ice has always assumed ICE_32 on amd64 instead?
Comment 10 Jan Beich freebsd_committer 2017-01-30 18:33:15 UTC
Created attachment 179433 [details]
workaround

(In reply to Dimitry Andric from comment #9)
> ... devel/ice has always assumed ICE_32 on amd64 instead?

devel/ice on 10.3, 11.0, 12.0 (for now) uses long long on amd64. So, we need a fix in base and a (temporary) workaround in the port.

$ uname -p
amd64
$ pkg install -qy ice
$ nm -D /usr/local/lib/libIceUtil.so | fgrep _ZN7IceUtil4TimeC1Ex
0000000000034d80 T _ZN7IceUtil4TimeC1Ex
Comment 11 Michael Gmelin freebsd_committer 2017-01-30 18:40:34 UTC
Before Ice 3.6.3, Ice always assumed ICE_32 on amd64. This has been fixed since then, but only if built with --std=c++11 (which is would we do in production, this is why I didn't notice).

Using your example:

e.g.

$ c++ --version
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Target: x86_64-unknown-freebsd10.2
Thread model: posix

$ c++ a.cc
$ ./a.out 
32
$ c++ -std=c++98 a.cc # same as before
$ ./a.out 
32
$ c++ -std=c++11 a.cc
$ ./a.out 
64
Comment 12 Jan Beich freebsd_committer 2017-01-30 18:40:58 UTC
Created attachment 179435 [details]
c++ -E output diff for Time.cpp

Previous one had __STDC_LIMIT_MACROS manually removed from /usr/include/c++/v1/stdint.h instead of using libc++ r282434, so it didn't show new includes properly.
Comment 13 Michael Gmelin freebsd_committer 2017-01-30 18:45:17 UTC
For completeness sake:

$ c++ -std=c++98 -D__STDC_LIMIT_MACROS a.cc
$ ./a.out 
64

So this accomplishes what we want (always use ICE_64 on amd64).

I'll approve the patch, but I'd suggest to add a line to UPDATING (when we switched from Ice 3.6.2 to Ice 3.6.3 we had some [overly optimistic] template code break in the process).
Comment 14 Michael Gmelin freebsd_committer 2017-01-30 18:47:54 UTC
Comment on attachment 179433 [details]
workaround

Looks good to me.
Comment 15 Michael Gmelin freebsd_committer 2017-01-30 18:50:40 UTC
Also: This difference in __WORDSIZE could explain a couple of runtime problems I've seen over the years when linking ports that where built using --std=c++98 with ones built using --std=c++11 and/or --std=c++14.
Comment 16 Jan Beich freebsd_committer 2017-01-30 18:54:05 UTC
(In reply to Michael Gmelin from comment #11)
> ... if built with --std=c++11 (which is would we do in production ...

Looking at the port Makefile and vendor source I see --std=c++11 being default only on Darwin. Was your production not on FreeBSD or using custom packages?

(In reply to Michael Gmelin from comment #13)
> I'd suggest to add a line to UPDATING (when we switched from Ice
> 3.6.2 to Ice 3.6.3 we had some [overly optimistic] template code
> break in the process).

As a maintainer you're better positioned to write an UPDATING entry. I can only check existing consumers in ports still build fine.
Comment 17 Michael Gmelin freebsd_committer 2017-01-30 19:00:15 UTC
(In reply to Jan Beich (mail not working) from comment #16)

We use C++14 in production and build all packages ourselves, as there were compatibility problems in the past (this obviously being one of them). We see more and more ports that define c++11/14 (USES=compiler:c++11), so fixing this in general would be a good idea.


I'll prepare an updated version of the port that applies your patch.
Comment 18 Michael Gmelin freebsd_committer 2017-01-30 21:17:59 UTC
This is the reason why this worked using --std=c++11:

From /usr/include/sys/cdefs.h
...
/* C++11 exposes a load of C99 stuff */
#if defined(__cplusplus) && __cplusplus >= 201103L
#define __LONG_LONG_SUPPORTED
#ifndef __STDC_LIMIT_MACROS
#define __STDC_LIMIT_MACROS
#endif
#ifndef __STDC_CONSTANT_MACROS
#define __STDC_CONSTANT_MACROS
#endif
#endif
...

Therefore, instead of using your patch, I changed the platform detection mechanism in IceUtil/Config.h to only use __WORDSIZE if C++11 or newer is used and fall-back to other detection mechanism in case it isn't. This should give users of the library a better experience (otherwise every custom project using the library would have to define __STDC_LIMIT_MACROS on build to get consistent results).
Comment 19 commit-hook freebsd_committer 2017-01-30 22:29:46 UTC
A commit references this bug:

Author: grembo
Date: Mon Jan 30 22:29:24 UTC 2017
New revision: 432888
URL: https://svnweb.freebsd.org/changeset/ports/432888

Log:
  Fix 64-bit platform detection for pre C++11 compilers.

  In version 3.6.3, Ice started detecting 64 bit platforms by
  checking __WORDSIZE. When using C++98/03, __STDC_LIMIT_MACROS isn't
  set by default and __WORDSIZE is always set to 32, even if the
  required headers weren't included beforehand. Until a proper
  fix is available in base (e.g. not setting __WORDSIZE at all if
  __STDC_LIMIT_MACROS isn't defined), we detect if C++11 or newer
  is used and only rely on __WORDSIZE in this case, otherwise
  we fall back to detecting the platform using other macros.

  PR:		216609
  Reported by:	jbeich

Changes:
  head/UPDATING
  head/devel/ice/Makefile
  head/devel/ice/files/patch-cpp-include-IceUtil-Config.h
Comment 20 Michael Gmelin freebsd_committer 2017-01-31 03:44:26 UTC
Created attachment 179461 [details]
poudriere build log devel/ice using clang 4.0
Comment 21 Michael Gmelin freebsd_committer 2017-01-31 03:45:42 UTC
Created attachment 179462 [details]
poudriere build log devel/py-ice using clang 4.0
Comment 22 Michael Gmelin freebsd_committer 2017-01-31 03:47:41 UTC
(In reply to Jan Beich (mail not working) from comment #3)

Tested using

$ poudriere jail -cj clang40-amd64 -v projects/clang400-import -m svn
$ poudriere bulk -Ctj clang40-amd64 devel/ice devel/py-ice

Couldn't run any unit tests manually (not surprising, as the host poudriere was running on is 10.old), but the build is ok now.

Attached logs and closing ticket.