Bug 203327

Summary: graphics/openjpeg: latest changes (from 2.1.0_1 to 2.1.1) break building of mupdf (and possibly other ports)
Product: Ports & Packages Reporter: dennis.noordsij
Component: Individual Port(s)Assignee: Po-Chuan Hsieh <sunpoet>
Status: Closed FIXED    
Severity: Affects Only Me CC: uzsolt, w.litter, wjenkner
Priority: --- Flags: bugzilla: maintainer-feedback? (sunpoet)
Version: Latest   
Hardware: Any   
OS: Any   
Attachments:
Description Flags
patch-load-jpx.c
none
Simplify last change none

Description dennis.noordsij 2015-09-25 09:18:29 UTC 
One of the recent changes to graphics/openjpeg causes graphics/mupdf to no longer build, with the following errors:

<.. snip ..>
cc  -fstack-protector -o build/release/mujstest build/release/platform/x11/jstest_main.o build/release/platform/x11/pdfapp.o build/release/libmupdf.a -L/usr/local/lib  -lm -L/usr/local/lib -lfreetype   -ljbig2dec -ljpeg -L/usr/local/lib -lopenjp2    -lz
build/release/libmupdf.a(load-jpx.o): In function `fz_load_jpx':
source/fitz/load-jpx.c:(.text+0x56): undefined reference to `opj_set_default_decoder_parameters'
source/fitz/load-jpx.c:(.text+0x69): undefined reference to `opj_create_decompress'
source/fitz/load-jpx.c:(.text+0x84): undefined reference to `opj_set_info_handler'
source/fitz/load-jpx.c:(.text+0x94): undefined reference to `opj_set_warning_handler'
source/fitz/load-jpx.c:(.text+0xa4): undefined reference to `opj_set_error_handler'
source/fitz/load-jpx.c:(.text+0xb1): undefined reference to `opj_setup_decoder'
source/fitz/load-jpx.c:(.text+0xc3): undefined reference to `opj_stream_default_create'
source/fitz/load-jpx.c:(.text+0xe5): undefined reference to `opj_stream_set_read_function'
source/fitz/load-jpx.c:(.text+0xf2): undefined reference to `opj_stream_set_skip_function'
source/fitz/load-jpx.c:(.text+0xff): undefined reference to `opj_stream_set_seek_function'
source/fitz/load-jpx.c:(.text+0x10e): undefined reference to `opj_stream_set_user_data'
source/fitz/load-jpx.c:(.text+0x119): undefined reference to `opj_stream_set_user_data_length'
source/fitz/load-jpx.c:(.text+0x129): undefined reference to `opj_read_header'
source/fitz/load-jpx.c:(.text+0x141): undefined reference to `opj_decode'
source/fitz/load-jpx.c:(.text+0x14b): undefined reference to `opj_stream_destroy'
source/fitz/load-jpx.c:(.text+0x153): undefined reference to `opj_destroy_codec'
source/fitz/load-jpx.c:(.text+0x42d): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x4e1): undefined reference to `opj_destroy_codec'
source/fitz/load-jpx.c:(.text+0x4f8): undefined reference to `opj_stream_destroy'
source/fitz/load-jpx.c:(.text+0x500): undefined reference to `opj_destroy_codec'
source/fitz/load-jpx.c:(.text+0x514): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x534): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x548): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x55c): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x570): undefined reference to `opj_image_destroy'
build/release/libmupdf.a(load-jpx.o):source/fitz/load-jpx.c:(.text+0x593): more undefined references to `opj_image_destroy' follow
build/release/libmupdf.a(load-jpx.o)build/release/libmupdf.a(load-jpx.o): In function `fz_load_jpx':
source/fitz/load-jpx.c:(.text+0x56): undefined reference to `opj_set_default_decoder_parameters'
source/fitz/load-jpx.c:(.text+0x69): undefined reference to `opj_create_decompress'
source/fitz/load-jpx.c:(.text+0x84): undefined reference to `opj_set_info_handler'
source/fitz/load-jpx.c:(.text+0x94): undefined reference to `opj_set_warning_handler'
source/fitz/load-jpx.c:(.text+0xa4): undefined reference to `opj_set_error_handler'
source/fitz/load-jpx.c:(.text+0xb1): undefined reference to `opj_setup_decoder'
source/fitz/load-jpx.c:(.text+0xc3): undefined reference to `opj_stream_default_create'
source/fitz/load-jpx.c:(.text+0xe5): undefined reference to `opj_stream_set_read_function'
source/fitz/load-jpx.c:(.text+0xf2): undefined reference to `opj_stream_set_skip_function'
source/fitz/load-jpx.c:(.text+0x: In function `fz_load_jpx':
source/fitz/load-jpx.c:(.text+0x56): undefined reference to `opj_set_default_decoder_parameters'
source/fitz/load-jpx.c:(.text+0x69): undefined reference to `opj_create_decompress'
source/fitz/load-jpx.c:(.text+0x84): undefined reference to `opj_set_info_handler'
source/fitz/load-jpx.c:(.text+0x94): undefined reference to `opj_set_warning_handler'
source/fitz/load-jpx.c:(.text+0xa4): undefined reference to `opj_set_error_handler'
source/fitz/load-jpx.c:(.text+0xb1): undefined reference to `opj_setup_decoder'
source/fitz/load-jpx.c:(.text+0xc3): undefined reference to `opj_stream_default_create'
source/fitz/load-jpx.c:(.text+0xe5): undefined reference to `opj_stream_set_read_function'
source/fitz/load-jpx.c:(.text+0xf2): undefined reference to `opj_stream_set_skip_function'
source/fitz/load-jpx.c:(.text+0xff): undefined reference to `opj_stream_set_seek_function'
source/fitz/load-jpx.c:(.text+0x10e): undefined reference to `opj_stream_set_user_data'
source/fitz/load-jpx.c:(.text+0x119): undefined reference to `opj_stream_set_user_data_length'
source/fitz/load-jpx.c:(.text+0x129): undefined reference to `opj_read_header'
source/fitz/load-jpx.c:(.text+0x141): undefined reference to `opj_decode'
source/fitz/load-jpx.c:(.text+0x14b): undefined reference to `opj_stream_destroy'
source/fitz/load-jpx.c:(.text+0x153): undefined reference to `opj_destroy_codec'ff
): undsource/fitz/load-jpx.c:e(f.texti+n0exd42d )r:e fuenrednecfei nteod  `refopj_stream_set_seek_functione'r
encesource/fitz/load-jpx.c: (t.texto+ 0`x10eopj_image_destroy)':
 undefinsource/fitz/load-jpx.c:e(d.text +r0x4e1): undefined reference to `opj_destroy_codec'
source/fitz/load-jpx.c:(.text+0x4f8): undefined reference to `opj_stream_destroy'
eference to `opj_stream_set_user_data'
source/fitz/load-jpx.c:(.text+0x119): undefined reference to `opj_stream_set_user_data_length'
source/fitz/load-jpx.c:(.text+0x129): undefined reference to `opj_read_header'
source/fitz/load-jpx.c:(.text+0x141): undefined reference to `opj_decode'
source/fitz/load-jpx.c:(.text+0x14b): undefined reference to `opj_stream_destroy'
source/fitz/load-jpx.c:(.text+0x153): undefined reference to `opj_destroy_codec'
source/fitz/load-jpx.c:(.text+0x42d): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x4e1): undefined reference to `opj_destroy_codec'
source/fitz/load-jpx.c:(.text+0x4f8): undefined reference to `opj_stream_destroy'
source/fitz/load-jpx.c:(.text+0x500): undefined reference to `opj_destroy_codec'
source/fitz/load-jpx.c:(.text+0x514): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x534): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x548): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x55c): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x570): undefined reference to `opj_image_destroy'
build/release/libmupdf.a(load-jpx.o):source/fitz/load-jpx.c:(.text+0x593): more undefined references to `opj_image_destroy' follow
source/fitz/load-jpx.c:(.text+0x500): undefined reference to `opj_destroy_codec'
source/fitz/load-jpx.c:(.text+0x514): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x534): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x548): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x55c): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x570): undefined reference to `opj_image_destroy'
build/release/libmupdf.a(load-jpx.o):source/fitz/load-jpx.c:(.text+0x593): more undefined references to `opj_image_destroy' follow
/usr/bin/ld: build/release/mutool: hidden symbol `opj_setup_decoder' isn't defined
/usr/bin/ld: final link failed: Nonrepresentable section on output
/usr/bin/ld: build/release/mujstest: hidden symbol `opj_setup_decoder' isn't defined
/usr/bin/ld: final link failed: Nonrepresentable section on output
/usr/bin/ld: build/release/mudraw: hidden symbol `opj_setup_decoder' isn't defined
/usr/bin/ld: final link failed: Nonrepresentable section on output
cc: error: linker command failed with exit code 1 (use -v to see invocation)
Makefile:248: recipe for target 'build/release/mutool' failed
gmake[2]: *** [build/release/mutool] Error 1
gmake[2]: *** Waiting for unfinished jobs....
cc: error: linker command failed with exit code 1 (use -v to see invocation)
cc: error: linker command failed with exit code 1 (use -v to see invocation)
Makefile:260: recipe for target 'build/release/mujstest' failed
gmake[2]: *** [build/release/mujstest] Error 1
Makefile:241: recipe for target 'build/release/mudraw' failed
gmake[2]: *** [build/release/mudraw] Error 1
build/release/libmupdf.a(load-jpx.o): In function `fz_load_jpx':
source/fitz/load-jpx.c:(.text+0x56): undefined reference to `opj_set_default_decoder_parameters'
source/fitz/load-jpx.c:(.text+0x69): undefined reference to `opj_create_decompress'
source/fitz/load-jpx.c:(.text+0x84): undefined reference to `opj_set_info_handler'
source/fitz/load-jpx.c:(.text+0x94): undefined reference to `opj_set_warning_handler'
source/fitz/load-jpx.c:(.text+0xa4): undefined reference to `opj_set_error_handler'
source/fitz/load-jpx.c:(.text+0xb1): undefined reference to `opj_setup_decoder'
source/fitz/load-jpx.c:(.text+0xc3): undefined reference to `opj_stream_default_create'
source/fitz/load-jpx.c:(.text+0xe5): undefined reference to `opj_stream_set_read_function'
source/fitz/load-jpx.c:(.text+0xf2): undefined reference to `opj_stream_set_skip_function'
source/fitz/load-jpx.c:(.text+0xff): undefined reference to `opj_stream_set_seek_function'
source/fitz/load-jpx.c:(.text+0x10e): undefined reference to `opj_stream_set_user_data'
source/fitz/load-jpx.c:(.text+0x119): undefined reference to `opj_stream_set_user_data_length'
source/fitz/load-jpx.c:(.text+0x129): undefined reference to `opj_read_header'
source/fitz/load-jpx.c:(.text+0x141): undefined reference to `opj_decode'
source/fitz/load-jpx.c:(.text+0x14b): undefined reference to `opj_stream_destroy'
source/fitz/load-jpx.c:(.text+0x153): undefined reference to `opj_destroy_codec'
source/fitz/load-jpx.c:(.text+0x42d): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x4e1): undefined reference to `opj_destroy_codec'
source/fitz/load-jpx.c:(.text+0x4f8): undefined reference to `opj_stream_destroy'
source/fitz/load-jpx.c:(.text+0x500): undefined reference to `opj_destroy_codec'
source/fitz/load-jpx.c:(.text+0x514): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x534): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x548): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x55c): undefined reference to `opj_image_destroy'
source/fitz/load-jpx.c:(.text+0x570): undefined reference to `opj_image_destroy'
build/release/libmupdf.a(load-jpx.o):source/fitz/load-jpx.c:(.text+0x593): more undefined references to `opj_image_destroy' follow
/usr/bin/ld: build/release/mupdf-x11: hidden symbol `opj_setup_decoder' isn't defined
/usr/bin/ld: final link failed: Nonrepresentable section on output
cc: error: linker command failed with exit code 1 (use -v to see invocation)
Makefile:268: recipe for target 'build/release/mupdf-x11' failed
gmake[2]: *** [build/release/mupdf-x11] Error 1
gmake[2]: Leaving directory '/usr/ports/graphics/mupdf/work/mupdf-1.7-source'
===> Compilation failed unexpectedly.



If I install the openjpeg binary pkg version (2.1.0_1) mupdf builds fine. If I install openjpeg from ports (version 2.1.1), the problem happens. (Tried in a poudriere box, jails, and vanilla FreeBSD, all on up to date 10.2-RELEASE).

I don't really know where to look for the problem, usually undefined reference means the library is not installed or can not be found, but that is not the case here. 


(Portsnap up to date as of  Fri Sep 25 11:13:39 CEST 2015)


Happy to provide more information
Comment 1 dennis.noordsij 2015-09-25 21:04:31 UTC 
Using git bisect the problem was introduced in:

  commit 3ed5858902055d3500a6ab183f1395686921d026
  Date:   Sun Jan 25 16:59:39 2015 +0000  

  [trunk] use hidden visibility for static library (fixes issue 354)


which references the following issue:
  https://github.com/uclouvain/openjpeg/issues/354


mupdf defines OPJ_STATIC, compiles static libraries libmupdf.a which contain openjp2, etc, and then links these into binaries (mutool, etc).

The same problem happens on a Linux distribution with the gcc toolchain.

I don't know what the correct place is to fix this.

Workarounds:
 - remove the visibility ("hidden") in include/openjp2/openjpeg.h  (basically partially undo the above commit)
 - (untested) do not define OPJ_STATIC in mupdf (but according to comments in load-jpx.c this breaks windows compilation)
 - do not remove the thirdparty/ folder in the mupdf sources and let it statically link against those instead of the system libraries (mupdf already needs a patch to be able to use the system openjpeg)
 - ifdef hacks


The easiest workaround for others who run into this:

--- usr/ports/graphics/mupdf/Makefile.old 2015-09-25 22:58:18.963166563 +0200
+++ usr/ports/graphics/mupdf/Makefile     2015-09-25 22:58:25.875451670 +0200
@@ -51,7 +51,6 @@
                ${WRKSRC}/Makerules
        @${REINPLACE_CMD} '/^CFLAGS/s|$$| -I${LOCALBASE}/include|' \
                ${WRKSRC}/Makefile
-       @${RM} -r ${WRKSRC}/thirdparty/*
 
 post-build:
        (cd ${WRKSRC} && ${SETENV} ${MAKE_ENV} ${MAKE_CMD} ${MAKE_FLAGS} \
Comment 2 Walter Schwarzenfeld 2015-09-27 08:28:01 UTC 
Created attachment 161454 [details]
patch-load-jpx.c
Comment 3 Walter Schwarzenfeld 2015-09-27 08:28:35 UTC 
This patch should fix it.
Comment 4 Walter Schwarzenfeld 2015-09-27 08:33:29 UTC 
May interesting: I the last time I found different errors in some ports, where the solution was the same (extern c). Something seems to went wrong (maybe, only with older ports).
Comment 5 Po-Chuan Hsieh freebsd_committer freebsd_triage 2015-09-28 07:57:25 UTC 
Add graphics/mupdf maintainer (udvzsolt@gmail.com) to CC List.
Comment 6 Zsolt Udvari freebsd_committer freebsd_triage 2015-09-28 08:40:32 UTC 
If Walter's patch fix the build I'll accept. Now I don't have time to test.
Comment 7 wjenkner 2015-09-28 15:43:50 UTC 
(In reply to dennis.noordsij from comment #1)
>  - (untested) do not define OPJ_STATIC in mupdf (but according to comments in load-jpx.c this breaks windows compilation)

Does it make sense at all to define OPJ_STATIC in load-jpx.c when linking with a libopenjp2.so which wasn't actually compiled with the kind of symbol visibility implied by that switch (by default, mupdf compiles its own private static versions of this and other libraries, see Makethird)?

If not, your suggestion above might be the best solution.
Comment 8 commit-hook freebsd_committer freebsd_triage 2015-10-04 18:02:47 UTC 
A commit references this bug:

Author: sunpoet
Date: Sun Oct  4 18:01:54 UTC 2015
New revision: 398615
URL: https://svnweb.freebsd.org/changeset/ports/398615

Log:
  - Fix build with openjpeg 2.1.1
  - Bump PORTREVISION
  - While I'm here:
    - Update MASTER_SITES: make 1.7 fetchable
    - Use DISTVERSIONSUFFIX instead of EXTRACT_SUFX
    - Convert to new options helper

  PR:		203327
  Submitted by:	Walter Schwarzenfeld <w.litter@aon.at>
  Approved by:	Zsolt Udvari <udvzsolt@gmail.com> (maintainer)

Changes:
  head/graphics/mupdf/Makefile
  head/graphics/mupdf/files/patch-source__fitz__load-jpx.c
Comment 9 Po-Chuan Hsieh freebsd_committer freebsd_triage 2015-10-04 18:04:40 UTC 
Committed. Thanks!
Comment 10 wjenkner 2015-10-05 14:12:56 UTC 
(In reply to Sunpoet Po-Chuan Hsieh from comment #9)
As I tried to convey in my comment above, I think this fix is wrong; it happens to work because load-jpx.c is compiled as a .c file by plain cc (not c++) and so the #ifdef __cplusplus ...#endif is just a fancy way to comment out the stuff in between.  IIUC, the link error happens because of symbol visibility, not because of symbol mangling (neither the symbols in load-jpx.o nor in libopenjp2.so are mangled).
Comment 11 Walter Schwarzenfeld 2015-10-05 16:35:49 UTC 
o..O sorry, I make nonsense and nearly nobody see it.
The wrong header is included in load-jpx.c. Change it to #include <openjpeg-2.1/openjpeg.h>.
And the there is another error:
source/fitz/load-jpx.c:120:38: error: too few arguments to function call, expected 3, have 2
        opj_stream_set_user_data(stream, &sb);

I am not sure, but I think the line 120 should be:
opj_stream_set_user_data(stream, sb.data, &sb);
Comment 12 Walter Schwarzenfeld 2015-10-05 16:41:47 UTC 
Thank you @wjenkner
Comment 13 Walter Schwarzenfeld 2015-10-05 17:32:52 UTC 
I correct the line 120 should be
opj_stream_set_user_data(stream, &sb, NULL);
Comment 14 dennis.noordsij 2015-10-05 17:35:21 UTC 
(In reply to Walter Schwarzenfeld from comment #11)

These 2 differences were already handled by files/patch-source__fitz__load-jpx.c of graphics/mupdf, and also in the new patch? Where do you run into problems?

What is the consensus on the proper fix? I agree with the reasoning of #10.
Comment 15 Walter Schwarzenfeld 2015-10-05 18:34:18 UTC 
You are  right, the change of the include is in the patch.
Comment 16 wjenkner 2015-10-06 15:43:43 UTC 
(In reply to dennis.noordsij from comment #14)
I think deleting the OPJ_STATIC definition in load-lpx.c is fine.

Effectively, this change is a no-op, but the point is to supply a (hopefully) correct explanation of the issue, see the attached patch.
Comment 17 wjenkner 2015-10-06 15:48:26 UTC 
Created attachment 161766 [details]
Simplify last change
Comment 18 Walter Schwarzenfeld 2015-10-06 16:04:57 UTC 
You forget the change I posted above in line 120 - in your attachment l think it is line 117.
Comment 19 commit-hook freebsd_committer freebsd_triage 2015-10-09 18:33:40 UTC 
A commit references this bug:

Author: sunpoet
Date: Fri Oct  9 18:33:09 UTC 2015
New revision: 398949
URL: https://svnweb.freebsd.org/changeset/ports/398949

Log:
  - Fix patch

  PR:		203327
  Submitted by:	Wolfgang Jenkner <wjenkner@inode.at>
  Approved by:	Zsolt Udvari <udvzsolt@gmail.com> (maintainer, implicit)

Changes:
  head/graphics/mupdf/files/patch-source__fitz__load-jpx.c