Bug 224606 - net-im/ejabberd 17.11 compile mod_vcard_sql.erl failed on amd64 FBSD11.1Stable with libressl
Summary: net-im/ejabberd 17.11 compile mod_vcard_sql.erl failed on amd64 FBSD11.1Stabl...
Status: Closed Works As Intended
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Ashish SHUKLA
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-12-27 00:37 UTC by dewayne
Modified: 2017-12-30 06:58 UTC (History)
1 user (show)

See Also:
bugzilla: maintainer-feedback? (ashish)


Attachments
possible fix (841 bytes, patch)
2017-12-28 15:39 UTC, Ashish SHUKLA
no flags Details | Diff
weak-patch-mod_vcard.erl (1.47 KB, text/x-erlang)
2017-12-29 00:59 UTC, Walter Schwarzenfeld
no flags Details
weak-patch-mod_vcard.erl_v2 (1.05 KB, text/x-erlang)
2017-12-29 01:05 UTC, Walter Schwarzenfeld
no flags Details
pkg info at the time of ejabberd failure (18.71 KB, application/octet-stream)
2017-12-29 07:09 UTC, dewayne
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description dewayne 2017-12-27 00:37:05 UTC
# Error message with context
Compiled src/node_dag.erl
Compiled src/mod_version.erl
Compiled src/mod_vcard_xupdate.erl
src/mod_vcard_sql.erl:63: unknown type specifier 'H'
src/mod_vcard_sql.erl:195: unknown type specifier 'H'
src/mod_vcard_sql.erl:207: unknown type specifier 'H'
src/mod_vcard_sql.erl:32: function export/1 undefined
src/mod_vcard_sql.erl:32: function get_vcard/2 undefined
src/mod_vcard_sql.erl:32: function remove_user/2 undefined
Compiling src/mod_vcard_sql.erl failed:
ERROR: compile failed while processing /usr/ports/net-im/ejabberd/work/ejabberd-17.11: rebar_abort
gmake[1]: *** [Makefile:100: deps/.built] Error 1
gmake[1]: Leaving directory '/usr/ports/net-im/ejabberd/work/ejabberd-17.11'
*** Error code 1

# an extract from "grep -n H src/mod_vcard_sql.erl"
64:                " where username=%(LUser)s and %(LServer)H")) of
196:                     " where username=%(LUser)s and %(LServer)H")),
199:                     " where lusername=%(LUser)s and %(LServer)H"))
204:      fun(Host, #vcard{us = {LUser, LServer}, vcard = VCARD})
205:            when LServer == Host ->
208:                    " where username=%(LUser)s and %(LServer)H;"),
213:         (_Host, _R) ->

# uname -aKU
FreeBSD b2.hs 11.1-STABLE FreeBSD 11.1-STABLE  r327075M  amd64 1101506 1101506

Build command
# make __MAKE_CONF=/dev/null -C /usr/ports/net-im/ejabberd -DBATCH -DMAKE_JOBS_UNSAFE -DUSE_CORE2 -VCC -VCFLAGS
cc
-O2 -pipe  -I/usr/local/include -fstack-protector -fno-strict-aliasing

# erl -V
Erlang/OTP 19 [erts-8.3.5.4] [source] [64-bit] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V8.3.5.4  (abort with ^G)
Comment 1 Ashish SHUKLA freebsd_committer freebsd_triage 2017-12-27 12:33:59 UTC
(In reply to dewayne from comment #0)

I just encountered this on one of my host as well, and I think if you uninstall existing ejabberd, and then build/install ejabberd, it should build. I will try to work on some fix, but my erlang-foo is not great.

Could you see if this works for you too ?

HTH
Ashish
Comment 2 dewayne 2017-12-28 01:59:36 UTC
(In reply to Ashish SHUKLA from comment #1)
Thanks Asish for looking into this, but I built & installed on a machine that didn't have ejabberd installed.  I did try with a make clean package with different options, cflags and ldflags without positive outcome.

My usual build only enables
# make -C /usr/ports/net-im/ejabberd -DBATCH -DMAKE_JOBS_UNSAFE -DUSE_CORE2 showconfig|grep =on
     ICONV=on: Encoding conversion support via iconv
#

Naively I tried with the options 
# make -C /usr/ports/net-im/ejabberd -DBATCH -DMAKE_JOBS_UNSAFE -DUSE_CORE2 showconfig|grep =on
     ICONV=on: Encoding conversion support via iconv
     PGSQL=on: PostgreSQL database support
#

but this resulted with the same
Compiling src/mod_vcard_sql.erl failed:
ERROR: compile failed while processing /var/ports/usr/ports/net-im/ejabberd/work/ejabberd-17.11: rebar_abort
gmake[1]: *** [Makefile:100: deps/.built] Error 1
gmake[1]: Leaving directory '/var/ports/usr/ports/net-im/ejabberd/work/ejabberd-17.11'
*** Error code 1

So I poked around inside /usr/ports/net-im/ejabberd/work/ejabberd-17.11/config.status
but this looks as it should (disabling anything to do with sql)
# grep -i sql config.status
ac_cs_config="'--localstatedir=/var' '--libdir=/usr/local/lib/erlang/lib' '--disable-full-xml' '--disable-graphics' '--enable-iconv' '--disable-mysql' '--disable-odbc' '--disable-pam' '--disable-pgsql' '--disable-redis' '--disable-riak' '--disable-sip' '--disable-sqlite' '--disable-stun' '--disable-tools' '--disable-zlib' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=amd64-portbld-freebsd11.1' 'build_alias=amd64-portbld-freebsd11.1' 'CC=cc' 'CFLAGS=-O2 -pipe  -I/usr/local/include -fstack-protector -fno-strict-aliasing' 'LDFLAGS= -L/usr/local/lib -liconv -Wl,-rpath,/usr/local/lib ' 'LIBS=' 'CPPFLAGS=-I/usr/local/include'"

I think its actually a coding bug within ejabberd! :(
Comment 3 Walter Schwarzenfeld freebsd_triage 2017-12-28 06:18:23 UTC
Could not be a coding bug, compiles on 10.4 fine (in the port and with poudriere). Either it is something with 11.1-STABLe or the enviroment, or with 
lang/erlang. 
Maybe, recompile erlang helps?
Comment 4 Walter Schwarzenfeld freebsd_triage 2017-12-28 07:16:01 UTC
Btw, I am wondering about make __MAKE_CONF=/dev/null

It results on my system in a warning:
/!\ WARNING /!\

You have security/libressl installed but do not have
DEFAULT_VERSIONS+=ssl=libressl set in your make.conf
Comment 5 Ashish SHUKLA freebsd_committer freebsd_triage 2017-12-28 15:39:30 UTC
Created attachment 189167 [details]
possible fix

Patch taken from:

https://github.com/processone/ejabberd/commit/d8ace67a50a2ee68c679dc6d1e33ed8818c5df66
Comment 6 Ashish SHUKLA freebsd_committer freebsd_triage 2017-12-28 15:41:29 UTC
Hi dewayne@ w.schwarzenfeld@ --

Could you please try the diff from attachment 189167 [details] and see if it fixes the issue ?

Thanks!
Comment 7 Walter Schwarzenfeld freebsd_triage 2017-12-28 16:07:08 UTC
Cannot really test, cause there is no issue on 10.4. But compiles also with the patch.
Comment 8 dewayne 2017-12-28 23:23:06 UTC
(In reply to Ashish SHUKLA from comment #6)
Thank-you Ashish for your prompt attention to this.

I copied/applied the patch. And reran the make command.  Unfortunately the same result. :(

I checked my options for erlang, which were
# make -C /usr/ports/lang/erlang -DUSE_CORE2 showconfig|grep =on
     DOCS=on: Build and/or install documentation
     HIPE=on: Build native HiPE compiler
     KQUEUE=on: Enable Kernel Poll (kqueue) support
     OPENSSL=on: SSL/TLS support via OpenSSL
     SCTP=on: Enable SCTP support
     SMP=on: Enable SMP support
     THREADS=on: Threading support

while the defaults are
# make __MAKE_CONF=/dev/null -C /usr/ports/lang/erlang -DUSE_CORE2 showconfig | grep =on
     DOCS=on: Build and/or install documentation
     DTRACE=on: Enable DTrace support
     HIPE=on: Build native HiPE compiler
     KQUEUE=on: Enable Kernel Poll (kqueue) support
     OPENSSL=on: SSL/TLS support via OpenSSL
     SCTP=on: Enable SCTP support
     SMP=on: Enable SMP support
     THREADS=on: Threading support

Good suggestion about rebuilding, but... I rebuild all ports during an upgrade cycle.  (Wastes a lot of cycles, but maintains a consistent set of libraries and dependencies.  Largely historical from the days of library version mismatches.)

Regarding use of __MAKE_CONF=/dev/null this ensures that there is no influence from a make file, which contains many (>250) changes to port options. This pretty well ensures that there are no build greeblies creeping in.  And also explains the reason for the libressl message.  To work around this, simply place your critical make configuration settings into say /etc/make_test.conf and use "make __MAKE_CONF=/etc/make_test.conf ..."  ;)

Ashish, I rebuilt lang/erlang without ccache or anything else; and tried to rebuild net-im/ejabberd unsuccessfully.  
# make __MAKE_CONF=/dev/null -C /usr/ports/net-im/ejabberd -DBATCH -DMAKE_JOBS_UNSAFE clean package
...
Compiled src/mod_vcard_xupdate.erl
src/mod_vcard_sql.erl:63: unknown type specifier 'H'
src/mod_vcard_sql.erl:195: unknown type specifier 'H'
src/mod_vcard_sql.erl:207: unknown type specifier 'H'
src/mod_vcard_sql.erl:32: function export/1 undefined
src/mod_vcard_sql.erl:32: function get_vcard/2 undefined
src/mod_vcard_sql.erl:32: function remove_user/2 undefined
Compiling src/mod_vcard_sql.erl failed:
ERROR: compile failed while processing /var/ports/usr/ports/net-im/ejabberd/work/ejabberd-17.11: rebar_abort

I think its time I learnt something about rebar.

The key environmental difference is that I use binutils 2.29,1 - but I fail to see how that would make an impact here. :(
Comment 9 Walter Schwarzenfeld freebsd_triage 2017-12-29 00:59:29 UTC
Created attachment 189187 [details]
weak-patch-mod_vcard.erl
Comment 10 Walter Schwarzenfeld freebsd_triage 2017-12-29 01:00:09 UTC
It's a weak idea. But try this patch instead the first.
Comment 11 Walter Schwarzenfeld freebsd_triage 2017-12-29 01:05:53 UTC
Created attachment 189188 [details]
weak-patch-mod_vcard.erl_v2

Was a typo in int.
Comment 12 Walter Schwarzenfeld freebsd_triage 2017-12-29 02:36:58 UTC
Comment on attachment 189188 [details]
weak-patch-mod_vcard.erl_v2

No, was wrong.
Comment 13 Ashish SHUKLA freebsd_committer freebsd_triage 2017-12-29 05:29:06 UTC
The error comes from ejabberd_sql_pt module:

324 parse_name([$), T | S], Acc, 0, IsArg, State) ->
325     Type =
326         case T of
327             $d -> integer;
328             $s -> string;
329             $b -> boolean;
330             $H when IsArg -> host;
331             _ ->
332                 throw({error, State#state.loc,
333                        ["unknown type specifier '", T, "'"]})
334         end,
335     {lists:reverse(Acc), Type, S, State};

which is used in compilation for certain ejabberd modules (including mod_vcard_sql):

 25 -module(mod_vcard_sql).
 26
 27 -compile([{parse_transform, ejabberd_sql_pt}]).
 28
Comment 14 Ashish SHUKLA freebsd_committer freebsd_triage 2017-12-29 06:07:32 UTC
(In reply to dewayne from comment #0)

Could you please also provide the list of packages installed (output of: pkg info) on the host ? I tried your instructions on 11.1-RELEASE (amd64) host, but unable to reproduce.
Comment 15 dewayne 2017-12-29 07:09:59 UTC
Created attachment 189193 [details]
pkg info at the time of ejabberd failure

Per your request, I've included the list of ports at the time of ejabbered failure. Please note that all ports are built during the same build sequence.
Comment 16 dewayne 2017-12-29 07:58:12 UTC
(In reply to dewayne from comment #15)
Please do not expend any further effort on this PR.  I've just gotten ejabberd to build on b2.  I'll investigate the cause of the problem and advise within 24 hours.  (Its nearly 7pm, I'm GMT+11)  

I strongly suspect a compiler directive during the erlang build as the culprit (or, and less likely, a corrupt ccache?).
Comment 17 Ashish SHUKLA freebsd_committer freebsd_triage 2017-12-29 14:28:11 UTC
(In reply to dewayne from comment #16)

I have a feeling it's one of the erlang-* ports, probably erlang-parse_trans, interfering with the build.

Waiting to hear more from you on how you got ejabberd to build on b2.

Happy weekend!
Comment 18 dewayne 2017-12-30 00:03:54 UTC
(In reply to Ashish SHUKLA from comment #17)
When I build applications, I define CFLAGS and LDFLAGS at the leaf node of the tree.  So for ejabberd I use (respectively):

-O2 -pipe -Wl,-m,elf_x86_64_fbsd -DOPENSSL_NO_SSL2 -DOPENSSL_NO_SSL3 -g0 -ggdb0 -DSTRIP_FBSDID -UDEBUGGING -UDEBUG -fno-math-errno -Wl,--hash-style=sysv -Wno-write-strings -Wno-error=unused-command-line-argument -Wno-ignored-optimization-argument -Wno-error=macro-redefined -fPIE -fPIC -fomit-frame-pointer -Wl,-z,relro -Wl,-z,now -Wl,-z,noexecstack -march=core2  -I/usr/local/include -fstack-protector-strong -fno-strict-aliasing

-pie -Wl,--strip-debug -Wl,--build-id=uuid -z relro -z now -z noexecstack -L/usr/local/lib -liconv -Wl,-rpath,/usr/local/lib

Unfortunately these were passed down to ejabberd's dependency lang/erlang.  Erlang being similar to PROLOG (which I use extensively) requires executable stack. This is where the problem lay.  To build ejabberd, erlang needs to behave correctly; which it does if we remove the noexecstack directive to the compiler.

Note: only binutils 2.29,1 (not in ports tree) supports the --strip-debug and --build-id=uuid features.

My sincere apologies for not tweaking to this (much) earlier. I will be more diligent.
Kind regards, Dewayne.
Comment 19 commit-hook freebsd_committer freebsd_triage 2017-12-30 06:58:56 UTC
A commit references this bug:

Author: ashish
Date: Sat Dec 30 06:58:29 UTC 2017
New revision: 457588
URL: https://svnweb.freebsd.org/changeset/ports/457588

Log:
  - Add a note to UPDATING about recently discovered ejabberd build issue

  PR:		224606
  Reported by:	dewayne@heuristicsystems.com.au

Changes:
  head/UPDATING