Bug 257051 - editors/neovim: neovim-0.5.0 fails to link if luajit is not enabled
Summary: editors/neovim: neovim-0.5.0 fails to link if luajit is not enabled
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Adam Weinberger
URL: https://reviews.freebsd.org/D36947
Keywords:
Depends on:
Blocks:
 
Reported: 2021-07-08 09:46 UTC by snafu
Modified: 2022-10-16 15:52 UTC (History)
6 users (show)

See Also:
linimon: maintainer-feedback? (adamw)


Attachments
Switch to luajit-openresty (2.77 KB, patch)
2021-07-21 22:01 UTC, Adam Weinberger
no flags Details | Diff
poudriere fail (359.68 KB, text/plain)
2021-07-27 06:57 UTC, Dave Cottlehuber
no flags Details
v0 (2.00 KB, patch)
2021-10-25 14:36 UTC, Mikael Urankar
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description snafu 2021-07-08 09:46:36 UTC
While first looking similar to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=251818 it seems to be different as I have build libluv without luajit.

Afaict it is related to rds parser which seems to be built against lua51 (statically and without option to change that) and the build trying to use lua52.

The relevant error messages seem to be:

ld: error: undefined symbol: luaL_register
>>> referenced by executor.c
>>>               lto.tmp:(nlua_enter.llvm.6927564360673786163)
>>> referenced by treesitter.c
>>>               lto.tmp:(tslua_init)
>>> referenced by treesitter.c
>>>               lto.tmp:(tslua_init)
>>> referenced 4 more times

ld: error: undefined symbol: lua_setfenv
>>> referenced by treesitter.c
>>>               lto.tmp:(push_tree)
>>> referenced by treesitter.c
>>>               lto.tmp:(parser_parse)
>>> referenced by treesitter.c
>>>               lto.tmp:(tree_root)
>>> referenced 11 more times

ld: error: undefined symbol: lua_objlen
>>> referenced by treesitter.c
>>>               lto.tmp:(parser_set_ranges)
>>> referenced by treesitter.c
>>>               lto.tmp:(parser_set_ranges)

ld: error: undefined symbol: lua_getfenv
>>> referenced by treesitter.c
>>>               lto.tmp:(tree_root)
>>> referenced by treesitter.c
>>>               lto.tmp:(node_field)
>>> referenced by treesitter.c
>>>               lto.tmp:(node_child)
>>> referenced 8 more times
cc: error: linker command failed with exit code 1 (use -v to see invocation)
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2021-07-09 12:00:57 UTC
^Triage: give complete Summary and assign.
Comment 2 Adam Weinberger freebsd_committer freebsd_triage 2021-07-21 21:53:54 UTC
(Adding in luv's maintainer)

Sorry about the long silence, IRL happened.

Thanks for the sleuthing here. Simply switching neovim (and luv) to lua:51 isn't enough to fix the build.

lua51: error loading module 'mpack' from file './mpack.so':
        ./mpack.so: Undefined symbol "luaL_setfuncs"
stack traceback:
        [C]: ?
        [C]: in function 'require'
        ...eovim-0.5.0/src/nvim/generators/gen_api_dispatch.lua:1: in main chunk
        [C]: ?

lua51: error loading module 'mpack' from file './mpack.so':
        ./mpack.so: Undefined symbol "luaL_setfuncs"
stack traceback:
        [C]: ?
        [C]: in function 'require'
        ...ovim-0.5.0/src/nvim/generators/gen_api_ui_events.lua:1: in main chunk
        [C]: ?

I know this is a bit of a sledgehammer fix, but what if we dropped the lua option entirely and switched neovim (and luv) to luajit-openresty? It's a drop-in replacement, and the Openresty fork should work nearly everywhere that luajit doesn't. The only casualty of this would be riscv64.

Neovim pushes luajit pretty hard and seems to make no effort or guarantee of PUC lua compatibility. Is there a strong use case for PUC lua here?
Comment 3 Adam Weinberger freebsd_committer freebsd_triage 2021-07-21 22:01:43 UTC
Created attachment 226597 [details]
Switch to luajit-openresty

I'm attaching a patch that removes the LUAJIT option (essentially disabling PUC lua support) and switches to luajit-openresty.
Comment 4 Dave Cottlehuber freebsd_committer freebsd_triage 2021-07-27 06:57:07 UTC
Created attachment 226733 [details]
poudriere fail

thanks for the patch; this doesn't link yet on arm64 13.0-RELEASE against latest ports tree.


ld: error: /usr/local/lib/libluv.so: undefined reference to lua_absindex [--no-allow-shlib-undefined]
ld: error: /usr/local/lib/libluv.so: undefined reference to lua_callk [--no-allow-shlib-undefined]
ld: error: /usr/local/lib/libluv.so: undefined reference to lua_getglobal [--no-allow-shlib-undefined]
ld: error: /usr/local/lib/libluv.so: undefined reference to lua_pcallk [--no-allow-shlib-undefined]
ld: error: /usr/local/lib/libluv.so: undefined reference to lua_rawlen [--no-allow-shlib-undefined]
cc: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
*** Error code 1

full log attached.
Comment 5 commit-hook freebsd_committer freebsd_triage 2021-09-26 11:32:41 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=ad4a6795e8d5177f1767802e58deb2a6ecc86122

commit ad4a6795e8d5177f1767802e58deb2a6ecc86122
Author:     Mikael Urankar <mikael@FreeBSD.org>
AuthorDate: 2021-09-26 11:31:09 +0000
Commit:     Mikael Urankar <mikael@FreeBSD.org>
CommitDate: 2021-09-26 11:32:27 +0000

    devel/luv: add luajit-openresty by default on aarch64

    PR:             257051
    Approved by:    portmgr (runtime fix blanket)

 devel/luv/Makefile | 2 ++
 1 file changed, 2 insertions(+)
Comment 6 Mikael Urankar freebsd_committer freebsd_triage 2021-09-26 11:34:23 UTC
(In reply to Dave Cottlehuber from comment #4)
It should be fixed with ports rad4a6795e8d5177f1767802e58deb2a6ecc86122
 https://cgit.FreeBSD.org/ports/commit/?id=ad4a6795e8d5177f1767802e58deb2a6ecc86122

and the switch to luajit-openresty
Comment 7 Dave Cottlehuber freebsd_committer freebsd_triage 2021-09-30 08:20:52 UTC
+1 please commit!

with aa0743dcc2fd9dc20e899887547ef9ca2846515b (devel/luv: un-break build on aarch64) and this patch, neovim-0.5 builds & runs fine on aarch64.
Comment 8 Adam Weinberger freebsd_committer freebsd_triage 2021-09-30 23:39:59 UTC
(In reply to Dave Cottlehuber from comment #7)
Is there any reason to keep PUC lua as an option? Or is it safe (enough) to just depend on luajit-openresty unconditionally?
Comment 9 Dave Cottlehuber freebsd_committer freebsd_triage 2021-10-01 21:07:45 UTC
honestly I can't comment on that, other than I didn't get this building with new patches on amd64, no time yet to look further into it sorry.
Comment 10 Mikael Urankar freebsd_committer freebsd_triage 2021-10-10 13:21:06 UTC
(In reply to Adam Weinberger from comment #8)
I think it's safe to depend on luajit-openresty unconditionally.
Comment 11 tech-lists 2021-10-24 14:15:46 UTC
Hi,

This still fails to build on stable/13 with todays portstree in poudriere testport. It's failing in lua but it's not the exact same error. I've put the full build log from testport here:

https://cloud.zyxst.net/~john/FreeBSD/ports/neovim/neovim-0.5.1.log

This is on aarch64 hardware. I've re-fetched the ports tree after deleting it and removed editors/neovim options. The snippet where it fails:

[...]
[00:04:41] ld: error: undefined symbol: luaL_register
[00:04:41] >>> referenced by executor.c
[00:04:41] >>>               lto.tmp:(nlua_enter.llvm.12998269152104470128)
[00:04:41] >>> referenced by treesitter.c
[00:04:41] >>>               lto.tmp:(tslua_init)
[00:04:41] >>> referenced by treesitter.c
[00:04:41] >>>               lto.tmp:(tslua_init)
[00:04:41] >>> referenced 4 more times
[00:04:41] 
[00:04:41] ld: error: undefined symbol: lua_setfenv
[00:04:41] >>> referenced by treesitter.c
[00:04:41] >>>               lto.tmp:(push_tree)
[00:04:41] >>> referenced by treesitter.c
[00:04:41] >>>               lto.tmp:(parser_parse)
[00:04:41] >>> referenced by treesitter.c
[00:04:41] >>>               lto.tmp:(tree_root)
[00:04:41] >>> referenced 11 more times
[00:04:41] 
[00:04:41] ld: error: undefined symbol: lua_objlen
[00:04:41] >>> referenced by treesitter.c
[00:04:41] >>>               lto.tmp:(parser_set_ranges)
[00:04:41] >>> referenced by treesitter.c
[00:04:41] >>>               lto.tmp:(parser_set_ranges)
[00:04:41] 
[00:04:41] ld: error: undefined symbol: lua_getfenv
[00:04:41] >>> referenced by treesitter.c
[00:04:41] >>>               lto.tmp:(tree_root)
[00:04:41] >>> referenced by treesitter.c
[00:04:41] >>>               lto.tmp:(node_field)
[00:04:41] >>> referenced by treesitter.c
[00:04:41] >>>               lto.tmp:(node_child)
[00:04:41] >>> referenced 8 more times
[00:04:41] 
[00:04:41] ld: error: /usr/local/lib/libluv.so: undefined reference to luaL_loadbuffer [--no-allow-shlib-undefined]
[00:04:41] ld: error: /usr/local/lib/libluv.so: undefined reference to lua_call [--no-allow-shlib-undefined]
[00:04:41] ld: error: /usr/local/lib/libluv.so: undefined reference to lua_pcall [--no-allow-shlib-undefined]
[00:04:41] ld: error: /usr/local/lib/libluv.so: undefined reference to lua_tointeger [--no-allow-shlib-undefined]
[00:04:41] ld: error: /usr/local/lib/libluv.so: undefined reference to lua_tonumber [--no-allow-shlib-undefined]
[00:04:41] cc: error: linker command failed with exit code 1 (use -v to see invocation)
[00:04:41] ninja: build stopped: subcommand failed.
Comment 12 Mikael Urankar freebsd_committer freebsd_triage 2021-10-25 12:15:38 UTC
(In reply to tech-lists from comment #11)
It fails even with the patch at 'attachment 226597 [details]' ?
Comment 13 tech-lists 2021-10-25 13:19:56 UTC
(In reply to Mikael Urankar from comment #12)
I've not tried that - thought the changes needed would be in the ports tree as per comments #6 and #7

I'll try the patch
Comment 14 tech-lists 2021-10-25 13:50:26 UTC
OK,

For testing purposes, I've made an arm64.aarch64 poudriere jail on a faster amd64 machine. My initial comment was in an arm64.aarch64 hardware context. The following is from a poudriere on amd64 with a jail compiled with -x and -a arm64.aarch64

I've downloaded a ports tree just for this jail. The ports tree I've called arm64. This was downloaded with git+https.

Then grabbed the patch like this, to apply from the top level of the arm64 ports tree:

wget -O patch1.txt "https://bugs.freebsd.org/bugzilla/attachment.cgi?id=226597&action=diff&collapsed=&headers=1&format=raw"

then applied the patch like this:

root@REDACTED:/usr/local/poudriere/ports/arm64# patch < patch1.txt
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- b/devel/luv/Makefile
|+++ b/devel/luv/Makefile
--------------------------
Patching file Makefile using Plan A...
Hunk #1 failed at 1.
Hunk #2 failed at 25.
2 out of 2 hunks failed--saving rejects to Makefile.rej
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- b/editors/neovim/Makefile
|+++ b/editors/neovim/Makefile
--------------------------
Patching file Makefile using Plan A...
Hunk #1 failed at 3.
Hunk #2 failed at 14.
Hunk #3 failed at 27.
Hunk #4 failed at 40.
Hunk #5 failed at 53.
5 out of 5 hunks failed--saving rejects to Makefile.rej
done
root@REDACTED:/usr/local/poudriere/ports/arm64#
Comment 15 Mikael Urankar freebsd_committer freebsd_triage 2021-10-25 14:36:59 UTC
Created attachment 229024 [details]
v0

Try this patch instead.
Comment 16 tech-lists 2021-10-25 15:19:50 UTC
Hi,

OK that patch applied. 

What's changed now is, to be on the safe side, I deleted the recently downloaded ports tree and fetched a pristine one. Then downloaded the patch with wget to the top of the ports tree, where it failed to apply. Copied the patch to editors/neovim and it applied successfully.

Running a build now
Comment 17 tech-lists 2021-10-25 15:27:21 UTC
(In reply to Mikael Urankar from comment #15)

That patch allowed neovim to build. Thank you!
Comment 18 Jeroen Ruigrok van der Werven freebsd_committer freebsd_triage 2022-04-27 07:26:38 UTC
What's the status of this? It's causing problems when trying to upgrade either rspamd or neovim due to the conflicting dependency they have on either luajit or luajit-openresty.
Comment 19 Adam Weinberger freebsd_committer freebsd_triage 2022-10-12 13:01:19 UTC
In https://reviews.freebsd.org/D36947 I propose a DEFAULT_VERSIONS for luajit. The default should probably become luajit-devel. Hopefully this can fix this issue properly once and for all.
Comment 20 Adam Weinberger freebsd_committer freebsd_triage 2022-10-14 22:35:22 UTC
After an exceptionally long time, this is (hopefully) finally fixed.

USES=luajit has landed, and both neovim and rspamd have been updated to use it. There are many other ports that need to be converted (it requires one-by-one testing and conversion) but we are on the path to a single version of luajit used across the tree.

Please let me know if you are still encountering problems with this.
Comment 21 Adam Weinberger freebsd_committer freebsd_triage 2022-10-16 15:52:46 UTC
Closing this out. Thanks to everyone involved for your patience.