Bug 239655 - lang/crystal: something weird in libevent
Summary: lang/crystal: something weird in libevent
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: Jan Beich
URL:
Keywords: regression
Depends on:
Blocks: 239599
  Show dependency treegraph
 
Reported: 2019-08-05 13:44 UTC by Petr Fischer
Modified: 2019-08-08 13:20 UTC (History)
4 users (show)

See Also:
greg: maintainer-feedback+


Attachments
svn-diff-crystal (2.66 KB, patch)
2019-08-07 21:45 UTC, Walter Schwarzenfeld
no flags Details | Diff
svn-diff-crystal_v2 (1.71 KB, patch)
2019-08-08 07:12 UTC, Walter Schwarzenfeld
no flags Details | Diff
svn-diff-crystal_v3 (1.73 KB, patch)
2019-08-08 07:14 UTC, Walter Schwarzenfeld
no flags Details | Diff
svn-diff-crystal_v4 (1.85 KB, patch)
2019-08-08 07:28 UTC, Walter Schwarzenfeld
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Petr Fischer 2019-08-05 13:44:59 UTC
When I run crystal 0.28 like:

crystal src/something.cr

Tons of this errors from libevent occurs:
...
Unhandled exception in spawn: Error reinitializing libevent[warn] event_reinit: forked from the event_loop.
 (Exception)
[warn] event_reinit: forked from the event_loop.
[warn] event_reinit: forked from the event_loop.
Unhandled exception in spawn: Unhandled exception in spawn:  (Unhandled exception in spawn: Error reinitializing libeventExceptionError reinitializing libevent ()
 (ExceptionException)
)
[warn] event_reinit: forked from the event_loop.
Unhandled exception in spawn: Error reinitializing libevent (Exception)
[warn] event_reinit: forked from the event_loop.
Unhandled exception in spawn: Error reinitializing libevent (Exception)
Unhandled exception in spawn: Error reinitializing libevent (Exception)
Unhandled exception in spawn: Error reinitializing libevent (Exception)
[warn] event_reinit: forked from the event_loop.
Unhandled exception in spawn: [warn] event_reinit: forked from the event_loop.
Error reinitializing libeventUnhandled exception in spawn: [warn] event_reinit: forked from the event_loop.
 (Error reinitializing libeventUnhandled exception in spawn: Exception (Error reinitializing libevent)
Exception ()
Exception[warn] event_reinit: forked from the event_loop.
[warn] event_reinit: forked from the event_loop.
Error reinitializing libeventUnhandled exception in spawn:  (Error reinitializing libeventException ()
Exception[warn] event_reinit: forked from the event_loop.
Unhandled exception in spawn: Error reinitializing libevent (Exception)
)
[warn] event_reinit: forked from the event_loop.
Unhandled exception in spawn: Error reinitializing libevent (Exception)
[warn] event_reinit: forked from the event_loop.
Unhandled exception in spawn: Error reinitializing libevent (Exception)
)
[warn] event_reinit: forked from the event_loop.
Unhandled exception in spawn: Error reinitializing libevent (Exception)
[warn] event_reinit: forked from the event_loop.
Unhandled exception in spawn: Error reinitializing libevent (Exception)
Unhandled exception in spawn: Error reinitializing libevent (Exception)
[warn] event_reinit: forked from the event_loop.
Unhandled exception in spawn: Error reinitializing libevent (Exception)
[warn] event_reinit: forked from the event_loop.
Unhandled exception in spawn: Error reinitializing libevent (Exception)
[warn] event_reinit: forked from the event_loop.
Unhandled exception in spawn: Error reinitializing libevent (Exception)
...


and there is also tons of dead processes in process list in state piperd:


...
 2382 pf            1  52    0   619M   494M piperd  2   0:00   0.00% crystal spec
 2416 pf            1  52    0   619M   494M piperd  0   0:00   0.00% crystal spec
 2443 pf            1  52    0   619M   494M piperd  1   0:00   0.00% crystal spec
 2140 pf            1  52    0   619M   494M piperd  2   0:00   0.00% crystal spec
 2113 pf            1  52    0   619M   494M piperd  1   0:00   0.00% crystal spec
 2436 pf            1  52    0   619M   494M piperd  0   0:00   0.00% crystal spec
 2368 pf            1  52    0   619M   494M piperd  1   0:00   0.00% crystal spec
 2384 pf            1  52    0   619M   494M piperd  2   0:00   0.00% crystal spec
 2430 pf            1  52    0   619M   494M piperd  3   0:00   0.00% crystal spec
 2401 pf            1  52    0   619M   494M piperd  1   0:00   0.00% crystal spec
 2347 pf            1  52    0   619M   494M piperd  3   0:00   0.00% crystal spec
 2357 pf            1  52    0   619M   494M piperd  3   0:00   0.00% crystal spec
 2440 pf            1  52    0   619M   494M piperd  1   0:00   0.00% crystal spec
 2377 pf            1  52    0   619M   494M piperd  2   0:00   0.00% crystal spec
 2375 pf            1  52    0   619M   494M piperd  0   0:00   0.00% crystal spec
 2371 pf            1  52    0   619M   494M piperd  2   0:00   0.00% crystal spec
 2439 pf            1  52    0   619M   494M piperd  1   0:00   0.00% crystal spec
 2376 pf            1  52    0   619M   494M piperd  3   0:00   0.00% crystal spec
 2316 pf            1  52    0   619M   494M piperd  2   0:00   0.00% crystal spec
 2373 pf            1  52    0   619M   494M piperd  0   0:00   0.00% crystal spec
 2409 pf            1  52    0   619M   494M piperd  0   0:00   0.00% crystal spec
 2298 pf            1  52    0   619M   494M piperd  2   0:00   0.00% crystal spec
...


Something wrong with libevent? This port is disfunctional now.

11.2-RELEASE-p11

Crystal 0.28.0 (2019-08-03)
LLVM: 6.0.1
Default target: x86_64-portbld-freebsd11.2
Comment 1 Walter Schwarzenfeld freebsd_triage 2019-08-05 14:02:14 UTC
Which libevent version? It was update on  2019-08-02 to 2.1.11.
Comment 2 Petr Fischer 2019-08-05 14:10:23 UTC
libevent-2.1.11 (libevent-2.1.so.7)
Comment 3 Walter Schwarzenfeld freebsd_triage 2019-08-05 14:23:39 UTC
Have you rebuild lang/crystal since the update of libevent?
Comment 4 Petr Fischer 2019-08-05 14:32:19 UTC
Yes, I had to rebuild crystal port, because old compiled  version needed libevent-2.1.so.6 - actual version is libevent-2.1.so.7.

Note: there is this line in the crystal port Makefile:

MAKE_ENV=   LD_LIBMAP="libevent-2.1.so.6=libevent-2.1.so.7"
Comment 5 Petr Fischer 2019-08-06 10:53:19 UTC
Next investigation - it looks like there is problem in transition from libevent <= 2.1.10 to the libevent-2.1.11.

I tested building crystal also on another box with FreeBSD 12 and:

- with libevent-2.1.10, everything is OK
- with libevent-2.1.11, lot of error messages from libevent, crystal does not work

Upstream github issue:
https://github.com/crystal-lang/crystal/issues/8044
Comment 6 Walter Schwarzenfeld freebsd_triage 2019-08-06 11:06:03 UTC
I tested on 11.X, here no problem.
Comment 7 Petr Fischer 2019-08-06 11:22:36 UTC
w.schwarzenfeld:

1) which version of libevent do you have?

2) did you run the crystal like: crystal xxx/yyy.cr? (no special options)
Comment 8 Walter Schwarzenfeld freebsd_triage 2019-08-06 11:40:04 UTC
1) 2.1.11
2) yes
Comment 9 Petr Fischer 2019-08-06 11:52:35 UTC
(In reply to w.schwarzenfeld from comment #8)

Try this crystal script:
---
require "http/client"
  
HTTP::Client.get "http://api.icndb.com/jokes/1"
---
Comment 10 Walter Schwarzenfeld freebsd_triage 2019-08-06 11:57:25 UTC
Ok, same error as you.
Comment 11 Walter Schwarzenfeld freebsd_triage 2019-08-06 12:24:26 UTC
CC'd libevent maintainer.
Comment 12 Walter Schwarzenfeld freebsd_triage 2019-08-06 12:49:47 UTC
It seems it is crystal and not libevent. Latest version is 0.30.

https://crystal-lang.org/blog/#release_notes

In version 0.29:

--> 
There were a couple of efforts to improve HTTP and URI. Although they are breaking-changes there should be easy to migrate.
Comment 13 Petr Fischer 2019-08-06 13:00:24 UTC
(In reply to w.schwarzenfeld from comment #12)

It looks even deeper - it's most probably not much about HTTP, but in internal fibers implementation (sort of cooperative green threads) and IO events (!). You can construct error generating script maybe also with another IO and fibers.

Don't forget, that with libevent-2.1.10, everyting is OK.

There is also ABI changes in libevent - that is probably source of problems:
https://raw.githubusercontent.com/libevent/libevent/release-2.1.11-stable/ChangeLog
Comment 14 Niclas Zeising freebsd_committer 2019-08-06 13:26:39 UTC
(In reply to Petr Fischer from comment #13)
> (In reply to w.schwarzenfeld from comment #12)
> 
> It looks even deeper - it's most probably not much about HTTP, but in
> internal fibers implementation (sort of cooperative green threads) and IO
> events (!). You can construct error generating script maybe also with
> another IO and fibers.
> 
> Don't forget, that with libevent-2.1.10, everyting is OK.
> 
> There is also ABI changes in libevent - that is probably source of problems:
> https://raw.githubusercontent.com/libevent/libevent/release-2.1.11-stable/
> ChangeLog

There was a ABI change in libevent, which makes it compatible with libevent 2.1.9 again, at least from my understanding of the release notes.
Have you tried recompiling crystal against the updated libevent version?
Comment 15 Petr Fischer 2019-08-07 18:11:04 UTC
News from upstream (crystal issue: https://github.com/crystal-lang/crystal/issues/8044)

Linux and OSX users are affected too - there is a little change in libevent-2.1.11, that broke the crystal.

So, we need to wait for fix from upstream - but it will be available as version 0.30.x, so we need to update crystal lang port from 0.28 to something newer...
Comment 16 Niclas Zeising freebsd_committer 2019-08-07 20:38:48 UTC
(In reply to Petr Fischer from comment #15)
> News from upstream (crystal issue:
> https://github.com/crystal-lang/crystal/issues/8044)
> 
> Linux and OSX users are affected too - there is a little change in
> libevent-2.1.11, that broke the crystal.
> 
> So, we need to wait for fix from upstream - but it will be available as
> version 0.30.x, so we need to update crystal lang port from 0.28 to
> something newer...

Do you know approximately how long this fix is going to take?  What are the chances it's possible to back port it to 0.28?

I'm not at all familiar with crystal.  How big and common is it?  How bothersome is this breakage?

The reasons I'm asking is to get a clearer view on whether to revert the libevent update or wait for crystal to be fixed.

Regards
Niclas
libevent maintainer
Comment 17 Walter Schwarzenfeld freebsd_triage 2019-08-07 21:01:17 UTC
Fix is there:

https://github.com/waj/crystal/commit/2fafc7d60825b4a7f10bb48c39107d0e8f08048b
Comment 18 Walter Schwarzenfeld freebsd_triage 2019-08-07 21:31:17 UTC
(In reply to Niclas Zeising from comment #16)
It is the question if this:

https://github.com/libevent/libevent/commit/497ef904d544ac51de43934549dbeccce8e6e8f8#diff-a49dae4bf6d105e1692f627817dc51af

causes other problems?
Comment 19 Niclas Zeising freebsd_committer 2019-08-07 21:33:24 UTC
(In reply to w.schwarzenfeld from comment #18)
> (In reply to Niclas Zeising from comment #16)
> It is the question if this:
> 
> https://github.com/libevent/libevent/commit/
> 497ef904d544ac51de43934549dbeccce8e6e8f8#diff-
> a49dae4bf6d105e1692f627817dc51af
> 
> causes other problems?

No, the question is if we can fix crystal in a resonable time frame, or if I should revert libevent.
This report is the only fallout I've heard from the libevent update thus far.
Comment 20 Walter Schwarzenfeld freebsd_triage 2019-08-07 21:44:33 UTC
Patch based on the link in  comment17 and seems to work.
Please, test it.
Comment 21 Walter Schwarzenfeld freebsd_triage 2019-08-07 21:45:19 UTC
Created attachment 206346 [details]
svn-diff-crystal
Comment 22 Jan Beich freebsd_committer 2019-08-08 05:48:09 UTC
Comment on attachment 206346 [details]
svn-diff-crystal

Better convert to PATCH_SITES in order to document the origin of the fix. See math/cvc4 for an example maintained by the same person.
Comment 23 Walter Schwarzenfeld freebsd_triage 2019-08-08 06:34:27 UTC
PATCH_SITES=            https://github.com/waj/crystal/commit/

and the syntax for 
PATCHFILES?
Comment 24 Jan Beich freebsd_committer 2019-08-08 06:43:22 UTC
(In reply to w.schwarzenfeld from comment #23)

PATCH_SITES=	https://github.com/${GH_ACCOUNT}/${GH_PROJECT}/commit/
PATCHFILES+=	2fafc7d60825.patch:-p1

then run "make makesum". Upstream fix was reviewed in https://github.com/crystal-lang/crystal/pull/8058

(In reply to Niclas Zeising from comment #19)
Why not use "portmgr blanket"? Whatever comes from upstream is often already reviewed. If don't use the port (even occasionally) then ask whoever reported to confirm the fix is enough and itself doesn't cause other regressions.

Otherwise, reverting the offending libevent commit is better idea as backing out updates is not risk-free. Don't forget to document the file containing the revert, so it can be dropped if future libevent updates make it unrebaseable. In such scenarios it's responsibility[1] of consumers' maintainers to avoid their ports becoming a burden by keeping up with upstream releases. Most upstream projects don't support old releases, especially those with 0 as major version.

[1] https://www.freebsd.org/doc/en/articles/contributing/ports-contributing.html#maintain-port
Comment 25 Walter Schwarzenfeld freebsd_triage 2019-08-08 06:55:08 UTC
Cryptic github...

2fafc7d60825 how do you get this number?

The url has 2fafc7d60825b4a7f10bb48c39107d0e8f08048b, the commit has 2fafc7d ??
Comment 26 Jan Beich freebsd_committer 2019-08-08 07:01:07 UTC
(In reply to Jan Beich from comment #24)
> ... and itself doesn't cause other regressions.

This is really just a precaution due to version differences i.e., can be ignored if no one is willing to test either the fix or the revert. Given everyone is a volunteer timeouts are very common.

(In reply to w.schwarzenfeld from comment #25)
> 2fafc7d60825 how do you get this number?

Git hash can be truncated as long as it still points to a unique commit. Depending on the size of the repo the length is usually between 7 to 12 digits. I often use 12-digit due to similarity with Mercurial. Hash collision with other projects is possible but unlikely due to how infrequently such patches are used.
Comment 27 Walter Schwarzenfeld freebsd_triage 2019-08-08 07:12:06 UTC
Created attachment 206352 [details]
svn-diff-crystal_v2
Comment 28 Walter Schwarzenfeld freebsd_triage 2019-08-08 07:14:42 UTC
Created attachment 206353 [details]
svn-diff-crystal_v3

Cause of portlint...
Comment 29 Jan Beich freebsd_committer 2019-08-08 07:16:47 UTC
Comment on attachment 206353 [details]
svn-diff-crystal_v3

>-SHA256 (crystal/crystal-0.27.2-freebsd12-amd64) = 67284ea4352efa8c6c8e50f6c57d589a7b86e21cf0dfe7fdadaa872dbcb3b3c6
>-SIZE (crystal/crystal-0.27.2-freebsd12-amd64) = 11263672
>-SHA256 (crystal/crystal-0.27.2-freebsd12-aarch64) = d50b23d2b7b0302e91d57d1a43cc8047d8c7a64e230b6a72c7a36da691386cb9
>-SIZE (crystal/crystal-0.27.2-freebsd12-aarch64) = 9692352

Please, look at the diff yourself before submitting to rule out such unrelated changes.
Comment 30 Walter Schwarzenfeld freebsd_triage 2019-08-08 07:28:18 UTC
Created attachment 206355 [details]
svn-diff-crystal_v4

Sorry!
Comment 31 Jan Beich freebsd_committer 2019-08-08 07:39:14 UTC
Comment on attachment 206355 [details]
svn-diff-crystal_v4

Looks OK now. Thanks.
Comment 32 Greg V 2019-08-08 13:00:18 UTC
Applying the upstream fix as PATCH_FILES looks good to me.
Comment 33 Jan Beich freebsd_committer 2019-08-08 13:19:38 UTC
> Assignee: ports-bugs@FreeBSD.orgzeising@FreeBSD.org

Sorry, my testing finished earlier and I've already written commit message.
Comment 34 Jan Beich freebsd_committer 2019-08-08 13:20:58 UTC
... but made a typo https://svnweb.freebsd.org/changeset/ports/508377