Bug 257525 - lang/ruby26: add DTRACE option
Summary: lang/ruby26: add DTRACE option
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-ruby (Nobody)
URL:
Keywords: patch
Depends on:
Blocks:
 
Reported: 2021-08-01 09:44 UTC by Evgeniy Khramtsov
Modified: 2021-09-06 21:09 UTC (History)
2 users (show)

See Also:
evgeniy: maintainer-feedback-
evgeniy: maintainer-feedback?


Attachments
v1 (use "git am") (2.48 KB, patch)
2021-08-01 09:44 UTC, Evgeniy Khramtsov
evgeniy: maintainer-approval?
Details | Diff
patch to ruby port (871 bytes, patch)
2021-09-06 18:54 UTC, Steve Wills
no flags Details | Diff
updated patch to ruby port (889 bytes, patch)
2021-09-06 20:46 UTC, Steve Wills
no flags Details | Diff
dtrace patch (1.39 KB, patch)
2021-09-06 21:09 UTC, Evgeniy Khramtsov
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Evgeniy Khramtsov 2021-08-01 09:44:15 UTC
Created attachment 226826 [details]
v1 (use "git am")

- add DTRACE option for user to decide

lang/ruby26 assumes that DTrace support is guarded by exists() check,
failing when base toolchain is removed, and instead external toolchain
is provided by ports via /usr/local/share/toolchains/*.mk inclusion.

- remove ${OPSYS} == "FreeBSD" check

DTrace does not exist on DragonFly BSD, the only known consumer of FreeBSD ports.

- Bump PORTREVISION per [1]

poudriere testport (post-patch) Tier 1 arch logs:
https://codeberg.org/ei/misc/commit/22ec334270277410067b24f9120022837cc1216d

poudriere bulk diff (post-patch) Tier 1 arch logs:
https://codeberg.org/ei/misc/commit/40a03285b80d6b49a49f8755ac97aefb493689b4

poudriere package diff (post-patch) Tier 1 arch logs [1]:
https://codeberg.org/ei/misc/commit/8f6a59914e4acd5f57633935d07ae73985c0af08

Note, 11.4 has modified usr/local/lib/libruby* entries. This is not caused
by this change, instead caused by any rebuild instead. Apparently, timestamp
or something else gets baked in on 11.4, so 11.4 Ruby is not reproducible. I
don't know any tool to diff binaries to tell what exactly is not reproducible.

RUBY_PORTREVISION bump is not included in this patch, git commit --amend
if committing.

UPDATING entry is not needed, the default behaviour of port did not change,
users who disable DTRACE globally will get notified via "options changed"
of poudriere.

lang/ruby27 change should follow shortly.
Comment 1 Steve Wills freebsd_committer 2021-08-04 17:06:47 UTC
(In reply to Evgeniy Khramtsov from comment #0)

> lang/ruby26 assumes that DTrace support is guarded by exists() check, failing when base toolchain is removed,

Are you saying that /usr/sbin/dtrace doesn't exist if WITHOUT_TOOLCHAIN is set?
Comment 2 Evgeniy Khramtsov 2021-08-05 01:34:44 UTC
(In reply to Steve Wills from comment #1)

WITHOUT_TOOLCHAIN is problematic (removes base headers). Instead, I use:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=257622#c2

dtrace does exist on such a system, but the port can't link DTRACE using non-base toolchain. Base toolchain links symbols that ports toolchain can't,
e.g. https://github.com/DankBSD/ports/commit/fef8031

Some ports already use DTRACE option, e.g. DTRACE_DESC in Mk/bsd.options.desc.mk,
so I think adding a DTRACE option to workaround this may be fine.
Comment 3 Evgeniy Khramtsov 2021-08-05 01:36:35 UTC
(In reply to Evgeniy Khramtsov from comment #2)

> the port can't link

Stderr is in the patch.
Comment 4 Evgeniy Khramtsov 2021-09-05 22:58:48 UTC
Maintainer timeout (1 month).

Maybe add this option for ports tree consistency,
e.g. do the same as databases/memcached, databases/postgresql14-server,
lang/perl5.34, lang/erlang-runtime24, lang/ponyc, www/node,
and many other ports that have optional DTRACE?
Comment 5 Steve Wills freebsd_committer 2021-09-06 01:36:17 UTC
(In reply to Evgeniy Khramtsov from comment #2)

> dtrace does exist on such a system, but the port can't link DTRACE using non-base toolchain.

Why not? What is the error? Was it posted previously and I'm missing it?

> Base toolchain links symbols that ports toolchain can't, e.g. https://github.com/DankBSD/ports/commit/fef8031

Sorry, I don't understand. The linked commit doesn't seem to explain much.

I really don't understand this issue overall. It seems like it would really be helpful if you could be more specific about the issue(s) you seem to be encountering.

You seem to be saying that you have set these options in src.conf:

CROSS_TOOLCHAIN=llvm12
WITHOUT_CLANG=yes
WITHOUT_CLANG_BOOTSTRAP=yes
WITHOUT_LLD=yes
WITHOUT_LLD_BOOTSTRAP=yes
WITHOUT_LLDB=yes
WITHOUT_LLVM_COV=yes

and an additional patch to enable using external LLVM toolchain. What isn't clear is why you are doing that, or exactly what problem you are encountering due to this configuration. Perhaps if you could post the error message and/or full build log you get when trying to build ruby with this configuration it might help?

I can say that ruby should be made reproducible if it isn't, in my opinion. I suggest using sysutils/py-diffoscope to investigate what is changing between builds; I've used it in the past and it's quite nice.

I wouldn't suggest invoking maintainer timeout here until the issue is better understood. Maybe someone besides me understands it better. True the option to disable DTRACE is harmless, at the same time, we try not to add unnecessary options as each one adds a maintenance work in the long term. I'd apply the same logic to similarly filed DTRACE option patches.
Comment 6 Steve Wills freebsd_committer 2021-09-06 03:46:57 UTC
(In reply to Steve Wills from comment #5)
I did compare multiple builds of ruby with diffoscope and dtrace does seem to be a source of non-reproducibility in the package. So that seems like a good reason for a DTRACE option, at least until we can fix dtrace to use reproducible symbol names.

But I don't understand why using the external compiler would cause the link issue, the symbols should be present regardless of which compiler builds them.
Comment 7 Evgeniy Khramtsov 2021-09-06 12:36:26 UTC
(In reply to Steve Wills from comment #5)

> What is the error? Was it posted previously and I'm missing it?

See v1 commit message:

compiling dmyenc.c
linking miniruby
ld: error: undefined symbol: __dtraceenabled_ruby___array__create
>>> referenced by array.c
>>>               array.o:(ary_new)
>>> referenced by array.c
>>>               array.o:(rb_ary_new)
>>> referenced by array.c
>>>               array.o:(rb_assoc_new)
>>> referenced 27 more times

> The linked commit doesn't seem to explain much.
> It seems like it would really be helpful if you could be more specific about the issue(s) you seem to be encountering.
> You seem to be saying that you have set these [...] options in src.conf
> and an additional patch to enable using external LLVM toolchain.

FreeBSD base system includes LLVM based toolchain, and if one follows CURRENT,
then each update from source can take hours due to LLVM build. While there are
different ways to solve this via meta mode or ccache, the most efficient way
to lessen compile times is to not build the base toolchain, and use devel/llvm12
from ports instead. LLVM from ports can also be fetched using poudriere-devel
pkg seeding via '-b' option, resulting in no compile time for LLVM. This allows
me to build both kernel+world in less than 5 minutes using cheap (QTJ2) hardware.

I did the same for my poudriere build jail and installed devel/llvm12 as toolchain. The ports LLVM != base LLVM; the base one can link DTrace symbols
(in www/node, etc) while ports one can't. Maybe the ports LLVM can't link them due to diff between base LLVM, or the base LLVM includes some headers by default, or I don't have related toolchain binaries in $PATH symlinked to /usr/local/llvm12/*.

This is not an issue to me; I don't use DTRACE options in ports and opted to turn them off instead. The only ports that I use that have non-optional DTRACE are lang/ruby2*.

> Full build log you get when trying to build ruby with this configuration it might help?

Sorry, but I can't expose my environment, as this is an infoleak that can help
crafting exploits and puts me in risk. I prefer to have unique binaries. I already posted the required steps to reproduce this.

> True the option to disable DTRACE is harmless, at the same time, we try not to add unnecessary options as each one adds a maintenance work in the long term.

Currently DTrace support is an "automagic" option (which some folks dislike),
but I think that the main rationale to include DTRACE as an option, is ports tree consistency. My use case happens to need this.

> sysutils/py-diffoscope

Thanks.
Comment 8 Steve Wills freebsd_committer 2021-09-06 13:13:50 UTC
(In reply to Evgeniy Khramtsov from comment #7)

Could you try running the scripts in this repo with your setup:

https://github.com/swills/dtrace_thing

and posting the results? You can leave out the particular md5 checksums if you want, but it would be helpful to know if they produce the same results on multiple runs or not.

I wonder if this is a problem with the ports tool chain, this might help show that.

I'm fine with adding the DTRACE option if we can better understand the issue and perhaps file a bug for the root cause, but simply turning it off as a workaround without understanding the issue seems like a bad idea, to me.
Comment 9 Steve Wills freebsd_committer 2021-09-06 13:28:19 UTC
I've tried to reproduce this issue, but so far I've been unable to reproduce it.
Comment 10 Evgeniy Khramtsov 2021-09-06 14:31:48 UTC
(In reply to Steve Wills from comment #8)

Sure. Although I changed $COMP because "cc" and "ld" don't exist here:
-COMP="cc"
+COMP="/usr/local/llvm12/bin/clang"
and
-COMP="c++"
+COMP="/usr/local/llvm12/bin/clang++"

1 output:
rm: app: No such file or directory
rm: app.o: No such file or directory
rm: probes.o: No such file or directory
sh: ld: not found
dtrace: failed to link script probes: failed to link probes.o: ld exited with status 127
clang: error: no such file or directory: 'probes.o'
dtrace: failed to execute ./app: No such file or directory
MD5 (app.o) = 4aa6f8c53e1786b9af667f38c03631de
md5: probes.o: No such file or directory
md5: app: No such file or directory

2 output:
rm: app: No such file or directory
rm: app.o: No such file or directory
rm: probes.o: No such file or directory
rm: probes.h: No such file or directory
sh: ld: not found
dtrace: failed to link script probes: failed to link probes.o: ld exited with status 127
clang++: error: no such file or directory: 'probes.o'
dtrace: failed to execute ./app: No such file or directory
MD5 (app.o) = e6c17b37918f8d953da79811b8131c0b
md5: probes.o: No such file or directory
md5: app: No such file or directory

Seems that dtrace calls "ld" regardless of what is set as LD in /usr/share/mk/sys.mk. LD set via env(1) is also ignored. Maybe a base issue?

Greg V of DankBSD could not experience issues with Ruby because of symlinks:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=257621#c1

In my setup I don't use symlinks.
Comment 11 Steve Wills freebsd_committer 2021-09-06 15:28:59 UTC
(In reply to Evgeniy Khramtsov from comment #10)

> Seems that dtrace calls "ld" regardless of what is set as LD in /usr/share/mk/sys.mk. LD set via env(1) is also ignored. Maybe a base issue?

Thank you, yes, this is the cause of the issue and what I was trying to get to. Indeed, dtrace expects an "ld" command:

https://cgit.freebsd.org/src/tree/cddl/contrib/opensolaris/lib/libdtrace/common/dt_open.c#n789

It would be nice if it checked LD environment variable, and we can file a bug requesting that, but expecting an "ld" command to exist somewhere seems like a pretty reasonable expectation to me.

I'm not even sure where the right place to file a bug on dtrace would be. What is the upstream source? Or perhaps it should be filed against FreeBSD base since this was added here:

https://cgit.freebsd.org/src/commit/cddl/contrib/opensolaris/lib/libdtrace/common/dt_open.c?id=fb24b944be9a0a5c0fe8a09b28321d1ff2c245fd

Seems like there are a number of ways around the issue for now.

You could patch this hard coded path in your setup for now, since you've already patched so many other things. Or perhaps create a patch to dtrace to have it check LD environment variable and fall back to "ld" and add that to a dtrace bug report. But, since we now understand the issue and disabling dtrace in the port avoids it and also helps with reproducibility, I'm OK with the patch to ruby to add the DTRACE option.

Thanks again for the help understanding the issue. I'll try to file a bug against FreeBSD base and create a patch when I have some time.
Comment 12 Steve Wills freebsd_committer 2021-09-06 18:54:53 UTC
Created attachment 227715 [details]
patch to ruby port

Could you test this patch and see if it allows ruby to build with dtrace enabled with your environment (non-system compiler, no system ld, etc.)?
Comment 13 Steve Wills freebsd_committer 2021-09-06 19:00:49 UTC
(In reply to Steve Wills from comment #12)
Sorry, never mind, that one seemed to work for ruby27 for me, but not for ruby26.
Comment 14 Steve Wills freebsd_committer 2021-09-06 19:02:50 UTC
(In reply to Steve Wills from comment #13)
Actually it does seem to work with ruby26, I was hitting a MAKE_JOBS issue locally when testing. If you could have a look, that'd be helpful.
Comment 15 Evgeniy Khramtsov 2021-09-06 20:11:31 UTC
(In reply to Steve Wills from comment #14)

Adding "-x ldpath=/usr/local/llvm12/bin/ld" fixes examples:

1:
dtrace: script 'script.d' matched 1 probe
a=1, b=2
a=2, b=3
dtrace: pid 92114 has exited

                1                2                1
                2                3                1
MD5 (app.o) = 4aa6f8c53e1786b9af667f38c03631de
MD5 (probes.o) = 9e27e17d7782614b07441578d49be643
MD5 (app) = 72cb8fb8f57b04d5c85405fc211c9e2a

2:
rm: app: No such file or directory
rm: app.o: No such file or directory
rm: probes.o: No such file or directory
rm: probes.h: No such file or directory
dtrace: script 'script.d' matched 1 probe
a=1, b=2
a=2, b=3
dtrace: pid 47097 has exited

                1                2                1
                2                3                1
MD5 (app.o) = e6c17b37918f8d953da79811b8131c0b
MD5 (probes.o) = 2ee59d958960e03ec4772462606a0c4f
MD5 (app) = 258f7505f4884f955ecc809d419f71e8

But it seems that Ruby needs something more?
[...]
dtrace: failed to compile script probes.d: Preprocessor not found
[...]

I took a look at build log:
[...]
checking whether dtrace USDT is available... yes(-x ldpath=ld -xnolibs)
[...]
LD = ld
[...]

Then I passed LD:

- CONFIGURE_ENV=  debugflags=
+ CONFIGURE_ENV=  debugflags= LD=${LD}

And the build log changed:

[...]
- checking whether dtrace USDT is available... yes(-x ldpath=ld -xnolibs)
+ checking whether dtrace USDT is available... yes(-x ldpath=/usr/local/llvm12/bin/ld -xnolibs)
- checking whether dtrace needs post processing... no
+ checking whether dtrace needs post processing... rebuild
[...]
        BASERUBY = echo executable host ruby is required.  use --with-baseruby option.; false
        CC = /usr/local/llvm12/bin/clang
-       LD = ld
+       LD = /usr/local/llvm12/bin/ld
        LDSHARED = /usr/local/llvm12/bin/clang -shared
        CFLAGS = -O2 -pipe -O3 -mbranches-within-32B-boundaries  -fstack-protector-all -fPIE -fPIC -isystem /usr/local/include -fno-strict-aliasing  -fPIC
        XCFLAGS = -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-strict-overflow -fvisibility=hidden -DRUBY_EXPORT -DCANONICALIZATION_FOR_MATHN
        CPPFLAGS = -I/usr/local/include -isystem /usr/local/include   -I. -I.ext/include/amd64-freebsd14 -I./include -I. -I./enc/unicode/12.1.0
        DLDFLAGS = -fstack-protector-all -pie -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro  -Wl,--compress-debug-sections=zlib -Wl,-E -Wl,-soname,libruby26.so.26 -fstack-protector-strong
        SOLIBS = -lelf -lunwind -lexecinfo -lprocstat -lz -lthr -lrt -lrt -ldl -lcrypt -lm -lpthread -L/usr/local/lib -L/usr/local/lib
        LANG = C
        LC_ALL = C
        LC_CTYPE =
clang version 12.0.1
Target: x86_64-portbld-freebsd14.0
Thread model: posix
InstalledDir: /usr/local/llvm12/bin
translating probes probes.d
dtrace: failed to compile script probes.d: Preprocessor not found
*** Error code 1

^ But still preprocessor (is it the $CPP one?) could not be found.
Passing $CPP to CONFIGURE_ENV did not help.

Did you try in a freshly created poudriere jail built using the patch
and src.conf? The jail should not have "cc", etc by default. "delete-old"
on existing jail does not delete "cc" and other binaries.

Thanks for trying to fix this. My priority in fixing DTRACE with ports LLVM
is low, but I will try any patches.
Comment 16 Evgeniy Khramtsov 2021-09-06 20:17:39 UTC
(In reply to Evgeniy Khramtsov from comment #15)

Nice, I forgot to remove CFLAGS and other things by accident.
Well, it doesn't matter now, here is the whole log (without make.conf):
https://codeberg.org/ei/misc/commit/862e8732b5ffff
Comment 17 Steve Wills freebsd_committer 2021-09-06 20:20:44 UTC
(In reply to Evgeniy Khramtsov from comment #16)
Oh, when I tested I didn't remove cpp also. I'll try that and see if I can come up with an updated patch.
Comment 18 Steve Wills freebsd_committer 2021-09-06 20:46:11 UTC
Created attachment 227718 [details]
updated patch  to ruby port

Here's an updated patch that seems to fix the cpp issue. Not ideal, but works. Need someone who knows autotools better to really make it good enough to consider up-streaming.
Comment 19 Evgeniy Khramtsov 2021-09-06 21:09:37 UTC
Created attachment 227720 [details]
dtrace patch

It built. But I had to also pass LD via CONFIGURE_ENV.
The working diff is attached.

If I don't report back about ruby27, then assume it also built fine.