NB: Do not close this issue as a duplicate until a solution is found, please! Or reopen the original issue (I cannot do it). Hi, I have recently upgraded to 14.2-RELEASE and am no longer able to use poudriere to upgrade my ports. In any jail dtrace -h fails: vasily@julia:~ % dtrace -h dtrace: failed to establish error handler: "/usr/lib/dtrace/ipfw.d", line 1: syntax error near "in_addr_t" In a usual environment (without a jail) it works: vasily@vonbraun:~ % dtrace -h dtrace: -h requires one or more scripts or enabling options There are plenty of reports like this one all leading to this one: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=232675 The problem is marked as resolved, but it isn't, obviously. Dtrace is required for some ports like lang/perl5.36 I provide the same dump as other people in that report: vasily@vonbraun:~ % ctfdump -S /boot/kernel/kernel [1] unexpected kind -- 1 - CTF Statistics ------------------------------------------------------------- total number of data objects = 30929 total number of functions = 1 total number of function arguments = 2 maximum argument list length = 2 average argument list length = 2.00 total number of types = 32546 total number of integers = 77 total number of floats = 0 total number of pointers = 8104 total number of arrays = 3180 total number of func types = 2351 total number of structs = 10858 total number of unions = 645 total number of enums = 1142 total number of forward tags = 46 total number of typedefs = 5196 total number of volatile types = 49 total number of const types = 738 total number of restrict types = 2 total number of unknowns (holes) = 158 total number of struct members = 77582 maximum number of struct members = 253 total size of all structs = 19133170 maximum size of a struct = 2971720 average number of struct members = 7.15 average size of a struct = 1762.13 total number of union members = 2204 maximum number of union members = 106 total size of all unions = 132627 maximum size of a union = 41584 average number of union members = 3.42 average size of a union = 205.62 total number of enum members = 12020 maximum number of enum members = 2032 average number of enum members = 10.53 total number of unique strings = 70398 bytes of string data = 1047475 maximum string length = 81 average string length = 14.88
> The problem is marked as resolved, but it isn't, obviously. It's possible for different problems to manifest with the same error message. How are you creating the jail? I can't reproduce this locally. root@freebsd:~ # uname -a FreeBSD freebsd 14.2-RELEASE FreeBSD 14.2-RELEASE releng/14.2-n269506-c8918d6c7412 GENERIC amd64 root@freebsd:~ # jail -c name=test command=dtrace -h dtrace: -h requires one or more scripts or enabling options jail: dtrace -h: failed
(In reply to Mark Johnston from comment #1) For example, you can do the following (works for me at least): 1) Download base.txz from here: https://download.freebsd.org/releases/amd64/14.2-RELEASE/base.txz 2) Create a new fs: $ zfs create mypool/jails/freshjail (it is mounted to /jails/freshjail) 3) Extract base.txz into /jails/freshjail 4) Add this to /etc/jail.conf freshjail { host.hostname = freshjail.local; # Hostname path = "/jails/freshjail"; # Path to the jail ip4 = inherit; ip6 = inherit; mount.devfs; # Mount devfs inside the jail exec.start = "/bin/sh /etc/rc"; # Start command exec.stop = "/bin/sh /etc/rc.shutdown"; # Stop command } 5) Start the jail: $ service jail onestart freshjail 6) Go to the jail and run dtrace -h
Addendum: by "go to the jail" I mean jexec
(In reply to shamaz.mazum from comment #2) Thanks for the repro hints. The "problem" is that there's no kernel in this jail, so dtrace has no access to the CTF info. But /usr/lib/dtrace/ipfw.d (parsed only when ipfw.ko is loaded) relies on some definitions, e.g., for in_addr_t and struct ip6_addr. If I install kernel.txz as well, "dtrace -h" works again. We've had these kinds of problems before. The workaround is generally to add "-x nolibs" to the "dtrace -h" invocation in various build scripts. "-x nolibs" means that the definitions in /usr/lib/dtrace won't be processed (as they aren't needed for usdt probes in general). In fact, it looks like the perl configure script knows about this and tries to add "-x nolibs" if dtrace(1) accepts that option. Why doesn't that work? Can you share the log where the perl build failed? In any case, I think adding "#pragma D depends_on module kernel" to ipfw.d is sufficient to prevent "dtrace -h" from trying to process the file.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=b64c5a0ace59af62eff52bfe110a521dc73c937b commit b64c5a0ace59af62eff52bfe110a521dc73c937b Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2024-12-16 22:09:39 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2024-12-16 22:09:39 +0000 ipfw.d: Add required dependency declarations ipfw.d requires definitions of, at least, in_addr_t and struct ip6_addr, which it must obtain from CTF info. This is found by looking at $(sysctl -n kern.bootfile)/kernel, which might not be available in a jail. These depends_on declarations ensure that dtrace(1) invocations will simply abort processing of ipfw.d if the required binaries can't be found, rather than raising an error. PR: 283359 MFC after: 1 week share/dtrace/ipfw.d | 2 ++ 1 file changed, 2 insertions(+)
(In reply to Mark Johnston from comment #4) Thanks! That really helped. But I was wondering why it worked fine before and came to the following discovery: I tried to reproduce the issue once more time, after a computer reboot. And it was fine this time, dtrace -h was working both in a jail and in "normal" environment (without the patch applied). Turns out, I debugged sbcl using dtrace that day, and all dtrace modules was in the kernel: vasily@vonbraun:~ % kldstat Id Refs Address Size Name 1 154 0xffffffff80200000 1f3d568 kernel 2 1 0xffffffff8213e000 5da658 zfs.ko 3 1 0xffffffff83800000 665888 amdgpu.ko 4 2 0xffffffff83e66000 85090 drm.ko 5 1 0xffffffff837f9000 22b8 iic.ko 6 2 0xffffffff83eec000 4120 linuxkpi_video.ko 7 3 0xffffffff83ef1000 7350 dmabuf.ko 8 3 0xffffffff837fc000 3378 lindebugfs.ko 9 1 0xffffffff83ef9000 c338 ttm.ko 10 1 0xffffffff83f06000 356d0 amdgpu_dimgrey_cavefish_sos_bin.ko 11 1 0xffffffff83f3c000 375e0 amdgpu_dimgrey_cavefish_ta_bin.ko 12 1 0xffffffff83f74000 3dd88 amdgpu_dimgrey_cavefish_smc_bin.ko 13 1 0xffffffff83fb2000 1d958 amdgpu_dimgrey_cavefish_dmcub_bin.ko 14 1 0xffffffff83fd0000 425e0 amdgpu_dimgrey_cavefish_pfp_bin.ko 15 1 0xffffffff84013000 425e0 amdgpu_dimgrey_cavefish_me_bin.ko 16 1 0xffffffff84056000 42560 amdgpu_dimgrey_cavefish_ce_bin.ko 17 1 0xffffffff84099000 238f0 amdgpu_dimgrey_cavefish_rlc_bin.ko 18 1 0xffffffff840bd000 43a90 amdgpu_dimgrey_cavefish_mec_bin.ko 19 1 0xffffffff84101000 43a90 amdgpu_dimgrey_cavefish_mec2_bin.ko 20 1 0xffffffff84145000 a5e0 amdgpu_dimgrey_cavefish_sdma_bin.ko 21 1 0xffffffff84150000 8f8d0 amdgpu_dimgrey_cavefish_vcn_bin.ko 22 1 0xffffffff841e0000 3160 amdtemp.ko 23 1 0xffffffff841e4000 2138 amdsmn.ko 24 1 0xffffffff841e7000 2278 pty.ko 25 1 0xffffffff841ea000 3390 acpi_wmi.ko 26 1 0xffffffff841ee000 3220 intpm.ko 27 1 0xffffffff841f2000 2178 smbus.ko 28 1 0xffffffff841f5000 3360 uhid.ko 29 1 0xffffffff841f9000 4364 ums.ko 30 1 0xffffffff841fe000 3360 wmt.ko 31 1 0xffffffff84202000 4d20 ng_ubt.ko 32 6 0xffffffff84207000 abb8 netgraph.ko 33 2 0xffffffff84212000 a250 ng_hci.ko 34 4 0xffffffff8421d000 2670 ng_bluetooth.ko 35 1 0xffffffff84220000 e268 ng_l2cap.ko 36 1 0xffffffff8422f000 1bf68 ng_btsocket.ko 37 1 0xffffffff8424b000 38f8 ng_socket.ko 38 1 0xffffffff8424f000 5be8 autofs.ko 39 1 0xffffffff84255000 2a68 mac_ntpd.ko 40 1 0xffffffff84258000 2f5c0 if_wg.ko 41 1 0xffffffff84288000 4850 nullfs.ko 42 1 0xffffffff8428d000 3560 fdescfs.ko 43 1 0xffffffff84291000 2298 dtraceall.ko 44 10 0xffffffff84294000 1c290 opensolaris.ko 45 10 0xffffffff842b1000 3ebe0 dtrace.ko 46 1 0xffffffff842f0000 21cc dtmalloc.ko 47 1 0xffffffff842f3000 2cd8 dtnfscl.ko 48 1 0xffffffff842f6000 3295 fbt.ko 49 1 0xffffffff842fa000 555d4 fasttrap.ko 50 1 0xffffffff84350000 33c0 kinst.ko 51 1 0xffffffff8435e000 2278 sdt.ko 52 1 0xffffffff84361000 91cc systrace.ko 53 1 0xffffffff8436b000 91cc systrace_freebsd32.ko 54 1 0xffffffff84375000 22c4 profile.ko So my reproduction instructions work only if dtrace modules are in the kernel (e.g. dtrace -l was called before). Is there any explanations for this? Anyway, the patch worked, thanks!
> So my reproduction instructions work only if dtrace modules are in the kernel (e.g. dtrace -l was called before). Is there any explanations for this? Anyway, the patch worked, thanks! If no dtrace modules are loaded (and in particular, sdt.ko is not loaded), then the ipfw probe provider will not be registered, and the existing depends_on guard in ipfw.d would have prevented the file from being loaded.
(In reply to Mark Johnston from comment #7) Thanks for the explanation! So I suppose the issue is fixed.
(In reply to shamaz.mazum from comment #8) I think so, yes. I'll close the bug report after this patch is merged to stable/14. Thanks again for providing a reproducer, tracking down these kinds of problems without one is difficult.
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=ae21b1f88e0227602017d9cc6d2b5f3f51a82e3c commit ae21b1f88e0227602017d9cc6d2b5f3f51a82e3c Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2024-12-16 22:09:39 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2024-12-23 16:51:33 +0000 ipfw.d: Add required dependency declarations ipfw.d requires definitions of, at least, in_addr_t and struct ip6_addr, which it must obtain from CTF info. This is found by looking at $(sysctl -n kern.bootfile)/kernel, which might not be available in a jail. These depends_on declarations ensure that dtrace(1) invocations will simply abort processing of ipfw.d if the required binaries can't be found, rather than raising an error. PR: 283359 MFC after: 1 week (cherry picked from commit b64c5a0ace59af62eff52bfe110a521dc73c937b) share/dtrace/ipfw.d | 2 ++ 1 file changed, 2 insertions(+)