Bug 263858

Summary: dtrace: Fails to compile script on main: operator -> cannot be applied to a forward declaration: no struct ip_fw_args definition is available
Product: Base System Reporter: Tomoaki AOKI <junchoon>
Component: binAssignee: freebsd-dtrace (Nobody) <dtrace>
Status: Closed Not A Bug    
Severity: Affects Only Me CC: dtrace, markj
Priority: --- Keywords: needs-qa
Version: CURRENTFlags: koobs: mfc-stable13-
koobs: mfc-stable12-
Hardware: Any   
OS: Any   
Attachments:
Description Flags
Output from `ctfdump -ht /boot/kernel/ipfw.ko` on stable/13 none

Description Tomoaki AOKI 2022-05-08 01:18:31 UTC
On main, dtrace fails to comile scripts.
Happenes only on main. stable/13 is not affected. Both are amd64.

Kernel configuration is based on GENERIC-NODEBUG.
Include GENERIC-NODEBUG, add "options CAM_IOSCHED_DYNAMIC" and set ident.

For stable/13 (working fine), copied old (before commit 6eebda3bbad96ab6ef001781ee1b6de45af062a2) GENERIC-NODEBUG with renaming, add "options CAM_IOSCHED_DYNAMIC" and set ident.

Patches applied to main (basically something from PR or differential revision) are also applied to stable/13 with minimal fix to apply, if any.


Error mode happenes:
operator -> cannot be applied to a forward declaration: no struct ip_fw_args definition is available


Output examples:

*On script for collecting iwlwifi debugging info, provided by bz@ at [1].

#dtrace -C -s iwlwifi-cmds2.d -o /var/tmp/YYYYMMDD-SS-iwlwifi.dout
dtrace: failed to compile script iwlwifi-cmds2.d: "/usr/lib/dtrace/ipfw.d", line 1: operator 
-> cannot be applied to a forward declaration: no struct ip_fw_args definition is available

[1] https://people.freebsd.org/~bz/wireless/iwlwifi-cmds2.d


*On ports java/openjdk11 build:

dtrace: failed to compile script /usr/ports/java/openjdk11/work/jdk11u-jdk-11.0.15-10-1/build/bsd-x86_64-normal-server-release/hotspot/variant-server/support/dtrace/hotspot.h.d: "/usr/lib/dtrace/ipfw.d", line 1: operator -> cannot be applied to a forward declaration: no struct ip_fw_args definition is available
dtrace: failed to compile script /usr/ports/java/openjdk11/work/jdk11u-jdk-11.0.15-10-1/build/bsd-x86_64-normal-server-release/hotspot/variant-server/support/dtrace/hotspot_jni.h.d: "/usr/lib/dtrace/ipfw.d", line 1: operator -> cannot be applied to a forward declaration: no struct ip_fw_args definition is available
dtrace: failed to compile script /usr/ports/java/openjdk11/work/jdk11u-jdk-11.0.15-10-1/build/bsd-x86_64-normal-server-release/hotspot/variant-server/support/dtrace/hs_private.h.d: "/usr/lib/dtrace/ipfw.d", line 1: operator -> cannot be applied to a forward declaration: no struct ip_fw_args definition is available
gmake[4]: *** [gensrc/GensrcDtrace.gmk:51: /usr/ports/java/openjdk11/work/jdk11u-jdk-11.0.15-10-1/build/bsd-x86_64-normal-server-release/hotspot/variant-server/gensrc/dtracefiles/hotspot.h] Error 1
gmake[4]: *** Waiting for unfinished jobs....
gmake[4]: *** [gensrc/GensrcDtrace.gmk:51: /usr/ports/java/openjdk11/work/jdk11u-jdk-11.0.15-10-1/build/bsd-x86_64-normal-server-release/hotspot/variant-server/gensrc/dtracefiles/hs_private.h] Error 1
gmake[4]: *** [gensrc/GensrcDtrace.gmk:51: /usr/ports/java/openjdk11/work/jdk11u-jdk-11.0.15-10-1/build/bsd-x86_64-normal-server-release/hotspot/variant-server/gensrc/dtracefiles/hotspot_jni.h] Error 1
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2022-05-08 01:42:27 UTC
Are you able to bisect or further narrow down the commit/regression window?
Comment 2 Tomoaki AOKI 2022-05-08 05:30:39 UTC
(In reply to Kubilay Kocak from comment #1)

IIRC, this didn't happen on previous java/openjdk11 update (bump), dated at 2022-02-12 04:28:15 +0000.

  ports main git: 399e3772d37e1ca597304d7e6784d4ee410ba869

dtrace-related portion on ports Makefile doesn't change with most recent commit, dated at 2022-05-05 01:50:04 +0000.

  ports main git: ec8ecb5b4a24746692cfd03a1f7bafeabe6409f6

Note that whether upstream modified the script on update or not is not checked.
(I've looked into ports side only.)


Both built fine on stable/13.

Disabling dtrace by forcibly adding --disable-dtrace to CONFIGURE_ARGS allowed build, so the problem is dtrace-related only.


For bz@'s script, I have no OK data point on main. But no faulure on stable/13.
Comment 3 Tomoaki AOKI 2022-05-08 12:36:08 UTC
(In reply to Tomoaki AOKI from comment #2)

Tried a bit more.

The affected scripts of java/openjdk11 are generated on early build stage.
Not yet exist on extract, patch nor configure stage.

Searching *.d files on ${WRKDIR} at extract stage resulted in:
/usr/ports/java/openjdk11/work/jdk11u-jdk-11.0.15-10-1/src/hotspot/os/solaris/dtrace/jhelper.d
/usr/ports/java/openjdk11/work/jdk11u-jdk-11.0.15-10-1/src/hotspot/os/posix/dtrace/hotspot.d
/usr/ports/java/openjdk11/work/jdk11u-jdk-11.0.15-10-1/src/hotspot/os/posix/dtrace/hotspot_jni.d
/usr/ports/java/openjdk11/work/jdk11u-jdk-11.0.15-10-1/src/hotspot/os/posix/dtrace/hs_private.d
/usr/ports/java/openjdk11/work/jdk11u-jdk-11.0.15-10-1/test/hotspot/jtreg/compiler/codecache/dtrace/SegmentedCodeCacheDtraceTestScript.d

All files, including anything existing at extract stage seems to have the same time stamp of release.

Looking into upstream github, above files seems to be 5 years old. [1]

So they would not be modified, unless following generation process is changed.
But looking into further beyonds me, unfortunately.

So what I can provide currently is what I wrote my previous post.

[1] https://github.com/openjdk/jdk/tree/master/src/hotspot/os/posix/dtrace
Comment 4 Mark Johnston freebsd_committer freebsd_triage 2022-05-09 12:33:48 UTC
Which revision of main are you testing?
Comment 5 Tomoaki AOKI 2022-05-09 17:54:58 UTC
(In reply to Mark Johnston from comment #4)

I've tried bz@'s script for testing iwlwifi at revs in NG list below with no success.

The rev I encountered java/openjdk11 build error is unclear, but any of the revs in NG list below.
The last successful java/openjdk11 is not recorded. Sorry.
But maybe at...

*Unclear (bz@'s script was not yet provided) but possibly OK rev.:
  a35bdd4489b97f14cb46ebd0f5354685e4488af3
    tcp: add sysctl interface for setting socket options
 or
  e01e8f911b935eabcc35b4d121951e4e21042ee5
    freebsd-update: improve BE creation feature


*Known to be NG revs:
  32c3e0f04921f1641c431433cf00ab2a1d9c100e
    nfscl: Clean up the code by removing unused arguments

  b75644771aa8bd362b9d22bc66c6bd6cc2beaf8a
    Add nullfs specific nocache option.

  350668eb09a274436922fd580cd958c6a8a7eb25
    rpc.tlsservd: Delete set but unused variable

  350668eb09a274436922fd580cd958c6a8a7eb25
    rpc.tlsservd: Delete set but unused variable

  9fb40baf60435c000415429220c57c754ce318fd
    cam_periph: Return ENXIO when peripheral is invalidated

  f44280bf5fbb0599871612e9286407981ad4aaa3
    libz: update the upgrade instructions to reflect reality

  1599fc904d35cfa8eecad92818d1f4b55de6818f
    iosched: Move bio_next() inside of the CAM_IOSCHED_DYNAMIC ifdef

  9a3583bfbd1740a158b3916432286190e0f2bf60
    OpenSSL: Merge OpenSSL 1.1.1o

  9f580526e45a28fdfabe1aa58798ad3883690af0
    ibcore: Remove set, but not used variable.

  2f44ad86f5553edb9e7854767424f22ba132ee92
    dmesg: Better wording from review
Comment 6 Mark Johnston freebsd_committer freebsd_triage 2022-05-10 13:16:16 UTC
I'm not able to reproduce the problem at commit c6df2176038.

There is one similar problem which was fixed recently by commit 6e563a1b608.  Are you sure that you have an updated libctf.so?

Also I presume that you have ipfw.ko loaded; can you attach output from "ctfdump -ht /path/to/ipfw.ko"?
Comment 7 Tomoaki AOKI 2022-05-10 16:33:25 UTC
(In reply to Mark Johnston from comment #6)
> There is one similar problem which was fixed recently by commit 6e563a1b608.  
> Are you sure that you have an updated libctf.so?

Sure. My last update at git: 2f44ad86f5553edb9e7854767424f22ba132ee92 was a clean rebuild. Backup /usr/obj/usr/src/amd64.amd64/sys/(confname)/version, delete everything under /usr/obj, then restore /usr/obj/usr/src/amd64.amd64/sys/(confname)/version to keep build No. monotonic.


> Also I presume that you have ipfw.ko loaded; can you attach output from
> "ctfdump -ht /path/to/ipfw.ko"?

Yes. ipfw.ko is loaded as I configure it via /etc/rc.conf.
The output from `ctfdump -ht /boot/kernel/ipfw.ko` are as below.
No difference between regular user (in wheel group) and root.

% ctfdump -ht /boot/kernel/ipfw.ko
ctfdump: /boot/kernel/ipfw.ko does not contain .SUNW_ctf data
: No error: 0

The above output seems to be via stderr.
Comment 8 Mark Johnston freebsd_committer freebsd_triage 2022-05-10 16:35:51 UTC
(In reply to Tomoaki AOKI from comment #7)
Oh.  How did you build ipfw.ko?  Did you build it as a standalone module, or as part of a buildkernel?  In the latter case, do you have "makeoptions WITH_CTF=1" in the kernel configuration?
Comment 9 Tomoaki AOKI 2022-05-10 16:37:06 UTC
Created attachment 233840 [details]
Output from `ctfdump -ht /boot/kernel/ipfw.ko` on stable/13

Output from `ctfdump -ht /boot/kernel/ipfw.ko` on stable/13 at git: 3c01fdcfc161b06732b0762e37d1beca4a1f214d, amd64.
dtrace works fine on stable/13.
Comment 10 Tomoaki AOKI 2022-05-10 16:56:08 UTC
(In reply to Mark Johnston from comment #8)

> Oh.  How did you build ipfw.ko?  Did you build it as a standalone module, or as > part of a buildkernel?  In the latter case, do you have "makeoptions
> WITH_CTF=1" in the kernel configuration?

As part of buildkernel process. I basically don't build modules in base standalone.

And my kernel configuration on main is as below, stripping comments.

include GENERIC-NODEBUG
ident   confname
options CAM_IOSCHED_DYNAMIC

As GENERIC-NODEBUG includes GENERIC, and /usr/src/sys/conf/std.nodebug doesn't seem to overwrite the makeoptions, there should be, theoretically, "makeoptions WITH_CTF=1".


Kerel configuration on stable/13 is different, as stable/13 doesn't have GENERIC-NODEBUG to include.
I've copied old-style GENERIC-NODEBUG from main (current, ATM) and applied minimal change (change ident line and add options CAM_IOSCHED_DYNAMIC).

include         GENERIC
ident           confname
nooptions       DDB
nooptions       GDB
nooptions       INVARIANTS
nooptions       INVARIANT_SUPPORT
nooptions       WITNESS
nooptions       WITNESS_SKIPSPIN
nooptions       DEADLKRES
options         CAM_IOSCHED_DYNAMIC
Comment 11 Tomoaki AOKI 2022-05-15 05:59:20 UTC
(In reply to Tomoaki AOKI from comment #10)

Problem solved.
I've missed one setting in /etc/make.conf only on main, which I believed I've commented out several years ago, when sys/amd64/conf/GENERIC-NODEBUG was introduced and switching to use it.

.if ${.CURDIR:M/usr/src/*}
DEBUG_FLAGS=
.endif

Commentig out this whole part solved the issue.
bz@'s script ran OK and java/openjdk11 builds fine.

The problematic part was one of the settings to disable debug settings.
I thought it also is needed to disable debug on -current ATM.

Remaining question is that why I've never bitten by this before, IIRC.

Anyway, I'll close this. Sorry for the noise.
Comment 12 Tomoaki AOKI 2022-05-15 06:01:31 UTC
Closing. Not a bug but mis-setting on /etc/make.conf.
Sorry for the noise. Hope this helps for someone bitten by the same mis-setting.