Bug 232675

Summary: dtrace: "failed to compile script" when more than 2^15 distinct types are in CTF info
Product: Base System Reporter: Yasuhiro Kimura <yasu>
Component: binAssignee: Mark Johnston <markj>
Status: Closed FIXED    
Severity: Affects Some People CC: FreeBSD, alexr, brooks, cem, dim, dtrace, emaste, freebsd, i.dani, koobs, ksbeattie, luporl, lwhsu, markj, mike-wesley, naito.yuichiro, ngie, np, pi, sigsys, zarychtam
Priority: --- Keywords: needs-patch, needs-qa
Version: CURRENTFlags: zarychtam: mfc-stable13?
koobs: mfc-stable12?
koobs: mfc-stable11?
Hardware: Any   
OS: Any   
URL: https://reviews.freebsd.org/D17877
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242689
Bug Depends on:    
Bug Blocks: 227312    

Description Yasuhiro Kimura freebsd_committer freebsd_triage 2018-10-25 09:13:27 UTC
On 13-CURRENT base r339548 build of lang/perl5.26 fails with configure error.

----------------------------------------------------------------------
yasu@rolling-vm-freebsd1[2100]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 13.0-CURRENT FreeBSD 13.0-CURRENT r339548 GENERIC_UTAHIME  amd64
yasu@rolling-vm-freebsd1[2101]% pwd
/usr/ports/lang/perl5.26
yasu@rolling-vm-freebsd1[2102]% make
===>  License ART10 GPLv1+ accepted by the user
===>   perl5.26-5.26.2 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by perl5.26-5.26.2 for building
===>  Extracting for perl5.26-5.26.2
=> SHA256 Checksum OK for perl/perl-5.26.2.tar.xz.
/bin/ln -s libperl.so.5.26.2 /usr0/freebsd/ports/work/net/freebsd/ports/head/lang/perl5.26/work/perl-5.26.2/libperl.so
/bin/ln -s libperl.so.5.26.2 /usr0/freebsd/ports/work/net/freebsd/ports/head/lang/perl5.26/work/perl-5.26.2/libperl.so.5.26
===>  Patching for perl5.26-5.26.2
===>  Applying FreeBSD patches for perl5.26-5.26.2
/usr/bin/sed -i.bak -e 's|/usr/local|/usr/local|g'  /usr0/freebsd/ports/work/net/freebsd/ports/head/lang/perl5.26/work/perl-5.26.2/Configure /usr0/freebsd/ports/work/net/freebsd/ports/head/lang/perl5.26/work/perl-5.26.2/hints/freebsd.sh
/usr/bin/sed -i.bak -e '/do_installprivlib = 0 if .versiononly/d;  /^if.*nopods.*versiononly || /s/.*/if (1) {/'  /usr0/freebsd/ports/work/net/freebsd/ports/head/lang/perl5.26/work/perl-5.26.2/installperl
===>  Configuring for perl5.26-5.26.2
First let's make sure your kit is complete.  Checking...

(snip)

Colon-separated list of additional directories for perl to search? [none]  
Checking out function prototypes...
Support DTrace if available? [y]  
Where is the dtrace executable? (~name ok) [/usr/sbin/dtrace]  

*** Configure:  Fatal Error:  /usr/sbin/dtrace doesn't support -h flag
***
*** Your installed dtrace doesn't support the -h switch to compile a D
*** program into a C header. Can't continue.

===>  Script "Configure" failed unexpectedly.
Please report the problem to mat@FreeBSD.org [maintainer] and attach the
"/usr0/freebsd/ports/work/net/freebsd/ports/head/lang/perl5.26/work/perl-5.26.2/config.log"
including the output of the failure of your make command. Also, it might be
a good idea to provide an overview of all packages installed on your system
(e.g. a /usr/local/sbin/pkg-static info -g -Ea).
*** Error code 1

Stop.
make: stopped in /net/freebsd/ports/head/lang/perl5.26
yasu@rolling-vm-freebsd1[2103]% 
----------------------------------------------------------------------

In perl-5.26.2/Configure there is following test.

----------------------------------------------------------------------
	if $test -f $dtrace
	then
		if $dtrace -h -s ../perldtrace.d \
			-o perldtrace.tmp >/dev/null 2>&1 \
			&& rm -f perldtrace.tmp
		then
			echo " "
			echo "Good: your $dtrace knows about the -h flag."						    
		else
			cat >&2 <<EOM

*** $me:  Fatal Error:	$dtrace doesn't support -h flag
***
*** Your installed dtrace doesn't support the -h switch to compile a D							    
*** program into a C header. Can't continue.

EOM
			exit 1
		fi
		break;
	fi
----------------------------------------------------------------------

And when I executed this test on command line, it result in following error.

----------------------------------------------------------------------
yasu@rolling-vm-freebsd1[2056]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 13.0-CURRENT FreeBSD 13.0-CURRENT r339548 GENERIC_UTAHIME  amd64
yasu@rolling-vm-freebsd1[2057]% pwd
/home/yasu/tmp/perl-5.26.2
yasu@rolling-vm-freebsd1[2058]% dtrace -h -s perldtrace.d -o perldtrace.tmp
dtrace: failed to compile script perldtrace.d: "/usr/lib/dtrace/mbuf.d", line 114: failed to copy type of 'm_data': Type information is in parent and unavailable
yasu@rolling-vm-freebsd1[2059]%
----------------------------------------------------------------------

But on 12.0-BETA1 it succeeds without any error.

----------------------------------------------------------------------
yasu@rolling-vm-freebsd3[2006]% uname -a
FreeBSD rolling-vm-freebsd3.home.utahime.org 12.0-BETA1 FreeBSD 12.0-BETA1 r339435 GENERIC_UTAHIME  amd64
yasu@rolling-vm-freebsd3[2007]% pwd
/home/yasu/tmp/perl-5.26.2
yasu@rolling-vm-freebsd3[2008]% dtrace -h -s perldtrace.d -o perldtrace.tmp
yasu@rolling-vm-freebsd3[2009]%
----------------------------------------------------------------------

So there is regression about dtrace between base r339436 and base r339548.
Comment 1 Mark Johnston freebsd_committer freebsd_triage 2018-10-25 14:25:58 UTC
This looks like an issue with the CTF embedded in your 13.0 kernel.  Could you please paste the output of "ctfdump -S /boot/kernel/kernel" here?
Comment 2 Yasuhiro Kimura freebsd_committer freebsd_triage 2018-10-25 14:47:44 UTC
(In reply to Mark Johnston from comment #1)

yasu@rolling-vm-freebsd1[2003]% ctfdump -S /boot/kernel/kernel

- CTF Statistics -------------------------------------------------------------

  total number of data objects        = 31888

  total number of functions           = 36329
  total number of function arguments  = 87818
  maximum argument list length        = 22
  average argument list length        = 2.42

  total number of types               = 38599
  total number of integers            = 75
  total number of floats              = 1
  total number of pointers            = 10290
  total number of arrays              = 3477
  total number of func types          = 3207
  total number of structs             = 12222
  total number of unions              = 650
  total number of enums               = 934
  total number of forward tags        = 44
  total number of typedefs            = 6849
  total number of volatile types      = 59
  total number of const types         = 646
  total number of restrict types      = 0
  total number of unknowns (holes)    = 145

  total number of struct members      = 94951
  maximum number of struct members    = 248
  total size of all structs           = 22179594
  maximum size of a struct            = 2971720
  average number of struct members    = 7.77
  average size of a struct            = 1814.73

  total number of union members       = 2258
  maximum number of union members     = 36
  total size of all unions            = 141759
  maximum size of a union             = 41576
  average number of union members     = 3.47
  average size of a union             = 218.09

  total number of enum members        = 9255
  maximum number of enum members      = 1023
  average number of enum members      = 9.91

  total number of unique strings      = 70147
  bytes of string data                = 1008600
  maximum string length               = 81
  average string length               = 14.38

yasu@rolling-vm-freebsd1[2004]%
Comment 3 Mark Johnston freebsd_committer freebsd_triage 2018-10-25 15:17:03 UTC
(In reply to Yasuhiro KIMURA from comment #2)
So, this is an old problem caused by having more than 2^15 distinct types in the CTF info (you have 38599).  At the moment the only workaround I can suggest is to remove some unneeded drivers from your kernel config.  I note that a plain 13-CURRENT GENERIC kernel does not hit this limit, so you must have added extra drivers to your config.
Comment 4 Yasuhiro Kimura freebsd_committer freebsd_triage 2018-10-25 16:08:18 UTC
(In reply to Mark Johnston from comment #3)

Yes, I use custom kernel config that adds AUTOFS, NULLFS and ZFS to GENERIC. So I will try your suggestion.

BTW I tried build of lang/perl5.26 with poudriere and 13-CURRENT base r339548 jail but build completed successfully. Why the problem doesn't happen inside jail?
Comment 5 Mark Johnston freebsd_committer freebsd_triage 2018-10-25 16:45:45 UTC
(In reply to Yasuhiro KIMURA from comment #4)
Are the /usr/lib/dtrace/*.d scripts present in your jail?
Comment 6 Yasuhiro Kimura freebsd_committer freebsd_triage 2018-10-25 16:59:47 UTC
(In reply to Mark Johnston from comment #5)

Yes, but drti.o is different.

yasu@rolling-vm-freebsd1[2056]% ls /usr/local/poudriere/jails/curamd64/usr/lib/dtrace
drti.o          ip.d            regs_x86.d      signal.d        udp.d
errno.d         mbuf.d          sctp.d          socket.d        udplite.d
io.d            psinfo.d        siftr.d         tcp.d           unistd.d
yasu@rolling-vm-freebsd1[2057]% diff -ru /usr/local/poudriere/jails/curamd64/usr/lib/dtrace /usr/lib/dtrace
Binary files /usr/local/poudriere/jails/curamd64/usr/lib/dtrace/drti.o and /usr/lib/dtrace/drti.o differ
yasu@rolling-vm-freebsd1[2058]%
Comment 7 Yasuhiro Kimura freebsd_committer freebsd_triage 2018-10-25 18:33:55 UTC
(In reply to Yasuhiro KIMURA from comment #4)

I switched to GENERIC config, recompiled kernel and rebooted. But total number of types (35336) is still larger than 2^15 and problem happens.

yasu@rolling-vm-freebsd1[2016]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 13.0-CURRENT FreeBSD 13.0-CURRENT r339548 GENERIC  amd64
yasu@rolling-vm-freebsd1[2017]% ctfdump -S /boot/kernel/kernel

- CTF Statistics -------------------------------------------------------------

  total number of data objects        = 30274

  total number of functions           = 32705
  total number of function arguments  = 78889
  maximum argument list length        = 22
  average argument list length        = 2.41

  total number of types               = 35336
  total number of integers            = 75
  total number of floats              = 0
  total number of pointers            = 9313
  total number of arrays              = 3267
  total number of func types          = 2954
  total number of structs             = 11418
  total number of unions              = 623
  total number of enums               = 819
  total number of forward tags        = 45
  total number of typedefs            = 6036
  total number of volatile types      = 58
  total number of const types         = 568
  total number of restrict types      = 0
  total number of unknowns (holes)    = 160

  total number of struct members      = 86220
  maximum number of struct members    = 248
  total size of all structs           = 17969864
  maximum size of a struct            = 2971720
  average number of struct members    = 7.55
  average size of a struct            = 1573.82

  total number of union members       = 2207
  maximum number of union members     = 36
  total size of all unions            = 140335
  maximum size of a union             = 41576
  average number of union members     = 3.54
  average size of a union             = 225.26

  total number of enum members        = 8206
  maximum number of enum members      = 1023
  average number of enum members      = 10.02

  total number of unique strings      = 64998
  bytes of string data                = 938044
  maximum string length               = 81
  average string length               = 14.43

yasu@rolling-vm-freebsd1[2018]% pwd
/home/yasu/tmp/perl-5.26.2
yasu@rolling-vm-freebsd1[2019]% dtrace -h -s perldtrace.d -o perldtrace.tmp
dtrace: failed to compile script perldtrace.d: "/usr/lib/dtrace/mbuf.d", line 114: failed to copy type of 'm_data': Type information is in parent and unavailable
yasu@rolling-vm-freebsd1[2020]%
Comment 8 Yasuhiro Kimura freebsd_committer freebsd_triage 2018-10-27 09:37:45 UTC
(In reply to Yasuhiro KIMURA from comment #7)

I tried update to base r339772 and total number of types got fewer than 2^15 with GENERAL config.

yasu@rolling-vm-freebsd1[2004]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 13.0-CURRENT FreeBSD 13.0-CURRENT r339772 GENERIC  amd64
yasu@rolling-vm-freebsd1[2005]% ctfdump -S /boot/kernel/kernel

- CTF Statistics -------------------------------------------------------------

  total number of data objects        = 30165

  total number of functions           = 32605
  total number of function arguments  = 78712
  maximum argument list length        = 22
  average argument list length        = 2.41

  total number of types               = 29947
  total number of integers            = 75
  total number of floats              = 0
  total number of pointers            = 7329
  total number of arrays              = 3205
  total number of func types          = 2030
  total number of structs             = 10035
  total number of unions              = 564
  total number of enums               = 807
  total number of forward tags        = 45
  total number of typedefs            = 5115
  total number of volatile types      = 53
  total number of const types         = 559
  total number of restrict types      = 0
  total number of unknowns (holes)    = 130

  total number of struct members      = 75099
  maximum number of struct members    = 248
  total size of all structs           = 17705982
  maximum size of a struct            = 2971720
  average number of struct members    = 7.48
  average size of a struct            = 1764.42

  total number of union members       = 1909
  maximum number of union members     = 36
  total size of all unions            = 132363
  maximum size of a union             = 41576
  average number of union members     = 3.38
  average size of a union             = 234.69

  total number of enum members        = 8078
  maximum number of enum members      = 1023
  average number of enum members      = 10.01

  total number of unique strings      = 64374
  bytes of string data                = 930045
  maximum string length               = 81
  average string length               = 14.45

yasu@rolling-vm-freebsd1[2006]%
Comment 9 Yasuhiro Kimura freebsd_committer freebsd_triage 2018-10-27 09:37:46 UTC
(In reply to Yasuhiro KIMURA from comment #7)

I tried update to base r339772 and total number of types got fewer than 2^15 with GENERAL config.

yasu@rolling-vm-freebsd1[2004]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 13.0-CURRENT FreeBSD 13.0-CURRENT r339772 GENERIC  amd64
yasu@rolling-vm-freebsd1[2005]% ctfdump -S /boot/kernel/kernel

- CTF Statistics -------------------------------------------------------------

  total number of data objects        = 30165

  total number of functions           = 32605
  total number of function arguments  = 78712
  maximum argument list length        = 22
  average argument list length        = 2.41

  total number of types               = 29947
  total number of integers            = 75
  total number of floats              = 0
  total number of pointers            = 7329
  total number of arrays              = 3205
  total number of func types          = 2030
  total number of structs             = 10035
  total number of unions              = 564
  total number of enums               = 807
  total number of forward tags        = 45
  total number of typedefs            = 5115
  total number of volatile types      = 53
  total number of const types         = 559
  total number of restrict types      = 0
  total number of unknowns (holes)    = 130

  total number of struct members      = 75099
  maximum number of struct members    = 248
  total size of all structs           = 17705982
  maximum size of a struct            = 2971720
  average number of struct members    = 7.48
  average size of a struct            = 1764.42

  total number of union members       = 1909
  maximum number of union members     = 36
  total size of all unions            = 132363
  maximum size of a union             = 41576
  average number of union members     = 3.38
  average size of a union             = 234.69

  total number of enum members        = 8078
  maximum number of enum members      = 1023
  average number of enum members      = 10.01

  total number of unique strings      = 64374
  bytes of string data                = 930045
  maximum string length               = 81
  average string length               = 14.45

yasu@rolling-vm-freebsd1[2006]%
Comment 10 Mark Johnston freebsd_committer freebsd_triage 2018-10-29 16:59:36 UTC
I get less than 30,000 types with GENERIC on both head/ and stable/12/.
Comment 11 Dimitry Andric freebsd_committer freebsd_triage 2018-11-05 19:02:03 UTC
(In reply to Mark Johnston from comment #10)
> I get less than 30,000 types with GENERIC on both head/ and stable/12/.

Just ran into the same problem with GENERIC on the clang700-import branch:

  total number of types               = 36917

This is with stock GENERIC on amd64.
Comment 12 Mark Johnston freebsd_committer freebsd_triage 2018-11-06 01:36:37 UTC
*** Bug 232995 has been marked as a duplicate of this bug. ***
Comment 13 Mark Johnston freebsd_committer freebsd_triage 2018-11-06 01:37:05 UTC
*** Bug 193109 has been marked as a duplicate of this bug. ***
Comment 14 Yasuhiro Kimura freebsd_committer freebsd_triage 2018-11-06 03:58:10 UTC
(In reply to Yasuhiro KIMURA from comment #9)

Last weekend I updated to base r340045 and total number of types got
larger than 2^15 with GENERAL config again.

yasu@rolling-vm-freebsd1[2025]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 13.0-CURRENT FreeBSD 13.0-CURRENT r340045 GENERIC  amd64
yasu@rolling-vm-freebsd1[2026]% ctfdump -S /boot/kernel/kernel| grep 'total number of types'
  total number of types               = 36905
Comment 15 Dimitry Andric freebsd_committer freebsd_triage 2018-11-06 06:58:17 UTC
I bisected last evening, building GENERIC from r335000 to r340156:

$ grep 'number of types' *.txt
types-r335000.txt:  total number of types               = 29914
types-r338000.txt:  total number of types               = 30049
types-r339000.txt:  total number of types               = 30086
types-r339300.txt:  total number of types               = 30087
types-r339400.txt:  total number of types               = 30110
types-r339450.txt:  total number of types               = 30110
types-r339460.txt:  total number of types               = 30111
types-r339465.txt:  total number of types               = 30111
types-r339467.txt:  total number of types               = 30111
types-r339468.txt:  total number of types               = 35162
types-r339470.txt:  total number of types               = 35162
types-r339475.txt:  total number of types               = 35162
types-r339500.txt:  total number of types               = 35162
types-r340156.txt:  total number of types               = 36685

So it seems r339468 ("Replace ttyprintf with sbuf_printf and tty drain routine
") by itself increases the number of types by almost 5000!  Strangely enough, nothing in the commit indicates this, and the author also didn't expect it.

There is likely something else going on too.
Comment 16 Mark Johnston freebsd_committer freebsd_triage 2018-11-06 19:21:55 UTC
(In reply to Dimitry Andric from comment #15)
With this commit I see many more duplicate type definitions than before.  This seems to be a bug in ctfmerge; if I merge all object files at once, the problem occurs, but if I incrementally merge one object at a time, it does not.
Comment 17 Mark Johnston freebsd_committer freebsd_triage 2018-11-06 21:49:48 UTC
https://reviews.freebsd.org/D17877
Comment 18 commit-hook freebsd_committer freebsd_triage 2018-11-06 23:42:07 UTC
A commit references this bug:

Author: markj
Date: Tue Nov  6 23:41:45 UTC 2018
New revision: 340208
URL: https://svnweb.freebsd.org/changeset/base/340208

Log:
  Avoid fixing the tty_info() buffer size in tty.h.

  Different compilation units may otherwise get a different view of the
  layout of struct tty depending on whether they include opt_printf.h.
  This caused a blowup in the number of types defined in the kernel's
  CTF file after r339468; thanks to dim@ for bisecting down to that
  revision.

  PR:		232675
  Reported by:	dim
  Reviewed by:	cem (previous version)
  Sponsored by:	The FreeBSD Foundation
  Differential Revision:	https://reviews.freebsd.org/D17877

Changes:
  head/sys/kern/tty.c
  head/sys/kern/tty_info.c
  head/sys/sys/tty.h
Comment 19 Yasuhiro Kimura freebsd_committer freebsd_triage 2018-11-07 19:25:28 UTC
(In reply to commit-hook from comment #18)

I applied this commit to source tree of base r340045 and recompiled kernel. Then total number of types got smaller than 2^15 with GENERAL config.

yasu@rolling-vm-freebsd1[2002]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r340045M: Thu Nov  8 03:49:00 JST 2018     rootz@rolling-vm-freebsd1.home.utahime.org:/usr0/freebsd/base/obj/usr0/freebsd/base/head/amd64.amd64/sys/GENERIC  amd64
yasu@rolling-vm-freebsd1[2003]% ctfdump -S /boot/kernel/kernel| grep 'total number of types'
  total number of types               = 29961
yasu@rolling-vm-freebsd1[2004]%
Comment 20 Mark Johnston freebsd_committer freebsd_triage 2018-11-07 21:03:37 UTC
(In reply to Yasuhiro KIMURA from comment #19)
Thanks for confirming that this fixes the problem.
Comment 21 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-28 05:22:34 UTC
Hitting this issue with: FreeBSD 13.0-CURRENT r348305 GENERIC-NODEBUG amd64

This causes the build (configure) failure of lang/perl5* (bug 227312)

What is the course of action to resolve the 2^15 limit permanently?

ctfdump -S /boot/kernel/kernel

- CTF Statistics -------------------------------------------------------------

  total number of data objects        = 29753

  total number of functions           = 32387
  total number of function arguments  = 78172
  maximum argument list length        = 22
  average argument list length        = 2.41

  total number of types               = 35498 <------
  total number of integers            = 76
  total number of floats              = 0
  total number of pointers            = 9481
  total number of arrays              = 3273
  total number of func types          = 3102
  total number of structs             = 11405
  total number of unions              = 633
  total number of enums               = 811
  total number of forward tags        = 49
  total number of typedefs            = 5893
  total number of volatile types      = 55
  total number of const types         = 588
  total number of restrict types      = 0
  total number of unknowns (holes)    = 132

  total number of struct members      = 87550
  maximum number of struct members    = 248
  total size of all structs           = 22415324
  maximum size of a struct            = 2971720
  average number of struct members    = 7.68
  average size of a struct            = 1965.39

  total number of union members       = 2091
  maximum number of union members     = 36
  total size of all unions            = 136952
  maximum size of a union             = 41576
  average number of union members     = 3.30
  average size of a union             = 216.35

  total number of enum members        = 8061
  maximum number of enum members      = 1023
  average number of enum members      = 9.94

  total number of unique strings      = 64074
  bytes of string data                = 926366
  maximum string length               = 81
  average string length               = 14.46
Comment 22 Conrad Meyer freebsd_committer freebsd_triage 2019-05-28 05:41:50 UTC
(In reply to Kubilay Kocak from comment #21)
> Hitting this issue with: FreeBSD 13.0-CURRENT r348305 GENERIC-NODEBUG amd64
> 
> What is the course of action to resolve the 2^15 limit permanently?

The problem generally indicates an error in the CTF type system, although we are getting fairly close to hitting that threshold anyway.

To get past 2^15 types we'd have to rev the format in some way.
Comment 23 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-28 05:51:46 UTC
(In reply to Conrad Meyer from comment #22)

Thanks Conrad.

@Triage: Reset assignee: Since this is a broader issue, Mark oughtn't be assumed Assignee (previous issue resolver with workaround) unless he wants to explicitly take it

If the format revision is something that definitely couldn't be MRC'd, the mfc-* flags may be cancelled.
Comment 24 Mark Johnston freebsd_committer freebsd_triage 2019-05-28 14:08:20 UTC
The problem was triggered by a certain src commit.  I reported the issue yesterday: https://lists.freebsd.org/pipermail/svn-src-all/2019-May/181386.html

It will probably be another day or two before it is resolved.  In the meantime all I can suggest is to temporarily disable DTrace support in perl (i.e., make it non-default), since I presume the issue is blocking pkg builds.
Comment 25 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-28 14:15:28 UTC
(In reply to Mark Johnston from comment #24)

Thanks for the update Mark.

Indeed lang/perl5.* ports default to DTRACE=on, so these will be effected for the head package builders post base r348300

Even after the short-term ctf count 'regression' is resolved though (please reference this PR), I'd like to this issue to remain open as a tracking issuefor permanent resolution, "format revision" or similar per comment 22, or whatever form that may take.
Comment 26 Mark Johnston freebsd_committer freebsd_triage 2019-05-28 16:59:41 UTC
(In reply to Kubilay Kocak from comment #25)
Kirk fixed the problem in src r348329, so we can avoid updating the port, I think.

I'm happy to have a PR for the general issue, but I think it should be a new one.  Most instances of this failure are caused by something introducing duplicate types, which is a problem in its own right regardless of CTF's limit on the size of the type graph.
Comment 27 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-29 03:05:52 UTC
(In reply to Mark Johnston from comment #26)

That may be one cause but the root issue is the 2^15 limit. It would be preferable (in this case) not to further complicate the bug hierarchy as this issue captures both the impact (ports failing to build), some of the causes (commits creating duplicate types), and the root cause already
Comment 28 Dimitry Andric freebsd_committer freebsd_triage 2019-06-22 06:57:54 UTC
For some reason, the effect is also much more pronounced on i386 than on amd64, which I noticed because a bunch of dtrace-enabled ports (such as perl and node) failed only on i386.

E.g.:

revision  amd64  i386
========  =====  =====
r348302   35900  39487
r349011   29976  36473

Something seems to have happened between r348302 and r348980 to lower the number of types on amd64 from 35900 to 29976, and it stayed there ever since.

But the trick, whatever it was, did not appear to work for i386...
Comment 29 Dimitry Andric freebsd_committer freebsd_triage 2019-07-01 17:46:03 UTC
(In reply to Dimitry Andric from comment #28)
> Something seems to have happened between r348302 and r348980 to lower the
> number of types on amd64 from 35900 to 29976, and it stayed there ever since.

Here's a more extended overview:

revision  types   committer  message
========  ======  =========  ===================================================
r346000   27,653
r347000   27,664
r347409   27,673
r347410   33,950  gallatin   Remove IPSEC from GENERIC due to performance issues
r347412   33,950
r347908   33,953
r347912   33,013  brooks     FCP-101: Remove ep(4)
r347913   35,427  brooks     FCP-101: Remove ex(4)
r347914   34,650  brooks     FCP-101: Remove fe(4)
r347915   30,190  brooks     FCP-101: Remove pcn(4)
r347916   29,953  brooks     FCP-101: Remove sf(4)
r347920   29,929  brooks     FCP-101: Remove txp(4)
r347921   27,405  brooks     FCP-101: Remove vx(4)
r347922   29,979  brooks     FCP-101: Remove wb(4)
r347923   36,415  brooks     FCP-101: Remove xe(4)
r347924   36,415
r349365   36,486

So for some reason, the removal of features (or moving them to modules) can greatly bloat the number of types.

For example, the removal of ex(4) in r347913 increases it by 2,414, most of the other removals decrease it, and lastly the removal of xe(4) increases it by 6,436!
Comment 30 Mark Johnston freebsd_committer freebsd_triage 2019-07-09 04:20:44 UTC
As usual, the problem is caused by duplicates in the type graph.  A recent i386 kernel has 10 copies of struct thread in its CTF info.  This is usually the result of a bug somewhere that was indirectly triggered or exacerbated by the referenced kernel changes.
Comment 31 Alan Somers freebsd_committer freebsd_triage 2019-12-19 04:00:48 UTC
*** Bug 242689 has been marked as a duplicate of this bug. ***
Comment 32 Mark Johnston freebsd_committer freebsd_triage 2019-12-20 22:38:52 UTC
(In reply to Mark Johnston from comment #30)
I spent some time digging into this; the problem appears to be that we define struct pmap, once for the PAE pmap and once for the non-PAE pmap, and this structure is embedded in struct vmspace, which results in bifurcation of the type graph.  I'll try to find a way to fix this that isn't too ugly.
Comment 33 Shane 2021-05-11 08:23:31 UTC
I have run into this building perl5.32 and tcl86 in poudriere. tcl86 reports the fail as a syntax error.

Both perl5.32 and tcl86 build on the host but fail in a matching poudriere jail.


host stable/13 e284a6234ffb - both build from ports

jail stable/13 e284a6234 - both fail
jail 13.0 release - both fail
jail 12.2 - both build
Comment 34 Kurt Jaeger freebsd_committer freebsd_triage 2021-12-01 20:05:24 UTC
(In reply to Shane from comment #33)
Still the same problem on 14.0:

/usr/sbin/dtrace -h  -o tclDTrace.h -s /wrkdirs/usr/ports/lang/tcl86/work/tcl8.6.12/generic/tclDTrace.d
dtrace: failed to compile script /wrkdirs/usr/ports/lang/tcl86/work/tcl8.6.12/generic/tclDTrace.d: "/usr/lib/dtrace/ipfw.d", line 1: syntax error near "in_addr_t"
*** [tclDTrace.h] Error code 1
Comment 35 Mark Johnston freebsd_committer freebsd_triage 2021-12-01 20:10:46 UTC
(In reply to Kurt Jaeger from comment #34)
Which revision?  Several libctf problems relating to ipfw.d have been fixed lately.  I can build tcl with dtrace enabled on main at 28dcccc1296.
Comment 36 Kurt Jaeger freebsd_committer freebsd_triage 2021-12-01 20:32:11 UTC
(In reply to Mark Johnston from comment #35)
version: 225639e7db68: Tue Oct 19 13:10:59 CEST 2021

Can it be that the build problem is tcl86 specific ?

There is PR#224857, which led me via PR#227312 to this PR.
Comment 37 Mark Johnston freebsd_committer freebsd_triage 2021-12-01 20:34:57 UTC
(In reply to Kurt Jaeger from comment #36)
Could you please show ctfdump -S output?
Comment 38 Kurt Jaeger freebsd_committer freebsd_triage 2021-12-01 20:38:00 UTC
(In reply to Mark Johnston from comment #37)
- CTF Statistics -------------------------------------------------------------

  total number of data objects        = 31755

  total number of functions           = 35492
  total number of function arguments  = 85612
  maximum argument list length        = 22
  average argument list length        = 2.41

  total number of types               = 32234
  total number of integers            = 78
  total number of floats              = 0
  total number of pointers            = 7847
  total number of arrays              = 3284
  total number of func types          = 2194
  total number of structs             = 10774
  total number of unions              = 632
  total number of enums               = 1003
  total number of forward tags        = 51
  total number of typedefs            = 5511
  total number of volatile types      = 56
  total number of const types         = 641
  total number of restrict types      = 6
  total number of unknowns (holes)    = 157

  total number of struct members      = 78966
  maximum number of struct members    = 248
  total size of all structs           = 19578781
  maximum size of a struct            = 2971720
  average number of struct members    = 7.33
  average size of a struct            = 1817.22

  total number of union members       = 2173
  maximum number of union members     = 90
  total size of all unions            = 134506
  maximum size of a union             = 41584
  average number of union members     = 3.44
  average size of a union             = 212.83

  total number of enum members        = 9544
  maximum number of enum members      = 1023
  average number of enum members      = 9.52

  total number of unique strings      = 69017
  bytes of string data                = 1008036
  maximum string length               = 81
  average string length               = 14.61
Comment 39 Mark Johnston freebsd_committer freebsd_triage 2021-12-01 20:45:25 UTC
(In reply to Kurt Jaeger from comment #38)
I am not sure what the problem is, then.  It's not the one from this PR, and I think you have the relevant libctf fixes (https://cgit.freebsd.org/src/commit/?id=105fd928b0b5b35ab529e5f6914788dc49582901 is the most recent), and I cannot reproduce the error by building either tcl86 or 87 with ipfw.ko loaded.

If it's possible that your userland/kernel are out of sync, or you have an out-of-date /usr/lib/dtrace/ipfw.d, then I'd suggest updating everything.
Comment 40 Kubilay Kocak freebsd_committer freebsd_triage 2021-12-01 23:12:49 UTC
@Mark Is there anything we can do, tests or otherwise, to make these build (ideally) or test (CI) time failures, so as to pick them up early and/or reduce or otherwise preclude false positives or user/environment specific build issues?
Comment 41 Mark Johnston freebsd_committer freebsd_triage 2021-12-03 14:57:50 UTC
(In reply to Kubilay Kocak from comment #40)
Certainly it'd be useful to add dtrace tests for new SDT providers, like we have for ip, tcp, sctp, etc..
Comment 42 Dani I. 2022-01-08 14:37:11 UTC
We are hitting the same bug after upgrading from 12.2 to 12.3 (same custom kernelconfig). Nothing else has changed from our side. Guess that has to do with the additional stuff that came to GENERIC with the update to 12.3. Is there a consistent resolution yet..?



- CTF Statistics -------------------------------------------------------------

  total number of data objects        = 31625

  total number of functions           = 35289
  total number of function arguments  = 85628
  maximum argument list length        = 22
  average argument list length        = 2.43

  total number of types               = 33462
  total number of integers            = 77
  total number of floats              = 0
  total number of pointers            = 8270
  total number of arrays              = 3473
  total number of func types          = 2341
  total number of structs             = 11377
  total number of unions              = 661
  total number of enums               = 954
  total number of forward tags        = 52
  total number of typedefs            = 5439
  total number of volatile types      = 58
  total number of const types         = 621
  total number of restrict types      = 0
  total number of unknowns (holes)    = 139

  total number of struct members      = 85074
  maximum number of struct members    = 248
  total size of all structs           = 23518133
  maximum size of a struct            = 2971720
  average number of struct members    = 7.48
  average size of a struct            = 2067.16

  total number of union members       = 2295
  maximum number of union members     = 90
  total size of all unions            = 137260
  maximum size of a union             = 41576
  average number of union members     = 3.47
  average size of a union             = 207.66

  total number of enum members        = 9414
  maximum number of enum members      = 1023
  average number of enum members      = 9.87

  total number of unique strings      = 70178
  bytes of string data                = 1017606
  maximum string length               = 81
  average string length               = 14.50
Comment 43 Dani I. 2022-01-08 14:39:34 UTC
If helpfull: Here are the stats from the same custom kernelconfig with FBSD 12.2:

- CTF Statistics -------------------------------------------------------------

  total number of data objects        = 31042

  total number of functions           = 33926
  total number of function arguments  = 82409
  maximum argument list length        = 22
  average argument list length        = 2.43

  total number of types               = 31478
  total number of integers            = 77
  total number of floats              = 0
  total number of pointers            = 7668
  total number of arrays              = 3340
  total number of func types          = 2091
  total number of structs             = 10738
  total number of unions              = 621
  total number of enums               = 886
  total number of forward tags        = 51
  total number of typedefs            = 5251
  total number of volatile types      = 55
  total number of const types         = 580
  total number of restrict types      = 0
  total number of unknowns (holes)    = 120

  total number of struct members      = 79250
  maximum number of struct members    = 248
  total size of all structs           = 19349614
  maximum size of a struct            = 2971720
  average number of struct members    = 7.38
  average size of a struct            = 1801.98

  total number of union members       = 2151
  maximum number of union members     = 90
  total size of all unions            = 133604
  maximum size of a union             = 41576
  average number of union members     = 3.46
  average size of a union             = 215.14

  total number of enum members        = 9025
  maximum number of enum members      = 1023
  average number of enum members      = 10.19

  total number of unique strings      = 67943
  bytes of string data                = 982977
  maximum string length               = 81
  average string length               = 14.47


Here's the kernelconfig (yes some stuff is double with GENERIC, we just want to make sure these options are set):
include GENERIC

ident CUSTOM

options         DDB

options         IPFIREWALL
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_VERBOSE_LIMIT=5
options         IPFIREWALL_DEFAULT_TO_ACCEPT
options         IPDIVERT

device          pf
device          pflog
device          pfsync

device          enc

device          cpuctl

device          tuntap
Comment 45 Mark Johnston freebsd_committer freebsd_triage 2022-03-15 19:56:13 UTC
*** Bug 227312 has been marked as a duplicate of this bug. ***
Comment 46 Marek Zarychta 2022-03-31 20:06:25 UTC
I am trying to run dtrace(1) on:
FreeBSD 14.0-CURRENT #7 main-n253931-9b597132aea: Thu Mar 24 10:24:28 CET 2022
 
#dtrace -l
    ID   PROVIDER            MODULE                          FUNCTION NAME
    1     dtrace                                                     BEGIN
    2     dtrace                                                     END
    3     dtrace                                                     ERROR

what still seems to be not correct while
# ctfdump -S /boot/kernel/kernel | grep "total number of types"
  total number of types               = 32821
Comment 47 Mark Johnston freebsd_committer freebsd_triage 2022-03-31 20:07:55 UTC
(In reply to Marek Zarychta from comment #46)
That would seem to suggest that you have only dtrace.ko loaded, and not fbt.ko, sdt.ko etc..  Try loading dtraceall.ko instead.
Comment 48 Marek Zarychta 2022-03-31 20:08:05 UTC
*** Bug 262961 has been marked as a duplicate of this bug. ***
Comment 49 Marek Zarychta 2022-03-31 20:14:17 UTC
(In reply to Mark Johnston from comment #47)
>That would seem to suggest that you have only dtrace.ko loaded, and not fbt.ko, >sdt.ko etc..  Try loading dtraceall.ko instead.

Thanks for the clue and for fixing this. I haven't used DTrace(1) for a while. I am sorry for making noise here.
Comment 50 Mark Johnston freebsd_committer freebsd_triage 2022-04-07 00:31:44 UTC
(In reply to Mark Johnston from comment #44)
These are now merged to stable/13.
Comment 51 Mike 2023-02-23 14:36:11 UTC
(In reply to Kurt Jaeger from comment #34)
I just got this exact same error.


===> Building for tcl86-8.6.13
/usr/sbin/dtrace -h  -o tclDTrace.h -s /usr/ports/lang/tcl86/work/tcl8.6.13/generic/tclDTrace.d
dtrace: failed to compile script /usr/ports/lang/tcl86/work/tcl8.6.13/generic/tclDTrace.d: "/usr/lib/dtrace/ipfw.d", line 1: syntax error near "in_addr_t"
*** [tclDTrace.h] Error code 1

make[2]: stopped in /usr/ports/lang/tcl86/work/tcl8.6.13/unix
1 error

make[2]: stopped in /usr/ports/lang/tcl86/work/tcl8.6.13/unix
===> Compilation failed unexpectedly.


Running TrueNAS 13.1-RELEASE-p4
Comment 52 Alex Rosenberg 2023-08-13 22:33:29 UTC
Also hitting this with my poudriere jail building perl5-5.34 on TrueNAS-13.0-U5.3.

  total number of data objects        = 37964

Reported as https://ixsystems.atlassian.net/browse/NAS-123532
Comment 53 Cory Albrecht 2023-12-24 00:26:18 UTC
I am also hitting the "in_addr_t" bug with ipfw.d building with poudriere in a 13.1-RELEASE-p9 jail on TrueNAS