Bug 232675

Summary: dtrace: "failed to compile script" when more than 2^15 distinct types are in CTF info
Product: Base System Reporter: Yasuhiro KIMURA <yasu>
Component: binAssignee: Mark Johnston <markj>
Status: Open ---    
Severity: Affects Some People CC: brooks, cem, dim, dtrace, koobs, markj, naito.yuichiro, ngie
Priority: --- Keywords: needs-patch, needs-qa
Version: CURRENTFlags: koobs: mfc-stable11?
koobs: mfc-stable12?
Hardware: Any   
OS: Any   
URL: https://reviews.freebsd.org/D17877
Bug Depends on:    
Bug Blocks: 227312    

Description Yasuhiro KIMURA 2018-10-25 09:13:27 UTC
On 13-CURRENT base r339548 build of lang/perl5.26 fails with configure error.

----------------------------------------------------------------------
yasu@rolling-vm-freebsd1[2100]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 13.0-CURRENT FreeBSD 13.0-CURRENT r339548 GENERIC_UTAHIME  amd64
yasu@rolling-vm-freebsd1[2101]% pwd
/usr/ports/lang/perl5.26
yasu@rolling-vm-freebsd1[2102]% make
===>  License ART10 GPLv1+ accepted by the user
===>   perl5.26-5.26.2 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by perl5.26-5.26.2 for building
===>  Extracting for perl5.26-5.26.2
=> SHA256 Checksum OK for perl/perl-5.26.2.tar.xz.
/bin/ln -s libperl.so.5.26.2 /usr0/freebsd/ports/work/net/freebsd/ports/head/lang/perl5.26/work/perl-5.26.2/libperl.so
/bin/ln -s libperl.so.5.26.2 /usr0/freebsd/ports/work/net/freebsd/ports/head/lang/perl5.26/work/perl-5.26.2/libperl.so.5.26
===>  Patching for perl5.26-5.26.2
===>  Applying FreeBSD patches for perl5.26-5.26.2
/usr/bin/sed -i.bak -e 's|/usr/local|/usr/local|g'  /usr0/freebsd/ports/work/net/freebsd/ports/head/lang/perl5.26/work/perl-5.26.2/Configure /usr0/freebsd/ports/work/net/freebsd/ports/head/lang/perl5.26/work/perl-5.26.2/hints/freebsd.sh
/usr/bin/sed -i.bak -e '/do_installprivlib = 0 if .versiononly/d;  /^if.*nopods.*versiononly || /s/.*/if (1) {/'  /usr0/freebsd/ports/work/net/freebsd/ports/head/lang/perl5.26/work/perl-5.26.2/installperl
===>  Configuring for perl5.26-5.26.2
First let's make sure your kit is complete.  Checking...

(snip)

Colon-separated list of additional directories for perl to search? [none]  
Checking out function prototypes...
Support DTrace if available? [y]  
Where is the dtrace executable? (~name ok) [/usr/sbin/dtrace]  

*** Configure:  Fatal Error:  /usr/sbin/dtrace doesn't support -h flag
***
*** Your installed dtrace doesn't support the -h switch to compile a D
*** program into a C header. Can't continue.

===>  Script "Configure" failed unexpectedly.
Please report the problem to mat@FreeBSD.org [maintainer] and attach the
"/usr0/freebsd/ports/work/net/freebsd/ports/head/lang/perl5.26/work/perl-5.26.2/config.log"
including the output of the failure of your make command. Also, it might be
a good idea to provide an overview of all packages installed on your system
(e.g. a /usr/local/sbin/pkg-static info -g -Ea).
*** Error code 1

Stop.
make: stopped in /net/freebsd/ports/head/lang/perl5.26
yasu@rolling-vm-freebsd1[2103]% 
----------------------------------------------------------------------

In perl-5.26.2/Configure there is following test.

----------------------------------------------------------------------
	if $test -f $dtrace
	then
		if $dtrace -h -s ../perldtrace.d \
			-o perldtrace.tmp >/dev/null 2>&1 \
			&& rm -f perldtrace.tmp
		then
			echo " "
			echo "Good: your $dtrace knows about the -h flag."						    
		else
			cat >&2 <<EOM

*** $me:  Fatal Error:	$dtrace doesn't support -h flag
***
*** Your installed dtrace doesn't support the -h switch to compile a D							    
*** program into a C header. Can't continue.

EOM
			exit 1
		fi
		break;
	fi
----------------------------------------------------------------------

And when I executed this test on command line, it result in following error.

----------------------------------------------------------------------
yasu@rolling-vm-freebsd1[2056]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 13.0-CURRENT FreeBSD 13.0-CURRENT r339548 GENERIC_UTAHIME  amd64
yasu@rolling-vm-freebsd1[2057]% pwd
/home/yasu/tmp/perl-5.26.2
yasu@rolling-vm-freebsd1[2058]% dtrace -h -s perldtrace.d -o perldtrace.tmp
dtrace: failed to compile script perldtrace.d: "/usr/lib/dtrace/mbuf.d", line 114: failed to copy type of 'm_data': Type information is in parent and unavailable
yasu@rolling-vm-freebsd1[2059]%
----------------------------------------------------------------------

But on 12.0-BETA1 it succeeds without any error.

----------------------------------------------------------------------
yasu@rolling-vm-freebsd3[2006]% uname -a
FreeBSD rolling-vm-freebsd3.home.utahime.org 12.0-BETA1 FreeBSD 12.0-BETA1 r339435 GENERIC_UTAHIME  amd64
yasu@rolling-vm-freebsd3[2007]% pwd
/home/yasu/tmp/perl-5.26.2
yasu@rolling-vm-freebsd3[2008]% dtrace -h -s perldtrace.d -o perldtrace.tmp
yasu@rolling-vm-freebsd3[2009]%
----------------------------------------------------------------------

So there is regression about dtrace between base r339436 and base r339548.
Comment 1 Mark Johnston freebsd_committer 2018-10-25 14:25:58 UTC
This looks like an issue with the CTF embedded in your 13.0 kernel.  Could you please paste the output of "ctfdump -S /boot/kernel/kernel" here?
Comment 2 Yasuhiro KIMURA 2018-10-25 14:47:44 UTC
(In reply to Mark Johnston from comment #1)

yasu@rolling-vm-freebsd1[2003]% ctfdump -S /boot/kernel/kernel

- CTF Statistics -------------------------------------------------------------

  total number of data objects        = 31888

  total number of functions           = 36329
  total number of function arguments  = 87818
  maximum argument list length        = 22
  average argument list length        = 2.42

  total number of types               = 38599
  total number of integers            = 75
  total number of floats              = 1
  total number of pointers            = 10290
  total number of arrays              = 3477
  total number of func types          = 3207
  total number of structs             = 12222
  total number of unions              = 650
  total number of enums               = 934
  total number of forward tags        = 44
  total number of typedefs            = 6849
  total number of volatile types      = 59
  total number of const types         = 646
  total number of restrict types      = 0
  total number of unknowns (holes)    = 145

  total number of struct members      = 94951
  maximum number of struct members    = 248
  total size of all structs           = 22179594
  maximum size of a struct            = 2971720
  average number of struct members    = 7.77
  average size of a struct            = 1814.73

  total number of union members       = 2258
  maximum number of union members     = 36
  total size of all unions            = 141759
  maximum size of a union             = 41576
  average number of union members     = 3.47
  average size of a union             = 218.09

  total number of enum members        = 9255
  maximum number of enum members      = 1023
  average number of enum members      = 9.91

  total number of unique strings      = 70147
  bytes of string data                = 1008600
  maximum string length               = 81
  average string length               = 14.38

yasu@rolling-vm-freebsd1[2004]%
Comment 3 Mark Johnston freebsd_committer 2018-10-25 15:17:03 UTC
(In reply to Yasuhiro KIMURA from comment #2)
So, this is an old problem caused by having more than 2^15 distinct types in the CTF info (you have 38599).  At the moment the only workaround I can suggest is to remove some unneeded drivers from your kernel config.  I note that a plain 13-CURRENT GENERIC kernel does not hit this limit, so you must have added extra drivers to your config.
Comment 4 Yasuhiro KIMURA 2018-10-25 16:08:18 UTC
(In reply to Mark Johnston from comment #3)

Yes, I use custom kernel config that adds AUTOFS, NULLFS and ZFS to GENERIC. So I will try your suggestion.

BTW I tried build of lang/perl5.26 with poudriere and 13-CURRENT base r339548 jail but build completed successfully. Why the problem doesn't happen inside jail?
Comment 5 Mark Johnston freebsd_committer 2018-10-25 16:45:45 UTC
(In reply to Yasuhiro KIMURA from comment #4)
Are the /usr/lib/dtrace/*.d scripts present in your jail?
Comment 6 Yasuhiro KIMURA 2018-10-25 16:59:47 UTC
(In reply to Mark Johnston from comment #5)

Yes, but drti.o is different.

yasu@rolling-vm-freebsd1[2056]% ls /usr/local/poudriere/jails/curamd64/usr/lib/dtrace
drti.o          ip.d            regs_x86.d      signal.d        udp.d
errno.d         mbuf.d          sctp.d          socket.d        udplite.d
io.d            psinfo.d        siftr.d         tcp.d           unistd.d
yasu@rolling-vm-freebsd1[2057]% diff -ru /usr/local/poudriere/jails/curamd64/usr/lib/dtrace /usr/lib/dtrace
Binary files /usr/local/poudriere/jails/curamd64/usr/lib/dtrace/drti.o and /usr/lib/dtrace/drti.o differ
yasu@rolling-vm-freebsd1[2058]%
Comment 7 Yasuhiro KIMURA 2018-10-25 18:33:55 UTC
(In reply to Yasuhiro KIMURA from comment #4)

I switched to GENERIC config, recompiled kernel and rebooted. But total number of types (35336) is still larger than 2^15 and problem happens.

yasu@rolling-vm-freebsd1[2016]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 13.0-CURRENT FreeBSD 13.0-CURRENT r339548 GENERIC  amd64
yasu@rolling-vm-freebsd1[2017]% ctfdump -S /boot/kernel/kernel

- CTF Statistics -------------------------------------------------------------

  total number of data objects        = 30274

  total number of functions           = 32705
  total number of function arguments  = 78889
  maximum argument list length        = 22
  average argument list length        = 2.41

  total number of types               = 35336
  total number of integers            = 75
  total number of floats              = 0
  total number of pointers            = 9313
  total number of arrays              = 3267
  total number of func types          = 2954
  total number of structs             = 11418
  total number of unions              = 623
  total number of enums               = 819
  total number of forward tags        = 45
  total number of typedefs            = 6036
  total number of volatile types      = 58
  total number of const types         = 568
  total number of restrict types      = 0
  total number of unknowns (holes)    = 160

  total number of struct members      = 86220
  maximum number of struct members    = 248
  total size of all structs           = 17969864
  maximum size of a struct            = 2971720
  average number of struct members    = 7.55
  average size of a struct            = 1573.82

  total number of union members       = 2207
  maximum number of union members     = 36
  total size of all unions            = 140335
  maximum size of a union             = 41576
  average number of union members     = 3.54
  average size of a union             = 225.26

  total number of enum members        = 8206
  maximum number of enum members      = 1023
  average number of enum members      = 10.02

  total number of unique strings      = 64998
  bytes of string data                = 938044
  maximum string length               = 81
  average string length               = 14.43

yasu@rolling-vm-freebsd1[2018]% pwd
/home/yasu/tmp/perl-5.26.2
yasu@rolling-vm-freebsd1[2019]% dtrace -h -s perldtrace.d -o perldtrace.tmp
dtrace: failed to compile script perldtrace.d: "/usr/lib/dtrace/mbuf.d", line 114: failed to copy type of 'm_data': Type information is in parent and unavailable
yasu@rolling-vm-freebsd1[2020]%
Comment 8 Yasuhiro KIMURA 2018-10-27 09:37:45 UTC
(In reply to Yasuhiro KIMURA from comment #7)

I tried update to base r339772 and total number of types got fewer than 2^15 with GENERAL config.

yasu@rolling-vm-freebsd1[2004]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 13.0-CURRENT FreeBSD 13.0-CURRENT r339772 GENERIC  amd64
yasu@rolling-vm-freebsd1[2005]% ctfdump -S /boot/kernel/kernel

- CTF Statistics -------------------------------------------------------------

  total number of data objects        = 30165

  total number of functions           = 32605
  total number of function arguments  = 78712
  maximum argument list length        = 22
  average argument list length        = 2.41

  total number of types               = 29947
  total number of integers            = 75
  total number of floats              = 0
  total number of pointers            = 7329
  total number of arrays              = 3205
  total number of func types          = 2030
  total number of structs             = 10035
  total number of unions              = 564
  total number of enums               = 807
  total number of forward tags        = 45
  total number of typedefs            = 5115
  total number of volatile types      = 53
  total number of const types         = 559
  total number of restrict types      = 0
  total number of unknowns (holes)    = 130

  total number of struct members      = 75099
  maximum number of struct members    = 248
  total size of all structs           = 17705982
  maximum size of a struct            = 2971720
  average number of struct members    = 7.48
  average size of a struct            = 1764.42

  total number of union members       = 1909
  maximum number of union members     = 36
  total size of all unions            = 132363
  maximum size of a union             = 41576
  average number of union members     = 3.38
  average size of a union             = 234.69

  total number of enum members        = 8078
  maximum number of enum members      = 1023
  average number of enum members      = 10.01

  total number of unique strings      = 64374
  bytes of string data                = 930045
  maximum string length               = 81
  average string length               = 14.45

yasu@rolling-vm-freebsd1[2006]%
Comment 9 Yasuhiro KIMURA 2018-10-27 09:37:46 UTC
(In reply to Yasuhiro KIMURA from comment #7)

I tried update to base r339772 and total number of types got fewer than 2^15 with GENERAL config.

yasu@rolling-vm-freebsd1[2004]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 13.0-CURRENT FreeBSD 13.0-CURRENT r339772 GENERIC  amd64
yasu@rolling-vm-freebsd1[2005]% ctfdump -S /boot/kernel/kernel

- CTF Statistics -------------------------------------------------------------

  total number of data objects        = 30165

  total number of functions           = 32605
  total number of function arguments  = 78712
  maximum argument list length        = 22
  average argument list length        = 2.41

  total number of types               = 29947
  total number of integers            = 75
  total number of floats              = 0
  total number of pointers            = 7329
  total number of arrays              = 3205
  total number of func types          = 2030
  total number of structs             = 10035
  total number of unions              = 564
  total number of enums               = 807
  total number of forward tags        = 45
  total number of typedefs            = 5115
  total number of volatile types      = 53
  total number of const types         = 559
  total number of restrict types      = 0
  total number of unknowns (holes)    = 130

  total number of struct members      = 75099
  maximum number of struct members    = 248
  total size of all structs           = 17705982
  maximum size of a struct            = 2971720
  average number of struct members    = 7.48
  average size of a struct            = 1764.42

  total number of union members       = 1909
  maximum number of union members     = 36
  total size of all unions            = 132363
  maximum size of a union             = 41576
  average number of union members     = 3.38
  average size of a union             = 234.69

  total number of enum members        = 8078
  maximum number of enum members      = 1023
  average number of enum members      = 10.01

  total number of unique strings      = 64374
  bytes of string data                = 930045
  maximum string length               = 81
  average string length               = 14.45

yasu@rolling-vm-freebsd1[2006]%
Comment 10 Mark Johnston freebsd_committer 2018-10-29 16:59:36 UTC
I get less than 30,000 types with GENERIC on both head/ and stable/12/.
Comment 11 Dimitry Andric freebsd_committer 2018-11-05 19:02:03 UTC
(In reply to Mark Johnston from comment #10)
> I get less than 30,000 types with GENERIC on both head/ and stable/12/.

Just ran into the same problem with GENERIC on the clang700-import branch:

  total number of types               = 36917

This is with stock GENERIC on amd64.
Comment 12 Mark Johnston freebsd_committer 2018-11-06 01:36:37 UTC
*** Bug 232995 has been marked as a duplicate of this bug. ***
Comment 13 Mark Johnston freebsd_committer 2018-11-06 01:37:05 UTC
*** Bug 193109 has been marked as a duplicate of this bug. ***
Comment 14 Yasuhiro KIMURA 2018-11-06 03:58:10 UTC
(In reply to Yasuhiro KIMURA from comment #9)

Last weekend I updated to base r340045 and total number of types got
larger than 2^15 with GENERAL config again.

yasu@rolling-vm-freebsd1[2025]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 13.0-CURRENT FreeBSD 13.0-CURRENT r340045 GENERIC  amd64
yasu@rolling-vm-freebsd1[2026]% ctfdump -S /boot/kernel/kernel| grep 'total number of types'
  total number of types               = 36905
Comment 15 Dimitry Andric freebsd_committer 2018-11-06 06:58:17 UTC
I bisected last evening, building GENERIC from r335000 to r340156:

$ grep 'number of types' *.txt
types-r335000.txt:  total number of types               = 29914
types-r338000.txt:  total number of types               = 30049
types-r339000.txt:  total number of types               = 30086
types-r339300.txt:  total number of types               = 30087
types-r339400.txt:  total number of types               = 30110
types-r339450.txt:  total number of types               = 30110
types-r339460.txt:  total number of types               = 30111
types-r339465.txt:  total number of types               = 30111
types-r339467.txt:  total number of types               = 30111
types-r339468.txt:  total number of types               = 35162
types-r339470.txt:  total number of types               = 35162
types-r339475.txt:  total number of types               = 35162
types-r339500.txt:  total number of types               = 35162
types-r340156.txt:  total number of types               = 36685

So it seems r339468 ("Replace ttyprintf with sbuf_printf and tty drain routine
") by itself increases the number of types by almost 5000!  Strangely enough, nothing in the commit indicates this, and the author also didn't expect it.

There is likely something else going on too.
Comment 16 Mark Johnston freebsd_committer 2018-11-06 19:21:55 UTC
(In reply to Dimitry Andric from comment #15)
With this commit I see many more duplicate type definitions than before.  This seems to be a bug in ctfmerge; if I merge all object files at once, the problem occurs, but if I incrementally merge one object at a time, it does not.
Comment 17 Mark Johnston freebsd_committer 2018-11-06 21:49:48 UTC
https://reviews.freebsd.org/D17877
Comment 18 commit-hook freebsd_committer 2018-11-06 23:42:07 UTC
A commit references this bug:

Author: markj
Date: Tue Nov  6 23:41:45 UTC 2018
New revision: 340208
URL: https://svnweb.freebsd.org/changeset/base/340208

Log:
  Avoid fixing the tty_info() buffer size in tty.h.

  Different compilation units may otherwise get a different view of the
  layout of struct tty depending on whether they include opt_printf.h.
  This caused a blowup in the number of types defined in the kernel's
  CTF file after r339468; thanks to dim@ for bisecting down to that
  revision.

  PR:		232675
  Reported by:	dim
  Reviewed by:	cem (previous version)
  Sponsored by:	The FreeBSD Foundation
  Differential Revision:	https://reviews.freebsd.org/D17877

Changes:
  head/sys/kern/tty.c
  head/sys/kern/tty_info.c
  head/sys/sys/tty.h
Comment 19 Yasuhiro KIMURA 2018-11-07 19:25:28 UTC
(In reply to commit-hook from comment #18)

I applied this commit to source tree of base r340045 and recompiled kernel. Then total number of types got smaller than 2^15 with GENERAL config.

yasu@rolling-vm-freebsd1[2002]% uname -a
FreeBSD rolling-vm-freebsd1.home.utahime.org 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r340045M: Thu Nov  8 03:49:00 JST 2018     rootz@rolling-vm-freebsd1.home.utahime.org:/usr0/freebsd/base/obj/usr0/freebsd/base/head/amd64.amd64/sys/GENERIC  amd64
yasu@rolling-vm-freebsd1[2003]% ctfdump -S /boot/kernel/kernel| grep 'total number of types'
  total number of types               = 29961
yasu@rolling-vm-freebsd1[2004]%
Comment 20 Mark Johnston freebsd_committer 2018-11-07 21:03:37 UTC
(In reply to Yasuhiro KIMURA from comment #19)
Thanks for confirming that this fixes the problem.
Comment 21 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-28 05:22:34 UTC
Hitting this issue with: FreeBSD 13.0-CURRENT r348305 GENERIC-NODEBUG amd64

This causes the build (configure) failure of lang/perl5* (bug 227312)

What is the course of action to resolve the 2^15 limit permanently?

ctfdump -S /boot/kernel/kernel

- CTF Statistics -------------------------------------------------------------

  total number of data objects        = 29753

  total number of functions           = 32387
  total number of function arguments  = 78172
  maximum argument list length        = 22
  average argument list length        = 2.41

  total number of types               = 35498 <------
  total number of integers            = 76
  total number of floats              = 0
  total number of pointers            = 9481
  total number of arrays              = 3273
  total number of func types          = 3102
  total number of structs             = 11405
  total number of unions              = 633
  total number of enums               = 811
  total number of forward tags        = 49
  total number of typedefs            = 5893
  total number of volatile types      = 55
  total number of const types         = 588
  total number of restrict types      = 0
  total number of unknowns (holes)    = 132

  total number of struct members      = 87550
  maximum number of struct members    = 248
  total size of all structs           = 22415324
  maximum size of a struct            = 2971720
  average number of struct members    = 7.68
  average size of a struct            = 1965.39

  total number of union members       = 2091
  maximum number of union members     = 36
  total size of all unions            = 136952
  maximum size of a union             = 41576
  average number of union members     = 3.30
  average size of a union             = 216.35

  total number of enum members        = 8061
  maximum number of enum members      = 1023
  average number of enum members      = 9.94

  total number of unique strings      = 64074
  bytes of string data                = 926366
  maximum string length               = 81
  average string length               = 14.46
Comment 22 Conrad Meyer freebsd_committer 2019-05-28 05:41:50 UTC
(In reply to Kubilay Kocak from comment #21)
> Hitting this issue with: FreeBSD 13.0-CURRENT r348305 GENERIC-NODEBUG amd64
> 
> What is the course of action to resolve the 2^15 limit permanently?

The problem generally indicates an error in the CTF type system, although we are getting fairly close to hitting that threshold anyway.

To get past 2^15 types we'd have to rev the format in some way.
Comment 23 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-28 05:51:46 UTC
(In reply to Conrad Meyer from comment #22)

Thanks Conrad.

@Triage: Reset assignee: Since this is a broader issue, Mark oughtn't be assumed Assignee (previous issue resolver with workaround) unless he wants to explicitly take it

If the format revision is something that definitely couldn't be MRC'd, the mfc-* flags may be cancelled.
Comment 24 Mark Johnston freebsd_committer 2019-05-28 14:08:20 UTC
The problem was triggered by a certain src commit.  I reported the issue yesterday: https://lists.freebsd.org/pipermail/svn-src-all/2019-May/181386.html

It will probably be another day or two before it is resolved.  In the meantime all I can suggest is to temporarily disable DTrace support in perl (i.e., make it non-default), since I presume the issue is blocking pkg builds.
Comment 25 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-28 14:15:28 UTC
(In reply to Mark Johnston from comment #24)

Thanks for the update Mark.

Indeed lang/perl5.* ports default to DTRACE=on, so these will be effected for the head package builders post base r348300

Even after the short-term ctf count 'regression' is resolved though (please reference this PR), I'd like to this issue to remain open as a tracking issuefor permanent resolution, "format revision" or similar per comment 22, or whatever form that may take.
Comment 26 Mark Johnston freebsd_committer 2019-05-28 16:59:41 UTC
(In reply to Kubilay Kocak from comment #25)
Kirk fixed the problem in src r348329, so we can avoid updating the port, I think.

I'm happy to have a PR for the general issue, but I think it should be a new one.  Most instances of this failure are caused by something introducing duplicate types, which is a problem in its own right regardless of CTF's limit on the size of the type graph.
Comment 27 Kubilay Kocak freebsd_committer freebsd_triage 2019-05-29 03:05:52 UTC
(In reply to Mark Johnston from comment #26)

That may be one cause but the root issue is the 2^15 limit. It would be preferable (in this case) not to further complicate the bug hierarchy as this issue captures both the impact (ports failing to build), some of the causes (commits creating duplicate types), and the root cause already
Comment 28 Dimitry Andric freebsd_committer 2019-06-22 06:57:54 UTC
For some reason, the effect is also much more pronounced on i386 than on amd64, which I noticed because a bunch of dtrace-enabled ports (such as perl and node) failed only on i386.

E.g.:

revision  amd64  i386
========  =====  =====
r348302   35900  39487
r349011   29976  36473

Something seems to have happened between r348302 and r348980 to lower the number of types on amd64 from 35900 to 29976, and it stayed there ever since.

But the trick, whatever it was, did not appear to work for i386...
Comment 29 Dimitry Andric freebsd_committer 2019-07-01 17:46:03 UTC
(In reply to Dimitry Andric from comment #28)
> Something seems to have happened between r348302 and r348980 to lower the
> number of types on amd64 from 35900 to 29976, and it stayed there ever since.

Here's a more extended overview:

revision  types   committer  message
========  ======  =========  ===================================================
r346000   27,653
r347000   27,664
r347409   27,673
r347410   33,950  gallatin   Remove IPSEC from GENERIC due to performance issues
r347412   33,950
r347908   33,953
r347912   33,013  brooks     FCP-101: Remove ep(4)
r347913   35,427  brooks     FCP-101: Remove ex(4)
r347914   34,650  brooks     FCP-101: Remove fe(4)
r347915   30,190  brooks     FCP-101: Remove pcn(4)
r347916   29,953  brooks     FCP-101: Remove sf(4)
r347920   29,929  brooks     FCP-101: Remove txp(4)
r347921   27,405  brooks     FCP-101: Remove vx(4)
r347922   29,979  brooks     FCP-101: Remove wb(4)
r347923   36,415  brooks     FCP-101: Remove xe(4)
r347924   36,415
r349365   36,486

So for some reason, the removal of features (or moving them to modules) can greatly bloat the number of types.

For example, the removal of ex(4) in r347913 increases it by 2,414, most of the other removals decrease it, and lastly the removal of xe(4) increases it by 6,436!
Comment 30 Mark Johnston freebsd_committer 2019-07-09 04:20:44 UTC
As usual, the problem is caused by duplicates in the type graph.  A recent i386 kernel has 10 copies of struct thread in its CTF info.  This is usually the result of a bug somewhere that was indirectly triggered or exacerbated by the referenced kernel changes.