Running into the following failures with my amd64 VM host running the DTrace tests. I'll hopefully get more data for the failures sometime in the next couple days. $ uname -a FreeBSD fbsd11 12.0-CURRENT FreeBSD 12.0-CURRENT #5 r304091:304099: Sun Aug 14 13:08:45 PDT 2016 ngie@fbsd11:/usr/obj/usr/src/svn/sys/GENERIC amd64 $ sysctl hw.physmem hw.physmem: 3187220480 $ sysctl kern.smp.cpus kern.smp.cpus: 3 ===> Broken tests cddl/usr.sbin/dtrace/common/probes/t_dtrace_contrib:tst_probestar_d -> broken: Test case body timed out [300.013s] ... ===> Failed tests cddl/usr.sbin/dtrace/common/dtraceUtil/t_dtrace_contrib:tst_DataModel32_d_ksh -> failed: Test case was expecting a failure but none were raised [0.108s] cddl/usr.sbin/dtrace/common/misc/t_dtrace_contrib:tst_macroglob_ksh -> failed: atf-check failed; see the output of the test for details [0.528s] cddl/usr.sbin/dtrace/common/pid/t_dtrace_contrib:err_D_PDESC_ZERO_badlib_d -> failed: atf-check failed; see the output of the test for details [0.148s] cddl/usr.sbin/dtrace/common/pid/t_dtrace_contrib:err_D_PROC_FUNC_badfunc_d -> failed: atf-check failed; see the output of the test for details [0.138s] cddl/usr.sbin/dtrace/common/pid/t_dtrace_contrib:err_D_PROC_LIB_libdash_d -> failed: atf-check failed; see the output of the test for details [0.142s] cddl/usr.sbin/dtrace/common/pid/t_dtrace_contrib:err_D_PROC_NAME_alldash_d -> failed: atf-check failed; see the output of the test for details [0.168s] cddl/usr.sbin/dtrace/common/pid/t_dtrace_contrib:err_D_PROC_NAME_badname_d -> failed: atf-check failed; see the output of the test for details [0.143s] cddl/usr.sbin/dtrace/common/pid/t_dtrace_contrib:err_D_PROC_NAME_globdash_d -> failed: atf-check failed; see the output of the test for details [0.144s] cddl/usr.sbin/dtrace/common/pid/t_dtrace_contrib:err_D_PROC_OFF_toobig_d -> failed: atf-check failed; see the output of the test for details [0.146s] cddl/usr.sbin/dtrace/common/pid/t_dtrace_contrib:tst_addprobes_ksh -> failed: atf-check failed; see the output of the test for details [0.137s] cddl/usr.sbin/dtrace/common/pid/t_dtrace_contrib:tst_args1_d -> failed: atf-check failed; see the output of the test for details [0.192s] cddl/usr.sbin/dtrace/common/pid/t_dtrace_contrib:tst_coverage_d -> failed: atf-check failed; see the output of the test for details [0.144s] cddl/usr.sbin/dtrace/common/pid/t_dtrace_contrib:tst_emptystack_d -> failed: atf-check failed; see the output of the test for details [0.142s] cddl/usr.sbin/dtrace/common/pid/t_dtrace_contrib:tst_provregex3_ksh -> failed: atf-check failed; see the output of the test for details [0.187s] cddl/usr.sbin/dtrace/common/pragma/t_dtrace_contrib:err_invalidlibdep_ksh -> failed: atf-check failed; see the output of the test for details [0.168s] cddl/usr.sbin/dtrace/common/privs/t_dtrace_contrib:tst_kpriv_ksh -> failed: atf-check failed; see the output of the test for details [0.190s] cddl/usr.sbin/dtrace/common/types/t_dtrace_contrib:tst_complex_d -> failed: atf-check failed; see the output of the test for details [0.158s] cddl/usr.sbin/dtrace/common/uctf/t_dtrace_contrib:err_user64mode_ksh -> failed: atf-check failed; see the output of the test for details [1.194s] cddl/usr.sbin/dtrace/common/uctf/t_dtrace_contrib:tst_pidprint_ksh -> failed: atf-check failed; see the output of the test for details [0.147s] cddl/usr.sbin/dtrace/common/uctf/t_dtrace_contrib:tst_pidprinttarg_ksh -> failed: atf-check failed; see the output of the test for details [0.140s] cddl/usr.sbin/dtrace/common/uctf/t_dtrace_contrib:tst_userstrings_ksh -> failed: atf-check failed; see the output of the test for details [0.146s] cddl/usr.sbin/dtrace/common/usdt/t_dtrace_contrib:tst_eliminate_ksh -> failed: atf-check failed; see the output of the test for details [0.291s] cddl/usr.sbin/dtrace/common/usdt/t_dtrace_contrib:tst_enabled2_ksh -> failed: atf-check failed; see the output of the test for details [0.148s] cddl/usr.sbin/dtrace/common/usdt/t_dtrace_contrib:tst_fork_ksh -> failed: atf-check failed; see the output of the test for details [1.376s] cddl/usr.sbin/dtrace/common/usdt/t_dtrace_contrib:tst_include_ksh -> failed: atf-check failed; see the output of the test for details [0.053s] cddl/usr.sbin/dtrace/common/ustack/t_dtrace_contrib:tst_spin_ksh -> failed: atf-check failed; see the output of the test for details [0.114s] FWIW, it looks like tst_probestar_d fails intermittently (its completion time is close to the 300 second time limit): $ sudo kyua debug -k cddl/usr.sbin/dtrace/common/probes/Kyuafile t_dtrace_contrib:tst_probestar_d Executing command [ /usr/tests/cddl/usr.sbin/dtrace/common/probes/../../dtest /usr/tests/cddl/usr.sbin/dtrace/common/probes/tst.probestar.d ] t_dtrace_contrib:tst_probestar_d -> passed $ sudo kyua test -k cddl/usr.sbin/dtrace/common/probes/Kyuafile t_dtrace_contrib:tst_probestar_d t_dtrace_contrib:tst_probestar_d -> load: 0.11 cmd: kyua 17221 [wait] 221.39r 0.00u 0.00s 0% 6656k passed [267.179s] Results file id is usr_tests_cddl_usr.sbin_dtrace_common_probes.20160815-174507-149272 Results saved to /root/.kyua/store/results.usr_tests_cddl_usr.sbin_dtrace_common_probes.20160815-174507-149272.db 1/1 passed (0 failed)
We have a continuously running CI job for DTrace tests at https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/ And some of the failures in this tickets have been fixed. Do you think it's reasonable to close this ticket and file a ticket for each failing test case?
Indeed, many tests have been fixed over the past two years. I think it'd be useful to have individual PRs for failing DTrace tests.
https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/ is usually green now and there bug237641 to tracking one flaky test case.