# uname -a FreeBSD upgrade-132 14.0-RC1 FreeBSD 14.0-RC1 #0 releng/14.0-n265234-d2025992ab68: Fri Oct 13 05:51:48 UTC 2023 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 I ran: # kyua test -k /usr/tests/Kyuafile ===> Summary Results read from /root/.kyua/store/results.usr_tests.20231019-214609-353496.db Test cases: 8026 total, 1268 skipped, 34 expected failures, 104 broken, 8 failed Total time: 1679.220s # kyua report --results-filter=failed -r /root/.kyua/store/results.usr_tests.20231019-214609-353496.db ===> Failed tests lib/libc/nss/gethostby_test:gethostbyname2_getaddrinfo_ipv4 -> failed: /root/freebsd-src/lib/libc/tests/nss/gethostby_test.c:1141: run_tests(_hostlist_file, ((void *)0), 2, TEST_GETHOSTBYNAME2_GETADDRINFO, 0) == 0 not met [0.044s] lib/libc/nss/gethostby_test:gethostbyname2_getaddrinfo_ipv6 -> failed: /root/freebsd-src/lib/libc/tests/nss/gethostby_test.c:1148: run_tests(_hostlist_file, ((void *)0), 28, TEST_GETHOSTBYNAME2_GETADDRINFO, 0) == 0 not met [0.046s] lib/libc/nss/gethostby_test:getipnodebyname_getaddrinfo_ipv6 -> failed: /root/freebsd-src/lib/libc/tests/nss/gethostby_test.c:1450: run_tests(_hostlist_file, ((void *)0), 28, TEST_GETHOSTBYNAME2_GETADDRINFO, 0) == 0 not met [0.043s] sys/fs/tmpfs/times_test:empty -> failed: Incorrect atime [1.051s] sys/netinet/output:output_raw_flowid_mpath_success -> failed: Balancing failure: 1: 26 2: 9 [0.262s] usr.sbin/mixer/mixer_test:S_flag -> failed: atf-check failed; see the output of the test for details [0.011s] usr.sbin/mixer/mixer_test:s_flag -> failed: atf-check failed; see the output of the test for details [0.010s] usr.sbin/mixer/mixer_test:set_empty_value -> failed: atf-check failed; see the output of the test for details [0.010s] # kyua report --results-filter=broken -r /root/.kyua/store/results.usr_tests.20231019-214609-353496.db ===> Broken tests sys/cddl/zfs/include/libtest_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.000s] sys/cddl/zfs/tests/acl/cifs/cifs_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/acl/nontrivial/nontrivial_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/acl/trivial/trivial_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/atime/atime_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/bootfs/bootfs_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cache/cache_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cachefile/cachefile_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/clean_mirror/clean_mirror_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zdb/zdb_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs/zfs_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_clone/zfs_clone_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_copies/zfs_copies_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_create/zfs_create_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_destroy/zfs_destroy_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_diff/zfs_diff_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_get/zfs_get_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_inherit/zfs_inherit_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_mount/zfs_mount_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_promote/zfs_promote_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_property/zfs_property_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_receive/zfs_receive_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_rename/zfs_rename_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_reservation/zfs_reservation_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_rollback/zfs_rollback_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_send/zfs_send_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_set/zfs_set_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_share/zfs_share_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_snapshot/zfs_snapshot_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_unmount/zfs_unmount_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_unshare/zfs_unshare_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zfs_upgrade/zfs_upgrade_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool/zpool_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_add/zpool_add_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_attach/zpool_attach_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_clear/zpool_clear_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_create/zpool_create_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_destroy/zpool_destroy_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_detach/zpool_detach_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_expand/zpool_expand_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_export/zpool_export_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_get/zpool_get_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_history/zpool_history_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_import/zpool_import_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_offline/zpool_offline_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_online/zpool_online_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_remove/zpool_remove_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_replace/zpool_replace_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_scrub/zpool_scrub_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_set/zpool_set_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_status/zpool_status_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_root/zpool_upgrade/zpool_upgrade_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_user/misc/misc_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_user/zfs_list/zfs_list_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_user/zpool_iostat/zpool_iostat_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/cli_user/zpool_list/zpool_list_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/compression/compression_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/ctime/ctime_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/delegate/zfs_allow_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/delegate/zfs_unallow_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/devices/devices_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/exec/exec_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/grow_pool/grow_pool_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/grow_replicas/grow_replicas_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/history/history_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/hotplug/hotplug_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/hotspare/hotspare_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/inheritance/inheritance_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/inuse/inuse_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/large_files/large_files_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/migration/migration_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/mmap/mmap_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/mount/mount_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/mv_files/mv_files_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/nestedfs/nestedfs_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/no_space/no_space_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/online_offline/online_offline_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/pool_names/pool_names_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/poolversion/poolversion_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/quota/quota_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/redundancy/redundancy_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/refquota/refquota_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/refreserv/refreserv_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/replacement/replacement_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/reservation/reservation_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/rootpool/rootpool_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/scrub_mirror/scrub_mirror_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/slog/slog_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/snapshot/snapshot_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/snapused/snapused_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/sparse/sparse_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/threadsappend/threadsappend_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/truncate/truncate_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/txg_integrity/txg_integrity_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/userquota/userquota_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/utils_test/utils_test_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/write_dirs/write_dirs_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/zfsd/zfsd_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/zil/zil_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/zvol/zvol_ENOSPC/zvol_ENOSPC_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/zvol/zvol_cli/zvol_cli_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/zvol/zvol_misc/zvol_misc_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/zvol/zvol_swap/zvol_swap_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s] sys/cddl/zfs/tests/zvol_thrash/zvol_thrash_test:__test_cases_list__ -> broken: Test program did not exit cleanly [0.001s]
The first three did not fail on a re-run # kyua test -k /usr/tests/lib/libc/nss/Kyuafile The others are reproducible on re-run # kyua test -k /usr/tests/sys/fs/tmpfs/Kyuafile # kyua test -k /usr/tests/usr.sbin/mixer/Kyuafile === For the tmpfs tests kyua report --verbose -r ... shows: Result: failed: Incorrect atime Start time: 2023-10-20T13:52:13.292723Z End time: 2023-10-20T13:52:14.368358Z Duration: 1.076s Metadata: allowed_architectures is empty allowed_platforms is empty description = Tests that creating an empty file and later manipulating it updates times correctly has_cleanup = false is_exclusive = false required_configs is empty required_disk_space = 0 required_files is empty required_memory = 0 required_programs is empty required_user = root timeout = 300 Standard output: Executing command [ mkdir /tmp/kyua.1N7Z2W/93/work/mntpt ] mount -t tmpfs tmpfs /tmp/kyua.1N7Z2W/93/work/mntpt Executing command [ touch a ] Executing command [ cat a ] Standard error: Files left in work directory after failure: mntpt, mounterr === For the mixer tests kyua report --verbose -r ... shows: Standard output: Executing command [ /bin/sh -c mixer vol | awk '{print $7}' ] vol.volume=1.00:1.00 vol.mute=0 Standard error: Fail: regexp ^[0-9]{1,3}:[0-9]{1,3}$ not in stdout Files left in work directory after failure: saved_vol mixer: : no such device
The tmpfs one is understood, it's due to the use of copy_file_range() in cat(1). I have a patch to fix it, just haven't posted it for review yet.
I think we should add specific bugs for the ones that are actionable and plan to fix, but probably close this specific bug after doing that. Maybe mark the other bugs as See Also for this. We can also then decide which of those specific bugs we might want to add as a blocker of the 14.0r bug.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f7c4f49649717000238184a975b2339bcd8c06d9 commit f7c4f49649717000238184a975b2339bcd8c06d9 Author: John Baldwin <jhb@FreeBSD.org> AuthorDate: 2023-10-20 18:33:11 +0000 Commit: John Baldwin <jhb@FreeBSD.org> CommitDate: 2023-10-20 18:36:04 +0000 ObsoleteFiles.inc: Remove old mixer(8) tests These were removed when the new mixer(3) library was imported. PR: 274602 Fixes: 903873ce1560 Implement and use new mixer(3) library for FreeBSD. ObsoleteFiles.inc | 5 +++++ 1 file changed, 5 insertions(+)
(In reply to John Baldwin from comment #3) The lib/libc/nss failures appear to be an intermittent issue, and the mixer tests were indeed left over from a 13.2 installation. For tmpfs we should indeed have a separate bug. I would like to have 0 failures in the release branch, even if they are just XFAIL against an appropriate PR.
I fully appreciate the goal of a clean test suite, I'm just not sure that multiple layers of nested meta-bugs in bugzilla is the best way to get there. One option in the future might be to include this information as comments in the original 14.0r bug rather than making a new bug?
With the mixer tests removed (i.e., jhb's change) the only remaining issue I observe is the tmpfs issue, tracked in 274615
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=e8daee32125301676fb246671532abffbdca3e77 commit e8daee32125301676fb246671532abffbdca3e77 Author: John Baldwin <jhb@FreeBSD.org> AuthorDate: 2023-10-20 18:33:11 +0000 Commit: John Baldwin <jhb@FreeBSD.org> CommitDate: 2023-10-24 19:03:19 +0000 ObsoleteFiles.inc: Remove old mixer(8) tests These were removed when the new mixer(3) library was imported. PR: 274602 Fixes: 903873ce1560 Implement and use new mixer(3) library for FreeBSD. (cherry picked from commit f7c4f49649717000238184a975b2339bcd8c06d9) ObsoleteFiles.inc | 5 +++++ 1 file changed, 5 insertions(+)
A commit in branch releng/14.0 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=dc5549231888f429af0b052dc6b255f044fa9c69 commit dc5549231888f429af0b052dc6b255f044fa9c69 Author: John Baldwin <jhb@FreeBSD.org> AuthorDate: 2023-10-20 18:33:11 +0000 Commit: John Baldwin <jhb@FreeBSD.org> CommitDate: 2023-10-24 19:29:56 +0000 ObsoleteFiles.inc: Remove old mixer(8) tests These were removed when the new mixer(3) library was imported. PR: 274602 Fixes: 903873ce1560 Implement and use new mixer(3) library for FreeBSD. (cherry picked from commit f7c4f49649717000238184a975b2339bcd8c06d9) (cherry picked from commit e8daee32125301676fb246671532abffbdca3e77) Approved by: re (gjb) ObsoleteFiles.inc | 5 +++++ 1 file changed, 5 insertions(+)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=2acd158d4b0ec832d54f69c7b245a688c612685f commit 2acd158d4b0ec832d54f69c7b245a688c612685f Author: Dimitry Andric <dim@FreeBSD.org> AuthorDate: 2024-01-23 18:40:01 +0000 Commit: Dimitry Andric <dim@FreeBSD.org> CommitDate: 2024-01-23 18:40:12 +0000 mtree: stop creating old mixer(8) test directories The tests themselves were added to ObsoleteFiles.inc in commit f7c4f4964971, but not yet removed from BSD.tests.dist. PR: 274602 Fixes: f7c4f4964971 etc/mtree/BSD.tests.dist | 2 -- 1 file changed, 2 deletions(-)