Bug 289696 - Failing Cirrus-CI jobs in main
Summary: Failing Cirrus-CI jobs in main
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: tests (show other bugs)
Version: 16.0-CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-testing (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2025-09-19 14:09 UTC by Ed Maste
Modified: 2025-12-23 15:49 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ed Maste freebsd_committer freebsd_triage 2025-09-19 14:09:33 UTC
See e.g. https://cirrus-ci.com/task/5050753301086208

Two jobs:
- amd64 smoke test using internal ci systems
- aarch64 smoke test using internal ci systems
are taking a combined 3 hours before failing. This is consuming all of the monthly Cirrus-CI credits that we have in very short order at the beginning of the month.

log snippet:

Building /usr/obj/tmp/cirrus-ci-build/amd64.amd64/usr.bin/kyua/kyuafile.5.gz
Building /usr/obj/tmp/cirrus-ci-build/amd64.amd64/usr.bin/kyua/kyua-test.1.gz
Building /usr/obj/tmp/cirrus-ci-build/amd64.amd64/usr.bin/kyua/kyua.full
Building /usr/obj/tmp/cirrus-ci-build/amd64.amd64/usr.bin/kyua/kyua.debug
Building /usr/obj/tmp/cirrus-ci-build/amd64.amd64/usr.bin/kyua/kyua
      325.52 real      2441.24 user       112.61 sys
--------------------------------------------------------------
>>> World build completed on Thu Sep 18 01:50:54 UTC 2025
>>> World built in 4516 seconds, ncpu: 8, make -j8
--------------------------------------------------------------
Building kernel for amd64
make TARGET=amd64 TARGET_ARCH=amd64 -j8 -C /tmp/cirrus-ci-build/tests/ci/../.. -DWITH_META_MODE   __MAKE_CONF=/dev/null SRCCONF=/etc/src.conf   KERNCONF=GENERIC  buildkernel 
make[1]: getcwd: No such file or directory.
*** Error code 2
Stop.
make: stopped in /tmp/cirrus-ci-build/tests/ci
Comment 1 Mark Johnston freebsd_committer freebsd_triage 2025-09-19 15:36:53 UTC
I see the same thing in my repo: https://github.com/markjdb/freebsd/runs/49407448292

That was on commit b197d2abcb6895d78bc9df8404e374397aa44748.  You had a successful build on commit da64f6e047b5f6cd9b9e8450d19072cb0b6e44d0, so we need to figure out what broke in between.
Comment 2 commit-hook freebsd_committer freebsd_triage 2025-09-21 13:35:33 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=2b8bfdcad1f46076eaf40938634538fbcb13cb1c

commit 2b8bfdcad1f46076eaf40938634538fbcb13cb1c
Author:     Ed Maste <emaste@FreeBSD.org>
AuthorDate: 2025-09-19 14:09:48 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2025-09-21 13:34:30 +0000

    Revert ".cirrus.yml: Reenable CI with pre-commit CI scripts"

    This reverts commit 676d64ee8327851063d92d0dd6a4ceee6b3a25e6.

    These tests take a combined 3h to run and are consuming our montly
    Cirrus-CI credits in short order, before failing.

    In the PR markj reports that the failure appeared somewhere between:

    good: da64f6e047b5f6cd9b9e8450d19072cb0b6e44d0
    bad:  b197d2abcb6895d78bc9df8404e374397aa44748

    Unfortunately, even after we find the offending commit and have a fix we
    still do not have sufficient monthly compute credit.  Disable these
    tests for now, pending a solution to both issues.

    PR:             289696
    Reviewed by:    ngie
    Differential Revision: https://reviews.freebsd.org/D52618

 .cirrus.yml | 2 ++
 1 file changed, 2 insertions(+)
Comment 3 Ed Maste freebsd_committer freebsd_triage 2025-09-21 13:55:48 UTC
Test is disabled for now (set to manual). We still need to understand and fix the failure, and then sort out the CPU time credits issue.
Comment 4 commit-hook freebsd_committer freebsd_triage 2025-10-14 15:01:54 UTC
A commit in branch stable/15 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=eb895ad4c47e0f3ee77dff638159e21dee1c501a

commit eb895ad4c47e0f3ee77dff638159e21dee1c501a
Author:     Ed Maste <emaste@FreeBSD.org>
AuthorDate: 2025-09-19 14:09:48 +0000
Commit:     Ed Maste <emaste@FreeBSD.org>
CommitDate: 2025-10-14 15:00:58 +0000

    Revert ".cirrus.yml: Reenable CI with pre-commit CI scripts"

    This reverts commit 676d64ee8327851063d92d0dd6a4ceee6b3a25e6.

    These tests take a combined 3h to run and are consuming our montly
    Cirrus-CI credits in short order, before failing.

    In the PR markj reports that the failure appeared somewhere between:

    good: da64f6e047b5f6cd9b9e8450d19072cb0b6e44d0
    bad:  b197d2abcb6895d78bc9df8404e374397aa44748

    Unfortunately, even after we find the offending commit and have a fix we
    still do not have sufficient monthly compute credit.  Disable these
    tests for now, pending a solution to both issues.

    PR:             289696
    Reviewed by:    ngie
    Differential Revision: https://reviews.freebsd.org/D52618

    (cherry picked from commit 2b8bfdcad1f46076eaf40938634538fbcb13cb1c)

 .cirrus.yml | 2 ++
 1 file changed, 2 insertions(+)
Comment 5 Muhammad Moinur Rahman freebsd_committer freebsd_triage 2025-12-18 23:30:58 UTC
As I was working on testing the new host from NetActuate as a persistent worker I have found the same error. So I did some digging and found this offending commit for this incident.
https://cgit.freebsd.org/src/commit/?id=ac7ed266dc73a1d55f2bc1e90bfce3c93757e8b4

I started with this test where it fails:
https://cirrus-ci.com/task/4852209881448448

And in the following task it actually passes and starts the make buildkernel although fails for other reasons:
https://cirrus-ci.com/build/5935516530507776

If someone can give me some ideas I can try to adopt and change the code in 
tests/ci/Makefile. The offending line in my code is this:

https://cgit.freebsd.org/src/tree/tests/ci/Makefile#n181

${IMAKE} -j${PARALLEL_JOBS} -C ${WORLDDIR} ${METAMODE} \
	${CROSS_TOOLCHAIN_PARAM} __MAKE_CONF=${MAKECONF} SRCCONF=${SRCCONF} \
	${EXTRA_MAKE_FLAGS} KERNCONF=${KERNCONF} \
	buildkernel ${LOG_TARGET}

And WORLDDIR is defined as:
WORLDDIR?=	${.CURDIR}/../..

I think we need to change value of WORLDDIR to something else. But not sure to what extent.
Comment 6 Ed Maste freebsd_committer freebsd_triage 2025-12-23 15:49:36 UTC
In general we want to use ${.CURDIR:H:H} not ${.CURDIR}../..

I opened https://reviews.freebsd.org/D54352 to set WORLDDIR using the standard approach, but I don't know if it will have an effect on this issue.