CI is too spammy to be enabled at this time. There are at least two problems: 1. Every failing build subsequent to a failing build re-spams every committer from the period during which the failure was introduced. There is no reason for this. Even mailing the actual committers from the build is probably low value considering the build was known to be broken previously. 2. We don't run a build for every revision, so broken build emails are not targetted to the actual build-breaker. This spams anybody who happens to commit nearby in time.
(In reply to Conrad Meyer from comment #0) Thanks for your suggestions, and sorry that this experimental feature makes you unsatisfied. We are working on improving it. For your suggestions, 1) There are two opinions about this, mailing the one breaks the build or all the following committers. Here I just keep what we did last time, and this is the default behavior of most the CI systems and services I have seen. One of the reasons is that committing to a broken head means the patch cannot be confirmed without any problem. We cannot guarantee it does not introduce new problems. This notification still has its point. The committers need to be aware their patches have not been tested (by CI system) yet. Basically, if you are not the one got "failure" mail and have confidence that your commit to a broken head is absolute fine, you can ignore most "still failing" notifications. The difference in the subject may help you triage and, perhaps, set the mail filter. 2) Yes, this is because we have too many committs and a full build takes a very long time. If we do a clean build for every revision, it will exhaust all the computing power we have currently. There is a plan to do an incremental build on every revision and I am working on it. However, in the end, we still need a clean build to ensure the whole code base can be built and run without any problem. That may still need to batch several commits in a build. And, if it fails, it may still have the similar problem unless we have a manual analysis (that's what I did in the past two years when turning the system and make sure there are no too many false positives), but this is not a long-term solution. I totally understand your points, and please give us some more time to work on improvements. After the previous version, we have adjusted the mail content, disabled sending to the lists like -current@ and have a pre-send filter script to prevent most false positives. There are some other comments like the necessary of the "fixed" mail. I will collect more inputs and ask core or write a CFP to get it finalized.
(In reply to Li-Wen Hsu from comment #1) For 1), I changed the setting to just send notification to the committers made change in a failing build. So one commit at most generates two notifications, one for notifying the commit cannot be confirmed tested and the other is notifying the build has back to normal, which means your commit has been verified. Hope this helps. Thanks.
(In reply to Li-Wen Hsu from comment #1) (1) Understood. However, in practice this is too spammy and does not provide much value. There is a reason CI emails were turned off last time, and this is part of it. In general we *expect* committers to commit compileable code, so in the absense of information ("cannot be confirmed"), our prior belief should be that the commit was correct and we should not spam. We already have a very effective and lower noise network of post-commit does-it-build CI via the freebsd-current@ mailing list. If I continue to get dozens of emails about a breakage I had no part in, I'm just going to filter the sender. (2) For CI and email notification, perhaps a -DNO_CLEAN build would be adequate? Clean builds can be done occasionally. I think -DNO_CLEAN would provider a much lower false positive report rate. It's important to have a low false positive rate or developers quickly learn to ignore or filter the reports. I would highly suggest reviewing https://ai.google/research/pubs/pub46576 , which is a good template for building these kinds of tools. (In reply to Li-Wen Hsu from comment #2) > For 1), I changed the setting to just send notification to the committers made change in a failing build. Yes, it is a big improvement. Thank you!
I think we should be mostly testing -DNO_CLEAN builds rather than clean builds (We probably should have a separate job for clean builds). If we were to test every commit, we'll want to have a tag in commit messages that causes CI builds to not be run as we have some classes of commits that are necessarily broken due to (e.g.) our rules about committing generated code.
(In reply to Brooks Davis from comment #4) Incremental build is also one of the things we want to have. In fact, all the build have -DNO_CLEAN on currently because we want to save the time doing cleanup before actual build. The main issue is how to keep the obj files between builds. Leaving obj files in the builders' workspace is not a scalable way because various build jobs will eventually use all the disk spaces of the builders. Each build is performing in a clean created jail and everything will be thrown away after build completes, except the artifacts we are interested in will be uploaded to artifact server. What I'm trying to do is also archiving the obj files on the artifact server and restoring them while setting up the build environment for the next build.
^Triage: this is more of a feature request than an "In Progress".