For a while now (a week? two? not sure exactly) my nightly ports build has been failing for lang/dotnet (specifically; lang/dotnet-host builds just fine). I don't know enough of the details of the build process to say for sure, but it *seems* like it might be more of an incidental failure rather than a "real" failure: The build log makes a lot of noises that sure make it seem like it has essentially completed successfully, but then dies while trying to clean up after itself: ============================= ===> Building for dotnet-9.0.6 cd /construction/xports/lang/dotnet/work/dotnet-9.0.6 && /usr/bin/env OPENSSLBASE=/usr OPENSSLDIR=/etc/ssl OPENSSLINC=/usr/include OPENSSLLIB=/usr/lib XDG_DATA_HOME=/construction/xports/lang/dotnet/work XDG_CONFIG_HOME=/construction/xports/lang/dotnet/work XDG_CACHE_HOME=/construction/xports/lang/dotnet/work/.cache HOME=/construction/xports/lang/dotnet/work TMPDIR="/tmp" PATH=/usr/local/libexec/ccache:/construction/xports/lang/dotnet/work/.bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin PKG_CONFIG_LIBDIR=/construction/xports/lang/dotnet/work/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES DEFAULT_AUTOCONF=2.72 PREFIX=/usr/local LOCALBASE=/usr/local CC="cc" CFLAGS="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing " CPP="cpp" CPPFLAGS="" LDFLAGS=" -Wl,-rpath,/usr/local/lib:/usr/lib " LIBS="" CXX="c++" CXXFLAGS="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing " CCACHE_DIR="/ccache" BSD_INSTALL_PROGRAM="install -s -m 555" BSD_INSTALL_LIB="install -s -m 0644" BSD_INSTALL_SCRIPT="install -m 555" BSD_INSTALL_DATA="install -m 0644" BSD_INSTALL_MAN="install -m 444" PYTHON=/usr/local/bin/python3.11 ./build.sh --release-manifest release.json --clean-while-building --with-sdk /construction/xports/lang/dotnet/work/bootstrap_sdk --with-packages /construction/xports/lang/dotnet/work/packages --source-build -- -maxcpucount:4 Using custom bootstrap SDK from '/construction/xports/lang/dotnet/work/bootstrap_sdk', version '9.0.103' Found bootstrap versions: SDK 9.0.103, Arcade 9.0.0-beta.25065.2, NoTargets 3.7.0 and Traversal 3.4.0 Welcome to .NET 9.0! --------------------- SDK Version: 9.0.103 ---------------- Installed an ASP.NET Core HTTPS development certificate. To trust the certificate, run 'dotnet dev-certs https --trust' Learn about HTTPS: https://aka.ms/dotnet-https ---------------- Write your first app: https://aka.ms/dotnet-hello-world Find out what's new: https://aka.ms/dotnet-whats-new Explore documentation: https://aka.ms/dotnet-docs Report issues and find source on GitHub: https://github.com/dotnet/core Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli -------------------------------------------------------------------------------------- Shutting down MSBuild server... Shutting down VB/C# compiler server... MSBuild server failed to shut down: sysctl 1 failed with 3 error. VB/C# compiler server shut down successfully. *** Error code 1 Stop. make: stopped in /xports/lang/dotnet ============================= Now, I should mention that this or at least something very, very like this has occasionally happened to me for a LONG time now; I think pretty much ever since the port was first published. But this is different: In the past, the next time the build ran (no matter whether via my automated nightly builds or manually), it would work just fine (at least the vast majority of the time). I always just attributed it (without any additional evidence) to the idea that the amount of time it waits for the server to shut down is not *quite* long enough for it to be 100% reliable. But now, it always fails. Or, at the very least, I haven't ever seen it succeed, in a lot of attempts.
If you're able to constantly reproduce this, can you please try this version of the port? https://github.com/arrowd/freebsd-ports/tree/dotnet The error should become more descriptive.
(In reply to Gleb Popov from comment #1) Thanks. I am able to constantly reproduce this, and while I can't promise that I can try that version of the port (see next paragraph), I can promise that I can *try* to try that version of the port. The main problem with trying that version of the port is that I don't know how to do so (nor do I know how to "recover" back to the standard ports-building process after having done so). If someone could please try to explain that to me, it would probably help me to be able to actually try that version of the port. I know very little about the internals of the ports-building process, and also I've been consciously avoiding knowing anything about git beyond extremely basic usage for literally decades now; I only use it when I have no other reasonable choice (I'm a subversion dinosaur). So if anyone decides to try to help me out with the explanation, please keep those facts in mind. For what it's worth, here's the basics of how I currently build ports: I have a nightly cron job that runs a dotnet program that I wrote pretty much as soon as dotnet was officially supported in the ports tree. It has served me well since that time, and the steps that it takes have served me well for quite a while longer (i.e. since well before I put it all into a dotnet program). Those steps are (stuff inside "{}" is not literal): (1) gitup ports (2) make -C /usr/ports index (3) env TERM=dumb synth status {list-of-ports-I-want} (4) env TERM=dumb synth just-build {list-of-ports-I-want} (5) env TERM=dumb synth rebuild-repository The program also gathers and saves some info about the results of all that, but it's obviously not relevant here. I later manually review it and "pkg upgrade" if appropriate. I can provide additional details (e.g. related configuration) upon request, but please be specific about what you're looking for. Thanks again.
(In reply to Robert William Vesterman from comment #2) Not a problem at all, here's the same patch that is ready to be dropped under files/ directory: fetch -o lang/dotnet/files/patch-sysctl https://arrowd.name/patch-sysctl Then just rebuild the port as usual.
(In reply to Gleb Popov from comment #3) Thanks. I just tried it out, but it doesn't seem to have made any difference. Still just "MSBuild server failed to shut down: sysctl 1 failed with 3 error." Just to be sure, when you wrote... fetch -o lang/dotnet/files/patch-sysctl https://arrowd.name/patch-sysctl ... that was intended to save the downloaded file as /usr/ports/lang/dotnet/files/patch-sysctl, right? That is where I put it, so if it should be somewhere else, please let me know. Random ignorant thoughts: * Might it need some other file (or setting or whatever) that indicates "patch-sysctl really is to be used as a patch for this"? * There are other things already in that "files" directory. Might the order in which patches are applied matter, and if so, might there be something missing that defines what the order should be? Thanks again.
Trying a couple more things to investigate: (1) I made a change to the patch file which, I believe, would cause the compile to fail. It seems to have had no effect. (2) Thinking maybe Synth was not seeing the patch file, I entered into its build and looked for it. It was indeed there, and moreover it did include the change I made in (1): # env ENTERAFTER=extract synth test lang/dotnet Scanning existing packages. After inspection, it has been determined that there are no packages that require rebuilding; the task is therefore complete. Starting interactive build of lang/dotnet Stand by, building up to the point requested ... Entering interactive test mode at the builder root directory. Type 'exit' when done exploring. # pushd /xports/lang/dotnet/files/ /xports/lang/dotnet/files / # ls extra-patch-base-openssl patch-src_roslyn-analyzers_src_PerformanceTests_Tests_PerformanceTests.csproj patch-src_runtime_src_native_libs_System.IO.Compression.Native_CMakeLists.txt patch-sysctl # grep asdf patch-sysctl + private static unsafe partiaasdfl int Sysctl(int* name, uint namelen, void* value, nuint* len); #
Can you provide Synth's build log?
(In reply to Gleb Popov from comment #6) Sure. I'll actually attach three different ones: (1) lang___dotnet.1-base.log (2) lang___dotnet.2-patched.log (3) lang___dotnet.3-badpatch.log They were generated in that order, and this is probably obvious, but they are respectively without the patch, with the patch, and with the intentionally screwed-up version of the patch that I imagine would cause a compilation error. There is no difference between #2 and #3 except for time stuff: # diff lang___dotnet.2-patched.log lang___dotnet.3-badpatch.log 2c2 < Started : Monday, 1 SEP 2025 at 17:04:55 UTC --- > Started : Monday, 1 SEP 2025 at 17:20:23 UTC 607,608c607,608 < Finished: Monday, 1 SEP 2025 at 17:06:41 UTC < Duration: 00:01:45 --- > Finished: Monday, 1 SEP 2025 at 17:22:09 UTC > Duration: 00:01:46 # Meanwhile there is a difference (besides that time sort of thing) between #1 and #2, which I hadn't noticed when I did this sort of thing yesterday (but I don't know for sure that it happened yesterday too, and I wouldn't have been looking for it in any case): # diff lang___dotnet.1-base.log lang___dotnet.2-patched.log 2c2 < Started : Monday, 1 SEP 2025 at 16:50:35 UTC --- > Started : Monday, 1 SEP 2025 at 17:04:55 UTC 595,603d594 < VB/C# compiler server shut down successfully. < MSBuild server shut down successfully. < MSBuild version 17.12.24+c4e5fd73f for .NET < < Determining projects to restore... < Restored /construction/xports/lang/dotnet/work/dotnet-9.0.6/eng/tools/tasks/Microsoft.DotNet.UnifiedBuild.MSBuildSdkResolver/Microsoft.DotNet.UnifiedBuild.MSBuildSdkResolver.csproj (in 893 ms). < Microsoft.DotNet.UnifiedBuild.MSBuildSdkResolver -> /construction/xports/lang/dotnet/work/dotnet-9.0.6/artifacts/bin/Microsoft.DotNet.UnifiedBuild.MSBuildSdkResolver/Release/Microsoft.DotNet.UnifiedBuild.MSBuildSdkResolver.dll < Shutting down MSBuild server... < Shutting down VB/C# compiler server... 616,617c607,608 < Finished: Monday, 1 SEP 2025 at 16:52:31 UTC < Duration: 00:01:56 --- > Finished: Monday, 1 SEP 2025 at 17:06:41 UTC > Duration: 00:01:45 # So the unpatched version has some extra part about "restoring" some stuff, and moreover indicates "MSBuild server shut down successfully." The latter surprised me when I noticed it. However, looking closer at it, it seems like it's some other use of MSBuild server, besides the one I've been having the problem with, and the problem still occurred at the end of the build. Thanks again.
Created attachment 263393 [details] synth log with no patch
Created attachment 263394 [details] synth log with patch
Created attachment 263395 [details] synth log with intentionally bad patch
Ah, silly me, it is the bootstrap dotnet that is failing, so all these patches change nothing. Hum. Can you please try reproducing the issue while building with plain make, without synth? To prevent accidental pollution of your system with build dependencies, always run "make USE_PACKAGE_DEPENDS_ONLY=yes" instead of just "make" and if you get an error about some dependency, just install it the usual way you install packages. Once you manage to reproduce the issue, prepend "truss -fo /tmp/truss.log" to your make invocation and run it again to create a log that would contain the info about the failure I need.
Created attachment 263496 [details] zipped truss log Sure, here's the truss log. Just to note: The build (outside of Synth) did in fact fail with the same error. Thanks again.
Great, this is useful and gives me a lead! I'll dive into debugging hopefully this weekend.
Thanks. I just noticed that in my automated nightly build for last night, it finally built successfully. I don't know of anything that I did between the night before and then that might have made a difference, with the possible exception of the ports tree update that is part of the automated nightly build -- but it looks to me like the port hasn't had any changes recently. On the other hand, maybe an update was made to something else that is used during the build of it. Given that (like I said in the original post) this problem would occur every once in a while for me since time immemorial, but until recently would happen rarely enough that I could always just try again and it would almost always work, I'm not sure if it is "really" fixed for me or if it just happened to be OK this one time. Just in case it might be helpful in some way, I'm attaching the successful build log. I notice that it includes a whole lot of stuff following the point that the error had been occurring at, so my guess in the original post that "it seem(s) like it has essentially completed successfully, but then dies while trying to clean up after itself" was incorrect.
Created attachment 263532 [details] synth log for a magically successful build
Welp, I think we can rule out the "really is fixed" possibility. An update was made to lang/dotnet yesterday, which made my automated ports update script try to rebuild it overnight. Same sort of error wound up happening. I'll attach the log.
Created attachment 263579 [details] synth log for 9.0.6_1; error is back
From what I gather, this error is caused by a race condition, so if you're lucky the port builds successfully, otherwise it fails. I started working towards fixing this, but this will probably take quite a time. Meanwhile you can just retry the build until it succeeds.
Hi, I'm sorry to be a pain, and really it's more an annoyance than an actual big deal, but it's still happening. I'm curious if there ha been any progress?
(In reply to Robert William Vesterman from comment #19) Unfortunately not, .NET stuff is PITA to work with, so I got derailed with other tasks. Hopefully I'll get back to this during summer vacation.
(In reply to Gleb Popov from comment #20) OK, no problem - again, I'm sorry to be a pain, and it's not really a big issue - I admit I'm definitely hoping for it, but I am not in a rush for it. Thanks!
(In reply to Robert William Vesterman from comment #21) This is quite a problem in reality, because it often causes the build to crash on FreeBSD official builders. What makes it hard to fix is its stochastic nature and a convoluted build system when building master branch.
There is a hope that this problem should be fixed by https://github.com/dotnet/runtime/pull/131534 I will include this change when updating the port to .NET 10
Great - thanks!