Hi there, `pkg search -e lang/go` show no results, and it's pretty new: I got some results last week. Neither `pkg search -g 'go1*'` I issued a `pkg update -f` before. Here is my `/etc/pkg/FreeBSD.conf`: ``` FreeBSD: { url: "pkg+https://pkg.FreeBSD.org/${ABI}/quarterly", mirror_type: "srv", signature_type: "fingerprints", fingerprints: "/usr/share/keys/pkg", enabled: yes } ``` Did lang/go vanished from repo? Did I missed something?
It looks as though several golang versions are failing to build, e.g.: https://portsfallout.com/fallout?port=go123&maintainer=&env=&category=&flavor= Numerous ports that depend on golang are also not present in the repos right now. Not sure what's going on, cc'ing adamw who I think might have more info?
I think the importance should be increased, this seems to affect everyone and ports that depend on go are missing
No, you're not missing something, go is unable to build on the package servers right now. We don't know why it's failing yet, but the result is that no packages for go or anything written in go are being produced. I'm unable to reproduce it, so troubleshooting it is a beast. Plus, almost all of the go team members have drifted away. It is absolutely a critical issue, but there's no light at the end of that tunnel yet...
Possibly related to https://github.com/golang/go/issues/69255#issuecomment-2543430338 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=283314 It seems to be a problem on certain VMs. Are the package builders also in a VM? Apparently GODEBUG=asyncpreemptoff=1 inside the VMs did help. Can this be tested on the package builders?
FYI, I just built the following at home in a FreeBSD 14.2 jail using poudriere. go: 1.21_7,2 go123: 1.23.7_4 go124: 1.24.2
Hi there, Thank you all for your answers, and investigations. If it can be of any help: - Every versions of Go compiles on my amd64 arch #ofcourse #itworksathome - I found this page : https://gitlab.com/qemu-project/qemu/-/issues/2560 Does FreeBSD compilations are made on x86_64 emulation running on aarch64 host ?
Asking the obvious questions: - does the package server use QEMU? - is it feasible to try disabling the garbage collector for builds? - what architecture is the package server? - who runs the package servers and are they aware? - can packages be built and inserted into the package repository or must they come from the build server?
(In reply to Ryan Shaw from comment #7) > - does the package server use QEMU? Yes, it relies heavily on QEMU for nearly every jail. > - is it feasible to try disabling the garbage collector for builds? Which garbage collector are you referring to? Poudriere is designed to work hand-in-hand with QEMU. If you were inclined you could test the above (plus it'd be nice to know if this is repeatable outside of the pkg builders). > - what architecture is the package server? AFAIK there are multiple physical pkg builders. I don't know what architecture(s) they are. > - who runs the package servers and are they aware? Of the problem? Absolutely.We are in the run-up to 14.3, so this is a problem on multiple teams' radars. I don't know what investigation or work they've done on it. I do my best with Go-related ports and its framework, but QEMU-related failures are definitely above my head. We have a pkgmgr team that oversees all things packages. > - can packages be built and inserted into the package repository or must they come from the build server? I doubt it. Integrity is critical in the pkg builders, which sign the packages.
(In reply to Adam Weinberger from comment #8) Appreciate the detailed response > Which garbage collector are you referring to? The Go garbage collector, to quote the gitlab issue previously linked: > Apps compiled for Go and the Go compiler/tool itself crash when they are run with qemu-x86_64 on an AARCH64 host system. > The crash from Go seems to be in the garbage collector, I cannot reproduce the issue when I disable the GC with GOGC=off.
(In reply to Stefan Schlosser from comment #4) > Are the package builders also in a VM? Yes, they're built in QEMU. It's only the amd64 packages that are failing, so I suspect they're built in QEMU. > Apparently GODEBUG=asyncpreemptoff=1 inside the VMs did help. Can this be tested on the package builders? While it wouldn't be difficult to test whether setting GODEBUG=asyncpreemptoff=1 fixes the problem, I'm not sure that this is actually related to bug #283314.
Created attachment 259453 [details] GOGC=off test (In reply to Ryan Shaw from comment #9) In reply to Adam Weinberger from comment #8) > > Which garbage collector are you referring to? > The Go garbage collector, to quote the gitlab issue previously linked: > Apps compiled for Go and the Go compiler/tool itself crash when they are run with qemu-x86_64 on an AARCH64 host system. > The crash from Go seems to be in the garbage collector, I cannot reproduce the issue when I disable the GC with GOGC=off. Ah, yes, thanks. Antoine, could you try building something small, like devel/shfmt, with the attached patch? No idea if it'll fix anything, but it's worth a try.
*** Bug 286019 has been marked as a duplicate of this bug. ***
*** Bug 285983 has been marked as a duplicate of this bug. ***
(In reply to Adam Weinberger from comment #8) In the meantime, it might be helpful if we can get more details on the builders setup (host hardware and OS, QEMU version, anything else that might be relevant) to try to find a repro that can be worked on. FWIW I tried under QEMU on aarch64, since that was reported to repro this bug (or something similar anyway), but wasn't successful. Happy to run a few more experiments if we can figure out which direction to go in.
I think we no longer use qemu in the cluster (was too broken). The builders run recent src main (from last week). The jails use releases updated with freebsd-update.
Hi! Could you restore a previous version? It’s strange that previous package versions aren’t saved.
(In reply to Eugene Kilimchuk from comment #16) For that, users should refer to /var/cache/pkg
(In reply to Dan Langille from comment #17) How does it help me if I and other users don’t have a local cache?
(In reply to Eugene Kilimchuk from comment #18) I'm sorry, I can only help based on information I know.
(In reply to Antoine Brodin from comment #15) Interesting, I didn't know that! Was I at least remembering correctly from when I was on portmgr that the builders relied on QEMU? Antoine, what's your sense of this problem? Are you suspecting it's a Go problem, or a problem in base? I just don't know where to even start with this.
(In reply to Adam Weinberger from comment #20) My feeling is that some compatibility between FreeBSD 14.2 amd64 and main src kernel is broken (i386, 13.4 amd64 and main amd64 userlands build go fine)
(In reply to Antoine Brodin from comment #21) At what point is it worth standing up a 14.2 host (or maybe a QEMU jail?)? I know that's a big deal, but there are quite a lot of packages that can't build, and I'm not seeing any obvious way to work around it at the port level.
https://lists.freebsd.org/archives/freebsd-ports/2025-April/007564.html Assuming this is the problem > I don't know what the bug is. But it's easy enough to reproduce: run > poudriere in a release/X jail on -CURRENT and watch it explode. I can't reproduce this... I set up a April 10 snapshot of 15 CURRENT in QEMU VM on an amd64 linux host, with a poudriere jail of 14.2-RELEASE-p3 amd64 So that the build logs state Host OSVERSION: 1500036 Jail OSVERSION: 1402000 And built lang/go* but they all succeeded. I then built a few thousand random packages, also not one failure. I have only just subscribed to the mailing list so not sure how to reply to this topic properly
(In reply to Ryan Shaw from comment #23) > Assuming this is the problem Based on subsequent replies to the list, it sounds like there are two problems. The one you linked seems to be sporadic, a subsequent build didn't run into it, but still couldn't build golang: > 142amd64-quarterly 359bbf7fc5af on beefy20 finished with > "done:" status: No crash. > > It built 26082 of 27764 queued packages. So it was an > incremental bulk build. > > It still has a large Skipped figure (1286). Some major > contributions were the failures of: > > go121-1.21.13_5 (skipped 542) > webkit2-gtk_40-2.46.6 (skipped 323) > go123-1.23.7_4 (skipped 100) https://lists.freebsd.org/archives/freebsd-ports/2025-April/007567.html Unless the GOGC patch from comment #11 works out, the best course of action might be to get to a repro on a system that can be used to debug, or at least test workarounds. I haven't been able to reproduce this yet myself, it sounds like it may require a very specific configuration: specific main snapshot + 14.2 jails, maybe some specific hardware as well?
(In reply to Nuno Subtil from comment #24) @Antoine @Bryan can you describe how the builders are set up? What is the host architecture, and how are the amd64 nails created (if not QEMU, then what is used instead)? This may help us be able to reproduce the problem. That said, folks please be aware of what Antoine said earlier about the problem most likely being a problem in base, not in Go.
beefy20 runs on this: Copyright (c) 1992-2025 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 15.0-CURRENT #0 main-n276176-6091e2aa9236: Mon Mar 31 09:41:59 UTC 2025 root@build-15.freebsd.org:/usr/obj/usr/src/sys/CLUSTER15 amd64 FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2) CPU: Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz (2400.00-MHz K8-class CPU) Origin="GenuineIntel" Id=0x606a6 Family=0x6 Model=0x6a Stepping=6 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x121<LAHF,ABM,Prefetch> Structured Extended Features=0xf3bfbfff<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,PQE,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,AVX512IFMA,CLFLUSHOPT,CLWB,PROCTRACE,AVX512CD,SHA,AVX512BW,AVX512VL> Structured Extended Features2=0x40417f5e<AVX512VBMI,UMIP,PKU,OSPKE,AVX512VBMI2,GFNI,VAES,VPCLMULQDQ,AVX512VNNI,AVX512BITALG,TME,AVX512VPOPCNTDQ,LA57,RDPID,SGXLC> Structured Extended Features3=0xbc040412<FSRM,MD_CLEAR,PCONFIG,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> IA32_ARCH_CAPS=0x6bdeb<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO,TSX_CTRL,TAA_NO> AMD Extended Feature Extensions ID EBX=0x200<WBNOINVD> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 549739036672 (524272 MB) avail memory = 535174987776 (510382 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <_ASUS_ OEMTABLE> FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs FreeBSD/SMP: 2 package(s) x 16 core(s) x 2 hardware threads beefy22 runs on this: Copyright (c) 1992-2025 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 15.0-CURRENT #0 main-n276176-6091e2aa9236: Mon Mar 31 09:41:59 UTC 2025 root@build-15.freebsd.org:/usr/obj/usr/src/sys/CLUSTER15 amd64 FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2) CPU: Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz (2400.00-MHz K8-class CPU) Origin="GenuineIntel" Id=0x606a6 Family=0x6 Model=0x6a Stepping=6 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x121<LAHF,ABM,Prefetch> Structured Extended Features=0xf3bfbfff<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,PQE,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,AVX512IFMA,CLFLUSHOPT,CLWB,PROCTRACE,AVX512CD,SHA,AVX512BW,AVX512VL> Structured Extended Features2=0x40417f5e<AVX512VBMI,UMIP,PKU,OSPKE,AVX512VBMI2,GFNI,VAES,VPCLMULQDQ,AVX512VNNI,AVX512BITALG,TME,AVX512VPOPCNTDQ,LA57,RDPID,SGXLC> Structured Extended Features3=0xbc040412<FSRM,MD_CLEAR,PCONFIG,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> IA32_ARCH_CAPS=0x6bdeb<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO,TSX_CTRL,TAA_NO> AMD Extended Feature Extensions ID EBX=0x200<WBNOINVD> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 549739036672 (524272 MB) avail memory = 535174987776 (510382 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <_ASUS_ OEMTABLE> FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs FreeBSD/SMP: 2 package(s) x 16 core(s) x 2 hardware threads The 142amd64 jails are created with poudriere jail -c -j 142amd64 -v 14.2-RELEASE -a amd64
Hi, Antoine said the other day that he is seeing this only in the newer builders, so I ran these tests to collect evidence, no jails/poudriere involved, just trying to build lang/go121 directly on the hosts. gohan06 (Newer builder): FreeBSD gohan06.chi.freebsd.org 15.0-CURRENT FreeBSD 15.0-CURRENT #0 main-n276176-6091e2aa9236: Mon Mar 31 09:41:59 UTC 2025 CPU: Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz (2400.00-MHz K8-class CPU) error: https://people.freebsd.org/~dbaio/go/go121-gohan06-logs.txt beefy19 (Not so old builder): FreeBSD beefy19.nyi.freebsd.org 15.0-CURRENT FreeBSD 15.0-CURRENT #0 main-n276176-6091e2aa9236: Mon Mar 31 09:41:59 UTC 2025 CPU: Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz (2200.00-MHz K8-class CPU) it works: https://people.freebsd.org/~dbaio/go/go121-beefy19-logs.txt My steps: ``` as root: # pkg install bsddialog portconfig as single user: $ export PORT_DBDIR=/home/dbaio/tmp/db $ git clone https://git.freebsd.org/ports.git $ cd ports/lang/go121 $ make stage ``` Both builders' hosts run the same version; they have the same packages, etc. One works, and the other doesn't. The main difference is the CPU. Does that make sense?
An update: Danilo (dbaio) may have found a work-around to allow Go ports to build again. Preliminary tests built go121 successfully. The change is being applied and a new build will start in a few hours. Hopefully we'll see some good news in the morning.
To quote from developers@: With vm.pmap.la57=0, lang/go121 builds on gohan06
To quote dbaio@ from developers@: https://github.com/golang/go/issues/49405 There is a comment that seems to describe the issue we are facing.
Hi, just to give more info. The newer package builders (beefy20, beefy21, beefy22) are set up with vm.pmap.la57=0. Last night, they started a new build and successfully built all lang/go* ports. See: https://pkg-status.freebsd.org/beefy20/build.html?mastername=142amd64-quarterly&build=4764684db30f https://pkg-status.freebsd.org/beefy22/build.html?mastername=142amd64-default&build=0aaaecafbdaf As pointed out in comment #26, these builders' CPUs have the LA57 feature, which is causing the go build issue on recent/current FreeBSD versions.
In addition to turning off la57 on the builders, we should mangle Go binaries we install by setting the la48 flag with elfctl in Mk/Uses/go.mk so they run on LA57 client boxes, until the issue is fixed up stream.
(In reply to Robert Clausecker from comment #32) Yeah, we are going to have to do something like that, though before we do is there any other better approach? While setting the la48 flag will fix any Go program built from the ports tree, it is not a complete solution: - If end-users use the go binary (even one that's la48-branded), they'll still compile apps that will cause crashes. People will have to be extremely diligent on their own about branding anything they compile that will run on -current. - Anything that links to an shlib compiled by Go (patched or not) will crash. - Reducing the address space to lower than what Go expects increases the risk of map key hash collisions, which are catastrophic (completely unpredictable behavior with no way of detecting or preventing it; see the upstream CL). And if I'm thinking about it right, the risk grows exponentially with every bit that's lost. @fuz (and anyone else, please!), before I add a brandelf block to go.mk, what other options should we consider? Some that I can think of: 1. Patch cmd/go to run brandelf itself after it renames a.out to the destination binary (the last step of the compilation chain). 2. Start discussing whether la57 was premature and needs to be rolled back temporarily. Conceptually, (1) would probably be in place of patching go.mk.
I thought the following background might be helpful for anyone following along at home: FreeBSD main extended the addressable address width, which caused Go binaries to crash immediately (and Go is built by Go, so the build crashed right away). The builders have been patched to limit pointer width, which in turn allows Go ports to build again. Go has addressed the problem upstream, but it likely won't appear until 1.25, which is months away. But, that change isn't somehow grandfathered: all we did was get them building. When they're run on a sufficiently-recent main, they'll crash. And anything those Go ports compile will crash (again, ONLY on a sufficiently-recent main). So, we have to do SOMETHING to stop the problem from progressing, because as time goes on more and more users will be on a new enough main to experience (Go application) panics.
From https://github.com/golang/go/issues/49405#issuecomment-2802384059 > Linux is much better about never breaking existing binaries, so any la57 > support would require explicit opt-in by the binary. This to me seems to be the best option. How many binaries would really *require* 5 level page tables? opt-in seems better than opt-out!
(In reply to Adam Weinberger from comment #33) Maybe it would be best to backport the la48 patch upstream is floating and then to force a rebuild of all Go packages. >> Linux is much better about never breaking existing binaries, so any la57 >> support would require explicit opt-in by the binary. > > > This to me seems to be the best option. How many binaries would really > *require* 5 level page tables? opt-in seems better than opt-out! I don't really like that, but it seems like it is an option. In this case, the toolchain should be patched such that it defaults to producing la57 enabled binaries.
(In reply to Robert Clausecker from comment #36) In the upstream CL, the Go developer said: > The only hesitation I have is that that CL is nontrivial enough that > backporting is dicey. If we want to backport something here (and I > suspect we do not, but I'm willing to hear arguments), the la48 bit > CL might be safer. I have no particular expectation that they'll backport it to 1.24 (though I certainly hope they will, and I requested it in the CL), and it touches internals that are way above my paygrade to modify.
(In reply to Adam Weinberger from comment #37) So what about backporting the LA48 CL then? AFAIK it just does the same ELF note tagging elfctl does.
(In reply to Robert Clausecker from comment #38) If the Go development team says backporting it is not simple and potentially perilous, I wouldn't feel comfortable just appling the patch myself. If you'd like to see it backported to 1.24, then join the CL and help them get it done. FreeBSD would be in your debt if you made it happen!
(In reply to Robert Clausecker from comment #38) Or did I misunderstand your question?
(In reply to Adam Weinberger from comment #40) If I understood comment #37 correctly, there are two CLs: one that fixes the cause of the issue and one that just has the Go linker set the LA48 bit on the binaries it generies. The former is tricky to port for sure, but as said in the part you cited, the other one is quite straightforward. So why not apply that as a site patch instead?
Hey team, lang/go* seems to be back in repos!
(In reply to FoxPower from comment #42) You are correct! Danilo fixed the remaining piece today (a mirror wasn't syncing) and the packages are live now.
(In reply to Robert Clausecker from comment #41) You know, I read his message many times and missed that every time! Do you know which CL he's referring to?
(In reply to Adam Weinberger from comment #44) Should be this one: https://go-review.googlesource.com/c/go/+/665475
(In reply to Robert Clausecker from comment #45) Awesome, thanks! Looks like this hasn't been accepted and committed upstream yet. Once it is, I agree with you that this looks like a good candidate for backporting immediately. Off topic: I added you to a phab review (https://reviews.freebsd.org/D49906 for anyone else who's interested) where I'm getting ready to gut go.mk, in case you have any thoughts or interest.
(In reply to Adam Weinberger from comment #46) The revision has two positive review ("Code Review +2"), which means it can be committed at any time according to the Go project policy. So the code is judged to be fine. The reason why it hasn't been committed is that they are currently debating whether to commit/backport this patch or go with adding support for LA57, which conflicts with adding this patch.
(In reply to Adam Weinberger from comment #46) So how do we proceed with this one? I recommend we apply https://go-review.googlesource.com/c/go/+/665475 as a site patch, which should fix the problem.
(In reply to Robert Clausecker from comment #48) Thanks for the ping. I'd like to pull in someone from src who understands vmem(9) subsystem better than I do to make sure that there's not any ramifications we'd be opening ourselves to. (Unless you can speak to that?)
(In reply to Adam Weinberger from comment #49) I wished I'd phrased that differently. Let me try again: any of the folks watching this PR, can you verify that that upstream patch is the correct approach for mitigating this issue for now?
(In reply to Adam Weinberger from comment #50) The setting has been reverted to off by default in main as of https://cgit.freebsd.org/src/commit/?id=7a8440bf0894f910f871e91a660a9f66dbb23f0b. With that, I would recommend we do /not/ attempt to solve this on the FreeBSD side because going forward it is a non-issue and only affects a small window of the unstable main branch. Anywhere dealing with the issue can fix it with sysctl or elfctl until updating their kernel. I'm less prescriptive on what the Go team should do. It was a trap to assume address bits can be used for anything other than addressing, but they have offered up two viable fixes of either requesting LA48 to reserve their bits or otherwise to reduce the size of the tag to fit in remaining space. Basically, the fire has been put out with the previous paragraph and what fireproofing they want to do is up to them based on their own expertise of the GC. They have the necessary FreeBSD APIs to make a choice.
(In reply to Kevin Bowling from comment #51) I apologize for a misunderstanding, I don't watch stable as closely but it looks like this is marked for MFC and will not apply to 14.2 and fixed in 14.3 so the same develop branch criteria will cover not rushing any FreeBSD local change to Go given it is easily worked around and limited to specific hardware and software.
(In reply to Kevin Bowling from comment #52) This patch sets the la48 flag on the binaries Go generates. The effect is that these binaries will be limited to 48 bits of address space, which is the intended outcome. But please feel free to consult kernel people.
(In reply to Robert Clausecker from comment #53) If that is the ultimate direction of the Go team, it will eventually come in via compiler updates. We could pull it in as a patch if necessary but given the circumstances I would recommend waiting to see if that is the approach they merge. With the kernel fixed to retain binary compatibility and an easy workaround there I don't see a need to rush this. stable/14 will be fixed without user involvement after the kernel MFC.