On Ubuntu Jammy Jellyfish, the golang test suite fails immediately after starting it: root@pustak:/# cd /usr/share/go/test && go run run.go -v # internal/reflectlite runtime: unexpected return pc for runtime.mallocgc called from 0x0 stack: frame={sp:0xc00045a3e0, fp:0xc00045a458} stack=[0xc000456000,0xc00045e000) 0x000000c00045a2e0: 0x0000000000000000 0x0000000000000000 0x000000c00045a2f0: 0x0000000000000000 0x0000000000000000 0x000000c00045a300: 0x0000000000000000 0x0000000000000000 0x000000c00045a310: 0x0000000000000000 0x0000000000000000 0x000000c00045a320: 0x0000000000000000 0x0000000000000000 0x000000c00045a330: 0x0000000000000000 0x0000000000000000 0x000000c00045a340: 0x0000000000000000 0x0000000000000000 0x000000c00045a350: 0x0000000000000000 0x0000000000000000 0x000000c00045a360: 0x0000000000000000 0x0000000000000000 0x000000c00045a370: 0x0000000000000000 0x0000000000000000 0x000000c00045a380: 0x0000000000000000 0x0000000000000000 0x000000c00045a390: 0x0000000000000000 0x0000000000000000 0x000000c00045a3a0: 0x0000000000000000 0x0000000000000000 0x000000c00045a3b0: 0x0000000000000000 0x0000000000000000 0x000000c00045a3c0: 0x0000000000000000 0x0000000000000000 0x000000c00045a3d0: 0x0000000000000000 0x000000000040d313 <runtime.mallocgc+0x0000000000000673> 0x000000c00045a3e0: <0x000000c0004549b0 0x0000000000000000 0x000000c00045a3f0: 0x0000000000000000 0x0000000000000000 0x000000c00045a400: 0x0000000000203000 0x000000082810ad20 0x000000c00045a410: 0x0000000000000000 0x0000000000000000 0x000000c00045a420: 0x0000000000000050 0x000000c00004c400 0x000000c00045a430: 0x0000000000000000 0x0000000000000000 0x000000c00045a440: 0x0000000000000000 0x000000c0000d0470 0x000000c00045a450: !0x0000000000000000 >0x0000000000000000 0x000000c00045a460: 0x0000000000cec5c0 0x0000000000000001 0x000000c00045a470: 0x000000c0000d04c8 0x0000000000bb19df <cmd/compile/internal/importer.(*iimporter).typAt+0x00000000000000bf> 0x000000c00045a480: 0x0000000000cab660 0x000000c000401470 0x000000c00045a490: 0x0000000000001670 0x000000c000454968 0x000000c00045a4a0: 0x000000c000454968 0x000000c00004c400 0x000000c00045a4b0: 0x000000c00040f260 0x000000000000000a 0x000000c00045a4c0: 0x660600c0000d04d8 0x000000c0000d04f0 0x000000c00045a4d0: 0x0000000000bb3533 <cmd/compile/internal/importer.(*importReader).typ+0x0000000000000033> 0x000000c00044a000 0x000000c00045a4e0: 0x0000000000001670 0x0000000000000000 0x000000c00045a4f0: 0x000000c0000d06d0 0x0000000000bb45df <cmd/compile/internal/importer.(*importReader).doType+0x0000000000000f5f> 0x000000c00045a500: 0x000000c000454960 0x0000000000000000 0x000000c00045a510: 0x0000000000000000 0x0000000000000000 0x000000c00045a520: 0x0000000000000000 0x0000000000000000 0x000000c00045a530: 0x0000000000000000 0x0000000000000000 0x000000c00045a540: 0x0000035f00000002 0x0000000000000000 0x000000c00045a550: 0x0000000000000006 fatal error: unknown caller pc runtime stack: runtime.throw({0xd2c6b3?, 0x12f82c0?}) /usr/lib/go-1.18/src/runtime/panic.go:992 +0x71 runtime.gentraceback(0xffffffffffff8000?, 0x6?, 0xc00036f380?, 0x828107dd8?, 0x0, 0x0, 0x7fffffff, 0xd54500, 0x0?, 0x0) /usr/lib/go-1.18/src/runtime/traceback.go:254 +0x1a36 runtime.copystack(0xc0000021a0, 0x800000002?) /usr/lib/go-1.18/src/runtime/stack.go:930 +0x2f5 runtime.newstack() /usr/lib/go-1.18/src/runtime/stack.go:1110 +0x497 runtime.morestack() /usr/lib/go-1.18/src/runtime/asm_amd64.s:547 +0x8b goroutine 1 [copystack]: runtime.heapBitsSetType(0xc0004549b0?, 0x0?, 0x0?, 0x0?) /usr/lib/go-1.18/src/runtime/mbitmap.go:832 +0xbcc fp=0xc00045a3e0 sp=0xc00045a3d8 pc=0x415d2c runtime: unexpected return pc for runtime.mallocgc called from 0x0 stack: frame={sp:0xc00045a3e0, fp:0xc00045a458} stack=[0xc000456000,0xc00045e000) 0x000000c00045a2e0: 0x0000000000000000 0x0000000000000000 0x000000c00045a2f0: 0x0000000000000000 0x0000000000000000 0x000000c00045a300: 0x0000000000000000 0x0000000000000000 0x000000c00045a310: 0x0000000000000000 0x0000000000000000 0x000000c00045a320: 0x0000000000000000 0x0000000000000000 0x000000c00045a330: 0x0000000000000000 0x0000000000000000 0x000000c00045a340: 0x0000000000000000 0x0000000000000000 0x000000c00045a350: 0x0000000000000000 0x0000000000000000 0x000000c00045a360: 0x0000000000000000 0x0000000000000000 0x000000c00045a370: 0x0000000000000000 0x0000000000000000 0x000000c00045a380: 0x0000000000000000 0x0000000000000000 0x000000c00045a390: 0x0000000000000000 0x0000000000000000 0x000000c00045a3a0: 0x0000000000000000 0x0000000000000000 0x000000c00045a3b0: 0x0000000000000000 0x0000000000000000 0x000000c00045a3c0: 0x0000000000000000 0x0000000000000000 0x000000c00045a3d0: 0x0000000000000000 0x000000000040d313 <runtime.mallocgc+0x0000000000000673> 0x000000c00045a3e0: <0x000000c0004549b0 0x0000000000000000 0x000000c00045a3f0: 0x0000000000000000 0x0000000000000000 0x000000c00045a400: 0x0000000000203000 0x000000082810ad20 0x000000c00045a410: 0x0000000000000000 0x0000000000000000 0x000000c00045a420: 0x0000000000000050 0x000000c00004c400 0x000000c00045a430: 0x0000000000000000 0x0000000000000000 0x000000c00045a440: 0x0000000000000000 0x000000c0000d0470 0x000000c00045a450: !0x0000000000000000 >0x0000000000000000 0x000000c00045a460: 0x0000000000cec5c0 0x0000000000000001 0x000000c00045a470: 0x000000c0000d04c8 0x0000000000bb19df <cmd/compile/internal/importer.(*iimporter).typAt+0x00000000000000bf> 0x000000c00045a480: 0x0000000000cab660 0x000000c000401470 0x000000c00045a490: 0x0000000000001670 0x000000c000454968 0x000000c00045a4a0: 0x000000c000454968 0x000000c00004c400 0x000000c00045a4b0: 0x000000c00040f260 0x000000000000000a 0x000000c00045a4c0: 0x660600c0000d04d8 0x000000c0000d04f0 0x000000c00045a4d0: 0x0000000000bb3533 <cmd/compile/internal/importer.(*importReader).typ+0x0000000000000033> 0x000000c00044a000 0x000000c00045a4e0: 0x0000000000001670 0x0000000000000000 0x000000c00045a4f0: 0x000000c0000d06d0 0x0000000000bb45df <cmd/compile/internal/importer.(*importReader).doType+0x0000000000000f5f> 0x000000c00045a500: 0x000000c000454960 0x0000000000000000 0x000000c00045a510: 0x0000000000000000 0x0000000000000000 0x000000c00045a520: 0x0000000000000000 0x0000000000000000 0x000000c00045a530: 0x0000000000000000 0x0000000000000000 0x000000c00045a540: 0x0000035f00000002 0x0000000000000000 0x000000c00045a550: 0x0000000000000006 runtime.mallocgc(0x0, 0xcec5c0, 0x1) /usr/lib/go-1.18/src/runtime/malloc.go:1117 +0x673 fp=0xc00045a458 sp=0xc00045a3e0 pc=0x40d313 # sync panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x5b4342] goroutine 46 [running]: cmd/compile/internal/ir.HasUniquePos({0x0, 0x0}) /usr/lib/go-1.18/src/cmd/compile/internal/ir/node.go:535 +0x22 cmd/compile/internal/ssagen.(*state).exprCheckPtr(0xc000b07a00, {0x0?, 0x0?}, 0x1) /usr/lib/go-1.18/src/cmd/compile/internal/ssagen/ssa.go:2564 +0x70 cmd/compile/internal/ssagen.(*state).expr(...) /usr/lib/go-1.18/src/cmd/compile/internal/ssagen/ssa.go:2560 cmd/compile/internal/ssagen.(*state).exprPtr(0xc000b07a00, {0xe9b778, 0xc000685450}, 0x0, {0x8?, 0x47a978?}) /usr/lib/go-1.18/src/cmd/compile/internal/ssagen/ssa.go:5419 +0x3f cmd/compile/internal/ssagen.(*state).exprCheckPtr(0xc000b07a00, {0xe9be80?, 0xc000689a40?}, 0x1) /usr/lib/go-1.18/src/cmd/compile/internal/ssagen/ssa.go:3048 +0x2a1a cmd/compile/internal/ssagen.(*state).expr(...) /usr/lib/go-1.18/src/cmd/compile/internal/ssagen/ssa.go:2560 cmd/compile/internal/ssagen.(*state).exprCheckPtr(0xc000b07a00, {0xe9c7e0?, 0xc000690960?}, 0x1) /usr/lib/go-1.18/src/cmd/compile/internal/ssagen/ssa.go:3115 +0x3b75 cmd/compile/internal/ssagen.(*state).expr(...) /usr/lib/go-1.18/src/cmd/compile/internal/ssagen/ssa.go:2560 cmd/compile/internal/ssagen.(*state).stmt(0xc000b07a00, {0xe99db0, 0xc00009bf40?}) /usr/lib/go-1.18/src/cmd/compile/internal/ssagen/ssa.go:1633 +0xb71 cmd/compile/internal/ssagen.(*state).stmtList(...) /usr/lib/go-1.18/src/cmd/compile/internal/ssagen/ssa.go:1399 cmd/compile/internal/ssagen.(*state).stmt(0xc000b07a00, {0xe9ae18, 0xc000696850?}) /usr/lib/go-1.18/src/cmd/compile/internal/ssagen/ssa.go:1417 +0x48ee cmd/compile/internal/ssagen.(*state).stmtList(...) /usr/lib/go-1.18/src/cmd/compile/internal/ssagen/ssa.go:1399 cmd/compile/internal/ssagen.buildssa(0xc00040edc0, 0x3) /usr/lib/go-1.18/src/cmd/compile/internal/ssagen/ssa.go:582 +0x1eb4 cmd/compile/internal/ssagen.Compile(0xc00040edc0, 0xc000397790?) /usr/lib/go-1.18/src/cmd/compile/internal/ssagen/pgen.go:183 +0x4c cmd/compile/internal/gc.compileFunctions.func4.1(0xc000421ce0?) /usr/lib/go-1.18/src/cmd/compile/internal/gc/compile.go:153 +0x3a cmd/compile/internal/gc.compileFunctions.func3.1() /usr/lib/go-1.18/src/cmd/compile/internal/gc/compile.go:140 +0x4d created by cmd/compile/internal/gc.compileFunctions.func3 /usr/lib/go-1.18/src/cmd/compile/internal/gc/compile.go:138 +0x78 root@pustak:/usr/share/go/test# root@pustak:/usr/share/go/test# cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.2 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.2 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy Golang test suite from an older Ubuntu release (Focal) works just fine.
Additional data point: same distro on aarch64 works a bit better - it hangs the same way the older aarch64 golang does instead of crashing: root@v3:/# cd /usr/share/go/test && go run run.go -v ok 235.go 0.103s ok 64bit.go 1.270s ok alg.go 0.236s ok alias.go 0.007s ok alias1.go 0.033s ok alias2.go 0.009s ok alias3.go 0.141s ok align.go 0.031s ok append.go 0.139s ok append1.go 0.007s ok args.go 0.175s ok armimm.go 0.077s ok asmhdr.go 0.084s ok assign.go 0.012s ok assign1.go 0.008s ok atomicload.go 0.057s ok bigalg.go 0.083s ok bigmap.go 0.064s ok blank.go 0.092s ok blank1.go 0.009s ok bom.go 0.256s ok bombad.go 0.010s ok bounds.go 0.043s ok cannotassign.go 0.008s ok chancap.go 0.057s ok chanlinear.go 0.120s ok char_lit.go 0.054s ok char_lit1.go 0.007s ok checkbce.go 0.000s ok clearfat.go 2.069s ok closedchan.go 0.089s ok closure.go 0.067s ok closure1.go 0.048s ok closure2.go 0.045s ok closure3.go 0.098s ^Csignal: interrupt root@v3:/usr/share/go/test# dpkg -l | grep golang ii golang:arm64 2:1.18~0ubuntu2 arm64 Go programming language compiler - metapackage ii golang-1.18 1.18.1-1ubuntu1 all Go programming language compiler - metapackage ii golang-1.18-doc 1.18.1-1ubuntu1 all Go programming language - documentation ii golang-1.18-go 1.18.1-1ubuntu1 arm64 Go programming language compiler, linker, compiled stdlib ii golang-1.18-src 1.18.1-1ubuntu1 all Go programming language - source files ii golang-doc 2:1.18~0ubuntu2 all Go programming language - documentation ii golang-go:arm64 2:1.18~0ubuntu2 arm64 Go programming language compiler, linker, compiled stdlib ii golang-src 2:1.18~0ubuntu2 all Go programming language - source files root@v3:/usr/share/go/test# cat /etc/os-release PRETTY_NAME="Ubuntu 22.04 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04 (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy
well, a quick look to trace shows that a problem in between go GC and a new go preemptive scheduler, i.e., we heve a problem in signals emulation. try env GODEBUG=asyncpreemtoff=1
(In reply to Dmitry Chagin from comment #2) env GODEBUG=asyncpreemptoff=1
Okay; the good news is, it seems to get rid of this problem; previously it would fail like above, or in another weird manner; with that env variable set things become reproducible again. The bad news is, it fails in a different way: # command-line-arguments /usr/lib/go-1.18/pkg/tool/linux_amd64/link: mapping output file failed: invalid argument This appears to be caused by fallocate(2) returning EINVAL; not sure why it’s doing that.
Okay, so the fallocate(2) problem is because it returns EINVAL instead of EOPNOTSUPP when running on ZFS, which doesn’t support fallocate(2). With that fixed and the env workaround the tests work :)
(In reply to Edward Tomasz Napierala from comment #5) Nice) Im trying to find the reason why goroutines fails after SIGURG, which is sent by preemptive scheduler. it looks like sometimes pc is 0, that means cpu context is broken after sigreturn. btw1, im trying to run tools/test/avx_sig test, it fails on Ubuntu 22.04, and it looks like avx registers is not preserved across context switch. I don't see any Linux kernel config options for avx registers which is turned off in Ubuntu 22.04. Do you know anything about that? btw2, could you please check arm64 also?
https://reviews.freebsd.org/D40443 https://reviews.freebsd.org/D40444
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=cbbac560911521c0ded3e06e713107176855fae4 commit cbbac560911521c0ded3e06e713107176855fae4 Author: Dmitry Chagin <dchagin@FreeBSD.org> AuthorDate: 2023-06-08 22:33:26 +0000 Commit: Dmitry Chagin <dchagin@FreeBSD.org> CommitDate: 2023-06-08 22:33:26 +0000 linux(4): Preserve fpu xsave state across signal delivery on amd64 PR: 270247 Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D40444 MFC after: 2 weeks sys/amd64/linux/linux_sysvec.c | 99 ++++++++++++++++++++++++++++++++++++-- sys/x86/linux/linux_x86_sigframe.h | 20 +++++++- 2 files changed, 114 insertions(+), 5 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=e40bcfdff7cc0596cd4e748e42a1947fd9f5d430 commit e40bcfdff7cc0596cd4e748e42a1947fd9f5d430 Author: Dmitry Chagin <dchagin@FreeBSD.org> AuthorDate: 2023-06-08 22:33:26 +0000 Commit: Dmitry Chagin <dchagin@FreeBSD.org> CommitDate: 2023-06-29 08:16:03 +0000 linux(4): Preserve fpu xsave state across signal delivery on amd64 PR: 270247 Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D40444 MFC after: 2 weeks (cherry picked from commit cbbac560911521c0ded3e06e713107176855fae4) sys/amd64/linux/linux_sysvec.c | 99 ++++++++++++++++++++++++++++++++++++-- sys/x86/linux/linux_x86_sigframe.h | 20 +++++++- 2 files changed, 114 insertions(+), 5 deletions(-)
Merged to stable/13