Bug 208421

Summary: lang/go gets stuck when running in a FreeBSD jail
Product: Ports & Packages Reporter: khaelin
Component: Individual Port(s)Assignee: Dmitri Goutnik <dmgk>
Status: Closed Overcome By Events    
Severity: Affects Only Me CC: arthur, chris, dmgk, freebsd8593, smh, snow, vangyzen, w.schwarzenfeld
Priority: --- Keywords: needs-qa
Version: LatestFlags: bugzilla: maintainer-feedback?
Hardware: amd64   
OS: Any   
See Also: https://github.com/golang/go/issues/15044
Attachments:
Description Flags
Full ktrace for go --help
none
Binary ktrace file for go --help
none
Working full ktrace for go --help none

Description khaelin 2016-03-31 14:16:08 UTC 
Created attachment 168820 [details]
Full ktrace for go --help

## Go version
1.6 (official freebsd pkg 1.6,1)

## System
FreeBSD 10.2 (10.2-RELEASE-p13).
Commands executed in a jail.

## Problem
Running any kind of go command (even something as simple as `go --help`) in a jail gets stuck and has to be killed with SIGKILL. There is no such problem when running `go` outside of a jail.

## Trace

Running `ktrace go --help`:

 72810 go       CALL  sigaction(SIG 32,0x7fffffffea18,0)
 72810 go       RET   sigaction 0
 72810 go       CALL  clock_gettime(0x4,0xc820020750)
 72810 go       RET   clock_gettime 0
 72810 go       CALL  sigprocmask(SIG_SETMASK,0x7fffffffe970,0x7fffffffe9a0)
 72810 go       RET   sigprocmask 0
 72810 go       CALL  _umtx_op(0x8010ad330,0x11,0,0,0)

Full trace attached.

I have the exact same problem when executing in a jail a binary built with `go build` outside a jail.

This is the first time we are trying to run our Go software on FreeBSD, so I have no idea whether it is a FreeBSD problem or a Go problem.

I also submitted a ticket on the go issue tracker (https://github.com/golang/go/issues/15044) in case this is a go bug.
Comment 1 khaelin 2016-03-31 14:16:40 UTC 
Created attachment 168821 [details]
Binary ktrace file for go --help
Comment 2 Steven Hartland freebsd_committer freebsd_triage 2016-05-12 14:21:05 UTC 
We run go in jails all over, so this is very odd.

Is there anything specific about your jail?
Comment 3 khaelin 2016-05-18 13:26:34 UTC 
Nothing special about our jails. They are very simple: we create them with release tarballs, without any external tool.

A quick google search shows that there seems to be multiple issues related to jails and _umtx_op.

As a workaround, we now build/run our software out of a jail. I will test in 10.3-RELEASE when I get the chance.
Comment 4 Eric van Gyzen freebsd_committer freebsd_triage 2016-05-18 15:01:53 UTC 
 72810 102291 go       CALL  _umtx_op(0x8010ad330,0x11,0,0,0)

0x11 is UMTX_OP_MUTEX_WAIT, so a mutex is locked and the thread is waiting for the owner to unlock it.  The problem is, there is only one thread, so he must be the owner, so he's waiting for himself, so he's deadlocked.  We'll need a stack trace from userland in order to debug this.  You'll need to build at least go, libc, and libthr with debug symbols, get it into this state, attach with gdb, and run "bt".
Comment 5 khaelin 2016-05-18 15:17:21 UTC 
Thank you for the help. I will put that in my TODO list and report when I get it done.
Comment 6 Steven Hartland freebsd_committer freebsd_triage 2016-05-18 15:24:30 UTC 
Just to check you're running on real hardware not a VM?
Comment 7 Steven Hartland freebsd_committer freebsd_triage 2016-05-18 15:34:34 UTC 
Created attachment 170450 [details]
Working full ktrace for go --help

This is a the output from jail here on a 10.2-RELEASE box with go 1.6.2

It seems odd my kdump is resolving symbols your's isn't are you sure the box has a matching kernel + world for both host and jail?
Comment 8 khaelin 2016-05-18 15:44:42 UTC 
> Just to check you're running on real hardware not a VM?
Yep, this is a simple server with a Core I5 CPU.
Comment 9 Eric van Gyzen freebsd_committer freebsd_triage 2016-05-18 18:44:24 UTC 
(In reply to Steven Hartland from comment #7)

> It seems odd my kdump is resolving symbols your's isn't are you sure
> the box has a matching kernel + world for both host and jail?

Are you referring to the umtx_op parameters like UMTX_OP_FOO_BAR?  Those were added after 10.2.  Are you running kdump on 10.2 or on a later build?
Comment 10 Eric van Gyzen freebsd_committer freebsd_triage 2016-05-18 18:47:05 UTC 
khaelin and Steven:  How many CPUs are in the root CPU set for your jails?  I wonder if processes in jails are getting mixed information about how many CPUs there are and making mixed decisions about whether to enable multi-threading mode.  This is a total shot in the dark, of course.
Comment 11 Steven Hartland freebsd_committer freebsd_triage 2016-05-18 19:03:16 UTC 
(In reply to Eric van Gyzen from comment #10)
We're not using any custom setup at all here, cpuset would be quite a custom thing to do IMO.

This is 10.2-RELEASE + a few patches but nothing which touches that area, and from what I can see those defines are in 10.2:
https://svnweb.freebsd.org/base/releng/10.2/usr.bin/truss/syscalls.c?revision=285830&view=markup#l422
Comment 12 Steven Hartland freebsd_committer freebsd_triage 2016-05-18 19:04:40 UTC 
FreeBSD 10.2-RELEASE-p14 #0: Thu Mar 17 16:08:51 UTC 2016
    root@xxx:/usr/obj/usr/src/sys/MULTIPLAY amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
CPU: Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz (2500.05-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206d7  Family=0x6  Model=0x2d  Stepping=7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x1fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 34359738368 (32768 MB)
avail memory = 33219543040 (31680 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <DELL   DCSRADON>
FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs
FreeBSD/SMP: 2 package(s) x 6 core(s) x 2 SMT threads
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
...
Comment 13 Eric van Gyzen freebsd_committer freebsd_triage 2016-05-18 19:09:31 UTC 
(In reply to Steven Hartland from comment #11)

> and from what I can see those defines are in 10.2:

Right you are.  Sorry about that.  (Serves me right for trusting my memory instead of asking Subversion!)
Comment 14 Walter Schwarzenfeld freebsd_triage 2018-01-13 23:54:52 UTC 
Is this relevant?
Comment 15 Steven Hartland freebsd_committer freebsd_triage 2018-01-14 01:08:14 UTC 
Is what relavent?
Comment 16 Walter Schwarzenfeld freebsd_triage 2018-01-14 01:13:07 UTC 
Is the problem still present?
Comment 17 James Snow 2018-03-21 19:55:51 UTC 
Possibly a different problem, or possibly related.... I am unable to compile go-1.10 from ports when running in a virtual machine. Physical systems seem to build it without issue.

This can be quickly reproduced in EC2...

root@ip-172-31-26-138:/usr/ports/lang/go # make
===>  License BSD3CLAUSE accepted by the user
===>   go-1.10,1 depends on file: /usr/local/sbin/pkg - found
=> go1.10.src.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch https://golang.org/dl/go1.10.src.tar.gz
go1.10.src.tar.gz                             100% of   17 MB   50 MBps 00m00s
===> Fetching all distfiles required by go-1.10,1 for building
===>  Extracting for go-1.10,1
=> SHA256 Checksum OK for go1.10.src.tar.gz.
===>  Patching for go-1.10,1
===>   go-1.10,1 depends on package: go14>=1.4 - found
===>  Configuring for go-1.10,1
===>  Building for go-1.10,1
cd /usr/ports/lang/go/work/go/src &&  GOROOT=/usr/ports/lang/go/work/go GOROOT_FINAL=/usr/local/go  GOROOT_BOOTSTRAP=/usr/local/go14  GOBIN= GOARCH=amd64 GOOS=freebsd  GO386=  /bin/sh make.bash
-ap: not found
go: not found
Building Go cmd/dist using /usr/local/go14.
Building Go toolchain1 using /usr/local/go14.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
gp 0xc42041d680 goid 6 status 4 gcscandone false gcscanvalid false
fatal error: scan missed a g

goroutine 18 [running]:
runtime.throw(0x6f511e, 0xf)
	/usr/local/go/src/runtime/panic.go:619 +0x81 fp=0xc420033ee0 sp=0xc420033ec0 pc=0x428f91
runtime.gcMarkRootCheck()
	/usr/local/go/src/runtime/mgcmark.go:165 +0x21a fp=0xc420033f20 sp=0xc420033ee0 pc=0x41a6ca
runtime.gcMarkDone()
	/usr/local/go/src/runtime/mgc.go:1465 +0xff fp=0xc420033f48 sp=0xc420033f20 pc=0x417cff
runtime.gcBgMarkWorker(0xc420022500)
	/usr/local/go/src/runtime/mgc.go:1912 +0x2e7 fp=0xc420033fd8 sp=0xc420033f48 pc=0x418da7
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420033fe0 sp=0xc420033fd8 pc=0x4542d1
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1723 +0x79

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc4201af8ac)
	/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc4201af8a0)
	/usr/local/go/src/sync/waitgroup.go:129 +0x72
cmd/go/internal/work.(*Builder).Do(0xc420422460, 0xc42049d7c0)
	/usr/local/go/src/cmd/go/internal/work/exec.go:173 +0x3b0
cmd/go/internal/work.InstallPackages(0xc42000e0e0, 0x4, 0x4, 0x0)
	/usr/local/go/src/cmd/go/internal/work/build.go:482 +0xb25
cmd/go/internal/work.runInstall(0x865de0, 0xc42000e0e0, 0x4, 0x4)
	/usr/local/go/src/cmd/go/internal/work/build.go:413 +0x49
main.main()
	/usr/local/go/src/cmd/go/main.go:140 +0x7e1

goroutine 4 [syscall]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 7 [running]:
runtime: unexpected return pc for runtime.systemstack_switch called from 0xc4203b8698
stack: frame={sp:0xc4203b8678, fp:0xc4203b8680} stack=[0xc4203b8000,0xc4203ba000)
000000c4203b8578:  000000000041187f <runtime.(*mspan).nextFreeIndex+111>  00000008008b2da8
000000c4203b8588:  0000000000000008  0000000000000200
000000c4203b8598:  0000000000000040  0000000000000000
000000c4203b85a8:  000000c4203b8600  000000000040f935 <runtime.(*mcache).nextFree+85>
000000c4203b85b8:  00000008008b2da8  0000000000000048
000000c4203b85c8:  0000000000000008  000000c4203b8600
000000c4203b85d8:  000000000041dac6 <runtime.gcmarknewobject+54>  000000c420172520
000000c4203b85e8:  0000000800a9ed9a  000000080084e604
000000c4203b85f8:  0000000000000052  000000c4203b86a0
000000c4203b8608:  000000000041004d <runtime.mallocgc+1293>  000000c420172520
000000c4203b8618:  0000000000000010  0000000000000000
000000c4203b8628:  00000008008b2da8  0000000000000000
000000c4203b8638:  010000000041dac6  0000000000000000
000000c4203b8648:  000000000000000e  000000c420172520
000000c4203b8658:  0000000000000200  0000000000000052
000000c4203b8668:  0000000000000200  0000000000000052
000000c4203b8678: <000000c4203b8698 >000000000042ac5a <runtime.gopark+282>
000000c4203b8688:  0000000000714250  000000080084e601
000000c4203b8698:  000000c4203b86d8  000000000042ad0e <runtime.goparkunlock+94>
000000c4203b86a8:  0000000000714258  0000000000877a40
000000c4203b86b8:  00000000006f32f7  000000000000000a
000000c4203b86c8:  000000c420172519  0000000000000004
000000c4203b86d8:  000000c4203b8748  000000000043aa14 <runtime.semacquire1+468>
000000c4203b86e8:  0000000000877a40  00000000006f32f7
000000c4203b86f8:  000000000000000a  0000000000000019
000000c4203b8708:  0000000000000004  0000000000000000
000000c4203b8718:  00000000000028c0  00000000000000a3
000000c4203b8728:  000000c420068060  0000000000877a40
000000c4203b8738:  0000000000877a50  0000000000877a40
000000c4203b8748:  000000c4203b8770  000000000043a72d <sync.runtime_SemacquireMutex+61>
000000c4203b8758:  00000000008894b4  000000000043ba00 <runtime.(*semaRoot).rotateRight+528>
000000c4203b8768:  0000000000000003  000000c4203b87b8
000000c4203b8778:  0000000000468068 <sync.(*Mutex).Lock+264>
runtime: unexpected return pc for runtime.systemstack_switch called from 0xc4203b8698
stack: frame={sp:0xc4203b8678, fp:0xc4203b8680} stack=[0xc4203b8000,0xc4203ba000)
000000c4203b8578:  000000000041187f <runtime.(*mspan).nextFreeIndex+111>  00000008008b2da8
000000c4203b8588:  0000000000000008  0000000000000200
000000c4203b8598:  0000000000000040  0000000000000000
000000c4203b85a8:  000000c4203b8600  000000000040f935 <runtime.(*mcache).nextFree+85>
000000c4203b85b8:  00000008008b2da8  0000000000000048
000000c4203b85c8:  0000000000000008  000000c4203b8600
000000c4203b85d8:  000000000041dac6 <runtime.gcmarknewobject+54>  000000c420172520
000000c4203b85e8:  0000000800a9ed9a  000000080084e604
000000c4203b85f8:  0000000000000052  000000c4203b86a0
000000c4203b8608:  000000000041004d <runtime.mallocgc+1293>  000000c420172520
000000c4203b8618:  0000000000000010  0000000000000000
000000c4203b8628:  00000008008b2da8  0000000000000000
000000c4203b8638:  010000000041dac6  0000000000000000
000000c4203b8648:  000000000000000e  000000c420172520
000000c4203b8658:  0000000000000200  0000000000000052
000000c4203b8668:  0000000000000200  0000000000000052
000000c4203b8678: <000000c4203b8698 >000000000042ac5a <runtime.gopark+282>
000000c4203b8688:  0000000000714250  000000080084e601
000000c4203b8698:  000000c4203b86d8  000000000042ad0e <runtime.goparkunlock+94>
000000c4203b86a8:  0000000000714258  0000000000877a40
000000c4203b86b8:  00000000006f32f7  000000000000000a
000000c4203b86c8:  000000c420172519  0000000000000004
000000c4203b86d8:  000000c4203b8748  000000000043aa14 <runtime.semacquire1+468>
000000c4203b86e8:  0000000000877a40  00000000006f32f7
000000c4203b86f8:  000000000000000a  0000000000000019
000000c4203b8708:  0000000000000004  0000000000000000
000000c4203b8718:  00000000000028c0  00000000000000a3
000000c4203b8728:  000000c420068060  0000000000877a40
000000c4203b8738:  0000000000877a50  0000000000877a40
000000c4203b8748:  000000c4203b8770  000000000043a72d <sync.runtime_SemacquireMutex+61>
000000c4203b8758:  00000000008894b4  000000000043ba00 <runtime.(*semaRoot).rotateRight+528>
000000c4203b8768:  0000000000000003  000000c4203b87b8
000000c4203b8778:  0000000000468068 <sync.(*Mutex).Lock+264>
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:363
created by cmd/go/internal/work.(*Builder).Do
	/usr/local/go/src/cmd/go/internal/work/exec.go:151 +0x38c

goroutine 8 [syscall]:
syscall.Syscall6(0x214, 0x0, 0xec7, 0x0, 0x18, 0x0, 0x0, 0xc4203bcbc8, 0x4e334f, 0x10)
	/usr/local/go/src/syscall/asm_unix_amd64.s:42 +0x5
os.(*Process).blockUntilWaitable(0xc420049a10, 0x0, 0x0, 0x2)
	/usr/local/go/src/os/wait_wait6.go:29 +0x5e
os.(*Process).wait(0xc420049a10, 0xc420321b80, 0xc42004c238, 0xc42004c238)
	/usr/local/go/src/os/exec_unix.go:22 +0x3c
os.(*Process).Wait(0xc420049a10, 0x713ed0, 0x713ed8, 0x713ec8)
	/usr/local/go/src/os/exec.go:123 +0x2b
os/exec.(*Cmd).Wait(0xc42004c160, 0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:461 +0x5c
os/exec.(*Cmd).Run(0xc42004c160, 0xc42008d5e0, 0xc420148e00)
	/usr/local/go/src/os/exec/exec.go:305 +0x5c
cmd/go/internal/work.(*Builder).toolID(0xc420422460, 0x6f1c71, 0x7, 0x2c, 0xc4203bd380)
	/usr/local/go/src/cmd/go/internal/work/buildid.go:183 +0x321
cmd/go/internal/work.(*Builder).buildActionID(0xc420422460, 0xc4203c8c80, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/cmd/go/internal/work/exec.go:219 +0xf69
cmd/go/internal/work.(*Builder).build(0xc420422460, 0xc4203c8c80, 0x0, 0x0)
	/usr/local/go/src/cmd/go/internal/work/exec.go:304 +0xb0
cmd/go/internal/work.(*Builder).Do.func1(0xc4203c8c80)
	/usr/local/go/src/cmd/go/internal/work/exec.go:106 +0x72
cmd/go/internal/work.(*Builder).Do.func2(0xc4201af8a0, 0xc420422460, 0xc420321b40)
	/usr/local/go/src/cmd/go/internal/work/exec.go:164 +0xbb
created by cmd/go/internal/work.(*Builder).Do
	/usr/local/go/src/cmd/go/internal/work/exec.go:151 +0x38c

goroutine 9 [IO wait]:
internal/poll.runtime_pollWait(0x8008c4ad8, 0x72, 0xc42002ece8)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc42030cd88, 0x72, 0xffffffffffffff01, 0x739840, 0x84e2a8)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc42030cd88, 0xc420401801, 0x200, 0x200)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc42030cd70, 0xc420401800, 0x200, 0x200, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
os.(*File).read(0xc420171e10, 0xc420401800, 0x200, 0x200, 0xc420401800, 0x0, 0x0)
	/usr/local/go/src/os/file_unix.go:226 +0x4e
os.(*File).Read(0xc420171e10, 0xc420401800, 0x200, 0x200, 0x0, 0xc42002ee78, 0xc42002ee60)
	/usr/local/go/src/os/file.go:107 +0x6a
bytes.(*Buffer).ReadFrom(0xc42008d490, 0x739420, 0xc420171e10, 0x8008c4bc8, 0xc42008d490, 0xc4203c8c01)
	/usr/local/go/src/bytes/buffer.go:205 +0xa0
io.copyBuffer(0x738f40, 0xc42008d490, 0x739420, 0xc420171e10, 0x0, 0x0, 0x0, 0x100000000000000, 0xc4203c8c80, 0xc4200c6580)
	/usr/local/go/src/io/io.go:386 +0x31a
io.Copy(0x738f40, 0xc42008d490, 0x739420, 0xc420171e10, 0x612eb1, 0x1, 0xc42002ef1f)
	/usr/local/go/src/io/io.go:362 +0x5a
os/exec.(*Cmd).writerDescriptor.func1(0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:275 +0x4d
os/exec.(*Cmd).Start.func1(0xc42004c160, 0xc420321be0)
	/usr/local/go/src/os/exec/exec.go:396 +0x27
created by os/exec.(*Cmd).Start
	/usr/local/go/src/os/exec/exec.go:395 +0x5df

goroutine 10 [IO wait]:
internal/poll.runtime_pollWait(0x8008c4938, 0x72, 0xc42002f4e8)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc42030cfb8, 0x72, 0xffffffffffffff01, 0x739840, 0x84e2a8)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc42030cfb8, 0xc420401601, 0x200, 0x200)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc42030cfa0, 0xc420401600, 0x200, 0x200, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
os.(*File).read(0xc420171e28, 0xc420401600, 0x200, 0x200, 0xc420401600, 0x0, 0x0)
	/usr/local/go/src/os/file_unix.go:226 +0x4e
os.(*File).Read(0xc420171e28, 0xc420401600, 0x200, 0x200, 0x0, 0x0, 0xc42002f660)
	/usr/local/go/src/os/file.go:107 +0x6a
bytes.(*Buffer).ReadFrom(0xc42008d5e0, 0x739420, 0xc420171e28, 0x8008c4bc8, 0xc42008d5e0, 0x1)
	/usr/local/go/src/bytes/buffer.go:205 +0xa0
io.copyBuffer(0x738f40, 0xc42008d5e0, 0x739420, 0xc420171e28, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/io.go:386 +0x31a
io.Copy(0x738f40, 0xc42008d5e0, 0x739420, 0xc420171e28, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/io.go:362 +0x5a
os/exec.(*Cmd).writerDescriptor.func1(0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:275 +0x4d
os/exec.(*Cmd).Start.func1(0xc42004c160, 0xc420321c20)
	/usr/local/go/src/os/exec/exec.go:396 +0x27
created by os/exec.(*Cmd).Start
	/usr/local/go/src/os/exec/exec.go:395 +0x5df
go tool dist: FAILED: /usr/ports/lang/go/work/go/pkg/tool/freebsd_amd64/go_bootstrap install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 2
*** Error code 2

Stop.
make[1]: stopped in /usr/ports/lang/go
*** Error code 1

Stop.
make: stopped in /usr/ports/lang/go
root@ip-172-31-26-138:/usr/ports/lang/go #
Comment 18 James Snow 2018-03-21 20:08:02 UTC 
The reason I thought my issue was possibly related is that in some cases I don't get a build failure. Instead, go_bootstrap will spin forever with the following in truss:

sched_yield()					 = 0 (0x0)
clock_gettime(4,{ 9309.502287093 })		 = 0 (0x0)
clock_gettime(4,{ 9309.502318796 })		 = 0 (0x0)
sched_yield()					 = 0 (0x0)
Comment 19 James Snow 2018-03-21 21:48:39 UTC 
In the event that anyone else stumbles down this path, I was able to resolve my problem by switching to the 4BSD scheduler.
Comment 20 Arthur Wiebe 2018-06-04 03:07:19 UTC 
I am experiencing something similar. Using an ARM poudriere jail inside of a VM. I don't have bare metal hardware handy to test on right now, and also not that experienced with freebsd to know how to switch the scheduler to 4BSD.

The output looks like this.

Building Go cmd/dist using /usr/local/go14.
Building Go toolchain1 using /usr/local/go14.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x0 addr=0x0 pc=0x383c0]

runtime stack:
runtime.throw(0x2b01d4, 0x2a)
        /usr/local/go/src/runtime/panic.go:616 +0x60
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:372 +0x22c
runtime.netpollunblock(0x0, 0x77, 0x1, 0x0)
        /usr/local/go/src/runtime/netpoll.go:377 +0x14
runtime.netpollready(0x1073ba14, 0x0, 0x77)
        /usr/local/go/src/runtime/netpoll.go:301 +0x74
runtime.netpoll(0x3f5e00, 0x60ad27db)
        /usr/local/go/src/runtime/netpoll_kqueue.go:111 +0xc8
runtime.sysmon()
        /usr/local/go/src/runtime/proc.go:4265 +0x558
runtime.mstart1(0x0)
        /usr/local/go/src/runtime/proc.go:1227 +0xb8
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1193 +0x68

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0x10a4e98c)
        /usr/local/go/src/runtime/sema.go:56 +0x2c
sync.(*WaitGroup).Wait(0x10a4e980)
        /usr/local/go/src/sync/waitgroup.go:129 +0x84
cmd/go/internal/work.(*Builder).Do(0x1082f320, 0x10900630)
        /usr/local/go/src/cmd/go/internal/work/exec.go:173 +0x338
cmd/go/internal/work.InstallPackages(0x1070e078, 0x4, 0x5, 0x0)
        /usr/local/go/src/cmd/go/internal/work/build.go:482 +0xa40
cmd/go/internal/work.runInstall(0x3f2970, 0x1070e078, 0x4, 0x5)
        /usr/local/go/src/cmd/go/internal/work/build.go:413 +0x38
main.main()
        /usr/local/go/src/cmd/go/main.go:140 +0x688

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:139 +0x130
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x14
created by os/signal.init.0
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x30

goroutine 8 [runnable]:
syscall.Syscall(0x3, 0xe, 0x108f12ec, 0x4, 0x0, 0x108f12ec, 0x0)
        /usr/local/go/src/syscall/asm_freebsd_arm.s:17 +0x8
syscall.readlen(0xe, 0x108f12ec, 0x4, 0x3, 0x10827300, 0x3e)
        /usr/local/go/src/syscall/zsyscall_freebsd_arm.go:1263 +0x3c
syscall.forkExec(0x109e8580, 0x3f, 0x10b14420, 0x2, 0x2, 0x108f13a8, 0x10, 0x0, 0x10a4ea10)
        /usr/local/go/src/syscall/exec_unix.go:203 +0x2cc
syscall.StartProcess(0x109e8580, 0x3f, 0x10b14420, 0x2, 0x2, 0x108f13a8, 0x2, 0x4, 0x0, 0x0)
        /usr/local/go/src/syscall/exec_unix.go:241 +0x44
os.startProcess(0x109e8580, 0x3f, 0x10b14420, 0x2, 0x2, 0x108f147c, 0x108fec00, 0x3d, 0x3d)
        /usr/local/go/src/os/exec_posix.go:46 +0x148
os.StartProcess(0x109e8580, 0x3f, 0x10b14420, 0x2, 0x2, 0x108f147c, 0x0, 0x0, 0x1)
        /usr/local/go/src/os/exec.go:102 +0x58
os/exec.(*Cmd).Start(0x109006e0, 0x2a1601, 0x1082f980)
        /usr/local/go/src/os/exec/exec.go:379 +0x390
os/exec.(*Cmd).Run(0x109006e0, 0x1082f980, 0x108fe800)
        /usr/local/go/src/os/exec/exec.go:302 +0x1c
cmd/go/internal/work.(*Builder).toolID(0x1082f320, 0x2a291f, 0x7, 0x2c, 0x1094a890)
        /usr/local/go/src/cmd/go/internal/work/buildid.go:183 +0x24c
cmd/go/internal/work.(*Builder).buildActionID(0x1082f320, 0x108b44d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/cmd/go/internal/work/exec.go:219 +0xb8c
cmd/go/internal/work.(*Builder).build(0x1082f320, 0x108b44d0, 0x0, 0x0)
        /usr/local/go/src/cmd/go/internal/work/exec.go:304 +0x64
cmd/go/internal/work.(*Builder).Do.func1(0x108b44d0)
        /usr/local/go/src/cmd/go/internal/work/exec.go:106 +0x58
cmd/go/internal/work.(*Builder).Do.func2(0x10a4e980, 0x1082f320, 0x10b14400)
        /usr/local/go/src/cmd/go/internal/work/exec.go:164 +0x84
created by cmd/go/internal/work.(*Builder).Do
        /usr/local/go/src/cmd/go/internal/work/exec.go:151 +0x318
go tool dist: FAILED: /wrkdirs/usr/ports/lang/go/work/go/pkg/tool/freebsd_arm/go_bootstrap install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 2
*** Error code 2
Comment 21 Dmitri Goutnik freebsd_committer freebsd_triage 2021-01-22 22:49:02 UTC 
Please reopen if this is still an issue.
Comment 22 Christos Chatzaras 2022-05-26 12:01:55 UTC 
I add this here as I believe it's the same issue.


CPU: AMD Ryzen 9 5950X 16 Cores (32 Threads)
OS: FreeBSD 13.1


I use this command to compress a lot of images:

find . -iname "*.jpg" -exec ~/graphicsmagick/bin/gm mogrify -strip -quality 75% {} > /dev/null 2>&1 \;


The issue is that GraphicsMagick (the same with ImageMagick) after some time hangs, the process usage is 100% and truss shows:

sched_yield() = 0 (0x0)
sched_yield() = 0 (0x0)
sched_yield() = 0 (0x0)