Bug 233998 - [hang] [shutdown] system hangs indefinitely on shutdown
Summary: [hang] [shutdown] system hangs indefinitely on shutdown
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 12.0-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: Conrad Meyer
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-12-14 00:39 UTC by Conor
Modified: 2019-01-11 01:33 UTC (History)
2 users (show)

See Also:


Attachments
Image of console output at shutdown (829.63 KB, image/jpeg)
2018-12-14 00:39 UTC, Conor
no flags Details
Output of acpidump -dtv (920.43 KB, text/plain)
2018-12-16 00:23 UTC, Conor
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Conor 2018-12-14 00:39:01 UTC
Created attachment 200096 [details]
Image of console output at shutdown

Since upgrading some hardware in a machine, my system hangs indefinitely when issuing `shutdown -p` such that it behaves like `shutdown -h`. Resetting the machine via `shutdown -r` works fine, and I've ruled out an electrical/hw issue by probing the hardware, and shutting down the system successfully whilst booted into other operating systems. This has been an issue in 11.{1.2}-RELEASE with this hardware, but is still a problem with 12.0-RELEASE.

H/W:
FreeBSD: 12.0-RELEASE
CPU: AMD Threadripper 1950X
Mobo: Asus Zenith Extreme (X399)

I'm unable to capture logs, but I've attached a capture of the console at shutdown. At the point in the image, the system hangs indefinitely.

If there is any further pertinent information I can provide, please let me know.
Comment 1 Conrad Meyer freebsd_committer 2018-12-14 02:06:34 UTC
If you can rebuild a test kernel, please try https://reviews.freebsd.org/D18506 and let me know if it resolves the issue.
Comment 2 Conrad Meyer freebsd_committer 2018-12-14 02:08:52 UTC
(byuu reported a similar issue just a few days ago on hackers@: https://lists.freebsd.org/pipermail/freebsd-hackers/2018-December/053689.html )
Comment 3 Conrad Meyer freebsd_committer 2018-12-14 02:12:04 UTC
FWIW, I also have an X399 motherboard and 1950X, but I have avoided this problem by never turning off my computer ;-).
Comment 4 Conor 2018-12-14 11:20:51 UTC
I've applied this patch and built a new kernel, but it doesn't seem to have rectified the issue for me, unfortunately.
Comment 5 Conrad Meyer freebsd_committer 2018-12-15 02:14:31 UTC
(In reply to Conor from comment #4)
Oh, shoot.  Do you know if you are booting in EFI mode?
Comment 6 commit-hook freebsd_committer 2018-12-15 05:46:19 UTC
A commit references this bug:

Author: cem
Date: Sat Dec 15 05:46:05 UTC 2018
New revision: 342108
URL: https://svnweb.freebsd.org/changeset/base/342108

Log:
  efirt: When present, attempt to use EFI runtime services to shutdown

  PR:		maybe related to 233998 (inconclusive at this time)
  Submitted by:	byuu <byuu AT tutanota.com> (previous version)
  Reviewed by:	imp
  Differential Revision:	https://reviews.freebsd.org/D18506

Changes:
  head/share/man/man9/efirt.9
  head/sys/contrib/zstd/lib/compress/zstd_compress_internal.h
  head/sys/contrib/zstd/lib/compress/zstd_opt.c
  head/sys/dev/efidev/efirt.c
  head/sys/dev/ipmi/ipmi.c
  head/sys/sys/efi.h
Comment 7 Conor 2018-12-15 12:07:24 UTC
This machine is booting with BIOS. I thought it was EFI, and went ahead with the patch, but on inspection, `machdep.bootmethod: BIOS`. I should've included this in the original report... Sorry about that.
Comment 8 Conrad Meyer freebsd_committer 2018-12-15 17:45:43 UTC
(In reply to Conor from comment #7)
Ok.  If it's possible to test the patch with efi boot (may require setting up an efi partition if you don't already have one), that might be interesting (and fix the issue).
Comment 9 Conor 2018-12-15 18:53:10 UTC
(In reply to Conrad Meyer from comment #8)
I posit that it doesn't fix this issue, per se, as this system is using BIOS, and not EFI. I can't comment on the effect on the patch on EFI systems, but perhaps that requires a separate bug. Is that reasonable?
Comment 10 Conrad Meyer freebsd_committer 2018-12-15 19:51:00 UTC
(In reply to Conor from comment #9)
> I posit that it doesn't fix this issue, per se, as this system is using BIOS,
> and not EFI. I can't comment on the effect on the patch on EFI systems, but
> perhaps that requires a separate bug. Is that reasonable?

Sure, it's totally reasonable.  I used sloppy language when I added "and fix the issue" in the parenthetical.

I was operating under the assumption that the root cause of the hang is a buggy BIOS ACPI spec/table/whatever that doesn't correctly implement S5 (poweroff), as reported by byuu@ on hackers.

But it could also be a bug in the ACPI parser or our implementation.  Notably, acpi_shutdown_final disables interrupts but the AcpiOsSleep implementation invokes pause(), which only works with interrupts enabled.  Probably acpi_shutdown_final <-> AcpiOsSleep should coordinate better.

Either way -- the EFI runtime services poweroff is a workaround for the underlying issue where ACPI shutdown fails.  So in that sense, yes -- the root cause isn't addressed.  But at the end of the day, your system might shutdown nicely.  Which is the goal (I think).  EFI provides other advantages, so you might want to consider it independent of this issue.

--

It sounds to me like you would prefer to keep this bug open to investigate / track the ACPI shutdown issue.  Is that fair?

Things I think we could take further action on:

1. Get an acpidump of some kind.  I'm unfamiliar with the tool and what options might be most useful for us to analyze the tables provided by your BIOS.  You could provide this (start with 'man acpidump' -- you know as much about the tool as I do :-)), or I could look into my own system (I have the same CPU and chipset).  But my system may well have different ACPI tables and has a different BIOS, so it may not have the same issue (ASRock X399 Taichi).

2. Fixing the acpi intr_disable <-> AcpiOsSleep interaction.  This is relatively straightforward and something we should do independent of this bug; I can take care of it.

3. Testing the EFI boot and shutdown patch, if possible, on your system.  If you're not interested in using EFI boot at all, just let me know and I'll stop suggesting it :).
Comment 11 Conor 2018-12-16 00:02:51 UTC
(In reply to Conrad Meyer from comment #10)
That all sounds very reasonable to me; I'm quite happy to keep this bug open and start on some of the investigations you've suggested. I appreciate the thorough explanation of the suspected issue.

I'll gather as much information as I can, attach it to this bug, and get started on debugging the issue further. My cycles will be a bit limited over the coming weeks, but I'd certainly like to take that EFI patch and test it in the event the we don't get to the bottom of the BIOS issue. I'll post again when I've got something from 1) as per your suggestion, and test 3) when I get the chance.

Thanks again.
Comment 12 Conor 2018-12-16 00:23:12 UTC
Created attachment 200139 [details]
Output of acpidump -dtv

I've attached the output of `acpidump -dtv`.
Comment 13 Rocco 2019-01-11 01:33:47 UTC
Hi,

I can confirm the issue can be observed also with my configuration:
FreeBSD: 12.0-RELEASE (also 11.2-RELEASE)
CPU: Threadripper 1950X
Mobo: ASUS X399 Prime

I tried buyy's workaround, shutting down via EFI, and it works. For completeness also this link:

FreeBSD forum thread: https://forums.freebsd.org/threads/acpi-shutdown-not-working-on-amd-x399-threadripper.69065/

+1 for adding the workaround as described by buyy.