Bug 248707 - Dell R630 - X2Apic enabled fails boot
Summary: Dell R630 - X2Apic enabled fails boot
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.1-STABLE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-08-17 18:12 UTC by unspam
Modified: 2021-01-29 22:00 UTC (History)
2 users (show)

See Also:


Attachments
Error1 (157.58 KB, image/jpeg)
2020-08-17 18:12 UTC, unspam
no flags Details
Another Error (196.98 KB, image/jpeg)
2020-08-17 18:14 UTC, unspam
no flags Details
HardenedBSD error (157.20 KB, image/jpeg)
2020-08-17 18:15 UTC, unspam
no flags Details
verbose boot output from Dell R730xd with X2APIC enabled FreeBSD 11.4 (224.06 KB, text/plain)
2021-01-29 18:57 UTC, Peter Eriksson
no flags Details
Verbose boot output from FreeBSD 14 (2020-01-29) (208.61 KB, text/plain)
2021-01-29 22:00 UTC, Peter Eriksson
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description unspam 2020-08-17 18:12:23 UTC
Created attachment 217285 [details]
Error1

Processor: E5-2650L V3
No Raidcard installed
BIOS: UEFI Mode
Firmware: all up to date
Version FreeBSD 12.1

Issue:
When trying to install FreeBSD12.1 with X2APIC enabled in BIOS, the kernel gets in a mess and hangs

With X2Apic turned off the OS boots just fine. However with X2APIC turned off I loose Hyper Threading abilities.

I've tried the following tunables.

kern.cam.boot_delay="10000"
vfs.root_mount_always_wait=1

Neither work.

Misc:
Debian boots fine

OmniOS fails with: 
No SOF interrupts have been received this USB ehci host-controller is unusable
Comment 1 unspam 2020-08-17 18:14:34 UTC
Created attachment 217286 [details]
Another Error
Comment 2 unspam 2020-08-17 18:15:00 UTC
Created attachment 217287 [details]
HardenedBSD error
Comment 3 unspam 2020-08-17 20:53:13 UTC
(In reply to unspam from comment #0)
Second R630 also has the same issue. 
All firmware upgraded, no raid
Processor: E5-2660 v3
Comment 4 unspam 2020-08-18 00:36:24 UTC
(In reply to unspam from comment #0)

Both Server 1 & 2 are reporting the same error.

I have tried the following tunables:

kern.cam.boot_delay="10000"
vfs.root_mount_always_wait=1
hw.usb.xhci.use_polling=1 

and can report that with X2APIC enabled no BSD variant will boot.

NetBSD, OpenBSD, FreeBSD and even OnmiOS all bail.
Linux however boots just fine.
Comment 5 Mark Johnston freebsd_committer 2020-08-28 13:07:09 UTC
Could you show the full bootup log please?
Comment 6 unspam 2020-08-30 14:06:00 UTC
(In reply to Mark Johnston from comment #5)
Hi Mark,

Not easily - the machine is unable to mount the root file system from
either USB or SATA, so I don't get a shell to provide a full bootlog. When it reaches the issue it: reboots, hangs and repeats. 

The attachments in this ticket are the best I can get via iDRAC virtual console. However I can also screen record the whole bootup if that would be useful. 

I can try getting non-maskable interrupt enabled on the server and I am more then happy to hand over credentials to the remote idrac if this would help too.
Comment 7 Mark Johnston freebsd_committer 2020-12-23 19:31:05 UTC
(In reply to unspam from comment #6)
Sorry for the delayed reply.  Presumably this is still a problem?  A recording of the bootup would be better than nothing.  Please try it with verbose messages enabled, i.e., boot -v at the loader prompt.
Comment 8 Peter Eriksson 2021-01-29 18:57:11 UTC
Created attachment 222012 [details]
verbose boot output from Dell R730xd with X2APIC enabled FreeBSD 11.4

Please find attached verbose boot output from a Dell R730xd (basically the same motherboard as the Dell R630) that also hangs when booting with X2APIC enabled.
Comment 9 Peter Eriksson 2021-01-29 22:00:38 UTC
Created attachment 222014 [details]
Verbose boot output from FreeBSD 14 (2020-01-29)

For comparison (also hangs if X2APIC is enabled) here is the verbose boot output from a FreeBSD current kernel as of today.

The server has updated BIOS & firmwares.