Bug 236333

Summary: ThunderX2 fails to boot after r343856 without options NUMA
Product: Base System Reporter: Ed Maste <emaste>
Component: armAssignee: freebsd-arm (Nobody) <freebsd-arm>
Status: Closed FIXED    
Severity: Affects Some People CC: Andrew
Priority: --- Keywords: regression
Version: CURRENT   
Hardware: arm64   
OS: Any   

Description Ed Maste freebsd_committer freebsd_triage 2019-03-06 18:20:41 UTC
Bisected to r343856, boot fails like so:

WARNING: WITNESS option enabled, expect reduced performance.^M
uhub1: 2 ports with 2 removable, self powered^M
uhub0: 2 ports with 2 removable, self powered^M
Root mount waiting for: usbus1 usbus0^M
Root mount waiting for: usbus1 usbus0^M
Root mount waiting for: usbus1 usbus0^M
Root mount waiting for: usbus1 usbus0^M
Root mount waiting for: usbus1 usbus0^M
Root mount waiting for: usbus1 usbus0^M
Root mount waiting for: usbus1 usbus0^M
Root mount waiting for: usbus1 usbus0^M
Root mount waiting for: usbus1 usbus0^M
Root mount waiting for: usbus1 usbus0^M
usbd_setup_device_desc: getting device descriptor at addr 1 failed, USB_ERR_TIMEOUT^M
usbd_setup_device_desc: getting device descriptor at addr 1 failed, USB_ERR_TIMEOUT^M
Root mount waiting for: usbus1 usbus0^M
Root mount waiting for: usbus1 usbus0^M
...

Bisect log:
5053141bcbea708eda40c20cc60a120f3a4c87f0 is the first bad commit
commit 5053141bcbea708eda40c20cc60a120f3a4c87f0
Author: jchandra <jchandra@FreeBSD.org>
Date:   Thu Feb 7 03:01:54 2019 +0000

    arm64 gicv3: add IORT and NUMA support
    
    acpi_iort.c has added support to query GIC proximity and MSI XREF
    ID for GIC ITS blocks. Use this when GIC ITS blocks are initialized
    from ACPI.
    
    Reviewed by:    andrew
    Differential Revision:  https://reviews.freebsd.org/D18003

:040000 040000 5bbbbc6dd15bad4c55d046e0d3b94079c7295370 e19676bca63f2c01d0f6d9aa427f63e5b7f4edb5 M      sys
$ git bisect log
git bisect start
# bad: [052f129679d4a3d705f9169a70dd53897840473a] Merge remote-tracking branch 'upstream/master' into wipbsd.20181109
git bisect bad 052f129679d4a3d705f9169a70dd53897840473a
# good: [ddd1f1d5f46cda5043027446abd96bcb0b740c84] Revert "Enable PIE by default in WIPBSD"
git bisect good ddd1f1d5f46cda5043027446abd96bcb0b740c84
# good: [73bcd05fc583be759e391941ac24918430ed021c] Fixes for very early use of the pthread_mutex_* and libthr malloc.
git bisect good 73bcd05fc583be759e391941ac24918430ed021c
# bad: [4b8675d5ed97289ca72267451f1f2d67631ad30f] Fix a slew of mdoc errors and warnings.
git bisect bad 4b8675d5ed97289ca72267451f1f2d67631ad30f
# bad: [fea91ac6136ea551592ea77b5a55ecf16e8036b6] Implement pmap_clear_modify() for RISC-V.
git bisect bad fea91ac6136ea551592ea77b5a55ecf16e8036b6
# bad: [c124a4991351b044aa1412e5a8ce022617843d44] Remove -R option which was added to sysctl(8) man page per r244106, but it is not implemented.
git bisect bad c124a4991351b044aa1412e5a8ce022617843d44
# bad: [5a66a899d51ffdc2ef3c5d846ff6b7c947dbe2af] pci_host_generic_acpi: use IORT data for MSI/MSI-X
git bisect bad 5a66a899d51ffdc2ef3c5d846ff6b7c947dbe2af
# good: [c558d0f965f55742092089bf56239b5169e23397] Add quirk for Sansisk X400 drives
git bisect good c558d0f965f55742092089bf56239b5169e23397
# good: [5b96e3b72dd52555255844e6c307c66b0a6ce38c] riscv: default to non-executable stack
git bisect good 5b96e3b72dd52555255844e6c307c66b0a6ce38c
# good: [47b24738b81eb479e169fe248afda102731a5207] arm64 acpi: Add support for IORT table
git bisect good 47b24738b81eb479e169fe248afda102731a5207
# bad: [e502a5bda0075bd48250cb86d3452bd0ec84b075] Report cache zones in UMA stats sysctl, that 'vmstat -z' uses. This should had been part of r251826.
git bisect bad e502a5bda0075bd48250cb86d3452bd0ec84b075
# bad: [5053141bcbea708eda40c20cc60a120f3a4c87f0] arm64 gicv3: add IORT and NUMA support
git bisect bad 5053141bcbea708eda40c20cc60a120f3a4c87f0
# good: [37387406493e76b36d7ef77b986894db4416502e] Use ifunc to select the barrier instruction for RDTSC.
git bisect good 37387406493e76b36d7ef77b986894db4416502e
# first bad commit: [5053141bcbea708eda40c20cc60a120f3a4c87f0] arm64 gicv3: add IORT and NUMA support
Comment 1 Ed Maste freebsd_committer freebsd_triage 2019-03-06 18:38:33 UTC
Confirmed, r344843 with r343856 reverted boots fine.
Comment 2 Andrew Turner freebsd_committer freebsd_triage 2019-03-06 18:38:57 UTC
Can you print the return value of acpi_iort_its_lookup and the values in di->di_gic_dinfo.gic_domain and di->di_gic_dinfo.msi_xref in gic_v3_add_children in gic_v3_acpi.c.

I'm guessing there's a conflict with the xref between the GIC ITS drivers.
Comment 3 Ed Maste freebsd_committer freebsd_triage 2019-03-06 19:13:47 UTC
Adding printfs in gic_v3_add_children I see entry->Type = 11, 12, 14 - never 15 (ACPI_MADT_TYPE_GENERIC_TRANSLATOR).

Note it so happens I am testing plain GENERIC. NUMA was not enabled.
Comment 4 Ed Maste freebsd_committer freebsd_triage 2019-03-06 19:26:46 UTC
This is on the 7.2 firmware
Comment 5 Ed Maste freebsd_committer freebsd_triage 2019-03-06 21:52:38 UTC
This was tested with plain GENERIC, i.e., without options NUMA. It works when options NUMA is added to the kernel config.
Comment 7 Ed Maste freebsd_committer freebsd_triage 2021-01-11 18:29:43 UTC
Referenced changes have been committed