Bug 210106
| Summary: | Current won't boot on PPC64 Power8 with sparse CPU IDs | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Base System | Reporter: | Luciano Mannucci <luciano> | ||||
| Component: | kern | Assignee: | Nathan Whitehorn <nwhitehorn> | ||||
| Status: | Closed FIXED | ||||||
| Severity: | Affects Some People | CC: | nwhitehorn, swills | ||||
| Priority: | Normal | ||||||
| Version: | CURRENT | ||||||
| Hardware: | Any | ||||||
| OS: | Any | ||||||
| Attachments: |
|
||||||
This is a result of disabling threads so that the CPU IDs seen by the system are non-consecutive. As a workaround, you can set threads=8 in your KVM configuration while we fix the underlying problem. Created attachment 171994 [details]
Patch to fix bug
I've just sent the attached patch to freebsd-hackers@ for review. It fixes this particular problem and hopefully will land in the tree before 11.0-BETA1.
Thank you very much for the report.
A commit references this bug: Author: nwhitehorn Date: Wed Jul 6 14:09:51 UTC 2016 New revision: 302372 URL: https://svnweb.freebsd.org/changeset/base/302372 Log: Replace a number of conflations of mp_ncpus and mp_maxid with either mp_maxid or CPU_FOREACH() as appropriate. This fixes a number of places in the kernel that assumed CPU IDs are dense in [0, mp_ncpus) and would try, for example, to run tasks on CPUs that did not exist or to allocate too few buffers on systems with sparse CPU IDs in which there are holes in the range and mp_maxid > mp_ncpus. Such circumstances generally occur on systems with SMT, but on which SMT is disabled. This patch restores system operation at least on POWER8 systems configured in this way. There are a number of other places in the kernel with potential problems in these situations, but where sparse CPU IDs are not currently known to occur, mostly in the ARM machine-dependent code. These will be fixed in a follow-up commit after the stable/11 branch. PR: kern/210106 Reviewed by: jhb Approved by: re (glebius) Changes: head/sys/amd64/include/counter.h head/sys/cddl/compat/opensolaris/sys/proc.h head/sys/dev/cpuctl/cpuctl.c head/sys/i386/include/counter.h head/sys/kern/subr_pcpu.c head/sys/kern/subr_taskqueue.c head/sys/net/flowtable.c head/sys/net/iflib.c head/sys/netinet/ip_id.c head/sys/powerpc/include/counter.h head/sys/powerpc/powerpc/mp_machdep.c head/sys/vm/uma.h head/sys/vm/uma_core.c Fixed for BETA1. Thank you for the report! A commit references this bug: Author: nwhitehorn Date: Wed Aug 31 04:02:53 UTC 2016 New revision: 305108 URL: https://svnweb.freebsd.org/changeset/base/305108 Log: Refix operation on sparse CPU mappings as in r302372, temporarily broken by r304716. PR: kern/210106 MFC after: 2 days Changes: head/sys/kern/imgact_elf.c head/sys/kern/subr_gtaskqueue.c A commit references this bug: Author: nwhitehorn Date: Thu Sep 1 22:27:48 UTC 2016 New revision: 305250 URL: https://svnweb.freebsd.org/changeset/base/305250 Log: MFC r305108,305109: Refix operation on sparse CPU mappings as in r302372, temporarily broken by r304716. PR: kern/210106 Changes: _U stable/11/ stable/11/sys/kern/subr_gtaskqueue.c A commit references this bug: Author: nwhitehorn Date: Fri Sep 2 01:41:57 UTC 2016 New revision: 305267 URL: https://svnweb.freebsd.org/changeset/base/305267 Log: MFS11 r305250: MFC r305108,305109: Refix operation on sparse CPU mappings as in r302372, temporarily broken by r304716. PR: kern/210106 Approved by: re (gjb) Changes: _U releng/11.0/ releng/11.0/sys/kern/subr_gtaskqueue.c |
The system hangs during the boot: OK boot -v Booting... Kernel entry at 0x1024f0 ... KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2016 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-ALPHA2 #0 r301230: Fri Jun 3 07:51:47 UTC 2016 root@releng2.nyi.freebsd.org:/usr/obj/powerpc.powerpc64/usr/src/sys/GENERIC64 powerpc gcc version 4.2.1 20070831 patched [FreeBSD] WARNING: WITNESS option enabled, expect reduced performance. VT: init without driver. Preloaded elf kernel "/boot/kernel/kernel" at 0x1656000. cpu0: IBM POWER8E revision 2.1, 3425.00 MHz cpu0: Features dc005180<PPC32,PPC64,ALTIVEC,FPU,MMU,SMT,ARCH205,ARCH206,VSX> cpu0: Features2 c2000000<ARCH207,HTM,VCRYPTO> real memory = 2110451712 (2012 MB) available KVA = 7650267135 (7295 MB) Physical memory chunk(s): 0x0000000000025000 - 0x00000000000fffff, 897024 bytes (219 pages) 0x0000000001c40000 - 0x0000000001c3ffff, 0 bytes (0 pages) 0x0000000003a49000 - 0x000000007ae67fff, 2000809984 bytes (488479 pages) 0x000000007ffa0000 - 0x000000007ffaffff, 65536 bytes (16 pages) 0x000000007ffb5000 - 0x000000007ffeffff, 241664 bytes (59 pages) 0x000000007fff1000 - 0x000000007fffffff, 61440 bytes (15 pages) 0x0000000001658000 - 0x0000000001bfffff, 5931008 bytes (1448 pages) avail memory = 1989722112 (1897 MB) FreeBSD/SMP: Multiprocessor System Detected: 6 CPUs cpu0: dev=7e5e00b8 (BSP) ULE: setup cpu 0 random: entropy device external interface snd_unit_init() u=0x00ff8000 [512] d=0x00007c00 [32] c=0x000003ff [1024] feeder_register: snd_unit=-1 snd_maxautovchans=16 latency=5 feeder_rate_min=1 feeder_rate_max=2016000 feeder_rate_round=25 firmware: 'isp_1040' version 1: 22944 bytes loaded at 0xc54204 firmware: 'isp_1080' version 1: 31350 bytes loaded at 0xc59ba4 firmware: 'isp_12160' version 1: 28050 bytes loaded at 0xc6161a firmware: 'isp_2100' version 1: 76770 bytes loaded at 0xc683ac firmware: 'isp_2200' version 1: 84566 bytes loaded at 0xc7af8e firmware: 'isp_2300' version 1: 125252 bytes loaded at 0xc8f9e4 firmware: 'isp_2322' version 1: 120814 bytes loaded at 0xcae328 firmware: 'isp_2400' version 1: 204384 bytes loaded at 0xccf690 firmware: 'isp_2500' version 1: 223908 bytes loaded at 0xd0f0c0 kbd0 at kbdmux0 mem: <memory> nfslock: pseudo-device null: <full device, null device, zero device> openfirm: <Open Firmware control device> random: harvesting attach, 8 bytes (4 bits) from nexus0 ofwbus0: <Open Firmware Device Tree> on nexus0 random: harvesting attach, 8 bytes (4 bits) from ofwbus0 xicp0: <PAPR virtual interrupt controller> on ofwbus0 random: harvesting attach, 8 bytes (4 bits) from xicp0 cpulist0: <Open Firmware CPU Group> on ofwbus0 cpu0: <Open Firmware CPU> on cpulist0 random: harvesting attach, 8 bytes (4 bits) from cpu0 cpu1: <Open Firmware CPU> on cpulist0 random: harvesting attach, 8 bytes (4 bits) from cpu1 cpu2: <Open Firmware CPU> on cpulist0 random: harvesting attach, 8 bytes (4 bits) from cpu2 cpu3: <Open Firmware CPU> on cpulist0 random: harvesting attach, 8 bytes (4 bits) from cpu3 cpu4: <Open Firmware CPU> on cpulist0 random: harvesting attach, 8 bytes (4 bits) from cpu4 cpu5: <Open Firmware CPU> on cpulist0 random: harvesting attach, 8 bytes (4 bits) from cpu5 random: harvesting attach, 8 bytes (4 bits) from cpulist0 pcib0: <RTAS Host-PCI bridge> on ofwbus0 pci0: <POWER Hypervisor PCI bus> on pcib0 pci0: domain=0, physical bus=0 found-> vendor=0x1af4, dev=0x1002, revid=0x00 domain=0, bus=0, slot=3, func=0 class=00-ff-00, hdrtype=0x00, mfdev=0 cmdreg=0x0100, statreg=0x0000, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=0 map[10]: type I/O Port, range 32, base rx20, size 5, port disabled found-> vendor=0x106b, dev=0x003f, revid=0x00 domain=0, bus=0, slot=2, func=0 class=0c-03-10, hdrtype=0x00, mfdev=0 cmdreg=0x0106, statreg=0x0000, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=0 map[10]: type Memory, range 32, base rxc0000000, size 8, enabled found-> vendor=0x10ec, dev=0x8139, revid=0x20 domain=0, bus=0, slot=1, func=0 class=02-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0100, statreg=0x0000, cachelnsz=0 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=0 map[10]: type I/O Port, range 32, base rx100, size 8, port disabled map[14]: type Memory, range 32, base rxc0000100, size 8, memory disabled pci0: <old> at device 3.0 (no driver attached) ohci0: <Apple KeyLargo/Intrepid USB controller> mem 0xc0000000-0xc00000ff irq 4100 at device 2.0 on pci0 ohci0: Mapping IOMMU domain 0x80000000 ofw_pci mapdev: start 100e0000000, len 256 usbus0 on ohci0 ohci0: usbpf: Attached random: harvesting attach, 8 bytes (4 bits) from usbus0 random: harvesting attach, 8 bytes (4 bits) from ohci0 re0: <RealTek 8139C+ 10/100BaseTX> port 0x100-0x1ff mem 0xc0000100-0xc00001ff irq 4099 at device 1.0 on pci0 ofw_pci mapdev: start 100e0000100, len 256 re0: MSI count : 0 re0: MSI-X count : 0 re0: Chip rev. 0x74800000 re0: MAC rev. 0x00000000 re0: Mapping IOMMU domain 0x80000000 miibus0: <MII bus> on re0 rlphy0: <RealTek internal media interface> PHY 0 on miibus0 rlphy0: OUI 0x000000, model 0x0000, rev. 0 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow random: harvesting attach, 8 bytes (4 bits) from rlphy0 random: harvesting attach, 8 bytes (4 bits) from miibus0 re0: Using defaults for TSO: 65518/35/2048 re0: bpf attached re0: Ethernet address: 52:54:00:1a:cd:48 random: harvesting attach, 8 bytes (4 bits) from re0 random: harvesting attach, 8 bytes (4 bits) from pci0 random: harvesting attach, 8 bytes (4 bits) from pcib0 rtas0: <Run-Time Abstraction Services> on ofwbus0 rtas0: registered as a time-of-day clock (resolution 2000us, adjustment 0.001000000s) random: harvesting attach, 8 bytes (4 bits) from rtas0 vdevice0: <POWER Hypervisor Virtual Device Root> on ofwbus0 vscsi0: <POWER Hypervisor Virtual SCSI Bus> irq 16781319 on vdevice0 vscsi0: Mapping IOMMU domain 0x2000 vscsi0: Queue depth 22 commands random: harvesting attach, 8 bytes (4 bits) from vscsi0 uart0: <POWER Hypervisor Virtual Serial Port> irq 16781320 on vdevice0 random: harvesting attach, 8 bytes (4 bits) from uart0 random: harvesting attach, 8 bytes (4 bits) from vdevice0 ofwbus0: <hypervisor> compat linux,kvm (no driver attached) procfs registered Timecounter "timebase" frequency 512000000 Hz quality 0 Event timer "decrementer" frequency 512000000 Hz quality 1000 Timecounters tick every 1.000 msec vlan: initialized, using hash tables with chaining tcp_init: net.inet.tcp.tcbhashsize auto tuned to 16384 lo0: bpf attached usbus0: 12Mbps Full Speed USB v1.0 (probe0:vscsi0:0:0:0): Down reving Protocol Version from 6 to 5? ugen0.1: <Apple> at usbus0 uhub0: <Apple OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 (probe0:vscsi0:0:0:2): Down reving Protocol Version from 6 to 5? (probe0:vscsi0:0:0:1): Down reving Protocol Version from 6 to 5? pass0 at vscsi0 bus 0 scbus0 target 0 lun 0 pass0: <QEMU QEMU TARGET 2.3> Fixed Uninstalled SPC-3 SCSI device (offline) pass0: 150.000MB/s transfers pass1 at vscsi0 bus 0 scbus0 target 0 lun 1 pass1: <QEMU QEMU CD-ROM 2.3.> Removable CD-ROM SPC-3 SCSI device pass1: 150.000MB/s transfers pass2 at vscsi0 bus 0 scbus0 target 0 lun 2 pass2: <QEMU QEMU HARDDISK 2.3.> Fixed Direct Access SPC-3 SCSI device pass2: 150.000MB/s transfers cd0 at vscsi0 bus 0 scbus0 target 0 lun 1 cd0: <QEMU QEMU CD-ROM 2.3.> Removable CD-ROM SPC-3 SCSI device cd0: 150.000MB/s transfers cd0: 695MB (356002 2048 byte sectors) da0 at vscsi0 bus 0 scbus0 target 0 lun 2 da0: <QEMU QEMU HARDDISK 2.3.> Fixed Direct Access SPC-3 SCSI device da0: 150.000MB/s transfers da0: 20480MB (41943040 512 byte sectors) Adding CPU 0, pir=28, awake=1 Waking up CPU 8 (dev=7e5e08d8) Adding CPU 8, pir=20, awake=1 Waking up CPU 16 (dev=7e5e10f8) Adding CPU 16, pir=20, awake=1 Waking up CPU 24 (dev=7e5e1918) Adding CPU 24, pir=20, awake=1 Waking up CPU 32 (dev=7e5e2138) Adding CPU 32, pir=20, awake=1 Waking up CPU 40 (dev=7e5e2958) Adding CPU 40, pir=20, awake=1 SMP: AP CPU #8 launched SMP: AP CPU #32 launched SMP: AP CPU #16 launched SMP: AP CPU #24 launched SMP: AP CPU #40 launched and freezes with the allocated cpus at 100%