Bug 19338

Customer reported an SMP "lock up" issue with "...FreeBSD-4.0 Stable Snapshot (from around 31-5-2000)..." on a Dual 800MHZ processor Proliant DL360.    

Subsequently, we were able to reproduce "the lock" up issue in our lab with FreeBSD 4.0-RELEASE #1 version on a bare bone 667MHZ DL360 with 512MB RAM.  We moved the drive from the DL360 to a DL380, and it booted the SMP kernel without issue.  To verify the DL 360 hardware, we loaded and ran the RedHat 6.2 SMP kernel without issue.

So, our focus moved to the differences between the DL360 & DL380 platforms with respect to SMP & APIC tables.  The only obvious difference between the DL 380 & DL 360 is APIC ID for the BSPs is 1 & 3 respectively with the APs both set to ID 0.

A summary of the main differnce is listed below:

"The package on the [DL360] FC PGA (Pentium III Flat Pack) is different from the [DL380] SECC2 (Pentium III Slot 1).  Intel ... required that the APIC IDs be 3 and 0 for a FCPGA dual processor design.  The hardware controls this, not the ROM."  

Original E-Mail from customer listed below:

> > Matthew,
> > 
> > I have FreeBSD 4.0 running in Uniprocessor mode running on the DL360.  It
> > should not be a big stretch to get it to run SMP. <big grin>   I will get
> > a second processor, and try it as soon as time permits.  My biggest issue
> > was 1510 SCSI driver support.  

We found that in order to get the Symbios SCSI working it was necessary to remove the RAID card.

> > Just curious:
> > 
> > What APIC mode is being used on the DL380 (Full Table or Mapped)?  
> > What Version of FreeBSD is the customer running on the DL380?
> > What SCSI controller is the customer using?

We tried the DL360 in all three possible APIC modes, Full Table, Mapped, and disabled. with no luck.

We were using a FreeBSD-4.0 Stable Snapshot (from around 31-5-2000) as we required a recent version for ida (Compaq RAID array adapter) support, which was not working for the 380 or 360 until the exisiting ida driver was "told" about the addresses etc for these machines by the one of the driver authors.

Snapshots are available from 

ftp://releng4.freebsd.org/pub/FreeBSD/snapshots/i386/

We were (obviously :o) using the ida RAID adapter for disk support.
 
There were no problems getting SMP to work on the 380, as we used the
'mptable' command to find out the correct number of APICs, BUSes, and
INTRupts for compilation into the kernel.

With the 360, it would pause at the point where the second CPU was launched,it would still respond to a <CTRL>-<ALT>-<DEL> key sequence though.

Please contact me if you need any further information.

Thanks,

James

Output from boot -hv DL360
--------------------------
connected
Copyright (c) 1992-2000 The FreeBSD Project.
Copyright (c) 1982, 1986, 1989, 1991, 1993
        The Regents of the University of California. All rights reserved.
FreeBSD 4.0-20000531-STABLE #0: Thu Jun  1 16:03:10 BST 2000
    root@fubar.noc.demon.net:/usr/src/sys/compile/CPQ
Calibrating clock(s) ... TSC clock: 797430926 Hz, i8254 clock: 1193118 Hz
CLK_USE_I8254_CALIBRATION not specified - using default frequency
Timecounter "i8254"  frequency 1193182 Hz
CLK_USE_TSC_CALIBRATION not specified - using old calibration method
CPU: Pentium III/Pentium III Xeon (797.48-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x683  Stepping = 3

Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,XMM>
real memory  = 1207943168 (1179632K bytes)
Physical memory chunk(s):
0x00001000 - 0x0009efff, 647168 bytes (158 pages)
0x002c3000 - 0x47ff3fff, 1205014528 bytes (294193 pages)
avail memory = 1170939904 (1143496K bytes)
Programming 35 pins in IOAPIC #0
IOAPIC #0 intpin 2 -> irq 0
IOAPIC #0 intpin 24 -> irq 2
SMP: CPU0 apic_initialize():
     lint0: 0x00000700 lint1: 0x00010400 TPR: 0x00000010 SVR: 0x000001ff
FreeBSD/SMP: Multiprocessor motherboard
 cpu0 (BSP): apic id:  3, version: 0x00040011, at 0xfee00000
 cpu1 (AP):  apic id:  0, version: 0x00040011, at 0xfee00000
 io0 (APIC): apic id:  8, version: 0x00220011, at 0xfec00000
bios32: Found BIOS32 Service Directory header at 0xc00ffee0
bios32: Entry = 0xf0000 (c00f0000)  Rev = 0  Len = 1
pcibios: PCI BIOS entry at 0x94
Other BIOS signatures found:
ACPI: 000f4f90
Preloaded elf kernel "kernel" at 0xc02a7000.
Pentium Pro MTRR support enabled
SMP: CPU0 bsp_apic_configure():
     lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000010 SVR: 0x000001ff
pci_open(1):    mode 1 addr port (0x0cf8) is 0x00000000
pci_open(1a):   mode1res=0x80000000 (0x80000000)
pci_cfgcheck:   device 0 [class=060000] [hdr=80] is there (id=00091166)
npx0: <math processor> on motherboard
npx0: INT 16 interface
pci_open(1):    mode 1 addr port (0x0cf8) is 0x00000000
pci_open(1a):   mode1res=0x80000000 (0x80000000)
pci_cfgcheck:   device 0 [class=060000] [hdr=80] is there (id=00091166)
pcib0: <RCC LE host to PCI bridge> on motherboard
found-> vendor=0x1166, dev=0x0009, revid=0x05
        class=06-00-00, hdrtype=0x00, mfdev=1
        subordinatebus=0        secondarybus=0
found-> vendor=0x1166, dev=0x0009, revid=0x05
        class=06-00-00, hdrtype=0x00, mfdev=1
        subordinatebus=0        secondarybus=0
Freeing (NOT implemented) redirected PCI irq 5.
found-> vendor=0x1000, dev=0x0010, revid=0x02
        class=01-04-00, hdrtype=0x00, mfdev=0
        subordinatebus=0        secondarybus=0
        intpin=a, irq=19
        map[10]: type 1, range 32, base 00002000, size  8
        map[14]: type 1, range 32, base c5000000, size 24
        map[18]: type 1, range 32, base c4000000, size 24
found-> vendor=0x1002, dev=0x4756, revid=0x7a
        class=03-00-00, hdrtype=0x00, mfdev=0
        subordinatebus=0        secondarybus=0
found-> vendor=0x0e11, dev=0xa0f0, revid=0x00
        class=08-80-00, hdrtype=0x00, mfdev=0
        subordinatebus=0        secondarybus=0
        intpin=a, irq=255
        map[10]: type 1, range 32, base 00001800, size  8
        map[14]: type 1, range 32, base c3ffef00, size  8
found-> vendor=0x8086, dev=0x0960, revid=0x05
        class=06-04-00, hdrtype=0x01, mfdev=1
        subordinatebus=1        secondarybus=1
Freeing (NOT implemented) redirected PCI irq 3.
found-> vendor=0x8086, dev=0x1960, revid=0x05
        class=05-80-00, hdrtype=0x00, mfdev=1
        subordinatebus=0        secondarybus=0
        intpin=a, irq=21
        map[10]: type 1, range 32, base c0fc0000, size 18
found-> vendor=0x1166, dev=0x0200, revid=0x4f
        class=06-01-00, hdrtype=0x00, mfdev=1
        subordinatebus=0        secondarybus=0
found-> vendor=0x1166, dev=0x0211, revid=0x00
        class=01-01-8a, hdrtype=0x00, mfdev=1
        subordinatebus=0        secondarybus=0
        map[20]: type 1, range 32, base 00002800, size  4
pci0: <PCI bus> on pcib0
ida0: <NCR/Compaq Integrated Array controller> port 0x2000-0x20ff mem
0xc4000000-0xc4ffffff,0xc5000000-0xc5ffffff irq 19 at device 1.0 on pci0
ida0: drives=1 firm_rev=1.40
idad0: <Compaq Logical Drive> on ida0
idad0: 8670MB (17756160 sectors), blocksize=512
Creating DISK idad0
pci0: <ATI Mach64-GV graphics accelerator> (vendor=0x1002, dev=0x4756) at
3.0
pci0: <unknown card> (vendor=0x0e11, dev=0xa0f0) at 4.0
pcib1: <PCI to PCI bridge (vendor=8086 device=0960)> at device 5.0 on pci0
found-> vendor=0x1002, dev=0x4756, revid=0x7a
        class=03-00-00, hdrtype=0x00, mfdev=0
        subordinatebus=0        secondarybus=0
        map[10]: type 1, range 32, base c2000000, size 24
        map[14]: type 1, range 32, base 00003000, size  8
        map[18]: type 1, range 32, base c6aff000, size 12
pci1: <PCI bus> on pcib1
pci1: <ATI Mach64-GV graphics accelerator> (vendor=0x1002, dev=0x4756) at
0.0
pci0: <unknown card> (vendor=0x8086, dev=0x1960) at 5.1 irq 21
isab0: <PCI to ISA bridge (vendor=1166 device=0200)> at device 15.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Unknown PCI ATA controller (generic mode)> port 0x2800-0x280f at
device 15.1 on pci0
ata0: iobase=0x01f0 altiobase=0x03f6 bmaddr=0x2800
ata0: mask=03 status0=20 status1=30
ata0: mask=03 status0=20 status1=30
ata0: devices = 0x0
ata0: probe allocation failed
ata1: iobase=0x0170 altiobase=0x0376 bmaddr=0x2808
ata1: mask=03 status0=50 status1=01
ata1: mask=03 status0=00 status1=01
ata1: devices = 0xc
ata1: at 0x170 irq 15 on atapci0
pci_open(1):    mode 1 addr port (0x0cf8) is 0x00000000
pci_open(1a):   mode1res=0x80000000 (0x80000000)
pci_cfgcheck:   device 0 [class=060000] [hdr=80] is there (id=00091166)
pcib3: <RCC LE host to PCI bridge> on motherboard
Freeing (NOT implemented) redirected PCI irq 7.
found-> vendor=0x8086, dev=0x1229, revid=0x08
        class=02-00-00, hdrtype=0x00, mfdev=1
        subordinatebus=0        secondarybus=0
        intpin=a, irq=17
        map[10]: type 1, range 32, base c6fff000, size 12
        map[14]: type 1, range 32, base 00004000, size  6
        map[18]: type 1, range 32, base c6e00000, size 20
Freeing (NOT implemented) redirected PCI irq 7.
found-> vendor=0x0000, dev=0x0000, revid=0x00
        class=07-00-00, hdrtype=0x00, mfdev=1
        subordinatebus=0        secondarybus=0
        intpin=a, irq=17
        map[10]: type 1, range 32, base 00004040, size  3
        map[14]: type 1, range 32, base c6dff000, size 12
Freeing (NOT implemented) redirected PCI irq 10.
found-> vendor=0x8086, dev=0x1229, revid=0x08
        class=02-00-00, hdrtype=0x00, mfdev=1
        subordinatebus=0        secondarybus=0
        intpin=a, irq=2
        map[10]: type 1, range 32, base c6dfe000, size 12
        map[14]: type 1, range 32, base 00004080, size  6
        map[18]: type 1, range 32, base c6c00000, size 20
Freeing (NOT implemented) redirected PCI irq 10.
found-> vendor=0x0000, dev=0x0000, revid=0x00
        class=07-00-00, hdrtype=0x00, mfdev=1
        subordinatebus=0        secondarybus=0
        intpin=a, irq=2
        map[10]: type 1, range 32, base 000040c0, size  3
        map[14]: type 1, range 32, base c6bff000, size 12
pci3: <PCI bus> on pcib3
fxp0: <Intel EtherExpress Pro 10/100B Ethernet> port 0x4000-0x403f mem
0xc6e00000-0xc6efffff,0xc6fff000-0xc6ffffff irq 17 at device 4.0 on pci3
fxp0: Ethernet address 00:01:fa:ff:f4:f3
bpf: fxp0 attached
pci3: <unknown card> (vendor=0x0000, dev=0x0000) at 4.1 irq 17
fxp1: <Intel EtherExpress Pro 10/100B Ethernet> port 0x4080-0x40bf mem
0xc6c00000-0xc6cfffff,0xc6dfe000-0xc6dfefff irq 2 at device 5.0 on pci3
fxp1: Ethernet address 00:01:fa:ff:f5:05
bpf: fxp1 attached
pci3: <unknown card> (vendor=0x0000, dev=0x0000) at 5.1 irq 2
eisa0: <EISA bus> on motherboard
mainboard0: <CPQ0685 (System Board)> on eisa0 slot 0
Trying Read_Port at 203
Trying Read_Port at 243
Trying Read_Port at 283
Trying Read_Port at 2c3
Trying Read_Port at 303
Trying Read_Port at 343
Trying Read_Port at 383
Trying Read_Port at 3c3
ata-: ata0 exists, using next available unit number
ata-: ata1 exists, using next available unit number
isa_probe_children: disabling PnP devices
isa_probe_children: probing non-PnP devices
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ata2: iobase=0x01f0 altiobase=0x03f6 bmaddr=0x0000
ata2: mask=03 status0=20 status1=30
ata2: mask=03 status0=20 status1=30
ata2: devices = 0x0
ata2: probe allocation failed
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
atkbd: the current kbd controller command byte 0065
atkbd: keyboard ID 0x41ab (2)
kbdc: RESET_KBD return code:00fa
kbdc: RESET_KBD status:00aa
kbd0: atkbd0, AT 101/102 (2), config:0x0, flags:0x1d0000
psm0: current command byte:0065
kbdc: TEST_AUX_PORT status:0000
kbdc: RESET_AUX return code:00fa
kbdc: RESET_AUX status:00aa
kbdc: RESET_AUX ID:0000
psm: status 00 02 64
psm: status 00 00 64
psm: status 00 03 64
psm: status 00 03 64
psm: data 08 00 00
psm: data 08 00 00
psm: status 00 02 64
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: model IntelliMouse, device ID 3-00, 3 buttons
psm0: config:00000000, flags:00000000, packet size:4
psm0: syncmask:08, syncbits:08
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
fb0: vga0, vga, type:VGA (5), flags:0x7007f
fb0: port:0x3c0-0x3df, crtc:0x3d4, mem:0xa0000 0x20000
fb0: init mode:24, bios mode:3, current mode:24
fb0: window:0xc00b8000 size:32k gran:32k, buf:0 size:32k
VGA parameters upon power-up
50 18 10 00 00 00 03 00 02 67 5f 4f 50 82 55 81 
bf 1f 00 4f 0d 0e 00 00 05 50 9c 8e 8f 28 1f 96 
b9 a3 ff 00 01 02 03 04 05 14 07 38 39 3a 3b 3c 
3d 3e 3f 0c 00 0f 08 00 00 00 00 00 10 0e 00 ff 
VGA parameters in BIOS for mode 24
50 18 10 00 10 00 03 00 02 67 5f 4f 50 82 55 81 
bf 1f 00 4f 0d 0e 00 00 00 00 9c 8e 8f 28 1f 96 
b9 a3 ff 00 01 02 03 04 05 14 07 38 39 3a 3b 3c 
3d 3e 3f 0c 00 0f 08 00 00 00 00 00 10 0e 00 ff 
EGA/VGA parameters to be used for mode 24
50 18 10 00 10 00 03 00 02 67 5f 4f 50 82 55 81 
bf 1f 00 4f 0d 0e 00 00 00 00 9c 8e 8f 28 1f 96 
b9 a3 ff 00 01 02 03 04 05 14 07 38 39 3a 3b 3c 
3d 3e 3f 0c 00 0f 08 00 00 00 00 00 10 0e 00 ff 
sc0: <System console> on isa0
sc0: VGA <16 virtual consoles, flags=0x0>
sc0: fb0, kbd0, terminal emulator: sc (syscons terminal)
sio0: irq maps: 0x8041 0x8051 0x8041 0x8041
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A, console
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: irq maps: 0x8041 0x8041 0x8041 0x8041
sio1: probe failed test(s): 0 1 2 4 6 7 9
ppc0: parallel port not found.
isa_probe_children: probing PnP devices
SMP: enabled INTs: 1, 2, 4, 6, 12, 15, 17, 19, apic_imen: 0x00f56fa9
BIOS Geometries:
 0:03fffe20 0..1023=1024 cylinders, 0..254=255 heads, 1..32=32 sectors
 0 accounted for
Device configuration finished.
APIC_IO: routing 8254 via IOAPIC #0 intpin 2
bpf: lo0 attached
SMP: AP CPU #1 Launched!
SMP: CPU1 apic_initialize():
     lint0: 0x00010700 lint1: 0x00010400 TPR: 0x00000010 SVR: 0x000001ff

A three fingered salute then adds the following

boot() called on cpu#1
Uptime: 0s
Rebooting...
cpu_reset called on cpu#1
cpu_reset: Stopping other CPUs
cpu_reset: Restarting BSP
 pu_reset_proxy: Grabbed mp locck uf_orr sBeSP:
BSP did not grab mp lock

---------------------------------



Output from mptable when booting from the kernel.GENERIC DL360

===============================================================================

MPTable, version 2.0.15

-------------------------------------------------------------------------------

MP Floating Pointer Structure:

  location:                     BIOS
  physical address:             0x000f4ff0
  signature:                    '_MP_'
  length:                       16 bytes
  version:                      1.4
  checksum:                     0x9f
  mode:                         Virtual Wire

-------------------------------------------------------------------------------

MP Config Table Header:

  physical address:             0x000f29c9
  signature:                    'PCMP'
  base table length:            324
  version:                      1.4
  checksum:                     0x88
  OEM ID:                       'COMPAQ  '
  Product ID:                   'PROLIANT    '
  OEM table pointer:            0x00000000
  OEM table size:               0
  entry count:                  28
  local APIC address:           0xfee00000
  extended table length:        184
  extended table checksum:      82

-------------------------------------------------------------------------------

MP Config Base Table Entries:

--
Processors:     APIC ID Version State           Family  Model   Step
Flags
                 3       0x10    BSP, usable     6       8       3
0x383fbff
                 0       0x10    AP, usable      6       8       3
0x383fbff
--
Bus:            Bus ID  Type
                 0       PCI   
                 3       PCI   
                 9       ISA   
--
I/O APICs:      APIC ID Version State           Address
                 8       0x11    usable          0xfec00000
--
I/O Ints:       Type    Polarity    Trigger     Bus ID   IRQ    APIC ID PIN#
                INT     active-lo       level        0   5:A          8   21
                INT     active-lo       level        0   5:B          8   20
                INT     active-lo       level        0   5:C          8   21
                INT     active-lo       level        0   5:D          8   20
                INT     active-lo       level        3   6:A          8   23
                INT     active-lo       level        3   6:B          8   22
                INT     active-lo       level        3   6:C          8   23
                INT     active-lo       level        3   6:D          8   22
                INT     active-lo       level        0   1:A          8   19
                INT     active-lo       level        0   1:B          8   18
                INT     active-lo       level        3   4:A          8   17
                INT     active-lo       level        3   5:A          8   24
                INT     active-hi        edge        9     1          8    1
                INT     active-hi        edge        9     0          8    2
                INT     active-hi        edge        9     4          8    4
                INT     active-hi        edge        9     6          8    6
                INT     active-hi        edge        9     8          8    8
                INT     active-hi        edge        9     9          8    9
                INT     active-hi        edge        9    11          8   11
                INT     active-hi        edge        9    12          8   12
                INT     active-lo       level        9    13          8   13
                INT     active-hi        edge        9    14          8   14
                INT     active-hi        edge        9    15          8   15
--
Local Ints:     Type    Polarity    Trigger     Bus ID   IRQ    APIC ID PIN#
                ExtINT   conforms    conforms        9     0        255    0
                NMI      conforms    conforms        9     0        255    1

-------------------------------------------------------------------------------

MP Config Extended Table Entries:

Extended Table HOSED!





-- 
James Guard						 System Administrator
Interactive Services				   Demon Internet, Demon@THUS
http://www.games.demon.net/				http://www.demon.net/

Fix: 

Possibly:

FreeBSD 4.0 might "assume" that APIC ID's be sequential in ascending order.  This is not a requirement per the Intel MP spec.  In fact, in a current ProLiant MP boxes (with 3 or more cpu's), if a processor were to fail, the ROM would map it out, and the system would still try to boot with non-sequential APIC ID's (Compaq Hot Spare Boot CPU Patent). If this is the problem, then it may not be limited to FC PGA designs.

Please contact me if you need more information.
How-To-Repeat: Configure an FreeBSD 4.0 RELEASE #1 SMP Kernel on a ProLiant DL360 with two FC PGA Pentium III processors configured for the Linux operating system, and attempt to boot it.