Bug 31535
| Summary: | Can't reboot system: Tyan Thunder K7+ Dual AMD Athlon 1.2 MP / FreeBSD 4.4 STABLE or 4.3 RELEASE | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Oleg Ilin <ilin> |
| Component: | i386 | Assignee: | freebsd-bugs (Nobody) <bugs> |
| Status: | Closed FIXED | ||
| Severity: | Affects Only Me | ||
| Priority: | Normal | ||
| Version: | Unspecified | ||
| Hardware: | Any | ||
| OS: | Any | ||
|
Description
Oleg Ilin
2001-10-27 16:40:00 UTC
Hello freebsd-gnats-submit, Info: #=========================================================================================================> Both GENERIC-based and custom-based kernels with SMP support do reboot in a little uptime (such as several minutes); however, after some work such as kernel build reboot requests lead to hang, sometimes with 'proxy_...' message, sometimes without it. #=========================================================================================================< last uname -a: #=========================================================================================================> test# uname -a FreeBSD test.seagull.ru 4.4-STABLE FreeBSD 4.4-STABLE #0: Sun Oct 28 16:38:52 MSK 2001 root@test.seagull.ru:/usr/src/sys/compile/TEST2 i386 #=========================================================================================================< Hardware info : #=========================================================================================================> Server : Appro 1124s [http://www.appro.com/1124s.html] Motherboard : Tyan Thunder K7 Processors : Two AMD® 1.2GHz Athlon MP processors HDD : 1 IBM DDYS-T36950M Memory : 2x 512 Mb Reg ECC (Infinion) #=========================================================================================================< Bios info: #=========================================================================================================> Memory ECC : Enable POWER : ACPI Disable SMP ver : 1.4 #=========================================================================================================< boot log [console]: #=========================================================================================================> Adaptec SCSI BIOS v3.10 (c) 2001 Adapt rved Build Time: 10/12/01 16:38:02 ÉÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ» º TYAN THUNDER K7 º º V2.08 BIOS º ÈÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍͼ CPU = AMD Athlon(tm) 1200 MHz 2 Processor(s) Detected 1024M System RAM Passed Console: serial port BIOS drive C: is disk0 BIOS 637kB/1047552kB available memory FreeBSD/i386 bootstrap loader, Revision 0.8 (root@test.seagull.ru, Sat Oct 27 19:34:32 GMT 2001) Loading /boot/defaults/loader.conf /kernel text=0x2a3cb5 data=0x44b58+0x2573c syms=[0x4+0x3c570+0x4+0x4228e] | Hit [Enter] to boot immediately, or any other key for command prompt. Booting [kernel]... Copyright (c) 1992-2001 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.4-STABLE #0: Sun Oct 28 16:38:52 MSK 2001 root@test.seagull.ru:/usr/src/sys/compile/TEST2 Timecounter "i8254" frequency 1193182 Hz CPU: AMD Athlon(tm) Processor (1194.68-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x662 Stepping = 2 Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE> AMD Features=0xc0480000<<b19>,AMIE,DSP,3DNow!> real memory = 1073741824 (1048576K bytes) config> q avail memory = 1041178624 (1016776K bytes) Programming 24 pins in IOAPIC #0 IOAPIC #0 intpin 2 -> irq 0 FreeBSD/SMP: Multiprocessor motherboard cpu0 (BSP): apic id: 1, version: 0x00040010, at 0xfee00000 cpu1 (AP): apic id: 0, version: 0x00040010, at 0xfee00000 io0 (APIC): apic id: 2, version: 0x00170011, at 0xfec00000 Preloaded elf kernel "kernel" at 0xc048f000. Preloaded userconfig_script "/boot/kernel.conf" at 0xc048f09c. Pentium Pro MTRR support enabled md0: Malloc disk Using $PIR table, 268435454 entries at 0xc00fdef0 npx0: <math processor> on motherboard npx0: INT 16 interface pcib0: <Host to PCI bridge> on motherboard pci0: <PCI bus> on pcib0 pcib1: <PCI to PCI bridge (vendor=1022 device=700d)> at device 1.0 on pci0 pci1: <PCI bus> on pcib1 isab0: <PCI to ISA bridge (vendor=1022 device=7410)> at device 7.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <AMD 766 ATA100 controller> port 0xf000-0xf00f at device 7.1 on pci0 ata0: at 0x1f0 irq 14 on atapci0 ata1: at 0x170 irq 15 on atapci0 chip1: <PCI to Other bridge (vendor=1022 device=7413)> at device 7.3 on pci0 ahc0: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x1000-0x10ff mem 0xf4000000-0xf4000fff irq 10 at device 13.0 on pci0 aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/255 SCBs ahc1: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x1400-0x14ff mem 0xf4001000-0xf4001fff irq 5 at device 13.1 on pci0 aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/255 SCBs xl0: <3Com 3c980C Fast Etherlink XL> port 0x1800-0x187f mem 0xf4002000-0xf400207f irq 3 at device 15.0 on pci0 xl0: Ethernet address: 00:e0:81:03:dd:cc miibus0: <MII bus> on xl0 ukphy0: <Generic IEEE 802.3u media interface> on miibus0 ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto xl1: <3Com 3c980C Fast Etherlink XL> port 0x1880-0x18ff mem 0xf4002400-0xf400247f irq 11 at device 16.0 on pci0 xl1: Ethernet address: 00:e0:81:03:dd:cd miibus1: <MII bus> on xl1 ukphy1: <Generic IEEE 802.3u media interface> on miibus1 ukphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fdc0: direction bit not set fdc0: cmd 3 failed at out byte 1 of 3 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 sio0: type 16550A, console sio1: configured irq 3 not in bitmap of probed irqs 0 ppc0: parallel port not found. APIC_IO: Testing 8254 interrupt delivery APIC_IO: routing 8254 via IOAPIC #0 intpin 2 SMP: AP CPU #1 Launched! acd0: CDROM <CD-224E> at ata0-master using PIO4 Waiting 15 seconds for SCSI devices to settle pass1 at ahc0 bus 0 target 9 lun 0 pass1: <QLogic GEM359 1.07> Fixed Processor SCSI-2 device pass1: 3.300MB/s transfers da0 at ahc0 bus 0 target 3 lun 0 da0: <IBM DDYS-T36950M S96H> Fixed Direct Access SCSI-3 device da0: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled da0: 35003MB (71687340 512 byte sectors: 255H 63S/T 4462C) Mounting root from ufs:/dev/da0s1a swapon: adding /dev/da0s3b as swap device Automatic boot in progress... /dev/da0s1a: FILESYSTEM CLEAN; SKIPPING CHECKS /dev/da0s1a: clean, 463398 free (510 frags, 57861 blocks, 0.1% fragmentation) /dev/da0s2e: FILESYSTEM CLEAN; SKIPPING CHECKS /dev/da0s2e: clean, 1071357 free (29045 frags, 130289 blocks, 1.4% fragmentation) /dev/da0s2f: FILESYSTEM CLEAN; SKIPPING CHECKS /dev/da0s2f: clean, 1010432 free (184 frags, 126281 blocks, 0.0% fragmentation) /dev/da0s4e: FILESYSTEM CLEAN; SKIPPING CHECKS /dev/da0s4e: clean, 30681312 free (8 frags, 3835163 blocks, 0.0% fragmentation) Doing initial network setup: hostname. xl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 inet 195.54.192.89 netmask 0xffffffe0 broadcast 195.54.192.95 inet6 fe80::2e0:81ff:fe03:ddcc%xl0 prefixlen 64 tentative scopeid 0x1 ether 00:e0:81:03:dd:cc media: Ethernet autoselect (10baseT/UTP) status: active lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 add net default: gateway 195.54.192.65 Additional routing options: tcp extensions=NO TCP keepalive=YES. Routing daemons:. Additional daemons: syslogd. Doing additional network setup:. Starting final network daemons:. ELF ldconfig path: /usr/lib /usr/lib/compat /usr/local/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout Starting standard daemons: inetd cron sendmail sshd. Initial rc.i386 initialization:. Configuring syscons: keymap/etc/rc.syscons: cannot open /dev/ttyv0: no such device or address keyrate/etc/rc.syscons: cannot open /dev/ttyv0: no such device or address font8x16/etc/rc.syscons: cannot open /dev/ttyv0: no such device or address font8x14/etc/rc.syscons: cannot open /dev/ttyv0: no such device or address font8x8/etc/rc.syscons: cannot open /dev/ttyv0: no such device or address blank_time/etc/rc.syscons: cannot open /dev/ttyv0: no such device or address . Additional ABI support:. Starting local daemons:. Local package initialization:. Additional TCP options:. #=========================================================================================================< bug reboot log [console]: #=========================================================================================================> boot() called on cpu#0 Waiting (max 60 seconds) for system process `bufdaemon' to stop...stopped Waiting (max 60 seconds) for system process `syncer' to stop...stopped syncing disks... done Uptime: 25m51s Rebooting... cpu_reset called on cpu#1 cpu_reset: Stopping other CPUs cpu_reset: Restarting BSP npu_reset_proxy: Grabbcepdu _mrpe sleotc:k BfSoPr dBiSdP ot grab mp lock #=========================================================================================================< ok reboot log [console]: #=========================================================================================================> boot() called on cpu#0 Waiting (max 60 seconds) for system process `bufdaemon' to stop...stopped Waiting (max 60 seconds) for system process `syncer' to stop...stopped syncing disks... done Uptime: 58s Rebooting... cpu_reset called on cpu#0 cpu_reset: Stopping other CPUs #=========================================================================================================< /sys/i386/conf/TEST2 #=========================================================================================================> # # GENERIC -- Generic kernel configuration file for FreeBSD/i386 # # For more information on this file, please read the handbook section on # Kernel Configuration Files: # # http://www.FreeBSD.org/handbook/kernelconfig-config.html # # The handbook is also available locally in /usr/share/doc/handbook # if you've installed the doc distribution, otherwise always see the # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the # latest information. # # An exhaustive list of options and more detailed explanations of the # device lines is also present in the ./LINT configuration file. If you are # in doubt as to the purpose or necessity of a line, check first in LINT. # # $FreeBSD: src/sys/i386/conf/GENERIC,v 1.246.2.35 2001/09/27 17:43:06 alfred Exp $ machine i386 cpu I686_CPU ident GENERIC maxusers 32 #makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols options MATH_EMULATE #Support for x87 emulation options INET #InterNETworking options INET6 #IPv6 communications protocols options FFS #Berkeley Fast Filesystem options FFS_ROOT #FFS usable as root device [keep this!] options SOFTUPDATES #Enable FFS soft updates support options MFS #Memory Filesystem options MD_ROOT #MD is a potential root device options NFS #Network Filesystem options NFS_ROOT #NFS usable as root device, NFS required options MSDOSFS #MSDOS Filesystem options CD9660 #ISO 9660 Filesystem options CD9660_ROOT #CD-ROM usable as root, CD9660 required options PROCFS #Process filesystem options COMPAT_43 #Compatible with BSD 4.3 [KEEP THIS!] options SCSI_DELAY=15000 #Delay (in ms) before probing SCSI options UCONSOLE #Allow users to grab the console options USERCONFIG #boot -c editor options VISUAL_USERCONFIG #visual boot -c editor options KTRACE #ktrace(1) support options SYSVSHM #SYSV-style shared memory options SYSVMSG #SYSV-style message queues options SYSVSEM #SYSV-style semaphores options P1003_1B #Posix P1003_1B real-time extensions options _KPOSIX_PRIORITY_SCHEDULING options ICMP_BANDLIM #Rate limit bad replies options KBD_INSTALL_CDEV # install a CDEV entry in /dev # To make an SMP kernel, the next two are needed options SMP # Symmetric MultiProcessor Kernel options APIC_IO # Symmetric (APIC) I/O device isa device eisa device pci # Floppy drives device fdc0 at isa? port IO_FD1 irq 6 drq 2 device fd0 at fdc0 drive 0 device fd1 at fdc0 drive 1 # # If you have a Toshiba Libretto with its Y-E Data PCMCIA floppy, # don't use the above line for fdc0 but the following one: #device fdc0 # ATA and ATAPI devices device ata0 at isa? port IO_WD1 irq 14 device ata1 at isa? port IO_WD2 irq 15 device ata device atadisk # ATA disk drives device atapicd # ATAPI CDROM drives device atapifd # ATAPI floppy drives device atapist # ATAPI tape drives options ATA_STATIC_ID #Static device numbering # SCSI Controllers device ahb # EISA AHA1742 family device ahc # AHA2940 and onboard AIC7xxx devices device amd # AMD 53C974 (Tekram DC-390(T)) device isp # Qlogic family device ncr # NCR/Symbios Logic device sym # NCR/Symbios Logic (newer chipsets) options SYM_SETUP_LP_PROBE_MAP=0x40 # Allow ncr to attach legacy NCR devices when # both sym and ncr are configured device adv0 at isa? device adw device bt0 at isa? device aha0 at isa? device aic0 at isa? device ncv # NCR 53C500 device nsp # Workbit Ninja SCSI-3 device stg # TMC 18C30/18C50 # SCSI peripherals device scbus # SCSI bus (required) device da # Direct Access (disks) device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct SCSI access) # RAID controllers interfaced to the SCSI subsystem device asr # DPT SmartRAID V, VI and Adaptec SCSI RAID device dpt # DPT Smartcache - See LINT for options! device mly # Mylex AcceleRAID/eXtremeRAID # RAID controllers device aac # Adaptec FSA RAID, Dell PERC2/PERC3 device ida # Compaq Smart RAID device amr # AMI MegaRAID device mlx # Mylex DAC960 family device twe # 3ware Escalade # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc0 at isa? port IO_KBD device atkbd0 at atkbdc? irq 1 flags 0x1 device psm0 at atkbdc? irq 12 device vga0 at isa? # splash screen/screen saver pseudo-device splash # syscons is the default console driver, resembling an SCO console device sc0 at isa? flags 0x100 # Enable this and PCVT_FREEBSD for pcvt vt220 compatible console driver #device vt0 at isa? #options XSERVER # support for X server on a vt console #options FAT_CURSOR # start with block cursor # If you have a ThinkPAD, uncomment this along with the rest of the PCVT lines #options PCVT_SCANSET=2 # IBM keyboards are non-std # Floating point support - do not disable. device npx0 at nexus? port IO_NPX irq 13 # Power management support (see LINT for more options) device apm0 at nexus? disable flags 0x20 # Advanced Power Management # PCCARD (PCMCIA) support device card device pcic0 at isa? irq 0 port 0x3e0 iomem 0xd0000 device pcic1 at isa? irq 0 port 0x3e2 iomem 0xd4000 disable # Serial (COM) ports device sio0 at isa? port IO_COM1 flags 0x10 irq 4 device sio1 at isa? port IO_COM2 irq 3 device sio2 at isa? disable port IO_COM3 irq 5 device sio3 at isa? disable port IO_COM4 irq 9 # Parallel port device ppc0 at isa? irq 7 device ppbus # Parallel port bus (required) device lpt # Printer device plip # TCP/IP over parallel device ppi # Parallel port interface device #device vpo # Requires scbus and da # PCI Ethernet NICs. device de # DEC/Intel DC21x4x (``Tulip'') device txp # 3Com 3cR990 (``Typhoon'') device vx # 3Com 3c590, 3c595 (``Vortex'') # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! device miibus # MII bus support device dc # DEC/Intel 21143 and various workalikes device fxp # Intel EtherExpress PRO/100B (82557, 82558) device pcn # AMD Am79C97x PCI 10/100 NICs device rl # RealTek 8129/8139 device sf # Adaptec AIC-6915 (``Starfire'') device sis # Silicon Integrated Systems SiS 900/SiS 7016 device ste # Sundance ST201 (D-Link DFE-550TX) device tl # Texas Instruments ThunderLAN device tx # SMC EtherPower II (83c170 ``EPIC'') device vr # VIA Rhine, Rhine II device wb # Winbond W89C840F device wx # Intel Gigabit Ethernet Card (``Wiseman'') device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'') # ISA Ethernet NICs. # 'device ed' requires 'device miibus' device ed0 at isa? port 0x280 irq 10 iomem 0xd8000 device ex device ep device fe0 at isa? port 0x300 # Xircom Ethernet device xe # PRISM I IEEE 802.11b wireless NIC. device awi # WaveLAN/IEEE 802.11 wireless NICs. Note: the WaveLAN/IEEE really # exists only as a PCMCIA device, so there is no ISA attachment needed # and resources will always be dynamically assigned by the pccard code. device wi # Aironet 4500/4800 802.11 wireless NICs. Note: the declaration below will # work for PCMCIA and PCI cards, as well as ISA cards set to ISA PnP # mode (the factory default). If you set the switches on your ISA # card for a manually chosen I/O address and IRQ, you must specify # those parameters here. device an # The probe order of these is presently determined by i386/isa/isa_compat.c. device ie0 at isa? port 0x300 irq 10 iomem 0xd0000 #device le0 at isa? port 0x300 irq 5 iomem 0xd0000 device lnc0 at isa? port 0x280 irq 10 drq 0 device cs0 at isa? port 0x300 device sn0 at isa? port 0x300 irq 10 # Pseudo devices - the number indicates how many units to allocate. pseudo-device loop # Network loopback pseudo-device ether # Ethernet support pseudo-device sl 1 # Kernel SLIP pseudo-device ppp 1 # Kernel PPP pseudo-device tun # Packet tunnel. pseudo-device pty # Pseudo-ttys (telnet etc) pseudo-device md # Memory "disks" pseudo-device gif # IPv6 and IPv4 tunneling pseudo-device faith 1 # IPv6-to-IPv4 relaying (translation) # The `bpf' pseudo-device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! pseudo-device bpf #Berkeley packet filter # USB support device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device usb # USB Bus (required) device ugen # Generic device uhid # "Human Interface Devices" device ukbd # Keyboard device ulpt # Printer device umass # Disks/Mass storage - Requires scbus and da device ums # Mouse device uscanner # Scanners device urio # Diamond Rio MP3 Player # USB Ethernet, requires mii device aue # ADMtek USB ethernet device cue # CATC USB ethernet device kue # Kawasaki LSI USB ethernet #=========================================================================================================< Best regards, Oleg mailto:ilin@rinet.ru The shutdown routine uses memory based syncronization to let the BSP
perform the real reset. Your problem looks like a livelock, which can
be avoided by flushing the cache after setting the variable used for
syncronization.
Index: sys/i386/i386/vm_machdep.c
===================================================================
RCS file: /home/ncvs/src/sys/i386/i386/vm_machdep.c,v
retrieving revision 1.132.2.5
diff -u -r1.132.2.5 vm_machdep.c
--- sys/i386/i386/vm_machdep.c 22 Sep 2001 09:21:48 -0000 1.132.2.5
+++ sys/i386/i386/vm_machdep.c 29 Oct 2001 03:41:00 -0000
@@ -406,12 +415,14 @@
u_int saved_mp_lock;
cpu_reset_proxy_active = 1;
+ wbinvd();
while (cpu_reset_proxy_active == 1)
; /* Wait for other cpu to disable interupts */
saved_mp_lock = mp_lock;
mp_lock = 1;
printf("cpu_reset_proxy: Grabbed mp lock for BSP\n");
cpu_reset_proxy_active = 3;
+ wbinvd();
while (cpu_reset_proxy_active == 3)
; /* Wait for other cpu to enable interrupts */
stop_cpus((1<<cpu_reset_proxyid));
@@ -454,6 +465,7 @@
started_cpus = (1<<0); /* Restart CPU #0 */
cnt = 0;
+ wbinvd();
while (cpu_reset_proxy_active == 0 && cnt < 10000000)
cnt++; /* Wait for BSP to announce restart */
if (cpu_reset_proxy_active == 0)
@@ -461,6 +473,7 @@
__asm __volatile("cli" : : : "memory");
cpu_reset_proxy_active = 2;
cnt = 0;
+ wbinvd();
while (cpu_reset_proxy_active == 2 && cnt < 10000000)
cnt++; /* Do nothing */
if (cpu_reset_proxy_active == 2) {
@@ -468,6 +481,7 @@
cpu_reset_real(); /* XXX: Bogus ? */
}
cpu_reset_proxy_active = 4;
+ wbinvd();
__asm __volatile("sti" : : : "memory");
while (1);
/* NOTREACHED */
Hello Tor, Monday, October 29, 2001, 6:51:11 AM, you wrote: > The shutdown routine uses memory based syncronization to let the BSP > perform the real reset. Your problem looks like a livelock, which can > be avoided by flushing the cache after setting the variable used for > syncronization. Hmm... Problem not fixed. /usr/src > patch < tor.patch Hmm... Looks like a unified diff to me... The text leading up to this was: -------------------------- |diff -u -r1.132.2.5 vm_machdep.c |--- sys/i386/i386/vm_machdep.c 22 Sep 2001 09:21:48 -0000 1.132.2.5 |+++ sys/i386/i386/vm_machdep.c 29 Oct 2001 03:41:00 -0000 -------------------------- Patching file sys/i386/i386/vm_machdep.c using Plan A... Hunk #1 succeeded at 415. Hunk #2 succeeded at 465. Hunk #3 succeeded at 473 with fuzz 2. Hunk #4 succeeded at 481. done I rebuild kernel and boot system. Next: test# reboot reboot log [console]: #========================================================================> boot() called on cpu#1 Waiting (max 60 seconds) for system process `bufdaemon' to stop...stopped Waiting (max 60 seconds) for system process `syncer' to stop...stopped syncing disks... 4 done Uptime: 30m49s Rebooting... cpu_reset called on cpu#1 cpu_reset: Stopping other CPUs cpu_reset: Restarting BSP npu_reset_proxy: Grabbcepdu _mrpe sleotc:k BfSoPr dBiSdP t grab mp lock #========================================================================< Long wait.. System not Restarting. Best regards, Oleg mailto:ilin@rinet.ru > Hello Tor,
>
> Monday, October 29, 2001, 6:51:11 AM, you wrote:
>
> > The shutdown routine uses memory based syncronization to let the BSP
> > perform the real reset. Your problem looks like a livelock, which can
> > be avoided by flushing the cache after setting the variable used for
> > syncronization.
>
> Hmm... Problem not fixed.
The AP probably managed to count to 10000000 before the BSP had
completed printing the message about having grabbed the MP lock. Thus
cpu_reset_proxy_active was not updated before the timeout had expired.
The fix for that is to move the printf call to after the wbinvd()
statement in cpu_reset_proxy().
Index: sys/i386/i386/vm_machdep.c
===================================================================
RCS file: /home/ncvs/src/sys/i386/i386/vm_machdep.c,v
retrieving revision 1.132.2.5
diff -u -r1.132.2.5 vm_machdep.c
--- sys/i386/i386/vm_machdep.c 22 Sep 2001 09:21:48 -0000 1.132.2.5
+++ sys/i386/i386/vm_machdep.c 29 Oct 2001 15:55:23 -0000
@@ -406,12 +415,14 @@
u_int saved_mp_lock;
cpu_reset_proxy_active = 1;
+ wbinvd();
while (cpu_reset_proxy_active == 1)
; /* Wait for other cpu to disable interupts */
saved_mp_lock = mp_lock;
mp_lock = 1;
- printf("cpu_reset_proxy: Grabbed mp lock for BSP\n");
cpu_reset_proxy_active = 3;
+ wbinvd();
+ printf("cpu_reset_proxy: Grabbed mp lock for BSP\n");
while (cpu_reset_proxy_active == 3)
; /* Wait for other cpu to enable interrupts */
stop_cpus((1<<cpu_reset_proxyid));
@@ -454,6 +465,7 @@
started_cpus = (1<<0); /* Restart CPU #0 */
cnt = 0;
+ wbinvd();
while (cpu_reset_proxy_active == 0 && cnt < 10000000)
cnt++; /* Wait for BSP to announce restart */
if (cpu_reset_proxy_active == 0)
@@ -461,6 +473,7 @@
__asm __volatile("cli" : : : "memory");
cpu_reset_proxy_active = 2;
cnt = 0;
+ wbinvd();
while (cpu_reset_proxy_active == 2 && cnt < 10000000)
cnt++; /* Do nothing */
if (cpu_reset_proxy_active == 2) {
@@ -468,6 +481,7 @@
cpu_reset_real(); /* XXX: Bogus ? */
}
cpu_reset_proxy_active = 4;
+ wbinvd();
__asm __volatile("sti" : : : "memory");
while (1);
/* NOTREACHED */
Hello Tor, Monday, October 29, 2001, 7:04:38 PM, you wrote: > The AP probably managed to count to 10000000 before the BSP had > completed printing the message about having grabbed the MP lock. Thus > cpu_reset_proxy_active was not updated before the timeout had expired. > The fix for that is to move the printf call to after the wbinvd() > statement in cpu_reset_proxy(). Hmm... /usr/src/sys/i386/i386 > mv vm_machdep.c.orig vm_machdep.c cd /usr/src /usr/src > patch < tor.patch2 Hmm... Looks like a unified diff to me... The text leading up to this was: -------------------------- |diff -u -r1.132.2.5 vm_machdep.c |--- sys/i386/i386/vm_machdep.c 22 Sep 2001 09:21:48 -0000 1.132.2.5 |+++ sys/i386/i386/vm_machdep.c 29 Oct 2001 15:55:23 -0000 -------------------------- Patching file sys/i386/i386/vm_machdep.c using Plan A... Hunk #1 succeeded at 415. Hunk #2 succeeded at 465. Hunk #3 succeeded at 473. Hunk #4 succeeded at 481. done Rebuild kernel, boot new kernel.. test# reboot reboot log [console] #==========================================================================> boot() called on cpu#1 Waiting (max 60 seconds) for system process `bufdaemon' to stop...stopped Waiting (max 60 seconds) for system process `syncer' to stop...stopped syncing disks... done Uptime: 32m55s Rebooting... cpu_reset called on cpu#1 cpu_reset: Stopping other CPUs cpu_reset: Restarting BSP cpu_reset_proxy: Grabbed mp lock for BSP cpu_reset_proxy: Stopped CPU 1 #==========================================================================< Long wait.. System not restarting. Best regards, Oleg mailto:ilin@rinet.ru > cpu_reset_proxy: Grabbed mp lock for BSP
> cpu_reset_proxy: Stopped CPU 1
> #==========================================================================<
>
> Long wait..
At least, the output is no longer intermingled.
Interrupts should probably be left disabled on CPU#1.
If that doesn't help then it's time to implement a proper shutdown
according to Appendix B.5 of the MP spec.
- Tor Egge
Index: sys/i386/i386/vm_machdep.c
===================================================================
RCS file: /home/ncvs/src/sys/i386/i386/vm_machdep.c,v
retrieving revision 1.132.2.5
diff -u -r1.132.2.5 vm_machdep.c
--- sys/i386/i386/vm_machdep.c 22 Sep 2001 09:21:48 -0000 1.132.2.5
+++ sys/i386/i386/vm_machdep.c 29 Oct 2001 20:27:03 -0000
@@ -406,12 +415,14 @@
u_int saved_mp_lock;
cpu_reset_proxy_active = 1;
+ wbinvd();
while (cpu_reset_proxy_active == 1)
; /* Wait for other cpu to disable interupts */
saved_mp_lock = mp_lock;
mp_lock = 1;
- printf("cpu_reset_proxy: Grabbed mp lock for BSP\n");
cpu_reset_proxy_active = 3;
+ wbinvd();
+ printf("cpu_reset_proxy: Grabbed mp lock for BSP\n");
while (cpu_reset_proxy_active == 3)
; /* Wait for other cpu to enable interrupts */
stop_cpus((1<<cpu_reset_proxyid));
@@ -454,6 +465,7 @@
started_cpus = (1<<0); /* Restart CPU #0 */
cnt = 0;
+ wbinvd();
while (cpu_reset_proxy_active == 0 && cnt < 10000000)
cnt++; /* Wait for BSP to announce restart */
if (cpu_reset_proxy_active == 0)
@@ -461,6 +473,7 @@
__asm __volatile("cli" : : : "memory");
cpu_reset_proxy_active = 2;
cnt = 0;
+ wbinvd();
while (cpu_reset_proxy_active == 2 && cnt < 10000000)
cnt++; /* Do nothing */
if (cpu_reset_proxy_active == 2) {
@@ -468,7 +481,7 @@
cpu_reset_real(); /* XXX: Bogus ? */
}
cpu_reset_proxy_active = 4;
- __asm __volatile("sti" : : : "memory");
+ wbinvd();
while (1);
/* NOTREACHED */
}
Here is an updated patch which tries to shutdown the AP in a way
closer to the MP spec. Interrupts are masked in the IOAPICs and an
attempt is made to set the APs in the default HALT state before
shutting down the BSP.
- Tor Egge
Index: sys/i386/include/smp.h
===================================================================
RCS file: /home/ncvs/src/sys/i386/include/smp.h,v
retrieving revision 1.50.2.5
diff -u -r1.50.2.5 smp.h
--- sys/i386/include/smp.h 13 Feb 2001 22:32:45 -0000 1.50.2.5
+++ sys/i386/include/smp.h 30 Oct 2001 05:40:34 -0000
@@ -42,9 +42,11 @@
/* global data in mpboot.s */
extern int bootMP_size;
+extern int shutdownMP_size;
/* functions in mpboot.s */
void bootMP __P((void));
+void shutdownMP __P((void));
/* global data in mplock.s */
extern u_int mp_lock;
@@ -128,6 +130,7 @@
void smp_invltlb __P((void));
int stop_cpus __P((u_int));
int restart_cpus __P((u_int));
+void shutdown_other_cpus __P((int));
#ifdef BETTER_CLOCK
void forward_statclock __P((int pscnt));
void forward_hardclock __P((int pscnt));
@@ -158,6 +161,11 @@
void io_apic_set_id __P((int, int));
int io_apic_get_id __P((int));
int ext_int_setup __P((int, int));
+void shutdown_ioapics __P((void));
+int focus_apic_interrupt __P((int, int));
+int unfocus_apic_interrupt __P((int));
+int focus_apic_interrupts __P((int));
+int unfocus_apic_interrupts __P((void));
#if defined(READY)
void clr_io_apic_mask24 __P((int, u_int32_t));
Index: sys/i386/i386/mpboot.s
===================================================================
RCS file: /home/ncvs/src/sys/i386/i386/mpboot.s,v
retrieving revision 1.13.2.3
diff -u -r1.13.2.3 mpboot.s
--- sys/i386/i386/mpboot.s 7 Sep 2000 01:18:26 -0000 1.13.2.3
+++ sys/i386/i386/mpboot.s 30 Oct 2001 05:40:34 -0000
@@ -302,3 +302,19 @@
.globl _bootMP_size
_bootMP_size:
.long BOOTMP2 - BOOTMP1
+
+SHUTDOWNMP1:
+
+NON_GPROF_ENTRY(shutdownMP)
+ .code16
+ cli
+1:
+ hlt
+ jmp 1b
+
+SHUTDOWNMP2:
+ .global CNAME(shutdownMP_size)
+CNAME(shutdownMP_size):
+ .long SHUTDOWNMP2 - SHUTDOWNMP1
+
+
Index: sys/i386/i386/mp_machdep.c
===================================================================
RCS file: /home/ncvs/src/sys/i386/i386/mp_machdep.c,v
retrieving revision 1.115.2.10
diff -u -r1.115.2.10 mp_machdep.c
--- sys/i386/i386/mp_machdep.c 5 Oct 2001 06:20:17 -0000 1.115.2.10
+++ sys/i386/i386/mp_machdep.c 30 Oct 2001 05:40:34 -0000
@@ -2107,6 +2130,78 @@
}
+void
+shutdown_other_cpus(int callercpu)
+{
+ u_char mpbiosreason;
+ u_long mpbioswarmvec;
+ int cpu;
+ u_int boot_addr;
+ u_long icr_lo, icr_hi;
+ int physical_cpu;
+
+ if (bsp_apic_ready == 0)
+ return;
+
+ boot_addr = boot_address; /* XXX */
+
+ /* save the current value of the warm-start vector */
+ mpbioswarmvec = *((u_long *) WARMBOOT_OFF);
+#ifndef PC98
+ outb(CMOS_REG, BIOS_RESET);
+ mpbiosreason = inb(CMOS_DATA);
+#endif
+
+ /* install the AP shutdown code */
+ install_ap_shutdown_tramp(boot_addr);
+
+ for (cpu = 0; cpu < mp_ncpus; cpu++) {
+
+ if (cpu == callercpu)
+ continue;
+
+ /* setup a vector to our boot code */
+ *((volatile u_short *) WARMBOOT_OFF) = WARMBOOT_TARGET;
+ *((volatile u_short *) WARMBOOT_SEG) = (boot_addr >> 4);
+#ifndef PC98
+ outb(CMOS_REG, BIOS_RESET);
+ outb(CMOS_DATA, BIOS_WARM); /* 'warm-start' */
+#endif
+ /* get the PHYSICAL APIC ID# */
+ physical_cpu = CPU_TO_ID(cpu);
+
+ /* setup the address for the target AP */
+ icr_hi = lapic.icr_hi & ~APIC_ID_MASK;
+ icr_hi |= (physical_cpu << 24);
+ lapic.icr_hi = icr_hi;
+
+ /* do an INIT IPI: assert RESET */
+ icr_lo = lapic.icr_lo & 0xfff00000;
+ lapic.icr_lo = icr_lo | 0x0000c500;
+
+ /* wait for pending status end */
+ while (lapic.icr_lo & APIC_DELSTAT_MASK)
+ /* spin */ ;
+
+ /* do an INIT IPI: deassert RESET */
+ lapic.icr_lo = icr_lo | 0x00008500;
+
+ /* wait for pending status end */
+ u_sleep(10000); /* wait ~10mS */
+ while (lapic.icr_lo & APIC_DELSTAT_MASK)
+ /* spin */ ;
+
+ u_sleep(100000); /* wait ~100mS */
+ }
+ /* restore the warmstart vector */
+ *(u_long *) WARMBOOT_OFF = mpbioswarmvec;
+#ifndef PC98
+ outb(CMOS_REG, BIOS_RESET);
+ outb(CMOS_DATA, mpbiosreason);
+#endif
+}
+
+
/*
* load the 1st level AP boot code into base memory.
*/
@@ -2168,6 +2263,22 @@
/*
+ * load the level AP shutdown code into base memory.
+ */
+
+static void
+install_ap_shutdown_tramp(u_int boot_addr)
+{
+ int x;
+ int size = *(int *) ((u_long) & shutdownMP_size);
+ u_char *src = (u_char *) ((u_long) shutdownMP);
+ u_char *dst = (u_char *) boot_addr + KERNBASE;
+
+ for (x = 0; x < size; ++x)
+ *dst++ = *src++;
+}
+
+/*
* this function starts the AP (application processor) identified
* by the APIC ID 'physicalCpu'. It does quite a "song and dance"
* to accomplish this. This is necessary because of the nuances
Index: sys/i386/i386/vm_machdep.c
===================================================================
RCS file: /home/ncvs/src/sys/i386/i386/vm_machdep.c,v
retrieving revision 1.132.2.6
diff -u -r1.132.2.6 vm_machdep.c
--- sys/i386/i386/vm_machdep.c 20 Oct 2001 17:37:29 -0000 1.132.2.6
+++ sys/i386/i386/vm_machdep.c 30 Oct 2001 05:40:34 -0000
@@ -406,17 +415,28 @@
u_int saved_mp_lock;
cpu_reset_proxy_active = 1;
+ wbinvd();
while (cpu_reset_proxy_active == 1)
; /* Wait for other cpu to disable interupts */
saved_mp_lock = mp_lock;
mp_lock = 1;
- printf("cpu_reset_proxy: Grabbed mp lock for BSP\n");
cpu_reset_proxy_active = 3;
+ wbinvd();
+ printf("cpu_reset_proxy: Grabbed mp lock for BSP\n");
while (cpu_reset_proxy_active == 3)
; /* Wait for other cpu to enable interrupts */
stop_cpus((1<<cpu_reset_proxyid));
printf("cpu_reset_proxy: Stopped CPU %d\n", cpu_reset_proxyid);
- DELAY(1000000);
+ DELAY(100000);
+ shutdown_ioapics();
+ printf("cpu_reset_proxy: Disabled IOAPICs\n");
+ DELAY(100000);
+ shutdown_other_cpus(cpuid);
+ DELAY(100000);
+ printf("cpu_reset_proxy: Shutdown other CPUs\n");
+ DELAY(100000);
+ printf("cpu_reset_proxy: Resetting BSP\n");
+ DELAY(100000);
cpu_reset_real();
}
#endif
@@ -442,7 +462,16 @@
}
if (cpuid == 0) {
- DELAY(1000000);
+ DELAY(100000);
+ shutdown_ioapics();
+ printf("cpu_reset: Disabled IOAPICs\n");
+ DELAY(100000);
+ shutdown_other_cpus(cpuid);
+ DELAY(100000);
+ printf("cpu_reset: Shutdown other CPUs\n");
+ DELAY(100000);
+ printf("cpu_reset: Resetting BSP\n");
+ DELAY(100000);
cpu_reset_real();
/* NOTREACHED */
} else {
@@ -454,20 +483,43 @@
started_cpus = (1<<0); /* Restart CPU #0 */
cnt = 0;
- while (cpu_reset_proxy_active == 0 && cnt < 10000000)
- cnt++; /* Wait for BSP to announce restart */
+ wbinvd();
+ /* Wait for BSP to announce restart */
+ while (cpu_reset_proxy_active == 0 && cnt < 2000) {
+ cnt++;
+ DELAY(1000);
+ }
if (cpu_reset_proxy_active == 0)
printf("cpu_reset: Failed to restart BSP\n");
__asm __volatile("cli" : : : "memory");
cpu_reset_proxy_active = 2;
+ wbinvd();
cnt = 0;
- while (cpu_reset_proxy_active == 2 && cnt < 10000000)
+ /* Wait for BSP to grab mp lock */
+ while (cpu_reset_proxy_active == 2 && cnt < 2000) {
cnt++; /* Do nothing */
+ DELAY(1000);
+ }
if (cpu_reset_proxy_active == 2) {
printf("cpu_reset: BSP did not grab mp lock\n");
+ DELAY(100000);
+ shutdown_ioapics();
+ printf("cpu#%d: Disabled IOAPICs\n",
+ cpuid);
+ DELAY(100000);
+ shutdown_other_cpus(cpuid);
+ DELAY(100000);
+ printf("cpu#%d: Shutdown other CPUs\n",
+ cpuid);
+ DELAY(100000);
+ printf("cpu#%d: Resetting self (AP)\n",
+ cpuid);
+ DELAY(100000);
cpu_reset_real(); /* XXX: Bogus ? */
}
cpu_reset_proxy_active = 4;
+ wbinvd();
+ /* Enable interrupts to allow BSP stopping this AP */
__asm __volatile("sti" : : : "memory");
while (1);
/* NOTREACHED */
Index: sys/i386/i386/mpapic.c
===================================================================
RCS file: /home/ncvs/src/sys/i386/i386/mpapic.c,v
retrieving revision 1.37.2.5
diff -u -r1.37.2.5 mpapic.c
--- sys/i386/i386/mpapic.c 13 Feb 2001 22:32:44 -0000 1.37.2.5
+++ sys/i386/i386/mpapic.c 30 Oct 2001 05:40:34 -0000
@@ -283,6 +294,149 @@
}
/* return GOOD status */
+ return 0;
+}
+
+
+/*
+ * Block all interrupts on all IOAPICs in preparation for shutdown.
+ */
+
+void
+shutdown_ioapics(void)
+{
+ int apic;
+ int maxpin;
+ int pin;
+ u_char select; /* the select register is 8 bits */
+ u_int eflags;
+
+ for (apic = 0; apic < mp_napics; ++apic) {
+ maxpin = REDIRCNT_IOAPIC(apic); /* pins in APIC */
+ printf("Disabling interrupt on %d pins in IOAPIC #%d ...",
+ maxpin, apic);
+ for (pin = 0; pin < maxpin; ++pin) {
+ select = pin * 2 + IOAPIC_REDTBL0; /* register */
+ eflags = read_eflags();
+ __asm __volatile("cli" : : : "memory");
+ s_lock(&imen_lock);
+ io_apic_write(apic, select,
+ (io_apic_read(apic, select) &
+ ~IOART_INTMASK)
+ |IOART_INTMSET);
+ s_unlock(&imen_lock);
+ write_eflags(eflags);
+ }
+ printf(" Done\n");
+ }
+}
+
+
+/* Bind an interrupt to a CPU */
+
+int
+focus_apic_interrupt(int irq, int cpu)
+{
+ int apic;
+ int pin;
+ int redtbl0;
+ int redtbl1;
+ u_char select;
+ u_int eflags;
+
+ if (irq < 0 || irq >= APIC_INTMAPSIZE)
+ return -1;
+
+ apic = int_to_apicintpin[irq].ioapic;
+ pin = int_to_apicintpin[irq].int_pin;
+
+ if (apic == -1)
+ return -1;
+
+ select = pin * 2 + IOAPIC_REDTBL0;
+
+ eflags = read_eflags();
+ __asm __volatile("cli" : : : "memory");
+ s_lock(&imen_lock);
+ redtbl0 = io_apic_read(apic, select);
+ redtbl1 = io_apic_read(apic, select + 1);
+
+ redtbl0 &= ~IOART_DELMOD;
+ redtbl0 |= IOART_DELFIXED;
+
+ redtbl1 &= ~IOART_DEST;
+ redtbl1 |= (CPU_TO_ID(cpu) << 24);
+
+ io_apic_write(apic, select, redtbl0);
+ io_apic_write(apic, select + 1, redtbl1);
+ s_unlock(&imen_lock);
+ write_eflags(eflags);
+
+ return 0;
+}
+
+
+/* Allow an interrupt to be delivered to any CPU */
+
+int
+unfocus_apic_interrupt(int irq)
+{
+ int apic;
+ int pin;
+ int redtbl0;
+ int redtbl1;
+ u_char select;
+ u_int eflags;
+
+ if (irq < 0 || irq >= APIC_INTMAPSIZE)
+ return -1;
+
+ apic = int_to_apicintpin[irq].ioapic;
+ pin = int_to_apicintpin[irq].int_pin;
+
+ if (apic == -1)
+ return -1;
+
+ select = pin * 2 + IOAPIC_REDTBL0;
+
+ eflags = read_eflags();
+ __asm __volatile("cli" : : : "memory");
+ s_lock(&imen_lock);
+ redtbl0 = io_apic_read(apic, select);
+ redtbl1 = io_apic_read(apic, select + 1);
+
+ redtbl0 &= ~IOART_DELMOD;
+ redtbl0 |= IOART_DELLOPRI;
+
+ redtbl1 |= IOART_DEST;
+
+ io_apic_write(apic, select, redtbl0);
+ io_apic_write(apic, select + 1, redtbl1);
+ s_unlock(&imen_lock);
+ write_eflags(eflags);
+
+ return 0;
+}
+
+
+int
+focus_apic_interrupts(int cpu)
+{
+ int irq;
+
+ for (irq = 0; irq < APIC_INTMAPSIZE; irq++)
+ focus_apic_interrupt(irq, cpu);
+ return 0;
+}
+
+
+int
+unfocus_apic_interrupts(void)
+{
+ int irq;
+
+ for (irq = 0; irq < APIC_INTMAPSIZE; irq++)
+ unfocus_apic_interrupt(irq);
return 0;
}
#undef DEFAULT_ISA_FLAGS
Hello Tor, Tuesday, October 30, 2001, 12:08:46 AM, you wrote: > At least, the output is no longer intermingled. > Interrupts should probably be left disabled on CPU#1. > If that doesn't help then it's time to implement a proper shutdown > according to Appendix B.5 of the MP spec. Report: /usr/src > patch < tor.patch3 Hmm... Looks like a unified diff to me... The text leading up to this was: -------------------------- |diff -u -r1.132.2.5 vm_machdep.c |--- sys/i386/i386/vm_machdep.c 22 Sep 2001 09:21:48 -0000 1.132.2.5 |+++ sys/i386/i386/vm_machdep.c 29 Oct 2001 20:27:03 -0000 -------------------------- Patching file sys/i386/i386/vm_machdep.c using Plan A... Hunk #1 succeeded at 415. Hunk #2 succeeded at 465. Hunk #3 succeeded at 473. Hunk #4 succeeded at 481. done Rebuild kernel, boot new kernel.. test# reboot reboot log [console]: #========================================================================> login: boot() called on cpu#1 Waiting (max 60 seconds) for system process `bufdaemon' to stop...stopped Waiting (max 60 seconds) for system process `syncer' to stop...stopped syncing disks... done Uptime: 8m50s Rebooting... cpu_reset called on cpu#0 cpu_reset: Stopping other CPUs #========================================================================< Long wait.. System not restarting. Best regards, Oleg mailto:ilin@rinet.ru Hello Tor, Tuesday, October 30, 2001, 9:02:00 AM, you wrote: > Here is an updated patch which tries to shutdown the AP in a way > closer to the MP spec. Interrupts are masked in the IOAPICs and an > attempt is made to set the APs in the default HALT state before > shutting down the BSP. > - Tor Egge Report: patch: #========================================================================> /usr/src > patch -l <tor.patch4 Hmm... Looks like a unified diff to me... The text leading up to this was: -------------------------- ..skip.. [all Hunk succeeded] done #========================================================================< Rebuild kernel, restart system .. test# reboot reboot log [console]: #========================================================================> boot() called on cpu#1 Waiting (max 60 seconds) for system process `bufdaemon' to stop...stopped Waiting (max 60 seconds) for system process `syncer' to stop...stopped syncing disks... done Uptime: 20m6s Rebooting... cpu_reset called on cpu#0 cpu_reset: Stopping other CPUs Disabling interrupt on 24 pins in IOAPIC #0 ... Done cpu_reset: Disabled IOAPICs cpu_reset: Shutdown other CPUs cpu_reset: Resetting BSP #========================================================================< Wait.. wait.. System not restarting. Best regards, Oleg mailto:ilin@rinet.ru State Changed From-To: open->feedback Does this still occur with more recent releases? Adding to the audit trail: In message <181321304046.20021202103846@rinet.ru>, Oleg Ilin writes: >Hello Ian, > >Sunday, December 01, 2002, 11:07:19 PM, you wrote: > >> Does this still occur with more recent releases? > >Yes. >uname -a >FreeBSD test.seagull.ru 4.7-STABLE FreeBSD 4.7-STABLE #0: Tue Oct 29 >14:12:20 MSK 2002 root@test.seagull.ru :/usr/src/sys/compile/TEST47 i386 > > >I'm reboot remote server over smart ups.. > >Best regards, > Oleg mailto:ilin@rinet.ru > State Changed From-To: feedback->open Feedback received Are you still having this issue with newer releases? Best Regards -- Rionda aka Matteo Riondato Disinformato per default G.U.F.I. Staff Member (http://www.gufi.org) FreeSBIE Developer (http://www.freesbie.org) State Changed From-To: open->closed Feedback timeout (> 5 months). |
