Installing 9.0-RELEASE on a UFS filesystem then creating a ZFS filesystem on a USB drive causes no evident problems. However, after upgrading to 9.0-STABLE the machine will not shutdown or reboot, getting "stuck" apparently trying to shut down the USB controller. IMPACT -- Cannot reboot or shutdown machine without physical access to manually reset or power-down Can be replicated on a "stock" install and with a memory card (spare 2GB SD used) in a reader (Elago used) in the USB port (Intel Atom 330 CPU and ICH7 chipset). Nothing has been added to the ZFS filesystem, just creating it using zpool is sufficient. Also reported as far back as February 11th, 2012 at http://forums.freebsd.org/showthread.php?t=29700 on different hardware and with ZFS root. Note that this does *not* require ZFS root, only that there be a ZFS filesystem on USB. I have seen this on three machines here, including both with i686 and amd64 architectures. How-To-Repeat: Fresh install of 9.0-RELEASE off FreeBSD-9.0-RELEASE-amd64-memstick.img Set hostname Decline custom keyboard map Select default distributions + src * games * lib32 * ports * src Guided partitioning tool, accept defaults offerred Observe base,txz, kernel.txz, games.txz, lib32.txz, ports.txz, src.txz install Set root password Configure IPv4 static for Class C private network and its gateway Configure resolv.conf to point at DNS on local network Timezone; CMOS set to UTC, PDT zone selected Services, only default sshd Enable crash dumps Add user 'jeff' Apply configuration and exit Decline making manual changes Reboot First boot: Add jeff to wheel ssh in and... 2 GB SD card in USB reader appears as /dev/da0 # dd if=/dev/zero of=/dev/da0 bs=64k * stick in and zeroed -- reboots fine * zfs_enable="YES" -- reboots fine # zpool create stick /dev/da0 # reboot now Reboots fine # uname -a FreeBSD usb.wagsky.com 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 ##### ##### Confirmed -- OK on "stock" 9.0-RELEASE ##### Copy /boot/kernel to /boot/GENERIC-RELEASE Make world using distributed sources (2012-05-06-1325) Make GENERIC kernel using distributed sources (2012-05-06-1833) Install as per http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html 25.7.1 The Canonical Way to Update Your System Reboot to single -- OK Only thing that was reported as different in mergemaster was MOTD, left original Reboot to live (no ZFS mounted) FreeBSD usb.wagsky.com 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Sun May 6 18:58:31 PDT 2012 jeff@usb.wagsky.com:/usr/obj/usr/src/sys/GENERIC amd64 Reboot after boot -- OK ##### ##### Confirmed -- OK on "home-built" 9.0-RELEASE ##### pkg_add -r cvsup-without-gui cvsup -h cvsup7.freebsd.org /usr/share/examples/cvsup/stable-supfile run it again, just to make sure no missed commits -- Sun May 6 19:58:37 PDT 2012 buildworld, buildkernel, installkernel (GENERIC) ##### ##### FAILS -- with just new kernel ##### (kernel alone tested as there have been many changes in /etc/rc.d/ and I wanted to see if it could be replicated without the changes) installworld, mergemaster ##### ##### FAILS -- with completed upgrade ##### Log output of shutdown/boot cycle (note that reset button or power switch is required to complete shutdown/reboot process) May 7 07:10:23 usb reboot: rebooted by jeff May 7 07:10:23 usb syslogd: exiting on signal 15 May 7 07:11:36 usb syslogd: kernel boot file is /boot/kernel/kernel May 7 07:11:36 usb kernel: Waiting (max 60 seconds) for system process `vnlru' to stop...done May 7 07:11:36 usb kernel: Waiting (max 60 seconds) for system process `bufdaemon' to stop...done May 7 07:11:36 usb kernel: Waiting (max 60 seconds) for system process `syncer' to stop... May 7 07:11:36 usb kernel: Syncing disks, vnodes remaining...1 1 0 done May 7 07:11:36 usb kernel: All buffers synced. May 7 07:11:36 usb kernel: Uptime: 2m23s May 7 07:11:36 usb kernel: usbus0: Controller shutdown May 7 07:11:36 usb kernel: uhub0: at usbus0, port 1, addr 1 (disconnected) May 7 07:11:36 usb kernel: usbus0: Controller shutdown complete May 7 07:11:36 usb kernel: usbus1: Controller shutdown May 7 07:11:36 usb kernel: uhub1: at usbus1, port 1, addr 1 (disconnected) May 7 07:11:36 usb kernel: usbus1: Controller shutdown complete May 7 07:11:36 usb kernel: usbus2: Controller shutdown May 7 07:11:36 usb kernel: uhub2: at usbus2, port 1, addr 1 (disconnected) May 7 07:11:36 usb kernel: usbus2: Controller shutdown complete May 7 07:11:36 usb kernel: usbus3: Controller shutdown May 7 07:11:36 usb kernel: uhub3: at usbus3, port 1, addr 1 (disconnected) May 7 07:11:36 usb kernel: usbus3: Controller shutdown complete May 7 07:11:36 usb kernel: usbus4: Controller shutdown ##### ##### -- Manual use of reset or power switch required here to complete ##### May 7 07:11:36 usb kernel: Copyright (c) 1992-2012 The FreeBSD Project. May 7 07:11:36 usb kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 May 7 07:11:36 usb kernel: The Regents of the University of California. All rights reserved. May 7 07:11:36 usb kernel: FreeBSD is a registered trademark of The FreeBSD Foundation. May 7 07:11:36 usb kernel: FreeBSD 9.0-STABLE #0: Mon May 7 00:17:49 PDT 2012 May 7 07:11:36 usb kernel: jeff@usb.wagsky.com:/usr/obj/usr/src/sys/GENERIC amd64 May 7 07:11:36 usb kernel: CPU: Intel(R) Atom(TM) CPU 330 @ 1.60GHz (1600.03-MHz K8-class CPU) May 7 07:11:36 usb kernel: Origin = "GenuineIntel" Id = 0x106c2 Family = 6 Model = 1c Stepping = 2 May 7 07:11:36 usb kernel: Features=0xbfe9fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> May 7 07:11:36 usb kernel: Features2=0x40e31d<SSE3,DTES64,MON,DS_CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE> May 7 07:11:36 usb kernel: AMD Features=0x20100800<SYSCALL,NX,LM> May 7 07:11:36 usb kernel: AMD Features2=0x1<LAHF> May 7 07:11:36 usb kernel: TSC: P-state invariant, performance statistics May 7 07:11:36 usb kernel: real memory = 2147483648 (2048 MB) May 7 07:11:36 usb kernel: avail memory = 2037760000 (1943 MB) May 7 07:11:36 usb kernel: Event timer "LAPIC" quality 400 May 7 07:11:36 usb kernel: ACPI APIC Table: <A_M_I_ OEMAPIC > May 7 07:11:36 usb kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs May 7 07:11:36 usb kernel: FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads May 7 07:11:36 usb kernel: cpu0 (BSP): APIC ID: 0 May 7 07:11:36 usb kernel: cpu1 (AP/HT): APIC ID: 1 May 7 07:11:36 usb kernel: cpu2 (AP): APIC ID: 2 May 7 07:11:36 usb kernel: cpu3 (AP/HT): APIC ID: 3 May 7 07:11:36 usb kernel: ioapic0: Changing APIC ID to 4 May 7 07:11:36 usb kernel: ioapic0 <Version 2.0> irqs 0-23 on motherboard May 7 07:11:36 usb kernel: kbd1 at kbdmux0 May 7 07:11:36 usb kernel: ctl: CAM Target Layer loaded May 7 07:11:36 usb kernel: acpi0: <A_M_I_ OEMXSDT> on motherboard May 7 07:11:36 usb kernel: acpi0: Power Button (fixed) May 7 07:11:36 usb kernel: acpi0: reservation of 0, a0000 (3) failed May 7 07:11:36 usb kernel: acpi0: reservation of 100000, 7f700000 (3) failed May 7 07:11:36 usb kernel: cpu0: <ACPI CPU> on acpi0 May 7 07:11:36 usb kernel: cpu1: <ACPI CPU> on acpi0 May 7 07:11:36 usb kernel: cpu2: <ACPI CPU> on acpi0 May 7 07:11:36 usb kernel: cpu3: <ACPI CPU> on acpi0 May 7 07:11:36 usb kernel: attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0 May 7 07:11:36 usb kernel: Timecounter "i8254" frequency 1193182 Hz quality 0 May 7 07:11:36 usb kernel: Event timer "i8254" frequency 1193182 Hz quality 100 May 7 07:11:36 usb kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 May 7 07:11:36 usb kernel: Event timer "RTC" frequency 32768 Hz quality 0 May 7 07:11:36 usb kernel: hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 May 7 07:11:36 usb kernel: Timecounter "HPET" frequency 14318180 Hz quality 950 May 7 07:11:36 usb kernel: Event timer "HPET" frequency 14318180 Hz quality 450 May 7 07:11:36 usb kernel: Event timer "HPET1" frequency 14318180 Hz quality 440 May 7 07:11:36 usb kernel: Event timer "HPET2" frequency 14318180 Hz quality 440 May 7 07:11:36 usb kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 May 7 07:11:36 usb kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 May 7 07:11:36 usb kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 May 7 07:11:36 usb kernel: pci0: <ACPI PCI bus> on pcib0 May 7 07:11:36 usb kernel: vgapci0: <VGA-compatible display> port 0xb800-0xb807 mem 0xefd00000-0xefd7ffff,0xd0000000-0xdfffffff,0xefd80000-0xefdbffff irq 16 at device 2.0 on pci0 May 7 07:11:36 usb kernel: agp0: <Intel 82945G (945G GMCH) SVGA controller> on vgapci0 May 7 07:11:36 usb kernel: agp0: aperture size is 256M, detected 7932k stolen memory May 7 07:11:36 usb kernel: hdac0: <Intel 82801G HDA Controller> mem 0xefdf8000-0xefdfbfff irq 19 at device 27.0 on pci0 May 7 07:11:36 usb kernel: pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 May 7 07:11:36 usb kernel: pci3: <ACPI PCI bus> on pcib1 May 7 07:11:36 usb kernel: pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0 May 7 07:11:36 usb kernel: pci2: <ACPI PCI bus> on pcib2 May 7 07:11:36 usb kernel: re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F PCIe Gigabit Ethernet> port 0xe800-0xe8ff mem 0xeffff000-0xefffffff irq 17 at device 0.0 on pci2 May 7 07:11:36 usb kernel: re0: Using 1 MSI message May 7 07:11:36 usb kernel: re0: Chip rev. 0x38000000 May 7 07:11:36 usb kernel: re0: MAC rev. 0x00000000 May 7 07:11:36 usb kernel: miibus0: <MII bus> on re0 May 7 07:11:36 usb kernel: rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0 May 7 07:11:36 usb kernel: rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow May 7 07:11:36 usb kernel: re0: Ethernet address: 00:26:18:bc:9e:44 May 7 07:11:36 usb kernel: uhci0: <Intel 82801G (ICH7) USB controller USB-A> port 0xb880-0xb89f irq 20 at device 29.0 on pci0 May 7 07:11:36 usb kernel: uhci0: LegSup = 0x2f00 May 7 07:11:36 usb kernel: usbus0 on uhci0 May 7 07:11:36 usb kernel: uhci1: <Intel 82801G (ICH7) USB controller USB-B> port 0xbc00-0xbc1f irq 17 at device 29.1 on pci0 May 7 07:11:36 usb kernel: uhci1: LegSup = 0x2f00 May 7 07:11:36 usb kernel: usbus1 on uhci1 May 7 07:11:36 usb kernel: uhci2: <Intel 82801G (ICH7) USB controller USB-C> port 0xc000-0xc01f irq 18 at device 29.2 on pci0 May 7 07:11:36 usb kernel: uhci2: LegSup = 0x2f00 May 7 07:11:36 usb kernel: usbus2 on uhci2 May 7 07:11:36 usb kernel: uhci3: <Intel 82801G (ICH7) USB controller USB-D> port 0xc080-0xc09f irq 19 at device 29.3 on pci0 May 7 07:11:36 usb kernel: uhci3: LegSup = 0x2f00 May 7 07:11:36 usb kernel: usbus3 on uhci3 May 7 07:11:36 usb kernel: ehci0: <Intel 82801GB/R (ICH7) USB 2.0 controller> mem 0xefdffc00-0xefdfffff irq 20 at device 29.7 on pci0 May 7 07:11:36 usb kernel: usbus4: EHCI version 1.0 May 7 07:11:36 usb kernel: usbus4 on ehci0 May 7 07:11:36 usb kernel: pcib3: <ACPI PCI-PCI bridge> at device 30.0 on pci0 May 7 07:11:36 usb kernel: pci1: <ACPI PCI bus> on pcib3 May 7 07:11:36 usb kernel: em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.3> port 0xdc00-0xdc3f mem 0xefee0000-0xefefffff,0xefec0000-0xefedffff irq 17 at device 0.0 on pci1 May 7 07:11:36 usb kernel: em0: Ethernet address: 00:1b:21:1d:f4:ec May 7 07:11:36 usb kernel: isab0: <PCI-ISA bridge> at device 31.0 on pci0 May 7 07:11:36 usb kernel: isa0: <ISA bus> on isab0 May 7 07:11:36 usb kernel: atapci0: <Intel ICH7 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0 May 7 07:11:36 usb kernel: ata0: <ATA channel> at channel 0 on atapci0 May 7 07:11:36 usb kernel: atapci1: <Intel ICH7 SATA300 controller> port 0xcc00-0xcc07,0xc880-0xc883,0xc800-0xc807,0xc480-0xc483,0xc400-0xc40f irq 23 at device 31.2 on pci0 May 7 07:11:36 usb kernel: ata2: <ATA channel> at channel 0 on atapci1 May 7 07:11:36 usb kernel: ata3: <ATA channel> at channel 1 on atapci1 May 7 07:11:36 usb kernel: pci0: <serial bus, SMBus> at device 31.3 (no driver attached) May 7 07:11:36 usb kernel: acpi_button0: <Power Button> on acpi0 May 7 07:11:36 usb kernel: atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 May 7 07:11:36 usb kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0 May 7 07:11:36 usb kernel: kbd0 at atkbd0 May 7 07:11:36 usb kernel: atkbd0: [GIANT-LOCKED] May 7 07:11:36 usb kernel: psm0: <PS/2 Mouse> irq 12 on atkbdc0 May 7 07:11:36 usb kernel: psm0: [GIANT-LOCKED] May 7 07:11:36 usb kernel: psm0: model IntelliMouse Explorer, device ID 4 May 7 07:11:36 usb kernel: uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 May 7 07:11:36 usb kernel: ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0 May 7 07:11:36 usb kernel: ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode May 7 07:11:36 usb kernel: ppbus0: <Parallel port bus> on ppc0 May 7 07:11:36 usb kernel: lpt0: <Printer> on ppbus0 May 7 07:11:36 usb kernel: lpt0: Interrupt-driven port May 7 07:11:36 usb kernel: ppi0: <Parallel I/O> on ppbus0 May 7 07:11:36 usb kernel: plip0: <PLIP network interface> on ppbus0 May 7 07:11:36 usb kernel: orm0: <ISA Option ROM> at iomem 0xcb000-0xcbfff on isa0 May 7 07:11:36 usb kernel: sc0: <System console> at flags 0x100 on isa0 May 7 07:11:36 usb kernel: sc0: VGA <16 virtual consoles, flags=0x300> May 7 07:11:36 usb kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 May 7 07:11:36 usb kernel: p4tcc0: <CPU Frequency Thermal Control> on cpu0 May 7 07:11:36 usb kernel: p4tcc1: <CPU Frequency Thermal Control> on cpu1 May 7 07:11:36 usb kernel: p4tcc2: <CPU Frequency Thermal Control> on cpu2 May 7 07:11:36 usb kernel: p4tcc3: <CPU Frequency Thermal Control> on cpu3 May 7 07:11:36 usb kernel: Timecounters tick every 1.000 msec May 7 07:11:36 usb kernel: hdacc0: <VIA VT1708S_4 HDA CODEC> at cad 0 on hdac0 May 7 07:11:36 usb kernel: hdaa0: <VIA VT1708S_4 HDA CODEC Audio Function Group> at nid 1 on hdacc0 May 7 07:11:36 usb kernel: pcm0: <VIA VT1708S_4 HDA CODEC PCM (Analog 2.0+HP/2.0)> at nid 28,29 and 26,31,30,27 on hdaa0 May 7 07:11:36 usb kernel: usbus0: 12Mbps Full Speed USB v1.0 May 7 07:11:36 usb kernel: usbus1: 12Mbps Full Speed USB v1.0 May 7 07:11:36 usb kernel: usbus2: 12Mbps Full Speed USB v1.0 May 7 07:11:36 usb kernel: usbus3: 12Mbps Full Speed USB v1.0 May 7 07:11:36 usb kernel: usbus4: 480Mbps High Speed USB v2.0 May 7 07:11:36 usb kernel: ugen0.1: <Intel> at usbus0 May 7 07:11:36 usb kernel: uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 May 7 07:11:36 usb kernel: ugen1.1: <Intel> at usbus1 May 7 07:11:36 usb kernel: uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 May 7 07:11:36 usb kernel: ugen2.1: <Intel> at usbus2 May 7 07:11:36 usb kernel: uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2 May 7 07:11:36 usb kernel: ugen3.1: <Intel> at usbus3 May 7 07:11:36 usb kernel: uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3 May 7 07:11:36 usb kernel: ugen4.1: <Intel> at usbus4 May 7 07:11:36 usb kernel: uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4 May 7 07:11:36 usb kernel: uhub0: 2 ports with 2 removable, self powered May 7 07:11:36 usb kernel: uhub1: 2 ports with 2 removable, self powered May 7 07:11:36 usb kernel: uhub2: 2 ports with 2 removable, self powered May 7 07:11:36 usb kernel: uhub3: 2 ports with 2 removable, self powered May 7 07:11:36 usb kernel: ada0 at ata2 bus 0 scbus1 target 0 lun 0 May 7 07:11:36 usb kernel: ada0: <M4-CT064M4SSD2 0309> ATA-9 SATA 3.x device May 7 07:11:36 usb kernel: ada0: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes) May 7 07:11:36 usb kernel: ada0: 61057MB (125045424 512 byte sectors: 16H 63S/T 16383C) May 7 07:11:36 usb kernel: ada0: Previously was known as ad4 May 7 07:11:36 usb kernel: SMP: AP CPU #2 Launched! May 7 07:11:36 usb kernel: SMP: AP CPU #1 Launched! May 7 07:11:36 usb kernel: SMP: AP CPU #3 Launched! May 7 07:11:36 usb kernel: Root mount waiting for: usbus4 May 7 07:11:36 usb kernel: uhub4: 8 ports with 8 removable, self powered May 7 07:11:36 usb kernel: Root mount waiting for: usbus4 May 7 07:11:36 usb kernel: ugen4.2: <Generic> at usbus4 May 7 07:11:36 usb kernel: umass0: <Generic USB2.0-CRW, class 0/0, rev 2.00/19.81, addr 2> on usbus4 May 7 07:11:36 usb kernel: umass0: SCSI over Bulk-Only; quirks = 0x4000 May 7 07:11:36 usb kernel: umass0:4:0:-1: Attached to scbus4 May 7 07:11:36 usb kernel: Trying to mount root from ufs:/dev/ada0p2 [rw]... May 7 07:11:36 usb kernel: (probe0:umass-sim0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 May 7 07:11:36 usb kernel: (probe0:umass-sim0:0:0:0): CAM status: SCSI Status Error May 7 07:11:36 usb kernel: (probe0:umass-sim0:0:0:0): SCSI status: Check Condition May 7 07:11:36 usb kernel: (probe0:umass-sim0:0:0:0): SCSI sense: UNIT ATTENTION asc:28,0 (Not ready to ready change, medium may have changed) May 7 07:11:36 usb kernel: da0 at umass-sim0 bus 0 scbus4 target 0 lun 0 May 7 07:11:36 usb kernel: da0: <Generic- SD/MMC 1.00> Removable Direct Access SCSI-0 device May 7 07:11:36 usb kernel: da0: 40.000MB/s transfers May 7 07:11:36 usb kernel: da0: 1962MB (4019200 512 byte sectors: 255H 63S/T 250C) May 7 07:11:36 usb kernel: ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; May 7 07:11:36 usb kernel: to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. May 7 07:11:36 usb kernel: ZFS filesystem version 5 May 7 07:11:36 usb kernel: ZFS storage pool version 28 May 7 07:11:37 usb kernel: .
Responsible Changed From-To: freebsd-bugs->freebsd-fs Over to maintainer(s).
Problem can be replicated by booting of a "memstick" (with a "spare" USB stick as /dev/da1) and then executing # dd if=/dev/zer of=/dev/da1 bs=64k # zpool create stick /dev/da1 # reboot Problem has been reliably reproduced on the Atom 330 previously mentioned, as well as on an AMD A8-3870 with A75 chipset. It also can be replicated using VirtualBox running under Ubuntu on the AMD A8-3870 system. It does not seem specific to one "flavor" of USB controller or driver. Using /usr/src/release/generate_release.sh and bisection, I have confirmed that * r227445 does not exhibit the behavior ("Copy stable/9 to releng/9.0 as part of the FreeBSD 9.0-RELEASE release cycle) * r229097 does not exhibit the behavior * r229281 -- FAIL by not rebooting under the conditions described above. Based on these results, I am suspicious of r229100 | hselasky | 2011-12-31 06:33:15 -0800 (Sat, 31 Dec 2011) | 6 lines MFC r228709, r228711 and r228723: - Add missing unlock of USB controller's lock, when doing shutdown, suspend and resume. - Add code to wait for USB shutdown to be executed at system shutdown. - Add sysctl which can be used to skip this waiting. as being what brought the issue to the forefront. I am presently building r229099 and r229100 to confirm this suspicion. A potential, though untested workaround would be # sysctl hw.usb.no_shutdown_wait=1
Not surprisingly: r229099 does *not* exhibit the symptom r229100 *does* exhibit the symptom # sysctl hw.usb.no_shutdown_wait=1 is confirmed as a workaround
Just a me too: FreeBSD fred 9.0-STABLE FreeBSD 9.0-STABLE #0 r237006: Wed Jun 13 20:38:08 BST 2012 root@fred:/usr/obj/usr/src/sys/GENERIC amd64 Setting hw.usb.no_shutdown_wait=1 shuts down properly. Without simply hangs.
FreeBSD 10.0-STABLE #0 r266463: Tue May 20 18:24:03 UTC 2014 and the issue is still here, but to repeat it need to be created zvol on some zfs pool, http://farm6.staticflickr.com/5506/14550830655_d517ab28f5_b.jpg - screenshot. the machine is shutting down fine, if no zvols present in system.
batch change: For bugs that match the following - Status Is In progress AND - Untouched since 2018-01-01. AND - Affects Base System OR Documentation DO: Reset to open status. Note: I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
A slight detour from the focus on ZFS … hw.usb.no_shutdown_wait=1 – generally, is there _ever_ an elevated risk of loss of data when there's no wait?
I just hit this issue with 13.1-STABLE. I have zpools that include USB disks. On reboot, system hangs with messages like "Solaris: WARNING: Pool 'pool' has encountered an uncorrectable I/O failure and has been suspended." This happens after 'All buffers synced' when there is a series of messages about detaching usb devices, e.g.: "ukbd0: detached", "uhid0: detached", "umass0: detached". I also noticed that single user mode with all zfs partitions mounted (doing /etc/rc.d/zfs start manually after logging in) this does not happen - no usb detached messages and system reboots normally.
*** Bug 211491 has been marked as a duplicate of this bug. ***
(In reply to Jeff Kletsky from comment #0) Is this reproducible with any supported RELEASE, or branch, of FreeBSD? (In reply to Vadym Khondar from comment #8) > … 13.1-STABLE. … include USB disks … uncorrectable I/O failure … Reproducible with more recent STABLE? I have never encountered the issue with FreeBSD 14.0-CURRENT. Can you provide additional information about your hardware? Output from commands such as these: gpart show geom disk list zpool list -v Thanks
In response to the direct query, I have not seen this issue in several years. However, as hardware configurations here have changed significantly, I can not confirm that the problem does not exist with current releases.
(In reply to Jeff Kletsky from comment #11) Thanks for the feedback, and for your patience. All things considered: let's close this report. If anyone finds recurrence of symptoms, we might reopen and reassign to fs@
Please reopen. I am seeing this bug today on 13.2-STABLE. I have a ZFS pool using five disks in a Sabrent enclosure. In terms of drivers, uhub5 on uhub0 uhub5: <VIA Labs, Inc. USB3.1 Hub, class 9/0, rev 3.20/90.13, addr 1> on usbus0 uhub5: 4 ports with 4 removable, self powered usb_msc_auto_quirk: UQ_MSC_NO_GETMAXLUN set for USB mass storage device ASMT ASM235CM (0x174c:0x55aa) ugen0.3: <ASMT ASM235CM> at usbus0 umass0 on uhub5 umass0: <ASMT ASM235CM, class 0/0, rev 3.10/1.00, addr 2> on usbus0 umass0: SCSI over Bulk-Only; quirks = 0x0100 umass0:20:0: Attached to scbus20 da0 at umass-sim0 bus 0 scbus20 target 0 lun 0 da0: <ASMT ASM235CM 0> Fixed Direct Access SPC-4 SCSI device da0: Serial Number 915000000C05 da0: 400.000MB/s transfers da0: 5723166MB (11721045168 512 byte sectors) da0: quirks=0x2<NO_6_BYTE> Other devices appear as uhub6 on uhub5 umass1 on uhub6 [= da1] umass2 on uhub6 [= da2] umass3 on uhub6 [= da3] umass4 on uhub6 [= da4] I infer the enclosure has a hub connected to a disk and a second hub. The second hub has the other 4 disks. When I reboot the system I see (retyped from photo) All buffers synced. Uptime: 3d3h57m14s uhub0: detached ukbd0: detached ... umass0: detached umass1: detached Solaris: WARNING: Pool 'sabrent' has encountered an uncorrectable I/O failure and has been suspended. Then the system hangs. Within the not especially long limit of my patience there is no more output. No detach message for uhub5, uhub6, umass2, umass3, and umass4. I have failmode=continue on this pool precisely to avoid having the system hang. There might be two bugs, shutdown order and failure to respect failmode=continue. This is repeatable.
Seen again on 13.2-STABLE
If I run "zpool export sabrent" on the pool described in my previous comment the command takes about 7 seconds to complete. ^T reports load: 7.17 cmd: zpool 66858 [tx->tx_sync_done_cv] 4.07r 0.00u 2.54s 20% 8008k mi_switch+0x152 sleepq_switch+0x104 _cv_wait+0x14a txg_wait_synced_impl+0xeb txg_wait_synced+0xb zil_close+0x128 zfsvfs_teardown+0xb1 zfs_umount+0x129 dounmount+0x396 kern_unmount+0x312 amd64_syscall+0x13c fast_syscall_common+0xf8 load: 6.76 cmd: zpool 66858 [running] 7.24r 0.00u 3.14s 20% 8024k uma_zfree_arg+0xd6 abd_free_chunks+0x44 abd_free+0x45 arc_hdr_free_abd+0x144 arc_evict_state+0xaca arc_flush+0x8c dsl_pool_close+0xe3 spa_unload+0x349 spa_export_common+0x2c8 zfsdev_ioctl_common+0x5ce zfsdev_ioctl+0x12a devfs_ioctl+0xd2 vn_ioctl+0x136 devfs_ioctl_f+0x1e kern_ioctl+0x286 sys_ioctl+0x140 amd64_syscall+0x13c fast_syscall_common+0xf8 I see after filesystems are unmounted there is still some ARC cleanup to be done. The shutdown path has a call to vfs_unmountall in bufshutdown. Does it have a call out to ZFS to quiesce pools? The hang is reproducible so I am able to test a fix if anybody has one.
Can confirm this is still on FreeBSD 14.1-p5. Output requested as per comment #10: # gpart show => 40 8000416 ada0 GPT (3.8G) 40 532480 1 efi (260M) 532520 7065600 2 freebsd-ufs (3.4G) 7598120 399360 3 freebsd-swap (195M) 7997480 2976 - free - (1.5M) => 40 976773088 da0 GPT (466G) 40 4056 - free - (2.0M) 4096 976281600 1 freebsd-zfs (466G) 976285696 487432 - free - (238M) => 40 976773088 da1 GPT (466G) 40 4056 - free - (2.0M) 4096 976281600 1 freebsd-zfs (466G) 976285696 487432 - free - (238M) # geom disk list Geom name: ada0 Providers: Geom name: da0 Providers: 1. Name: da0 Mediasize: 500107862016 (466G) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e3 descr: WDC WD50 00LPLX-66ZNT lunname: WDC WD5000LPLX-66ZNT 02.0 lunid: WDC WD5000LPLX-66ZNT 02.0 ident: 0020041324C6 rotationrate: unknown fwsectors: 63 fwheads: 255 Geom name: da1 Providers: 1. Name: da1 Mediasize: 500107862016 (466G) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e3 descr: WDC WD50 00LPLX-66ZNT lunname: WDC WD5000LPLX-66ZNT 01.0 lunid: WDC WD5000LPLX-66ZNT 01.0 ident: 0020041325C3 rotationrate: unknown fwsectors: 63 fwheads: 255 # zpool list -v NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT storage 464G 50.6G 413G - - 23% 10% 1.00x ONLINE - mirror-0 464G 50.6G 413G - - 23% 10.9% - ONLINE gpt/WD-WX91AC56DFUZ 466G - - - - - - - ONLINE gpt/WD-WXR1AB6AP8HH 466G - - - - - - - ONLINE On reboot the attached message appears.
Created attachment 254343 [details] Screenshot of error on FreeBSD 14.1-p5 error showing it is still happening on FreeBSD 14.1-p5
Also have the following in /boot/loader.conf: zfs_load="YES"
In my simple ZFS context I'v enver gotten such behavior. Here I'm using a da0 based UFS boot and later adding da1 and importing its pool: # gpart show -p da1 => 40 1875384928 da1 GPT (894G) 40 532480 da1p1 efi (260M) 532520 8152 - free - (4.0M) 540672 62914560 da1p2 freebsd-swap (30G) 63455232 184549376 da1p4 freebsd-swap (88G) 248004608 20971520 - free - (10G) 268976128 1468006400 da1p3 freebsd-zfs (700G) 1736982528 138402440 - free - (66G) # zpool import pool: zwdk23 id: 8373431646426325674 state: ONLINE status: Some supported features are not enabled on the pool. (Note that they may be intentionally disabled if the 'compatibility' property is set.) action: The pool can be imported using its name or numeric identifier, though some features will not be available without an explicit 'zpool upgrade'. config: zwdk23 ONLINE gpt/wdk23CA78Czfs ONLINE Note: I historically use a /usr/share/zfs/compatibility.d/openzfs-2.*-freebsd as the feature set -- or the likes of /usr/share/zfs/compatibility.d/openzfs-2.2 when there is no linux vs. freebsd distinction for where I have things set. Thus main tends to produce the some-not-enabled message. # zpool import -t -f -R /zptemp-mnt/ zwdk23 zptemp # zpool list -v NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zptemp 696G 574G 122G - - 20% 82% 1.00x ONLINE /zptemp-mnt/ gpt/wdk23CA78Czfs 700G 574G 122G - - 20% 82.5% - ONLINE # zfs list # shortened listing via ". . ." for omissions NAME USED AVAIL REFER MOUNTPOINT zptemp 574G 100G 96K /zptemp-mnt//zwdk23 . . . zptemp/ROOT 4.94G 100G 96K none zptemp/ROOT/build_area_for-main-CA76 364K 100G 3.00G none zptemp/ROOT/main-CA76 4.94G 100G 2.88G none zptemp/poudriere 516G 100G 112K /zptemp-mnt//usr/local/poudriere zptemp/poudriere/data 516G 100G 128K /zptemp-mnt//usr/local/poudriere/data zptemp/poudriere/data/.m 112K 100G 112K /zptemp-mnt//usr/local/poudriere/data/.m zptemp/poudriere/data/cache 16.4M 100G 16.4M /zptemp-mnt//usr/local/poudriere/data/cache . . . zptemp/poudriere/ports 96K 100G 96K /zptemp-mnt//usr/local/poudriere/ports zptemp/tmp 14.8M 100G 14.8M /zptemp-mnt//tmp zptemp/usr 13.9G 100G 96K /zptemp-mnt//usr . . . zptemp/usr/home 906M 100G 906M /zptemp-mnt//usr/home zptemp/usr/local 2.37G 100G 2.37G /zptemp-mnt//usr/local . . . zptemp/usr/ports 6.81G 100G 6.81G /zptemp-mnt//usr/ports zptemp/usr/src 96K 100G 96K /zptemp-mnt//usr/src zptemp/var 47.0M 100G 96K /zptemp-mnt//var zptemp/var/audit 360K 100G 360K /zptemp-mnt//var/audit zptemp/var/crash 184K 100G 184K /zptemp-mnt//var/crash zptemp/var/db 42.0M 100G 96K /zptemp-mnt//var/db zptemp/var/db/pkg 41.8M 100G 41.8M /zptemp-mnt//var/db/pkg zptemp/var/db/ports 148K 100G 148K /zptemp-mnt//var/db/ports zptemp/var/log 2.75M 100G 2.75M /zptemp-mnt//var/log zptemp/var/mail 1.24M 100G 1.24M /zptemp-mnt//var/mail zptemp/var/tmp 404K 100G 404K /zptemp-mnt//var/tmp Over the years I've never had a problem with shutdown or reboot for such a context. But I've not used mirroring or other such. I experimented with ZFS to experiment with bectl use, not other usage that is typical.
I'm also getting the Pool 'XXXX' has encountered an uncorrectable I/O failure and has been suspended during shutdown. Half my bulk storage pool is on USB devices and the USB subsystem appears to get shutdown before the pool is fully flushed.