This is in response to Warner's closure of bug 205166 The test host I used was a Pi3 booted from a usb mechanical hard drive connected via a powered hub running FreeBSD pelorus.zefox.org 14.0-RELEASE-p4 FreeBSD 14.0-RELEASE-p4 #0 releng/14.0-n265400-4edf3b80733e: Wed Dec 27 20:21:26 PST 2023 bob@pelorus:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64 The first test was to connect a former boot disk similar to the one the machine was currently running from. No problem, normal device discovery chatter on the console. Next experiment was to connect a usb-serial adapter with an old Raspian microSD already plugged into it. Response was an immediate reboot, from the Raspian microSD Las experiment was to plug in an FTDI usb-serial adapter, resulting in login: ugen1.4: <GenesysLogic USB2.1 Hub> at usbus1 (disconnected) (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 27 e0 b0 28 00 00 18 00 uhub2: at uhub1, port 2, addr 4 (disconnected) (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error (da0:umass-sim0:0:0:0): Retrying command, 3 more tries remain ugen1.5: <JMicron SABRENT> at usbus1 (disconnected) (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 27 e0 b0 28 00 00 18 00 umass0: at uhub2, port 2, addr 5 (disconnected) (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error (da0:umass-sim0:0:0:0): Retrying command, 2 more tries remain da0 at umass-sim0 bus 0 scbus0 target 0 lun 0 da0: <SABRENT 1214> s/n 000000000000A detached g_vfs_done(): da0p2 converting all errors to ENXIO g_vfs_done():da0p2[WRITE(offset=342274080768, length=12288)]error = 6 supressing further ENXIO panic: UFS: root fs would be forcibly unmounted cpuid = 0 time = 1705223363 KDB: stack backtrace: #0 0xffff00000050d52c at kdb_backtrace+0x58 #1 0xffff0000004b9044 at vpanic+0x19c #2 0xffff0000004b8ea4 at panic+0x44 #3 0xffff0000007fd290 at ffs_fsfail_cleanup+0x178 #4 0xffff0000007d1748 at ffs_update+0x300 #5 0xffff00000080534c at ffs_syncvnode+0x62c #6 0xffff000000803ce4 at ffs_fsync+0x28 #7 0xffff0000005cdd84 at kern_fsync+0x1b0 #8 0xffff000000896628 at do_el0_sync+0x5b0 #9 0xffff000000874110 at handle_el0_sync+0x44 Uptime: 6m28s Resetting system ...had to pull the plug. All three tests were done via the powered usb hub. I'll add that even Pi4 RasPiOS isn't immune to such mischief. Plugging in the same usb-microSD adapter with a card inserted stops the mouse and keyboard working, with power cycling the easiest recovery. Plugging in the adapter without a card and then inserting the card allows normal recognition. Haven't tried that with FreeBSD recently, it used to work also. The FT232 also crashed RasPiOS, disabling keyboard and mouse. The RasPiOS experiments didn't use a powered hub, but neither device draws much power. USB still seems a work in progress.
So there's several things going on here. First, da0 is disappearing because the usb system thinks that it has gone away. Since root was mounted on it, it's a fatal error to the system. Given that it went away, the panic is expected and not a bug. We cannot run without a root filesystem. Second, the system didn't reset properly once the panic happened. This is indeed a bug. It should do so. On the email thread, I suggested it might be sitting at the ddb> prompt, but that was before I saw the 'resetting system' message. Third, we shouldn't be detaching USB when something else is plugged in. However, there's a problem: RPi are well known for being quite picky about power and other devices dropping off the bus is a well known reaction to increased load. It seems to be a hardware problem, given that Linux (?) is having similar issues. If there's a repeatable test case, with a powered hub, then we might have a bug in USB somehow, though I consider that unlikely given Raspian reboots as well. This suggests that there's nothing wrong in hardware with the reset, and that FreeBSD is doing something wrong and not resetting, though what I can't say.
I'm confused: QUOTE Next experiment was to connect a usb-serial adapter with an old Raspian microSD already plugged into it END QUOTE I've never seen a usb-serial adapter than had its own microSD card slot. Was it a reader/writer for microsd cards instead of a serial cable connection device?
(In reply to Warner Losh from comment #1) QUOTE RPi are well known for being quite picky about power and other devices dropping off the bus is a well known reaction to increased load. END QUOTE I think that Bob P. is hoping that the powered hub use and plugging into the powered hub instead avoids that issue with RPi3B and before. (Not as much of a problem for RPi4B's.) The reference to a "Pi4" also having an issue leads me to wonder what the load is like as seen by the RPi4B and what power supply is in use for the RPi4B. (I use a 5.1V 3.5A power supply, somewhat more margin than the official 5.1V 3.0A power supply.) If a RPi4B has problems with the load, I doubt that a RPi3B has any chance of handling the load (if it is a loading issue).
(In reply to Bob Prohaska from comment #0) QUOTE The first test was to connect a former boot disk similar to the one the machine was currently running from. No problem, normal device discovery chatter on the console. Next experiment was to connect a usb-serial adapter with an old Raspian microSD already plugged into it. Response was an immediate reboot, from the Raspian microSD END QUOTE It is unclear if you unplugged the "former boot disk" first vs. if you just added yet another device to the two disks already connected (boot and former boot). Also: was the powered hub the only USB device directly plugged into the RPi3B, other than what is built-in (such as the built-in RPi3B Ethernet)? I assume that the "from the Raspian microSD" is just indicating the content of the microsd that you happened to plug in, not that Raspian was involved in the FreeBSD issue. If any of the drives are spinning rust, what are the startup current requirements for the drive(s) to get things spinning and such? How does it compare to the powered-hub's power ratings for the load(s) it supports?
(In reply to Mark Millard from comment #2) The adapter was usb-microSD, with an old Raspbian microSD inserted. The machine crashed and rebooted before I could get back to the console. Perhaps naturally, the reboot started from the Raspbian microSD, probably because it was discovered first. A later test involved an FTDI usb-serial adapter.
Focus. None of the commentary so far is remotely actionable.
(In reply to Warner Losh from comment #6) I think the problem here is how to get to something remotely actionable. Suggestions? May be something for investigating the potential "FreeBSD is doing something wrong and not resetting" (even if it is not the overall solution of itself)?
(In reply to Mark Millard from comment #7) So far "not resetting" is the only bug that's been disclosed. Raspian and FreeBSD have the same behavior when a serial usb dongle is plugged in, so there's nothing actionable there. The panic when root goes away is expected behavior, and there's no bug to fix. All that's left is reboot doesn't work, which I can't recreate.... which makes it really hard to even know where to start.
(In reply to Warner Losh from comment #8) Isn't having root go away when an elsewhere-functional usb device unrelated to storage is connected an actionable event? FTDI usb-serial adapters are generally well-regarded. In principle it shouldn't be hard to record power supply voltage during the experiment. I'm willing to try if there's interest. Other suggestions are welcome.
(In reply to Bob Prohaska from comment #9) > Isn't having root go away when an elsewhere-functional usb device unrelated to storage is connected an actionable event? FreeBSD has no support for that today, and is unlikely to grow support for that due to the large effort it qould require. We currently panic rather than do a forced unmount. That's what's supported and we did that. It might be a nice to have feature, but where would you go? Unless you configured an alternate root in advance, the system could read any new files in. And if you are configuring such a root, then you could just as easily reboot -r to re-root to the ram disk so that the root could never go away. And if root goes away, what happens to all the other mount mounts? It's really a thorny problem. > In principle it shouldn't be hard to record power supply voltage during the experiment. I'm willing to try if there's interest. I think you'd have to connect a scope to the power supply to make sure it doesn't brown out. And even then, you'd likely need to connect something to the USB bus that can measure these things as well as record the transactions so that we understand what caused the da0 device to drop off the bus. But if this is also happens with Raspian or similar other OSes, it might not give us something we can work with. At best, it would confirm or refute the suspected hardware issue. I don't think a voltage measurement would tell us what to do, and given the USB bus traces I've looked at in the past, I'm skeptical that it would either (in all likelihood the da device is seeing a brownout and reinitializing and reacting to any commands to it like an uninitialized device would, but my memory of what those things are on the USB bus is dim at best).
(In reply to Warner Losh from comment #10) It seems like there might be an undeclared constraint that I'm not aware of. The idea that baffles me is "Unless you configured an alternate root in advance, the system could read any new files in." Is it a concern about eavesdropping by the added device? If that isn't the stumbling block alluded to please tell me what is. It's understood that servers in crewed datacenters are different from private workstations, and USB is insecure. Thanks for writing!
(In reply to Bob Prohaska from comment #5) I tried to replicate on a RPi3B (so: old) with no powered hub involved: Just the USB boot drive and Ethernet and the serial console connected before boot, no keyboard and no mouse (my usual configuration). Then, plugging in the reader/writer that had a microsd card already in place: usb_msc_auto_quirk: UQ_MSC_NO_TEST_UNIT_READY set for USB mass storage device Kingston Multi-Reader (0x11b0:0x6368) usb_msc_auto_quirk: UQ_MSC_NO_PREVENT_ALLOW set for USB mass storage device Kingston Multi-Reader (0x11b0:0x6368) usb_msc_auto_quirk: UQ_MSC_NO_SYNC_CACHE set for USB mass storage device Kingston Multi-Reader (0x11b0:0x6368) usb_msc_auto_quirk: UQ_MSC_NO_START_STOP set for USB mass storage device Kingston Multi-Reader (0x11b0:0x6368) ugen1.5: <Kingston Multi-Reader> at usbus1 umass1 on uhub1 umass1: <Bulk-In, Bulk-Out, Interface> on usbus1 umass1: SCSI over Bulk-Only; quirks = 0xc005 umass1:1:1: Attached to scbus1 da1 at umass-sim1 bus 1 scbus1 target 0 lun 0 da1: < Multi-Reader -0 1.00> Removable Direct Access SPC-2 SCSI device da1: Serial Number 08738181902155 da1: 40.000MB/s transfers da1: Attempt to query device size failed: NOT READY, Medium not present da1: quirks=0x2<NO_6_BYTE> da2 at umass-sim1 bus 1 scbus1 target 0 lun 1 da2: < Multi-Reader -1 1.00> Removable Direct Access SPC-2 SCSI device da2: Serial Number 08738181902155 da2: 40.000MB/s transfers da2: Attempt to query device size failed: NOT READY, Medium not present da2: quirks=0x2<NO_6_BYTE> da3 at umass-sim1 bus 1 scbus1 target 0 lun 2 da3: < Multi-Reader -2 1.00> Removable Direct Access SPC-2 SCSI device da3: Serial Number 08738181902155 da3: 40.000MB/s transfers da3: Attempt to query device size failed: NOT READY, Medium not present da3: quirks=0x2<NO_6_BYTE> da4 at umass-sim1 bus 1 scbus1 target 0 lun 3 da4: < Multi-Reader -3 1.00> Removable Direct Access SPC-2 SCSI device da4: Serial Number 08738181902155 da4: 40.000MB/s transfers da4: 30528MB (62521344 512 byte sectors) da4: quirks=0x2<NO_6_BYTE> So, no problem. Then plugging in an official RPi keyboard with an official RPi mouse already plugged in to one of its ports: ugen1.6: <vendor 0x05e3 USB2.0 Hub> at usbus1 uhub2 on uhub1 uhub2: <vendor 0x05e3 USB2.0 Hub, class 9/0, rev 2.00/32.98, addr 6> on usbus1 uhub2: MTT enabled uhub2: 4 ports with 4 removable, self powered ugen1.7: <vendor 0x04d9 RPI Wired Keyboard 4> at usbus1 ukbd0 on uhub2 ukbd0: <vendor 0x04d9 RPI Wired Keyboard 4, class 0/0, rev 2.00/1.40, addr 7> on usbus1 kbd1 at ukbd0 ugen1.8: <PixArt USB Optical Mouse> at usbus1 uhid0 on uhub2 uhid0: <vendor 0x04d9 RPI Wired Keyboard 4, class 0/0, rev 2.00/1.40, addr 7> on usbus1 uhid0: no report descriptor device_attach: uhid0 attach returned 12 ums0 on uhub2 ums0: <PixArt USB Optical Mouse, class 0/0, rev 1.10/1.00, addr 8> on usbus1 ums0: 3 buttons and [XYZ] coordinates ID=0 uhid0 on uhub2 uhid0: <vendor 0x04d9 RPI Wired Keyboard 4, class 0/0, rev 2.00/1.40, addr 7> on usbus1 uhid0: no report descriptor device_attach: uhid0 attach returned 12 Still no issue with the boot drive. I then unplugged the microsd card reader. Then I plugged it back in. I did not remove the microsd card from the reader at all. The keyboard and mouse were still plugged in to the RPi3B the whole time. Still no problem. (I'll not repeat the text.) The problem appears to be specific to your particular combination of equipment or to handling of powered hubs being in the chain. I may see about attempting use of a powered hub being involved in a while.
(In reply to Bob Prohaska from comment #5) So I found and tried using a powered hub. I had the boot drive and the keyboard plugged in the hub, and the mouse plugged into to the keyboard before booting. So only the powered hub, ethernet, and the serial console connection were at the RPi3B. It booted just fine. I then plugged in the microsd card reader that already had a microsd card in place. Again no problem. So, just having the powered hub stage did not change the behavior to be like you are reporting. The problem appears to be specific to your particular combination of equipment. I'll note that the boot media is a USB SSD that meets the USB2 power constraints but is USB3 capable. The microsd card reader/writer is also like that. In essence, my checks are an attempt to see if anything is broken as you report when staying inside the power constraints. I got no evidence of problems for a RPi3B. (So I doubt that a RPi4B would have problems.) Extending my prior point: The problem appears to be specific to your particular combination of equipment's power use, at least for the RPi3B context.
(In reply to Mark Millard from comment #13) Thank you for checking. I hope it's the power supply, which is replaceable. Apologies for the fuss!
(In reply to Bob Prohaska from comment #14) FYI: To be explicit, this being an RPi3B, its power supply was a standard RPi one. Its label shows: STONTRONICS Switching Adapter MODEL-DSA-13PFC-05 FCA 051250 P/N: T5875DV OUTPUT: +5.1V 2.5A