Bug 236191 - FreeBSD Pine64 images SD/eMMC problems
Summary: FreeBSD Pine64 images SD/eMMC problems
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: 12.0-RELEASE
Hardware: arm64 Any
: --- Affects Some People
Assignee: freebsd-arm mailing list
URL: https://forums.freebsd.org/threads/fr...
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2019-03-04 03:50 UTC by twistedpair
Modified: 2019-09-07 02:07 UTC (History)
3 users (show)

See Also:


Attachments
Boot log on Pine64 LTS (29.66 KB, text/plain)
2019-03-18 12:34 UTC, twistedpair
no flags Details
Verbose boot log (35.33 KB, text/plain)
2019-03-18 13:29 UTC, twistedpair
no flags Details
Verbose boot log with SD, not eMMC (31.58 KB, text/plain)
2019-03-18 13:55 UTC, twistedpair
no flags Details
Boot log with hw.regulator.disable_unused=0 (30.67 KB, text/plain)
2019-03-18 14:26 UTC, twistedpair
no flags Details
FreeBSD-13.0-CURRENT-arm64-aarch64-PINE64-LTS-20190314-r345110.log (14.74 KB, text/plain)
2019-03-18 23:49 UTC, Emmanuel Vadot
no flags Details
FreeBSD-12.0-RELEASE-arm64-aarch64-PINE64-LTS.log (13.14 KB, text/plain)
2019-03-18 23:53 UTC, Emmanuel Vadot
no flags Details
FreeBSD-13.0-CURRENT-arm64-aarch64-PINE64-LTS failed boot (12.63 KB, text/plain)
2019-03-19 03:41 UTC, twistedpair
no flags Details
FreeBSD-13.0-CURRENT-arm64-aarch64-PINE64-LTS successful boot (15.21 KB, text/plain)
2019-03-19 10:03 UTC, twistedpair
no flags Details
FreeBSD-13.0-CURRENT-arm64-aarch64-PINE64-LTS eMMC unsuccessful again (13.00 KB, text/plain)
2019-03-19 10:23 UTC, twistedpair
no flags Details
FreeBSD-13.0-CURRENT-etc with different eMMC (33.77 KB, text/plain)
2019-03-27 09:29 UTC, twistedpair
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description twistedpair 2019-03-04 03:50:55 UTC
This is moving [this forum post](https://forums.freebsd.org/threads/freebsd-pine64-images-wont-actually-boot-on-a-pine64.69540) across to make it a bug report:

I recently got a Pine64 LTS in order to try the official FreeBSD images , specifically FreeBSD-12.0-RELEASE-arm64-aarch64-PINE64-LTS. This goes through part of the boot process (the information scrolls off the screen too fast to see where it gets to, it's perhaps 3-4 screenfuls) and then the screen goes blank and the keyboard goes dead. There are occasional bursts of network traffic of some kind, but nothing useful such as the device requesting a DHCP address, and there's no difference when the USB keyboard and/or network cable are unplugged.

I retried with the current snapshot, FreeBSD-13.0-CURRENT-arm64-aarch64-PINE64-LTS-20190207-r343862, but got the same result. Others have reported similar issues, as per this thread in the Pine forums. Note that I can't use the RaspBSD image (which in any case seems to have the same issue) because that's for Pine64, not Pine64-LTS.

Does anyone know whether there's any official image for Pine64-LTS that works? Failing that, is there anything I can do to diagnose the problem?
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2019-03-18 01:51:09 UTC
@Twistedpair Could you try a verbose boot to see whether the output is more informative, and include the output in an attachment if possible (screenshot if necessary)
Comment 2 Emmanuel Vadot freebsd_committer 2019-03-18 02:16:14 UTC
If you have a uart adapter that would help too.
I guess that you are using an HDMI monitor, could you test another one ? (one with a lower max resolution could help, maybe ...).
Comment 3 twistedpair 2019-03-18 12:34:17 UTC
Created attachment 202954 [details]
Boot log on Pine64 LTS
Comment 4 twistedpair 2019-03-18 12:37:26 UTC
Here's the boot log, run twice just to be sure.  The screen goes dead at around the following extract, which is also where the boot process stops, full dump in the attachment:

CPU  0: ARM Cortex-A53 r0p4 affinity:  0
arc4random: no preloaded entropy cache
 Instruction Set Attributes 0 = <AES+PMULL,SHA1,SHA2,CRC32>
 Instruction Set Attributes 1 = <>
         Processor Features 0 = <AdvSIMD,Float,EL3 32,EL2 32,EL1 32,EL0 32>
Trying to mount root from ufs:/dev/ufs/rootfs [rw]...
         Processor Features 1 = <0>
      Memory Model Features 0 = <4k Granule,64k Granule,MixedEndian,S/NS Mem,16bit ASID,1TB PA>
AW_MMC_INT_DATA_CRC_ERR       Memory Model Features 1 = <>

      Memory Model Features 2 = <32b CCIDX,48b VA>
mmcsd0:              Debug Features 0 = <2 CTX Breakpoints,4 Watchpoints,6 Breakpoints,PMUv3,Debug v8>
             Debug Features 1 = <0>
         Auxiliary Features 0 = <0>
         Auxiliary Features 1 = <0>
Error indicated: 4 Failed
CPU  1: ARM Cortex-A53 r0p4 affinity:  1
AW_MMC_INT_DATA_CRC_ERR
CPU  2: ARM Cortex-A53 r0p4 affinity:  2
aw_mmc1: CPU  3: ARM Cortex-A53 r0p4 affinity:  3
mmcsd0: Spurious interrupt - no active request, rint: 0x00000000

Error indicated: 4 Failed
WARNING: WITNESS option enabled, expect reduced performance.
AW_MMC_INT_DATA_CRC_ERR
mmcsd0: Error indicated: 4 Failed
AW_MMC_INT_DATA_CRC_ERR
aw_mmc1: mmcsd0: Error indicated: 4 Failed
Spurious interrupt - no active request, rint: 0x00000000

AW_MMC_INT_DATA_CRC_ERR
aw_mmc1: mmcsd0: Error indicated: 4 Failed
Spurious interrupt - no active request, rint: 0x00000000

AW_MMC_INT_DATA_CRC_ERR
aw_mmc1: Spurious interrupt - no active request, rint: 0x00000000

AW_MMC_INT_DATA_CRC_ERR
aw_mmc1: Spurious interrupt - no active request, rint: 0x00000000

AW_MMC_INT_DATA_CRC_ERR
AW_MMC_INT_DATA_CRC_ERR
mountroot: waiting for device /dev/ufs/rootfs...
Mounting from ufs:/dev/ufs/rootfs failed with error 19.

[...]

panic: mountroot: unable to (re-)mount root.
Comment 5 Emmanuel Vadot freebsd_committer 2019-03-18 12:45:23 UTC
Well you seems to have a bad sdcard otherwise you wouldn't see CRC errors.
I suggest you try with another one, if you can from another batch if you bought multiple sd at once.
Comment 6 twistedpair 2019-03-18 12:54:37 UTC
It's eMMC, not an SD card.  Discussions I've seen elsewhere (can't find the thread at the moment) indicates it could be problems with the Allwinner eMMC driver.  It seems to be somewhat hairy in general, see e.g. the descussion here:

https://reviews.freebsd.org/D15891

which contains among other things:

-- Snip --

08:13 < kibab> And one more thing (c): on AllWinner, MMC devices are not enumerated at boot yet.
08:13 < kevans91> Ah, ok, that explains that, then
08:13 < kibab> That means: you cannot have root on SD/MMC yet.
08:13 < kibab> This is on my list.

So you get dropped into a mountroot prompt.

-- Snip --

which is exactly what I'm experiencing.  However that was from last July, so I'm not sure what the current status should be, i.e. is it still known to be broken or should it have been fixed by now?
Comment 7 Emmanuel Vadot freebsd_committer 2019-03-18 13:01:45 UTC
This is related to MMCCAM, another sd/mmc stack in FreeBSD which isn't active by default.
I don't have any problem with my eMMC, how did you install on it ?
Could you try booting from the sd and using the eMMC to see if it works ?
Comment 8 twistedpair 2019-03-18 13:09:11 UTC
Just wrote the image, i.e. FreeBSD-12.0-RELEASE-arm64-aarch64-PINE64-LTS and FreeBSD-13.0-CURRENT-arm64-aarch64-PINE64-LTS to the eMMC and plugged it in.  I'll try and dig up a micro SD card that isn't already used by something else and try booting off that.
Comment 9 Emmanuel Vadot freebsd_committer 2019-03-18 13:12:13 UTC
Using an usb<->eMMC adapter ?
Comment 10 Emmanuel Vadot freebsd_committer 2019-03-18 13:14:11 UTC
Could you boot -v on loader prompt with the 13-CURRENT image ? This will give more debug output.
Thanks
Comment 11 twistedpair 2019-03-18 13:14:40 UTC
To write the image?  Yes, USB -> eMMC, the Pine64 one.
Comment 12 twistedpair 2019-03-18 13:29:13 UTC
Created attachment 202956 [details]
Verbose boot log
Comment 13 twistedpair 2019-03-18 13:29:58 UTC
Here's the verbose log, just a single boot this time.  Found a spare micro SD card, I'll try that...
Comment 14 twistedpair 2019-03-18 13:55:25 UTC
Created attachment 202957 [details]
Verbose boot log with SD, not eMMC
Comment 15 twistedpair 2019-03-18 13:57:05 UTC
Just uploaded the boot log when booting from micro SD.  It still dies at the same location, there's just less eMMC-related errors before it gets there.
Comment 16 Emmanuel Vadot freebsd_committer 2019-03-18 14:06:08 UTC
Could you test 'set hw.regulator.disable_unused=0' at loader and then boot ?
That's for 13-CURRENT, if you could try 12.0 on the SD (with an without the tunable) that would help me too.
Thanks
Comment 17 twistedpair 2019-03-18 14:26:08 UTC
Created attachment 202958 [details]
Boot log with hw.regulator.disable_unused=0
Comment 18 twistedpair 2019-03-18 14:27:54 UTC
See the attached log, it's identical to the previous one apart from a few "regulator: shutting down xxx" messages.

That's for 13-CURRENT, 12 will take a while because I need to get another SD card and re-download the image for it.
Comment 19 Emmanuel Vadot freebsd_committer 2019-03-18 23:49:18 UTC
Created attachment 202967 [details]
FreeBSD-13.0-CURRENT-arm64-aarch64-PINE64-LTS-20190314-r345110.log

This is the bootlog on the latest snapshot on my Pine64-LTS.
Comment 20 Emmanuel Vadot freebsd_committer 2019-03-18 23:53:55 UTC
Created attachment 202968 [details]
FreeBSD-12.0-RELEASE-arm64-aarch64-PINE64-LTS.log

And the 12.0 log.
So I don't have a problem using SD.
Comment 21 twistedpair 2019-03-19 00:35:26 UTC
So the only real difference I can see is that I'm getting:

GEOM_PART: partition 1 on (mmcsd0s2, BSD) is not aligned on 4194304 bytes

with further GEOM warnings later on.  While your boot moves on to:

Growing root partition to fill device
[...]
GEOM_PART: mmcsd0s2 was automatically resized.

and fsck, mine stops with:

mountroot: waiting for device /dev/ufs/rootfs...
ugen3.2: <Dell Dell USB Keyboard> at usbus3
ukbd0 on uhub2
ukbd0: <EP1 Interrupt> on usbus3
kbd: new array size 4
kbd1 at ukbd0
kbd1: ukbd0, generic (0), config:0x0, flags:0x3d0000
Mounting from ufs:/dev/ufs/rootfs failed with error 19.

Let me see if the keyboard is the problem, it's some generic USB keyboard that
I used to change the boot config...
Comment 22 twistedpair 2019-03-19 03:41:21 UTC
Created attachment 202973 [details]
FreeBSD-13.0-CURRENT-arm64-aarch64-PINE64-LTS failed boot
Comment 23 twistedpair 2019-03-19 03:43:28 UTC
Here's a clean boot (no keyboard or monitor connected) to match your one, it's
practically identical except that your one has:

-- Snip --

Growing root partition to fill device
[...]
GEOM_PART: mmcsd0s2 was automatically resized.

-- Snip --

at the point where mine has:

-- Snip --

mountroot: waiting for device /dev/ufs/rootfs...
Mounting from ufs:/dev/ufs/rootfs failed with error 19.

-- Snip --

Another strange thing is that all the GEOM_PART warnings have been silenced in
mine, but I'm not sure if that's related to removing the USB/HDMI devices or just coincidence.

I'll try again later with the latest -RELEASE and -CURRENT.
Comment 24 Emmanuel Vadot freebsd_committer 2019-03-19 04:03:51 UTC
All the GEOM_PART don't appears in yours because it seems that the sdcard (and the eMMC) aren't present for you.
Since I don't think that both your sdcard and eMMC are bad (since you can load from them) maybe it's power related ? How are you powering the board ?
Comment 25 twistedpair 2019-03-19 08:52:49 UTC
Shouldn't be power-related, I'm using a 5V 3A PSU through the barrel jack power connector, so not a flaky micro USB thing.
Comment 26 twistedpair 2019-03-19 10:03:09 UTC
Created attachment 202980 [details]
FreeBSD-13.0-CURRENT-arm64-aarch64-PINE64-LTS successful boot
Comment 27 twistedpair 2019-03-19 10:07:38 UTC
Now it works, boot log identical to your one.  The two things that changed were that I downloaded a more current CURRENT (20190207 -> 20190314) and that I switched from the eMMC and an 8GB Apacer microSD I had lying around to a 32GB Samsung SD because I noticed you'd used a 32GB micro SD as well.  I'll retry with the current image on the eMMC (16GB) and 8GB Apacer to try and narrow down what the magic factor was.
Comment 28 twistedpair 2019-03-19 10:23:37 UTC
Created attachment 202981 [details]
FreeBSD-13.0-CURRENT-arm64-aarch64-PINE64-LTS eMMC unsuccessful again
Comment 29 Emmanuel Vadot freebsd_committer 2019-03-19 10:26:55 UTC
The two different CURRENT shouldn't make a difference for SD/MMC.
If you can boot successfully multiple times with the new SD I guess the other one is bad.
Can you try booting from the SD (the good one) and accessing the eMMC from it ?
Do stuff like gpart, newfs, dd etc ...
Thanks
Comment 30 twistedpair 2019-03-19 10:31:28 UTC
Finally, the same image on the 16GB eMMC, failing as before.

Successful: 32GB SD.
Failed: 16GB eMMC, 8GB SD.

Could there be some size dependency in the image, i.e. for some reason it requires 32GB media to run?  The boot logs for the 8GB and 32GB SDs are identical up to the point where the 8GB stops with:

mountroot: waiting for device /dev/ufs/rootfs...
Mounting from ufs:/dev/ufs/rootfs failed with error 19.

while the 32GB continues with:

Growing root partition to fill device
[...]
GEOM_PART: mmcsd0s2 was automatically resized.
Comment 31 twistedpair 2019-03-19 10:33:32 UTC
It seems unlikely that both the eMMC and 8GB SD would be bad in exactly the same way, but I'll try booting from the 32GB SD and see what I can get to on the eMMC.  Will the Pine64 preferentially boot from SD if it's present or do I need to change config settings?
Comment 32 Emmanuel Vadot freebsd_committer 2019-03-19 10:37:31 UTC
They aren't bad the same way, with the eMMC you have CRC errors.
The board will always boot on the SD if u-boot is present, if not it will try the eMMC.
Comment 33 Emmanuel Vadot freebsd_committer 2019-03-19 10:39:32 UTC
Also note that the second boot from eMMC don't have the same error, now it's can even correctly detect the eMMC.
Comment 34 twistedpair 2019-03-19 10:43:05 UTC
OK, I'll try an SD boot.  The reason I asked is that there's conflicting info on the Pine64 forums about the boot order: https://forum.pine64.org/showthread.php?tid=6566

The eMMC from the Pine64 store had Windows malware on it, specifically the Win32/FlyStudio trojan, when I got it (!!) so I can well believe that it may not be quite 100%, but the SD cards should be OK, they're from an authorised reseller, not eBay/Aliexpress.
Comment 35 Emmanuel Vadot freebsd_committer 2019-03-19 10:49:38 UTC
This thread is about the Rock64, this have a different SoC.
Comment 36 twistedpair 2019-03-19 10:57:38 UTC
Uhh, it's about the Pine64 LTS, https://www.pine64.org/?product=pine-a64-lts.  I'm not aware of there being Rock64 FreeBSD images at http://ftp.freebsd.org/pub/FreeBSD/releases/arm64/aarch64/ISO-IMAGES/12.0/.
Comment 37 twistedpair 2019-03-19 11:00:00 UTC
Oh, sorry, you meant the thread on the Pine64 forums :-).  Yeah, but there's a certain amount of overlap in the hardware, e.g. the USB serial pinouts so I assumed they're work the same way.
Comment 38 twistedpair 2019-03-19 11:22:12 UTC
Final update (I hope): Tried with another SD card, this time a 16GB Sandisk, and it works as well.  So the problem really is that both the Pine eMMC *and* the Apacer SD are bad.  Given that the eMMC had malware on it when it arrived that's quite plausible for the eMMC, but I'm surprised at the Apacer, although admittedly they're not known for high-quality media.  I'll run both through a flash stress tester in a minute to see what turns up.
Comment 39 twistedpair 2019-03-19 13:01:01 UTC
Just finished running the write/read test on the eMMC:

Pass #1 completed, 0 errors found.

Test completed, total 0 errors found.

So the boot process says there are errors but a R/W test indicates no errors.  In any case I'll have a second eMMC module to try out next week some time, in the meantime I'll run off the Samsung SD card...
Comment 40 twistedpair 2019-03-19 13:37:32 UTC
OK, not the last one after all, I have a clue about why there could be errors
booting but not on a R/W test.  To access the eMMC module I'm using a USB <->
eMMC bridge that uses the GL823 controller, which doesn't mention which
version of eMMC it supports but given that it's from 2010 it's unlikely to be
a recent one, and table 5.7 of the GL823 data sheet mentions a clock frequency
of 24MHz.

The Pine64 with FreeBSD OTOH uses what looks like HS400 / eMMC 5.0:

mmcsd0: 16GB <MMCHC M8B16G 2.8 SN 3E0A3527 MFG 03/2018 by 136 0x0003> at mmc1 52.0MHz/8bit/4096-block
[...]
mmc1:  bus: 8bit, 200MHz (HS200 timing)
[...]
mmc1: setting transfer rate to 52.000MHz (dual data rate timing)

So it could be that it's being run beyond what it's actually capable of, or at
least what it's safely capable of, thus producing various errors that don't
occur at the lower speed.

Is there something like a device.hints setting to restrict eMMC
support/performance to a lower level, or would it require a kernel rebuild?
Comment 41 twistedpair 2019-03-20 03:16:48 UTC
Just ran another boot with the eMMC, this:

mmc1: setting transfer rate to 52.000MHz (dual data rate timing)
aw_mmc1: error rint: 0x000000AC
AW_MMC_INT_DATA_CRC_ERR
aw_mmc1: error rint: 0x000000AC
AW_MMC_INT_DATA_CRC_ERR
aw_mmc1: error rint: 0x000000AC
AW_MMC_INT_DATA_CRC_ERR

would seem to indicate that it's a problem with HS400 mode, the CRC errors start appearing as soon as it's enabled.

What's the best way to move ahead with this?  I'm not really set up to do custom kernel builds, but can run test builds and report back... one way to test things would be to mask out MMC_CAP_MMC_HS400 and/or MMC_CAP_MMC_DDR52 from host_caps / slot->host.caps in sdhci.c and see if that affects things.
Comment 42 Emmanuel Vadot freebsd_committer 2019-03-20 23:45:14 UTC
Allwinner A64 doesn't use sdhci, the driver is in sys/arm/allwinner/aw_mmc.c
It doesn't support HS200 or HS400 for now, the highest speed that it can do is DDR52 (which is the one selected in your case) and I doubt that this is the root of your problems.
Comment 43 twistedpair 2019-03-27 09:29:58 UTC
Created attachment 203184 [details]
FreeBSD-13.0-CURRENT-etc with different eMMC
Comment 44 twistedpair 2019-03-27 09:33:37 UTC
Just got a different eMMC module, this one from ODroid instead of Pine.  Boot log from boot -v is attached, now it works fine, so presumably there's an incompatibility with the Pine-sourced eMMC module.  If it's useful I can send you the Pine eMMC for testing/development of the driver...
Comment 45 Daniel Zeisig 2019-09-06 21:29:46 UTC
Having the same issue with the Pine emmc. Mine is 64GB. 
Which emmc exactly is it which is working for you now ?
Comment 46 twistedpair 2019-09-07 02:07:48 UTC
The Pine eMMC is the problem, it also came with Windows malware preinstalled on it.  The ODroid one works fine.  I'm guessing Pine are sourcing their eMMC from less-than-reputable vendors.