Bug 204521 - [new driver] [request] Port rtsx from OpenBSD to FreeBSD
Summary: [new driver] [request] Port rtsx from OpenBSD to FreeBSD
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Many People
Assignee: freebsd-bugs mailing list
URL: https://lists.freebsd.org/pipermail/f...
Keywords: feature, patch
: 161719 (view as bug list)
Depends on:
Blocks:
 
Reported: 2015-11-13 13:02 UTC by evil.lombo
Modified: 2020-03-22 18:48 UTC (History)
33 users (show)

See Also:


Attachments
Patch to add SD card insertion/removal (7.19 KB, patch)
2019-12-12 10:33 UTC, Gary Jennejohn
gljennjohn: maintainer-approval?
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description evil.lombo 2015-11-13 13:02:29 UTC
Hi,

in my quest to make FreeBSD fully operational in my laptop, I would like to know whether anyone is working on porting the OpenBSD rtsx to FreeBSD.

I have asked to Robert (https://lists.freebsd.org/pipermail/freebsd-questions/2015-February/264014.html) about its working on the module but he has not yet got the chance to start it.

If not, I'd try to develop it personally, but I have to be sure the better workflow: is it better to integrate it with sdhci/mmc modules or start it brand new?

Thanks,

Simone
Comment 1 Stefan Hagen 2018-06-14 11:39:43 UTC
Hello,

the request to port this driver came up again on freebsd-hackers again. Warner Losh gave some insight about what needs to be done to port rtsx.

https://lists.freebsd.org/pipermail/freebsd-hackers/2018-April/052520.html

This SD Card reader is built into a wide variety of Lenovo Laptops (x240 and up, t440/t540 and up, L440/L540 and up) Dell XPS Series, HP Spectre x360 and up.

Best Regards,
Stefan
Comment 2 Gleb Popov freebsd_committer 2018-09-15 15:06:37 UTC
Adding myself to CC, as I also have this card reader in my notebook.

Simone, do you still plan to work on this?
Comment 3 Gleb Popov freebsd_committer 2019-06-18 12:02:12 UTC
I have posted a bounty of $100 for this bug on BountySource: https://www.bountysource.com/issues/75687739-new-driver-request-port-rtsx-from-openbsd-to-freebsd

Hope this will give a spin.
Comment 4 Gleb Popov freebsd_committer 2019-07-03 10:07:22 UTC
*** Bug 161719 has been marked as a duplicate of this bug. ***
Comment 5 Kamila Součková 2019-11-28 10:52:10 UTC
Started work on this: https://github.com/AnotherKamila/freebsd-rtsz . Very much WIP right now. I intend to make progress on this over the weekend, please ping me if you can help with testing.
Comment 6 Gleb Popov freebsd_committer 2019-11-28 15:53:41 UTC
(In reply to Kamila Součková from comment #5)

Hi Kamilla, thanks for your interest. It worth noting that former FreeBSD member already started working on this, and it seems that he has made more progress than you. I'm not know if that means that you should abandon your work, though.
Comment 7 Kamila Součková 2019-11-28 16:06:22 UTC
(In reply to Gleb Popov from comment #6)

Hi, thank you for letting me know! Do you know who that was? I am aware of https://github.com/ruupert/rtsx/ , which seems to have been abandoned (and I am currently at the same stage, so also able to detect card presence, plus my code might be structured in a way that is more likely to be merged). Are you referring to this, or to something else?

Thanks a lot!
Comment 8 Gleb Popov freebsd_committer 2019-11-28 16:10:42 UTC
(In reply to Kamila Součková from comment #7)

His name is Gary Jennejohn, here is relevant thread on the mailing list: https://lists.freebsd.org/pipermail/freebsd-hackers/2019-November/055200.html

His work is not public, but he reported that has quite a progress.
Comment 9 hlh 2019-11-28 18:34:31 UTC
I have also a try at this problem:

https://github.com/hlh-restart/rtsx
Comment 10 Gary Jennejohn 2019-11-29 09:56:46 UTC
To hlh

I've been working on this driver for about 3 weeks.  I looked at your code and it looks quite good.  I'll give it a try and send some feedback.
Comment 11 Gary Jennejohn 2019-11-29 11:26:10 UTC
(In reply to Gary Jennejohn from comment #10)
The driver from hlh is much further along than mine.  Loading the module with a SD card plugged in actually results in MMC seeing the card and reading its capacity.  Plugging in a card after the module is loaded results in a spurious interrupt error, but that can probably be fixed without much trouble.

I had to add ``#include <sys/mutex.h>'' to get it to compile in my FreeBSD13 tree.

It doesn't work too well with my 522A and I see timeouts waiting for interrupts quite often.

In any case, my opinion is that this driver is the way forward and hlh should continue working on it.
Comment 12 hlh 2019-11-29 12:09:44 UTC
(In reply to Gary Jennejohn from comment #11)
The detection of a new card inserted after the driver is loaded will be dealt later.

My main problem is that any read io return all 0 on timeout.
So I postpone the write problem.

I try to read the source of openbsd, netbsd and linux to no avail for now.

I almost lost...

Henri
Comment 13 Gary Jennejohn 2019-11-29 14:48:57 UTC
(In reply to hlh from comment #12)

I noticed that, immediately after loading the module, I can successfully run e.g.
dd if=/dev/mmcsd0 of=junk count=20 without errors or timeouts.  But, as you say,
the bytes are all 0x00.  Any further attempt to access the SD card results in a
CRC error.

In fact, I see LOTS of CRC errors (all RTSX_SD_CRC16_ERR) during start up.  I also see CMD7 failing quite often.

I've spent many hours looking at the Linux driver, which was written by Realtek.
There is so much special handling for each of the controllers that it almost seems hopeless to get a driver which works as well as what Linux has.

Whether the errors I see are due to pecularities of the 522a (Linux has lots of special code for this one) I can't say.
Comment 14 Gary Jennejohn 2019-11-30 11:49:24 UTC
(In reply to hlh from comment #12)

I installed OpenBSD on a sacrifical disk and it also returns all 0s when I read from the SD card.  I know there are data on the SD card because I can dump them using a USB2 multi-card reader.

Evidently, OpenBSD does not support my 522A all too well.

Hard to tell whether any errors occur because there's no debug output from the driver.  But then again, I was able to run dd multiple times without an error being reported (which may mean nothing if the driver suppresses errors).
Comment 15 Gary Jennejohn 2019-12-02 16:49:15 UTC
(In reply to Gary Jennejohn from comment #14)

I installed Debian Linux to check whether my Raltek controller really works.

The first read I did from the SD card returned all 0s.

After playing around for some time I discovered that the first megabyte (2048 sectors) was filled with 0s.  Apparently a result of the vendor formatting it with FAT32.

So, OpenBSD may also work if I run dd with skip=2048.

This might also be the reason why hlh is seeing 0s when he reads from his card.

Unfortunately, with the code from hlh I still see CRC errors which prevent me from using the my controller.
Comment 16 hlh 2019-12-02 17:30:16 UTC
(In reply to Gary Jennejohn from comment #15)

I dump the sd card on another device and I get:


# hd sdcard.data
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 82  |................|
000001c0  03 00 0c fe ff ff 00 20  00 00 00 04 b7 03 00 00  |...þÿÿ. ........|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............Uª|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002000

With `sysctl debug.bootverbose=1` I display some part of the buffer
and it is all 0.

On my pc with rtsx I get:

# dd bs=512 if=/dev/mmcsd0 of=sd.dump count=20
20+0 records in
20+0 records out
10240 bytes transferred in 0.785274 secs (13040 bytes/sec)
# hd sd.dump 
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002800

:-(
Comment 17 Gary Jennejohn 2019-12-03 07:44:24 UTC
(In reply to hlh from comment #16)

My Realtek 522A does work correctly under OpenBSD.

There is some difference in behavior between the rtsx code in OpenBSD and what there is in the FreeBSD code we now have.  For example, I do not see CRC errors with OpenBSD.

But there may also be differences between the mmc/mmcsd code in OpenBSD and the corresponding code in FreeBSD.  Hard to say.
Comment 18 Lars Engels freebsd_committer 2019-12-09 20:03:03 UTC
I have a Thinkpad T450s which has a rtsx reader supported by OpenBSD:

rtsx0 at pci1 dev 0 function 0 "Realtek RTS5227 Card Reader" rev 0x01: msi

I'm running FreeBSD 12.1 on it, so if I can test anything, ping me please.
Comment 19 Lars Engels freebsd_committer 2019-12-09 20:03:43 UTC
Change state to "In Progress" as now there are some people working on it.
Comment 20 Alexey Dokuchaev freebsd_committer 2019-12-10 03:21:00 UTC
I've given Henri's current GitHub code a spin today, on an RTS5227 (card=0x1992103c chip=0x522710ec).  I've built and loaded all three modules, with the card removed; it was not able to detect card insertion (every time I pushed the card in/out it just complained about spurious interrupt):

> rtsx0: <Realtek RTS5227 PCI MMC/SD Card Reader> mem 0xd4000000-0xd4000fff
> at device 0.0 on pci2
> rtsx0: Card removed
> rtsx0: Spurious interrupt - no active request
> rtsx0: Spurious interrupt - no active request
> rtsx0: Spurious interrupt - no active request
> rtsx0: Spurious interrupt - no active request
> rtsx0: Spurious interrupt - no active request
Things changed when I've unloaded rtsx.ko, plugged the card, and then loaded it again:

> rtsx0: <Realtek RTS5227 PCI MMC/SD Card Reader> mem 0xd4000000-0xd4000fff
> at device 0.0 on pci2
> rtsx0: Card inserted
> rtsx0: Attaching MMC bus failed [6]
> mmc0: <MMC/SD bus> on rtsx0
> mmcsd0: 129GB <SDHC 00000 1.0 SN A123456Z MFG 05/2014 by 27 SM> at mmc0
> 0.4MHz/1bit/256-block
> rtsx0: CRC error
> rtsx0: Soft reset
> mmcsd0: Error indicated: 2 Bad CRC
> rtsx0: CRC error
> rtsx0: Soft reset
> mmcsd0: Error indicated: 2 Bad CRC
> rtsx0: CRC error
> rtsx0: Soft reset
> mmcsd0: Error indicated: 2 Bad CRC
Those errors come from trying to read anything from /dev/mmcsd0 with dd(1), to no avail.  Sometimes it's possible to read one 512-byte sector off of it, but most of the times it yields an input/output error.
Comment 21 Gary Jennejohn 2019-12-10 08:13:23 UTC
(In reply to Alexey Dokuchaev from comment #20)
I've been looking at Henri's driver for some time.

Unfortunately, there are DMA errors when trying to read larger amounts of data from the SD card.  I've been adding debug output to the driver but haven't yet been able to pinpoint the cause of the errors.

The DMA errors result in the Controller Timeouts you've seen because no interrupt is being raised.

As a result e.g. CMD7 fails.  This is a very important command which is part of the card intialization process.

The SD card protocols are rather complex which makes understanding what is going on kind of difficult.

Since FreeBSD policy dictates that new drivers have to land in HEAD first, it will probably be some time before the driver appears in older versions.
Comment 22 Alexey Dokuchaev freebsd_committer 2019-12-10 08:55:12 UTC
(In reply to Gary Jennejohn from comment #21)
> Since FreeBSD policy dictates that new drivers have to land in HEAD first,
> it will probably be some time before the driver appears in older versions.
I guess this part was reply to Lars Engels from comment #18:
> I'm running FreeBSD 12.1 on it, so if I can test anything, ping me please.
I don't think 12.1 vs -CURRENT would cause a difference here; all one has to do is to clone Henri's GitHub repo, type "make", and play with it on any relatively recent FreeBSD version/branch.  There's even no need to wait for it to land in HEAD (which, given the current state of the driver, is not immediately foreseeable anyway).
Comment 23 Gary Jennejohn 2019-12-10 09:28:00 UTC
(In reply to Alexey Dokuchaev from comment #22)
Yes, that's a good point.  If users of older FreeBSD versions could clone the driver and give it a spin then we'd have some useful information.
Comment 24 Alexey Dokuchaev freebsd_committer 2019-12-12 01:16:39 UTC
Another strange observation which I didn't pay attention earlier: upon "kldload rtsx", the following lines appear in the kernel log before anything else:

> hdac1: <ATI (0x9902) HDA Controller> mem 0xd6044000-0xd6047fff at device 1.1
> on pci0
> hdac1: hdac_get_capabilities: Invalid corb size (0)
> device_attach: hdac1 attach returned 6
Admittedly, I have zero ideas what rtsx(4) has to do with HDA. :-/
Comment 25 Gary Jennejohn 2019-12-12 06:16:37 UTC
(In reply to Alexey Dokuchaev from comment #24)
rtsx has nothing at all to do with hda.  But the "kldload rtsx" results in the kernel rescanning all devices to find a match.

There were recently some changes in the hda code.  How current is your kernel?
Comment 26 Alexey Dokuchaev freebsd_committer 2019-12-12 06:41:17 UTC
(In reply to Gary Jennejohn from comment #25)
> But the "kldload rtsx" results in the kernel rescanning all devices
> to find a match.
That should explain it, thanks.

> There were recently some changes in the hda code.  How current is your kernel?
Not the very latest one (septemberish r352614), I should probably update, but a bit afraid of some potentially boot-breaking USB/CAM problems being reported on the lists*.

*) https://lists.freebsd.org/pipermail/freebsd-current/2019-December/074914.html
Comment 27 Gary Jennejohn 2019-12-12 06:57:59 UTC
(In reply to Alexey Dokuchaev from comment #26)
I sort of understand your reluctance.

I never install a new kernel over the top of the old kernel.  Instead I do this
(as root, of course) in a bash alias:

mkdir -p /boot/test
cd /usr/src
make -s installkernel KODIR=/boot/test
nextboot -k test
cd

Then a reboot will boot the new kernel from /boot/test.  If it fails the next boot will use the old, known-good kernel.

If it succeeds then you can simply run "make installkernel" so that the standard kernel in /boot/kernel will be updated and used.
Comment 28 Gary Jennejohn 2019-12-12 10:33:23 UTC
Created attachment 209887 [details]
Patch to add SD card insertion/removal

A minor patch to Henri's code to add SD card insertion/removal handling.  Doesn't fix the DMA problems, but I felt like doing something else.

Shamelessly stolen from dwmmc.c
Comment 29 Alexey Dokuchaev freebsd_committer 2019-12-12 10:49:06 UTC
(In reply to Gary Jennejohn from comment #28)
> A minor patch to Henri's code to add SD card insertion/removal handling.
Thanks, I confirm it does fix card detection problem for me.
Comment 30 Greg V 2019-12-12 12:05:03 UTC
(In reply to Gary Jennejohn from comment #27)

> I never install a new kernel over the top of the old kernel.  Instead I do this

installkernel does this for you! the old kernel is always moved to /boot/kernel.old and you can select that in the bootloader.
Comment 31 hlh 2019-12-12 13:32:07 UTC
(In reply to Gary Jennejohn from comment #28)

I update the git repository with the patch

Thanks a lot
Henri
Comment 32 Gary Jennejohn 2019-12-12 14:21:06 UTC
(In reply to Greg V from comment #30)
I'm well aware of what installkernel does.  But my way saves me the trouble of telling the bootloader to boot from kernel.old if the new kernel fails.  I can simply reboot and the known-working kernel under /boot/kernel will be loaded.
Comment 33 Gary Jennejohn 2019-12-21 08:12:49 UTC
I tried implementing rtsx_xfer_bounce() from the OpenBSD code.

OpenBSD uses this function when the upstream code does not pass in a list of DMA scatter-gather addresses.  Normally, the SD code in OpenBSD allocates its own DMA buffers and passes them to the driver.

Since FreeBSD does not do that - it either uses buffers on the kernel stack or buffers from a struct bio - it seemed logical to me to try rtsx_xfer_bouce().

The DMA transfer raises an interrupt, but the target buffer is always filled with 0s.

I'm running out of ideas on how to get DMA to work.

If I had a laptop I could tear apart I'd put a logic analyzer on the controller pins and compare the behavior between OpenBSD and FreeBSD.  Unfortunately, the laptop I have is a loaner and I can't do that.
Comment 34 Gary Jennejohn 2020-01-08 13:37:37 UTC
I reimplemented rtsx_xfer_bounce() so that it's pretty much one-to-one the same as the way OpenBSD does it.  But now the transfer always times out.  The DMA addresses returned by the kernel all look good and are within the 32-bit range.

If anyone is interested I can attach my code.  It's full of ifdef's and trace output.  Maybe other eyes can see what I can't.  I've looked at the code so much that I may be blind to any errors.

Comparing OpenBSD with the code I'm using doesn't reveal any obvious errors.  The DMA setup for using bounce buffers appears to be the same and the steps preceeding the DMA transfer also seem identical.

I'm at a loss to explain why DMA in FreeBSD does not work.

What I haven't yet tried is to do the DMA setup like dwmmc.c does it.  Guess I'll try that next.
Comment 35 Lyubomir 2020-01-09 16:10:31 UTC
Hi Gary,
All the best to all for the New 2020!
I am not a developer so I will not be able to help with the code but I have access to Lenovo T450 (mine), to a logic analyzer SALEAE Logic pro 16 (not mine :)) and I can also find another hdd to install OpenBSD on it. I would like to help but I would definitely need your guidance.
Do you think this will work?

Best regards,

Lyubo
Comment 36 Gary Jennejohn 2020-01-10 08:54:42 UTC
(In reply to Lyubomir from comment #35)
Thanks for the offer.  I did some research and the Realtek controllers used in laptops are basically black boxes.  They consist of a module with the electronics and the SD card holder and then a rather long cable which gets routed to a connecotr on the motherboard.  I was not able to find any documentation on the cable pinout.  Without that information it would not be possible to connect the SALEAE logic analyzer (I have two of them) to get useful traces.  So my idea to do this was not so good.

At one time there were PCIe cards with suitable Realtek controllers, but I haven't been able to find any on e.g. eBay.  The cards now available no longer use Realtek.
Comment 37 Gary Jennejohn 2020-01-18 09:24:33 UTC
(In reply to hlh from comment #31)
You should add sc->rtsx_intr_status = 0; prior to the call to rtsx_wait_intr() in rtsx_xfer()to be consistent.  This is the only place where that was not being done.  It doesn't change the behavior, but it can prevent false positives in rtsx_wait_intr().
Comment 38 hlh 2020-01-18 09:58:39 UTC
(In reply to Gary Jennejohn from comment #37)
Update done. Thanks
Comment 39 Sergey V. Dyatko 2020-01-23 20:49:37 UTC
Hi, I have lenovo thinkpad t470p with 
none4@pci0:4:0:0:       class=0xff0000 rev=0x01 hdr=0x00 vendor=0x10ec device=0x522a subvendor=0x17aa subdevice=0x505d
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTS522A PCI Express Card Reader'
and can test too
Comment 40 dirkx 2020-03-19 12:34:50 UTC
(In reply to Gary Jennejohn from comment #34)

if (len == 0) set it to min(maxbuff,len) at https://github.com/hlh-restart/rtsx/blob/fa2c5b66d00b7a4bec6eb0e5539b8edbd7a06a9b/rtsx.c#L1472 looks rather odd to me.
Comment 41 Gary Jennejohn 2020-03-19 13:46:26 UTC
(In reply to dirkx from comment #40)
The code is correct.  The {Net,Open}BSD upper layer code always sets xfer_len to the correct value.  FreeBSD does indeed have xfer_len, but the upper layer never sets it.  To simplify implementing the OpenBSD code (and avoid kernel panics) it was convenient for hlh to set xfer_len to a reasonable value.
I must admit that what's on github looks nothing like the code I've tested.  I've made massive changes to DMA code, added lots of variations on how transfers are done based on examining what OpneBSD and Linux do in various situations and added large amounts of debugging.  Unfortunately, nothing I've tried so far results in successful DMA transfers.
One test I did was to set a byte pattern in the virtual address of the DMA buffer, start the transfer (the controller claims that no errors occurred), and then dump the physical address of the buffer using /dev/mem.  The result was that NO bytes were transferred at all, i.e. the contents of the virtual and physical addresses were identical.
There is much more to the {Net,Open}BSD implementation than just the driver.  There is an upper layer which implements the SD Card Host Controller Specification.  It is somewhat different than what was implemented in FreeBSD.  I'm beginning to suspect that these differences may be contributing to the DMA problems.
Comment 42 Gary Jennejohn 2020-03-20 10:35:48 UTC
(In reply to hlh from comment #38)
The message from dirkx led me to look more closely at the OpenBSD code.
OpenBSD never sets intr_status to 0 before calling rtsx_wait_intr().
That makes sense because, theoretically, the interrupt could be raised before rtsx_wait_intr() is entered.  In that case, setting intr_status to 0 would overwrite its setting in the interrupt handler and result in a false timeout error.
I tested NOT setting intr_status to 0 and the driver works just as well (or badly) as before.  Actually, setting it to zero is not really necessary due to this code in rtsx_wait_intr(): sc->rtsx_intr_status &= ~status;
So, I suggest, but do not demand, at least commenting out the lines where intr_status is set to 0.  I leave it up to you.
Comment 43 hlh 2020-03-20 13:18:19 UTC
(In reply to Gary Jennejohn from comment #42)

I am not sure this is a problem because sc->rtsx_intr_status is set
to 0 _before_ sending the command and so before the relevant
interrupt can be raised.
Comment 44 dirkx 2020-03-20 13:20:47 UTC
(In reply to hlh from comment #43)

So I put a few 'if (sc->rtsx_intr_status) printf..' just before the  sc->rtsx_intr_status=0; lines to check if there was (I assuse) something pending/non zero at the point this happens. And it seems to happy once or twice - exactly before the timeout.
Comment 45 dirkx 2020-03-20 13:21:18 UTC
(In reply to hlh from comment #43)

So I put a few 'if (sc->rtsx_intr_status) printf..' just before the  sc->rtsx_intr_status=0; lines to check if there was (I assuse) something pending/non zero at the point this happens. And it seems to happy once or twice - exactly before the timeout.
Comment 46 hlh 2020-03-20 13:31:14 UTC
(In reply to dirkx from comment #44)
Strange, if you comment the sc->rtsx_intr_status=0 does the timeout disappear?
Comment 47 Gary Jennejohn 2020-03-20 13:57:50 UTC
(In reply to hlh from comment #46)
I see timeouts whether rtsx_intr_status is set to 0 or not.  But dirkx makes a valid point, if setting it to 0 occurs before the command is sent.  In any case, I didn't observe any adverse effects by not setting it to 0, but then again, setting it to 0 may also be harmless.
The timeouts semm to only occur when a transfer using rtsx_xfer_short() is attempted.  But these errors are only visible when boot_verbose is set to 1.
I also see quite a few CRC errors being generated (may be due to the SD card I'm using).
Comment 48 hlh 2020-03-20 14:07:23 UTC
(In reply to Gary Jennejohn from comment #4(In reply to Gary Jennejohn from comment #47)

I also encounter CRC errors but if I boot with tails (https://tails.boum.org/)
I can play with the sdcard without problem. So CRC is a problem with the
Freebsd driver/system. Now I think that our problem is due to some wrong
initialization... But I can't find what :-(
Comment 49 Gary Jennejohn 2020-03-20 14:34:14 UTC
(In reply to hlh from comment #48)
Tails is Linux, and the Linux driver(s) were developed by Realtek.  Not surprising that it works.
I also tested with OpenBSD, which also works without error.
So I agree, something is missing in FreeBSD, although your driver seems to faithfully implement the OpenBSD rtsx.c.
But, as I wrote in a previous comment, there's a lot more to the OpenBSD code than just the driver itself.