Bug 259129 - FreeBSD 13.0: Corruption in SGL observed while running IO with smartqpi driver
Summary: FreeBSD 13.0: Corruption in SGL observed while running IO with smartqpi driver
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.0-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-10-13 07:35 UTC by Hermes T K
Modified: 2021-10-13 07:40 UTC (History)
1 user (show)

See Also:


Attachments
Attaching kernel logs for reference (50.48 KB, text/plain)
2021-10-13 07:35 UTC, Hermes T K
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Hermes T K 2021-10-13 07:35:51 UTC
Created attachment 228651 [details]
Attaching kernel logs for reference

In FreeBSD 13.0, while running IO with blocksize 1MB, observed corruption in SGL. We were running IO with FIO (version 3.28)IO tool.
Command used :

fio --filename=/dev/da0: -direct=1 -iodepth=32 -ioengine=posixaio -rw=randrw -bs=1024k -numjobs=8 -runtime=30 -group_reporting -name=stress

Transfer size used in smartpqi  driver:

In update_sim_properties FUNCTION
cpi->maxio = (softs->pqi_cap.max_sg_elem - 1) * PAGE_SIZE;
ie: Max transfer size: 0x21f000  (2.24 MB)

While running IO with 1mb block size , getting EINPROGRESS status for bus_dmamap_load_ccb in pqi_map_request function.

function stack => smartpqi_cam_action() -> pqisrc_io_start() -> pqi_map_request() -> bus_dmamap_load_ccb() ->  pqi_request_map_helper

After receiving the EINPROGRESS status for bus_dmamap_load_ccb(),  pqi_request_map_helper callback function is called .
Then observed corruption in the SGL received.
Eg: While running FIO with 1 MB block size, the total mapped SGL length received is 0x3002c0 (3.14MB), where the total transfer size for the IO was  0x100000 (1.04MB).  
SGL dump(Added debug prints to dump sgl):

05:30:29 freebsd13 kernel: [NOTE]:[ pqi_map_request ] [ 615 ]Total transfer size : 100000 for tag value 37, Max transfer size: 21f000
Oct  6 05:30:29 freebsd13 kernel: [NOTE]:[ pqi_request_map_helper ] [ 577 ]Submitting IO With tag 30 after EINPROGRESS status[NOTE]:[ pqisrc_build_sgl ] [ 108 ]SGL_Count :256 for tag 30
Oct  6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 131 ]SGL Chaining
Oct  6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[0]: 5930000
Oct  6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[0]: 1000
Oct  6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[1]: 592f000
Oct  6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[1]: 1000


Oct  6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[255]: 5831000
Oct  6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[255]: 1000
Oct  6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 147 ]Total sg length = 3002c0 for tag value 30
Oct  6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_show_cdb ] [ 1615 ]TAG: 30


When we tried in FreeBSD 12.2, the maximum block size allowed to run in fio is 128k.

We are suspecting some issue in SGL handling with FreeBSD 13.0.