Overview: If the LUN block size in ctl.conf is set to 512, the physical block size reported to iSCSI initiators is the volblocksize. Steps to reproduce: Mirrored vdevs with ashift=12 on 2 x Samsung 870 QVO 4 TB Zpool on 2 x said vdevs Zvol with volblocksize=8K CTL LUN on the zvol with backend=block, block size=512 Connect Windows or ESXi initiator to the target and list the device parameters. Actual result: Block size is reported as 512. Expected results: Block size is reported as 8K -- same as the volblocksize. Build date and hardware: XigmaNAS 13.2.05 on FreeBSD 13.2-RELEASE-p1 running on Dell PowerEdge R730XD.
Sorry, I had the Actual and Expected sections reversed! Actual result: Block size is reported as 8K -- same as the volblocksize. Expected results: Block size is reported as 512.
I am not sure what exactly you are complaining about. As you have told, CTL by default reports logical block size of 512 bytes and physical block size equal to ZVOL volblocksize. That is how it is designed to work. If you want to report different physical block size, you can set it with pblocksize LUN option.
Hi Alexander. I hope this is a better illustration of the issue. As you can see, the logical block size reported to ESXi is always 512, but the physical block size is 4K when set to 4K, and 8K when set to 512. volblocksize=8K LUN Block Size=4K LUN Physical Block Size=<blank> [root@localhost:~] esxcli storage core device capacity list Device Physical Blocksize Logical Blocksize Logical Block Count Size Format Type --------------------------------- ------------------ ----------------- ------------------- ----------- ----------- t10.FreeBSD_MYDEVID00000000000000 4096 512 3865470576 1887436 MiB Unknown volblocksize=8K LUN Block Size=512 LUN Physical Block Size=<blank> [root@localhost:~] esxcli storage core device capacity list Device Physical Blocksize Logical Blocksize Logical Block Count Size Format Type --------------------------------- ------------------ ----------------- ------------------- ----------- ----------- t10.FreeBSD_MYDEVID00000000000000 8192 512 3865470576 1887436 MiB Unknown
How exactly do you set "LUN Block Size"?
Show your /etc/ctl.conf .
xigmanas: etc# cat ctl.conf # global settings # auth-group configuration # portal-group configuration portal-group "iscsi" { discovery-auth-group "no-authentication" listen "0.0.0.0" } # lun configuration lun "test" { backend "block" blocksize "512" device-type "disk" option "vendor" "FreeBSD" option "product" "iSCSI Disk" option "revision" "0123" option "rpm" "7200" option "formfactor" "3" path "/dev/zvol/test2/test2vol" } # target configuration target "iqn.2001-01-01.no.ste.test" { auth-group "no-authentication" portal-group "iscsi" lun "0" "test" }
I see `blocksize "512"` here. I suppose the questions are to xigmanas, not FreeBSD.
How do you mean? Isn't `blocksize "512"` valid?
Valid for what? xigmanas via the ctl.conf explicitly requested logical block size of 512 bytes -- CTL done it in both cases. If physical block size is not specified in config, it is taken from underlying ZVOL. So far everything makes sense if we assume the quoted ctl.conf is from the second ESXi output. Please show ctl.conf for the case you think is wrong, otherwise I still don't see a problem. PS: Last time I heard ESXi did not like increased block sizes, so all this experiment may be moot.
Changed LUN block size to 4K. xigmanas: etc# cat ctl.conf # global settings # auth-group configuration # portal-group configuration portal-group "iscsi" { discovery-auth-group "no-authentication" listen "0.0.0.0" } # lun configuration lun "test" { backend "block" blocksize "4096" device-type "disk" option "vendor" "FreeBSD" option "product" "iSCSI Disk" option "revision" "0123" option "rpm" "7200" option "formfactor" "3" path "/dev/zvol/test2/test2vol" } # target configuration target "iqn.2001-01-01.no.ste.test" { auth-group "no-authentication" portal-group "iscsi" lun "0" "test" } [root@localhost:~] esxcli storage core device capacity list Device Physical Blocksize Logical Blocksize Logical Block Count Size Format Type --------------------------------- ------------------ ----------------- ------------------- ----------- ----------- t10.FreeBSD_MYDEVID00000000000000 4096 512 3865470576 1887436 MiB Unknown
Here's what Ubuntu 22.04 has to say about the same iSCSI LUN that I connected to ESXi. Does this make sense to you? It seems inconsistent to me in Linux as well. ===================================================================================================== volblocksize=8K LUN Block Size=512 LUN Physical Block Size=<blank> balchen@blocksize-test:~$ sudo dmesg [....] [ 15.960107] Loading iSCSI transport class v2.0-870. [ 16.341346] iscsi: registered transport (tcp) [ 16.412853] scsi host33: iSCSI Initiator over TCP/IP [ 16.500664] scsi 33:0:0:0: Direct-Access FreeBSD iSCSI Disk 0123 PQ: 0 ANSI: 7 [ 16.503518] scsi 33:0:0:0: alua: supports implicit TPGS [ 16.503527] scsi 33:0:0:0: alua: device t10.FreeBSD MYDEVID0000\0\0\0\0\0 port group 1 rel port 3 [ 16.503752] sd 33:0:0:0: Power-on or device reset occurred [ 16.504341] sd 33:0:0:0: [sdb] 3865470576 512-byte logical blocks: (1.98 TB/1.80 TiB) [ 16.504346] sd 33:0:0:0: [sdb] 8192-byte physical blocks [ 16.504427] sd 33:0:0:0: [sdb] Write Protect is off [ 16.504432] sd 33:0:0:0: [sdb] Mode Sense: 7f 00 10 08 [ 16.504585] sd 33:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 16.504799] sd 33:0:0:0: [sdb] Optimal transfer size 1048576 bytes [ 16.505500] sd 33:0:0:0: Attached scsi generic sg2 type 0 [ 16.509214] sdb: sdb1 [ 16.511646] sd 33:0:0:0: [sdb] Attached SCSI disk [ 16.515220] sd 33:0:0:0: alua: transition timeout set to 60 seconds [ 16.515228] sd 33:0:0:0: alua: port group 01 state A non-preferred supports TolUSNA [ 65.600593] loop5: detected capacity change from 0 to 8 balchen@blocksize-test:~$ sudo blockdev --report /dev/sdb RO RA SSZ BSZ StartSec Size Device rw 4096 512 4096 0 1979120934912 /dev/sdb balchen@blocksize-test:~$ sudo sg_format -v /dev/sdb FreeBSD iSCSI Disk 0123 peripheral_type: disk [0x0] PROTECT=0 Unit serial number: MYSERIAL0000 LU name: @Pr▒ mode sense(10) cdb: [5a 00 01 00 00 00 00 00 fc 00] Mode Sense (block descriptor) data, prior to changes: Number of blocks=3865470576 [0xe6666670] Block size=512 [0x200] read capacity(10) cdb: [25 00 00 00 00 00 00 00 00 00] Read Capacity (10) results: Number of logical blocks=3865470576 Logical block size=512 bytes No changes made. To format use '--format'. To resize use '--resize' balchen@blocksize-test:~$ sudo blockdev -v --getbsz /dev/sdb get blocksize: 4096 balchen@blocksize-test:~$ sudo blockdev -v --getpbsz /dev/sdb get physical block (sector) size: 8192 balchen@blocksize-test:~$ sudo blockdev -v --getss /dev/sdb get logical block (sector) size: 512 ===================================================================================================== volblocksize=8K LUN Block Size=4K LUN Physical Block Size=<blank> balchen@blocksize-test:~$ sudo dmesg [....] [ 15.026385] Loading iSCSI transport class v2.0-870. [ 15.428041] iscsi: registered transport (tcp) [ 15.484360] scsi host33: iSCSI Initiator over TCP/IP [ 15.570188] scsi 33:0:0:0: Direct-Access FreeBSD iSCSI Disk 0123 PQ: 0 ANSI: 7 [ 15.574942] scsi 33:0:0:0: alua: supports implicit TPGS [ 15.574951] scsi 33:0:0:0: alua: device t10.FreeBSD MYDEVID0000\0\0\0\0\0 port group 1 rel port 3 [ 15.575206] sd 33:0:0:0: Power-on or device reset occurred [ 15.575843] sd 33:0:0:0: [sdb] 483183822 4096-byte logical blocks: (1.98 TB/1.80 TiB) [ 15.575849] sd 33:0:0:0: [sdb] 8192-byte physical blocks [ 15.575934] sd 33:0:0:0: [sdb] Write Protect is off [ 15.575939] sd 33:0:0:0: [sdb] Mode Sense: 7f 00 10 08 [ 15.576096] sd 33:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 15.576349] sd 33:0:0:0: [sdb] Optimal transfer size 1048576 bytes [ 15.579762] sd 33:0:0:0: [sdb] Attached SCSI disk [ 15.580794] sd 33:0:0:0: Attached scsi generic sg2 type 0 [ 15.583250] sd 33:0:0:0: alua: transition timeout set to 60 seconds [ 15.583258] sd 33:0:0:0: alua: port group 01 state A non-preferred supports TolUSNA [ 62.026528] loop5: detected capacity change from 0 to 8 balchen@blocksize-test:~$ sudo sg_format -v /dev/sdb FreeBSD iSCSI Disk 0123 peripheral_type: disk [0x0] PROTECT=0 Unit serial number: MYSERIAL0000 LU name: mode sense(10) cdb: [5a 00 01 00 00 00 00 00 fc 00] Mode Sense (block descriptor) data, prior to changes: Number of blocks=483183822 [0x1cccccce] Block size=4096 [0x1000] read capacity(10) cdb: [25 00 00 00 00 00 00 00 00 00] Read Capacity (10) results: Number of logical blocks=483183822 Logical block size=4096 bytes No changes made. To format use '--format'. To resize use '--resize' balchen@blocksize-test:~$ sudo blockdev --report /dev/sdb RO RA SSZ BSZ StartSec Size Device rw 4096 4096 4096 0 1979120934912 /dev/sdb balchen@blocksize-test:~$ sudo blockdev -v --getbsz /dev/sdb get blocksize: 4096 balchen@blocksize-test:~$ sudo blockdev -v --getpbsz /dev/sdb get physical block (sector) size: 8192 balchen@blocksize-test:~$ sudo blockdev -v --getss /dev/sdb get logical block (sector) size: 4096
Linux output seems reasonable on a quick look. Though I would use `sg_readcap --16 /dev/sdb` to avoid any questions. Seems ESXi (at least version you are using) is still broken.
PS: Linux has problems with sectors above page size, that is why I guess you see 4096 in the first case, which should not be there.
So when volblocksize=8K and LUN Block Size=512, Linux reports a block size=4K, physical block size=8K and sector size=512. I don't know how these three parameters are supposed to work together, but these are reasonable to you? Where did 4K come from?
SCSI target can report only 2 values, logical and physical, should be 512 and 8K here. Linux kernel by its current design can not handle logical blocks above 4096. But don't ask me why it reports as a block size here, ask somebody there. My responsibility ends at `sg_readcap --16 /dev/sdb` output, which should return proper values.
This is with LUN Block Size=4K. balchen@blocksize-test:~$ sudo sg_readcap --16 /dev/sdb Read Capacity results: Protection: prot_en=0, p_type=0, p_i_exponent=0 Logical block provisioning: lbpme=1, lbprz=1 Last LBA=483183821 (0x1ccccccd), Number of logical blocks=483183822 Logical block length=4096 bytes Logical blocks per physical block exponent=1 [so physical block length=8192 bytes] Lowest aligned LBA=0 Hence: Device size: 1979120934912 bytes, 1887436.8 MiB, 1979.12 GB balchen@blocksize-test:~$ sudo sg_modes -aa /dev/sdb FreeBSD iSCSI Disk 0123 peripheral_type: disk [0x0] Mode parameter header from MODE SENSE(10): Mode data length=212, medium type=0x00, WP=0, DpoFua=1, longlba=0 Block descriptor length=8 > Direct access device block descriptors: Density code=0x0 00 1c cc cc ce 00 00 10 00 >> Read-Write error recovery, page_control: current 00 01 0a c0 00 00 00 00 80 00 00 00 00 >> Format (obsolete), page_control: current 00 03 16 00 00 00 00 00 00 00 00 01 00 10 00 00 00 10 00 00 00 00 40 00 00 00 >> Rigid disk geometry (obsolete), page_control: current 00 04 16 00 39 99 80 00 00 00 00 00 00 00 00 00 00 10 00 00 00 00 1c 20 00 00 >> Verify error recovery, page_control: current 00 07 0a 00 00 00 00 00 00 00 00 00 00 >> Caching, page_control: current 00 08 12 14 00 ff ff 00 00 ff ff ff ff 00 00 00 00 10 00 00 00 00 >> Control, page_control: current 00 0a 0a 00 00 00 40 00 00 00 00 00 00 >> Control extension, page_control: current 00 4a 01 00 1c 00 00 00 00 00 00 00 00 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> Informational exceptions control, page_control: current 00 1c 0a 10 00 00 00 00 00 00 00 00 01 >> Logical block provisioning, page_control: current 00 5c 02 00 2c 00 00 00 00 00 00 00 00 00 00 00 00 10 00 01 00 00 00 00 00 00 00 02 00 00 00 00 00 00 20 00 f1 00 00 00 00 00 00 00 f2 00 00 00 00 00 00
The Format page from MODE SENSE seems to contain a different physical block size, unless I'm reading this incorrectly. balchen@blocksize-test:~$ sudo sginfo -f /dev/sdb Format Device mode page (0x3) ----------------------------- Tracks per Zone 0 Alternate sectors per zone 0 Alternate tracks per zone 0 Alternate tracks per lu 0 Sectors per track 256 Data bytes per physical sector 4096 Interleave 0 Track skew factor 0 Cylinder skew factor 0 Supports Soft Sectoring 0 Supports Hard Sectoring 1 Removable Medium 0 Surface 0
Format Device mode page reports physical sector size, not physical block size. That concept is used for bad block marking/relocation and appeared very early in SCSI specs, long before logical blocks being formalized in SBC-3. Since Format Device mode page is obsolete since SBC-2 specification, released almost 20 years ago, it is hard to guess what would be its meaning in today's terms, but I suppose physical sector smaller than physical block would not make sense. So I'll update the code to report physical block size there instead of logical block size. But after that I have a strong itch to remove Format Device mode page together with Rigid Disk Geometry mode page, obsoleted for the same 20 years.
Thinking more, those physical sectors should somehow fit into disk CHS geometry, based on logical sectors, and I don't want to get even close to that legacy disaster. I think I'll just remove it all together.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=7c667affb7b09fcdcb81f6f2635e9dfab7bc1fa8 commit 7c667affb7b09fcdcb81f6f2635e9dfab7bc1fa8 Author: Alexander Motin <mav@FreeBSD.org> AuthorDate: 2024-02-27 18:28:44 +0000 Commit: Alexander Motin <mav@FreeBSD.org> CommitDate: 2024-02-27 18:28:44 +0000 CTL: Drop Format Device and Rigid Disk Geometry mode pages Those mode pages are obsolete since SBC-2 specification almost 20 years ago. First I was trying to understand possible relations between physical block and physical sector terms in different specs. Then was thinking about possible relations to device CHS geometry and compatibility issues. Finally I just decided that none of it worth the efforts and should rest in piece. PR: 276524 sys/cam/ctl/ctl.c | 194 ---------------------------------------------- sys/cam/ctl/ctl_private.h | 63 --------------- 2 files changed, 257 deletions(-)
> Where did 4K come from? man blockdev says: --getbsz Print the blocksize in bytes. This size does not describe device topology. It’s the size used internally by the kernel and it may be modified (for example) by filesystem driver on mount. To me it sounds like they just invented it out of thin air. I don't see anything else actionable here, so I am closing the PR. I you think I am wrong, I am open to hear more arguments.