Bug 204646 - 10.2 iSCSI backed zpool shows imporper warnings about non-native block sizes that 10.1 doesn't show
Summary: 10.2 iSCSI backed zpool shows imporper warnings about non-native block sizes ...
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: 10.2-RELEASE
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-scsi mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-11-17 22:01 UTC by Christopher Forgeron
Modified: 2015-11-30 19:40 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Christopher Forgeron 2015-11-17 22:01:05 UTC
This is similar to Bug 197513, but that is a 9.3 system.

Consider this scenario:

Virtual FreeBSD Machine Initiator, with a zpool created out of iSCSI disks.
Physical FreeBSD Machine Target, with a zpool holding a sparse file that is the target for the iSCSI disk. 

- The 10.2 Machines are 10.2-p7 RELEASE, updated via freebsd-update, no custom.
- The 10.1 Machine are 10.1-p24 RELEASE, updated via freebsd-update, no custom.
- iSCSI is all CAM iSCSI, not the old istgt platform. 
- The iSCSI Target is a sparse file, stored on a zpool (not a vdev Target)

The target machine is the same physical machine, with the same zpools - I either boot 10.1 or 10.2 for testing, and use the same zpool/disks 

to ensure nothing is changing. 

If I have a 10.2 iSCSI Initiator (client) connected to a 10.2 iSCSI Target, I get eroneous warnings. 
If I have a 10.1 iSCSI Initiator (client) connected to a 10.1 iSCSI Target, I don't get the warnings. 

On the iSCSI Target, the file that backs the iSCSI disk is stored in a zpool that has a recordsize=64k set. 

It's something in the CAM iSCSI Target code that is reporting the zfs recordsize as the sector size. 

If you use a 10.1 iSCSI Initiator connected to a 10.2 iSCSI Target, or 10.1 iSCSI Target, this is what you see on the Initiator:


#zpool status
  pool: iscsi-nfs
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Tue Nov 17 15:25:51 2015
config:

        NAME                              STATE     READ WRITE CKSUM
        iscsi-nfs                         ONLINE       0     0     0
          diskid/DISK-MYSERIAL%20%20%201  ONLINE       0     0     0

errors: No known data errors

  pool: iscsi-nfs1
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Tue Nov 17 15:25:51 2015
config:

        NAME                              STATE     READ WRITE CKSUM
        iscsi-nfs1                        ONLINE       0     0     0
          diskid/DISK-MYSERIAL%20%20%200  ONLINE       0     0     0

errors: No known data errors

With a 10.2 iSCSI Initiator connected to a 10.2 iSCSI Target, this is what I see on zpool status:

# zpool status
  pool: iscsi-nfs
 state: ONLINE
status: One or more devices are configured to use a non-native block size.
        Expect reduced performance.
action: Replace affected devices with devices that support the
        configured block size, or migrate data to a properly configured
        pool.
  scan: scrub repaired 0 in 0h0m with 0 errors on Tue Nov 17 15:25:51 2015
config:

        NAME                              STATE     READ WRITE CKSUM
        iscsi-nfs                         ONLINE       0     0     0
          diskid/DISK-MYSERIAL%20%20%200  ONLINE       0     0     0  block size: 4096B configured, 65536B native

errors: No known data errors

  pool: iscsi-nfs1
 state: ONLINE
status: One or more devices are configured to use a non-native block size.
        Expect reduced performance.
action: Replace affected devices with devices that support the
        configured block size, or migrate data to a properly configured
        pool.
  scan: scrub repaired 0 in 0h0m with 0 errors on Tue Nov 17 15:25:51 2015
config:

        NAME                              STATE     READ WRITE CKSUM
        iscsi-nfs1                        ONLINE       0     0     0
          diskid/DISK-MYSERIAL%20%20%201  ONLINE       0     0     0  block size: 4096B configured, 65536B native

errors: No known data errors


Notice the block size warnings. 


The /etc/ctl.conf on both of the target machines is:


portal-group pg0 {
        discovery-auth-group no-authentication
        listen 0.0.0.0
        listen [::]
}

        lun 0 {
                path /pool92/iscsi/iscsi.zvol
                blocksize 4K
                size 5T
                option unmap "on"
                option scsiname "pool92"
                option vendor "pool92"
                option insecure_tpc "on"
        }
}


target iqn.iscsi1.zvol {
        auth-group no-authentication
        portal-group pg0

        lun 0 {
                path /pool92_1/iscsi/iscsi.zvol
                blocksize 4K
                size 5T
                option unmap "on"
                option scsiname "pool92_1"
                option vendor "pool92_1"
                option insecure_tpc "on"
        }
}


Note the 4k size for my iSCSI Blocks. 


I believe this error message to be incorrect, as recordsize=64k on the Target will not affect the ashift value at all, which is what this warning is really designed for, correct? It's going to warn when you have ashift=9 on a 4k drive, or similar mis-matches. 

The last interesting bit is that somehow the iSCSI Initiator is believing that the Target's sector size _is_ the zfs recordsize, as a zpool creation with a iSCSI drive in this situation trys zet the zpool's ashift to vfs.zfs.max_auto_ashift of 13 (8k Sectors)


Thanks.
Comment 1 Christopher Forgeron 2015-11-18 19:14:51 UTC
Continuing to dig on this, I see that ctl is intentionally reporting volblocksize as the physical sector size. The 'blocksize' in ctl.conf is the logical sector size. 

I'm not sure I agree with this setup - However, perhaps I misunderstand volblocksize.

With recordsize, the size of the write to disk will be a multiple of ashift, up to a max of recordsize.

With volblocksize, is the write _always_ volblocksize in size? ie it really is acting like a sector size?

If so, this behaviour may be correct - If not, then ctl should be advertising the translated ashift value.
Comment 2 Alexander Motin freebsd_committer 2015-11-30 19:40:23 UTC
recordsize and volblocksize properties are functionally equal, except that one of them is used for datasets, while another is for zvols.  Both of them CTL report to initiator as a physical sector size, since that is smallest item that can be written to disks or read back, exactly as for real physical sectors.

So I don't see any problem from CTL side here. The only problem I see is in ZFS on initiator side -- it never sets ashift above 13 (8K), that is reasonable, but it still complains about that as an error, that is pointless.  I think ZFS code should be fixed to just ignore any physical block sizes above 8K, as if they would not be reported.