Bug 204642 - 10.2 iSCSI connected initiator to a 10.1 iSCSI Target generates excessive UNMAP/TRIM commands, system unresponsive.
Summary: 10.2 iSCSI connected initiator to a 10.1 iSCSI Target generates excessive UNM...
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: 10.2-RELEASE
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-scsi mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-11-17 20:44 UTC by Christopher Forgeron
Modified: 2015-11-18 13:48 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Christopher Forgeron 2015-11-17 20:44:58 UTC
A FreeBSD 10.2-p7 RELEASE machine connected via CAM iSCSI Initiator to a 10.1p24 RELEASE iSCSI file-based target, on a zpool generates a crazy amount of TRIM commands, locking the system. 

This is a gstat -d on the 10.2 Initiator, once the zpools are mounted, and just a basic bit of activity is started (simple file write, etc - no real load)

dT: 12.546s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s   kBps   ms/d   %busy Name
    0      3      0      0    0.0      3     46   58.3      0      0    0.0    4.2| da0
    0      0      0      0    0.0      0      0    0.0      0      0    0.0    0.0| da0p1
    0      0      0      0    0.0      0      0    0.0      0      0    0.0    0.0| da0p2
    0      3      0      0    0.0      3     46   58.3      0      0    0.0    4.2| da0p3
    0 333485      0      0    0.0      0      0    0.0 333485      0    0.0   48.0| da1
    0 306605      0      0    0.0      0      0    0.0 306605      0    0.0   47.7| da2

Notice the huge number of deletes per second. 

The iSCSI Target doesn't seem to be doing anything, so I suspect these are internally dying/retrying. They don't seem to show up in the sysctl:

# sysctl -a | grep trim
vfs.zfs.trim.max_interval: 1
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.txg_delay: 32
vfs.zfs.trim.enabled: 1
vfs.zfs.vdev.trim_max_pending: 10000
vfs.zfs.vdev.trim_max_active: 64
vfs.zfs.vdev.trim_min_active: 1
vfs.zfs.vdev.trim_on_init: 1
kstat.zfs.misc.zio_trim.failed: 0
kstat.zfs.misc.zio_trim.unsupported: 110
kstat.zfs.misc.zio_trim.success: 11
kstat.zfs.misc.zio_trim.bytes: 118784

I do notice that the delete_method is now switched to 'DISABLE'. It originally was 'UNMAP' after boot, and before the zpools had activity. 

# sysctl -a | grep cam.da
kern.cam.da.2.minimum_cmd_size: 6
kern.cam.da.2.delete_max: 0
kern.cam.da.2.delete_method: DISABLE
kern.cam.da.1.error_inject: 0
kern.cam.da.1.sort_io_queue: 0
kern.cam.da.1.minimum_cmd_size: 6
kern.cam.da.1.delete_max: 0
kern.cam.da.1.delete_method: DISABLE
kern.cam.da.0.error_inject: 0
kern.cam.da.0.sort_io_queue: -1
kern.cam.da.0.minimum_cmd_size: 6
kern.cam.da.0.delete_max: 131072
kern.cam.da.0.delete_method: NONE



For reference, here is the /etc/ctl.conf on the 10.1 iSCSI Target:


portal-group pg0 {
        discovery-auth-group no-authentication
        listen 0.0.0.0
        listen [::]
}

        lun 0 {
                path /pool92/iscsi/iscsi.zvol
                blocksize 4K
                size 5T
                option unmap "on"
                option scsiname "pool92"
                option vendor "pool92"
                option insecure_tpc "on"
        }
}


target iqn.iscsi1.zvol {
        auth-group no-authentication
        portal-group pg0

        lun 0 {
                path /pool92_1/iscsi/iscsi.zvol
                blocksize 4K
                size 5T
                option unmap "on"
                option scsiname "pool92_1"
                option vendor "pool92_1"
                option insecure_tpc "on"
        }
}

Thanks.
Comment 1 Christopher Forgeron 2015-11-18 13:48:18 UTC
Please note Bug 204641, where is is noted that the new ctl in 10.2 doesn't allow UNMAP on a file based iSCSI - Perhaps something is overlooked, and it's not completely disabled. 

I'll be running the tests with a zvol backed iSCSI connection today and update once I have results.