Bug 195135

Summary: ctl.conf have set eui/naa option but it is not detected by vmware esxi 5.1/5.5
Product: Base System Reporter: patdung100
Component: binAssignee: Edward Tomasz Napierala <trasz>
Status: Closed FIXED    
Severity: Affects Only Me CC: emaste, trasz
Priority: ---    
Version: 10.1-RELEASE   
Hardware: amd64   
OS: Any   
Attachments:
Description Flags
sample ctl.conf
none
screen capture in vcenter side
none
for comment 2
none
further testing result for comment 4 and 5
none
test with naa, with quote on none

Description patdung100 2014-11-18 12:01:15 UTC
Created attachment 149553 [details]
sample ctl.conf

There is two lun/disk assigned to a target.
But in the vmware esxi iscsi initiator side, it cannot detect the correct EUI/NAA option. Sometimes only single LUN is detected. Sometimes two LUN is detected but the EUI is not the one specified in the config.

For the version:
FreeBSD 10.1 x64.
Comment 1 patdung100 2014-11-18 12:02:13 UTC
Created attachment 149554 [details]
screen capture in vcenter side
Comment 2 patdung100 2014-11-19 04:11:39 UTC
I tried FreeNAS 9.3 nightly 20141119.
I take a look on their ctl.conf and I changed my own setting on FreeBSD 10.1 as below. The result is that the two disk/lun is seen in ESXi correctly.
I am not sure the root cause is quote "" or device-id has to match with option product+serial.

target iqn.2010-09.org.openindiana:02:gp2esx-clu {
  auth-group gp2esx-clu
  portal-group pg-s1-s2
 lun 3 {
  option vendor "FreeBSD"
  option product "iSCSI Disk"
  option revision "1234"
  option insecure_tpc on
  option unmap on
  device-id "iSCSI Disk      0050562df2d800"
  serial 0050562df2d800
  path /dev/zvol/storage/gp2esx1
 }
 lun 4 {
  option vendor "FreeBSD"
  option product "iSCSI Disk"
  option revision "1234"
  option insecure_tpc on
  option unmap on
  option naa 0x6589cfc0000000006b1df6aa2d0f83ed
  device-id "iSCSI Disk      0050562df2d801"
  serial 0050562df2d800
  path /dev/zvol/storage/gp2esx2
 }
}
Comment 3 patdung100 2014-11-19 04:12:13 UTC
Created attachment 149574 [details]
for comment 2
Comment 4 Edward Tomasz Napierala freebsd_committer 2014-11-19 10:23:53 UTC
I don't think it's either: quotes get stripped by ctld, and device-ids in your working configuration don't actually match serial - you have the same serial for both disks.

Can you do a little experiment to determine the root cause of the problem?  First, try to remove the "naa" clause from targets (right now you have it set only for one of them) and see if the problem comes back.  If it doesn't, remove "device-id" lines, without adding "naa" clauses back, and try again.  Then remove "serial" lines, without adding "naa" and "device-id" back, and try again.
Comment 5 patdung100 2014-11-19 12:34:31 UTC
Created attachment 149584 [details]
further testing result for comment 4 and 5
Comment 6 patdung100 2014-11-19 12:40:46 UTC
Comment on attachment 149584 [details]
further testing result for comment 4 and 5

There are two issues or goals in here:

1) sometimes only one lun is seen
After removing device_id or removing serial. Two lun is seen.

2) customizing / recognize by naa
For FreeBSD 10.1 release, the ESXi iscsi client side is always using t10.
I cannot get it to be identified as eui/naa in the ESXi side.

FYI, in FreeNAS 9.3 BETA, I can specify the option naa 0x3[15 char hex]. And it is recognized by ESXi iscsi client side. I am not sure if it is related to VPD support (freeNAS has added some VPD-080/083?h) support.
Comment 7 patdung100 2014-11-19 12:45:09 UTC
Additional testing:

For the same serial number on two LUN in Comment 2 to Comment 4:

The FreeNAS GUI only create a serial number for a target, so all the LUN under the same target have the same serial (but their device ID is different when generated by the GUI).

You can get the FreeNAS software and test.
I extract the config generated by FreeNAS in here:

target iqn.2010-09.org.openindiana:02:gp1esx-clu {
  alias gp1esx-clu
  auth-group no-authentication
  portal-group pg1
  lun 0 {
    option umap on
    path /dev/zvol/storage/gp1esx1-vol1
    blocksize 512
    serial 0050562df2d800
    device-id "iSCSI Disk      0050562df2d8000                 "
    option vendor "FreeBSD"
    option product "iSCSI Disk"
    option revision "0123"
    option naa 0x6589cfc000000000e3e3a9275df70fae
    option insecure_tpc on
  }
  lun 1 {
    option umap on
    path /dev/zvol/storage/gp1esx2-vol1
    blocksize 512
    serial 0050562df2d800
    device-id "iSCSI Disk      0050562df2d8001                 "
    option vendor "FreeBSD"
    option product "iSCSI Disk"
    option revision "0123"
    option naa 0x6589cfc0000000006b1df6aa2d0f83ed
    option insecure_tpc on
}
Comment 8 Edward Tomasz Napierala freebsd_committer 2014-11-20 06:37:56 UTC
Ok, so, to sum up, the problem with VMWare recognizing only one LUN instead of two seems to be caused by using the same device-id for both LUNs, do I get this right?

As for NAA - what happens if you use the FreeNAS config (or at least the naa clauses) in FreeBSD 10.1?
Comment 9 patdung100 2014-11-20 07:26:27 UTC
1.
>Ok, so, to sum up, the problem with VMWare recognizing only one LUN instead of >two seems to be caused by using the same device-id for both LUNs, do I get this >right?

Yes.

2.
>As for NAA - what happens if you use the FreeNAS config (or at least the naa >clauses) in FreeBSD 10.1?

Tried. For FreeBSD 10.1's ctld, esxi only display t10.xxx instead of displaying naa.

If my remembered correctly (it was tested in yesterday):
When trying Linux iscsi client+multipath with FreeBSD 10.1's ctld,
Linux multipath cannot see the naa (a 32char hex string). The client see a combination of the vendor + device-id as the (WWID) identifier (checked by multipath -ll).
Comment 10 Edward Tomasz Napierala freebsd_committer 2014-11-22 15:48:37 UTC
Can you try again with NAA values in quotes, like this:

option naa "0x6589cfc0000000006b1df6aa2d0f83ed"
Comment 11 patdung100 2014-11-23 04:04:02 UTC
Created attachment 149746 [details]
test with naa, with quote on

Below is
1. the setting 

2. result in Linux multipath
3. attached screenshot of result in esxi

Linux/esxi cannot detect the naa option.
Strongly suggest the developer(s) to find a Linux machine to do client side testing.

option naa "0x6589cfc0000000006b1df6aa2d0f83ed"

auth-group alltest {
  auth-type none
}

target iqn.2010-09.org.openindiana:02:alltest {
  auth-group alltest
  portal-group pg-s1-s2
 lun 3 {
  option vendor FreeBSD
  option product "iSCSI Disk"
  option revision "1234"
  option insecure_tpc on
  option unmap on
  option naa "0x6589cfc0000000006b1df6aa2d0f83ed"
  device-id "iSCSI Disk      ODJLMJNJND"
  serial ODJLMJNJND
  path /dev/zvol/storage2/test
 }
 lun 4 {
  option vendor FreeBSD
  option product "iSCSI Disk"
  option revision "1234"
  option insecure_tpc on
  option unmap on
  option naa "0x6589cfc0000000006b1df6aa2d0f83ee"
  device-id "iSCSI Disk      YWM5NJQ5OD"
  serial YWM5NJQ5OD
  path /dev/zvol/storage2/test2
 }

}

LUN Backend       Size (Blocks)   BS Serial Number    Device ID
  0 block               2097152  512 ODJLMJNJND       iSCSI Disk      ODJLMJNJND
      lun_type=0
      num_threads=14
      vendor=FreeBSD
      product=iSCSI Disk
      revision=1234
      insecure_tpc=on
      unmap=on
      naa=0x6589cfc0000000006b1df6aa2d0f83ed
      file=/dev/zvol/storage2/test
      cfiscsi_target=iqn.2010-09.org.openindiana:02:alltest
      cfiscsi_lun=3
      scsiname=iqn.2010-09.org.openindiana:02:alltest,lun,3
  1 block               2097152  512 YWM5NJQ5OD       iSCSI Disk      YWM5NJQ5OD
      lun_type=0
      num_threads=14
      vendor=FreeBSD
      product=iSCSI Disk
      revision=1234
      insecure_tpc=on
      unmap=on
      naa=0x6589cfc0000000006b1df6aa2d0f83ee
      file=/dev/zvol/storage2/test2
      cfiscsi_target=iqn.2010-09.org.openindiana:02:alltest
      cfiscsi_lun=4
      scsiname=iqn.2010-09.org.openindiana:02:alltest,lun,4

Linux multipath result:

# multipath -ll
mpathi (3ffffffffffffffff) dm-4 FreeBSD,iSCSI Disk
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 18:0:0:3 sdc 8:32 active ready running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 17:0:0:3 sdd 8:48 active ready running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 18:0:0:4 sde 8:64 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 17:0:0:4 sdf 8:80 active ready running
Comment 12 Edward Tomasz Napierala freebsd_committer 2014-11-23 12:32:41 UTC
Turns out it's a bug, fixed in r273640 in HEAD, and MFC-ed to 10-STABLE in r273979.  So, upgrading to 10-STABLE should fix it.
Comment 13 Edward Tomasz Napierala freebsd_committer 2014-11-29 21:30:47 UTC
LUN enumeration problem was caused by setting the same device-id for several devices; the problem with NAA not being visible to the initiator was fixed in HEAD and 10-STABLE.