Bug 227313

Summary: net/isboot-kmod works with net/istgt but not with ctld(8)
Product: Ports & Packages Reporter: Maurizio <maurizio1018>
Component: Individual Port(s)Assignee: freebsd-ports-bugs (Nobody) <ports-bugs>
Status: New ---    
Severity: Affects Some People CC: darius, john, maurizio1018, me, milios
Priority: ---    
Version: Latest   
Hardware: Any   
OS: Any   
Attachments:
Description Flags
Let isboot-kmod working with ctld(8)
none
Patch loader to load isboot.ko if iBFT is present. none

Description Maurizio 2018-04-06 08:34:57 UTC
I am not shure this bug is related to net/isboot-kmod or ctld(8).

I am successfully running a diskless TrueOS, a FreeBSD 12-CURRENT based OS, desktop with the net/istgt port installed in a FreeBSD 11-RELEASE server. I am using this setup without any error, but it is a bit slow.
I want to test ctld on my server, but when the diskless PC loads the isboot driver it loops with a lot of these error messages:
“soreceive BHS is not complete, remaining byte(s)=48
do login failed
soreceive BHS is not complete, remaining byte(s)=48
do login failed”
The source code, of the isboot-kmod driver, that prints the error message is:

/* BHS */
    flags = MSG_WAITALL;
    uio.uio_resid = ISCSI_BHS_LEN;
    error = soreceive(sess->so, NULL, &uio, &mp, NULL, &flags);
    if (error) {
        ISBOOT_ERROR("soreceive BHS error %d\n", error);
        return (error);
    }
    if (uio.uio_resid != 0) {
        ISBOOT_ERROR("soreceive BHS is not complete, remaining "
            "byte(s)=%d\n", (int) uio.uio_resid);
        return (EIO);
    }

source file: iscsi.c, proc: isboot_recv_pdu()
The ISCSI_BHS_LEN constant is 48 and seems that no bytes are read.

On the server, in /var/log/messages, I can read a lots of messages like:
“Mar 28 16:32:39 clover-nas2 ctld[59634]: 192.168.0.164: protocol error: received invalid opcode 0x83
Mar 28 16:32:39 clover-nas2 ctld[40784]: child process 59634 terminated with exit status 1
Mar 28 16:32:40 clover-nas2 ctld[59637]: 192.168.0.164: protocol error: received invalid opcode 0x83
Mar 28 16:32:40 clover-nas2 ctld[40784]: child process 59637 terminated with exit status 1”

The isboot driver send an unrecognized opcode 0x83 to the cltd daemon ..., but I cannot continue, I need some help from the community. :-)

Note: the patch for compiling and running isboot-kmod in FreeBSD 12 is at: bug #226982
Comment 1 Chad Jacob Milios 2018-06-01 15:55:41 UTC
I can confirm I am experiencing this same bug between two FreeBSD 11.1-RELEASE-p10/amd64 systems. I had to apply the same patch to isboot or else I'd get fatal trap 12 during boot at the point isboot does its thing. Both happen to be VIMAGE kernels though I am running ctld/istgt on the host for this testing, not a jail.

The ctld and istgt daemons are configured identically as far as I can get them (CHAP required, no restriction for initiator name or ip address, one portal group, one target, one LUN of 512 byte blocksize which is the same zvol of 4096 byte zfs blocksize.) I can confirm CHAP is indeed authenticating in both cases. To be explicit, istgt allows isboot to succeed and ctld doesn't.

I get the exact console and log messages as Maurizio. Furthermore, I then enabled DEBUG in the isboot module to get the following console output when booting through ctld:

...
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
iSCSI boot driver version 0.2.13
IS: Initiator name: iqn.2011-11.org.nuos:08-00-27-a7-cb-a7
NIC0: IP address: 192.168.1.71
NIC0: Prefix: 24
NIC0: Gateway: 192.168.1.1
NIC0: MAC address: 08:00:27:a7:cb:a7
TGT0: Target IP address: 23.111.168.34
TGT0: Target Port: 3260
TGT0: Target LUN: 0
TGT0: Target name: iqn.2011-11.org.nuos:target0
Boot NIC: em0
Configure IPv4 by NIC0
CHAP Type: CHAP
isboot start, thread id=186a0
kproc_start
Attempting to login to iSCSI target and scan all LUNs.
isboot kproc start, thread id=186d9
isboot iscsi start, thread id=186d9
main loop, thread id=186d9
initialize session, thread id=186d9
Initiator: iqn.2011-11.org.nuos:08-00-27-a7-cb-a7
Target: iqn.2011-11.org.nuos:target0
Target IP=23.111.168.34, Port=3260, LUN=0
strdup(joe)3
strdup(passwordhere)12
strdup(iqn.2011-11.org.nuos:08-00-27-a7-cb-a7)38
strdup(iqn.2011-11.org.nuos:target0)28
strdup(CHAP,None)9
strdup(None,CRC32C)11
strdup(None,CRC32C)11
isboot_connect
open socket
try connect...(186d9)
wait connect...
em0: link state changed to UP
old so=0, new so=0xfffff80006424360
isboot_do_login
login start
xmit PDU
recv PDU
isboot_free_mbufext
soreceive BHS is not complete, remaining byte(s)=48
do login failed
boot retry (59)
isboot_connect
open socket
try connect...(186d9)
wait connect...
em0: link state changed to UP
old so=0, new so=0xfffff80006424000
isboot_do_login
login start
xmit PDU
recv PDU
isboot_free_mbufext
soreceive BHS is not complete, remaining byte(s)=48
do login failed
boot retry (58)
isboot_connect
open socket
try connect...(186d9)
wait connect...
em0: link state changed to UP
old so=0, new so=0xfffff80006423a20
isboot_do_login
login start
xmit PDU
recv PDU
isboot_free_mbufext
soreceive BHS is not complete, remaining byte(s)=48
do login failed
boot retry (57)
...repeats...

I haven't gone so far as to dig into what's happening on the wire but if I'm asked I'll be glad to follow up with any assistance or testing.
Comment 2 Maurizio 2018-06-07 12:52:34 UTC
Created attachment 194065 [details]
Let isboot-kmod working with ctld(8)
Comment 3 Maurizio 2018-06-07 13:05:24 UTC
In https://jelmer.uk/klaus/wireshark/blob/4a812d4ad5e61143d7a18b52f1f32a2a369784f6/packet-iscsi.c
are defined the iscsi opcodes:
 {0x03, "Login Command"},
 {0x83, "Login Command (Retry)"},

In my opinion there are two bugs, one in isboot-kmod and one in ctld:
isboot-kmod send the "Login Command (Retry)" (0x83) opcode and not the  "Login Command" (0x03) opcode, ctld accepts only the "Login Command" (0x03) opcode and not the "Login Command (Retry)" (0x83) opcode.

The above patch let the isboot-kmod driver to send the "Login Command" (0x03) opcode and then, if fails, the "Login Command (Retry)" (0x83) opcode.
Comment 4 darius 2018-09-06 13:05:12 UTC
Hi,
I tested this on an 11.1 ESXi VM guest and it works great.

BTW if compiling on 12 it should probably default to VIMAGE being on since that is in GENERIC.

Better yet would be to get it into base, and then have the loader detect the iBFT table and load it automatically.

That would allow unmodified images to be booted which would be Pretty Neat (tm) IMO.
Comment 5 darius 2018-09-07 04:14:29 UTC
Created attachment 196934 [details]
Patch loader to load isboot.ko if iBFT is present.
Comment 6 darius 2018-09-07 04:17:02 UTC
I added a patch for the loader which sets isboot_load to YES if it finds the iBFT.


I am going to see how much work it is to create a patch to merge isboot into base as well.
Comment 7 darius 2018-09-07 04:28:11 UTC
(In reply to darius from comment #4)
Oops, I spoke too soon.. 
isboot_load gets set but the loader doesn't seem to notice for some reason.