After upgrading to FreeBSD 15, ctld behavior has changed compared to FreeBSD 12, 13. The same configuration works correctly on previous releases but fails on FreeBSD 15. Two issues are observed: ctld no longer allows multiple physical ports to be assigned to the same target. Frontend CAM target ports are not automatically set Online when ctld starts and must be enabled manually using ctladm. Steps to reproduce: Create the following /etc/ctl.conf: lun example_1 { path /dev/zvol/zroot/lun1 option naa 0x50015178f369f093 } target naa.50015178f369f092 { port isp0 port isp1 lun 0 example_1 } Start ctld with: service ctld onestart Actual result: ctld fails to start with the error: ctld: cannot set multiple physical ports for target "naa.50015178f369f092" ctld: configuration error; exiting If ctld is started with a single port, CAM target ports remain Offline until manually enabled with: ctladm port -o on Expected result: ctld should allow multiple physical ports (for example isp0 and isp1) to be assigned to a single target, as in FreeBSD 12–13. ctld should automatically bring frontend ports Online when the service starts, without requiring manual ctladm commands. Additional information: ctladm portlist -v output is identical on FreeBSD 12, 13, 14, and 15. The difference is only in runtime behavior on FreeBSD 15. The issue was reproduced with both QLogic ISP and Emulex ocs_fc drivers, indicating it is not driver-specific.
To help narrow down where a regression may have occurred, have you tested FreeBSD 14 as well? Or did you move from 13 directly to 15?
This was discussed outside of bugzilla, and mav believes that the regression was introduced in 969876fcee57ea1cb1c7b4d2ee757793cbfbe353 .
I have a patch to support multiple ports per target. It compiles, but I have no way to test it. The patch is available here: https://reviews.freebsd.org/D55767 The lack of enabling the port I'm a bit more puzzled by. In theory, conf::apply should call port::kernel_add which should bring the port up. My guess is that for some reason conf::apply thinks the port already exists in the old configuration. Can you please run ctld with syslog enabled to log all messages (including debug) for ctld to a log file? For example: ``` !ctld *.* /var/log/ctld.log ``` in /etc/syslog.d/ctld.log.
Hi, Multiple port issue seems fixed, however port online still there. as you said port::kernel_add() is only called for new ports, not existing ones. On first start, ports already exist in the kernel like this case not enabled. Also ctld reload case could be an issue in this situation. # ctld -d -f /etc/ctl.conf ctld: obtaining configuration from /etc/ctl.conf ctld: /etc/ctl.conf is world-readable ctld: auth-group "default" not defined; going with defaults ctld: portal-group "default" not defined; going with defaults ctld: transport-group "default" not defined; going with defaults ctld: opening pidfile /var/run/ctld.pid ctld: obtaining previously configured CTL luns from the kernel ctld: CTL port 0 "camsim" wasn't managed by ctld; ctld: CTL port 1 "ioctl" wasn't managed by ctld; ctld: CTL port 2 "tpc" wasn't managed by ctld; ctld: CTL port 3 "ocs_fc0" wasn't managed by ctld; ctld: CTL port 4 "ocs_fc1" wasn't managed by ctld; ctld: adding lun "example_1" ctld: adding port "default-naa.50015178f369f092" ctld: listening on 0.0.0.0, portal-group "default" ctld: listening on [::], portal-group "default" ctld: not listening on transport-group "default", not assigned to any target # ctladm portlist -v Port Online Frontend Name pp vp 0 NO camsim camsim 0 0 naa.5000000ccf2b8f01 Target: naa.5000000ccf2b8f00 All LUNs mapped port_type=8 1 YES ioctl ioctl 0 0 All LUNs mapped port_type=4 2 YES tpc tpc 0 0 All LUNs mapped port_type=8 3 NO camtgt ocs_fc0 0 0 All LUNs mapped port_type=1 4 NO camtgt ocs_fc1 0 0 All LUNs mapped port_type=1 5 YES iscsi iscsi 256 1 naa.50015178f369f092,t,0x0100 Target: naa.50015178f369f092 LUN 0: 0 port_type=16 cfiscsi_state=1 cfiscsi_target=naa.50015178f369f092 ctld_portal_group_name=default cfiscsi_portal_group_tag=256 # ctladm port -o on Front End Ports enabled # ctladm portlist -v Port Online Frontend Name pp vp 0 YES camsim camsim 0 0 naa.5000000ccf2b8f01 Target: naa.5000000ccf2b8f00 All LUNs mapped port_type=8 1 YES ioctl ioctl 0 0 All LUNs mapped port_type=4 2 YES tpc tpc 0 0 All LUNs mapped port_type=8 3 YES camtgt ocs_fc0 0 0 naa.100000109b223c61 Target: naa.200000109b223c61 All LUNs mapped port_type=1 4 YES camtgt ocs_fc1 0 0 naa.100000109b223c62 Target: naa.200000109b223c62 All LUNs mapped port_type=1 5 YES iscsi iscsi 256 1 naa.50015178f369f092,t,0x0100 Target: naa.50015178f369f092 LUN 0: 0 port_type=16 cfiscsi_state=1 cfiscsi_target=naa.50015178f369f092 ctld_portal_group_name=default cfiscsi_portal_group_tag=256
Just to help clarify, how are the physical ports being created? Is the driver adding them directly, are you using ctladm to create them? In particular, I assume the ports are already present before you start ctld the first time? ctld reload should be ok as the ports should in theory already be "up" so there's no need to do anything. Restarting ctld entirely won't remove/readd these ports though it seems, instead it seems like it just wants to do a disable/enable. My understanding of the code flow is that for these physical ports, conf_new_from_kernel() should add the port to `kports` without adding a corresponding port in the configuration returned by conf_new_from_kernel(). And we are seeing this log message: void add_iscsi_port(struct kports &kports, struct conf *conf, const struct cctl_port &port, std::string &name) { if (port.cfiscsi_target.empty()) { log_debugx("CTL port %u \"%s\" wasn't managed by ctld; ", port.port_id, name.c_str()); if (!kports.has_port(name)) { if (!kports.add_port(name, port.port_id)) { log_warnx("kports::add_port failed"); return; } } return; } Back in main(), this call should add ports to the `newconf`: if (!newconf->add_pports(kports)) log_errx(1, "Error associating physical ports; exiting"); Can you either use stepping with gdb/lldb or add some tracing to see if conf::add_pports() is actually adding ports? Something like: diff --git a/usr.sbin/ctld/ctld.cc b/usr.sbin/ctld/ctld.cc index 551a70b3fa8f..96e3885e92fe 100644 --- a/usr.sbin/ctld/ctld.cc +++ b/usr.sbin/ctld/ctld.cc @@ -2140,6 +2140,7 @@ conf::apply(struct conf *oldconf) port *newport = it->second.get(); if (newport->is_dummy()) { + log_debugx("skipping dummy port \"%s\"", name.c_str()); it++; continue; } @@ -2630,6 +2631,8 @@ conf::add_pports(struct kports &kports) "for %s", targ->label()); return (false); } + log_debugx("added ioctl port \"%s\" for %s", + pport.c_str(), targ->label()); continue; } @@ -2651,6 +2654,8 @@ conf::add_pports(struct kports &kports) pport.c_str(), targ->label()); return (false); } + log_debugx("added kernel port \"%s\" for %s", + pport.c_str(), targ->label()); } } return (true);
(In reply to John Baldwin from comment #5) I will try to capture with gdb next week. Meanwhile i realize that even ctl.conf target config port is ocs_fc, ctladm portlist show this target mapped to iscsi ! # cat /etc/ctl.conf lun example_1 { path /dev/zvol/zroot/lun1 option naa 0x50015178f369f093 } target naa.123456789 { port ocs_fc0 port ocs_fc1 lun 0 example_1 } # ctladm portlist -v Port Online Frontend Name pp vp 0 NO camsim camsim 0 0 naa.5000000a87f74f01 Target: naa.5000000a87f74f00 All LUNs mapped port_type=8 1 YES ioctl ioctl 0 0 All LUNs mapped port_type=4 2 YES tpc tpc 0 0 All LUNs mapped port_type=8 3 NO camtgt ocs_fc0 0 0 All LUNs mapped port_type=1 4 NO camtgt ocs_fc1 0 0 All LUNs mapped port_type=1 5 YES iscsi iscsi 256 1 naa.123456789,t,0x0100 Target: naa.123456789 LUN 0: 0 port_type=16 cfiscsi_state=1 cfiscsi_target=naa.123456789 ctld_portal_group_name=default cfiscsi_portal_group_tag=256
(In reply to Ken J. Thomson from comment #6) I do think that if ctld fails to create the actual internal ports for the physical ports, then any targets gets an "automatic" iscsi port created. So I think the issue is that for some reason the internal ports aren't being created for the physical ports. This would also explain why they aren't being enabled. I still don't know why they aren't being added though.
Reading symbols from ctld... Reading symbols from /usr/lib/debug//usr/sbin/ctld.debug... (gdb) break conf::apply Breakpoint 1 at 0x46965: file /usr/src/usr.sbin/ctld/ctld.cc, line 1967. (gdb) run -d -f /etc/ctl.conf Starting program: /usr/sbin/ctld -d -f /etc/ctl.conf ctld: obtaining configuration from /etc/ctl.conf ctld: /etc/ctl.conf is world-readable ctld: auth-group "default" not defined; going with defaults ctld: portal-group "default" not defined; going with defaults ctld: transport-group "default" not defined; going with defaults ctld: opening pidfile /var/run/ctld.pid ctld: obtaining previously configured CTL luns from the kernel ctld: CTL port 0 "camsim" wasn't managed by ctld; ctld: CTL port 1 "ioctl" wasn't managed by ctld; ctld: CTL port 2 "tpc" wasn't managed by ctld; ctld: CTL port 3 "ocs_fc0" wasn't managed by ctld; ctld: CTL port 4 "ocs_fc1" wasn't managed by ctld; ctld: added kernel port "ocs_fc0" for target "naa.123456789" ctld: added kernel port "ocs_fc1" for target "naa.123456789" Breakpoint 1, conf::apply (this=0x801a1f000, oldconf=0x801a1f1c0) at /usr/src/usr.sbin/ctld/ctld.cc:1967 1967 int cumulated_error = 0; (gdb) n 1969 if (oldconf->conf_debug != conf_debug) { (gdb) n 1982 if (!oldconf->conf_pidfile_path.empty() && (gdb) 2001 for (auto &kv : conf_portal_groups) { (gdb) n 2002 struct portal_group &newpg = *kv.second; (gdb) n 2004 if (newpg.tag() != 0) (gdb) n 2006 auto it = oldconf->conf_portal_groups.find(kv.first); (gdb) n 2007 if (it != oldconf->conf_portal_groups.end()) (gdb) n 2010 newpg.allocate_tag(); (gdb) n 2001 for (auto &kv : conf_portal_groups) { (gdb) n 2012 for (auto &kv : conf_transport_groups) { (gdb) n 2013 struct portal_group &newpg = *kv.second; (gdb) n 2015 if (newpg.tag() != 0) (gdb) n 2017 auto it = oldconf->conf_transport_groups.find(kv.first); (gdb) n 2018 if (it != oldconf->conf_transport_groups.end()) (gdb) n 2021 newpg.allocate_tag(); (gdb) n 2012 for (auto &kv : conf_transport_groups) { (gdb) n 2025 for (auto &kv : oldconf->conf_isns) { (gdb) n 2041 for (const auto &kv : oldconf->conf_ports) { (gdb) n 2065 for (auto it = oldconf->conf_luns.begin(); (gdb) n 2066 it != oldconf->conf_luns.end(); ) { (gdb) n 2065 for (auto it = oldconf->conf_luns.begin(); (gdb) n 2108 for (auto it = conf_luns.begin(); it != conf_luns.end(); ) { (gdb) n 2109 struct lun *newlun = it->second.get(); (gdb) n 2111 auto oldit = oldconf->conf_luns.find(it->first); (gdb) n 2112 if (oldit != oldconf->conf_luns.end()) { (gdb) n 2125 log_debugx("adding lun \"%s\"", newlun->name()); (gdb) n ctld: adding lun "example_1" 2126 if (!newlun->kernel_add()) { (gdb) n 2132 it++; (gdb) n 2108 for (auto it = conf_luns.begin(); it != conf_luns.end(); ) { (gdb) n 2138 for (auto it = conf_ports.begin(); it != conf_ports.end(); ) { (gdb) n 2139 const std::string &name = it->first; (gdb) print it->first.c_str() $1 = (const std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::value_type *) 0x801a2f161 "ocs_fc1-naa.123456789" (gdb) print it->second->is_dummy() $2 = true (gdb) print it->second->p_type There is no member or method named p_type. (gdb) break port::kernel_add Breakpoint 2 at 0x10a028e: file /usr/src/usr.sbin/ctld/kernel.cc, line 850. (gdb) continue Continuing. ctld: skipping dummy port "ocs_fc1-naa.123456789" ctld: skipping dummy port "ocs_fc0-naa.123456789" ctld: adding port "default-naa.123456789" Breakpoint 2, port::kernel_add (this=0x801a2f180) at /usr/src/usr.sbin/ctld/kernel.cc:850 850 struct target *targ = p_target;
Ah, try this fix (though I'm not currently sure why an iscsi port is created now): diff --git a/usr.sbin/ctld/ctld.hh b/usr.sbin/ctld/ctld.hh index 7ff6a0bc6d5f..7eb86e6cc535 100644 --- a/usr.sbin/ctld/ctld.hh +++ b/usr.sbin/ctld/ctld.hh @@ -269,7 +269,7 @@ struct port { virtual struct auth_group *auth_group() const { return nullptr; } virtual struct portal_group *portal_group() const { return nullptr; } - virtual bool is_dummy() const { return true; } + virtual bool is_dummy() const { return false; } virtual void clear_references(); Please let me see the output of ctladm portliest -v after trying this.
(In reply to John Baldwin from comment #9) As you said iscsi still creating the target. And ctld reload didnt enabled the ports. However ctld start now opened the ports after last patch. # ctladm portlist -v Port Online Frontend Name pp vp 0 NO camsim camsim 0 0 naa.5000000c7fee4b01 Target: naa.5000000c7fee4b00 All LUNs mapped port_type=8 1 YES ioctl ioctl 0 0 All LUNs mapped port_type=4 2 YES tpc tpc 0 0 All LUNs mapped port_type=8 3 NO camtgt ocs_fc0 0 0 All LUNs mapped port_type=1 4 NO camtgt ocs_fc1 0 0 All LUNs mapped port_type=1 # service ctld onestart Starting ctld. ctld: /etc/ctl.conf is world-readable [root@bsd15 ~]# ctladm portlist -v Port Online Frontend Name pp vp 0 NO camsim camsim 0 0 naa.5000000c7fee4b01 Target: naa.5000000c7fee4b00 All LUNs mapped port_type=8 1 YES ioctl ioctl 0 0 All LUNs mapped port_type=4 2 YES tpc tpc 0 0 All LUNs mapped port_type=8 3 YES camtgt ocs_fc0 0 0 naa.100000109b223c61 Target: naa.200000109b223c61 LUN 0: 0 port_type=1 4 NO camtgt ocs_fc1 0 0 All LUNs mapped port_type=1 5 YES iscsi iscsi 256 1 naa.123456789,t,0x0100 Target: naa.123456789 LUN 0: 0 port_type=16 cfiscsi_state=1 cfiscsi_target=naa.123456789 ctld_portal_group_name=default cfiscsi_portal_group_tag=25 #-- add second interface to target --# # cat /etc/ctl.conf lun example_1 { path /dev/zvol/zroot/lun1 option naa 0x50015178f369f093 } target naa.123456789 { port ocs_fc0 port ocs_fc1 lun 0 example_1 } root@bsd15 ~]# service ctld onereload [root@bsd15 ~]# ctladm portlist -v Port Online Frontend Name pp vp 0 NO camsim camsim 0 0 naa.5000000c7fee4b01 Target: naa.5000000c7fee4b00 All LUNs mapped port_type=8 1 YES ioctl ioctl 0 0 All LUNs mapped port_type=4 2 YES tpc tpc 0 0 All LUNs mapped port_type=8 3 YES camtgt ocs_fc0 0 0 naa.100000109b223c61 Target: naa.200000109b223c61 LUN 0: 0 port_type=1 4 NO camtgt ocs_fc1 0 0 All LUNs mapped port_type=1 5 YES iscsi iscsi 256 1 naa.123456789,t,0x0100 Target: naa.123456789 LUN 0: 0 port_type=16 cfiscsi_state=1 cfiscsi_target=naa.123456789 ctld_portal_group_name=default cfiscsi_portal_group_tag=256 # -- ctld reload do not enabled the port !!! [root@bsd15 ~]# service ctld onerestart Stopping ctld. Waiting for PIDS: 10307. Starting ctld. ctld: /etc/ctl.conf is world-readable [root@bsd15 ~]# ctladm portlist -v Port Online Frontend Name pp vp 0 NO camsim camsim 0 0 naa.5000000c7fee4b01 Target: naa.5000000c7fee4b00 All LUNs mapped port_type=8 1 YES ioctl ioctl 0 0 All LUNs mapped port_type=4 2 YES tpc tpc 0 0 All LUNs mapped port_type=8 3 YES camtgt ocs_fc0 0 0 naa.100000109b223c61 Target: naa.200000109b223c61 LUN 0: 0 port_type=1 4 YES camtgt ocs_fc1 0 0 naa.100000109b223c62 Target: naa.200000109b223c62 LUN 0: 0 port_type=1 5 YES iscsi iscsi 256 1 naa.123456789,t,0x0100 Target: naa.123456789 LUN 0: 0 port_type=16 cfiscsi_state=1 cfiscsi_target=naa.123456789 ctld_portal_group_name=default cfiscsi_portal_group_tag=256
I tried below approach, now iscsi didnt mapped to default portal group. target::verify() { if (t_auth_group == nullptr) { t_auth_group = t_conf->find_auth_group("default"); assert(t_auth_group != nullptr); } - if (t_ports.empty()){ + if (t_ports.empty() && t_pports.empty()) { struct portal_group *pg = default_portal_group(); assert(pg != NULL); t_conf->add_port(this, pg, nullptr); }
I tried below approach at ctld.cc, now iscsi didnt mapped to default portal group. target::verify() { if (t_auth_group == nullptr) { t_auth_group = t_conf->find_auth_group("default"); assert(t_auth_group != nullptr); } - if (t_ports.empty()){ + if (t_ports.empty() && t_pports.empty()) { struct portal_group *pg = default_portal_group(); assert(pg != NULL); t_conf->add_port(this, pg, nullptr); }
Created attachment 269544 [details] 4 patch to address current issues Can someone review them ? In my testing these are performed as expected. Thanks.