Bug 248630 - sesutil does not report all drives in an MSA60, locate LED is off by +1
Summary: sesutil does not report all drives in an MSA60, locate LED is off by +1
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.1-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-08-12 16:47 UTC by Mike Biesele
Modified: 2020-08-26 20:01 UTC (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mike Biesele 2020-08-12 16:47:56 UTC
On an HPE MSA60 fully populated with 12 SATA disks, sesutil map only shows 11 disks.  Possibly related, using 'sesutil locate' using either disk or sesid affects the wrong slot in the shelf by +1.  That is, given disk da1 in physical slot 1 in the shelf, 'sesutil locate da1 on' illuminates the locate LED in slot 2.  sas2ircu reports the shelf contents accurately, however.  

System is an HPE ProLiant DL360p Gen8.  HBA is an Avago LSI 2008 flashed with the IT firmware.  

Disks da1-da12 are in the MSA60. 

library ~ # camcontrol devlist
<HP RAID 1(1+0) OK>                at scbus1 target 0 lun 0 (pass0,da0)
<ATA WDC WD2000FYYZ-0 1K02>        at scbus3 target 31 lun 0 (pass1,da1)
<ATA WDC WD2000FYYZ-0 1K02>        at scbus3 target 33 lun 0 (pass3,da3)
<ATA WDC WD2000FYYZ-0 1K02>        at scbus3 target 34 lun 0 (pass4,da4)
<ATA WDC WD2000FYYZ-0 1K02>        at scbus3 target 35 lun 0 (pass5,da5)
<ATA WDC WD2000FYYZ-0 1K02>        at scbus3 target 36 lun 0 (da6,pass6)
<ATA WDC WD2000FYYZ-0 1K02>        at scbus3 target 37 lun 0 (pass7,da7)
<ATA WDC WD2000FYYZ-0 1K02>        at scbus3 target 38 lun 0 (pass8,da8)
<ATA WDC WD2000FYYZ-0 1K02>        at scbus3 target 39 lun 0 (pass9,da9)
<ATA WDC WD2000FYYZ-0 1K02>        at scbus3 target 40 lun 0 (pass10,da10)
<ATA WDC WD2000FYYZ-0 1K02>        at scbus3 target 41 lun 0 (pass11,da11)
<ATA WDC WD2000FYYZ-0 1K02>        at scbus3 target 42 lun 0 (pass12,da12)
<HP MSA60 2.18>                    at scbus3 target 43 lun 0 (ses0,pass13)
<IBM-ESXS ST32000444SS BC29>       at scbus3 target 44 lun 0 (da2,pass2)
<hp DVD ROM UJ8C2 DC02>            at scbus4 target 0 lun 0 (cd0,pass14)

library ~ # sesutil map
ses0:
	Enclosure Name: HP MSA60 2.18
	Enclosure ID: 5001438001571280
	Element 0, Type: Array Device Slot
		Status: Unsupported (0x00 0x00 0x00 0x00)
	Element 1, Type: Array Device Slot
		Status: OK (0x01 0x00 0x00 0x00)
	Element 2, Type: Array Device Slot
		Status: OK (0x01 0x00 0x02 0x00)
		Device Names: da1,pass1
		Extra status:
		- LED=locate
	Element 3, Type: Array Device Slot
		Status: OK (0x01 0x00 0x00 0x00)
		Device Names: da2,pass2
	Element 4, Type: Array Device Slot
		Status: OK (0x01 0x00 0x00 0x00)
		Device Names: da3,pass3
	Element 5, Type: Array Device Slot
		Status: OK (0x01 0x00 0x00 0x00)
		Device Names: da4,pass4
	Element 6, Type: Array Device Slot
		Status: OK (0x01 0x00 0x00 0x00)
		Device Names: da5,pass5
	Element 7, Type: Array Device Slot
		Status: OK (0x01 0x00 0x00 0x00)
		Device Names: da6,pass6
	Element 8, Type: Array Device Slot
		Status: OK (0x01 0x00 0x00 0x00)
		Device Names: da7,pass7
	Element 9, Type: Array Device Slot
		Status: OK (0x01 0x00 0x00 0x00)
		Device Names: da8,pass8
	Element 10, Type: Array Device Slot
		Status: OK (0x01 0x00 0x00 0x00)
		Device Names: da9,pass9
	Element 11, Type: Array Device Slot
		Status: OK (0x01 0x00 0x00 0x00)
		Device Names: da10,pass10
	Element 12, Type: Array Device Slot
		Status: OK (0x01 0x00 0x00 0x00)
		Device Names: da11,pass11
	Element 13, Type: Power Supply
		Status: Unsupported (0x00 0x00 0x00 0x00)
	Element 14, Type: Power Supply
		Status: OK (0x11 0x00 0x00 0x20)
		Extra status:
		- Swapped
	Element 15, Type: Power Supply
		Status: OK (0x11 0x00 0x00 0x20)
		Extra status:
		- Swapped
	Element 16, Type: Cooling
		Status: Unsupported (0x00 0x00 0x00 0x00)
		Extra status:
		- Speed: 0 rpm
	Element 17, Type: Cooling
		Status: OK (0x01 0x01 0xa4 0x21)
		Extra status:
		- Speed: 4200 rpm
	Element 18, Type: Cooling
		Status: OK (0x01 0x01 0x68 0x21)
		Extra status:
		- Speed: 3600 rpm
	Element 19, Type: Cooling
		Status: Unsupported (0x00 0x00 0x00 0x00)
		Extra status:
		- Speed: 0 rpm
	Element 20, Type: Cooling
		Status: OK (0x01 0x01 0x68 0x21)
		Extra status:
		- Speed: 3600 rpm
	Element 21, Type: Cooling
		Status: OK (0x01 0x01 0x68 0x21)
		Extra status:
		- Speed: 3600 rpm
	Element 22, Type: Temperature Sensors
		Status: Unsupported (0x00 0x00 0x00 0x00)
		Extra status:
		- Temperature: -reserved-
	Element 23, Type: Temperature Sensors
		Status: OK (0x01 0x00 0x2f 0x00)
		Extra status:
		- Temperature: 27 C
	Element 24, Type: Temperature Sensors
		Status: OK (0x01 0x00 0x2f 0x00)
		Extra status:
		- Temperature: 27 C
	Element 25, Type: Temperature Sensors
		Status: OK (0x01 0x00 0x2c 0x00)
		Extra status:
		- Temperature: 24 C
	Element 26, Type: Temperature Sensors
		Status: OK (0x01 0x00 0x2c 0x00)
		Extra status:
		- Temperature: 24 C
	Element 27, Type: Enclosure Services Controller Electronics
		Status: Unsupported (0x00 0x00 0x00 0x00)
	Element 28, Type: Enclosure Services Controller Electronics
		Status: OK (0x01 0x00 0x01 0x00)
	Element 29, Type: Enclosure
		Status: Unsupported (0x00 0x00 0x00 0x00)
	Element 30, Type: Enclosure
		Status: OK (0x01 0x00 0x00 0x00)
	Element 31, Type: SAS Expander
		Status: Unsupported (0x00 0x00 0x00 0x00)
	Element 32, Type: SAS Expander
		Status: OK (0x01 0x00 0x00 0x00)
	Element 33, Type: SAS Connector
		Status: Unsupported (0x00 0x00 0x00 0x00)
	Element 34, Type: SAS Connector
		Status: OK (0x01 0x20 0x01 0x00)
	Element 35, Type: SAS Connector
		Status: OK (0x01 0x20 0x01 0x00)
	Element 36, Type: SAS Connector
		Status: OK (0x01 0x20 0x01 0x00)
	Element 37, Type: SAS Connector
		Status: OK (0x01 0x20 0x01 0x00)
	Element 38, Type: SAS Connector
		Status: OK (0x01 0x20 0x01 0x00)
	Element 39, Type: SAS Connector
		Status: OK (0x01 0x20 0x01 0x00)
	Element 40, Type: SAS Connector
		Status: OK (0x01 0x20 0x01 0x00)
	Element 41, Type: SAS Connector
		Status: OK (0x01 0x20 0x01 0x00)
	Element 42, Type: SAS Connector
		Status: OK (0x01 0x20 0x01 0x00)
	Element 43, Type: SAS Connector
		Status: OK (0x01 0x20 0x01 0x00)
	Element 44, Type: SAS Connector
		Status: OK (0x01 0x20 0x01 0x00)
	Element 45, Type: SAS Connector
		Status: OK (0x01 0x20 0x01 0x00)
	Element 46, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 47, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 48, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 49, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 50, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 51, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 52, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 53, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 54, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 55, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 56, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 57, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 58, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 59, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 60, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 61, Type: SAS Connector
		Status: Unsupported (0x20 0x00 0x00 0x00)
		Extra status:
		- Disabled
	Element 62, Type: SAS Connector
		Status: OK (0x01 0x02 0x00 0x00)
	Element 63, Type: SAS Connector
		Status: OK (0x01 0x02 0x01 0x00)
	Element 64, Type: SAS Connector
		Status: OK (0x01 0x02 0x02 0x00)
	Element 65, Type: SAS Connector
		Status: OK (0x01 0x02 0x03 0x00)
	Element 66, Type: SAS Connector
		Status: OK (0x01 0x02 0x00 0x00)
	Element 67, Type: SAS Connector
		Status: OK (0x01 0x02 0x01 0x00)
	Element 68, Type: SAS Connector
		Status: OK (0x01 0x02 0x02 0x00)
	Element 69, Type: SAS Connector
		Status: OK (0x01 0x02 0x03 0x00)
	Element 70, Type: <Type 0x80>
		Status: Unsupported (0x00 0x00 0x00 0x00)
	Element 71, Type: <Type 0x80>
		Status: OK (0x01 0x20 0xff 0xff)

library ~ # sas2ircu 1 display
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.

Read configuration has been initiated for controller 1
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type                         : SAS2008
  BIOS version                            : 7.33.00.00
  Firmware version                        : 20.00.07.00
  Channel description                     : 1 Serial Attached SCSI
  Initiator ID                            : 0
  Maximum physical devices                : 255
  Concurrent commands supported           : 3432
  Slot                                    : 2
  Segment                                 : 0
  Bus                                     : 7
  Device                                  : 0
  Function                                : 0
  RAID Support                            : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0

Device is a Enclosure services device
  Enclosure #                             : 2
  Slot #                                  : 0
  SAS Address                             : 5001438-0-0157-12a5
  State                                   : Standby (SBY)
  Manufacturer                            : HP
  Model Number                            : MSA60
  Firmware Revision                       : 2.18
  Serial No                               :
  GUID                                    : N/A
  Protocol                                : SAS
  Device Type                             : Enclosure services device

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 1
  SAS Address                             : 5001438-0-0157-1281
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA
  Model Number                            : WDC WD2000FYYZ-0
  Firmware Revision                       : 1K02
  Serial No                               : WDWMC1P0054868
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 2
  SAS Address                             : 5000c50-0-34cd-2731
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : IBM-ESXS
  Model Number                            : ST32000444SS
  Firmware Revision                       : BC29
  Serial No                               : 9WM78S1V0428BC29
  GUID                                    : N/A
  Protocol                                : SAS
  Drive Type                              : SAS_HDD

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 3
  SAS Address                             : 5001438-0-0157-1283
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA
  Model Number                            : WDC WD2000FYYZ-0
  Firmware Revision                       : 1K02
  Serial No                               : WDWMC1P0087547
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 4
  SAS Address                             : 5001438-0-0157-1284
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA
  Model Number                            : WDC WD2000FYYZ-0
  Firmware Revision                       : 1K02
  Serial No                               : WDWMC1P0087107
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 5
  SAS Address                             : 5001438-0-0157-1285
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA
  Model Number                            : WDC WD2000FYYZ-0
  Firmware Revision                       : 1K02
  Serial No                               : WDWMC1P0212052
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 6
  SAS Address                             : 5001438-0-0157-1286
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA
  Model Number                            : WDC WD2000FYYZ-0
  Firmware Revision                       : 1K02
  Serial No                               : WDWMC1P0213822
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 7
  SAS Address                             : 5001438-0-0157-1287
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA
  Model Number                            : WDC WD2000FYYZ-0
  Firmware Revision                       : 1K02
  Serial No                               : WDWMC1P0215894
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 8
  SAS Address                             : 5001438-0-0157-1288
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA
  Model Number                            : WDC WD2000FYYZ-0
  Firmware Revision                       : 1K02
  Serial No                               : WDWMC1P0053015
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 9
  SAS Address                             : 5001438-0-0157-1289
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA
  Model Number                            : WDC WD2000FYYZ-0
  Firmware Revision                       : 1K02
  Serial No                               : WDWMC1P0053035
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 10
  SAS Address                             : 5001438-0-0157-128a
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA
  Model Number                            : WDC WD2000FYYZ-0
  Firmware Revision                       : 1K02
  Serial No                               : WDWMC1P0116027
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 11
  SAS Address                             : 5001438-0-0157-128b
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA
  Model Number                            : WDC WD2000FYYZ-0
  Firmware Revision                       : 1K02
  Serial No                               : WDWMC1P0115889
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 12
  SAS Address                             : 5001438-0-0157-128c
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA
  Model Number                            : WDC WD2000FYYZ-0
  Firmware Revision                       : 1K02
  Serial No                               : WDWMC1P0115969
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
  Enclosure#                              : 1
  Logical ID                              : 500605b0:0400cf00
  Numslots                                : 8
  StartSlot                               : 0
  Enclosure#                              : 2
  Logical ID                              : 50014380:01571280
  Numslots                                : 13
  StartSlot                               : 1
------------------------------------------------------------------------
SAS2IRCU: Command DISPLAY Completed Successfully.
SAS2IRCU: Utility Completed Successfully.

library ~ # getencstat -v /dev/ses0
/dev/ses0: Enclosure Status <OK>
Element 0x0: Array Device Slot, status: Unsupported (0x00 0x00 0x00 0x00)
Element 0x1: Array Device Slot, status: OK (0x01 0x00 0x00 0x00)
Element 0x2: Array Device Slot, status: OK (0x01 0x00 0x02 0x00), dev: 'da1,pass1'
Element 0x3: Array Device Slot, status: OK (0x01 0x00 0x00 0x00), dev: 'da2,pass2'
Element 0x4: Array Device Slot, status: OK (0x01 0x00 0x00 0x00), dev: 'da3,pass3'
Element 0x5: Array Device Slot, status: OK (0x01 0x00 0x00 0x00), dev: 'da4,pass4'
Element 0x6: Array Device Slot, status: OK (0x01 0x00 0x00 0x00), dev: 'da5,pass5'
Element 0x7: Array Device Slot, status: OK (0x01 0x00 0x00 0x00), dev: 'da6,pass6'
Element 0x8: Array Device Slot, status: OK (0x01 0x00 0x00 0x00), dev: 'da7,pass7'
Element 0x9: Array Device Slot, status: OK (0x01 0x00 0x00 0x00), dev: 'da8,pass8'
Element 0xa: Array Device Slot, status: OK (0x01 0x00 0x00 0x00), dev: 'da9,pass9'
Element 0xb: Array Device Slot, status: OK (0x01 0x00 0x00 0x00), dev: 'da10,pass10'
Element 0xc: Array Device Slot, status: OK (0x01 0x00 0x00 0x00), dev: 'da11,pass11'
Element 0xd: Power Supply, status: Unsupported (0x00 0x00 0x00 0x00)
Element 0xe: Power Supply, status: OK, Swapped (0x11 0x00 0x00 0x20)
Element 0xf: Power Supply, status: OK, Swapped (0x11 0x00 0x00 0x20)
Element 0x10: Cooling, status: Unsupported (0x00 0x00 0x00 0x00)
Element 0x11: Cooling, status: OK (0x01 0x01 0x68 0x21)
Element 0x12: Cooling, status: OK (0x01 0x01 0x68 0x21)
Element 0x13: Cooling, status: Unsupported (0x00 0x00 0x00 0x00)
Element 0x14: Cooling, status: OK (0x01 0x01 0x68 0x21)
Element 0x15: Cooling, status: OK (0x01 0x01 0x68 0x21)
Element 0x16: Temperature Sensors, status: Unsupported (0x00 0x00 0x00 0x00)
Element 0x17: Temperature Sensors, status: OK (0x01 0x00 0x2f 0x00)
Element 0x18: Temperature Sensors, status: OK (0x01 0x00 0x2f 0x00)
Element 0x19: Temperature Sensors, status: OK (0x01 0x00 0x2c 0x00)
Element 0x1a: Temperature Sensors, status: OK (0x01 0x00 0x2c 0x00)
Element 0x1b: Enclosure Services Controller Electronics, status: Unsupported (0x00 0x00 0x00 0x00)
Element 0x1c: Enclosure Services Controller Electronics, status: OK (0x01 0x00 0x01 0x00)
Element 0x1d: Enclosure, status: Unsupported (0x00 0x00 0x00 0x00)
Element 0x1e: Enclosure, status: OK (0x01 0x00 0x00 0x00)
Element 0x1f: SAS Expander, status: Unsupported (0x00 0x00 0x00 0x00)
Element 0x20: SAS Expander, status: OK (0x01 0x00 0x00 0x00)
Element 0x21: SAS Connector, status: Unsupported (0x00 0x00 0x00 0x00)
Element 0x22: SAS Connector, status: OK (0x01 0x20 0x01 0x00)
Element 0x23: SAS Connector, status: OK (0x01 0x20 0x01 0x00)
Element 0x24: SAS Connector, status: OK (0x01 0x20 0x01 0x00)
Element 0x25: SAS Connector, status: OK (0x01 0x20 0x01 0x00)
Element 0x26: SAS Connector, status: OK (0x01 0x20 0x01 0x00)
Element 0x27: SAS Connector, status: OK (0x01 0x20 0x01 0x00)
Element 0x28: SAS Connector, status: OK (0x01 0x20 0x01 0x00)
Element 0x29: SAS Connector, status: OK (0x01 0x20 0x01 0x00)
Element 0x2a: SAS Connector, status: OK (0x01 0x20 0x01 0x00)
Element 0x2b: SAS Connector, status: OK (0x01 0x20 0x01 0x00)
Element 0x2c: SAS Connector, status: OK (0x01 0x20 0x01 0x00)
Element 0x2d: SAS Connector, status: OK (0x01 0x20 0x01 0x00)
Element 0x2e: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x2f: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x30: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x31: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x32: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x33: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x34: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x35: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x36: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x37: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x38: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x39: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x3a: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x3b: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x3c: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x3d: SAS Connector, status: Unsupported, Disabled (0x20 0x00 0x00 0x00)
Element 0x3e: SAS Connector, status: OK (0x01 0x02 0x00 0x00)
Element 0x3f: SAS Connector, status: OK (0x01 0x02 0x01 0x00)
Element 0x40: SAS Connector, status: OK (0x01 0x02 0x02 0x00)
Element 0x41: SAS Connector, status: OK (0x01 0x02 0x03 0x00)
Element 0x42: SAS Connector, status: OK (0x01 0x02 0x00 0x00)
Element 0x43: SAS Connector, status: OK (0x01 0x02 0x01 0x00)
Element 0x44: SAS Connector, status: OK (0x01 0x02 0x02 0x00)
Element 0x45: SAS Connector, status: OK (0x01 0x02 0x03 0x00)
Element 0x46: <Type 0x80>, status: Unsupported (0x00 0x00 0x00 0x00)
Element 0x47: <Type 0x80>, status: OK (0x01 0x20 0xff 0xff)

library ~ # cat /var/run/dmesg.boot
---<<BOOT>>---
Copyright (c) 1992-2019 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.1-RELEASE-p2 GENERIC amd64
FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1)
VT(vga): resolution 640x480
CPU: Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz (2494.05-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206d7  Family=0x6  Model=0x2d  Stepping=7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x1fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 68719476736 (65536 MB)
avail memory = 66833526784 (63737 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <HP     ProLiant>
FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs
FreeBSD/SMP: 2 package(s) x 6 core(s)
random: unblocking device.
Firmware Warning (ACPI): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20181213/tbfadt-850)
Firmware Warning (ACPI): Invalid length for FADT/Pm2ControlBlock: 32, using default 8 (20181213/tbfadt-850)
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
Launching APs: 6 2 8 10 3 11 5 7 4 1 9
Timecounter "TSC-low" frequency 1247023495 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
000.000023 [4335] netmap_init               netmap: loaded module
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0xffffffff8112e0f0, 0) error 19
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <HP ProLiant> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> numa-domain 0 on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Event timer "HPET3" frequency 14318180 Hz quality 340
Event timer "HPET4" frequency 14318180 Hz quality 340
Event timer "HPET5" frequency 14318180 Hz quality 340
Event timer "HPET6" frequency 14318180 Hz quality 340
Event timer "HPET7" frequency 14318180 Hz quality 340
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0
pcib0: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pci0: <ACPI PCI bus> numa-domain 0 on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 numa-domain 0 on pci0
pci1: <ACPI PCI bus> numa-domain 0 on pcib1
mps0: <Avago Technologies (LSI) SAS2008> port 0x6000-0x60ff mem 0xf7ef0000-0xf7ef3fff,0xf7e80000-0xf7ebffff irq 26 at device 0.0 numa-domain 0 on pci1
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib2: <ACPI PCI-PCI bridge> at device 1.1 numa-domain 0 on pci0
pci2: <ACPI PCI bus> numa-domain 0 on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 2.0 numa-domain 0 on pci0
pci3: <ACPI PCI bus> numa-domain 0 on pcib3
bge0: <HP Ethernet 1Gb 4-port 331FLR Adapter, ASIC rev. 0x5719001> mem 0xf6bf0000-0xf6bfffff,0xf6be0000-0xf6beffff,0xf6bd0000-0xf6bdffff irq 32 at device 0.0 numa-domain 0 on pci3
bge0: APE FW version: NCSI v1.4.12.0
bge0: CHIP ID 0x05719001; ASIC REV 0x5719; CHIP REV 0x57190; PCI-E
miibus0: <MII bus> numa-domain 0 on bge0
brgphy0: <BCM5719C 1000BASE-T media interface> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: Using defaults for TSO: 65518/35/2048
bge0: Ethernet address: 2c:44:fd:7b:59:5c
bge1: <HP Ethernet 1Gb 4-port 331FLR Adapter, ASIC rev. 0x5719001> mem 0xf6bc0000-0xf6bcffff,0xf6bb0000-0xf6bbffff,0xf6ba0000-0xf6baffff irq 36 at device 0.1 numa-domain 0 on pci3
bge1: APE FW version: NCSI v1.4.12.0
bge1: CHIP ID 0x05719001; ASIC REV 0x5719; CHIP REV 0x57190; PCI-E
miibus1: <MII bus> numa-domain 0 on bge1
brgphy1: <BCM5719C 1000BASE-T media interface> PHY 2 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge1: Using defaults for TSO: 65518/35/2048
bge1: Ethernet address: 2c:44:fd:7b:59:5d
bge2: <HP Ethernet 1Gb 4-port 331FLR Adapter, ASIC rev. 0x5719001> mem 0xf6b90000-0xf6b9ffff,0xf6b80000-0xf6b8ffff,0xf6b70000-0xf6b7ffff irq 32 at device 0.2 numa-domain 0 on pci3
bge2: APE FW version: NCSI v1.4.12.0
bge2: CHIP ID 0x05719001; ASIC REV 0x5719; CHIP REV 0x57190; PCI-E
miibus2: <MII bus> numa-domain 0 on bge2
brgphy2: <BCM5719C 1000BASE-T media interface> PHY 3 on miibus2
brgphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge2: Using defaults for TSO: 65518/35/2048
bge2: Ethernet address: 2c:44:fd:7b:59:5e
bge3: <HP Ethernet 1Gb 4-port 331FLR Adapter, ASIC rev. 0x5719001> mem 0xf6b60000-0xf6b6ffff,0xf6b50000-0xf6b5ffff,0xf6b40000-0xf6b4ffff irq 36 at device 0.3 numa-domain 0 on pci3
bge3: APE FW version: NCSI v1.4.12.0
bge3: CHIP ID 0x05719001; ASIC REV 0x5719; CHIP REV 0x57190; PCI-E
miibus3: <MII bus> numa-domain 0 on bge3
brgphy3: <BCM5719C 1000BASE-T media interface> PHY 4 on miibus3
brgphy3:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge3: Using defaults for TSO: 65518/35/2048
bge3: Ethernet address: 2c:44:fd:7b:59:5f
pcib4: <ACPI PCI-PCI bridge> at device 2.1 numa-domain 0 on pci0
pci4: <ACPI PCI bus> numa-domain 0 on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 2.2 numa-domain 0 on pci0
pci5: <ACPI PCI bus> numa-domain 0 on pcib5
ciss0: <HP Smart Array P420i> port 0x5000-0x50ff mem 0xf7d00000-0xf7dfffff,0xf7cf0000-0xf7cf03ff irq 34 at device 0.0 numa-domain 0 on pci5
ciss0: PERFORMANT Transport
pcib6: <ACPI PCI-PCI bridge> at device 2.3 numa-domain 0 on pci0
pci6: <ACPI PCI bus> numa-domain 0 on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 3.0 numa-domain 0 on pci0
pci7: <ACPI PCI bus> numa-domain 0 on pcib7
mps1: <Avago Technologies (LSI) SAS2008> port 0x7000-0x70ff mem 0xf7ff0000-0xf7ff3fff,0xf7f80000-0xf7fbffff irq 40 at device 0.0 numa-domain 0 on pci7
mps1: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps1: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib8: <ACPI PCI-PCI bridge> at device 3.1 numa-domain 0 on pci0
pci8: <ACPI PCI bus> numa-domain 0 on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 3.2 numa-domain 0 on pci0
pci9: <ACPI PCI bus> numa-domain 0 on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 3.3 numa-domain 0 on pci0
pci10: <ACPI PCI bus> numa-domain 0 on pcib10
pcib11: <ACPI PCI-PCI bridge> at device 17.0 numa-domain 0 on pci0
pci11: <ACPI PCI bus> numa-domain 0 on pcib11
ehci0: <Intel Patsburg USB 2.0 controller> mem 0xf6c60000-0xf6c603ff irq 21 at device 26.0 numa-domain 0 on pci0
usbus0: EHCI version 1.0
usbus0 numa-domain 0 on ehci0
usbus0: 480Mbps High Speed USB v2.0
pcib12: <ACPI PCI-PCI bridge> at device 28.0 numa-domain 0 on pci0
pci12: <ACPI PCI bus> numa-domain 0 on pcib12
pcib13: <ACPI PCI-PCI bridge> at device 28.7 numa-domain 0 on pci0
pci13: <ACPI PCI bus> numa-domain 0 on pcib13
vgapci0: <VGA-compatible display> mem 0xf5000000-0xf5ffffff,0xf7be0000-0xf7be3fff,0xf7000000-0xf77fffff irq 16 at device 0.1 numa-domain 0 on pci13
vgapci0: Boot video device
uhci0: <HP iLO Standard Virtual USB controller> port 0x3c00-0x3c1f irq 16 at device 0.4 numa-domain 0 on pci13
usbus1 numa-domain 0 on uhci0
usbus1: 12Mbps Full Speed USB v1.0
ehci1: <Intel Patsburg USB 2.0 controller> mem 0xf6c50000-0xf6c503ff irq 20 at device 29.0 numa-domain 0 on pci0
usbus2: EHCI version 1.0
usbus2 numa-domain 0 on ehci1
usbus2: 480Mbps High Speed USB v2.0
pcib14: <PCI-PCI bridge> at device 30.0 numa-domain 0 on pci0
pci14: <PCI bus> numa-domain 0 on pcib14
isab0: <PCI-ISA bridge> at device 31.0 numa-domain 0 on pci0
isa0: <ISA bus> numa-domain 0 on isab0
atapci0: <Intel Patsburg SATA600 controller> port 0x4000-0x4007,0x4008-0x400b,0x4010-0x4017,0x4018-0x401b,0x4020-0x402f,0x4030-0x403f irq 17 at device 31.2 numa-domain 0 on pci0
ata2: <ATA channel> at channel 0 on atapci0
ata3: <ATA channel> at channel 1 on atapci0
pcib15: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
pci15: <ACPI PCI bus> numa-domain 1 on pcib15
pcib16: <PCI-PCI bridge> at device 0.0 numa-domain 1 on pci15
pci16: <PCI bus> numa-domain 1 on pcib16
pcib17: <ACPI PCI-PCI bridge> at device 1.0 numa-domain 1 on pci15
pci17: <ACPI PCI bus> numa-domain 1 on pcib17
pcib18: <ACPI PCI-PCI bridge> at device 1.1 numa-domain 1 on pci15
pci18: <ACPI PCI bus> numa-domain 1 on pcib18
pcib19: <ACPI PCI-PCI bridge> at device 2.0 numa-domain 1 on pci15
pci19: <ACPI PCI bus> numa-domain 1 on pcib19
pcib20: <ACPI PCI-PCI bridge> at device 2.1 numa-domain 1 on pci15
pci20: <ACPI PCI bus> numa-domain 1 on pcib20
pcib21: <ACPI PCI-PCI bridge> at device 2.2 numa-domain 1 on pci15
pci21: <ACPI PCI bus> numa-domain 1 on pcib21
pcib22: <ACPI PCI-PCI bridge> at device 2.3 numa-domain 1 on pci15
pci22: <ACPI PCI bus> numa-domain 1 on pcib22
pcib23: <ACPI PCI-PCI bridge> at device 3.0 numa-domain 1 on pci15
pci23: <ACPI PCI bus> numa-domain 1 on pcib23
pcib24: <ACPI PCI-PCI bridge> at device 3.1 numa-domain 1 on pci15
pci24: <ACPI PCI bus> numa-domain 1 on pcib24
pcib25: <ACPI PCI-PCI bridge> at device 3.2 numa-domain 1 on pci15
pci25: <ACPI PCI bus> numa-domain 1 on pcib25
pcib26: <ACPI PCI-PCI bridge> at device 3.3 numa-domain 1 on pci15
pci26: <ACPI PCI bus> numa-domain 1 on pcib26
acpi_tz0: <Thermal Zone> on acpi0
ipmi0: <IPMI System Interface> port 0xca2-0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
acpi_syscontainer0: <System Container> port 0x2e-0x2f on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <Non-standard ns8250 class UART with FIFOs> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0
uart1: <Non-standard ns8250 class UART with FIFOs> at port 0x2f8 irq 3 on isa0
uart1: non-PNP ISA device will be removed from GENERIC in FreeBSD 12.
est0: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 238c00001c00
device_attach: est0 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 238c00001c00
device_attach: est1 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 238c00001c00
device_attach: est2 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 238c00001c00
device_attach: est3 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 238c00001c00
device_attach: est4 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 238c00001c00
device_attach: est5 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 238c00001c00
device_attach: est6 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 238c00001c00
device_attach: est7 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 238c00001c00
device_attach: est8 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 238c00001c00
device_attach: est9 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 238c00001c00
device_attach: est10 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 238c00001c00
device_attach: est11 attach returned 6
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ugen2.1: <Intel EHCI root HUB> at usbus2
ugen1.1: <HP UHCI root HUB> at usbus1
ugen0.1: <Intel EHCI root HUB> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
uhub2: <HP UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ipmi0: IPMI device rev. 2, firmware rev. 2.50, version 2.0, device support mask 0x8f
ipmi0: Number of channels 1
ipmi0: Attached watchdog
ipmi0: Establishing power cycle handler
uhub2: 2 ports with 2 removable, self powered
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
ugen2.2: <vendor 0x8087 product 0x0024> at usbus2
uhub3 numa-domain 0 on uhub0
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus2
ugen0.2: <vendor 0x8087 product 0x0024> at usbus0
uhub4 numa-domain 0 on uhub1
uhub4: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
uhub4: 6 ports with 6 removable, self powered
uhub3: 8 ports with 8 removable, self powered
ugen2.3: <vendor 0x0424 product 0x2660> at usbus2
uhub5 numa-domain 0 on uhub3
uhub5: <vendor 0x0424 product 0x2660, class 9/0, rev 2.00/8.01, addr 3> on usbus2
uhub5: 2 ports with 1 removable, self powered
ses0 at mps1 bus 0 scbus3 target 43 lun 0
ses0: <HP MSA60 2.18> Fixed Enclosure Services SPC-3 SCSI device
ses0: Serial Number SGA817005M
ses0: 300.000MB/s transfers
ses0: SES Device
da1 at mps1 bus 0 scbus3 target 31 lun 0
da1: <ATA WDC WD2000FYYZ-0 1K02> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number WD-WMC1P0054868
da1: 150.000MB/s transfers
da1: Command Queueing enabled
da1: 1907729MB (3907029168 512 byte sectors)
da5 at mps1 bus 0 scbus3 target 35 lun 0
da5: <ATA WDC WD2000FYYZ-0 1K02> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number WD-WMC1P0212052
da5: 150.000MB/s transfers
da5: Command Queueing enabled
da5: 1907729MB (3907029168 512 byte sectors)
da4 at mps1 bus 0 scbus3 target 34 lun 0
da4: <ATA WDC WD2000FYYZ-0 1K02> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number WD-WMC1P0087107
da4: 150.000MB/s transfers
da4: Command Queueing enabled
da4: 1907729MB (3907029168 512 byte sectors)
da6 at mps1 bus 0 scbus3 target 36 lun 0
da6: <ATA WDC WD2000FYYZ-0 1K02> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number WD-WMC1P0213822
da6: 150.000MB/s transfers
da6: Command Queueing enabled
da6: 1907729MB (3907029168 512 byte sectors)
da8 at mps1 bus 0 scbus3 target 38 lun 0
da8: <ATA WDC WD2000FYYZ-0 1K02> Fixed Direct Access SPC-4 SCSI device
da8: Serial Number WD-WMC1P0053015
da8: 150.000MB/s transfers
da8: Command Queueing enabled
da8: 1907729MB (3907029168 512 byte sectors)
da9 at mps1 bus 0 scbus3 target 39 lun 0
da9: <ATA WDC WD2000FYYZ-0 1K02> Fixed Direct Access SPC-4 SCSI device
da9: Serial Number WD-WMC1P0053035
da9: 150.000MB/s transfers
da9: Command Queueing enabled
da9: 1907729MB (3907029168 512 byte sectors)
da10 at mps1 bus 0 scbus3 target 40 lun 0
da10: <ATA WDC WD2000FYYZ-0 1K02> Fixed Direct Access SPC-4 SCSI device
da10: Serial Number WD-WMC1P0116027
da10: 150.000MB/s transfers
da10: Command Queueing enabled
da10: 1907729MB (3907029168 512 byte sectors)
da3 at mps1 bus 0 scbus3 target 33 lun 0
da3: <ATA WDC WD2000FYYZ-0 1K02> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number WD-WMC1P0087547
da3: 150.000MB/s transfers
da3: Command Queueing enabled
da3: 1907729MB (3907029168 512 byte sectors)
da11 at mps1 bus 0 scbus3 target 41 lun 0
da11: <ATA WDC WD2000FYYZ-0 1K02> Fixed Direct Access SPC-4 SCSI device
da11: Serial Number WD-WMC1P0115889
da11: 150.000MB/s transfers
da11: Command Queueing enabled
da11: 1907729MB (3907029168 512 byte sectors)
cd0 at ata2 bus 0 scbus4 target 0 lun 0
cd0: <hp DVD ROM UJ8C2 DC02> Removable CD-ROM SCSI device
cd0: Serial Number HP34 021570
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
da12 at mps1 bus 0 scbus3 target 42 lun 0
da12: <ATA WDC WD2000FYYZ-0 1K02> Fixed Direct Access SPC-4 SCSI device
da12: Serial Number WD-WMC1P0115969
da12: 150.000MB/s transfers
da12: Command Queueing enabled
da12: 1907729MB (3907029168 512 byte sectors)
da2 at mps1 bus 0 scbus3 target 32 lun 0
da2: <ATA WDC WD2000FYYZ-0 1K02> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number WD-WMC1P0212306
da2: 150.000MB/s transfers
da2: Command Queueing enabled
da2: 1907729MB (3907029168 512 byte sectors)
da7 at mps1 bus 0 scbus3 target 37 lun 0
da7: <ATA WDC WD2000FYYZ-0 1K02> Fixed Direct Access SPC-4 SCSI device
da7: Serial Number WD-WMC1P0215894
da7: 150.000MB/s transfers
da7: Command Queueing enabled
da7: 1907729MB (3907029168 512 byte sectors)
da0 at ciss0 bus 0 scbus1 target 0 lun 0
da0: <HP RAID 1(1+0) OK> Fixed Direct Access SPC-3 SCSI device
da0: Serial Number 001438027E7FB80
da0: 135.168MB/s transfers
da0: Command Queueing enabled
da0: 858210MB (1757614684 512 byte sectors)
ses0: da1,pass1 in Array Device Slot 2, SAS Slot: 2+ phys at slot 1
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5001438001571280 addr 5001438001571281
ses0:  phy 1: SAS device type 0 phy 0
ses0:  phy 1: parent 0 addr 0
ses0: da2,pass2 in Array Device Slot 3, SAS Slot: 2+ phys at slot 2
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5001438001571280 addr 5001438001571282
ses0:  phy 1: SAS device type 0 phy 0
ses0:  phy 1: parent 0 addr 0
ses0: da3,pass3 in Array Device Slot 4, SAS Slot: 2+ phys at slot 3
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5001438001571280 addr 5001438001571283
ses0:  phy 1: SAS device type 0 phy 0
ses0:  phy 1: parent 0 addr 0
ses0: da4,pass4 in Array Device Slot 5, SAS Slot: 2+ phys at slot 4
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5001438001571280 addr 5001438001571284
ses0:  phy 1: SAS device type 0 phy 0
ses0:  phy 1: parent 0 addr 0
ses0: da5,pass5 in Array Device Slot 6, SAS Slot: 2+ phys at slot 5
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5001438001571280 addr 5001438001571285
ses0:  phy 1: SAS device type 0 phy 0
ses0:  phy 1: parent 0 addr 0
ses0: da6,pass6 in Array Device Slot 7, SAS Slot: 2+ phys at slot 6
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5001438001571280 addr 5001438001571286
ses0:  phy 1: SAS device type 0 phy 0
ses0:  phy 1: parent 0 addr 0
ses0: da7,pass7 in Array Device Slot 8, SAS Slot: 2+ phys at slot 7
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5001438001571280 addr 5001438001571287
ses0:  phy 1: SAS device type 0 phy 0
ses0:  phy 1: parent 0 addr 0
ses0: da8,pass8 in Array Device Slot 9, SAS Slot: 2+ phys at slot 8
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5001438001571280 addr 5001438001571288
ses0:  phy 1: SAS device type 0 phy 0
ses0:  phy 1: parent 0 addr 0
ses0: da9,pass9 in Array Device Slot 10, SAS Slot: 2+ phys at slot 9
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5001438001571280 addr 5001438001571289
ses0:  phy 1: SAS device type 0 phy 0
ses0:  phy 1: parent 0 addr 0
ses0: da10,pass10 in Array Device Slot 11, SAS Slot: 2+ phys at slot 10
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5001438001571280 addr 500143800157128a
ses0:  phy 1: SAS device type 0 phy 0
ses0:  phy 1: parent 0 addr 0
ses0: da11,pass11 in Array Device Slot 12, SAS Slot: 2+ phys at slot 11
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5001438001571280 addr 500143800157128b
ses0:  phy 1: SAS device type 0 phy 0
ses0:  phy 1: parent 0 addr 0
Trying to mount root from zfs:zroot/ROOT/default []...
lo0: link state changed to UP
bge0: link state changed to DOWN
bge0: link state changed to UP
ioat0: <SNB IOAT Ch0> mem 0xf6cf0000-0xf6cf3fff irq 31 at device 4.0 numa-domain 0 on pci0
ioat0: Capabilities: 2f7<PQ,Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat1: <SNB IOAT Ch1> mem 0xf6ce0000-0xf6ce3fff irq 39 at device 4.1 numa-domain 0 on pci0
ioat1: Capabilities: 2f7<PQ,Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat2: <SNB IOAT Ch2> mem 0xf6cd0000-0xf6cd3fff irq 31 at device 4.2 numa-domain 0 on pci0
ioat2: Capabilities: f7<Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat3: <SNB IOAT Ch3> mem 0xf6cc0000-0xf6cc3fff irq 39 at device 4.3 numa-domain 0 on pci0
ioat3: Capabilities: f7<Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat4: <SNB IOAT Ch4> mem 0xf6cb0000-0xf6cb3fff irq 31 at device 4.4 numa-domain 0 on pci0
ioat4: Capabilities: f7<Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat5: <SNB IOAT Ch5> mem 0xf6ca0000-0xf6ca3fff irq 39 at device 4.5 numa-domain 0 on pci0
ioat5: Capabilities: f7<Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat6: <SNB IOAT Ch6> mem 0xf6c90000-0xf6c93fff irq 31 at device 4.6 numa-domain 0 on pci0
ioat6: Capabilities: f7<Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat7: <SNB IOAT Ch7> mem 0xf6c80000-0xf6c83fff irq 39 at device 4.7 numa-domain 0 on pci0
ioat7: Capabilities: f7<Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat8: <SNB IOAT Ch0> mem 0xfbff0000-0xfbff3fff irq 55 at device 4.0 numa-domain 1 on pci15
ioat8: Capabilities: 2f7<PQ,Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat9: <SNB IOAT Ch1> mem 0xfbfe0000-0xfbfe3fff irq 63 at device 4.1 numa-domain 1 on pci15
ioat9: Capabilities: 2f7<PQ,Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat10: <SNB IOAT Ch2> mem 0xfbfd0000-0xfbfd3fff irq 55 at device 4.2 numa-domain 1 on pci15
ioat10: Capabilities: f7<Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat11: <SNB IOAT Ch3> mem 0xfbfc0000-0xfbfc3fff irq 63 at device 4.3 numa-domain 1 on pci15
ioat11: Capabilities: f7<Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat12: <SNB IOAT Ch4> mem 0xfbfb0000-0xfbfb3fff irq 55 at device 4.4 numa-domain 1 on pci15
ioat12: Capabilities: f7<Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat13: <SNB IOAT Ch5> mem 0xfbfa0000-0xfbfa3fff irq 63 at device 4.5 numa-domain 1 on pci15
ioat13: Capabilities: f7<Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat14: <SNB IOAT Ch6> mem 0xfbf90000-0xfbf93fff irq 55 at device 4.6 numa-domain 1 on pci15
ioat14: Capabilities: f7<Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
ioat15: <SNB IOAT Ch7> mem 0xfbf80000-0xfbf83fff irq 63 at device 4.7 numa-domain 1 on pci15
ioat15: Capabilities: f7<Extended_APIC_ID,Block_Fill,Move_CRC,DCA,Marker_Skipping,CRC,Page_Break>
Comment 1 Alexander Motin freebsd_committer 2020-08-13 14:03:05 UTC
My guess is that ses driver can not properly parse the Additional Element Status diagnostic page on this enclosure. Could you please show also output of `sg_ses -p 1 /dev/ses0`, `sg_ses -p 2 /dev/ses0`, `sg_ses -p 7 /dev/ses0` and `sg_ses -p 0xa /dev/ses0`.  `sg_ses` tool is from sysutils/sg3_utils port.
Comment 2 Mike Biesele 2020-08-13 19:06:15 UTC
library ~ # sg_ses -p 1 /dev/ses0
  HP        MSA60             2.18
Configuration diagnostic page:
  number of secondary subenclosures: 0
  generation code: 0x0
  enclosure descriptor list
    Subenclosure identifier: 0 [primary]
      relative ES process id: 1, number of ES processes: 1
      number of type descriptor headers: 10
      enclosure logical identifier (hex): 5001438001571280
      enclosure vendor: HP        product: MSA60             rev: 2.18
      vendor-specific data:
        48 50 20 4d 53 41 20 53  74 6f 72 61 67 65 20 45    HP MSA Storage E
        6e 63 6c 6f 73 75 72 65  20 20 20 20 20 20 20 20    nclosure
        20 00 00 00                                          ...

  type descriptor header and text list
    Element type: Array device slot, subenclosure id: 0
      number of possible elements: 12
      text: Bay
    Element type: Power supply, subenclosure id: 0
      number of possible elements: 2
      text: Power Supply
    Element type: Cooling, subenclosure id: 0
      number of possible elements: 2
      text: Fans
    Element type: Cooling, subenclosure id: 0
      number of possible elements: 2
      text: Fans
    Element type: Temperature sensor, subenclosure id: 0
      number of possible elements: 4
      text: Temperature Sensor
    Element type: Enclosure services controller electronics, subenclosure id: 0
      number of possible elements: 1
      text: Enclosure Service Controller
    Element type: Enclosure, subenclosure id: 0
      number of possible elements: 1
      text: Enclosure
    Element type: SAS expander, subenclosure id: 0
      number of possible elements: 1
      text: SAS Expander
    Element type: SAS connector, subenclosure id: 0
      number of possible elements: 36
      text: SAS Expander Phy
    Element type: vendor specific [0x80], subenclosure id: 0
      number of possible elements: 1
      text: HP 7-Segment Display

library ~ # sg_ses -p 2 /dev/ses0
  HP        MSA60             2.18
Enclosure Status diagnostic page:
  INVOP=0, INFO=0, NON-CRIT=0, CRIT=0, UNRECOV=0
  generation code: 0x0
  status descriptor list
    Element type: Array device slot, subenclosure id: 0 [ti=0]
      Overall descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: Unsupported
        OK=0, Reserved device=0, Hot spare=0, Cons check=0
        In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
        App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
        Ready to insert=0, RMV=0, Ident=0, Report=0
        App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
        Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
      Element 0 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        OK=0, Reserved device=0, Hot spare=0, Cons check=0
        In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
        App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
        Ready to insert=0, RMV=0, Ident=0, Report=0
        App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
        Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
      Element 1 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        OK=0, Reserved device=0, Hot spare=0, Cons check=0
        In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
        App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
        Ready to insert=0, RMV=0, Ident=1, Report=0
        App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
        Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
      Element 2 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        OK=0, Reserved device=0, Hot spare=0, Cons check=0
        In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
        App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
        Ready to insert=0, RMV=0, Ident=0, Report=0
        App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
        Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
      Element 3 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        OK=0, Reserved device=0, Hot spare=0, Cons check=0
        In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
        App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
        Ready to insert=0, RMV=0, Ident=0, Report=0
        App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
        Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
      Element 4 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        OK=0, Reserved device=0, Hot spare=0, Cons check=0
        In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
        App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
        Ready to insert=0, RMV=0, Ident=0, Report=0
        App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
        Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
      Element 5 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        OK=0, Reserved device=0, Hot spare=0, Cons check=0
        In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
        App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
        Ready to insert=0, RMV=0, Ident=0, Report=0
        App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
        Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
      Element 6 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        OK=0, Reserved device=0, Hot spare=0, Cons check=0
        In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
        App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
        Ready to insert=0, RMV=0, Ident=0, Report=0
        App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
        Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
      Element 7 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        OK=0, Reserved device=0, Hot spare=0, Cons check=0
        In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
        App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
        Ready to insert=0, RMV=0, Ident=0, Report=0
        App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
        Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
      Element 8 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        OK=0, Reserved device=0, Hot spare=0, Cons check=0
        In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
        App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
        Ready to insert=0, RMV=0, Ident=0, Report=0
        App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
        Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
      Element 9 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        OK=0, Reserved device=0, Hot spare=0, Cons check=0
        In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
        App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
        Ready to insert=0, RMV=0, Ident=0, Report=0
        App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
        Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
      Element 10 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        OK=0, Reserved device=0, Hot spare=0, Cons check=0
        In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
        App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
        Ready to insert=0, RMV=0, Ident=0, Report=0
        App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
        Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
      Element 11 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        OK=0, Reserved device=0, Hot spare=0, Cons check=0
        In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
        App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
        Ready to insert=0, RMV=0, Ident=0, Report=0
        App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
        Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
    Element type: Power supply, subenclosure id: 0 [ti=1]
      Overall descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: Unsupported
        Ident=0, Do not remove=0, DC overvoltage=0, DC undervoltage=0
        DC overcurrent=0, Hot swap=0, Fail=0, Requested on=0, Off=0
        Overtmp fail=0, Temperature warn=0, AC fail=0, DC fail=0
      Element 0 descriptor:
        Predicted failure=0, Disabled=0, Swap=1, status: OK
        Ident=0, Do not remove=0, DC overvoltage=0, DC undervoltage=0
        DC overcurrent=0, Hot swap=0, Fail=0, Requested on=1, Off=0
        Overtmp fail=0, Temperature warn=0, AC fail=0, DC fail=0
      Element 1 descriptor:
        Predicted failure=0, Disabled=0, Swap=1, status: OK
        Ident=0, Do not remove=0, DC overvoltage=0, DC undervoltage=0
        DC overcurrent=0, Hot swap=0, Fail=0, Requested on=1, Off=0
        Overtmp fail=0, Temperature warn=0, AC fail=0, DC fail=0
    Element type: Cooling, subenclosure id: 0 [ti=2]
      Overall descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: Unsupported
        Ident=0, Do not remove=0, Hot swap=0, Fail=0, Requested on=0
        Off=0, Actual speed=0 rpm, Fan stopped
      Element 0 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Do not remove=0, Hot swap=0, Fail=0, Requested on=1
        Off=0, Actual speed=4200 rpm, Fan at lowest speed
      Element 1 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Do not remove=0, Hot swap=0, Fail=0, Requested on=1
        Off=0, Actual speed=3600 rpm, Fan at lowest speed
    Element type: Cooling, subenclosure id: 0 [ti=3]
      Overall descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: Unsupported
        Ident=0, Do not remove=0, Hot swap=0, Fail=0, Requested on=0
        Off=0, Actual speed=0 rpm, Fan stopped
      Element 0 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Do not remove=0, Hot swap=0, Fail=0, Requested on=1
        Off=0, Actual speed=3600 rpm, Fan at lowest speed
      Element 1 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Do not remove=0, Hot swap=0, Fail=0, Requested on=1
        Off=0, Actual speed=3600 rpm, Fan at lowest speed
    Element type: Temperature sensor, subenclosure id: 0 [ti=4]
      Overall descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: Unsupported
        Ident=0, Fail=0, OT failure=0, OT warning=0, UT failure=0
        UT warning=0
        Temperature: <reserved>
      Element 0 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Fail=0, OT failure=0, OT warning=0, UT failure=0
        UT warning=0
        Temperature=27 C
      Element 1 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Fail=0, OT failure=0, OT warning=0, UT failure=0
        UT warning=0
        Temperature=27 C
      Element 2 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Fail=0, OT failure=0, OT warning=0, UT failure=0
        UT warning=0
        Temperature=24 C
      Element 3 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Fail=0, OT failure=0, OT warning=0, UT failure=0
        UT warning=0
        Temperature=24 C
    Element type: Enclosure services controller electronics, subenclosure id: 0 [ti=5]
      Overall descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: Unsupported
        Ident=0, Fail=0, Do not remove=0, Report=0, Hot swap=0
      Element 0 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Fail=0, Do not remove=0, Report=1, Hot swap=0
    Element type: Enclosure, subenclosure id: 0 [ti=6]
      Overall descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: Unsupported
        Ident=0, Time until power cycle=0, Failure indication=0
        Warning indication=0, Requested power off duration=0
        Failure requested=0, Warning requested=0
      Element 0 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Time until power cycle=0, Failure indication=0
        Warning indication=0, Requested power off duration=0
        Failure requested=0, Warning requested=0
    Element type: SAS expander, subenclosure id: 0 [ti=7]
      Overall descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: Unsupported
        Ident=0, Fail=0
      Element 0 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Fail=0
    Element type: SAS connector, subenclosure id: 0 [ti=8]
      Overall descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 0 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
        Connector physical link=0x1, Mated=0, Fail=0, OC=0
      Element 1 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
        Connector physical link=0x1, Mated=0, Fail=0, OC=0
      Element 2 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
        Connector physical link=0x1, Mated=0, Fail=0, OC=0
      Element 3 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
        Connector physical link=0x1, Mated=0, Fail=0, OC=0
      Element 4 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
        Connector physical link=0x1, Mated=0, Fail=0, OC=0
      Element 5 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
        Connector physical link=0x1, Mated=0, Fail=0, OC=0
      Element 6 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
        Connector physical link=0x1, Mated=0, Fail=0, OC=0
      Element 7 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
        Connector physical link=0x1, Mated=0, Fail=0, OC=0
      Element 8 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
        Connector physical link=0x1, Mated=0, Fail=0, OC=0
      Element 9 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
        Connector physical link=0x1, Mated=0, Fail=0, OC=0
      Element 10 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
        Connector physical link=0x1, Mated=0, Fail=0, OC=0
      Element 11 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
        Connector physical link=0x1, Mated=0, Fail=0, OC=0
      Element 12 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 13 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 14 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 15 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 16 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 17 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 18 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 19 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 20 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 21 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 22 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 23 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 24 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 25 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 26 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 27 descriptor:
        Predicted failure=0, Disabled=1, Swap=0, status: Unsupported
        Ident=0, No information
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 28 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Mini SAS 4x receptacle (SFF-8088) [max 4 phys]
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 29 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Mini SAS 4x receptacle (SFF-8088) [max 4 phys]
        Connector physical link=0x1, Mated=0, Fail=0, OC=0
      Element 30 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Mini SAS 4x receptacle (SFF-8088) [max 4 phys]
        Connector physical link=0x2, Mated=0, Fail=0, OC=0
      Element 31 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Mini SAS 4x receptacle (SFF-8088) [max 4 phys]
        Connector physical link=0x3, Mated=0, Fail=0, OC=0
      Element 32 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Mini SAS 4x receptacle (SFF-8088) [max 4 phys]
        Connector physical link=0x0, Mated=0, Fail=0, OC=0
      Element 33 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Mini SAS 4x receptacle (SFF-8088) [max 4 phys]
        Connector physical link=0x1, Mated=0, Fail=0, OC=0
      Element 34 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Mini SAS 4x receptacle (SFF-8088) [max 4 phys]
        Connector physical link=0x2, Mated=0, Fail=0, OC=0
      Element 35 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Ident=0, Mini SAS 4x receptacle (SFF-8088) [max 4 phys]
        Connector physical link=0x3, Mated=0, Fail=0, OC=0
    Element type: vendor specific [0x80], subenclosure id: 0 [ti=9]
      Overall descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: Unsupported
        Vendor specific element type, status in hex: 00 00 00 00
      Element 0 descriptor:
        Predicted failure=0, Disabled=0, Swap=0, status: OK
        Vendor specific element type, status in hex: 01 20 ff ff

library ~ # sg_ses -p 7 /dev/ses0
  HP        MSA60             2.18
sg_ses failed: Illegal request

library ~ # sg_ses -p 0xa /dev/ses0
  HP        MSA60             2.18
  Primary enclosure logical identifier (hex): 5001438001571280
Additional element status diagnostic page:
  generation code: 0x0
  additional element status descriptor list
    Element type: Array device slot, subenclosure id: 0 [ti=0]
      Element index: 1  eiioe=0
        Transport protocol: SAS
        number of phys: 2, not all phys: 1, device slot number: 1
        phy index: 0
          SAS device type: no SAS device attached
          initiator port for:
          target port for: SATA_device
          attached SAS address: 0x5001438001571280
          SAS address: 0x5001438001571281
          phy identifier: 0x0
        phy index: 1
          SAS device type: no SAS device attached
          initiator port for:
          target port for:
          attached SAS address: 0x0
          SAS address: 0x0
          phy identifier: 0x0
      Element index: 2  eiioe=0
        Transport protocol: SAS
        number of phys: 2, not all phys: 1, device slot number: 2
        phy index: 0
          SAS device type: end device
          initiator port for:
          target port for: SSP
          attached SAS address: 0x5001438001571280
          SAS address: 0x5000c50034cd2731
          phy identifier: 0x1
        phy index: 1
          SAS device type: no SAS device attached
          initiator port for:
          target port for:
          attached SAS address: 0x0
          SAS address: 0x0
          phy identifier: 0x0
      Element index: 3  eiioe=0
        Transport protocol: SAS
        number of phys: 2, not all phys: 1, device slot number: 3
        phy index: 0
          SAS device type: no SAS device attached
          initiator port for:
          target port for: SATA_device
          attached SAS address: 0x5001438001571280
          SAS address: 0x5001438001571283
          phy identifier: 0x0
        phy index: 1
          SAS device type: no SAS device attached
          initiator port for:
          target port for:
          attached SAS address: 0x0
          SAS address: 0x0
          phy identifier: 0x0
      Element index: 4  eiioe=0
        Transport protocol: SAS
        number of phys: 2, not all phys: 1, device slot number: 4
        phy index: 0
          SAS device type: no SAS device attached
          initiator port for:
          target port for: SATA_device
          attached SAS address: 0x5001438001571280
          SAS address: 0x5001438001571284
          phy identifier: 0x0
        phy index: 1
          SAS device type: no SAS device attached
          initiator port for:
          target port for:
          attached SAS address: 0x0
          SAS address: 0x0
          phy identifier: 0x0
      Element index: 5  eiioe=0
        Transport protocol: SAS
        number of phys: 2, not all phys: 1, device slot number: 5
        phy index: 0
          SAS device type: no SAS device attached
          initiator port for:
          target port for: SATA_device
          attached SAS address: 0x5001438001571280
          SAS address: 0x5001438001571285
          phy identifier: 0x0
        phy index: 1
          SAS device type: no SAS device attached
          initiator port for:
          target port for:
          attached SAS address: 0x0
          SAS address: 0x0
          phy identifier: 0x0
      Element index: 6  eiioe=0
        Transport protocol: SAS
        number of phys: 2, not all phys: 1, device slot number: 6
        phy index: 0
          SAS device type: no SAS device attached
          initiator port for:
          target port for: SATA_device
          attached SAS address: 0x5001438001571280
          SAS address: 0x5001438001571286
          phy identifier: 0x0
        phy index: 1
          SAS device type: no SAS device attached
          initiator port for:
          target port for:
          attached SAS address: 0x0
          SAS address: 0x0
          phy identifier: 0x0
      Element index: 7  eiioe=0
        Transport protocol: SAS
        number of phys: 2, not all phys: 1, device slot number: 7
        phy index: 0
          SAS device type: no SAS device attached
          initiator port for:
          target port for: SATA_device
          attached SAS address: 0x5001438001571280
          SAS address: 0x5001438001571287
          phy identifier: 0x0
        phy index: 1
          SAS device type: no SAS device attached
          initiator port for:
          target port for:
          attached SAS address: 0x0
          SAS address: 0x0
          phy identifier: 0x0
      Element index: 8  eiioe=0
        Transport protocol: SAS
        number of phys: 2, not all phys: 1, device slot number: 8
        phy index: 0
          SAS device type: no SAS device attached
          initiator port for:
          target port for: SATA_device
          attached SAS address: 0x5001438001571280
          SAS address: 0x5001438001571288
          phy identifier: 0x0
        phy index: 1
          SAS device type: no SAS device attached
          initiator port for:
          target port for:
          attached SAS address: 0x0
          SAS address: 0x0
          phy identifier: 0x0
      Element index: 9  eiioe=0
        Transport protocol: SAS
        number of phys: 2, not all phys: 1, device slot number: 9
        phy index: 0
          SAS device type: no SAS device attached
          initiator port for:
          target port for: SATA_device
          attached SAS address: 0x5001438001571280
          SAS address: 0x5001438001571289
          phy identifier: 0x0
        phy index: 1
          SAS device type: no SAS device attached
          initiator port for:
          target port for:
          attached SAS address: 0x0
          SAS address: 0x0
          phy identifier: 0x0
      Element index: 10  eiioe=0
        Transport protocol: SAS
        number of phys: 2, not all phys: 1, device slot number: 10
        phy index: 0
          SAS device type: no SAS device attached
          initiator port for:
          target port for: SATA_device
          attached SAS address: 0x5001438001571280
          SAS address: 0x500143800157128a
          phy identifier: 0x0
        phy index: 1
          SAS device type: no SAS device attached
          initiator port for:
          target port for:
          attached SAS address: 0x0
          SAS address: 0x0
          phy identifier: 0x0
      Element index: 11  eiioe=0
        Transport protocol: SAS
        number of phys: 2, not all phys: 1, device slot number: 11
        phy index: 0
          SAS device type: no SAS device attached
          initiator port for:
          target port for: SATA_device
          attached SAS address: 0x5001438001571280
          SAS address: 0x500143800157128b
          phy identifier: 0x0
        phy index: 1
          SAS device type: no SAS device attached
          initiator port for:
          target port for:
          attached SAS address: 0x0
          SAS address: 0x0
          phy identifier: 0x0
      Element index: 12  eiioe=0
        Transport protocol: SAS
        number of phys: 2, not all phys: 1, device slot number: 12
        phy index: 0
          SAS device type: no SAS device attached
          initiator port for:
          target port for: SATA_device
          attached SAS address: 0x5001438001571280
          SAS address: 0x500143800157128c
          phy identifier: 0x0
        phy index: 1
          SAS device type: no SAS device attached
          initiator port for:
          target port for:
          attached SAS address: 0x0
          SAS address: 0x0
          phy identifier: 0x0
Comment 3 Alexander Motin freebsd_committer 2020-08-17 14:33:12 UTC
It looks like a bug in Additional element status diagnostic page.  You may see they use element indexes from 1 to 12, while with eiioe=0 it has to be 0 to 11.  It would have sense if eiiie=1, but it is not.  I don't know what they were thinking about.  I would recommend to ask HPE about the firmware update.  I don't see error from our side here, the only thing we could do is a workaround, that I hate to do.
Comment 4 Mike Biesele 2020-08-17 20:33:42 UTC
Unfortunately, I have other MSA60s that are on the latest firmware (2.28, according to https://support.hpe.com/hpsc/swd/public/detail?swItemId=co_119008_1#tab-history ) that exhibit this same behavior.  I think they are past EOL/EOS so a firmware update is not possible.  I assume this enclosure falls under the category described at http://sg.danny.cz/sg/sg_ses.html : 

"Prior to the EIIOE bit's introduction in ses3r06 several arrays acted as if this bit was present and set. If 'force' is given the Additional Element Status page's element index field is treated as if EIIOE=1. If 'auto' is given a heuristic is used to decide whether to set EIIOE or not."

It looks like ses3r06 was introduced in August 2013, and the 2.28 firmware for this enclosure was released in May 2013.  

I think the heuristic described in the sg_ses documentation is documented in a large comment in sg_ses.c in sg3_utils :

/* Join array has four "element index"ing stategies:
 *   [1] based on all descriptors in the Enclosure Status (ES) dpage
 *   [2] based on the non-overall descriptors in the ES dpage
 *   [3] based on the non-overall descriptors of these element types
 *       in the ES dpage: DEVICE_ETC, ARRAY_DEV_ETC, SAS_EXPANDER_ETC,
 *       SCSI_IPORT_ETC, SCSI_TPORT_ETC and ENC_SCELECTR_ETC.
 *   [4] based on the non-overall descriptors of the SAS_CONNECTOR_ETC
 *       element type
 *
 * The indexes are all origin 0 with the maximum index being one less then
 * the number of status descriptors in the ES dpage. Table of supported
 * permutations follows:
 *
 *  ==========|===============================================================
 *  Algorithm |              Indexes                  |    Notes
 *            |Element|Connector element|Other element|
 *  ==========|=======|=================|=============|=======================
 *   [A]      |  [2]  |       [4]       |    [3]      | SES-2, OR
 *   [A]      |  [2]  |       [4]       |    [3]      | SES-3,EIIOE=0
 *  ----------|-------|-----------------|-------------|-----------------------
 *   [B]      |  [1]  |       [1]       |    [1]      | SES-3, EIIOE=1
 *  ----------|-------|-----------------|-------------|-----------------------
 *   [C]      |  [2]  |       [2]       |    [2]      | SES-3, EIIOE=2
 *  ----------|-------|-----------------|-------------|-----------------------
 *   [D]      |  [2]  |       [1]       |    [1]      | SES-3, EIIOE=3
 *  ----------|-------|-----------------|-------------|-----------------------
 *   [E]      |  [1]  |       [4]       |    [3]      | EIIOE=0 and
 *            |       |                 |             | --eiioe=force, OR
 *   [E]      |  [1]  |       [4]       |    [3]      | {HP JBOD} EIIOE=0 and
 *            |       |                 |             | --eiioe=auto and
 *            |       |                 |             | AES[desc_0].ei==1 .
 *  ----------|-------|-----------------|-------------|-----------------------
 *   [F]      | [2->3]|       [4]       |    [3]      | "broken_ei" when any
 *            |       |                 |             | of AES[*].ei invalid
 *            |       |                 |             | using strategy [2]
 *  ----------|-------|-----------------|-------------|-----------------------
 *   [Z]      |  -    |       [4]       |    [3]      | EIP=0, implicit
 *            |       |                 |             | element index of [3]
 *  ==========================================================================
 *
 *
 */

Which does look pretty cumbersome, and appears to call out some HP gear for specific treatment as well.  Ugh.

Would a sysctl tunable be plausible to implement?  Something like dev.mps.ses.X.eiioe=1 to mimic the --eiioe=force behavior from ses_util and allow working around the broken behavior?  I realize it may not be worth it at this point, of course.