3/23/23, 11:05 AM Document 2087727.

1
Copyright (c) 2023, Oracle. All rights reserved. Oracle Confidential.

FC HBA - SAN Troubleshooting in an Oracle Linux x86/x64 Environment (Doc ID 2087727.1)

In this Document

Main Content
References

APPLIES TO:

Emulex FC HBA - Version All Versions and later

Qlogic FC HBA - Version All Versions and later
Linux OS - Version Oracle Linux 5.0 to Oracle Linux 8.0 [Release OL5 to OL8]
Linux x86

MAIN CONTENT

This Document was created to assist in collecting SAN Connectivity data on Oracle Linux V5 and V6 and Redhat Linux V6
systems to aid in troubleshooting.

Note : Systools provides an easy way to get vital information. If Systools is not installed : yum install sysfsutils .
Install takes approx. 5 minutes.

Note : lshw provides an easy way to get slot information. If lshw is not installed : yum -y install lshw . Install takes approx.
5 minutes.

NOTE: Many of the example command outputs are not coordinated and are just for illustration purposes

When opening a new Oracle Service Request (SR) see following document for recommended support data
collection.

Document 2590638.1 Oracle Linux: Fibre Channel Server-Based Connectivity Issues - Basic information and Data Needed
by Oracle support

1 - Is Linux OS Oracle or NON-Oracle? What version of Oracle Linux?

# ls -la /etc/*release    -list release files, files contain kernel versions on the system

# ls -la /etc/*release
-rw-r--r--. 1 root root 152 Dec 16 07:10 /etc/lsb-release
-rw-r--r--. 1 root root  32 Jul 24  2015 /etc/oracle-release
-rw-r--r--. 1 root root 402 Jul 24  2015 /etc/os-release
-rw-r--r--. 1 root root  55 Jul 24  2015 /etc/redhat-release
lrwxrwxrwx. 1 root root  14 Dec 16 07:09 /etc/system-release -> oracle-release
#

Oracle VM (OVM) is Oracle Linux with integrated Virtualization features

The Role of Oracle VM Server for x86 in a Virtualization Strategy

# more /etc/*release
::::::::::::::
/etc/lsb-release
::::::::::::::
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 1/19
3/23/23, 11:05 AM Document 2087727.1
amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4
.0-noarch
::::::::::::::
/etc/oracle-release
::::::::::::::
Oracle Linux Server release 6.7
::::::::::::::
/etc/os-release
::::::::::::::
NAME="Oracle Linux Server"
VERSION="6.7"
ID="ol"
VERSION_ID="6.7"
PRETTY_NAME="Oracle Linux Server 6.7"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:6:7"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 6"

ORACLE_BUGZILLA_PRODUCT_VERSION=6.7
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=6.7
::::::::::::::
/etc/redhat-release
::::::::::::::
Red Hat Enterprise Linux Server release 6.7 (Santiago)
::::::::::::::
/etc/system-release
::::::::::::::
Oracle Linux Server release 6.7
#

# rpm -qf /etc/*release   -proper way to display the kernel version of each release

# rpm -qf /etc/*release

redhat-lsb-core-4.0-7.0.1.el6.x86_64
oraclelinux-release-6Server-7.0.5.x86_64
oraclelinux-release-6Server-7.0.5.x86_64
oraclelinux-release-6Server-7.0.5.x86_64
oraclelinux-release-6Server-7.0.5.x86_64
#

It's Oracle Linux 5 if /etc/enterprise_release exists.

It’s Oracle Linux 6 (or above) if /etc/oracle_release exists but no /etc/enterprise_release exists.
It’s Red Hat Linux if only redhat-release exists. Redhat-release exists to indicate redhat content.

See docs below:

Document 560992.1 Comparison of Red Hat and Oracle Linux kernel versions and release strings
Document 207296.1 How to Find out Linux Version Information?
 

Note: Output of "uname -a" command can also be used to identify Linux.

# uname -a
Linux host 4.1.12-61.1.18.el7uek.x86_64 #2 SMP Fri Nov 4 15:48:30 PDT 2016 x86_64 x86_64 x86_64
GNU/Linux
#

el7 = Enterprise Linux 7

uek = Unbreakable Enterprise Kernel (ie can only be Oracle Linux)

2 - Is Server Oracle or NON-Oracle? How determine the server model?

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 2/19
3/23/23, 11:05 AM Document 2087727.1

# dmidecode | grep -A8 "System Information" | egrep "Manufacturer|Name|Serial"

# dmidecode | grep -A8 "System Information" | egrep "Manufacturer|Name|Serial"

        Manufacturer: Sun Microsystems
        Product Name: SUN FIRE X4250
        Serial Number: <SN>
#

SosReport, SosCommands, Hardware/Dmidecode

Server Manufacturer / Model Number is in 2nd or 3rd entry

3 - Are FC HBA card(s) in system? Are FC HBA card(s) Oracle or NON-Oracle?

# lspci -nn | egrep -i "fibre|10df|1077"

05:00.0 Fibre Channel [0c04]: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter
[10df:fc40] (rev 03)
05:00.1 Fibre Channel [0c04]: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter
[10df:fc40] (rev 03)
0c:00.0 Fibre Channel [0c04]: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA
[1077:2432] (rev 02)
0c:00.1 Fibre Channel [0c04]: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA
[1077:2432] (rev 02)
#

When doing Live Linux system troubleshooting,

we can use the following script

perl -e '$s = "/sys/class/fc_host/host";$st = 'ls -d $s*';chomp($st);@a1 = split("\n", $st);@a1 = sort{$a <=> $b}
map{(/(\d{1,2})/)?($1):(0)} @a1;print "\n\nPort Port Name" . " " x 17 . "Node Name Type State Speed\n" . "-" x 87 .
"\n";foreach(@a1){$port = substr('cat $s$_/port_name',2,16);$port =~ s/..\K(?=.)/:/g;$node = substr('cat
$s$_/node_name',2,16);$node =~ s/..\K(?=.)/:/g;$type = substr('cat $s$_/port_type',0,5);$type =~
s/^U.*/unknown/;$state = 'cat $s$_/port_state';chomp($state);$speed = 'cat $s$_/speed';chomp($speed);printf
"%-6s %s %s %-7s %s %s\n", "host$_", $port, $node, $type, $state, $speed};print "\n\n";'

that would give an output like

Port Port Name Node Name Type State Speed

---------------------------------------------------------------------------------------
host2 10:00:00:00:xx:xx:xx:a0 20:00:00:00:xx:xx:xx:a0 NPort Online 2 Gbit
host3 10:00:00:00:xx:xx:xx:a1 20:00:00:00:xx:xx:xx:a1 NPort Online 4 Gbit
host10 10:00:00:90:xx:xx:xx:58 20:00:01:20:xx:xx:xx:58 NPort Online 2 Gbit
host11 10:00:00:90:xx:xx:xx:59 20:00:01:20:xx:xx:xx:59 NPort Online 2 Gbit

See also: MaxRep: How to map FC WWNs to the HBAs (Doc ID 2169732.1)

Note: Unlike Qlogic, Emulex list chipset names instead of card speed. Use table below to determine card speed.

ChipSet BUS  Speed

------- ---  -----
  Thor PCI-X 2Gb
Helios PCI-X 4Gb    
Zephyr PCI-E 4Gb

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 3/19
3/23/23, 11:05 AM Document 2087727.1
Saturn PCI-E 8Gb
Lancer PCI-E 16Gb

Each line on the lspci output above shows a PCI device. Each device is given a bus number, a device number and a
function number. On Linux, PCI devices are also given domain numbers, but they are usually omitted by lspci since very
often all devices have the same domain number (usually zero). These four numbers are assigned by Linux to each device
either on boot or when a device is hot-plugged. Since the PCI specification permits a system to host up to 256 buses,
nonzero domain numbers are only used to group PCI buses in very large systems. Each bus can host up to 32 devices, and
a PCI device can have up to eight functions. In more technical terms, a device's location is specified by a 16-bit domain
number, an 8-bit bus number, a 5-bit device number and a 3-bit function number; the last three numbers are commonly
referred to as the device's BDF or B/D/F (for bus/device/function).

The name of the slot where the device resides ([domain:]bus:device.function). This will need to be correlated to a physical
slot number on the server so the card can be located.

In this case, using the Emulex card as an example, the function number shows us the first port (port 0) of dual port fc hba
card in slot location 05:00.0.

And the second port (port 1) of the same dual port fc hba card as location 05:00.1.

Note: See item 15 below for command to correlate ([domain:]bus:device.function) to physical slot on system chassis.

Then check if fc hba cards are Oracle or NON-Oracle branded cards.

# lspci -v | grep -A 1 -i fibre

21:00.0 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)
        Subsystem: Emulex Corporation Device fc42
--
21:00.1 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)
        Subsystem: Emulex Corporation Device fc42
--
2f:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
        Subsystem: QLogic Corp. Device 0171
--
2f:00.1 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
        Subsystem: QLogic Corp. Device 0171

"Subsystem: QLogic Corp. Device xxxx" = subsystem ID (SSID) this value determines if it is an Oracle branded fc hba card per Oracle Knowledge Base document below.

# lspci -vmm -s 05:00.0

Slot:   05:00.0
Class:  Fibre Channel
Vendor: Emulex Corporation
Device: Saturn-X: LightPulse Fibre Channel Host Adapter
SVendor:        Emulex Corporation
SDevice:        Device fc42
Rev:    03

# lspci -vmm -s 0c:00.0

Slot:   0c:00.0
Class:  Fibre Channel
Vendor: QLogic Corp.
Device: ISP2432-based 4Gb Fibre Channel to PCI Express HBA
SVendor:        QLogic Corp.
SDevice:        Device 0143
Rev:    02

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 4/19
3/23/23, 11:05 AM Document 2087727.1

SDevice = subsystem ID (SSID) this value determines if it is an Oracle branded fc hba card per Oracle Knowledge Base
document below.

Or use following command which provides even more useful information. The end of the line that begins with "Subsystem:"
list the SSID.

# lspci -vvvs 05:00.0 | egrep "Fibre|Subsystem:|Name:|Part|\[V1\]|\[V2\]|driver|Slot"

05:00.0 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03)
        Subsystem: Emulex Corporation Device fc42
                Product Name: Sun StorageTek 8Gb FC PCIe HBA, dual port, Emulex
                        [PN] Part number: 371-4306-01
                        [V1] Vendor specific: Sun StorageTek LPe12002-S 8Gb 2-port PCIe FC HBA
                        [V2] Vendor specific: LPe12002-S
        Kernel driver in use: lpfc
#

# lspci -vvvs 0c:00.0 | egrep "Fibre|Subsystem:|Name:|Part|\[V1\]|\[V2\]|driver|Slot"

0c:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 02)
        Subsystem: QLogic Corp. Device 0143
                Product Name: SG-(X)PCIE2FC-QF4, Sun StorageTek 4 Gb FC Enterprise PCI-Express Dual
Channel Host Bus Adapter
                        [PN] Part number: 375-3356-01
                        [V1] Vendor specific: 01.00
        Kernel driver in use: qla2xxx
#

Document 1282491.1 Solaris: How to Identify Oracle[TM] Branded Fibre Channel (FC) HBA, CNA/FCoE and Universal 16GB
HBA Cards and Their Slot Locations

Check "subsystem-id" field value (ie SSID) of FC HBA cards against the HBA Table pdf file attached to determine if hba is
Oracle or NON-Oracle.

SosReport, SosCommands, Hardware, lspci : Search for ql or lp then scroll up to find SSID ( Also Part # )

SosReport, SosCommands, Hardware, usr.share...hardware.py : Search for ql or lp to find SSID

4 - Is/are the FC HBA port(s) online or offline? HBA Port Type ? , WWPN ? Speed ?  FC Switch WWN? Port
Address?

Each fc hba port is assigned an alias with format of "hostnn" nn being a unique number.

To list these fc hba port aliases run following command.

Note: If additional fc hba cards are added the "hostnn" aliases may change.
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 5/19
3/23/23, 11:05 AM Document 2087727.1

"Port" = end point ie. World Wide Port number (WWPN) therefore, a physical cable connected to a physical port may
have one or more virtual ports (ie WWPN's) associated with it.

host numbers (ie fc hba OS port numbers) are designated by H

bus numbers are B
targets are T
logical unit numbers (LUNs) are L
remote port numbers are R

Host: /sys/class/fc_host/hostH/
Remote Port: /sys/class/fc_remote_ports/rport-H:B-R/
Transport: /sys/class/fc_transport/targetH:B:T/

# ls -la /sys/class/fc_host
total 0
drwxr-xr-x  2 root root 0 Dec 16 13:52 .
drwxr-xr-x 46 root root 0 Dec 16 13:52 ..
lrwxrwxrwx  1 root root 0 Dec 16 13:52 host10 ->
../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/0000:0b:02.0/0000:0c:00.0/host10/fc_host/host10
lrwxrwxrwx  1 root root 0 Dec 16 13:52 host11 ->
../../devices/pci0000:00/0000:00:04.0/0000:0a:00.0/0000:0b:02.0/0000:0c:00.1/host11/fc_host/host11
lrwxrwxrwx  1 root root 0 Dec 16 13:52 host12 ->
../../devices/pci0000:00/0000:00:02.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/0000:04:02.0/0000:05:00
lrwxrwxrwx  1 root root 0 Dec 16 13:52 host13 ->
../../devices/pci0000:00/0000:00:02.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/0000:04:02.0/0000:05:00
#

Or

# ls /sys/class/fc_host
host10  host11  host12  host13
#

Then check each port for status and other information as well.

Quick check shows only host10 and host12 are online others are Linkdown

# cd /sys/class/fc_host/
[root]# for x in `ls`
> do
> echo $x
> cat $x/port_state
> done
host10
Online
host11
Linkdown
host12
Online
host13
Linkdown
[root]#

Or

# systool -c fc_host -A "port_state"

Class = "fc_host"

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 6/19
3/23/23, 11:05 AM Document 2087727.1

  Class Device = "host10"

    port_state          = "Linkdown"

    Device = "host10"

  Class Device = "host11"

    port_state          = "Linkdown"

    Device = "host11"

  Class Device = "host12"

    port_state          = "Online"

    Device = "host12"

  Class Device = "host14"

    port_state          = "Online"

    Device = "host14"

  Class Device = "host15"

    port_state          = "Linkdown"

    Device = "host15"

  Class Device = "host9"

    port_state          = "Online"

    Device = "host9"

Example: First port of the dual port Qlogic fc hba card which is online

fabric_name is WWN of the device hba port is directly connected to, in this case it is a fc switch

port_id is fc switch port address (domain id/port #/alpa) the hba is connected to

# systool -c fc_host -v host10

Class = "fc_host"

  Class Device = "host10"

  Class Device path =
"/sys/devices/pci0000:00/0000:00:04.0/0000:0a:00.0/0000:0b:02.0/0000:0c:00.0/host10/fc_host/host10"
    dev_loss_tmo        = "30"
    fabric_name         = "0x1000000xxxxxx000"
    issue_lip           = <store method only>
    max_npiv_vports     = "127"
    node_name           = "0x200000e0xxxxxx7b"
    npiv_vports_inuse   = "0"
    port_id             = "0x3c0d00"
    port_name           = "0x210000e0xxxxxx7b"
    port_state          = "Online"
    port_type           = "NPort (fabric via point-to-point)"
    speed               = "4 Gbit"
    supported_classes   = "Class 3"
    supported_speeds    = "1 Gbit, 2 Gbit, 4 Gbit"
    symbolic_name       = "QLE2462 FW:v7.04.00 DVR:v8.07.00.16.39.0-k"
    system_hostname     = ""
    tgtid_bind_type     = "wwpn (World Wide Port Name)"
    uevent              =

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 7/19
3/23/23, 11:05 AM Document 2087727.1
    vport_create        = <store method only>
    vport_delete        = <store method only>

    Device = "host10"

    Device path =
"/sys/devices/pci0000:00/0000:00:04.0/0000:0a:00.0/0000:0b:02.0/0000:0c:00.0/host10"
      fw_dump             =
      nvram               = "ISP "
      optrom_ctl          = <store method only>
      optrom              =
      reset               = <store method only>
      sfp                 = ""
      uevent              = "DEVTYPE=scsi_host"
      vpd                 = "▒^"

Example: Second port of the dual port Emulex fc hba card which is Linkdown

# systool -c fc_host -v host13

Class = "fc_host"

  Class Device = "host13"

  Class Device path =
"/sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/0000:04:02.0/0000:05:00
    active_fc4s         = "0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0x00
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 "
    dev_loss_tmo        = "30"
    fabric_name         = "0x0"
    issue_lip           = <store method only>
    max_npiv_vports     = "255"
    maxframe_size       = "2048 bytes"
    node_name           = "0x20000000xxxxxx17"
    npiv_vports_inuse   = "0"
    port_id             = "0x000000"
    port_name           = "0x10000000xxxxxx17"
    port_state          = "Linkdown"
    port_type           = "Unknown"
    speed               = "unknown"
    supported_classes   = "Class 3"
    supported_fc4s      = "0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0x00
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 "
    supported_speeds    = "2 Gbit, 4 Gbit, 8 Gbit"
    symbolic_name       = "Emulex LPe12002-S FV1.11A5 DV10.6.61.0 HN:(none) OS:Linux"
    tgtid_bind_type     = "wwpn (World Wide Port Name)"
    uevent              =
    vport_create        = <store method only>
    vport_delete        = <store method only>

    Device = "host13"

    Device path =
"/sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/0000:04:02.0/0000:05:00
      uevent              = "DEVTYPE=scsi_host"

Or

cd /sys/class/fc_host/hostnn/

then cat the file with info desired

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 8/19
3/23/23, 11:05 AM Document 2087727.1

5 - Is the FC HBA port seeing storage target WWN?

Note: If the port is not online it can not see any storage.

systool -c fc_remote_ports -v

Example showing no port is seeing any storage

# systool -c fc_remote_ports -v
Class = "fc_remote_ports"

Example of host seeing 4 tape drives

# systool -c fc_remote_ports -v | egrep "remote|port_id|port_name|port_state|roles"

Class = "fc_remote_ports"

  Class Device path = "/sys/devices/pci0000:00/0000:00:02.0/0000:05:00.0/host1/rport-1:0-

0/fc_remote_ports/rport-1:0-0"
    port_id             = "0x560500"
    port_name           = "0x500104f0xxxxxxxx"
    port_state          = "Online"
    roles               = "FCP Target"
  Class Device path = "/sys/devices/pci0000:00/0000:00:03.2/0000:0b:00.0/host3/rport-3:0-
0/fc_remote_ports/rport-3:0-0"
    port_id             = "0xa20600"
    port_name           = "0x500104f0xxxxxxxx"
    port_state          = "Online"
    roles               = "FCP Target"
  Class Device path = "/sys/devices/pci0000:00/0000:00:02.0/0000:05:00.0/host1/rport-1:0-
1/fc_remote_ports/rport-1:0-1"
    port_id             = "0x560600"
    port_name           = "0x500104f0xxxxxxxx"
    port_state          = "Online"
    roles               = "FCP Target"
  Class Device path = "/sys/devices/pci0000:00/0000:00:03.2/0000:0b:00.0/host3/rport-3:0-
1/fc_remote_ports/rport-3:0-1"
    port_id             = "0xa20700"
    port_name           = "0x500104f0xxxxxxxx"
    port_state          = "Online"
    roles               = "FCP Target"
#

Example of host only seeing some "Well Known" addresses of the fc switch

Note: Usually FC Emulex FC HBA cards will display "Well Known" addresses but not Qlogic FC HBA cards.

# systool -c fc_remote_ports -v | egrep "remote|port_id|port_name|port_state|roles"

Class = "fc_remote_ports"
  Class Device path =
"/sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/0000:10:02.0/0000:11:00.1/host12/rport-12:0-
0/fc_remote_ports/rport-12:0-0"
    port_id             = "0xfffffe"
    port_name           = "0x200c0005xxxxxxxx"
    port_state          = "Online"
    roles               = "Fabric Port"
  Class Device path =
"/sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/0000:10:02.0/0000:11:00.1/host12/rport-12:0-

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 9/19
3/23/23, 11:05 AM Document 2087727.1
1/fc_remote_ports/rport-12:0-1"
    port_id             = "0xfffffc"
    port_name           = "0x21fc0005xxxxxxxx"
    port_state          = "Online"
    roles               = "Directory Server"
  Class Device path =
"/sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/0000:04:02.0/0000:05:00
9:0-1/fc_remote_ports/rport-9:0-1"
    port_id             = "0xfffffc"
    port_name           = "0x21fc0005xxxxxxxx"
    port_state          = "Online"
    roles               = "Directory Server"
  Class Device path =
"/sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/0000:04:02.0/0000:05:00
9:0-12/fc_remote_ports/rport-9:0-12"
    port_id             = "0xfffffe"
    port_name           = "0x200a0005xxxxxxxx"
    port_state          = "Online"
    roles               = "Fabric Port"
  Class Device path =
"/sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/0000:04:02.0/0000:05:00
9:0-3/fc_remote_ports/rport-9:0-3"
    port_id             = "0xfffffe"
    port_name           = "0x200a0005xxxxxxxx"
    port_state          = "Online"
    roles               = "Fabric Port"
  Class Device path =
"/sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/0000:04:02.0/0000:05:00
9:0-7/fc_remote_ports/rport-9:0-7"
    port_id             = "0xfffffe"
    port_name           = "0x200a0005xxxxxxxx"
    port_state          = "Online"
    roles               = "Fabric Port"
#

/sys/class/fc_remote_ports/rport(x)/port_name ( WWPN ) or port_state ( Online )

6 - Are the storage target WWNs presenting LUNs? Are those LUNs configured or unconfigured for use by
OS?

fdisk -l, then fdisk -l /dev/sd(x)

SosReport, SOSCommands, Devicemapper, udevinfo or pvscan_-v or pvs_-a_-v

Systool –c fc_transport –v

/sys/class/fc_transport/target(x)/node_name or port_name

systool -c scsi_disk -v

/sys/class/scsi_host/host(x) directory

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 10/19
3/23/23, 11:05 AM Document 2087727.1
/proc/scsi/scsi

/proc/partitions : List of LUN's + Major / Minor Numbers

/dev/disk/by-path and /dev/disk/by-id

7 - How can device tree be refreshed for SAN devices?

Automatically

echo 1 > /sys/class/fc_host/host(x)/issue_lip ( Did not work in Oracle Linux 6 )

Rescan Scsi Bus : echo - - - > /sys/class/scsi_host/host(x)/scan  ( Works in Oracle Linux 6 )

8 - Software versions installed on/for fc hba card? How is software upgraded?

# systool -c scsi_host -v host4 | egrep "scsi_host|_version|model|fwrev"

Class = "scsi_host"
  Class Device path = "/sys/devices/pci0000:00/0000:00:0d.0/0000:03:00.0/host4/scsi_host/host4"
    84xx_fw_version     = ""
    driver_version      = "8.07.00.16.39.0-k"
    fw_version          = "7.05.00 (90d5)"
    model_desc          = "Sun StorageTek 8Gb FC PCIe HBA, single port"
    model_name          = "QLE2560"
    mpi_version         = ""
    optrom_bios_version = "2.02"
    optrom_efi_version  = "2.01"
    optrom_fcode_version= "2.03"
    optrom_fw_version   = "4.03.02 128"
    optrom_gold_fw_version= ""
    phy_version         = ""
      uevent              = "DEVTYPE=scsi_host"
#

# systool -c scsi_host -v

# systool -c scsi_host -v host5 | egrep "scsi_host|_version|model|fwrev"

Class = "scsi_host"
  Class Device path = "/sys/devices/pci0000:00/0000:00:0e.0/0000:04:00.0/host5/scsi_host/host5"
    fwrev               = "1.00A9 (U3D1.00A9), sli-3"
    lpfc_drvr_version   = "Emulex LightPulse Fibre Channel SCSI driver 10.6.61.0"
    modeldesc           = "Sun StorageTek LPe12002-S 8Gb 2-port PCIe FC HBA"
    modelname           = "LPe12002-S"
    option_rom_version  = "5.03a0"
      uevent              = "DEVTYPE=scsi_host"
#

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 11/19
3/23/23, 11:05 AM Document 2087727.1
 /sys/class/scsi_host/host(x)/x 

 Where x = driver_version and fw_version for qla and lpfc_drvr_verson and fwrev for lpfc

 ====

Firmware update:

How to Update QLogic HBA Firmware in Oracle Linux (Doc ID 2765892.1)

 
9 - Is there a "/var/adm/messages" file in Linux?

/var/log/messages and /var/log/dmesg

10 - How is support data collected? What version of sosreport? How to upgrade sosreport?

Sosreport at command line

11 - Any previous sosreports saved anywhere like explorer?

/tmp

Survives reboot

12 - Is fc switch ports location for fc hba and other devices available (like luxadm -e dump_map) ?

TBD

13 - How do I pull up link error counters for fc hba ports?

/sys/class/fc_host/host(x)/statistics ( Directory )

14 - Is there a format command in Linux? How can we query devices?

fdisk –l

/proc/scsi/scsi

SosReport, SosCommands, Devicemapper, pvs_-a_-v

Systool –c fc_transport -v

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 12/19
3/23/23, 11:05 AM Document 2087727.1
 

15 - FC HBA port host ( Instance ) numbers? Slot Locations ? What controller number is assigned to each fc
hba port?

See also
Document 2590209.1 Oracle Linux: Locating Fibre Channel (FC) HBA Card Physical Slot and Port location
in Oracle x86 Server

# lshw -numeric | egrep "\*-pci:|description|bus|capabilities|fiber|product|vendor|qla|lpfc" | grep

-B6 -A6 "\*-fiber"
     *-pci:2
          description: PCI bridge
          product: CK804 PCIE Bridge [10DE:5D]
          vendor: NVIDIA Corporation [10DE]
          bus info: pci@0000:00:0d.0
          capabilities: pci pm msi ht pciexpress normal_decode bus_primary cap_list
        *-fiber
             description: Fibre Channel
             product: ISP2532-based 8Gb Fibre Channel to PCI Express HBA [1077:2532]
             vendor: QLogic Corp. [1077]
             bus info: pci@0000:03:00.0
             capabilities: pm pciexpress msi vpd msix bus_primary cap_list rom
             configuration: driver=qla2xxx latency=0
     *-pci:3
          description: PCI bridge
          product: CK804 PCIE Bridge [10DE:5D]
          vendor: NVIDIA Corporation [10DE]
          bus info: pci@0000:00:0e.0
          capabilities: pci pm msi ht pciexpress normal_decode bus_primary cap_list
        *-fiber:0
             description: Fibre Channel
             product: Saturn-X: LightPulse Fibre Channel Host Adapter [10DF:FC40]
             vendor: Emulex Corporation [10DF]
             bus info: pci@0000:04:00.0
             capabilities: pm msi msix vpd pciexpress bus_primary cap_list rom
             configuration: driver=lpfc latency=0
        *-fiber:1
             description: Fibre Channel
             product: Saturn-X: LightPulse Fibre Channel Host Adapter [10DF:FC40]
             vendor: Emulex Corporation [10DF]
             bus info: pci@0000:04:00.1
             capabilities: pm msi msix vpd pciexpress bus_primary cap_list rom
             configuration: driver=lpfc latency=0
#

Note: the above output needs to be double checked via dmidecode command output below by comparing bus 
xxxx.aa.bb.c  ignore xxxx digits.

# dmidecode -t slot | grep -E "dmidecode|SMBIOS|Designation|Usage|Bus Address"

# dmidecode 3.0
SMBIOS 2.6 present.
        Designation: PCIE0
        Current Usage: In Use
        Bus Address: ffff:13:00.0
        Designation: PCIE1
        Current Usage: In Use
        Bus Address: ffff:21:00.0
        Designation: PCIE2
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 13/19
3/23/23, 11:05 AM Document 2087727.1
        Current Usage: In Use
        Bus Address: ffff:2f:00.0
        Designation: PCIE3
        Current Usage: Available
        Designation: PCIE4
        Current Usage: Available
        Designation: PCIE5
        Current Usage: Available

Note: If dmidecode command output does not list Bus Address then most likely the server bios is to old. In that case
correlating the fc hba card Bus:Device.Function location to a physical external slot will have to be done via observation
and deductive reasoning.

# dmidecode -s bios-version
07060315
#
# dmidecode -s bios-release-date
04/14/2016
#

systool -c fc_host -v : If the output does not have Slot Number, then look at the line named Class Device Path ;

Example : Class Device path = "/sys/devices/pci0000:00/0000:00:02.2/0000:30:00.0/host2/fc_host/host2"

Search Sosreport, Soscommands, hardware, dmidecode for the Bus Address, which in this case is 0000:00:02.2 to find slot
#

15.1 - Enable/Disable Beacon LED pattern to locate FC HBA card

Qlogic

To enable beacon LED blink on "host#" HBA, run:

echo 1 > /sys/class/scsi_host/host#/beacon

To disable it again, run:

echo 0 > /sys/class/scsi_host/host#/beacon

Emulex

to be determined

16 - How do I tell if HBA driver is loaded ?

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 14/19
3/23/23, 11:05 AM Document 2087727.1
SosReport, SosCommands, Kernel, lsmod  Search for qla or lpfc

17 - Is there FMA in Linux?

Ipmitool

If not installed : yum install ipmitool

18 - Where can I find Multipathing information ?

Default verbosity:

# multipath –ll

Debug verbosity:

# multipath –ll –v 3

/etc/multipath.conf

/etc/multipath/wwids

If not installed : yum install device-mapper-multipath

Oracle Manual ‘Configuring Multipathing for Oracle Linux and the Oracle ZFS Storage Appliance

How to Decode multipath -ll output ? 

multipath_-l
::::::::::::::
360000970000296700327533030303241 dm-1 EMC,SYMMETRIX
size=20G features='1 queue_if_no_path' hwhandler='0' wp=rw
'-+- policy='round-robin 0' prio=0 status=active
|- 1:0:1:1 sdc 8:32 active undef running
'- 1:0:0:1 sdf 8:80 active undef running

[a:b:c:d] --> scsi_host : channel_no : target_number : LUN_id

[1:0:1:1] --> HBA_port_no : Channel_no : target_port_id : Lun_ID

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 15/19
3/23/23, 11:05 AM Document 2087727.1
 

19 - How to View HBA Parameters ?

/sys/module/qlx2xxx/parameters : Each parameter is a file containing a value.

/sys/module/lpfc/parameters : Each parameter is a file containing a value.

                                                                 

20 - How to enable extended logging ?

echo 1 > /sys/module/qla2xxx/parameters/ql2xextended_error_logging : Will not survive a reboot

echo 0xffff > /sys/class/scsi_host/host(x)/lpfc_log_verbose

To enable extended scsi logging : echo x > /proc/sys/dev/scsi/logging_level  Where x = 1 to 7 verbosity

To disable, echo 0 into the file

21 - Find Uptime, Date SosReport was run, Hostname

SosReport, SosCommands, general

Common Support Data to Collect

 

1 - Sosreport

2 - systool -c fc_host -v

                - Slot Location

                - HBA WWPN

                - Port State ( Online / Offline )

                - Port Type ( N Port )

                - Speed ( Negotiated Speed )

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 16/19
3/23/23, 11:05 AM Document 2087727.1
3 - systool -c scsi_host -v

                - Instance Name ( host0, host1, etc )

                - Mode ( Initiator, Target )

                - Link Speed

                - Link Up / Down

                - Loop or Fabric

                - Driver Name and Version

                - Firmware Version

                - PCI Bus Type / Speed ( PCIe ( 2.5GT/s x4 ))

                - HBA Parameters

                - HBA Model Name

                - HBA Option Number ( Part Number )

                - Slot #

4 - systool -c fc_remote_ports -v

                - Remote Port WWPN's

                - System Rport #

                - Timeout Values

                - Port State ( Online / Offline )

                - Device Path maps to Host Instance Number ( HBA Port )

5 - systool –c fc_transport –v

-       Target WWPN’s

-       System Target #

6 - systool -c scsi_disk -v

                - Attached Disk Devices

                - Disk Device Parameters

                - Disk Model Name

                - Failover Mode ( ALUA )

7 - multipath –ll –v3, /etc/multipath.conf

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 17/19
3/23/23, 11:05 AM Document 2087727.1
 

                - Multipathing Information

                - Target WWPN's

8 – systool –c infiniband –v ( If necessary )

9 – systool –c iscsi_connection –v ( If necessary )

-       Also iscsi_endpoint, iscsi_host, iscsi_session, iscsi_transport

10 - /sys/module/lpfc or ql2xxx/ parameter

-       HBA Parameters

-       View or echo {new value} > {parameter file name}

11 – Vendor Utility Programs

-       QConverge Console for Qlogic HBA’s

-       One Command Manager for Emulex HBA’s

Useful Commands/Files within SOSreport output

 

/var/log/boot.log <<-- boot log !!

/var/log/dmesg <<--- message file (detailed view)
/var/log/messages <<--- message file
/var/log/yum.log <<-- gives information which packages removed, installed, any new changes done to any packages, etc !!

-- /sos_commands <<--- important file for SAN and Storage troubleshooting

dmidecode -> sos_commands/hardware/dmidecode

last -> sos_commands/last/last
lsmod -> sos_commands/kernel/lsmod
lspci -> sos_commands/pci/lspci_-nnvv
mount -> sos_commands/filesys/mount_-l
ip_addr -> sos_commands/networking/ip_-o_addr
route -> sos_commands/networking/route_-n
uptime -> sos_commands/general/uptime
uname -> sos_commands/kernel/uname_-a
date -> sos_commands/general/date
df -> sos_commands/filesys/df_-al

-- Configuration related files under SOSreport

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 18/19
3/23/23, 11:05 AM Document 2087727.1
/etc/firewalld <<--- firewall configuration files !!
/etc/multipath.conf <<--- multipath configuration file !!
/etc/NetworkManager/NetworkManager.conf <<--- Network configuration file !!
/etc/openldap/ldap.conf <<--- LDAP configuration file !!
/etc/os-release <<-- current running OS release verion details !!
/etc/resolv.conf <<-- nameserver details information !!
/etc/rsyslog.conf <<-- mail server, messsage log, boot log, etc configuration file details !!
/etc/selinux <<--- Selinus security log and configuration files !!
/etc/snmp <<--- SNMP log and configuration files !!
/etc/ssh <<--- SSH log and configuration files !!
/etc/systemd <<-- system all configuration of various activity files !!
/etc/yum.repos.d <<--- Yum repository details and configuration files !!
/etc/yum.conf <<-- Yum configuration file !!

sys/devices/system/cpu <<-- CPU, memmory, Power detail directory and files !!

/sos_logs/sos.log <<--- sos log information that shows what all are the command being executed while running SOSreport.

<Snip>
2018-03-15 11:02:36,807 INFO: [plugin:pci] collecting output of 'lspci'
2018-03-15 11:02:36,880 INFO: [plugin:pci] collecting output of 'lspci -nvv'
2018-03-15 11:02:37,082 INFO: [plugin:pci] collecting output of 'lspci -tv'
<Snip>

/sos_reports/sos.txt <<--- readable text file

            
 
Didn't find what you are looking for?

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=lm4i08351_462&id=2087727.1 19/19