Troubleshooting :

Virtual Interface Card (VIC)

SAN Boot and SAN Connectivity Issues

Yogesha MG
-29Mar’17
1
 Quick Quiz Question

• How many logical devices we can create on

Cisco VIC UCSM mode?
 2
 16
 128
 256
 VIC Overview
• Designed exclusively for Cisco UCS B-Series Blade Servers
and UCS C-Series Blade Servers

• Cisco UCS Virtual Interface Card (VIC) is a 2-port, 10/40

Gigabit Ethernet, Fibre Channel over Ethernet (FCoE)-
capable mezzanine adapter

• Over 256 PCIe standards-compliant interfaces to the host

that can be dynamically configured as either network
interface cards (NICs) or host bus adapters (HBAs)

• It Supports Virtual Machine Fabric Extender (VM-FEX)

technology, which extends the Cisco UCS fabric interconnect
ports to virtual machines, simplifying server virtualization
deployment and management.
CISCO UCS VIC Architecture
 VIC Terminology
• vNIC: Virtual NIC
–Host presented PCI device that is centrally
managed within UCSM

• LIF: Logical Interface

– A logical channel end point on an adapter

• VIF: Virtual Interface

– Network-facing entity
 B-Series VIC adapter

Note:
1. Mix of Gen-2 & Gen-3 adapters is not supported.
2. Gen-1 adapters are not supported in new releases.
3. 3rd Party Adapters
• Only support 2 vEth & 2 vFC devices
• No failover & nictool support [for fw upgrades]
 C-Series UCSM Managed VIC adapters

Note:
1. VIC 1385 & 1387 are supported on DelNorte-2 & Imperial Servers
 Identifying VIC — GUI

 UCSM displays adapter details from the Equipment

tab view
 Adapter-specific information can be viewed for each
installed adapter card
 Identifying VIC — CLI
• Adapter type and state information can be
obtained from the UCS CLI:
H16-Bala-Mammoth-A# scope server 2/1
H16-Bala-Mammoth-A /chassis/server # show adapter detail
Id: 2
Product Name: Cisco UCS VIC 1380
PID: UCSB-VIC-M83-8P
VID: V00
Vendor: Cisco Systems Inc
Serial: FCH1844JXQ6
Revision: 0
Mfg Date: 2014-11-10T00:00:00.000
Slot: N/A
Overall Status: Operable
Conn Path: A,B
Conn Status: A,B
Managing Instance: A
Product Description: PCI Express Dual Port Cisco Converged
Network Adapter Supporting 40Gb Ethernet and 40Gb Fibre Channel
 vNIC Types
The VIC supports seven different types of
virtual NICS:
• Enic
• Dynamic vnic
• iSCSI vnic
• RNIC
• usNIC
• sNIC
• fnic
 Adapter Debug CLI (Cont.)
• The Master Control Program is the main
firmware application, which runs on the VIC.
‘attach-mcp’ will provide access into the
VIC’s MCP:
adapter 1/1/1 (top):3# attach-mcp

• As the VIC does not have a traditional BIOS

utility, ‘attach-fls’ will connect to the Fibre
Channel Port services on the adapter to
verify and troubleshoot SAN boot issues:
adapter 1/1/1 (top):4# attach-fls
VIC Data Path/Circuit Trace

12
 Service Profile Circuit Paths
• A server’s virtual interface (VIF) mapping can be displayed with the ‘show service-profile circuit’
command.
• The VIF number can be used in the NXOS level of the FI to trace a path to the upstream.

H16-Bala-Mammoth-A# show service-profile circuit server 2/1

Service Profile: B200M4_C2B1_Vmware
Server: 2/1
Fabric ID: A
Path ID: 2
VIF vNIC Link State Oper State Prot State Prot Role Admin Pin Oper Pin Transport
---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- ---------

9109 Up Active No Protection Unprotected 0/0 0/0 Ether

912 eth3 Up Active Active Primary 0/0 1/9 Ether
915 eth4 Up Active Passive Backup 0/0 1/9 Ether
917 test2 Offline Unknown No Protection Unprotected 0/0 0/0 Fc
Fabric ID: B
Path ID: 1
VIF vNIC Link State Oper State Prot State Prot Role Admin Pin Oper Pin Transport
---------- --------------- ----------- ---------- ------------- ----------- ---------- ---------- ---------
907 eth0 Up Active Passive Backup 0/0 1/9 Ether
909 eth1 Up Active Passive Backup 0/0 1/9 Ether
910 eth2 Up Active Active Primary 0/0 1/9 Ether
 VIF View from NXOS
H16-Bala-Mammoth-A# connect nxos

• Ethernet VIFs will be prefaced with

‘vethernet’
• FC VIFs will be prefaced with ‘vfc’
• The virtual interface output will display the IO
Module port a particular VIF is bound to:
H16-Bala-Mammoth-A# show interface vethernet 965
vethernet965 is up
Bound Interface is Ethernet1/1/1
Hardware: VEthernet
Encapsulation ARPA
Port mode is trunk
Last link flapped 3week(s) 2day(s)
Last clearing of "show interface" counters never
13 interface resets
 Active Border Interfaces
• The interface a VIF will use to reach the
upstream network can be determined by
analyzing the active border-interfaces on
the FI:
H16-Bala-Mammoth-A# show pinning border-interfaces active

--------------------+---------+----------------------------------------
Border Interface Status SIFs
--------------------+---------+----------------------------------------
Po3 Active veth961 veth965 veth973 veth977 veth981
veth985 veth989 veth993 veth997
veth1001 veth1005 veth1009 veth1013
veth1017 veth1021 veth1025 veth1029
veth1033 veth1037 veth1041 veth1045
veth1049 veth1053 veth1057 veth1061
veth1065 veth1069 veth1073 veth1081
veth1085

Total Interfaces : 1
VIC Logs

16
 Adapter Log
• VIC log can be viewed after connecting to
a server’s adapter:
adapter 1/1/1 (top):2# show-log

• This will show a series of steps and

different configuration points that the
adapter has gone through during boot
up, as well as vNIC status changes, and
connection details
 Collecting Adapter Tech-Support

• A tech-support can be generated for the

VIC to view logs and details around the
adapter card:
H16-Bala-Mammoth-A# connect local-mgmt
H16-Bala-Mammoth-A(local-mgmt)# show tech-support chassis 1
cimc 1 adapter 1 detail

• It is generally recommended to collect the full chassis

tech-support for TAC to provide complete details of the
environment, as well as UCSM logs:

H16-Bala-Mammoth-A# connect local-mgmt

H16-Bala-Mammoth-A(local-mgmt)# show tech-support chassis 1
all detail
H16-Bala-Mammoth-A(local-mgmt)# show tech-support ucsm detail
 Host Side Logs
• Driver Version Info [eNIC, fNIC]
• Enable debug logs
• Dmesg, /var/log/messages – Linux, VMKernel Log
– ESX & Event Viewer Messages – Windows
• Host OS info
• FCC, eNIC & fNIC stats, Multipath info.
• Command line options in tools
• Enable kernel dumps for OS crash issues.
• Check the event logs
Note: All info to be captured after reproducing the
issue.
Troubleshooting SAN Boot on VIC

20
SAN boot troubleshooting with VIC
• With CISCO VIC, once you enable a boot FNIC and reboot
the blade, the Option ROM for that FNIC gets enabled and
tells FW to log into the fabric to discover the boot LUNs.

• If the SAN switches are configured correctly, LUNs

mapped, and boot targets configured correctly, VIC Option
ROM reports the boot LUN to the System BIOS and you
will see it in the boot menu.

• CISCO VIC does not have a traditional "BIOS utility", you

use UCSM CLI commands and the blade navigator "Boot
Order Details" dialogue to monitor, and troubleshoot SAN
booting issues.
SAN boot troubleshooting with VIC
• Use Boot Order Details to verify the current BIOS boot order configuration
• "Boot Order Details" is a real time view into the current status of the BIOS boot
order configuration
– Rather than have to stop the boot and enter a separate interface one can
view the BIOS live.
– If you observe it as a blade cycles you will see it transition several times
• Boot Order Details is under Equipment->Server->General
SAN boot troubleshooting with VIC
• Use Boot Order Details to verify the current BIOS boot order configuration
 SAN boot troubleshooting with VIC

• Check option ROM Execution[Note : Quite boot should be disabled]

• Lunmap and lunlist will only work after the CiscoVIC initializes.
• You can run lunmap and lunlist after you see this screen
adapter 1/2/1 (fls):2# lunmap
lunmapid: 0 port_cnt: 1
lif_id: 4
PORTNAME NODENAME LUN PLOGI
50:0a:09:86:86:78:39:66 00:00:00:00:00:00:00:00 0011000000000000 Y

lunmapid: 0 port_cnt: 1
lif_id: 5
PORTNAME NODENAME LUN PLOGI
50:0a:09:85:86:78:39:66 00:00:00:00:00:00:00:00 0011000000000000 N
SAN boot troubleshooting with VIC
• Connect to the adapter chassis/Blade slot/card:
phx2-dc-ucs-A# connect adapter 2/2/1
• C-series - Standalone
ucs-c220-m3# connect debug-shell <1|2>
adapter (top):1#
• Open rlogin session to VIC interface:
adapter 2/2/1 # connect
• Attach to the FC port services:
adapter 2/2/1 (top):1# attach-fls
• Query vNIC interfaces to get numbering:
adapter 2/2/1 (fls):2# vnic
---- ---- ---- ------- -------
vnic ecpu type state lif
---- ---- ---- ------- -------
7 1 fc active 5
8 2 fc active 6
SAN boot troubleshooting with VIC

• Get port login information for each interface - target

PWWN and FCID:

adapter 2/2/1 (fls):3# login 7

lifid: 5
ID PORTNAME NODENAME FID
0: 50:0a:09:83:99:fa:e1:e3 00:00:00:00:00:00:00:00 0xe60000

adapter 2/2/1 (fls):4# login 8

lifid: 6
ID PORTNAME NODENAME FID
0: 50:0a:09:83:89:fa:e1:e3 00:00:00:00:00:00:00:00 0xef0001
SAN boot troubleshooting with VIC
• Have the interfaces logged into the target?

adapter 2/2/1 (fls):5# lunmap 7

lunmapid: 0 port_cnt: 1
lif_id: 5
PORTNAME NODENAME LUN PLOGI
50:0a:09:83:99:fa:e1:e3 00:00:00:00:00:00:00:00 0000000000000000 Y

adapter 2/2/1 (fls):6# lunmap 8

lunmapid: 0 port_cnt: 1
lif_id: 6
PORTNAME NODENAME LUN PLOGI
50:0a:09:83:89:fa:e1:e3 00:00:00:00:00:00:00:00 0000000000000000 Y
 FC Login Information – VIC
• Working Example

Configured
Target details

Boot LUN

All visible Array info

LUNs

WWPN targets are visible

to the server
 FC Login Information – VIC
Incorrect LUN Masking : server that is having an issue
with incorrect LUN masking. The host has not been allowed access to
the LUN.
 FC Login Information – VIC
• Incorrect SAN : The host is not zoned correctly

PLOGI is
unsuccessful

no response returned for either of these queries

 FC Login Information – VIC
• Incorrect SAN Boot Target
 FC Login Information – VIC
• Incorrect LUN ID
Summary Troubleshooting SAN
Boot on VIC

40
 Adapter Connect debug-shell
• For troubleshooting, you must connect to the debug shell
after the adapter has initialized (success of fail)

• But, before any OS loads and the interface drivers load

• After the OS boots and the host drivers are loaded, it logs
out of the fabric and the host drivers take over

• **If you want more time to investigate issues, hit F2

(Enter Setup) during boot. This will allow the adapters to
initialize and then stop POST at the BIOS screen before
the OS loads
iSCSI boot config and debug

51
iSCSI Adapter Policy

• iSCSI adapter policies are under the Service Profile tab

under the Adapter Policies
iSCSI Adapter Policy

• Create a new iSCSI policy. You will get the following window
 iSCSI Troubleshoot : Quite boot
• Disable quiet boot for your blade models
• This makes troubleshooting so much easier
– Allows you to see if the Cisco VIC has initialized correctly
– Will allow you to look into the OptionROM session
 iSCSI Boot Service Policy
• Use expert mode
 iSCSI Boot Service Policy
• Create vNIC0 and vNIC1, no failover, specify native
 iSCSI Boot Service Policy
• With vNIC0 and vNIC1 created go to Add iSCSI vNICs
 iSCSI Boot Service Policy
• Overlay vNIC – the iSCSI
vNIC uses the Service
Profile vNIC.
• iSCSI adapter policy
(Broadcom has specific
requirements)
• Identify native VLAN
• MAC address:
– Cisco VIC – none (do not
set)
– M51KR-B – must be set
(manual or pool)
 iSCSI Devices
• You can create multiple iSCSI vNICs for HA. Only one path will
be used during install. This is how iBFT works
iSCSI Boot Service Policy
• Create the boot policy
– Cisco VIC, CD can be first or second.
 iSCSI Boot Service Policy
• Assign the boot policy to the SP
 iSCSI Boot Service Policy
• Set the iSCSI parameters – Select the first iSCSI boot devices
(you can have multiple iSCSI initiators)
 iSCSI Boot Service Policy
• Initiator name (EUI or iQN formats).
• Initiator IP Address
• Target name (Query your array for the correct name, iQN format)

 CHAP Authentication
 Initiator Auth – to iSCSI array (E.g.
NetApp inbound)
 Target Auth – from iSCSI array (E.g.
NetApp outbound) Initiator Auth

 Target Port, IP and LUN ID

 This is a simple example. Refer to
configuration guide for allowable
syntax
Target Auth
 iSCSI Authentication Profile
• Specify a name for the Auth profile and
provide the correct username/password
credentials that match that configured on the
iSCSI Array
 Initiator IQN Configurations
• Option here to use Pool or manual
• Note still must be unique even if pooled
 iSCSI Boot Service Policy
• Make sure you set the Initiator IP address, DHCP, pool, or
static. Repeat for second iSCSI initiator
• Pool IP will not display after you hit OK. You can go back and
check what was assigned after finishing SP
 Cisco VIC Initialization
• With quiet boot disabled, if your service profile and iSCSI
config is correct, you will see this during POST
 Cisco VIC Initialization - Successful
• Can verify Cisco VIC configuration by connecting to the
adapter and viewing the configuration
cae-sj-ca1-A# conn adapter 1/8/1
adapter 1/8/1 # connect
adapter 1/8/1 (top):1# attach-mcp
adapter 1/8/1 (mcp):1# iscsi_get_config

vnic iSCSI Configuration:

----------------------------
vnic_id: 5
link_state: Up
Initiator Cfg:
initiator_state: ISCSI_INITIATOR_READY
initiator_error_code: ISCSI_BOOT_NIC_NO_ERROR
vlan: 0
dhcp status: false
IQN: eui.87654321ab1234cd
IP Addr: 172.25.183.142
Subnet Mask: 255.255.255.0
Gateway: 172.25.183.1
 Cisco VIC Initialization Success –
cont’d
Target Cfg:
Target Idx: 0
State: ISCSI_TARGET_READY
Prev State: ISCSI_TARGET_DISABLED
Target Error: ISCSI_TARGET_NO_ERROR
IQN: iqn.1992-08.com.netapp:sn.101202278
IP Addr: 172.25.183.49
Port: 3260
Boot Lun: 0
Ping Stats: Success (9.698ms)
Session Info:
session_id: 0
host_number: 0
bus_number: 0
target_id: 0
Cisco VIC Initialization - failure
adapter 1/4/1 (mcp):2# iscsi_get_config

vnic iSCSI Configuration:

----------------------------

vnic_id: 13
link_state: Up

Initiator Cfg:
initiator_state: ISCSI_INITIATOR_READY
initiator_error_code: ISCSI_BOOT_NIC_NO_ERROR
vlan: 0
dhcp status: false
IQN: eui.feeddeadbeefcafe
IP Addr: 1.1.1.170
Subnet Mask: 255.255.255.0
Gateway: 1.1.1.1

Target Cfg:
Target Idx: 0
State: ISCSI_TARGET_LOGIN
Prev State: ISCSI_TARGET_DISABLED
Target Error: ISCSI_TARGET_NO_ERROR
IQN: iqn.1992-08.com.netapp:sn.101202278
IP Addr: 10.29.17.52
Port: 3260
Boot Lun: 2
Ping Stats: Failed - Connection refused
adapter 1/4/1 (mcp):3#
 Cisco VIC Initialization – Troubleshooting

• You can also test connectivity between the Adapter and Target.
• Blade must be booted and entered into the BIOS Menu (F2)
• Ensure iSCSI Target IP is correct from “iscsi_get_config” output

adapter 1/4/1 (mcp):4# iscsi_ping 172.25.183.49

id name tgt address port tcp ping status

--- -------------- --- --------------- ----- ---------------------
13 vnic_1 0 10.29.177.52 3260 Success (11.900ms)
Thank you.