IBM System Networking Easy Connect Mode
Download as PPT, PDF•1 like•12,516 views
IBM Easy-Connect provides simplified networking capabilities for IBM PureFlex systems. It operates in either transparent mode or multi-chassis mode. Transparent mode makes the system appear as a "dumb" switch to the edge switch, eliminating the need for spanning tree protocol. Multi-chassis mode allows traffic consolidation across chassis. Easy-Connect supports both intra-chassis and inter-chassis switching for high performance without requiring complex network integration or configuration. Customer examples show how it can provide simple, redundant, or dedicated uplink connectivity for various workloads.
IBM System Networking Easy Connect Mode
- 1. IBM Easy-Connect System Networking - Transparent Mode - Multi-chassis Mode (PureFlex ONLY) - Customer Examples EN/CN4093 and Virtual Fabric Switch Module Design and configuration by Scott Irwin & Igor Marty © 2013 IBM Corporation
- 2. What is Easy-Connect • Clever pass-through = Simple I/O module connecting servers, storage and the Core network • Aggregation = Server ports aggregated together towards Core Network • No network integration = Seen as one big pipe with server traffic coming in and out. • No Spanning-Tree = No loops, no network integration • Similar to Cisco FEX – with one important exception, server to server traffic stays local to chassis. • Specific Networking features (Virtual Fabric Mode, VMready etc…) can still be activated on selected ports when/if needed. 2 © 2013 IBM Corporation
- 3. Transparent Mode © 2013 IBM Corporation
- 4. Easy-Connect - Transparent Mode Easy-Connect Transparent Mode • Looks like a “dumb” device to Nexus 5K/Edge Switch • No Spanning Tree Protocol (STP) – eliminates Network admin loop concerns • Provides traffic consolidation in the chassis to minimize ToR port utilization • Provides intra-chassis switching, even in Transparent Mode Client Benefits • Allows client to manage networking from the Edge/Nexus management system • Allows for intra-chassis switching for optimal performance (i.e. vMotion) • Allows for pay as you grow expandability in the chassis, both to the network and the compute nodes. Networking Infrastructure • VLAN’s are transparent when in “Transparent Mode” between the NIC and Edge Switch • Allows for pNIC or Switch Independent vNIC to be utilized • Note: Power Nodes supported in pNIC mode only • Allows for growth into “Virtual Fabric Mode” using multiple groups and uplinks • Allows for FCoE if ONLY using a single Uplink (FCoE currently not supported over PortChannel) How to set the I/O Switch into Transparent Mode • Set the uplinks as one logical unit via Static Port-Channel (note: VFSM support LACP when using vNIC Groups) • Turn on vNIC and configure all internal “ports” and external “port-channel” into vNIC “Group 1” • Turn on Spanning Tree BPDU Guard and Edge on the Edge/Nexus Switch for additional protection Note: If connecting to a Nexus 2k both BPDU Guard and Edge are already enabled by default and cannot be disabled. • Works with PureFlex Chassis EN/CN4093 and Virtual Fabric Switch Modules • If using FCoE over a single link simply enable CEE Globally (cee enable) © 2013 IBM Corporation
- 5. Easy-Connect Configuration to convert from Switch Mode to Transparent Mode Configuration for the IBM VFSM and EN/CN4093 •ssh to each VFSM,EN/CN4093 •boot the VFSM,EN/CN4093 to use isCLI (Cisco-Like CLI) for a more familiar look • /boot/mode iscli • /boot/reset • /boot/prompt enable enable if using FSM on PureFlex •Set the following commands to convert to “Transparent Mode” • ssh to each VFSM • enter into enable mode (e.g. enable) • enter into configuration terminal mode (e.g. config t) • type the following; (start with factory default configuration) vLAG Networking Infrastructure Switch Switch vPC “boot conf factory” ===========Begin Script =========== spanning-tree mode disable LACP interface port ext1-ext10 lacp key 1001 VFSM 1 VFSM 2 lacp mode active vnic enable vnic vnicgroup 1 vlan 4091 port int1-int14 adminkey 1001 Node enable failover exit Chassis write memory ===========End Script =========== © 2013 IBM Corporation
- 6. Easy-Connect Configuration to convert from Switch Mode to Transparent Mode w/FCoE Configuration for the IBM VFSM and EN/CN4093 •ssh to each VFSM •boot the VFSM to use isCLI (Cisco-Like CLI) for a more familiar look • /boot/mode iscli • /boot/reset •Set the following commands to convert the VFSM to “Transparent Mode” • ssh to each VFSM • enter into enable mode (e.g. enable) • enter into configuration terminal mode (e.g. config t) • type the following; (start with factory default configuration) “boot conf factory” vLAG Networking Infrastructure Switch Switch vPC ===========Begin Script =========== spanning-tree mode disable interface port ext1-ext4 LACP lacp key 1001 lacp mode active VFSM 1 VFSM 2 vnic enable vnic vnicgroup 1 vlan 4091 port int1-int14 adminkey 1001 enable Node failover exit cee enable Chassis write memory ===========End Script =========== © 2013 IBM Corporation
- 7. Multi-Chassis Mode © 2013 IBM Corporation
- 8. Easy-Connect - Multi-chassis Mode Easy-Connect Multi-chassis Mode • Allows G8264 and EN/CN4093 look like a “dumb” device to Edge/Nexus Switches • No Spanning Tree Protocol (STP) Required – eliminates Network admin loop concerns • Provides traffic consolidation in the chassis to minimize ToR port utilization • Provides intra chassis switching, even in Easy-Connect Mode • Works with EN/CN4093 ONLY as VFSM does NOT support vLAG Client Benefits • Allows client to manage networking from the Edge/Nexus management system • Allows for intra/external chassis switching for optimal performance (i.e. vMotion) Networking Infrastructure • Allows for pay as you grow expandability in the chassis, both to the network and the compute nodes. • VLAN’s are transparent on both G8264 and EN/CN4093 when in “Easy-Connect Mode” between the NIC and Upstream AGG/Core Switch • Allows for pNIC or Switch Independent vNIC to be utilized • Allows for growth into Virtual Fabric Mode • Note: Power Nodes supported in pNIC mode only Use cases for Multi-Chassis Mode • Works with environments that require Active/Active NIC Teaming and or Bonding using 802.3ad and or Static EtherChannels on the NODE Ports • Works with both Intel and Power Nodes • ALL local Layer 2 Traffic, even if using different I/O Bays, will remain within the same enclosure with the use of the ISL Peer-Links between the EN/CN4093’s • Each Chassis appears to upstream Network as a single entity © 2013 IBM Corporation
- 9. Easy-Connect What you end up with in Multi-chassis Mode with vLAG and vPC How-To: EN/CN4093 Easy-Connect Multi-chassis Mode Disable Global Spanning Tree Set all VFSM ports, both EXT Ports facing the Network (Top of Rack) and INT Ports, into tagpvid-ingress using VLAN 4091 as Switch Switch vLAG or vPC the PVID Enable TAGGING on the two EXT Ports being used as the vLAG Peer Link and set 4090 as the PVID adding VLAN 4091 as a LACP TAGGED/TRUNKED member (VLAN 4090 is used as the vLAG ISL VLAN) EN/CN4093 EN/CN4093 vLAG Configure all required LACP PortChannels (Peer Link, EXT, and LACP INT Ports) Configure a dummy IP Address used by the MGT EXT Port Node vLAG Health Check (e.g. 1.1.1.1 (SW1) and 1.1.1.2 (SW2)) X or P Configure vLAG ISL, Health Check peer-ip, and all associated Chassis vLAG pairs NOTE: Turning on Spanning Tree BPDU Guard and Edge on the EDGE G8264 / Nexus 5K Switches can provide further protection © 2013 IBM Corporation
- 10. Easy-Connect If using G8264’s in ToR also in Multi-Chassis Transparent Mode How-To: G8264 Easy-Connect Multi-chassis Mode vPC AGG/Core AGG/Core Disable Global Spanning Tree (not required (default is using Rapid PVST+ (i.e. PVRST))) LACP Set all G8264 ports, both uplink facing network and enclosure vLAG G8264-1 G8264-2 facing EN/CN4093’s, into tagpvid-ingress using VLAN 4091 as the PVID Enable TAGGING on the two ISL Ports being used as the vLAG LACP LACP Peer-Link and set 4090 as the PVID adding VLAN 4091 as a TAGGED/TRUNKED member (VLAN 4090 is used as the vLAG vLAG EN/CN4093 EN/CN4093 ISL VLAN) LACP Configure all required LACP PortChannels (Peer Link, Uplinks, and EN/CN4093 facing ports) Node Configure a dummy IP Address used by the MGT Port vLAG X or P Health Check (e.g. 1.1.1.1 (SW1) and 1.1.1.2 (SW2)) NOTE: if using customers MGT Network set IP’s appropriately Chassis Configure vLAG ISL, Health Check peer-ip, and all associated vLAG pairs NOTE: Turning on Spanning Tree BPDU Guard and Edge on the EDGE G8264 / Nexus 5K Switches can provide further protection © 2013 IBM Corporation
- 11. Customer requirements to implement Easy-Connect 1. If using FSM in same chassis – since the FSM is NOT capable of TAGGING (Cisco Trunking) customer MUST enable the Top-of-Rack Port “Native VLAN ID” with the VLAN that the FSM needs to be configured on. 2. ALL other OS’s (i.e. VMware, VIO, and even Bare Metal OS’s) within the PureFlex Chassis are required to TAG/Trunk VLAN’s. 3. If using multiple vNIC Groups for traffic either traffic separation and or if using Virtual Fabric Mode - each vNIC Group requires it’s own uplink/PortChannel or DUM (Dedicated Uplink Mode) 4. If Multi-Tenant security is a concern within the same PureFlex Chassis Easy-Connect may/may not be a good option. Why, because each vNIC Group is a single broadcast domain. 5. If IGMP Multicast Snooping is required Easy-Connect may not be a best option. Since Easy-Connect is VLAN Agnostic IGMP Snooping is not supported in any Q-n-Q Architecture. However, Multicast will still pass through the Easy Connect enabled Switches. 11 © 2013 IBM Corporation
- 12. Easy Connect Customer Examples © 2013 IBM Corporation
- 13. Easy-Connect EN/CN4093 Transparent Mode – Telecom Customer (Very Simple) Requirements included: •No Spanning Tree and or any other protocols seen by the network (Easy-Connect) •Connection into Nexus 2k (no vPC or vLAG requirement) •EN4093 is to be a total transparent device requiring no management by any group © 2013 IBM Corporation
- 14. Easy-Connect EN/CN4093 Transparent Mode – State Government Requirements included: •Using LoM in Virtual Fabric Mode so bandwidth can be adjusted on the fly for each vNIC as required •Dedicated uplink vPC PortChannel from each EN4093 for each vNIC Group for separation of traffic •EN4093’s in Transparent Mode with minimal hands-on requirement (vNIC BW ONLY) © 2013 IBM Corporation
- 15. Easy-Connect EN/CN4093 Storage Mode w/NIC Redundancy – Medical Center Requirements included: •Dedication and Separation of Fiber Channel and Ethernet from Each NODE and Enclosure •Total Hardware Redundancy including NIC and ASIC on every NODE (CN4054 Mezz) •Transparency on both Ethernet (Easy-Connect) and Fiber Channel (NPV) © 2013 IBM Corporation
- 16. Backup Slides © 2013 IBM Corporation
- 17. Four Scalable switches enable high speed connectivity – Ethernet (FCoE, iSCSI), Fibre Channel and InfiniBand Four high performance Scalable Switch Modules 3 2 4 1 17 17 © 2013 IBM Corporation
- 18. Robust connectivity: Switch, asic and adapter level redundancy Redundant pair asic level redundancy Redundant pair EN4093 (base) EN4093 (base) EN4093 (base) EN4093 (base) asic 1 asic 2 CN4054 EN4093 (Upgrade 1) EN4093 (Upgrade 1) EN4093 (Upgrade 1) EN4093 (Upgrade 1) asic 1 asic 2 CN4054 1 3 2 4 Adapter level redundancy © 2013 IBM Corporation
Editor's Notes
- #2: March 20, 2013 BLADE Network Technologies, An IBM Company
- #4: March 20, 2013 BLADE Network Technologies, An IBM Company
- #8: March 20, 2013 BLADE Network Technologies, An IBM Company
- #10: EN/CN4093 configuration to support Multi-Chassis Mode (OS Version 7.5.1) ssh to each EN/CN4093 boot to isCLI (Cisco-Like CLI) /boot/mode iscli /boot/reset Set the following commands to convert the EN/CN4093 into “Multi-chassis Mode” ssh to each EN/CN4093 enter into enable mode (e.g. enable) enter into configuration terminal mode (e.g. config t) type the following; (note: the IP Address of the health check “IP 127” Address will be different on both switch, everything else should remain the same) ===========Begin Script =========== spanning-tree mode disable interface port ext9,ext10 ISL vLAG Peer-Link Ports pvid 4090 tagging lacp key 1001 lacp mode active vlan 4090 enable name Peer-Link vlan 4091 Note: if using Power-Nodes also and want in separate domain (including uplinks (a MUST)) use VLAN 4092 enable name Intel-Nodes member int1-int14,ext1-ext4,ext9,ext10 interface port inta1-inta14,ext1-ext4 tagpvid-ingress interface port ext1-ext4 uplink ports to AGG/Core lacp key 4091 use SAME key on both VFSM INTEL Uplinks (4091) lacp mode active interface port inat1 INTa1 on both Switches will be in same PortChannel using vLAG (lacp key MUST match) lacp key 101 lacp mode active interface port inat2 lacp key 102 lacp mode active interface ip 127 IP 127 is dedicated to the MGT Port used for vLAG health check ip address 1.1.1.1 enable vlag ena vlag isl peer-ip 1.1.1.2 other switch will use 1.1.1.1 vlag isl vlan 4090 vlag isl adminkey 1001 vlag tier-id 10 each pair of switches connecting to each other should be a different Tier-ID vlag adminkey 4091 enable vlag adminkey 101 enable vlag adminkey 102 enable repeat for each Server using 802.3ad / LACP write memory ===========End Script ===========
- #11: G8264 configuration to support Multi-Chassis Mode (OS Version 7.5.1) ssh to each G8264 Note: G8264 uses isCLI by default Set the following commands to convert the G8264 into “Multi-chassis Mode” ssh to each G8264 enter into enable mode (e.g. enable) enter into configuration terminal mode (e.g. config t) type the following; (note: the IP Address of the health check “IP 128” Address will be different on both switch, everything else should remain the same) ===========Begin Script =========== spanning-tree mode disable Optional interface port 1,5 2x 40Gb ISL (e.g. between G8264 ’s) tagging pvid 4090 lacp key 4090 lacp mode active vlan 4090 enable name Peer-Link vlan 4091 enable name “Transparent-Ports” interface port 17-64 Uplinks and EN/CN4093 facing Ports ONLY tagpvid-ingress interface port 17,18 uplink ports to AGG/Core lacp key 1001 lacp mode active interface port 19,20 Ports facing first PureFlex enclosure lacp key 1920 lacp mode active interface port 21,22 lacp key 2122 Ports facing second PureFlex enclosure lacp mode active vlag ena vlag isl adminkey 4090 vlag tier-id 1 vlag adminkey 1001 ena Uplink PortChannel to AGG/Core vlag adminkey 1920 ena vlag adminkey 2122 ena repeat for each Port-Channel to each EN/CN4093 write memory ===========End Script ===========
- #13: March 20, 2013 BLADE Network Technologies, An IBM Company
- #17: March 20, 2013 BLADE Network Technologies, An IBM Company