Application Centric Infrastructure (ACI), the policy driven data centre

© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Application Centric Infrastructure (ACI), the
Policy Driven Data Center
Mike Herbert - Principal Engineer, Cisco
Dave Cole, Consulting Systems Engineer, Cisco
Sean Comrie, Technical Solutions Architect, Cisco

House Keeping Notes
• Thank you for attending Cisco Connect Toronto 2015, here are a few
housekeeping notes to ensure we all enjoy the session today.
•  Please ensure your cellphones / Laptops are set on silent to ensure
no one is disturbed during the session
•  A power bar is available under each desk in case you need to charge
your laptop

•  Cisco dCloud is a self-service platform that can be accessed via a browser, a high-speed Internet
connection, and a cisco.com account
•  Customers will have direct access to a subset of dCloud demos and labs
•  Restricted content must be brokered by an authorized user (Cisco or Partner) and then shared
with the customers (cisco.com user).
•  Go to dcloud.cisco.com, select the location closest to you, and log in with your cisco.com
credentials
•  Review the getting started videos and try Cisco dCloud today: https://dcloud-cms.cisco.com/help
dCloud
Customers now get full dCloud experience!

Evolution of the Data Center

IT Challenges and Opportunities
IT’s ability to deliver innovation
IT’s budget
Need:ITSimplification
Better alignment of IT with rapidly changing business needs requires dynamic and
automated policy-based control of DC and Cloud infrastructure.

Switch
ASICs
X86
CPUs
2013
2014/15
2015+

28nm 16nm65nmCisco
40nm 28nm65nmOthers
14nm22nmIntel
Capacity and Cost – Impact of Mega Scale DC’s

What’s the DNA of your applications ?
7FUTURE< 2000 2003 2006 2008 2010 2012 2013 20142011
?

The on-going “IT pain”
•  High cost, heterogeneous systems
•  Redundant functionality
•  Lack of agility to innovate
•  Slow time to market
•  Rising maintenance costs
•  Rising regulatory and compliance costs,
multiplied by:
•  Heterogeneous systems
•  Geographic expansion / local laws
•  Falling IT Budgets
8

What
Happened
?

•  Separation of IT areas / buying-
centers / silos preventing IT to
move at the speed demanded by
the business
•  Focus changed from
Consolidation to Automation and
now to Consumption
•  Business owners and Apps
Developers started to go straight
to public cloud to meet agility and
demand. Security and Data
Sovereignty arise.
•  Operations become further
relevant. Shift from “what it
does / how it works” to “how to
use / how to consume it”.
DevOps

App Development via DevOps is Changing the Behavior
DevOps

DevOps: Where does each “tool” fit ?
CONTINUOUS
INTEGRATION
CONFIGURATION
MANAGEMENT ORCHESTRATION &
MANAGEMENT (O&M)
Infrastructure as Code

… so, let’s talk about the elephant in the room…
Current networks are not inflexible nor expensive. Operational process around
them makes them just like that. ACI simplifies IT and becomes an enabler.
“Elephants can dance”.

Abstraction, the real objective of “SDN”
How to Avoid Death by Micromanagement
You can not mask
complexity with
complexity
Less Networks, Not
More

Control & Audit Connectivity
(Security – Firewall, ACL, …)
IP Address, VLAN, VRF
Enable Connectivity
(The Network)
Application Requirements
IP Addressing
Application Requirements
Application Specific Connectivity
Dynamic provisioning of
connectivity explicitly defined for
the application
Application RequirementsApplication Requirements
Redirect and Load Balance
Connectivity
IP Address, VLAN, VRF
ACI directly maps the application
connectivity requirements onto the
network and services fabric
Why Networks are Complex
Overloaded Network Constructs

Why Network Provisioning is Slow
Application Language Barriers
Developers
Application
Tiers
Provider /
Consumer
Relationships
Infrastructure Teams
VLANs
Subnets
Protocols
Ports
Developer and infrastructure teams must translate between disparate languages.

What is ACI

“Users”“Files”
ACI Fabric
Logical Endpoint Groups by
Role
Heterogeneous clients, servers,
external clouds; fabric controls
communication
Every device is one hop away, microsecond
latency, no power or port availability
constraints, ease of scaling
Flexible Insertion
ACI Controller manages all
participating devices, change control
and audit capabilities
Unified Management and Visibility
Fabric Port Services
Hardware filtering and bridging; default
gateway; seamless service insertion,
“service farm” aggregation
Flat Hardware Accelerated
Network
Full abstraction, de-coupled from
VLANs and Dynamic Routing, low
latency, built-in QoS
Application Centric Infrastructure Fabric

“Users”“Files”
ACI Fabric
Define Endpoint Groups
Any endpoints anywhere within the
fabric, virtual or physical
Enforce Ingress Fabric Rules
Hardware rules on each port, security in
depth, embedded QoS
Single Point of Orchestration
Different administrative groups
use same interface, high level
of object sharing
Application Policy Infrastructure
Controller (APIC)
Create Contracts Between Endpoint
Groups
Port-level rules: drop, prioritize, push to
service chain; reusable templates
Service Graph
Single Pass Services
Security administrator defines
generic templates in APIC,
availed to contract creation
All TCP/UDP: Accept, Redirect
UDP/16384-32767: Prioritize
All Other: Drop
Policy Contract “Users → Files”
ACI is a Fabric which provides a new communication
abstraction model

ACI How to build it and how it works

ACI – Components
A Policy Based IP Network
PayloadIPVXLANVTEP
AVS

VTEP
APIC - Policy Controller &
Distributed Management
Information Tree (DMIT)
Physical and Virtual L4-7
Service Nodes
Physical and Virtual VTEP’s
(Policy & Forwarding Edge
Nodes)
Proxy (Directory)
Services
Physical and Virtual Endpoints
(Servers) & VMM (Hypervisor vSwitch)
VTEP
IP Network & Integrated
VXLAN
WAN/DCI
Services
VTEP
VTEP
AVS

VTEP

ACI - Components
Logical network provisioning of stateless hardware
22
Outside
(Tenant
VRF)
App DBWeb
QoS
Filter
QoS
Service
QoS
Filter
ACI Fabric
Application Policy
Infrastructure Controller
Integrated GBP VXLAN Overlay
APIC

Policy Instantiation: Each device
dynamically instantiates the required
changes based on the policies
Application Policy Model: Defines
the application requirements
(Application Network Profile)
VM VM
ACI – 21st Century Distributed Systems in Action
23
App TierWeb Tier DB Tier
Storage Storage
Application
Client
VM
10.2.4.7
VM
10.9.3.37
VM
10.32.3.7
VMVM
•  All forwarding in the fabric is managed via the Application Network Profile
•  IP addresses are fully portable anywhere within the fabric
•  Security & Forwarding are fully decoupled from any physical or virtual network attributes
•  Devices autonomously update the state of the network based on configured policy requirements
APIC

Application Policy Infrastructure Controller
Centralized Automation and Fabric Management
Layer 4..7 System
Management
Storage
Management
Orchestration
Management
Storage SME Server SME Network SME
Security SME App. SME OS SME
Open RESTful API
Policy-Based
Provisioning
APIC
•  Unified point of Data Center network
automation and management:
•  Data Model based declarative
provisioning
•  Application, Topology Monitoring, &
Troubleshooting
•  3rd party Integration (L4-L7 Services,
Storage, Compute, WAN, …)
•  Image Management (Spine / Leaf)
•  Fabric Inventory
•  Single APIC cluster supports one million+
end points, 200,000+ ports, 64,000+ tenants
•  Centralized Access to ‘all’ Fabric information
- GUI, CLI and RESTful API’s
•  Extensible to compute and storage
management

APIC Communicating to the Network
25
•  Infra VRF – Used for inband APIC to switch node communication, non routable outside the fabric
currently (Multi-Fabric, Remote Leaf will both allow extension of the Infra VRF - Future)
•  Inband Management Network – ‘tenant’ VRF created for inband access to switch nodes
•  OOB Management Network – APIC and switch node dedicated mgmt ports
OOB Management Network
APIC will have:
1.  2 attached to fabric for data
2.  2 for mgmt (OOB)
3.  1 console ethernet port (can be only used
for direct laptop hookup)
4.  CIMC/IPMI ports
Inband Management VRF
Infrastructure VRF
Switch nodes will have:
1.  Inband access to Infra & Mgmt VRF
2.  Mgmt Port (OOB)
3.  Console port
APIC APIC APIC

APIC first time Setup
•  APIC one time setup is via UCS console access
•  Cluster configuration
•  Fabric Name
•  Number of controllers [1..9]
•  Controller ID [1..9]
•  TEP Address pool [10.0.0.1/16]
•  Infra VLAN ID [4093]
•  Out-of-band management configuration
•  Management IP address [192.168.10.1/254]
•  Default gateway [192.168.10.254]
•  Admin user configuration
•  Enable strong passwords (Y/N)
•  Password
After first time setup, APIC UI is
accessible via URL
https://<APIC-mgmt-IP>
APIC

Login Screen

Fabric Initialization & Maintenance
28
•  ACI Fabric supports discovery, boot, inventory and systems maintenance processes via the
APIC
•  Fabric Discovery and Addressing
•  Image Management
•  Topology validation through wiring diagram and systems checks
Loopback and VTEP IP Addresses
allocated from “Infra VRF” via DHCP from
APIC
APIC Cluster
Topology Discovery via LLDP
using ACI specific TLV’s (ACI
OUI)
APICAPICAPIC

29
APIC bootstrap configuration
1)  APIC Cluster Configuration
2)  Fabric Name
3)  TEP Address space (Infra-VRF)
4)  …
Leaf switch discovers attached
APIC via LLDP, requests TEP
address and boot file via DHCP
2
1
Spine switch discovers attached
Leaf via LLDP, requests TEP
address and boot file via DHCP
3
All nodes in the same APIC cluster should
contain same bootstrap information if they are
intended to form a cluster
4
Fabric can be discovered and initialized
from multiple sources concurrently
5
6 Fabric will self assemble starting from
multiple APIC sources
APIC Cluster
7
APIC Cluster will form when members
discovery each other via Appliance
Vector (AV)
APIC APIC APIC

Node Identity Policy
•  Assigns ID/Name to switches based on serial number
•  Controls which switches can join the fabric
•  Allows zero touch provisioning of switches
POST: https://192.168.10.1/api/node/mo/uni/controller.xml
<fabricNodeIdentPol>
<fabricNodeIdentP serial=”TNAX234ZA"
name="leaf1" nodeId=”101"/>
<fabricNodeIdentP serial=” JNAX234ZZ"
name="leaf2" nodeId=”102"/>
<fabricNodeIdentP serial=“KLAX234ZZ”
name="spine1" nodeId=”103"/>
</fabricNodeIdentPol>

31
•  ACI Fabric leverages the same Global Catalogue methodology as UCS, the supported HW/SW matrix, image
versioning, …
•  APIC and switch node image management controlled via APIC policies
•  Policies control which images should be on which groupings of devices, when the images should be
upgraded/downgraded
•  Also control the upgrade process, automatic, manual step by step, …
“All-APICs”
APIC Cluster
“All-Leafs”
“All-Spines”
APIC APIC APIC

Policy Upgrade of Fabric
•  Catalogue Based Software Management

Policy Upgrade of Fabric
•  Automated Software Management of all components

APIC - Unified Management and Visibility
•  APIC creates a single point of orchestration for entire network
•  Controls underlying fabric topology, service consumer instances, and their policies
•  Application, Network, and Security administrators use a single entity to configure their
devices
•  High degree of element reuse and templating between different roles and workflows
•  Embedded Role Based Access Control (RBAC) and change management
•  Audit and event correlation capabilities
•  Trace specific network events to prior changes, no more management fragmentation/
unknowns
•  Flexible programmability for any managed device or management system
•  XML/JSON for Northbound API
•  Python scripting for custom device management

ACI Routed Access with Host Based
Granularity

ACI Fabric – Integrated Overlay
Decoupled Identity, Location & Policy
•  ACI Fabric decouples the tenant end-point address, it’s “identifier”, from the location of that
end-point which is defined by it’s “locator” or VTEP address
•  Forwarding within the Fabric is between VTEPs (ACI VXLAN tunnel endpoints) and leverages
an extender VXLAN header format referred to as the ACI VXLAN policy header
•  The mapping of the internal tenant MAC or IP address to location is performed by the VTEP
using a distributed mapping database
PayloadIPVXLANVTEP
APIC
VTEP VTEP VTEP VTEP VTEP VTEP

ACI leverages VXLAN
IETF Draft for Group Based Policy

Location Independent Forwarding
Layer 2 and Layer 3
•  Forward based on destination IP Address for intra and inter subnet (Default Mode)
•  Bridge semantics are preserved for intra subnet traffic (no TTL decrement, no MAC
header rewrite, etc.)
•  Non-IP packets will be forwarded using MAC address. Fabric will learn MAC’s for non-IP
packets, IP address learning for all other packets
•  Route if MAC is router-mac, otherwise bridge (standard L2/L3 behaviour)
IP Forwarding:
Forwarded using DIPi
address, HW learning of IP
address
10.1.3.11 10.6.3.210.1.3.35 10.6.3.17
MAC Forwarding:
Forwarded using DMAC
address, HW learning of
MAC address

10.1.1.10 10.1.3.11 10.6.3.2
Distributed Default Gateway
•  ACI Fabric supports full layer 2 and layer 3 forwarding semantics, no changes required to applications or end point IP
stacks
•  ACI Fabric provides optimal forwarding for layer 2 and layer 3
•  Fabric provides a pervasive SVI which allows for a distributed default gateway
•  Layer 2 and layer 3 traffic is directly forwarded to destination end point
•  IP ARP/GARP packets are forwarded directly to target end point address contained within ARP/GARP header
(elimination of flooding)
10.1.3.35 10.1.1.10
10.1.3.11
10.6.3.2
Directed ARP Forwarding
10.1.3.35
Location Independent Forwarding
Layer 2 and Layer 3

10.1.3.11 10.6.3.2
Pervasive SVI
•  Default Gateway can reside internal or external to the Fabric
•  Pervasive SVI provides a distributed default gateway (anycast gateway)
•  Subnet default gateway addresses are programmed in all Leaves with end points present for the specific
Tenant IP subnet
•  Layer 2 and layer 3 traffic is directly forwarded to destination end point
•  External Gateway is used when Fabric is configured to provide layer 2 transport only for a specific Tenant
10.1.3.35 10.1.1.10
10.1.3.11
10.6.3.2
External Default Gateway
10.1.3.35
Pervasive SVI’s
10.6.3.2
10.6.3.110.1.3.1

Host Routing - Inside
Inline Hardware Mapping DB - 1,000,000+ hosts
10.1.3.11 fe80::462a:60ff:fef7:8e5e10.1.3.35 fe80::62c5:47ff:fe0a:5b1a
•  The Forwarding Table on the Leaf Switch is divided between local (directly attached) and
global entries
•  The Leaf global table is a cached portion of the full global table
•  If an endpoint is not found in the local cache the packet is forwarded to the ‘default’
forwarding table in the spine switches (1,000,000+ entries in the spine forwarding table)
Local Station Table
contains addresses of
‘all’ hosts attached
directly to the Leaf
10.1.3.11
10.1.3.35
Port 9
Leaf 3
Proxy A*
Global Station Table
contains a local cache
of the fabric endpoints
10.1.3.35 Leaf 3
10.1.3.11 Leaf 1
Leaf 4
Leaf 6
fe80::8e5e
fe80::5b1a
Proxy Station Table contains
addresses of ‘all’ hosts attached
to the fabric
Proxy Proxy Proxy Proxy

Proxy Scaling
Scaled based on number of Fabric NFE’s per chassis
Spine Proxy Total Host
Entries in the
Mapping DB
Network
Forwarding
Engines Per Fabric
9336 200K* 2 x NFE
9504 (6 fabrics) 300K 1
9508 (6 fabrics) 600K 2
9516 (6 fabrics) 1M+ 4
NFE
Fabric Module for Nexus 9504
NFE NFE
NFE NFE
NFE NFE
*9336 maintains a single copy of each host entry in the HW proxy DB, 950x maintains
redundant copies sharded across Fabric NFE’s

Proxy Database Adjacencies (APIC GUI)

Proxy Database (Oracle)
Spine-1# show coop internal info global Spine-1# show coop internal event-history oracle-adj <IP>
•  You still have full access to all forwarding, adjacency, ..., information via
CLI and debug commands when you want them

Endpoint Repository (APIC GUI)

Multicast repository (on APIC GUI)

ACI Endpoint Tracker Application
•  Tracks all attachment, detachment,
movement of Endpoints in ACI fabric
•  Stores activity in open source MySQL
Database, allowing query capabilities
•  Provides foundation for visualization and
query tools
•  Some questions that could be solved:
•  What are all the Endpoints on network?
•  Where is a specific Endpoint?
•  What was connected last Thursday
between 3:30am and 4:00am?
•  What is the history of a given Endpoint?

Using Atomic Counters
•  Detect fabric misrouting, debug & isolate application connectivity issues
•  Per-application, per-EP, per-EPG real-time, comprehensive traffic counters
•  Example:
•  Configure atomic counters on all leafs to count packets EP1->EP2
•  Any counts NOT on Leaf03 or Leaf06 highlight misrouted packets
•  Drill-down to Leaf03, Leaf01 and check routing, forwarding entries
•  Configure via policy in appropriate context
10.1.1.10 10.1.3.12 10.6.3.2 10.1.3.35
EP1
Leaf01 Leaf06
EP2

Heatmap
49

Fabric Traceroute
•  Traditional traceroute does not cover multipath technologies; can’t see devices in overlay network
•  ACI Traceroute
•  Accurately represents physical & virtual environments
•  Complete path visibility
•  Configured via policy in appropriate context
•  Fabric
•  Infra
•  Tenants
10.1.1.10 10.1.3.12 10.6.3.2 10.1.3.35

SPAN
•  How to span traffic between EPGs?
•  Could manually config on each leaf node that has a port in target EPG
•  Manually reconfig with every move/add/change
•  APIC automatically pushes span configs to every leaf which needs it
•  Configure via policy in appropriate context
10.1.1.10 10.1.3.12 10.6.3.2 10.1.3.35
EPG_A

Troubleshooting Wizard
•  https://www.youtube.com/watch?v=Gm9vvHj3LGM

ACI Improved vPC

vPC Behaviour – Standalone & ACI Differences
vSwitch
vSwitch

No vPC Peer
Link
Required
Standard vPC ACI Based vPC
Orphan Port
‘No’ Orphan
Ports (Single
Homed Servers
‘not’ orphans)
Implicit Uplink
Tracking
Hardware Based
Recovery for server link
failures (no STP no vPC
state updates)

FEX Topology Support Roadmap
6.1(2)I2(3) Future Future6.1(2)I2(3)
Straight Through
(Single Homed)
vPC (Dual Homed) EvPC
Active/Standby
Teaming
Nexus 9300
Standalone
Nexus 9300
ACI Leaf
11.1(x) - 1HCY1511.0(1d) - Shipping Future Future

Classical vPC
•  In classical vPC host addresses are scoped to a
VLAN
•  Traffic is recovered based on updating the VLAN
forwarding topology
•  On loss of all of the locally attached members of the
vPC MAC address table is updated to forward
frames for the vPC across the vPC Peer Link
N5K-1# sh mac-address-table vlan 101
VLAN MAC Address Type Age Port
---------+-----------------+-------+---------+-----
101 001b.0cdd.387f dynamic 0 Po30
101 0023.ac64.dda5 dynamic 30 Po201
Total MAC Addresses: 4
N5K-2# sh mac-address-table vlan 101
VLAN MAC Address Type Age Port
---------+-----------------+-------+---------+-----
101 001b.0cdd.387f dynamic 0 Po20
101 0023.ac64.dda5 dynamic 30 Po201
Total MAC Addresses: 4
MAC_C
MAC_A
N5K-1 N5K-2
1
3
2

vPC in ACI Fabric
•  ACI Leaves support virtual port channels (vPC)
interfaces similar to Nexus (802.3ad port
channels with links split across two devices)
•  Differences between ACI vPC and standard
vPC
•  No Peer Link is required
•  Peer communication happens via the
Fabric
•  Path recovery also happens via the Fabric
and not peer link
•  CFS (Cisco Fabric Services) is replaced by
IFS (ACI Fabric Services) which is based
on Zero Message Queue (ZMQ)
•  Forwarding selection (which peer will
forward a frame
•  Within the Fabric the vPC interfaces use an
anycast VTEP which is active on both vPC
peers
ACI Fabric Services (ZMQ)
Host or Switch
VTEP VTEP
vPC Anycast
VTEP
vPC Anycast
VTEP

vPC in ACI Fabric •  Traffic is both sourced and destined to the
anycast vPC VTEP address from remote
Leaves
•  A hardware hash in the spine will determine
which of the two peers forwards a specific flow
downstream to the attached device (flow
hashing between the peers via spine
•  In the event of a downlink failure on one of the
peers (all local member ports are down)
1.  A bounce entry is created for the end
points reachable via the port channel
pointing to the peers VTEP
2.  All MAC/IP to Leaf bindings for the specific
vPC are removed from the COOP
database and the spine proxy
•  On failure of a peer the remaining Leaf
converts all vPC ports to non-VPC local ports
Host or Switch
VTEP VTEP
vPC Anycast
VTEP
vPC Anycast
VTEP
Traffic within the Fabric is sent to
the vPC anycast address

ACI Networking and Policy Terms

Backbone
vPC
vPC
vPC
•  Layer 2 and Layer 3 interoperation between ACI Fabric and Existing Data Center builds
•  Layer 3 interconnect via standard routing interfaces,
OSPF, Static, iBGP (Supported)
MP-BGP, EIGRP, OSPF (1HCY15)
•  Layer 2 interconnect via standard STP or via VXLAN overlays
vSwitch
Hyper-‐V
AVS

Connecting the ACI Network
Layer 2 and Layer 3
Extend Layer 2 VLAN’s
where required
Interconnect at
Layer 3

Fabric Infrastructure
Understanding Networks and Groups APIC
Outside
(Tenant
VRF)
App DBWeb
QoS
Filter
QoS
Service
QoS
Filter
Location for Endpoints that are ‘Inside’ the
Fabric are found via the Proxy Mapping DB
(Host Level Granularity)
Location for Endpoints that are
‘Outside’ the Fabric are found via
redistributed routes sourced from
the externally peered routers
(Network Level Granularity)
‘Outside’ EPG associated
with external network
policies (OSPF, BGP, …
peering)
Forwarding Policy for ‘inside’ EPG’s defined by associated
Bridge Domain network policies

Fabric Infrastructure
Understanding Networks and Groups
EP EP
EPGEPG
Application
Profile
EP EP
Bridge Domain
EP EP
EPGEPG
Application
Profile
EP EP EP EP
EPGEPG
Application
Profile
EP EP
Bridge Domain
Tenant
Private
Network
Private
Network

A Tenant is a container for all
network, security,
troubleshooting and L4 – 7
service policies.
Pepsi-Tenant Coke-Tenant
Tenant
Tenant resources are isolated
from each other, allowing
management by different
administrators.

Private Network 1
Private Network 2
Private Network 1
Private Network 2
Private networks (also called
VRFs or contexts) are defined
within a tenant to allow
isolated and potentially
overlapping IP address space.
Private Networks

Private Network 1
Private Network 2
Private Network 1
Private Network 2
Within a private network, one
or more bridge domains must
be defined.
A bridge domain is a L2
forwarding construct within the
fabric, used to constrain
broadcast and multicast traffic.
Bridge Domain 1
Bridge Domain 2
Bridge Domain 3
Bridge Domain 4
Bridge Domain 1
Bridge Domain 2
Bridge Domain 3
Bridge Domain 4
Bridge Domain

EPG Definition
EP EP
EPGEPG
Application
Profile
EP EP
EPs are devices which attach to the network either virtually or physically, e.g:
•  Virtual Machine
•  Physical Server (running Bare Metal or Hypervisor)
•  External Layer 2 device
•  External Layer 3 device
•  VLAN
•  Subnet
•  Firewall
•  Load balancer
Virtual Port, Physical Ports, External L2 VLAN, External L3 subnet

Private Network 1
Private Network 2
Private Network 1
Private Network 2
EPGs exist within a single
bridge domain only – they do
not span bridge domains.
Bridge Domain 1
Bridge Domain 2
Bridge Domain 3
Bridge Domain 4
Bridge Domain 1
Bridge Domain 2
Bridge Domain 3
Bridge Domain 4
EPG
End Point Groups
EPG
EPGEPG
EPG
EPG
EPG
EPG EPG
EPG

Mapping the Configuration to the Packet
M/LB/SPFlags
Flags/
DRE
VNID == BD/VRFSource Class ID == EPG
•  ACI Fabric leverages an application centric policy
model
•  VXLAN Source Group is used as a tag/label to
identify the specific end point for each application
function (EPG)
•  Policy is enforced between an ingress or source
application tier (EPG) and an egress or destination
application tier (EPG)
•  Policy can be enforced at source or destination
Coke-Tenant
Private Network 1
Private Network 2
Bridge Domain 1
Bridge Domain 2
Bridge Domain 3
Bridge Domain 4
EPG
EPG
EPG EPG
EPG

ACI Integration and Connecting to existing
Networks

Connecting/Extending ACI via Layer 2
Layer 2
Layer 2
Layer 2
Extend L2 domain beyond ACI fabric - 2 options
1.  Manually assign a port to a VLAN which in turn mapped to an EPG. This extend EPG beyond ACI fabric
(EPG == VLAN)
2.  Create a L2 connection to outside network. Extend bridge domain beyond ACI fabric. Allow contract
between EPG inside ACI and EPG outside of ACI
Lets Look at
the Links

Bridge any VLAN/VXLAN to any VLAN/VXLAN
71
•  Forwarding is ‘not’ limited to nor constrained by the encapsulation type or
encapsulation specific ‘overlay’ network
•  VLAN’s are local to the leaf switch
802.1Q
VLAN 10
VXLAN
VNID = 5789
VXLAN
VNID = 11348
NVGRE
VSID = 7456
Any to Any
802.1Q
VLAN 50
Normalized
Encapsulation
Localized
Encapsulation
APIC

VXLAN
VNID = 5789
VXLAN
VNID = 11348
NVGRE
VSID = 7456
Any to Any
802.1Q
VLAN 50
Normalized
Encapsulation
Localized
Encapsulation
IP Fabric Using
VXLAN Tagging
PayloadIPVXLANVTEP
•  All traffic within the ACI Fabric is encapsulated with an extended VXLAN header
•  External VLAN, VXLAN, NVGRE tags are mapped at ingress to an internal VXLAN tag
•  Forwarding is not limited to, nor constrained within, the encapsulation type or
encapsulation ‘overlay’ network
•  External identifies are localized to the Leaf or Leaf port, allowing re-use and/or translation
if required
Payload
Payload
Payload
Payload
Payload
Eth
IP
VXLAN
Outer
IP
IPNVGRE
Outer
IP
IP802.1Q
Eth
IP
Eth
MAC
Normalization of Ingress
Encapsulation
Bridge any VLAN/VXLAN to any VLAN/VXLAN
72

An Example of
Interconnecting and Migrating
Logical Design
HSRP
Default GW
VLAN / Subnet
P P VM VM VM
P
VM
vPC
N7k
N5k
L3 HSRP
P
VM
vPC
N7k
N5k
L3 HSRP
N2k
P
VM
N7k
FEX
L3 HSRP
P
VM
Cat6500
L3 HSRP
Many Different Physical Designs

Extend the EPG
Option 1
VLAN 30
Layer 2
100.1.1.3 100.1.1.5
EPG
100.1.1.7100.1.1.99
•  VLAN’s are localized to the leaf nodes
•  The same subnet, bridge domain, EPG can be configured as a ‘different’ VLAN on each leaf
switch
•  In 1HCY15 VLAN’s will be port local
100.1.1.3
BD
Existing
App
VLAN 20

Extend the EPG
Option 1
Layer 2
VLAN 10
100.1.1.3 100.1.1.5
EPG
100.1.1.7100.1.1.99
•  Single Policy Group (one extended EPG)
•  Leverage vPC for interconnect (diagram shows a single port-channel which is an option)
•  BPDU should be enabled on the interconnect ports on the ‘vPC’ domain
100.1.1.3
VLAN 30
VLAN 20
BD
Existing
App
VLAN 10 VLAN 10 VLAN 10

Assign Port to an EPG
•  With VMM integration, port is assigned to EPG by
APIC dynamically.
•  In all other cases, such as connecting to switch,
router, bare metal, port need to be assigned to
EPG manually or use API
•  Use “Static Binding” under EPG to assign
port to EPG
•  The example assigns traffic received on port
eth1/32 with vlan tagging 100 to EPG VLAN
100

Assign Port to EPG
VLAN Tagging Mode
•  Tagged. Trunk mode
•  Untagged. Access mode. Port can only be in one
EPG
•  802.1P Tag. Native VLAN.
•  No Tagged and Untagged(for different port) config
for same EPG with current software
•  Assign port eth1/1 with VLAN 100 tagged mode
and port eth1/2 with VLAN 100 untagged mode to
EPG WEB is not supported
•  Use 802.1P Tag. Port eth1/1 vlan 100 tagged,
eth1/2 vlan 100 902.1P Tag
•  VLAN to EPG mapping is switch wide significant

C
Extend the Bridge Domain
Option 2
Layer 2
100.1.1.3 100.1.1.5100.1.1.7100.1.1.99
•  External EPG (policy between the L2 outside EPG and internal EPG)
•  Leverage vPC for interconnect (diagram shows a single port-channel which is an option)
•  BPDU should be enabled on the interconnect ports on the ‘vPC’ domain
•  L2 outside forces the same external VLAN << fewer operational errors
100.1.1.3
BD
Existing
App
EPG
Inside
EPG
Outside
VLAN 30
VLAN 10
VLAN 20

L2 Outside Connection Configuration Example
•  Step 1. Create L2 Outside
connection.
•  Associate with BD.
•  Specify VLAN ID to connect to
outside L2 network
•  External Bridge Domain is a way
to specify the VLAN pool for
outside connection.
•  It is NOT a Bridge Domain.

•  Step 2. Specify leaf node
and interface providing
L2 outside connection

•  Step 3. Create external EPG
under L2 outside connection
•  Step 4. Create contract
between external EPG and
internal EPG

Configure ACI Bridge Domain settings
•  Temporary Bridge Domain
specific settings while we are
using the HSRP gateways in
the existing network.
•  Select Forwarding to be
“Custom” which allow
•  Enable Flooding of L2
unknown unicast
•  Enble ARP flooding
•  Disable Unicast routing
Tenant “Red”
Context “Red”
Bridge Domain “10”
Subnet 10 EPG-10

Migrate Workloads
Existing Design
HSRP
Default GW
VLAN 10 / Subnet A
P P
VM VM VM
APIC
EPG “10”
P P
VM VM VM
APIC point of view, the policy model
VM’s will need to be connected to new Port
Group under APIC control (AVS or DVS).

Complete the Migration
Change BD settings back to normal for ACI mode
•  Change BD settings back to default.
•  No Flooding
•  Unicast Routing enabled.

Migrating Default Gateway to the ACI Fabric
Change GW MAC address. By default, All
fabric and all BD share same GW MAC
Enable Routing and ARP flooding

ACI Interaction with STP
BPDU
STP Root
Switch
Same L2 Outside
EPG
(e.g. VLAN 10)
•  No STP running within ACI fabric
•  BPDU frames are flooded between
ports configured to be members of the
same external L2 Outside (EPG)
•  No Explicit Configuration required
•  Hardware forwarding, no interaction
with CPU on leaf or spine switches
for standard BPDU frames
•  Protects CPU against any L2 flood
that is occurring externally
•  External switches break any potential
loop upon receiving the flooded BPDU
frame fabric
•  BPDU filter and BPDU guard can be
enabled with interface policy
APIC
BPDU
BPDU
BPDU

ACI Fabric Loopback Protection
STP Loop
Detection
LLDP Loop
Detection
•  Multiple Protection Mechanisms against
external loops
•  LLDP detects direct loopback cables
between any two switches in the same
fabric
•  Mis-Cabling Protocol (MCP) is a new link
level loopback packet that detects an
external L2 forwarding loop
•  MCP frame sent on all VLAN’s on all Ports
•  If any switch detects MCP packet arriving on
a port that originated from the same fabric the
port is err-disabled
•  External devices can leverage STP/
BPDU
•  MAC/IP move detection and learning
throttling and err-disable
APIC
BPDULLDP
MCP Loop
Detection
(supported with
11.1 release)
MCP

C
Managing Flooding Within the BD
Layer 2
100.1.1.3 100.1.1.5100.1.1.7100.1.1.99
•  In a classical network traffic is flooded with the Bridge Domain (within the VLAN)
•  You have more control in an ACI Fabric but need to understand what behaviour you want
100.1.1.3
BD
Multi
EPG
EPG
App 1
EPG
Outside
VLAN 30
VLAN 10
EPG
App 2
VLAN 20

Managing Flooding Within the Fabric
ARP Unicast
ARP Flooding Disabled
(Default)
•  Disable ARP Flooding – ARP/GARP is
forwarded as a unicast packet within the
fabric based on the host forwarding DB
•  On egress the ARP/GARP is forwarded as a
flooded frame (supports hosts reachable via
downstream L2 switches)
Firewall Configured as
the Default Gateway
ARP

ARP Flooding
ARP Flooding Enabled
•  Enabling ARP Flooding – ARP/GARP is
flooded within the BD
•  Commonly used when the default GW is
external to the Fabric
Firewall Configured as
the Default Gateway
ARP

Unknown Unicast Proxy Lookup
Unknown Unicast
Lookup via Proxy
•  Hosts (MAC, v4, v6) that are not known by a
specific ingress leaf switch are forwarded to
one of the proxies for lookup and inline
rewrite of VTEP address
•  If the host is not known by any leaf in the
fabric it will be dropped at the proxy (allows
honeypot for scanning attacks)
Unknown
Unicast
Proxy
HW Proxy
Lookup

Unknown Unicast Flooding
•  Hosts (MAC, v4, v6) that are not known by a
specific ingress leaf switch are flooded to all
ports within the bridge domain
•  Silent hosts can be installed as static entries
in the proxy (flooding not required for silent
hosts)
Unknown Unicast
Flooded
Unknown
Unicast
Unknown
Unicast
Flooded

Unknown Multicast – Mode 1 (Flood)
•  Unknown Multicast traffic is flooded locally to
all ports in the BD on the same leaf the
source server is attached to
•  Unknown Multicast traffic is flooded to all
ports in the BD on leaf nodes with a ‘multicast
router port’
Unknown Multicast
Flooded
Unknown
Multicast

Unknown Multicast – Mode 2 (OMF ‘or’ Optimized Flood)
•  Unknown Multicast traffic is only flooded to
‘multicast router ports’ in this mode
Unknown Multicast
Optimized Flooding
Unknown
Multicast

Scoping Broadcasts to a micro segment
100.1.1.3 100.1.1.5100.1.1.7100.1.1.99 100.1.1.3
EPG
B
EPG
A
EPG
C
100.1.1.72
Traffic Type 11.0(x) Behaviour 11.1(x) Behaviour
ARP Flood or Unicast Flood or Unicast
Unknown Unicast Flood or Leverage Proxy Lookup Flood or Leverage Proxy Lookup
Unknown IP Multicast Flood or OMF Flood or OMF
L2 MCAST, BCAST, Link Local Flood Flood within the BD, Flood within the EPG,
Disable Flooding within the BD/EPG

Multi Destination Flooding (Supported with 11.1(x) – Q2CY15)
•  Link Level Traffic is either
•  Contained within the EPG
•  Contained within the Bridge Domain
•  Dropped
•  Security Segmentation for Link Level Traffic
Link Level
BCAST
Manage
Flooding within
the BD
100.1.1.3
100.1.1.5
100.1.1.7100.1.1.99
100.1.1.72
100.1.1.52
EPG ‘A’
100.1.1.4
EPG ‘A’EPG ‘B’ EPG ‘B’
EPG ‘B’

Flooding scoped to the EPG
100.1.1.3 100.1.1.5100.1.1.7100.1.1.99 100.1.1.3
EPG
B
EPG
A
EPG
C
100.1.1.72
•  Link Local, BCAST & L2 Multicast traffic can be managed on a micro-segment basis
•  As an example:
•  EPG A, EPG B & EPG C - Link Level traffic is flooded ‘only’ to the endpoints within the
EPG

Extension and Connecting
It’s a Network with any VLAN Anywhere
Anycast Default Gateway
10.10.10.8 10.20.20.32
10.10.10.9
10.20.20.33
10.20.20.3110.10.10.6
Any IP - Anywhere

Application
Client
Subnet
10.20.20.0/24
Subnet
10.10.10.0/24
Subnet
10.30.30.0/24
Subnet
10.40.40.0/24
Subnet
10.50.50.0/24
External Networks
(Outside)
Redirect to Pre-
configured FW
Redirect to Pre-
configured FW
Critical Users
(Outside)
Middle Ware
Servers
Web
Servers
Oracle
DB Contract
Redirect to dynamically
configured FW
NFS ContractRedirect to
dynamically
configured FW
Default Users
(Outside)
NFS
Servers
Subnet
10.20.20.0/24
Subnet
10.10.10.0/24
Subnet
10.30.30.0/24
Permit TCP any
any
Redirect to Pre-
configured FW
Policy can be added gradually starting with what you have
today

Simple Policy During Migration - Any-to-Any
Configuration
Contracts
Provided
Filter Contracts
Provided
Contracts
consumed
Filter
EPG “VLAN 10” VLAN10 Default ALL ALL Default
EPG “VLAN 20” VLAN20 Default ALL ALL
ALLVLAN 10
VLAN 20
VLAN 30

I want to have a very open configuration with VLAN10 talking
to anything (Step 1)
•  Create “Contract” ALL
if it doesn’t exist yet
•  Use filter “common/
default”

I want to have a very open configuration with VLAN10 talking
to anything (Step 2)
•  EPG VLAN 10
provides and
consumes “ALL”

Dynamic Distributed ACL’s
Permit ACL is applied on
all ports between VLAN
10, 20 & 30
10.10.10.8 10.20.20.32
10.10.10.9
10.20.20.33
10.20.20.3110.10.10.6
All Subnets are allowed to communicate with this policy applied

Later if I want to put an ACL between VLAN 10 and 20
ALL
VLAN 10
VLAN 20
VLAN 30
Contracts Provided Filter Contracts Provided Contracts
consumed
Filter
EPG “VLAN 10” VLAN10 Default VLAN20 Port 80
EPG “VLAN 20” VLAN20 Default ALL ALL Default

Dynamic ACL’s
Dynamic ACL is applied
between all endpoints
only allowing port 80
10.10.10.8 10.20.20.32
10.10.10.9
10.20.20.33
10.20.20.3110.10.10.6
Traffic is controlled between VLAN 10 & 20 to HTTP (port 80)

ACI Routing

Backbone
vPC
vPC
vPC
vSwitch
Hyper-‐V
AVS

Connecting via Layer 3
Interconnect at
Layer 3
•  Layer 3 interconnect via standard routing
interfaces,
OSPF NSSA, Static, iBGP - 11.0(x) FCS
OSPF, eBGP, EIGRP & Transit Routing –
11.1(x) (1HCY15)
Border Leaf
•  Any leaf can be border leaf
•  No limit for number of border leaf in the
fabric
•  L3 interface & sub-interface
•  VRF-lite for multi-tenancy
•  SVI Interface for L2 and L3 outside connection
on same port

Connecting ACI via Layer 3 - Routing J
Steps to Enabling Routing
1.  Active Internal Fabric Route
Redistribution (MP-BGP)
2.  Configure Routing Peer and
Protocol to external WAN/Core
routers
3.  Define which internal networks
should be advertised to the outside
and via which routing peers
4.  Define the outside policy groups
(which external networks should be
able to communicate to which
internal hosts
Border Leaf
Router Peering

109© 2013-2014 Cisco and/or its affiliates. All rights reserved.
§  Fabric runs MP-BGP between
spines and leaves
§  Each L3 out is a separate L3
domain
§  Routes learned from L3 outs
are redistributed into BGP on
border leaves
§  OSPF domains are not joined
via the fabric. Leaf switches
are ASBRs
ACI fabric is a transit network, supported with 11.1
OSPF
Area 0
OSPF
Area 0
Different OSPF domains
ACI Fabric as transit
MP-BGP
OSPF ASBR OSPF ASBR

OSPF
Area 0
Border Leaf
§  Redistribution of
routes into MP-
BPG (per VRF)
§  Routes are
redistributed from
MP-BGP to leaf
only if VRF is
deployed on that
leaf.
Redistribution of routes into MP-BGP
BGP RRBGP RR
AS-400
EBGP
Border Leaf
Border LeafBorder Leaf Border Leaf
AS-200
OSPF
Area 10
IBGP
AS-200
MP-BGP
Peering
Protocol Peering
for VRF1
Protocol Peering
for VRF2
Routes
redistributed
into BGP at
border leaf
Per VRF
Routes
redistributed
from MP-BGP
to border leaf
for VRF 2.
VRF 1 routes
are not
redistributed
on this leaf

Manage the Fabric MP-BGP Configuration

MP-BGP in ACI Fabric
•  MP-BGP is not on by default. Assign BGP ASN and specify spine nodes as BGP RR
to turn on MP-BGP
•  APIC provisions the rest (BGP sessions, RD, import and export target, VPNV4
address family, route-map for route redistribution etc.)
•  MP-BGP doesn’t carry end point tables(MAC and IP)
MP-BGP sessions
with two spine nodes

External Routed Networks (L3outside) Configuration
Tenant
External Routed Networks
L3Outside (l3extOut)
Logical Node Profile (l3extLNodeP)
Logical Interface Profile (l3extLIfP)
BGP Peer Connectivity Profile (bgpPeerP)
External Network Instances
Profile (l3extInstP)
L3out Name
Private Network association
External Routed Domain association
Protocol selection (i.e OSPF area)
Node selection
Router ID configuration
Loopback Interface configuration
Interface selection (routed interface, sub-
interface, SVI)
IP address configuration
Association to protocol policy
(authentication, network type, etc)
BGP peer configuration
BGP settings
Remote AS
Import/Export route control subnets
Import security subnets
Contracts: (provided, consumed, taboo)

Import and Export Route Control Example
100.1.1.0/24
100.2.2.0/24
100.3.3.0/24
Tenant-1:VRF-1
L3 EPG 1:
Import route control:
100.1.1.0/24
100.2.2.0/24
BGP
Neighbor
BGP
Neighbor
Only prefix 1001.1.0/24 added
to MP-BGP MP-BGP table.
Tenant-1:VRF-1
>i100.1.1.0/24
>i100.2.2.0/24
Tenant-1:VRF-1
L3 EPG 2
Export route control:
100.1.1.0/24
100.1.1.0/24
100.2.2.0/2
100.3.3.0/24
100.1.1.0/24
100.2.2.0/24
100.1.1.0/24
100.2.2.0/24
100.1.1.0/24

§  Route control is configured at the L3out EPG object (L3extInstP)
§  A “route-map” is created for the L3out.
§  An “ip prefix-list” is created for each L3out EPG (L3extInstP)
Export Route Control Configuration Example

§  Policy control enforcement is enabled per Private Network (VRF)
§  If policy control is unenforced for the Private Network all data plane
traffic is permitted between L3out EPGs.
§  If policy control is enforced contracts are required between L3out EPGs
to allow transit traffic and between Application Profile EPGs for fabric to
L3out traffic.
§  Security Policy is enforced for IP prefixes not L4 ports.
§  Filters (L4 port filters) are not supported for L3out EPG contracts
§  Security Policy subnets are configured on the L3out EPGs
Security Policy Control Enforcement

Security Policy Subnet Configuration
Zoning rules are created for
Security Import Subnets
when contracts are
configured between L3 outs

ACI Topologies

Interfacing to WAN/DCI Routing (Planned 11.2, Q1CY16)
Extending VXLAN to the PE
Direct Connect
from Spine to PE
Web/App
DB
•  GBP VXLAN hand off from border leaf to WAN/DCI
•  Direct Connection between ‘Spine’ and ASR9K and N7K (ASR1K EC is in progress)
•  BGP-EVPN L3 route exchange (Layer 2 post 11.2)
MP-BGP – GBP
VXLAN
DCI
OTV/VPLS
WAN
DC Site 2
Client
PE

PE

PE

PE

•  Direct connect to Spine with GBP VXLAN to PE
•  EPG/VRF == Fabric Scale
•  Endpoint and LPM == COOP (LISP DB) Scale
Leaf
VT
EP
VT
EP
VT
EP
VT
EP
VT
EP
VT
EP
SpineR
R
R
R
Bor
der
Leaf
EVPN
iBGP

Multi-Fabric Scenarios
In-Region ‘and’ Out-of-Region
Fabric ‘A’ Fabric ‘B’
Web/App
DB
Web/App
Web/App
DB
Web/App
•  In-Region (Same Room, Building, Campus, Metro)
< 10 msec RTT
•  Out of Region Data Centers
> 10 msec RTT

Single Fabric Scenarios
Multi-Site (Stretched) Fabric
Site/Room ‘A’ Site/Room ‘B’
•  Single Fabric + Multi-Site
•  Single Operational Zone (VMM, Storage, FW/LB are all treated as if it is ‘one’ zone)
•  e.g. Single vCenter with Synchronized Storage
•  Interconnect between sites
•  Direct Fiber (40G), DWDM (40G or multiple 10G), Pseudo Wire (10G or 40G)
Interconnect Leaf
Nodes
HYPERVISOR HYPERVISOR HYPERVISOR
10 msec. Round Trip

Site ‘A’ Site ‘B’
Multi-Fabric – Current Options
L2/L3 Classification
Web1
App1
dB1
Web2
App2
dB2
L2_Outside
Classify Based on
VLAN
L3_Outside
Classify Based on
Network/Mask
Classify traffic arriving from a remote
site (fabric) based on the incoming
VLAN or layer 3 prefix (LPM)

Site ‘A’ Site ‘B’
Multi-Fabrics – Current Options
External Synchronization of Fabric Policy
Symmetrical XML
Configuration will
maintain consistent
operation between
fabrics
Externally triggered
Export and Import
between Fabrics is
another option to
maintain consistency

Multi-Site
Traffic
mBGP - EVPN
Multi-Fabric Extended GBP VXLAN
(Target Q1CY16)
mBGP is used to advertise host & network
level reachability between fabrics
Central Policy Control
to coordinate across
multiple fabrics
VTEP
IP
VNID
Tenant
Packet
Group
Policy
•  Multiple APIC Clusters (N+1 Redundancy
for each Fabric)
•  Single Operational Domain via Hierarchical
Controller
•  VXLAN is extended between fabrics (EPG
information is communicated between
fabrics)
•  VXLAN translation permits independent
fabrics while maintaining full policy
VTEP
IP
VNID
Tenant
Packet
Group
Policy
VTEP
IP
VNID
Tenant
Packet
Group
Policy

Hypervisor Integration

Hypervisor Interaction with ACI
Two modes of Operation
•  ACI Fabric as an IP-Ethernet
Transport
•  Encapsulations manually allocated
•  Separate Policy domains for Physical
and Virtual
VLAN 10 VLAN 10 VXLAN 10000
Non-Integrated Mode
•  ACI Fabric as a Policy Authority
•  Encapsulations Normalized and
dynamically provisioned
•  Integrated Policy domains across
Physical and Virtual
APP WEB DB
Integrated Mode
DB
126

vCenter DVS SCVMM
§  Relationship is formed between
APIC and Virtual Machine Manager
(VMM)
§  Multiple VMMs likely on a single
ACI Fabric
§  Each VMM and associated Virtual
hosts are grouped within APIC
§  Called VMM Domain
§  There is 1:1 relationship between a
Virtual Switch and VMM DomainVMM Domain 1
Hypervisor Integration with ACI
Control Channel - VMM Domains
vCenter AVS
VMM Domain 2 VMM Domain 3
127

L/B
EPG
APP
EPG
DBF/W
EPG
WEB
Application Network Profile
VM VM VM
WEB PORT GROUP APP PORT GROUP DB PORT GROUP
APIC
§  ACI Fabric implements policy on Virtual
Networks by mapping Endpoints to
EPGs
§  Endpoints in a Virtualized environment
are represented as the vNICs
§  VMM applies network configuration by
placement of vNICs into:
§  Port Groups (VMWare),
§  VM Networks (Hyper-V)
§  Networks (OpenStack)
§  EPGs are exposed to the VMM as a 1:1
mapping to Port Groups, VM Networks
or OpenStack Networking.
128

VMWare Integration
Three Different Options
+
Distributed Virtual Switch
(DVS)
vCenter + vShield
Application Virtual Switch
(AVS)
•  Encapsulations: VLAN
•  Installation: Native
•  VM discovery: LLDP
•  Software/Licenses:
vCenter with
EnterprisePlus License
•  Encapsulations: VLAN,
VXLAN
•  Installation: Native
•  VM discovery: LLDP
vCenter with
EnterprisePlus License,
vShield Manager with
vShield License
•  Encapsulations: VLAN,
VXLAN
•  Installation: VIB through
VUM or Console
•  VM discovery: OpFlex
vCenter with
EnterprisePlus License
129

APIC Admin
VI/Server Admin Instantiate VMs,
Assign to Port Groups
L/B
EPG
APP
EPG DB
F/W
EPG
WEB
Create Application Policy
WebWebWeb App
HYPERVISOR HYPERVISOR
VIRTUAL DISTRIBUTED SWITCH
vCenter
Server / vShield
8
5
1
9
ACI
Fabric
Automatically Map
EPG To Port Groups
Push Policy
Create VDS2
Cisco APIC and VMware
vCenter Initial
Handshake
6
DB DB
7
Create Port
Groups
ACI Hypervisor Integration – VMware DVS/vShield
APIC
3
Attach Hypervisor
to VDS
4
Learn location of ESX
Host through LLDP
130

Southbound
OpFlex API
VMVM VM VM
N1KV VEM
vSphere
Hypervisor
Manager
§  OpFlex Control protocol
-  Control channel
-  VM attach/detach, link state
notifications
§  VEM extension to the fabric
§  vSphere 5.0 and above
§  BPDU Filter/BPDU Guard
§  SPAN/ERSPAN
§  Port level stats collection
§  Remote Virtual Leaf Support
(future)
Application Virtual Switch (AVS)
Integration Overview
131

APIC Admin
VI/Server Admin Instantiate VMs,
Assign to Port Groups
L/B
EPG
APP
EPG
DB
F/W
EPG
WEB
WebWebWeb App
Application Virtual Switch (AVS)
vCenter
Server
8
5
1
9
ACI
Fabric
Automatically Map
EPG To Port Groups
Push Policy
Create AVS
VDS
2
Cisco APIC and VMware
vCenter Initial
Handshake
6
DB DB
7
Create Port
Groups
ACI Hypervisor Integration – AVS
APIC
3
Attach Hypervisor
to VDS
4
Learn location of ESX
Host through OpFlex
OpFlex Agent OpFlex Agent
132

VM Attribute EPG Classification with AVS
11.1

End-Points end EPG membership
Server
Virtual Machines & Containers
Storage
Client 134
•  Endpoint == Workload unit connected to network directly
or indirectly
•  An endpoint has address (identity), location, attributes
(version, patch level)
•  Can be physical or virtual or container
•  End Point Group (EPG) membership defined by:
•  Ingress physical port (Leaf or FEX)
•  Ingress logical port (VM port group)
•  VLAN ID
•  VXLAN (VNID)
•  IP Prefix/Subnet (so far only applicable to external/border
leaf connectivity)
•  VM-based attributes (11.1 release)
•  IP address (planned for 11.1(MR2) – Sept 2015)

Hypervisor Integration with ACI 11.0
EPG Classification via Port Groups
•  VM’s are placed within the Port Group defined for each EPG
•  Traffic is encapsulated with the specific VLAN or VXLAN assigned to that port group
on that port and forwarded upstream to the TOR
VXLAN
VNID = 5789
VXLAN
VNID = 11348
802.1Q
VLAN 50
PayloadIP
GBP
VXLAN
VTEP
VXLAN
Leaf
VTEP
802.1Q
vSwitch
WEB PORT
GROUP
APP PORT
GROUP
vSwitch
WEB PORT
GROUP
APP PORT
GROUP
802.1Q
VLAN 125
PayloadIP
PayloadIP
Port Groups
Created for
Each EPG

EPG Classification via VM Attributes
•  End Point Groups (EPG’s) can leverage
multiple methods to ‘classify’ an endpoint or
traffic from an endpoint
•  VM Port Groups Provide a simple mechanism
to correlate a VM to a specific policy group
•  VM Attributes can also be used to classify a
VM as a member of an EPG
•  Leverage ACI release 11.1 with AVS (initial
deployment)
•  Support for other Hypervisor switches
VMware vDS, Microsoft vSwitch, OVS
(future)
VM Attribute
Guest OS
VM Name
VM (id)
VNIC (id)
Hypervisor
DVS port-group
DVS
Datacenter
Custom Attribute
MAC Address
IP Address
vCenterVMAttributes
VMTraffic
Attributes

•  There are two categories of Attributes
supported with the 11.1 release
•  VM Attributes (set by server administrator
on creation of the VM)
•  VM Traffic Attributes (VM MAC/IP address
or L4 port being used by the application)
•  Any endpoint placed within a Port Group on the
vSwitch can be micro-classified based on the
specific VM Attributes
•  Dynamic classification or re-classification
•  e.g. Re-classify an endpoint that has been
detected to have a security exposure
(move to quarantine security group)
VM Attribute
Guest OS
VM Name
VM (id)
VNIC (id)
Hypervisor
DVS port-group
DVS
Datacenter
Custom Attribute
MAC Address
IP Address
vCenterVMAttributes
VMTraffic
Attributes

AVS with ACI 11.1
vSwitch (AVS)
Port Group
EPG == VM
Attribute ‘x’
EPG == VM
Attribute ‘y’
APIC Admin
Create an EPG ==
VM Attribute ‘x’ on
VMM Domain ‘A’
3
4
APIC Distributes VM
Attribute Policies to
Leaf nodes
AVS notifies Leaf of
VM Attach via
OpFlex Channel
6
Leaf Determines
Attribute to EPG
Classification
7
Leaf Pushes EPG
encapsulation
binding to AVS via
OpFlex Channel
8
802.1Q
VLAN 50
AVS forwards traffic
with the correct EPG
label (encapsulation)
9
APIC Retrieves
Hypervisor State
(VM State & VM
Attributes) & Initiate
a Listener Process
for any changes/
updates
2
Administrator
Creates new vDS
(AVS)
1
VI/Server Admin
Boot new VM with
desired VM
Attributes
5

ACI Hypervisor Integration – Vmware vCenter View
139

VMware vCenter Plugin View
140

141

142

Microsoft SCVMM and Azure Pack
Integration

Cisco Confidential 144
Microsoft Interaction with ACI
Two modes of Operation
•  Policy Management: Through APIC
•  Software / License: Windows Server with
HyperV, SCVMM
•  VM Discovery: OpFlex
•  Encapsulations: VLAN, VXLAN and
NVGRE (Future)
•  Plugin Installation: Manual
Integration with SCVMM
APIC
Integration with Azure Pack
APIC
•  Superset of SCVMM
•  Policy Management: Through APIC or
through Azure Pack
•  Software / License: Windows Server with
HyperV, SCVMM, Azure Pack (free)
•  VM Discovery: OpFlex
•  Encapsulations: VLAN, VXLAN and
NVGRE (Future)
•  Plugin Installation: Integrated
+
144

APIC Admin
SCVMM Admin Instantiate VMs,
Assign to VM Networks
L/B
EPG
APP
EPG
DB
F/W
EPG
WEB
MSFT SCVMM
8
5
1
9
ACI
Fabric
Automatically Map
EPG To VM Networks
Push Policy
Create Virtual
Switch
2
Cisco APIC and MSFT
SCVMM Initial
Handshake
6
ACI and SCVMM Integration in 11.1 Release
APIC
3 Attach Hypervisor
to Virtual Switch
4
Learn location of HyperV
Host through OpFlex
OpFlex Agent
HYPERV VIRTUAL SWITCH
7
Create VM
Networks
OpFlex Agent
WEB VM NETWORK APP VM NETWORK DB VM NETWORK
145
Web Web AppApp DB

APIC Admin
(Basic Infrastructure)
Azure Pack Tenant
3
6
ACI
Fabric
Push Network
Profiles to APIC
Pull Policy on leaf
where EP attaches
Indicate EP Attach to attached leaf
when VM starts
1
2
ACI Azure Pack Integration in 11.1 Release
APIC
Get VLANs allocated
for each EPG
Create Application
Policy
7
Azure Pack SPF
SCVMM PluginAPIC Plugin
OpFlex Agent OpFlex Agent OpFlex Agent
Instantiate VMs
5
1
4
Create VM Networks
4
146
Web WebWebWeb AppApp DB DB

Microsoft Azure Pack Integration
§  Integration with Microsoft requires:
-  Windows Server 2012
-  Systems Center 2012 R2 with
SPF
-  Windows Azure Pack
§  Azure Pack provides single pane of glass
for Definition, creation, management of
their cloud service
§  Divided into Provider (Admin) portal and
Consumer Self-Service (Tenant) portal
§  Cisco ACI Service Plugin enables
management of Network Infrastructure
through APIC REST API
R2 w/ Service Provider
Foundation
Web
Sites
Service
Plans
Users
Provider
Portal
Consumer
Self-Service
Portal
Web Sites
Apps
Database
VMs
ACI
Service Provider
Customer
VMs SQL
Service
Bus …
147

Cisco ACI Network Offerings
Features Shared Network Virtual Private Network
Isolated Networks ✓ ✓
Firewall ✓ ✓
Shared DHCP ✓ ✓
Shared Load Balancer ✓ ✓
Shared Services ✓ ✓
Public Internet Access ✓ ✓
Private Address Space ✓
Private DHCP Server ✓

Use Cases
Shared Network and Virtual Private Network
WEB
WEB
APP
APP
Finance Tenant
DB
MONGO
DB
Shared Services
Tenant
DHCP
DNS
ACI Common
services
LB
FW
WEB
WEB
APP
APP
DevTest Tenant
192.168.0.0/16
APPAPP
Finance Tenant
DHCP
DNS
ACI Common
services
LB
FW
WEB WEB
APPAPP
DevTest Tenant
192.168.0.0/16
WEB WEB WEB WEBDB
MONGO
DB
Shared Services
Tenant
10.0.10.0/24 10.0.10.0/24

Admin Experience
Add & Configure
APIC, tenants,
and VLAN ranges
Usage & Billing statistics per user and other admin functions
150
Role Based Access Control for Shared Services

Admin Experience
Network and
Compute
resources
tenant has
access to
Application Network Profiles are created through Azure Pack, and
pushed to APIC using REST APIs
ACI constructs
available to
tenant
F5 or Citrix
Load Balancer
that is part of
ACI Fabric
Shared
Services

Tenant Experience
Network and
Compute
resources
tenant has
access to
Application Network Profiles are created through Azure Pack, and
pushed to APIC using REST APIs
ACI constructs
available to
tenant

Openstack and KVM/OVS Integration

Why Cisco ACI and OpenStack
TELEMETRY
AND
OPERATIONS5
•  Health Metrics
•  Visibility
•  Troubleshooting
•  Automation
•  Intent-driven
GROUP-BASED
POLICY
SUPPPORT
1
•  Service chaining
•  App Acceleration
SERVICE
CHAINING4
PHYSICAL +
VIRTUAL
•  Zero-touch
Performance
•  Physical server
•  Multi-hypervisor
2
•  Automatic
VXLAN
•  Distributed L2
FABRIC
TUNNELS3
•  Service
chaining and
redirection

W
eb
W
eb
W
eb
W
eb
A
pp
A
pp
D
B
D
B
NEUTRON
ROUTER
SECURITY
GROUP
NEUTRON
NETWORK
Contract Contract Contract
DBAPPWEB
ADC
F/W
ADC
APIC
Driver OVS Driver
Neutron
Networking
Group Policy
OVS Driver
Neutron
Networking
APIC Group Driver
W
eb
W
eb
W
eb
W
eb
A
pp
A
pp
D
B
D
B
Two Options for ACI
APIC Driver (ML2) Group Policy Plugin

NEUTRON
ROUTER
SECURITY
GROUP
W
eb
W
eb
W
eb
W
eb
A
pp
A
pp
D
B
D
B
NEUTRON
NETWORK
APIC
Driver OVS Driver
Neutron
Networking
•  ML2 (modular level 2) driver
supporting existing Neutron
APIs: network, router, security
group, LBaaS, etc.
•  Automation of neutron ports for
virtual machines
•  Relies on OVS in hypervisor
•  Shipping today from Cisco
•  Available on Openstack
IceHouse, Juno, etc.
APIC Driver for OpenStack
APIC Driver (ML2)

APIC Driver Details
Neutron Workflow
1.  User creates a network / router / etc. through Neutron CLI / Horizon / Heat
2.  OVS Driver selects VLAN from VLAN pool. VLAN is configured in Open vSwitch
3.  APIC Driver maps neutron object to APIC policy model
4.  IP Tables in Linux Hypervisor provides host-based security group enforcement
5.  Open vSwitch tags each Neutron network with VLAN
6.  ACI ToR translates VLAN into VXLAN, providing distributed L2 and distributed default gateway support.
OVS Driver
Neutron
Networking
APIC
Driver
Hypervisor Hypervisor Hypervisor Hypervisor Hypervisor Hypervisor
ACI Fabric Offers:
•  VXLAN tunnels
•  Distributed default
gateway
Hypervisor:
•  Enforces security
groups

What’s Wrong with OpenStack Networking Today?
Service A
Service B Service C
Cloud Application Model Neutron Model
Network /
subnet
Router
External Network
Network /
subnet
•  L2 / Broadcast is the base API!
•  Network / routers / subnets
•  Based on existing networking models
•  No concept of dependency mapping or
intent
•  No broadcast / multicast
•  Resilient / Fault Tolerant
•  Scalable Tiers
•  Built around loosely coupled services
•  Don’t care about IP addresses

Where Can We Do Better
§  Build self-documenting
dependency maps of
tiers of an application
§  Define network service
chains between tiers of an
application without low level
configuration
§  Separate application
requirements from low
level APIs
§  Separate tenant from
operator
Separation of Concerns Enable Network
Services
Dependency
Mapping
Service
A
Service
C
Abstract Application API
Low level / Detailed API
Service
A
Service
C
Service A consumes service B
and Service C
Service B
Service
A
Service
C
FIREWALL
Operator /
Admin
OpenStack
Tenant

Introducing Group-Based Policy
•  Intent-based API for describing application requirements
•  Separates concerns of tenants and operators
•  Captures dependencies between tiers of an application
•  Plugin model
•  Supports mapping to Neutron APIs
•  Supports “native” SDN drivers
Policy Rules Set
Web
Group
Classifier Action
FIREWALL
DB
Group
Classifier Action
Service
Chain

OpenStack GBP Architecture
Neutron Driver maps GBP
to existing Neutron API and
offers compatibility with any
existing Neutron Plugin
Native Drivers exist for
OpenDaylight as well as
multiple vendors (Cisco,
Nuage Networks, and One
Convergence)
Group Policy
CLI Horizon Heat
Neutron Driver
Neutron
Any Existing Plugins
and ML2 Drivers
Open model that is compatible with ANY physical or
virtual networking backends
Native Driver
1
1
2
2

Group-Based Policy Model
Policy Group: Set of endpoints with the same
properties. Often a tier of an application.
Policy RuleSet: Set of Classifier / Actions
describing how Policy Groups communicate.
Policy Classifier: Traffic filter including
protocol, port and direction.
Policy Action: Behavior to take as a result of a
match. Supported actions include “allow” and
“redirect”
Service Chains: Set of ordered network
services between Groups.
L2 Policy: Specifies the boundaries of a
switching domain. Broadcast is an optional
parameter
L3 Policy: An isolated address space
containing L2 Policies / Subnets
L3 Policy
Policy
Rule Set
Policy Rule
Policy Rule
Service Chain
Classifier Action
Classifier Action
L2 Policy
Policy
Group
Policy Target
Policy Target
Policy Target
Policy
Group
Policy Target
Policy Target
Policy Target
L2 Policy
provide consume
Node Node

Contract Contract Contract
DBAPPWEB
ADC
F/W
ADC
Group Policy
OVS Driver
Neutron
Networking
APIC Group Driver
W
eb
W
eb
W
eb
W
eb
A
pp
A
pp
D
B
D
B
•  OpenStack extensions on top of Neutron
exposing a policy API
•  Supports policy API to APIC
•  Backwards compatible with existing neutron
plug-ins (works with Nexus 9000
standalone)
•  Available for Openstack Juno (Q1 CY 15)
•  Open approach
•  Enables Openstack customers to deploy,
scale and modify policy across teams fast
Group-Based Policy APIC Driver (ML2)

Group Policy Plugin
ACI Fabric Offers:
•  VXLAN tunnels
•  Distributed default
gateway
•  Security enforcement
Neutron Workflow
1.  User creates Group-Based Policy through CLI / Horizon / Heat.
2.  OVS Driver selects VLAN from VLAN pool. VLAN is configured in Open vSwitch
3.  APIC Driver maps GBP to APIC policy
4.  Non-OpFlex: All inter-EPG traffic sent to ToR for enforcement (note, with OpFlex switching and enforcement may occur in OVS).
5.  Open vSwitch tags each group with VLAN
6.  ACI ToR translates VLAN into VXLAN, providing distributed L2, security policy, and distributed default gateway support.
OVS Driver
Neutron
Networking
APIC Group
Driver
Hypervisor Hypervisor Hypervisor Hypervisor Hypervisor Hypervisor
Group Policy

Install and try GBP now!
•  Available with OpenStack Juno release via StackForge
•  https://github.com/stackforge/group-based-policy
Runs with ML2 / OVS in a VM!
Try it now:
•  git clone http://github.com/group-policy/devstack -b juno-gbp
•  cd devstack;
•  stack.sh
Packaging and support available through Cisco and its partners Red Hat, Mirantis,
Canonical in progress

OpenStack Partners
Support for major OpenStack Distributions
Testing and Integration
Working closely with vendors to
test and qualify APIC Plugin on
OpenStack distributions
Easy Deployment
Integrating with existing
deployment tools used by
each distribution
Customization to ACI
Evaluating ways to expose features that
ACI can leverage such as Group Policy
and OpFlex
For Your
Reference

Support Matrix
Vendor Distribution Deployment
ToolChain
Base Operating
System
Ubuntu OpenStack Juju Ubuntu 14.04
Red Hat OS 5 Foreman RHEL 7
Mirantis OpenStack 5 Fuel Ubuntu 12.04
Mirantis OpenStack 5 Fuel Centos 6.5
Mirantis 6 + RHEL OSP 6 testing in progress
For Your
Reference

LINUX Container Integration

Hypervisors vs. Linux Containers
Hardware
Operating System
Hypervisor
Virtual Machine
Operating
System
Bins / libs
App App
Virtual Machine
Operating
System
Bins / libs
App App
Hardware
Hypervisor
Virtual Machine
Operating
System
Bins / libs
App App
Virtual Machine
Operating
System
Bins / libs
App App
Hardware
Operating System
Container
Bins / libs
App App
Container
Bins / libs
App App
Type 1 Hypervisor Type 2 Hypervisor Linux Containers (LXC)
Containers share the OS kernel of the host and thus are lightweight.
However, each container must have the same OS kernel.
Containers are isolated, but share
OS and, where appropriate, libs /
bins.

Hypervisor VM vs. LXC vs. Docker containers

•  Open-Source Container for Dummies
•  Open Source engine to commoditize LXC
•  Create lightweight, portable, isolated, self-
sufficient container from any application.
•  Delivers on full DevOps goal:
•  Build once… run anywhere.
•  Configure once… run anything
•  Ecosystems! OS, VM’s, PaaS, IaaS…
What is containers ?

SECURITY
Trusted
Zone
DB
Tier
DMZ
External
Zone
APP DBWEB
EXTERNAL
ACI
Policy
ACI
Policy
ACI
Policy
17
Abstracting / Mapping via ACI’s Application Network Profiles
! ! !
FW
ADC
Virtual Machines Docker Containers Bare-Metal Server
17
HYPERVISORHYPERVISORHYPERVISOR

SECURITY
Trusted
Zone
DB
Tier
DMZ
External
Zone
APP DBWEB
EXTERNAL
ACI
Policy
ACI
Policy
ACI
Policy
Option 1: Supporting Containers with ACI policy model via OpFlex on OVS
! ! !
FW
ADC
Virtual Machines Docker Containers Bare-Metal Server
HYPERVISORHYPERVISORHYPERVISOR
ACI Virtual Leaf: OpFlex + OVS
H1CY15

ACI Fabric
EPG
A
EPG
B
EPG = VLAN
ACI Contract 1)  Load the ACI Toolkit on your machine (documentation is at
http://datacenter.github.io/acitoolkit/docsbuild/html/genindex.html)
2)  Run the Toolkit to automate the following:
1)  Create the ACI constructs:
Tenant, BD, context, Application Network Profile, EPG, Contract
2) Attach physical interfaces to EPG(s)
3) Create a VLAN interface:
4) Attach the logical interface (VLAN) to the Physical Interface
5) Attach the EPG to the logical interface
Option 2: Supporting Containers with ACI policy model via MACVLAN on Linux

ACI Fabric
! !! ! ! !! ! ! ! !
20 20 3030
EPG
A
EPG
B
EPG = VLAN
ACI Contract
3)  Example with LXC
# Show the EPGs on the APIC
aci-show-epgs.py
# Create the container
lxc-create --template ubuntu --name container_name
# Attach the container to the EPG
aci-attach-epg.py --container container_name --epg epg_name
# Start the container
lxc-start --name container_name
4)  Example with Docker
“docker run” with “macvlan” network type
•  allows to map the docker container (MAC) to a VLAN by the “fire up” of
the Docker container
•  VLAN got previously mapped to EPG via interface (physical or trunk)
•  Connectivity is done without “virtual switching” which increases
performance
•  cross-server / cross-racks policy consistency granted via ACI.
•  P.S.: you may consider to previously run a network type “empty” to remove the masquerade
rule and not have the default docker0 associated with br0 linux bridge
Option 2: Supporting Containers with ACI policy model via MACVLAN on Linux

Cisco Confidential 176© 2014 Cisco and/or its affiliates. All rights reserved.
ACI Fabric – DC 01 ACI Fabric – DC 02
Docker-based Web Application Docker-based Web Application
ACI Application Network Profile
Data Center 01 Data Center 02
Multi-site abstraction and portability of Network Metadata and
Docker-based Applications

http://www.cisco.com/c/en/us/solutions/
collateral/data-center-virtualization/application-
centric-infrastructure/white-paper-
c11-732697.html
Docker and ACI

ACI Integration of Layer 4 – 7 Services

What is NOT Simple Today?
Challenges with Network Service Insertion
Router
Router
Switch
LB
FW
vFW
servers
1.  Configure Network to Insert Firewall
2.  Configure firewall network parameters
3.  Configure firewall rules as required by the
application
4.  Configure Load Balancer Network
Parameters
5.  Configure Router to steer traffic to/from Load
Balancer
6.  Configure Load Balancer as required by the
application

Intended design
Physical server Virtual Server
I want virtual firewalling in between
with ASA version a.b
I want physical firewalling in between
with F5 version a.b and Firewall version
c.d.
180

Automate Service Insertion Through APIC
APP DBWEBEXTERNAL
APIC Policy Model
Endpoint Group (EPG): Collection of similar End Points identifying a particular
Application Tier. Endpoint could represent VMs, VNICs , IP, DNS name etc
Application Profile: Collection of Endpoint Groups and the policies that define way
Endpoint group communicate with each other
Application profile
PolicyPolicyPolicy
181

ACI Service Insertion via Policy
•  Automated and scalable L4-
L7 service insertion
•  Packet match on a
redirection rule sends the
packet into a services
graph.
•  Service Graph can be one
or more service nodes pre-
defined in a series.
•  Service graph simplifies and
scales service operations
Begin End
Stage
1
FW_ADC1
EPG
2
EPG
1
Application
Admin
Service
Admin
ASA
5585
Netscaler
VPX
Chain
“FW_ADC 1”
Policy-based
Redirection
Stage
2

Intended Design Goal
Default Gateway
Transparent firewall with virtual ASA

Create Service Graph
184

Associate Graph to a Contract
185

APIC
L4-7 Plugin API (Device Package)
•  APIC interfaces with the device using
python scripts
•  APIC calls device specific python script
function on various events
•  APIC uses device configuration model
provided in the device package to pass
appropriate configuration to the device
scripts
•  Device script handlers interface with the
device using its REST or CLI interface
•  Open Specification
Device Spec
(XML)
Device Script
(Python / CLI)
Uses
Device’s
native API
186

Device Package Example
Following functions can be configured through APIC 187

Configure Function Parameters
188

Bridge Domain Outside Bridge Domain Inside
L3Out
L3InstP
Server EPG
service graph
Contract ProviderConsumer
VRF
This is just to make the Policy model happy
ARP flooding
unicast flooding
no ip routing
subnet, i.e. default gateway for servers
hardware proxy
Service Graph with the Policy Model

Service Configuration before the Service Graph
192.168.1.1 192.168.1.100
10.1.1.1
172.16.1.1
192.168.100.1
HTTP (TCP/80)
HTTPS (TCP/443)
DCERPC (TCP/135)
SSH (TCP/22)
ICMP
access-list OUT permit tcp host 192.168.1.1 host 10.1.1.1 eq 80
[…]
access-list OUT permit icmp host 192.168.1.100 host 192.168.100.1
30 ACL Rules
172.18.20.13
[…]
access-list OUT permit icmp host 172.18.20.13 host 192.168.100.1
15 ACL Rules
45 ACL Rules
Network Admin Security Admin
Add client 172.18.20.13,
call Security Admin to
enable access
Remove client 192.168.1.1, “no
other action necessary”
Add ASA rules for client
172.18.20.13
Original ASA rules never
change4
1
2
2
3
4
Files
Users

Automatic endpoint addition/removal with ACI
10.1.1.1
172.16.1.1
192.168.100.1
Servers
192.168.1.1
192.168.1.100
172.18.20.13
HTTP (TCP/80)
HTTPS (TCP/443)
DCERPC (TCP/135)
SSH (TCP/22)
ICMP
Source EPG
Leaf 1, port 1 Users
Destination EPG
Leaf 3, port 2 Servers
Network Admin
Add client 172.18.20.13, use
existing ASA instance
Remove client
192.168.1.1
Security Admin
Insert ASA instance in the service
graph with desired policies
Same 5 service rules and
actions
ASA1
Clients
Port Rules
access-list OUT permit tcp any any eq 80
access-list OUT permit icmp any any

FirePOWER in ACI

Advanced Threat Protection with FirePOWER + ACI
FireSIGHT
Management Center
Alerts
Network Visibility
Policy Management
Analytics
Remediation
•  Situation
–  Advanced threats that are not detected by
conventional security products
–  Limited security resources
•  ACI Solution
–  Automated provisioning of NGIPS and
Advanced Malware Protection
–  Visibility and awareness with FireSIGHT
–  Continuous analysis
–  Physical and virtual appliances
•  Benefits
–  Industry-leading security efficacy
–  Automation and correlation for reduced TCO
–  Retrospective security helps scope, contain
and remediate
Automated Feedback Loop
for Intelligent Threat
Response
WEB
WEB WEB WEB
DB
DB DB DB
APP
APP APP APP
AMP
NGIPS
AMP
NGIPS

Preserve Separation of Duties
SecOps
DevOps/
Network Admin
FireSIGHT Management
Center
Configuration Model
Device Interface: REST/CLI
APIC Script Interface
Python Scripts
Script Engine
APIC– Policy Manager
Physical Virtual

EPG
“Internet”
EPG
“Web”
FirePOWER Services For ACI – Intelligent Threat
Defense
FireSIGHT Management
Center
Alerts
Network Visibility
Policy Management
Analytics
Remediation
Application Policy
Infrastructure
Controller (APIC)
Service GraphContracts
NGIPS/NGFW
Advanced Malware Protection
Policy and events
Basic configuration
and health
Intelligent Remediation

UNT
PUBLIC

Trusted – No Graph
CORP

APIC

172.28.199.30

Move IP to Quarantine
Defense

Center

10.0.0.244

FW

NGIPS

10.1.0.234

Relaxed
REST calls to
APIC NB API
ACI Fabric
N9K
Leaf
Switch

FirePOWER

Appliance

10.0.1.30

SPAN Traffic
AttackESXi – 10.1.0.44
1.1.1.6 1.1.1.7
FW

QUA

Strict
REM

1.1.1.3
Security Feedback Loop

§  Cisco® ASAv running Release 9.2(1) and later and Cisco ASA 5585-X running
Release 8.4(1) and later
§  Cisco ASA Release 9.2(2) and later is recommended for all appliances
§  Device specification
§  Hierarchical model of the device capabilities in Cisco APIC
§  E.g., the list of supported features that are configurable by the Cisco APIC user
§  Function-independent vs. function-specific parameters
§  Device script
§  Converts Cisco APIC specific API function calls into Cisco ASA CLI script over HTTPS
§  E.g., how to configure an ACL or interface on Cisco ASA with the given parameters from
Cisco APIC
§  Add/delete/modify or monitor health

Routed Mode
Transparent Mode
External
EPG E1
App-A
EPG
FW
Graph A
10.0.0.0/24
10.0.0.1 20.0.0.1
20.0.0.0/24
Tenant A
Consumer Provider
EPG A EPG BFW
Graph A
10.0.0.0/24
Consumer Provider
Tenant A

•  Routed Mode
•  Transparent Mode
EPG

A
EPG
B
FW

10.0.0.0/24

Tenant

A

External

Internal

EPG

A
EPG
A
FW

10.0.0.1
20.0.0.1

Tenant

A

External

Internal

VRF
VRF

OSPF/BGP
OSPF/BGPOSPF/BGP
VRF
VRF

10.0.0.2
20.0.0.2

10.0.0.10
10.0.0.11
100.0.0.0/24
200.0.0.0/24

201.0.0.0/24

202.0.0.0/24

203.0.0.0/24

101.0.0.0/24

102.0.0.0/24

103.0.0.0/24

200.0.0.0/24

201.0.0.0/24

202.0.0.0/24

203.0.0.0/24

100.0.0.0/24

101.0.0.0/24

102.0.0.0/24

103.0.0.0/24

Cisco® ACI Fabric
Cisco ASA ClusterFlow Symmetry Within
Service Graph
Stateless Load
Balancing
Stateful Flow Asymmetry
on Changes
Elastic Scalability
Asymmetry
Compensation

Cisco Security + ACI Roadmap
ASA, FP, NGFW
= EC/AC = CC/BC = Roadmap
Q2CY15 4QCY15
Release & Commit Status FCS+9 (ACI 11.1) FCS+12 - ACI 11.1(1)
ASA •  Support for Multi-context
•  Support for BGP
•  Support for OSPF support
•  Support for ASA + FirePOWER
Services (5585)
•  Support for SGACL/SXP
configuration
•  Support for S2S VPN
•  Support for RAVPN
FirePOWER •  Device Package 1.0
•  FirePOWER Threat Capabilities
•  Switched interfaces
•  Usability Enhancements
•  Add missing management
functions
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 201
Cisco Confidential – Redistribution Prohibited

ACI L4-L7 – Device Package Update
Device Package ETA
F5 (Big IP physical and virtual) Now
ASA (5585 8.4 and ASAv 9.2.1) Now
Citrix (NetScaler MPX, SDX, VPX, NetScaler 1000v) Now
A10 Now
Radware ADC Now
Avi Networks Now
Cisco Sourcefire Q2 CY15
Fortinet Q2 CY15
Palo Alto Networks Q2 CY15
Check Point Q3 CY15
Radware DefensePro Q3 CY15
Intel Security - McAfee Q3 CY15
Symantec Data Loss Prevention Q3 CY15

Programmability and ACI

Virtual
Machines
LXC / Docker
Containers
Apps Portability, Cross-
Platform & Automation
Applications PaaS
Two Market Transitions – One DC Network
Traditional
Data Center
Networking
Network
Apps Policy
Application Centric
Infrastructure (ACI)
Network + Services
Abstraction & Automation
Infrastructure HyperScale
Data Centers
DC
Switching

We currently have :
•  REST API
•  Full Object Model exposed
•  JSON or XML
•  Python SDK for accessing object
model
PROGRAMMABILITY & ACI

DBAPPADC
WEBF/W
ADC
Typical Application Network Profile on ACI

EPG 100 EPG 200
App 1 App 2
10.10.40/24
10.10.30/24
10.10.20/24
10.10.10/24
VLAN 400
10.10.40/24
VLAN 300
10.10.30/24
VLAN 200
10.10.20/24
VLAN 100
10.10.10/24
Apps Coupled
to Location
ACL-based Policy Per
Interface
Visibility At Network or
VLAN Level
No Address Independence
or Policy Mobility
Apps Decoupled
from Location
Visibility At App or Group
Level
Policy Between Groups
Complete Address
Independence & Policy
Mobility
Traditional Network Model Application Centric Infrastructure
EPG 100
EPG 200
EPG 300
EPG 400
EPGs @ ACI bring true network abstraction, as needed
207

From Development to Test to Production
EPG Dev
DEV DEV
EPG Test
TEST TEST
EPG Prod
PROD PROD
Development lifecycle push as code progresses
EPGs can be used to segregate separate development phases.
208

209
Many times, it’s the same way it’s being done already

Leveraging Declarative Modeling for Application Profiles
WAN
Firewall
LB to EPG 2
Connect to EPG 3
Connect to EPG 2
High Priority
EPG1 EPG2 EPG3
APPLICATION PROFILE
Security GovernanceService Level ScalabilityAvailabilityPerformance
ADC
F/W
ADC
WEB APP DB

WEB APP DatabaseLoad
Balancer
User/Client
Browser
Example of EPG allocation and associated ACI contracts
on a 3-Tier video application
External EPG Front-End-Scale EPG Web EPG APP EPG DB EPG

On-going App Development evolution towards Cloud model
From Traditional Monolithic Multi-tier App to Cloud-Aware App

Load
Balancer
Client
Product Info
Service
Order
Service
Feedback
Loop
Management
Cache-Fill
Cache
Control
Streaming
OLTP
OLAP
Real Time
Historical
REST
REST
Thrift
API Gateway
Rest Proxy
Event
Publishing
Browser
REST
Client
Content
Router
Product Info
UI
Order Service
UI
Feedback
Loop UI
Service Registry
Load
Balancer
Same video application example as microservices-based Cloud-App

Load
Balancer
Client
Product Info
Service
Order
Service
Feedback
Loop
Management
Cache-Fill
Cache
Control
Streaming
OLTP
OLAP
Real Time
Historical
API Gateway
Rest Proxy
Event
Publishing
Browser
REST
Client
Content
Router
Product Info
UI
Order Service
UI
Feedback
Loop UI
Service Registry
Load
Balancer
Potential ACI EPG and contracts allocation on a Cloud-App

We currently have :
•  REST API
•  Full Object Model exposed
•  JSON or XML
•  Python SDK for accessing object
model
But….
•  Steep learning curve
•  5000+ classes
•  New concepts, etc.
PROGRAMMABILITY & ACI

•  Ease the learning curve
•  Remove some initial frustration
•  Address 80% of the use cases
•  Provide examples and sample scripts for
customers
•  Accelerate ACI adoption
ACI TOOLKIT – GOALS

Cisco ACI Toolkit
Infrastructure as Code
https://github.com/datacenter/acitoolkit
http://datacenter.github.io/acitoolkit/

•  Simple toolkit built on top of APIC API
•  Set of simple python classes
Python Library
Used to generate REST API calls
Runs locally
•  Small number of classes
~30 currently
“Intuitive” names
•  Not full functionality, most common
Focused primarily on configuration
•  Preserves the ACI basic concepts
Tenants, EPGs, Contracts, etc.
•  Expose ACI to DevOps as a library / code
APIC
ACI Toolkit
Linux
Commands
NX-OS like
CLI
Do It
Yourself
Cisco ACI Toolkit

Tenant
Context /
VRF
Subnet
Bridge
Domain
Outside
EPG
EPG
App. Profile
TabooContract
Filter Entry
L3 Interface
L2 InterfaceInterface
Node
Link
POD
1
**
1
* *1
1
* *
***
*
Provide /
Consume
1
* *
*
*
*
1 1
* *
1
*
Provide /
Consume
*
*
1
1
*
1
1
1
1
1
1
*
1
*
1
2
1
*
1
*
1
*
NetworkPhysical Policy
ACI Toolkit
Policy Model
Jan/2015

http://goo.gl/agx8gZ
Docker and ACI
https://registry.hub.docker.com/u/dockercisco/aci/

ACI + UCS are unique as the foundation for the App market transition
ACI + UCS
Traditional Monolithic Multi-tier App Cloud-Aware App
ACI supports physical, virtual and container based workloads as well as API and code / library based consumption. Allows business to change app models at their speed.

GitHub – a resource for ACI scripts and tools
•  ACI Toolkit:
https://github.com/datacenter/acitoolkit
•  ACI Diagram
https://github.com/cgascoig/aci-diagram
•  ACI Endpoint Tracker
docsbuild/html/endpointtracker.html
225

ACI Toolkit – using the APIC API’s
•  Fulfill prerequisites:
Python 2.7+
setuptools package (apt-get install python-pip – installs setuptools too)
requests library (pip install requests)
websocket-client library (pip install websocket-client)
•  Get acitoolkit:
git clone https://github.com/datacenter/acitoolkit.git
•  Install acitoolkit:
cd acitoolkit
python setup.py install
226

Application Centric Infrastructure (ACI), the policy driven data centre

More Related Content

What's hot (20)

Similar to Application Centric Infrastructure (ACI), the policy driven data centre (20)

More from Cisco Canada (20)

Recently uploaded (20)

Application Centric Infrastructure (ACI), the policy driven data centre