BRKRST-2362

Deploying Performance Routing

Jean-Marc Barozet

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Business and IT are Changing Like Never Before

Drastic Change in Application Type, Delivery, and Consumption

Public/Hybrid Cloud
SaaS/IaaS
Storage

Users/ Machines
Proliferation of Devices

THE NETWORK

Private Cloud
VDI | IaaS
Database

60% of IT professional cites performance as key challenge for cloud

BRKRST-2362

How Application applications Type of applications are areConsumed Delivered

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

WAN Challenges
Data Centers

Public SaaS

I dont know, if I am getting my SLA

I dont know, the applications running in my network I dont know, how to isolate performance problems

I dont know, how much non-business traffic is consuming

WAN

Internet

Branch with no direct Internet access

Branch with direct Internet access

I cant do, anything about poor and inconsistent performance

I cant do, anything about my Network outages

I cant do, anything about under utilization of my Expensive WAN links I cant do, anything about unreliability for my WAN links

Network Admin
BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Why we need Performance Routing?

Visibility

Control

Report

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Introducing Performance Routing (PfR)

Application aware adaptive routing
Full utilization of expensive WAN bandwidth
Efficient distribution of traffic based upon load, circuit cost and path preference Improved Application Performance Per application best path based on delay, loss, jitter measurements Increased Application Availability Protection from carrier black holes and brownouts
PfR MCs

WAE Cluster
Email VMs

ISR G2

ASR1K ASR1K

Internet DMVPN

Email Path Video Path

Branch
PfR MC/BR

PfR BRs

Headquarter
ASR1K ASR1K

Master Controller (MC) Border Router (BR)

SP A MPLS SP B GETVPN MPLS GETVPN

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Performance Routing 101

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Performance Routing Components

The Decision Maker: Master Controller (MC)

Apply policy, verification, reporting No packet forwarding/ inspection required
MC

The Forwarding Path: Border Router (BR)

Gain network visibility in forwarding path (Learn, measure) Enforce MCs decision (path enforcement)
BR BR

Optimize by:
Reachability, Delay, Loss, Jitter, MOS, Throughput, Load, and/or $Cost
WAN1 WAN2

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Performance Routing Topologies

Internet Edge
ISP1
ISP2

Enterprise WAN
MC/BR

Branch

WAN1
(IP-VPN)
BR BR

MC
BR

MC/BR

WAN2
MC

BR

(IPVPN, DMVPN)

BR

MC/BR

HQ

Optimize by: Reachability, Loss, Delay, Jitter, MOS, Throughput, Load, and/or $Cost
2013 Cisco and/or its affiliates. All rights reserved.

BRKRST-2362

Cisco Public

Basic Configuration
MC (Decision Maker)
key chain pfr key 0 key-string cisco ! pfr master ! border 10.4.5.4 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external border 10.4.5.5 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external ! learn throughput ! max-range-utilization 15 ! ! Load-Balancing enabled by default !

HQ

BR (Forwarding Path)
MC
! key chain pfr key 0 key-string cisco ! pfr border local Ethernet0/0 master 10.2.3.3 key-chain pfr !
WAN1 (IP-VPN)

BR

BR

1. Provisioning 2. Learning 3. Monitoring and Optimization

MC/BR

WAN2 (IPVPN, DMVPN)

MC/BR

MC/BR

BR

Learning enabled by default with 15.2(3)T and XE 3.6

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

10

What you Need to Enable PfR + WAAS

WCCP establishes tunnels between router and WAAS device PfR needs to be aware of the WCCP Tunnel interfaces
router#show tunnel groups 2 tunnel groups active WCCP : service group 317 in "Default", ver v2, assgnmnt: mask-value set intf: Tunnel0, locally sourced WCCP : service group 318 in "Default", ver v2, assgnmnt: mask-value set intf: Tunnel2, locally sourced

PfR cannot control WAAS Express traffic which is sourced from the router
pfr master border 192.168.254.2 key-chain pfr-keychain interface GigabitEthernet0/2 external max-xmit-utilization percentage 80 link-group secondary interface GigabitEthernet0/1.34 internal interface GigabitEthernet0/1.32 internal interface Tunnel0 internal interface Tunnel2 internal
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Add both interfaces as PfR internal interfaces

Cisco Public

11

Performance Routing The Journey

Learning

Get the Traffic Classes in the MC database

Monitoring (Passive Active)

Get the Traffic Classes Performance Metrics

Check Delay, loss, threshold, Bandwidth and more Use a good performing path per Traffic Class

Choosing Your Policies

Enforcing the Path

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

12

Step #1 Learning
Automatic vs Static
Traffic Classes
MC

HQ

Learning
Prefixes ACL DSCP Based Applications Static Automatic
WAN1 (IP-VPN) WAN2 (IPVPN, DMVPN)

BR

BR

Learning Monitoring (Passive Active)

MC/BR MC/BR MC/BR

Choosing Your Policies Enforcing the Path

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

13

Learning
PfR Operates on Traffic Classes
PfR determines the traffic classes from the
Traffic Classes
MC

HQ

traffic flowing through the border routers Subsets of the total traffic must be identified, and these traffic subsets are named traffic classes
Dest. IP DSCP Delay 0 Loss Jitter BW

BR

BR

Voice, Video, Critical

The Rest of the Traffic

Prefixes or

10.2.2.0/24

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

Dest. IP 10.2.2.0/24

DSCP EF

AppID

Delay 0

Loss

Jitter

BW

Applications
Dest. IP 10.2.2.0/24 DSCP AF31 AppID Delay 0 Loss Jitter BW

MC/BR

MC/BR

MC/BR

BR

10.1.1.0/24 10.1.2.0/24
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

10.2.1.0/24 10.2.9.0/24

14

Learning Traffic Classes

On a global basis
MC

HQ

Learn

Global

BR

BR

Global learning Aka Similar to the default class in QoS

MC/BR

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

MC/BR

MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

15

Learning Traffic Classes

Or Per Service Class
MC

HQ

Voice - Video

Service Class

Critical Application Rest of the Traffic

Voice, Video, Critical

BR

BR

The Rest of the Traffic

Or define Service Classes

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

Aka Similar to the class-map concept in QoS

Allows to define:
Specific policies per group
MC/BR MC/BR MC/BR

BR

Specific thresholds per group

Specific monitoring mode per group
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

16

Learning Traffic Classes Definition

Automatic learning is enabled by default Default is to learn based on destination prefixes with aggregation mask /24
Type Destination Prefix (Mandatory) ACL Application (Optional) Example 10.0.0.0/8 20.1.1.0/24

10.1.1.0/24 dscp ef
10.1.1.0/24 dst-port 50 10.1.1.0/24 telnet

Well-Known

20.1.0.0/16 ssh
10.1.1.0/24 nbar RTP 20.1.1.0/24 nbar citrix

NBAR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

17

Traffic Class Learning

Configuration Sample
MC pfr master !

HQ

learn throughput
list seq 10 refname <CLASS_NAME_1> traffic-class access-list <ACL1> filter <PREFIX_LIST1> aggregation-type prefix-length <LENGTH1> throughput list seq 20 refname <CLASS_NAME_2> traffic-class access-list <ACL2> filter <PREFIX_LIST2> aggregation-type prefix-length <LENGTH2> throughput [Rest of the traffic]
MC/BR

BR

BR

#1

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

#2

MC/BR

MC/BR

BR

DEFAULT

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

18

Traffic Class Learning

Disable Global Learning (Optional)
MC pfr master learn throughput traffic-class filter access-list DENY_GLOBAL_LEARN_LIST list seq 10 refname <CLASS_NAME_1> traffic-class access-list <ACL1> filter <PREFIX_LIST1> aggregation-type prefix-length <LENGTH1> throughput ! ! Access-list for disabling global learn. ! ip access-list extended DENY_GLOBAL_LEARN_LIST BR

HQ

BR

Voice, Video, Critical

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

deny ip any any

MC/BR MC/BR MC/BR

BR

When you want to use PfR for a subset of the traffic (ie Voice traffic only, rest is controlled by the routing protocol)
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

19

Traffic Class Learning

Using NetFlow on BRs
Traffic Classes
Destination Prefix DSCP App

HQ

MC

Id

Delay

Loss

Ingress

Egress

BW

BW

BR

Exit

NetFlow Cache
Traffic Classes
WAN1 (IP-VPN)

BR

BR

NetFlow Cache

Automatic Learning is enabled by default once you enable PfR and add the Border Routers MC commands BRs to learn Traffic Classes
MC/BR

WAN2 (IPVPN, DMVPN)

MC/BR

MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

20

Learning Traffic Classes

Database Filled with TCs
MC
Destination Prefix 10.1.1.1/32 DSCP EF App Id Delay Loss Ingress BW Egress BW BR BR1 Exit E0/0

HQ

20.2.1.0/24
30.1.1.0/24

AF31
0

BR2
BR1

E0/0
E0/0

NetFlow Cache

BR

BR

NetFlow Cache

Traffic Classes
WAN1 (IP-VPN) WAN2 (IPVPN, DMVPN)

BRs use their NetFlow cache

Top Talkers based on throughput

BRs aggregate based on the configured destination mask Send the reports to the MC every minute

MC/BR

MC/BR

MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

21

Step #2 Measurement
Passive Monitoring
Traffic Classes
MC

HQ

Passive
Reachability Egress BW PfR Netflow Monitoring Flows Need not be symmetrical Delay Loss Ingress BW

Passive Performance Metrics

NetFlow Cache

BR

BR

NetFlow Cache

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

Learning Monitoring (Passive Active) Choosing Your Policies Enforcing the Path
MC/BR MC/BR MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

22

Measuring Passive Mode

For Data, Best Effort Applications
MC
Destination Prefix 10.1.1.1/32 10.1.10.0/24 DSCP 0 AF11 App Id Delay Loss Ingress BW Egress BW BR BR1 BR1 Exit Gi1/1 Gi1/2

HQ

BR

BR

0
BR BR1 BR2 Links Gig1/1 Gig1/2 Ingress

BR2
Egress

Gi1/1

Traffic Classes

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

PfR uses NetFlow to collect and aggregate passive monitoring statistics on a per traffic class basis.
MC Instructs BRs to monitor the performance

MC/BR

MC/BR

MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

23

Measuring Passive Mode

For Data, Best Effort Applications
MC
Destination Prefix 10.1.1.1/32 10.1.10.0/24 DSCP 0 AF11 App Id Delay 60 110 Loss 0 0 Ingress BW 20 52 Egress BW 40 60 BR BR1 BR1 Exit Gi1/1 Gi1/2

HQ

89
BR BR1 BR2

1
Links Gig1/1 Gig1/2

34

10
Ingress 200 130

BR2
Egress 40 60

Gi1/1

Traffic Classes

NetFlow Cache

BR

BR

NetFlow Cache

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

Border routers collect and report passive monitoring statistics to the master controller approximately once per minute.
BRs gather performance measurements using Netflow BRs report Performance Metrics for Traffic Classes to the Master Controller

MC/BR

MC/BR

MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

24

Step #2 Measurement
Active Mode
Active
Reachability Jitter Delay Loss MOS
Traffic Classes Active Performance Metrics
MC

HQ

SLA

IP SLA Responder

BR

BR

PfR enables IP SLA feature

Probes sourced from BR ICMP probes learned or configured TCP, UDP, JITTER need ip sla responder

Probe active path

WAN1 (IP-VPN) WAN2 (IPVPN, DMVPN)

Learning Monitoring (Passive - Active) Choosing Your Policies Enforcing the Path
MC/BR MC/BR MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

25

Measuring Active Mode

Hybrid Modes
Fast
Active probes on all path all the time
Passive to measure BW only
Traffic Classes Active Performance Metrics
MC

HQ

SLA

BR

BR

Probe all paths

Active Throughput
Passive to measure BW only Active probing on current exit
WAN1 (IP-VPN) WAN2 (IPVPN, DMVPN)

MC/BR

MC/BR

MC/BR

BR

Fast Mode is used when fast failover is needed

BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

26

Measuring Active Mode

For Voice, Video and Critical Apps
MC
App Id Ingress BW Egress BW

HQ

SLA

Destination Prefix

DSCP

Delay

Jitter

Loss

BR

Exit

10.1.1.1/32
10.1.10.0/24

EF
AF31 0

BR1
BR1 BR2

Gi1/1
Gi1/2 Gi1/1

BR

BR

Traffic Classes
WAN1 (IP-VPN) WAN2 (IPVPN, DMVPN)

Active monitoring involves creating a stream of synthetic traffic (IP SLA probes) that replicates a traffic class as closely as possible. MC Instructs BRs to monitor the performance
MC/BR MC/BR MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

27

Measuring Active Mode

For Voice, Video and Critical Apps
MC
App Id Ingress BW Egress BW

HQ

SLA

Destination Prefix

DSCP

Delay

Jitter

Loss

BR

Exit

10.1.1.1/32
10.1.10.0/24

EF
AF31 0

60
110 89

10
15 26

0
0 1

20
52 34

40
60 10

BR1
BR1 BR2

Gi1/1
Gi1/2 Gi1/1

BR

BR

Traffic Classes
WAN1 (IP-VPN) WAN2 (IPVPN, DMVPN)

BRS gather performance measurements using IP SLA probes The performance metrics of the synthetic traffic are measured The results are applied to the traffic class entry in the Master Controller database BRs report Performance Metrics for Traffic Classes
MC/BR MC/BR MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

28

PfR and Monitor Mode

Delay Loss

For Your Reference

Unreachable

Jitter

MOS

Bandwidth

Monitor Mode

When to use?

Passive

Internet presence deployment Destination prefixes are all over the internet. These destinations may not respond to ICMP probes. Traffic is TCP traffic Enterprise deployment Traffic is Between Head-quarter/Branch. If there is TCP traffic Traffic is Between Head-quarter/Branch. The only optimization criteria is traffic-class performance (i.e. no optimization based on bandwidth).

Both

Active

Active Throughput
Fast

Traffic is Between Head-quarter/Branch. Traffic is UDP or it is Encrypted

Traffic is time-sensitive like video-stream, VoIP. Traffic is Between Head-quarter/Branch

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

29

Step #3 Policy Definitions

Choosing Your Policies
Traffic Classes
MC

HQ

Link
Load balancing Max utilization Link grouping $Cost

Application Performance
Reachability
Delay Loss MOS Voice, Video, Critical
BR BR

The Rest of the Traffic

Jitter

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

Learning Monitoring (Passive Active) Choosing Your Policies Enforcing the Path
MC/BR MC/BR MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

30

Choosing Your Policies

Resolvers per Group
1. Link-Group
2. Loss
MC

HQ

Voice - Video

3. Jitter 4. Delay 1. Link-Group

Voice, Video, Critical

BR

BR

The Rest of the Traffic

Critical Application

2. Loss 4. Delay

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

Rest of the Traffic

Load-Balancing
MC/BR MC/BR MC/BR

Multiple resolvers can be assigned Set of Policies per Application Group Resolver Priority
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

BR

Cisco Public

31

Performance Routing Policies

Option1 - Global - Interface
Global

HQ

MC

pfr master max-range-utilization percent 30 delay relative 200 loss threshold 50000 mode <monitor-mode> resolve delay resolve loss
Per Interface

BR

BR

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

pfr master border 10.2.4.4 interface Ethernet0/1 external max-xmit-utilization percentage 80 maximum utilization receive percentage 80 link-group WAN1
IOS 15.2(3)T
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

MC/BR

MC/BR

MC/BR

BR

Cisco Public

32

Performance Routing Policies

Option2 - Per Group (pfr-map)
Application Group #1

HQ

MC

pfr-map MYMAP 10 match pfr learn list <CLASS_NAME_1> <Threshold definitions> <Monitoring mode definition> <Policy definitions> <Preferred Path and Fallback option> <Probe frequency> <Periodic timer definition>
Application Group #2
WAN1 (IP-VPN)

BR

BR

WAN2 (IPVPN, DMVPN)

pfr-map MYMAP 20 match pfr learn list <CLASS_NAME_2> <Threshold definitions> <Monitoring mode definition> <Policy definitions> <Preferred Path and Fallback option> <Probe frequency> <Periodic timer definition>
IOS 15.2(3)T

MC/BR

MC/BR

MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

33

Define Your Policy Thresholds?

Absolute vs. Relative
The required characteristic can be represented in two ways Absolute Short Avg exceeds the threshold of N msec.
Used for delay, loss, unreachable, jitter

Relative Short Avg exceeds the Long Avg by X percent.

Used for delay, loss, unreachable

Delay Absolute 70 60 50 40 30 20 10 0 1 3 5 7 9 11 13 15 17 19

OOP Detected Short Avg is higher than the absolute threshold

70 60 50 40 30

Delay Relative

OOP Detected Short Avg is higher than Long Avg by X Percent

Short Term Avg (5 minute)

Short Term Avg (5 minute)

Threshold 35 ms

20 10 0

Long Term Avg (60 minute)

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33
34

21 23

25 27 29

31 33

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Step #4 Enforce the Path

Automatic or force PBR
MC

HQ

Destination Prefix
BGP
Egress: route injection or Modifying the BGP Local Preference attribute Ingress: BGP AS-PATH Prepend or AS Community

Application
Dynamic PBR NBAR/CCE
BR BR

Voice, Video, Critical

The Rest of the Traffic

EIGRP Route Control Static Route Injection PIRO

WAN1 (IP-VPN) WAN2 (IPVPN, DMVPN)

Learning Monitoring (Passive Active) Choosing Your Policies Enforcing the Path
35

MC/BR

MC/BR

MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Performance Routing The Journey

Learning

pfr master learn <learn-list>

Monitoring (Passive Active)

pfr-map match <list> parameters policy definitions

Choosing Your Policies

Enforcing the Path

Automatic, PBR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

36

Performance Routing Domain

Moving PfR to multi-Sites

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Active Mode
Simplifying the Configuration
Active Fast Active Throughput TCP, UDP, JITTER probe need ip sla responder Whats needed: Configure a pfr-map that matches prefixes or applications @ Remote-site1 Define the policies Define the jitter probes And REPEAT for each remote site
MC/BR

HQ pfr-map MYMAP 10 match pfr learn list LEARN_LIST_VIDEO_BRANCH1 set periodic 90 set delay threshold 200 MC set loss threshold 50000 IP SLA Responder SLA set jitter threshold 30 set mode monitor fast set resolve loss priority 2 variance 5 set resolve jitter priority 3 variance 5 set resolve delay priority 4BR variance 5 BR no set resolve range no set resolve utilization set probe frequency 4 set active-probe jitter 20.9.9.9 target-port 2000

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

MC/BR

MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

38

PfR Multisite System Evolution

Peering & Discovery
Multisite MC Peering Framework
MC to MC Peering Framework can be used to exchange policies, services and feedback Remote Site Discovery
Automatic discovery of branch routers
Simplifies Configuration prefix and target discovery Probing Efficiency sharing of probe data across policies Enhance PfR remote site bandwidth discovery
MC/BR
WAN1 (IP-VPN)

HQ

MC

SLA

IP SLA Responder

BR

BR

WAN2 (IPVPN, DMVPN)

MC/BR

MC/BR

BR

IP SLA Responder
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

39

PfR Domain

HQ

Site HQ

Active Fast Active Throughput

BR

MC

Publish Prefix H1, H2, H3 Responder H

SLA

BR

Each MC announces its inside prefixes, together with probe target address and site names
MC/BR

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

Site 1

Site 2

MC/BR

MC/BR

Site 3

BR

Publish Prefix A Responder 1

Publish Prefix B Responder 2

Publish Prefix C, D, E Responder 3, 4

IP SLA Responder
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

40

PfR Domain
Target Discovery

HQ

MC

Prefixes Prefix A Prefix B Prefix C, D, E

Responders Responder1 Responder2 Responder3, 4

BW 1.2 Mbps 2.3 Mbps 10 Mbps

Sites Site 1 Site2 Site3

WAN1 (IP-VPN)

SLA

BR

BR

WAN2 (IPVPN, DMVPN)

Mapping table built on each site Allows automatic jitter probe configuration

Allows automatic probe generation

Remote Bandwidth discovery per remote site

MC/BR

MC/BR

MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

41

With Target Discovery Enabled

pfr-map MAP-TEST3 10 match pfr learn list LEARN_LIST_BRANCH1 set periodic 90 set mode route control set delay threshold 200 set loss threshold 50000 set jitter threshold 30 set mode monitor fast set resolve loss priority 2 variance 5 set resolve jitter priority 3 variance 5 set resolve delay priority 4 variance 5 no set resolve range no set resolve utilization set probe frequency 4 set active-probe jitter 20.9.9.9 target-port 2000 pfr-map MAP-TEST3 15 pfr master policy-rules MYMAP mc-peer head-end Loopback1 target-discovery [SNIP] ! pfr-map MYMAP 10 match pfr learn list LEARN_LIST_BRANCH

pfr-map MAP-TEST3 15 pfr-map MAP-TEST3 15 match pfr learn list LEARN_LIST_BRANCH2 pfr-map MAP-TEST3 15 match pfr learn list LEARN_LIST_BRANCH2 pfr-map MAP-TEST3 15 match pfr learn list LEARN_LIST_BRANCH2 set periodic 90 pfr-map MAP-TEST3 15 match pfr-map pfr learnMAP-TEST3 list LEARN_LIST_BRANCH2 set periodic 90 15 match pfr learn list LEARN_LIST_BRANCH2 set periodic 90 set delay threshold 200 pfr-map MAP-TEST3 15 match pfr learn list LEARN_LIST_BRANCH2 set periodic 90 set delay threshold 200 pfr-map MAP-TEST3 15 match pfr learnMAP-TEST3 list LEARN_LIST_BRANCH2 set periodic 90 set threshold delay threshold 200 pfr-map 15 set loss 50000 match pfr learn list LEARN_LIST_BRANCH2 set periodic 90 set delay threshold 200 pfr-map MAP-TEST3 15 set lossset threshold 50000 match pfr learnMAP-TEST3 list LEARN_LIST_BRANCH2 set periodic 90 delay threshold 200 pfr-map MAP-TEST3 pfr-map 1515 set loss threshold 50000 match pfr learn list LEARN_LIST_BRANCH2 set jitter threshold 30 set periodic 90 set delay threshold 200 set loss threshold 50000 match pfr learn list LEARN_LIST_BRANCH2 set jitter threshold 30 set periodic 90 set delay threshold 200 set loss threshold 50000 match pfr learn LEARN_LIST_BRANCH2 match pfr learn listlist LEARN_LIST_BRANCH2 set jitter threshold 30 set periodic 90 set mode fast set delay threshold 200 set loss threshold 50000 setmonitor jitter threshold 30 set periodic 90 set mode monitor fast set delay threshold 200 set loss threshold 50000 set jitter threshold 30 set periodic 90 set periodic 90 200 set mode monitor fast set delay threshold set resolve loss priority 2 threshold variance 5 set loss 50000 set jitter threshold 30 set mode monitor fast set delay threshold 200 set resolve loss priority 2 threshold variance 5 set loss 50000 set jitter threshold 30 set mode monitor fast set mode control set delay threshold 200 set resolve loss priority 2 threshold variance 5 set 50000 set resolve jitter priority 3loss variance 5 set jitter threshold 30 route set mode monitor fast set resolve loss priority 2 variance 5 set loss threshold 50000 set resolve jitter priority 3 variance 5 set jitter threshold 30 set mode monitor fast set resolve loss priority 2 threshold variance set delay threshold 200 set loss 50000 set resolve jitter priority 3 variance 5 set jitter threshold 30 5 set resolve delay priority 4 variance 5 variance set mode monitor fast set resolve loss priority 2 set resolve jitter priority 3 threshold variance 5 set jitter 30 5 5 set resolve delay priority 4 variance 5 set mode monitor fast set resolve loss priority 2 variance set resolve jitter priority 3 threshold variance 5 set loss threshold set jitter 3050000 set resolve delay priority 4 variance 5 variance set mode monitor fast no set resolve set resolve loss priority 2 setrange resolve jitter priority 3 variance 5 5 5 set resolve delay priority 4 variance 5 variance set mode monitor fast no set resolve range set resolve loss priority 2 set resolve jitter priority 3 variance set resolve delay priority 4 variance 5 set jitter threshold 305 5 5 set mode monitor fast no set resolve range set resolve loss priority 2 variance no set resolve utilization set resolve jitter priority 3 variance set resolve delay priority 4 variance 5 variance no set resolve range set resolve loss priority 2 no set resolve utilization set resolve jitter priority 3 variance 5 5 set resolve delay priority 4 variance 5 variance no set resolve range set mode monitor fast set resolve loss priority 2 no set resolve utilization set resolve jitter priority 3 variance set probe frequency 4 set resolve delay priority 4 variance 5 5 55 no set resolve range no set resolve utilization set resolve jitter priority 3 variance set probe 4utilization set resolve delay priority 4 variance 5 no set resolve range no frequency set resolve set resolve loss priority 2 variance resolve jitter priority 3 variance set probe 4set set resolve delay priority 4 variance 5 55 set active-probe jitter 20.9.9.9 target-port 2000 no set resolve range no frequency set resolve utilization set probe frequency 4 set resolve delay priority 4 variance 5 set active-probe jitter 20.9.9.9 target-port 2000 no set resolve range no set resolve utilization set probe frequency 4utilization set resolve jitter priority variance set resolve delay priority variance 55 set active-probe jitter 20.9.9.9 target-port 2000 43 no set resolve range no set resolve set probe frequency 4 set active-probe jitter 20.9.9.9 target-port 2000 no set resolve range no set resolve utilization set probe frequency 4 set active-probe jitter 20.9.9.9 target-port 2000 set resolve delay priority 4 variance 5 no set resolve range no set resolve utilization set probe frequency 4utilization set active-probe jitter 20.9.9.9 target-port 2000 no set resolve set probe frequency 4 set active-probe jitter 20.9.9.9 target-port 2000 set resolve nono set resolve utilization set probe frequency 4range set active-probe 20.9.9.9 2000 set probe jitter frequency 4 target-port set active-probe jitter 20.9.9.9 target-port 2000 no set resolve utilization set probe frequency 4 set active-probe jitter 20.9.9.9 target-port 2000 set active-probe jitter 20.9.9.9 target-port 2000 set probe frequency 4 set active-probe jitter 20.9.9.9 target-port 2000 2000 BRKRST-2362 set active-probe jitter 20.9.9.9 target-port 2013

set periodic 90 set delay threshold 200 set loss threshold 50000 set jitter threshold 30 set mode monitor fast set resolve loss priority 2 variance 5 set resolve jitter priority 3 variance 5 set resolve delay priority 4 variance 5 no set resolve range no set resolve utilization set probe frequency 4

Cisco and/or its affiliates. All rights reserved.

Cisco Public

42

PfR Domain Summary

PfR is now multi-site aware
MC to MC Peering Framework can be used to exchange policies, services and feedback Target Discovery is just the first feature running over this framework Automatic configuration and generation of all jitter probes Reduces IP SLA target configuration per destination and per policy. Improves IP SLA probing efficiency through the sharing of probe data across multiple policies Possible future features: Solving the Asymmetric Routing issue PfR Policies distribution QoS Policies distribution

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

43

Internet Edge Load-Balancing

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Performance Routing Topologies

Internet Edge
Branch

ISP1

ISP2
MC/BR

WAN1
(IP-VPN)
BR BR

MC BR

MC/BR

WAN2
MC

BR

(IPVPN, DMVPN)

BR

MC/BR

HQ

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

45

Manual Traffic Engineering

Overview
Problem Statement
Ingress/Egress path are under/over utilized Maximize bandwidth utilization (uplinks with different BW) ISP X ISP Z

ISP Y

Manual Solution
Consider The Traffic Patterns of the Enterprise.
Does the Enterprise Host Content? Does the Enterprise Access Content?

ISP A

ISP B

Not Sure?
Graph Interface Byte Count Use NetFlow Even better, use Flexible NetFlow

Egress

1GE
R1

100M
R2

In General, sites have a 80:20 traffic volume (in bytes) mix. Fix only the 80% Direction.
20% direction doesnt matter unless links are widely varying speeds.
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Ingress

Cisco Public

46

Manual Traffic Engineering

BGP Policy Instruments
Tool/Attribute
Longest Match Local Preference AS_Path Communities

Direction of Traffic Flow Affected

Inbound and Outbound Outbound

Implementation
Static or Redistribution / Received
Direction Applied: Inbound Direction Applied: Outbound

Inbound
Inbound

Direction Applied: Outbound

Direction Applied, works together with Route Maps

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

47

Automatic Traffic Engineering Load-Balancing with PfR

Solution: PfR used to load balance the traffic New default policies based on load-balancing Cisco ASR1k is typical BR/MC with BR terminating WAN connections BGP routing BRs must be iBGP peers Default routing or Partial routes or Full routes PfR can actively manage up to 20k Prefixes concurrently (with ASR1000) 12.4T/15.0.1M IOS-XE 3.3.0
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

HQ

R3

iBGP
R4 R5

eBGP

eBGP

ISP1 ISP3 ISP4

ISP2

ISP5

Cisco Public

48

Egress Load Balancing

PfR Solution Used
R3

HQ

Learning

Dest Prefixes (NetFlow)

Passive Global
Egress BW eBGP
R4

iBGP
R5

Monitoring

55% 45%

eBGP

Policies Path Enforcement

Load-Balancing (range) BGP

Inject BGP Route

ISP1 ISP3 ISP4

ISP2

ISP5

BGP Local Pref

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

49

Internet Presence
Outbound Load Balancing Only
pfr master max-range-utilization percent 25 logging ! border 10.4.5.4 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external max-xmit-utilization percentage 90 ! border 10.4.5.5 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external max-xmit-utilization percentage 90 ! learn prefixes 1000 applications 0 expire after time 300 ! ! max prefix total 10000 learn 10000 exporter MYEXPORTER mode monitor passive periodic 600

Link Range Utilization Keep the usage on a set of exit links within a certain percentage range of each other

Max Link Utilization Upper threshold on the amount of traffic a specific link can carry

Max Prefixes Learn 1000 Prefixes Delete Prefix if not relearned in 60 Minutes

Global Policies Load Balancing enabled by default Link OOP if : % Util > Lowest + 10 % Util > 90 Revaluate Exit every 10 Minutes
2013 Cisco and/or its affiliates. All rights reserved.

BRKRST-2362

Cisco Public

50

Ingress Load Balancing

PfR Solution Used
R3

HQ

Learning

Inside Prefixes (BGP)

Passive Global
Ingress BW
eBGP
R4

iBGP
R5

Monitoring

20%

17%

eBGP

Policies Path Enforcement

Load-Balancing (range) BGP

BGP AS-PATH Prepend

ISP1 ISP3 ISP4

ISP2

ISP5

BGP Community

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

51

Internet Presence
Inbound Load Balancing
Learning Inside Prefix
pfr master policy-rules MYMAP max-range-utilization percent 25 logging ! [BR PROVISIONNING] ! learn inside bgp prefixes 1000 applications 0 expire after time 300 ! max prefix total 10000 learn 10000 max range receive percent 35 exporter MYEXPORTER mode monitor passive periodic 600 ! pfr-map MYMAP 10 match pfr learn inside !

Configuring Inside Prefix

pfr master policy-rules MYMAP max-range-utilization percent 10 logging ! [BR PROVISIONNING] ! learn prefixes 1000 expire after time 300 ! max prefix total 10000 learn 10000 max range receive percent 5 exporter MYEXPORTER mode monitor passive periodic 600 ! pfr-map MYMAP 10 match ip address prefix-list HQ_PREFIX inside ! ip prefix-list HQ_PREFIX seq 5 permit 10.10.1.0/24 ip prefix-list HQ_PREFIX seq 10 permit 10.10.2.0/24 ip prefix-list HQ_PREFIX seq 15 permit 10.10.3.0/24 ip prefix-list HQ_PREFIX seq 20 permit 10.10.4.0/24

Link Range Utilization - Ingress Keep the usage on a set of exit links within a certain percentage range of each other

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

52

Internet Presence
Downgrade Method
AS-PATH PREPEND BGP Community pfr master max-range-utilization percent 10 logging ! border 10.4.5.4 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external downgrade bgp community 3:100 ! border 10.4.5.5 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external downgrade bgp community 4:200 !

Nothing required

Downgrade Community Specific value per Provider

Check the following:

BRKRST-2362

Send-community enabled under bgp neighbor <ip> soft-reconfiguration inbound per neighbor max range receive percent X under pfr master
Cisco Public
53

2013 Cisco and/or its affiliates. All rights reserved.

PfR Master Controller Database

Display Traffic Classes Performance Metrics
R3#sh pfr master traffic-class OER Prefix Statistics: Pas - Passive, Act - Active, S - Short term, L - Long term, Dly - Delay (ms), P - Percentage below threshold, Jit - Jitter (ms), MOS - Mean Opinion Score Los - Packet Loss (packets-per-million), Un - Unreachable (flows-per-million), E - Egress, I - Ingress, Bw - Bandwidth (kbps), N - Not applicable U - unknown, * - uncontrolled, + - control more specific, @ - active probe all # - Prefix monitor mode is Special, & - Blackholed Prefix % - Force Next-Hop, ^ - Prefix is denied DstPrefix Appl_ID Dscp Prot SrcPort DstPort SrcPrefix Flags State Time CurrBR CurrI/F Protocol PasSDly PasLDly PasSUn PasLUn PasSLos PasLLos EBw IBw ActSDly ActLDly ActSUn ActLUn ActSJit ActPMOS ActSLos ActLLos -------------------------------------------------------------------------------20.20.0.0/24 N N N N NN INPOLICY 32 10.4.4.4 Et0/1 BGP 53 53 0 0 0 0 12 1 N N N N N N 20.20.8.0/24 53 N 20.20.16.0/24 105 N N N INPOLICY 53 0 N N N 42 0 N 0 N N NN 10.4.4.4 Et0/1 0 12 1 N BGP

Passive Delay (from TCP Syn/Ack)

No Active Delay Passive Mode used

N N N INPOLICY 59 105 0 0 N N N

N NN 10.5.5.5 Et0/1 0 0 27 N N

BGP 1

PfR has calculated the per traffic class (which is per /24 destination in this case) egress bandwidth usage as well as the ingress

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

54

PfR Master Controller Database

Traffic Class Performance
R3#sh pfr master traffic-class performance ip any 20.20.0.0/24 ==============================================================================================

Traffic-class: Destination Prefix : 20.20.0.0/24 Source Prefix : N/A Destination Port : N/A Source Port : N/A DSCP :N Protocol : N/A Application Name: : N/A
General: Control State : Controlled using BGP Traffic-class status : INPOLICY Current Exit : BR 10.4.4.4 interface Et0/1, Tie breaker was Range Time on current exit : 0d 0:4:49 Time remaining in current state : @10 seconds Traffic-class type : Learned Improper config : None Last Out-of-Policy event: No Out-of-Policy Event Average Passive Performance Current Exit: (Average for last 5 minutes) Unreachable : 0% -- Threshold: 50% Delay : 60% -- Threshold: 50% Loss : 0% -- Threshold: 10% Egress BW : 13 kbps Ingress BW : 1 kbps Time since last update : 0d 0:0:15

TC Performance Performance Details Last Resolver decisions Filter on any type of traffic

Average Active Performance Current Exit: (Average for last 5 minutes) Unreachable : 0% -- Threshold: 50% Delay : 50% -- Threshold: 50%
============================================================================================== R3#
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

55

Load Balancing
Checking Load Balancing Accuracy
R3#sh pfr master exits ============================================================================================== PfR Master Controller Exits:
General Info: ============= E - External I - Internal N/A - Not Applicable Up/ ID Name Border Interface ifIdx IP Address Mask Policy Type Down --- ------------ --------------- ----------- ----- --------------- ---- ----------- ---- ---2 10.4.4.4 Et0/1 2 100.4.81.4 24 Util E UP 1 10.5.5.5 Et0/1 2 100.5.82.5 24 Util E UP Global Exit Policy: =================== Cost: In Policy Exits Performance: ================== Egress Ingress ---------------------------------------------------- -----------------------------------ID Capacity MaxUtil Usage % RSVP POOL OOP Capacity MaxUtil Usage % OOP --- -------- -------- -------- --- -------------- ----- -------- -------- -------- --- ----2 1000 900 232 23 N/A N/A 1000 1000 42 4 N/A 1 1000 900 317 31 N/A N/A 1000 1000 16 1 N/A TC and BW Distribution: ======================= # of TCs BW (kbps) Probe Active Name/ID Current Controlled InPolicy Controlled Total Failed Unreach (count) (fpm) ---- ---------------------------- ---------------------- ------ -------2 23 23 23 509 232 0 0 1 26 26 26 1203 317 0 0 Exit Related TC Stats: ====================== Priority highest nth -----------------Number of TCs with range: 0 Number of TCs with util: 0 Number of TCs with cost: 0

HQ

R3

iBGP
R4 R5

eBGP

eBGP

ISP1 ISP3 ISP4

ISP2

ISP5

0 0 0

Total number of TCs:

R3#

59
2013 Cisco and/or its affiliates. All rights reserved.

BRKRST-2362

Cisco Public

56

Enterprise WAN
Application Based Routing

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Performance Routing Topologies

ISP1

ISP2

Enterprise WAN
MC/BR

Branch

WAN1
(IP-VPN)
BR BR

MC BR

MC/BR

WAN2
MC

BR

(IPVPN, DMVPN)

BR

MC/BR

HQ

Voice - Video Critical Application Rest of the Traffic

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

58

Enterprise WAN Use Case

Blackout and Brownout
Problem Statement:
Recent carrier routing problem cause a network outage (Blackout). Fluctuating performance over the WAN is causing intermittent application problems (Brownout) Secondary/Backup WAN path under utilized

Voice - Video
Critical Application Rest of the Traffic
MC

HQ

BR

BR

Voice, Video, Critical

The Rest of the Traffic

Solution: PfR Application based optimization

Protect Voice and Video traffic:
primary path, check delay, loss, jitter fallback secondary
WAN1 (IP-VPN) WAN2 (IPVPN, DMVPN)

Protect Business Applications:

primary path, check loss, utilization fallback secondary

Best effort Applications Maximize bandwidth utilization:

load balanced across SPs or use the secondary path
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

MC/BR

MC/BR

MC/BR

BR

Cisco Public

59

Deployment With Target Discovery (Recommended)

ISP-1 ISP-2
MC/BR

WAN1
(IP-VPN)
BR BR
MC

Voice - Video
MC/BR

Critical Application Rest of the Traffic

BR

WAN2
MC

BR

(IPVPN, DMVPN)

BR

HQ

MC/BR

Multisite MC Peering Framework MC to MC Peering Framework can be used to exchange policies, services and feedback Remote Site Discovery Simplifies Configuration prefix and target discovery Probing Efficiency sharing of probe data across policies
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

60

Enabling Target Discovery

Ease Configuration for Active Probing
The MC peering system must configure a source loopback interface with an IP address that is reachable (routed) through the network The Head-End MC is configured into unicast-listen mode and each of the Branch MCs will have a static unicast neighbor configuration with the HeadEnd MC peer address
MC

HQ

10.10.0.0/16

R3 LISTEN Shadow Router IP Sla Responder

10.3.3.3

SLA BR

Voice, Video, Critical

BR

The Rest of the Traffic

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

MC/BR

MC/BR

MC/BR

BR

BRANCH SETUP The peering to the head-end

BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

20.9.9.9

30.10.10.10

20.20.0.0/16
Cisco Public

30.30.0.0/16
61

Target Discovery
Configuration Example
! pfr master policy-rules MYMAP mc-peer head-end Loopback0 target-discovery <responder-list HQ_TARGET> <insideprefixes HQ_PREFIX> border 10.4.4.4 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external link-group SP1 ! border 10.5.5.5 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external link-group SP2 ! MC

HQ

10.10.0.0/16 10.3.3.3
SLA BR BR

Shadow Router IP Sla Responder

Voice, Video, Critical

The Rest of the Traffic

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

pfr master policy-rules MYMAP mc-peer 10.3.3.3 Loopback0 target-discovery

BRKRST-2362

MC/BR

MC/BR

MC/BR

BR

20.9.9.9
IOS 15.2(3)T

30.10.10.10

20.20.0.0/16
2013 Cisco and/or its affiliates. All rights reserved.

30.30.0.0/16
62

Cisco Public

Target Discovery
Hub Site
MC

HQ

10.10.0.0/16 10.3.3.3
SLA BR BR

R3#sh pfr master target-discovery PfR Target-Discovery Services Mode: Static Domain: 59501 Responder list: HQ_TARGET Inside-prefixes list: HQ_PREFIX SvcRtg: client-handle: 7 sub-handle: 6 pub-seq: 1
PfR Target-Discovery Database (local) Local-ID: 10.3.3.3 Desc: R3 Target-list: 10.4.5.5, 10.4.5.4 Prefix-list: 10.10.4.0/24, 10.10.3.0/24, 10.10.2.0/24, 10.10.1.0/24 PfR Target-Discovery Database (remote) MC-peer: 30.10.10.10 Target-list: 30.30.0.10 Prefix-list: 30.30.0.0/16 Desc: R10

Shadow Router IP Sla Responder

Voice, Video, Critical

The Rest of the Traffic

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

MC/BR

MC/BR

MC/BR

BR

MC-peer: 20.9.9.9 Desc: R9 Target-list: 20.20.0.9 Prefix-list: 20.20.0.0/16 R3#

20.9.9.9

30.10.10.10

20.20.0.0/16
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

30.30.0.0/16
63

Cisco Public

Target Discovery
Branch
MC

HQ

10.10.0.0/16 10.3.3.3
SLA BR BR

R10#sh pfr master target-discovery PfR Target-Discovery Services Mode: Dynamic Domain: 59501 SvcRtg: client-handle: 2 sub-handle: 1 pub-seq: 1
PfR Target-Discovery Database (local)

Shadow Router IP Sla Responder

Voice, Video, Critical

The Rest of the Traffic

Local-ID: 30.10.10.10 Desc: R10 Target-list: 30.30.0.10 Prefix-list: 30.30.0.0/16

PfR Target-Discovery Database (remote)

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

MC-peer: 20.9.9.9 Desc: R9 Target-list: 20.20.0.9 Prefix-list: 20.20.0.0/16

MC-peer: 10.3.3.3 Desc: R3 Target-list: 10.4.5.5, 10.4.5.4 Prefix-list: 10.10.4.0/24, 10.10.3.0/24, 10.10.2.0/24, 10.10.1.0/24 R10#
MC/BR MC/BR MC/BR

BR

20.9.9.9

30.10.10.10

20.20.0.0/16
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

30.30.0.0/16
64

Cisco Public

Step1 Learning Traffic

Defines your groups
Voice - Video
Traffic Classes
MC

HQ

Traffic Classes Definition

Prefixes Prefixes + DSCP Applications
EXAMPLE Critical Apps DSCP AF31 Mask 24 11 TCs Mask 16 2 TCs
WAN1 (IP-VPN)

SLA

Critical Application
BR BR

Rest of the Traffic

Voice, Video, Critical

The Rest of the Traffic

WAN2 (IPVPN, DMVPN)

Check Prefixes and application/DSCP per branch

This will give an idea of the number of Traffic Classes Tune Aggregation Mask as needed
BRKRST-2362

MC/BR

MC/BR

MC/BR

BR

10.10.1.0/24 10.10.2.0/24
2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

10.20.1.0/24 10.20.9.0/24

65

Traffic Class Learning

Configuration Example
Traffic Classes
pfr master ! MC

HQ

SLA BR BR

learn throughput ! list seq 10 refname LEARN_VOICE_VIDEO traffic-class access-list VOICE_VIDEO filter BRANCH_PREFIX aggregation-type prefix-length <LENGTH> throughput ! list seq 20 refname LEARN_CRITICAL traffic-class access-list CRITICAL filter BRANCH_PREFIX aggregation-type prefix-length <LENGTH> throughput
! [Rest of the traffic]

Voice, Video, Critical

The Rest of the Traffic

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

MC/BR

MC/BR

MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

66

Traffic Class Learning

Configuration Example
Access-List Traffic Automatically learn based on DSCP values Classes for Voice, Video and Critical Applications Rest of the Traffic falls under global learning (kind of default class)
! ip access-list extended VOICE_VIDEO permit ip any any dscp ef ! ip access-list extended CRITICAL permit ip any any dscp af31 !
MC

HQ

SLA BR BR

Voice, Video, Critical

The Rest of the Traffic

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

ip prefix-list BRANCH_PREFIX seq 5 permit 20.20.0.0/16 ip prefix-list BRANCH_PREFIX seq 10 permit 30.30.0.0/16 !

MC/BR

MC/BR

MC/BR

BR

Branch Prefixes Filters on remote prefixes

BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

67

Traffic Class Learning

Configuration Example with NBAR
MC

HQ

! pfr master learn list seq 10 refname VOICE_VIDEO traffic-class application nbar rtp-audio filter BRANCH thoughput list seq 20 refname CRITICAL traffic-class application nbar citrix filter BRANCH thoughput !

SLA BR BR

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

Use NBAR to identify Application Traffic

NBAR is activated automatically on BR NBAR Support was added in 12.4(20T) release
MC/BR MC/BR MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

68

Learning Traffic Classes

Example
MC#sh pfr master learn list Learn-List seq 10 refname LEARN_VOICE_VIDEO Configuration: Traffic-Class Access-list: VOICE_VIDEO Filter: BRANCH_PREFIX Aggregation-type: prefix-length 32 Learn type: throughput Session count: 1000 Max count: 1000 Policies assigned: 10 Status: ACTIVE Stats: Traffic-Class Count: 2 Traffic-Class Learned: Appl Prefix 20.20.1.0/24 ef 256 Appl Prefix 20.20.2.0/24 ef 256 Learn-List seq 20 refname LEARN_CRITICAL Configuration: Traffic-Class Access-list: CRITICAL Filter: BRANCH_PREFIX Aggregation-type: prefix-length 24 Learn type: throughput Session count: 50 Max count: 100 Policies assigned: 20 Status: ACTIVE Stats: Traffic-Class Count: 37 Traffic-Class Learned: Appl Prefix 20.20.14.0/24 af31 256 Appl Prefix 20.20.6.0/24 af31 256 Appl Prefix 30.30.5.0/24 af31 256 Appl Prefix 20.20.8.0/24 af31 256 Appl Prefix 30.30.14.0/24 af31 256 [SNIP]
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

HQ

MC SLA BR BR

Voice, Video, Critical

The Rest of the Traffic

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

MC/BR

MC/BR

MC/BR

BR

Cisco Public

69

Step2 Performance Measurement

PfR Solution Used
MC tells the BR to configure and generate all Jitter probes needed
Traffic Classes
MC

HQ

Monitoring
TD Enabled

Fast Voice/Video
NetFlow

SLA BR BR

Shadow Router IP Sla Responder

NetFlow

Active Critical Apps Passive Rest

Destination DSCP Prefix 10.1.1.0/24 10.1.10.0/24 EF AF31 App Id Delay 60 110 Jitter 10 20 Loss 0 0 Ingress BW 20 52 Egress BW 40 60 BR BR1 BR1 Exit Gi1/1 Gi1/2

Probes

Probes

WAN1 (IP-VPN)

WAN2 (IPVPN, DMVPN)

Destination Prefix 10.1.1.0/24 10.1.10.0/24

EF
DSCP App Id

89
Delay 60 110 89

35
Loss 0 0 1

34
Ingress BW 20 52 34

10
Egress BW 40 60 10

BR2
BR BR1 BR1 BR2

Gi1/1
Exit Gi1/1 Gi1/2 Gi1/1 MC/BR MC/BR MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

70

Step3 Policies
Define your policies per group
MC

HQ

Policies
Voice, Video, Critical
BR BR

SLA

Shadow Router IP Sla Responder The Rest of the Traffic

1. Link-Group

Voice - Video

2. Loss 3. Jitter 4. Delay 1. Link-Group

WAN1 (IP-VPN) WAN2 (IPVPN, DMVPN)

Critical Application

2. Loss 4. Delay
MC/BR MC/BR MC/BR

BR

Rest of the Traffic

Load-Balancing

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

71

Configuration Sample
Policies Voice and Video
Policies Thresholds Applied to the voice and video traffic Loss, delay and jitter Monitor mode fast Actively probe all exits to get performance metrics Policies Definition List all policies Assign priority Administrative policy: SP1 is the primary path, fallback to SP2 if OOP

pfr-map MYMAP 10 match pfr learn list LEARN_VOICE_VIDEO

set delay threshold 200 set loss threshold 50000 set jitter threshold 30 set mode monitor fast set resolve loss priority 2 variance 5 set resolve jitter priority 3 variance 5 set resolve delay priority 4 variance 5 set link-group SP1 fallback SP2 set probe frequency 4 set periodic 90
IOS 15.2(3)T

Jitter Probe Target Discovery is used No need to manually define the probe target

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

72

Configuration Sample
Policies Critical Applications
Policies Thresholds Applied to the voice and video traffic Loss, delay and jitter

pfr-map MYMAP 20 match pfr learn list LEARN_CRITICAL set delay threshold 120 set loss threshold 200000 set mode monitor active throughput set resolve delay priority 1 variance 20 set resolve loss priority 5 variance 10 set link-group SP1 fallback SP2 set probe frequency 4 set periodic 90

Monitor mode Active Actively probe the current exit to get performance metrics. Mode fast is also an option
Policies Definition List all policies Assign priority Administrative policy: SP1 is the primary path, fallback to SP2 if OOP

IOS 15.2(3)T

Active Probes Automatic configuration and generation of probes

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

73

Configuration Sample
Policies Rest of the Traffic

! pfr master policy-rules MYMAP max-range-utilization percent 22 ! mc-peer head-end Loopback0 target-discovery ! logging ! ! Default Policies ! mode route protocol pbr !
IOS 15.2(3)T

Link Range Utilization Keep the usage on a set of exit links within a certain percentage range of each other

Global Policies Apply for the rest of the traffic Load Balancing enabled by default

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

74

Target Discovery
Hub Site Jitter probes Generated
R3#sh pfr master active-probes target-discovery PfR Master Controller active-probes (TD) Border = Border Roter running this probe MC-Peer = Remote MC associated with this target Type = Probe Type Target = Target Address TPort = Target Port N - Not applicable Destination Site Peer Addresses: MC-Peer 30.10.10.10 20.9.9.9 Targets 30.30.0.10 20.20.0.9
WAN1 (IP-VPN)

HQ

10.10.0.0/16 10.3.3.3
SLA BR BR

MC

Shadow Router IP Sla Responder

Voice, Video, Critical

The Rest of the Traffic

WAN2 (IPVPN, DMVPN)

The following Probes are running: Border 10.4.4.4 10.4.4.4 10.5.5.5 10.4.4.4 10.4.4.4 10.5.5.5 R3#
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Idx 2 2 2 2 2 2

State MC-Peer TD-Actv 30.10.10.10 TD-Actv 30.10.10.10 TD-Actv 30.10.10.10 TD-Actv 20.9.9.9 TD-Actv 20.9.9.9 TD-Actv 20.9.9.9

Type Target TPort jitter 30.30.0.10 5000 jitter 30.30.0.10 5000 jitter 30.30.0.10 5000 jitter 20.20.0.9 5000 jitter 20.20.0.9 5000 jitter 20.20.0.9 5000

MC/BR

MC/BR

MC/BR

BR

20.9.9.9

30.10.10.10

20.20.0.0/16
Cisco Public

30.30.0.0/16
75

PfR Master Controller Database

Display Traffic Classes Performance Metrics
MC#sh pfr master traffic-class OER Prefix Statistics: Pas - Passive, Act - Active, S - Short term, L - Long term, Dly - Delay (ms), P - Percentage below threshold, Jit - Jitter (ms), MOS - Mean Opinion Score Los - Packet Loss (packets-per-million), Un - Unreachable (flows-per-million), E - Egress, I - Ingress, Bw - Bandwidth (kbps), N - Not applicable U - unknown, * - uncontrolled, + - control more specific, @ - active probe all # - Prefix monitor mode is Special, & - Blackholed Prefix % - Force Next-Hop, ^ - Prefix is denied DstPrefix Appl_ID Dscp Prot SrcPort DstPort SrcPrefix Flags State Time CurrBR CurrI/F Protocol PasSDly PasLDly PasSUn PasLUn PasSLos PasLLos EBw IBw ActSDly ActLDly ActSUn ActLUn ActSJit ActPMOS ActSLos ActLLos -------------------------------------------------------------------------------30.1.8.0/24 N defa 256 N N 0.0.0.0/0 INPOLICY 17 10.4.5.4 Et0/1 PBR 52 52 0 0 0 0 65 7 53 52 0 0 N N N N 30.1.1.0/24 52 54 30.1.9.0/24 52 52 30.1.3.0/24 52 52 30.1.4.0/24 52 54 N ef 256 N N 0.0.0.0/0 INPOLICY 41 10.4.5.5 Et0/1 52 0 0 0 0 66 7 51 0 0 N N N N N defa 256 N N 0.0.0.0/0 INPOLICY @47 10.4.5.4 Et0/1 52 0 0 0 0 64 7 51 0 0 N N N N N af31 256 N N 0.0.0.0/0 INPOLICY 113 10.4.5.4 Et0/1 52 0 0 0 0 65 7 52 0 0 N N N N PBR

Passive Delay (from TCP Syn/Ack)

Active Delay (from probes)

PBR

PBR

N af31 256 N N 0.0.0.0/0 INPOLICY 75 10.4.5.5 Et0/1 PBR 52 0 0 0 0 64 7 51 0 0 N N N N

PfR has calculated the per traffic class (which is per /24 destination in this case) egress bandwidth usage as well as the ingress

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

76

PfR Master Controller Database

Traffic Class Performance
MC#sh pfr master traffic-class performance ip any 20.20.0.0/24 dscp ef ============================================================================================== Traffic-class: Destination Prefix : 20.20.0.0/24 Source Prefix : 0.0.0.0/0 Destination Port : N Source Port :N DSCP : ef Protocol : 256 Application Name: : N/A General: Control State : Controlled using PBR Traffic-class status : INPOLICY Current Exit : BR 10.4.4.4 interface Et0/1, Tie breaker was None Time on current exit : 0d 0:9:32 Time remaining in current state : @34 seconds Last uncontrol reason : Probe frequency changed Time since last uncontrol : 0d 1:12:1 Traffic-class type : Learned Improper config : None Last Out-of-Policy event: No Out-of-Policy Event Average Passive Performance Current Exit: (Average for last 5 minutes) Unreachable : 0% -- Threshold: 50% Loss : 0 ppm -- Threshold: 50000 ppm Delay : 52 msecs -- Threshold: 200 msecs Egress BW : 7 kbps Ingress BW : 5 kbps Time since last update : 0d 0:0:24 Average Active Performance Current Exit: (Average for last 5 minutes) Unreachable : 0% -- Threshold: 50% Loss : 0 ppm -- Threshold: 50000 ppm Jitter : 0 msec -- Threshold: 3000 msec Delay : 52 msec -- Threshold: 200 msec Last Resolver Decision: BR Interface Status Reason Performance Threshold --------------- ------------ ------------ ------------ ----------- --------10.5.5.5 Et0/1 Eliminated Link Group N/A N/A 10.4.4.4 Et0/1 Best Exit Unreachable N/A N/A ============================================================================================== MC# BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

TC Performance Performance Details Last Resolver decisions Filter on any type of traffic

Cisco Public

77

Enforcing the Path

Dynamic PBR MC instructs BRs to apply Dynamic Route Maps on their inside interfaces Direct connection between BRs is needed
Direct connection GRE/mGRE tunnel
WAN1 (IP-VPN)

HQ

MC

Voice, Video, Critical

BR

BR

The Rest of the Traffic

WAN2 (IPVPN, DMVPN)

MC/BR

MC/BR

MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

78

PBR Example
Troubleshooting Path Enforcement
R4#sh route-map dynamic route-map OER_INTERNAL_RMAP, permit, sequence 0, identifier 402653185 Match clauses: ip address (access-lists): oer#1 Set clauses: ip next-hop 10.4.5.5 interface Ethernet0/0 Policy routing matches: 98809 packets, 67823285 bytes route-map OER_INTERNAL_RMAP, permit, sequence 1, identifier 3439329282 Match clauses: ip address (access-lists): oer#2 Set clauses: ip next-hop 100.4.8.8 interface Ethernet0/1 Policy routing matches: 75247 packets, 51786342 bytes Current active dynamic routemaps = 1 R4#

A dynamic route-map is created by PfR to enforce traffic to the chosen exit point Next hop = R5 A dynamic route-map is created by PfR to enforce traffic to the chosen exit point Next hop = WAN

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

R4#sh ip access-lists dynamic Extended IP access list oer#1 67108863 permit ip any 30.1.6.0 0.0.0.255 dscp af31 (17981 matches) 134217727 permit ip any 30.1.4.0 0.0.0.255 dscp af31 (17984 matches) 268435455 permit ip any 30.1.5.0 0.0.0.255 dscp af31 (17975 matches) 536870911 permit ip any 30.1.2.0 0.0.0.255 dscp ef (17938 matches) 1073741823 permit ip any 30.1.1.0 0.0.0.255 dscp ef (18002 matches) Extended IP access list oer#2 67108863 permit ip any 30.1.3.0 0.0.0.255 dscp af31 (15000 matches) 134217727 permit ip any 30.1.7.0 0.0.0.255 dscp default (15034 matches) 268435455 permit ip any 30.1.10.0 0.0.0.255 dscp default (15958 matches) 536870911 permit ip any 30.1.9.0 0.0.0.255 dscp default (15937 matches) 1073741823 permit ip any 30.1.8.0 0.0.0.255 dscp default (15966 matches) R4#
Cisco Public

Design Considerations
Scalability on the hub
Number of jitter probes
CPU impact on the BRs
Traffic Classes
MC

HQ

SLA BR BR

Shadow Router IP Sla Responder

Probe Frequency
CPU impact on the BRs

Probes

Probes

Number of Traffic Classes

TCAM space on ASR1k BRs CPU/ Memory usage on MC
WAN1 (IP-VPN) WAN2 (IPVPN, DMVPN)

How fast is Fast Mode?

Min Probe Frequency: 2 sec
MC/BR MC/BR MC/BR

BR

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

80

Network Management
NetFlow v9 Export

SNMP MIB

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

PfR NetFlow v9 Export

PfR NetFlow v9 Export
The PfR NetFlow v9 Exporter resides on the PfR Master Controller Exports Passive Metrics, Active Metrics, Events, Configuration
BR

HQ

NetFlow
MC

SNMP Read
BR

PfR SNMP MIB (Traps coming)

TC as a row of cpfrTrafficClassTable TC Status as a row of cpfrTrafficClassStatusTable performance metrics as a row of cpfrTrafficClassMetricsTable
WAN1 (IP-VPN) WAN2 (IPVPN, DMVPN)

NMS application vendors engaged!

MC/BR

MC/BR

MC/BR

BR

Cisco Prime Assurance engaged!

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

82

Traffic Class Identifier

Each TC will have its own Identifier Traffic-Class ID is encoded into application tag as follows:

Classification Engine ID (1 Byte) Classification Engine ID: 17

Type (1 Byte) Type: Type=1: Passive Update, Type=2: Passive Performance, Type=3: Active Update, Type=4: Active Performance, Type=5: Traffic-Class Event

Traffic Class Id (4 Bytes)

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

83

Configuration Templates Options

Traffic Class Config Encoded Traffic-Class ID, Source prefix, Source mask, Destination prefix, Destination mask, Protocol, DSCP, Source port min, Source port max, Destination port min, Destination port max, Application Name, Policy id External Interface Config
Internal Interface Config

BR address Internal interface id BR interface name

Policy Config

BR address, External interface id, BR interface name, Link capacity, RSVP reserved bandwidth pool, Maximum ingress bandwidth, Maximum egress bandwidth, BGP Community Link group Name, Cost nickname, Cost type Cost discard rollup count

Policy id pfr-map name State Threshold Priority Variance MOS Link group Name

Reason Config

Reason id Reason text

http://docwiki.cisco.com/wiki/AVC-Export:PfR
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

84

Passive/Active Reports
Passive Update Encoded Traffic-Class ID BR Address, External interface ID, Direction State First switched time Last switched time Sum of round-trip-time, Samples, Loss, Unreachable, Bytes, Packets, Flows, Next hop address Passive Performance Encoded Traffic-Class ID BR Address, External interface ID Direction Routing protocol State Reason ID Timestamp Left time Passive short-term unreachable Passive short-term loss Passive short-term round-trip-time Passive long-term unreachable Passive long-term loss Passive long-term round-trip-time Egress bandwidth Ingress bandwidth BGP Prepend BGP Community Active Update Encoded Traffic-Class ID BR Address External Interface ID State Average round-trip-time Minimum round-trip-time Maximum round-trip-time Sum of round-trip-time Unreachable Loss Jitter MOS below counts MOS total counts Initiations Completes Passive Performance Encoded Traffic-Class ID BR Address External interface ID Routing protocol State Reason ID Timestamp Left time Active short-term unreachable Active short-term loss Active short-term round-trip-time Active short-term jitter Active Short-term MOS Active long-term unreachable Active long-term loss Active long-term round-trip-time

http://docwiki.cisco.com/wiki/AVC-Export:PfR
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

85

Interface Update, Various

External Interface Update BR Address External interface ID State Reason ID Egress bandwidth Ingress bandwidth Cost target bandwidth RSVP bandwidth pool TC total counts Controlled TC counts In Policy TC counts Controlled bandwidth Internal Interface Update BR Address Internal interface ID State Reason ID Traffic Class Performance Encoded Traffic-Class ID BR address External interface ID Direction Routing protocol State Reason ID Timestamp Left time BGP Prepend BGP Community Cost Minimization Performance BR Address External interface ID State Timestamp Counter of left rollup Ingress rollup bandwidth (kbps) Egress rollup bandwidth (kbps) k-th rollup bandwidth (kbps) k-th tier percentage (%) k-th fee

BR Up/Down Alert BR address State

http://docwiki.cisco.com/wiki/AVC-Export:PfR
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

86

Leverage NetFlow Partners Eco-system

Traffic Analysis

Denial of Service

Billing

More info: http://www.cisco.com/warp/public/732/Tech/nmp/netflow/partners/commercial/

BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

87

PfR Reporting Tools

NetFlow Partners Plixer, ActionPacked
Cisco Prime Infrastructure 2.x Future (H1/CY13)

LiveAction
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

88

Conclusion
Platform Support

Software Packages
URLs

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

PfR Platform Support

Cisco ASR 1000 BR in IOS-XE 2.6.1 MC in IOS-XE 3.3.0

Cisco 3900
Cisco 3900 Cisco 3800 12.4, 12.4T 15M/T
Cisco 6500
12.2(33)SXH (Deprecated)

Cisco 2900
Cisco 1900 Cisco 1800 12.4, 12.4T, 15M/T Cisco 2800 12.4, 12.4T 15M/T

Cisco 3800 12.4, 12.4T 15M/T

Cisco 7600
12.2(33)SRB (Limited Support)

Cisco 7200-NPE-G2, 7301 12.4, 12.4T 15M/T

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

90

PfR Software Packaging

New Cisco ISR G2 Simplified Feature Sets
New ISR-G2 1900, 2900, 3900 A single IOS Universal Image for all ISR Generation 2

Classic Cisco IOS Software Feature Sets

Existing ISR 1800, 2800, 3800, 7200

ISR Platforms PfR is within the DATA package.

PfR

ASR 1000 Series

ASR1001: Use Universal Image (U or UK9) with Advanced IP Services (AIS) or Advanced Enterprise

Services (AES) technology package license All other ASR1000 (ASR1002-F, ASR1002, ASR1004, ASR1006, ASR1013): Use Advanced IP Services (AIS/AISK9) or Advanced Enterprise Services (AES/AESK9) images
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

91

Technical References
Docwiki Performance Routing Home
Technology Overview, Solution Guides, Troubleshooting Guides, FAQ
http://docwiki.cisco.com/wiki/PfR:Home

Performance Routing Technology Overview

http://docwiki.cisco.com/wiki/PfR:Technology_Overview

Performance Routing Solution Guides

http://docwiki.cisco.com/wiki/PfR:Solutions

Performance Routing Troubleshooting Guide

http://docwiki.cisco.com/wiki/PfR:Troubleshooting

Configuration
Understanding Performance Routing
http://www.cisco.com/en/US/docs/ios/pfr/configuration/guide/pfr-understand.html

Basic Configuration
http://www.cisco.com/en/US/docs/ios/pfr/configuration/guide/pfr-basic.html

Advanced Configuration
http://www.cisco.com/en/US/docs/ios/pfr/configuration/guide/pfr-advanced.html

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

92

Recommended Reading for BRKRST-2362

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

93

Call to Action

Visit the Cisco Campus at the World of Solutions

to experience Cisco innovations in action

Get hands-on experience attending one of the Walk-in Labs Schedule face to face meeting with one of Ciscos engineers at the Meet the Engineer center Discuss your projects challenges at the Technical Solutions Clinics

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

94

BRKRST-2362

2013 Cisco and/or its affiliates. All rights reserved.

Cisco Public

95