Lab Guide

Advanced Network Automation Solutions using

Cisco IOS EEM

LABNMS-2001

Arie Vayner, [email protected]

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
Advanced Network Automation Solutions using Cisco IOS EEM
LABNMS-2001

Contents
Advanced Network Automation Solutions using Cisco IOS EEM ............................... 2
LABNMS-2001 ........................................................................................................ 2
Session Abstract ..................................................................................................... 2
Introduction to EEM ................................................................................................ 3
EEM References..................................................................................................... 3
EEM Debugging Commands .................................................................................. 4
Lab Structure .......................................................................................................... 5
Task 1 – Block a CLI Command ............................................................................. 6
Task 2 – Control CLI Command Execution Rate .................................................... 8
Task 3 – Scheduling Events ................................................................................... 9
Task 4 – Manually Triggered EEM Scripts............................................................ 11
Task 5 – Monitor Interface Parameters................................................................. 12
Task 6 – Switch between Primary and Backup Paths........................................... 15
Task 7 – Consolidated Custom Status Command ................................................ 20
Task 8 – Secure Automatic Provisioning .............................................................. 23
Appendix I ................................................................................................................ 28

Session Abstract
In this session we will review advanced automation and manageability solutions based
on Cisco IOS Embedded Event Manager (EEM) functionality. The session will allow
delegates to gain hands-on experience of the implementation of advanced solutions
including high availability, network performance optimization, network monitoring and
efficient automation. The session would provide a relevant tool set for enhancing
network operations within networks built on Cisco IOS based routers and switches.

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
Introduction to EEM
Cisco IOS Embedded Event Manager (EEM) is a powerful and flexible subsystem that
provides real-time network event detection and onboard automation. It gives you the
ability to adapt the behaviour of your network devices to align with your business
needs.

Your business can benefit from the capabilities of IOS Embedded Event Manager
without upgrading to a new version of Cisco IOS Software. It is available on a wide
range of Cisco platforms.
Figure 1 Event Detectors

Syslog SNMP Watchdog Counter CLI

OIR None ERM EOT RF

GOLD SNMP Proxy XML RPC Routing Netflow

Neighbor
IP SLA SNMP Object Discovery Identity MAC

IOS Embedded Event Manager supports more than 20 event detectors that are highly
integrated with different Cisco IOS Software components to trigger actions in response
to network events.
Figure 2 EEM Architecture

EEM
Server Subsystem

EEM EEM
Event Detector Policies

Your business logic can be injected into network operations using IOS Embedded
Event Manager policies. These policies are programmed using either simple
command-line interface (CLI) or using a scripting language called Tool Command
Language (Tcl).

Harnessing the significant intelligence within Cisco devices, IOS Embedded Event
Manager helps enable creative solutions, including automated troubleshooting, fault
detection, and device configuration.

EEM References
Embedded Event Manager Overview
http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_eem_overvie
w_ps10591_TSD_Products_Configuration_Guide_Chapter.html

Writing Embedded Event Manager Policies Using Cisco IOS CLI Configuration
Guide

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_eem_policy_
cli.html

Embedded Event Manager (EEM) Scripting Community (Cisco Beyond)

http://forums.cisco.com/eforum/servlet/EEM?page=main

Relevant Command Reference Guides

http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_01.html
http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_02.html
http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_05.html
http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_06.html

EEM Debugging Commands

The following commands can be used to debug and display the operations of the
different scripts used in this lab:
 debug event manager action cli
 debug event manager detector <event detector type>
 show event manager detector <event detector type> detailed
 show event manager policy registered
 show track <id>

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
Lab Structure
The different tasks in this lab provide introduction to different elements of Embedded
Event Manager applet programming.

Each task would present a problem that may solve some operational challenge,
providing an example using an EEM CLI applet. Each applet would present new
programming elements and tools available in EEM.

It is recommended to try and configure the different examples in the lab, and then try
executing them (see the output examples). It is also highly recommended to turn on
relevant debugging commands (see the above debugging reference) and use the
different show commands provided above.
Feel free to experiment and modify the applets to create more advanced solutions.

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
Task 1 – Block a CLI Command

Goal: Use an EEM applet to block the “show beep ...” command

Script Logic:
 Use the “event cli pattern” event detector to catch any relevant command by
matching a regular expression.
 The syslog action allows publishing a customer syslog event
 The puts actions writes a string to the active terminal
 Setting the _exit_status variable to “0” would block the executed command
(“1” would allow the original command to run after script execution)

Introduced EEM Elements:

 event cli
 action syslog
 action puts
 Using _exit_status

Example:

!
event manager applet BLOCK-CLI-SHOW-BEEP
event cli pattern "^show beep" mode "exec" enter
action 1.0 syslog msg "BEEP!"
action 1.1 puts "BEEP!"
action 2.0 set _exit_status "0"
!

Output Example:

Router#show beep
BEEP!

Router#
*Nov 24 20:58:18.424: %HA_EM-6-LOG: BLOCK-CLI: BEEP!

Tip:
If the cli pattern is in another mode than “exec” it is possible to identify the mode by
running “debug event manager all” and execute the required command:

Router#debug event manager all

Router#show beep
…
*Nov 24 20:56:29.856: check_eem_cli_key: line=show beep mode=exec

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
Tip 2:
Using _exit_status=0 allows creating new CLI commands. Matching for an undefined
CLI string can trigger a script which can perform custom tasks. A common example
is combining the partial (using | include) output of multiple show commands creating
an overview “show status” command (see Error! Reference source not found.)

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
Task 2 – Control CLI Command Execution Rate

Goal: Use an EEM applet to control the rate allowed for any “show” command

Script Logic:
 Adding the “occurs 3 period 10” to the “event cli” detector allows triggering the
script only if it occurs at least 3 times in a period of 10 seconds

Introduced EEM Elements:

 event ... occurs ... period
 puts nonewline

Example:

!
event manager applet TOO-FAST-SHOW
event cli pattern "show" sync yes occurs 3 period 10
action 1.0 puts nonewline "Too fast show commands, please slow down"
action 2.0 set _exit_status "0"
!

Output Example:

Router#show clock
*21:14:07.560 CET Tue Nov 24 2009
Router#show clock
*21:14:09.720 CET Tue Nov 24 2009
Router#show clock
Too fast show commands, please slow down

Tip:
The “occurs X period Y” option is available on other event detectors, including syslog
events, track objects etc. It can be used to detect different kinds of repeating events
allowing detection of events such as link flaps, routing update flapping etc.

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
Task 3 – Scheduling Events

Goal: Use an EEM applet to execute scripts at specific time of day. The example script
would perform a daily configuration backup task.

Script Logic:
 Script is executed every day, Monday to Friday, at 23:55
 “enable” has to be executed as the script is executed in a separate VTY, and
starts in low privilege level

Introduced EEM Elements:

 event timer cron

Example:

!
event manager applet PERIODIC-CONFIG-SAVE
event timer cron name CONFIG-SAVE-TIMER cron-entry "55 23 * * 1-5"
action 1.0 cli command "enable"
action 2.0 cli command "copy running-config startup-config"
!

Note This script has a minor mistake which breaks its operation. This is done on
purpose.
If you do not see the issue, try using “debug event manager cli” to see how
the CLI interaction is working, and where it breaks.
If you are unable to find the issue consult Appendix I for the detailed
solution

Tips:
 The cron scheduler syntax is: “Minute, Hour, Day, Month, Day of Week”
(similar to unix cron syntax)
o Some shortcuts are available (@hourly, @weekly etc)
o Full details can be reviewed at
http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm
_06.html#wp1157622
 In order to test the script you can change the timer to save the config every
minute (60 seconds) by using the following event timer CLI:
event timer watchdog time 60

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved

Note EEM scripts do not pass authentication. If the script is in the configuration,
it means that the person who configured it had full config rights

CLI commands executed by an EEM script (“action cli command”) can get
authorized (for example in case of per-command authorization with
TACACS).
The username used for authorization is set using the “event manager
session cli username <username>” command

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
Task 4 – Manually Triggered EEM Scripts

Goal: Use an EEM applet to create a single line command to perform “clear counters”
hiding the [confirm] prompt

Script Logic:
 “event none” allows manual execution of the script from a CLI command using
“event manager run <script-name>”
 The “pattern” keyword allows catching a string written to the VTY terminal
 Using the alias command will allow easy execution of the new command

Introduced EEM Elements:

 “event none” and manually running EEM applets
 Matching CLI prompts (pattern)
 Using a CLI alias to run EEM applets

Example:

!
event manager applet CLEAR-COUNTERS
event none
action 1.0 cli command "enable"
action 2.0 cli command "clear counters" pattern "\[confirm\]"
action 3.0 cli command "y"

!
alias exec cc event manager run CLEAR-COUNTERS
!

Tips:
 “debug event manager action cli” would allow seeing the operation of the scripts
 Note the escape sequence “\[“ instead of just using a “[“ character. The “[“ (and
“]”) have a special meaning in regular expressions, and have to be escaped
using “\”.

Output Example:

Router#cc

Router#
*Nov 25 09:02:09.517: %CLEAR-5-COUNTERS: Clear counter on
all interfaces by on vty0 (EEM:CLEAR-COUNTERS)

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
Task 5 – Monitor Interface Parameters

Goal: Use an EEM applet to react to crossing a counter threshold on an interface.

This script would monitor the input bit per second (BPS) and packet per second (PPS)
counters of an interface, and if the exceed a given threshold, an alert would be
generated.

Script Logic:
 Event tags define different events that may be correlated in order to have a
combined trigger event for the applet.
 The “trigger” statement defines the correlation between the 2 different events.
 We would be monitoring two different events:
o Ethernet0/0 receive BPS rate crossing 128000 bps
o Ethernet0/1 receive PPS rate crossing 100 pps
 If the above thresholds have been crossed, we also want to know when the
issue has been resolved, so if BPS rate on E0/0 drops below 32000 bps, or
PPS rate on E1/0 drops below 50, an exit event would be triggered
 The “exit-event true” statement in the events would trigger the script also when
the lower threshold has been reached
 We use the boolean system variable $_interface_exit_event to detect if the
event is an “entry” or “exit” event (high or low threshold). This is done using an
“if” action.
 For reporting the event, we use some other system variables which are
populated automatically when an “interface” event is triggered.

Introduced EEM Elements:

 event interface name
 multiple events and event correlation triggers
 using event detector specific system variables
 using the if/else conditional syntax

Example:

!
event manager applet MONITOR-INTERFACES

event tag ETH0-0-RX-BPS interface name Ethernet0/0 parameter

receive_rate_bps entry-op gt entry-val 128000 entry-type value exit-op lt
exit-val 32000 exit-type value exit-event true poll-interval 1

event tag ETH0-1-RX-PPS interface name Ethernet1/0 parameter

receive_rate_pps entry-op gt entry-val 100 entry-type value exit-op lt
exit-val 50 exit-type value exit-event true poll-interval 1
trigger
correlate event ETH0-0-RX-BPS or event ETH0-1-RX-PPS
action 1.0 if $_interface_exit_event eq 0
action 1.1.1 syslog priority alerts msg "RED ALERT: $_interface_name
$_interface_parameter is $_interface_value"

action 1.2 else

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
 action 1.3.1 syslog priority informational msg "GREEN ALERT:
$_interface_name $_interface_parameter is $_interface_value"

action 1.4 end

!

Output Example:

*Jul 29 08:34:32.979: %HA_EM-1-LOG: MONITOR-INTERFACES: RED

ALERT: Ethernet0/0 receive_rate_pps is 320

How to Test:
In order to generate traffic on the links being monitored by the script, it is possible to
log in into the opposite router on the other side of the link, and use the ping command
to generate traffic.
In order to generate a high rate of packets, it is recommended to use an extended ping
command using a high count of ping packets and a timeout of 0. This will make the
opposite router to generate a high rate of packets.
In order to get a high BPS rate, it is also possible to increase the ping packet size.

R102#ping
Protocol [ip]:
Target IP address: 10.1.5.100
Repeat count [5]: 10000
Datagram size [100]: 1000
Timeout in seconds [2]: 0
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 10000, 1000-byte ICMP Echos to 10.1.5.100, timeout is 0
seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

If the ping operation is taking too long to complete (due to a high repeat count), it is
possible to break it using the break sequence CTRL-SHIFT-6.

Tips:
 In order to see which system variables are available for each of the different
event detectors it is possible to use the “show event manager detector <name>
detailed”

Router#show event manager detector interface detailed

....
Applet Built-in Environment Variables:
$_event_id
$_event_type
$_event_type_string
$_event_pub_time
LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
 $_event_pub_sec
$_event_pub_msec
$_event_severity
$_interface_name
$_interface_parameter
$_interface_is_increment
$_interface_value
$_interface_delta_value
$_interface_exit_event

 It is possible to use the “elseif” action for more conditions in the “if” structure.

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
Task 6 – Switch between Primary and Backup Paths

Goal: Use an EEM applet to react on primary link failure and restoration.
 When the primary path fails, the backup path should be activated
 When the primary path is restored, the backup path should be deactivated, but
only after verifying the primary path has fully converged.

Script Logic:
Figure 3 Logical Lab Topology

Primary
Lo0: 10.10.10.100/32
2001::100/128
R100
E0/0-E0/3
10.1.1-4.100/24

R102

10.1.1-4.101/24
E0/0-E0/3

Lo0: 10.10.10.101/32
2001::101/128
R101

Backup
 On the spoke router, R102, a pair of tunnels are used to provide connectivity to
the remote hub sites. Tunnel100 is the primary tunnel, and should be enabled
at all times. Tunnel101 is the backup tunnel, and is kept shutdown as long as
the primary path is active.
 One script should detect the primary path failing, and enable (no shut) Tunnel
101
 As soon as the primary path is restored, we should probe the primary path and
make sure it is restored, and only then shut down Tunnel101 again. This is
done by another EEM script.
 Track object 10 tracks the line protocol of Tunnel 100 (note that Tunnel 100 is
configured with Keepalive enabled).
 Track 10 going down would trigger “ENABLE-BACKUP-PATH”, which would
perform “no shut” to interface Tunnel101
 When Tunnel100 recovers, the directly connected subnet 10.20.20.0/24 would
be added into the routing table, triggering the “event routing” in “RECOVER-
PRIMARY-PATH”.
 The script would use a “while” loop to ping the remote side of Tunnel100, and
only after it becomes available would perform “shut” for interface Tunnel101.
 The script uses a “regexp” action to match the “!!!!” output of the ping command.

Introduced EEM Elements:

 track objects and the track event detector
LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
  event routing event detector
 using the “regexp” action
 using while loops
 using the _cli_result system variable

Example:

!
track 10 interface Tunnel100 line-protocol
!
event manager applet ENABLE-BACKUP-PATH
event track 10 state down
action 001 cli command "enable"
action 002 cli command "conf t"
action 003 cli command "int tunnel 101"
action 004 cli command "no shut"
action 005 syslog priority alerts msg "PRIMARY LINK IS
DOWN. BACKUP LINK ACTIVATED"
!
event manager app RECOVER-PRIMARY-PATH
event routing network 10.20.20.0/24 type add maxrun 30
action 001 cli command "enable"
action 002 set done 0
action 003 while $done ne 1
action 004 wait 5
action 005 cli command "ping 2001:20:20::100"
action 006 regexp "!!!!!" "$_cli_result"
action 007 if $_regexp_result eq 1
action 008 cli command "config t"
action 009 cli command "int Tunnel101"
action 010 cli command "shut"
action 011 cli command "end"
action 012 set done 1
action 013 end
action 014 end
action 015 syslog priority alerts msg "PRIMARY LINK IS
RESTORED. BACKUP LINK DEACTIVATED"
!

Tips:
 The “routing” event detector can detect not only specific prefixes, but any prefix
which falls inside a predefined subnet range. For the complete syntax please
refer to:
http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_06.ht
ml#wp1156862
 The “track” object can be enhanced to delay any down or up event from
propagating into the triggered event by configuring the “delay up/down”
command under the track object configuration. This can allow more advanced
policies which trigger events only after a certain condition is stable for a while.
The “show track <id>” command would show how much time left for the state
delay to expire.
http://www.cisco.com/en/US/docs/ios/ipapp/command/reference/iap_t1.html#
wp1163288

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
 Stub tracking objects can be configured to be used through the “track set/read”
EEM actions. These track objects can maintain states inside or between
different runs of EEM applets using the EEM actions “track set” and “track read”.
http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_21.ht
ml#wp1098882

Track objects can also track ip routes and IP SLA probes

(http://www.cisco.com/en/US/docs/ios/ipapp/command/reference/iap_t1.html#
wp1163396)
(http://www.cisco.com/en/US/docs/ios/ipapp/command/reference/iap_t1.html#
wp1163503)

Multiple track objects can be combined to create complex logical conditions and
trigger EEM applets with the “track list” command:
http://www.cisco.com/en/US/docs/ios/ipapp/command/reference/iap_t1.html#
wp1163622

In order to get faster response from track object events the “track timer”
command should be used.
http://www.cisco.com/en/US/docs/ios/ipapp/command/reference/iap_t1.html#
wp1158894

Output Example:

R102#debug event manager action cli

Debug EEM action cli debugging is on
R102#debug event manager detector routing
Debug EEM Routing Event Detector debugging is on

(Perfromed “shut” on interface Tunnel100 on R100)

R102#
R102#
*Nov 24 09:14:45.438: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel100,
changed state to down
*Nov 24 09:14:45.438: %TRACKING-5-STATE: 10 interface Tu100 line-protocol Up->Down
*Nov 24 09:14:45.438: %DUAL-5-NBRCHANGE: EIGRP-IPv6 1: Neighbor
FE80::A8BB:CCFF:FE00:6400 (Tunnel100) is down: interface down
*Nov 24 09:14:45.446: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.20.20.100
(Tunnel100) is down: interface down

...

*Nov 24 09:14:45.466: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : CTL :

cli_open called.
*Nov 24 09:14:45.478: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : OUT :
R102>
*Nov 24 09:14:45.478: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : IN :
R102>enable
*Nov 24 09:14:45.598: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : OUT :
R102#
*Nov 24 09:14:45.598: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : IN :
R102#conf t
*Nov 24 09:14:45.738: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : OUT :
Enter configuration commands, one per line. End with CNTL/Z.
*Nov 24 09:14:45.738: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : OUT :
R102(config)#

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
*Nov 24 09:14:45.738: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : IN :
R102(config)#int tunnel 101
*Nov 24 09:14:45.990: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : OUT :
R102(config-if)#
*Nov 24 09:14:45.990: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : IN :
R102(config-if)#no shut
*Nov 24 09:14:46.126: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : OUT :
R102(config-if)#
*Nov 24 09:14:46.126: %HA_EM-1-LOG: ENABLE-BACKUP-PATH: PRIMARY LINK IS DOWN. BACKUP
LINK ACTIVATED
*Nov 24 09:14:46.126: %HA_EM-6-LOG: ENABLE-BACKUP-PATH : DEBUG(cli_lib) : : CTL :
cli_close called.
R102#
*Nov 24 09:14:46.130: %SYS-5-CONFIG_I: Configured from console by vty0
R102#
*Nov 24 09:14:48.054: %LINK-3-UPDOWN: Interface Tunnel101, changed state to up
*Nov 24 09:14:48.070: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel101,
changed state to up
R102#
R102#
R102#
R102#
(Perfromed “no shut” on interface Tunnel100 on R100)

R102#
*Nov 24 09:15:00.546: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel100,
changed state to up
*Nov 24 09:15:00.546: %TRACKING-5-STATE: 10 interface Tu100 line-protocol Down->Up
*Nov 24 09:15:00.550: EEM routing ED: event to match: type=add, network=10.20.20.0,
mask=255.255.255.0
*Nov 24 09:15:00.550: EEM routing ED: network/mask: 10.20.20.0/255.255.255.0 matched
pattern network/len/, ge/le/ne: 10.20.20.0/24, 0/0/0

...

R102#
*Nov 24 09:15:00.562: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : CTL :
cli_open called.
*Nov 24 09:15:00.586: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT :
R102>
*Nov 24 09:15:00.586: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : IN :
R102>enable
*Nov 24 09:15:00.722: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT :
R102#
R102#
*Nov 24 09:15:03.302: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.20.20.100
(Tunnel100) is up: new adjacency

...

*Nov 24 09:15:04.434: %DUAL-5-NBRCHANGE: EIGRP-IPv6 1: Neighbor

FE80::A8BB:CCFF:FE00:6400 (Tunnel100) is up: new adjacency
R102#
*Nov 24 09:15:05.754: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : IN :
R102#ping 2001:20:20::100
*Nov 24 09:15:06.002: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT :
Type escape sequence to abort.
*Nov 24 09:15:06.002: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT :
Sending 5, 100-byte ICMP Echos to 2001:20:20::100, timeout is 2 seconds:
*Nov 24 09:15:06.002: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT :
!!!!!
*Nov 24 09:15:06.002: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT :
Success rate is 100 percent (5/5), round-trip min/avg/max = 0/0/4 ms
*Nov 24 09:15:06.002: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT :
R102#
*Nov 24 09:15:06.002: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : IN :
R102#config t

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
*Nov 24 09:15:06.134: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT :
Enter configuration commands, one per line. End with CNTL/Z.
*Nov 24 09:15:06.134: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT :
R102(config)#
*Nov 24 09:15:06.134: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : IN :
R102(config)#int Tunnel101
*Nov 24 09:15:06.262: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT :
R102(config-if)#
*Nov 24 09:15:06.262: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : IN :
R102(config-if)#shut
*Nov 24 09:15:06.310: %DUAL-5-NBRCHANGE: EIGRP-IPv6 1: Neighbor
FE80::A8BB:CCFF:FE00:6500 (Tunnel101) is down: interface down
*Nov 24 09:15:06.318: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.20.21.101
(Tunnel101) is down: interface down
*Nov 24 09:15:06.390: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT :
R102(config-if)#
*Nov 24 09:15:06.390: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : IN :
R102(config-if)#end
*Nov 24 09:15:06.422: %SYS-5-CONFIG_I: Configured from console by on vty0
(EEM:RECOVER-PRIMARY-PATH)
R102#
*Nov 24 09:15:06.522: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : OUT :
R102#
*Nov 24 09:15:06.522: %HA_EM-1-LOG: RECOVER-PRIMARY-PATH: PRIMARY LINK IS RESTORED.
BACKUP LINK DEACTIVATED
R102#
*Nov 24 09:15:06.522: %HA_EM-6-LOG: RECOVER-PRIMARY-PATH : DEBUG(cli_lib) : : CTL :
cli_close called.
R102#
*Nov 24 09:15:08.310: %LINK-5-CHANGED: Interface Tunnel101, changed state to
administratively down
*Nov 24 09:15:08.330: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel101,
changed state to down
R102#
*Nov 24 09:15:09.894: EEM routing ED: RIB update: event=2, proc=2048, table=0,
*Nov 24 09:15:09.894: EEM routing ED: event to match: type=remove, network=10.20.21.0,
mask=255.255.255.0
*Nov 24 09:15:09.894: EEM Routing ED: num_matches = 0

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
Task 7 – Consolidated Custom Status Command

Goal: Use an EEM applet to create a custom consolidated “show status” command

Script Logic:
 Add a new CLI command (“show status”) by using the “cli” event detector
 The event detector would match a non-existing command, and would execute
the required actions
 The command matching uses a regular expression that would allow partial
command matching (for example “sh stat” or “sho statu”)
 We use a global environment variable (_SERVICE_INF_LIST) which is defined
in the router configuration. This variable holds a list of interfaces grouped into
logical groups. The group names are marked with <<GROUP NAME>>.
 A “foreach” loop iterates through all the values in _SERVICE_INF_LIST.
o Each time a group name is found (<<xxx>>) a group header is printed.
The rest of the operations inside the loop instance are skipped using the
“continue” action.
o Each time a regular interface is found, the interface operational status is
extracted, and printed in a custom format

Introduced EEM Elements:

 Matching an unknown command pattern using a regular expression
 Using environment variables for global script parameters
 foreach loop to iterate through a list of values
 continue keyword inside a loop
 regexp matching and extraction of sub-strings

Example:

!
event manager environment _SERVICE_INF_LIST <<Core
Interfaces>>,Eth0/0,Eth0/1,Eth0/2,Eth0/3,<<Spoke
Interfaces>>,Eth1/0,<<Spoke Tunnels>>,Tun100,
!
event manager applet SHOW_STATUS
event cli pattern "(sh|sho|show)\s+(stat|statu|status)" mode "exec"
enter
action 001.1 cli command "ena"

action 003.1 foreach _inf "$_SERVICE_INF_LIST" ","

action 004.1 regexp "<<(.*)>>" $_inf _match _str1

action 004.2 if $_regexp_result eq 1
action 004.3 puts "\n$_str1"
action 004.4 puts "------------------------"
action 004.5 continue
action 004.6 end
action 005.1 cli command "show interface $_inf"

action 006.1 foreach _line $_cli_result "\n"

action 007.1 regexp "^(.* is .*, line protocol is .*)\r$" $_line
_match _str1
action 007.2 if $_regexp_result eq 1
action 007.3 puts $_str1
action 007.4 end

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
 action 008.1 regexp "^(.*Description.*)\r$" $_line _match
action 008.2 if $_regexp_result eq 1
action 008.3 puts $_match
action 008.4 end

action 009.1 regexp "^(.* input rate .*)\r$" $_line _match

action 009.2 if $_regexp_result eq 1
action 009.3 puts $_match
action 009.4 end

action 010.1 regexp "^(.* output rate .*)\r$" $_line _match

action 010.2 if $_regexp_result eq 1
action 010.3 puts $_match
action 010.4 end

action 099.1 end

action 099.2 end
!

Output Example:

R100#show stat
Core Interfaces
------------------------
Ethernet0/0 is up, line protocol is up
Description: TO HUB2-R101
30 second input rate 0 bits/sec, 0 packets/sec
30 second output rate 0 bits/sec, 0 packets/sec
Ethernet0/1 is up, line protocol is up
Description: TO HUB2-R101
30 second input rate 0 bits/sec, 0 packets/sec
30 second output rate 0 bits/sec, 0 packets/sec
Ethernet0/2 is up, line protocol is up
Description: TO HUB2-R101
30 second input rate 0 bits/sec, 0 packets/sec
30 second output rate 0 bits/sec, 0 packets/sec
Ethernet0/3 is up, line protocol is up
Description: TO HUB2-R101
30 second input rate 0 bits/sec, 0 packets/sec
30 second output rate 0 bits/sec, 0 packets/sec
Spoke Interfaces
------------------------
Ethernet1/0 is up, line protocol is up
5 minute input rate 1000 bits/sec, 2 packets/sec
5 minute output rate 1000 bits/sec, 2 packets/sec
Spoke Tunnels
------------------------
Tunnel100 is up, line protocol is up
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec

Tips:

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
 The regexp action matches a pattern (1st parameter) in the string that follows
it (2nd parameter).
The 3rd parameter would be populated with whatever part of the string was
matched by the whole pattern, allowing content extraction from the matched
string portion:
o Any other parameters (4th and on) on the regexp command would be
populated by extracted parts of the string (marked with ‘(‘ and ‘)’ in the
pattern)
o The $_regexp_result system variable would hold a Boolean (0/1) result
of the last regexp match.
o http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm
_01.html#wp1139025

 The different regular expression strings used in the example include the
following shortcuts:
o \s – match any white space
o \r – carriage return
o \n – new line (note that a show output end of line is matched with \r\n)
o .* would match any (also zero) number of any characters
o ^ - beginning of line
o $ - end of line
o _ - space (used in CLI “| include” syntax) – note the “ | inc rate_”

A more complete reference to Cisco regular expression support can be found

at
http://www.cisco.com/en/US/docs/ios/12_2/termserv/configuration/guide/tcfaa
pre_ps1835_TSD_Products_Configuration_Guide_Chapter.html

 The foreach loop action would iterate through all the fields in the provided string
list, separated by the provided field separator
o The field separator used in the foreach action could be “\n”. This would
allow iterating through a list of lines (for example all the lines in the output
of a show command or as it is used later in task 4)
o http://www.cisco.com/en/US/docs/ios-xml/ios/eem/command/eem-cr-
a1.html#GUID-A9FDB5DC-ED8F-422B-BE4B-B59DE5A1D0B4

 The continue action would skip over later command, and would make the
current loop skip to the next iteration.
http://www.cisco.com/en/US/docs/ios-xml/ios/eem/command/eem-cr-
a1.html#GUID-5A6D5C59-2EEF-44FE-B336-2F66FDC38757

 You can also use the “while” loop to match for a loop condition
http://www.cisco.com/en/US/docs/ios-xml/ios/eem/command/eem-cr-
a2.html#GUID-5010385D-272C-48F7-BDE6-F413A44F3523

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
 Task 8 – Secure Automatic Provisioning

Goal: Provision link configuration only when a specific neighbor is detected

Script Logic:
 Use the “neighbor-discovery” event detector to detect new or expired CDP
neighbors.
 Correlate either one of the two different events:
o CDP add neighbor
o CDP delete neighbor
 If a new neighbor is detected on the uplink interfaces, the remote hostname is
checked, and only if it matches the expected hostname, the interface is
provisioned
 If an unexpected hostname is detected, the interface is disabled as a security
measure, and the script is aborted
 If a neighbor is deleted (for example if interface is down), the IP configuration
is deleted

Introduced EEM Elements:

 Neighbor-discovery event detector
 Event correlation (using “event tag”, and “trigger”)
 if/else/elseif logical operations

Example:

Note This example should be deployed on router R102 in the topology

event manager applet AUTO_PROVISION_LINKS

event tag CDP-ADD neighbor-discovery interface regexp (Ethernet0/0|Ethernet1/0) cdp
add
event tag CDP-DEL neighbor-discovery interface regexp (Ethernet0/0|Ethernet1/0) cdp
delete

trigger
correlate event CDP-ADD or event CDP-DEL

action 001 cli command "ena"

action 002 cli command "conf t"
action 003 cli command "interface $_nd_local_intf_name"

action 004 if $_nd_notification eq "cdp-delete"

action 005 cli command "interface $_nd_local_intf_name"

action 006 cli command "no ip address"
action 007 syslog priority alerts msg "Uplink device $_nd_cdp_entry_name on
interface $_nd_local_intf_name is no longer detected. IP Address configuration
removed."

action 008 elseif $_nd_notification eq cdp-add

action 009 if $_nd_local_intf_name eq "Ethernet0/0"

action 010 if $_nd_cdp_entry_name eq "R101"

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
 action 011 set ip_addr "10.1.6.102"
action 012 set ip_mask "255.255.255.0"
action 013 else
action 014 syslog priority alerts msg "Unexpected device detected on interface
$_nd_local_intf_name ($_nd_cdp_entry_name). Interface is disabled."
action 015 cli command "shut"
action 016 exit
action 017 end

action 018 elseif $_nd_local_intf_name eq "Ethernet1/0"

action 019 if $_nd_cdp_entry_name eq "R100"

action 020 set ip_addr "10.1.5.102"
action 021 set ip_mask "255.255.255.0"
action 022 else

action 023 syslog priority alerts msg "Unexpected device detected on interface
$_nd_local_intf_name ($_nd_cdp_entry_name). Interface is disabled."
action 024 cli command "shut"
action 025 exit

action 026 end

action 027 end

action 028 cli command "ip address $ip_addr $ip_mask"

action 029 syslog priority informational msg "Uplink device $_nd_cdp_entry_name on
interface $_nd_local_intf_name detected. IP Address configuration applied."

action 030 end

Tips:
 All system variables used by a specific event detector can be seen in IOS by
using the following command:

router#show event manager detector neighbor-discovery detailed

No. Name Version Node Type
1 neighbor-discovery 01.00 node0/0 RP

<skipped>
Applet Built-in Environment Variables:
$_event_id
$_job_id
$_event_type
$_event_type_string
$_event_pub_time
$_event_pub_sec
$_event_pub_msec
$_event_severity
COMMON VARIABLES:
$_nd_notification
$_nd_intf_linkstatus
$_nd_intf_linestatus
$_nd_local_intf_name
$_nd_short_local_intf_name
$_nd_port_id
CDP EVENT VARIABLES:
$_nd_protocol
$_nd_proto_notif
$_nd_proto_new_entry
$_nd_cdp_entry_name
$_nd_cdp_hold_time
$_nd_cdp_mgmt_domain

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
 $_nd_cdp_platform
$_nd_cdp_version
$_nd_cdp_capabilities_string
$_nd_cdp_capabilities_bits
$_nd_cdp_capabilities_bits_[0-31]
LLDP EVENT VARIABLES:
$_nd_protocol
$_nd_proto_notif
$_nd_proto_new_entry
$_nd_lldp_chassis_id
$_nd_lldp_system_name
$_nd_lldp_system_description
$_nd_lldp_ttl
$_nd_lldp_port_description
$_nd_lldp_system_capabilities_string
$_nd_lldp_enabled_capabilities_string
$_nd_lldp_system_capabilities_bits
$_nd_lldp_enabled_capabilities_bits
$_nd_lldp_capabilities_bits
$_nd_lldp_capabilities_bit_[0-31]

 In order to figure out how all the different parameters used by the neighbor-
discovery event detector, the following test script can be used:

event manager applet TEST-ND

event tag CDP-ADD neighbor-discovery interface regexp .* cdp add
event tag CDP-DELETE neighbor-discovery interface regexp .* cdp delete
trigger
correlate event CDP-ADD or event CDP-DELETE
action 100 puts "_nd_notification=$_nd_notification"
action 101 puts "_nd_intf_linkstatus=$_nd_intf_linkstatus"
action 102 puts "_nd_intf_linestatus=$_nd_intf_linestatus"
action 103 puts "_nd_local_intf_name=$_nd_local_intf_name"
action 104 puts "_nd_short_local_intf_name=$_nd_short_local_intf_name"
action 105 puts "_nd_port_id=$_nd_port_id"
action 110 puts "_nd_protocol=$_nd_protocol"
action 111 puts "_nd_proto_notif=$_nd_proto_notif"
action 112 puts "_nd_proto_new_entry=$_nd_proto_new_entry"
action 113 puts "_nd_cdp_entry_name=$_nd_cdp_entry_name"
action 114 puts "_nd_cdp_hold_time=$_nd_cdp_hold_time"
action 115 puts "_nd_cdp_mgmt_domain=$_nd_cdp_mgmt_domain"
action 116 puts "_nd_cdp_platform=$_nd_cdp_platform"
action 117 puts "_nd_cdp_version=$_nd_cdp_version"
action 118 puts "_nd_cdp_capabilities_string=$_nd_cdp_capabilities_string"
action 119 puts "_nd_cdp_capabilities_bits=$_nd_cdp_capabilities_bits"

Output Example:

R102#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R102(config)#int e0/0
R102(config-if)#no shut
R102(config-if)#
*May 11 21:10:54.364: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up
*May 11 21:10:55.372: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0,
changed state to up
R102(config-if)#
R102(config-if)#

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
*May 11 21:11:26.356: %HA_EM-6-LOG: AUTO_PROVISION_LINKS: Uplink device R101 on
interface Ethernet0/0 detected. IP Address configuration applied.
R102(config-if)#
R102(config-if)#do show run int e0/0
Building configuration...

Current configuration : 87 bytes

!
interface Ethernet0/0
description TO R101
ip address 10.1.6.102 255.255.255.0
end
R102(config-if)#int e0/0
R102(config-if)#shut
R102(config-if)#
*May 11 21:12:04.956: %HA_EM-1-LOG: AUTO_PROVISION_LINKS: Uplink device R101 on
interface Ethernet0/0 is no longer detected. IP Address configuration removed.
R102(config-if)#
*May 11 21:12:06.052: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to
administratively down
*May 11 21:12:07.052: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0,
changed state to down
R102(config-if)#
R102(config-if)#
R102(config-if)#do show run int e0/0
Building configuration...
Current configuration : 75 bytes
!
interface Ethernet0/0
description TO R101
no ip address
shutdown
end

-------------------------------------------------------------------------

R102#show cdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
D - Remote, C - CVTA, M - Two-port Mac Relay
Device ID Local Intrfce Holdtme Capability Platform Port ID
R101 Eth 0/0 168 R Solaris U Eth 1/0
R100 Eth 1/0 140 R Solaris U Eth 1/0
R102#show run int e1/0
Building configuration...
Current configuration : 105 bytes
!
interface Ethernet1/0
description TO R100
ip address 10.1.5.102 255.255.255.0
load-interval 30
end

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
R100(config)#hostname NOT_R100

R102#
*May 11 21:21:40.127: %HA_EM-1-LOG: AUTO_PROVISION_LINKS: Unexpected
device detected on interface Ethernet1/0 (NOT_R100). Interface is disabled.

R102#show run int e1/0

Building configuration...

Current configuration : 93 bytes

!
interface Ethernet1/0
description TO R100
no ip address
load-interval 30
shutdown
end

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
Appendix I
As mentioned in “Task 3 – Scheduling Events”, the provided example has a minor
error.
When issuing the “copy running-config startup-config” CLI command, the router waits
for some input, and would not execute the command unless the information is
provided:

Router#copy running-config startup-config

Destination filename [startup-config]?

The following script example shows an alternative version of the script which works
correctly:

!
event manager applet PERIODIC-CONFIG-SAVE
event timer cron name CONFIG-SAVE-TIMER cron-entry "55 23 * * 1-5"
action 1.0 cli command "enable"
action 2.0 cli command "copy running-config startup-config" pattern config
action 2.1 cli command “startup-config”
!

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
Appendix II
Figure 4 provides a reference to the different EEM event detectors available in the
different Cisco software trains and versions:

Figure 4 Event Detector Support Matrix

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved
Figure 5 shows the mapping between the EEM version and the different IOS version
trains available:
Figure 5 IOS to EEM Version Mapping

The following command provides the EEM version on an IOS device:

Router#show event manager version

Embedded Event Manager Version 3.10
Component Versions:
eem: (v310_throttle)4.1.10
eem-gold: (v310_throttle)1.0.7
eem-call-home: (v310_throttle)1.0.6
Event Detectors:
Name Version Node Type
application 01.00 node0/0 RP
syslog 01.00 node0/0 RP
track 01.00 node0/0 RP
resource 01.00 node0/0 RP
routing 02.00 node0/0 RP
...

LABNMS-2001
© 2014 Cisco Systems, Inc. All rights reserved