0% found this document useful (0 votes)

162 views

Untitled

Uploaded by

Faisal Khan

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

162 views

Untitled

Uploaded by

Faisal Khan

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 156

Troubleshooting Cisco Unified

Contact Center Enterprise

BRKCCT-3005

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
Agenda
Troubleshooting UCCE

 Serviceability
‒ Performance Monitors
‒ SNMP
‒ Logging Improvements
‒ OPC Capture
‒ Changes with Windows 2008 R2
‒ Diagnostic Framework
 Flagship Features in UCCE 8.5 and 9.0
‒ Agent Greeting
‒ Finesse
‒ Precision Routing

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
Troubleshooting UCCE

 Multimedia Integration
‒ Interruptibility & Handle Time
 Network Optimization
‒ The Perils of TCP Offloading
 Solution Approach to Troubleshooting
‒ Crisis Management, Communication, Collaboration, Customer Success
‒ Problem Isolation
 Q&A

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
“The problem with troubleshooting is that trouble
shoots back.”

“Tell me, and I will forget. Show me, and I may

remember. Involve me, and I will understand.”

Authors Unknown
Serviceability
Performance Monitors
Grade Your System in PhysEd

 Performance counters are measurements of

system state or activity. – Microsoft TechNet

 Reactive Usage: UCCE Serviceability Guide section 8.3

recommends which PerfMon counters to enable when diagnosing
issues.

 Proactive Benefits: Counters can also be enabled for trending and

predicting upcoming capacity/performance requirements.

 Counters added in UCCE 8.0 provide richer visibility into system

activity and health.
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Performance Monitors
Grade Your System in PhysEd

 Platform Diagnostic Counters still (7.0) collected automatically by Node Manager

‒ 1 Minute Intervals / 1MB Per File / 45 Days Max
Component Counter Name Units Threshold

Processor %Processor Time (_Total) Percentage <50% | 50% - 60% | >60% (sustained)

Process Handle Count (_Total) # Handles <=2 x [# CPUs] (continuous)

Memory Page Faults / sec # Faults <10 | >=10 | >10 (sustained)

Memory Committed Bytes # Bytes Relative to paging, demand for virtual memory

Memory Pages / sec # Pages / sec <10 | >10 (continuous)

System Threads # Threads Reference

System Processor Queue Length # Threads <2 x [# CPUs] | >= 2 x [#CPUs] (sustained)

System Processes # Processes Reference

‒ Saved to \icm\log\Perf_MACHINENAME_YYYYMMDDHHMMSS.CSV
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Performance Monitors in 8.0
Grade Your System in PhysEd

 Component-specific Counters for Further Diagnosis

Cisco Router Object Counter Default State
Agents Logged On* On
Calls In Progress* On
Calls/sec* On
Calls In Router On
Router State Size Off
Messages Processed/sec Off
Bytes Processed/sec Off
Avg Process Time/Message (ms) Off

 * = Useful for Capacity Planning. Available Prior to UCCE 8.0.

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Performance Monitors in 8.0
Grade Your System in PhysEd

 Component-specific Counters for Further Diagnosis

 * = Useful for Capacity Planning. Available Prior to UCCE 8.0.

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Performance Monitors in 8.0
Grade Your System in PhysEd
Administration & Data Server Default OPC Default Logger Default
New Counters State New Counters State New Counters State

CallType DB Write Records Processed On Call Count On Number of DB Write Records On

DB Write Average Time On
Route Queue Depth On Agent Count On
DB Write Records Processed On
Route DB Write Average Time On Skill Group Count On
Route DB Write Records Processed On Services Count On Outbound Option Dialer Default
New Counters State
Service Queue Depth On Logged-In Agent Count On
Number of DB Write Records On
Service DB Write Average Time On Ready Agent Count On DB Write Average Time On
Agent Queue Depth On Not-Ready Agent Count On DB Write Records Processed On
…and many more! …and many more! …and many more!

 Other new objects available for

‒ PIM – EA & VRU
‒ CTI Server, CTI OS
‒ Outbound Option Campaign Mgr, Import Process
 Reference the UCCE Serviceability Guide for more information
http://www.cisco.com/en/US/products/sw/custcosw/ps1844/products_installation_and_configuration_guides_list.html

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Simple Network Management Protocol
SNMP Crash Course

 SNMP is a family of specifications that provide a means for collecting

network management data from the devices residing in a network.
– Microsoft TechNet
 SNMP Agent installed on “managed devices” can report problems via
“traps” to the Network Management System – and you(r team)!
 Subagent Management Information Bases (MIBs) Installed:
‒CISCO-CONTACT-CENTER-APPS-MIB
‒CISCO-CVP-MIB
 Agent supports Versions 1, 2c, and 3
‒ Community strings for v1 and v2c
‒ v3 supports authorization (MD4, SHA-1) & privacy (3DES, AES)
 BONUS! Extra slides at end of deck – Tips on MIBs, OIDs, Config
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Simple Network Management Protocol
SNMP Crash Course

 UCCE SNMP Event Flow – High Level

Syslog
Collector
syslog
PG EMS

SNMP
EMS Logger Network
Router Management
SNMP
System

EMS
Cisco
ADS Unified
Operations
Manager

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Simple Network Management Protocol
SNMP Crash Course

 UCCE SNMP Event Flow – Digging Deeper

CW2K
syslog
Feed
Syslog Collector
CSFS
Router ProTip: Cisco SNMP Service
CCCA starts SNMPDM, which starts
EMS Logger Agent
CCCAAgent Other
MDS SNMP SNMP Agents
DM
(Generic traps,
platform-specific
SNMP NMS subagents)
EMS EMS EMS t_Event

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
CW2K
Simple Network Management Protocol Feed

SNMP Crash Course CSFS

CCCA
 Logging Differences for SNMP vs Syslog Events Logger Agent

‒ Certain events hard-coded as “Objects”

‒ SNMP looks for this flag and will only process “Objects”
 Example: Syslog-only “Simple” Event (no Object)
03:37:06:096 la-csfs Trace: ProcName:mds SysName:85PG2B StatusCode:The remote process has exited. Severity:Error Category:Message Delivery
03:37:06:096 la-csfs Trace: ID: e1048013 Msg: Unexpected disconnect from client pim1.
03:37:06:096 la-csfs Trace: EMS ID [e1048013] in Primary Positive Table
03:37:06:096 la-csfs Trace: Using [e104801385PG2Bpim1] as String for Suppression Event

 CW2KFeed picks the event from the pipe and pushes it out to Syslog collector
03:37:06:096 la-cw2kfeed Trace: CSysMsg::PutQMsg at Tail -- Q size = 1.
03:37:06:096 la-cw2kfeed Trace: CSyslog::GetCSFSMsg Reading data from CSFS pipe feed.
03:37:06:096 la-cw2kfeed Trace: CSysMsg::GetQMsg -- event for data in Q has been signalled.
03:37:06:096 la-cw2kfeed Trace: CSysMsg::GetQMsg -- Retrieving Q item # 1.
03:37:06:096 la-cw2kfeed Trace: <131>101: 85pg2b: Mar 8 2012 00:37:03.679 +0000: ICM_PG_MessageDelivery-3-1048013: [comp=PG2-
B][pname=mds][iid=ucce][mid=1048013][sev=error]: Unexpected disconnect from client pim1.
03:37:06:096 la-cw2kfeed Trace: CSysMsg::GetQMsg -- Q is empty. Waiting for data.

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
CW2K
Simple Network Management Protocol Feed

SNMP Crash Course CSFS

CCCA
 Logging Differences for SNMP vs Syslog Events Logger Agent

‒ Certain events hard-coded as “Objects”

‒ SNMP looks for this flag and will only process “Objects”
 Example: SNMP Event (Object!)
03:37:06:096 la-csfs Trace: ProcName:nm SysName:85PG2B StatusCode: Severity:Warning Category:Node Manager
03:37:06:096 la-csfs Trace: ID: a102c113 Msg: ICM\ucce\PG2B node process pim1 exited from Control-C or window close.
03:37:06:096 la-csfs Trace: Using [3_2_85PG2B_ICM\ucce\PG2B_pim1] as Unique Object String

03:37:06:096 la-csfs Trace: Created New Base Object '3_2_85PG2B_ICM\ucce\PG2B_pim1'

03:37:06:096 la-csfs Trace: Using [a102c113_3_2_85PG2B_ICM\ucce\PG2B_pim1] as String for Object Suppression Event

 CW2KFeed still picks the event from the pipe and sends to Syslog collector
03:37:06:096 la-cw2kfeed Trace: <132>102: 85pg2b: Mar 8 2012 00:37:03.695 +0000: ICM_PG_NodeManager-4-102C113: [comp=PG2-
B][pname=nm][iid=ucce][mid=102C113][sev=warning]: ICM\ucce\PG2B node process pim1 exited from Control-C or window close.

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
CW2K
Simple Network Management Protocol Feed

SNMP Crash Course CSFS

CCCA
 Logging Differences for SNMP vs Syslog Events Logger Agent

‒ Certain events hard-coded as “Objects”

03:37:06:096 la-csfs Trace: Created New Base Object '3_2_85PG2B_ICM\ucce\PG2B_pim1'

03:37:06:096 la-csfs Trace: Using [a102c113_3_2_85PG2B_ICM\ucce\PG2B_pim1] as String for Object Suppression Event

 Events for Host\Instance\Process use same Object created since last CSFS restart
03:37:15:596 la-csfs Trace: ProcName:nm SysName:85PG2B StatusCode: Severity:Warning Category:Node Manager
03:37:15:596 la-csfs Trace: ID: a102c110 Msg: ICM\ucce\PG2B node process pim1 successfully reinitialized after restart.
03:37:15:596 la-csfs Trace: Using [3_2_85PG2B_ICM\ucce\PG2B_pim1] as Unique Object String

03:37:15:596 la-csfs Trace: Event for Existing Base Object '3_2_85PG2B_ICM\ucce\PG2B_pim1'

03:37:15:596 la-csfs Trace: Using [a102c110_3_2_85PG2B_ICM\ucce\PG2B_pim1] as String for Object Suppression Event

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
 Peripheral Gateway JGW
Logging Improvements Enable in Procmon:
Stop The Insanity!
‒ trace JT_TPREQUESTS /on
 Peripheral Gateway OPC: ‒ trace JT_JTAPI_EVENT_USED* /on
 Call Router RTRTrace
‒ OPCTEST debug /agent ‒ trace JT_PIM_EVENT /on
Check boxes for:
/routing /closedcalls /cstacer
‒ Config Changes /ctimsg /inrcmsg /calls ‒ trace JT_ROUTE_MESSAGE /on

‒ Route Request ‒ Registry EMSTraceMask 0x40 ‒ trace CONF /on

‒ Network VRU  Peripheral Gateway CTI Gateway  Peripheral Gateway EA PIM
‒ Translation Route
‒ Registry EMSTraceMask 0xF8 Enable in Procmon:
‒ Call Queueing
‒ trace tp* /on
 Peripheral Gateway CTI OS
‒ CallType RealTime
‒ trace precall /on
‒ Registry EMSTraceMask 0x60A0F
‒ INRC (RC Dialogue)
‒ trace *event /on
 Peripheral Gateway VRU PIM ‒ trace csta* /on
 Peripheral Gateway MR PIM
‒ Registry EMSUserData 7F F7 C0
‒ trace periph* /on
‒ Registry EMSTraceMask 0xF8
 Logger Configlogger ‒ trace opc* /on
‒ Registry EMSUserData DF
‒ Registry EMSTraceMask 0xFFFF
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Logging Improvements
Stop The Insanity!

 Simplified Tracing Introduced in UCCE 8.0(1)

‒ Level 0 – Default / Error :: Zero to minimal performance impact
‒ Level 1 – Warning :: Small performance impact, Cumulative tracing
‒ Level 2 – Informational :: Medium performance impact, Cumulative tracing
‒ Level 3 – Debug :: High performance impact, Cumulative tracing

 How do they translate?

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Logging Improvements
Stop The Insanity! Reference the UCCE Serviceability
Guide for more information!
http://www.cisco.com/en/US/products/sw/custcosw/ps1844/prod
 Simplified Tracing Introduced in UCCE 8.0(1) ucts_installation_and_configuration_guides_list.html

‒ Level 0 – Default / Error PG Process

Level 0
(Default/Error)
Level 1
(Warning)
Level 2
(Informational)
Level 3
(Debug)
‒ Level 1 – Warning JT_JTAPI_EVENT_ JT_JTAPI*
USED JT_HEX
‒ Level 2 – Informational JTAPIGW
JT_TPREQUESTS JT_CONNECTION
JT_ROUTE* JT*
JT_PIM_EVENT *CONF*
‒ Level 3 – Debug JT_ROUTE_
MESSAGE
JT_TERM*
JT_LOW*
agent, inrcmsg,
calls, NCT, Missingdata,
default, cstacer closedcalls,
 How do they translate? OPC
0x40 tpmsg, routing
simplified
0x40
halfhour
0x40
0x40
PGAGENT 0x00 0x03 0x0F 0xFF
 Goal: Only ask for logs CTI SVR 0xF0 0xF6 0xFE 0xFF

ONCE CTI OS 0x60A0F 0x240A2F 0x260A2F 0x2E0A2F

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Logging Improvements
Compression Rules!

 EMS Log Compression = Increased Retention (x10!) for:

‒ CTI OS Server ‒ OPC-TDM ‒ EAGTPIM ‒ VRUPIM
‒ OPC-CCE ‒ CTISVR ‒ JGW
 OPC Capture Binary Retention also increased x4
 ~10% increase in CPU and Disk I/O during high-water mark load testing.

 Introduced with 7.5(10), 8.0(3), 8.5(1)

 Corresponding Dumplog enhancements made on Peripheral Gateways
‒ Uses \icm\bin\gzip.exe to unzip compressed EMS before dumping
‒ User must manually decompress files if using dumplog on non-PG machine.

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
OPC Capture
OPC Is Watching You

 How is OPC Capture different from traditional component logs?

opc_120301_170156.ems(.gz)
Debug Level 1 10101000100101100010
Dumplog
Missing lots of trace messages!

Free Disk
Space > 5GB

OPCdata_opc_0301_170156.cap
Merge Debug
111111111111111111111111
Also: OPCconfig, Pgreg, OPCXferData
Not a Quick Process!
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
OPC Capture
OPC Is Watching You

 Where are the OPC Capture files?

PG Service (Re)Start, OPC State Transfer

M
K Triggers dated folder creation containing:
• Capture data
D
• OPC Config Export
I
• Registry Export
R • State Transfer (optional)

Timestamps must match!

Non-matching OPCdata = Capture File Wrap
Root Cause Unlikely!

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
OPC Capture Pop Quiz!
OPC Is Watching You

 If OPC asserted at 6:39PM on February 15th, which folder will have the capture
files that include the assertion?

A.
B.
C.
D.

 Dated Folder is created when OPC starts, so 18:39:44 will have the capture
data AFTER the assertion. The folder from the PREVIOUS start is correct.
 Side note: Minidump vs Full Dump slides in Appendix!
 But what if the Capture Files have wrapped & timestamps don’t match?
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
OPC Capture
OPC Is Watching You

 Hard-Coded 2GB Capture Limit Per Dated Folder

‒ OPCdata_opc_xxxx_yyyyyy.cap files limit ~10MB each. Wrapping = Failure.
 How can we baseline the Capture Files?
 7.2(5) & 7.5(1) through 8.5(1): „exit_opc‟
‒ Intrusive / downtime required
‒ Could clear / delay condition recurrence

 8.5(2) & Later: „newcap‟

‒ Does not kill OPC, but WILL delay requests from
agent desktop (i.e. consult calls) for 1-2 seconds!
These requests typically have a 60 second timer, so 1-2 seconds will not impact
‒ Perform during low call volume / agent activity
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
OPC Capture
OPC Is Watching You Updated!
NewCap in
 Where are the OPC Capture files?
8.5(2) or later

PG Service (Re)Start, OPC State Transfer

M
K Triggers dated folder creation containing:
• OPC Config Export
D
• Registry Export
I
• Capture data
R • State Transfer (optional)

Timestamps must match!

Non-matching OPCdata = Capture File Wrap
Root Cause Unlikely!

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Updated Operating System Support
Party Like It‟s 2008!

 8.5(2) adds support for Windows 2008 R2 SP1 or greater (64-Bit Only)
‒ Enhanced Native Security = no need for UCCE Installer OS Hardening
‒ TCP Chimney Offload disabled by default! (Ref: Cisco FN 63215)
‒ Unsupported PGs on 2008: Aspect, Spectrum, Symposium (Ref: Cisco ICM ACD Matrix)
 OS architecture change = UCCE process windows no longer appear

 Title bar indicators gone with the wind(ows).

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Updated Operating System Support
Party Like It‟s 2008!

 OS architecture change = UCCE

EMSmon
process windows no longer appear
 Title bar indicators gone with the wind(ows).
Usage Syntax:
emsmon <cust_instance> <node_name> <process> {system}

Note: „system‟ argument is optional and only

needed if connecting to remote server

Example (local):
EMSMON ucce RouterA rtr

Example (remote):
EMSMON ucce PG1A pim1 UCCEPG1A

*Max of 1 local and 5 remote connections

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Updated Operating System Support
Party Like It‟s 2008!

 OS architecture change = UCCE

EMSmon
process windows no longer appear
 Title bar indicators gone with the
? ?wind(ows).
?
Usage Syntax:
emsmon <cust_instance> <node_name> <process> {system}

Note: „system‟ argument is optional and only

needed if connecting to remote server

Example (local):
EMSMON ucce RouterA rtr

Example (remote):
EMSMON ucce PG1A pim1 UCCEPG1A

*Max of 1 local and 5 remote connections

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Updated Operating System Support
Enter…The Diagnostic Framework Portico
Direct Link
Bookmark
Services Installed Locally Your Servers
Not Bold = No Processes Running

Smart refresh
Remembers
current location

Uptime
Red < 10 minutes
10 < Yellow < 30
Process Title Bar
Blue: Parentheses status
change within 10 min

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Diagnostic Framework
Portico

 Single-system “Support Tool” – More than just Process Status!

 Set tracing for single process

Level 99 = Custom
(Does Not Match
Defined Trace Levels)

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Diagnostic Framework
Portico

 Single-system “Support Tool” – More than just Process Status!

 Set tracing for single process … and Verify

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Diagnostic Framework
Portico

 Dump logs for a single process

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Diagnostic Framework
A Whole New API

 How does it work? RoggerA RoggerB

HTTPS PG1A PG1B

ADS 1 ADS 2

 Other devices also RTMT HTTPS CUCM PUB CUCM SUB

have their own Web
Services for similar
functions
RTMT HTTPS IP IVR 1 IP IVR 2
 Isn‟t there a tool to
bring this all together?
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Analysis Manager
Solution Tool Option #1

 Tool within the CUCM Real-Time Monitoring Tool

(RTMT)
 Node, Trace template configuration steps required
 Set and collect traces in real-time or scheduled
 Collect system environment info
 Supported Products:
‒ UCM – IOS Voice Gateways
‒ UCCE – Unity Connection
‒ UCCX – Unified Presence
‒ CVP

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Analysis Manager
Solution Tool Option #1

 How does it work? RoggerA RoggerB

HTTPS PG1A PG1B

ADS 1 ADS 2

RTMT HTTPS CUCM PUB CUCM SUB

RTMT HTTPS IP IVR 1 IP IVR 2

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Analysis Manager
Solution Tool Option #1 Node Type Nodes

 How does it work? RoggerA RoggerB

RTMT Trace Template ICM PG1A PG1B

ADS 1 ADS 2

UCM CUCM PUB CUCM SUB

Node Group

UCCX IP IVR 1 IP IVR 2

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Analysis Manager
Solution Tool Option #1

 Add Nodes
Node Type
Node

Node Group

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Analysis Manager
Solution Tool Option #1

 Create Trace
Template

Select
Processes

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Analysis Manager
Solution Tool Option #1

 Create Trace
Template

Set Trace
Levels
Level 0 – Default / Error
Level 1 – Warning
Level 2 – Informational
Level 3 – Debug
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Analysis Manager
Solution Tool Option #1

 Schedule
Periodic Select Node or
Collection Node Group

 Trace level Select Trace Template

applied at
Start Time Set Duration
 Set back to
original at Set Interval
End Time

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Analysis Manager
Solution Tool Option #1

 So this supports all of the core “Cisco Unified” contact center products, right?
‒ Analysis Manager does NOT support CUIC at this time.
 What about the other products that use RTMT? Is this version compatible?

 What if I have multiple ICM instances?

‒ Analysis Manager does not support multiple ICM instances.

 Let‟s check out Option #2!!

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
System CLI
Solution Tool Option #2

 Solution-wide tool
Multi-Instance Support
‒ System Mode – run commands on multiple devices
‒ Easily Add/Remove devices.
 Syntax similar to Cisco IOS gateway CLI
‒ Without Tab auto-complete
 Capture, debug, trace from:
‒ All UCCE servers – Gateways – UCM
‒ CVP / CUPS – IP IVR – CUIC
 Low Priority Tasks Throttled – Reduced Priority, Low CPU threshold
‒ Will only use the IDLE CPU on the system. Max of 20 concurrent requests
‒ No timeouts on the client; controlled by the server.
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
System CLI
Solution Tool Option #2

 How does it work? RoggerA RoggerB

HTTPS PG1A PG1B

ADS 1 ADS 2

RTMT HTTPS CUCM PUB CUCM SUB

RTMT HTTPS IP IVR 1 IP IVR 2

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
System CLI
Solution Tool Option #2

 How does it work? RoggerA RoggerB

PG1A ADS PG1B
PG1A PG1B

ADS 1 ADS 2

CUCM PUB CUCM SUB

IP IVR 1 IP IVR 2
CUIC PUB CUIC SUB

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
System CLI
Solution Tool Option #2 Device Type Devices

 How does it work? ICM RoggerA RoggerB

PG1A ADS PG1B

CLI UCM CUCM PUB CUCM SUB

Device List
UCCX IP IVR 1 IP IVR 2
Location differs
with/without CVP
in deployment

CUIC CUIC PUB CUIC SUB

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
System CLI
Deployments Without CVP Device Type Devices

 How does it work? ICM RoggerA RoggerB

PG1A ADS PG1B
CLI
UCM CUCM PUB CUCM SUB

Device List
devices.csv
UCCX IP IVR 1 IP IVR 2

CUIC CUIC PUB CUIC SUB

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
System CLI
Deployments Without CVP

 Devices.csv must be present for CLI to use System Mode (without CVP)
‒ Otherwise, the CLI will run as a single server tool
 Create from Devices-sample.csv (included with every UCCE installation)

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
System CLI ProTip: If System CLI will be run only on ADS,
Deployments Without CVP then devices.csv is only required on ADS.

Required

<server or IP>, <description>, <device type>, <group>, <username>, <password>

10.10.10.30, RoggerA, ICM, ucce-sideA, VMLOAD\Administrator, cvpcisco

10.10.10.31, PG1A, ICM, ucce-sideA, VMLOAD\Administrator, cvpcisco

Recommended

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
System CLI
Deployments Without CVP Device Type Devices
Trace Template
 How does it work? ICM RoggerA RoggerB
PG1A ADS PG1B
CLI Device Type
“All ICM” UCM CUCM PUB CUCM SUB

Device List
devices.csv
Group UCCX IP IVR 1 IP IVR 2

CUIC CUIC PUB CUIC SUB

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
System CLI
Device Type Devices

 How does it work? ICM RoggerA RoggerB

PG1A ADS PG1B
CLI Device Type
“All ICM” UCM CUCM PUB CUCM SUB

Device List
devices.csv
Group

CUIC CUIC PUB CUIC SUB

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
System CLI
Deployments With CVP Device Type Devices

 How does it work? ICM RoggerA RoggerB

CVP Remote PG1A ADS PG1B
CLI
Operations Device Type
Manager “All ICM” UCM CUCM PUB CUCM SUB

Group CVP CVP 1 CVP 2

Device List

CVP
OAMP CUIC CUIC PUB CUIC SUB

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
System CLI
Deployments With CVP

 Devices Managed from CVP OAMP

Encrypted!

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
System CLI
Useful Commands

 Initial Login is to single machine only (CVP OAMP or a UCCE server)

Enter System Mode to connect to solution devices (defined in OAMP or local CSV)
admin:system
Initializing system mode …
Retrieving device list. This process may take a few minutes to complete.
Success . . .
Fail Unable to connect to 10.10.10.130:7890(icm)
Success . . . . .

 Once in System Mode, commands will be run against all devices.

admin(system):show version
Retrieving [version] data from device [uccergra30] ProductType [icm] …
ICM 8.5(2) BuildNumber=37931 ES=0 PatchInstallerVer=465

Retrieving [version] data from device [uccepg1a31] ProductType [icm] …

ICM 8.5(2) BuildNumber=37931 ES=0 PatchInstallerVer=465

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
System CLI
Useful Commands

 Commands can still be limited to a single device…

admin(system):show debug server 10.10.10.30
Retrieving [debug] data from device [10.10.10.30] ProductType [icm] …
Component: icm:Router A, subcomponent: icm:rtr
Trace level = 0  …Group…
admin(system):debug level 1 group ucce-sidea
 …Device Type… Component: icm:Logger A, subcomponent: icm:hlgr
admin(system):show devices devicetype icm Debug level is set successfully
Device host name: uccergra30 …repeats for all Logger A processes
Port: 7890 Component: cvp:CallServer, subcomponent: cvp:MSGBUS
Product type: ICM Debug level is set successfully
Group: ucce-sidea …repeats for all CVP processes
Component: icm:CTI Server 1A, subcomponent: icm:ctisvr
Device host name: uccepg1a31 Debug level is set successfully
Port: 7890 …repeats for all CTI Server 1A processes
Product type: ICM Component: icm:PG1A, subcomponent: icm:jgw1
Group: ucce-sidea Debug level is set successfully
BRKCCT-3005
…repeats for all PG1A
© 2012 Cisco and/or its affiliates. All rights reserved.
processes
Cisco Public 55
System CLI
Useful Commands

 …or on-the-fly combination of devices

admin(system):show config server 10.10.10.30|10.10.10.31
redirect dir c:\temp
File Download
Specify Save
Retrieving [configuration] data from device [10.10.10.30] ProductType [icm] … Location
Fetching configuration resource for component DumpCfg
Fetching configuration resource for component ExportICMCfg Rogger
Fetching configuration resource for component ConfigExport
DumpCfg,
Fetching configuration resource for component Registry
DB Export
Retrieving [configuration] data from device [10.10.10.31] ProductType [icm] …
Fetching configuration resource for component Registry Peripheral Gateway
Registry Export Only
Output is saved to “c:\temp\clioutput0.zip”

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
System CLI
Useful Commands

 Best Of All, The Game-Changer YouTube Search “ciscocc system cli” for Demo!
admin(system):show tech-support http://www.youtube.com/watch?v=iqHZzM_tank
Retrieving [Every Single Piece Of] data from Instant 24-hour Snapshot
device [Your Entire Deployment]
Produces zip of ALL defined server
ProductType [ICM, UCM, CVP, IOS, UCCX, CUIC, ETC!] …
component logs (including OPC Capture),
Output is saved to ICM config, dumpcfg, versions, registries,
“C:\Cisco\CVP\wsm\CLI\download\clioutput1.zip” OS info, PerfMon, debug levels, etc.

For >24 hours, add:

“absdatetime = MM-DD-YYYY:hh:mm”

Organized By Device
EVERYTHING TAC/DE would
need to start investigation.

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
System CLI
Scheduling With Batch Files

 Samples in C:\icm\serviceability\wsccli\sample

 Bat file can contain commonly-used commands

‒ Set tracing on CVP
‒ Collect logs from all ICM devices
‒ Endless possibilities
 Bat file can be run manually or
automatically with Windows Scheduler
 Bring quick resolution to an ongoing issue!

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Diagnostic Framework
Three Serviceability Tools Enter a Ring…

Positives Positives Positives

 Ability to schedule  No Configuration Needed  Multi-instance support
collections  Browser Accessible  System Mode
Negatives  Quick Single-Process  Supports CORE solution
No multi-instance support Transactions endpoints (No OEMs)
Requires UCM Negatives  ANY command can be
Thick-Client No multi-instance support scheduled
Limited Scope (Single Negatives
Device) …No GUI?
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Flagship Features in UCCE 8.5 and 9.0
Agent Greeting
What‟s My Name Again?

 High-level Flow

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Agent Greeting Not Playing
Digging Deeper

 Problem Statement: Agent Greeting does not play

 Business Impact: Inconsistent customer experience. Manual greeting leads to
agent fatigue, adding approximately 4 seconds to every call. Contact center
handles 2000 calls per day. Significant loss to productivity.
 Versions:
‒ UCCE 8.5(3)
‒ CVP 8.5(1)
‒ CUCM 8.5(1)
 Troubleshooting Performed: Call is being routed to available agent correctly.
Once connected, agent and caller cannot hear each other. Greeting does not play.
 What‟s the call flow?
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Agent Greeting Not Playing
Digging Deeper

PG PG

 Agent answers call. PIM knows to play greeting from variable passed in script
19:22:48:286 PG1A-pim2 Trace: CSTA ESTABLISHED, TelephonyDriver::Established: SENT TO OPC
CallID = 17142218 DeviceID = 2007 DeviceType = Static
19:22:48:286 PG1A-pim2 Trace: NeedPlayAgentGreeting: GREET=1 CustID=1

 Agent PIM sends Route Request to OPC for “PlayAgentGreeting” script

19:22:48:286 PG1A-pim2 Trace: CSTA ROUTE REQUEST, TelephonyDriver::RequestPostRoute: by PIM for the DNP SENT TO OPC
CallID = 2130706433 DeviceID = 2007 DeviceType = Static
CurrentRoute = PlayAgentGreeting
CallingDevice = 2007
RequestType = 59
AG =1
CustomerID =1
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Agent Greeting Not Playing
Digging Deeper

PG PG

 Router receives NEW_CALL from OPC and returns Route Select

19:22:48:367 ra-rtr Trace: (65535 x 0 : 0 0) NewCall: CID=(150202,54426100), DN=PlayAgentGreeting, RCID=5001, OpCode=0, GREET=1,
PrefAGSTID=8031, CustomerID=1
19:22:48:301 PG1A-pim2 Trace: ProcessCSTARouteSelect: crossRefID=65535, RTRCallKey=150202.54426100, label 888040000010002
callID=2130706433

 Agent PIM instructs CUCM to add media stream from CVP to agent call
19:22:48:301 PG1A-pim2 Trace: TelephonyDriver::SendMediaStreamRequest - AgentCallID=17142218, AgentCallConnectionDeviceID=2007,
AgentCallConnectionDeviceIDType=0, InvokeID=536642

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Agent Greeting Not Playing
Digging Deeper

PG PG

 CVP Requests Instructions from the Router and receives a CONNECT

133: 10.10.10.20: Mar 29 2012 19:22:48.919 -0400: %CVP_8_5_ICM-7-CALL: {Thrd=pool-1-thread-176-ICM-11} CALLGUID =
1638AC800001000000000002280A0A0A, DLGID = 3 [SIP_LEG_PRERTE_CORRID] - Publishing ,, [ICM_REQUEST_INSTRUCTION], dialogueId=3,
sendSeqNo=1, trunkGroupId=200, trunkNumber=0, serviceId=2, uui=, correlationId=10002, location=, locationpkid=, pstntrunkgroupid=10.10.10.40 ,
pstntrunkgroupchannelnum=2147483647, sipheader=,, LEGID = 1638ac80-f741eec6-2-280a0a0a, DNIS = 888040000010002, ANI = 2007

135: 10.10.10.20: Mar 29 2012 19:22:48.919 -0400: %CVP_8_5_ICM-7-CALL: {Thrd=pool-1-thread-177-ICM-12} CALLGUID =

1638AC800001000000000002280A0A0A, DLGID = 3 [SIP_LEG_PRERTE_CORRID] - Publishing ,, [MsgBus:CONNECT], ssId=SYS_SIP1, labelType=,
label=9990200000, correlationId=10003, command=null, rckey=54426100, rcday=150202, rcseq=4, agentGreetingHint=true, CallContext:,
CallVariable1: 1, user.microapp.caller_input: 1, user.microapp.app_media_lib: app/ag_gr, user.media.id: 10DB62000001000000000001280A0A0A,
user.microapp.ToExtVXML[1]: testpath=1, user.microapp.ToExtVXML[0]: application=CVPMenu,, LEGID = 1638ac80-f741eec6-2-280a0a0a, DNIS =
888040000010002, ANI = 2007

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Agent Greeting Not Playing
Digging Deeper

PG PG

 CVP identifies the Agent Greeting leg, which quickly disconnects

236: 10.10.10.20: Mar 29 2012 19:22:48.919 -0400: %CVP_8_5_SIP-7-CALL: {Thrd=pool-1-thread-178-SIP-661} CALLGUID =
1638AC800001000000000002280A0A0A LEGID = 1638ac80-f741eec6-2-280a0a0a - [INBOUND] Marking the leg as an outbound agent greeting call.

140: 10.10.10.20: Mar 29 2012 19:22:49.387 -0400: %CVP_8_5_ICM-7-CALL: {Thrd=pool-1-thread-187-ICM-14} CALLGUID =

1638AC800001000000000002280A0A0A, DLGID = 3 [SIP_LEG_PRERTE_CORRID] - Deleted dialogue. Duration: 0 hrs, 0 mins, 0 secs, 484 msecs

‒ Less than ½ second … that‟s a SHORT greeting!

 Caller and Agent are connected but cannot hear each other.
‒ What happened to our RTP stream?

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Agent Greeting Not Playing
Digging Deeper

PG PG

 PIM shows RTP started between phones, but 1-way & stopped within half a second.
19:22:48:692 PG1A-pim2 Trace: RTPStarted: CID=17142218, ConnDevID=2007/0, Addr=10.10.10.250:25528, Direction=OUT, RtpType=Audio, BitRate=1,
EchoCan=ON, PktSize=20, Payload=4 , UniqueID=00000000000591CA01D2D03000000000
19:22:48:692 PG1A-pim2 Trace: RTPStarted: CID=17142218, ConnDevID=2007/0, Addr=10.10.10.248:20630, Direction=IN, RtpType=Audio, BitRate=1,
EchoCan=ON, PktSize=20, Payload=4 , UniqueID=00000000000591CA01D2D03000000000

19:22:49:083 PG1A-pim2 Trace: RTPStopped: CID=17142218, ConnDevID=2007/0, Addr=10.10.10.248:20630, Direction=IN,

UniqueID=00000000000591CA01D2D03000000000
19:22:49:083 PG1A-pim2 Trace: RTPStopped: CID=17142218, ConnDevID=2007/0, Addr=10.10.10.250:25528, Direction=OUT,
UniqueID=00000000000591CA01D2D03000000000

 Why did the RTP stop? Do we see anything else in the PIM log?

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Agent Greeting Not Playing
Digging Deeper

PG PG

 PIM receives an error from CUCM (also in Event Viewer)

19:22:49:426 PG1A-pim2 Failed to play Agent Greeting (Type: 2007) for the call (ID:1) to Agent (ID:17142218) with the extension (ext:2001) since no
conference bridge is avaiable.

‒ Oops…let‟s rearrange those variables and fix that typo

19:22:49:426 PG1A-pim2 Failed to play Agent Greeting (Type: 1) for the call (ID:17142218) to Agent (ID:2001) with the extension (ext:2007) since no
conference bridge is available.

 Agent PIM error says something is wrong with the conference bridge.
‒ Bridge…. what features does agent greeting use?
‒ Built-in bridge!!
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Agent Greeting Not Playing
Digging Deeper

 CUCM Built-in Bridge shows “Default” in Device settings for extension 2007

 Never assume! What is the default?

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Agent Greeting Not Playing
Digging Deeper

PG PG

 Place another call with BiB=On. Two-way audio now, but still no Agent Greeting!
 Agent PIM thinks the greeting played successfully.
19:32:18:941 PG1A-pim2 Trace: CSTA Call Agent Greeting, RecvCallAgentGreetingEventMsg: SENT TO OPC
CallID = 17142224 DeviceID = 2007 DeviceType = Static
EventCode = EC_AGENT_GREETING_STARTED GreetingType = 1 PeripheralErrorCode=0x0(0)

19:32:19:347 PG1A-pim2 Trace: CSTA Call Agent Greeting, RecvCallAgentGreetingEventMsg: SENT TO OPC
CallID = 17142224 DeviceID = 2007 DeviceType = Static
EventCode = EC_AGENT_GREETING_DONE GreetingType = 1 PeripheralErrorCode=0x0(0)

 What does CVP show?

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Agent Greeting Not Playing
Digging Deeper

PG PG

 CVP receives RUN_SCRIPT but a MicroApp Format Error occurs.

121: 10.10.10.20: Mar 29 2012 19:32:19.341 -0400: %CVP_8_5_IVR-3-MICROAPP_FORMAT_ERROR: [CONFIG_PARAM(7)]
CALLGUID=695F2F000001000000000006280A0A0A: Invalid Barge-in option: null. Valid options are 'Y' or 'N'
{VRUScriptName: PM,-a ConfigParam: null } [id:3016]

 ConfigParam should be Y or N, not “null”

 Where is this “Barge-in option” configured?
‒ To the Administration and Data Server! (aka Admin Workstation)

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Agent Greeting Not Playing
Digging Deeper

 ICM Config Manager > Network VRU Script List > ConfigParam set incorrectly.

 Replace as “N” and place another test call.

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Agent Greeting Not Playing
Digging Deeper

PG PG

 CVP generates RUN_SCRIPT for JDoe‟s Agent Greeting media file

155: 10.10.10.20: Mar 29 2012 19:33:51.419 -0400: %CVP_8_5_IVR-7-CALL: {Thrd=SubscriptionMgr} CallMsgListener:onMsg:
CALLGUID=A03545000001000000000008280A0A0A DNIS=999020000010012 Received Call message : >>HEADERS: (JMSType)=MsgBus:RUN_SCRIPT_REQ
(JMSDestination)=Topic(CVP.ICM.CC.REQ.SYS_IVR1) (JMSTimestamp)=1333064031419 (ServerID)=cvpcsvxml20.SYS_ICM1:ICM:ICM1:cvpcsvxml20.MsgBus001
>>BODY: featuretype=1 ced= dnis=999020000010012 floatingvars=>>HEADERS: (JMSType)=MsgBus:FLOATINGVARS >>BODY: callvars=[2, , , , , , , , , ]
user.microapp.caller_input=2 user.microapp.app_media_lib=app/ag_gr user.media.id=9F9CAE800001000000000007280A0A0A
user.microapp.FromExtVXML=[null, null, null, null] user.microapp.ToExtVXML=[application=CVPMenu, testpath=1, null, null, null] >>STATE: isTabular=false
isWriteable=true cursor=-1 featureparam2=1 featureparam1=JDoe RouterCallKeySequenceNumber=5 version=CVP_8_5
callguid=A03545000001000000000008280A0A0A RouterCallKeyDay=150202 scriptconfiguration=N switch_leg_ani=2007 requestid=1 calllegid=
timezone=America/New_York scriptid=PM,-a ani=2007 localOffset=-240 switch_leg_dnis=888040000010011 RouterCallKey=54426103 >>STATE: isTabular=false
isWriteable=false cursor=-1

Agent Name + Greeting Type =

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Agent Greeting Not Playing
Digging Deeper

PG PG

 CVP instructs VXML Gateway to play media file

156: 10.10.10.20: Mar 29 2012 19:33:51.434 -0400: %CVP_8_5_IVR-7-CALL: {Thrd=http-8000-1} VXMLManager:generateVXML:
CALLGUID=A03545000001000000000008280A0A0A Generated VXML from template 'PlayMediaIOS.template' for client: 10.10.10.150 clientType: IOS

 VXML Gateway plays Greeting for both parties via agent phone Built-in-Bridge
153: 10.10.10.20: Mar 29 2012 19:33:51.419 -0400: %CVP_8_5_IVR-7-CALL: {Thrd=http-8000-1} CallSession:addCall:
CALLGUID=A03545000001000000000008280A0A0A Added IVRCall ( DNIS=999020000010012 client: 10.10.10.150) to CallSession at slot: 0

164: 10.10.10.20: Mar 29 2012 19:33:53.309 -0400: %CVP_8_5_IVR-7-CALL: {Thrd=http-8000-1} CallSession:removeCallByDNIS:

CALLGUID=A03545000001000000000008280A0A0A Removed IVRCall from CallSession (DNIS=999020000010012 eventId:6 causeCode:1). Removed from slot: 0

 Call continues normally from this point forward.

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Finesse
Not Windows!
Finesse Cisco Unified
Widget Way Of Life Voice Operating
System (VOS)

 High Level

Authentication
ADS JDBC
Finesse REST / XMPP

GED-188 (CTI)
‒ Requests
‒ Events
PG ‒ Everything else

SIP/SCCP

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Finesse
Widget Way Of Life Finesse Server

 Under the hood Separate Agent & Admin Apps

Tomcat
JDBC Web Services
REST
WebApps
ADS
GED-188
XMPP
Informix DB Notification
Service
PG

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Finesse
Widget Way Of Life Finesse Server

 Under the hood Separate Agent & Admin Apps

Tomcat
JDBC Web Services
REST
WebApps
ADS
GED-188
XMPP
XMPP
Notifications
Informix DB Notification
Service
PG

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
3rd Party
Finesse Application
Widget Way Of Life Finesse Server

 Under the hood Separate Agent & Admin Apps

Tomcat
JDBC Web Services
REST
WebApps
ADS
GED-188
XMPP
XMPP
Notifications
Informix DB Notification
Service
PG

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Finesse Problem Scenario
Intermittent System-wide Agent Login Failures

 Problem Statement: Finesse agent desktop does not load. Agent login
authenticates fine for CTI OS desktop. Agent receives pop-up error:
‒ “Unable to connect to the server. You will be
redirected to the sign-in page.“

 Business Impact: No calls can be handled.

Agent PCs do not have thick CTI OS desktop installed.
 Versions:
‒ Finesse 8.5(3)
‒ UCCE 8.5(3)
 Troubleshooting Performed: I found my Finesse server!
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Finesse Problem Scenario Agent Web
Intermittent System-wide Agent Login Failures Services PG

 Where to start?
 Web Services logs show Finesse connecting to CTI Server
16:35:52.681 CONNECT_TO_RESOURCE: %[resource_name=CTI_SERVER][noname=192.168.25.72]
[noname=42027]: Connecting to resource

 CTI Server creates the session for the Finesse connection

16:35:52:861 cg1A-ctisvr SESSION 1: Opening from client rsuser at (192.168.25.10:50780)
16:35:52:861 cg1A-ctisvr Trace: *** Created ClientCB 1 rsuser

 Finesse sends OPEN_REQ to CTI Server

16:35:52.681 MESSAGE_TO_CTI_SERVER: %[cti_message=Invoke id :6449 , perpheralId: 5000, versionNumber : 16, idleTimeout : 120,
servicesRequested: 0x1140196, callMsgMask: 0x85efff, configMsgMask: 0x3, clientId: rsuser, clientPass: password, clientSignature: null, agentExtension:
null, agentId: null, agentInstrument: null] [cti_message_name=OpenReq]: Message going to the backend cti server

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Finesse Problem Scenario Agent Web
Intermittent System-wide Agent Login Failures Services PG

 CTI Server processes OPEN_REQ, then closes the connection ??

16:35:52:861 cg1A-ctisvr Trace: CTIApplication::ProcessOpenRequest - sessionID 1 from rsuser at (192.168.25.10:50780), version 16
16:35:52:861 cg1A-ctisvr Trace: CTIApplication::OpenValidAgent from sessionID 1 (rsuser) - ACCEPTED (0 ms)
16:35:52:877 cg1A-ctisvr SESSION 1: Closing client rsuser at (192.168.25.10:50780)

 Finesse reports that there is a configuration issue.

0000047814: 192.168.25.10: Mar 01 2012 16:35:52.686 -0800: %CCBU_CTIMessageEventExecutor-0-6-MESSAGE_FROM_CTI_SERVER:
%[cti_message_name=OpenConf][cti_sequence_id=8379]: Message coming to Finesse from backend cti server

0000047816: 192.168.25.10: Mar 01 2012 16:35:52.686 -0800: %CCBU_pool-7-thread-1-3-CTI_OPEN_FAILURE:

%[ERROR_DESCRIPTION=][Response=-1]: CTI Server rejected open request ; There is a configuration mismatch between
this server and the CTI server

 …but where?
‒ IP Addresses, Ports obviously correct
‒ What caused CTI Server to suddenly close the connection?
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Finesse Problem Scenario Agent Web
Intermittent System-wide Agent Login Failures Services PG

 Just before the connection issues we see 4 peripherals offline

16:35:48:861 cg1A-ctisvr PG System Event: Peripheral Online (PeripheralID 5000), PG Status: NORMAL
16:35:48:861 cg1A-ctisvr PG System Event: Peripheral Offline (PeripheralID 5001), PG Status: PERIPHERAL(S) OFFLINE
16:35:48:861 cg1A-ctisvr PG System Event: Peripheral Offline (PeripheralID 5002), PG Status: PERIPHERAL(S) OFFLINE
16:35:48:861 cg1A-ctisvr PG System Event: Peripheral Offline (PeripheralID 5003), PG Status: PERIPHERAL(S) OFFLINE
16:35:48:861 cg1A-ctisvr PG System Event: Peripheral Offline (PeripheralID 5004), PG Status: PERIPHERAL(S) OFFLINE

 What type of PG is this?

 With Generic PG, if any of the PIMs are not active, Finesse will not go In Service
‒ Finesse 8.5(3) does not support Generic PG due to the way it determines PG status
‒ This feature was added in Finesse 8.5(3) ES1 and 9.0 (out soon!)

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
Precision Routing
Precision Routing
The Best of the Best (Available)

 Traditional Skill Group Routing

Sales

Support

Returns

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
Precision Routing Attribute Value
The Best of the Best (Available) English 9
Sales 8
 Attribute Routing Returns 5
‒ Simpler scripting Boston True
‒ Multidimensional Attribute Value
‒ Multi-step English 9
Sales 5
Returns 8
Boston True

Attribute Value
English 7
Spanish 8
Support 10
Baltimore True
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Precision Routing Problem Scenario
Agents & Attributes

 Problem Statement: Agents not receiving call routed to Precision Queue.

 Business Impact: Calls are staying in queue and not reaching agents.
Customers are hanging up in queue. Losing an average of $1000 per call and
impacting metrics.
 Versions:
‒ UCCE 9.0(1)
‒ CVP 9.0(1)
‒ CUCM 9.0(1)
 Troubleshooting Performed:
‒ Verified calls are coming into the routing script
‒ Real-time Data shows agent available in PQ

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
Precision Routing Problem Scenario
Agents & Attributes

 Precision Queue node is routing statically

‒ Can rule out any dynamic formulas in troubleshooting process

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
Precision Routing Problem Scenario
Agents & Attributes

 What does the Router log show?

ra-rtr Trace: (99693 56226 13213 : 0 0) RequestInstr: CID=(150210,2982), CallState=1
ra-rtr Trace: (56226 56226 13213 : 0 0) Dialog resuming (Request Instruction received.) status (0)
ra-rtr Trace: (56226 56226 13213 : 0 0) CID=(150210,2982):PQ1:CurStep 0
ra-rtr Trace: (56226 56226 13213 : 0 0) CID=(150210,2982):PQ1:Step 1:Eval true
ra-rtr Trace: (56226 56226 13213 : 0 0) CID=(150210,2982):PQ1:Step 1:No Agents Logged In
ra-rtr Trace: (56226 56226 13213 : 0 0) CID=(150210,2982):PQ1:Step 2:Eval true
ra-rtr Trace: (56226 56226 13213 : 0 0) CID=(150210,2982):PQ1:Step 2:No Agents Logged In
ra-rtr Trace: (56226 56226 13213 : 0 0) CID=(150210,2982):PQ1:Step 3:Last Step:Eval true
ra-rtr Trace: (56226 56226 13213 : 0 0) CID=(150210,2982):PQ1:Step 3:No Agents Logged In
ra-rtr Trace: (56226 56226 13213 : 0 0) Queuing Call CID=(150210,2982) for PQStep PQ1.3,Pos=1
ra-rtr Trace: (56226 56226 13213 : 0 0) CID=(150210,2982):PQ1:Step 3:Last Step, no timer

 Agent is Available in PQ1

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
Precision Routing Problem Scenario
Agents & Attributes

 What attribute set is PQ1 looking for?

= Agent pool smaller than previous step

‒ Subsequent steps should expand the agent pool

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
Precision Routing Problem Scenario
Agents & Attributes

 What attribute(s) does our agent have?

5 is not > 5!

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
Precision Routing Problem Scenario
Agents & Attributes

 Two Options:
‒ Adjust the Precision Queue Step. “Lower the bar” to include agent
‒ Time for an attribute promotion. Make the agent a Superstar!

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
Precision Routing Problem Scenario
Agents & Attributes

 Agent now belongs to “Superstar” attribute pool in third step, receives call.
ra-rtr Trace: (99716 56249 13236 : 0 0) RequestInstr: CID=(150210,3005), CallState=1
ra-rtr Trace: (56249 56249 13236 : 0 0) Dialog resuming (Request Instruction received.) status (0)
ra-rtr Trace: (56249 56249 13236 : 0 0) CID=(150210,3005):PQ1:CurStep 0
ra-rtr Trace: (56249 56249 13236 : 0 0) CID=(150210,3005):PQ1:Step 1:Eval true
ra-rtr Trace: (56249 56249 13236 : 0 0) CID=(150210,3005):PQ1:Step 1:No Agents Logged In
ra-rtr Trace: (56249 56249 13236 : 0 0) CID=(150210,3005):PQ1:Step 2:Eval true
ra-rtr Trace: (56249 56249 13236 : 0 0) CID=(150210,3005):PQ1:Step 2:No Agents Logged In
ra-rtr Trace: (56249 56249 13236 : 0 0) CID=(150210,3005):PQ1:Step 3:Last Step:Eval true
ra-rtr Trace: (56249 56249 13236 : 0 0) CID=(150210,3005):PQ1:Step 3 sending call to Agent 38213
ra-rtr Trace: (56249 56249 13236 : 0 0) RouteComplete:
ra-rtr Trace: Route: DN=903000, ANI=3009, CED=, Label=3001
ra-rtr Trace: Route: CID=(150210,3005), Labels=1

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
Multimedia Integration
Multimedia Integration
Combining Email, Chat, and Voice

 High-level Flow
GED-188 (CTI)
‒ Agent
availability
for Task Agent
PG

MR
GED-125 (MR) PG
‒ New Task
‒ Do This With Task

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
Multimedia Integration
Combining Email, Chat, and Voice

Key Differences from Voice: Email

MRD
 Email/Chat Media Routing Domains (MRD)
A collection of skill groups and services associated
Sales Support Billing
with a specific communication channel.
– UCCE Scripting and Media Routing Guide

 Concurrency
How many tasks can an agent handle in an MRD?

 Interruptibility
Can one task be interrupted by another task?
Concept of “Available” vs. “ICM Available”
I.E. Agent “Available” for email but not “ICM Available”

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
Interruptibility & Handle Time
Combining Email, Chat, and Voice

 Problem Statement: Agent Productivity confusion.

 Business Impact: Agents paid based on activity efficiency. Reported
numbers are unclear to supervisors.
 Versions:
‒ UCCE 8.5(3)
‒ EIM/WIM 4.4(1)
 Troubleshooting Performed:
‒ Verified interruptibility settings for Chat and Email MRDs are correct.
‒ Verified concurrent task limits defined in EIM/WIM.
‒ Paid attention to this crash course!

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
Interruptibility & Handle Time
Combining Email, Chat, and Voice
Media Routing Domain Email Chat
Concurrent Task Limit 3 2

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
Interruptibility & Handle Time
Combining Email, Chat, and Voice
Media Routing Domain Email Chat
Concurrent Task Limit 3 2

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
Interruptibility & Handle Time
Combining Email, Chat, and Voice
Media Routing Domain Email Chat
Concurrent Task Limit 3 2

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
Interruptibility & Handle Time
Combining Email, Chat, and Voice
Media Routing Domain Email Chat
Concurrent Task Limit 3 2

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
Interruptibility & Handle Time
Combining Email, Chat, and Voice
Media Routing Domain Email Chat
Concurrent Task Limit 3 2

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
Interruptibility & Handle Time
Combining Email, Chat, and Voice
Media Routing Domain Email Chat
Concurrent Task Limit 3 2

Interrupted!

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
Interruptibility & Handle Time
Combining Email, Chat, and Voice
Media Routing Domain Email Chat
Concurrent Task Limit 3 2

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
Interruptibility & Handle Time
Combining Email, Chat, and Voice
Media Routing Domain Email Chat
Concurrent Task Limit 3 2

Interrupted!

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
Interruptibility & Handle Time
Combining Email, Chat, and Voice
Media Routing Domain Email Chat
Concurrent Task Limit 3 2

Interrupted!

 External Agent Assignment Service GED-125 MR PIM

18:38:19:693 pg2A-pim2 Trace: Application->PG:
Message = NEW_TASK; Length = 102 bytes
DialogueID = (10) Hex 0000000a
SendSeqNo = (1) Hex 00000001 18:38:19:693 pg2A-pim2 Trace: PG->Application:
MRDomainID = (5003) Hex 0000138b Message = DO_THIS_WITH_TASK; Length = 121 MR Agent
PreviousTask = -1:-1:-1 bytes PG PG
PreferredAgent = Undefined DialogueID = (10) Hex 0000000a
Service = (0) Hex 00000000 SendSeqNo = (1) Hex 00000001
CiscoReserved = (0) Hex 00000000 IcmTaskID = 150083:310: 1 Router CallKey
ScriptSelector: 8000 SkillGroup = (5036) Hex 000013ac
ECC Variable Name: user.cim.activity.id Service = Undefined
Value: 1372 Agent = (5003) Hex 0000138b Agent STID
AgentInfo: 1003
EIM/WIM AgentID
Label:
ECC Variable Name: user.cim.activity.id
 MR PIM Finished!
Value: 1372
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
Interruptibility & Handle Time
How does UCCE see all of this?

 Listener GED-188 CTI Server

 Pre-Call Indicator, Links The RouterCallKey & DialedNumber
18:38:19:693 cg1A-ctisvr Trace: DEVICE_TARGET_PRE_CALL_IND - Instrument= RouterCallKey=150083 310
NetworkTargetID=0
18:38:19:709 cg1A-ctisvr Trace: SvSkTargID=-1(-1) SkGroupSkTarID=5036(531)
18:38:19:709 cg1A-ctisvr Trace: AgSkTargID=5003 ANI= CED= Agent
PG
18:38:19:709 cg1A-ctisvr SESSION 2: MsgType:AGENT_PRE_CALL_EVENT (MonitorID:0 NumNamedVariables:2
NumNamedArrays:0 ServiceNumber:N/A
18:38:19:709 cg1A-ctisvr SESSION 2: ServiceID:N/A SkillGroupNumber:531 SkillGroupID:5036
SkillGroupPriority:0 MRDID:5003
18:38:19:709 cg1A-ctisvr SESSION 2: AgentInstrument:"" RouterCallKeyDay:150083 RouterCallKeyCallID:310
18:38:19:709 cg1A-ctisvr SESSION 2: RouterCallKeySequenceNum:1 DialedNumber:"8000" )

 Indicators are generally sent from an integrated application

‒ EXCEPT DEVICE_TARGET_PRE_CALL_IND which is sent from Router.
 Events are calculations that occur within OPC for interval reporting
& All Event Clients
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
Interruptibility & Handle Time
How does UCCE see all of this?

 Listener GED-188 CTI Server

 Email Offered & Started
18:38:19:724 cg1A-ctisvr SESSION 3: MsgType:OFFER_TASK_IND (InvokeID:0x2c767 MRDID:5003 ICMAgentID:5003
18:38:19:724 cg1A-ctisvr SESSION 3: TaskID:150083/310/1ARMSkillGroupID:5036 ARMServiceID:N/A )

18:38:19:724 cg1A-ctisvr Trace: OFFER_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003 Agent

18:38:19:724 cg1A-ctisvr Trace: TaskID=150083:310:1 ICMAgentID=5003 SkillGroupID=5036 PG
18:38:19:724 cg1A-ctisvr Trace: ServiceID=5617416 ApplicationRouted

18:38:25:537 cg1A-ctisvr SESSION 3: MsgType:START_TASK_IND (InvokeID:0x2c768 MRDID:5003 ICMAgentID:5003

18:38:25:537 cg1A-ctisvr SESSION 3: TaskID:150083/310/1ARMSkillGroupID:5036 ARMServiceID:N/A )

18:38:25:537 cg1A-ctisvr Trace: START_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003

18:38:25:537 cg1A-ctisvr Trace: TaskID=150083:310:1 ICMAgentID=5003 SkillGroupID=5036
18:38:25:537 cg1A-ctisvr Trace: ServiceID=5617416 ApplicationRouted

 Handle Time Starts Counting

 Listener GED-188 CTI Server

 Agent “Active” In Multimedia SG
18:38:25:537 cg1A-ctisvr Trace: AGENT_EVENT: ID=21 Periph=5000 Ext= Inst= Sig=
18:38:25:537 cg1A-ctisvr Trace: SGState=ACTIVE SGDuration=0 OverallState=ACTIVE OverallDuration=0 Reason=-1
18:38:25:537 cg1A-ctisvr Trace: MRDID=5003 NumTasks=1 MaxTaskLimit=2 AgtMode=1 AgtAvailStatus=1
Agent
ICMAgentID=5003
PG
18:38:25:537 cg1A-ctisvr Trace: SkTgtID=5003 SkGrpNo=0x213 SkGrpID=5036 NumLines=0 CurLine=-1
ClientStatus=0x0

 Agent Can Still Receive A Second Task

‒ NumTasks=1, MaxTaskLimit=2

 Listener GED-188 CTI Server

 Agent Gets Second Email (Pre_Call, Offer) and Reads It
‒ Handle Time “pauses” for original email and starts for new email
18:40:33:632 cg1A-ctisvr Trace: PAUSE_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003
18:40:33:632 cg1A-ctisvr Trace: TaskID=150083:310:1 ICMAgentID=5003 Reason=5000 Agent
PG
18:40:33:647 cg1A-ctisvr Trace: START_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003
18:40:33:647 cg1A-ctisvr Trace: TaskID=150083:311:1 ICMAgentID=5003 SkillGroupID=5036
18:40:33:647 cg1A-ctisvr Trace: ServiceID=5617416 ApplicationRouted

 Second Email Completed, Original Email Resumed

18:40:39:929 cg1A-ctisvr Trace: END_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003
18:40:39:929 cg1A-ctisvr Trace: TaskID=150083:311:1 PreviousTaskID=-1:-1:-1
18:40:39:929 cg1A-ctisvr Trace: ICMAgentID=5003

18:40:42:788 cg1A-ctisvr Trace: RESUME_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003

18:40:42:788 cg1A-ctisvr Trace: TaskID=150083:310:1 ICMAgentID=5003 Reason=5000

 Listener GED-188 CTI Server

 Original Email Interrupted By Chat
18:40:49:710 cg1A-ctisvr SESSION 3: MsgType:AGENT_INTERRUPT_ADVISORY_EVENT (InvokeID:0x2
MRDID:5001 InterruptingMRDID:5002 ICMAgentID:5003 )
18:40:49:710 cg1A-ctisvr SESSION 3: MsgType:AGENT_INTERRUPT_ADVISORY_EVENT (InvokeID:0x2
MRDID:5003 InterruptingMRDID:5002 ICMAgentID:5003 ) Agent
PG
 What are these MRDs? Where can we find their names?
USE bxb_awdb select * from t_Media_Routing_Domain

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 112
MRD Media
ID
Interruptibility & Handle Time 5001 Outbound
How does UCCE see all of this? Email
5002 Chat
5003 Inbound
 Listener GED-188 CTI Server Email

 Agent Reserved For Chat SG; Interrupted For Email SG

18:40:49:710 cg1A-ctisvr Trace: AGENT_EVENT: ID=21 Periph=5000 Ext= Inst= Sig=
18:40:49:710 cg1A-ctisvr Trace: SGState=RESERVED SGDuration=0 OverallState=RESERVED OverallDuration=0 Reason=-1
18:40:49:710 cg1A-ctisvr Trace: MRDID=5002 NumTasks=1 MaxTaskLimit=1 AgtMode=1 AgtAvailStatus=0 ICMAgentID=5003
18:40:49:710 cg1A-ctisvr Trace: SkTgtID=5003 SkGrpNo=0x21d SkGrpID=5039 NumLines=0 CurLine=-1 ClientStatus=0x0 Agent
PG
18:40:49:725 cg1A-ctisvr SESSION 3: MsgType:AGENT_INTERRUPT_ACCEPTED_IND (InvokeID:0x2 MRDID:5001 ICMAgentID:5003 )
18:40:49:725 cg1A-ctisvr SESSION 3: MsgType:AGENT_INTERRUPT_ACCEPTED_IND (InvokeID:0x2 MRDID:5003 ICMAgentID:5003 )

18:40:49:725 cg1A-ctisvr Trace: AGENT_EVENT: ID=21 Periph=5000 Ext= Inst= Sig=

18:40:49:725 cg1A-ctisvr Trace: SGState=INTERRUPTED SGDuration=0 OverallState=INTERRUPTED OverallDuration=0 Reason=0
18:40:49:725 cg1A-ctisvr Trace: MRDID=5003 NumTasks=1 MaxTaskLimit=2 AgtMode=1 AgtAvailStatus=0 ICMAgentID=5003
18:40:49:725 cg1A-ctisvr Trace: SkTgtID=5003 SkGrpNo=0x213 SkGrpID=5036 NumLines=0 CurLine=-1 ClientStatus=0x0

 Offer, Start, End Task Indicators/Events and

state transitions mimic Email
 Note: Handle time for chat increments for all running sessions.
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
Interruptibility & Handle Time
How does UCCE see all of this?

 How Can a Failure Event Be Mapped to a Task/Activity?

18:38:19:693 pg2A-pim2 Trace: PG->Application:
Message = NEW_TASK_FAILURE_EVENT; Length = 12 bytes
DialogueID = (10) Hex 0000000a
SendSeqNo = (1) Hex 00000001
ReasonCode = (209) Hex 000000d1
MR Agent
 Use DialogueID to Correlate PG PG
18:38:19:693 pg2A-pim2 Trace: Application->PG: 18:38:19:693 pg2A-pim2 Trace: PG->Application:
Message = NEW_TASK; Length = 102 bytes Message = DO_THIS_WITH_TASK; Length = 121 bytes
DialogueID = (10) Hex 0000000a DialogueID = (10) Hex 0000000a
SendSeqNo = (1) Hex 00000001 SendSeqNo = (1) Hex 00000001
MRDomainID = (5003) Hex 0000138b IcmTaskID = 150083:310: 1
PreviousTask = -1:-1:-1 SkillGroup = (5036) Hex 000013ac
PreferredAgent = Undefined Service = Undefined
Service = (0) Hex 00000000 Agent = (5003) Hex 0000138b
CiscoReserved = (0) Hex 00000000 AgentInfo: 1003
ScriptSelector: 8000 Label:
ECC Variable Name: user.cim.activity.id ECC Variable Name: user.cim.activity.id
Value: 1372 Value: 1372
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 114
Network Optimization

115
Network Optimization
You‟ve Heard It A Thousand Times…+1

 NIC Teaming Not Supported

‒ Load balancing can negatively affect critical HW I/O continuity
 Binding Order
‒ Public NIC should be listed first followed by Private NIC
 Speed And Duplex Settings
‒ Always full duplex (exceptions with some peripherals, e.g. Avaya)
‒ Statically set and must match on both the NIC and Switch Port, respectively
‒ Use Auto/Auto ONLY when running Gigabit NIC/Switch
 Best Practices tech tip
‒ http://www.cisco.com/en/US/products/sw/custcosw/ps1001/products_tech_note09
186a00808160f4.shtml
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 116
Network Optimization
Important! Urgent! Please Read! Nota Bene!

 TCP/IP Offloading, Potential Loss of Server Network Connection

‒ Offloads some TCP/IP processing to network adapters
‒ Causes serious issues with UCCE & CVP components
‒ Enabled by default with Windows 2003 SP2
‒ http://www.cisco.com/en/US/ts/fn/632/fn63215.html
‒ http://blogs.technet.com/b/networking/archive/2008/11/14/the-effect-of-tcp-
chimney-offload-on-viewing-network-traffic.aspx
‒ http://en.wikipedia.org/wiki/TCP_Offload_Engine
 Microsoft TCP/IP Stack Logic Affects MDS TCP Characteristics (RTT)
‒ http://www.cisco.com/en/US/ts/fn/620/fn62508.html
 Microsoft Scalable Networking Pack With TCP/IP Offloading
‒ http://support.microsoft.com/default.aspx?scid=kb;EN-US;948496
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 117
Network Optimization
A Few of the Many Faces of TCP Offload / TCP Chimney

 Intermittent Silent Monitoring Disconnections, Failures

‒ 10:15:17.517 552 SilentMonitorService Thd(1292) CUDPPort(008AFB90)::ReceiveData, recvfrom,
Blocking call interrupted by WSACancelBlockingCall. (WSAEINTR), Error Number(10004).

 Lost Heartbeats, Test-Other-Side (ToS) Seen in MDS

‒ ra-mds Trace: EMT 10: Missing heartbeat, countdown is 5, last received seqno 269795
‒ pg1b-mds Trace: Sending TOS request: sequence = 1.
pg1b-mds Trace: Received TOS response: sequence=1 status=ENABLED.
 Logger Sync issues during business day, then clearing up at night
‒ Recovery keys different between A-side and B-side
Validate All Windows 2003 Servers: netsh int ip show offload + NIC Drivers!
Validate All Windows 2008 Servers: netsh in tcp show global + NIC Properties!
 http://www.cisco.com/en/US/ts/fn/632/fn63215.html
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 118
Solution Approach to Troubleshooting
“If I find 1,000 ways something won‟t work, I
haven‟t failed. I am not discouraged, because
every wrong attempt discarded is another step
forward.”
Thomas Edison
U.S. Inventor (1847 – 1931)
Communicate, Collaborate, Succeed
Resolving Problems with a Solution-based Approach

 “Trust, but verify.” ~Ronald Reagan

“What and when did this happen?” “What changes have you made?”
“Don’t ask me.” “None!”
“What version are you running?” “Did you enable tracing?”
“9.b(7c)” “Yes.”
“Did you look at the logs?”
“Who is affected?”
“Yep.”
“Everyone!”
“It’s been 10 minutes. When are you going to fix this?!”
 Accurately Confirming Critical Items Helps With…
 Baseline(s)
 Accurate troubleshooting
 Improving feedback and collaboration
 Reducing cycles, guess work, relying on luck, etc.
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
Communicate, Collaborate, Succeed
Resolving Problems with a Solution-based Approach

Clarify Think Beyond

Problem Problem
Description the Fix…

Determine
Test Likely
Possible
Causes
Causes

Investigate Solution

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 122
Communicate, Collaborate, Succeed
Resolving Problems with a Solution-based Approach
Situational • Brokering Reasonable Expectations
Analysis • Fair and Balanced Approach

Emotions
Technical • Formulate Baseline
Analysis • Deliver Logs, Recordings, Screenshots
Politics
• Clear Expectations
Action Plan • Formulate Next Steps, Workarounds
Fear
Resource • Alignment and Utilization
Management

Technical Solution • Collaborate on Big Picture

Approach

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 123
Communicate, Collaborate, Succeed
Communicating Effectively
Listen
• Establishes respect
• Acts as a pressure valve
• Shows you care

Understand
• Ask questions to re-enforce what you were told
• Echo and summarize your understanding

Check And Assess Action Plan

• Log Validation
• Reproduce
• Test Cause & Effect
• Remember, “Trust, but verify.”

Collaborate
• Avoid transactional dialog
• “Tell me, I will forget. Show me, I may remember. Involve me and I will understand.”

 Have a clear, concise problem description.

‒ Helps to know what we are troubleshooting.

 Take Logical, Methodological Approach

‒ Think through the problem, trial and error is not efficient

 Technical with Solution Management

‒ Clear communication with proper expectations

 Problem Statement: Agents getting stuck in RESERVED state for 30

seconds, not receiving the call being routed to them.
 Business Impact: Calls being rerouted introduce delays, exceed
SLAs. This is affecting 20% of calls, causing financial penalties for the
call center.
 Versions:
‒ UCCE 8.5(3)
‒ CVP 8.5(1)
‒ CUCM 8.5(1)
 Troubleshooting Performed: Isolated issue to single site.
 What‟s the call flow?
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 126
Solution Problem Isolation
Narrowing Down The Problem

PG PG

 Call comes in to CVP from Voice Gateway

Aug 30 2011 15:20:52.360 +0530: %CVP_8_0_SIP-7-CALL: {Thrd=DATAI.11} NEW CALL with guid=1A1935C8100001326975748653ED69C3
legid=8D03395A-D22311E0-9734A76F-421B7E24 dn=61401211 ani=+918946861782 uui=null calldate=Tue Aug 30 15:20:52 IST 2011 video=false
cachecallcontext = false is_postcallsurvey = false RouterCallKey = null RouterCallKeyDay = null RouterCallKeySequenceNumber = null

 Caller requests to speak with an agent

Aug 30 2011 15:26:12.031 +0530: %CVP_8_0_ICM-7-CALL: {Thrd=pool-1-thread-34-ICM-16843525} CALLGUID =
1A1935C8100001326975748653ED69C3, DLGID = 401702 [IVR_LEG] - Publishing ,, [ICM_RUN_SCRIPT_RESULT], dialogueId=401702,
sendSeqNo=3, invokedId=2, resultCode=true, ced=-1, newTransaction=false, CallContext:, user.microapp.caller_input: 1,
user.microapp.error_code: 0, user.microapp.metadata: N|000|01|00|00|019390|GS,Server,V, user.microapp.FromExtVXML[0]: +918946861782,
user.microapp.FromExtVXML[1]: hin,, LEGID = , DNIS = 10582222217939, ANI = sip:[email protected]:5060

PG PG

 Router sends a Connect message for extension 19797

15:26:16:045 ra-rtr Trace: (401693 401702 17939 : 0 0) DeviceTargetPreCall_V11: CID=(149990,405494), PerID=5000, SGSTID=7037, AGSTID=8294,
NetworkCallFlags=0, InvokeID=1830299, GREET=
15:26:16:045 ra-rtr Trace: (401693 401702 17939 : 0 0) Connect: CID=(149990,405494), EventSelect=0, ServiceType=0, RCID=5005,
Label(s)=19797

 CVP receives the Connect message

Aug 30 2011 15:26:15.922 +0530: %CVP_8_0_ICM-7-CALL: {Thrd=pool-1-thread-459-ICM-16843638} CALLGUID = 1A1935C8100001326975748653ED69C3,
DLGID = 401693 [SIP_LEG] - Publishing ,, [MsgBus:CONNECT], ssId=SYS_SIP2, labelType=NORMAL, label=19797, correlationId=null,
command=null, rckey=405494, rcday=149990, rcseq=0, CallContext:, CallVariable1: 8946861782, CallVariable2: NA|NA, CallVariable3:
1A1935C8100001326975748653ED69C3, etc...…..

PG PG

 Router sends a Connect message for extension 19797

15:26:16:045 ra-rtr Trace: (401693 401702 17939 : 0 0) DeviceTargetPreCall_V11: CID=( 149990,405494
CID=(149990,405494), ), PerID=5000,
PerID=5000, SGSTID=7037,
SGSTID=7037, AGSTID=8294,
AGSTID=8294,
NetworkCallFlags=0,
NetworkCallFlags=0, InvokeID=1830299,
InvokeID=1830299 GREET=
, GREET=
15:26:16:045
15:26:16:045 ra-rtr
ra-rtr Trace: (401693 401702
Trace: (401693 401702 17939
17939 :: 0 0) Connect:
0 0) Connect: CID=(149990,405494),
CID=(149990,405494), EventSelect=0,
EventSelect=0, ServiceType=0,
ServiceType=0, RCID=5005,
RCID=5005,
Label(s)=19797
Label(s)=19797
 CVP receives the Connect message
Aug 30 2011 15:26:15.922 +0530: %CVP_8_0_ICM-7-CALL: {Thrd=pool-1-thread-459-ICM-16843638} CALLGUID = 1A1935C8100001326975748653ED69C3,
DLGID = 401693 [SIP_LEG] - Publishing ,, [MsgBus:CONNECT], ssId=SYS_SIP2, labelType=NORMAL, label=19797, correlationId=null,
command=null, rckey=405494, rcday=149990, rcseq=0, CallContext:, CallVariable1: 8946861782, CallVariable2: NA|NA, CallVariable3:
1A1935C8100001326975748653ED69C3, etc...…..

PG PG

 Use InvokeID or RCK to match PreCall message in CTI Server logs

15:26:15:723 cg1B-ctisvr Trace: DEVICE_TARGET_PRE_CALL_IND - Instrument=19797 RouterCallKey=149990 405494
15:26:15:723 cg1B-ctisvr Trace: SvSkTargID=-1(-1) SkGroupSkTarID=7037(617)
15:26:15:723 cg1B-ctisvr Trace: AgSkTargID=8294 ANI=+918946861782 CED=-1
15:26:15:723 cg1B-ctisvr Trace: CallTypeID=5535 PreCallInvokeID=1830299

 Agent was AVAILABLE in call‟s destination SG just before this PreCall

15:26:15:504 cg1B-ctisvr Trace: AGENT_EVENT: ID=20415 Periph=5000 Ext=19797 Inst=19797 Sig=CTIOSServer
15:26:15:504 cg1B-ctisvr Trace: SkgState=AVAILABLE SkgDuration=0 OverallState=AVAILABLE OverallDuration=0 Reason=0
15:26:15:504 cg1B-ctisvr Trace: MRDID=1 NumTasks=0 MaxTaskLimit=1 AgtMode=1 AGTAvailabilityStatus=1 ICMAgtID=8294
15:26:15:504 cg1B-ctisvr Trace: SkTgtID=8294 SkGrpNo=0x269 SkGrpID=7037 NumLines=1 CurLine=0 ClientStatus=0x1

PG PG

 Use InvokeID or RCK to match PreCall message in CTI Server logs

 Agent was RESERVED in call‟s destination SG concurrent with PreCall

15:26:15:723 cg1B-ctisvr Trace: AGENT_EVENT: ID=20415 Periph=5000 Ext=19797 Inst=19797 Sig=CTIOSServer
15:26:15:723 cg1B-ctisvr Trace: SkgState=RESERVED SkgDuration=0 OverallState=RESERVED OverallDuration=0 Reason=0
15:26:15:723 cg1B-ctisvr Trace: MRDID=1 NumTasks=1 MaxTaskLimit=1 AgtMode=1 AGTAvailabilityStatus=0 ICMAgtID=8294
15:26:15:723 cg1B-ctisvr Trace: SkTgtID=8294 SkGrpNo=0x269 SkGrpID=7037 NumLines=1 CurLine=0 ClientStatus=0x1

PG PG

 UCCE and CUCM notify CVP that call has been Answered
Aug 30 2011 15:26:16.172 +0530: %CVP_8_0_ICM-7-CALL: {Thrd=pool-1-thread-284-ICM-16843644} CALLGUID =
1A1935C8100001326975748653ED69C3, DLGID = 401693 [SIP_LEG] - Publishing ,, [ICM_EVENT_REPORT], dialogueId=401693, sendSeqNo=3,
eventId=ANSWER, causeCode=NONE,, LEGID = 8D03395A-D22311E0-9734A76F-421B7E24, DNIS = 61401211, ANI = +918946861782
From UCCE
Aug 30 2011 15:26:16.172 +0530: %_UserCB-6-com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.LlSM.client.UserCB: SIP/2.0 200 OK
Via: SIP/2.0/UDP 172.18.150.61:5060;branch=z9hG4bKI5Y12NeaMEV6dImqSWb3TA~~8933838
To: "--CVP_8_0_1_0_0_0_1440" <sip:[email protected]>;tag=A269BB6C-77D From CUCM

 But the Agent PG is never notified that this call arrived at device 19797.
 What does a normal call look like? What do we expect to see?
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 132
Solution Problem Isolation
Narrowing Down The Problem

PG PG

 For a NORMAL call, we should see after the Reservation:

‒ CSTA_DELIVERED
‒ CSTA_ESTABLISHED
‒ RTP_STARTED / RTP_STOPPED
‒ CSTA_CONNECTION_CLEARED
‒ Agent transition to NOT_READY state (or wrap-up etc)
 None of these messages are seen for the call in question.
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 133
Solution Problem Isolation
Narrowing Down The Problem

PG PG

 PGs internal timer expires in 30 seconds, reverting agent to AVAILABLE

15:26:45:945 PG1A-pim1 Trace: RemovePreRoutedCall: Head Inst: N19797 CreatedByPreCall: T PeriphCID: -1 Queue Count: 0
RouterCallKey=(149990/405494)
15:26:45:945 PG1A-pim1 Trace: ProcessReservedTimeout: No call arrived to match PreCall message. netTargID=100780 Ext=19797
AgentID=20415 RTRCallKey=149990.405494 DtAborts=1
15:26:45:945 PG1A-pim1 Trace: SendDeviceTargetAbortInd: agentSkillTargetID = 8294
15:26:45:945 PG1A-pim1 Trace: SetPrevalentAgentState: (Agent 20415 ext 19797) newAgentState=AS_AVAILABLE(3) reasoncode=0
stateDuration=0 currentLineHandle=-1 lineType=LT_UNKNOWN(3) newLineState=LS_UNKNOWN(5) devString=
15:26:45:945 PG1A-pim1 Trace: SetPrevalentState: newState=AS_AVAILABLE(3) reasoncode=0
15:26:45:945 PG1A-pim1 Trace: CancelAgentTimer: type = agentTimerReserved

 Agent Reserved, PG waiting, CVP Answered. So what‟s left?

PG PG

 CUCM logs show agent going off-hook and on-hook repeatedly.

15:26:15.135 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164601^
15:26:16.252 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085009.|6,100,50,1.4164606^
15:26:17.372 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164651^
15:26:18.490 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085041.|6,100,50,1.4164655^
15:26:19.522 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164719^
15:26:20.609 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085084.|6,100,50,1.4164725^

 CUCM rejects routed call while off-hook

15:26:17.740 |LineControl(2351) - 1 calls, 0 CiReq, busyTrigger=1, maxCall=2|6,100,57,1.399447^172.18.150.120^*
15:26:17.740 |//SIP/SIPTcp/wait_SdlSPISignal: Outgoing SIP TCP message to 172.18.150.120 on port 37776 index 13 [1449373,NET]
SIP/2.0 486 Busy here

PG PG

 CUCM logs show agent going off-hook and on-hook repeatedly.

15:26:15.135 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164601^
15:26:16.252 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085009.|6,100,50,1.4164606^
15:26:17.372 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164651^
15:26:18.490 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085041.|6,100,50,1.4164655^
15:26:19.522 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164719^
Hit the Floor!
15:26:20.609 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085084.|6,100,50,1.4164725^

 CUCM rejects routed call while off-hook

PG PG

 CUCM logs show agent going off-hook and on-hook repeatedly.

15:26:15.135 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164601^
15:26:16.252 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085009.|6,100,50,1.4164606^
15:26:17.372 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164651^
15:26:18.490 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085041.|6,100,50,1.4164655^
15:26:19.522 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164719^
Hit the Floor!
15:26:20.609 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085084.|6,100,50,1.4164725^

 CUCM rejects routed call while off-hook

PG PG

 CUCM logs show agent going off-hook and on-hook repeatedly.

‒ Agents do not have enough time in between calls
‒ Agent Desk Settings issue, possible staffing issue
 ACTION PLAN:
‒ After-Call Wait Time should be increased to meet agent needs.
‒ Call volume should be evaluated to align with agent staffing.

 Serviceability
‒ New Tools
‒ Old Tools with New Tricks
 Real Customer Scenarios
‒ Basic Agent Greeting Call Flow
‒ Finesse Configuration Fundamentals
‒ Multimedia 101
‒ TCP Offload Anomalies
 Solution Approach to Issue Resolution
 Time for Q&A!

Effective Tools
Efficient Easier
Troubleshooting
Life
 Keep This Presentation as a Reference!
 Homework: Implement CLI in your environment & USE IT!

Thank You
 Enjoy tonight‟s events, have a safe trip home.
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 141
Complete Your Online
Session Evaluation
 Give us your feedback and you
could win fabulous prizes.
Winners announced daily.
 Receive 20 Passport points for each
session evaluation you complete.
 Complete your session evaluation
online now (open a browser through
our wireless network to access our Don‟t forget to activate your
portal) or visit one of the Internet Cisco Live Virtual account for access to
stations throughout the Convention all session material, communities, and
on-demand and live activities throughout
Center. the year. Activate your account at the
Cisco booth in the World of Solutions or visit
www.ciscolive.com.

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 142
Connect with Your Peers and Cisco
 Discuss business, IT, architecture, adoption
and product topics with peers Cisco Collaboration Community and
 conferencing, customer care, enterprise social software,
IP communications, messaging, mobile applications, User Group
telepresence

 Interact with Cisco Product Managers,

Technical Marketing Engineers and Services
Consultants
 Learn about new product announcements
 Join the Collaboration User Group
 Influence product direction
communities.cisco.com
 Access to early product releases and info
 Get VIP perks and exclusive sessions at Cisco Live
Visit the Cisco Collaboration Booth (#1289) at Cisco Live:
Learn more about the community. Sign up for the user group.
Pick up your polo shirt and badge ribbon (user group members only).
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Simple Network Management Protocol
It‟s a Trap!!

1.3.6.1.4.1.9.9.473.0.1
1.3.6.1.4.1.9.9.473 Root Of CISCO-CONTACT-CENTER-APPS-MIB
iso.organization.dod.internet.private.enterprises.cisco.ciscoMgmt.ciscoCcaMIB

1.3.6.1.4.1.9.9.473.0.1
ciscoCcaMIBNotifs(0).cccaIcmEvent(1)
 Think Of It As An Address: earth.usa.ca.sanjose.westtasmandr.170
 Unique Object Identifier (OID) For Each Data Object In An SNMP Message
‒ Not unique per alarm
 Applications Passes OID
 Manufacture‟s MIB is required to translate OID‟s to human-readable events

 <drive>:\icm\snmp\CCCA-Notifications.txt
----------------------- E=error, A=warning, 6=info
NOTIFICATION : 102C001
cccaEventMessageId : 3775053825 (0xE102C001)
DESCRIPTION : A critical process needed to run the ICM software
: on this node has died. The Node Manager is
: forcing a reboot of the node.
cccaEventState : Raise Process Name
SUBSTITUTION STRING : Critical process %1 died. Rebooting node.
cccaEventComponentId: { cccaEventOrginatingNode %1 }
CorrelationId : { CLASS_NM_REBOOT_ON_FAIL cccaEventOrginatingNode %1 }
-----------------------

 Know The Requirements (RFC‟s And Other MIBs)

‒ CISCO-CONTACT-CENTER-APPS-MIB
‒ CISCO-CVP-MIB
 Configure UCCE SNMP/Syslog Agent Management
‒ Reference the UCCE SNMP Guide for step-by-step instructions:
http://www.cisco.com/en/US/products/sw/custcosw/ps1844/products_installatio
n_and_configuration_guides_list.html
‒ Note: On Windows 2008 R2, use “mmc /32” to launch 32-bit Snap-in!
 Configure CVP SNMP/Syslog
 View UCCE and CVP SNMP In Action

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 148
Setting Up SNMP And Syslog
What RFC‟s / MIBs are required to support this device / MIB?
-- ******************************************************************
-- CISCO-CONTACT-CENTER-APPS-MIB.my:
-- Cisco Contact Center Applications MIB file.
--
-- March 2005, Steven Harter
 Open MIB In Text Editor
--
-- Copyright (c) 2005 by Cisco Systems, Inc.
-- All rights reserved.  Check for FROM tags under IMPORTS
-- ******************************************************************

CISCO-CONTACT-CENTER-APPS-MIB DEFINITIONS ::= BEGIN

IMPORTS
MODULE-IDENTITY,
OBJECT-TYPE,
NOTIFICATION-TYPE,
Integer32,
Unsigned32,
Gauge32
FROM SNMPv2-SMI
CiscoURLString
FROM CISCO-TC
InetAddressDNS
FROM INET-ADDRESS-MIB
TEXTUAL-CONVENTION, FROM denotes
TruthValue, the MIB/RFC
DateAndTime required
FROM SNMPv2-TC
SnmpAdminString
FROM SNMP-FRAMEWORK-MIB
MODULE-COMPLIANCE,
OBJECT-GROUP,
NOTIFICATION-GROUP
FROM SNMPv2-CONF
ciscoMgmt
FROM CISCO-SMI;
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 149
Missing MIBs?

 Cisco Has A Great Library of MIBs Available

 Search and Download
‒ by MIB Name
‒ by MIB Object
‒ by OID

http://tools.cisco.com/Support/SNMP/do/BrowseOID.do?local=en
BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 150
UCCE – Configuration Mapping
Device Objects

CCE UCM IP IVR

Dialed Number CTI Route Point (PG) NONE
Trunk Group CTI Port JTAPI Call Control Group
Device Target IP Phone NONE
Translation Route NONE ICM Translation Routing Application
DNIS (Temp. Label) CTI Route Point (IP IVR) JTAPI Trigger
Network VRU Script NONE ICM Network VRU Script
NONE NONE Dialog Control Group
NONE CSS/Partition NONE

CCE UCM IP IVR

PG CUCM Dialed Numbers CTI Route Points
Directory NONE!
Device Targets
User Agent Phones

Translation Route
IP IVR CUCM CTI Route Points JTAPI Triggers
DNIS
Directory User JTAPI Call Control
Network Trunk
Groups CTI Ports Groups

 Dump size determined in the registry paths:

‒ CTISvr: HKEY_LOCAL_MACHINE\SOFTWARE\Cisco Systems,
Inc.\ICM\<customer_instance>\CGXY\EMS\...
‒ CTIOS: HKEY_LOCAL_MACHINE\SOFTWARE\Cisco Systems,
Inc.\ICM\ctios\CTIOS1\EMS\...
‒ ICM Core: HKEY_LOCAL_MACHINE\SOFTWARE\Cisco Systems,
Inc.\ICM\<customer_instance>\<component>\EMS…
 Look for “EMSGenerateSmallMemoryDump”
‒ Default 0x20000001, stack dump (low RCA probability)
‒ Recommended 0x20000002, full variable and memory dump (high RCA
probability)

Enabled

BRKCCT-3005 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 155
Mini Dumps vs Full Dumps
Getting to Root Cause Faster
 RCA Rule #1 – If you see a mdmp, enable the full dump!
‒ Stack dump does NOT guarantee root cause
‒ Full dump will take a few seconds longer than usual to restart process
… but getting closer to root cause is worth it.
 Always Collect The Following
‒ .mdmp file

‒ <process>.exe file \\icm\bin

Needed to confirm exact build info to match
binaries when debugging

‒ process>.pdb file \\icm\bin

Symbol file contains debugging info

Instructions For Installing VisionMaster FT in The Field
No ratings yet
Instructions For Installing VisionMaster FT in The Field
53 pages
Tecucc 3000
No ratings yet
Tecucc 3000
449 pages
Tecucc 3000
No ratings yet
Tecucc 3000
376 pages
portspan
No ratings yet
portspan
152 pages
CUCM_Troubleshooting
No ratings yet
CUCM_Troubleshooting
131 pages
BRKTRS 2811
No ratings yet
BRKTRS 2811
150 pages
NetBrain System Setup Guide Two-Server Deployment
100% (1)
NetBrain System Setup Guide Two-Server Deployment
52 pages
JSEC C2 Arch 10.a.pps
No ratings yet
JSEC C2 Arch 10.a.pps
34 pages
Soft Core Processor Microarchitecture
No ratings yet
Soft Core Processor Microarchitecture
18 pages
Call Processing
No ratings yet
Call Processing
56 pages
Attckwio
No ratings yet
Attckwio
7 pages
MA BRKUCT-2002 284382 156-1 v1
100% (2)
MA BRKUCT-2002 284382 156-1 v1
132 pages
NIIT BS+IT Program
No ratings yet
NIIT BS+IT Program
14 pages
PSDIRECT_Driver_Specification_2.6
No ratings yet
PSDIRECT_Driver_Specification_2.6
55 pages
5 Valmet Valimaki 18112015
No ratings yet
5 Valmet Valimaki 18112015
12 pages
Data Logger-Mill Parameter Project Progress (updated 02-Feb-2023)
No ratings yet
Data Logger-Mill Parameter Project Progress (updated 02-Feb-2023)
6 pages
Ribbon-MSTeams - Config - Guide v2.2-BCedits
No ratings yet
Ribbon-MSTeams - Config - Guide v2.2-BCedits
78 pages
PSDIRECT Driver Specification Rev 2.5.0
No ratings yet
PSDIRECT Driver Specification Rev 2.5.0
46 pages
SDP Test Automation Demo: Oct 04 Monday, 2010
No ratings yet
SDP Test Automation Demo: Oct 04 Monday, 2010
12 pages
ctrdk
No ratings yet
ctrdk
37 pages
Maepgetst-Eng V102 12 2009
No ratings yet
Maepgetst-Eng V102 12 2009
34 pages
IBM Integration Bus V10 Performance: How To Analyse Your System To Optimise Performance and Throughput
No ratings yet
IBM Integration Bus V10 Performance: How To Analyse Your System To Optimise Performance and Throughput
67 pages
6 IOM - IDirect NMS IMonitor Module, V6.0, 030106
No ratings yet
6 IOM - IDirect NMS IMonitor Module, V6.0, 030106
78 pages
ABB Standard Ten Byte Master Protocol
No ratings yet
ABB Standard Ten Byte Master Protocol
21 pages
Datek Master Protocol
No ratings yet
Datek Master Protocol
17 pages
Azbil BMS-Network Protocols
No ratings yet
Azbil BMS-Network Protocols
38 pages
Lecture05
No ratings yet
Lecture05
28 pages
Nokia-Sbts Deletion and Recreation
No ratings yet
Nokia-Sbts Deletion and Recreation
25 pages
IBM Hardware List 23 Oct 17
No ratings yet
IBM Hardware List 23 Oct 17
15 pages
PLCFundamental
No ratings yet
PLCFundamental
34 pages
Tf13 Factorytalk System Design Considerations Rsteched
No ratings yet
Tf13 Factorytalk System Design Considerations Rsteched
101 pages
Troubleshooting the Msbr Configuration Note Ver 72
No ratings yet
Troubleshooting the Msbr Configuration Note Ver 72
43 pages
Multi-Protocol Substation Demonstration: DNP3 Key Manager DNP3 Forge
No ratings yet
Multi-Protocol Substation Demonstration: DNP3 Key Manager DNP3 Forge
2 pages
Ystem Equirements Pecification: Specifying The Specifications
No ratings yet
Ystem Equirements Pecification: Specifying The Specifications
24 pages
Densify Report Infrastructure Impact Analysis Recommendations
No ratings yet
Densify Report Infrastructure Impact Analysis Recommendations
6 pages
ETMA IP Attribute Configuration (Version-003)
No ratings yet
ETMA IP Attribute Configuration (Version-003)
9 pages
Ystem Equirements Pecification: Specifying The Specifications
No ratings yet
Ystem Equirements Pecification: Specifying The Specifications
24 pages
DELTA INFRASUITE EN (Br-InfraSuite DCIM EMS3000) PDF
No ratings yet
DELTA INFRASUITE EN (Br-InfraSuite DCIM EMS3000) PDF
2 pages
Schneider Ecostruxure IT DCIM - 23nov2020
No ratings yet
Schneider Ecostruxure IT DCIM - 23nov2020
29 pages
NetBrain System Setup Guide Two-Server Deployment
No ratings yet
NetBrain System Setup Guide Two-Server Deployment
49 pages
The Show Processes Command: Prerequisites
No ratings yet
The Show Processes Command: Prerequisites
12 pages
8086 Microprocessor
No ratings yet
8086 Microprocessor
100 pages
Seer Verilog
No ratings yet
Seer Verilog
59 pages
SOC Design
No ratings yet
SOC Design
23 pages
Putting It All Together Some EJB™ Design Issues and J2EE Security
No ratings yet
Putting It All Together Some EJB™ Design Issues and J2EE Security
31 pages
systems
No ratings yet
systems
13 pages
GS36J04A10-01E_037
No ratings yet
GS36J04A10-01E_037
15 pages
Server Checklist
100% (1)
Server Checklist
22 pages
BRKIOT-2026
No ratings yet
BRKIOT-2026
55 pages
ESD - DAY 1-Portal
No ratings yet
ESD - DAY 1-Portal
35 pages
Esx-3Xl: Flexibility / Scalability
No ratings yet
Esx-3Xl: Flexibility / Scalability
4 pages
IBM Z Systems Processor Optimization Primer
No ratings yet
IBM Z Systems Processor Optimization Primer
50 pages
037 Prosalus Easton
No ratings yet
037 Prosalus Easton
23 pages
3573_PF00603
No ratings yet
3573_PF00603
8 pages
General Specifications: NTPB001 - NTPB010 Exaquantum/Batch Plant Information Management System
No ratings yet
General Specifications: NTPB001 - NTPB010 Exaquantum/Batch Plant Information Management System
6 pages
Solution-Brief SuperCloud Composer
No ratings yet
Solution-Brief SuperCloud Composer
13 pages
ECE153a/253 Embedded Systems Class Overview
No ratings yet
ECE153a/253 Embedded Systems Class Overview
41 pages
QN432 PC-Based ICCP Survalent
No ratings yet
QN432 PC-Based ICCP Survalent
26 pages
EcoStruxure Plant TVDA Catalog - 2020 - 02
No ratings yet
EcoStruxure Plant TVDA Catalog - 2020 - 02
63 pages
CPA Security Procedures - Egress Switch - Issue 1.3 Oct 2018
No ratings yet
CPA Security Procedures - Egress Switch - Issue 1.3 Oct 2018
28 pages
Transit Quarters
From Everand
Transit Quarters
Shyamala Nemana
4/5 (8)
Self Provision Lab
No ratings yet
Self Provision Lab
19 pages
BRKUCC-2932 TG - SIP Trunk
No ratings yet
BRKUCC-2932 TG - SIP Trunk
162 pages
BRKCRT-2602 Configuring and Troubleshooting Cisco Jabber Mobile Remote Access Using Collaboration-Edge Deployment Model
No ratings yet
BRKCRT-2602 Configuring and Troubleshooting Cisco Jabber Mobile Remote Access Using Collaboration-Edge Deployment Model
143 pages
BRKCCT-2015 TG Uccx
No ratings yet
BRKCCT-2015 TG Uccx
125 pages
Collaboration Edge Troubleshooting: Philip Smeuninx Technical Leader Services
No ratings yet
Collaboration Edge Troubleshooting: Philip Smeuninx Technical Leader Services
221 pages
Lab Guide: 44+ Labs For Amazon AWS Associates Amazon AWS Lab Guide
No ratings yet
Lab Guide: 44+ Labs For Amazon AWS Associates Amazon AWS Lab Guide
451 pages
Sistema Mega
No ratings yet
Sistema Mega
26 pages
Types of Cursors
No ratings yet
Types of Cursors
1 page
CBCL CAD Standards - Tool Palettes Tutorial
No ratings yet
CBCL CAD Standards - Tool Palettes Tutorial
12 pages
QA Training
No ratings yet
QA Training
8 pages
Citrix XenApp and XenDesktop 7.15 LTSR Architecture and Components
100% (1)
Citrix XenApp and XenDesktop 7.15 LTSR Architecture and Components
8 pages
Docker-Command Notebook
No ratings yet
Docker-Command Notebook
10 pages
TC 10 1 Release Bulletin PDF
No ratings yet
TC 10 1 Release Bulletin PDF
500 pages
Preload Installer
No ratings yet
Preload Installer
1 page
Codeproject Pdf2textconvert
No ratings yet
Codeproject Pdf2textconvert
7 pages
2.9 (en)
No ratings yet
2.9 (en)
4 pages
Operating System Research
No ratings yet
Operating System Research
8 pages
Nhập môn Công nghệ Thông tin: Chapter 9 - Operating Systems (Hệ điều hành)
No ratings yet
Nhập môn Công nghệ Thông tin: Chapter 9 - Operating Systems (Hệ điều hành)
22 pages
OmniStudio Meet The Data Source Wizard
No ratings yet
OmniStudio Meet The Data Source Wizard
13 pages
Infrastructure Assessment Worksheet: - x000D - x000D - 42U Cabinet
No ratings yet
Infrastructure Assessment Worksheet: - x000D - x000D - 42U Cabinet
2 pages
DO Automating SALT PDF
No ratings yet
DO Automating SALT PDF
96 pages
Tutorials - Day 1
No ratings yet
Tutorials - Day 1
40 pages
Refresh STQ Qual
No ratings yet
Refresh STQ Qual
21 pages
281027
No ratings yet
281027
258 pages
How To Add Managed Oacore in Oracle Apps R12.2.x
No ratings yet
How To Add Managed Oacore in Oracle Apps R12.2.x
3 pages
Managing Windows 10 With Intune and Autopilot v2106 - Labs
No ratings yet
Managing Windows 10 With Intune and Autopilot v2106 - Labs
380 pages
XENA XER Import Utility Install Guide
No ratings yet
XENA XER Import Utility Install Guide
30 pages
KX v741525 ReleaseNote KDC
No ratings yet
KX v741525 ReleaseNote KDC
17 pages
Ready Reckoner Remote Assessment v.1.0 TCS
No ratings yet
Ready Reckoner Remote Assessment v.1.0 TCS
6 pages
Dominik Joe Pantucek-Ceph
No ratings yet
Dominik Joe Pantucek-Ceph
32 pages
OJT Weekly Report Edit For Hardbound
No ratings yet
OJT Weekly Report Edit For Hardbound
8 pages
Create A Windows Service in C#
No ratings yet
Create A Windows Service in C#
1 page
Configuring Local Services
No ratings yet
Configuring Local Services
3 pages
Dm00403500-Stm32cubeprogrammer-Software-Description-Stmicroelectronics (1) - Compressed
No ratings yet
Dm00403500-Stm32cubeprogrammer-Software-Description-Stmicroelectronics (1) - Compressed
169 pages

Untitled

Uploaded by

Untitled

Uploaded by

Troubleshooting Cisco Unified

Contact Center Enterprise

“Tell me, and I will forget. Show me, and I may

 Performance counters are measurements of

 Reactive Usage: UCCE Serviceability Guide section 8.3

 Proactive Benefits: Counters can also be enabled for trending and

 Counters added in UCCE 8.0 provide richer visibility into system

 Platform Diagnostic Counters still (7.0) collected automatically by Node Manager

Process Handle Count (_Total) # Handles <=2 x [# CPUs] (continuous)

Memory Page Faults / sec # Faults <10 | >=10 | >10 (sustained)

Memory Pages / sec # Pages / sec <10 | >10 (continuous)

System Threads # Threads Reference

System Processes # Processes Reference

 Component-specific Counters for Further Diagnosis

 * = Useful for Capacity Planning. Available Prior to UCCE 8.0.

 Component-specific Counters for Further Diagnosis

 * = Useful for Capacity Planning. Available Prior to UCCE 8.0.

CallType DB Write Records Processed On Call Count On Number of DB Write Records On

 Other new objects available for

 SNMP is a family of specifications that provide a means for collecting

 UCCE SNMP Event Flow – High Level

 UCCE SNMP Event Flow – Digging Deeper

SNMP Crash Course CSFS

‒ Certain events hard-coded as “Objects”

SNMP Crash Course CSFS

‒ Certain events hard-coded as “Objects”

03:37:06:096 la-csfs Trace: Created New Base Object '3_2_85PG2B_ICM\ucce\PG2B_pim1'

SNMP Crash Course CSFS

‒ Certain events hard-coded as “Objects”

03:37:06:096 la-csfs Trace: Created New Base Object '3_2_85PG2B_ICM\ucce\PG2B_pim1'

03:37:15:596 la-csfs Trace: Event for Existing Base Object '3_2_85PG2B_ICM\ucce\PG2B_pim1'

‒ Route Request ‒ Registry EMSTraceMask 0x40 ‒ trace *CONF* /on

 Simplified Tracing Introduced in UCCE 8.0(1)

 How do they translate?

‒ Level 0 – Default / Error PG Process

ONCE CTI OS 0x60A0F 0x240A2F 0x260A2F 0x2E0A2F

 EMS Log Compression = Increased Retention (x10!) for:

 Introduced with 7.5(10), 8.0(3), 8.5(1)

 How is OPC Capture different from traditional component logs?

 Where are the OPC Capture files?

PG Service (Re)Start, OPC State Transfer

Timestamps must match!

 Hard-Coded 2GB Capture Limit Per Dated Folder

 8.5(2) & Later: „newcap‟

PG Service (Re)Start, OPC State Transfer

Timestamps must match!

 Title bar indicators gone with the wind(ows).

 OS architecture change = UCCE

Note: „system‟ argument is optional and only

*Max of 1 local and 5 remote connections

 OS architecture change = UCCE

Note: „system‟ argument is optional and only

*Max of 1 local and 5 remote connections

 Single-system “Support Tool” – More than just Process Status!

 Single-system “Support Tool” – More than just Process Status!

 Dump logs for a single process

 How does it work? RoggerA RoggerB

HTTPS PG1A PG1B

 Other devices also RTMT HTTPS CUCM PUB CUCM SUB

 Tool within the CUCM Real-Time Monitoring Tool

 How does it work? RoggerA RoggerB

HTTPS PG1A PG1B

RTMT HTTPS CUCM PUB CUCM SUB

RTMT HTTPS IP IVR 1 IP IVR 2

 How does it work? RoggerA RoggerB

RTMT Trace Template ICM PG1A PG1B

UCM CUCM PUB CUCM SUB

UCCX IP IVR 1 IP IVR 2

 Trace level Select Trace Template

 What if I have multiple ICM instances?

 Let‟s check out Option #2!!

 How does it work? RoggerA RoggerB

HTTPS PG1A PG1B

RTMT HTTPS CUCM PUB CUCM SUB

RTMT HTTPS IP IVR 1 IP IVR 2

 How does it work? RoggerA RoggerB

‒ Route Request ‒ Registry EMSTraceMask 0x40 ‒ trace CONF /on