Active Directory domain name problems 841

Active Directory integration troubleshooting

and overview 837
Active Directory schema modications 840
Active Directoryusers added in Active Directory
dont show up in CallManager Administration 843
Adding a user fails 824
Alarms (red and yellow) on a digital interface 208
Already in conference message 597
Attendant Console client conguration 781
Attendant Consolefast busy on calls to a pilot point 782
Attendant Consoleline states wont update 782
Attendant Consolelines are disabled 782
Attendant Consolelogin failed 782
Attendant Consolelongest idle algorithm is not
working properly 782
Attendant Consolenew user doesnt display 782
Attendant Console server conguration 780
Attendant Consolesome line states show
Unknown status 782
Attendant Console troubleshooting methodology 782
Attendant Consolewrong directory list displays 782
Audio problems 384, 389, 396, 400, 402, 405, 410
Audio Translator problems 617
Automated alternate routing (AAR) troubleshooting 637
Busy signal not heard on an IP phone 310
Calling name display problems 270
Calling search spaces, overview 469
CallManager Serviceability 82
CallManager wildcard summary 460
Call preservation, overview 551
Call routing problems 515
CCM traceshow to read 42
CCM traceshow to read Skinny messages 148
CDRs are not being written properly 813
CDRs are not generated by Subscriber 810
Choppy audio 396
CM Down, Features Disabled message 158
CM Fallback Service Operating message 707
CMIreading traces 674
Codec selection between devices 570
Conference bridgeout of resources 578
Conference Connection doesnt work 789
Corporate directoryadd or delete users fails in
CallManager Administration 843
Corporate directoryError: The phone administrator
is currently not allowed to add or delete users 843
CRA Administration page does not load 738
CRA Application Engine problems 745
CRA directory conguration troubleshooting 741
CRA trace les (MIVR) 748
Customer Directory Conguration Plugin
troubleshooting 839, 842
Database Layer Monitor is not running properly 812

Database replication problems 796, 804, 807

D-channel wont establish on PRI 210
DC Directoryreconguring in CallManager 3.3 828, 835
DC Directoryreconguring in pre-3.3 CallManager 830, 835
Directory access troubleshooting 823
Directory troubleshooting 824
Delayed audio 384
Delayed routing 466
Dial peer matching in IOS, overview 175
Dick Tracy 101
Digit discard instructions (DDIs), overview 486
directories button doesnt work 160
Disconnected calls with cause code 0xE6, Recovery on
timer expiry. 271
DPA 7610/7630MWI problems 702
Dropped calls 157
Dropped packets 397
DTMF relay, overview 303
E1 interface troubleshooting 208
Echo problems 410, 418, 421, 428
Exceeds maximum parties message 597
Extension mobilitycommon error messages 765, 777
Extension mobility problems on CallManager
release 3.1 or 3.2 756
Extension mobility problems on CallManager
release 3.3 773
Extension mobility troubleshooting methodology
for CallManager release 3.1 or 3.2 765
Failoverphone behavior and causes 155, 158
Fax machine troubleshooting methodology 446, 447, 449
Fax passthrough conguration 441
Fax passthrough, overview 437
Fax relay debugs, enabling 455
Fax relay, overview 444
Fax takes twice as long to complete 451
FXO port will not disconnect a completed call 205
Garbled audio 396
Gatekeeper call admission control 638, 640
H.323 call ow (H.225 and H.245) 283, 284, 287
Hold and resume problems 522
Hold doesnt play music 611
Intercluster trunk troubleshooting 311
IOS gatewaycall routing and dial peer debugs 196
IOS gateway debugs 185
IOS gatewaydebugs and show commands 184
IOS gatewaydiagnosing the state of ports 187
IOS gatewayTDM interfaces 187
IOS gateway wont register with CallManager (MGCP) 240
iPlanet integration troubleshooting 844
ISDN cause codes (Q.850) 262
ISDN messages, overview 258
ISDN timers, overview 271
Jitter 389, 400

Live audio source problems 619

LMHOSTS le, overview 796
Locations-based call admission control 624, 636
Masks, overview 495
Methodology for troubleshooting 4
MGCP overview 218
Microsoft Performance (PerfMon) 68
Modem passthrough conguration 441
Modem passthrough, overview 437
Modem troubleshooting methodology 447
MOHlive audio source problems 619
MOHmulticast and unicast problems 615
MOHno music when calls are on hold 611, 617
MOHreading CCM traces 607
MOHtroubleshooting methodology 611
MWI problems (Personal Assistant) 659
MWI problems (SMDI) 682
MWI problems (Unity) 659
MWI problems (VG248) 690
No conference bridge available message 587
No-way audio 405, 410
Octel integration 693
One-way audio 405, 406, 410
Outside dial tone played at the wrong time 465
Park problems 531
Partitions, overview 469
Personal Assistant is not intercepting calls 785
Personal AssistantMWI problems 659
Phonebusy signal not heard 310
Phonefailover and failback 154, 155, 158
Phoneinline power problems 114
Phonenetwork connectivity and Skinny registration 117
Phone stuck in SRST mode 730
Phoneswitch port operation 161
PhoneTFTP conguration le 121
Phoneunderstanding the difference between
restart and reset 156
Phoneunderstanding the Skinny protocol 139
PhoneVLAN conguration 118
Phone wont register 127
Pickup/group pickup problems 533
PRI backhaul channel status 256
PRICallManager sends the proper digits to the PSTN,
but call wont route properly 269
PRI signaling troubleshooting 210, 262
Publisher-Subscriber model, overview 793, 796
Q.931 Translator 95
Registration problems on IP phone 127
Replication problems 796, 804, 807
Reset vs. restart 156
Ringback problems 307, 309
Route lters, overview 506
SDL traceshow to read 60

Search for a user fails 824

services button doesnt work 160
Silence suppressioneffect on voice quality 402
SMDIcheck conguration parameters 686
SMDIintegration 662
SMDI integration with VG248 686
SMDIMWI problems 682
SoftPhone has no lines 788
SoftPhoneone-way audio over VPN 787
SoftPhone shows line but wont go off-hook 786
SQL database replication problems 796, 804, 807
SQLre-establishing a broken subscription 807
SQLreinitializing a subscription 809
SRST and phone registration 712
SRSTDHCP issues 731
SRSTfeatures lost during operation 707
SRSTphones still registered after WAN connection
is restored 730
SRSTtransfer problems 729
SRSTvoice mail and forwarding issues 731
T1 CAS signaling troubleshooting 214
T1 interface troubleshooting 208
Temporary Failure message 561
Time synchronization 38
Toll fraud prevention 544, 548
Tone on hold plays instead of music 615, 617
Transcoderout of resources 578
Transcoderunderstanding codec selection between
devices 570
Transfer problems 529
Transformation troubleshooting 513
Transformations and masks, overview 486
Transformations, overview 496, 500
Translation pattern troubleshooting 501
UnityMWI problems 659
UnityTSP conguration 656
VADeffect on voice quality 402
VG248MWI problems 690
Voice mailMWI problems (DPA 7610/7630) 702
Voice mailMWI problems (SMDI) 682
Voice mailMWI problems (Unity) 659
Voice mailMWI problems (VG248) 690
Voice mailOctel integration 693
Voice quality problems 384, 389, 396, 400, 402, 405, 410
WS-X6608/6624 gateway troubleshooting 313, 314, 320, 324
WS-X6608D-channel is down 326, 337, 340, 343, 344
WS-X6608dropped calls 326
WS-X6608T1 CAS problems 359
WS-X6608 T1/E1 conguration troubleshooting 325, 326
WS-X6608unexpected resets 326, 345
WS-X6624 FXS analog gateway conguration 367

Troubleshooting Cisco IP Telephony

Paul Giralt, CCIE No. 4793
Addis Hallmark
Anne Smith

Cisco Press
Cisco Press
201 West 103rd Street
Indianapolis, IN 46290 USA

ii

Troubleshooting Cisco IP Telephony

Paul Giralt, CCIE No. 4793
Addis Hallmark
Anne Smith
Copyright 2003 Cisco Systems, Inc.
Published by:
Cisco Press
800 East 96th Street
Indianapolis, IN 46240 USA
All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or
mechanical, including photocopying and recording, or by any information storage and retrieval system, without written
permission from the publisher, except for the inclusion of brief quotations in a review.
Printed in the United States of America 5 6 7 8 9 0
Twelfth Printing May 2011
Library of Congress Cataloging-in-Publication Number: 2001096407
ISBN: 1-58705-075-7

Trademark Acknowledgments
All terms mentioned in this book that are known to be trademarks or service marks have been appropriately capitalized.
Cisco Press or Cisco Systems, Inc., cannot attest to the accuracy of this information. Use of a term in this book should
not be regarded as affecting the validity of any trademark or service mark.
Figure 7-11 in Chapter 7 comes from ITU-T Recommendation G.131, Figure 1, p.8, and has been reproduced with the
prior authorization of the Union as copyright holder; the sole responsibility for selecting extracts for reproduction lies
with Cisco Press alone and can in no way be attributed to the ITU.
The complete volume of ITU-T Recommendation G.131, from which the gure reproduced is extracted, can be obtained
from:
International Telecommunication Union
Sales and Marketing Division
Place des Nations - CH-1211 GENEVA 20 (Switzerland)
Telephone: + 41 22 730 61 41 (English) / +41 22 730 61 42 (French) / +41 22 730 61 43 (Spanish)
Telex: 421 000 uit ch / Fax: +41 22 730 51 94
E-mail: [email protected] / http://www.itu.int/publications

iii

Warning and Disclaimer

This book is designed to provide information about troubleshooting the various components of a Cisco IP Telephony
network. Every effort has been made to make this book as complete and accurate as possible, but no warranty or tness
is implied.
The information is provided on an as is basis. The authors, Cisco Press, and Cisco Systems, Inc., shall have neither
liability nor responsibility to any person or entity with respect to any loss or damages arising from the information
contained in this book or from the use of the discs or programs that may accompany it or be referenced by it.
The opinions expressed in this book belong to the authors and are not necessarily those of Cisco Systems, Inc.
Portions of Chapter 6 are extracted from RFC 2705 which denes MGCP. The following copyright statement applies to
any information derived from RFC 2705:
Full Copyright Statement
Copyright The Internet Society (1999). All Rights Reserved.
This document and translations of it may be copied and furnished to others, and derivative works that comment on or
otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in
part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such
copies and derivative works. However, this document itself may not be modied in any way, such as by removing the
copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for copyrights dened in the Internet Standards process must
be followed, or as required to translate it into languages other than English.
The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or
assigns.
This document and the information contained herein is provided on an AS IS basis and THE INTERNET SOCIETY
AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR
FITNESS FOR A PARTICULAR PURPOSE.

Feedback Information
At Cisco Press, our goal is to create in-depth technical books of the highest quality and value. Each book is crafted with
care and precision, undergoing rigorous development that involves the unique expertise of members of the professional
technical community.
Reader feedback is a natural continuation of this process. If you have any comments regarding how we could
improve the quality of this book, or otherwise alter it to better suit your needs, you can contact us through e-mail at
[email protected]. Please be sure to include the book title and ISBN in your message.
We greatly appreciate your assistance.

iv

Publisher
Editor-In-Chief
Cisco Representative
Cisco Press Program Manager
Cisco Marketing Communications Manager
Cisco Marketing Program Manager
Acquisitions Editor
Production Manager
Development Editor
Copy Editor
Technical Editors

Team Coordinator
Book Designer
Cover Designer
Compositor
Indexer

Corporate Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 526-4100

John Wait
John Kane
Anthony Wolfenden
Sonia Torres Chavez
Tom Geitner
Edie Quiroz
Amy Moss
Patrick Kanouse
Christopher Cleveland
Gayle Johnson
Shawn Armstrong, Dave Goodwin, Christina Hattingh,
Phil Jensen, Ketil Johansen, Chris Pearce, Ana Rivas,
Markus Schneider, Gert Vanderstraeten, Liang Wu
Tammi Ross
Gina Rexrode
Louisa Klucznik
Mark Shirar
Tim Wright

European Headquarters
Cisco Systems Europe
11 Rue Camille Desmoulins
92782 Issy-les-Moulineaux
Cedex 9
France
http://wwweurope.cisco.com
Tel: 33 1 58 04 60 00
Fax: 33 1 58 04 61 00

Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-7660
Fax: 408 527-0883

Asia Pacic Headquarters

Cisco Systems Australia,
Pty., Ltd
Level 17, 99 Walker Street
North Sydney
NSW 2059 Australia
http://www.cisco.com
Tel: +61 2 8448 7100
Fax: +61 2 9957 4350

Cisco Systems has more than 200 ofces in the following countries. Addresses, phone numbers, and fax numbers are listed on
the Cisco Web site at www.cisco.com/go/ofces
Argentina Australia Austria Belgium Brazil Bulgaria Canada Chile China Colombia Costa
Rica Croatia Czech Republic Denmark Dubai, UAE Finland France Germany Greece Hong
Kong Hungary India Indonesia Ireland Israel Italy Japan Korea Luxembourg Malaysia
Mexico The Netherlands New Zealand Norway Peru Philippines Poland Portugal Puerto Rico
Romania Russia Saudi Arabia Scotland Singapore Slovakia Slovenia South Africa Spain Sweden
Switzerland Taiwan Thailand Turkey Ukraine United Kingdom United States Venezuela Vietnam
Zimbabwe
Copyright 2000, Cisco Systems, Inc. All rights reserved. Access Registrar, AccessPath, Are You Ready, ATM Director, Browse with Me, CCDA, CCDE, CCDP, CCIE, CCNA,
CCNP, CCSI, CD-PAC, CiscoLink, the Cisco NetWorks logo, the Cisco Powered Network logo, Cisco Systems Networking Academy, Fast Step, FireRunner, Follow Me Browsing,
FormShare, GigaStack, IGX, Intelligence in the Optical Core, Internet Quotient, IP/VC, iQ Breakthrough, iQ Expertise, iQ FastTrack, iQuick Study, iQ Readiness Scorecard, The
iQ Logo, Kernel Proxy, MGX, Natural Network Viewer, Network Registrar, the Networkers logo, Packet, PIX, Point and Click Internetworking, Policy Builder, RateMUX,
ReyMaster, ReyView, ScriptShare, Secure Script, Shop with Me, SlideCast, SMARTnet, SVX, TrafficDirector, TransPath, VlanDirector, Voice LAN, Wavelength Router,
Workgroup Director, and Workgroup Stack are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, Empowering the Internet Generation, are
service marks of Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, Cisco, the Cisco Certified Internetwork Expert Logo, Cisco IOS, the Cisco IOS logo, Cisco Press, Cisco
Systems, Cisco Systems Capital, the Cisco Systems logo, Collision Free, Enterprise/Solver, EtherChannel, EtherSwitch, FastHub, FastLink, FastPAD, IOS, IP/TV, IPX, LightStream,
LightSwitch, MICA, NetRanger, Post-Routing, Pre-Routing, Registrar, StrataView Plus, Stratm, SwitchProbe, TeleRouter, are registered trademarks of Cisco Systems, Inc. or its
affiliates in the U.S. and certain other countries.
All other brands, names, or trademarks mentioned in this document or Web site are the property of their respective owners. The use of the word partner does not imply a partnership
relationship between Cisco and any other company. (0010R)

About the Authors

Paul Giralt, CCIE No. 4793, is an escalation engineer at the Cisco Systems Technical Assistance Center in Research
Triangle Park, N.C., where he has worked since 1998. He has been troubleshooting complex IP Telephony networks
since the release of CallManager 3.0 as a TAC engineer, a technical lead for the Enterprise Voice team, and now as an
escalation engineer supporting the complete Cisco line of IP Telephony products. Paul has troubleshot problems in some
of Ciscos largest IP Telephony deployments and has provided training for TAC teams around the globe. Prior to working on IP Telephony, he was a TAC engineer on the LAN Switching team. He holds a B.S. in computer engineering from
the University of Miami.
Addis Hallmark, CCNA, CIPT, is a senior technical marketing engineer with Cisco Systems. He has been installing,
conguring, administering, and troubleshooting the Cisco IP Telephony solution since the 2.3 release of CallManager.
He has contributed to numerous design guides, application notes, and white papers on a variety of IP Telephony subjects, including CallManager, IP Phones, and IP gateways.
Anne Smith is a technical writer in the CallManager engineering group at Cisco Systems. She has written technical
documentation for the Cisco IP Telephony solution since CallManager release 2.0 and was part of the Selsius Systems
acquisition in 1998. Anne writes internal and external documents for CallManager, IP phones, and other
Cisco IP Telephony products. She is a co-author of Cisco CallManager Fundamentals (ISBN: 1-58705-008-0)
and Developing Cisco IP Phone Services (ISBN: 1-58705-060-9), both from Cisco Press.

vi

About the Technical Reviewers

Shawn Armstrong is an IT engineer working in Ciscos Core Hosting group. She has been with Cisco for four years
and is responsible for managing NT and Windows 2000 servers within Ciscos Information Technology group.
Dave Goodwin, CCIE No. 4992, is a customer diagnostic engineer for Ciscos Advanced Engineering Services. He is
responsible for discovering and resolving problems in new Cisco IP Telephony products while administering internal
eld trials for these systems. He also works closely with Ciscos development and TAC support teams to provide support
for anything from troubleshooting to quality issues to tools. He has been at Cisco for almost ve years and has worked
as a network engineer for eight years.
Christina Hattingh is a member of the Technical Marketing organization at Cisco Systems. In this role she works
closely with product management and engineering. Christina focuses on helping Cisco sales engineers, partners, and
customers design and tune enterprise and service provider Voice over Packet network infrastructures with particular
focus on QoS. Prior to this she was a software engineer and engineering manager of PBX Call Center products at Nortel
Networks. Her earlier software development experience in X.25 and network management systems provide background
for the issues involved today in migrating customers traditional data and voice networks to packet-based technologies.
Christina has a graduate degree in computer science and mathematical statistics.
Phil Jensen, CCIE No. 2065, is a consulting systems engineer for Cisco in the southeastern U.S. He has focused on
helping Ciscos largest customers design and troubleshoot AVVID IP Telephony solutions for the past three years. He
has worked as a network engineer for more than 14 years.
Ketil Johansen, CCIE No. 1145, is a business development manager with Cisco Systems, working with companies
integrating their applications with Cisco CallManager. He has worked with networking technologies for more than
18 years and has been a CCIE since 1994. The last three years he has focused on IP Telephony technologies.
Chris Pearce is a technical leader in the Cisco CallManager software group at Cisco Systems, Inc. He has ten years of
experience in telecommunications. His primary areas of expertise include call routing, call control, and telephone features. He was a member of the team that developed and implemented the Cisco CallManager software from its early
stages, and he was directly involved in developing the system architecture and design.
Ana Rivas, CCIE No. 3877, is an escalation engineer in Ciscos EMEA region. She is one of the technical leaders for
AVVID solutions in the Cisco TAC. She is responsible for technically leading the resolution of some of the most critical
problems in voice and IP Telephony, spreading technical knowledge to other teams, and working with Cisco business
units and the eld to head IP Telephony solutions. She has been working as a network engineer for more than ve years.
Markus Schneider, CCIE No. 2863, is a diagnostic engineer for Ciscos Advanced Engineering Services. He is responsible for helping Cisco customers design, implement, and troubleshoot IP Telephony solutions in their environment. He
has been working for Cisco as a network engineer for more than six years.
Gert Vanderstraeten has been working as a telecom/datacom engineer for companies such as Alcatel, Bell, and Lucent
Technologies since 1993. Since 1998 he has been an independent contractor for the Cisco Systems IT department. During the course of his tenure, his main focus has been the design, implementation, and maintenance of VoIP, IP Telephony, voice and video applications, and the integration of AVVID technologies into solutions. He is currently operating
within the Cisco Systems global Enterprise Architecture Solutions team.
Liang Wu is a software engineer in the CallManager software group at Cisco Systems, Inc. For the last seven years, he
has been focusing on PBX/Enterprise communication systems. He spent more than eight years in the Class 4/5/AIN
telephone switching industry.

vii

Dedications
Paul Giralt
I dedicate this book to my parents, Vicia and Pedro, for being the best parents anyone could ask for and always providing the opportunity and encouragement to continue learning.
Addis Hallmark
I want to dedicate this book to my lovely wife, Stephanie. Her companionship is the most precious thing in the world to
me. Her patience, understanding, and encouragement helped me write this book. I love and appreciate her dearly.
Anne Smith
For Herb for seeing me through the long nights and weekends without complaint. And, of course, for all those backrubs.

viii

Acknowledgments
Paul Giralt
I want to rst thank Anne Smith for all her hard work and guidance throughout this entire project. There is no way this
book would exist without her constant dedication and attention to detail.
Thanks to Chris Cleveland for his excellent work as development editor on this book and for being so exible when it
comes to the unpredictable schedules of a TAC engineer.
Thank you to the worldwide Enterprise Voice and AVVID TAC teams, especially the RTP Enterprise Voice team for
being such a world-class group of engineers to work with.
Thanks to the RTP Voice Network Team (VNT) for all the excellent VoX documentation. Special thanks to Gonzalo Salgueiro and Mike Whitley for the VoX boot camp material and to Steve Penna for knowing everything.
Thanks to Dave Hanes for his excellent fax troubleshooting presentations and Andy Pepperell for his explanation of fax
and modem passthrough.
Thanks to all the technical reviewersAna Rivas, Chris Pearce, Dave Goodwin, Ketil Johansen, Markus Schneider, Phil
Jensen, Gert Vanderstraeten, Liang Wu, Shawn Armstrong, and especially Christina Hattinghfor always being on top
of everything in the world of Cisco IOS gateways.
Thanks to all the developers in Richardson and San Jose that I have worked with over the years. Your insight into the
inner workings of CallManager has helped me understand how to better troubleshoot the product. Special thanks to Bill
Benninghoff for always answering any question I throw his way and for always being so thorough in his explanations.
Also thanks to Chris Pearce for his excellent grasp on the intricacies of call routing.
Thank you to all the contributors to the VNT Voice University website as well as the AVVID TAC tips website on
Cisco.com. Also thanks to all the other unnamed authors for the documentation scattered throughout various web pages.
Thanks to all the customers I have worked with over the past several years on AVVID issues for being my teachers.
Every customer I work with helps me understand a little more about IP Telephony.
Addis Hallmark
First, Id like to thank Paul Hahn and Richard Platt for bringing me on at Cisco. Paul in particular spent a lot of time
with me, bringing me up to speed on these technologies, and for that, I am indebted to him.
Id also like to thank all the brilliant development engineers who patiently helped me understand CallManager so well
over the past few years.
Id like to thank Susan Sauter. She is a brilliant engineer, and so much of what I know about IP Phones came from her
patient instruction.
Chris Pearce has also helped me so much over the last few years in understanding dial plans.
The chapter on applications is based on the hard work of Dave Bicknell. Without his efforts, that chapter would not be
even close to what it should be.
Manish Gupta and his team were a tremendous source of help on the LDAP Directory chapter. Stefano Giorcellis excellent directory documentation also was so very helpful!
The TAC is on the front lines of troubleshooting, and much of the help I received was from the experiences that only
solid TAC engineers could provide.
Also, the technical reviewers of this book were so helpful. Thank you so much to everyone for their hard work!
I really believe this is a great book, and one of the biggest reasons for that is Paul Giralts invaluable contribution and
hard work on this project. I couldnt have done this without him!

ix

My manager, Shaik Kaleem, was very supportive of this project that I undertook on my own time, and I greatly appreciate that support.
Finally, Id like to thank Anne Smith. This project would never have happened without her tireless work and skillful
help. I am so grateful for Annes effort. She worked so very hard over this past year, and Paul Giralt and I would have
been lost without her.
Anne Smith
My many thanks go to Paul Giralt and Addis Hallmark for making this book a reality with their knowledge, experience,
hard work, and sacrice. In particular, I thank Paul for a highly enjoyable working experience. Pauls dedication to the
quality, accuracy, and comprehensiveness of this book was unsurpassed; he spent countless hours reviewing every page
of technical information and his experience with the many components in the Cisco AVVID IP Telephony solution made
his extensive contribution invaluable. At every turn, Pauls dedication, commitment to quality, tireless drive for accuracy,
and constant positive attitude made working with him a rewarding experience.
As always, my thanks and great admiration go to Richard Platt and Scott Veibell. Without their continued support there
would be no Cisco IP Telephony-related Cisco Press books.
I would like to thank Chris Pearce for his help on the Call Routing chapter, Travis Amsler for his assistance on the Cisco
CRA and extension mobility sections, and Brian Sedgley and Ken Pruski for their help with CCM and SDL tracing.
Appreciation and recognition also go to the engineers who created and developed Dick Tracy: Rick Baugh, Jim Brasher,
Long Huang, and David Patton.

Contents at a Glance
Foreword

xxv

Introduction

xxvi

Chapter 1

Troubleshooting Methodology and Approach 3

Chapter 2

IP Telephony Architecture Overview 23

Chapter 3

Understanding the Troubleshooting Tools 37

Chapter 4

Skinny Client Registration 113

Chapter 5

IP Phones 139

Chapter 6

Voice Gateways 169

Chapter 7

Voice Quality 383

Chapter 8

Fax Machines and Modems 433

Chapter 9

Call Routing 459

Chapter 10

Call Preservation 551

Chapter 11

Conference Bridges, Transcoders, and Media Termination Points 565

Chapter 12

Music on Hold 601

Chapter 13

Call Admission Control 623

Chapter 14

Voice Mail 655

Chapter 15

Survivable Remote Site Telephony (SRST) 707

Chapter 16

Applications 735

Chapter 17

SQL Database Replication 793

Chapter 18

LDAP Integration and Replication 819

Appendix A

Cisco IP Telephony Protocol and Codec Information and References 849

Appendix B

NANP Call Routing Information 857

Appendix C

Decimal to Hexadecimal and Binary Conversion Table 881

Appendix D

Performance Objects and Counters 891

Glossary 927
Index 947

xi

Contents
Foreword

xxv

Introduction
Chapter 1

xxvi

Troubleshooting Methodology and Approach 3

Developing a Troubleshooting Methodology or Approach 4
Production Versus Nonproduction Outages 5
Step 1: Gathering Data About the Problem 6
Identifying and Isolating the Problem 6
Using Topology Information to Isolate the Problem 7
Gathering Information from the User 10
Determining the Problems Timeframe 10
Step 2: Analyzing the Data Collected About the Problem 11
Using Deductive Reasoning to Narrow the List of Possible Causes 11
Verifying IP Network Integrity 12
Determining the Proper Troubleshooting Tool 13
Case Study: Resolving a Problem Using Proper Troubleshooting Methodology 13
Gathering the Data 14
Analyzing the Data 18
Conclusions 20
Summary 21

Chapter 2

IP Telephony Architecture Overview 23

Network Infrastructure 23
IP Telephony Infrastructure 23
Call Processing 24
Single-Site Deployment Model 24
Multiple-Site Deployment Model 25
Centralized Deployment Model 26
Distributed Deployment Model 27
Cisco AVVID IP Telephony Infrastructure 28
Clients 29
Cisco IP Phone Models 7960 and 7940 31
Cisco IP Phone Expansion Module 7914 31
Cisco IP Phone 7910 32
Cisco IP Conference Station 7935 32
Voice Gateways 32
Cisco AVVID IP Telephony Applications 33
Summary 34

xii

Chapter 3

Understanding the Troubleshooting Tools 37

Time Synchronization 38
Configuring Automatic Time Synchronization on CallManager Servers 39
Synchronizing Time Manually on CallManager Servers 40
Synchronizing Time on Cisco IOS Devices 40
Synchronizing Time on CatOS Devices 41
Reading CCM (or SDI) Traces 42
Setting the Appropriate Trace Level and Flags 42
Reading CCM Traces 50
A Sample CCM Trace for a Call Between Two IP Phones 51
Tracing a Call Through an MGCP T1 PRI Gateway 58
Reading SDL Traces 60
SDL Overview 60
Enabling SDL Trace and Setting the Appropriate SDL Trace Level 63
Microsoft Performance (PerfMon) 68
Comparing PerfMon and the Real-time Monitoring Tool (RTMT) 68
PerfMon Advantages 68
RTMT Advantages 68
Using PerfMon to View Real-Time Statistics 69
Using Counter Logs 71
Using Alerts 75
CCEmail 76
Alerting Methods During Production and Non-production Hours 81
Acquiring CCEmail 82
CallManager Serviceability 82
Alarms 82
Tracing 83
Using XML-enabled Traces 83
Searching for Devices with XML Traces 84
Web-based Q.931 Translator 84
Service Activation 84
Control Center 85
Real-Time Monitoring Tool (RTMT) 85
Performance Tab 86
Devices Tab 86
CTI Apps Tab 88
Call Detail Records (CDR) and the CDR Analysis and Reporting (CAR) Tool 89
CDR Time Converter 90
Acquiring the CDR Time Converter 91

xiii

Event Viewer 91
Q.931 Translator and Enhanced Q.931 Translator 95
Enhanced Q.931 Translator 98
Acquiring Enhanced Q.931 Translator 100
Dick Tracy 101
Using the Dick Tracy Tool 102
Using the CLI Tracy/Embedded Tracy Tool 105
Acquiring Dick Tracy 105
Sniffer Traces 106
Voice Codec Bandwidth Calculator 106
Bug Toolkit (Formerly Bug Navigator) 106
Remote Access Tools 107
Terminal Services 107
Virtual Network Computing (VNC) 108
Websites and Further Reading 108
Best Practices 109
VNC Best Practices 109
Summary 110
Chapter 4

Skinny Client Registration 113

Troubleshooting Inline Power 114
Troubleshooting Network Connectivity and
Skinny Registration 117
Verifying VLAN configuration 118
Verifying IP Addressing Information 118
Verifying TFTP Configuration File Download 121
Understanding Skinny Registration 127
Troubleshooting Skinny Registration 130
Additional Tools for Troubleshooting Skinny Client Registration Problems 133
Checking IP Phone Status Messages 133
Checking Registration with the Real-Time Monitoring Tool 135
Best Practices 137
Summary 137

xiv

Chapter 5

IP Phones 139
Understanding IP Phone Behavior 139
Understanding the Skinny Protocol 139
Call Processing Behavior 140
Examining Skinny Protocol Messages in a CCM Trace 148
Understanding Failover and Failback 154
Failover Behavior 155
Failback Behavior 156
Understanding the Difference Between Restart and Reset 156
Troubleshooting IP Phone Problems 157
Dropped Calls 157
CM Down, Features Disabled 158
Reasons for Failover 158
Directory and Service Problems 160
79xx Series IP Phone 3-port Switch Operation 161
Best Practices 165
Check Your Firmware 165
Press the Help (i or ?) Button Twice During Active Calls 165
Use a Custom Phone Service That Tracks Voice Quality Statistics 166
Check the IP Phone Configuration Via Web Browser 167
Summary 167

Chapter 6

Voice Gateways 169

Cisco IOS Voice Gateways 169
Cisco VG200 170
Cisco 2600 Series Routers 171
Cisco 3600 Series Routers 172
Cisco 3700 Series Routers 173
Cisco Catalyst 4224 173
Cisco Catalyst 4000 Access Gateway Module (AGM) 174
Cisco WS-SVC-CMM Communications Media Module (CMM) 174
Other Cisco IOS Gateways 174
Understanding Dial Peer Matching in Cisco IOS Software 175
Understanding Cisco IOS Debugs and show Commands 184
Correctly Setting the Timestamps 185
Enabling Cisco IOS Software Debugs 185
Troubleshooting TDM Interfaces on Cisco IOS Gateways 187
Useful show Commands for Troubleshooting TDM Interfaces 187
Using debug Commands to Troubleshoot TDM Interfaces 192
Understanding Cisco IOS CCAPI Debugs 196

xv

Understanding the FXO Disconnect Problem 205

Troubleshooting Digital Interfaces 208
Checking Physical Layer Connectivity on Digital Interfaces 208
Troubleshooting ISDN PRI Signaling 210
Troubleshooting T1 CAS 214
Understanding MGCP 218
MGCP Endpoint Identifiers 219
MGCP Commands 219
MGCP Parameter Lines 221
MGCP Packages 229
Generic Media Package (G) 231
DTMF Package 231
MF Package (M) 232
Trunk Package (T) 233
Line Package (L) 234
Handset Emulation Package (H) 235
RTP Package (R) 236
DTMF Trunk Package (DT) 236
MF Trunk Package (MS) 237
MGCP Response Headers and Response Codes 238
Cisco IOS MGCP Gateways 240
MGCP FXS/FXO 249
Cisco IOS MGCP PRI 256
Reading ISDN Messages 258
Table of Q.850 Cause Codes 262
Numbering Type and Plan Mismatches 269
Troubleshooting Calling Name Display Problems 270
Understanding ISDN Timers 271
Cisco IOS MGCP T1 CAS 276
Cisco IOS Gateways Using the H.323 Protocol 281
H.225 Signaling 283
H.225 Messages 283
H.225 Information Elements 284
H.225 Call Flow 287
H.245 Signaling 295
Master/Slave Determination 296
Terminal Capabilities Exchange 297
Logical Channel Signaling 300
DTMF Relay 303
Additional H.323 Debugs in Cisco IOS Software 305
Troubleshooting Problems with Ringback and Other Progress Tones 307
No Ringback on an IP Phone When Calling the PSTN 308
No Ringback on a PSTN Phone When Calling an IP Phone 309

xvi

No Ringback When Transferring a Call 309

The IP Phone User Does Not Hear In-band Messages When a Call Is Disconnected 310
Intercluster Trunks 311
Troubleshooting the WS-X6608 and WS-X6624 Voice Gateways 313
Recognizing and Powering the Module 313
Troubleshooting DHCP, TFTP, and Registration Problems 314
Troubleshooting DHCP Problems 314
Troubleshooting TFTP Problems 320
Troubleshooting Registration Problems 324
Catalyst WS-X6608 T1/E1 Digital Gateway Configuration 325
Troubleshooting Configuration Issues 326
Getting the D-channel Established 337
Checking Physical Layer Statistics on the WS-X6608 340
Verifying D-channel Configuration 343
Advanced Troubleshooting for D-channel Problems 344
Unexpected Resets 345
Using Dick Tracy to Analyze a WS-X6608 Port 345
Troubleshooting T1 CAS Problems on the WS-X6608 359
Catalyst WS-X6624 FXS Analog Gateway Configuration 367
Best Practices 380
Summary 381
Chapter 7

Voice Quality 383

Fixed and Variable Delays 384
Fixed Delay Sources 385
Coder (Processing) Delay 386
Packetization Delay 386
Serialization Delay 387
Propagation Delay 389
Variable Delay Sources 389
Queuing/Buffering Delay 390
Low-speed Links 391
Dejitter Delay 393
The Effects of Delay on Signaling 395
Analyzing and Troubleshooting Choppy and Garbled Audio 396
Packet Drops 397
Queuing Problems 400
The Effect of VAD on Voice Quality 402
Troubleshooting Problems with One-way or No-way Audio 405
Verifying IP Connectivity 405
One-way Audio on Cisco IOS Software Gateways 406
NAT, PAT, and Firewalls 410

xvii

Troubleshooting Echo Problems 410

Sources of Echo 411
Electrical Echo 411
Acoustic Echo 412
Talker Versus Listener Echo 412
What Makes Echo a Problem 414
How an Echo Canceller Works 416
Eliminating Echo 418
Eliminating Echo on Cisco IOS Software Gateways 421
Eliminating Echo on the WS-X6608 and DT-24+/DE-30+ 424
Eliminating Echo Problems on Cisco IP SoftPhone 428
Best Practices 429
Summary 430
Chapter 8

Fax Machines and Modems 433

Understanding Fax Machine Operation 433
Basic Fax Machine Operation 434
T.30 Messages 435
Understanding Fax/Modem Passthrough Versus Fax Relay 437
Fax/Modem Passthrough 437
Named Service Events and Named Telephony Events 438
Basic Fax/Modem Passthrough Operation 439
Modem Passthrough Operation 439
Fax Passthrough Operation 440
Verifying Fax and Modem Passthrough Configuration 441
Fax Relay Basics 444
The Effect of Packet Loss and Jitter on Fax
Machines and Modems 446
First Steps in Troubleshooting Fax and Modem Problems 447
Checking for Physical Layer Problems on Digital Circuits 447
Isolating and Troubleshooting Fax Problems 449
Adjusting the Fax Relay Data Rate 451
Disabling Error Correction Mode 452
Changing the Nonstandard Facilities Field 453
Changing the Fax Protocol 454
Checking the fax interface-type Command 454
Enabling Fax Relay Debugs 455
Best Practices 457
Summary 457

xviii

Chapter 9

Call Routing 459

Understanding Closest-match Routing 461
Common Problems Associated with Closest-match Routing 465
Outside Dial Tone Played at the Wrong Time 465
Delayed Routing When Placing Seven-digit Local Calls 466
Understanding Calling Search Spaces and Partitions 469
Calling Search Space/Partition Rules 474
The First Partition Takes Precedence 474
The Line-level Calling Search Space Takes Precedence over the Device-level
Calling Search Space 476
Event-specific Calling Search Spaces 478
Call Forwarding Calling Search Spaces 479
Call Forward No Answer (CFNA) 479
Call Forward Busy (CFB) 480
Call Forward All (CFA) 480
Call Forward on Failure (CTI Ports and CTI Route Points Only) 485
Understanding and Troubleshooting Transformations and Masks 486
Digit Discard Instructions (DDIs) 486
Understanding the Concept of Masks 495
Transformation Rules 496
Order of Applied Transformations 496
Cumulative Transformations 497
Cumulative Transformation on Calling Party Number Example 497
Cumulative Transformation on Called Party Number Example 498
Overwritten Transformations 499
Service Parameter-related Transformations 500
Understanding and Troubleshooting Translation Patterns 501
Understanding Route Filters 506
Digit Transformation Troubleshooting 513
Call Routing Troubleshooting 515
Reading CCM Traces for Call Routing Information 516
Troubleshooting Hold, Transfer, Park, and Call Pickup 521
Call Hold and Resume 522
Call Transfer 529
Call Park 531
Call Pickup 533
Getting the Dialing Forest Traces 538
Best Practices 544
Toll Fraud Prevention 544
Preventing Transfers to Extension 9011 or Your Equivalent International
Access Code 545

xix

Using PLAR to Control Rogue Auto-registered IP Phones 545

Restricting the Call Forward All Field on IP Phones 546
Restricting Voice Mail Systems by Using Calling Search Spaces 547
Blocking Certain Area Codes 548
Summary 549
Chapter 10 Call Preservation 551
Understanding Call Preservation 551
Survivable Endpoints 552
IP Phones 552
MGCP Gateways 553
Nonsurvivable Endpoints 557
Skinny Gateways 557
H.323 Gateways 558
CTI/TAPI Endpoints 559
Media Processing Resources 560
Troubleshooting Call Preservation Issues 561
Best Practices 562
Summary 562
Chapter 11 Conference Bridges, Transcoders, and Media Termination Points 565
Media Resource Groups (MRGs) and Media Resource Group Lists (MRGLs) 566
MRGL Selection 567
Understanding Codec Selection 568
Transcoder Resources 570
Regions and the Regions Codec Matrix 570
Out-of-resource Conditions 578
Use of Transcoders in Conjunction with Other Media Resources 580
Transcoders in Conjunction with Conference Bridge Resources 581
Transcoders in Conjunction with MOH Servers 585
Conference Bridge Devices 586
Types of Conference Bridges 586
Troubleshooting No Conference Bridge Available 587
Troubleshooting Conference Failures 591
Other Conferencing Error Messages 597
Already In Conference 597
Exceeds maximum parties 597
Best Practices 598
Summary 598

xx

Chapter 12 Music on Hold 601

Understanding MOH 601
Troubleshooting Data Points 603
Performance Counters 604
CCM Trace Files 607
Troubleshooting MOH 611
Resolving Problems Related to Multicast and Unicast 615
Determining Why Tone on Hold Is Playing 617
Troubleshooting the Audio Translator 617
Troubleshooting the Live Audio Source 619
Configuring the Correct MOH Fixed Audio Source Device 619
Selecting the Proper Recording Input 620
Best Practices 620
Summary 621
Chapter 13 Call Admission Control 623
Locations-based CAC 624
Setting LocationsTraceDetailsFlag and CDCC Values 626
The Role of Regions in CAC 627
Locations-based CAC in Action 627
Locations Reservations for Media Resources 631
Locations-based CAC Reservations for Music on Hold Resources 631
Locations-based CAC Reservations for Ad Hoc or Meet-Me Conferences 633
Finding Bandwidth Leaks 635
Locations and Call Preservation Interaction 636
Troubleshooting Automated Alternate Routing 637
Gatekeeper Call Admission Control 638
Checking Gatekeeper Configuration 640
Verifying Gatekeeper Configuration on CallManager 641
CallManager Registration with Gatekeeper 645
Call Setup with Gatekeeper 647
Best Practices 652
Summary 652
Chapter 14 Voice Mail 655
Cisco Unity 655
CallManager Integration 655
Verifying Version Compatibility 656
Verifying TSP Configuration 656

xxi

Verifying Cisco Unity Switch Configuration 658

Message Waiting Indicator (MWI) 659
Dual-Tone Multifrequency (DTMF) Relay Problems 661
Additional Unity Troubleshooting 662
More Troubleshooting Resources for Unity 662
SMDI Integration 662
Understanding SMDI Messages 663
Call History Information for Calls to Voice Mail from CallManager 664
Message Waiting Indicator On/Off Messages 665
Error Messages 666
Cisco Messaging Interface 666
CMI Configuration Parameters 667
Reading CMI Traces 674
Using HyperTerminal to Diagnose SMDI Problems 679
Message Waiting Indicator Problems 682
Cisco VG248 SMDI Integration 686
Verifying Configuration Parameters 686
Message Waiting Indicator Problems 690
Octel Voice Mail Digital Integration Via a DPA Voice Mail Gateway 693
Verify Cabling 693
Check Port Status 697
Troubleshooting DPA MWI Problems 702
Using the DPA Event Log 703
Best Practices 703
Summary 704
Chapter 15 Survivable Remote Site Telephony (SRST) 707
SRST Operation 707
SRST Configuration 709
IP Phone Registration 712
SRST Dial Plan 718
Debugging Call Control in SRST Mode 719
Problems with Transferring Calls in SRST Mode 729
IP Phones Stuck in SRST Mode 730
Voice Mail and Forwarding Features in SRST Mode 731
DHCP Considerations When Using SRST 731
Best Practices 732
Summary 733

xxii

Chapter 16 Applications 735

Customer Response Applications (CRA) 736
Checking TSP or JTAPI Plugin Versions 736
IP IVR and IP AA 737
CRA Administration Problems 738
Directory Configuration 741
Verifying Configuration 744
Engine Status 745
Collecting Traces 748
Extension Mobility for CallManager 3.1 and 3.2 756
CallManager Extension Mobility Configuration 758
CRA Extension Mobility Configuration 759
Configuration Summary 762
Understanding the Login and Logout process 763
Troubleshooting Extension Mobility on CallManager 3.1 and 3.2 765
Extension Mobility for CallManager 3.3 773
Understanding the Login and Logout Process 775
Troubleshooting Extension Mobility on CallManager 3.3 777
Cisco CallManager Attendant Console 779
Understanding the Server Components 780
Understanding the Attendant Console Client 781
Troubleshooting Attendant Console 782
Cisco Personal Assistant 785
Call Routing Problems and Personal Assistant 785
Personal Assistant and Message Waiting Indicator Issues 786
Cisco IP SoftPhone 786
Line Number Displays, But No Dial Tone 786
Echo Problems with Cisco IP SoftPhone 787
One-way Audio and Using Cisco IP SoftPhone over VPN 787
Cisco IP SoftPhone Has No Lines 788
Cisco IP Phone Services 788
Cisco IP Videoconferencing (IP/VC) 789
Cisco Conference Connection 789
Ensure the Necessary Services Are Started 790
Using Event Viewer with Conference Connection 791
Cisco Emergency Responder (ER) 791
Summary 791

xxiii

Chapter 17 SQL Database Replication 793

Understanding the Publisher-Subscriber Model 793
Troubleshooting the Publisher-Subscriber Relationship 796
The Role of Name Resolution and Passwords in Replication 796
Microsoft SQL Server Enterprise Manager 802
Correcting Replication Errors 804
Re-establishing a Broken SQL Replication Subscription 807
Deleting the Subscription from the Publisher 807
Adding the Subscription to the Subscriber SQL Server 808
Starting the Snapshot Agent 809
Reinitializing a Subscription 809
CDR Replication Issues 809
Subscriber Is Not Configured to Generate CDRs 810
Database Layer Monitor Is Not Running Properly 812
Additional Problems with Writing CDRs 813
Best Practices 815
Summary 816
Chapter 18 LDAP Integration and Replication 819
Directory Integration Versus Directory Access 820
Providing Endpoints with Corporate Directory Access 821
Troubleshooting Corporate Directory Access 823
Using the CallManager Embedded Directory 823
Troubleshooting the CallManager Embedded Directory 824
Reconfiguring DC Directory on the Publisher 827
CallManager 3.3 Reconfiguration Steps 828
CallManager 3.03.2 Reconfiguration Steps 830
Reconfiguring DC Directory on Subscribers 835
Understanding and Troubleshooting Active Directory Integration 837
Troubleshooting Common Problems with Installing the Customer Directory
Configuration Plugin 839
Preparing Active Directory to Allow Schema Modifications 840
Ensuring Domain Name Accuracy for Active Directory 841
Verifying Distinguished Name Administrative Rights After Cisco Customer
Directory Configuration Plugin Failure 842
Checking Log Files for Errors 842
Miscellaneous Troubleshooting Items 843
Understanding and Troubleshooting Netscape iPlanet Integration 844
Best Practices 845
Summary 846

xxiv

Appendix A Cisco IP Telephony Protocol and Codec Information and References 849
Protocols 849
Codecs 855
Appendix B NANP Call Routing Information 857
Appendix C Decimal to Hexadecimal and Binary Conversion Table 881
Appendix D Performance Objects and Counters 891
Cisco Performance Objects and Counters 891
Cisco Analog Access Object 892
Cisco CallManager Object 893
Cisco CallManager Attendant Console ObjectRelease 3.3(2) 898
Cisco CallManager System Performance Object 900
Cisco CTI Manager Object 903
Cisco Gatekeeper Object 904
Cisco H.323 Object 904
Cisco HW Conference Bridge DeviceRelease 3.3(2) 904
Cisco Lines Object 905
Cisco Locations Object 906
Cisco Media Streaming App Object 906
Cisco Media Termination Point ObjectThrough Release 3.3(2) 909
Cisco Messaging Interface Object 910
Cisco MGCP FXO Device Object 911
Cisco MGCP FXS Device Object 912
Cisco MGCP Gateways Object 912
Cisco MGCP PRI Device Object 913
Cisco MGCP T1 CAS Device Object 914
Cisco MOH Device Object 915
Cisco MTP Device Object 916
Cisco Music on Hold Server ObjectThrough Release 3.3(2) 916
Cisco Phones Object 918
Cisco SW Conference Bridge ObjectThrough Release 3.3(2) 918
Cisco SW Conference Bridge Device ObjectRelease 3.3(2) 919
Cisco TFTP Object 920
Cisco Transcode Device Object 923
Cisco Unicast Hardware Conference Object 924
Cisco Unicast Software Conference Bridge Device ObjectThrough Release 3.3(1) 924
Cisco WebAttendant ObjectThrough Release 3.3(1) 924
Windows 2000 Objects 924
Glossary

927

Index

947

xxv

Foreword
In November of 1998, Cisco Systems acquired a small startup called Selsius Systems. For over a year this small company had been shipping the worlds rst IP phones and Windows NT-based call management software consisting of
close to a million lines of C++ code with a small development staff of about 40 engineers. Since the acquisition, the
code base has evolved into many millions of lines of C++, XML, and Java code, and the development staff now has over
500 engineers. The level of sophistication and capability has increased dramatically and is a key component of the Cisco
Architecture for Voice, Video, and Integrated Data (AVVID). Current deployments range from extremely distributed
enterprises with hundreds of remote ofces to small 50-person ofces. Geographically, systems are deployed across the
world, including exotic locations such as Antarctica and the International Space Station!
AVVIDs IP Telephony components (including the IP phones, gateways, and Cisco CallManager) comprise a telephony
system that is both richer than and different from traditional TDM-based phone systems. For example, manageability
and serviceability are achieved through either a browseable interface or an XML SOAP-based protocol for integration
with existing IT systems. Geography disappears as a problem because telephony functions, manageability, and serviceability all traverse the IP network. Proprietary databases disappear in favor of standard SQL databases and LDAP directories. Nevertheless, this unication and standardization of telephony on IP networks also presents unique challenges.
Voice quality can be impacted by poor IP network design. Capacity planning requires consideration of IP address numbering. Music on Hold as a multicast stream requires proper switch and router conguration. These are only a few
examples of the unique considerations that must be given to IP Telephony deployments.
This book incorporates the authors real-life experiences in planning and troubleshooting IP Telephony within the
AVVID solution. The wisdom contained herein has been gained over the course of thousands of real customer experiences. Paul Giralt and Addis Hallmark are two of the very best troubleshooters in the industry, and Anne Smith has written
about and worked with the system since the earliest releases. Paul has been with Ciscos customer support organization
for several years. His depth and breadth of knowledge across all Cisco products are legendary, including his most recent
focus on IP Telephony. I have seen him in action at some very large and sensitive customer installations, where he
resolved extremely difcult problems and provided excellent guidance during upgrades and installations. We were fortunate to get him back, inasmuch as our customers were loathe to let him leave! Addis has been involved in the development and testing of many AVVID products. He has been personally engaged with many key customers during
deployment and operation and has received numerous rave reviews from customers. Addis also has been instrumental in
the security design aspects of Cisco CallManager. Anne is an author and the technical editor for this and several other
AVVID books. She has been engaged with the technology since its inception at Selsius Systems.
I highly recommend this book to any individual or organization involved in installing, operating, or troubleshooting one
of the most exciting advances in the long history of telephony. Written by three of its pioneers, this book serves as a
guide for the rest of the pioneers who arent afraid to help their organization communicate in its own way, the better
way, the IP way.
Richard B. Platt
Vice President for Enterprise Voice, Video Business Unit
Cisco Systems, Inc.

xxvi

Introduction
This book teaches you the troubleshooting skills you need to isolate and resolve IP telephony problems. IP
telephony is a relatively new technology with many different components. The Cisco IP Telephony (CIPT)
solution revolves around Cisco CallManager, the core call processing engine. CIPT includes many different
endpoints, such as IP phones, various gateways, and various applications such as Cisco IP IVR,
Cisco CallManager Attendant Console, Cisco IP SoftPhone, Cisco Conference Connection, extension
mobility, and more. Additionally, the network infrastructure plays an important role in prioritizing voice
packets to ensure quality of service (QoS).
With all these components involved in transmitting voice across packet networks, it is essential that you be
able to identify and resolve issues in the entire solution. This requires knowledge of the functionality of
these components and how they interact with each other, as well as what tools are available to help you nd
the root cause when problems arise. This book educates you about the techniques, tools, and methodologies
involved in troubleshooting an IP telephony system.

Target CallManager Release

This book is written to CallManager release 3.3. Updates to this book may be provided after publication.
You should periodically check the ciscopress.com web site for updates (go to ciscopress.com and search for
Troubleshooting Cisco IP Telephony).

Goals and Methods

This book intends to deliver a methodology you can follow when troubleshooting problems in an IP telephony network, particularly a Cisco IP Telephony solution. This book provides detailed troubleshooting
information that applies to a variety of problems that can occur in any IP telephony deployment.
Best Practices sections in each chapter provide tips and design considerations to help you avoid common
conguration problems.

Who Should Read This Book?

This book is designed to teach you how to isolate and correct problems in an IP telephony network. If you
are a networking professional responsible for administering a Cisco IP Telephony (CIPT) system, this book
is for you. Although this books main focus is on CIPT, some concepts apply to IP telephony in general as
well.
You will best be able to assimilate the information in this book if you already have a working knowledge of
a CIPT network.

How This Book Is Organized

Although you could read this book cover-to-cover, it is designed to help you nd solutions to specic problems. The chapters are organized by the various components of a Cisco IP Telephony solution. Four appendixes provide reference information.
Chapter 1, Troubleshooting Methodology and ApproachYou can troubleshoot
even the most complex problems if you have a good methodology in place for nding the
root cause. This chapter focuses on teaching that methodology: learning how to nd clues
and track down your suspect by breaking the problem into smaller pieces and tackling
each piece individually.

xxvii

Chapter 2, IP Telephony Architecture OverviewCisco AVVID includes many

different components that come together to form a comprehensive architecture for voice,
video, and integrated data. This chapter covers the basic components of the IP Telephony
architecture in order to provide a big-picture view of the system.

Chapter 3, Understanding the Troubleshooting ToolsTo effectively troubleshoot

problems in a Cisco IP Telephony network, you must be familiar with the many tools at
your disposal. In addition, you need to know how to best use those tools to achieve
maximum results. This chapter describes the various tools and their different uses.

Chapter 4, Skinny Client Registration IP phone registration is a common source

of problems. This chapter describes how Skinny protocol-based device registration works,
including discussions of inline power, network connectivity, and potential TFTP and
CallManager issues.

Chapter 5, IP PhonesIP phones can encounter various problems, from unexpected

resets to directory and service problems, and more. This chapter explains proper IP phone
behavior and examines problems that can occur after an IP phone successfully registers.

Chapter 6, Voice GatewaysVoice gateways are the interface that bridges the Voice
over IP (VoIP) world with the Public Switched Telephone Network (PSTN). Voice
gateways can be Cisco IOS Software gateways or modules within voice-enabled LAN
switches. They can be analog or digital, and they can use a wide variety of signaling
protocols. This chapter teaches you how to identify and resolve gateway problems by
breaking these components into logical groups and following a methodical troubleshooting approach.

Chapter 7, Voice QualityVoice quality is a broad term that covers the following
conditions: delayed audio, choppy or garbled audio, static and noise, one-way or no-way
audio, and echo. This chapter focuses on the information you need to investigate and
resolve voice quality problems in an IP Telephony network.

Chapter 8, Fax Machines and ModemsFax machines and modems present unique
challenges when carried over an IP Telephony network, primarily due to their unforgiving
nature concerning any modication to the audio stream. This chapter discusses the effect
of packet loss and jitter, fax passthrough, fax relay, and how to troubleshoot modems and
faxes.

Chapter 9, Call RoutingPossessing a strong understanding of call routing is

arguably one of the most important aspects of a smooth-operating CIPT solution. This
chapter discusses closest-match routing, calling search spaces and partitions, transformations, and translation patterns as well as troubleshooting hold, transfer, park, and
call pickup.

Chapter 10, Call PreservationCall preservation is easier to predict when you

understand the protocol interaction with CallManager. This chapter provides guidelines
for determining call survivability based on endpoint type and protocol.

xxviii

Chapter 11, Conference Bridges, Transcoders, and Media Termination Points

Conference bridges, transcoders, and media termination points are media resources. This
chapter discusses the role of media resource groups and media resource group lists, codec
selection, and troubleshooting transcoder and conference bridge resources.

Chapter 12, Music on HoldThe Music on Hold feature allows callers to hear
streaming audio while on hold. This chapter describes this feature and provides steps to
take if you encounter problems.

Chapter 13, Call Admission ControlCall admission control is used in situations

where a limited amount of bandwidth exists between telephony endpoints such as phones
and gateways. This chapter discusses the two types of call admission controllocationsbased and gatekeeperand the mechanisms available to reroute calls through the PSTN
in the event of WAN congestion.

Chapter 14, Voice MailCallManager is compatible with a variety of voice mail

systems that integrate with CallManager through various methods. This chapter focuses
on troubleshooting the integration of CallManager and three types of voice mail systems:
Cisco Unity, third-party voice mail systems integrated via Simple Message Desk Interface
(SMDI), and Octel Voice Mail, integrated through Cisco DPA Voice Mail gateways.

Chapter 15, Survivable Remote Site Telephony (SRST)SRST allows a router at a

remote branch to assume call processing responsibilities in the event that phones at a
remote site are unable to contact the central CallManager. This chapter describes SRST
and provides detailed information about the various problems that can occur.

Chapter 16, ApplicationsCisco AVVID allows for the creation of many different
applications to interoperate within the converged network. This chapter discusses some of
the primary applications in a Cisco AVVID IP Telephony solution, such as IP AA and IP
IVR, extension mobility, Cisco IP SoftPhone, Personal Assistant, and Cisco CallManager
Attendant Console.

Chapter 17, SQL Database ReplicationThe SQL relational database stores the
majority of CallManager conguration information. This chapter discusses the PublisherSubscriber model for database replication, name resolution, Enterprise Manager,
Replication Monitor, broken subscriptions, and CDR database replication.

Chapter 18, LDAP Integration and ReplicationUser information is stored in a

Lightweight Directory Access Protocol (LDAP) database. This chapter describes
directory integration versus directory access, using the CallManager embedded directory,
and integrating with Active Directory and Netscape iPlanet.

Appendix A, Cisco IP Telephony Protocol and Codec Information and

ReferenceCisco IP Telephony employs many different protocols and codecs. This
appendix provides a list of applicable protocols and codecs with descriptions and the
standards body corresponding to the protocol or the Request for Comments (RFC)
number. Compression rates are given for each codec.

xxix

Appendix B, NANP Call Routing InformationCallManager provides a built-in dial

plan for the North American numbering plan (NANP). This appendix provides
information from the NANP le located in the C:\Program Files\Cisco\Dial Plan
directory. This le shows you how each part of an NANP number corresponds to a specic
placeholder. It is particularly useful when youre learning how to apply route lters.

Appendix C, Decimal to Hexadecimal and Binary Conversion TableThis

appendix provides a cheat sheet that shows you how to quickly convert between decimal,
hexadecimal, and binary values.

Appendix D, Performance Objects and CountersMicrosoft Performance

(PerfMon) and the Real-Time Monitoring Tool allow you to monitor your system through
the use of performance counters. This appendix lists and describes the performance
objects and counters in a Cisco IP Telephony network. Some pertinent Windows 2000
counters are also described.

GlossaryThe glossary denes terms and acronyms used in this book.

Best Practices
In a perfect world, there would be no need for this book, because systems would always run perfectly. Unfortunately, in the real world, problems do arise, and they usually dont go away on their own. However, an
administrator/installer can proactively take steps to ensure reliability and high availability and minimize the
number of problems that arise.
Best practices include not only design considerations but also monitoring and management. A properly monitored system can detect failures before they become service-affecting. Each chapter contains a section outlining best practices as they apply to the chapter topic.
In a properly designed network, you can achieve 99.999 percent reliabilitya rating that is expected of a telephone system.

High Availability in an IP Telephony Environment

High availability for IP telephony is based on distribution and core layers in the network and
servers (call processing, application servers, and so on). BellCore Specication GR-512 denes
what criteria must be met to achieve ve 9s (99.999 percent) reliability. A careful examination
of this document is recommended if you are interested in understanding 99.999 percent
reliability. Note that many events are not counted against ve 9s reliability. Some of these
events include the following:
Outages of less than 64 devices

Outages less than 30 seconds in duration

Outages due to planned maintenance

Outages due to outside causes, such as power loss from utility or network circuit failures
caused by the provider

The Cisco AVVID IP Telephony solution can achieve 99.999 percent reliability per the
BellCore FR-512 specication.

xxx

Command Syntax Conventions

The conventions used to present command syntax in this book are the same conventions used in the IOS
Command Reference. The Command Reference describes these conventions as follows:
Vertical bars | separate alternative, mutually exclusive elements.

Square brackets [ ] indicate an optional element.

Italic indicates arguments for which you supply actual values.

Braces { } indicate a required choice.

Braces within brackets [{ }] indicate a required choice within an optional element.
Boldface indicates commands and keywords that are entered literally as shown. In actual
conguration examples and output (not general command syntax), boldface indicates
commands that the user inputs (such as a show command).

OSI Reference Model

Throughout the book, a few references are made to the OSI model. Table I-1 provides a brief primer on the
OSI reference model layers and the functions of each. You can learn more about the OSI model in any of the
Cisco Press books that target the CCNA certication.
Table I-1

OSI Reference Model Overview

OSI Layer Name

Functional Description

Examples

Physical (Layer 1)

Responsible for moving bits of data between devices. Also

species characteristics such as voltage, cable types, and
cable pinouts.

EIA/TIA-232, V.35

Data link (Layer 2)

Combines bytes of data into frames.

802.3/802.2, HDLC

Provides access to the physical media using a Media Access

Control (MAC) address, which is typically hard-coded into
a network adapter. Also performs error detection and
recovery for the data contained in the frame.
Network (Layer 3)

Uses logical addressing which routers use for path

determination. Can fragment and reassemble data if the
upper-layer protocol is sending data larger than the data link
layer can accept.

IP, IPX

Transport (Layer 4)

Provides reliable or unreliable delivery of data packets.

Allows for multiplexing of various conversations using a
single network-layer address. Can also ensure data is
presented to the upper layers in the same order it was
transmitted. Can also provide ow control.

TCP, UDP

Session (Layer 5)

Sets up, coordinates, and terminates network connections

between applications. Also deals with session and
connection coordination between network endpoints.

Operating systems
and application
access scheduling

xxxi

Table I-1

OSI Reference Model Overview (Continued)

OSI Layer Name

Functional Description

Examples

Presentation (Layer 6)

Denes how data is presented to the application layer.

JPEG, ASCII

Can perform special processing, such as encryption, or can

perform operations such as ensuring byte-ordering is
correct.
Application (Layer 7)

Interface between network and application software.

Telnet, HTTP

Comments for the Authors

The authors are interested in your comments and suggestions about this book. Please send feedback to the
following address:
[email protected]

Further Reading
The authors recommend the following sources for more information.

Cisco Documentation
This book provides comprehensive troubleshooting information and methodology. However, details about
common procedures might not be provided. You should be familiar with and regularly use the documentation that is provided with the Cisco IP Telephony system to supplement the information in this book.
You can nd Cisco IP Telephony documentation by searching for a specic product on Cisco.com or by
starting at the following link:
www.cisco.com/univercd/cc/td/doc/product/voice/index.htm
You can examine the following books at a technical bookseller near you or online by entering the title in the
search box at www.ciscopress.com.

Cisco CallManager Fundamentals: A Cisco AVVID Solution

You can nd detailed information about CallManagers inner workings in the book Cisco CallManager
Fundamentals (ISBN 1-58705-008-0).

Developing Cisco IP Phone Services: A Cisco AVVID Solution

You can nd instructions and tools for creating custom phone services and directories for Cisco
IP Phones in the book Developing Cisco IP Phone Services (ISBN 1-58705-060-9).

Cisco IP Telephony
You can nd installation, conguration, and maintenance information for Cisco IP Telephony networks in
the book Cisco IP Telephony (ISBN 1-58705-050-1).

xxxii

Integrating Voice and Data Networks

You can nd information on how to integrate and congure packetized voice networks in the book Integrating Voice and Data Networks (ISBN 1-57870-196-1).

Cisco Router Conguration, Second Edition

Cisco Router Conguration, Second Edition (ISBN 1-57870-241-0) provides example-oriented Cisco IOS
Software conguration for the three most popular networking protocols used todayTCP/IP, AppleTalk,
and Novell IPX.

xxxiii

Icons Used in This Book

Throughout this book, you will see a number of icons used to designate Cisco-specic and general networking devices,
peripherals, and other items. The following icon legend explains what these icons represent.
Network Device Icons

CallManager

SRST Router

IP Phone

Used For:
Application Server
DHCP
DNS
MOH Server
MTP
SW Conference Bridge
Voice Mail Server

Stations

Router

Switch

PIX Firewall
Layer 3 Switch

Modem
Access Server

Gateway or
3rd-party
H.323
Server

PBX/PSTN Switch

PBX (Small)

Cisco
Directory
Server

ATM Switch

Local Director

V
Used For:
Analog Gateway
Gatekeeper
Gateway
H.323 Gateway
Voice-enabled Router

DAT Tape

V
PC

Laptop

Server
PC w/Software

POTS Phone

Relational
Database

Fax Machine

Media/Building Icons
Ethernet Connection

Serial Connection

Network Cloud
Telecommuter

Building

Branch
Office

Used For:
HW Conference Bridge
Transcoder
Voice-enabled Switch

CHAPTER

Understanding the
Troubleshooting Tools
To effectively troubleshoot problems in a Cisco IP Telephony (CIPT) network, you must be
familiar with the many tools at your disposal. In addition, you need to know how best to use
those tools to achieve maximum results. This chapter describes the various tools and their
different uses.
Depending on the problem you encounter and your particular skill set, you might nd
certain tools more helpful than others. Nevertheless, knowing what tools are available and
how they can help you solve the problem is essential to a successful resolution. Also, some
tools might be more useful to you based on your past experience. If you are strong in IP, a
sniffer trace might be your preferred tool when applicable. However, if you are stronger in
call processing-related traces, the Cisco CallManager (CCM, also sometimes called SDI)
traces might prove helpful once you understand how they work.
Later chapters demonstrate the use of these tools in different scenarios you might encounter.
This chapter covers the following topics:

Time synchronizationExplains how to synchronize the clocks on all devices in an

IP telephony network to ensure that timestamps on your trace les and debugs are
synchronized with each other.

Reading CCM (or SDI) tracesDescribes one of the most important trace les you
use to troubleshoot CallManager-related problems. CCM trace les provide information about call processing events and all messages exchanged between Skinny,
MGCP, and H.323 endpoints.

Reading SDL tracesDiscusses the components of the less-used Signal Distribution

Layer (SDL) trace les in CallManager. SDL traces describe the events occurring in
the CallManager software at a code level. These traces are usually reserved for Cisco
development engineering use; however, there are a few key pieces of information you
can use to your advantage when troubleshooting CallManager problems.

Microsoft Performance (PerfMon)Details the capabilities of PerfMon, a built-in

Windows 2000 utility that helps you troubleshoot CallManager.

CCEmailDetails the third-party alerting tool that can be used in conjunction with
PerfMon to congure alerts for the performance counters.

38

Chapter 3: Understanding the Troubleshooting Tools

CallManager ServiceabilityDiscusses various web-enabled tools provided with

CallManager for reading alarms and XML-based tracing.

Real Time Monitoring Tool (RTMT)Describes the Cisco web-based monitoring

application that allows you to view CallManager cluster details, monitor performance
objects (much like PerfMon), and monitor devices and CTI applications.

Call detail records (CDR) and the CDR Analysis and Reporting (CAR) Tool
Describes the CAR tool that helps you analyze the raw data that comprises the CDR
database and create reports based on your search criteria.

CDR Time ConverterDescribes how to use this small utility that allows you to
convert the UNIX Epoch-based date and time format stored in a CDR to standard date
and time format.

Event ViewerBriey explains the function of another built-in Windows 2000 tool
that plays a key role in troubleshooting CallManager.

Q.931 Translator and Enhanced Q.931 TranslatorDescribes two tools that read
a CCM trace le and analyze all Q.931 and H.225 messages. The various information
elements are decoded for ease of reading. The Enhanced Q.931 Translator also adds
additional search and ltering options and decodes more information elements than
the original Q.931 Translator.

Dick TracyDescribes an important tool used to troubleshoot the WS-X6608 and

WS-X6624 voice gateways for the Catalyst 6000 series switches.

Sniffer tracesDiscusses when and why to use a network packet-capture tool.

Cisco Bug Toolkit (formerly Bug Navigator)Describes the web-based tool that
allows you to nd known bugs based on software version, feature set, and keywords.
The resulting matrix shows when each bug was integrated or xed if applicable.

Remote Access ToolsDescribes applications like Terminal Services and Virtual

Network Computing, which allow you to access a server from a remote location.

Websites and Further ReadingProvides URLs for websites that contain

additional troubleshooting information. Also points you to a section in the
Introduction with a recommended reading list.

Voice Codec Bandwidth CalculatorDescribes how to use the Voice Codec

Bandwidth Calculator to determine the bandwidth used by different codecs with
various voice protocols over different media.

Time Synchronization
Time synchronization is simply making sure that all the participating CallManager servers
and network devices have the same exact time. Time synchronization is critical. A large
CallManager cluster can have eight or more separate servers, not including any voice mail
servers or application servers. This distributed architecture creates a highly available and

Time Synchronization

39

scalable system. It also makes the troubleshooting process more involved because you have
to collect traces from all participating servers to see the full picture of what happened.
As endpoints such as IP phones and voice gateways call each other, signaling occurs between CallManager and the endpoint device. Signaling also occurs between the respective
CallManagers of each endpoint device. If a problem occurs, you need to consolidate trace
les from all involved servers. CallManager Serviceability, discussed later in this chapter,
can collect this information into one le. However, if the timestamps of each le are mismatched, it can be impossible to tell what the real series of call processing events is.
All the CallManager servers can be time-synched using the Network Time Protocol (NTP).
When you synchronize the time on all involved servers, all the trace les are timestamped
the same. When trace les are collected for analysis, you can follow the true series of call
processing events with accuracy.
In addition to CallManager servers, you should ensure all network devices such as switches,
routers, and voice gateways are synchronized to the same time source as the CallManager
servers. This ensures consistent timestamps regardless of which device you are looking at.

Conguring Automatic Time Synchronization on CallManager

Servers
Use the following steps to congure the CallManager server to automatically synchronize
and stay synchronizedwith a Time Server.
Step 1 Verify that the NetworkTimeProtocol service is congured to launch

automatically upon startup. Right-click My Computer and select

Manage.
Step 2 Expand the Services and Applications section, and select Services.
Step 3 Double-click the NetworkTimeProtocol service, and ensure that Startup

Type is set to Automatic.

Step 4 Congure the C:\WINNT\ntp.conf le. This le contains the list of time

servers that CallManager will synchronize with. You can congure

CallManager to point to specic time servers (see Example 3-1), or you
can congure it to receive NTP broadcasts (see Example 3-2) on the local
LAN segment from the router (as long as the router is congured to do
so).
Example 3-1 Sample ntp.conf File Using Static Time Servers
server 10.0.0.10
server 10.1.0.10
driftfile %windir%\ntp.drift

40

Chapter 3: Understanding the Troubleshooting Tools

Example 3-2 Sample ntp.conf File Using an NTP Broadcast Router

broadcastclient
driftfile %windir%\ntp.drift

Step 5 Go to the Services Control Panel and stop/start the

NetworkTimeProtocol service. Allow several minutes for the update to

take place.

Synchronizing Time Manually on CallManager Servers

Use the following steps to manually congure time synchronization.
Step 1 Stop the NetworkTimeProtocol service in the Services Control Panel.
Step 2 Synchronize the clock by using one of the following commands from a

command prompt.
To synchronize with a remote Time Server, use the following command
where x.x.x.x is the IP address of the Time Server:
ntpdate x.x.x.x

To synchronize with a Broadcast Router, use the command where x.x.x.x

is the IP address of the Ethernet port of the router:
ntpdate x.x.x.x

Step 3 Restart the NetworkTimeProtocol service in the Services Control Panel.

Synchronizing Time on Cisco IOS Devices

In addition to your CallManager servers, you should also ensure the time is synchronized
on all your network devices, including IOS voice gateways, switches, and routers.
For Cisco IOS devices, you can use the Network Time Protocol (NTP) to synchronize the
time. To enable NTP, you must congure the time zone the IOS device is in along with the
IP address of the NTP server. You should congure the IOS device to take daylight savings
time into account as well if you live in a time zone that observes daylight savings time.
First you must congure the time zone. Use the command clock timezone
name_of_time_zone offset_ from_GMT to configure the time zone. For example, the
following command congures the IOS device for Eastern Standard Time (EST):
clock timezone EST -5

If you are in a time zone that observes daylight savings time, use the command clock
summer-time name_of_time_zone recurring to enable automatic daylight savings time.
For example, the following congures Eastern Daylight Savings Time:

Time Synchronization

41

clock summer-time EDT recurring

You can determine time zones by performing a search at Google.com; search for a string
such as time zone GMT. Any one of the many hits should point you to a table showing
the various time zones around the world, including daylight savings time.
Once your time zone is congured properly, you can enable NTP. If you do not have a
device on the network running NTP, you can make an IOS device into an NTP master clock.
You can do this by conguring the command ntp master on the IOS device. If you are not
making the device an NTP master, you should congure the IP address of the NTP server
using the command ntp server NTP_server_IP_address. For example, the following tells
the IOS device to synchronize its clock with the NTP server at IP address 172.18.109.1:
ntp server 172.18.109.1

Once you have enabled NTP, all IOS devices should synchronize their clocks with the
central NTP server. You should also synchronize the clocks on non-IOS network devices
such as switches running CatOS software.

Synchronizing Time on CatOS Devices

You can use NTP to synchronize the clocks on switches running CatOS software. The
conguration is similar to conguring NTP on a device running Cisco IOS Software.
First you must congure the time zone the switch is in using the command set timezone
name_of_time_zone offset_ from_GMT. For example, the following sets the clock to
Eastern Standard Time:
set timezone EST -5

You can also congure daylight savings time using the command set summertime enable
name_of_time_zone. For example, the following enables Eastern Daylight Savings Time:
set summertime enable EDT

You can determine time zones by performing a search at Google.com; search for a string
such as time zone GMT. Any one of the many hits should point you to a table showing
the various time zones around the world, including daylight savings time.
Once you have the time zone congured properly, you can set the NTP server IP address
and enable the NTP client on the switch. First set the NTP server IP address using the command set ntp server NTP_server_IP_address. Then enable the NTP client using the
command set ntp client enable. The following shows the conguration to use the NTP
server with IP address 172.18.109.1:
set ntp server 172.18.109.1
set ntp client enable

Once you have enabled proper time synchronization in your network, you can move on
to reading the variety of trace les available to you for troubleshooting problems in an
IP Telephony network.

42

Chapter 3: Understanding the Troubleshooting Tools

Reading CCM (or SDI) Traces

CCM (also known as System Diagnostic Interface or SDI) traces are the user-friendliest call
processing trace les you have available to you. After you learn a few tricks, it is easy to
follow the call ows and nd the potential problem. Although you might see SDI and
CCM used interchangeably to refer to this type of trace, in this book we primarily refer
to this type of trace as a CCM trace. Some pages in CallManager Serviceability refer to
CCM traces as SDI traces.
Because CallManager is at the heart of a CIPT network, CCM traces are usually the rst
place to look when troubleshooting most problems. You can analyze problems related to
device registration, call ow, digit analysis, and related devices such as IP phones, gateways, gatekeepers, and more. By the end of this section, you should be able to follow some
basic call ows in the CCM trace. Future chapters continue to show CCM trace examples
as you learn how to troubleshoot more problems. Later in this chapter you also learn about
the Q.931 Translator, which is useful for quickly troubleshooting a variety of gateway
problems. It is not a substitute for learning how to read the CCM trace, but it helps you
quickly examine a trace in a graphical format without having to wade through irrelevant
debugs. See the later Q.931 Translator and Enhanced Q.931 Translator section for more
information.

Setting the Appropriate Trace Level and Flags

CallManager allows you to select from a variety of different options that adjust which
events are logged to the CCM trace les. If you know exactly what you are looking for, you
can congure CallManager to log only specic events in the CCM trace le. If you congure the trace to collect too much information, the trace becomes more difcult to analyze.
However, if you do not congure the trace to collect enough information, you might miss
the problem you are trying to nd. When you are beginning to learn how to read CCM trace
les, it is best to enable more debugs than less. As you learn which trace settings are
required for specic problems, you can enable just the settings you need.
Unfortunately you can never predict what problems will come up, so if you do not have a
high level of tracing enabled, you may have to wait for a problem to happen a second time
after enabling the appropriate trace settings before you can troubleshoot. For this reason, it
is usually best to leave a majority of the trace ags enabled during normal operating conditions so you have trace data available if a problem occurs.
You congure a CCM trace in Cisco CallManager Serviceability (Trace > Conguration).
Figure 3-1 shows the top half of the Trace Conguration page, and this section discusses
the relevant elds and settings on that page.

Reading CCM (or SDI) Traces

Figure 3-1

43

Trace Conguration (Part 1) in CallManager Serviceability

First thing to note is the Trace On checkbox. This must be selected for any of the trace
settings to be available.
The Apply to All Nodes checkbox allows you to apply the specied trace settings to all
CallManager servers in the cluster. This is useful when you are troubleshooting a problem
that is occurring on more than one CallManager server. You generally want to keep the same
level of tracing enabled on all the servers in the cluster unless you are absolutely certain the
problem is isolated to a single server in the cluster. Because of the distributed nature of
CallManager, you may think a process only involves a single server when in reality part of
the processing for a particular call is occuring on another server in the cluster.
The Trace Filter Settings area allows you to specify the exact parameters of your trace.
First, you set the level of tracing you want to perform in the Debug Trace Level eld.

44

Chapter 3: Understanding the Troubleshooting Tools

CallManager Serviceability provides six different levels, but for nearly every kind of
problem that youll want traces for, the recommended levels are either Detailed or
Arbitrary:

DetailedProvides detailed debug information and highly repetitive messages that

are primarily used for debugging, including KeepAlives and responses. For
CallManager releases 3.1 and earlier, be cautious about using this level during normal
production hours on a heavily loaded system. It can cause performance degradation
on CallManager. In release 3.2 and later, CallManager uses asynchronous tracing to
reduce the impact that trace le generation has on call processing.

ArbitraryProvides low-level debug traces. This level is best suited for debugging
difcult problems. This level includes nearly everything that is included in Detailed
with the exception of KeepAlives. If you are not troubleshooting problems related to
missed KeepAlives, this level is good for day-to-day troubleshooting.

Other trace levels are provided, but generally, they dont offer as much information as the
Detailed and Arbitrary levels. The other levels are

ErrorProvides traces generated in abnormal conditions, such as coding errors or

other errors that normally should not occur.

SpecialProvides traces for all informational, non-repetitive messages such as

process startup messages, registration messages, and so on. All system and device
initialization traces are at this level.

State TransitionProvides traces for call processing events or normal events traced
for the subsystem (signaling layers).

SignicantProvides traces for media layer events.

Second, make sure the Enable CallManager Trace Fields checkbox is selected, which
gives you the opportunity to select which specic traces you want to run and to choose your
traces. Table 3-1 describes the trace elds to choose from. Most are reasonably selfexplanatory. For example, if youre having problems with an H.323 device, you would
enable the H245 Message Trace and Enable H225 & Gatekeeper Trace options.
Likewise, for music on hold (MOH) issues, the Enable Music on Hold option would
display trace relating to all MOH activity.
Table 3-1

Cisco CallManager Trace Fields

Field Name

Description

Enable H245 Message Trace

Debugs H.245 signaling for H.323 calls, including the

media processing messages.

Enable DT-24+/DE-30+ Trace

Activates the logging of events related to the legacy

gateways, Cisco Digital Access DT-24+/DE-30+.

Enable PRI Trace

Activates a trace of Primary Rate Interface (PRI) devices.

Enable ISDN Translation Trace

Activates a Layer 3 trace of Q.931 (ISDN messages).

Reading CCM (or SDI) Traces

Table 3-1

45

Cisco CallManager Trace Fields (Continued)

Field Name

Description

Enable H225 & Gatekeeper Trace

Activates a trace showing H.225 signaling messaging for

H.323 calls.

Enable Miscellaneous Trace

Activates a trace of miscellaneous devices.

Enable Conference Bridge Trace

Activates a trace of the conference bridges. Use this level

to trace conference bridge statuses such as
Registered with CallManager
Unregistered with CallManager
Resource allocation processed successfully
Resource allocation failed

Enable Music on Hold Trace

Activates a trace of MOH devices. Use this level to trace

MOH device statuses such as
Registered with CallManager
Unregistered with CallManager
Resource allocation processed successfully
Resource allocation failed

Enable CM Real-Time
Information Server Trace

Activates CallManager real-time information traces used

by the real-time information server.

Enable CDR Trace

Enables tracing of call detail record (CDR) processing.

However, this trace ag does not provide much information
about CDRs because CDR processing is mostly handled by
the Database Layer Monitor and CDR Insert services
discussed in Chapter 17, SQL Database Replication.

Enable Analog Trunk Trace

Activates a trace of all MGCP-based devices using an

analog interface.

Enable All Phone Device Trace

Activates a trace of phone devices, including Cisco IP

SoftPhones, and shows events such as on-hook, off-hook,
key presses, and so on.

Enable MTP Trace

Activates a trace of media termination point devices and

transcoders. Use this level to trace MTP device statuses
such as
Registered with CallManager
Unregistered with CallManager
Resource allocation processed successfully
Resource allocation failed

Enable All Gateway Trace

Activates a trace of all analog and digital gateways.

continues

46

Chapter 3: Understanding the Troubleshooting Tools

Table 3-1

Cisco CallManager Trace Fields (Continued)

Field Name

Description

Enable Forward and

Miscellaneous Trace

Activates a trace of call forwarding and all subsystems not

covered by another checkbox.

Enable MGCP Trace

Activates a trace showing media gateway control protocol

(MGCP) messages for MGCP-based devices.

Enable Media Resource Manager

Trace

Activates a trace for media resource manager (MRM)

activities.

Next, you can trace based on a specic device name by selecting the Device Name Based
Trace Monitoring checkbox. When that checkbox is selected, you can click the Select
Devices button to choose from a list of devices to trace. Tracing based on specic devices
is very useful when you know which devices are involved in the problem, such as specic
phones or gateways, and want to see trace output only for those devices.
The non-device option is a catch-all; if youre having issues that are not device-related, you
can select the Include Non-device Traces checkbox to see traces not related to devices.
Figure 3-2 shows the bottom half of the Trace Conguration page.
Figure 3-2

Trace Conguration (Part 2) in CallManager Serviceability

Reading CCM (or SDI) Traces

47

For traces to be logged to a le, the Enable File Trace Log checkbox must be selected. The
trace output is then sent to the path specied in the File Name eld, which is C:\Program
Files\Cisco\Trace\CCM\ccm.txt by default.
Were going to skip discussion of the Enable XML Formatted Output for Trace
Analysis checkbox for a moment to nish the elds related to le settings.
The Trace Conguration page in CallManager Serviceability validates the lename and
ensures that it has a .txt extension. Do not use a lename that exists on another computer.
Use a lename that exists on the computer running the trace. It is a good practice to
have each server use a lename format that has the server name in it. This could be
servera-ccm.txt, for example. That way, if trace les are collected from several servers,
each le is easily distinguished from those collected on a different server. One downside
to this, though, is that you cant use the Apply to all nodes option. If you did, you would
have to go back and change the names of the les on all nodes.
The Maximum No. of Files eld defaults to 250. Normally, a maximum of 250 is never
enough les because the les write in round-robin fashionwhen the maximum limit is
met, the next le starts at the beginning and overwrites the rst le. Frequently, this means
that the trace information you need to troubleshoot a problem that began occurring yesterday or the day before has already been overwritten by new les. For each server on which
youre running trace, determine how much free disk space you have, deduct a safety net of
500 MB, and then use the rest of your disk space for trace space. Assuming you dont go
above 10,000 lineswhich we dont recommendyou can calculate the size of your trace
les by guring the average CCM trace at 10,000 lines consumes about 2.5 MB; the
average SDL trace at 10,000 lines consumes about 3.5 MB. We dont recommend going
above 10,000 lines because you want to keep your trace les at a manageable size. At these
average sizes, you can easily zip the le and e-mail it if needed.
The default of 1440 for the Maximum No. of Minutes per File is adequate; youll probably never reach it.
The Enable XML Formatted Output for Trace Analysis checkbox takes the trace
output and formats it in XML, which is required if you want use the Trace Analysis feature
in CallManager Serviceability. With Trace Analysis, you can view the trace les in a web
page, and the XML tagging lets you lter the trace results. The downside, however, is that
the number of lines per le is limited to 2000 instead of 10,000. In most cases, youll
probably want to stick with standard text-based tracing because the 2000 line limit severely
restricts the amount of information in the trace le. Also, reading XML-formatted traces
manually can be far more time-consuming than non-XML-formatted traces because you
need to weed out all the extra XML tags that get added to each trace line. If you do not
enable XML-formatted output, the log le compiles in text format.
The Enable Debug Output String checkbox sends debugging information to a Microsoft
development tool useful only to Cisco development engineers. You should never need to or
be asked to enable this checkbox.

48

Chapter 3: Understanding the Troubleshooting Tools

Click Update to save your settings. The new trace settings take effect immediately when
you click Update.
Once youve specied the trace settings, you can go to the Trace Collection page (Trace >
Collection) to collect traces from one or more servers after an event has occurred. Set the
service you want to retrieve the trace les for (such as Cisco CallManager), along with the
date and time period you want to trace and click the Submit Form button. Depending on
the time period you specied, tracing can impact performance, so be sure to trace short
intervals that wont impact CallManager or during non-production hours.
It is usually quicker and easier to manually collect the traces yourself using either Terminal
Services or VNC (discussed later in this chapter) to access the CallManager server and copy
the les to another machine for analysis.
Once the trace has been gathered, you are given the option to view it in a new window or
use Save As to save the output to a le. If you choose to view the text-based output in a new
window, it looks similar to Figure 3-3.
Figure 3-3

Viewing Text-based Trace Output in Web Browser

If you choose the Save As option, the text-based output displays in the CallManager
Serviceability window in the same format as Figure 3-3, and you can click File > Save As
to save the le.

Reading CCM (or SDI) Traces

49

We generally recommend that you do not use the trace collection utility for anything other
than very small traces on a system that is not busy. Manually copy the traces off the
server(s) in question and analyze them ofine.
If you selected the Collect XML Trace File(s) checkbox and choose to view the output in
a new window, the Trace Analysis dialog appears. In this dialog box, you can lter the trace
results based on the options described in Table 3-2.
Table 3-2

Trace Analysis Filtering

Field Name

Description

CallManager Host

Choose ALL CallManagers or select just one CallManager.

Device Name

Choose ALL or specify the devices name. If you specify a device name,
only trace information pertaining to that device name appears in the
search results.

IP Address

Choose ALL or a specic servers IP address.

Trace Type

Choose ALL, Alarm, or Trace. Trace shows all events as specied in the
trace settings. Alarm shows only specic messages that meet the criteria
of being an information Alarm message.

Cluster

Select this checkbox if you want to include the cluster name in the trace
output.

Date and Time

Select this checkbox if you want to include the date and time of each
event listed in the trace output.

CM Node

Select this checkbox if you want to include the CallManager node (IP
address or host name) in the trace output.

Trace Type

Select this checkbox if you want to include the trace type in the trace
output.

IP Address

Select this checkbox if you want to include the devices source IP address
in the trace output.

Correlation Tag

Select this checkbox if you want to include the number that correlates
traces with each other in the trace output.

Application Name

Select this checkbox if you want to include the directory numbers (DNs)
and other service-specic information in the trace output.

Information

Select this checkbox if you want to include a description of what the

trace found in the trace output.

Device Name

Select this checkbox if you want to include the device name in the trace
output.

Figure 3-4 shows a trace le formatted as XML output and ltered to display only one
CallManager host, one IP address, and the elds Cluster, Date and Time, CM Node, Source
IP, and Information.

50

Chapter 3: Understanding the Troubleshooting Tools

Figure 3-4

Filtered Trace Output in XML Format

You can click the Back to Selection link to return to the Trace Analysis dialog and lter
based on different criteria.
As you can see, reading through a large amount of trace les using the trace collection
utility can be very cumbersome and time-consuming in relation to reading the trace les
manually. You should learn how to read the CCM traces directly from the text les to help
you troubleshoot problems more quickly and accurately. For that reason, all the examples
of CCM traces that we show in this book are in plain text les. The following section
describes how to read a text-based CCM trace.

Reading CCM Traces

This section shows you a few call ow examples and highlights key information in each
example that helps you understand the CCM trace.
For brevity, the header and tail of the trace line have been omitted in many examples. For
example, the complete trace would look like the following:
03/15/2001 05:34:41.956 CCM | StationInit: 1a3e8b54 OffHook.|
<CLID::DLS2-CM152-SRV4-Cluster><NID::DLS2-CM152-SRV4><CT::1,100,96,1.69>
<IP::172.28.238.62><DEV::SEP003094C2D11F>

But for many examples, the header and tail do not add value. So the same trace is shown in
this book as
StationInit: 1a3e8b54 OffHook.

Reading CCM (or SDI) Traces

51

The header portion of the trace line just species the date and time when the trace event was
generated and which trace le you are looking at. For a CCM trace le, every line starts
with the date and time followed by the letters CCM. Prior to CallManager 3.3 the trace
les begin with the date and time followed by the words Cisco CallManager.
You should understand a few things about CCM traces:

Many places in the trace les use hexadecimal equivalents for the IP addressesThe
IP address 172.28.232.164 is shown in trace les as a4e81cac, which is a hexadecimal
representation of 172.28.232.164. You can determine the IP address by working
backwards: take the last two digits, ac, which is hex for 172; then 1c, which is hex for
28; then e8, which is hex for 232; and nally, a4, which is hex for 164. The IP address
is 172.28.232.164. Appendix C, Decimal to Hexadecimal and Binary Conversion
Table, provides a quick cheat sheet to determine how to quickly convert between
decimal, hexadecimal, and binary values.

Trace les sometimes use ASCII for directory numbersConsider the value 33 30 30
31, which is how the directory number 3001 is sometimes displayed in the trace le.

Trace les may include messages for a variety of protocolsDetails on each of these
protocols is described in the appropriate chapters. For example, the Skinny protocol
is discussed in Chapter 4, Skinny Client Registration, and Chapter 5, IP Phones.
Other protocols such as H.323 and MGCP are discussed in Chapter 6, Voice Gateways. Appendix A, Cisco IP Telephony Protocol and Codec Information and
References, lists the protocols in an IP Telephony environment and the standards
body or specication governing the protocol.

When you rst open a CCM trace le, you might feel intimidated by the large amount of
information presented in the trace le. We recommend that you use the default trace settings
for CCM traces except you set the trace level to either Arbitrary or Detailed. Click the
SetDefault button on the Trace Conguration page for CCM traces in CallManager
Serviceability (Trace > Conguration) and then change the Debug Trace Level setting to
Detailed.

A Sample CCM Trace for a Call Between Two IP Phones

As a rst example at looking at CCM traces, go through a simple call between two IP
phones. IP phones use the Skinny protocol to communicate with CallManager. All
messages to and from a Skinny device are preceded by either the words StationInit or
StationD.
For example, assume you have two phones, Phone A and Phone B. Phone A calls Phone B,
Phone A goes off-hook and you see the following line in the CCM trace:
StationInit: 1a3e8b54 OffHook.

StationInit means that an inbound Transmission Control Protocol (TCP) message from a
Skinny station reached CallManager. A Skinny station is any endpoint that uses the Skinny
protocol to communicate with CallManager. This includes the Cisco 79xx family of

52

Chapter 3: Understanding the Troubleshooting Tools

IP phones. In other words, any message that starts with StationInit is a message from an
IP phone.
1a3e8b54 is probably the most important piece of this trace example. It is called a TCP
handle and it represents a unique value that identies a specic IP phone registered to this
CallManager server. With the TCP handle, you can follow every message to and from that
IP phone and see the full series of messages exchanged between CallManager and the
phone. When searching through a CCM trace le in Notepad, copy the TCP handle to the
clipboard (Ctrl+C), then open the Find box in Notepad (Ctrl+F), and paste the TCP handle
(Ctrl+V). Once you get into the habit of highlighting the TCP handle and then pressing
Ctrl+C, Ctrl+F, and Ctrl+V to enter the TCP handle into the Find window, you will be able
to search for Skinny messages related to a device very quickly.
If you want to nd the TCP handle for a particular IP phone, obtain the MAC address of the
phone from either CallManager Administration or from the phone itself and search for the
MAC address in the CCM trace until you nd a KeepAlive to that phone. For example,
StationInit - InboundStim - KeepAliveMessage
Send KeepAlive to Device Controller. DeviceName=SEP003094C2D11F,
TCPHandle=1a3e8b54, IPAddr=10.80.1.147, Port=51763,
Device Controller=[2,89,2992]

Notice the KeepAlive message contains both the TCP handle and the device name for the
IP phone. Once you nd the KeepAlive message, copy the TCPHandle eld and use that to
search through the CCM trace.
The OffHook message means that CallManager received a Skinny message indicating the
phone went off-hook.
The next message is
StationD:

1a3e8b54 DisplayText text=

3000

Notice that instead of StationInit you see StationD. This signies that CallManager is
sending a Skinny message to the phone. StationInit messages are sent from the IP phone to
CallManager, while StationD messages are sent from CallManager to the IP phone. Skinny
message transmission such as this between the IP phone and CallManager occurs for every
action undertaken by the IP phone, including initialization, registration, on-hook, off-hook,
dialing of digits, key presses on the phone, and so much more.
Again you see the same TCP handle, 1a3e8b54, listed for this message.
The number 3000 represents the directory number of the phone. If you know the phone
number of the calling IP phone, you can often nd the beginning of a call by simply
searching for the calling phones directory number. In CallManager 3.3 and later the
DisplayText message actually shows
StationOutputDisplayText dont need to send, because mIsALegacyDevice = 0

The reason for this message is that the 79xx series Cisco IP Phones never paid attention to
the DisplayText message in the rst place, so in CallManager 3.3 and beyond, the message

Reading CCM (or SDI) Traces

53

is no longer sent. This means that if you search for the directory number of the IP phone,
you might not nd exactly the beginning of the sequence of events. It is best to search for
the device name you are looking for and nd a KeepAlive to get the TCP handle as
discussed earlier.
Other messages sent to the IP phone include the following:
StationD:
1a3e8b54 StartTone tone=33(InsideDialTone), direction=0
StationD:
1a3e8b54 SetLamp stimulus=9(Line) stimulusInstance=1
lampMode=2(LampOn).
StationD:
1a3e8b54 CallState callState=1 lineInstance=1 callReference=16777217
StationD:
1a3e8b54 DisplayPromptStatus timeOutValue=0
promptStatus=Enter number lineInstance=1 callReference=16777217.
StationD:
1a3e8b54 SelectSoftKeys instance=1 reference=16777217
softKeySetIndex=4 validKeyMask=-1.
StationD:
1a3e8b54 ActivateCallPlane lineInstance=1.

Again you see that all the trace lines begin with StationD indicating that these are messages
from CallManager to the IP phone and you see each line has the same TCP handle.
Do not concern yourself at this point about exactly what each of the pieces in the trace
mean. These are all Skinny messages sent to the IP phone. At this point, you should just
familiarize yourself with the basic call ow to understand how to read the trace les, not
necessarily what each piece of the trace le means. Chapters 4 and 5 provide additional
detail relating to the Skinny messaging you see in the preceding output. In particular, see
Table 5-1, Skinny Message Denitions, and the section, Examining Skinny Protocol
Messages in a CCM Trace in Chapter 5 for detailed explanations.
You will notice, however, that many of the trace messages are relatively self-explanatory.
For example, the StartTone message with tone=33(InsideDialTone) tells the IP phone to
start playing dial tone.
Note the callReference ID. A callReference ID is created for each participant in a call and
you can use this ID to track a particular call through a CCM trace. A new callReference ID
is created for each participant in a call and when some features are invoked, such as transfer
and conference. Each leg of a call gets its own callReference ID assigned, so in a call
between two IP phones, each phone gets assigned a separate callReferenceID.
So far you have only seen Skinny protocol messages; however, this callReference ID can
help you correlate the Skinny messages with other messages to devices involved in the same
call.
Next, the user on the IP phone begins dialing digits. This time, notice the difference
between StationD and StationInit messages, indicating communication back and forth
between CallManager and the IP phone.
StationInit: 1a3e8b54 KeypadButton kpButton=3
StationD:
1a3e8b54 StopTone
StationD:
1a3e8b54 SelectSoftKeys instance=1 reference=16777217
softKeySetIndex=6 validKeyMask=-1
StationInit: 1a3e8b54 KeypadButton kpButton=0
StationInit: 1a3e8b54 KeypadButton kpButton=0
StationInit: 1a3e8b54 KeypadButton kpButton=1
Digit analysis: match(fqcn=3000, cn=3000, pss=IPMA:PA:Line1, dd=3001)

54

Chapter 3: Understanding the Troubleshooting Tools

Notice that a 3 is dialed and a tone is then stopped. The Skinny protocol does not provide
a mechanism to specify which tone to stop, so it sends a generic StopTone message. This
stops any tones the IP phone happened to be playing at the time. In this case, remember you
saw CallManager instruct the IP phone to play inside dial tone in the previous trace section.
The kpButton= message is always followed by the dialed digit. As soon as the rst digit is
dialed, the phone is told to stop playing dial tone. That makes sense because when you pick
up the phone, you hear dial tone, but as soon as you dial a digit, the tone stops. Notice that
after the phone is told to stop the tone, the soft keys are updated on the display. By looking
at all the kpButton= messages, you can see that 3001 is the number dialed.
Note that the digit * is shown in a trace as the letter e and a # is shown as f. So for example,
the message kpButton=e means the user entered the * key.
CallManager is constantly analyzing the digits the user dials, and once it nds an exact
match, digit analysis returns the results for the match. Now it is time to ring Phone B, which
is congured with DN 3001. Chapter 9, Call Routing provides additional detail about
how digit analysis works in CallManager. There is, however, one important concept you
should understand about digit analysis: Whenever digit analysis makes a match for a call,
it displays the digit analysis results in the CCM trace. For example,
Digit analysis: analysis results
| PretransformCallingPartyNumber=3000
| CallingPartyNumber=3000
| DialingPartition=Line1
| DialingPattern=3001
| DialingRoutePatternRegularExpression=(3001)
| DialingWhere=
| PatternType=Enterprise
| PotentialMatches=NoPotentialMatchesExist
| DialingSdlProcessId=(2,34,3500)
| PretransformDigitString=3001
| PretransformTagsList=SUBSCRIBER
| PretransformPositionalMatchList=3001
| CollectedDigits=3001
| UnconsumedDigits=
| TagsList=SUBSCRIBER
| PositionalMatchList=3001
| VoiceMailbox=
| VoiceMailCallingSearchSpace=IPMA:PA:Line1
| VoiceMailPilotNumber=5678
| DisplayName=James
| RouteBlockFlag=RouteThisPattern
| InterceptPartition=
| InterceptPattern=
| InterceptWhere=
| InterceptSdlProcessId=(0,0,0)
| InterceptSsType=0
| InterceptSsKey=0
| WithTags=
| WithValues=
| CgpnPresentation=NotSelected
| CallManagerDeviceType=UserDevice

Do not be concerned about what each of the elds means at this point. Chapter 9 explains
some of the concepts such as partitions and calling search spaces. The important concept to
grasp here is that any time digit analysis makes a match, you see a digit analysis result

Reading CCM (or SDI) Traces

55

similar to this one in the CCM trace. These digit analysis results are easy to spot in a CCM
trace because of the white space to the right of the digit analysis results. If you look at a
CCM trace, you will see that the majority of trace lines are over 100 characters in length;
however, the digit analysis results are usually no more than 20 to 30 characters long,
making the digit analysis results easy to nd while scrolling quickly through a CCM trace
le.
Because CallManager has collected all the required digits, it is ready to notify the destination IP phone there is an incoming call. The next example shows CallManager sending
Skinny messages to Phone B.
StationD:
StationD:
StationD:

1a3e8af0 DisplayText text=

3001

1a3e8af0 CallState callState=4 lineInstance=1 callReference=16777218

1a3e8af0 CallInfo callingPartyName=James callingParty=3000
cgpnVoiceMailbox=
calledPartyName=Mary calledParty=3001 cdpnVoiceMailbox=
originalCalledPartyName=Mary originalCalledParty=3001
originalCdpnVoiceMailbox= originalCdpnRedirectReason=0
lastRedirectingPartyName=Mary lastRedirectingParty=3001
lastRedirectingVoiceMailbox=
lastRedirectingReason=0
callType=1(InBound) lineInstance=1
callReference=16777218
StationD:
1a3e8af0 SetLamp stimulus=9(Line) stimulusInstance=1
lampMode=5(LampBlink)
StationD:
1a3e8af0 SetRinger ringMode=2(InsideRing)
StationD:
1a3e8af0 DisplayNotify timeOutValue=10 notify=From 3000
StationD:
1a3e8af0 DisplayPromptStatus timeOutValue=0 promptStatus=From 3000
lineInstance=1 callReference=16777218
StationD:
1a3e8af0 SelectSoftKeys instance=1 reference=16777218
softKeySetIndex=3 validKeyMask=-1

Notice rst that a StationD message is generated. This means that a message is sent from
CallManager to the IP phone. Also notice that the TCP handle is different than in the preceding trace output. Each IP phone has a unique TCP handle assigned to it at registration.
You can use the unique TCP handle to differentiate between the Skinny messages sent to
and from Phone A (TCP handle 1a3e8b54) and those sent to and from Phone B (TCP handle
1a3e8af0). A new TCP handle is assigned any time an IP phone such as Phone A unregisters
and reregisters to CallManager, resets, or fails over or back from one CallManager to
another.
Notice also that the callReference value (16777218) is different from the previous output.
As we mentioned earlier, this is because each leg of a call is assigned a different call reference. In this case, this is the call reference for Phone B. This call reference persists for
the duration of the call on Phone B.
Phone B rings and the call information shows that James called Mary. You see several
messages that seem to suggest the call is being redirected; its not. These are just standard
messages sent by CallManager.
Once again, do not be concerned about exactly what each message means at this point.
Future chapters go into detail about each message; however, you can see from reading the
trace that Phone B is being told to ring (SetRinger ringMode=2(InsideRing)) and display
From 3000 on the prompt line of the IP phone (promptStatus=From 3000).

56

Chapter 3: Understanding the Troubleshooting Tools

Now that a call is in progress, Phone A gets some updated information, including display
information such as called and calling party names.
StationD:
1a3e8b54 SelectSoftKeys instance=1 reference=16777217
softKeySetIndex=8 validKeyMask=-1.
StationD:
1a3e8b54 CallState callState=12 lineInstance=1 callReference=16777217
StationD:
1a3e8b54 CallInfo callingPartyName=James callingParty=3000
cgpnVoiceMailbox= calledPartyName=Mary calledParty=3001
cdpnVoiceMailbox= originalCalledPartyName= originalCalledParty=
originalCdpnVoiceMailbox= originalCdpnRedirectReason=0
lastRedirectingPartyName= lastRedirectingParty=
lastRedirectingVoiceMailbox= lastRedirectingReason=0
StationD:
1a3e8b54 StartTone tone=36(AlertingTone).
StationD:
1a3e8b54 CallState callState=3 lineInstance=1 callReference=16777217
StationD:
1a3e8b54 SelectSoftKeys instance=1 reference=16777217
softKeySetIndex=8 validKeyMask=-1.
StationD:
1a3e8b54 DisplayPromptStatus timeOutValue=0 promptStatus=Ring Out
lineInstance=1 callReference=16777217.

Basically, these messages perform two actions. First, the display on Phone A changes now
that the call is in progress. Second, the phone is told to play an alerting tone. The alerting
tone is the standard ringback tone you hear when placing a call. You also see the rst
callReference value, 16777217, indicating the original call placed by Phone A. Remember each call reference is only valid for one leg of the call.
StationInit: 1a3e8af0
StationD:
1a3e8af0
StationD:
1a3e8af0
StationD:
1a3e8af0
lampMode=2(LampOn)
StationD:
1a3e8af0
StationD:
1a3e8af0

OffHook
ClearNotify
SetRinger ringMode=1(RingOff)
SetLamp stimulus=9(Line) stimulusInstance=1
CallState callState=1 lineInstance=1 callReference=16777218
ActivateCallPlane lineInstance=1

The StationInit OffHook message indicates that Phone B goes off-hook and answers the
call. CallManager sends a SetRinger ringMode=1(RingOff) message, which tells Phone
B to stop ringing, and the preparation is now complete for the actual media connection.
Next, well examine how the audio stream is set up. As with all VoIP protocols, Skinny uses
Real-Time Transport Protocol (RTP) streams over User Datagram Protocol (UDP) packets
to send and receive Voice over IP (VoIP) samples. Each RTP stream is called a logical channel. A logical channel is a unidirectional RTP stream, so to have a two-way conversation,
you must have two logical channels openedone from the calling device to the called
device and one from the called device to the calling device.
In a call involving a Skinny device, CallManager asks the IP phone to open a connection to
receive RTP streams. CallManager asks the IP phone for specic parameters for this connection, including the codec and packet size. You can see the following
OpenLogicalChannel messages from CallManager to each IP phone requesting that they
open a connection to receive RTP packets using G.711.
StationD:
1a3e8b54 OpenReceiveChannel conferenceID=0
passThruPartyID=17 millisecondPacketSize=20
compressionType=4(Media_Payload_G711Ulaw64k)
qualifierIn=?.
myIP: 3eee1cac (172.28.238.62)
StationD:
1a3e8af0 OpenReceiveChannel conferenceID=0
passThruPartyID=33 millisecondPacketSize=20
compressionType=4(Media_Payload_G711Ulaw64k)
qualifierIn=?.
myIP: 2fee1cac (172.28.238.47)

Reading CCM (or SDI) Traces

57

Upon receiving an OpenReceiveChannel message, the IP phone selects the UDP port
number it wants to use to receive RTP packets and reports this information back to
CallManager in an OpenReceiveChannelAck message. Phone A responds rst:
StationInit: 1a3e8b54 OpenReceiveChannelAck
Status=0, IpAddr=0x3eee1cac, Port=20096, PartyID=17

Once CallManager receives this information from Phone A, it can tell Phone B where to
send its RTP stream. Until this point, CallManager could not tell Phone B which UDP port
number to use because Phone A had not reported it to CallManager. Once CallManager
receives the port number in the OpenReceiveChannelAck message from Phone A, it sends
a StartMediaTransmission message to Phone B giving it the IP address and port number
of Phone A along with information about which voice codec to use.
StationD:
1a3e8af0 StartMediaTransmission
conferenceID=0 passThruPartyID=33
remoteIpAddress=3eee1cac(172.28.238.62)
remotePortNumber=20096 milliSecondPacketSize=20
compressType=4(Media_Payload_G711Ulaw64k)
qualifierOut=?. myIP: 2fee1cac (172.28.238.62)

Next, CallManager receives an OpenReceiveChannelAck from Phone B containing the

UDP port number information and passes this on to Phone A in a
StartMediaTransmission message.
StationInit: 1a3e8af0 OpenReceiveChannelAck Status=0,
IpAddr=0x2fee1cac, Port=19648, PartyID=33
StationD:
1a3e8b54 StartMediaTransmission
conferenceID=0 passThruPartyID=17
remoteIpAddress=2fee1cac(172.28.238.47)
remotePortNumber=19648 milliSecondPacketSize=20
compressType=4(Media_Payload_G711Ulaw64k) qualifierOut=?.
myIP: 3eee1cac (172.28.238.47)

So at this point, Phone A (TCP handle 1a3e8b54) is sending RTP packets to 172.28.238.47,
which happens to be Phone B, and Phone B (TCP handle 1a3e8af0) is sending RTP packets
to 172.28.238.62, which happens to be Phone A.
Notice that for the duration of this call, Phone A has never sent nor received any Skinny
signaling to or from Phone B. This is because all the signaling goes through CallManager.
The only time IP phones send packets to each other is for the actual voice stream. This is
what allows CallManager to set up calls between devices that use different signaling protocols. For example, if Phone A called a phone number on the PSTN instead of another IP
phone, the signaling between CallManager and Phone A remains the same. Phone A has no
idea that it is sending RTP packets to a voice gateway and vice versa.
When reading CCM traces, you can usually separate each leg of the call and concentrate on
one part at a time. For example, if you have a call that goes out through a voice gateway,
once you have veried that the IP phone dialed the correct digits and CallManager is
routing the call to a voice gateway, you can focus on the gateway debugs to determine how
the call gets set up.

58

Chapter 3: Understanding the Troubleshooting Tools

Also it is important to separate the signaling aspects of a call set up from the RTP media
streams. All VoIP devices are blindly told to send RTP packets to an IP address and port
number without knowing what type of device they are sending these packets to. As long
as the terminating device provided the correct IP address and port number and CallManager
relayed this information correctly, everything works properly. However, pay attention to the
signaling aspects to ensure that the port number received from the terminating device is the
same as the port number reported to the originating device.

Tracing a Call Through an MGCP T1 PRI Gateway

The next call to dissect is an IP phone (3000) making a call through a WS-X6608-T1 PRI
MGCP gateway. MGCP is not described in detail in this example; however, you will see
ISDN Q.931 messages because this is the protocol used over the PRI D-channel. A detailed
discussion on troubleshooting ISDN PRI problems can be found in Chapter 6. Appendix A
includes the standards body and specication for ISDN Q.931.
All Q.931 and H.323 (including intercluster trunk) calls look basically the same in a CCM
trace. Also, some non-ISDN gateways such as the WS-X6624 and the WS-X6608 when
using T1 CAS also use Q.931 messages to communicate with CallManager. This makes
understanding the basic structure of this kind of trace message important.
The majority of information presented in the CCM trace le for a gateway call is in hexadecimal notation. The Q.931 specication states that phone numbers should be encoded in
ASCII as well as character strings, such as display names. The CCM trace also uses hexadecimal equivalents for the IP addresses on some occasions. Do not be intimidated by the
hexadecimal values. Once you understand how to decode them, they are actually easy to
understand. Appendix C provides a cheat sheet for conversions.
Whenever an H.323 or Q.931 call is made, you will see a section of trace similar to the
following outgoing ISDN setup message:
Out Message -- PriSetupMsg -- Protocol= PriNi2Protocol.
Ie - Ni2BearerCapabilityIe IEData= 04 03 80 90 A2
Ie - Q931ChannelIdIe IEData= 18 03 A9 83 97
Ie - Q931CallingPartyIe IEData= 6C 06 00 80 33 30 30 30
Ie - Q931CalledPartyIe IEData= 70 05 80 33 30 30 31
MMan_Id= 0. (iep= 0 dsl= 0 sapi= 0 ces= 0 IpAddr=a4e81cac IpPort=2427)
IsdnMsgData2= 08 02 00 09 05 04 03 80 90 A2 18 03 A9 83 97 6C 06 00 80 33 30 30
30 70 05 80 33 30 30 31

The rst line gives you information about the direction of the call (either Out Message or
In Message) followed by the type of message (PriSetupMsg) and the protocol used for
the message (PriNi2Protocol). The direction is from the perspective of CallManager, so
this means a device registered to CallManager is placing a call out through a gateway.
As with the IP phone, a unique identier is used to keep track of the call. Use the hexadecimal IP address IpAddr=a4e81cac in conjunction with call reference number 02 00 09
to track the call throughout the trace. When viewing an ISDN trace, look at the
IsdnMsgData2 line to nd the call reference ID. In some cases you will see IsdnMsgData

Reading CCM (or SDI) Traces

59

or IsdnMsgData1 instead of IsDNMsgData2. They are equivalent. Ignore the rst two
numbers (usually 08). The next two numbers are the call reference length (02), and the next
four are the call reference value (00 09).
You might be wondering how a4e81cac is converted to an IP address in dotted decimal
notation. This is a hexadecimal representation of the IP address. You can gure out the
IP address by working backwards. In this example, rst, take the last two digits, ac, which
is hex for 172. Next, consider the 1c, which is hex for 28. Third, take e8, which is hex for
232. Finally, take a4, which is hex for 164. The IP address is 172.28.232.164.

NOTE

Appendix C provides a quick cheat sheet to determine how to quickly convert between
decimal, hexadecimal, and binary values.

So far you know you are looking at an outbound setup message on a PRI congured for the
NI2 ISDN protocol, all of which you determined from the rst line of the trace. You also
know the call reference from decoding the rst few bytes of the IsdnMessageData2. The
lines in the middle that begin with Ie are ISDN Q.931 information elements. Information
elements are covered in detail in Chapter 6; however, they are all formatted the same way
in the CCM trace. For example, the following is the called party information element (IE):
Ie - Q931CalledPartyIe IEData= 70 05 80 33 30 30 31

Each information element line begins with the letters Ie followed by the name of the information element, in this case Q931CalledPartyIe. After the name follows the data
contained in that information element. The format of the data that follows is dependent on
the particular information element. The Q.931 and H.225 specications describe the format
of each information element. Having a copy of the ITU-T Q.931 specication can prove to
be invaluable when troubleshooting ISDN problems because you can get down to the exact
details of each bit in the IE data.
Fortunately for most day-to-day activities you do not need to reference the Q.931 specication because the information you need is easily identiable. For example, in the called
party number information element shown above, notice the IE data contains the sequence
33 30 30 31, which is 3001 in ASCII. This represents the directory number that is being
called. Now you can follow the calls events.
Just as you searched through the CCM trace for the TCP handle of the IP phone to nd the
next message associated with a particular phone, you can do the same for a Q.931 or H.323
call. The call reference for a call remains the same for the duration of a call. The rst bit,
however, of the call reference is ipped depending on the direction of the call. Chapter 6
goes into detail about how this works, so dont worry about it right now. All you need to
know is to search for the call reference minus the rst digit. In this case the call reference
is 00 09, so search for 0 09.

60

Chapter 3: Understanding the Troubleshooting Tools

If you search through the CCM trace, you come to the next message. Notice that the
message that follows is an In Message. This means it is an inbound message sent to
CallManager by the ISDN network:
In Message -- PriCallProceedingMsg -- Protocol= PriNi2Protocol
Ie - Q931ChannelIdIe -- IEData= 18 03 A9 83 97
MMan_Id= 0. (iep= 0 dsl= 0 sapi= 0 ces= 0 IpAddr=a4e81cac IpPort=2427
IsdnMsgData1= 08 02 80 09 02 18 03 A9 83 97

Notice that the call reference is now 02 80 09. The most-signicant bit (MSB) is set on the
call reference value. This bit determines if this message is the originating or terminating
side; however, you do not need to know what this bit means because CallManager clearly
tells you which direction the message is going when it says In Message or Out Messsage.
When searching, you need to look for 02 00 09 or 02 80 09 to track all events relating to
this call event, although searching for just 0 09 is usually good enough.
These are just some of the tricks that help you follow call ows through the CCM trace.
Subsequent chapters provide additional trace examples as we investigate other troubleshooting scenarios. Half the battle in reading a CCM trace is knowing which pieces of the
trace le to ignore so that you can focus on the important messages in the trace. As you read
through the following chapters, you will get a better understanding of the different
messages you might nd in a CCM trace.

Reading SDL Traces

An SDL trace is a very detailed trace mainly used by Cisco development engineers for
code-level analysis of call processing events. To the average CallManager administrator,
SDL trace les are far too detailed for normal practical use.
If youre working with the Cisco Technical Assistance Center (TAC) on problem resolution,
TAC might ask you to collect SDL traces so that it can forward them to CallManager developers for analysis. You need to know how to congure the trace les to capture the right
information for TAC.
Although the SDL trace les are generally not used for troubleshooting purposes, you will
nd a few occasions in this book where you must look in the SDL trace le to get a full
understanding of a particular problem. In cases like this, we will provide details on exactly
what to look for.

SDL Overview
SDL traces provide a C programming language interface to alarms and trace information in
CallManager. Alarms are used to inform a TAC engineer or CallManager developer of
unexpected events, such as being unable to access a le, database, Winsock, or other
operating system resources.
SDL traces can span multiple servers, allowing a process on one server to communicate
with a process on another server transparently. This mechanism is supported by the use of

Reading SDL Traces

61

SDL links. An SDL link spans from one server supporting SDL to another server supporting SDL.
SDL maintains a circular queue of les to log information. Over time, a le will be
overwritten. The number of les (determined by the CallManager service parameter
SdlTraceTotalNumFiles) and the number of lines per le (determined by the CallManager
service parameter SdlTraceMaxLines) governs how long it takes to overwrite old log les.
SDL generates two types of les:

Log lesContain the actual tracing information.

Index lesIndicate which log le in the circular queue is currently being used for
writing. An index le allows SDL logging to start where it left off each time an
application is restarted.

Log lenames are composed of the following:

SDLnnn_qqq_xxxxxx.txt
where
nnn represents the node ID
qqq represents the application ID (CallManager = 100)
xxxxxx represents the unique le index
Index lenames are composed of the following:
SDLnnn_qqq.index
where
nnn represents the node ID
qqq represents the application ID (CallManager = 100)
The actual SDL log and index les are text-based. All columns in a log entry are delimited
by a vertical bar (|) character (0x7c). A log entry is broken into two components, prex and
detail, which appear in the following format:
| Prefix Component | Detail Component |

Every log entry contains a prex. The prex always has the same format. The common
prex of a trace line is as follows:
uuuuuuuuu | yy/mm/dd-hh:mm:ss:vvv | nnn | xxxx |

where
uuuuuuuuu represents a unique line sequence timestamp
yy represents the year
mm represents the month
dd represents the day
hh represents the hour
mm represents the minutes
ss represents the seconds

62

Chapter 3: Understanding the Troubleshooting Tools

vvv represents the milliseconds

nnn represents the node ID
xxxx represents the log entry type (which denes the type of entry being logged)
The SDL process running on each CallManager allows each node in the cluster to
communicate with other nodes in the cluster to exchange information. SDL is what allows
CallManager clustering to work by allowing processing for tasks to run on any node in the
cluster. Generally, processing for an event occurs on the node where the device performing
the given action is registered. For example, if an IP phone is registered to a particular
CallManager in a cluster, that CallManager usually handles all call processing operations
for that phone. But what if an IP phone registered to one CallManager in the cluster places
a call out a gateway registered to a different CallManager in the cluster? In this case, some
of the processing might occur on one node while the rest of the processing occurs on
another node.
CallManager functions by sending signals from one process to another. These signals are
just messages from one piece of software to another internal to CallManager. These signals
can be sent to a piece of code on an entirely different CallManager node in the cluster. The
SDL trace lets you see this happening. The best way to see this is by looking at the log entry
type eld at the beginning of an SDL trace le. For example,
011381785 | 02/09/16 14:51:37.874 | 002 | SdlSig-O

The log entry type is SdlSig-O. This means this CallManager server sent an SDL signal to
another node (O means outgoing). Conversely, if you see SdlSig-I, this tells you the
CallManager node you are looking at received an SDL signal from another node or another
process. The CallManager and CTI Manager process communicate with each other using
SDL links as well, so you see communication to and from CTI Manager in the SDL trace
as SdlSig-O and SdlSig-I.
So how do you know which node the signal was coming from or destined to? After the
SdlSig-O or SdlSig-I message you should see several columns of text separated by the
vertical bar (|) character. For example,
SdlSig-O

| DbDeviceClose

| initialized

| Db(1,100,20,1)

| Db(2,100,20,1)

An actual SDL trace has many more columns and spaces between columns than what is
shown here. We have condensed the trace to t on the printed page. The two columns to
examine are the third and fourth columns after SdlSig-O. These describe the process
sending the signal and the process to which the signal is being sent. The third column shows
the destination (Db(1,100,20,1)) and the fourth shows the source (Db(2,100,20,1)).
Now you need to understand what you are looking at when you see Db(2,100,20,1). Db is
the process name. You do not need to understand what the various process names signify.
The important part is the four comma-separated numbers that follow the process name. The
four numbers represent (in this order)

Reading SDL Traces

63

Node number
Application number (100 = CallManager, 200 = CTI Manager)
Process type
Process instance

The only ones you should be concerned with are the node number and the application
number. In this case, you can see the signal is from CallManager on node 2 to CallManager
on node 1.
You might be wondering why you need to know all this. Because CallManager is a distributed architecture, you might nd that when looking at a CCM trace the events occurring
dont seem to make sense. For example, you might see an IP phone making a call but never
see the call go out a gateway to its destination, or you might see an IP phone told to play
reorder for no apparent reason. When you see an unexplained event in the CCM trace, look
at the same timestamp in the SDL trace to see if there was a signal from another node at
that time. If so, look in the CCM trace on the other node at the same time to see if there is
any additional detail about the call on the other node. This is the reason why having the
clocks synchronized on all servers is so vitally important. Without time synchronization,
you would have a very difcult task trying to match the events shown in the CCM trace on
one server with the events on another server.
Although you will never read through an SDL trace the way you read through a CCM trace,
you might occasionally have to look at the SDL trace to see what triggered a particular
event in the CCM trace. In some cases, you will take the timestamp for an event in the CCM
trace and match up that same timestamp in the SDL trace.

Enabling SDL Trace and Setting the Appropriate SDL Trace Level
Now that you know what the SDL trace is for, you need to know how to turn on SDL tracing
and set the appropriate bit mask for the data you need. A bit mask is a string of bits that each
represent a particular trace setting. For example, 10010011 is a bit mask. Each 1 in the bit
mask indicates that a particular trace should be enabled, while each 0 indicates that a
particular trace should be disabled. Bit masks are usually represented as strings of hex
digits. For example, 10010011 is 0x93.
Detailed SDL tracing consumes a lot of disk space and affects the processor on the
CallManager server, which can result in performance degradation under very high call
volumes. However, as of CallManager 3.3, SDL trace writing is performed asynchronously
which means CallManager is allowed rst access to the disk and CPU for call processing.
If insufcient disk or CPU bandwidth exists to write the trace le, lines are skipped in the
trace, but call processing is not affected.

64

Chapter 3: Understanding the Troubleshooting Tools

SDL traces are enabled in the Service > Service Parameter area in CallManager
Administration. Table 3-3 shows the parameters you can adjust.
Table 3-3

SDL Service Parameters

SDL Service Parameter

Description

SdlListeningPort

This is the TCP port with which SDL links can be established between nodes in a cluster. The port is 8002 by
default. There is rarely any reason to change this value.

SdlMaxRouterLatencySecs

Indicates the maximum number of seconds of signal latency

before forcing a restart of the CallManager service. The
default is 20.

SdlMaxUnHandledExceptions

Species the maximum number of CallManager exceptions

that can occur before CallManager stops running. The
default is 5.

SdlTraceDataFlags

This is a bit mask used to enable the tracing of SDL nonapplication-specic components or to modify the behavior of
SDL tracing.
The recommended value for normal system debugging
is 0x110.
The recommended value when tracking problems
with SDL links is 0x13D.
See Table 3-4 for details.

SdlTraceFlushImmed

Determines whether SDL trace entries are to be ushed to

disk immediately. If this parameter is set to False, SDL trace
entries are ushed to disk when there is spare disk bandwidth
not being used for call processing. Setting this parameter to
True causes higher disk input/output but ensures that all
entries are written to the disk in the unlikely event of a
software error. You should set this to False during normal
operating conditions. The default is True.

SdlXmlTraceFlag

Determines whether XML-formatted tracing is allowed for

SDL traces. The default is False.

SdlTraceDataSize

For signal types, this constrains the number of bytes that can
be dumped from the data portion of a signal. This
information appears in the freeform information column at
the end of each line in the SDL trace le. The default is 100
bytes.

SdlTraceFilePath

This is the directory path that SDL uses to generate the log
les. If this path is not dened or is dened incorrectly, SDL
uses the default root path: C:\ProgramFiles\Cisco\Trace\SDL\.

SdlTraceFlag

This is a Boolean ag that indicates if SDL tracing is enabled

or disabled. Set this ag to True to turn tracing on or False to
turn tracing off. The default is True.

Reading SDL Traces

Table 3-3

65

SDL Service Parameters (Continued)

SDL Service Parameter

Description

SdlTraceMaxLines

This value indicates the maximum number of lines written to

a log le before a new log le is created. The default is
10,000 lines.

SdlTraceTotalNumFiles

This value indicates the maximum number of les that can

be created for logging purposes. The default is 250 les.
Normally, a maximum of 250 les is not enough because the
les write in round-robin or circular fashionwhen the
maximum limit is met, the next le starts at the beginning
and overwrites the rst le.
Determine how much free disk space you have, deduct a
safety net of 500 MB, and then use the rest of your disk
space for trace space. You can calculate the size of your trace
les by guring the average CCM trace at 10,000 lines
consumes about 2.5 MB; the average SDL trace at 10,000
lines consumes about 3.5 MB. Be sure to monitor free disk
space by using other tools mentioned in this chapter such as
PerfMon or CCEmail.

SdlTraceTypeFlags

This eld indicates the bit mask value for collecting the trace
type ag of choice.
The recommended value for normal call debugging is
SdlTraceTypeFlags=0x00000B04.
The recommended value for low-level debugging or the
debugging of voice gateways is 0xA000EB15.
See Table 3-5 for more details.

The bit mask denitions shown in Table 3-4 correlate to the Trace Characteristics on the
SDL Trace Conguration page in CallManager Serviceability (Trace > Conguration >
select a server > Cisco CallManager > click the link to SDL Conguration).
Table 3-4

Non-application-specic Bits

Name

Trace Characteristic
in CallManager
Serviceability

Bit Mask

Description

traceSdlLinkState

Enable SDL Link

States Trace

0x00000001

Enables the tracing of SDL

link states.

traceSdlLowLevel

Enable Low-level SDL

Trace

0x00000002

Enables low-level SDL

tracing.

traceSdlLinkPoll

Enable SDL Link Poll

Trace

0x00000004

Enables the tracing of SDL

link poll.
continues

66

Chapter 3: Understanding the Troubleshooting Tools

Table 3-4

Non-application-specic Bits (Continued)

Name

Trace Characteristic
in CallManager
Serviceability

Bit Mask

Description

traceSdlLinkMsg

Enable SDL Link

Messages Trace

0x00000008

Enables the tracing of SDL

Link messages.

traceRawData

Enable Signal Data

Dump Trace

0x00000010

Enables signal data dump.

traceSdlTagMap

Enable Correlation Tag

Mapping Trace

0x00000020

Enables the tracing of

correlation tag mapping.

traceCreate

Enable SDL Process

States Trace

0x00000100

Enables the tracing of SDL

process states.

traceNoPretyPrint

Disable Pretty Print of

SDL Trace

0x00000200

Enables no pretty print of the

SDL trace. Pretty print adds
tabs and spaces in a trace le
without performing post
processing.

traceSdlEvent

Enable SDL TCP Event

Trace

0x80000000

Enables TCP event traces.

The bit mask denitions shown in Table 3-5 correlate to the Trace Filter Settings on the
SDL Trace Conguration page in CallManager Serviceability (Trace > Conguration >
select a server > Cisco CallManager > click the link to SDL Conguration).
Table 3-5

Bit Mask Denitions

Name

Trace Filter
Setting in
CallManager
Serviceability

Bit Mask

Description

traceLayer1

Enable All Layer 1

Trace

0x00000001

Enables all Layer 1 traces.

traceDetailLayer1

Enable Detailed
Layer 1 Trace

0x00000002

Enables detailed Layer 1 trace.

Not used

0x00000008

This bit is not used.

traceLayer2

Enable All Layer 2

Trace

0x00000010

Enables all Layer 2 traces.

traceLayer2Interface

Enable Layer 2
Interface Trace

0x00000020

Enables Layer 2 interface trace.

traceLayer2TCP

Enable All Layer 2

TCP Trace

0x00000040

Enables Layer 2 TCP trace.

Reading SDL Traces

Table 3-5

Bit Mask Denitions (Continued)

Name

Trace Filter
Setting in
CallManager
Serviceability

Bit Mask

Description

traceDetailLayer2

Enable Detailed
Dump Layer 2
Trace

0x00000080

Enables a detailed dump of

Layer 2 frames.

traceLayer3

Enable All Layer 3

Trace

0x00000100

Enables all Layer 3 traces.

traceCc

Enable All Call

Control Trace

0x00000200

Enables all Call Control traces.

traceMiscPolls

Enable
Miscellaneous
Polls Trace

0x00000400

Enables miscellaneous polls

traces.

traceMisc

Enable
Miscellaneous
Trace (Database
Signals)

0x00000800

Enables miscellaneous traces

(database signals).

traceMsgtrans

Enable Message
Translation Signals
Trace

0x00001000

Enables message translation

signals
TranslateIsdnToSdlReq,
TranslateIsdnToSdlRes,
TranslateSdlToIsdnReq,
TranslateSdlToIsdnRes traces.

traceUuie

Enable UUIE
Output Trace

0x00002000

Enables UUIE output traces.

traceGateway

Enable Gateway
Signals Trace

0x00004000

Enables gateway signals traces.

traceCti

Enable CTI Trace

0x00008000

Enables CTI signal traces.

traceNetworkSvc

Enable Network
Service Data Trace

0x10000000

Enables network service data

traces.

traceNetworkEvent

Enable Network
Service Event
Trace

0x20000000

Enables network service event

traces.

traceIccpAdmin

Enable ICCP
Admin Trace

0x40000000

Enables Intracluster Control

Protocol (ICCP) administration
traces.

traceDefault

Enable Default
Trace

0x80000000

Enables default traces.

67

68

Chapter 3: Understanding the Troubleshooting Tools

Microsoft Performance (PerfMon)

Microsoft Performance is an administrative tool provided by the Windows 2000 operating
system. It is colloquially referred to as PerfMon. PerfMon can be used to monitor a variety
of performance objects. A performance object is a set of counters reported by a process or
application running on the system that can be monitored. An example of a standard
performance object is the Processor object, which contains a variety of processor-related
counters such as processor utilization. PerfMon allows you to look at real-time statistics. It
contains logging facilities to take snapshots of any counter at user-dened intervals.
The StatisticsEnabled service parameter in CallManager Administration (Service >
Service Paramaters > select a server > Cisco CallManager) must be set to True to
generate data in the counters. If statistics are disabled, neither PerfMon nor the Real-Time
Monitoring Tool (RTMT) can collect data. Statistics are enabled by default.
CallManager includes several performance objects that let you monitor various counters
related to the operation of the CallManager services and associated devices. The RTMT,
which is discussed later in this chapter as part of the CallManager Serviceability section,
provides much the same functionality as PerfMon.

Comparing PerfMon and the Real-Time Monitoring Tool (RTMT)

Although PerfMon and the RTMT allow you to view the same performance objects, each
tool has its strengths and weaknesses.

PerfMon Advantages
You can congure PerfMon to log specic counters to a comma-separated values (CSV)
le. This CSV le can then be imported into a spreadsheet application for further analysis.
A third-party tool called CCEmail (discussed in the next section) allows you to congure
alerts in PerfMon.
Another advantage PerfMon has over RTMT is that the RTMT web browser must always
be running for the counters to be monitored and alerts to be sent. However, that will likely
be xed in releases of RTMT subsequent to release 3.3(3).

RTMT Advantages
You can run the RTMT from a web browser on any PC that has IP connectivity to
CallManager. PerfMon can be run only from a Windows NT/2000/XP PC that has PerfMon
installed.
You can congure specic counters and save the congurations in RTMT. Then, each time
you run RTMT, the pre-dened congurations are available.

Microsoft Performance (PerfMon)

69

In addition to performance-monitoring capabilities, RTMT provides two tabsDevices

and CTI Apps. The tabs allow you to search for devices and check the state of TAPI and
JTAPI applications that are connected to CallManager.
Youll read more about the additional capabilities of the RTMT later in this chapter.

Using PerfMon to View Real-time Statistics

PerfMons most basic function is to view real-time statistics on a machine. For example,
you might want to know how many calls are currently active on CallManager or the status
of the channels on a PRI. This kind of information can easily be obtained through PerfMon.
An example of how to view some real-time statistics with PerfMon will better familiarize
you with the tool. Launch PerfMon from a CallManager server by selecting Start >
Programs > Administrative Tools > Performance. The application looks similar to
Figure 3-5.
Figure 3-5

PerfMon

PerfMon has three formats to display data: chart, histogram, and report. Three buttons on
the toolbar shown in Figure 3-5 are used to switch between the different formats. For viewing real-time statistics, you usually want to use the report format. Click the View Report
button on the toolbar to gray out the area below.
Next, add the counters you want to monitor. Click the Add button on the toolbar, as shown
in Figure 3-5. You see a dialog box similar to Figure 3-6.

70

Chapter 3: Understanding the Troubleshooting Tools

Figure 3-6

PerfMon Add Counters Dialog Box

From this screen, you can add counters from either the machine on which you are running
PerfMon or any other machine that has IP connectivity to the machine on which you are
running PerfMon. For you to be able to view counters on remote machines, the account you
are logged in with must have administrator privileges on the machine you want to monitor.
Next, select the object that contains the counter(s) you want to monitor. For example, if you
want to monitor the number of active calls on a CallManager, select the Cisco CallManager
object.
Below the object selection you can choose which counter to monitor. If you select the
Cisco CallManager object, you see a counter labeled CallsActive. Select this counter and
click Add. The dialog box remains open so that you can continue selecting and adding
counters. To monitor all the counters for a particular object, click the All counters button.
You can click the Explain button to get a short description of the counter, as shown in
Figure 3-6. When you are done adding counters, click Close. Figure 3-7 shows all the
counters in the Cisco CallManager object. You can see that the counters are updated every
second or so.
Appendix D, Performance Objects and Counters, describes the meanings of each of these
counters, as well as the rest of the CallManager-related objects.
Objects and counters are only available for installed components. For example, if you do
not have Cisco CallManager Attendant Console installed on the server you are trying to
monitor, you will not see the Cisco CallManager Attendant Console object.

Microsoft Performance (PerfMon)

Figure 3-7

71

PerfMon Displaying the CallManager Performance Object

Some counters give you the option of selecting a specic instance of the counter to monitor.
For example, the Cisco Lines object has only one counter, named Active. However, for this
one counter, you can select one or more instances to monitor. In this case, each instance
corresponds to a particular line on CallManager. You can select a range by holding down
the Shift key and clicking the rst and last instances in the range. Or you can select several
individual instances by holding down the Ctrl key while clicking to select specic
instances.
See Appendix D for detailed descriptions of all the Cisco CallManager-related performance
objects and counters available in either PerfMon or the RTMT.

Using Counter Logs

One of PerfMons most powerful features is the ability to periodically log performance
information to a le. This can be useful for monitoring trends or determining exactly what
time a problem occurred. Probably the most common use of counter logs is to monitor
memory and CPU utilization for trends. If memory utilization continues to increase, you
might be running into a memory leak. If the CPU spikes during specic times, you might
be encountering one of many problems that cause high CPU utilization, such as a call
routing loop.

72

Chapter 3: Understanding the Troubleshooting Tools

Use the following steps to congure a counter log to monitor memory and CPU utilization
on a per-process basis:
Step 1 On a CallManager server, open PerfMon by selecting Start > Programs >

Administrative Tools > Performance.

Step 2 In the left column, in the Performance Logs and Alerts section, select

Counter Logs.
Step 3 Select Action > New Log Settings.
Step 4 In the New Log Settings dialog box, type a name for the log, such as CPU

and Memory Logging.

Step 5 A dialog box similar to the one shown in Figure 3-8 appears. Click Add.
Figure 3-8

Counter Log Conguration Dialog Box

Step 6 The Select Counters dialog box appears (previously shown in Figure 3-6).

The Processor object and % Processor Time counter are selected by default.
Click Add to add this counter to the log.
Step 7 In the Performance object eld, select the Memory object.
Step 8 In the list of counters, select the Available MBytes counter.
Step 9 Click Add and then Close to return to the counter log conguration

dialog box.

Microsoft Performance (PerfMon)

Step 10 In the Sample data every area, adjust the interval depending on how

often you want to monitor. For example, set it to 5 seconds to take a

snapshot of the selected counters every 5 seconds. Microsoft recommends this counter not be set lower than 2 seconds when logging many
different counters.
Step 11 Select the Log Files tab.
Step 12 In the Location eld, type the path where you want to save the log les

(the default is C:\PerfLogs).

Step 13 In the File Name eld, type a lename for this log le, such as

CPU_and_Memory_Logging.
Step 14 The End le names with checkbox and popup menu let you append a

number to the end of the lename based on either the date and time the
le is created or just an arbitrary number that increments each time a new
log le is created. Select the End le names with checkbox, and choose
a date format from the popup menu.
Step 15 Change the Log le type from Binary File to Text File CSV. With the

data in a CSV format. you can take the resulting data and import it into a
variety of applications. Figure 3-9 shows the Log Files tab as described
in the preceding steps.
Figure 3-9

Log Files Tab for CPU and Memory Logging in PerfMon

73

74

Chapter 3: Understanding the Troubleshooting Tools

Step 16 Click the Schedule tab. If this is the rst log on this system, youll be

asked if you want to create the directory, C:\PerfLogs. Click Yes.

Step 17 Instead of creating one huge log le, you can have PerfMon

automatically create a new le periodically. To do this, in the Stop log

area, click After and select how often you want to create a new le. For
example, to create a new le every day, enter 1 and set the Units to days.
Step 18 Check the Start a new log le checkbox so that PerfMon starts a new le

for the frequency you specied.

Step 19 Click OK.

If you followed the preceding instructions, you should now see a log with a green icon to
the left of it on the Counter Logs screen, as shown in Figure 3-10.
Figure 3-10 CPU and Memory Logging Counter Log in PerfMon

If the icon is red, right-click it and select Start. An error message should appear, telling you
PerfMon is unable to start the log. The error will likely refer you to the Windows event log
to get more details on the error. Correct the error, right-click the icon again, and select
Start.

CAUTION

When logging a small number of objects, as in the preceding example, setting the logging
interval to something small such as 1 or 2 seconds is not a problem. However, if you are
logging a large number of counters, be careful not to set the interval too low because this
can lead to performance degradation.

Microsoft Performance (PerfMon)

CAUTION

75

Unlike CCM trace les, PerfMon log les do not have a way to do circular wrapping (also
known as round-robin). This means that if left unattended, PerfMon log les can use up all
the available hard drive space on the server, leading to a multitude of other potential problems. If you are using PerfMon logging, ensure you periodically delete old les to free up
hard drive space.

Using Alerts
PerfMon allows you to congure alerts based on selected counters. For example, you can
monitor the OutOfResources counter in the Cisco HW Conference Bridge Device object
and be alerted when the value in that counter crosses a threshold you specify. Figure 3-11
shows some common alerts you may want to congure.
Figure 3-11 Alerts in PerfMon

Other common things you might want to set an alert for are low disk space, low available
memory, the D-channel going out of service on an MGCP gateway, and the number of registered phones falling below a certain value. You can place an alert on any counter in
PerfMon, so the reasons and counters you may want to monitor vary based on your system.
The default action for an alert is to log an entry in Event Viewer. You can also congure the
alert to send a network message to a computer you specify, run a counter log, or have a
specied program run when the alert is triggered.
The Help in PerfMon (Help > Help Topics) provides detailed conguration steps for
setting alerts.

76

Chapter 3: Understanding the Troubleshooting Tools

The RTMT in CallManager Serviceability also allows you to congure alerts based on
counters. However, in RTMT, you can have the alert sent as a network message to a
computer you specify (just like in PerfMon) or to a pager or via e-mail to a specied
recipient. You can also achieve this functionality using the CCEmail tool in combination
with PerfMon.

CCEmail
You can use the Windows 2000-based tool CCEmail in conjunction with PerfMon to
congure alerts for the performance counters. Alerts can be sent via pager or e-mail so long
as the CallManager node for which the alert is congured has connectivity to an SMTP
server. Also, the pager specied to receive alerts must be capable of receiving alphanumeric
pages through an e-mail.
As part of the free tools provided with this book, you can download a copy of CCEmail
from the Cisco Press web site. See the Acquiring CCEmail section for details.
Perform the following steps to install and congure CCEmail:
Step 1 Create a folder called ccemail on the C: drive of each CallManager server

where monitoring is to take place.

Step 2 Download the CCEmail program from the Cisco Press website (as

specied in the Acquiring CCEmail section) and save it to the

C:\ccemail directory on each CallManager server.
Step 3 Open PerfMon (Start > Programs > Administrative Tools >

Performance) and click on Performance Logs and Alerts and then

Alerts.
Step 4 Right-click in the window and select New Alert Settings.
Step 5 In the Name eld, type minor. Once you nish all the steps in this

section, you are asked to repeat them again to create two alert settings,
minor and major. The second time you perform this step, type major
instead.
Step 6 Click OK.
Step 7 (Optional) Add a comment, such as Alert settings for minor alarms or

Alert settings for major alarms.

Step 8 Click Add. The Select Counters dialog box appears.
Step 9 Make sure that Select counters from computer is selected and select the

name of the CallManager server.

Step 10 In the Performance object list, choose Cisco CallManager.

CCEmail

Step 11 With the Select counters from list button selected, select counters you

want to include in the alert and click Add. Add as many counters as you
want to monitor with this alert. You can press and hold the Control key
while clicking on counters from the list.
Some recommended counters to include for the minor alert are
RegisteredAnalogAccess
RegisteredDigitalAccess
RegisteredPhones
Some recommended counters to include for the major alert are
CallManagerHeartBeat
RegisteredAnalogAccess
RegisteredDigitalAccess
RegisteredPhones
Step 12 In the Performance object list, choose Process. Select the Private

Bytes counter and the instance _Total.

Step 13 In the Performance object list, choose Processor. Select the %

Processor Time counter and the instance _Total.

Step 14 Click Add and then Close.
Step 15 Click on the rst counter in the Counters list and set the threshold for

when to be notied. Refer to Table 3-6 for suggestions for determining

thresholds.
Table 3-6

Threshold Recommendations
Counter

Minor Alert

Major Alert

RegisteredAnalogAccess

Under 1 less than total

Under 50 percent less than total

RegisteredDigitalAccess

Under 1 less than total

Under 50 percent less than total

RegisteredPhones

Under 20 percent of total

or 50, whichever is
smaller

Under 40 percent of total or 150,

whichever is smaller

Private Bytes (1 GB Total

Memory)

900 MB

950 MB

Private Bytes (512 MB Total

Memory)

450 MB

500 MB

% Processor Time

85 percent

95 percent

77

78

Chapter 3: Understanding the Troubleshooting Tools

Step 16 Repeat the previous step until the thresholds have been set for all

counters.
Step 17 Set the interval for how often the counters should be monitored. The

recommended time is 15 minutes but you can choose any interval.

Figure 3-12 shows the General tab for the minor alert.
Figure 3-12 Minor Alert Settings, General Tab

Step 18 Click the Action tab.

Step 19 Select the Log an entry in the application event log checkbox.
Step 20 Select the Run this program checkbox and click Browse.
Step 21 Navigate to the C:\ccemail directory and choose the proper .bat le.

If you only want to use e-mail alerts (no paging), choose email.bat. If you
only want to use paging alerts (no e-mail), choose page.bat. You can also
use one method for minor alarms and the other method for major alarms.
For example, major alerts always page someone and minor alerts always
e-mail someone. In that case, choose page.bat for the major alerts and
email.bat for the minor alerts.
Figure 3-13 shows the Action tab for the minor alert.
Step 22 Click the Schedule tab.

CCEmail

Figure 3-13 Minor Alert Settings, Action Tab

Step 23 In the Start scan area, make sure that At is selected.

Step 24 In the Stop scan area, make sure that After is selected and that the setting

is 1 day. Also, select the Start a new scan checkbox. Figure 3-14 shows
the Schedule tab for the minor alert.
Step 25 Click OK.
Step 26 The alert settings for minor alarms are now complete. Repeat Steps 4

through 25 to create the alert settings for major alarms.

79

80

Chapter 3: Understanding the Troubleshooting Tools

Figure 3-14 Minor Alert Settings, Schedule Tab

Perform the following steps to congure the .bat les for CCEmail:
Step 1 In the C:\ccemail directory, double-click Setup.exe.
Step 2 The CCEmail Setup Program displays. Click Next.
Step 3 In the Sender Information window, type the name of the SMTP server to

be used and the senders e-mail address.

Step 4 Click Next.
Step 5 Conrm the settings you just entered and click Next again.
Step 6 In the Email Addresses window, type the e-mail addresses, if any, to be

e-mailed. For multiple addresses, separate entries with this string -to:.
For example: [email protected] -to:[email protected]
Step 7 Click Next.
Step 8 Conrm the settings you just entered and click Next again.
Step 9 In the Pager Addresses window, type the pager addresses, if any, to be

paged. For multiple addresses, separate entries with this string, as

described in Step 6 -to:.
Step 10 Click Next.
Step 11 Conrm the settings you just entered and click Next again.
Step 12 Click Finish.

CCEmail

81

Alerting Methods During Production and Non-production Hours

You can congure CCEmail to automatically switch between paging and e-mailing alerts
depending on the time of day. Major alerts are typically important enough that a page
should be received when the alert is triggered, regardless of the hour of day. Minor alerts
are still important, but theyre normally not considered critical enough to warrant a page
during non-production hours, such as the middle of the night.
The following steps explain how to use .bat les and the Windows Task Scheduler so that
minor alerts send pages during business hours and e-mails at night.
Step 1 Download the ccemail_auto.exe le from the Cisco Press website (as

specied in the Acquiring CCEmail section) le and save it to the

CallManager server where the changes are to be made.
Step 2 Double-click ccemail_auto.exe and make sure that the Unzip to folder

eld is set to C:\ccemail.

Step 3 Open the Windows Task Scheduler (Start > Settings > Control Panel >

Scheduled Tasks > Add Scheduled Task).

Step 4 The Scheduled Task Wizard displays. Click Next.
Step 5 Click Browse, choose the C:\ccemail\today.bat le, and click Open.
Step 6 Type a name for the task (such as today), select Daily and click Next.
Step 7 For Start time, enter the beginning of business hours, such as 8:00 AM,

select Every Day, and set the start date as todays date.
Step 8 Click Next.
Step 9 Enter the username and password of any user that has read/write

permission for the C: drive and click Next.

Step 10 Click Finish.
Step 11 Repeat Steps 3 through 10 with the following changes:

In Step 5, choose tonight.bat instead of today.bat.

In Step 7, enter the end of business hours instead of the beginning,
such as 6:00 PM.
Step 12 Open PerfMon (Start > Programs > Administrative Tools >

Performance) and click on Performance Logs and Alerts and then

click Alerts.
Step 13 Double-click on minor and click on the Action tab.
Step 14 In the Run this program eld, click Browse and choose the

C:\ccemail\minor.bat le.
Step 15 Click OK and close PerfMon.

82

Chapter 3: Understanding the Troubleshooting Tools

Acquiring CCEmail
Check the Cisco Press website for a free downloadable le containing this tool
(www.ciscopress.com > type 1587050757 in the Search eld > click the link to
Troubleshooting Cisco IP Telephony). Check the site regularly as there may also be
updates to the tool or the book chapters.

CAUTION

This is not an ofcially supported tool. If you download, install, or use this tool, you do so
at your own risk. Cisco Systems, Inc., is not responsible for correcting problems that may
arise as a result of using this unsupported tool.

CallManager Serviceability
CallManager Serviceability is a collection of tools that help you troubleshoot various
aspects of your CIPT system. CallManager Serviceability provides end user documentation
online (make a selection from the Help menu) and on Cisco.com at the following location:
www.cisco.com/univercd/cc/td/doc/product/voice/c_callmg/index.htm > select
your CallManager release > Serviceability
CallManager Serviceability provides the following basic services:

Alarms
Tracing and the web-based Q.931 Translator
Service Activation
Control Center
Real-Time Monitoring Tool

Alarms
Options under the Alarm menu let you congure the destination for the alarms (Alarm >
Conguration) and search for alarm message denitions (Alarm > Denitions). Alarms
are messages that notify you of basic errors. The messages can be inserted into CCM (SDI)
and SDL traces and the Windows Event Viewer.
CallManager Serviceability provides pre-dened alarms, set at pre-dened levels. Use the
Alarm Conguration page to set up which level of alarm you want to receive and where you
want those alarms sent. Table 3-7 describes the alarm event levels.

CallManager Serviceability

Table 3-7

83

Alarm Event Levels

Level

Description

Emergency

This level designates the system as unusable.

Alert

This level indicates that immediate action is needed.

Critical

This level indicates that CallManager detects a critical condition.

Error

This level signies that an error condition exists.

Warning

This level indicates that a warning condition is detected.

Notice

This level designates a normal but signicant condition.

Informational

This level designates information messages only.

Debug

This level designates detailed event information used for debugging.

Tracing
Options under the Trace menu let you congure trace levels and parameters for
CallManager, Database Layer, CTI Manager, and other core services. CallManager
Serviceability then allows you to collect this information from one node or all nodes in the
cluster when necessary. You can select trace information at the device level to target one or
more specic devices in the trace output. The log les generated by the trace can be .txt
format or XML-enabled for detailed analysis. We have already discussed text-based and
XML-based tracing in the previous sections Reading CCM Traces and Reading SDL
Traces. In this section, we focus on XML-formatted tracing.

Using XML-enabled Traces

Using XML-enabled traces can make reading traces easier. Earlier in this chapter, you
learned how to read trace les in their raw, text-based format and saw comparisons of
standard text-based tracing and XML-formatted tracing. In some cases, you might nd it
easier to use XML-enabled traces and let the system do the searching for you. With XMLbased tracing, you are given certain trace lters that you can apply to the trace output. Also,
you can have the system search and compile trace information on just the devices you need
(this feature is also available for standard text-based tracing). We dont normally
recommend XML-based tracing because of a 2000-line limit in the trace les and the fact
that searching through a large number of trace les is very slow. If you are searching a large
amount of data, for instance, a problem that occurred over the course of several hours or
several days, depending on the size of your system, you will probably need to use textbased tracing instead of XML-based tracing. If you select trace collection criteria that
causes the trace output to be larger than XML-based tracing can handle, a message displays
in CallManager Serviceability advising you that trace analysis cannot be performed on les

84

Chapter 3: Understanding the Troubleshooting Tools

greater than 2 MB in size. You are then given the option to save the result without ltering
using File > Save As in your web browser.
The previous section, Reading CCM Traces detailed the steps used to congure XMLenabled tracing. Once trace les are collected, XML-based tracing allows you to lter the
trace output using Trace Analysis.

Searching for Devices with XML Traces

Using the Trace Analysis feature, you can search for devices in the XML traces and narrow
the scope of the trace search by choosing different search criteria and display elds. Instead
of scanning thousands of lines of trace les looking for an event here and there, the XMLenabled trace can compile only the lines that relate to the devices or analysis criteria you
have selected. In other words, all trace le lines you arent looking for are ltered out so
that you can concentrate on what is important to your search. Trace Analysis was discussed
in detail in the previous section, Reading CM Traces.

Web-based Q.931 Translator

The Trace menu in CallManager Serviceability provides a link to a web-based interface to
the Q.931 Translator. Although convenient, the web-based interface may slow the performance of the Q.931 Translator. See the Q.931 Translator and Enhanced Q.931 Translator
section for more information about this tool.

Service Activation
Service Activation in CallManager Serviceability (Tools > Service Activation) lets you
activate and deactivate CallManager services. You can activate or deactivate CallManagerrelated services from Automatic mode by selecting or deselecting checkboxes next to specic services and then clicking the Update button. Then you can use the Control Center or
the Windows Services Microsoft Management Console (right-click My Computer and
select Manage > double-click Services and Applications > click Services) to start and stop
services.

CAUTION

If you need to deactivate a service, you should do so from the Service Activation page. If
you deactivate services from the Service Control Manager, you get an error message saying
that some of the services are not congured properly. This is because deactivating services
from the Service Control Manager does not remove the entries from the CallManager
database; therefore, the services are out of sync with the congured services in the
CallManager database.

CallManager Serviceability

85

Control Center
The Control Center in CallManager Serviceability (Tools > Control Center) lets you start
and stop CallManager services and view their activation status. You can stop and start
services in the Windows Services Microsoft Management Console as well if you have local
access to the server. If not, you can use the Control Center web page in CallManager
Serviceability to do the same thing.

Real-Time Monitoring Tool (RTMT)

The RTMT is a web-based application that provides up-to-the-second information about
the state of a CallManager cluster, including run-time information about CallManager and
CallManager-related components such as IP phones and gateways. This includes dozens of
counters on items such as call activity, trunk usage, and even memory and processor
utilization.
Appendix D describes the meanings of each of the CallManager-related performance
objects and counters, as well as some commonly used Windows counters.
The RTMT provides much the same functionality as PerfMon in an easy-to-use web-based
tool. See the section Comparing PerfMon and the Real-Time Monitoring Tool (RTMT),
for more information about the differences the two tools provide. You may nd yourself
using PerfMon for some tasks and RTMT for others. The StatisticsEnabled service parameter in CallManager Administration (Service > Service Paramaters > select a server >
Cisco CallManager) must be set to True to generate data in the counters. If statistics are
disabled, neither the RTMT nor PerfMon can collect data. Statistics are enabled by default.
One very useful feature not provided by PerfMon (unless you use a third-party tool such as
CCEmail described in this chapter) is the ability to congure alerts based on the objects.
These alerts can be set to send a notication via pager, e-mail, or system message popups.
These alerts can be congured for any performance object and can be set to trigger if a
counter is greater or less than a specic threshold. The alerts can be congured to run only
at specic times or all the time. Alerts are useful when you are troubleshooting problems
because you can set the RTMT to alert you when a specic event you are trying to
troubleshoot occurs. For example, if you are running out of channels on a gateway
interface, you can set the RTMT to page or e-mail you when the counter for total calls to
that gateway exceeds a certain value. Through release 3.3, the RTMT web browser must
remain open and running for alerts to be sent. This requirement may change in future
releases of CallManager Serviceability.
Objects and counters are only available for installed components. For example, if you do
not have Cisco CallManager Attendant Console installed on the server you are trying to
monitor, you will not see the Cisco CallManager Attendant Console object.

86

Chapter 3: Understanding the Troubleshooting Tools

Performance Tab
The Performance tab in RTMT allows you to view CallManager cluster info as shown in
Figure 3-15. To view this information, right-click on the cluster name and select
Properties. The CallManager Cluster Info window displays basic statistic about the cluster,
broken down by server.
Figure 3-15 RTMT, Cluster Info

This is also the tab where performance monitoring occurs, much the same functionality as
with PerfMon. However, in RTMT, you can congure tabs for specic counter congurations. Each time you use RTMT, the pre-dened congurations are displayed. Figure 3-16
shows several counters used to monitor general activities on a system.
To add a new category, right-click on one of the existing tabs and select New Category.
Building the category is as easy as selecting a counter and then dragging and dropping it
onto the tabs frame. Multiple counters can be piled in a single frame, and six frames per
tab are provided.

Devices Tab
The Devices tab in RTMT allows you to view device information that you congure for
various device typesphones, gateways, H.323 devices, CTI applications, voice mail, and
Cisco IP Voice Media Streaming Application devices such as MOH servers, MTP resources, and conference bridges. The Devices tab is shown in Figure 3-17.

CallManager Serviceability

Figure 3-16 RTMT, Performance Tab

Figure 3-17 RTMT, Devices Tab

87

88

Chapter 3: Understanding the Troubleshooting Tools

To add a new category, click on one of the existing tabs and select New Category. You build
the category by rst double-clicking on one of the device types and then making selections
in a series of wizard-like screens that display, as shown in Figure 3-18 and 3-19.
Figure 3-18 Device Search Criteria Screens, Part 1

Figure 3-19 Device Search Criteria Screens, Part 2

The screens shown in Figure 3-18 and 3-19 allow you to search for real-time information
about devices in the cluster regardless of their registration status. You can search for devices
based on device name, IP address, DN, IP subnet, and so on.

CTI Apps Tab

The CTI Apps tab in RTMT allows you to view application-, device-, and line-based
information for CTI applications. You can check the registration status of TAPI and JTAPI
applications such as Personal Assistant, Cisco IP Manager Assistant, Cisco IP SoftPhone,
and so on. The CTI Apps tab shows whether the application has an open connection with
CallManager and whos using it. The CTI Apps tab also shows you applications attempting
to log in using an invalid username or password. Figure 3-20 shows the CTI Apps tab.
To add a new category, right-click on one of the existing tabs and select New Category. You
build the category by rst double-clicking on one of Applications, Devices, or Lines in the
left frame and then making selections in a series of wizard-like screens that appear.

Call Detail Records (CDR) and the CDR Analysis and Reporting (CAR) Tool

89

Figure 3-20 RTMT, CTI Apps Tab

Call Detail Records (CDR) and the CDR Analysis and

Reporting (CAR) Tool
CallManager can be congured to store CDRs for all calls generated throughout a
CallManager cluster. Typically CDRs are used for billing and accounting purposes, but they
can also be useful when troubleshooting certain types of problems. Usually CDRs do not
provide enough information to diagnose a problem, but they can help you narrow down a
problem and provide information about the specic time when a problem occurred, leading
you to other trace les or debugging tools that might give you more details on the problems
root cause.
CDRs are stored in their own SQL database on the Publisher server in the CallManager
cluster. For CallManager to generate CDRs, you must set the CdrEnabled service
parameter on each CallManager to True (select Service > Service Parameters). This
service parameter is set to False by default. For additional details, you should also enable
the Call Diagnostics Enabled service parameter. If the Call Diagnostics Enabled service
parameter is set to True, CallManager also generates Call Management Records (CMRs,
also known as Diagnostic CDRs). CMRs contain data such as packets sent and received,
packets lost, and jitter for the duration of the call. One CDR might have multiple CMRs
associated with it because each media stream creates a CMR. This means that if a call is
placed on hold and then resumed, two CMRs are createdone for the media stream before

90

Chapter 3: Understanding the Troubleshooting Tools

the call is on hold, and one for the media stream after the call is on hold. CMRs are especially useful for diagnosing voice quality problems because they allow you to see patterns.
For example, you might notice that all the phones across a specic WAN link are experiencing high jitter or packet loss. This can indicate a possible quality of service (QoS)
misconguration or line errors on the WAN link.
The CDR Analysis and Reporting (CAR) tool (CallManager Serviceability > Tools >
CDR Analysis and Reporting) can help you analyze the raw data that comprises the CDR
database and create reports based on your search criteria. For example, if you are receiving
complaints of poor voice quality, one of the rst things you should do is nd out which
phones or gateways are experiencing the poor voice quality. Although you can wait to
collect data from additional user complaints, you can proactively use the data in the CDRs
and CMRs to identify any trends in high jitter or packet loss to hone in on where the
problem is.
For example, if a user tells you they had a problem calling someone, you can use CAR to
search for the call in the CDRs. This gives you the time the call occurred, which helps you
when you examine the CCM traces related to the problem.
You can search CDRs by user or specic extensions for the period that you specify. This
helps you trace calls placed from specic extensions for diagnostic or informational
purposes. All associated records, such as transfer and conference calls, appear together as
a logical group.
CAR can also be used to send you alerts if the number of calls with poor QoS is exceeded
or if the CDR database size exceeds a percentage of the maximum number of records.
For more information on the various features available in CAR, review the CAR section in
the CallManager Serviceability documentation at
www.cisco.com/univercd/cc/td/doc/product/voice/c_callmg/index.htm > select
your CallManager release > Serviceability > Serviceability System Guide > CDR
Analysis and Reporting
If you look at the raw timestamp information stored in the CallManager CDRs, you notice
they are stored in a format that is not easily recognizable in standard date and time format.
To quickly convert the date and time from the format in the CDRs to standard format, use
the CDR Time Converter utility.

CDR Time Converter

Timestamps in the CallManager CDRs are stored in a format known as Epoch time. This is
the number of milliseconds that have elapsed since midnight January 1, 1970 GMT. While
this might be a convenient format for things like computers, humans usually prefer a more
readable date and time format.

Event Viewer

91

The CDR Time Converter utility allows you to enter the time as stored in a CDR
for example, 1030565084and convert it to standard date and time formatsuch as
8/28/2002 3:04:44 PM. Figure 3-21 shows the output of the CDR Time Converter tool.
Figure 3-21 CDR Time Converter Tool

Notice the tool converts the number in Epoch time to Greenwich Mean Time (GMT) and
the local time zone of the PC on which the tool is installed for both standard and daylight
savings time.

Acquiring the CDR Time Converter

Check the Cisco Press website for a free downloadable le containing this tool
(www.ciscopress.com > type 1587050757 in the Search eld > click the link to
Troubleshooting Cisco IP Telephony). Check the site regularly as there may also be
updates to the tool or the book chapters.

CAUTION

This is not an ofcially supported tool. If you download, install, or use this tool, you do so
at your own risk. Cisco Systems, Inc., is not responsible for correcting problems that may
arise as a result of using this unsupported tool.

Event Viewer
Microsoft Event Viewer is a Windows 2000 Server application that displays system,
security, and application events (including CallManager) for the Windows 2000 Server.
These events are alarm messages generated by CallManager. CallManager Serviceability is
used to congure alarm messages to be sent to the Event Viewer (Alarm > Conguration).

92

Chapter 3: Understanding the Troubleshooting Tools

Open Event Viewer on the server running CallManager by clicking Start > Settings >
Control Panel > Administrative Tools > Event Viewer. CallManager errors are logged in
the Application log. You can double-click an event in the log to learn more about it.
Alarm denitions can be found in CallManager Serviceability (Alarm > Denitions).
Figure 3-22 shows an example of an alarm message in Event Viewer.
Figure 3-22 Event Viewer

Notice the App ID and the Error message. This tells you that the IP phone specied in the
Device Name details has unregistered from CallManager.
All alarms fall into seven catalogs, as shown in Table 3-8.
Table 3-8

Alarm Denition Catalogs

Catalog Name

Description

CallManager

All CallManager alarm denitions such as CallManagerFailure,

DChannelOOS, DeviceUnregistered, and RouteListExhausted.

TFTPAlarmCatalog

All Cisco TFTP alarm denitions such as kServingFileWarning

and kCTFTPConnectSendFileTimeoutOccurred.

Event Viewer

Table 3-8

Alarm Denition Catalogs (Continued)

Catalog Name

Description

CMIAlarmCatalog

All Cisco Messaging Interface (CMI) alarm denitions such as

kCCMConnectionError, kSMDIMessageError and
kSerialPortOpenningError.

CtiManagerAlarmCatalog

All Cisco Computer Telephony Integration (CTI) alarm denitions

such as kCtiProviderOpenRej, kCtiMaxConnectionReached, and
kCtiProviderOpenFailure.

DBAlarmCatalog

All Cisco database alarm denitions such as kPrimaryDbIsLost,

kUnableToConnectToDB, and kErrorBuildingCnfFile.

GenericAlarmCatalog

All generic alarm denitions shared by all applications such as

OutOfMemory, ServiceStopped, and ServiceStartupFailed.

IpVmsAlarmCatalog

All Cisco IP Voice Media Streaming Application alarm denitions,

including MOH, conference, media termination point, and
transcoder alarms, such as kIPVMSDeviceDriverNotFound,
CreateAudioSourcesFailed, and kDeviceDriverError.

JavaApplications

All Java applications that run on the CallManager server including

extension mobility and Cisco IP Manager Assistant (IPMA), such
as EMAppServiceError and IPMAApplicationError.

Based on the information given in the Event log entry in Figure 3-22 you can go to
CallManager Serviceability to search for a better denition of the problem in question.
To search for alarm denitions, perform this procedure:
Step 1 In CallManager Administration, choose Application >

Cisco CallManager Serviceability. The Cisco CallManager

Serviceability window appears.
Step 2 Choose Alarm > Denitions.
Step 3 Choose the catalog of alarm denitions from the Find alarms where

drop-down box, or click the Enter Alarm Name eld to enter the alarm
name. Figure 3-23 shows this interface.

93

94

Chapter 3: Understanding the Troubleshooting Tools

Figure 3-23 CallManager Alarm Message Denitions

Step 4 Click the Find button. The denitions list appears for the alarm catalog

or search string you entered.

Step 5 In the list, click the alarm denition for which you want alarm details.

Based on the Event Viewer entry shown in Figure 3-22, choose the
CallManager catalog and the DeviceUnregistered error. You see Figure
3-24, showing the severity, an explanation, and the recommended action.
All alarm messages found in the Event Viewer can be researched in this
fashion.

Q.931 Translator and Enhanced Q.931 Translator

95

Figure 3-24 CallManager Alarm Details

Q.931 Translator and Enhanced Q.931 Translator

Q.931 Translator is an application that takes a CCM trace and decodes the hex-formatted
Q.931 messages into human-readable format. Depending on the version of CallManager
you are using, there are two ways to access the application. Q.931 Translator is bundled
with every CallManager installation. It can be found in the C:\Program Files\Cisco\bin
directory. The le is called Q931 translator.exe. In CallManager version 3.2 and later, the
application is also part of the CallManager Serviceability web page (select Trace > Q931
Translator).
Q.931 Translator is useful for quickly troubleshooting a variety of gateway problems. It is
not a substitute for learning how to read the CCM trace, but it helps you resolve some
gateway signaling problems without ever having to look in the CCM trace.
The name Q.931 Translator is a bit misleading because this application does more than just
translate ISDN messages into a human-readable format. The Q.931 Translator also helps
you troubleshoot problems with hardware that does not use the Q.931 protocol, including
the WS-X6624 analog FXS card, T1 CAS on the WS-X6608 card, and calls to and from an

96

Chapter 3: Understanding the Troubleshooting Tools

H.323 gateway. The Translator is helpful for troubleshooting problems with these hardware
components because some of these gateways convert their native signaling to Q.931
messages. H.225 messages used in the H.323 protocol are based on the Q.931 specication;
hence, the Q.931 Translator can decode H.225 messages.
The amount of information contained in a CCM trace can be intimidating. Q.931 Translator
helps you quickly examine a trace in a graphical format without having to wade through
irrelevant debugs. For example, the following sample is from a CCM trace of an outgoing
call setup:
Out Message -- Pri250SetupMsg -- Protocol= Pri250Protocol
Ie - Ni2BearerCapabilityIe IEData= 04 03 80 90 A2
Ie - Q931ChannelIdIe IEData= 18 03 A9 83 94
Ie - Q931DisplayIe IEData= 28 0C B2 50 61 75 6C 20 47 69 72 61 6C 74
Ie - Q931CallingPartyIe IEData= 6C 0C 21 80 39 31 39 35 35 35 35 36 34 34
Ie - Q931CalledPartyIe IEData= 70 0D A1 39 31 32 31 30 35 35 35 32 35 30 30
MMan_Id= 0. (iep= 0 dsl= 0 sapi= 0 ces= 0 IpAddr=346812ac IpPort=2427)
IsdnMsgData2= 08 02 05 C8 05 04 03 80 90 A2 18 03 A9 83 94 28 0C B2 50 61 75
6C 20 47 69 72 61 6C 74 6C 0C 21 80 39 31 39 35 35 35 35 36 34 34 70 0D A1
39 31 32 31 30 35 35 35 32 35 30 30

Without a copy of the ITU Q.931 specication, which explains each bit in the trace
information elements, this sample looks like a bunch of numbers and letters. However, the
Q.931 Translator decodes the output into a readable format, as shown in Figure 3-25.
Figure 3-25 Q.931 Translator Application

One thing you should know right away is not to trust the Direction column. Information in
the Direction column can occasionally be inaccurate because of how the tool decodes hex
messages. To be sure of the direction, look at the rst line in the debug, which states Out

Q.931 Translator and Enhanced Q.931 Translator

97

Message. This indicates that this is an outbound (TX) setup message. Similarly, an inbound
(RX) message would show In Message in the CCM trace.
Calls in the trace can be distinguished by the call reference value. The most signicant bit
(MSB) of the call reference toggles between 0 and 1, depending on whether the message
was inbound to CallManager or outbound from CallManager to the gateway. Table 3-9
shows the difference between the rst bit being 1 versus being 0. This table also shows the
binary representation of the hexadecimal digits to clearly show how the last three binary
digits in the left and right columns are identical. The only difference between the left and
right columns is that the MSB for all the digits in the left column is 0, and the MSB for all
the digits in the right column is 1.
Table 3-9

Binary Representation of Hexadecimal Digits

MSB = 0

MSB = 1

0 (0000)

8 (1000)

1 (0001)

9 (1001)

2 (0010)

A (1010)

3 (0011)

B (1011)

4 (0100)

C (1100)

5 (0101)

D (1101)

6 (0110)

E (1110)

7 (0111)

F (1111)

The output in Figure 3-25 shows that the call reference is 0x05C8 in the transmit (TX)
direction and 0x85C8 in the receive (RX) direction. Any row that has a 0x05C8 or 0x85C8
in the Call Reference column is a message for the same call. As Table 3-9 shows, 0 and 8
are equivalent, with the exception of the MSB, which is different. Call reference values are
eventually reused, but it is impossible for the same call reference to appear in a single CCM
trace le for one gateway because several hundred or thousand calls must occur before the
value is reused.
Q.931 Translator shows the calling and called party numbers converted to easily readable
text, along with the bearer channel identier, bearer capability, and display information
element. The channel identier and bearer capability are not fully decoded. You need a copy
of the Q.931 specication to understand what the various bits in those elds signify. Dont
worry if you dont have this specication. Chapter 6 explains how to decode some of these.
Some of these are decoded for you in the Enhanced Q.931 Translator explained in the next
section.
Q.931 Translator is most useful for decoding cause codes that are sent as an information
element (IE) in various Q.931 messages. These cause codes are always sent as part of a
DISCONNECT message and may be included as part of other messages as well. Decoding
the cause code IE gives you some insight into why the call was disconnected.

98

Chapter 3: Understanding the Troubleshooting Tools

The following is a sample of a CCM trace showing a Q.931 DISCONNECT message:

Out Message -- PriDisconnectMsg -- Protocol= Pri250Protocol
Ie - Q931CauseIe IEData= 08 02 80 81
MMan_Id= 0. (iep= 0 dsl= 0 sapi= 0 ces= 0 IpAddr=a86a12ac IpPort=2427)
IsdnMsgData2= 08 01 AA 45 08 02 80 81

The rst thing you should notice is that this is a DISCONNECT message being sent by
CallManager to the PSTN. You can see that the hex representation of the data in the
Q931CauseIe is 08 02 80 81. These cause codes are dened in the ITU Q.850 specication;
however, Q.931 Translator decodes these for you. Opening the trace le containing the
DISCONNECT message in Q.931 Translator reveals the following:
DISCONNECT pd = 8 callref = 0xAA
Cause i = 0x8081 - Unallocated/unassigned number

You can see that 0x8081 is decoded to Unallocated/unassigned number, meaning that
CallManager does not have a phone or route pattern that matches the digits that were sent
to CallManager by the ISDN network.

NOTE

See Chapter 6 for additional information about what various cause codes mean and how to
continue troubleshooting problems like these.

Enhanced Q.931 Translator

Although the Q.931 Translator allows you to quickly observe Q.931 and H.225 events, its
functionality is somewhat limited. Two Cisco TAC engineers took the original source code
for Q.931 Translator and enhanced it to provide additional functionality not available from
the ofcial Q.931 Translator product bundled with CallManager.
The Enhanced Q.931 Translator offers the following advantages over the standard Q.931
Translator:

Direction column is correctAs mentioned earlier, the method by which the

standard Q.931 Translator decodes the direction is awed, and therefore, the direction
column is sometimes incorrect. The Enhanced Q.931 Translator properly decodes the
messages as In or Out.

Protocol columnTells you whether the message is a Q.931 message or an H.225

message.

Expanded IE decodingDecodes far more Q.931 information elements than the

original Q.931 Translator, including bearer capability, channel ID, numbering plan
and type in the calling and called party number IEs, call state, and many more.

Find in messagesAllows you to search for any text that appears in the decoded
message data. This means you can search for a calling or called party phone number,
disconnect cause code, or any other value and quickly nd the message you are
looking for.

Q.931 Translator and Enhanced Q.931 Translator

99

Filter messagesAllows you to lter messages based on a specic call reference or

protocol type. When ltering on call reference, Enhanced Q.931 Translator automatically includes all messages for that call reference regardless of the setting of the
MSB of the call reference.

Raw ISDN message dataAfter the decoded information, the raw hex bytes for the
ISDN message are presented.

Resizable windowAllows you to resize the window to view more messages.

File name displayThe currently-open lename is displayed in the title bar in case
you forget where you left off when searching through several traces in a folder.

Figure 3-26 shows the same trace shown in Figure 3-25, this time as it looks in the
Enhanced Q.931 Translator.
Figure 3-26 Enhanced Q.931 Translator Application

Notice the amount of additional data presented in the bottom pane of the tool. The bearer
capability (ITU-T standard, Speech, Circuit mode, 64k, -law), channel ID (PRI interface,
Exclusive channel 20), calling party numbering plan and type (Plan: ISDN, Type: National,
Presentation Allowed, User-provided, not screened), and called party numbering plan and
type (Plan: ISDN, Type: National) are all decoded for you automatically.

100

Chapter 3: Understanding the Troubleshooting Tools

Future versions of CallManager may include the Enhanced Q.931 Translator, but until then,
you must download the tool from the Cisco Press website (see Acquiring Enhanced Q.931
Translator later in this chapter).
One thing to remember is that the Q.931 Translator and Enhanced Q.931 Translator tools
are not a replacement for the CCM trace. They are just another tool you can use to make
decoding the CCM trace easier and nding the location of your problem quicker. Often you
will have to refer back to the CCM trace after nding the call in Q.931 Translator to get
additional detail regarding the events that surround a message. For example, if you nd a
message in Q.931 Translator indicating CallManager disconnected a call with a cause code
of 0xAF, Resources Unavailable, Q.931 Translator tells you the cause code and the
timestamp in the CCM trace. You must then go to the CCM trace at that timestamp and
determine what resource was unavailable that caused CallManager to disconnect the call.
As mentioned before, Q.931 Translator decodes more than just Q.931 messages. The H.225
protocol used to communicate between H.323 gateways and other CallManager clusters
uses messages similar to ISDN Q.931. Q.931 Translator translates H.225 messages that
appear in CCM traces.
Some gateways use Q.931 to communicate with CallManager even though the gateways
interface to the PSTN is not actually using Q.931. For example, the WS-X6624 24-port
FXS gateway uses analog FXS signaling to communicate with analog phones, fax machines, and modems; however, this analog signaling is converted to Q.931 messages
between the gateway and CallManager. These Q.931 messages appear in the CCM trace
and are translated by the Q.931 Translator. The same is true of the WS-X6608-T1 gateway
when communicating with the PSTN using channel associated signaling (CAS). The
gateway converts CAS to Q.931 messages.
So if the gateways are converting these various protocols to Q.931, you might wonder how
you can troubleshoot the signaling before it is converted to Q.931. This is one of the various
uses of the Dick Tracy tool discussed in the section Dick Tracy.

Acquiring Enhanced Q.931 Translator

Check the Cisco Press website for a free downloadable le containing this tool
(www.ciscopress.com > type 1587050757 in the Search eld > click the link to
Troubleshooting Cisco IP Telephony). Check the site regularly as there may also be
updates to the tool or the book chapters.

CAUTION

This is not an ofcially supported tool. If you download, install, or use this tool, you do so
at your own risk. Cisco Systems, Inc., is not responsible for correcting problems that may
arise as a result of using this unsupported tool.

Dick Tracy

101

Dick Tracy
The Dick Tracy tool is a complex and powerful tool used to troubleshoot problems on
various gateways based on the Skinny or MGCP protocols. Specically, the Dick Tracy tool
is used on the following voice gateways:

DT-24+
DE-30+
WS-X6608-T1
WS-X6608-E1
WS-X6624

There is little to no documentation on the Dick Tracy tool because it was created as an
internal development tool. However, it has been released under the condition that the tool
itself is unsupported. This means that no formal documentation or release mechanism exists
for the tool. This also means that the tools behavior might change from one release to
another without warning, but as long as you understand the basics of how the tool works,
you should be able to pick up any changes without too much difculty. The Acquiring
Dick Tracy section explains how you can download a copy of the Dick Tracy tool.
There are two versions of Dick Tracy:

A standalone Windows 95/98/NT/2000/XP application that connects to the gateway

you want to monitor over TCP/IP. This version is commonly called Dick Tracy.

A version is embedded in the Catalyst 6000 operating system (CatOS). It can be

invoked from the Catalyst 6000 command-line interface (CLI) to diagnose problems
on WS-X6608 and WS-X6624 gateways in the chassis. This version is commonly
called CLI Tracy or embedded Tracy.

The CLI Tracy tool can be used to connect only to gateways that are in the same Catalyst
6000 chassis to which you are connected. Using the Windows-based Dick Tracy tool is
recommended here because it is more exible and easier to use than CLI Tracy.
Why do you need the Dick Tracy tool? In case youve never seen a Cisco DT-24+ gateway,
it is a PCI card that plugs into a PCI slot in any PC chassis. The gateway only uses the PCI
slot to draw power. The PCI bus is not used for any kind of data transfer to or from the
DT-24+. The DT-24+ also provides an Ethernet port and a T1 port. There is no console port
or other method of out-of-band communication to the gateway; therefore, you need a tool
to access the gateway so that you can determine what is going on inside the gateway. The
WS-X6608 and WS-X6624 are similar to the DT-24+ because they use the Catalyst 6000
chassis they reside in for power and IP connectivity. Other than that, the Catalyst 6000
Supervisor (the management interface on the Catalyst 6000) has no out-of-band
management interface to the gateways.

102

Chapter 3: Understanding the Troubleshooting Tools

CAUTION

Before we discuss the tool itself, you should know that the Dick Tracy tool is not the most
user-friendly piece of software. It also can damage your gateway if its not used properly.
A good working rule is that if you dont know what something does, you probably shouldnt
mess with itparticularly most of the set commands that are available. This book covers a
few useful set commands, but other than those, you should not need to use Dick Tracy to
set commands on the gateways.

Using the Dick Tracy Tool

Figure 3-27 shows the Windows version of the Dick Tracy tool. The tool itself is a very
simple program. Clicking the Connect button on the toolbar reveals a box indicating
Connect to remote target device. Enter the IP address of the gateway you want to
troubleshoot in the Target IP Address box. The port number should always be 2005.
Figure 3-27 Dick Tracy Tool for Windows

After you connect to a gateway, you see a text box labeled Live trace on ip address, where
ip address is your gateways IP address. You can open additional connections to other
gateways using the same procedure. You will likely see several lines of tracing when you
connect. This is because the gateway buffers the last few lines of tracing and shows them
to you when you connect.

Dick Tracy

103

After you connect to a gateway, it is a good idea to enable logging of the live trace. Click
Options > Start Logging. A dialog box appears, asking where you want to save the logged
traces. Choose a convenient location on your hard drive that has enough space available.
The traces are just a text log of everything you see on your screen while connected to the
gateway. They usually do not get very big unless you are running a debug for an extended
period of time. Logging does not affect the performance of Dick Tracy or the gateway.
Enabling logging keeps a record of all the traces you capture until you close the trace
window or stop logging using the menu.
At this point, youve started a trace, and the tool is up and running. On the menu bar you
see a one-line text box and a Send button. Think of this eld as the command box to communicate with the gateway. To communicate with the gateway, you must understand the
concept of task IDs. Each gateway has various tasks that are responsible for various
components of the software. For example, one of the tasks is responsible for sending and
receiving messages to and from the digital signal processors (DSPs) on the gateway.
To view the list of available tasks on a gateway, enter the command 0 show tl (this command uses the number 0, not the letter O, and the letter L, not the number 1) in the command
box and click Send or press Enter. You should see something similar to the following
sample:
03:15:15.450
0 :
1 :
2 :
3 :
4 :
5 :
6 :
7 :
8 :
9 :
10 :
11 :
12 :
13 :

(GEN) Lennon Tasks

GEN
AUD
TRC
SNMP
SPAN
NMP
DSP
LINE
CFG
GMSG
SOCK
TMR
Q921
XA

The 0 (zero) is the number of the task ID you want to issue a command to. In this case, you
are issuing a command to the GEN task (ID 0). Each task has various commands that can
be issued to it. Some tasks do not respond to commands at all. The show tl portion is the
command being issued to task ID 0.
Dick Tracy offers context-sensitive help for the tasks that accept commands. For example,
to see which commands are available for task 0, enter 0 ? in the command box and press
Enter. You see something similar to the following sample:
03:22:21.320
03:22:21.320
03:22:21.320
03:22:21.320
03:22:21.320
03:22:21.320
03:22:21.320

GEN --> Help -> Available Commands:

reset <hard/soft>
read func[index] <count>
write func[index] <value>
show func
clear func
set func

104

Chapter 3: Understanding the Troubleshooting Tools

You can see that show is one of the available commands for task ID 0. You can get
additional context-sensitive help by entering 0 show ?. You see the following:
03:24:31.500
03:24:31.500
03:24:31.500
03:24:31.500
03:24:31.500
03:24:31.500

GEN
Show
Show
Show
Show
Show

Help: show modifiers ->

Task List : show tl
Ethernet Stats : show ether
Versions : show ver
Time : show time
Reset : show reset

You can see that the command show tl performs the command Show Task List according
to the context-sensitive help. It is difcult to provide a list of the available task IDs and
Tracy commands because they vary from gateway to gateway and can change from one
version of CallManager to the next. Chapter 6 covers some specic task IDs when
discussing troubleshooting gateways.
You should always precede each command with a task ID. If you do not precede a command
with a task ID, the gateway applies the command to the last task ID you sent a command to.
In addition to the various show commands, a few other commands are useful. One in particular is set mask. You use the set mask command to turn on various debugs. To view what
debugs you can turn on for a particular task, issue the show mask command for the task in
question. For example, issuing the command 6 show mask details the various trace bits for
the DSP task:
(DSP) Mask<0x0>
Where Bit0 =
Bit1 =
Bit2 =
Bit3 =
Bit4 =
Bit5 =
Bit6 =
Bit7 =

Debug Msgs
Call Progression Msgs
Boot Msgs
Stat Msgs
Cmd Msgs
RTP Msgs
SID Frames
Status Msgs

As you can see from this output, the mask is currently set to 0x0, and eight different trace
bits can be enabled. The mask is set as a hexadecimal digit. In this case because you have
eight bits, 256 possible combinations of tracing can be turned on, depending on which bits
are enabled. For example, to enable Debug Msgs (Bit0) and Call Progression Msgs (Bit1),
the mask must be set to 00000011 in binary. This translates to 0x03 in hex. Therefore, the
command is 6 set mask 0x03.

CAUTION

Be sure to set the masks back to 0x0 after enabling any trace masks. The traces continue to
run until the gateway is reset, even if you close your Dick Tracy session. Failure to turn off
the debug masks could create a performance impact on the gateway.

For the time being, dont worry about what each of these trace masks or task IDs does.
Chapter 6 covers the masks and IDs in detail.

Dick Tracy

105

Using the CLI Tracy/Embedded Tracy Tool

The less-used version of Dick Tracy is the CLI Tracy or embedded Tracy tool, available on
the Catalyst 6000 CLI. To enable tracing for a particular port, enter the command
tracy_start module port, where module and port are the module and port numbers of the
port you want to debug. For example, for the gateway on port 6/3, you would enter
tracy_start 6 3

Notice that the syntax differs from the traditional Catalyst Operating System (CatOS)
notation, in which ports are specied using module/port.
After you start CLI Tracy for a port, all the debug information for that port appears on your
console or Telnet session. You can send Tracy commands to the port using the command
tracy_send_cmd module port taskid command. When you need to send commands to a
port, you should really use the Windows-based tool because sending commands via CLI
Tracy can lead to switch instability. When you are done using the CLI Tracy tool, enter the
command tracy_close module port to end the session. You can have only one CLI Tracy
session open at any given time on the whole chassis. So if someone has a Tracy session open
on the console port, you cannot start one from a Telnet session. Be sure to close the CLI
Tracy session when you are done.
CLI Tracy does have one advantage over the Windows-based Dick Tracy tool: It can monitor a port before it obtains an IP address. For that reason, you can use CLI Tracy to
troubleshoot problems where the port cannot obtain an IP address. The regular Dick Tracy
tool cannot accomplish this because you need IP connectivity to the port in question before
you can gather information from the port.

Acquiring Dick Tracy

Check the Cisco Press website for a free downloadable le containing this tool
(www.ciscopress.com > type 1587050757 in the Search eld > click the link to
Troubleshooting Cisco IP Telephony). Check the site regularly as there may also be
updates to the tool or the book chapters.

CAUTION

This is not an ofcially supported tool. If you download, install, or use this tool, you do so
at your own risk. Cisco Systems, Inc., is not responsible for correcting problems that may
arise as a result of using this unsupported tool.

106

Chapter 3: Understanding the Troubleshooting Tools

Sniffer Traces
One of the most powerful tools for troubleshooting a large number of CallManager
problems is a packet capture/analyzer tool such as Network Associates Sniffer Pro
(www.nai.com), Finisar Surveyor (www.nisar.com/product/product.php?product_id=104),
or Ethereal (www.ethereal.com). Because of Sniffer Pros widespread appeal, a trace le
generated by any packet-capture tool is commonly called a sniffer trace. This term is used
here to refer to any packet-capture software, not just Sniffer Pro.
A sniffer trace lets you see exactly what is happening on the network at any given time.
Examining a sniffer trace requires a good understanding of the various layers of the OSI
model, which were briey described in the Introduction to this book.
To get the most benet from a packet-capture tool, you should use a tool that can decode
the various protocols you might encounter in an IP telephony network. This includes, but is
not limited to, H.323 (H.225 and H.245), MGCP, RTP, SQL, and LDAP. Also extremely
important is the capability to decode Skinny packets. Most newer versions of commercially
available network capture application can decode Skinny. Finisar Surveyor can decode
Skinny as part of the standard package while Network Associates Sniffer Pro requires you
purchase the Sniffer Voice add-on to get Skinny decodes. Also, the free protocol analyzer
Ethereal supports decoding Skinny as of version 0.8.20.
This book does not teach you how to use the network analysis software, but instead focuses
on the kind of information you can obtain using network analysis software. Sniffer traces
are most important when youre troubleshooting problems that cant be examined using
standard trace les or debugs because the problem is either network-related or the appropriate diagnostic tool is not built into the product. For example, device registration problems are much easier to troubleshoot with a sniffer trace, as are most voice quality
problems.

Voice Codec Bandwidth Calculator

Use the Voice Codec Bandwidth Calculator to determine the bandwidth used by different
codecs with various voice protocols over different media.
To get a detailed analysis of all the headers for your particular medium, use the automated
Voice Codec Bandwidth Calculator available on Cisco.com. You must be a registered user
on Cisco.com to access the tool.
http://tools.cisco.com/Support/VBC/jsp/Codec_Calc1.jsp

Bug Toolkit (Formerly Bug Navigator)

Cisco provides a bug search feature that allows you to nd known bugs based on software
version, feature set, and keywords. The resulting matrix shows when each bug was integrated or xed, if applicable. It also allows you to save the results of a search in Bug Groups
and also create persistent Alert Agents that can feed those Groups with new defect alerts.

Remote Access Tools

107

Bug Toolkit is only available to registered users on Cisco.com. Access the bug toolkit by
searching for bug toolkit on Cisco.com or at the following link:
www.cisco.com/cgi-bin/Support/Bugtool/launch_bugtool.pl

Remote Access Tools

There are several tools you can use to remotely access your system. In this chapter, we
briey discuss Terminal Services and Virtual Network Computing (VNC).

Terminal Services
Windows Terminal Services is a feature that comes standard on all servers running
Windows 2000 Server software. Terminal Services allows you to remotely access the
Windows interface on a CallManager server.
Terminal Services is extremely useful for remotely troubleshooting problems on
CallManager when you do not have immediate access to the console. To use Terminal
Services, you must install the Terminal Services Client on any PC running most Microsoft
Windows operating systems.
The installer for the Terminal Services Client is included on all CallManager servers in the
C:\WINNT\system32\clients\tsclient folder. There are several folders for the 32-bit and
16-bit versions. Use the 32-bit client on any PC running Windows 95 or later.
Once installed, launch the Terminal Services Client and enter the IP address of
CallManager and the screen resolution you want to connect with. You receive a login
prompt for the server. Enter the administrator username and password to authenticate. Once
authenticated, you have access to the CallManager desktop almost as if you were on the
console.
If you need to open a hole through your rewall to access CallManager via Terminal
Services, open TCP port 3389. This is the only port needed to access a Windows 2000
Server via Terminal Services.

CAUTION

Cisco does not support installing any CallManager applications, CallManager software
patches, or operating system patches via Terminal Services. Terminal Services is designed
for remote access to the CallManager server for troubleshooting purposes; however, some
portions of the CallManager installer are not compatible with Terminal Services. You can
use VNC to access the console of the server for performing upgrades and patches.

108

Chapter 3: Understanding the Troubleshooting Tools

Virtual Network Computing (VNC)

VNC is basically a remote display system that allows you to view a remote desktop
environment. VNC allows you to use one computer to drive actions on a target computer
but differs from Terminal Services because, with VNC, any actions performed by you that
occur on the target computer can be seen equally by the local user. All computers must have
a local copy of VNC installed. You can use VNC to install, upgrade, or apply patches to
CallManager.
You can access the VNC application and documentation les on the OS version 2000 2.2
and later CD or download. If youre running an older version of the OS, run the OS upgrade
for version 2000 2.2 or later to gain access to the VNC les. OS upgrades are available on
CCO at the following link:
www.cisco.com/cgi-bin/tablebuild.pl/cmva-3des

CAUTION

Using VNC can expose you to a security risk. Please review the Security Best Practices
section in the Cisco-produced document for installing VNC, which is available on the OS
2000 version 2.2 and later CD or at the download link previously shown.

Websites and Further Reading

Theres a wealth of information available on the Internet. When youre looking for more
information, use the following resources:

Cisco Technical Assistance websiteProvides the latest information and technical

documentation from Ciscos Technical Assistance Center (TAC). Use the TAC
website to search for tech tips and documentation, download software updates, or
open a TAC case to obtain additional information. Access the TAC website by
searching for TAC on Cisco.com or at
www.cisco.com/tac

Technical tips for IP Telephony applications, servers and associated

technologiesThis site contains a variety of technical documents written by TAC
engineers that are useful for solving commonly encountered problems. Access the
IP Telephony Applications website by searching for IP Telephony Applications
on Cisco.com or at
www.cisco.com/pcgi-bin/Support/browse/index.pl?i=Technologies&f=1533

Networking Professionals ConnectionThis site is the gathering place for networking professionals to share questions, suggestions, and information about
networking solutions, products, and technologies. Access the forum by searching for
networking professionals on Cisco.com or at
www.cisco.com/go/netpro

Best Practices

109

AnswerMonkeyYoull nd detailed information about Cisco Unity at the home

page of one Unitys creators, Jeff Lindborg. Access the site by searching for
answermonkey on Google.com or at
www.answermonkey.net

Updates to this bookCheck the Cisco Press website regularly for updated
information pertaining to the chapters in this book (www.ciscopress.com >
type 1587050757 in the Search eld > click the link to Troubleshooting Cisco
IP Telephony).

Check out the section Further Reading in the Introduction to this book for additional
books about IP Telephony and VoIP.

Best Practices

Become familiar with the various troubleshooting tools at your disposal. Be sure to
try each of them before you encounter a problem to understand how they work so you
do not have to waste time learning the tool when under pressure to resolve a network
outage.

Keep a copy of all the tools in a centralized location and check frequently for updates
that may add additional functionality.

The only way to become efcient at reading CCM traces is by practicing. The more
you look through CCM traces to troubleshoot problems, the better you will
understand the intricacies of the CCM trace.

Ensure remote access to your CallManager cluster is available before a problem

occurs. If you need to be able to access the CallManager cluster from outside the
ofce, make provisions for VPN or dialup access and use Terminal Services or VNC
to access your CallManagers. Terminal Services works amazingly well even over slow
dialup connections.

VNC Best Practices

If youre using VNC and no longer plan to use Terminal Services for remote
management, disable Terminal Services.

Set the VNC service to Manual startup and start it only during remote management.
This adds another layer of protection by requiring that users access the environment
via Windows username/password authentication to start the VNC service.

Use a complex alphanumeric password for VNC. VNC does not have a username/
password structure; it only uses a single password, so make sure the password you
choose is difcult to crack. VNC limits the password to eight characters. A good
password includes numbers, upper- and lowercase letters, and special characters and
does not use any known word. For example, 123eye67 is not as good a password
choice as 4hW9Lv#g.

110

Chapter 3: Understanding the Troubleshooting Tools

Summary
This chapter covered the various troubleshooting tools and resources you have at your
disposal to troubleshoot a CIPT network. Which tool you use depends largely on the
problem at hand. However, some problems can be resolved using more than one tool. As
you become more familiar with each of the tools in your tool belt, you will begin to favor
some tools over others for specic tasks. No matter how much you read about these tools,
you will not learn about them until you use them to troubleshoot a real problem on your
own. As you advance through the rest of this book, you will see frequent references to the
tools presented in this chapter because they play an integral part in CIPT troubleshooting.

This page intentionally left blank

INDEX

Symbols
! wildcard, 460
. wildcard, 461
@ wildcard, 461
DDIs, 487494
route filters, 506507
multiple clauses, 512
NANP tags, 508510

Numerics
3-port switch operation, Cisco 79xx series IP
phones, 161164
7-digit local calls, delayed routing, 466469
10-10-Dialing DDI, 487
10-10-Dialing Trailing-# DDI, 487
11/10D->7D DDI, 487
11/10D->7D Trailing-# DDI, 487
11D->10D DDI, 488
11D->10D Trailing-# DDI, 488
802.1Q protocol, 850
911 routing, Cisco ER, 478
6608 T1/E1 module
configuring, 325337
D-channel establishment, 337, 340343
advanced troubleshooting, 344359
T1 CAS, 359367
6624 Port FXS Analog Interface Module
configuring, 367379
7960/7940 IP Phones
extension mobility, 756758
conguring, 758763
login/logout process, 763765
resolving common problems,
765768, 772
79xx IP Phones
3-port switch operation, 161164
network settings, 123126

A
AA (Auto Attendant), 737
traces, collecting, 748752
AAR (automatic alternate routing), 478
acknowledgments, 238
ACOM (combined loss), 417
acoustic echo, isolating, 412
acquiring
Dick Tracy tool, 105
Q.931 Translator, 100
active connections, 155
AD (Active Directory)
Customer Directory Configuration plugin,
troubleshooting, 839844
LDAP integration, 837839
Ad Hoc conferences, 565
error messages, 597598
locations-based CAC bandwidth reservations,
633635
adjusting
fax relay data rate, 451452
interdigit timeout, 467
Administrative Reporting Tool (ART), 795
alarms
configuring on CallManager Serviceability, 82
StationAlarmMessage field definitions,
158160
alerts
configuring on CCEmail, 81
enabling on PerfMon, 75
algorithmic delay, 386
Already In Conference error messages,
troubleshooting, 597
analog gateways, VG248 SMDI integration,
686692
Analog Ground Start, 850
Analog Loop Start, 850
analyzing collected data
case study, 1819
CCM traces, 42, 5057
through MGCP T1 PRI gateways, 5860
CMI traces, 674679
deductive reasoning, 1112
ISDN traces, 258262

948 analyzing collected data

calling name display, 270

cause codes, 262269
numbering type/plan mismatches,
269270
timer information, 271276
locations-based CAC trace information,
628631
SDL traces, 6063
verifying IP network integrity, 1213
Anlagenanschluss, 213
ANS (answer tone), 439
ANSI (American National Standards Institute) web
site, 849
appearances, held calls, 524
applications
CallManager Serviceability, 82
alarms, 82
Control Center, 85
RTMT, 8588
Service Activation, 84
traces, conguring, 83
CAR, 90
CCC, 790791
CCEmail
alerting methods, 81
conguring, 7680
Cisco Attendant Console, 779780
client, 781782
resolving common problems, 782784
server components, 780781
Cisco AVVID IP Telephony, 34
Cisco ER, 791
Cisco IP SoftPhone, 786788
Cisco Personal Attendant, resolving call routing
problems, 785786
CRA, 736
CTI, 736
Dick Tracy, 101104
CLI/embedded Tracy, 105
directory-enabled, 819820
Enhanced Q.931 Translator, 98100
Event Viewer, 91
alarm denitions, 9293
PerfMon
alerts, 75
counter logging, 7175

versus RTMT, 6869

viewing real-time statistics, 6971
Q.931 Translator, 9597
VNC, 108
Windows Terminal Services, 107
applying transformations
cumulative effect, 497499
order of application, importance of, 496497
area codes
blocking, 548549
versus local area code, 510
ART (Administrative Reporting Tool), 795
assigning calling search spaces to devices, 471473
audio sources (MOH), 601603
Audio Translator, troubleshooting, 618
live
selecting recording input, 620
troubleshooting, 619620
mulitcast
troubleshooting, 616
versus unicast, 615
Audio Translator, troubleshooting, 618
automatic alternate routing (AAR), 478
automatic time synchronization, conguring on
CallManager servers, 39
auto-registration, controlling with PLAR, 545546

B
backhauling, 553
on MGCP PRI gateways, 256258
backup CallManager, 154
BackupCallManagerName parameter (CMI), 667
bandwidth requirements, locations-based CAC,
624626
BaudRate parameter (CMI), 667
best practices, troubleshooting Cisco IP Phones,
165166
binary values, converting to decimal and
hexadecimal values, 881889
bit masks, 63
configuring for SDL traces, 6367
blind transfers, 529531
blocked calls, 473
blocking area codes, 548549
buffering delay. See queueing delay

calling search spaces

Bug Toolkit, 106107

busy calls, forwarding, 480

C
CAC (call admission control), 24, 623
gatekeeper CAC, 638
call setup, 647651
CallManager registration, 645647
RAS messages, 639
verifying conguration, 640645
locations-based, 623624
bandwidth requirements, 624626
call preservation, 636637
CCM traces, enabling, 626
conference bandwidth reservations,
633635
conguring, 630
detecting bandwidth leaks, 635636
location identiers, 628
MOH bandwidth reservations, 631633
regions, 627629
trace information, analyzing, 628631
call admission control. See CAC
call control, CCAPI debug commands, 196205
call forwarding, 479
CFA, 480485
restricting, 546547
CFB, 480
CFF, 485486
CFNA, 479480
to voice mail, reading CMI traces, 678
call history information messages (SMDI), 665
call hold feature (CallManager), 522529
call legs, 175. See also dial peers
call park feature (CallManager), 531
troubleshooting, 532533
call pickup feature (CallManager), troubleshooting,
533538
call preservation
locations-based CAC, 636637
SRST, 562
troubleshooting, 561

949

call routing
called party transformations, effect on, 513514
Cisco Personal Attendant, 785786
closest-match routing, 461464
unexpected outside dial tone,
troubleshooting, 465466
dial peers
call legs, 175
destination-pattern parameter, 176179
incoming called number command,
181184
matching, 175
optional parameters, 179181
NANP, 857879
pattern matching
blocked calls, 473
multiple partitions within calling search
space, 474475
problem resolution methodology, 515516
reading CCM traces, 516521
route patterns
urgent priority, 502
wildcards, 460461
toll fraud, preventing, 544549
translation patterns, 501506
call setup, gatekeeper CAC, 647651
call statistics menu (Cisco IP Phones), 165
call transfer feature (CallManager), 529531
called party transformations
effect on call routing, 513514
masks, 495496
cumulative effect of changes, 497499
order of application, 496497
overriding, 499
CallerID service parameter transformation, 500
calling party transformations, 513514
masks, 495496
cumulative effect of changes, 497499
order of application, 496497
overwriting, 499
calling search spaces, 469473
AAR, 637
applying to voice mail systems, 547
call forwarding, 479
CFA, 480485, 546547
CFB, 480

950 calling search spaces

CFF, 485486
CFNA, 479480
device-level, 476477
event-specific, 478
line-level, 476477
multiple partitions, pattern-matching rules,
474475
CallManager. See also CallManager Serviceability
audio sources
mulitcast versus unicast, 615
selecting recording input, 620
troubleshooting live sources, 619620
call hold feature, 522529
call park feature, 531533
call pickup feature, 533538
call processing messages, 140, 144147
call transfer feature, 529531
calling IP phone interaction, 150
Cisco AVVID IP Telephony call processing, 24
centralized deployment model, 26
distributed deployment model, 27
multiple-site deployment model, 25
single-site deployment model, 24
closest-match routing, 461464
unexpected outside dial tone,
troubleshooting, 465466
Database Layer Monitor
CDR replication, troubleshooting,
813815
verifying operation, 812813
delayed routing, 466469
digit analysis behavior, 463464
embedded LDAP directory, 823825
logon failures, troubleshooting, 827
reconguring on Publisher server,
828835
reconguring on Subscriber server,
835837
endpoints, 551
MOH, troubleshooting, 611615
nonsurvivable endpoints, 557
CTI/TAPI endpoints, 559
H.323 gateways, 558559
Skinny gateways, 557

object counters, 893898

Cisco CallManager Attendant Console
object counters, 898900
Cisco CallManager System Performance
object counters, 900902
Cisco CIT Manager object counters, 903
partitions, 470
service parameters, transformations, 500501
survivable endpoints, 552
IP Phones, 552553
MGCP gateways, 553557
TOH, 602
investigating instances of, 617
trace files
analyzing SCCP messages, 148154
call state eld values, 525
conguring in CallManager
Serviceability, 4250
digit analysis results, 149
elds, 4446
for MGCP T1 PRI gateways, 5860
MOH, troubleshooting, 608611
reading, 42, 5057
reviewing for call routing problems,
516521
unregistered IP Phones, troubleshooting
checking inline power, 114117
verifying network connectivity, 117127
CallManager Serviceability, 82
alarms, 82
configuring CCM traces, 4250
Control Center, 85
RTMT, 85
CTI Apps tab, 88
Devices tab, 86
Performance tab, 86
Service Activation, 84
traces, configuring, 83
CallManagerName parameter (CMI), 667
capability bits, 200
DTMF relay, 202
fax, 201
capturing IP IVR/AA traces, 748752
CAR (CDR analysis and reporting), 90
case studies
data analysis, 1819
data collection, 1418

Cisco CallManager System Performance object counters

Catalyst 4000 series switches

AGM, hardware conferencing, 587
Catalyst 4224 switch, voice gateway
functionality, 173174
Catalyst 6000 series
6608 T1/E1 modules
conguring, 325337
D-channel establishment, 337, 340359
T1 CAS, troubleshooting, 359367
6608/6624 voice gateway modules
DHCP, troubleshooting, 314320
powering up, 313314
registration, troubleshooting, 324325
TFTP, troubleshooting, 320324
6624 FXS Analog Interface Module,
configuring, 367379
CMM switch, voice gateway functionality, 174
CatOS switches, time synchronization, 41
CCAPI (call control application programming
interface) debugs, 196205
CCC (Cisco Conference Connection), 790791
CCEmail
alerting methods, 81
configuring, 7680
CCMAdmin
reset command, 156
restart command, 156
CDCC (Call Dependent Call Control) processes,
tracing locations-based CAC, 626
CDR Time Converter, 9091
CDRs (call detail records), 89
CAR, 90
configuring Subscriber replication, 810812
replication, troubleshooting, 813815
storing in Publisher server, 795
timestamps, 9091
centralized CallManager architecture, locationsbased CAC, 26, 623624
AAR, troubleshooting, 637
analyzing, 628631
bandwidth requirements, 624626
call preservation, 636637
CCM traces
analyzing, 628631
enabling, 626

951

conference bandwidth reservations, 633635

configuring, 627631
detecting bandwidth leaks, 635636
MOH bandwidth reservations, 631633
regions, 627
CFA (call forward all), 480485
restricting, 546547
CFB (call forwardbusy), 480
CFF (call forward on failure), 485486
CFNA (call forward no answer), 479480
CgpnScreeningIndicator service parameter
transformation, 500
choppy voice quality, sources of
packet drops, 397400
queuing delay, 401
VAD, 402404
Cisco 7910 IP Phone, 32
Cisco 7914 IP Phone Expansion Module, 32
Cisco 7935 IP Conference Station, 32
Cisco 7960/7940 IP Phones, 31
Cisco Attendant Console, 779780
client, 781782
resolving common problems, 782784
server components, 780781
Cisco AVVID IP Telephony
applicaitons, 34
call processing
centralized deployment model, 26
distributed deployment model, 27
multiple-site deployment model, 25
single-site deployment model, 24
clients, 2931
Cisco 7910 IP Phone, 32
Cisco 7914 IP Phone Expansion
Module, 32
Cisco 7935 IP Conference Station, 32
Cisco 7960/7940 IP Phones, 31
IP Telephony infrastructure, 2324
network infrastructure, 23
voice gateways, 32
Cisco CallManager Administration, viewing Route
Plan Report, 466
Cisco CallManager Attendant Console object
counters, 898900
Cisco CallManager System Performance object
counters, 900902

952 Cisco CIT Manager object counters

Cisco CIT Manager object counters, 903

Cisco Customer Directory Conguration plugin,
troubleshooting installation, 839843
Cisco DPA 7630 voice mail gateway, 702
Octel voice mail system integration with
CallManager, 693, 697703
MWI problems, troubleshooting, 702
port statuses, 700702
verifying cabling, 693
Cisco ER (Emergency Responder), 478, 791
Cisco Gatekeeper object counters, 904
Cisco H.323 object counters, 904
Cisco HW Conference Bridge Device object
counters, 905
Cisco IOS Software
debugs, enabling, 185187
dial peers
call legs, 175
destination-pattern parameter legs,
176179
incoming called number command,
181184
matching, 175
optional parameters, 179181
Cisco IOS voice gateways, 169
2600 series routers, 171172
3600 series routers, 172
3700 series routers, 173
digital interfaces
ISDN PRI signaling, 210214
T1 CAS, 214218
timestamps, configuring, 185
verifying physical layer connectivity,
208210
eliminating sources of echo, 421424
H.323, 281
H.225 call ow, 288294
H.225 signaling, 283284
H.245 call signaling, 295307
IEs, 284287
MGCP
cause codes (traces), 262269
commands, 219221
DTMF packages, 231232
DTMF trunk packages, 236237
endpoint identiers, 218219
FXO/FXS signaling, 249256

generic media packages, 231

handset emulation packages, 235236
line packages, 234235
MF packages, 232238
packages, 229230
parameter lines, 221229
PRI backhaul, 256258
numbering type/plan mismatches,
269270
reading ISDN traces, 258262
response codes, 239240
response headers, 238
RTP packages, 236
T1 CAS, 276281
timers, 271276
trunk packages, 233
verifying registration status, 240249
resolving one-way/no-way audio problems,
407410
TDM interfaces, troubleshooting, 187
with debug commands, 192205
with show commands, 187192
VG200, 170
Cisco IP Phone Services SDK, 822
Cisco IP Phones
79xx series
3-port switch operation, 161164
call processing messages, 140, 144148
network settings, 123126
SCCP, troubleshooting, 139140
active connections, 155
best practices for troubleshooting, 165166
directory problems, troubleshooting, 160161
dropped calls, troubleshooting, 157
failback, 156
failover, 155
troubleshooting, 158160
resetting, 156
restarting, 156
service problems, troubleshooting,
160161, 789
Skinny client registration process
messages, 127132
verifying with status messages, 133135
soft keys, 147
TCP handle, deriving from CCM traces, 148
Temporary Failure messages, 561562

commands

Cisco IP SoftPhone, 786788

eliminating sources of echo, 428429
Cisco IP/VC products, 789
Cisco Lines object counters, 905
Cisco Locations object counters, 906
Cisco Media Streaming App object counters,
906909
Cisco Media Termination Point object counters,
909910
Cisco Messaging Interface object counters, 910911
Cisco MGCO FXI Device object counters, 911
Cisco MGCO FXS Device object counters, 912
Cisco MGCP Gateways object counters, 912
Cisco MGCP PRI Device object counters, 913914
Cisco MGCP T1 CAS Device object counters,
914915
Cisco MOH Device object counters, 915918
Cisco MTP Device object counters, 916
Cisco Personal Attendant, resolving call routing
problems, 785786
Cisco Phones object counters, 918
Cisco SW Conference Bridge Device object
counters, 918920
Cisco TFTP object counters, 920923
Cisco Transcoder Device object counters, 923
Cisco Unity, 655
DTMF, 661662
MWI, 659661
switch configuration, verifying, 658659
troubleshooting resouces, 662
TSP
compatibility, verifying, 655656
conguring, 656657
Cisco WebAttendant. See Cisco Attendant Console
Cisco WS-X6608 gateway, eliminating sources of
echo, 424427
CLI Tracy, 105
clients
Cisco 7910 IP Phone, 32
Cisco 7914 IP Phone Expansion Module, 32
Cisco 7935 IP Conference Station, 32
closest-match routing, 461464
unexpected outside dial tone, troubleshooting,
465466

953

clusters
database replication, Publisher-Subscriber
model, 793796
intercluster trunks, 311
codec mismatches, 312
master/replica relationship, 823
passwords, configuring on nodes, 798 802
CM Down, Features Disabled message (Cisco IP
Phones), 158
CMI (Cisco Messaging Interface), 666667
service parameters, 667671, 674
traces, reading, 674679
troubleshooting with HyperTerminal, 679682
CMRs (Call Management Records), 89
codec complexity, 171
codecs
CallManager selection process, 568
capability bits, 200
configuring between regions, 569
GSM, 855
transcoding, 565
wideband, 855
coder delay, isolating, 386
collecting data, 4
analyzing, 11
CCM traces, 42, 5057
CMI traces, 674679
ISDN traces, 258262
locations-based CAC traces,
628631
SDL traces, 6063
case study, 1418
IP IVR/AA traces, 748752
isolating root cause of problems, 6
deductive reasoning, 1112
earliest occurence of problem, referencing
device-based time, 1011
with topology information, 79
user information, 10
verifying IP network integrity, 1213
comfort noise, 402
commands
debug ephone, 713
debug ephone detail, 723728
debug ephone register, 714717
debug ephone state, 719722
debug vstp tone, 192196

954 commands

fax interface-type, 454

fax nsf, 453
fax rate, 451
fax-relay ecm disable, 452
frame-clock-select, 210
incoming called number, 182184
show call active voice, 191, 404
show call active voice brief, 449
show ephone, 720
show gatekeeper calls, 649
show gatekeeper endpoints, 645
show gatekeeper zone status, 649
show voice port summary, 188189
conferencing
Ad Hoc, error messages, 597598
failures, troubleshooting, 592597
conguration parameters, VG248, 686690
conguring
6608 T1/E1 digital gateway, 325337
D-channel establishment, 337, 340359
T1 CAS, troubleshooting, 359367
6624 FXS Analog Gateway, 367379
CallManager Serviceability
alarms, 82
CCM traces, 4250
Service Activation, 84
traces, 83
CCEmail, 7680
CMI, service parameters, 667671, 674
codecs between regions, 569
CRA, LDAP directories, 741745
dial peers, 176177
incoming called number command,
181184
optional parameters, 179181
variable-length matching, 178179
extension mobility, 758763
fax/modem passthrough on WS-X6608
port, 441
locations-based CAC, 627631
MWI, parameters, 682685
passwords on cluster nodes, 798802
regions, 571
SDL traces, 6367

SRST, 709712
DHCP support, 732
transfer patterns, 730
Subscriber CDR replication, 810812
connectivity
troubleshooting unregistered Skinny clients,
117127
conguration les, 121127
IP addressing, 118121
VLAN conguration, 118
verifying, 1213
Control Center (CallManager Serviceability), 85
converting decimal values
to binary, 881889
to hex, 881889
CoR (class of restriction), 708
corporate directories
Cisco IP phone directory integration, 820
LDAP integration
with Active Directory, 837839
with Netscape iPlanet, 844
providing endpoint access, 821823
troubleshooting, 823
counters
Cisco analog access, 892
Cisco CallManager Attendant Console object,
898900
Cisco CallManager object, 893898
Cisco CallManager System Performance object,
900902
Cisco CTI Manager object, 903
Cisco Gatekeeper object, 904
Cisco H.323 object, 904
Cisco HW Conference Bridge Device
object, 905
Cisco Lines object, 905
Cisco Locations object, 906
Cisco Media Streaming App object, 906909
Cisco Media Termination Point object,
909910
Cisco Messaging Interface object, 910911
Cisco MGCP FXO Device object, 911
Cisco MGCP FXS Device object, 912
Cisco MGCP Gateways object, 912
Cisco MGCP PRI Device object, 913914
Cisco MGCP T1 CAS Device object, 914915
Cisco MOH Device object, 915918

dejitter delay, isolating source of

Cisco MTP Device object, 916

Cisco Phones object, 918
Cisco SW Conference Bridge Device object,
918920
Cisco TFTP object, 920923
Cisco Transcoder Device object, 923
enabling logging on PerfMon, 7175
Windows 2000 objects, 924925
CRA (customer response application), 736
AA, 737
compatibility with CCM, verifying, 737
extension mobility
conguring, 759763
login/logout process, 763765
resolving common problems,
765768, 772
LDAP directory, configuring, 741745
CRA Administration, 738741
engine status, verifying, 745748
CTI (Computer Telephony Interface)
applications, 736
CRA
AA, 737
extenion mobility, 759769, 772
IVR, 737
LDAP directory, conguring, 741745
CRA Administration, troubleshooting, 738741
CTI Manager, 738
nonsurvivable CTI/TAPI endpoints, 559
verifying TSP version, 736
CTI Apps tab (RTMT), 88
CTI Manager, 738
CTIQBE (Computer Telephony Interface Quick
Buffer Encoding), 736
cumulative transformations, 497499
Customer Directory Conguration plugin,
troubleshooting installation, 839843

D
data analysis, 45
case study, 1419
CCM traces, 42, 5057
through MGCP T1 PRI gateways, 5860
CMI traces, 674679
deductive reasoning, 1112

ISDN traces, 258262

calling name display, 270
cause codes, 262269
numbering type/plan mismatches,
269270
timer information, 271276
isolating root cause, 6
with topology information, 79
locations-based CAC trace information,
628631
SDL traces, 6063
user information, 10
verifying IP network integrity, 1213
Database Layer Monitor
troubleshooting CDR replication, 813815
verifying operation, 812813
database replication
name resolution, 796797
passwords, changing, 798802
Publisher-Subscriber model, 793796
troubleshooting with Microsoft SQL Server
Enterprise Manager, 802803
DataBits parameter (CMI), 667
DC Directory, 823825
logon failures, troubleshooting, 827
reconfiguring
on Publisher server, 828835
on Subscriber server, 835837
DC Directory Administrator, launching, 826
DDIs (digit discard instructions), 486494
alternate expansion of DDI acronym, 486
debug ephone command, 713
debug ephone detail command, 723728
debug ephone register command, 714717
debug ephone state command, 719722
debug vtsp tone command, 192196
debugs
enabling on Cisco IOS Software, 185187
SRST call control, 719720
decimal values, converting to hexadecimal and
binary values, 881889
deductive reasoning, 1112
default MRGL, 568
dejitter delay, isolating source of, 393395

955

956 delay

delay, 384. See also echo

effect on signaling, 395396
isolating sources of
dejitter delay, 393395
xed delay, 385389
variable delay, 390395
delayed routing, troubleshooting, 466469
Delayed Start (E&M), 850
deriving TCP handles of Cisco IP Phones from
CCM traces, 148
destination-pattern parameter (dial peers), 176179
developing phone services, 789
DEVICE_RESET message (SCCP), 157
DEVICE_RESTART message (SCCP), 157
device-level calling search spaces, 476477
devices
Cisco IOS Software gateways
eliminating sources of echo, 421424
resolving one-way/no-way audio
problems, 407410
Cisco IP SoftPhones, eliminating sources of
echo, 428429
codecs
CallManager selection process, 568
capability bits, 200
conguring between regions, 569
G.711, 855
G.723, 855
G.726, 855
G.729, 855
G.729a, 855
G.729ab, 855
G.729b, 855
transcoding, 565
echo cancellers, 384
operation of, 416418
fax machines, 433
encoding schemes, 434
fax/modem passthrough, 437439
isolating problems, 449450
jitter, 446
negotiation, 434
NSF eld, modifying, 453
packet loss, 446
page transmission, 434
passthrough, 440

physical layer errors, troubleshooting,

447449
switching fax protocol, 454
T.30 transmissions, 435437
media resources, 565
modems
passthrough, 439
physical layer errors, troubleshooting,
447449
MOH fixed audio sources, verifying
configuration, 619620
time synchronization
CatOS, 41
Cisco IOS, 4041
transcoders, 571577
out-of-resource conditions, 578580
with conference bridge resources,
581585
with MOH servers, 585
Devices tab (RTMT), 86
DHCP (Dynamic Host Conguration Protocol), 850
troubleshooting on 6608/6624 modules,
314320
dial peers
answer-address command, 182
call legs, 175
callID assignment, 199200
capabilities, 200203
inbound, disconnects, 204
tear down, 204205
destination-pattern parameter, 176179
inbound peer matching, 181182
based on port conguration, 183
incoming called-number command, 181184
interdigit timeout, 177178
longest-match routing, 177
optional parameters, 179181
outbound peer matching, 182183
peer ID 0, characteristics, 183184
POTS, 175
as destination, 179
optional parameters, 180181
priority, assigning, 177
session-target ipv4 command, 179
temporary dial peers, viewing, 719
variable-length destination patterns, 178

embedded LDAP directory

viewing configuration with CCAPI debugs,

197200
VoIP, 175
Dial Plan Path service parameter
transformation, 501
dialing forest traces, 538542
verbose mode, 543
dialing transformations. See transformations
DialingPlan parameter (CMI), 667
Dick Tracy, 101104
acquiring, 105
CLI/embedded Tracy, 105
digit analysis, 461464
CCM trace results, 149
dialing forest traces, 538542
verbose mode, 543
identifying potential route pattern matches
within partitions, 517521
digital integration, Octel voice mail systems and
CallManager, 693, 698700
digital interfaces
ISDN PRI signaling, 210212
conguring, 213214
physical layer, verifying connectivity, 208210
T1 CAS, 214218
directories, LDAP
access, 820
integration, 820
schema, 819
Directories button (Cisco IP Phones), request
failures, 160161
directory-enabled Cisco applications, 819
disabling ECM on fax relay, 452453
disconnected FXO interfaces, troubleshooting, 205
displaying
e-phone-dn configuration, 718
SRST polling statistics, 722
distributed CallManager architecture, gatekeeper
CAC, 623, 638
call setup, 647651
CallManger registration, 645647
RAS messages, 639
verifying configuration, 640645
distributed deployment model, CallManager, 27
distribution agent, 804
DNS name resolution in database replication, 798

957

DPA voicemail gateway

event logging, 703
Octel/CallManager integration, 693, 697703
MWI problems, troubleshooting, 702
port statuses, 700702
verifying cabling, 693
dropped calls, 551552
media processing resources, 560
troubleshooting, 157, 561
DSPs (digital signal processors), codec
complexity, 171
DT-24+/DE-30+ gateways, eliminating sources of
echo, 424427
DTMF (Dual-Tone MultiFrequency) tones,
importance to voice mail systems, 661662
DTMF packages (MGCP), 231232
trunk packages, 236237
DTMF relay (H.245), 303307
Dynamic Host Conguration Protocol. See DHCP

E
E&M Delayed Start, 850
echo
eliminating sources of, 418429
perception of as problem, 414416
sources of, isolating, 411
acoustic echo, 412
electrical echo, 411412
echo cancellers, 384
operation, 416418
ECM (error control mode), disabling on fax relay,
452453
EIA/TIA web site, 849
EIGRP (Enhanced Interior Gateway Routing
Protocol), 850
electrical echo, isolating, 411412
eliminating
possible causes using deductive reasoning,
1112
sources of echo, 418429
embedded LDAP directory, 823825
logon failures, troubleshooting, 827
reconfiguring
on Publisher server, 828835
on Subscriber server, 835837

958 embedded Tracy tool

embedded Tracy tool, 105

empty capabilities set, 565
enabling
debugs on Cisco IOS Software, 185187
fax relay debugs, 455456
H.323 Fast Connect, 396
encoding schemes, fax machines, 434
endpoint directory access, 821823
endpoint identiers (MGCP), 219
endpoints
nonsurvivable, 557
CTI/TAPI endpoints, 559
H.323 gateways, 558559
Skinny gateways, 557
survivable, 552
IP Phones, 552553
MGCP gateways, 553557
end-to-end delay, ITU-T specications, 384
Enhanced Interior Gateway Routing Protocol
(EIGRP), 850
Enhanced Q.931 Translator, 98100
Enterprise Manager, troubleshooting database
replication errors, 802804
ephone-dn conguration, viewing, 718
Epoch time, 9091
ER (Emergency Responder), 478
ERL (echo return loss), 417
ERLE (echo return loss enhancement), 417
error codes, Extension Mobility (CallManager 3.3),
777779
error messages, SMDI, 666
event-specic calling search spaces, 478
exceeds maximum parties error messages, 597
extension mobility, 756758
CallManager 3.1/3.2, 756772
login/logout process, 763765
CallManager 3.3, 772773
error codes, 777779
login/logout process, 774777
configuring, 758763
resolving common problems, 765772

F
failback behavior in Cisco IP Phones, 156
failed conferences, troubleshooting, 592597
failover
behavior in Cisco IP Phones, 155
troubleshooting, 158160
Fast Connect, enabling, 396
Fax Group 3, 433
fax interface-type command, 454
fax machines, 433
encoding schemes, 434
isolating problems, 449450
jitter, 446
negotiation, 434
NSF field, modifying, 453
packet loss, 446
page transmission speed, 434
physical layer errors on digital interfaces,
447449
switching fax protocol, 454
T.30 transmissions, 435437
fax nsf command, 453
fax preamble, 440
fax rate command, 451
fax relay, 444445
adjusting data rate, 451452
debugs, enabling, 455456
ECM, disabling, 452453
switching to fax passthrough, 450
T.38, 445446
troubleshooting, 450
fax/modem passthrough, 437, 440
NSE, 439
NTE, 438
troubleshooting, 450
verifying configuration, 441444
fax-relay ecm disable command, 452
features of CallManager
call hold, 522529
call park, 531533
call pickup, 533538
call transfer, 529531
elds of CCM traces, 4446
ltering CCM trace results, 4950
rewalls, resolving one-way/no-way audio
problems, 410

H.323

rmware, Cisco IP Phones, 165

xed delay, 384
coder delay, isolating, 386
effect on signaling, 395396
packetization delay, isolating, 386387
propagation delay, isolating, 389
serialization delay, isolating, 387389
sources of, isolating, 385
formatting
called/calling party numbers with
transformations, 501506
called/calling party tranformations with masks,
495496
cumulative effect of changes, 497499
order of application, 496497
forwarding in SRST mode, 731
frame-clock-select command, 210
FXO interface
disconnects, 205
supervisory disconnect tone, 207
FXO/FXS signaling on MGCP gateways, 249256
FXS (Foreign Exchange Station) gateways, applying
restrictive calling search spaces, 547548

G
G.711 codecs, 855
fax passthrough, 437
G.723 codecs, 855
G.726 codecs, 855
G.729 codecs, 855
G.729a codecs, 855
G.729ab codecs, 855
G.729b codecs, 855
garbled audio, sources of
packet drops, 397400
queuing delay, 401
VAD, 402404
gatekeeper CAC, 638
call setup, 647651
CallManager registration, 645647
RAS messages, 639
verifying configuration, 640645

959

gateways, Cisco IOS MGCP

FXO/FXS signaling, 249256
PRI backhaul, 256258
reading ISDN traces, 258276
T1 CAS, 276281
verifying registration status, 240249
gathering data, 4
analyzing collected data
case study, 1819
CCM traces, 42, 5060
CMI traces, 674679
deductive reasoning, 1112
ISDN traces, 258262
case study, 1418
earliest occurence of problem, referencing
device-based time, 1011
isolating root cause, 69
user information, 10
verifying IP network integrity, 1213
generic media packages (MGCP), 231
Group 3 fax devices, 433
group pickup, 533538
GSM (Global System for Mobile Communications)
codecs, 855

H
H.225 signaling, 283, 850
call flow, 288294
call setup messages, 283284
H.245, 851
call signaling, 295
DTMF relay, 303307
logical channel signaling, 300303
maser/slave determination, 296
terminal capabilities exchange, 297300
H.323, 281, 851
gatekeepers, 638
H..225 signaling
call ow, 288294
H..245 signaling, 295
DTMF relay, 303307
logical channel signaling, 300303
master/slave determination, 296
terminal capabilities exchange, 297300

960 H.323

H.225 signaling, 283

call setup messages, 283284
IEs, 284287
nonsurvivable endpoints, 558559
null capabilities set, 565
versus MG CP, 281
H.323 Fast Connect, enabling, 396
handset emulation packages (MGCP), 235236
hardware conferencing
No Conference Bridge Availablemessages,
587591
Catalyst 4000 AGM, 587
held calls, 522525
held party, 602
hexadecimal conversion table, 881889
high complexity calls, 171
high-compression codecs, 437
fax relay, 444445
T.38, 445446
holding party, 602
hub-and-spoke topology, locations-based CAC, 624
AAR, troubleshooting, 637
analyzing trace information, 628631
bandwidth requirements, 624626
call preservation, 636637
CCM traces, enabling, 626
conference bandwidth reservations, 633635
configuring, 627631
detecting bandwidth leaks, 635636
location identifier assignments, 628
MOH bandwidth reservations, 631633
regions, 627
HyperTerminal, troubleshooting CMI problems,
679682

I
i button (Cisco IP Phones), logging call
statistics, 165
identifying root cause of problems, 69
IEEE (Institute of Electrical and Electronic
Engineers) web site, 849
IEs (information elements), 284287
Immediate Start (E&M), 850

inbound call legs, 175

incoming called-number command, dial peer
conguration, 182184
initiating transactional replication, 804
inline power problems (Cisco IP Phones),
troubleshooting unregistered Skinny clients,
114117
InputDnSignicantDigits parameter (CMI), 668
inside dial tone, 465
installation of Cisco Customer Directory
Conguration plugin, troubleshooting,
839843
intercluster trunks, 311
codec mismatches, 312
interdigit timeout, adjusting, 466467
international numbers, preventing unauthorized
access, 545
Intl TollBypass DDI, 488
Intl TollBypass Trailing-# DDI, 488
investigating sources of delay, 385
fixed delay, 385
coder delay, 386
packetization delay, 386387
propagation delay, 389
serialization delay, 387389
variable delay
dejitter delay, 393395
low-speed links, 391393
queuing delay, 390391
IP addressing
SRST, DHCP support, 732
resolving one-way/no-way audio problems,
405406
verifying IP Phone configuration, 118121
IP IVR traces, collecting, 748752
IP Phones
auto-registration, controlling, 545546
call forward fields, 479
CFA, 480485
CFB, 480
CFF, 485486
CFNA, 479480
directory access, 820

local area code versus area code

extension mobility, 756758

conguring, 758763
login/logout process, 763765
resolving common problems,
765768, 772
Skinny client registration
troubleshooting inline power, 114117
troubleshooting network connectivity,
117127
verifying, 133
IP Telephony infrastructure, call processing, 24
centralized deployment model, 26
distributed deployment model, 27
multiple-site deployment model, 25
single-site deployment model, 24
IP/VC products, 789
iPlanet (Netscape), LDAP integration, 844
IPV MSApp (Cisco IP Voice Media Streaming
Application), software conferencing, 586
ISDN (Integrated Services Digital Network)
Anlagenanschluss, 213
PRI signaling, 210212
conguring on Cisco IOS voice gateways,
213214
traces, reading from MGCP gateways, 258276
isolating
fax problems, 449450
root cause of problems, 6
case study, 16
with topology information, 79
sources of echo, 411
acoustic echo, 412
electrical echo, 411412
sources of fixed delay, 385
coder delay, 386
packetization delay, 386387
propagation delay, 389
serialization delay, 387389
sources of variable delay
dejitter delay, 393395
low-speed links, 391393
queuing delay, 390391
voice quality problems
packet drops, 397400
queuing delay, 401
VAD, 402404

961

ITS (IOS Telephony Services), 707

ITU-T
H.225 specification, 651
Recommendation G.114, delay
specifications, 384
web site, 849
IVR (Integrated Voice Response) scripts, 737

J
jitter
effect on fax machines and modems, 446
isolating source of, 391392
JTAPI (Java Telephony Application Programming
Interface), 852
verifying CRA engine status, 745748

K-L
KeepAliveDn parameter (CMI), 668
LDAP (Lightweight Directory Access Protocol), 852
Active Directory integration, 837839
corporate directory access, 821823
Customer Directory Conguration plugin,
troubleshooting, 839844
directories, 819
conguring, 741743
verifying conguration, 745
directory integration versus directory
access, 820
embedded directories, 823825
logon failures, troubleshooting, 827
reconguring on Publisher server,
828 835
reconguring on Subscriber server,
835837
iPlanet integration, 844
LFI (link fragmentation and interleaving), 391
line packages (MGCP), 234235
line-level calling search spaces, 476477
listener echo, isolating sources of, 412413
live audio sources, troubleshooting, 619620
LMHOSTS le, name resolution, 796797
local area code versus area code, 510

962 local calls, delayed routing

local calls, delayed routing, 466469

locating alarm denitions, 9394
locations-based CAC, 623624
bandwidth requirements, 624626
call preservation, 636637
CCM traces, enabling, 626
conference bandwidth reservations, 633635
configuring, 627631
detecting bandwidth leaks, 635636
location identifier assignments, 628
MOH bandwidth reservations, 631633
regions, 627
trace information, analyzing, 628631
log reader agent, 804
logging call statistics on Cisco IP Phones, 166
logical channel signaling (H.245), 300303
login/logout process
Extension Manager (CallManager 3.3),
774777
extension mobility, 763765
low-speed links, isolating delay source, 391393

M
manual time synchronization, conguring on
CallManager servers, 40
masks, 495496
master/replica relationship in clusters, 823
master/slave determination in H.245 call
signaling, 296
MatchingCgpnWithAttendantFlag service parameter
transformation, 500
MCM (Multimedia Conference Manager), 638
media processing resources, 560
media resource group lists (MRGLs), 566, 602
media resource groups (MRGs), 566, 602
media resources, 565
selecting, 567
medium complexity calls, 171
Meet-Me conferences, 565
locations-based CAC bandwidth reservations,
633635
Message Waiting Indicator On/Off Messages
(SMDI), 665

MessageDeskNumber parameter (CMI), 668

messages
CM Down, Features Disabled,
troubleshooting, 158
in H.225 call setup, 283284
IP Phone status, verifying registration, 133135
SCCP, 140, 144147
in CCM traces, 148154
Skinny client registration process, 127132
SMDI, 664666
T.30, 435437
Temporary Failure, troubleshooting dropped
calls, 561562
messaging. See voice mail systems
methodology for resolving call routing problems,
515516
reading CCM traces, 516521
MF packages (MGCP), 232233
MF trunk packages (MGCP), 237238
MGCP (Media Gateway Control Protocol), 852
commands, 219221
endpoint identifiers, 218219
packages, 229230
DTFM package, 231232
DTMF trunk package, 236237
generic media package, 231
handset emulation package, 235236
line package, 234235
MF package, 232233
MF trunk package, 237238
RTP package, 236
trunk package, 233
parameter lines, 221229
response codes, 239240
response headers, 238
See also Cisco IOS MGCP gateways
MGCP gateways, survivable endpoints, 553557
MGCP T1 PRI gateways, tracing calls, 5860
Microsoft AD (Active Directory)
Customer Directory Configuration plugin,
troubleshooting, 839844
LDAP integration, 837839
Microsoft Event Viewer, 91
alarm definitions, 9293

MwiSearchSpace parameter (CMI)

Microsoft PerfMon, 68
alerts, 75
counter logging, 7175
versus RTMT, 6869
viewing real-time statistics, 6971
Microsoft SQL Server Enterprise Manager, 802803
Replication Monitor
correcting replication errors, 804806
reestablishing broken replication
subscription, 807809
reinitializing subscriptions, 809
miscongured 6608 T1/E1 modules,
troubleshooting, 326337
MIVR traces, capturing, 748752
models of Cisco 2600 series routers, 171172
modems
jitter, 446
packet loss, 446
passthrough, 437
ANS, 439
NSE, 439
NTE, 438
verifying conguration, 441444
physical layer errors, troubleshooting, 447449
modules
6608 T1/E1
advanced troubleshooting, 344359
conguring, 325337
D-channel establishment, 337, 340343
T1 CAS, troubleshooting, 359367
6608/6624 voice gateways
DHCP, troubleshooting, 314320
powering up, 313314
registration, troubleshooting, 324325
TFTP, troubleshooting, 320324
6624 Port FXS Analog Interface Module,
conguring, 367379
MOH (Music On Hold). See also TOH
audio sources, 601603
multicast versus unicast, 615616
selecting recording input, 620
Audio Translator, troubleshooting, 618
CAC bandwidth reservations, 631633
performance counters, 915918

963

troubleshooting, 611615
CCM trace les, 608611
performance counters, 605607
MOHAudioSourcesActive counter
(CallManager 3.3), 604
MOHConnectionsLost counter
(CallManager 3.3), 606
MOHConnectionState counter
(CallManager 3.3), 604
MOHHighestActiveResources counter
(CallManager 3.3), 607
MOHMulticastResourceActive counter
(CallManager 3.3), 606
MOHMulticastResourceAvailable counter
(CallManager 3.3), 607
MOHOutOfResources counter
(CallManager 3.3), 607
MOHStreamsActive counter (CallManager 3.3), 605
MOHStreamsAvailable counter
(CallManager 3.3), 605
MOHStreamsTotal counter (CallManager 3.3), 606
MOHTotalMulticastResources counter
(CallManager 3.3), 606
MOHTotalUnicastResources counter
(CallManager 3.3), 606
MOHUnicastResourceActive counter
(CallManager 3.3), 606
MOHUnicastResourceAvailable counter
(CallManager 3.3), 607
MRGLs (media resource group lists), 566, 602
MRGs (media resource groups), 566, 602
MTPs (media termination points), null capabilities
set, 565
multicast audio sources (MOH), troubleshooting,
615616
multiple-site deployment model (CallManager), 25
MWIs (Message Waiting Indicators), 709
configuration parameters, 682685
toggling on/off, 659, 661
VG248 platform, troubleshooting, 690692
MwiSearchSpace parameter (CMI), 668

964 name resolution

N
name resolution
LMHOSTS file, 796797
NetBIOS in database replication, 796798
NANP (North American Numbering Plan)
call routing information, 857879
route filters, 506510
multiple clauses, 512
tags, 507
NAT (Network Address Translation), resolving
one-way/no-way audio problems, 410
negotiation process, fax machines, 434
NetBIOS name resolution in database
replication, 796798
Netscape iPlanet, LDAP integration, 844
network diagrams, required information, 79
network hold MOH audio source, 601
network integrity, verifying, 1213
network settings, Cisco 79xx IP Phones, 123126
Network Time Protocol. See NTP
No Conference Bridge Available, troubleshooting,
587591
NoDigits DDI, 488
nonproduction hours, troubleshooting
methodologies, 56
nonsurvivable endpoints, 557
CTI/TAPI endpoints, 559
H.323 gateways, 558559
Skinny gateways, 557
no-way audio, isolating sources of
Cisco IOS Software gateways, 408410
firewalls, 410
IP connectivity, 405406
NAT, 410
PAT, 410
NSE (Named Service Event), 439
NSF (Nonstandard Facilities) eld, modifying, 453
NTE (Named Telephony Event), 438
NTP (Network Time Protocol), 852
time synchronization, 39
on CatOS devices, 41
on Cisco IOS devices, 4041
null capabilities set, 565

numbering plans
NANP, call routing information, 857879
route filters, 506507
multiple clauses, 512
NANP tags, 508510

O
object counters
Cisco analog access, 892
Cisco CallManager, 893898
Cisco CallManager Attendant Console,
898900
Cisco CallManager System Performance,
900902
Cisco CIT Manager, 903
Cisco Gatekeeper, 904
Cisco H.323, 904
Cisco HW Conference Bridge Device, 905
Cisco Lines, 905
Cisco Locations, 906
Cisco Media Streaming App, 906909
Cisco Media Termination Point, 909910
Cisco Messaging Interface, 910911
Cisco MGCP FXO Device, 911
Cisco MGCP FXS Device, 912
Cisco MGCP Gateways, 912
Cisco MGCP PRI Device, 913914
Cisco MGCP T1 CAS Device, 914915
Cisco MOH Device, 915918
Cisco MTP Device, 916
Cisco Phones, 918
Cisco SW Conference Bridge Device, 918920
Cisco TFTP, 920923
Cisco Transcoder Device, 923
logging on PerfMon, 7175
Windows 2000, 924925
obtaining
Dick Tracy tool, 105
Enhanced Q.931 Translator, 100
Octel voice mail systems, CallManager integration,
693, 698700
OffHookMessage message, SCCP call
processing, 144

performance counters

one-way audio, isolating sources of

Cisco IOS Software gateways, 408410
firewalls, 410
IP connectivity, 405406
NAT, 410
PAT, 410
open trees, 673
operating systems, Windows 2000
CCEmail, 7681
performance counters, 924925
operation of echo cancellers, 416418
optional dial peer parameters, 179181
OSI reference model, verifying connectivity at every
layer, 1213
OSPF (Open Shortest Path First), 852
outbound call legs, 175
out-of-resource conditions, 578580
OutputDnFor parameter (CMI), 668
OutputExternalFormat parameter (CMI), 669
OverlapReceivingForPriFlag service parameter
transformation, 501
overwriting transformations, 499

P
packages (MGCP), 229230
DTMF, 231232
DTMF trunkRTP, 236237
generic media, 231
handset emulation, 235236
line, 234235
MF, 232233
MF trunkRTP, 237238
RTP, 236
trunk, 233
packet drops
as source of voice quality degradation, 397400
effect on fax machines and modems, 446
packet-capture software, 106
packetization delay, isolating, 386387
page transmission speed, 434
parameter lines (MGCP), 221229
Parity parameter (CMI), 669
parked calls, 531
troubleshooting, 532533

965

partitions, 469470. See also calling search spaces

calling search spaces, AAR, 637
identifying potential route pattern matches,
517519
pattern-matching rules, 474475
passwords, conguring on cluster nodes, 798802
PAT, resolving one-way/no-way audio
problems, 410
pattern matching. See also calling search spaces
blocked calls, 473
closest-match routing, 461464
delayed routing, troubleshooting, 466469
multiple partitions within a calling search space,
474475
variable-length, 178179
wildcards, 460461
PBXs (private branch exchanges), troubleshooting
calling name display problems, , 270
perception of echo as problematic, 414416
PerfMon, 68. See also CCEmail
alerts, 75
counter logging, 7175
versus RTMT, 6869
viewing real-time statistics, 6971
performance counters
Cisco analog access, 892
Cisco CallManager, 893898
Cisco CallManager Attendant Console,
898900
Cisco CallManager System Performance,
900902
Cisco CIT Manager, 903
Cisco Gatekeeper, 904
Cisco H.323, 904
Cisco HW Conference Bridge Device, 905
Cisco Lines, 905
Cisco Locations, 906
Cisco Media Streaming App, 906909
Cisco Media Termination Point, 909910
Cisco Messaging Interface, 910911
Cisco MGCP FXO Device, 911
Cisco MGCP FXS Device, 912
Cisco MGCP Gateways, 912
Cisco MGCP PRI Device, 913914
Cisco MGCP T1 CAS Device, 914915
Cisco MOH Device, 915918
Cisco MTP Device, 916

966 performance counters

Cisco Phones, 918

Cisco SW Conference Bridge Device, 918920
Cisco TFTP, 920923
Cisco Transcoder Device, 923
logging on PerfMon, 7175
MOH, monitoring, 605607
Windows 2000, 924925
Performance tab (RTMT), 86
phone registration, SRST, 712717
phone services, 789
physical layer
connectivity, verifying on digital interfaces,
208210
troubleshooting fax/modem errors, 447449
pinpointing earliest occurence of problems, 10
referencing device-based time, 11
plain-text protocols, 218
PLAR (Private Line Automatic Ringdown),
controlling IP phone auto-registration, 545546
plugins, Cisco Customer Directory Conguration,
troubleshooting installation, 839843
polling statistics (SRST), viewing, 722
POTS dial peers, 175
variable-length pattern matching, 179
power denial, 206
powering 6608/6624 voice gateway modules,
313314
PreAt 10-10-Dialing DDI, 489
PreAt 10-10-Dialing Trailing-# DDI, 489
PreAt 11/10D->7D DDI, 490
PreAt 11/10D->7D Trailing-# DDI, 490
PreAt 11D->10D DDI, 491
PreAt 11D->10D Trailing-# DDI, 491
PreAt DDI, 489
PreAt Intl TollBypass DDI, 492
PreAt Intl TollBypass Trailing-# DDI, 492
PreAt Trailing-# DDI, 489
PreDot 10-10-Dialing DDI, 493
PreDot 10-10-Dialing Trailing-# DDI, 493
PreDot 11/10D->7D DDI, 493
PreDot 11/10D->7D Trailing-# DDI, 493
PreDot 11D->10D DDI, 494
PreDot 11D->10D Trailing-# DDI, 494
PreDot DDI, 487, 492
PreDot Intl TollBypass DDI, 494

PreDot Intl TollBypass Trailing-# DDI, 495

PreDot IntlAccess IntlDirectDial DDI, 494
PreDot Trailing-# DDI, 493
preventing
service-affecting problems, 56
toll fraud, 544549
PRI backhaul on MGCP gateways, 256258
primary CallManager, 154
processing delay, isolating, 386
production hours, troubleshooting
methodologies, 56
progress tones, 307
propagation delay, isolating, 389
Publisher server
DC, reconfiguring, 828835
Replication Monitor
correcting replication errors, 804806
reestablishing broken replication
subscription, 807809
reinitializing subscriptions, 809
Publisher-Subscriber model
database replication, 793796
name resolution
LMHOSTS le, 796797
NetBIOS, 796798
Subscriber server, configuring CDR
replication, 810812

Q
Q.850, 852
Q.921, 853
Q.931 Translator, 9597. See also Enhanced Q.931
Translator
queuing delay
as source of voice quality degradation, 401
isolating source of, 390391

RouteFilter parameter (CMI)

R
RAS (Registration, Admission, and Status)
messages, 639, 853
reading traces
CCM traces, 42, 5057
through MGCP T1 PRI gateways, 5860
CMI traces, 674679
ISDN traces from MGCP gateways, 258262
calling name display, 270
cause codes, 262269
numbering type/plan mismatches,
269270
timer information, 271276
SDL traces, 6063
Real-Time Monitoring Tool, monitoring MOH
performance counters, 605607
real-time statistics, viewing with PerfMon, 6971
reconguring DC
on Publisher server, 828835
on Subscriber server, 835837
recording input of live audio sources, selecting, 620
redirecting calls, group pickup, 533538
reestablishing broken replication subscription,
807809
regions, 571
codec configuration, 568569
codec matrix, 571577
configuring for locations-based CAC, 627
registration (Skinny clients)
608/6624 modules, 324325
checking phone status display, 133
inline power, troubleshooting, 114117
messages, 127132
network connectivity, 117, 120127
conguration les, 121127
IP addressing, 118121
VLAN conguration, 118
verifying with IP Phone status messages,
133135
reinitializing subscriptions, 809
remote access tools
VNC, 108
Windows Terminal Services, 107

967

replication
correcting with replication agents, 804806
name resolution, 796797
of CDRs
conguring, 810812
troubleshooting, 813815
passwords, configuring on cluster nodes,
798802
Publisher-Subscriber model, 793796
reestablishing broken subscription, 807809
troubleshooting with Microsoft SQL Server
Enterprise Manager, 802803
Replication Monitor
reestablishing broken replication subscription,
807809
reinitializing subscriptions, 809
troubleshooting replication errors, 804806
resetting
Cisco IP Phones, 156
NSF field, 453
resolving call routing problems, 515516
reading CCM traces, 516521
response codes, 239240
response headers, 238
restarting Cisco IP Phones, 156
restrictions of SRST, 708
ringback, troubleshooting absence of, 307
during call transfer, 309
on IP phones calling PSTN, 308
on PSTN phones calling IP phones, 309
RIP (Routing Information Protocol), 853
robbed-bit signaling, 214
rollover cables, 679
route lters, 506507
multiple clauses, 512
NANP tags, 508510
route patterns. See also translation patterns
closest-match routing, 461464
pattern-matching, delayed routing, 466469
urgent priority, 502
wildcards, 460461
Route Plan Report, viewing in Cisco CallManager
Administration, 466
RouteFilter parameter (CMI), 669

968 routers, voice gateway functionality

routers, voice gateway functionality

Cisco 2600 series, 171172
Cisco 3600 series, 172
Cisco 3700 series, 173
routing calls to voice mail (SRST), 731
RSVP (Resource Reservation Protocol), 853
RTMT (Real-Time Monitoring Tool), 85
CTI Apps tab, 88
Devices tab, 86
Performance tab, 86
verifying Skinny client registration, 135137
RTP (Real-Time Protocol), 853
dropped calls, 551552
packages (MGCP), 236
RUDP (Reliable User Datagram Protocol), 853

S
sa (system administrator) user account, changing
password, 802
sample CCM trace, 51
SCCP (Skinny Client Control Protocol), 139
messages
analyzing in CCM traces, 148154
call processing, 140, 144148
DEVICE_RESET, 157
DEVICE_RESTART, 157
Skinny client registration, 127135
608/6624 modules, 324325
checking phone status display, 133
conguration les, 121127
inline power, troubleshooting, 114117
IP addressing, 118121
messages, 127132
network connectivity, troubleshooting,
117, 120127
verifying with RTMT, 135137
verifying with status messages, 133135
scheduled outages, preventing service-affecting
problems, 56
SDI traces, reading, 42, 5057
SDKs, Cisco IP Phone Services, 822
SDL traces
configuring, 6367
reading, 6063
troubleshooting held calls, 527529

secondary CallManager, 154

selecting
appropriate troubleshooting tools, 13
MRGLs, 567
recording input for live audio sources, 620
serialization delay, isolating, 387389
SerialPort parameter (CMI), 669
Service Activation, conguring on CallManager
Serviceability, 84
service parameters
CMI, 667671, 674
transformations, 500501
VG248, 686690
service-affecting problems, preventing, 56
services button (Cisco IP Phones), request failures,
160161
SGCP (Skinny Gateway Control Protocol), 853
shared lines, calling search spaces, 477
show call active voice brief command, 449
show call active voice command, 191, 404
show ephone command, 720
show gatekeeper calls command, 649
show gatekeeper endpoints command, 645
show gatekeeper zone status command, 649
show voice port summary command, 188189
signaling
H.225, 283284
problem isolation, 18
silence suppression. See VAD
single-site deployment model (CallManager), 24
Skinny Client Control Protocol. See SCCP
Skinny clients, registration
608/6624 modules, 324325
checking phone status display, 133
conguration les, 121127
inline power, troubleshooting, 114117
IP addressing, 118121
messages, 127132
network conenctivity, troubleshooting,
117, 120127
verifying with RTMT, 135137
verifying with status messages, 133135
Skinny gateways, nonsurvivable endpoints, 557

StaionSetRing message, SCCP call processing

SMDI (Simple Message Desk Interface), 854

CallManager integration, 662666
messages, 664666
MWI, configuration parameters, 682685
VG248 SMDI integration, 686
conguration parameters, 686690
MWI problems, troubleshooting, 690692
snapshot agent, 804
sniffer traces, 106
soft keys, 147
events, 522523
software conferencing
No Conference Bridge Available messages,
troubleshooting, 587591
IPV MSApp, 586
sources of delay, investigating
fixed delay, 385
coder delay, 386
packetization delay, 386387
propagation delay, 389
serialization delay, 387389
variable delay
dejitter delay, 393395
low-speed links, 391393
queuing delay, 390391
sources of echo
eliminating, 418429
isolating, 411
acoustic echo, 412
electrical echo, 411412
SQL servers
database replication
changing passwords, 798802
name resolution, 796798
Microsoft SQL Server Enterprise Manager,
802803
Publisher-Subscriber model, 793796
SRST (Survivable Remote Site Telephony), 562
call control, debugging, 719720
call tranfer, debugging, 729730
configuring, 709712
CoR, 708
DHCP support, 732
ephone-dn configuration, viewing, 718
forwarding calls, 731
phone registration, 712717
polling, 722

969

restrictions, 708
routing calls to voice mail system, 731
transfer patterns, configuring, 730
SsapiKeepAliveInterval parameter (CMI), 669
standards
ITU-T H.225 specification, 651
standby CallManager, 154
StationActivateCallPlane message, SCCP call
processing, 144
StationAlarmMessage, 129
field definitions, 158160
StationCallInfo message, SCCP call processing, 145
StationCallState message, SCCP call
processing, 145
StationClearNotify message, SCCP call
processing, 146
StationClearPromptStatus message, SCCP call
processing, 146
StationCloseReceiveChannel message, SCCP call
processing, 146
StationConnectionStatisticsRequest message, SCCP
call processing, 146
StationConnectionStatisticsResponse message,
SCCP call processing, 147
StationDisplayNotify message, SCCP call
processing, 145
StationDisplayPromptStatus message, SCCP call
processing, 144
StationKeepAliveAck messages, 129
StationKeepAliveMsg, 129
StationKeypadButtonMessage message, SCCP call
processing, 144
StationOpenReceiveChannel message, SCCP call
processing, 146
StationOpenReceiveChannelAck message, SCCP
call processing, 146
StationOutputDisplayText message, SCCP call
processing, 144
StationRegisterAck messages, 129
StationRegisterMessage, 129
StationRegisterReject messages, 129
StationSelectSoftKeys message, SCCP call
processing, 144
StationSetLamp message, SCCP call
processing, 144
StationSetRinger message, SCCP call
processing, 145

970 StationSetSpeakerMode message, SCCP call processing

StationSetSpeakerMode message, SCCP call

processing, 146
StationSoftKeyEventMessage message, SCCP call
processing, 147
StationStartMediaTransmission message, SCCP call
processing, 146
StationStartTone message, SCCP call
processing, 144
StationStopMediaTransmission message, SCCP call
processing, 146
StationStopTone message, SCCP call
processing, 146
status messages (IP Phones), verifying Skinny client
registration, 133135
StopBits parameter (CMI), 669
StripPoundCalledPartyFlag service parameter
transformation, 501
Subscriber server, 794
CDR replication, configuring, 810812
DC, reconfiguring, 835837
subscriptions, reinitializing, 809
substrings, 507
tags, 507510
supervisory disconnect tone, 207208
survivable endpoints, 552
dropped calls, troubleshooting, 561562
IP Phones, 552553
MGCP gateways, 553557
switching. fax protocol, 454

T
T.30 fax transmissions, 435437, 854
T.38 fax relay, 445446, 854
T1 CAS (Channel Associated Signaling),
troubleshooting
on 6608 module, 359367
on Cisco IOS voice gateways, 214218
on MGCP-enabled ports, 276281
tags, 507510
tail circuits, 416
talker echo, isolating sources of, 412413
TAPI (Telephony Application Programming
Interface), 854

TCP (Transmission Control Protocol), 854

backhauling, 554
failback, 156
failover, 155
TCP handle, 52
TDM interfaces
ISDN PRI, 210212
conguring on Cisco IOS voice gateways,
213214
on Cisco IOS voice gateways, 187
debug commands, 192205
show commands, 187192
Telcordia web site, 849
temporary dial peers, viewing, 719
Temporary Failure messages (IP Phones), 561562
terminal capabilities exchange in H.245 call
signaling, 297300
terminal emulation, troubleshooting HyperTerminal
CMI problems, 679682
Terminal Services, 107
TFTP (Trivial File Transfer Protocol), 854
configuration files, 154
troubleshooting on 6608/6624 modules,
320324
third-party voice mail systems, applying restrictive
calling search spaces, 547548
time synchronization, 38
on CallManager servers, 3940
on CatOS devices, 41
on Cisco IOS devices, 4041
timestamps
configuring, 185
on CDRs, 9091
toggling MWI on/off, 659, 661
TOH (tone on hold), 602
investigating instances of, 617
toll fraud, preventing, 544549
topologies, required documentation, 9
traces. See also CDRs
CCM
analyzing SCCP messages, 148154
call state field values, 525
conguring for CallManager
serviceability, 4250
digit analysis results, 149
elds, 4446
reading, 42, 5057

urgent priority route patterns

reviewing for call routing problems,

516521
through MGCP T1 PRI gateways, 5860
CMI, reading, 674679
configuring
for locations-based CAC, 626
for CallManager Serviceability, 83
dialing forests, 538542
verbose mode, 543
IP IVR/AA, capturing, 748752
ISDN, analyzing, 258276
MOH, troubleshooting, 608611
SDL
conguring, 6367
reading, 6063
troubleshooting held calls, 527529
sniffer traces, 106
Trailing-# DDI, 495
training, 434
transactional replication, 794
initiating, 804
transcoders, 565, 571577
out-of-resource conditions, 578580
with conference bridge resources, 581585
with MOH servers, 585
transfer patterns, conguring, 730
transferred calls, 529531
transformations, 513514
DDIs, 486494
overriding, 499
rules
cumulative effect of changes, 497499
order of application, 496497
service parameter-related, 500501
translation patterns, 501506
translation patterns, 501506
transmission rates
fax devices, 434
fax relay, adjusting, 451452
transmitting faxes through voice codecs, 437
troubleshooting methodologies
data analysis, case study, 1819
data collection, 45
analyzing collected data, 11
case study, 1418
earliest occurence of problem, 1011
identifying root cause of problem, 6

971

isolating root cause of problem, 79

user information, 10
production versus nonproduction outages, 56
trunk packages (MGCP), 233
trunks, intercluster, 311
codec mismatches, 312
TSP (TAPI service provider)
verifying compatibility with Cisco Unity,
655656
verifying configuration, 656657

U
UDP (User Datagram Protocol), 854
umbrella recommendations, H.323, 281
unanswered calls, forwarding, 479480
unauthorized access to international numbers,
preventing, 545
unexpected outside dial tone, troubleshooting,
465466
unicast audio sources (MOH), troubleshooting,
615617
Unity voice mail systems
applying restrictive calling search spaces, 547
DTMF, 661662
MWI, 659661
troubleshooting resources, 662
verifying switch configuration, 658659
verifying TSP compatibility, 655656
verifying TSP configuration, 656657
UnknownCallerId service parameter
transformation, 501
UnknownCallerIdFlag service parameter
transformation, 501
UnknownCallerIdText service parameter
transformation, 501
unregistered IP Phones, tracing, 131132
unregistered Skinny clients
troubleshooting inline power problems,
114117
troubleshooting network connectivity, 117,
120127
conguration les, 121127
IP addressing, 118121
VLAN conguration, 118
urgent priority route patterns, 502

972 user hold audio source (MOH)

user hold audio source (MOH), 601

user information, collecting, 10
user search requests, directory access, 820
UseZerosForUnknownDn parameter (CMI), 670
utilities, CDR Time Converter, 91.
See also applications

V
V.21 HDLC, 854
VAD (voice activity detection)
as source of voice quality degradation, 402404
comfort noise, 402
ValidateDns parameter (CMI), 670
variable delay, 384
dejitter delay, isolating, 393395
effect on signaling, 395396
low-speed links, isolating, 391393
queuing delay, isolating, 390391
variable-length matching (dial peers), 178179
VAT (Voice Anomaly Tracking), 166
verbose dialing forest traces, 543
verifying
CAC configuration, 640645
Cisco IOS MGCP registration status, 240249
Cisco IP Phone firmware, 165
Cisco Unity switch configuration, 658659
CRA engine status, 745748
Database Layer Monitor operation, 812813
fax/modem passthrough configuration,
441444
IP network integrity, 13
LDAP directory configuration, 745
MOH fixed audio source device configuration,
619620
physical layer connectivity on digital interfaces,
208210
Skinny client registration with RTMT, 135137
SRST configuration, 709712
TSP compatibility with Cisco Unity, 655656
TSP configuration, 656657
TSP version on CTI applications, 736
VG200 voice gateway, 170

VG248 voice gateway, 521

SMDI integration, 686
conguration parameters, 686690
MWI problems, troubleshooting, 690692
viewing
ephone-dn configuration, 718
real-time statistics with PerfMon, 6971
Route Plan Report in Cisco CallManager
Administration, 466
SRST polling statistics, 722
virtual dial peers, viewing, 719
VNC (Virtual Computer Networking), 108
Voice Codec Bandwidth Calculator, 106
voice codecs, fax/modem passthrough, 437
voice gateways
Catalyst
Catalyst 4224, 173174
Catalyst 6000 CMM, 174
conguring 6624 Analog Interface
Module, 367379
Cisco AVVID IP Telephony, 32
Cisco IOS, 169
2600 series routers, 171172
3600 series routers, 172
3700 series routers, 173
H.323, 281307
MGCP, 218240
T1 CAS, troubleshooting, 214218
timestamps, conguring, 185
troubleshooting TDM interfaces, 187205
VG200, 170
Dick Tracy tool, 101104
CLI/embedded Tracy, 105
FXO interface, troubleshooting
disconnects, 205
voice mail systems
applying restrictive calling search spaces, 547
Cisco Unity, 655
DTMF, 661662
MWI, 659661
troubleshooting resources, 662
verifying switch conguration, 658659
verifying TSP compatibility, 655656
verifying TSP conguration, 656657

X wildcard

CMI, 666667
service parameters, 667671, 674
traces, reading, 674679
troubleshooting with HyperTerminal,
679682
Octel, CallManager integration, 693, 698700
SMDI
CallManager integration, 662666
messages, 664666
MWI, 682685
VG248 integration, 686692
voice quality
choppy audio, isolating sources of, 397404
echo
acoustic echo, 412
electrical echo, 411412
eliminating sources of, 418429
isolating sources of, 411
perception of as problem, 414416
one-way/no-way audio, isolating sources of,
405410
voice streaming
dropped calls
media processing resources, 560
RTP/UDP, 551552
nonsurvivable endpoints, 557
CTI/TAPI endpoints, 559
H.323 gateways, 558559
Skinny gateways, 557
survivable endpoints, 552
IP Phones, 552553
MGCP gateways, 553557
VoiceMailDn parameter (CMI), 670
VoiceMailPartition parameter (CMI), 670
VoIP dial peers, 175
variable-length pattern matching, 179
VSTP (Voice Telephony Service Provider) states,
190191
debug commands, 193196

W-X-Y-Z
WANs, fax relay, 444445
wideband codecs, 855
wildcards, 460
! wildcard, 460
. wildcard, 461
@ wildcard, 461
DDIs, 487494
route lters, 506512
multiple clauses, 512
X wildcard, 460
NANP tags, 508510
Windows 2000
CCEmail
alerting methods, 81
conguring, 7680
object counters, 924925
Windows Terminal Services, 107
Wink Start (E&M), 850
winks, 214
WS-X6608 module, 587
X wildcard, 460

973