Method of Procedure

Customer:
Change Request:

www.atsg.net

(888) 504-9559

ATSG MOP V1.0 1

TABLE OF CONTENTS

Table of Contents.................................................................................................................... 2

Revision History...................................................................................................................... 3

Change details........................................................................................................................ 3

Affected devices...................................................................................................................... 3

Associated devices.................................................................................................................. 4

Contacts.................................................................................................................................. 4

Supplemental information....................................................................................................... 4

Terminal Servers.................................................................................................................. 4

Vendor Documentation........................................................................................................ 5

Purpose................................................................................................................................... 5

Description........................................................................................................................... 5

Reason for change............................................................................................................... 5

Technical impact/Risk............................................................................................................. 5

Requirements.......................................................................................................................... 5

Preliminary steps.................................................................................................................... 5

Pre-change output collection................................................................................................... 6

Cisco CUCM - Pre-Change Output Collection:.......................................................................6

Cisco IM&P - Pre-Change Output Collection..........................................................................7

Implementation plan............................................................................................................... 7

prerequisties........................................................................................................................ 7

Cisco Collaboration System compatibility matrix..............................................................7

common space requirement............................................................................................. 8

Required files.................................................................................................................... 8

free disk space.................................................................................................................. 9

upgrade............................................................................................................................... 9

run upgrade readiness cop file (pre-upgrade)...................................................................9

upgrade sccucmsempub cluster.......................................................................................9

ATSG MOP V1.0 2

 run upgrade readiness COP file (Post Upgrade)..............................................................10

upgrade CUCM cluster and IMPs.....................................................................................10

Verification......................................................................................................................... 11

Backout Plan......................................................................................................................... 11

Backout Implementation.................................................................................................... 11

Backout Verification........................................................................................................... 12

post-change output collection............................................................................................... 12

Final Steps............................................................................................................................ 12

ATSG MOP V1.0 3

REVISION HISTORY
Revisi Tea
Date Engineer Notes
on m

CHANGE DETAILS
Information Details
ATSG Ticket #
Customer Ticket #
Vendor Ticket #
Estimated Duration of
Pre-Checks
Estimated Change
Duration

AFFECTED DEVICES

The following list contains devices that will be modified:

Device Name IP Address Device Type/Function

ATSG MOP V1.0 4

ASSOCIATED DEVICES

The following list contains devices that will NOT be modified but may be indirectly impacted.

Device Name IP Address Device Type/Function

CONTACTS

The following list contains all individuals involved with the change (including vendors):

Organiza Primary
Name Role Email Address Phone Number
tion Contact

SUPPLEMENTAL INFORMATION

TERMINAL SERVERS

 SIML DMA will be used as sftp server (not tested)

 Port forwarding requires host file edit or use ICM device as jump box

VENDOR DOCUMENTATION

 Security Advisory -
https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-
cucm-dos-kkHq43We
 Installation Guide -
https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/upgrade/12_5_1/
cucm_b_upgrade-migration-guide-1251su7/cucm_b_upgrade-guide-
1251su2_chapter_01100.html

PURPOSE

ATSG MOP V1.0 5

DESCRIPTION

 Apply 14 SU4 to mitigate UCM DoS Vulnerability.

REASON FOR CHANGE

 A new vulnerability on UCM was release by Cisco on August 21, 2024.

 Patch is needed to address the issue

TECHNICAL IMPACT/RISK

The expected impact is high, UCM, IMP servers will be reloaded as part of the change, call
service will be disrupted.

REQUIREMENTS
Function Required Details (if function required)
Onsite support ☐

Vendor or TAC ☐

Hardware RMA ☐

Customer/User/App ☒ Test calls, jabber, Q function before/after the

Testing change
Video/Audio ☒
Conference Bridge
Customer contact ☒ Pre/Post Upgrade Notifications & E-mail
required before Notification for any issues
change
Additional ☐
Requirement

PRELIMINARY STEPS

 Review all steps in the implementation plan before proceeding

 If applicable, create outage for associated device(s) to suppress alarms

 If required, join bridge

 Update case with details from the above steps taken and change case status to “Work in
progress”.
 Notify all listed in the Contact(s) section via email, please follow the format specified in
the Change Management documentation.

PRE-CHANGE OUTPUT COLLECTION

 Take the necessary backups/screenshots of the current configurations that will be
modified and any informative “show” commands. Please reference, MOP Workflow for
more details.
 Filename should utilize a similar format to the below:
[devicename]_[pre,post]-change_[date].log

ATSG MOP V1.0 6

 Date format: yyyy.mm.dd
Example: Router001_pre-change_2014.12.3.log

CISCO CUCM - PRE-CHANGE OUTPUT COLLECTION:

The devices have been listed under “Affected Devices” on previous steps.
The actual commands and details are found in: https://tacconnect > Home > Homepage >
Health Checks
Perform a Pre-change Health Check from each device as follows: (Gather, Review and
Document)

o System Status
o System Versions (Active and Inactive)
o Hardware from Application perspective
o Run a Diagnose Test
o Gather a list of Services and Status
o NTP Status
o Check Last backup and backup history
o Cluster Replication
o Check for any core dumps

Via RTMT/CLI/GUI:

o System Resources: CPU, Memory, Disk Usage

o System Health from Alerts– Errors, Warnings, Critical
 Take appropriate action based on Alert Severity
o Phone/Device/End-Point Registration counts
o Licensing information and Status
o Phone Firmware information

run sql select count(d.tkmodel), tp.name, defaults.tkdeviceprotocol as

SignalingProtocol, defaults.loadinformation as DeviceDefault, d.tkmodel as
tkmodel from device as d INNER JOIN typeproduct as tp on d.tkmodel=tp.tkmodel
INNER JOIN defaults as defaults on tp.tkmodel=defaults.tkmodel where d.name
like '%' and defaults.loadinformation != "" group by d.tkmodel, tp.name,
defaults.loadinformation, defaults.tkdeviceprotocol order by tp.name

Backup

o During the upgrade, List.xml file could be overwritten, which affects

customized background for phones.
o Download List.xml file from tftp servers and save it.
o Ensure there is a recent successful backup file for UC clusters

Other Information:

o System summary: https://pub/ast/ASTIsapi.dll?

GetPreCannedInfo&Items=getPhoneSummary

CISCO IM&P - PRE-CHANGE OUTPUT COLLECTION

ATSG MOP V1.0 7

The devices have been listed under “Affected Devices” on previous steps.
The actual commands and details are found in: https://tacconnect > Home > Homepage >
Health Checks
Perform a Pre-change Health Check from each device as follows: (Gather, Review and
Document)

o System Status
o System Versions (Active and Inactive)
o Hardware from Application perspective
o Run a Diagnose Test
o Gather a list of Services and Status
o NTP Status
o Check Last backup and backup history
o Cluster Replication (IMDB)
o HA Status

Via RTMT/CLI/GUI:

o System Resources: CPU, Memory, Disk Usage

o System Health from Alerts– Errors, Warnings, Critical
 Take appropriate action based on Alert Severity
o IM and Presence Summary
o Jabber Summary
 Sip trunk
 Media resource
o Licensing information and Status
o CUCM GUI
 HA Status
 User counts and Server States
 SIP Trunk status
o IM GUI
 Health status of each Imp server
 Notification
 Overall Cluster Health
 Diagnostics > System Troubleshooter

ATSG MOP V1.0 8

IMPLEMENTATION PLAN

PREREQUISTIES

CISCO COLLABORATION SYSTEM COMPATIBILITY MATRIX

1. Cisco compatibility Requirements for UCM v14

IMP CUC ISR

4300/4400

Min. Version

Recommended

2. Information collected from the UC devices:

CUCM CUC IMP ISR 4300/4400

Current
Version

COMMON SPACE REQUIREMENT

1. Min. space requirement is 25GB, use

ciscocm.free_common_space_v<latest_verion>.cop.sgn file to clean the disk prior to
the change window
2. UCM cluster
3. IMP cluster

REQUIRED FILES

1. Free common space file both UCM and IMP:

a. ciscocm.free_common_space_v1.11.k4.cop.sha512
b. f58511d7c3ab88e007dce64ff1bc257b
2. UCM updates:
a. UCSInstall_UCOS_14.0.1.14900-94.sha512.iso
b. MD5: cc180b7d2872510a1ced1ecc7c4dd7c9
3. IMP updates:
a. UCSInstall_CUP_14.0.1.14900-4.sha512.iso
b. MD5:
4. preUpgradeCheck:

ATSG MOP V1.0 9

 a. ciscocm.preUpgradeCheck-00043.cop.sha512
b. MD5: 03e595106e13acdd61cde463af378130
5. postUpgradeCheck:
a. ciscocm.postUpgradeCheck-00043.cop.sha512
b. MD5: 8f70b14b143fc182d59ff795492a9cb1
6. V14 CVE-2024-6387 cop file
a. ciscocm.V14_CVE-2024-6387_v1.1.cop.sha512
b. MD5:

SIML DMA INFO

7. Required to proceed with the change.

8. Staging files to DMA
a. Need CUCM file for restrtricted version of CUCM 14 SU4 uploaded to the DMA.
9. sampson@a1pmopt01:/home/transfer/files$ pwd
a. /home/transfer/files
10. sampson@a1pmopt01:/home/transfer/files$ ls -lh
a. total 7.5G
b. drwxr-xr-x 3 transfersftp transfer 4.0K Aug 2 10:32 10.10.70.55
c. -rw-r--r-- 1 sampson sampson 4.9K Sep 14 16:40
ciscocm.free_common_space_v1.11.k4.cop.sha512
d. -rw-r--r-- 1 sampson sampson 646K Sep 14 16:40
ciscocm.postUpgradeCheck-00043.cop.sha512
e. -rw-r--r-- 1 sampson sampson 646K Sep 14 16:40
ciscocm.preUpgradeCheck-00043.cop.sha512
f. -rw-r--r-- 1 sampson sampson 1.6M Aug 12 08:01 ciscocm.V14_CVE-2024-
6387_v1.1.cop.sha512
g. -rw-r--r-- 1 sampson sampson 1.6M Aug 12 08:01 ciscocm.V14_CVE-2024-
6387_v1.1_revert.cop.sha512
h. -rw-r--r-- 1 sampson sampson 2.9G Sep 14 16:42
UCSInstall_CUP_14.0.1.14900-4.sha512.iso
i. -rw-r--r-- 1 sampson sampson 4.7G Sep 14 16:43
UCSInstall_UCOS_14.0.1.14900-94.sha512.iso
j. sampson@a1pmopt01:/home/transfer/files$
11. Access the customer OPTX platform and enabled sftp via the walled directory page.
a. Username transfersftp
b. Password is system generated
12. Server IP address collected
a. 10.10.70.204

FREE DISK SPACE

If no jump box is available, use CLI to install the cop file

1. Go to the nodes highlighted under common space requirement

2. If CLI is used, issue the command utils system upgrade
3. If GUI is used, go to CUCM OS administration > Software Upgrades >
Install/Upgrade
a. Source: SFTP
b. Directory: /home/transfer/files
c. Server IP:

ATSG MOP V1.0 10

 d. Username: transfersftp
e. Password: [system generated]
f. Transfer protocol: SFTP
g. Choose file ciscocm.free_common_space_v1.11.k4.cop.sha512
4. Use show tech runtime disk to verify available common disk space

UPGRADE

Notes:

1. Proceed only when the followings were complete:

2. There was a recent successful backup, list.xml file(s) was backed up if customized
phone background or ring tongs were used.
3. Report of cluster summary and registered phone was collected via pre-healthchack
script.

UPGRADE CUCM CLUSTER AND IMPS

Sequence for upgrade

ATSG MOP V1.0 11

 1. Before beginning upgrade verify with customer about
phone upgrades
a. Note the customer expectation here.
b. This has been prepared to include phone upgrades
2. Run the preUpgrade COP file on all nodes starting from CUCM Pub, CUCM subs, then
IMP Pub and subs.
a. Resolve any reported issue and rerun the cop file, if needed.
3. If CLI is used, issue the command utils system upgrade
4. If GUI is used, go to CUCM OS administration > Software Upgrades >
Install/Upgrade
5. Enter the sftp information
6. Choose the installation file
b. UCM: UCSInstall_UCOS_14.0.1.14900-94.sha512.iso
c. IMP: UCSInstall_CUP_14.0.1.14900-4.sha512.iso
7. DO NOT check Version switching box.
8. DO NOT reboot the node when prompted.
9. When the upgrade is complete on all UCM, IMP servers, switch software version
manually:
a. Starting from UCM Pub, subs, then IMP Pub and sub
b. Disable HA and failover users prior to reboot IMP servers
c. If CLI is used, issue the command utils system switch-verison
d. If GUI is used, go to Settings > Version.
i. Click Switch Version to switch versions and restart the node.
10. Ensure that database replication is complete and functioning.
11. Enable HA and balance users for IMP cluster
12. Run postUpgrade COP file on all nodes.

REFESH THE UNIFIED COMMUNICATIONS ZONE

13. Log in to the Expressway C GUI and navigate to Configuration > Unified
Communications > Unified CM Servers
a. NOTE CoOp has two Expressway C clusters
14. Place a check next to the CUCM Publisher FQDN and select refresh.
15. Refresh connections to all Unity Connection Servers
16. Refresh connection to IMP nodes
17. If any report to have an issue report this to the customer.

If at any point during implementation there are unexpected results that may cause impact
to production:
a. Spot check the appliance to assist in determining the impact
b. Take steps noted in the Unsuccessful Implementation section under Final Steps
c. Call the customer listed in the Contact(s) section
d. Do your best to remediate the issue

VERIFICATION

1. Check cluster health:

a. Check database replication status via CLI.
a. Check services list.
2. Confirm sip trunk status on CUCM clusters.

ATSG MOP V1.0 12

3. Test inbound/outbound calls as well as supplementary features – conference, transfer etc
4. Test voice mail, mwi and unified messaging.
5. Test Jabber, MRA (if any) login/off, call featuers
6. Agent login/off, Q call.
7. If verification fails:
a. Do your best to remediate the issue
b. If unable to remediate, jump to the “Back-out Plan” section
8. If verification succeeds, jump to the “Post Output Collection” section

BACKOUT PLAN

BACKOUT IMPLEMENTATION

1. Roll back to the previous version:

a. Starting from the Publisher followed the subscriber
b. Go to Settings > Version.
c. Click Switch Version to switch versions and restart the node.

BACKOUT VERIFICATION

1. Confirm call functionalities

2. Test voice mail, mwi and unified messaging.
3. Test Jabber, MRA and agent login/off as well call feature

POST-CHANGE OUTPUT COLLECTION

 Take the necessary backups/screenshots of the current configurations and any
informative “show” commands that were run in the “Pre-change Output Collection”
section. Verify for any differences between the pre and post output.
 Filename should utilize a similar format to pre-change filenames

FINAL STEPS
 Ensure that all debugs and elevated traces are disabled unless explicitly stated
otherwise
 Please refer to the Operations documentation, Change Management, and complete all
steps to finalize the change and to communicate effectively on the change
status/completion.

ATSG MOP V1.0 13