Method of Procedure

Customer:
Change Request:

TABLE OF CONTENTS

Table of Contents.................................................................................................................... 2

ATSG MOP V1.0 1

Revision History...................................................................................................................... 4

Change details........................................................................................................................ 4

Affected devices...................................................................................................................... 4

Associated devices.................................................................................................................. 4

Contacts.................................................................................................................................. 5

Supplemental information....................................................................................................... 5

Terminal Servers.................................................................................................................. 5

Vendor Documentation........................................................................................................ 5

Purpose................................................................................................................................... 5

Description........................................................................................................................... 5

Reason for change............................................................................................................... 5

Technical impact/Risk............................................................................................................. 5

Requirements.......................................................................................................................... 5

Preliminary steps.................................................................................................................... 6

Virtualization requirements.................................................................................................. 6

UC and Endpoint requirements............................................................................................7

certificates........................................................................................................................... 7

UCM.................................................................................................................................. 7

IMP.................................................................................................................................... 7

exp....................................................................................................................................... 7

ciphers.............................................................................................................................. 8

license.............................................................................................................................. 8

Pre-change output collection................................................................................................... 8

Implementation plan............................................................................................................... 9

Implementation.................................................................................................................... 9

Verification......................................................................................................................... 10

Backout Plan......................................................................................................................... 10

Backout Implementation.................................................................................................... 10

ATSG MOP V1.0 2

 Backout Verification........................................................................................................... 11

post-change output collection............................................................................................... 11

Final Steps............................................................................................................................ 11

ATSG MOP V1.0 3

REVISION HISTORY
Revisi Tea
Date Engineer Notes
on m

CHANGE DETAILS
Information Details
ATSG Ticket # ESR Case
Customer Ticket #
Vendor Ticket #
Estimated Duration of
00 hrs and 00 mins
Pre-Checks
Estimated Change
00 hrs and 00 mins
Duration

AFFECTED DEVICES

The following list contains devices that will be modified:

Device Name IP Address Device Type/Function

ASSOCIATED DEVICES

The following list contains devices that will NOT be modified but may be indirectly impacted.

Device Name IP Address Device Type/Function

CONTACTS

The following list contains all individuals involved with the change (including vendors):

ATSG MOP V1.0 4

 Organizati
Name Role Email Address Phone Number
on

SUPPLEMENTAL INFORMATION

TERMINAL SERVERS

 Jump box

VENDOR DOCUMENTATION

 Security advisory:
https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-
expressway-csrf-KnnZDMj3
 Upgrade Guide:
https://www.cisco.com/c/en/us/support/docs/unified-communications/expressway/217743-
upgrade-of-video-communication-server-v.html#anc2
 Smart Licensing
https://www.cisco.com/c/en/us/support/docs/unified-communications/expressway/215633-
configuring-smart-licensing-on-cisco-exp.html

PURPOSE

DESCRIPTION

 Upgrade Expressways to mitigate the security vulnerability

REASON FOR CHANGE

 Customer’s expressways version is affected by the recent released security advisory.

 Upgrade the devices to fixed version to address the vulnerabilities

TECHNICAL IMPACT/RISK

 The change should be implemented in a change window to reduce the impact

REQUIREMENTS
Function Required Details (if function required)
Onsite support ☒

Vendor or TAC ☐

Hardware RMA ☐

ATSG MOP V1.0 5

 Customer/User/App ☒
Testing
Video/Audio ☒
Conference Bridge
Customer contact ☒
required before
change
Additional ☐
Requirement

PRELIMINARY STEPS
 Review all steps in the implementation plan before proceeding
 If applicable, create outage for associated device(s) to suppress alarms
 If required, join bridge
 Update case with details from the above steps taken and change case status to “Work in
progress”.
 Notify all listed in the Contact(s) section via email, please follow the format specified in
the Change Management documentation.

VIRTUALIZATION REQUIREMENTS

 Production settings meet the requirement, but both venter and ESXi run on EoL
version.
o Potential bug -https://bst.cisco.com/bugsearch/bug/CSCvy07347

vCenter ESXi Version VM configuration

Verson
Physical
vCPU vRAM vDisk1 vDisk2
CPU

6.7 update 6.5 update 3

2.6 GHz 2 6GB 4 GB 128 GB
3t

ATSG MOP V1.0 6

UC AND ENDPOINT REQUIREMENTS

 Ref: https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/expressway/config_guide/X14-
3/mra/exwy_b_mra-deployment-guide-x143/exwy_m_requirements-for-mra.html
 UC servers meet the requirements.
o Push Notification is not enabled on UCM.
 Endpoints meet minimum requirements.

 No CMS WebRTC Proxy is used

CERTIFICATES

UCM

 Reference:
o Bug CSCvz20720 -- https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvz20720
o https://www.cisco.com/c/en/us/support/docs/unified-communications/expressway/
218018-troubleshoot-expressway-traffic-server-c.html
 Due to the bug CSCvz20720, Root and intermediate certificates of exp-c are required to
be uploaded to UCM publisher.
o Snaponglobal Enterprise Certificate Authority is the CA, it’s uploaded as
tomcat-trust but not callmanager-trust
 Upload root certificate to the Publisher as callmanager-trust (pending)
o Confirm it’s distributed to other UCM nodes
o If not, upload it to each node manually
 Restart below services on all UCM nodes (pending)
o Cisco Tomcat
o Cisco callmanager
o Cisco TFTP
o Cisco HAProxy is restarted automatically when Tomcat is restarted.
 If it’s not restarted, restart it via cli utils service restart Cisco HAProxy

IMP

 Upload root certificate to the Publisher as cup-xmpp-trust (pending)

o Confirm it’s distributed to other IMP node
o If not, upload it to each node manually

EXP

CIPHERS

 Existing chipher order under Maintenance > Security > Ciphers

ATSG MOP V1.0 7

 EECDH:EDH:HIGH:-AES256+SHA:!MEDIUM:!LOW:!3DES:!MD5:!PSK:!eNULL:!
aNULL

LICENSE

 Starting from x14.2, smart license is the only available license mode (pending)
o If the EXP is used only for MRA, no license is needed
o If any other feature is used, e.g B2B, it’s customer’s responsibility to make sure that
EXPs are able to communicate with Cisco smart software Manager.
o It’s also customer’s responsibility to get license assigned and be able to access
license portal for any troubleshooting if needed.

Note: starting from x14.2, exp is limited to 2500 crypto sessions.

PRE-CHANGE OUTPUT COLLECTION

 Take the necessary backups/screenshots of the current configurations that will be
modified and any informative “show” commands. Please reference, MOP Workflow for
more details.
 Filename should utilize a similar format to the below:
[devicename]_[pre,post]-change_[date].log
Date format: yyyy.mm.dd
Example: Router001_pre-change_2014.12.3.log

Cisco Expressway-VCS - Pre-Change Output Collection:

The devices have been listed under “Affected Devices” on previous steps.
The actual commands and details are found in: https://tacconnect > Home > Homepage >
Health Checks
Perform a Pre-change Health Check from each node as follows: (Gather, Review and
Document)
Gather system details; status, hostname, version, resource usage, etc.
Gather Zone/registrations/connections status
Gather hardware details
Gather the last 100 config/event/network logs
Gather Cluster status

VIA GUI:
Review the following and save screenshots:

o Status, overview, alarms

o Hardware/Software Resource and Performance status
o Configuration > Zones > Zones
o Unified Communications status / Calls / Zone Status
o Licensing details
o System Events

Note: if there is any alarm, fix the alarm first.

VIA CLI:

ATSG MOP V1.0 8

 o Ssh to each exp-c and exp-e as root and issue the below command:

cd / && ./sbin/verify-syskey

o It must not provide any output. If you got an “error” as a result, open CTAC tickt to fix
the issue prior to upgrade.
o At the time of this writing, there was no error output from all exps. This verification
needs to perform again prior to the upgrade.

Backups:

o Perform a backup of the configurations prior to any upgrades/changes/reboots.

o Make sure there is a good backup before starting the upgrade. Run
manual backup if needed. Don’t proceed without a backup.
o Maintenance>Backup and restore

IMPLEMENTATION PLAN

IMPLEMENTATION

UPGRADE

1. Go to Cisco software download portal and download s42700x14_3_4.tar.gz

https://software.cisco.com/download/home/286255326/type/286332039/release/X14.3.4

2. Start upgrade from EXP-E primary following by EXP-E secondary, EXP-C primary and EXP-C
secondary one at a time.
3. Go to Maintenance > Maintenance mode

a. Set Maintenance mode to ON

b. Click Save
4. Go to Maintenance > Upgrade
a. Click Browse to find the upgrade file that was downloaded earlier and click Upgrade

b. Follow the prompt, press Continue to proceed with the upgrade and Reboot to switch
to new version.
c. After the server is up run xConfiguration Security CSRFProtection status: “Enabled”

ATSG MOP V1.0 9

5. Refresh UC servers

a. Go to Configuration > Unified Communication

b. Choose Unified CM servers
c. Check the checkbox beside the Publisher name and click Refresh servers
d. Repeat above steps to refresh IMP and UCX servers.
6. Confirm that Maintenance mode is OFF.

SMART LICENSING

1. After upgrade, smart licensing is automatically enabled.

2. Go to Maintenance > Smart licensing
3. Confirm Smart licensing mode is ON
4. Transport settings: choose one based on customer’s settings (UC devices use direct)

a. if it’s Direct, make sure FW allows the traffic

b. Customer creates a token on CSSM

c. Paste the token and click Register. It may take a while to register
d. Leave DO not share my hostname or IP address with Cisco unchecked
5. Confirm registration status via GUI or CLI
a. xstatus //license
b. Maintenance > Smart Licensing
6. If at any point during implementation there are unexpected results that may cause impact
to production:
a. Spot check the appliance to assist in determining the impact
b. Take steps noted in the Unsuccessful Implementation section under Final Steps
c. Call the customer listed in the Contact(s) section
d. Do your best to remediate the issue

VERIFICATION
1. go to Status > Overview, make sure all EXPs are on x14.3.4
2. go to Status > Alarms, confirm no cluster alarms
3. go to System > Clustering and confirm cluster status
4. go to Configuration > Zones > Zones, ensure that Unified communication
traversal shows as Active
5. Test MRA log in, calls. If everything works fine, jump to step 8. Otherwise, go to step 6.
6. go to Maintenance > Security > Ciphers, confirm ciphers order.
a. After the upgrade, the ECDSA could be preferred, which uses callmanager-ECDSA
and tomcat-ECDSA certificates (self-signed) to be uploaded to EXPs.

Note: EXP has a known bug – CSCwa12905 and it’s impossible to upload two different
certificates with the same common name. So CA signed certificate is preferred.

b. If the cipher doesn’t prepend “ECDHE-RSA-AES256-GCM-SHA384:”, then ECDSA is

preferred. The fix is to prefix it to the Ciphers List
i. Test MRA first, if it fails, modify ciphers list
ii. Copy the existing ciphers to notepad

ATSG MOP V1.0 10

 iii. Prefix ECDHE-RSA-AES256-GCM-SHA384: to ciphers list for all ciphers
7. If verification fails:
a. Do your best to remediate the issue
b. If unable to remediate, jump to the “Back-out Plan” section
8. If verification succeeds, jump to the “Post Output Collection” section

BACKOUT PLAN

BACKOUT IMPLEMENTATION

Note: EXP keeps two sets of partitions after an upgrade: one for upgraded version and one
for previous version.
1. Ssh to EXP with root account.
2. Issue the command selectsw to identify the active set.
a. If the output is “1”, then the current version is set 1.
3. To switch to another version, issue command selectsw <>
a. The output is based on the output of step 2.
b. If output is 1, then issue the command selectsw 2; if the output is 2, then issue
selectsw 1.
4. Reboot the device
a. After switch version, if CLI prompt for restart, press Y and enter
b. If no prompt, go to GUI to reboot

BACKOUT VERIFICATION

 Perform the same verification steps documented earlier.

POST-CHANGE OUTPUT COLLECTION

 Take the necessary backups/screenshots of the current configurations and any
informative “show” commands that were run in the “Pre-change Output Collection”
section. Verify for any differences between the pre and post output.
 Filename should utilize a similar format to pre-change filenames

FINAL STEPS
 Ensure that all debugs and elevated traces are disabled unless explicitly stated
otherwise
 Please refer to the Operations documentation, Change Management, and complete all
steps to finalize the change and to communicate effectively on the change
status/completion.

ATSG MOP V1.0 11