SD-WAN Orchestrator Upgrade process

Step 1: Prepare for the Orchestrator Upgrade

Step 2: Send Upgrade Announcement
Step 3: Prepare for the SD-WAN Orchestrator Upgrade
Known issues
Upgrade image is automatically deleted after every run of the upgrade command.
Insufficient disk space when upgrading from a 4.x release lower than 4.3.1 to 4.5 or up
Step 4: Proceed with the SD-WAN Orchestrator Upgrade
4.1 Upgrade standby VCO (It’s important that this one is done first):
4.2 Upgrade Primary VCO
Final notes:

Step 1: Prepare for the Orchestrator Upgrade

The procedure for this first step is outlined in the Operator Guide documentation.

https://docs.vmware.com/en/VMware-SD-WAN-by-VeloCloud/3.4/vmware-sd-wan-operator-guide/GUID-495C8232-3D3C-4598-AB72-55ED0D929824.html

Step 2: Send Upgrade Announcement

This one is done at the customer’s discretion prior to the upgrade and comprises of two stages. and it serves the following purposes:

Upgrade Announcement: To inform the users of the VCO of the upgrade activity that will be taking place. It can be done any amount of time in
advance as it doesn’t have any impact in the VCO’s operation.
Upgrade Actions: To prevent changes to be pushed while the upgrade is in process. Should be done during the maintenance window where the
update is actually taking place as it affects the VCO’s operation (as mentioned by virtue of not allowing configuration changes to be delivered)

To send an upgrade announcement:

1. From the SD-WAN Orchestrator, select Orchestrator Upgrade from the navigation panel.
2. In the Upgrade Announcement area, type in your message in the Banner Message text box.

3. Click the Announce Orchestrator Upgrade button.

A popup message appears indicating that you have successfully created your announcement, and that your banner message displays at the top
of the SD-WAN Orchestrator.

Step 3: Prepare for the SD-WAN Orchestrator Upgrade

All commands presented here are intended to be executed by "root"
While the process can be executed over SSH (and might be the only option in certain environments), it is recommended to be done from the VM
console to avoid potential issues that might happen if the SSH connection is interrupted for any reason.
You may obtain software images from "http://myvmware.com"
 Starting with release 4.5.0 significant changes were introduced to the vco upgrade command "/opt/vc/bin/vco_software_update". The upgrade
process is now capable of using signed upgrade packages. This adds an extra layer of security to ensure the integrity of the software being
applied.

In order to make use of that additional security layer, please make sure to download the respective public key and place it in the following location.
This key should be available to you from the same location where you downloaded the upgrade package.

/var/lib/velocloud/software_update/keys/software.key

You may need to create the respective directory if it doesn't exists:

mkdir /var/lib/velocloud/software_update/keys

The upgrade command provides options should you wish to bypass the signed package verification (–untrusted) or define a different location for
the public key to use (--pubkey)

root@vco1:~# /opt/vc/bin/vco_software_update --help

Software Upgrade
Usage:
/opt/vc/bin/vco_software_update [--untrusted] [--batch] [--debug] [--pubkey=KEY]

Copy the VCO upgrade image to: /var/lib/velocloud/software_update/vco_update.tar

cp <image location>/image_name.tar /var/lib/velocloud/software_update/vco_update.tar

Known issues

Upgrade image is automatically deleted after every run of the upgrade command.
Every time that you execute upgrade command "vco_software_update" the image you placed in "/var/lib/velocloud/software_update/" directory. This
means that if by any reason you execute the command but the upgrade is not conducted, for example if you choose to cancel it because you didn't have
the software key in place:

root@vco5:~# /opt/vc/bin/vco_software_update
2022-03-02 15:23:33,999 - UPGRADE - WARNING - Verification key does not exist: /var/lib/velocloud
/software_update/keys/software.key
WARNING: failed to verify package identity. Proceed as untrusted [y/n] [n]: n
2022-03-02 15:23:42,708 - UPGRADE - ERROR - Aborting installation of untrusted package

You will have to copy the image file vco_update.tar again to "/var/lib/velocloud/software_update/" before attempting to run the upgrade command again.

Insufficient disk space when upgrading from a 4.x release lower than 4.3.1 to 4.5 or up
When upgrading from a 4.x release lower than 4.3.1 to 4.5 or up, you may encounter a issue while trying to perform an upgrade due to insufficient disk
space (Bug ID 65998) . If this happens the upgrade will fail to be executed (no harm is done to the running system). You will see the following error ending
the upgrade command execution:

OSError: [Errno 28] No space left on device

To address this you may perform one of the following options:

Option 1: Upgrade first to release 4.3.1

Option 2: Create a symbolic link to move the software_update directory out of /var. To do that, simply run the following script:

#!/bin/bash
if [[ ! -L "/var/lib/velocloud/software_update" ]]
 then
mkdir -p /store2/velocloud
if [[ -d "/var/lib/velocloud/software_update" ]]
then
mv /var/lib/velocloud/software_update /store2/velocloud/
fi
ln -s /store2/velocloud/software_update /var/lib/velocloud/software_update
fi

This will move the directory to /store2, which is a filesystem that should have plenty of space to manage this extra usage.

Once you execute this script, you should be able to proceed with the upgrade process

Step 4: Proceed with the SD-WAN Orchestrator Upgrade

4.1 Upgrade standby VCO (It’s important that this one is done first):
4.1.1 Execute upgrade command:

/opt/vc/bin/vco_software_update

4.1.2 Reboot. The upgrade mechanism will determine if this is required and will ask for it to be performed. The user can chose to perform it immediately
(advised) or at a later time . The following message appears:

Reboot is required. Reboot? (y/n) [y]

4.2 Upgrade Primary VCO

4.2.1 Execute upgrade command

/opt/vc/bin/vco_software_update

4.2.2 Reboot. The upgrade mechanism will determine if this is required and will ask for it to be performed. The user can chose to perform it immediately
(advised) or at a later time . The following message appears:

Reboot is required. Reboot? (y/n) [y]

Do not proceed to perform the step below until the reboot (if required) is completed

If you are upgrading from 4.x to a subsequent version (e.g. from 4.0.2 to 4.2.0) please run the following command at this point. This step is not
required in previous versions.

4.2.3 Run the schema update script present on the VCO inside the path: /opt/vc/scripts/ with option -e

/opt/vc/scripts/vco_schema_update.sh -e

More options with script

#/opt/vc/scripts/vco_schema_update.sh -h
Usage: vco_schema_update.sh [OPTIONS]
 Options:
-h, --help Print this help
-d, --dry-run Print update schema query that is going to execute and exit.
-e, --execute Printe update schema query and execute it.
Actual query execution will be carried out by MySQL event scheduler starting in 60 seconds
after executing the query
This script would return immidiately after the execution and delete the schema update sql
file
-s, --status Fetch and print the status of the ongoing scheduler execution

If desired, use -d options to see what changes are going to be applied.

To get the current status of the schema changes progress, execute script with -s options and it will print out the current status of schedule
schema change. It will also show any ongoing schema changes and progress for it using performance schema. MySQL Documentation with
sample

# /opt/vc/scripts/vco_schema_update.sh -s
*************************** 1. row ***************************
Db: velocloud
Name: longRunningOperations
Definer: velocloud-root@localhost
Time zone: SYSTEM
Type: ONE TIME
Execute at: 2022-05-12 07:30:56
Interval value: NULL
Interval field: NULL
Starts: NULL
Ends: NULL
Status: DISABLED
Originator: 1
character_set_client: utf8mb4
collation_connection: utf8mb4_0900_ai_ci
Database Collation: utf8mb4_0900_ai_ci

Final notes:
1. The log of the upgrade process will be located at: /var/log/vco_software_update.log
2. Although not exactly a part of the upgrade process: If a customer doesn’t have the “Enabled on Orchestrator Upgrade” checkbox checked prior to
starting the upgrade process:

Once the upgrade process is done and the customer click on “Complete Orchestrator Upgrade”:

That customer will be left with the following configuration:

Which in turn will cause for configuration updates to not be propagated to Edges of that Customer. To address this, depending on the desired behavior:

1. Make sure the “Enabled on Orchestrator Upgrade” is checked prior to starting the upgrade process if it is desired that configuration updates are
pushed to Edges once the upgrade is done (I.E return to normal operation)
2. If it is not desired for updates to be pushed to Edges after the upgrade is done and rather the Customer or partner wants to enable this at a later
time. Then leave the checkbox "Enabled on Orchestrator Upgrade" unchecked and then check the "Enabled" checkbox at the times where normal
push of updates and configurations wants to be resumed.