Task details commands Description

1.Clear the old backup files by executing the command:svcconfig clear -all

2.Take the new backup by executing the command:svcconfig backup

3.Create the folder(backup) in local system and copy the files.

e,g:IBM_2145:XYZCCSVC03:admin>svc_snap
Collecting system information...
Copying files, please wait...
Copying files, please wait...
Listing files, please wait...
Save the Data Dumping error log...
collection Creating snap package...
Snap data collected in /dumps/snap.112008.110829.045733.tgz
IBM_2145:XYZCCSVC03:admin>
 
2.Copy file to local system.
pscp -load SSPC_IP
admin@SSPC_IP:/dumps/snap.112193.110811.085901.tgz c:\backup
("backup" is your folder under C drive where the snap will be saved)
cleardumps -prefix /home/admin/upgrade
pscp -load SSPC_IP C:\firmware_77733_2145_75.9.1402170001
admin@SSPC_IP:/home/admin/upgrade
Upload New
pscp -load SSPC_IP C:\IBM2145_INSTALL_upgradetest_11.10
software files to admin@SSPC_IP:/home/admin/upgrade
SVC Cluster  
check the uploaded file with command
svcinfo lssoftwaredumps
   
Upgradation of GUI Automatically it will upgrade during the code upgrade
   
Install and run the
test utility svcservicetask applysoftware -file IBM2145_INSTALL_upgradetest_12.13
(Fix any errors
before upgrade) svcupgradetest -v 7.2.0.7
   
Upgrade the
firmware svcservicetask applysoftware -file firmware_77733_2145_75.9.1402170001
   
Check the status svcinfo lssoftwareupgradestatus
   
svcinfo lscluster,svcinfo lsnode
Check and confirm
the new code svcinfo lsnodevpd
   
svctask finderr
check for errors If required create the error dump and copy to local system
svctask dumperrlog -prefix svcerrlog
   
Health check verify MM,Vdisk,Mdisk status

Who Performs
Description
Task
Pre task.  
This change is to install the firmware on BHPCCSVC03 as recommended by
the SVC L3 team - Any issue that will be identifed for mutipath as part of post
or pre PVT account team will be talking to GBIS for approval to reboot the
 
server and fix the issue as part of this change
Work with IBM SVC support and provide required information for health check to
SAN TEAM
confirm the cluster is healthy and online.
Receive confirmation of good health check from SVC L2 support and communicate
to entire team. Followed by the health check confirmation take the SVC SAN TEAM
configuration back up.
Engage BRM/SRM along with the respective platform and application teams on
SAN TEAM
standby, to perform any recovery or PVT as required.
Validate the multipath/Application functionality on all attached servers on
BHPCCSVC03. (App,Intel & UNIX) 1.Verify Event Logs is there any
SAN,
errors
Intel,AIX,LINUX,
2.Collect the details of all the drives present in the server (will be done with script)
Application
3.Take output of Datapath Query (will be done with script)
4. Applicaton Teams need to perform PVT
If there are any servers identified with multipath issues, highlight those servers to
 
management and take a call to fix it or shut them prior to the upgrade.
India steady state team will perform Heath check for BHPCCSVC03 for normal
operation. Confirm all nodes were healthy:
svcinfo lsnode SAN TEAM
svcinfo lsvdsk
svcinfo lscontroller
Open a SR and work with BUR team to validate the backup status of all servers
attached to this SVC cluster, any server that has missed or failed backup will be SAN TEAM
notified to the account management team.
India steady state team will confirm good health check to entire platform team SAN TEAM
Coordinate appropriate action to correct issues; may result in deferral of change  
Actual Task.  
Upgarde of firmware for BHPCCSVC03  
Execute test utility V12.3 from SVC command line using the following command
(svcservicetask applysoftware -file IBM2145_INSTALL_upgradetest_12.3).
SAN TEAM
Test utility review output: SAN TEAM
Test output should be No errors and Proceed with the upgrade.
Test the upgrade for known issues that may prevent a
software upgrade from completing successfully and check for the problems in error log Before
executing the error. If any unfixed errors identified, We will stop the change there itself and
engage the IBM support to fix .
Any issues that reported during the execution of test utility will be reviewed and analyzed by
the SVC L2 support team.
SAN TEAM
Proceed with firmware patch installation in BHPCCSVC03 using the following
command (svctask applysoftware -file "filename"). Once the cmd is executed the svc
firmware package which contains the script will lead the upgrade one node at a
SAN TEAM
time. During the upgrade process SVC will pick one node at a time from different I/O
groups while the partner nodes of the same i/o grp will remain online and serve the
I/O's.
Pick few non-critical servers and check for the hosts paths during the firmware
SAN TEAM
patch installation in progress, engage the platform team to perform this action.
If there were any issues reported on the server end like path lost or disk
unavailability, engage the respective service lines to restore the services by SAN TEAM
following the recovery procedure.
Complete post Upgrade checks
- Check Upgrade Status(svcinfo lssoftwareupgradestatus)
- Verify Cluster Status(svcinfo lscluster)
SAN TEAM
- Generate Dump logs (refer to Runbook R1 Tab)
- Run maintenance procedure (refer to Runbook R1 Tab)
- Copy error logs at local workstation(refer to Runbook R1 Tab)
   
PVT checks & Ticket update and closure  
Perform health check on BHPCCSVC03
VERIFY NORMAL OPERATION OF CLUSTER USING THE FOLLOWING COMMANDS TO
COMPARE WITH EXPECTED RESULTS:
svcinfo lsnode
SAN TEAM
svcinfo lsvdsk
svcinfo lscontroller
Login via SVC GUI, go to Maintenance window and under Analyze error logs check
SAN TEAM
for any new errors.
Post Pvt Tasks
SAN team has verified redundant path of all these servers in SVC and Fabric
Pre/Post Pvt for Documentum, Intel /UNIXservers: SAN,
1.Verify Event Logs is there any errors
Intel,AIX,LINUX,
2.Collect the details of all the drives present in the server (will be done with script)
3.Take output of Datapath Query (will be done with script) and compare with Pre- Application
PVT
4. Applicaton Teams need to perform PVT
UPDATE CHANGE RECORD IMPLEMENTATION STATUS
VERIFY CHANGE RECORD STATUS SAN TEAM
CORRECT CHANGE RECORD STATUS
India SAN team and Sys Ops team will monitor for any critical events for this device for the SAN TEAM & Sys
next 24 hours after successful completion of this change. Ops team
Regression Strategies  
Any informational or errors in log will be reviewed by IBM Support and India SAN
SAN TEAM
team along with US SAN team.
If errors cannot be corrected within the change window the change will be
considered failed and issues will be handled under IBM Standard Incident  
Procedures
Backout plan / Recovery  
Installation failing due to a faulty node:
For example:
During the installation incase of any issues, i.e., Where we start the installation and
done with the first 4 nodes, When it comes to the 5th node and installation stops
due to an issue. In this case SVC will run with firmware patch in the installed 4
SAN TEAM
nodes. Problemetic node will be taken offline and the load will be balanced with the
partner nodes and remaining 3 nodes will be running without the firmware patch.
And installation process stops there itself. Refer to Hosts list for the server/VIO
details. Recovery process to be invoked.
Windows and UNIX startup procedure is attached in RUNBOOK