The National Commercial Bank – ITD

IT Production Department
Document1
Rev Date: 01-Jun-2015 Rev: 2015-2 Doc No: SYS-153 Page No: 1 of 11

Document: MQ Work Instruction

Author: Sanjay Bardhan
Contributor: Yahya M. Asfour
Adan Republica
Reviewer: Joey Licyayo
Owner: IT Production Department

NCB Internal
 The National Commercial Bank – ITD
IT Production Department
Document1
Rev Date: 01-Jun-2015 Rev: 2015-2 Doc No: SYS-153 Page No: 2 of 11

TABLE OF CONTENTS
1.0 OVERVIEW ............................................................................................................................................................. 3
1.1 Introduction ................................................................................................................................................................ 3
1.2 Systems Architecture .................................................................................................................................................. 3
1.2.1 MQ Enterprise ............................................................................................................................................. 3
1.2.2 MQ Tadawul ................................................................................................................................................ 3
1.3 Technical Information ................................................................................................................................................. 4
2.0 ROLES AND RESPONSIBILITIES ................................................................................................................................ 4
2.1 Network Operations Center Team (L1 Support) ......................................................................................................... 4
2.2 Systems Engineering Team (L2 Support) .................................................................................................................... 4
3.0 MONITORING......................................................................................................................................................... 4
3.1 MQ Enterprise ............................................................................................................................................................. 4
3.2 MQ Tadawul ................................................................................................................................................................ 5
4.0 TROUBLESHOOTING ............................................................................................................................................... 5
4.1 Application .................................................................................................................................................................. 5
4.1.1 Checking MQ Services ................................................................................................................................. 5
4.1.2 Application Logs........................................................................................................................................... 6
5.0 DISASTER RECOVERY .............................................................................................................................................. 6
6.1 DR Activation............................................................................................................................................................... 6
6.2 DR Checklist................................................................................................................................................................. 6
6.0 ESCALATION..........................................................................................................................................................11

NCB Internal
 The National Commercial Bank – ITD
IT Production Department
Document1
Rev Date: 01-Jun-2015 Rev: 2015-2 Doc No: SYS-153 Page No: 3 of 11

1.0 OVERVIEW
1.1 INTRODUCTION
This is the queuing interface for Enterprise and Tadawul applications.

1.2 SYSTEMS ARCHITECTURE

1.2.1 MQ Enterprise
MQ Enterprise

BANCS 14 CORDYS

Prod: 10.70.125.34 (rhlux3166)

DR: 10.80.125.34 (rhlux4166)
DNS: QUICKPAYMQPRD

SmartHub SSS

1.2.2 MQ Tadawul
MQ Tadawul

ETS CARONTE GBS

Middleware

MQ Tadawul Servers

Prod: 10.70.125.23 (rhlux3115) Prod: 10.70.125.24 (rhlux3116) Prod: 10.70.125.25 (rhlux3117)

DR: 10.80.125.23 (rhlux4115) DR: 10.80.125.24 (rhlux4116) DR: 10.80.125.25 (rhlux4117)
DNS: ETSMQ1PRD DNS: ETSMQ2PRD DNS: ETSMQ3PRD

NCB Internal
 The National Commercial Bank – ITD
IT Production Department
Document1
Rev Date: 01-Jun-2015 Rev: 2015-2 Doc No: SYS-153 Page No: 4 of 11

1.3 TECHNICAL INFORMATION

Refer to the NOC Database for managing the Windows servers. Individual login account is provided to NOC DB
support for database management.

Server Details:
MITC K10
Server Role DNS Name
Host Name IP Address Host Name IP Address
Tadawul MQ rhlux3115 10.70.125.23 ETSMQ1PRD rhlux4115 10.80.125.23
Tadawul MQ rhlux3116 10.70.125.24 ETSMQ2PRD rhlux4116 10.80.125.24
Tadawul MQ rhlux3117 10.70.125.25 ETSMQ3PRD rhlux4117 10.80.125.25
Enterprise MQ rhlux3166 10.70.125.34 QUICKPAYMQPRD rhlux4166 10.80.125.34

2.0 ROLES AND RESPONSIBILITIES

2.1 NETWORK OPERATIONS CENTER TEAM (L1 SUPPORT)
This team is responsible for monitoring the application and performs the necessary troubleshooting for any
problem as defined in this operations guide manual.

2.2 SYSTEMS ENGINEERING TEAM (L2 SUPPORT)

The system owner shall provide the necessary technical assistance for any problem that could not be resolved
after performing the troubleshooting procedure.

3.0 MONITORING
The following services are monitored by HP Openview and refer to the Troubleshooting section for the action required
when an alert is received in the OVO console:
3.1 MQ ENTERPRISE
Alert Severity / Message Description Action
[Critical]
Connection to the remote host is unexpectedly terminated.

The Connection between MQ client or BaNCs and MQ Server (10.70.x.x)

Inform the BaNCS team to re-establish the
is terminated, the automatic action is initiated. The auto-action result
connection.
will be sent once shortly.

Impact: Customer information is momentarily delayed to synchronize

between BaNCS and SmartHub.
[Major]
The <process name> is not running that may result to MQ service to
terminate if not restarted.
 Login to MQ server as user mqm
and run the command
MQ Processes Monitored:
/opt/mqm/bin/strmqm <qmgr>
/opt/mqm/bin/runmqchi
 Monitor that the service was restarted
/opt/mqm/bin/amqpcsea
after 5 to 10 minutes when alert was
/opt/mqm/bin/amqzmuc0
received.
/opt/mqm/bin/amqzlaa0
 If failed to restart then escalate by
/opt/mqm/bin/amqzxma0
phone and email to the Application
/opt/mqm/bin/runmqlsr
Admin to start the MQ service.
Impact: Customer Information (Cust_Info) will not be synchronized
between BaNCS and SmartHub.

NCB Internal
 The National Commercial Bank – ITD
IT Production Department
Document1
Rev Date: 01-Jun-2015 Rev: 2015-2 Doc No: SYS-153 Page No: 5 of 11

[Minor]
<monitoring object> is pending to read for a longer time than expected.
If message is not delivered and stuck in DeadLetterQueue.

/var/opt/OV/bin/instrumentation/MQ.sh
HighQueueDepth (more than 0)
DeadLetterQueue (more than 0
Send email to BANCS and SmartHub
HighMessageAge (more than 0)
Admins for further checking.
BANCS TO SmartHub Expiry
SmartHub to BANCS Expiry
SmartHub to BANCS Batch Expiry

Impact: Specific Cust_Info in Q will not be synchronized between BaNCS

and SmartHub. MQ Admin needs to retrieve and analyze the Qs' during
business hour.

3.2 MQ TADAWUL
Alert Severity / Message Description
[Minor] Escalate to Tadawul Team (Selex)
The script will monitor the Deal Letter Queue (DLQ) process in the 3
MQ Tadawul servers.

Script path:/var/mqm/script/
10.70.125.23: qdepth_DLQ_MQPRDGBS2
10.70.125.24: qdepth_DLQ_MQPRDGBS
10.70.125.25: qdepth_DLQ_MQPRDGBS3
[Critical] Escalate to MQ App Owner
This script will monitor the critical MQ processes to be running in the 3
MQ Tadawul servers.
 runmqchi
 runmqlsr
 amqpcsea
 amqzmuc0
 amqzlaa0
 amqzxma0

Script path:/var/mqm/script/ mqRunningProcess.sh

4.0 TROUBLESHOOTING
4.1 APPLICATION
4.1.1 Checking MQ Services
The following table lists the actions to be taken when an error is received:
ERROR Alert is received that one of the following services is not running or terminated
unexpectedly:
runmqchi
runmqlsr
amqpcsea
amqzmuc0
amqzlaa0
amqzxma0
ACTION Restart the service from the Linux Console.

NCB Internal
 The National Commercial Bank – ITD
IT Production Department
Document1
Rev Date: 01-Jun-2015 Rev: 2015-2 Doc No: SYS-153 Page No: 6 of 11

4.1.2 Application Logs

From the MQ application server, go to the log file:
/var/mqm/qmgrs/BANCSCUSTQMGR/errors/AMQERR0*.LOG

5.0 DISASTER RECOVERY

5.1 DR ACTIVATION
The switching of the application/system from Production to DR site takes place during the following scenarios:
 Outages where disaster was declared by the Systems Engineering Manager upon consultation with the
Emergency Response Team (refer to Disaster Recovery Procedure for detailed roles).
 Scheduled DR exercise (upon advise from IT Production management)

5.2 DR CHECKLIST FOR ENTERPRISE MQ

Date Application Enterprise MQ Activity Type Disaster Exercise
Done/
No Action Host Owner User ID
Time
Moving the Application to DR Site
Received from management an order to failover MQ Enterprise N/A N/A N/A
1 service to DR. Ask OE-Monitoring team to stop the monitoring from
MITC and turn on the probing and alerting in DR side.
OSL sends a notification (upon coordination with the Application N/A SE- N/A
2 Admin) regarding the failover of the MQ service from MITC to DR. Engineeri
ng
Confirm SSS Admin to stop their listener to connect MQ server in Back N/A
3
PROD. office
Confirm BaNCS team stops jobs to connect MQ server in PROD. Back N/A
4 BaNCS team to stop MQ0101 & MQ0201 jobs to stop connection to office
MQ server in Prod.
MQ Admin stops the MQ service in MITC by manually or using the SE- mqm
Control M automation. Channels
5
/var/mqm/scripts/ EndMQScript.sh

Unmount application FS in PROD server in MITC SE- N/A

(/appmqm) Technical
6 Support
Services-
UNIX
Failover to K10 SE- N/A
Enable RDM Disk in K10 Engineeri
7 PRD 10.70.125.33 EDC2 60000970000295700528533036324341 /appmqm ng
DR 10.80.125.34 EDC2 60000970000295700836533036324341 /appmqm

mount application File System in DR server in K10 SE- N/A

Technical
8 /dev/mapper/vg_appmqm-lvol0 /appmqm Support
Services-
UNIX
Point the DNS name K10 SE- N/A
9 From MITC To K10 DNS Name Engineeri
10.70.125.3 10.80.125.3 ng
rhlux3166 rhlux4166 QUICKPAYMQPRD
4 4
Start MQ service in DR by manually or using the Control M SE- mqm
10
automation. Channels
NCB Internal
 The National Commercial Bank – ITD
IT Production Department
Document1
Rev Date: 01-Jun-2015 Rev: 2015-2 Doc No: SYS-153 Page No: 7 of 11

/var/mqm/scripts/ EndMQScript.sh

SSS Application Admin need to start application to connect to MQ in N/A Back N/A
11 DR office

BaNCS team to start MQ0101 & MQ0201 jobs to start connection to Back N/A
12
MQ server in DR. office
MQ Admin ensures that all the services are started and ports are Clause SE- mqm
listening 1.2 Channels
 Check all the application systems (SSS/Cordys, BaNCS/SmartHub)
are connected.
 Check the status and transection statistics to Queue.

13

Check the Synchronization between the user created in NBL and N/A App N/A
CCR are synchronized and the process is completed Admin
Check the Synchronization between SSS and Cordys.

14

Trouble Shooting Scenario’s N/A App N/A

a) Check you are receiving the requests from NBL- Bancs14- Admin/M
15 MQ-SMARTHUB WAdmin/
b) Check you are receiving the requests from CCR/.NetMW – MQAdmin
Smarthub
Moving the Enterprise MQ Application back to MITC Site
Received from management an order to return the MQ Enterprise N/A N/A N/A
1 service to MITC. Ask OE-Monitoring team to return the application
monitoring to MITC and stop it in DR .
 OSL sends a notification (upon coordination with the Application N/A SE- N/A
2 Admin) regarding the return of the MQ service from MITC to DR. Engineeri
ng
 Confirm SSS Admin to start their listener to connect MQ server in Back N/A
3 DR. office

 Confirm BaNCS admin to stop their application(MQ Client) Back N/A

4  BaNCS team to stop MQ0101 & MQ0201 jobs to stop connection office
to MQ server in DR.
 Stop MQ service in DR. N/A SE- N/A
5
Channels
 umount /appmqm FS in DR MQ server. SE- N/A
6
Technical

NCB Internal
 The National Commercial Bank – ITD
IT Production Department
Document1
Rev Date: 01-Jun-2015 Rev: 2015-2 Doc No: SYS-153 Page No: 8 of 11

Support
Services-
UNIX
 Failback to MITC Clause SE- admin
 Enable RDM Disk in MITC 1.2 Engineeri
7 PRD 10.70.125.33 EDC2 60000970000295700528533036324341 /appmqm ng
DR 10.80.125.34 EDC2 60000970000295700836533036324341 /appmqm

 mount /appmqm File System in Prod MQ server in MITC SE- N/A

Technical
8 Support
Services-
UNIX
 Point the DNS name to MITC server. SE- N/A
Engineeri
9 From K10 To MITC DNS Name ng
10.70.125.3
10.80.125.34 Rhlux4166 Rhlux3166 QUICKPAYMQPRD
4
 Start MQ service in MITC SE- mqm
10
Channels
 SSS Application Admin need to start the application App N/A
11
Admin
 BaNCS team to start MQ0101 & MQ0201 jobs to start App N/A
12
connection to MQ server in MITC Admin
MQ Admin ensures that all the services are started and ports are MQAdmin mqm
listening

13  Check all the application systems (SSS/Cordys, BaNCS/SmartHub)

are connected.
 Check the status and transection statistics to Queue.

Check the Synchronization between the user created in NBL and N/A App N/A
CCR are synchronized and the process is completed Admin

14

Trouble Shooting Scenario’s N/A App N/A

c) Check you are receiving the requests from NBL- Bancs14- Admin/M
15 MQ-SMARTHUB WAdmin/
d) Check you are receiving the requests from CCR/.NetMW – MQAdmin
SmartHub

5.3 DR CHECKLIST FOR TADAWUL MQ

Date Application MQ_Tadawul Activity Type Disaster Exercise
Done/
No Action Host Owner User ID
Time
Moving the Application to DR Site
1 Received from management an order to failover MQ Tadawul N/A N/A N/A

NCB Internal
 The National Commercial Bank – ITD
IT Production Department
Document1
Rev Date: 01-Jun-2015 Rev: 2015-2 Doc No: SYS-153 Page No: 9 of 11

service to DR. Ask OE-Monitoring team to stop the monitoring from

MITC and turn on the probing and alerting in DR side.
Teams Notification (upon coordination with the Application Admin) N/A SE- N/A
regarding the failover of the MQ service for Tadawul from MITC to Engineeri
2
DR. ng

Confirm Application team has stopped the ETS service. N/A Back N/A
3
office
Confirm Application team has stopped the GBS service. N/A Back N/A
4
office
MQ Admin stops the MQ Tadawul service in MITC by manually or N/A SE- mqm
using the Control M automation. Channels
5 /var/mqm/scripts/ EndMQScript.sh

Unmount application FS in PROD server in MITC SE- N/A

Technical
(/appmqm)
6 Support
Services-
UNIX
 Failover to K10 SE- N/A
 Enable RDM Disk in K10 Engineeri
PRD 10.70.125.23 EDC1 60000970000295700528533036324337 /appmqm ng
PRD 10.70.125.24 EDC1 60000970000295700528533036324338 /appmqm

7 PRD 10.70.125.25 EDC1 60000970000295700528533036324339 /appmqm

DR 10.80.125.23 EDC4 60000970000295700836533036324337 /appmqm

DR 10.80.125.24 EDC4 60000970000295700836533036324338 /appmqm

DR 10.80.125.25 EDC4 60000970000295700836533036324339 /appmqm


mount application File System in DR server in K10 SE- N/A
Technical
8 Support
 /appmqm
Services-
UNIX
Point following the DNS to K10 SE- mqm
From MITC To K10 DNS Name Engineeri
10.70.125.2 rhlux311
rhlux4115
10.80.125.2 ng
9 3 5 3 ETSMQ1PRD
10.70.125.2 rhlux311 10.80.125.2
rhlux4116
4 6 4 ETSMQ2PRD
10.70.125.2 rhlux311 10.80.125.2
rhlux4117
5 7 5 ETSMQ3PRD
 Start MQ service in DR SE- mqm
/var/mqm/scripts/ StartMQScript.sh Channels
10

Confirm Application team has started the ETS service. App

11
Admin
Confirm Application has started the GBS service. App
12
Admin
13 MQ Admin ensures that all the services are started and ports are Clause SE- mqm

NCB Internal
 The National Commercial Bank – ITD
IT Production Department
Document1
Rev Date: 01-Jun-2015 Rev: 2015-2 Doc No: SYS-153 Page No: 10 of 11

listening 1.2 Channels

 Using login to the DR server and check the service, port and
Listener and connected MQ clients.
 Check if the MW, ETS and GBS IPs are connected to MQ

Check Queue and channel statics in DR servers. SE- mqm

14
Channels
Moving the MQ_tadawul Application back to MITC Site
Received from management an order to return the MQ Tadawul N/A N/A N/A
1 service to MITC. Ask OE-Monitoring team to return the application
monitoring to MITC and stop it in DR .
Teams Notification (upon coordination with the Application Admin) N/A SE- N/A
regarding the failover of the MQ service for Tadawul from MITC to Engineeri
2
DR. ng

Confirm Application team has stopped the ETS service. N/A SE- N/A
3
Channels
Confirm Application team has stopped the GBS service. N/A App admin
4
Admin
MQ Admin stops the MQ Tadawul service in Dr by manually or using N/A SE- mqm
the Control M automation. Channels
5
/var/mqm/scripts/ EndMQScript.sh

Unmount application FS in PROD server in MITC SE- N/A

(/appmqm) Technical
6 Support
Services-
UNIX
 Failback to MITC SE- N/A
 Enable RDM Disk in MITC Engineeri
PRD 10.70.125.23 EDC1 60000970000295700528533036324337 /appmqm ng
PRD 10.70.125.24 EDC1 60000970000295700528533036324338 /appmqm
7 PRD 10.70.125.25 EDC1 60000970000295700528533036324339 /appmqm

DR 10.80.125.23 EDC4 60000970000295700836533036324337 /appmqm

DR 10.80.125.24 EDC4 60000970000295700836533036324338 /appmqm

DR 10.80.125.25 EDC4 60000970000295700836533036324339 /appmqm

Mount RDM Disk in MITC by OS team. SE- N/A

Technical
8 Support
Services-
UNIX
 Point the DNS name to MITC N/A SE- N/A
To MITC From K10 DNS Name Engineeri
10.70.125.23 rhlux3115 rhlux4115 10.80.125.23
ng
ETSMQ1PRD
9 10.70.125.24 rhlux3116 rhlux4116 10.80.125.24
ETSMQ2PRD
10.70.125.25 rhlux3117 rhlux4117 10.80.125.25
ETSMQ3PRD

NCB Internal
 The National Commercial Bank – ITD
IT Production Department
Document1
Rev Date: 01-Jun-2015 Rev: 2015-2 Doc No: SYS-153 Page No: 11 of 11

 Start MQ service in DR App mqm

10 /var/mqm/scripts/ StartMQScript.sh Admin

Confirm Application team has started the ETS service. Back N/A
11
Office
Confirm Application team has started the GBS service. Back N/A
12
Office
MQ Admin ensures that all the services are started and ports are MQAdmin mqm
listening

13  Using login to the DR server and check the service, port and
Listener and connected MQ clients.
 Check if the MW, ETS and GBS IPs are connected to MQ

14 Check Queue and channel statics in prod in MITC servers MQAdmin mqm

6.0 ESCALATION
For any problem that could not be resolved according to the troubleshooting procedure, contact the system owner.

End of Document

NCB Internal