NetAct™ 15.

Preventive Health Check

DN09144478
Issue: 1-0
The information in this document is subject to change without notice and describes only the product
defined in the introduction of this documentation. This documentation is intended for the use of
Nokia Solutions and Networks customers only for the purposes of the agreement under which
the document is submitted, and no part of it may be used, reproduced, modified or transmitted
in any form or means without the prior written permission of Nokia Solutions and Networks. The
documentation has been prepared to be used by professional and properly trained personnel, and
the customer assumes full responsibility when using it. Nokia Solutions and Networks welcomes
customer comments as part of the process of continuous development and improvement of the
documentation.

The information or statements given in this documentation concerning the suitability, capacity, or
performance of the mentioned hardware or software products are given “as is“ and all liability arising
in connection with such hardware or software products shall be defined conclusively and finally in
a separate agreement between Nokia Solutions and Networks and the customer. However, Nokia
Solutions and Networks has made all reasonable efforts to ensure that the instructions contained
in the document are adequate and free of material errors and omissions. Nokia Solutions and
Networks will, if deemed necessary by Nokia Solutions and Networks, explain issues which may not
be covered by the document.

Nokia Solutions and Networks will correct errors in this documentation as soon as possible. I N N O
E V EN T WI LL N OKI A S O L UT I ON S A N D N E T W O R K S B E L I A B L E F O R E R R O R S
I N T H IS DO CU M EN TAT IO N OR F O R A N Y D A M A G E S , I N C L U D I N G B U T N O T
L IM I T ED TO SPEC IAL, D IR E C T, IN D I R E C T, I N C I D E N TA L O R C O N S E Q U E N T I A L
O R AN Y LOSSES, SU CH A S B U T N O T L I M I T E D TO L O S S O F P R O F I T, R E V E N U E,
B USI NESS IN T ER RU P T I ON , B U S I N E S S O P P O RT U N I T Y O R D ATA , T H AT M AY
A RI SE F RO M T H E USE OF T HI S D O C U M E N T O R T H E I N F O R M AT I O N I N I T.

NSN is a trademark of Nokia Solutions and Networks. Nokia is a registered trademark of Nokia
Corporation. Other product names mentioned in this document may be trademarks of their
respective owners, and they are mentioned for identification purposes only.

Copyright © Nokia Solutions and Networks 2015/2/10. All rights reserved.

Nokia Solutions and Networks is continually striving to reduce the adverse environmental effects of
its products and services. We would like to encourage you as our customers and users to join us
in working towards a cleaner, safer environment. Please recycle product packaging and follow the
recommendations for power use and proper disposal of our products and their components. If you
should have questions regarding our Environmental Policy or any of the environmental services we
offer, please contact us at Nokia Solutions and Networks for additional information.
Preventive Health Check

Contents
1 About Preventive Health Check.....................................................................................................................4
1.1 Preventive Health Check contents............................................................................................................4

2 Running Preventive Health Check.................................................................................................................7

2.1 Report directory structure....................................................................................................................... 10
2.1.1 Report cleanup................................................................................................................................11
2.2 Test cases............................................................................................................................................... 12
2.2.1 NetAct module................................................................................................................................ 12
2.2.1.1 Fault management..................................................................................................................13
2.2.1.2 High availability.......................................................................................................................15
2.2.1.3 NetAct system information..................................................................................................... 20
2.2.1.4 Performance management..................................................................................................... 21
2.2.1.5 WebSphere Application Server.............................................................................................. 24
2.2.1.6 Configuration Management (NAC)......................................................................................... 25
2.2.1.7 License Manager & SWAM.................................................................................................... 26
2.2.2 Optimizer module............................................................................................................................27
2.2.3 RHEL module..................................................................................................................................28
2.2.4 ORACLE..........................................................................................................................................30

3 Scheduling Preventive Health Check.......................................................................................................... 34

3
Preventive Health Check About Preventive Health Check

1 About Preventive Health Check

Preventive Health Check (PHC) is a command line tool to check the health of the NetAct system.
Preventive Health Check is designed to monitor the health of cluster by execution of test suite.

Preventive Health Check enables you to:

• generate summary report on the state of the system.

• collect symptoms to resolve the problems.
• check the health of the system after NetAct deployment (after fresh installation or upgrade).
• verify all nodes using central remote execution utility.
• generate consolidated summary report and collect logs from all the nodes on the node where the
tool is invoked.

Preventive Health Check Tool script covers the following products:

• NetAct
• NetAct Configurator (both Core and Radio)
• Optimizer

1.1 Preventive Health Check contents

Preventive Health Check is part of NetAct media. After installation, it is available on all nodes and can
be invoked as omc user from any NetAct node except performance management nodes.

After installation, the tool can be found in /opt/oss/NSN-mhcf directory on all the nodes of NetAct
Cluster.

The Directory folders figure below shows the contents of the Preventive Health Check installation di-
rectory.

Figure 1: Directory folders

Preventive Health Check installation directory consists of the following:

Issue: 1-0 DN09144478 4

Preventive Health Check About Preventive Health Check

Directory Description

lib Contains Perl and Shell li-

braries used in Preventive
Health Check.

product Contains configuration files re-

quired to execute the Preven-
tive Health Check as a com-
plete testsuite. It also con-
tains sub folders with configu-
ration files specific to respec-
tive OSS products divided in-
to modules or sub-modules.
Each folder and its sub folder
contain test scripts (Perl) re-
quired to execute the test cas-
es.

Note:

Each product re-

lated health check
scripts are present
under respective
folder names:

• netact 8(includ-
ing NAC scripts)
• optimizer

netact8 lib Contains perl modules with

helper functions

tools Contains scripts such as Shell,

SQL, and Perl that are used
as tools in test scripts.

xslt Contains files that transforms

the generic XML into HTML
format.

net- Script enables the mhcf user

act8-root_ to run the programs with the
privileges. security privileges of roo-
pl tuser.

Issue: 1-0 DN09144478 5

Preventive Health Check About Preventive Health Check

Directory Description

optimizer lib Contains perl modules with

helper functions

tools Contains scripts such as Shell,

SQL, and Perl that are used
as tools in test scripts.

xslt Contains files that transforms

the generic XML into HTML
format.

optimiz- Script enables the mhcf user

er-root_ to run the programs with the
privileges. security privileges of root user.
pl

mhcf.pl Main script that executes Pre-

ventive Health Check.

Table 1: Directory structure

Issue: 1-0 DN09144478 6

Preventive Health Check Running Preventive Health Check

2 Running Preventive Health Check

To run Preventive Health Check:

1. Log in as omc user to the node where you want to run the health check. This is applicable to all
NetAct nodes except the performance management nodes.

2. Start Preventive Health Check by executing:

mhcf.pl -c <path of configuration file>

where:

• mhcf.pl is tool executable

• <configuration file> is configuration file for the complete testsuite

3. To run Preventive Health Check with log collection execute:

mhcf.pl -c <path of configuration file> -loglevel <value>

where:

<value> is value of the log level. Example: small, medium and all

For example,

• To execute the PHC for NetAct scenarios (including NetAct for Core and Performance Man-
agement) the command is:

mhcf.pl -c /opt/oss/NSN-mhcf/product/netact8/cfg_netact8.xml

Note:

During NetAct Upgrade, the PHC tool should not be executed. If PHC is scheduled
for automatic execution, such scheduling should be disabled.

• To execute the PHC for Optimizer the command is:

mhcf.pl -c /opt/oss/NSN-mhcf/product/optimizer/cfg.optimizer.xml

Note:

If Password Policy is enforced in the system, then it may happen that the password of
internal backend user such as mhcf is expired on any node, the test cases executed
on that node fails as Preventive Health Check tool cannot make SSH connection to that
node.

In such cases the following error message is displayed on the screen and in the output
files.

1.1.2 Node: <nodename>

Checking connection...WARNING: Your password has expired.

Issue: 1-0 DN09144478 7

Preventive Health Check Running Preventive Health Check

Password change required but no TTY available.

[ERROR] Can't establish SSH connection to node. Cannot run
tests on this host.
[FAILED] Empty answer from script

You must manually change the password on all nodes of the system. The user security
hardening measures mentioned in the Administering NetAct System Security
document is also applicable for mhcf user.

Password expiry duration is as per password policy, which is 180 days by default

Expected outcome

The command output is either passed or failed. A result folder is created in /var/opt/nokia/
oss/global/NSN-mhcf/<user>/<product>/ directory. For more information, see Report
directory structure on page 10.

Note:

You can use timeout option in the configuration to define maximum execution time for sin-
gle test case. If test case execution exceeds the defined maximum execution time then
the test case fails.

<defaultTimeOut>90</defaultTimeOut>

Few test cases might have its own timeout defined in the test configuration.

Minimum or basic symptoms for quick analysis of the problem are collected on every execution
of the tool with loglevel = small, regardless of a test case pass or fail. If test case fails,
Preventive Health Check can be executed with loglevel = medium and filter=module
name or testSet=testset name.

Example

Additional Information

• To view the list of supported commands of the Preventive Health Check, execute:

mhcf.pl -help

• To run specific configuration file that contains the test case, execute:

mhcf.pl -c /opt/oss/NSN-mhcf/product/netact8/cfg_netact8.xml

• To run specific test set, execute:

mhcf.pl -c /opt/oss/NSN-mhcf/product/netact8/cfg_netact8.xml -t
product/netact8/subcfg_netact8.xml

Issue: 1-0 DN09144478 8

Preventive Health Check Running Preventive Health Check

Note:

subcfg_netact8.xml is a sample testset configuration, which you can refer to create

customized test set configuration for different requirements.

• To run test case for NetAct for Core based on filterTags=cm specified in the test case,
execute:

mhcf.pl -c /opt/oss/NSN-mhcf/product/netact8/cfg_netact8.xml -f cm

Note:

filterTags can be cm, rhel, nas_common, fm, ha, was, pm. You can provide
multiple filterTags as follows:

mhcf.pl -c /opt/oss/NSN-mhcf/product/netact8/cfg_netact8.xml -
f cm,rhel,nas_common,fm,ha,was,pm

For NetAct for Core testcases, the filter option is cm

For Performance Management testcases, the filter option is pm

• To list all the available test cases, execute:

mhcf.pl -c /opt/oss/NSN-mhcf/product/netact8/cfg_netact8.xml -list

For information on the available test cases, see Test cases on page 12.

• To run specific test case assigned to specific role, such as db, execute:

mhcf.pl -c /opt/oss/NSN-mhcf/product/netact8/cfg_netact8.xml -r db

Note:

Role can be dmgr, fm_pipe and was. Multiple role values can be provided in the
same command as follows:

mhcf.pl -c /opt/oss/NSN-mhcf/product/netact8/cfg_netact8.xml -
r db,dmgr,fm_pipe,was

• To suppress the output, execute:

mhcf.pl -c /opt/oss/NSN-mhcf/product/netact8/cfg_netact8.xml -q

• To run specific test case based on the test case number, execute:

mhcf.pl -c opt/oss/NSN-mhcf/product/netact8/cfg_netact8.xml -single

1.2

where 1.2 is the test case number.

Note:

To view test case number, execute the command with -list option.

Issue: 1-0 DN09144478 9

Preventive Health Check Running Preventive Health Check

• To collect logs, execute the tool with -loglevel option:

mhcf.pl -c /opt/oss/NSN-mhcf/product/netact8/cfg_netact8.xml -loglevel

small

• To list available logs use -listlogs option:

mhcf.pl -c /opt/oss/NSN-mhcf/product/netact8/cfg_netact8.xml -listlogs

Note:

• Allowed Log Levels:

– small: quick logs; defined as level=4 in each module log configuration file
– medium: detailed logs, including core dumps; defined as level=2 in each
module log configuration file
– all: both small & medium logs
• Logs are collected based on the tag collectLogs="always" or
collectLogs="onError" in the log configuration file:
• The most preferred and default command is mhcf.pl -c /opt/oss/NSN-mhcf/
product/netact8/cfg_netact8.xml -loglevel small.

To see simple statistics at the end of the report use -stats option:

mhcf.pl -c /opt/oss/NSN-mhcf/product/netact8/cfg_netact8.xml -stats

2.1 Report directory structure

The report directory contains the summary report and symptoms collected by Preventive Health
Check.

The PHC reports are located in /var/opt/nokia/oss/global/NSN-mhcf/<user>/<product>/

directory. Each OSS product will have its own log sub-folder, for example, netact8 or optimizer.

Example:

/var/opt/nokia/oss/global/NSN-mhcf/omc/netact8/

The summary report lists the test cases executed on each node and provides a summary of the test
results. The summary contains the description of each test and result of the test execution. If any test
case fails, the reason for the failure and suggested action are described.

The symptoms collected are stored in .tar.gz file. Summary report is available in the following
formats:

• .txt - Summary report in text format is same as the console output . Open the file to view the
report on a console.
• .html - Summary report in .html format enables to view the report in a browser.
• .xml - Summary report in .xml format is ASWS compatible for integration to care tools.

Issue: 1-0 DN09144478 10

Preventive Health Check Running Preventive Health Check

Contents of logs directory is shown below.

Figure 2: Log directory contents

Example of a summary report:

Figure 3: Summary report

2.1.1 Report cleanup

Reports are being cleaned based on the configuration in the /opt/oss/NSN-mhcf/product/
netact8/cfg_netact8.xml file.

There are several fields responsible for cleanup configuration:

1. Reports cleanup policy [time or size or both]

time: reports and logs are cleaned when older than value in reportsRetentionTimeInDays

size: reports and logs are cleaned when size of the report directory exceeds value given in report-
sSizeThresholdInMB. Files are removed starting from the oldest until the size of the report direc-
tory become smaller than above threshold.

Issue: 1-0 DN09144478 11

Preventive Health Check Running Preventive Health Check

both: both of the above options are used <reportsCleanUpPolicy>both</

reportsCleanUpPolicy>
2. Reports cleanup range [all or logsonly]

This value tells the tool which files should be removed during clean up.

all: all files are removed

logsonly: only log files archives are removed, the reports are retained.
<reportsCleanUpRange>all</reportsCleanUpRange>
3. Reports cleanup retention time in days

Reports older than this value are removed <reportsRetentionTimeInDays>5</

reportsRetentionTimeInDays>
4. Reports cleanup size threshold

Reports are removed when size of the reports directory exceeds this value
<reportsSizeThresholdInMB>200</reportsSizeThresholdInMB>

2.2 Test cases

Preventive Health Check can be executed by using the following testcases (or product areas)::

• RHEL test cases checks the operating system and hardware. For more information, see RHEL
module on page 28.
• NetAct test case cover NetAct for Core, Fault Management, high availability, NetAct system infor-
mation, Performance Management, and, WebSphere application server. For more information, see
NetAct module on page 12.
• Optimizer testcases are listed in Optimizer module on page 27.

For information on how to run a test case, see Running Preventive Health Check on page 7.

2.2.1 NetAct module

NetAct module is categorized into:

• Configuration management
• Fault management
• High Availability
• NetAct system information
• Performance management
• WebSphere application server
• License Management
• Software Asset Monitoring (SWAM)

Issue: 1-0 DN09144478 12

Preventive Health Check Running Preventive Health Check

Note:

Check the filterTag in the test case for the module name. The filterTag can be cm,
rhel, nas_common, fm, ha, was, pm. You can execute the modules separately by providing
the module name as filter in the command. For more information on how to execute modules,
see Running Preventive Health Check on page 7.

2.2.1.1 Fault management

Execute on
Test case title node with Test case description
Role

FM Alarm Threshold dmgr Checks that the average alarm

insertion time is less than the
threshold of 120 seconds for
each alarm in the database.
Test fails if insertion time is
greater than the threshold.

DB Buffer checks and fm_pipe Checks that:

FM alarm checks
• the FX alarm table buffer
hit ratio is greater than 98
• the active alarm count is
less than 100000 thresh-
old.

Test fails even if one of the

condition is not met.

GEP Status dmgr Checks that the GEP instance

of servicemix is running. Test
fails if GEP instance of ser-
vicemix is not running.

FM Trap Delivery fm_pipe Checks that the trap delivery

Check is working. This is done by
checking if mediation is lis-
tening to the traps and traps
are auto forwarded. Test fails
if any of these checks fail be-
cause the OFaS files are not
written to the import directory.

GEP Process Memory fm_pipe Logs the Used Memory, Com-

Check mitted Memory, and Max
Memory. If Preventive Health

Issue: 1-0 DN09144478 13

Preventive Health Check Running Preventive Health Check

Execute on
Test case title node with Test case description
Role
Check is unable to retrieve
these data, the test fails.

Internal Alarm List dmgr Display the NetAct internal/

SelfMon Alarms raised past
(default 48) hour(s). Change
the default duration in /opt/oss/
NSN-mhcf/product/netact8/
fm/fm_config.xml. Parameter:
<var id="internalAlarmHours"
name="Internal Alarm duration
in hours" value="48" />

Daily Alarm Rate dmgr Lists 'daily alarm rate' for last
7 days excluding present-day.
This is calculated based on
alarm insertion time in FM
database. This report also
displays additional informa-
tion about daily count of ac-
tive-non-acknowledged, ac-
knowledged and cancelled
alarms for last 7 days. Test
fails if daily alarm rate cross-
ing threshold limit of 300K/
day. The report shows "fail"
against the dates alarm rate
is exceeding limit. Thresholds
can be changed in file: /opt/
oss/NSN-mhcf/product/net-
act8/fm/fm_config.xml. Para-
meters:

• <var id="days"
name="Number of days"
value="7" />
• <var
id="dailyAlarmThreshold"
name="Threshold for
daily alarm rate" val-
ue="300000" />

Issue: 1-0 DN09144478 14

Preventive Health Check Running Preventive Health Check

Execute on
Test case title node with Test case description
Role

NEs with most alarms dmgr Lists top 10 network elements

having most number of open
alarms at present. Test fails if
open alarm count for any NE
is crossing a threshold limit of
100K/NE. Thresholds can be
changed in file: /opt/oss/NSN-
mhcf/product/netact8/fm/fm_
config.xml. Parameters:

• <var id="numNE"
name="This value in-
dicates number of high
alarm NEs to be shown in
report" value="10" />
• <var
id="highAlarmThreshold"
name="Threshold for high
alarm rate per NE" val-
ue="100000" />

FM Import File Count fm_pipe Checks that the number of

files in FM Import folders is
less than 10000 threshold.
Test fails if file count exceeds
the threshold and the current
number of files is recorded in
the south FM import directory.

Table 2: Fault management test cases

2.2.1.2 High availability

Execute on
Test case title node with Test case description
Role

Nfs service check db (This test- Checks that nfs is running.

case needs Test fails if it is not running.
to be run on-
Netfs service check Checks that netfs-<node
ly once hence
name> is running. Test fails if it
a node with
is not running.
unique role

Issue: 1-0 DN09144478 15

Preventive Health Check Running Preventive Health Check

Execute on
Test case title node with Test case description
Role

OMAgent AF service like 'db' is se- Checks that OMA-

check lected here) gentAF-<node name> is
running. Test fails if it is not
running.

Dirsrv service check Checks that dirsrv is run-

ning. Test fails if it is not run-
ning.

tomcat_vcentplg ser- Checks that netfs-<node

vice check name> is running. Test fails if it
is not running.

db service check Checks that db is running.

Test fails if it is not running.

lb-unify-primary service Checks that lb-unify-primary

check is running. Test fails if it is not
running.

lb-unify-backup service Checks that lb-unify-backup

check is running. Test fails if it is not
running.

OM Agent service Checks that OMAgent is run-

check ning. Test fails if it is not run-
ning.

hpsim service check Checks that hpsim is running.

Test fails if it is not running.

pingServer service Checks that pingServ-

check er-<node name> is running.
Test fails if it is not running.

tnameserver service Checks that tnameserv-

check er-<node name> is running.
Test fails if it is not running.

Nbi3gc service check Checks that nbi3gc is run-

ning. Test fails if it is not run-
ning.

Rac3gp service check Checks that rac3gp is run-

ning. Test fails if it is not run-
ning.

Issue: 1-0 DN09144478 16

Preventive Health Check Running Preventive Health Check

Execute on
Test case title node with Test case description
Role

ftp service check Checks that ftp-<node

name> is running. Test fails if it
is not running.

Nwi3 service check Checks that nwi3 is running.

Test fails if it is not running.

Nwi3-http service check Checks that nwi3-http is

running. Test fails if it is not
running.

jacorb service check Checks that jacorb is run-

ning. Test fails if it is not run-
ning.

osi service check Checks that osi is running.

Test fails if it is not running.

Q3user service check Checks that q3user is run-

ning. Test fails if it is not run-
ning.

socks service check Checks that socks is running.

Test fails if it is not running.

fmascii service check Checks that fmascii is run-

ning. Test fails if it is not run-
ning.

nbisnmp service check Checks that nbisnmp is run-

ning. Test fails if it is not run-
ning.

Nbi3gcpm service Checks that nbi3gcpm is run-

check ning. Test fails if it is not run-
ning.

Common_mediations Checks that common_media-

service check tions-<node name> is run-
ning. Test fails if it is not run-
ning.

httpd service check Checks that httpd is running.

Test fails if it is not running.

Issue: 1-0 DN09144478 17

Preventive Health Check Running Preventive Health Check

Execute on
Test case title node with Test case description
Role

FM_pipe service check Checks that fm_pipe-<node

name> is running. Test fails if it
is not running.

dmgr service check Checks that dmgr is running.

Test fails if it is not running.

was service check Checks that was-<node

name> is running. Test fails if it
is not running.

ihs service check Checks that ihs-<node

name> is running. Test fails if it
is not running.

lb service check (Per- Checks that lb-<node name>

formance management) is running. Test fails if it is not
running.

db_crons service check Checks that db_crons is run-

(Performance manage- ning. Test fails if it is not run-
ment) ning.

system service check Checks that system is running.

(Performance manage- Test fails if it is not running.
ment)

message service check Checks that message is run-

(Performance manage- ning. Test fails if it is not run-
ment) ning.

base service check Checks that base-<node

(Performance manage- name> is running. Test fails if it
ment) is not running.

httpd service check Checks that httpd-<node

(Performance manage- name> is running. Test fails if it
ment) is not running.

cmessage service Checks that cmessage is run-

check (Performance ning. Test fails if it is not run-
management) ning.

dns-namesurfer service Checks that dns-namesurfer

check (Performance is running. Test fails if it is not
management) running.

Issue: 1-0 DN09144478 18

Preventive Health Check Running Preventive Health Check

Execute on
Test case title node with Test case description
Role

osscore service check Checks that osscore is run-

(Performance manage- ning. Test fails if it is not run-
ment) ning.

osscore2 service check Checks that osscore2 is run-

(Performance manage- ning. Test fails if it is not run-
ment) ning.

nww service check Checks that nww is running.

(Performance manage- Test fails if it is not running.
ment)

mvi service check (Per- Checks that mvi is running.

formance management) Test fails if it is not running.

rep service check (Per- Checks that rep is running.

formance management) Test fails if it is not running.

dmgr_linas service Checks that dmgr_linas-<node

check (Performance name> is running. Test fails if it
management) is not running.

was_linas service Checks that was_linas-<node

check (Performance name> is running. Test fails if it
management) is not running.

nodeagent_linas ser- Checks that nodeagent_linas-

vice check (Perfor- <node name> is running. Test
mance management) fails if it is not running.

ihs_linas service check Checks that ihs_linas-<node

(Performance manage- name> is running. Test fails if it
ment) is not running.

DNS-Master service Checks that DNS-Master is

check running. Test fails if it is not
running.

DNS-Slave service Checks that DNS-Slave is run-

check ning. Test fails if it is not run-
ning.

Load Balance lookup 1. Check for nslookup of

and url check the WAS Load Balancer IP
(CPF.LB.WAS_ACCESS_
ADDRESS)

Issue: 1-0 DN09144478 19

Preventive Health Check Running Preventive Health Check

Execute on
Test case title node with Test case description
Role
2. Check this url https:/
/<loadbalancer_IP>/
opens without any error

Table 3: High availability test cases

Note: High availability testcases must be run only once. Therefore, execute these testcases
on the node with unique role db.

2.2.1.3 NetAct system information

Execute on
Test case title node with Test case description
Role

NetAct System Info db Lists NetAct system information such as ClusterID, Cluster-
Name, TargetID , Domain name. Also display Oracle & WAS
versions. Test fails if it is not able to fetch the required informa-
tion.

NetAct Release Info dmgr Lists the Software release information. Test fails if it is not able
to fetch the required information.

DB Size Check db Checks that the DB usage is below 80% threshold. Test fails if
usage exceeds threshold.

Global Disk Size Check nfs Check folders mounted on NFS Server are less than 95%
(threshold) full. Folders considered:

• /home/
• /d/oss/global
• /home
• /packages/var/opt/nokia/oss/global

The threshold of 80% can be changed in /opt/oss/NSN-mhcf/

product/netact8/nas_common/nas_common_config.xml

NFS Server check for nfs Check /home/ and /d/oss/global folders exported on NFS
exported folders Server.

1. The above folders must be listed in /etc/export

2. They are mounted successfully. showmount -e command
must list above folders.

NFS Client check for netfs Checks if /home and /d/oss/global folders are mounted as
mounted folders 'nfs' file system successfully.

Issue: 1-0 DN09144478 20

Preventive Health Check Running Preventive Health Check

Table 4: NetAct system information related test cases

2.2.1.4 Performance management

Execute on
Test case title node with Test case description
Role

PM files in import di- rep Check the number of files in

rectories PM Import directories (/var/
opt/nokia/oss/osscore/
iscirp/pmupld_osscore/
*) and ensure it's less than the
given threshold 10000

PM files in download Check the number of files

directories in PM Download directo-
ries (/var/opt/nokia/
oss/rep*/etload/pg_
repetl*/work/download)
and ensure it's less than the
given threshold 10000

PM files in load direc- Check the number of files in

tories PM Load directories (/var/
opt/nokia/oss/rep*/
etload/pg_repetl*/
work/load_*) and ensure it's
less than the given threshold
10000

PM files in notify direc- Check the number of files in

tories PM Notify directories (/var/
opt/nokia/oss/rep*/
etload/pg_repetl*/
work/notify) and ensure it's
less than the given threshold
10000

PM files in backup di- Check the number of files in

rectories PM Backup directories (/var/
opt/nokia/oss/rep*/
etload/pg_repetl*/
work/backup) and ensure it's
less than the given threshold
10000

Issue: 1-0 DN09144478 21

Preventive Health Check Running Preventive Health Check

Execute on
Test case title node with Test case description
Role

PM files in temporary Check the number of files in

error directories PM Temporary Error direc-
tories (/var/opt/nokia/
oss/rep*/etload/pg_
repetl*/work/temp_err)
and ensure it's less than the
given threshold 10000

PM files in permanent Check the number of files in

error directories PM Permanent Error direc-
tories (/var/opt/nokia/
oss/rep*/etload/pg_
repetl*/work/perm_err)
and ensure it's less than the
given threshold 10000

PM files in export di- Check the number of files in

rectories PM Export directories (/var/
opt/nokia/oss/rep*/
etload/pg_repetl*/
work/export) and ensure it's
less than the given threshold
10000

ETL process check Check the PM-core processes,

log files and metadata tables.
In case of process malfunction
the corresponding werlog file
is displayed

Daily/Hourly Aggrega- Check the duration of Hourly

tions and Daily aggregations using
cosprcmx.pl capacity log files
processed and collected by
SelfPM. The threshold of max-
imum hourly/daily Aggregation
can be changed in file: /opt/
oss/NSN-mhcf/product/
netact8/pm/pm_config.
xml

Parameters:

Issue: 1-0 DN09144478 22

Preventive Health Check Running Preventive Health Check

Execute on
Test case title node with Test case description
Role
• <var
id="pmHourlyAggregation"
name="PM Maximum Ag-
gregation Duration Hourly"
value='900' />
• <var
id="pmDailyAggregation"
name="PM Maximum Ag-
gregation Duration Daily"
value='10800' />

PM deletion time - Check deletion duration using

partitioned partition manager (pmpmgr-
mx.pl) log files processed and
collected by SelfPM. The du-
ration is default 500 seconds
can be changed in file /opt/
oss/NSN-mhcf/product/
netact8/pm/pm_config.
xml

Parameter: <var
id="pmDeletionPart"
name="PM Maximum Deletion
Duration of Partitioned tables"
value='500' />

PM deletion time - non Check deletion duration using

partitioned pm cleanup process (etload_
pm_meas_clean.pl) log files.
The duration is default 1800
seconds can be changed in
file /opt/oss/NSN-mhcf/
product/netact8/pm/pm_
config.xml

Parameter: <var
id="pmDeletionNonPart"
name="PM Maximum Deletion
Duration of Non-Partitioned ta-
bles" value='1800' />

Issue: 1-0 DN09144478 23

Preventive Health Check Running Preventive Health Check

Execute on
Test case title node with Test case description
Role

PM amount of inserted Display the amount of records

data and counters of inserted PM
data per day per adaptation

PM LinAS NetAct Sta- dmgr_linas Check the status of LinAS-

tus cluster executing linas_status.
sh script

Table 5: Performance management test cases

2.2.1.5 WebSphere Application Server

Execute on
Test case title node with Test case description
Role

Websphere status : was Checks that the Node Agent

nodeagent process is running. Test fails if
the process is not running.

Websphere status : Checks that the JMSCluster

cluster and SOLCluster processes are
running. Test fails if the either
of these process is not run-
ning.

Websphere status : lis- Checks that the Websphere

tener listener process is running.
Test fails if the process is not
running.

Websphere status : jd- Checks that the WAS JDBC

bc_connection is connected to the database.
Test fails if the WAS JDBC is
not connected.

Temporary directory Checks that the free space is

disk space Check 1048576 bytes in the /var/
tmp directory of the node
where was services are run-
ning. Test fails if free disk
space is less than the thresh-
old.

Issue: 1-0 DN09144478 24

Preventive Health Check Running Preventive Health Check

Execute on
Test case title node with Test case description
Role

NTP Sync Check Checks that the nodes are

synchronized with NTP serv-
er. Test fails if it is not synchro-
nized.

Jetty Webserver Test Checks that the Jetty webserv-

er is running. Test fails if it is
not running.

WAS Heap Dump Checks that there are any

Checks heap dumps in NetAct appli-
cations, WAS phd, and WAS
javacore. Test fails if heap
dump exists.

Table 6: WebSphere Application Server test cases

2.2.1.6 Configuration Management (NAC)

Execute on
Test case title node with Test case description
Role

CM applications status was Check websphere CM applica-

tions status

Check mediation or respective Check NWI, Q3 and NE3S

processes status mediation mediation or processes status
nodes and also search for ERROR
string in respective log checks.
The test fails if it finds any ER-
ROR string in log files

common_me- Check list of deployed SA

diations (Service Assemblies) into
SMX

CM Topology nwi3 Displays the Number of NE in-

tegrations. The table is sort-
ed based on Adaptation, Ver-
sion and Operational state.
The test fails if there are no in-
tegrations. It also generates a
detailed CM\topology.log file
when logging (loglevel=small)

Issue: 1-0 DN09144478 25

Preventive Health Check Running Preventive Health Check

Execute on
Test case title node with Test case description
Role
is enabled. The log contains
the number of NE integrations
sorted on the Adaptation, Ab-
brevation, Version and Opera-
tional state.

CM Adaptation Check dmgr Check installed adaptations

status using the script: /opt/
oss/d3b2v2/install/
bin/sort_adaptations.
sh

Testcase will fail if any of the

adaptations check fails from
above script. Detailed out-
put of the sort_adaptations.sh
script is shown in the log folder
CM\sort_adaptations.log (dur-
ing loglevel=small option)

Doctormx Check dmgr Check doctormx log files from

past 1 day(s). The default du-
ration can be changed in file: /
opt/oss/NSN-mhcf/prod-
uct/netact8/cm/cm_doc-
tormx_config.xml

<var id="days"
name="Duration (in days)
of Doctor logs to check" val-
ue="1" />

DB Checks db List number of ongoing and

querying CM operations : up-
load and provisioning

Table 7: NetAct for Core test cases

2.2.1.7 License Manager & SWAM

The following table lists the License Manager & SWAM test cases.

Issue: 1-0 DN09144478 26

Preventive Health Check Running Preventive Health Check

Execute on node
Test case title Test case description
with Role

Main NetAct SW License and dmgr Check if NetAct SW

CJLA Check main(mother) license exists
and is valid - feature code
5146. This is main license
needed in order to use NetAct.
Also checks if CJLA service
is working. This service is de-
livering license information for
NetAct application.

Check Number of Conflicting NE/ db Check if in LIC database ex-

NMS Licenses ists entries which indicate NE
or NMS Licenses which are in
state conflicting.

Check Time of Validity of NetAct Check if in LIC database ex-

SW Licenses ists entries which indicate that
there is no NetAct SW licens-
es which have time of validity
greater than 30 days. It means
that validity of NetAct SW li-
censes will end up soon.

Check If SWAM default Reports Check if in SWAM database

Are Generated exist entry for "NetAct Ra-
dio" report with date of cre-
ation equal or newer than
yesterday. It means SWAM
is working and at least "Net-
Act Radio" (default) report
is generated according to
schedule(daily).

Table 8: License Manager & SWAM test cases

2.2.2 Optimizer module

Execute
Test case title on node Test case description
with Role

List of servers dmgr Check and display all avail-

able nodes in the system (from
smanager.pl script)

Issue: 1-0 DN09144478 27

Preventive Health Check Running Preventive Health Check

Execute
Test case title on node Test case description
with Role

Optimizer Build Number and Customer Name Check the latest bulid number

Disk space details of UMA and Optimizer ta- Check and display table space
bles space in Optimizer

List of KPIs selected for Auto Retrieval – as Check and point all active KPI
zipped filed attached to log

Total Number of Plans in Optimizer and Total Check and display all avaliable
number of plans in optimizer which are older plans in Optimizer and point all
than 60 days older than 60 days

Count of Managed Objects - as zipped filed at- Check and display amount of
tached to log Managed Objects

Row Space Consumed in OPTDB – as zipped Check and point Row Space
file attached to log Consumed in OPTD

Installed version on all nodes Check and point active version

of upgraded software

Check of LinAs application status Check the LinaAs status and

display

Table 9: Optimizer test cases

2.2.3 RHEL module

The following table lists the RHEL test cases.

Execute on
Test case title Node(s) with Test case description
ROLE

HW check ALL Nodes Checks the hardware spec-

(both NetAct & ification such as processor
Performance count, RAM size, and hard
Management) disk drive size. There also
a check for swap usage per
process and information is
collected in log folder RHEL
\swap_users.log (when log-
ging is enabled)

RHEL message Log Checks for specific error

Check keywords in last x (default
500) lines of the /var/log/

Issue: 1-0 DN09144478 28

Preventive Health Check Running Preventive Health Check

Execute on
Test case title Node(s) with Test case description
ROLE
message file. The keywords
can be specified in /opt/
oss/NSN-mhcf/product/
netact8/rhel/rhel_
config.xml; change the
value for id=errString1
<var id="errString1"
name="Error pattern
to check in /var/log/
messages file; pattern
separated with /"
value="error/Error/
ERROR" />

To change the line count,

the id errorString1Count
value can be changed in
the same xml file: <var
id="errString1Count"
name="Trailing number
of lines to check in
the /var/log/messages
file; default 500"
value="500" />

RHEL secure Log Checks for error keywords

Check in last x (default 500) lines
of /var/log/secure
file. The keywords can be
specified in /opt/oss/
NSN-mhcf/product/
netact8/rhel/rhel_
config.xml ; change the
value for id=errString1
<var id="errString1"
name="Error pattern
to check in /var/log/
secure file; pattern
separated with /"
value="error/Error/
ERROR" />

Issue: 1-0 DN09144478 29

Preventive Health Check Running Preventive Health Check

Execute on
Test case title Node(s) with Test case description
ROLE
To change the line count,
the id errorString2Count
value can be changed in
the same XML file: <var
id="errString1Count"
name="Trailing number
of lines to check in
the /var/log/secure
file; default 500"
value="500" />

Node Performance Checks that the load on the

Check – System Load CPU (application and operat-
Check ing system) is less than 80%.
Test fails if it exceeds 80%.
You can check the current
CPU load in the log file.

Duplicate IP Check Checks that the system IP is

used in another machine in the
network and reports the MAC
address of the same. Test fails
if the test case finds same IP.

NTP Synch Check Checks that the system time

is in synchronization with NTP
Server. Test fails if the test
case finds any system not
in synchronization with NTP
server.

DNS Check DNS-Master & Check forward and reverse

DNS-Slave name resolution for DNS-Mas-
ter and DNS-Slave VMs.

Table 10: RHEL test cases

Issue: 1-0 DN09144478 30

Preventive Health Check Running Preventive Health Check

2.2.4 ORACLE
Execute on
Test case title node with Test case description
Role

Oracle RDA db Execute Oracle RDA Data

Collection for PERF, LOG
modules

Oracle Processes Uti- Check utilization of Oracle

lization Processes is below thresh-
old 95% default. This thresh-
old can be changed in: file: /
opt/oss/NSN-mhcf/prod-
uct/netact8/oracle/or-
acle_config.xml

<var id="proc_threshold"
name="Oracle Process
Threshold%" value="95" />

Top Oracle Processes Top 20 (default) Oracle

Processes exceeding mini-
mum 1000 (default) hundreds
of seconds are listed here.
The top count and minimum
seconds can be changed in:
file: /opt/oss/NSN-mhcf/
product/netact8/ora-
cle/oracle_config.xml

• <var id="topCount"
name="Top Oracle
Processes Count" val-
ue="20" />
• <var id="minTime"
name="Minimum TIME
in hundreds of second to
be listed in top command"
value="1000" />

Archive Folder checks INFO: Number of archive logs

present in system

CHECK-1: Checking for Inter-

val between archive files are

Issue: 1-0 DN09144478 31

Preventive Health Check Running Preventive Health Check

Execute on
Test case title node with Test case description
Role
above threshold 1200 (default)
seconds

CHECK-2: Calculating Archive

Filling Rate if number of
archives are > 4

CHECK-3: Calculating time to

fill archive partition (minimum
72hours)

The default time interval can

be changed in file: /opt/
oss/NSN-mhcf/product/
netact8/oracle/oracle_
config.xml

• <var id="minInterval"
name="Minimum In-
terval (seconds) be-
tween archive files" val-
ue="1200" />
• <var id="hoursToFull"
name="Minimum Hours
until archive partition is
full" value="72" />

Archive File Check Find any archive files older

than 1440 minutes (default)
indication that oracle online
backup was not executed or
not successful. The default
time interval can be changed
in file: /opt/oss/NSN-mhcf/
product/netact8/ora-
cle/oracle_config.xml

<var id="oldestFile"
name="Time Limit (minutes)
for archive file before backup"
value="1440" />

Redo Log Switch Count Determine Redo Log switchs

per hour past 24 hrs

Issue: 1-0 DN09144478 32

Preventive Health Check Running Preventive Health Check

Execute on
Test case title node with Test case description
Role

Redo Log Switch inter- Determine Shortest redo log

val switch interval in living memo-
ry

Table 11: ORACLE related test cases

Issue: 1-0 DN09144478 33

Preventive Health Check Scheduling Preventive Health Check

3 Scheduling Preventive Health Check

The PHC tool is scheduled to run automatically (cron added during installation for mhcf user) daily at
22:23 hrs, on the node with role DNS-Master.

The scheduled PHC reports are located in /var/opt/nokia/oss/global/NSN-mhcf/mhcf/

netact8/<datetime folder>/ directory.

Note:

Before starting NetAct upgrade, scheduling of Preventive Health Check must be disabled for
mhcf user on the node with DNS-Master role.

1. Login as root to the VM with DNS-Master role (use smanager.pl status service dns
command to get the DNS-Master VM node name)
2. Disable the execute permission for the cron job file:

chmod 640 /opt/oss/NSN-mhcf/product/netact8/cron/

mhcf_netact8_cron_daily.sh

After the NetAct upgrade is completed, the Preventive Health Check should be enabled again. Enable
the PHC cron job for mhcf user again on the node with DNS-Master role

1. Login as root to the VM with DNS-Master role (use smanager.pl status service dns
command to get the DNS-Master VM node name)
2. Enable the execute permission for the cron job file:

chmod 750 /opt/oss/NSN-mhcf/product/netact8/cron/

mhcf_netact8_cron_daily.sh

Issue: 1-0 DN09144478 34