Unit 7: Common Integrations

© 2010 IBM Corporation

 IBM Software Group Tivoli software

Objectives

After you complete this unit, you should be able to:

• Describe how common integrations are implemented in
Netcool/OMNIbus
• Describe the basic operation of a probe
• Describe how probes are configured and modified
• Identify the sources for pre-defined integrations
• Understand how to install and configure the Netcool Knowledge
Library (NCKL)
• Understand how to install the UNIX Syslog Probe and configure it
to use the NCKL integration
• Describe the new event rate detection capabilities new in v7.3

2
 IBM Software Group Tivoli software

Common Integrations
• The question most often asked
• Can Netcool/OMNIbus manage vendor XYZ device

• Most Common Integrations Handled via Probes

• Syslog Probe (UNIX / Linux)
• MTTrapd Probe
• Tivoli EIF Probe

• Integrations Implemented via Rules Files

• Shipped with probes
• Netcool Knowledge Library (NCKL)
• Open Process Automation Library (OPAL)

3
 IBM Software Group Tivoli software

Probe Operation
A Probe is lightweight software used to collect and pre-process event data.
START PROBE $OMNIHOME/probes/nco_p_<probename> -server SERVERNAME

Binary starts with <probename>.props properties; reads in

READ .props, .rules, etc.
<probename>.rules, additional files into memory

VALIDATE Verify sanity of rules file and Object Server fields

SOURCE

READ EVENT FROM SOURCE Connect to source (API, logfile, etc) and read an event

TOKENIZE Break event into its component fields ($fields, or tokens)

Interpret tokens via rules file. One pass through the

PROCESS VIA .rules rules for each event, with $tokens assigned to @fields

BUILD INSERT COMMAND The @fields are collected into an SQL Insert command

Which is forwarded to the Object Server

SEND TO OBJECT SERVER Buffers are flushed in anticipation of the next event

4
 IBM Software Group Tivoli software

Probe Basics
A Probe consists of a binary, a .rules and a .props file:
Binaries - retrieve and tokenise event streams($OMNIHOME/probes/nco_p_<probename>)
Properties - run-time settings of probe ($OMNIHOME/probes/<arch>/<probename>.props)
Rules - instructions for processing event ($OMNIHOME/probes/<arch>/<probename>.rules)

Probes may have additional files in $OMNIHOME/probes/<arch>/

If the probe is on a separate machine from the Object Server

• Install common components, process control
• Install the probe itself
• Identify the Object Server(s) in the interfaces file (with nco_xigen):
OBJ_SERV hostname 4100
• Edit the properties and rules files of the probe
• Run the probe:
$OMNIHOME/probes/nco_p_probename
 IBM Software Group Tivoli software

Probe Properties
Properties can be set:
- with command line switches at probe startup
( -help , -dumpprops lists switches)

- in the probe .props file (using VI or nco_xprops)

- or by letting them default by not specifying either
Property formats:
command-line switch: in the props file:
-server NCOMS Server : 'NCOMS'
secure-mode properties:
-authusername root AuthUserName: 'root'
-authpassword Lkfnjv0d AuthPassword: 'Lkfnjv0d'
useful debugging properties:
-messagelevel debug MessageLevel: 'debug'
-messagelog stdout MessageLog : 'stdout'
 IBM Software Group Tivoli software

Rules File
• Rules Files contain program steps, executed for each event, to
manipulate incoming data and assign it to alerts.status fields
• Found in $OMNIHOME/probes/<arch>/<probename>.rules
• Inbound tokens start with $ , Object Server Fields start with @
• Major function of rules file is to define Identifier field
• Field values of alerts.status may be set by the rules file
• Additional information can be added using the rules file
• Probe can have multiple associated rules files (include files)
• Netcool Knowledge Library (NCKL) uses this technique
• One master rules file
• Contains multiple “include” statements for individual vendor / technology rules
files
 IBM Software Group Tivoli software

Testing and Implementing New Rules

• nco_p_syntax check rules file syntax:
• Valid field names
• Command constructs
• Brace matching
$OMNIHOME/probes/nco_p_syntax -server NCOMS -
rulesfile $OMNIHOME/probes/<arch>/myprobe.rules
• To apply new rules to a running probe, restart probe
process with SIGHUP:

$kill –HUP <probe PID>

Note: nco_p_syntax is installed when the Probe Support feature is

selected during the OMNIbus Core installation
 IBM Software Group Tivoli software

A Rules File example

@Node=$Node
@Summary=$Summary
switch($Severity) {
case "Critical":
@Severity=5
case "Major":
@Severity=4
default:
@Severity=2
}
if (regmatch(@Summary,"interface.*down")) {
@AlertKey=extract(@Summary,"interface (.*) is")
}
@Identifier = @Node + @AlertKey + @Summary

Or, look at: $OMNIHOME/probes/<arch>/simnet.rules

9
 IBM Software Group Tivoli software

Tokens and Fields

• Probes split the event stream into tokens
• All Tokens are created as Strings

• Tokens in the rules file are denoted with a $ prefix

• for example, $Node is a token holding the Node name
• Tokens can be created on the fly: $MyElement=somevalue
• Fields in alerts.status are denoted with a @ prefix
• To populate fields, you assign values through:
• Direct assignment: @Node = $Node
• Concatenation: @Summary = $Summary + $Group
• Concatenation with literals:
@Summary = $Node + " has problem" + $Summary
 IBM Software Group Tivoli software

Testing and Logic in rules files

• Tests for string values:
• match(@Node,"router1") Exact match
• nmatch(@Node,"router") Begins with
• regmatch(@Node,"^router[0-9]") Full regex matching
if statement incorporates conditional logic:
if (<test1>)
{ <action> }
else if (<test2>)
{ <action> }
else
{ <action> }
Example: set Class field based on the value of $EquipType
if (match($EquipType,“Router")) {
@Class = 3303
}
else if (nmatch($EquipType,“Switch")) {
@Class = 3301
}
else if (regmatch($EquipType,"^[Hh]ub.*")){
@Class = 3302
}
 IBM Software Group Tivoli software

switch / case Statements

• switch processes statements that exactly match the value of an expression:
switch($EquipType)
{
case “framerelay”:
@Class = 3303
case “hub” | “switch” | “router”:
@Class = 3302
default:
@Class = 3300
}
Note: default: must always be present; be careful with punctuation and braces

Example: Nesting switch and if

switch ($EquipType) {
case “router":
@Class=3303
if (regmatch($Summary,".*offline.*"))
{
@Class=3304
}
default:
@Class = 3300
}
 IBM Software Group Tivoli software

Control Functions: discard, recover, update, details

• If an event is not required it can be discarded:
if (match($EquipType,“hub"))
{ discard }

Note: Be sure to use discard inside a test, or all events will be discarded.
• A discard can be negated with a recover:
if (match(@CustSLA,“Platinum"))
{ recover }

• Event-specific deduplication is enabled with the update function:

update(@Location)

• The details table can be inserted into with the details function:
if (regmatch(@Summary,"Link .* down"))
{details($Card, $Slot)}
The remove function excludes a token from further consideration:
if (match(@Node,"newdevice"))
{ remove($Slot)
details($*) }
Note: Using details($*) impacts performance – use during testing only.
 IBM Software Group Tivoli software

The extract and exists Function

Sometimes information required is embedded in a string:
$Summary = “Port is down on Port 1 Board 2”
Use the extract() function to extract the Port value of 1:

extract($Summary,"Port ([0-9]+)")

You can create temporary elements to build statements:

$temp1=extract($Summary,"Port ([0-9]+)")
$temp2=extract($Summary,"Board ([0-9]+)")
@AlertKey = $temp1 + "." + $temp2
Use the exists() function to test if a token was created:
(not all tokens are created for every event)
if (exists($port))
 IBM Software Group Tivoli software

Lookup Tables
 At the very top of the rules file, place a file reference or the table
itself:
File Reference: table name ="$OMNIHOME/probes/<arch>/file"
file should look like: key[tab]value
key[tab]value
Table in Rules: table name = {{"key","value"},{"key","value"}}
where key - item looked up, value - item returned.
 Then, to use the table:
@result = lookup(key, name)

where @result is assigned by searching for key in tablename

For example, a rules table would have this definition at the top:
table DeptTab = {{"batman","Technical"},
{"robin","Finance"} }
And the lookup would occur on executing this command:
@Department=lookup(@Node,DeptTab)
 IBM Software Group Tivoli software

Multi-Column Lookups and Defaults

Multi Column Lookup tables return more than one value per key:
In a Rules Table:
table xnodes = {{"mia1","Miami","Marketing"},
{"bos7","Boston","Finance"} }
Or, in a File Table:
mia1[TAB]Miami[TAB]Marketing
bos7[TAB]Boston[TAB]Finance
Field Assignment:
[@Location,@Department] = lookup(@Node,xnodes)

default sets values returned when key is not found

Add default values immediately after the table definition

In a Single Column lookup table:

table DeptTab = "/tmp/dept.lookup"
default = "UNKNOWN"
In a Multi-Column lookup table:
table DeptTab = "/tmp/dept.lookup"
default = {"UNKNOWN","UNKNOWN"}

16
 IBM Software Group Tivoli software

Properties and Writable Properties

The values of probe startup properties can be accessed in rules as:

%Manager, %MessageLevel, %MessageLog …

This is useful for host-specific processing, or debugging probe rules

• A Writable Property (persistent variable) is created if it does not exist

• Writable Properties retain their values across different events and
when probe re-reads rules but not when cold started.
Example uses: total events counter, discarded events counter

if (match(%TotalEvents),"")) { %TotalEvents=1 }
else
{ %TotalEvents=int(%TotalEvents)+1 }
@RunningTotal = %TotalEvents
 IBM Software Group Tivoli software

Associative Arrays
• Store and use data in dynamic arrays in rules
• Retain values when rules are re-read, but not when cold started
• Array elements are keyed by a string; Array must be declared

• Example: Calculate interval between similar events

# Declare array at top of rules file
array prevtime
# First time round initialize array element
if (match(prevtime[@Identifier], ""))
{ prevtime[@Identifier] = 0 }
# Calculate and store interval
@Interval = int(@LastOccurrence) –
int(prevtime[@Identifier])
# Update array element
prevtime[@Identifier] = @LastOccurrence
18
 IBM Software Group Tivoli software

Additional Functions

• String Functions
• Converts format of string functions

• Time Functions
• Converts UNIX time to time and vice versa
• Get current time

• Utility Functions
• Obtain platform specific information such as hostname

19
 IBM Software Group Tivoli software

Events to Alternate ObjectServers and Tables

All ObjectServers must be defined in .props or command :
-server NY1:LON1:TOK1

To select an ObjectServer for all subsequent events:

setdefaultobjectserver("NY1")

To select an ObjectServer for this event:

setobjectserver("LON1")

To send events to alternate tables and ObjectServers:

Usa = registertarget("NY1","",
"alerts.status","alerts.details")

Asia = registertarget("TOK1","LON1",
"alerts.stat_asia","alerts.det_asia")

setdefaulttarget("Usa")
....
if (match(@Location,"Tokyo")) {settarget("Asia")}

20
 IBM Software Group Tivoli software

Generic Clear Trigger

• Generic Clear Trigger requires configuration at the probe
• Uses these ObjectServer fields:
AlertGroup, AlertKey, Manager, Type
• Probe rules identify problem and resolution events by setting
Type = 1 for problems, and Type = 2 for resolutions:

if (regmatch(@Summary,"interface.*down"))
{ @AlertGroup="Interface"
@Type = 1 }

else if (regmatch($Summary, "interface.*up"))

{ @AlertGroup="Interface"
@Type = 2 }

21
 IBM Software Group Tivoli software

Student Exercise: 7-1

22
 IBM Software Group Tivoli software

Syslog Probe
• Install the probe with
$NCHOME/omnibus/install/nco_install_integration
• The syslog Probe can obtain its input from a log file or fifo
• Create either a file or a fifo
File: touch /var/adm/xyz
Fifo: mkfifo /var/adm/xyz
• Add the following line to /etc/syslog.conf
*.debug /var/adm/xyz
• Restart the syslog daemon
• Run the probe
$OMNIHOME/probes/nco_p_syslog -logfile /var/adm/xyz
$OMNIHOME/probes/nco_p_syslog -fifo /var/adm/xyz
 IBM Software Group Tivoli software

Syslog Next Generation

• SUSE Linux uses Syslog-NG
• Configuration for use with the Syslog Probe is slightly different
• To configure Syslog-NG to send output to a new fifo file
• Create fifo
• Fifo: mkfifo /var/adm/xyz
• Change ownership: chown netcool:ncoadmin /var/adm/xyz
• Add the following lines to /etc/syslog-ng/syslog-ng.conf
destination netcool { pipe(“/var/log/nco” owner(netcool)
group(ncoadmin)); };
log { source(src); filter(f_messages); destination(netcool); };
• Restart the syslog-ng daemon

24
 IBM Software Group Tivoli software

MTTrapd Probe
• Install the probe via
$NCHOME/omnibus/install/nco_install_integration
• The most common method of integration
• Supports SNMP v1, v2 and v3
• Configured in property settings
• Supports UDP or TCP communication
• Configured in property settings
• Probe must run as the root user if using default port – 162
• UNIX operating system requirement (port < 1024)
• Possible to configure for non-root user
 IBM Software Group Tivoli software

Tivoli EIF Probe

• Install the probe via
$NCHOME/omnibus/install/nco_install_integration
• Event Integration Facility
• Used for heritage Tivoli product integration
• Supports
• IBM Tivoli Monitor (ITM)
• IBM Tivoli Composite Application Manager (ITCAM)
• Tivoli Enterprise Console (TEC)
• Tivoli Event Pump
• Requires ObjectServer modifications
• New fields (8)
• Modification to deduplication trigger
• This is a java-based probe
• nco_p_nonnative runs
• Calls java and passes nco_p_tivoli_eif.jar file
 IBM Software Group Tivoli software

Netcool / OMNIbus Knowledge Library - NCKL

• Distributed with OMNIbus as separate download
• Uses the include technique to incorporate multiple individual files
• One master rules file contains multiple include statements
• Predefined rules files for multiple vendors and technologies
• Packaged as individual files (over 2000 in current version)
• Easy to configure to add or remove
• Rules provided for MTTrapd and Syslog probes
• Installation
• Unpack rules files into target directory - $NC_RULES_HOME
• Run supplied sql file to add ObjectServer customizations
• Configure probe to use NCKL “master” rules file
$NC_RULES_HOME/syslog.rules – Syslog Probe
$NC_RULES_HOME/snmptrap.rules – MTTrapd Probe

27
 IBM Software Group Tivoli software

Student Exercise: 7-2

28
 IBM Software Group Tivoli software

New Probe Features

• Event flood detection
• Detects high event rate based on a preset threshold.
• User can divert/discard lower priority events during flood.

• Anomalous event rate detection

• Detects an unusually low or unusually high rate of receipt of events.
• Anomaly detection based on the ‘normal’ rate of receipt of events.
• Anomalous event rate detection reported in a new ObjectServer event.

29
 IBM Software Group Tivoli software

Installation

• Event flood detection is defined in two rules file fragments

• flood.config.rules
• flood.rules
• After installing OMNIbus 7.3
• Rules file fragments are located in
$OMNIHOME/extensions/eventflood

30
 IBM Software Group Tivoli software

Configuration
• Configuration is done in flood.config.rules
• Configure the time windows for collecting events for flood and
anomalous event rate detection
• Configure the length of the ‘training period’ during which the ‘normal’ rate
of receipt of events is determined
• Configure the threshold (events/sec) for determining when an event
flood is in progress
• Configure what percentage of the ‘normal’ event rate constitutes an
unusually low or unusually high rate of receipt of events.
• Configure remedial behavior to take during the event flood.
• For example, discard all events with severity less than major

Note: flood.rules should not need to be altered in most cases

31
 IBM Software Group Tivoli software

Usage

• flood.config.rules
• Included at the beginning of the user’s set of rules
• Necessary because the rules file fragment defines an array
• flood.rules
• Included towards the end of the user’s set of rules
• Necessary because the remedial action to take during the event
flood may be based on the event severity
• Rules file needs to be processed first to determine this severity
• The probe is run as normal

32
 IBM Software Group Tivoli software

Limitations
• There may be work involved in adjusting the configuration
variables for the user’s environment
• How long the probe should be running before flood detection is
enabled to cope with an initial burst of alarms when the probe is
started
• How long the training period should be to determine the normal
rate of receipt of events
• Once the training period has completed and the normal rate of
receipt of events has been determined
• This normal rate is only valid while the probe is running
• Not persisted between subsequent restarts of the probe

33
 IBM Software Group Tivoli software

Example Usage – Syslog Probe

• Edit syslog.rules
• Add to the beginning of the file
include “flood.config.rules”
• Add to the end of the file
include “flood.rules”
• Edit flood.config.rules
• Adjust the configuration variables to suitable values:
$average_event_rate_time_window = 60
(This is the anomaly detection training period in seconds)
$flood_detection_event_rate_flood_threshold =
50
• Run the Syslog Probe as usual

34
 IBM Software Group Tivoli software

Example Usage (Continued)

• Flood Condition
• Probe receives more than 50 events per second
• ObjectServer event indicating that a flood has been detected
• Flood Subsides
• ObjectServer event indicating that an event flood has finished
• Describes number of events received during flood condition and
the duration
• Low Event Condition
• Probe receives less than 10% of the expected rate of events
• ObjectServer event indicating low event rate
• High Event Condition
• Probe receives more than five times the expected rate of events
• ObjectServer event indicating high event rate

35
 IBM Software Group Tivoli software

Summary

You should now be able to:

• Describe how common integrations are implemented in
Netcool/OMNIbus
• Describe the basic operation of a probe
• Describe how probes are configured and modified
• Identify the sources for pre-defined integrations
• Understand how to install and configure the Netcool Knowledge
Library (NCKL)
• Understand how to install the UNIX Syslog Probe and configure it
to use the NCKL integration
• Describe the new event rate detection capabilities new in v7.3

36