Remote Log Analytics Using DDS, ELK, and RxJS

Your systems. Working as one.
Log Analytics Using DDS and RxJS
Sumant Tambe
Jan. 2015

ProDigy4IT
Autonomous Probing and Diagnostics for IT
• Flexible Data Acquisition, Smart Analysis, and Adaptive Data
Transmission
• Intuitive Dashboards for Status and Remote Diagnostics
• Autonomous, Customizable, Programmable, and Highly
Extensible
• Agent-based, distributed, server-less, stream-oriented, and
dynamic
• COTS, standards-based open architecture, Secure
8/25/2015 2

Data
Acquisition
Streaming
Analysis
Prioritized
Intelligent
Comms
Expert
Local
Tools
Stream
Probes
Snapshot
Agents
IT
Systems
ProDigy4IT
• Extensible Architecture Leverages existing COTS / Open Source Tools (Data Acquisition / local visualization)
• No need to build proprietary one-off IT tools
• Data Acquisition Tools – ELK, SolarWinds, SMARTS, etc.
• Local Visualization/Analysis Tools – Kibana, Netuitive, Nlyte, etc.
• Our focus is on the challenges – solving the problem of getting the right data to local
• Customizable Dynamic Stream Probes – Monitors the IT streams for expert-defined important events
• Customizable Snapshot Agents – Grabs the right data before and after the detected error occurs & sends it
to local (a snapshot for a “hot cpu” error may include process logs, motherboard heat logs, and power supply
logs for a given machine to be sent)
• Intelligent Compression – Bandwidth-sensitive compression over disadvantaged links, sends only high priority
attributes to local when bandwidth constrained
• Built Upon our RTI Connext (TRL9) Middleware
Rapidly Getting The RIGHT Data
RTI Connext - OMG Data Distribution Service
A Highly Extensible, Open Architecture For Remote IT Management
Data Analytics Challenges

ProDigy4IT Architecture
8/25/2015 4
RTI DDS Data Bus
(standard-based protocols DDS/RTPS)
Secondary
Storage and
Visualization
Remote
RTI DDS Low-
Bandwidth
Plugins
Local
Intelligent
Priority-aware
Bandwidth-
aware Data
Transmission
Agent (T1)
RTI
DDS
Data
Bus
Primary Data
Archive
T1
DB’
S1
UI
REST
HTTP
JSON
REST
HTTP
JSON
ProDigy4IT Agents
Logs,
SNMP
N1
A1
Cn An
datacenters
C1
Scriptable probes
• Watch for ‘important’ events
Snapshot Agents
• Queue up the ‘right’
information
Data Collection
Data
Normalization
SNMP
Logs
Streaming Analysis

Key RTI Technologies Used
Building Upon A Mature Foundation of Commercial Software
• RTI Connext Data Distribution Service (DDS)
– Market-leading implementation of the OMG DDS standard
– TRL9 Middleware. Widely deployed.
– Remote distributed agents, low-bandwidth transmission
• RTI DDS Node.js Connector (Javascript)
– Web-friendly, dynamic, scripting language interface to DDS
• RTI Routing Service
– Bridge two or more DDS domains
8/25/2015 5

Key Open-Source Technologies Used
• Elasticsearch
– Document-oriented, scalable, NoSQL database
– Persist data remote and local
• Logstash
– Data collection (log, snmp, etc.)
– Configured for Win/Linux logs and snmp
• Kibana
– Browser-hosted visualization for time-stamped data
– Configured for dashboards
• Reactive Extensions (Rx)
– A library for composing asynchronous data streams
– Used to implement the dynamic stream probing (analysis) agent
8/25/2015 6© RTI

Key Innovations
Detecting Problems & Getting the Right Information to the Experts
• Dynamic Stream Probing (DSP) Agents
– A light-weight, distributed, embedded stream processing engine
– Executes probes
– Dynamic, extensible
– Combines DDS and Rx; Written in Javascript (Node.js)
• Key Concepts
– Data Stream  Potentially infinite data samples from a data source
• E.g., a temperature sensor, application event log from a Windows host
– Probe  A named declarative “stream processor” that analyzes one or more data
streams
• Written using JSON syntax; Probes have data and alert dependencies
– Alert  A notification produced by a probe indicating that data samples in a stream
satisfy a complex precondition
• E.g., average temperature over last 30 seconds >= 65oC
– Snapshot A collection of samples in a data stream that fall within a window of time
before and after the alert
• Informally, a thin slice through time and space (when agents are distributed)
8/25/2015 7© RTI

8/25/2015 8
IT Systems
+/- 1 min window +/- 1 min window
Alert Stream
Source Data
Streams
3
2
1
Dynamic
Stream
Probes
time
Snapshot Stream
+/- 1 min window +/- 1 min window
Experts
 snapshots 
© RTI
Acquiring The Right Information To Help Resolve Problems
Probe Agents decide what & how much to send

Key Innovations Developed
• The Probe Language for Authoring Probes
– A Domain-specific language for stream processing and coordination
– Embodies “Save Query; Run Data” paradigm; flow-based programming
– Declarative English-like syntax; Little/No programming experience necessary
– Compose operators using pipes similar to the Unix command line
– Scripted
• No compilation
– Dynamic
• A probe may handle 100s of data streams dynamically
• Probes can be added at run-time to existing running agent processes
• Probes may depend on existing running probes
– Extensible through a library of operators
• Arithmetic, window, conditionals, I/O, filtering, grouping, throttling, staleness detection ,etc.
• Operators may coordinate one or more streams
– Efficient
• Compiles just-in-time to native code (V8 Javascript engine)
8/25/2015 9© RTI

Key Innovations Developed
• Operator Library for Simplifying Probe Authoring
– Extends the probe language
– 30+ operators developed during Phase I
8/25/2015 10© RTI
• Conditionals
• >, <, >=, <=, ==, !=
• Time-based Windows
• insert avg(celsius) over 30 sec as avg_celsius
• Counter-based Windows
• insert avg(celsius) over last 3000 as avg_celsius
• Data manipulation
• insert … as
• avg(), min(), max(), eval(), JSONPath()
• delete <member>
• I/O
• source <DDS topic>
• source <pre-existing probe>
• mergesources <1> <2> …
• output dds <topic>
• output console
• Filtering
• has <member>
• hastype <something>
• match <a partial object>
• contains <field> <data>
• Data Partitioning (very important)
• groupby <key-field>
• Allows downstream processing on each partition
independently.
• Also natively supported by DDS
• Data Stream Capture
• snapshot timerange(past-time, future-time)
• Captures alert dependencies between probes
• Alert Throttling
• interval 120 sec
• Avoid repeated firing of an alert in quick
succession
• Staleness Detection
• liveliness 30 sec
Operators coordinating multiple streams in Red

A Completely Customizable Dashboard
8/25/2015 11
1 = Queries; 2 = Time range selector; 3 = Alerts; 4= Data Visualization
1
2
3
4
© RTI

Demo Data Sources
• SNMP Numerical Data
– CPU temperature readings from the climate monitor
• “Core 0”, “Core 1”, “Physical Id 0” (motherboard), and
“temp1” (GPU)
• Textual Data
– Log files from Linux VM
– Eventlog from the Windows Host
• SNMP Networking Data
– IP addresses of connected hosts
– Address Resolution Protocol (ARP) table entries
8/25/2015 12
Router
Climate
Monitor
© RTI

Demo Data Types
8/25/2015 13
temperature_readings SNMP Router Alert
{
"message” : string,
"@version” : integer,
"@timestamp” : date,
"type” : string,
"host” : string,
“celsius” : double,
“sensor_id” : string,
"seq_num” : integer
}
{
"message” : string,
"@version” : integer,
"@timestamp” : date,
"type” : string,
"host_ip” : string,
"seq_num” : integer
}
{
“type” : string,
“@timestamp” : date,
“causedByProbe”: string,
“causedByData” :{ ... },
“seq_num” : integer
“tags” : array
}
Optionally,
“causedBySensor” : string
“hostStatus” : string
© RTI

Demo 1: Temperature Diagnostics
• Scenario – Climate monitor reports rising temperatures
• How – Stress CPU to 100% utilization
• Trigger – CPU avg. temperature >= 65oC over 30 seconds
8/25/2015 14© RTI

{
"name" : "TemperatureDataSource",
"filter" : "source AggregateFlow |
hastype temperature_readings |
has sensor_id |
has celsius_x_1000 |
match { "host" : “remote" } |
insert eval(celsius_x_1000 / 1000) as celsius“,
“trigger” : false
},
{
"name" : "TempAvgProbe",
"filter" : "source TemperatureDataSource |
groupby sensor_id |
insert avg(celsius) over 30 sec as extra.degree_avg |
snapshot timerange(-55 sec,+55 sec) |
greater_than_equal extra.degree_avg 65 |
interval 120 sec",
"trigger" : true,
"tags" : [ "cpu_overheat", "cpu_heat_warning" ]
}
8/25/2015 15© RTI
Developing Custom Probes
Example – Temperature Probe(s)

{
"name" : "TemperatureDataSource",
hastype temperature_readings |
has sensor_id |
has celsius_x_1000 |
match { "host" : “remote" } |
insert eval(celsius_x_1000 / 1000) as celsius“,
“trigger” : false
},
{
"name" : "TempAvgProbe",
"filter" : "source TemperatureDataSource |
groupby sensor_id |
insert avg(celsius) over 30 sec as extra.degree_avg |
greater_than_equal extra.degree_avg 65 |
interval 120 sec",
"trigger" : true,
"tags" : [ "cpu_overheat", "cpu_heat_warning" ]
}
8/25/2015 16
Refer DDS Topic
Type
Precondition
Precondition
Preprocessing
Precondition
Refer probe
Data Partitions
Analysis
SnapshotBuffer
Final alert test
Throttle
Queryable reason
No alert here
Produce Alerts
© RTI
Example – Temperature Probe(s)

Example Temperature Dashboard
8/25/2015 17© RTI

ProDigy4IT Highlights in Demo 1
• Numerical Data Monitoring using ProDigy4IT
• Collects, analyzes numerical SNMP data
• Readable, composable, dynamic probes for event detection
• Asset-specific analysis (e.g., “core 0”, “core 1”, etc.)
– Discover and analyze assets at run-time
• Intuitive Dashboard
– Alert timeline, timestamps, trending up/down, etc.
– Alert counters
– Alert classification (warning, critical)
– Identify assets in need of the most urgent attention
– Trend comparison among assets in the same class
– Trends of numerical data before and after the alert
– Likely cause displayed in “Logs” table
8/25/2015 18© RTI

Demo 2: Logs Diagnostics
• Scenario – Analyze textual log data for alerts
• How – Remote login to Windows/Linux host
• Trigger
– On Linux log contains “ … session opened …”
– On Windows log contains “ … new logon … ”
8/25/2015 19© RTI

Example – Log Analysis Probe(s)
{
"name" : "Win32EventLogSource",
hastype win32_eventlogs |
snapshot timerange(-120 sec,+60 sec)",
"trigger" : false
}, {
"name" : "Win32LoginEventProbe",
"filter" : "source Win32EventLogSource |
contains Logfile 'Security' |
contains Message 'new logon'",
"trigger" : true,
"tags" : [ "login_attempt", "win32_login" ]
}, {
"name" : "LinuxSecurityLogsProbe",
hastype linux_securitylogs |
contains message 'session opened'",
"trigger" : true,
"tags" : [ "login_attempt", "linux_login" ]
}
8/25/2015 20© RTI

Example – Log Analysis Probe(s)
{
"name" : "Win32EventLogSource",
hastype win32_eventlogs |
snapshot timerange(-120 sec,+60 sec)",
"trigger" : false
}, {
"name" : "Win32LoginEventProbe",
"filter" : "source Win32EventLogSource |
contains Logfile 'Security' |
contains Message 'new logon'",
"trigger" : true,
"tags" : [ "login_attempt", "win32_login" ]
}, {
"name" : "LinuxSecurityLogsProbe",
hastype linux_securitylogs |
contains message 'session opened'",
"trigger" : true,
"tags" : [ "login_attempt", "linux_login" ]
}
8/25/2015 21
Refer DDS Topic
Type
Buffer Eventlogs
No alerts here
Refer Probe
Precondition
Produce Alert
Queryable reason
Refer DDS Topic
Precondition
Queryable reason
Precondition
Precondition
Buffer logs
© RTI

• Textual Data collection and processing using ProDigy4IT
• Collects, analyzes textual log data
• Readable, composable, dynamic probes for log diagnostics
• Log stream specific analysis (e.g., “event logs”, “security logs”, etc.)
• Capture Implicit/Explicit Alert Dependencies
– An alert causes snapshot of all other streams to aid diagnosis (e.g., login
alert causes snapshot of temperature readings)
8/25/2015 22© RTI

Demo 3: Availability Diagnostics
• Scenario – Analyze SNMP data from Router
• How – Auto-detect
– Poll Router SNMP end-point for Cisco-specific object-ids
• Trigger
– Auto-detect connected hosts
– Detect loss of connectivity
8/25/2015 23© RTI

Example – Availability Analysis Probe
{
"name" : "AvailabilityProbe",
hastype host_availability_readings |
groupby host_ip |
liveliness 30 sec",
"trigger" : true,
"tags" : [ "host_availability_change" ]
},
{
"name" : "AlertsStream",
"filter" : "source alerts |
hastype Alert |
insert jsonpath($..host_ip) as availabilityChangedHost |
insert jsonpath($..stale) as hostStatus |
output console |
delete causedByData |
output dds"
}
8/25/2015 24© RTI

Example – Availability Analysis Probe
{
"name" : "AvailabilityProbe",
hastype host_availability_readings |
groupby host_ip |
liveliness 30 sec",
"trigger" : true,
"tags" : [ "host_availability_change" ]
},
{
"name" : "AlertsStream",
"filter" : "source alerts |
hastype Alert |
insert jsonpath($..host_ip) as availabilityChangedHost |
insert jsonpath($..stale) as hostStatus |
output console |
delete causedByData |
output dds"
}
8/25/2015 25
Refer DDS Topic
Type
Data partition
Preconditio
Queryable reason
Refer Alert Stream
Type
Extract-Transform
Extract-Transform
output
postprocessing
publish
© RTI

• SNMP Data collection and processing using ProDigy4IT
– Collects, analyzes SNMP from networking equipment
• Readable, composable, dynamic probes for SNMP diagnostics
• Declarative liveliness monitoring
– Per host
– Edge triggered – No need to send periodic pings to local
• Alerts post-processing
– Alerts are just data streams
– Apply transformations to the alerts before transmitting
8/25/2015 26© RTI

References
• StreamCoCo: A DSL for Processing Data-Centric
Streams for Industrial IoT Edge Applications
[link]
8/25/2015 © RTI 27

Remote Log Analytics Using DDS, ELK, and RxJS

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Remote Log Analytics Using DDS, ELK, and RxJS (20)

More from Sumant Tambe (13)

Recently uploaded (20)

Remote Log Analytics Using DDS, ELK, and RxJS