Q radar architecture deep dive

How QRadar SIEM Collects
Security Data
Luis Latas
WW Technical Sales Enablement

From an Appliance Perspective
Event Collector Capabilities
Event Collector
(ecs-ec-ingress)
Event Collector
(ecs-ec)

Event/Flow Processor Capabilities
Event Collector
(ecs-ec-ingress)
+
Event Collector
(ecs-ec)
Event Processor
(ecs-ep)

AIO/Console Capabilities
Event Collector
(ecs-ec-ingress)
+
Event Collector
(ecs-ec)
Event Processor
(ecs-ep)
Magistrate

High-level component architecture and data stores
Flow and event data is stored in the Ariel
database on the event processors
– If accumulation is required, accumulated data is stored
in Ariel accumulation data tables
– As soon as data is stored, it cannot be changed (tamper
proof)
– Data can be selectively indexed
Offenses, assets, and identity information are
stored in the master PostgreSQL database on the
Console
– Provides one master database with copies on each
processor for backup and automatic restore
Secure SSH communication between appliances in
a distributed environment is supported
Console services
User interface
Magistrate
Reporting
Event processor
Flow collector Event collector
Identities
Assets
Offenses
Configuration
Flows
Events
Accumulations
Network packet
interface, sFlow,
and 3rd party
Events
from log
sources
6

QRadar Data Flow - Overall
IBM Security / © 2020 IBM Corporation 7

Collecting and Normalizing raw events
An event is a record from a device that describes an action on a network or host.
QRadar SIEM normalizes the varied information found in raw events.
– Normalizing means to map information to common field names, for example:
• SRC_IP, Source, IP, and others are normalized to Source IP.
• user_name, username, login, and others are normalized to Username.
– Normalized events are mapped to high-level and low-level categories to facilitate further
processing.
– After raw events are normalized, it is easy to search, report, and cross-correlate these
normalized events.

9
Event data pipeline
IBM Security / © 2020 IBM Corporation
Event data is sent to or pulled by QRadar
Event Collector Ingress – Responsible for
collecting data at all times (zero event loss)
Data is collected and buffered during patch and
deploys and processed once the operation is
complete
Protocols – Reads or pulls raw data from network
devices (e.g: Windows Servers, Firewalls, etc)
Throttle Filter - Licensing - On a second-by-
second basis, slows down the incoming rate so
it does not exceed the license on the appliance.
Events are sent to ecs-ec-parse to be parsed
Event
Data
Protocols
Throttle
Filter
Licensing
Event
Collector –
Ingress (ecs-
ec-ingress)
Event
Collector
(ecs-ec)

10
Event data pipeline
Parsing
(DSM, LSX,
CEP)
Coalescing
Forwarding
Log
Only/Data
Sore
Event
Collector
(ecs-ec)
Event
Processor
Event data is received from the ecs-ec-ingress
Parsing – DSMs / LSX / CEP – take the raw data
and normalize it into a common structure.
Coalescing - “Event Compression”. Find nearly
identical events and delete one and increase the
event count on the record. Key is: source IP, dest
IP, dest port, QID, username
Forwarding - Applies routing rules for the system,
such as sending event data to offsite targets,
external Syslog systems, JSON systems, and other
SIEMs.
Log Only/Data Store supports the storage of an
unlimited number of logs without counting against
the EPS License
Event
Collector –
Ingress (ecs-
ec-ingress)
Events are then sent to the Event Processor
component and pass through the Custom Rules
Engine (CRE).

Events not counted against the EPS licenses
- The list of log source types that do not incur EPS hits are as follows:
- System Notification
- Custom Rule Engine (CRE)
- SIM Audit
- Anomaly Detection Engine
- Asset Profiler
- Search Results from scheduled searches
- Health Metrics
- Sense DSM
- Risk Manager questions, Simulations and internal logging
- Log Only/Data Store
- Supports the storage of an unlimited number of logs without counting against the EPS QRadar
SIEM license
- Enables an organization to build custom apps and reports based on this stored data to gain
deeper insights into IT environments.

Event Coalescing
- Event Coalescing is a method of reducing the data going through the pipeline.
- As data arrives in the pipeline QRadar will attempt to group like events together into a
single event.
- Coalescing occurs after licensing and parsing
- Coalescing is indexed by Log Source, QID, Source IP, Destination IP, Destination Port
and Username.
- If more than 4 events arrive within a 10 second window with these properties being
identical any additional events beyond the 4th will be collapsed together.
- Coalesced events can be identified by looking at the Event Count column in the log
viewer, if the Event Count is >1 the event has been coalesced.
- Coalescing can be turned on or off per log source or by changing the default setting in
the system setting page.

13
© Copyright IBM Corporation 2017

Flow collection and processing
A flow is a communication session between two hosts
QFlow Collectors read packets from the wire or receive flows from other devices
QFlow Collectors convert all gathered network data to flow records similar normalized
events; they include such details as:
– when, who, how much, protocols, and options.

Flow pipeline
Flows
Qflow
Asymmetric
Recombination
De-Duplication
Flow Governor
(Licensing)
CFP Parsing
Flow
Forwarding
Event
Collector
Event Processor
The QFlow component collects and creates flow information from
internal and external flow sources
Event Collector – Responsible for parsing and normalizing incoming
flows
Asymmetric recombination - Responsible for combining two sides of
each flow when data is provided asymmetrically
Deduplication - Flow deduplication is a process that
removes duplicate flows when multiple Flow
Collectors provide data to Flow Processors appliances.
Flow Governor - Monitors the number of incoming flows
to the system to manage input queues and licensing.
Custom flow properties – extracts any properties defined
in the Custom Flow Properties
Forwarding - Applies routing rules for the system, such as
sending flow data to offsite targets, external Syslog
systems, JSON systems, and other SIEMs.
Flows are then sent to the Event Processor component
and pass through the Custom Rules Engine (CRE). They
are tested and correlated against the rules that are
configured

16

Event & Flow Correlation and Processing
Event
Collector
Licensing
CRE
Storage and
Indexing
Host Profiling
Real Time
streaming
Event
Processor
Magistrate
Asset Profiler
Ariel
Licensing is applied again on ingress to the EP
After Events and Flows are normalized they are then
sent to the Event Processor for processing
The CRE or Custom Rules Engine Applies the
correlation rules that were created in the UI.
Flow data is then sent to the Ariel Database for
storage.
Host Profiling – Also called passive profiling or
passive scanning. Watches flows on the network in
order to make educated guesses about which
IPs/assets exist and what ports are open.
Streaming – Responsible for the “real time
(streaming)” view in User Interface
If an event matches a rule, the Magistrate component
generates the response that is configure in the custom
rule

Magistrate
• The Magistrate creates and stores offenses in the PostgreSQL database; these
offenses are then brought to the analyst’s attention in the interface
• The Magistrate instructs the Ariel Proxy Server to gather information about all
events and flows that triggered the creation of an offense
• The Vulnerability Information Server (VIS) creates new assets or adds open
ports to existing assets based on information from the EPs
• The Anomaly Detection Engine (ADE) searches the Accumulator databases for
anomalies, which are then used for offense evaluation

Ariel Components
Ariel
Ariel
Event
Processor
Accumulator
Historical
Correlation
Ariel Proxy Server Ariel Query Server
Offline Forwarder
Report Runner

Ariel Components
Ariel Proxy
Server
Console
Ariel query
Server
Managed Host
Ariel query
Server
Managed Host
Ariel Components
Ariel
Ariel Ariel

Asset and Vulnerability Flow
ecs-ep
Event Processor
Scanners/QVM/3rd
Party
ecs-ec
(event collector
Asset Profiler
vis
(Vulnerability
integration service)
Identity Data
Passive
Profiling
POSTGRES

Gathering asset information
Active scanners
QRadar Vulnerability Manager scanner,
Nessus, Nmap, Qualys, and others
Provide:
• List of hosts with risks and potential
vulnerabilities
• IP and MAC addresses
• Open ports
• Services and versions
• Operating system
Pros
• Detailed host information
• Policy and compliance information
Cons
• Out of date quickly
• Full network scans can take weeks
• Active scanners cannot scan past firewalls
• User can hide from active scans
Passive detection
Flows from QFlow, or other flow
sources in accounting technologies
such as IPFIX/NetFlow, sFlow, and
others
Provide:
• IP addresses in use
• Open ports in use
Pros
• Real-time asset profile updates
• Firewalls have no impact
• End system cannot hide
• Policy and compliance information
Cons
• Not as detailed as active scans
• Does not detect installed but
unused services or ports

Hostcontext
Reporting Executor
Report Runner
Tomcat
“Owns” the host it is responsible for starting and stopping
processes and for overall system health and backups.
A stopwatch responsible for keeping track of reports and
when they should run and then instantiating the report
runner
The process that actually generates the reports, querying
postgres, Ariel, etc.
Process that drives our web UI and serves up web pages.
Historical Correlation
Processor
Process that is responsible for historical correlation. Runs a
specified search, runs the results through CRE rules (based
on QRadar time or device time) and generates offenses
The Remainder

© Copyright IBM Corporation 2020. All rights reserved. The information contained in these materials is provided for
informational purposes only, and is provided AS IS without warranty, of any kind, express or implied. Any statement of
direction represents IBM’s current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM,
the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in
the United States, other countries or both. Other company, product, or service names may be trademarks or service marks
of others.
Statement of Good Security Practices: IT system security involves protecting systems and information through prevention,
detection and response to improper access from within and outside your enterprise. Improper access can result in
information being altered, destroyed, misappropriated or misused or can result in damage to or misuse of your systems,
including for use in attacks on others. No IT system or product should be considered completely secure and no single
product, service or security measure can be completely effective in preventing improper use or access. IBM systems,
products and services are designed to be part of a lawful, comprehensive security approach, which will necessarily involve
additional operational procedures, and may require other systems, products or services to be most effective. IBM does not
warrant that any systems, products or services are immune from, or will make your enterprise immune from, the malicious
or illegal conduct of any party.
Thank you
Follow us on:
ibm.com/security
securityintelligence.com
ibm.com/security/community
xforce.ibmcloud.com
@ibmsecurity
youtube.com/ibmsecurity

Q radar architecture deep dive

Q radar architecture deep dive

Recommended

More Related Content

What's hot (20)

Similar to Q radar architecture deep dive (20)

Recently uploaded (20)

Q radar architecture deep dive