Apache Eagle: Architecture Evolvement and New Features
Download as PPTX, PDF1 like540 views
Apache Eagle is a distributed real-time monitoring framework for Hadoop clusters. It analyzes data activities, applications, metrics, and logs to identify security breaches, performance issues, and provide insights. The architecture uses streaming ingestion of data, processing, alerting through complex event processing and machine learning, and storage and dashboards for insights. New features include support for additional data sources, applications for specific use cases, and evolving the architecture for improved scalability and flexibility.
![Alert Engine
Apache Eagle - Distributed Alert Engine
from MetricStream[(name ==
'ReplLag') and (value > 1000)]
select * insert into
outputStream;
Messaging
Notification
Slack
Insight
Action
● Real-time Streaming: Apache Storm
(Execution Engine) + Kafka (Messaging)
● Declarative Policy: CEP and Extensible
Alert Model in streaming way
● Dynamical Onboarding & Correlation:
Connect to new stream and change
Stream Grouping in Runtime
● Hot Deploy & No Downtime: Metadata-
driven and lightweight alert logic
assignment](https://image.slidesharecdn.com/apacheeagle-gops-201716x9v2-170506080701/85/Apache-Eagle-Architecture-Evolvement-and-New-Features-21-320.jpg)
![Apache Eagle – Policy Examples
from hadoopJmxMetricEventStream
[metric == "hadoop.namenode.fsnamesystemstate.capacityused" and value > 0.9] select
metric, host, value, timestamp, component, site insert into alertStream;
Example 1: Alert if hadoop namenode capacity usage exceed 90 percentages
from every
a = hadoopJmxMetricEventStream[metric=="hadoop.namenode.fsnamesystem.hastate"]
->
b = hadoopJmxMetricEventStream[metric==a.metric and b.host == a.host and a.value !=
value)]
within 10 min
select a.host, a.value as oldHaState, b.value as newHaState, b.timestamp as timestamp,
b.metric as metric, b.component as component, b.site as site insert into alertStream;
Example 2: Alert if hadoop namenode HA switches](https://image.slidesharecdn.com/apacheeagle-gops-201716x9v2-170506080701/85/Apache-Eagle-Architecture-Evolvement-and-New-Features-22-320.jpg)
![define stream SystemMetricStream (
metric string,
host string,
device string
value double);
from SystemMetricStream
[name = "disk.usage.metric" and value > 0.99 ]
#window.time(30 min)
group by host, device
insert into SystemAlertStream;
Apache Eagle - Distributed Alert Engine
SourceSpec
PartitionSpec
SortSpec
PolicySpec
PublishSpec
From Policy Definition in User View to Engine View
Schedule
Assignment
Source: kafka topic
Schema:
SystemMetricStream
Window: 5min
Margin: 1min
Partition:
Group by host, device
Publish:
SystemAlertStream
Process:
CEP Execution Plan](https://image.slidesharecdn.com/apacheeagle-gops-201716x9v2-170506080701/85/Apache-Eagle-Architecture-Evolvement-and-New-Features-26-320.jpg)
![Apache Eagle - TSDB Storage Engine
• Light-weight ORM Framework for HBase/RDMBS
• Full-function SQL-Like REST Query
• Optimized Rowkey design for time-series data
• Native HBase Coprocessor
• Secondary Index Support
@Table("alertdef")
@ColumnFamily("f")
@Prefix("alertdef")
@Service(AlertConstants.ALERT_DEFINITION_SERVICE_ENDPOINT_NAME)
@JsonIgnoreProperties(ignoreUnknown = true)
@TimeSeries(false)
@Tags({"site", "dataSource", "alertExecutorId", "policyId", "policyType"})
@Indexes({
@Index(name="Index_1_alertExecutorId", columns = { "alertExecutorID" },
unique = true),
})
public class AlertDefinitionAPIEntity extends TaggedLogAPIEntity{
@Column("a")
private String desc;
@Column("b")
private String policyDef;
@Column("c")
private String dedupeDef;
Query=AlertDefinitionService[@dataSource="hiveQueryLog"]{@policyDef}](https://image.slidesharecdn.com/apacheeagle-gops-201716x9v2-170506080701/85/Apache-Eagle-Architecture-Evolvement-and-New-Features-28-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Apache Eagle: Architecture Evolvement and New Features (20)
Ad
Recently uploaded (20)
Ad
Apache Eagle: Architecture Evolvement and New Features
- 2. Apache Eagle Architecture Evolvement and New Features Hao Chen, Lead PMC and Committer of Apache Eagle http://people.apache.org/~hao
- 3. 个人简介 Hao Chen / 陈浩 Apache Eagle 联合发起人（PMC 以及Committer） eBay基础架构部 资深工程师 (Staff Engineer, Member of Technical Staff) QCon, Hadoop Summit (中国/北美/日本), GOPS 等国内外会议 讲师
- 4. Agenda Introduction New Features and Use Cases Architecture Evolvement What’s Next Q&A
- 5. Apache Eagle - Introduction Apache® Eagle™ analyzes data activities, yarn applications, JMX metrics, and daemon logs etc., provides state-of-the-art alert engine to identify security breach, performance issues and shows insights. Oct, 2015 Apache Incubation Apr, 2016 Apache Eagle v0.3 Release Jul, 2016 Apache Eagle v0.4 Release Jan, 2017 Apache Top Level Project Apache Eagle v0.5 Release March, 2017
- 6. Apache Eagle - Introduction Apache® Eagle™ analyzes data activities, yarn applications, JMX metrics, and daemon logs etc., provides state-of-the-art alert engine to identify security breach, performance issues and shows insights. Ingestion (Metric/Log/Event) Processing (Parsing/Enrich/Aggregation) Alerting (CEP/Correlation/ML) Insight (Storage/Dashboard)
- 7. Global Marketplace 42% GMV VIA MOBILE 291M MOBILE APP DOWNLOADS GLOBALLY 1.3B LISTINGS CREATED VIA MOBILE EVERY WEEK 162M ACTIVE BUYERS 25M ACTIVE SELLERS 800M ACTIVE LISTINGS 8.8M NEW LISTINGS EVERY WEEK Q3 2016 Q3 2016 http://www.ebay.com
- 8. Trustable Ecommerce Platform Metric Log Event Critical Event BIG DATA PLATFORM APPLICATIONS / DATABASE CLOUD PLATFORM Vision • Availability • Security Capability • Monitoring • Alerting
- 9. DevOps becomes a big data problem Eagle was initialized by end of 2013 for hadoop ecosystem monitoring as any existing tool like zabbix, ganglia can not handle the huge volume of metrics/logs generated by hadoop system in eBay. 2015/2016 10,000+ nodes 150,000+ cores 250 PB 2000+ user 3000+ nodes 10,000+ cores 50+ PB 2012 2011 1000+ nodes 10,000+ cores 10+ PB 100+ nodes 1000 + cores 1 PB 2010 2009 50+ nodes 2007 1-10 nodes Hadoop Data • Security • Activity Hadoop Platform • Heath • Availability • Performance Hadoop @ eBay Inc 7+ CLUSTERS 10000+ NODES 250+ PB DATA 10 B+ EVENTS / DAY 500+ METRIC TYPES 50,000+ JOBS / DAY 50,000,000+ TASKS / DAY
- 10. Apache Eagle - Overview Daemon Log Audit Log JMX Metrics Job Log Ingested in real-time YARN ...... System Metrics Service Heath Job Performance Downgrade Node Failure Security Breach ...... ...... Flexible Policy Definition Dynamic and Scalable Engine 1 2 3 Source processing and policy enforcement in real-time Extensible Monitoring Use Cases
- 11. Apache Eagle - Typical Use Cases 3 4 Security Monitoring Instantly identify sensitive data access and malicious operations Job Performance Monitoring Hadoop, Spark job profiling and performance analysis 2 Bad Node Detection Detect soft failure issues, Linux filesystem ACL, disk full 1 Service Health Check Service and process aliveness, JMX status as well as JVM GC
- 12. Apache Eagle – Security Monitoring CaseData Loss Prevention Get alerted and stop a malicious user trying to copy, delete, move sensitive data from the Hadoop cluster. Malicious Logins Detect login when malicious user tries to guess password. Eagle creates user profiles using machine learning algorithm to detect anomalies Unauthorized access Detect and stop a malicious user trying to access classified data without privilege. User Privileges Common Data Sets Patterns CommandsZones Query Columns Malicious user operation Detect and stop a malicious user trying to delete large amount of data. Operation type is one parameter of Eagle user profiles. Eagle supports multiple native operation types.
- 13. Apache Eagle –Bad Node Detection CaseUse Case Detect node anomaly by analyzing task failure ratio across all nodes Assumption Task failure ratio for every node should be approximately equal Algorithm Node by node compare (symmetry violation) and per node trend
- 14. Apache Eagle – Bad Node Alerting& AutomationAlerting: Detection Alerting Insight: Task failure drill-down Automation: Node Remediation
- 15. Apache Eagle - Case Onboarding 1. Register a new Monitored Site 2. Choose and Install Application 3. Configure Application 4. Administrate Application 5. Define Alert Policies and Explore Alerts 6. Analyze with Dashboards and Insights
- 16. Apache Eagle - Architecture Applications (Process/Job/Policy/View) Messaging (Kafka) Alerting (Storm) Storage (HBase) Metadata (Metadata-driven) Metric/Entity/Log PolicyConfig Stream Onboarding Administration Monitoring Insights Alerts Daemon Log Audit Log JMX Metrics Job Log YARN ...... System Metrics ...... System Metrics
- 17. Job Performance AppsHadoop Monitoring Apps Apache Eagle - Components App Framework Lifecycle Manage Alert Engine Streaming and Real time Storage Engine Easy and Fast Query StaticResourceApp Dashboard/Analytics Features/Policies SchedulingApp HealthCheck/Anomal y Detector Jobs StreamingApp Ingest/Process/Aggre gation Eagle Interface UI Dashboard API Eagle Core Scheduling Engine Job Workflow Scheduling Hadoop Security Apps Eagle Apps Integration JMX/System Hardware Metric Service/Process/Topology Availability HealthCheck MR Job Monitoring Spark Job Monitoring Cluster Capacity Analytics Hadoop Audit/Security HBase Audit/Security Job Access Security
- 18. Apache Eagle - Application Framework An “Application” is case-oriented solution package Installation: Application user guide, configuration, management Ingestion: Provide data ingestion/collection approaches to integrate any kinds of monitor data sources Process: Analyze data source based on Storm Topology or Spark Streaming App Alerting: Stream: Structured stream exported for alerting with eagle alerting engine or persistence in eagle storage Model: Complex built-in policies or policy templates defined in SQL/Java code/ML model, etc. Insight: Monitoring Analytics UI or Dashboard
- 19. Apache Eagle - Application Framework
- 20. Execution RuntimeEagle AppsEagle Server Apache Eagle - Application Execution StreamingApp Ingest/Process/Aggreg ation StaticResourceApp Dashboard/Analytics Features/Policies SchedulingApp HealthCheck/Detector Jobs and Workflows Application Manager ● Loader (SPI) ● Lifecycle/Admin ● Configuration ● Monitoring Eagle UI REST API
- 21. Alert Engine Apache Eagle - Distributed Alert Engine from MetricStream[(name == 'ReplLag') and (value > 1000)] select * insert into outputStream; Messaging Notification Slack Insight Action ● Real-time Streaming: Apache Storm (Execution Engine) + Kafka (Messaging) ● Declarative Policy: CEP and Extensible Alert Model in streaming way ● Dynamical Onboarding & Correlation: Connect to new stream and change Stream Grouping in Runtime ● Hot Deploy & No Downtime: Metadata- driven and lightweight alert logic assignment
- 22. Apache Eagle – Policy Examples from hadoopJmxMetricEventStream [metric == "hadoop.namenode.fsnamesystemstate.capacityused" and value > 0.9] select metric, host, value, timestamp, component, site insert into alertStream; Example 1: Alert if hadoop namenode capacity usage exceed 90 percentages from every a = hadoopJmxMetricEventStream[metric=="hadoop.namenode.fsnamesystem.hastate"] -> b = hadoopJmxMetricEventStream[metric==a.metric and b.host == a.host and a.value != value)] within 10 min select a.host, a.value as oldHaState, b.value as newHaState, b.timestamp as timestamp, b.metric as metric, b.component as component, b.site as site insert into alertStream; Example 2: Alert if hadoop namenode HA switches
- 23. Apache Eagle - Policy Definition/Alert Example
- 24. Apache Eagle - Distributed Alert Engine Extensible Data Source Dynamic Sorting/Grouping Declarative CEP Policy Elastic Resource Pool MetadataCoordinator Topology Manager REST API (Schema/Policy)ZK Notify (Schedule) START/STOP AlertInsight Management Services DataSource User Interface: Register Data Source -> Design Stream Model -> Define Alert Policy REST API
- 25. Apache Eagle - Distributed Alert Engine MetadataCoordinator Topology Manager Topology Resource Pool REST API (Schema/Policy)ZK Notify (Schedule) START/STOP Stream Receiver Stream Router Alert Publisher Policy Evaluator Stream Receiver Stream Router Alert Publisher Policy Evaluator Stream Receiver Stream Router Alert Publisher Policy Evaluator AlertInsight Management Services DataSource
- 26. define stream SystemMetricStream ( metric string, host string, device string value double); from SystemMetricStream [name = "disk.usage.metric" and value > 0.99 ] #window.time(30 min) group by host, device insert into SystemAlertStream; Apache Eagle - Distributed Alert Engine SourceSpec PartitionSpec SortSpec PolicySpec PublishSpec From Policy Definition in User View to Engine View Schedule Assignment Source: kafka topic Schema: SystemMetricStream Window: 5min Margin: 1min Partition: Group by host, device Publish: SystemAlertStream Process: CEP Execution Plan
- 27. Apache Eagle - Distributed Alert Engine 3 Rebuild Assignments 2 Trigger ScheduleDefine new Policy 3 21 Stream Receiver Stream Router Alert Publisher Policy Evaluator Metadata Coordinator Notify with latest version 5. Watch notification 6. Pull notified version of metadata 7. Update components runtime according to metadata changes SourceSpec PartitionSpec PolicySpec PublishSpec AlertZKRoot/ receiver/ router/ evaluator/ publisher/ 4 6 5 78 Connect and flow stream through alert engine
- 28. Apache Eagle - TSDB Storage Engine • Light-weight ORM Framework for HBase/RDMBS • Full-function SQL-Like REST Query • Optimized Rowkey design for time-series data • Native HBase Coprocessor • Secondary Index Support @Table("alertdef") @ColumnFamily("f") @Prefix("alertdef") @Service(AlertConstants.ALERT_DEFINITION_SERVICE_ENDPOINT_NAME) @JsonIgnoreProperties(ignoreUnknown = true) @TimeSeries(false) @Tags({"site", "dataSource", "alertExecutorId", "policyId", "policyType"}) @Indexes({ @Index(name="Index_1_alertExecutorId", columns = { "alertExecutorID" }, unique = true), }) public class AlertDefinitionAPIEntity extends TaggedLogAPIEntity{ @Column("a") private String desc; @Column("b") private String policyDef; @Column("c") private String dedupeDef; Query=AlertDefinitionService[@dataSource="hiveQueryLog"]{@policyDef}
- 29. Apache Eagle - What's Next Eagle Alert Engine on Apache Beam • Unified streaming on Spark/Flink Eagle Integration with Ambari/Cloudera Manager • Seamless connect monitoring data source Eagle on Cloud • Support deployment and monitor service on AWS Unified Monitoring Applications • Monitor real-time/online platform like Storm/Kafka/Database, etc.
- 30. Apache Eagle - Learn more Community • Website: http://eagle.apache.org • Github: http://github.com/apache/eagle • Mailing list: [email protected] Publications • EAGLE: USER PROFILE-BASED ANOMALY DETECTION IN HADOOP CLUSTER (IEEE) • EAGLE: DISTRIBUTED REAL-TIME MONITORING FRAMEWORK FOR HADOOP CLUSTER
- 31. Apache Eagle - Community And more ….
- 32. Open Source If you want to go fast, go alone. If you want to go far, go together. -- African Proverb Open Sourced By
- 33. Thanks and We are Hiring! http://eagle.apache.org [email protected] apache/incubator-eagle @TheApacheEagle