Complex Event Processor 3.0.0 - An overview of upcoming features
7 likes1,838 views
This document provides an overview of upcoming features in WSO2 Complex Event Processor 3.0.0. It discusses the Siddhi CEP engine, event processing queries including filters, windows, joins, patterns and event tables. It also covers high availability, persistence, scaling, integration with BAM, and performance comparisons with Esper. The document concludes with a demo of monitoring stock prices and tweets to detect significant stock price changes when a company is highly discussed on Twitter.
![Filters
from <stream-name> [<conditions>]*
select <attributes>
insert into <stream-name>
! Filters the events by conditions, use to detect simple
condition
! Conditions
o >, <, = , <=, <=, !=
o contains, instanceof
o and, or, not
! Example
from cseEventStream[price >= 20 and symbol==’IBM’]
select symbol, volume
insert into StockQuote](https://image.slidesharecdn.com/complexeventprocessor3-131008012538-phpapp01/85/Complex-Event-Processor-3-0-0-An-overview-of-upcoming-features-18-320.jpg)
![Window
from <stream-name> [<conditions>]#window.<window-name>(<
parameters>)
select <attributes>
Insert into <stream-name>
Types of Windows
● (Time | Length) (Sliding| Batch) windows
● Type of aggregate functions
● sum, avg, max, min
Example
from cseEventStream[price >= 20]#window.lengthBatch(50)
select symbol, avg(price) as avgPrice
group by symbol
having avgPrice>50
insert into StockQuote](https://image.slidesharecdn.com/complexeventprocessor3-131008012538-phpapp01/85/Complex-Event-Processor-3-0-0-An-overview-of-upcoming-features-19-320.jpg)
![Join
from <stream>#<window> [unidirectional] join <stream>#<window>
on <condition> within <time>
insert into <stream>
! Use to join two streams based on a condition. There
must be at least one window defined
! Unidirectional – event arriving only to the
unidirectional stream triggers join
! Example
from TickEvent[symbol==’IBM’]#window.length(2000)
join NewsEvent#window.time(5 min)
on TickEvent.symbol==NewsEvent.company
select *
insert into JoinStream *](https://image.slidesharecdn.com/complexeventprocessor3-131008012538-phpapp01/85/Complex-Event-Processor-3-0-0-An-overview-of-upcoming-features-20-320.jpg)
![Pattern
from [every] <condition> → [every] <condition> … <condition> within
<time>
select <attributes>
insert into StockQuote
! Use to Check condition A happen before/after
condition B.
! Can do iterative checks via “every” keyword.
! Here with “within <time>”, SIddhi emits only events
that are within that time of each other
! Example
from every (a1 = purchase[price < 10] ) -> a2 = purchase [price >10000 and
a1.cardNo==a2.cardNo] within 1 day
select a1.cardNo as cardNo, a2.price as price, a2.place as place
insert into potentialFraud
y1 a1 x1 k5 a2 n7](https://image.slidesharecdn.com/complexeventprocessor3-131008012538-phpapp01/85/Complex-Event-Processor-3-0-0-An-overview-of-upcoming-features-21-320.jpg)
![Sequence
from <event-regular-expression> within <time>
select <attributes>
Insert into <stream>
! Regular Expressions supported
o * - Zero or more matches (reluctant).
o + - One or more matches (reluctant).
o ? - Zero or one match (reluctant).
o or – either event
! Here we have to refer events returned by * , + using square
brackets to access a specific occurrence of that event
from a1 = requestOrder[action == "buy"],
b1 = cseEventStream[price > a1.price and symbol==a1.symbol]+,
b2 = cseEventStream[price <b1.price]
select a1. symbol as symbol, b1[0].price as firstPrice, b2.price as orderPrice
insert into purchaseOrder
y1 a1 b1 b1 b2 n7](https://image.slidesharecdn.com/complexeventprocessor3-131008012538-phpapp01/85/Complex-Event-Processor-3-0-0-An-overview-of-upcoming-features-22-320.jpg)
![Working with Event Tables
from <stream> (select <attribute-name> (,<attribute-name>)*
)? insert into <table-name>
Inserts the selected attributes from the input stream into the
event table.
from cseEventCheckStream[symbol==cseEventTable.symbol in
cseEventTable] insert into outStream;
For update and delete
from <stream> update <table-name> (on <condition>)?
from <stream> delete <table-name> (on <condition>)?](https://image.slidesharecdn.com/complexeventprocessor3-131008012538-phpapp01/85/Complex-Event-Processor-3-0-0-An-overview-of-upcoming-features-25-320.jpg)
![Performance sending event within same JVM
Simple filter without window
from StockTick[prize >6] return symbol, price](https://image.slidesharecdn.com/complexeventprocessor3-131008012538-phpapp01/85/Complex-Event-Processor-3-0-0-An-overview-of-upcoming-features-27-320.jpg)
![Queries
from allStockQuotes[win.time(60000)]
select symbol,price, avg(price) as averagePrice
group by symbol
having ((price > averagePrice*1.02) or (averagePrice*0.98 > price ))
insert into fastMovingStockQuotes
from twitterFeed[win.time(60000)]
select company as company, sum(wordCount) as words
group by company
having (words > 10)
insert into highFrequentTweets
from fastMovingStockQuotes[win.time(60000)] as fastMovingStockQuotes
join highFrequentTweets[win.time(60000)] as highFrequentTweets
on fastMovingStockQuotes.symbol==highFrequentTweets.company
select fastMovingStockQuotes.symbol as company,
fastMovingStockQuotes.averagePrice as amount,
highFrequentTweets.words as words
insert into predictedStockQuotes](https://image.slidesharecdn.com/complexeventprocessor3-131008012538-phpapp01/85/Complex-Event-Processor-3-0-0-An-overview-of-upcoming-features-40-320.jpg)
Complex Event Processor 3.0.0 - An overview of upcoming features
- 1. WSO2 Complex Event Processor 3.0.0 An overview of upcoming features By S. Suhothayan Associate Technical Lead, Team Lead CEP.
- 2. About WSO2 • Providing the only complete open source componentized cloud platform – Dedicated to removing all the stumbling blocks to enterprise agility – Enabling you to focus on business logic and business value • Recognized by leading analyst firms as visionaries and leaders – Gartner cites WSO2 as visionaries in all 3 categories of application infrastructure – Forrester places WSO2 in top 2 for API Management • Global corporation with offices in USA, UK & Sri Lanka – 200+ employees and growing • Business model of selling comprehensive support & maintenance for our products
- 3. 150+ globally positioned support customers
- 4. Outline ! Scenarios of Event Processing ! WSO2 CEP Server & SOA integrates ! The Siddhi Runtime CEP Engine. ! High availability, Persistence and Scalability of WSO2 CEP ! How CEP can be combined with Business Activity Monitoring (BAM). ! Demo
- 5. CEP Is & Is NOT! ! Is NOT! o Simple filters • Simple Event Processing • E.g. Is this a gold or platinum customer? o Joining multiple event streams • Event Stream Processing ! Is ! o Processing multiple event streams o Identify meaningful patterns among streams o Using temporal windows • E.g. Notify if there is a 10% increase in overall trading activity AND the average price of commodities has fallen 2% in the last 4 hours
- 6. WSO2 CEP Server ! Enterprise grade server for CEP runtimes ! Supports several transports (network access) ! Supports several data formats ! Support for multiple CEP runtimes ! Governance ! Monitoring ! Tools (WSO2 Dev Studio)
- 8. Siddhi CEP Runtime ! Apache License, a java library, Tuple based event model ! Supports distributed processing ! Supports multiple query models • Based on a SQL-like language • Supports ! Partitions ! Filters ! Windows ! Joins ! Ordering ! Output Rate Limiting ! and others
- 9. CEP Event Adaptors ! Is an adaptor for receiving and publishing events ! Has the configurations to connect to external endpoints ! Its many-to-many with CEP engine
- 10. CEP Event Adaptors Support for several transports (network access) and data formats ● SOAP/WS-Eventing XML messages ● REST JSON messages ● JMS Map messages XML messages Text messages JSON messages ● SMTP (Email) Text messages JSON messages XML messages ● Thrift - WSO2 data format High Performant Event Capturing & Delivery Framework supports Java/C/C++/C# via Thrift language bindings WSO2 Event
- 11. CEP Event Adaptors ● Cassandra (from CEP 3.0.0) Map messages ● Fix (from CEP 3.0.0+) Map messages ● MYSQL (from CEP 3.0.0) Map messages ● HBase (from CEP 3.0.0+) Map messages & Event adaptors are pluggable !
- 12. CEP Event Builders • Event builder converts different input event types to a type compatible with the execution plan. • Subscribes to an Input Event Adaptor to listen for events and sends a converted WSO2 Event or Basic Event to the Execution Plan • Receives events in different formats and exposes those input streams as stream definitions • Has a one to many relationship with execution plans in execution plan.
- 13. CEP Execution Plan ● Is an isolated logical execution unit ● Each execution plan has a set of Queries Input & Output stream mappings. ● Its one-to-one with a CEP Backend Runtime Engine ● It deals with Siddhi processing engine.
- 14. CEP Event Formatter Event formatter does the inverse – listens to events coming from event processor and sends converted events to Event adaptors. There are 5 types of output mapping types are available ● Map ● Text ● WSO2Event ● XML ● JSON
- 15. Monitoring (Event Tracer & Event Statistics) ! Provides real-time statistical visual illustrations of request & response counts per time based on CEP server, execution plan, transport adaptor, event builder and formatter.
- 16. Writing CEP Queries (using Siddhi Runtime)
- 17. Siddhi Queries ! Filters and Projection ! Windows o Events are processed within temporal windows. (e.g. for aggregation and joins) Time window vs. length window. ! Joins - Join two streams ! Event ordering - Identify event sequences and patterns ! Event Partitions ! Event Tables
- 18. Filters from <stream-name> [<conditions>]* select <attributes> insert into <stream-name> ! Filters the events by conditions, use to detect simple condition ! Conditions o >, <, = , <=, <=, != o contains, instanceof o and, or, not ! Example from cseEventStream[price >= 20 and symbol==’IBM’] select symbol, volume insert into StockQuote
- 19. Window from <stream-name> [<conditions>]#window.<window-name>(< parameters>) select <attributes> Insert into <stream-name> Types of Windows ● (Time | Length) (Sliding| Batch) windows ● Type of aggregate functions ● sum, avg, max, min Example from cseEventStream[price >= 20]#window.lengthBatch(50) select symbol, avg(price) as avgPrice group by symbol having avgPrice>50 insert into StockQuote
- 20. Join from <stream>#<window> [unidirectional] join <stream>#<window> on <condition> within <time> insert into <stream> ! Use to join two streams based on a condition. There must be at least one window defined ! Unidirectional – event arriving only to the unidirectional stream triggers join ! Example from TickEvent[symbol==’IBM’]#window.length(2000) join NewsEvent#window.time(5 min) on TickEvent.symbol==NewsEvent.company select * insert into JoinStream *
- 21. Pattern from [every] <condition> → [every] <condition> … <condition> within <time> select <attributes> insert into StockQuote ! Use to Check condition A happen before/after condition B. ! Can do iterative checks via “every” keyword. ! Here with “within <time>”, SIddhi emits only events that are within that time of each other ! Example from every (a1 = purchase[price < 10] ) -> a2 = purchase [price >10000 and a1.cardNo==a2.cardNo] within 1 day select a1.cardNo as cardNo, a2.price as price, a2.place as place insert into potentialFraud y1 a1 x1 k5 a2 n7
- 22. Sequence from <event-regular-expression> within <time> select <attributes> Insert into <stream> ! Regular Expressions supported o * - Zero or more matches (reluctant). o + - One or more matches (reluctant). o ? - Zero or one match (reluctant). o or – either event ! Here we have to refer events returned by * , + using square brackets to access a specific occurrence of that event from a1 = requestOrder[action == "buy"], b1 = cseEventStream[price > a1.price and symbol==a1.symbol]+, b2 = cseEventStream[price <b1.price] select a1. symbol as symbol, b1[0].price as firstPrice, b2.price as orderPrice insert into purchaseOrder y1 a1 b1 b1 b2 n7
- 23. Event Paritions define <partition-id> by <partition-type> (,<partition-type>)* Partition types can be one of two types • Variable Partitions - Partitions are created by the discrete values that are encountered for a variable define partition StockSymbol by StockStream.symbol • Range partitions - Partitions are created according to predefined ranges of variables define partition stockVolume by range volume < 10 as 'SMALL', range volume > 10 and volume < 100 as 'MEDIUM', range volume > 100 as 'LARGE'
- 24. Event Tables define table <table-name> (<attribute-name> <type> {, <attribute-name> <type>}*) ( from <table-type>.<datasource-name>:< database-name>.<table-name>)? Event tables can be used in the same manner as an event stream, with the difference being that events sent to an event table being persisted to a data source. CEP supports event tables for • In Memory • Relational o MySQL o H2 define table cseEventTable(symbol string, price int, volume float) from MYSQL.cepDataSource:cepdb.cepEventTable0
- 25. Working with Event Tables from <stream> (select <attribute-name> (,<attribute-name>)* )? insert into <table-name> Inserts the selected attributes from the input stream into the event table. from cseEventCheckStream[symbol==cseEventTable.symbol in cseEventTable] insert into outStream; For update and delete from <stream> update <table-name> (on <condition>)? from <stream> delete <table-name> (on <condition>)?
- 26. Performance Results ! We compared Siddhi with Esper, the widely used opensource CEP engine ! For evaluation, we did setup different queries using both systems, push events in to the system, and measure the time till all of them are processed. ! We used Intel(R) Xeon(R) X3440 @2.53GHz , 4 cores 8M cache 8GB RAM running Debian 2.6.32-5-amd64 Kernel
- 27. Performance sending event within same JVM Simple filter without window from StockTick[prize >6] return symbol, price
- 28. Performance sending event within same JVM State machine query for pattern matching From f=FraudWarningEvent -> p=PINChangeEvent(accountNumber=f.accountNumber) return accountNumber;
- 29. Performance Sending Events over the network ! Here we publihsed data from two client publisher nodes to the CEP Sever node and sent the triggered notifications of CEP to a client subscriber node. ! To test the worsecase sinario, 100% of the data published to CEP is recived at the subscriber node after processing (No data is filtered) ! We used Intel® Core™ i7-2630QM CPU @ 2.00GHz, 8 cores, 8GB RAM running Ubnthu 12.04, 3.2.0-32- generic Kernel, for running CEP and used Intel® Core™ i3-2350M CPU @ 2.30GHz, 4 cores, 4GB RAM running Ubnthu 12.04, 3.2.0-32-generic Kernel, for the three client nodes.
- 30. HA/ Persistence ! Ability to recover runtime state in the case of a failure. ! Enables queries to span lifetimes much greater than server uptime. ! Takes periodic snapshots and stores all state information to a scalable persistence store (Apache Cassandra). ! Supports pluggable persistent stores.
- 31. Scaling ! Vertically scaling o Can be distributed as a pipeline ! Horizontally scaling o Queries like windows, patterns, and Join have shared states, hence hard to distribute! o Use distributed cache (Hazelcast) to achieve this • shared memory and batch processing
- 32. Event Recording ! Ability to record all/some of the events for future processing ! Few options o Publish them to Cassandra cluster using WSO2 data bridge API or BAM (can process data in Cassandra with Hadoop using WSO2 BAM). o Write them to distributed cache o Custom thrift based event recorder
- 33. Integration with WSO2 BAM Data Receiving Data Analyzing Data Presentation Data Publishing
- 34. CEP Role within WSO2 Platform
- 35. DEMO
- 36. Scenario ! Monitoring stock exchange for game changing moments ! Two input event streams. o Event stream of Stock Quotes from a stock exchange o Event stream of word count on various company names from twitter pages ! Check whether the last traded price of the stock has changed significantly(by 2%) within last minute, and people are twitting about that company (> 10) within last minute
- 38. Input events ! Input events are JMS Maps o Stock Exchange Stream Map<String, Object> map1 = new HashMap<String, Object>(); map1.put("symbol", "MSFT"); map1.put("price", 26.36); publisher.publish("AllStockQuotes", map1); o Twitter Stream Map<String, Object> map1 = new HashMap<String, Object>(); map1.put("company", "MSFT"); map1.put("wordCount", 8); publisher.publish("TwitterFeed", map1);
- 39. Queries
- 40. Queries from allStockQuotes[win.time(60000)] select symbol,price, avg(price) as averagePrice group by symbol having ((price > averagePrice*1.02) or (averagePrice*0.98 > price )) insert into fastMovingStockQuotes from twitterFeed[win.time(60000)] select company as company, sum(wordCount) as words group by company having (words > 10) insert into highFrequentTweets from fastMovingStockQuotes[win.time(60000)] as fastMovingStockQuotes join highFrequentTweets[win.time(60000)] as highFrequentTweets on fastMovingStockQuotes.symbol==highFrequentTweets.company select fastMovingStockQuotes.symbol as company, fastMovingStockQuotes.averagePrice as amount, highFrequentTweets.words as words insert into predictedStockQuotes
- 41. Alert ! As a Email Hi Within last minute, people being twitting about {company} {words} times, and the last traded price of {company} has changed by 2% and now being trading at ${amount}. From CEP ! As a SMS
- 42. Useful links ! WSO2 CEP 3.0.0 http://ec2-54-224-94-128.compute-1.amazonaws.com/chunk-02/ N-25_09_2013/wso2cep-3.0.0.zip ! WSO2 CEP http://wso2.com/products/complex-event-processor/ ! CEP Performance Info http://srinathsview.blogspot.com/2013/08/cep-performance-processing-100k-to. html ! Distributed Processing Sample With Siddhi CEP and ActiveMQ JMS Broker. http://suhothayan.blogspot.com/2012/08/distributed-processing-sample-for-wso2. html ! Creating Custom Data Publishers to BAM/CEP http://wso2.org/library/articles/2012/07/creating-custom-agents-publish-events- bamcep ! WSO2 BAM 2.3.0 http://wso2.com/products/business-activity-monitor/
- 43. Engage with WSO2 • Helping you get the most out of your deployments • From project evaluation and inception to development and going into production, WSO2 is your partner in ensuring 100% project success
- 44. Questions?
- 45. Thank you.