Event Stream Processing SAP
2 likes703 views
This document provides an introduction to Sybase Event Stream Processing (ESP). ESP is a technology for analyzing streams of event data in real-time to derive continuous intelligence. It allows defining logic to combine data sources, compute values, detect patterns, and produce summaries. The ESP runtime continuously processes incoming event streams before storing to disk, enabling low latency analysis. It is not a replacement for databases but rather complements them by supporting continuous, real-time analysis of fast data streams. ESP uses data-flow programming where streams and operations on the data are defined as it flows from sources to outputs.
Ad
More Related Content
What's hot (18)
Similar to Event Stream Processing SAP (20)
Ad
More from Gaurav Ahluwalia (11)
Ad
Event Stream Processing SAP
- 1. Event Stream Processor as seen by SAP An evolution of event driven architecture a white paper Author- Gaurav Ahluwalia Table of Contents Abstract ...............................................................................................................................................2 What is Sybase Event Stream Processing? an Introduction.................................................................3 Event Stream Processor as a Development Platform...........................................................................4 Event Stream Processor Runtime Environment...................................................................................4 Event Streams.......................................................................................................................................5 Event Stream Processor Compared to Databases.................................................................................5 Data-Flow Programming......................................................................................................................6 ESP Projects: Adapters, Streams, Windows, and Continuous Queries.................................................7 Streams Versus Windows......................................................................................................................7 Input, Output, and Local Streams and Windows..................................................................................8 Schemas................................................................................................................................................8 Product Components............................................................................................................................9 Input and Output Adapters..................................................................................................................10 Authoring Methods.............................................................................................................................11 Continuous Computation Language...................................................................................................12 SPLASH.............................................................................................................................................12
- 2. Abstract Event stream processing, or ESP, is a set of technologies designed to assist the construction of event-driven information systems. ESP technologies include event visualization, event databases, event-driven middleware, and event processing languages, or complex event processing (CEP). In practice, the terms ESP and CEP are often used interchangeably. ESP deals with the task of processing streams of event data with the goal of identifying the meaningful pattern within those streams, employing techniques such as detection of relationships between multiple events, event correlation, event hierarchies, and other aspects such as causality, membership and timing. ESP enables many different applications such as algorithmic trading in financial services, RFID event processing applications, fraud detection, process monitoring, and location-based services in telecommunications. Event Stream can be visualized by the under stated imagination. Stream of events, a regular burst of events which would handled by processor routines. Its like firing multiple arrows which initiate processor routines. Like a microprocessor handles interrupts which can instead trigger interrupt routines. You need lot of computing power to process these event streams because they are on application layer and HANA is one the power houses for SAP which provide stupendous computing speeds.This might be the next version of event driven architecture.
- 3. What is Sybase Event Stream Processing? an Introduction Sybase® Event Stream Processor enables you to create and run your own complex event processing (CEP) applications to derive continuous intelligence from streaming event data in real time. Please think this way intelligence in philosophy is considered as answering the puzzles or questions which come past to you in life. Events are generally created by subtle questions and requirements from day to day business operations. So Complex Event Processing will give you continuous intelligence as events are a sort of questions or puzzles. This intelligence would help in decision making for a business and organization. Event Stream Processing and CEP Event stream processing is a form of CEP, a technique for analyzing information about events, in real time, for situational awareness. When vast numbers of event messages are flooding in, it is difficult to see the big picture. With event stream processing, you can analyze events as they stream in and identify emerging threats and opportunities as they happen. Event Stream Processor Server filters, aggregates, and summarizes data to enable better decision making based on more complete and timely information. Event Stream Processor is not a presentation layer stuff. People might not interact with event stream. Its a highly scalable technology which enables (by providing tools) to deploy complex project in a tweak. Think like a responsive web design. Its highly scalable runs on every bit of hardware on any screen resolution. Event Stream Processor as a Development Platform As a platform for developing CEP projects, Event Stream Processor provides high-level tools for defining how events are processed and analyzed. Analyses of Event Streams can be done either through visual modelling or by text oriented authoring environment. You can define logic that is applied to incoming events to: 1) Combine data from multiple sources, producing derived event streams that include richer and more complete information. So event streams can produce evolved event streams.
- 4. 2) Compute value-added information to enable rapid decision making. You use these bits and pieces of intelligence generated for your purpose. 3) Watch for specific conditions or patterns to enable instantaneous response. 4) Produce high-level information, such as summary data, statistics, and trends to see the big picture, or the net effect, of many individual events. New trend of analytics management might see a different BI/BW coming into action going foward these years. 5) Continuously recompute key operating values based on complex analysis of incoming data. You want a specific ratio to be followed, production target, sales target you can do a backward scheduling from it. Event Stream Processor Runtime Environment As an engine for an event-driven architecture (EDA), Event Stream Processor can absorb, aggregate, correlate, and analyze events to produce new high-level events that can trigger responses, and high-level information that shows the current state of the business. Event Stream Processor. Processes data continuously as it arrives. • Processes data before it is stored on disk, thus achieving extremely high throughput and low latency, enabling better decision making based on more complete and timely information. • Separates business logic from data management, making it easier to maintain the business logic and reducing total cost of ownership. • Provides enterprise class scalability, reliability, and security. Event Streams An business event is a message that contains information about an actual business event that occurred. Many business systems produce events when things happen. A quantum of event stream is like when there some raw material out of stock in storage bins. It raises a business event. It has all the information about Material such ID, Type, its classifications, attributres which are required at to sent to processor routine for futher processing. There can be many such quantums generated in a split of a second. More examples for these streams would be: • Financial market data feeds that transmit trade and quote events. Events may consist of ticket symbol, price, quantity, time, and so on. • Radio Frequency Identification System (RFID) sensors that transmit events indicating that
- 5. an RFID tag was sensed nearby. • Click streams, which transmit a message (a click event) each time a user clicks a link, button, or control on a Web site. • Transaction events, which occur each time a record is added to a database or updated in a database. Event Stream Processor Compared to Databases Sybase Event Stream Processor complements traditional databases to help solve new classes of problems where continuous, event-driven data analysis is required. Its not an alternative to databases. Its doing analysis of real time data before storing it in a disk. databases are not effective at continuously analyzing fast moving streams of data. • Process of analysing data has reversed. For example of a relational database, data is stored on commits and then it gets fetched from database for analysis. For Event Streams real time data is analysed and part of domain and ranges of the analysis are stored in Database for future use. • Databases do not use preregistered continuous queries. Database queries are "one- timeonly" queries. To ask a question ten times a second, you must issue the query ten times a second. This model breaks down when one or more such queries need to be executed continuously as polling the database faster results in a performance impact to the source systems. Additionally, the polling approach has latency. • Databases do not use incremental processing. Event Stream Processor can evaluate queries incrementally as data arrives. Event Stream Processor is not an in-memory database. It shares some traits with in-memory databases in that it operates in and holds all data in memory, to achieve desired speed. Data-Flow Programming In data-flow programming, you define a set of event streams and the connections between them, and apply operations to the data as it flows from sources to outputs. Its just like a multiplexer absorbing electrical signals out of a different sources and getting output. The bit of it would be a logic gate. There can multiple steps to different descisions just like a flowchart.
- 6. Data-flow programming breaks a potentially complex computation into a sequence of operations with data flowing from one operation to the next. This technique also provides scalability and potential parallelization, since each operation is event driven and independently applied. Example of a Visual Modeling tool for data flow programming ESP Projects: Adapters, Streams, Windows, and Continuous Queries. An ESP project defines a set of event streams, any other required datasources, and the business logic applied to incoming event data to produce results. At its most basic level, a project consists of: • Input streams and windows – where the input data flows into the project. • Adapters – connect an input stream or window to a datasource. The plugs. • Derived streams and windows – take data from one or more streams or windows and apply a continuous query to produce a new stream or window. The Evolved creatures. Getting Results from an ESP Project Event Stream Processor has four ways to get output from a running project: • Applications receive information automatically from internal output adapters attached to a stream when you build the project. Adapters are like plugs to inserted into pipes which has fast running water. You can insert or take out water from the pipe though plug. Similar case with adapters.
- 7. • Applications can subscribe to data streams by means of an external subscriber, which users can create using subscription APIs provided with the product. Its like public event handler for streams. • Users can start a new project that binds (connects) to a stream in a running project, without reconfiguring the project. • Users can run on-demand queries using the esp_query tool to query output windows in a running ESP project. For more information see the Utilities Guide. Streams Versus Windows Both streams and windows process events. The difference is that windows have state, meaning they can retain and store data, while streams are stateless and cannot. Windows Streams Stateful and can retain data Stateless and can't retain data Window consists of a table where incoming events can add rows, update existing rows, or delete rows. Process incoming events and produce output events according to the continuous query that is attached to the stream. You can set the size of the window based on time, or on the number of events recorded. Nothing like this sort with streams. Input, Output, and Local Streams and Windows Streams and windows can be designated as input, output, or local. Input Output Streams/Windows Local Streams/Windows Input streams are the point at which data enters the project from external sources via adapters. Local and output streams and windows take their input from other streams or windows, rather than from adapters. A project may have any number of input streams. They apply a continuous query to produce their output. Input streams do not have continuous queries attached to them, although you can define filters for them. When a subscriber selects which stream or window to subscribe to, only output streams and windows are available. Local streams and windows are identical to output streams and windows.except that local streams and windows are hidden from outside subscribers.
- 8. Schemas By defination a schema is a metadata object which describe a Stream or Window. Each stream or window has a schema, which defines the columns in the events produced by the stream or window. Each column has a name and datatype. All events that output from a single stream or window have an identical set of columns. Inserts, Updates, and Deletes Operation Codes (opcodes) associate insert, update, and delete events with a window. In many Event Stream Processor use cases, events are independent of each other each carries information about something that happened. If you define a window on this type of event stream, each incoming event is inserted into the window. Relationship between Window and events So events are the rows of the Window. Events deliver new information about previous events. The ESP Server needs to maintain a current view of the set of information as the incoming events and continuously update it. Exmaple open orders in a fulfillment system. Event Stream Processor recognizes insert, update, and delete operations associated with incoming events. • A special field that indicates whether the event is an insert event, an update event, or a delete event. There is also an upsert opcode, which either updates an existing record with a matching key, or inserts a new record. • Input windows apply insert, update, and delete events to the data in the window directly,
- 9. as events arrive. Inserts, updates, and deletes are propagated through the query graph, that is, all downstream derived windows. Product Components Event Stream Processor includes a server component for processing and correlating streams of data, a Studio environment for developing, testing, and starting applications that run on the server, and administrative tools. Components include: • ESP Server – the software that processes and correlates data streams at runtime. • ESP Studio – an integrated development environment for creating, modifying, and testing ESP projects. • CCLcompiler – the compiler that translates and optimizes projects for processing by ESP Server. It is invoked by ESP Studio or from the command line. • Input and output adapters – the components that establish connections between Event Stream Processor and datasources, as well as the connections between the ESP Server and the consumers that will receive output from Event Stream Processor. • Integration SDK – a set of APIs for creating custom adapters in C/C++, Java, and .NET,for integrating custom function libraries, and for managing and monitoring live projects. • Utilities – a set of executables that offer command line access to many administrative, project development, publishing and subscription, and other features. Input and Output Adapters The plugs in the system can be both ways.Input and output adapters enable Event Stream Processor to send and receive messages from dynamic and static external sources and destinations. External sources or destinations can include: • Data feeds • Sensor devices • Messaging systems • Radio frequency identification (RFID) readers • E-mail servers • Relational databases
- 10. Input Adapter Output Adapter Custom Adapter Input adapters connect to an external datasource and translate incoming messages from the external sources into a format that is accepted by the ESP server. Output adapters translate rows processed by Event Stream Processor into message formats that are compatible with external destinations and send those messages downstream. In addition to the adapters provided by Event Stream Processor, you can write your own adapters to integrate into the server. Example of a adapter circuit for ESP. Authoring Methods Visual Modeling( Process chains) Text Based Authoring( Coding ) In the visual authoring environment, you can develop projects using graphical tools to define streams and windows, connect them, integrate with input and output adapters, and create a project consisting of queries. In the text authoring environment, you can develop projects in the Continuous Computation Language (CCL), as you would in any text editor. Create data streams and windows, develop queries, and organize them in hierarchical modules and projects. You can swtich between both editors very easily. Changes made in one editor are reflected in the other. You can also compile projects within Studio. Continuous Computation Language • CCLis the primary event processing language of the Event Stream Processor. ESP projects
- 11. are defined in CCL. • CCLis the primary event processing language of the Event Stream Processor. ESP projects are defined in CCL. • CCL supports sophisticated data selection and calculation capabilities, including features such as: data grouping, aggregations, and joins. • CCL also includes features that are required to manipulate data during real-time continuous processing, such as windows on data streams, and pattern and event matching. • The key distinguishing feature of CCL is its ability to continuously process dynamic data. • A SQL query typically executes only once each time it is submitted to a database server and must be resubmitted every time a user or an application needs to reexecute the query. • Although CCL borrows SQL syntax to define continuous queries, the ESP server does not use an SQL query engine. • Instead, it compiles CCL into a highly efficient byte code that is used by the ESP server to construct the continuous queries within the data-flow architecture. • CCL queries are converted to an executable form by the CCL compiler. ESP servers are optimized for incremental processing, hence the query optimization is different than for databases. SPLASH • Stream Processing Language SHell (SPLASH) is a scripting language that brings extensibility to CCL, allowing you to create custom operators and functions that go beyond standard SQL. • The ability to embed SPLASH scripts in CCL provides tremendous flexibility, and the ability to do it within the CCL editor maximizes user productivity. • SPLASH also allows you to define any complex computations that are easier to define using procedural logic rather than a relational paradigm. • SPLASH is a simple scripting language comprised of expressions used to compute values from other values, as well as variables, and looping constructs, with the ability to organize instructions in functions.