The State of Streaming Analytics: The Need for Speed and Scale
2 likes1,731 views
Peter Vescuso, CMO, VoltDB and Mike Gualtieri Principal Analyst, Forrester Research talk about the latest on the streaming analytics market, in-memory database technology, and the emergence of ‘translytics’. Also hear how the latest release of VoltDB, the in-memory operational database, simplifies the distributed system complexity of building high-speed data pipelines with rapid ingestion of data, real-time analytics, Apache Kafka support, and Elasticsearch support.
Ad
More Related Content
What's hot (20)
Similar to The State of Streaming Analytics: The Need for Speed and Scale (20)
Ad
More from VoltDB (19)
Ad
Recently uploaded (20)
![Get & Download Wondershare Filmora Crack Latest [2025]](https://cdn.slidesharecdn.com/ss_thumbnails/revolutionizingresidentialwi-fi-250422112639-60fb726f-250429170801-59e1b240-thumbnail.jpg?width=560&fit=bounds)
![Download Wondershare Filmora Crack [2025] With Latest](https://cdn.slidesharecdn.com/ss_thumbnails/neo4j-howkgsareshapingthefutureofgenerativeaiatawssummitlondonapril2024-240426125209-2d9db05d-250419-250428115407-a04afffa-thumbnail.jpg?width=560&fit=bounds)
The State of Streaming Analytics: The Need for Speed and Scale
- 1. page THE STATE OF STREAMING ANALYTICS: THE NEED FOR SPEED AND SCALE
- 2. page© 2015 VoltDB OUR SPEAKERS Peter Vescuso CMO, VoltDB 2 Mike Gualtieri Principal Analyst Forrester Research
- 3. page © 2015 VoltDB page OVERVIEW • The Streaming Analytics Market • Solution for Analytics and Transactions • VoltDB -- New Features! • Use Cases 3
- 4. The State Of Streaming Analytics The Need For Speed and Scale Mike Gualtieri, Principal Analyst Twitter: @mgualtieri
- 5. #Analytics
- 6. © 2015 Forrester Research, Inc. Reproduction Prohibited 6 Learn Infer Detect Adapt True BI means having all kinds of analytics Predictive Analytics Streaming Analytics Descriptive Analytics (Advanced Analytics) Prescriptive Analytics Batch Real-time Most firms invest here They must invest here too
- 7. © 2015 Forrester Research, Inc. Reproduction Prohibited 7 Source: Forrester Research Advanced analytics is rightly surging “What is your firm's/business unit's current use of the following technologies?” Source: Forrester's Global Business Technographics Data And Analytics Survey, 2015 and 2014 Base: 1805 (2015), 1063 (2014) Sweet! Most of your competitors still haven’t started!19% 19% 24% 31% 34% 22% 22% 35% 31% 43% 53% 54% 50% 50% 69% 39% 42% 42% 42% 42% 43% 43% 46% 48% 52% 54% 55% 56% 57% 69% Non modeled data exploration and discovery Search/interactive discovery Streaming analytics Metadata generated analytics OLAP Advanced visualization Text analytics Location analytics Predictive analytics Process analytics Embedded analytics Web analytics Dashboards Performance analytics Reporting 2015 2014
- 8. #FastData
- 9. Some say that data is the new oil.
- 10. It’s more like the Sun – virtually limitless.
- 11. © 2015 Forrester Research, Inc. Reproduction Prohibited 11 Every industry is graced with more data › Rich transactional data from business applications › Usage and behavior data from web and mobile apps › Social media data › Log data › IoT device sensor and event data › Data economy – firms buying and selling data
- 12. Using your best estimate, what is the size of all data stored within your company? Source: Forrester Research, September 2015 Base: 100 US Managers and above currently using Hadoop for processing and analyzing data. Enterprises have plenty of data from both internal and external sources 10-49 Terabytes 5% 50-99 Terabytes 12% 100-500 Terabytes 54% Greater than 500 Terabytes 29% Internal business data 49% External source data 51% What % of the data available is from internal business applications (ERP and business applications) versus external sources (social, IoT)?
- 13. All data originates fast!
- 14. 110010011011001 010010011 010011001101 0100 CustomerData Transactions DataWarehosue IoT But, analytics is usually done much later.
- 15. #WhyWait
- 16. Perishable insights can have exponentially more value than after-the-fact traditional historical analytics.
- 17. #Perishable
- 18. How can you prevent this dude from fleecing you right now?
- 19. What are movers and shakers saying about equities that we cover right now?
- 20. How can you know if your baby is sleeping soundly or if something is wrong right now?
- 21. What offers should you make to your customer if they are within proximity of your store right now?
- 22. All data starts off fast!
- 23. #Problem
- 24. © 2015 Forrester Research, Inc. Reproduction Prohibited 24 What are the technical challenges impeding you from processing and analyzing more data? A. Difficulty integrating data from multiple sources. B. Data volume is too large. C. Difficulty in creating data models and/or preparing data for analytics. D. Too may data formats to integrate effectively. E. Data is difficult to access from multiple sources Quiz
- 25. #Streaming
- 26. Streaming analytics can capture and act on perishable insights.
- 27. DEFINITION FORRESTERStreaming analytics filter, aggregate, enrich, and analyze a high throughput of data from disparate live data sources to identify patterns, detect urgent situations, and automate immediate actions in real-time.
- 28. © 2015 Forrester Research, Inc. Reproduction Prohibited 28 Real-time means business time › A customer walks into a shopping mall › A shopper clicks on an online add › A temperature sensor spikes › A stock price rises › A customer uses a credit card › A customer wakes up
- 29. © 2015 Forrester Research, Inc. Reproduction Prohibited 29 Thinking in streams is different… › Ingest › Filter › Transform › Normalize › Link › Enrich › Correlate › Location/motion (geofencing) › Time windows › Temporal pattern detection › Business logic/rules execution › Action interfaces Real-time ETL Real-time Analytics
- 30. © 2015 Forrester Research, Inc. Reproduction Prohibited 30 Streaming analytics-to-action architecture
- 31. #Translytical
- 32. DEFINITION FORRESTERA single, unified database that supports transactions and analytics in real-time without sacrificing transactional integrity, performance, and scale.
- 33. In-memory streaming provides near-zero latency for complex data & analytical operations.
- 34. © 2015 Forrester Research, Inc. Reproduction Prohibited 34 S S S B B B D G GG DDD Streaming Analytics General-purpose data processing cluster Scale-up Database Data And Compute Grid Clustered Database Batch Real-time Dataflow
- 35. © 2015 Forrester Research, Inc. Reproduction Prohibited 35 Traditional analytic architectures must move data to analyze it Transactional Operational Analytical Data Movement Transactional Apps Operational Reporting Analytical/BI Insights
- 36. © 2015 Forrester Research, Inc. Reproduction Prohibited 36 Analytical silos have proliferated Transactional Operational Analytical Transactional Operational Analytical Transactional Operational Analytical CRM Stack BI Stack Mobile App Stack
- 37. © 2015 Forrester Research, Inc. Reproduction Prohibited 37 Translytical databases collapse the stack to reduce complexity and deliver analytics in real-time Transactional Operational Analytical Translytical Database Traditional Stack Single data layer
- 38. © 2015 Forrester Research, Inc. Reproduction Prohibited 38 Translytical databases are transactional and analytical Transactional Operational reporting Analytics/BI Streaming data On-premise data Cloud data In-Memory Scale-outMulti-modal Translytical Database Tiered DataCompression
- 40. Translytic databases use RAM to be real-time.
- 41. Transactions + Analytics = Translytical
- 42. #Opportunity
- 43. © 2015 Forrester Research, Inc. Reproduction Prohibited 43 Learn Infer Detect Adapt Start by adopting translytic databases for real- time applications Predictive Analytics Streaming Analytics Descriptive Analytics Prescriptive Analytics Past Present Future
- 45. page© 2015 VoltDB THE VOLTDB ARCHITECTURE FOR FAST DATA In-Memory performance (never lose data) Scale-out, shared nothing ACID & SQL & Java Real-time analytics Reliability and fault tolerance Hadoop ecosystem integration VoltDB is really different than everything else
- 46. page© 2015 VoltDB VOLTDB: A BEAUTIFUL ARCHITECTURE Work Queue Execution Engine Table and Index Data VoltDB Cluster Server 1 Partition 1 Partition 2 Partition 3 Server 2 Partition 4 Partition 5 Partition 6 Server 3 Partition 7 Partition 8 Partition 9 Inside a Partition
- 47. page© 2015 VoltDB THE MODERN OPERATIONAL APPLICATION Ingest Analyze Decide Streaming Analytics Transactions } } Fast Data =
- 48. page© 2015 VoltDB 1ST GENERATION FAST DATA: STREAMING ANALYTICS • Examples: Spark Streaming, Storm, Kinesis, Tibco Streambase, et al • Technical: • Lack “state” for transaction processing (operational) • Complex programming model • No ability to do ad hoc queries • Functional: • 1st Gen only offers streaming analytics • Separate database required for any meaningful work • Proprietary interface is inconsistent with the rest of the data pipeline • Does not support applications requirement for interaction 1stGen Streaming Stream Analytics Query Predefined
- 49. page© 2015 VoltDB 2ND GENERATION FAST DATA: STREAMING ANALYTICS & OPERATIONAL WORK • Streaming Analytics converges with the operational applications • Convergence is necessary to use data in real-time • Automated application interactions are informed by data • Brings the application into the “data analytics” world • Streaming Analytics alone is passive, Fast Data is interactive 1stGen2ndGen Streaming Stream Analytics Query Predefined Ad hoc Support Operational Work VoltDB
- 50. page© 2015 VoltDB VOLTDB’S SOLUTION FOR FAST DATA Single System • Operational DB (high speed) • Streaming Analytics Approached the problem from a database perspective • Same way low speed apps have always solved the problem • Performance scales to streaming levels (millions of tps) • Fully Integrated with Big Data Ecosystem • Import and Export are core functionalities
- 51. page© 2015 VoltDB VOLTDB 5.6 • Fast Data Pipeline • Distributed Database Replication
- 52. page© 2015 VoltDB FAST DATA PIPELINE VoltDB takes over the distributed system complexity of building high-speed ingestion fast data pipelines
- 53. page© 2015 VoltDB FAST DATA PIPELINE Data Lake HDFS/OLAP ExportIngest VoltDB
- 54. page© 2015 VoltDB FAST DATA PIPELINE Data Lake HDFS/OLAP Export VoltDB Customer-Facing - Personalization - Customer experience Operations-Facing - Network optimization - API monitoring - Sensors
- 55. page© 2015 VoltDB FAST DATA PIPELINE Data Lake HDFS/OLAP Export VoltDB Customer-Facing - Personalization - Customer experience Operations-Facing - Network optimization - API monitoring - Sensors Streaming Analytics - Filtering - Windowing - Aggregation - Enrichment - Correlations + Transactions - Context-aware - Personal - Real-time
- 56. page© 2015 VoltDB FAST DATA PIPELINE Export VoltDB Customer-Facing - Personalization - Customer experience Operations-Facing - Network optimization - API monitoring - Sensors Streaming Analytics + Transactions Batch/Iterative Analytics - Statistical correlations - Multi-dimensional analysis - Predictive analytics Data at Rest
- 57. page© 2015 VoltDB FAST DATA PIPELINE What’s New? Simpler, faster connections to your data pipeline infrastructure • New Kafka Importer • Without writing code, import multiple Apache Kafka streams; built-in, distributed, and fault tolerant. • Import different Kafka topics from a single set of brokers, or from different brokers, to separate tables or stored procedures. • Fast, synchronized export • Supporting the latest Kafka release, synchronized export avoids the connector hanging if the Kafka infrastructure becomes unresponsive
- 58. page© 2015 VoltDB FAST DATA PIPELINE What’s New? Simpler, faster connections to your data pipeline infrastructure • Multi-stream import and export • Import multiple streams to different tables or stored procedures • Import different Kafka topics from a single set of brokers, or from different brokers, to separate tables • Export data to multiple targets simultaneously, e.g., • Export de-duped sensor data to Hadoop once it has been processed • “Export” alerts regarding unusual events to HTTP for distribution via SMS, email, or other notification service
- 59. page© 2015 VoltDB FAST DATA PIPELINE What’s New? Simpler, faster connections to your data pipeline infrastructure • Elasticsearch Export Connector • Receives serialized data from the export tables and inserts it into an Elasticsearch server or cluster • Export data to Elasticsearch to perform full-text searches on VoltDB data more flexibly and faster.
- 60. page© 2015 VoltDB DATABASE REPLICATION
- 61. page© 2015 VoltDB DATABASE REPLICATION What’s New? Active/Active database replication for mission critical applications • Enables multiple database instances, including geographically distributed instances, to support the same application • Scalable, with no single point of failure, and supporting binary log-based disaster recovery (DR) Uses • “Hot standby" to improve uptime in case of failure • Protecting against catastrophic events -> disaster recovery • Offload read-only workloads, such as reporting
- 62. page© 2015 VoltDB DATABASE REPLICATION Server X p3p1 p2 Server Y p6p4 p5 Server Z p9p7 p8 East Coast Server A p3p1 p2 Server B p6p4 p5 Server C p9p7 p8 West Coast The VoltDB advantage: Replication by partition • Faster - parallel processing improves performance • More durable – individual partition streams can be redirected without interfering with the replication of the other partitions ReplicaMaster
- 63. page© 2015 VoltDB DATABASE REPLICATION What’s New? Rack-Aware Partitioning to improve availability • Partition database replicas with rack-awareness to prevent having replicas on the same physical rack, thereby improving availability in the event of a rack, chassis or power failure.
- 64. page© 2015 VoltDB RACK-AWARE PARTITIONING Old “All Nodes are Equal”
- 65. page© 2015 VoltDB RACK-AWARE PARTITIONING Old “All Nodes are Equal”
- 66. page© 2015 VoltDB RACK-AWARE PARTITIONING Old “All Nodes are Equal” New “Rack-aware” Replicas assigned to different racks
- 67. page© 2015 VoltDB page USE CASES
- 68. page© 2015 VoltDB DIY DISTRIBUTED DATA INFRASTRUCTURE? How to ensure data consistency, accuracy?
- 69. page© 2015 VoltDB SIMPLIFYING THE LAMBDA ARCHITECTURE Use Case • Content delivery network service provider • Counting content views Why VoltDB? • Real-time analytics+ transactions w/scale • Replaced Storm, Cassandra with VoltDB for real-time streaming aggregations with “exactly once” semantic Bottom line • Accurate – guaranteed correct results with VoltDB’s ‘exactly-once’ semantics • Faster time to market • 32 TB of data processed with 7 servers • 1/10th the resources of the alternatives
- 70. page© 2015 VoltDB MOBILE Use Case • The Emagine real-time event decision making platform for Communications Service Providers (CSPs) Why VoltDB? • Real-time analytics+ transactions • Scale - billions of network events per day, analyzing hundreds of thousands of transactions simultaneously, and then intelligently interacting with customers Bottom line • 3 ms system response time • 253% increase in offer purchases Real-Time Event Decisioning
- 71. page© 2015 VoltDB EAGLE INVESTMENT SYSTEMS, BNY MELON Use Case • Financial portfolio management and performance measurement • Over 160 clients, supporting over $21 trillion in assets • Performance tracking and risk management Why VoltDB? • Performance, scalability to meet application workloads and client SLAs • High speed data cache for risk calculations with large and rapidly changing data sets Bottom Line • Lower TCO “We plumb Eagle data with third party analytic engines with VoltDB as part of our overall architecture to collect, manufacture, validate and then distribute this information as fast as possible, as fast as the operations could support it.” Marc Firenze, CTO
- 72. page© 2015 VoltDB AIRPUSH Use Case • Managing online mobile advertising • Over 120,000 live applications Why VoltDB? • Replaced costly MySQL infrastructure with scalable VoltDB cluster Bottom line • Enabled accurate ad-campaign balance tracking, dramatically improving “last-dollar” decisions • Eliminated the opportunity cost of placing wrong ads • Reduced infrastructure cost by 93% (7 servers vs. 100) “Achieved a previously impossible level of ad budget management accuracy”
- 73. page© 2015 VoltDB WHY VOLTDB? Faster Smarter Better • Superior architecture for fast data/translytics • In-Memory, Scale-out, ACID, SQL+JSON • Rapid data ingestion with transactions • Data durability and HA • VoltDB customers realize exceptional business value
- 74. page© 2015 VoltDB QUESTIONS? • Use the chat window to type in your questions • Try VoltDB yourself • www.voltdb.com/Download • Download our new ‘recipes’ eBook • https://voltdb.com/ebook/fast-data-recipes • Email us your questions: [email protected] [email protected]