How jKool Analyzes Streaming Data in Real Time with DataStax
Download as PPTX, PDF2 likes1,433 views
In this webinar, Charles Rich, VP of Product Management at jKool will share their journey with DataStax; how jKool knew from the start that traditional relational databases wouldn’t work for the scalability and availability demands of time-series data, and why they turned to DataStax Enterprise for blazing performance and powerful enterprise search and analytics capabilities.
Ad
More Related Content
What's hot (20)
Viewers also liked (8)
Ad
Similar to How jKool Analyzes Streaming Data in Real Time with DataStax (20)
Ad
More from DataStax (20)
Recently uploaded (20)
How jKool Analyzes Streaming Data in Real Time with DataStax
- 1. How jKool Analyzes Streaming Data in Real Time with DataStax Charles Rich VP of Product Management jKool – jKoolcloud.com Thank you for joining. We will begin shortly.
- 2. All attendees placed on mute Input questions at any time using the online interface Webinar Housekeeping
- 3. © 2015 jKool, All Rights Reserved. 3 Agenda • jKool Overview • jKool Technology • Challenges • Why We Selected Cassandra and DataStax • Demo
- 4. jKool Overview © 2015 jKool, All Rights Reserved. 4 • jKool – Founded 2014 as an spin-off from Nastel Technologies – Expertize in building scalable real-time analytics • Initial Vision – Address the big data problems we saw at customers • Inability to analyze data fast enough to take action and address problems • Too much data – Too little time – Provide real-time, in-memory analytics (our heritage) – Leverage open-source – SaaS (or on-premises) – Simplicity
- 5. © © 2015 jKool, All Rights Reserved. 5 What is jKool? A solution to Find and Fix Problems Faster (operational intelligence) DevOps can use jKool to get real-time diagnostics for entire applications: logs, metrics and transactions. – Detect anomalies, 2-clicks to root-cause – Discover log, transaction topologies – Analyze app behavior – Diagnose and determine causality • An alternative to Splunk or Elasticsearch – Fraction of the cost of Splunk – Much easier to use than Elasticsearch
- 6. © 2015 jKool, All Rights Reserved. 6 Business Value: Instant Insight Provide high quality app experiences for customers - Improve customer satisfaction Enable DevOps to: – Fix problems faster • Faster problem resolution, eliminate false alarms – Deliver releases sooner • Less time patching and more time innovating – Be proactive • Spot trends and prevent problems
- 7. © 2015 jKool, All Rights Reserved 7 Features • Web-based, mobile-friendly dashboard – Designed for simplicity and power • Real-time & historical visualization – Flexible, user configurable • Analytics immediately detect outliers – Aggregation, summarization, comparison, including: count, min, max, avg., bucketing, filtering and Bollinger • Ease of use – Talk to your data using English-like query language • Scale to handle the largest volumes of data – NoSQL architecture provides elastic scalability
- 8. © 2015 jKool, All Rights Reserved. 8 jKool Does Machine Data • Sequence, Order, Group, Store • Relationships • Compute Timing • Summarization, comparisons • Triggers based on continuous queries (CEP) – Subscribe to events min elapsedtime, avg elapsedtime, max elapsedtime where eventname="Buy" show as linechart
- 9. © 2015 jKool, All Rights Reserved 9 Real-time, In-Memory Analytics jKool Analyzes Time-Series Data
- 10. Technology • Elastic Architecture – Linear scalability – Highly extensible – Fast, in-memory analysis • Open Source – NoSQL DB, tools and instrumentation – No schema to maintain • FatPipes – Micro-services for ultimate flexibility, change and configuration © 2015 jKool, All Rights Reserved. 10 RESTful
- 11. © 2015 jKool, All Rights Reserved. 11 Key to Real-time Analytics • Process streams as they come while at the same time avoiding IO – Streams are split into real-time queue and persistence queue with eventual consistency • Both have to be processed in parallel – Writing to persistence layer and then analyzing will not achieve near real-time processing
- 12. © 2015 jKool, All Rights Reserved 12 Why clustered computing platforms? • STORM paired with Kafka/JMS and CEP – Clustered way to process incoming real-time streams • STORM handles clustering/distribution • Kafka/JMS for a messaging between grids – Split streaming workload across the cluster – Achieve linear scalability for incoming real-time streams • Apache Spark (alternative to MapReduce) – For distributing queries and trend analysis – Micro batching for historical analytics – Loading large dataset into memory (across different nodes) – Running queries against large data-sets
- 13. Web Interface: DevOps Application Owner 13© 2015 jKool, All Rights Reserved
- 14. © 2015 jKool, All Rights Reserved. 14 Challenges: Meeting our Objectives • Store everything, analyze everything… • Combined real-time & historical analytics • Fast response, flexible query capabilities – Target – for business user – Insulate us from underlying software – Hide complexity • Scale for ingesting data-in-motion • Scale for storing data-at-rest • Elasticity & Operational efficiency • Ease of monitoring & management
- 15. © 2015 jKool, All Rights Reserved 15 Challenges: What we experienced • So many technology options (…so little time…) – Deciding on the right combination is key early on • Cassandra/Solr deployment — (it was a learning experience for us) – Lots of configuration, memory management, replication options • Monitoring, managing clusters – Cassandra/Solr, STORM, Zookeeper, Messaging – +Leverage parent company’s AutoPilot Technology • Achieving near real-time analytics proved extremely challenging – but we did it! – Keeping track of latencies across cluster – Estimating computational capacity required to crunch incoming streams
- 16. © 2015 jKool, All Rights Reserved 16 Challenges: DB was the bottleneck • Needed high performance DB platform • SQL (Oracle, MySQL, etc.) – No scale. We have had a lot of experience our customer’s issues with this at our parent company Nastel… – RAM was “the” bottleneck. Commits take too long and while that is happening everything else stops • NoSQL – Cassandra/Solr (DSE) – Hadoop/MapReduce – MongoDB • Clustered Computing Platforms – STORM – MapReduce – Spark (we learned about this while building jKool)
- 17. Why we chose Cassandra/Solr? • Pros: – Simple to setup & scale for clustered deployments – Scalable, resilient, fault-tolerant (easy replication) – Ability to have data automatically expire (TTL – necessary for our pricing model) – Configurable replication strategy – Great for heavy write workloads • Write performance was better than Hadoop. • Insert rate was of paramount importance for us – get data in as fast as possible was our goal • Java driver balances the load amongst the nodes in a cluster for us (master-slave would never have worked for us) – Solr provides a way to index all incoming data - essential – DSE provides a nice integration between Cassandra and Solr • Cons: – Susceptible to GC pauses (memory management) • The more memory the more GC pauses • Less memory and more nodes seems a better approach than one big “honking” server (we see 6-8GB optimal, so far) – Data compaction tasks may hang © © 2015 jKool, All Rights Reserved 17
- 18. © 2015 jKool, All Rights Reserved 18 Why not Hadoop MapReduce? • MapReduce too slow for real-time workloads – Ok for batch, not so great for real-time – Need to be paired with other technologies for query (Hive/Pig) – Complex to setup, run and operate • Our goals were simplicity first… • Opted for STORM/Spark wrapped with our own micro services platform FatPipes instead of the Map Reduce functionality
- 19. © 2015 jKool, All Rights Reserved 19 Why we chose Cassandra/Solr vs. Mongo? • Why not Mongo? – Global write-lock performance concerns… • Cassandra/Solr – Java based (our project was in Java) – Easy to scale, replicate data, – Flexible write & write consistency levels (ALL, QUORUM, ANY, etc.) – Did we say Java? Yes.(we like Java…) • Flexible choice of platform coverage – Great for time-series data streams (market focus for jKool) • Inherent query limitations in Cassandra solved via Solr integration (provided with DSE – as mentioned earlier)
- 20. © 2015 jKool, All Rights Reserved 20 What we learned • Consider your application – Read heavy or write heavy? Both? • Evaluate performance of course, but consider the user – We needed simplicity: setup and scale (us and end user) – We needed reliability – not planning on targeting data engineers – We needed auto pruning (TTL) – We needed easy search • DSE had this…the others did not provide all of this – We choose DSE.
- 21. © 2015 jKool, All Rights Reserved 21 jKool in Real Time – A Live Demo
- 22. Thank you! Input questions at any time using the online interface More information on jKool at: jKoolCloud.com
Editor's Notes
- #2: Choices we had to make and the architectural decisions to build a system for both real-time and historical…
- #5: For Java applications, initially with RESTful for any apps Open source collectors Log4J, SLF4J, Logback, JMX, HTTP Spark RESTful API… More coming…
- #6: Real-time, in-memory analytics Operational Intelligence for machine data Analyze & Visualize: Logs & Metrics & Transactions Gain insight, root cause, understand application behavior Reduce MTTR (mean-time-to-problem-resolution) Leverage NoSQL and Open source Deliver Operational Intelligence for machine data Analyze your logs & metrics in real-time (& historical) Spot patterns, trends, behavior SaaS or On-Premise Built ground up on Big data analytics platforms NoSQL, STORM, Spark, Kafka Light weight, simple, open source instrumentation Improved cost/benefit
- #7: Keep developers developing and enable App support to analyze app behavior, determine causality and resolve is Reduce time associated with manually analyzing logs Improve productivity of your DevOps, Application teams Keep developers coding…enable app support Benefits: - Fix faster Release sooner Be proactive For the Business: Focus your time on what matters to your business issues Quickly identify risks and opportunities Learn what’s important – what you didn’t know… Exploit hidden & perishable insights Turn machine data into insight Detect preventable losses… if you knew, you could act now… Know your application and how it is used Just deployed a new feature? Are people using it? Was it worth the cost?
- #9: Relationships Splitting & morphing Causality Tree, topology Compute Timing Elapsed Time (ev1..evN) Summarization, comparisons High/low bands, outliers, counts (max, min, avg) Interval Bucketing (second, min, hour) Compare Ev1..EvN
- #12: Real-time means analyzing before data is persisted… We created FatPipes to manage this around STORM/Spark with message infrastructure Kafka/JMS Process data but don’t wait till after a write – no disk IO, split, analyze 2 parallel architectures to handle historical and one for real-time (eventually… both real-time and historical must reconcile) User interacts with Real-time via JKQL (jKool Query Language) English like query language for analyzing data in motion and at rest. “Subscribe” verb for real-time updates
- #13: Clustered computing was selected to scale with the demands of the workload STORM – distribution of CEP (also helpful for distributing data to specific tasks, conditionally) JMS/Kafka for distributing data amongst nodes in our real-time grids CEP for processing streams and publishing results to clients via JMS/Kafka Spark jobs will crunch the data and then write back to Cassandra Created our own micro-services architecture (FatPipes) which runs on top of: STORM/JMS/Kafka STORM – distributes the CEP (also helpful for distributing data to specific tasks, conditionally) JMS/Kafka for distributing data amongst nodes in our real-time grids CEP for processing streams and publishing results to clients via JMS/Kafka FatPipes can be embedded or distributed Real-time Grid Feeds tracking data and real-time queries to CEP and back
- #15: Customers from our experience didn’t know what they needed to store until they actually need it…but then it is to late…so hence, store everything… Historical requirements for architecture very different thank real-time to deliver with fast response time and to provide user defined KPIs Scale must there for interaction as it comes in – not how many TB’s you can analyze but fast can you go and keep up with data streams Can’t build everything, so to accelerate time to market, how much open-source could we leverage? For Elasticity we can add nodes horizontally.
- #16: Can’t test all possibilities and then select…not agile…not enough time. Long term analytics needs different than real-time and weed out what would slow down real-time Providing this as a service, estimating capacity also a challenge.
- #18: We are on DSE 4.6.5 and going shortly to 4.7 (today is: 10.13.15 …) We tried using CQL CQL (Cassandra Query Language) Ad-hoc query would be very hard ad CQL query capabilities are very limited. We would need to define all the tables and indexes for every possible query permutation and the user would need to know the event_id. – too much to be usable Too slow. We only use CQL for admin tasks Lucene addresses the above problem, but adds its own issues. We started with Lucene and did inline inserts and the time to index was too long For each Cassandra insert ,we had to write a Lucene doc…since there is no rewrite, we had to read, delete and then write – a series of batch ops and too slow for our real-time goals Solr helped with this – we write to Cassandra and Solr handles the indexed (automagically) for us Solr is a Web app on top of Lucene We do use Solr indexes jKQL does invoke Solr queries. But we needed to enhance this as we are a multi-tenancy solution and pass it our repository_id to ensure we get the data appropriate to that tenant. We use 3 nodes in a Cassandra Cluster and data ingested is replicated to Solr clusters with 3 nodes (they have both Cassandra and Solr) Data-at-rest – we can ingest as fast as Cassandra can handle it using eventual availability. The data is distributed across Cassandra and Solr. We use DSJava driver. Data is written to coordinator node and he handles the distribution to other nodes. Quorum means 1/2 + 1. You would say that "we use consistency level "quorum" for queries", which means half +1 of the replicas must respond. Like if you were taking a vote and in order for the vote to be valid, you need a quorum of members to be present. Has the same meaning here. If your replication factor is 3, you need 2 of the 3 nodes to respond. 1/2 + 1 using integer division, so half of 3, using integer division is 1 (1.5 truncated) + 1 = 2 We use consistency level #1 for writes For reads we use quorum (admin tasks) All other reads use Solr - the jKQL queries you see on dashboard are all coming from Solr.
- #19: STORM for ingesting Spark for processing data (compute framework)
- #21: Simple to setup & scale for clustered deployments Scalable, resilient, fault-tolerant (easy replication) Ability to have data automatically expire (TTL – necessary for our pricing model) Configurable replication strategy Great for heavy write workloads Write performance was better than Hadoop. Insert rate was of paramount importance for us – get data in as fast as possible was our goal Java driver balances the load amongst the nodes in a cluster for us (master-slave would never have worked for us) Solr provides a way to index all incoming data - essential DSE provides a nice integration between Cassandra and Solr