Analyzing Time-Series Data with Apache Spark and Cassandra - StampedeCon 2016
10 likes2,936 views
The document discusses the integration and utilization of Apache Spark with Cassandra for analyzing time-series data, particularly from sensor measurements or financial instruments. It explains Cassandra's architecture and how data is stored and retrieved, alongside Spark's capabilities for data processing, including RDDs and Spark Streaming. Additionally, it highlights practical use cases for weather station data analysis and emphasizes the benefits of this technology stack for handling large-scale, high-frequency data.
![Cassandra architecture is
Shared nothing[1]
Materless peer-to-peer
Shard Free
Based on Amazon Dynamo and Google BigTable
[1] 1986 paper “The Case for Shared Nothing” -http://db.cs.berkeley.edu/papers/hpts85-
nothing.pdf.](https://image.slidesharecdn.com/analyzingtime-seriesdatawithapachesparkandcassandra-stampedecon2016-160805121857/85/Analyzing-Time-Series-Data-with-Apache-Spark-and-Cassandra-StampedeCon-2016-10-320.jpg)
![ Transformations - Similar to scala collections API
• Produce new RDDs
• filter, flatmap, map, distinct, groupBy, union, zip,
reduceByKey, subtract
Actions
• Require materialization of the records to generate
a value
• collect: Array[T], count, fold, reduce..
RDD Operations](https://image.slidesharecdn.com/analyzingtime-seriesdatawithapachesparkandcassandra-stampedecon2016-160805121857/85/Analyzing-Time-Series-Data-with-Apache-Spark-and-Cassandra-StampedeCon-2016-30-320.jpg)
Analyzing Time-Series Data with Apache Spark and Cassandra - StampedeCon 2016
- 1. Analyzing Time-Series Data with Apache Spark and Cassandra Andrew Psaltis HDF / IoT Product Solution Architect @itmdata StampedeCon 2016
- 2. If you every wanted to…. Build models over measurements coming in every second from sensors across the world? Dig into intra-day trading prices of millions of financial instruments? Compare hourly page view statistics across every page on Wikipedia?
- 3. You need to do it over a large sequence of measurements over time.
- 4. A problem perfect for Cassandra and Spark
- 5. Time-series data consists of sequences of measurements, each occurring at a point in time.
- 6. Example: Weather Station Weather station collects data Cassandra stores in sequence Application reads in sequence
- 7. Query use cases Weather Station ID Get weather data given: Weather Station ID and Time Weather Station ID and Range of Time
- 8. Aggregation use cases Weather Station ID Get temperature stats given: Weather Station ID and Time Weather Station ID and Range of Time
- 10. Cassandra architecture is Shared nothing[1] Materless peer-to-peer Shard Free Based on Amazon Dynamo and Google BigTable [1] 1986 paper “The Case for Shared Nothing” -http://db.cs.berkeley.edu/papers/hpts85- nothing.pdf.
- 11. Row
- 12. Partition
- 13. Table
- 14. Keyspace Table 1 Table 2
- 15. Partition Key Clustering Columns Order Override
- 16. Partition Key Clustering Columns 10010:99999 2016:07:28:1 2 -5.6 2016:07:28:1 1 -5.1 2016:07:28:1 0 -4.9 2016:07:28:0 9 -5.3 Primary key relationship
- 17. Tokens Consistent hash between 2-63 and 264 Each node owns a range of those values The token is the beginning of that range to the next node’s token value Virtual Nodes break these down further
- 18. Replication Node Primary 10.0.0.1 00-25 10.0.0.2 26-50 10.0.0.3 51-75 10.0.0.4 76-100 10.0.0.1 00-25 10.0.0.2 26-50 10.0.0.4 76-100 10.0.0.3 51-75 DC 1 DC 1 RF: 1
- 19. Replication Node Primary Replica 10.0.0.1 00-25 76-100 10.0.0.2 26-50 00-25 10.0.0.3 51-75 26-50 10.0.0.4 76-100 51-75 10.0.0.1 00-25 76-100 10.0.0.2 26-50 00-25 10.0.0.4 76-100 51-75 10.0.0.3 51-75 26-50 DC 1 DC 1 RF: 2
- 20. Replication Node Primary Replica Replica 10.0.0. 1 00-25 76-100 51-75 10.0.0. 2 26-50 00-25 76-100 10.0.0. 3 51-75 26-50 00-25 10.0.0.1 00-25 76-100 51-75 10.0.0.2 26-50 00-25 76-100 10.0.0.4 76-100 51-75 26-50 10.0.0.3 51-75 26-50 00-25 DC 1 DC 1 RF: 3
- 21. Replication 10.0.0.1 00-25 76-100 51-75 10.0.0.2 26-50 00-25 76-100 10.0.0.4 76-100 51-75 26-50 10.0.0.3 51-75 26-50 00-25 DC 1 RF: 3 Client Write to partition 15
- 22. Multi-Datacenter 10.0.0.1 00-25 76-100 51-75 10.0.0.2 26-50 00-25 76-100 10.0.0.4 76-100 51-75 26-50 10.0.0.3 51-75 26-50 00-25 DC 1 RF: 3 Client Write to partition 15 10.0.0.1 00-25 76-100 51-75 10.0.0.2 26-50 00-25 76-100 10.0.0.4 76-100 51-75 26-50 10.0.0.3 51-75 26-50 00-25 DC 2 RF: 3
- 23. Query use cases Weather Station ID Get weather data given: Weather Station ID and Time Weather Station ID and Range of Time
- 24. Spark Overview
- 27. What is Spark used for? Fast and general purpose engine for large scale data processing Provides a framework that supports In-Memory Cluster Computing Designed for iterative computations and interactive data mining
- 28. Resilient Distributed Dataset (RDD) • Created through transformations on data (map,filter..) or other RDDs • Immutable • Partitioned • Reusable
- 29. RDD Partitioning Number of RDD partitions will control how many parallel tasks can be run against the data stored in the RDD Hint: in general make it at least as large as the # of cpu cores in your cluster
- 30. Transformations - Similar to scala collections API • Produce new RDDs • filter, flatmap, map, distinct, groupBy, union, zip, reduceByKey, subtract Actions • Require materialization of the records to generate a value • collect: Array[T], count, fold, reduce.. RDD Operations
- 31. Data Locality Spark asks an RDD for a list of its partitions (splits) Each split consists of one or more token-ranges For every partition: •Spark gets a list of preferred nodes to process on from RDD •Spark creates a task and sends it to one of the nodes for execution
- 32. What is Spark Streaming? •Provides efficient, fault-tolerant stateful stream processing •Provides a simple API for implementing complex algorithms •Integrates with Spark’s batch and interactive processing •Integrates with other Spark extensions
- 35. Discretized Streams (DStreams) •The basic abstraction provided by Spark Streaming •Continuous series of RDDs
- 39. Spark on Cassandra • Server-Side filters (where clauses) • Cross-table operations (JOIN, UNION, etc.) • Data locality-aware (speed) • Data transformation, aggregation, etc. • Natural Time Series Integration
- 40. Spark Cassandra Connector • Loads data from Cassandra to Spark • Writes data from Spark to Cassandra • Implicit Type Conversions and Object Mapping • Implemented in Scala (offers a Java API) • Open Source • Exposes Cassandra Tables as Spark RDDs + Spark DStreams
- 43. Locating a Row
- 44. Cassandra RDD Use the Token Range to Create Node Local Spark Partitions
- 45. The Spark Executor uses the Java Driver to Pull Rows from the Local Cassandra Instance
- 47. Batch Weather Station Analysis
- 48. Weather Station Analysis Weather station collects data Cassandra stores in sequence Spark rolls up data into new tables
- 50. Setup Connection
- 51. Get data and aggregate
- 52. Store back into Cassandra
- 54. Aggregation use cases Weather Station ID Get temperature stats given: Weather Station ID and Time Weather Station ID and Range of Time
- 55. Weather Station Stream Analysis Weather station collects data Data processed in stream Cassandra stores in sequence
- 56. Weather Station Stream Analysis Counter
- 58. Thank You
Editor's Notes
- #2: Analyzing Time-Series Data with Apache Spark and Cassandra
- #6: What is Time-Series Data? Time-series data consists of sequences of measurements, each occurring at a point in time. A variety of terms are used to describe time-series data, and many of them apply to conflicting or overlapping concepts. In the interest of clarity, in spark-ts , we stick to a particular vocabulary: A time series is a sequence of floating-point values, each linked to a timestamp.
- #18: Consistent hash between 2-63 and 264 •Each node owns a range of those values •The token is the beginning of that range to the next node’s token value •Virtual Nodes break these down further Each partition is a 128 bit value
- #37: Since we are really just creating discrete RDD’s we will see how we have the opportunity to combine streaming with the rest of the stack