Spark with Cassandra by Christopher Batey
8 likes3,517 views
This document discusses using Spark with Cassandra for processing time series weather data. It describes storing raw weather data in Cassandra, then using Spark to perform analytics like aggregations that require querying large portions of the data. Spark partitions the data by Cassandra token ranges to allow efficient distributed processing on each node's local data. Daily aggregate tables are also stored in Cassandra and maintained through Spark streaming to enable fast queries on rolled-up metrics.
Ad
More Related Content
What's hot (20)
Similar to Spark with Cassandra by Christopher Batey (20)
Ad
More from Spark Summit (20)
Ad
Recently uploaded (20)
Spark with Cassandra by Christopher Batey
- 2. @chbatey Christopher Batey - @chbatey Freelance Software engineer/Devops/Architect Loves: Breaking software Distributed systems Hates: Fragile software Untested code :( Introduction
- 3. Audience
- 5. Overview Cassandra architecture Modelling time series - weather data for many stations What can be done with pure C* When to introduce Spark
- 6. What do we use Spark for? Batch processing Machine Learning Ad-hoc querying of large datasets Streaming processing
- 7. What do we use Cassandra for? Operational Database OLTP
- 12. @chbatey Consistent hashing jim age: 36 car: ford gender: M carol age: 37 car: bmw gender: F johnny age: 12 gender: M suzy age: 10 gender: F Partition Key Hash value jim 350 carol 998 johnny 50 suzy 600 Partition Key
- 14. Example Node Start range End range Primary key Hash value A 0 249 johnny 50 B 250 499 jim 350 C 500 749 suzy 600 D 750 999 carol 998
- 15. @chbatey Fault tolerance Replicate each price of data on multiple nodes Keep replicas on different racks Datacenter aware
- 16. DC2 client RF3 RF3 C C WRITE CL = 1 We have replication! DC1
- 17. Storing weather data CREATE TABLE raw_weather_data ( weather_station text, year int, month int, day int, hour int, temp double, PRIMARY KEY ((weather_station), year, month, day, hour) ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC);
- 18. @chbatey Primary key relationship PRIMARY KEY ((weatherstation_id),year,month,day,hour)
- 19. Data Locality weatherstation_id=‘10010:99999’ ? 1000 Node Cluster You are here!
- 20. Primary key relationship PRIMARY KEY ((weatherstation_id),year,month,day,hour) Partition Key Clustering Columns 10010:99999
- 21. 2005:12:1:8:temp 2005:12:1:7:temp -5.6 PRIMARY KEY ((weatherstation_id),year,month,day,hour) Partition Key Clustering Columns 10010:99999 -5.1 2005:12:1:10:temp -5.3 2005:12:1:9:temp -4.9 Primary key relationship
- 22. I have a question!! What happens if I want to do an adhoc query??
- 23. I’ve stored the data partitioned by weather id… … now I want a report for all stations
- 24. I’ve stored the raw weather data… … now I want rollups/ aggregates
- 26. Deployment - Spark worker on each of the Cassandra nodes - Partitions made up of LOCAL cassandra data S C S C S C S C
- 27. Cassandra Data is Distributed By Token Range 0 500 Node 1 Node 2 Node 3 Node 4
- 28. Cassandra Data is Distributed By Token Range 0 500 Node 1 Node 2 Node 3 Node 4 Without vnodes
- 29. Cassandra Data is Distributed By Token Range 0 500 Node 1 Node 2 Node 3 Node 4 With vnodes
- 30. Cassandra RDD
- 31. Each Spark partition is made up of token ranges that live on the same node
- 32. Each Spark partition is made up of Cassandra partitions that are on the same node
- 33. Storing weather data CREATE TABLE raw_weather_data ( weather_station text, year int, month int, day int, hour int, temp double, PRIMARY KEY ((weather_station), year, month, day, hour) ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC);
- 34. (count: 24, mean: 14.428150, stdev: 7.092196, max: 28.034969, min: 0.675863) Partition key = Single node
- 35. (count: 11242, mean: 8.921956, stdev: 7.428311, max: 29.997986, min: -2.200000) No partition key = Every node
- 37. daily_aggregate_precip CREATE TABLE daily_aggregate_precip ( weather_station text, year int, month int, day int, precipitation counter, PRIMARY KEY ((weather_station), year, month, day) ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC); SELECT precipitation FROM daily_aggregate_precip WHERE weather_station='010010:99999' AND year=2005 AND month=12 AND day>=1 AND day <= 7;
- 41. Saving the raw data
- 42. Building an aggregate CREATE TABLE daily_aggregate_precip ( weather_station text, year int, month int, day int, precipitation counter, PRIMARY KEY ((weather_station), year, month, day) ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC); CQL Counter
- 44. Conclusion Cassandra = OLTP database for the large scale Spark can be used to do complex queries in a partition Or analytical queries for an entire table Spark streaming to keep tables up to date
- 45. Thanks for listening Questions later? @chbatey