Feeding Cassandra with Spark-Streaming and Kafka
5 likes2,682 views
The document discusses integrating Cassandra with Spark Streaming and Kafka, highlighting the speaker's background and experience as a solutions engineer at Datastax. It emphasizes the capabilities of Kafka as a robust messaging system and outlines the steps for producing and consuming messages through Kafka. The presentation also covers storing data in Cassandra, processing with Spark, and future steps for improving analytics workflows.
Feeding Cassandra with Spark-Streaming and Kafka
- 1. Feeding Cassandra with Spark Streaming & Kafka Cary Bourgeois Solutions Engineer DataStax, Central Region
- 2. Who Am I • Datastax < 2 Years • Not a “developer” • Legacy BI/Database • Business Objects • SAP • Demo Development • R • Java (If I have to) • Scala (Someday) 2
- 3. 3 Cassandra Summit 2015 September 22-24, Santa Clara Convention Center 7,000 Attendees
- 4. Last Week - Mission Impossible? A Stretch but possible. 4
- 5. Sunday Afternoon - I’m getting my A$$ kicked 5
- 6. Monday Afternoon - Arghhhhh! 6
- 7. Monday Night - I got this! 7
- 9. Why Mess with Success? • Spark 1.3+ • New/Improved Kafka Support • Dataframes • Datastax Enterprise 4.8 • Spark 1.4 support 9 https://databricks.com/blog/2015/03/30/improvements-to-kafka-integration-of-spark-streaming.html
- 10. Why Mess with Success? • Spark 1.3+ • New/Improved Kafka Support • Dataframes • Datastax Enterprise 4.8 • Spark 1.4 support 10https://databricks.com/blog/2015/02/17/introducing-dataframes-in-spark-for-large-scale-data-science.html
- 11. Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. Fast A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. Scalable Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers Durable Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact. Distributed by Design Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees. 11
- 12. • Producers • Consumers • Persistence • Topics • Partitions • Replication 12 http://kafka.apache.org/documentation.html
- 13. • Create a Kafka topic bin/kafka-topics.sh --zookeeper localhost:2181 --create --replication-factor 1 --partitions 1 --topic stream_ts • List all topics bin/kafka-topics.sh --zookeeper localhost:2181 --list • Monitor a topic bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic stream_ts --from-beginning 13
- 14. Confidential Kafka and the Producer 14
- 15. The Producer App • Lots of Options • I chose • Scala • Not steep enough • Akka • Producing this message 15 Edge 1;1;401843;2015-11-04 06:23:49.001;64.44286233060423;82.79653847181152
- 16. Destination - Cassandra Tables 16 CREATE TABLE demo.data ( edge_id text, sensor text, epoch_hr text, ts timestamp, depth double, value double, PRIMARY KEY (( edge_id, sensor, epoch_hr ), ts) ) CREATE TABLE demo.last ( edge_id text, sensor text, ts timestamp, depth double, value double, PRIMARY KEY (( edge_id, sensor )) ) CREATE TABLE demo.count ( pk int, ts timestamp, count bigint, count_ma double, PRIMARY KEY (pk, ts) )
- 17. DSE Analytics => Spark • No ETL • Spark 1.4.1 certification • Simplified map and reduce • Very developer Friendly • SparkSQL • Spark Streaming • Machine Learning • DSE Analytics and Search Integration • Cassandra benefits (scaling, availability) 17 “I want to do processing on data before it hits Cassandra.” “I need my sums, avgs, group by’s ETC.” “I want to run real-time analytics on my Cassandra data.”
- 18. Processing the Stream • Simple Scala Job • Deal with the raw flow • Capture the raw data • Capture the latest sensor reading • Summarize and Analyze • Windowing the Stream • Count Records every x seconds • Calculate a moving average of every x seconds over a number of periods. 18
- 20. Next Steps • SparkR • MLLib workflows • Notebooks • Spark • Jupyter 20
- 21. If you would like the code: 21 https://github.com/CaryBourgeois/KafkaSparkCassandraDemo