![43
Spark Streaming Example
Questions? tiny.cloudera.com/streaming-singapore-questions
1. val conf = new SparkConf().setMaster("local[2]”)
2. val sc = new SparkContext(conf)
3. val lines = sc.textFile(path, 2)
4. val words = lines.flatMap(_.split(" "))
5. val pairs = words.map(word => (word, 1))
6. val wordCounts = pairs.reduceByKey(_ + _)
7. wordCounts.print()](https://image.slidesharecdn.com/streamingarchitecturepatternsv4-151130093924-lva1-app6891/85/Architectural-Patterns-for-Streaming-Applications-43-320.jpg)
![44
Spark Streaming Example
Questions? tiny.cloudera.com/streaming-singapore-questions
1. val conf = new SparkConf().setMaster("local[2]”)
2. val ssc = new StreamingContext(conf, Seconds(1))
3. val lines = ssc.socketTextStream("localhost", 9999)
4. val words = lines.flatMap(_.split(" "))
5. val pairs = words.map(word => (word, 1))
6. val wordCounts = pairs.reduceByKey(_ + _)
7. wordCounts.print()
8. SSC.start()](https://image.slidesharecdn.com/streamingarchitecturepatternsv4-151130093924-lva1-app6891/85/Architectural-Patterns-for-Streaming-Applications-44-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Architectural Patterns for Streaming Applications (20)
Ad
More from hadooparchbook (10)
Ad
Recently uploaded (20)
Architectural Patterns for Streaming Applications
- 1. Architectural Patterns for Streaming applications Strata+Hadoop World, Singapore – December 02, 2015 tiny.cloudera.com/streaming-singapore tiny.cloudera.com/streaming-singapore-questions Mark Grover | @mark_grover Ted Malaska | @TedMalaska
- 2. 2 About the book • @hadooparchbook • hadooparchitecturebook.com • github.com/hadooparchitecturebook • slideshare.com/hadooparchbook Questions? tiny.cloudera.com/streaming-singapore-questions
- 3. 3 About the presenters • Principal Solutions Architect at Cloudera • Done Hadoop for 6 years – Worked with > 70 companies in 8 countries • Previously, lead architect at FINRA • Contributor to Apache Hadoop, HBase, Flume, Avro, Pig and Spark • Contributor to Apache Hadoop, HBase, Flume, Avro, Pig and Spark • Marvel fan boy, runner • Software Engineer at Cloudera, working on Spark • Committer on Apache Bigtop, PMC member on Apache Sentry (incubating) • Contributor to Apache Hadoop, Spark, Hive, Sqoop, Pig and Flume Questions? tiny.cloudera.com/streaming-singapore-questions Ted Malaska Mark Grover
- 4. 4 Goal
- 5. 5 Understand common use- cases for streaming and their architectures
- 7. 7 When to stream, and when not to • We are looking for a SLA sweet spot • Multi milliseconds to seconds • Not minutes • Not constant low milliseconds or under • Doesn’t come for free Questions? tiny.cloudera.com/streaming-singapore-questions
- 9. 9 Use-case categories • Ingestion – Transformation – Decision (e.g. Anomaly detection) • Simple counts – Lambda, etc. • Advanced usage – Machine Learning – Windowing Questions? tiny.cloudera.com/streaming-singapore-questions
- 10. 10 Ingestion
- 11. 11 What is ingestion? Questions? tiny.cloudera.com/streaming-singapore-questions IngestSource Systems Destination system
- 12. 12 But there multiple sources Questions? tiny.cloudera.com/streaming-singapore-questions Source System 1 Destination systemSource System 2 Source System 3 Ingest
- 13. 13 But.. • Sources, sinks, ingestion channels may go down • Sources and sinks may be producing/consuming at different rates • Regular maintenance windows may need to be scheduled • We need a resilient message broker Questions? tiny.cloudera.com/streaming-singapore-questions
- 14. 14 Need for a message broker Questions? tiny.cloudera.com/streaming-singapore-questions Source System 1 Destination systemSource System 2 Source System 3 Ingest Extract Message broker
- 15. 15 Kafka Questions? tiny.cloudera.com/streaming-singapore-questions Source System 1 Destination systemSource System 2 Source System 3 Ingest Extract Message broker
- 16. 16 But ‘queue’ doesn’t ‘push’ Questions? tiny.cloudera.com/streaming-singapore-questions Source System 1 Storage systemSource System 2 Source System 3 Ingest Extract Streaming ingestion process Push Message broker
- 17. 17 Streaming data ingestion process Questions? tiny.cloudera.com/streaming-singapore-questions Source System 1 Storage systemSource System 2 Source System 3 Ingest Extract Streaming ingestion process Push Kafka Connect Apache Flume Message broker
- 18. 18 Streaming architecture for ingestion Questions? tiny.cloudera.com/streaming-singapore-questions Source System 1 Storage systemSource System 2 Source System 3 Ingest Extract Streaming ingestion process Push Kafka Connect Apache Flume Message broker
- 20. 20 Streaming architecture for ingestion Questions? tiny.cloudera.com/streaming-singapore-questions Source System 1 Storage systemSource System 2 Source System 3 Ingest Extract Streaming ingestion process Push Kafka connect Apache Flume Message broker
- 21. 21 Streaming architecture for ingestion Questions? tiny.cloudera.com/streaming-singapore-questions Source System 1 Storage systemSource System 2 Source System 3 Ingest Extract Streaming ingestion process Push Kafka connect Apache Flume Message broker Can be used to do simple transformations
- 22. 22 Two types of transformations Atomic • Need to work with one event at a time • Example – mask a credit card number With context • Need to refer to external context • Example - convert zip code to state, by looking up a cache Questions? tiny.cloudera.com/streaming-singapore-questions
- 23. 23 Atomic transformations • Require no context • Can be simply done within Flume interceptors, Kafka connect or Spark streaming Questions? tiny.cloudera.com/streaming-singapore-questions
- 24. 24 Flume Interceptors • Mask fields • Validate information against external source • Extract fields • Modify data format • Filter or split events Questions? tiny.cloudera.com/streaming-singapore-questions
- 25. 25 Streaming architecture for ingestion Questions? tiny.cloudera.com/streaming-singapore-questions Source System 1 Storage systemSource System 2 Source System 3 Ingest Extract Streaming ingestion process Push Kafka connect Apache Flume Message broker Can be used to do simple transformations
- 26. 26 Transformations with context Questions? tiny.cloudera.com/streaming-singapore-questions
- 27. 27 Exactly once, at least once, at most once (In the context of data ingestion)
- 28. 28 Streaming architecture for ingestion Questions? tiny.cloudera.com/streaming-singapore-questions Source System 1 Storage systemSource System 2 Source System 3 Ingest Extract Streaming ingestion process Push Copycat Apache Flume Message broker Can be used to do simple transformations
- 29. 29 Semantic types • At most once – Not good for many cases – Only where performance/SLA is more important than accuracy • Exactly once – Expensive to achieve but desirable • At least once – Easiest to achieve Questions? tiny.cloudera.com/streaming-singapore-questions
- 30. 30 Categories of storage systems “Puts” based • Can be re-inserted without side effects since re-inserted record will have duplicate key “Appends” based • Can not be re-inserted Questions? tiny.cloudera.com/streaming-singapore-questions
- 31. 31 How to achieve exactly once? • For “puts” based storage systems – At least once is enough (keys have to be unique though i.e. primary key) – Re-inserted records will have duplicate keys – Will simply overwrite the exist record with the same value • For “appends” based storage systems (e.g. HDFS) – Still easiest to do at least once – Need to de-duplicate before processing Questions? tiny.cloudera.com/streaming-singapore-questions
- 33. 33Questions? tiny.cloudera.com/streaming-singapore-questions Hadoop Cluster II Storage Batch Processing Hadoop Cluster I Flume (Sink) HBase and/or Memory Store HDFS HBase Impala Map/Reduce Spark Automated & Manual Analytical Adjustments and Pattern detection Fetching & Updating Profiles/Rules Batch Time Adjustments NRT/Stream Processing Spark Streaming Adjusting NRT stats Kafka Events Reporting Flume (Source) Interceptor(Rules) Flume (Source) Flume (Source) Interceptor (Rules) Kafka Alerts/Events Flume Channel Events Alerts Hadoop Cluster I HBase and/or Memory Store
- 34. 34 Counting
- 35. 35 Streaming and Counting • Counting is easy right? • Back to Only once Questions? tiny.cloudera.com/streaming-singapore-questions
- 36. 36 We started with Lambda Pipe Speed Layer Batch Layer Persist Results Speed Results Batch Results Serving Layer Questions? tiny.cloudera.com/streaming-singapore-questions
- 37. 37 Why did Streaming Suck • Increments with Cassandra • Double increment • No strong consistency • Storm with out Kafka • Not only once • Not at least once • Batch would have to re-process EVERY record to remove dups Questions? tiny.cloudera.com/streaming-singapore-questions
- 38. 38 We have come a long way • We don’t have to use Increments any more and we can have consistency • HBase • We can have state in our streaming platform • Spark Streaming • We don’t lose data • Spark Streaming • Kafka • Other options • Full universe of Deduping • Again HBase with versions Questions? tiny.cloudera.com/streaming-singapore-questions
- 41. 41 Advanced Streaming • Ad-hoc will produce Identify Value • Ad-hoc will become batch • The value will demand less latency on batch • Batch will become Streaming Questions? tiny.cloudera.com/streaming-singapore-questions
- 42. 42 Advanced Streaming • Requirements for Ideal Batch to Streaming frameworks • Something that can snap both paradigms • Something that can use the tools of Ad-hoc • SQL • MlLib • R • Scala • Java • Development through a common IDE • Debugging • Unit Testing • Common deployment model Questions? tiny.cloudera.com/streaming-singapore-questions
- 43. 43 Spark Streaming Example Questions? tiny.cloudera.com/streaming-singapore-questions 1. val conf = new SparkConf().setMaster("local[2]”) 2. val sc = new SparkContext(conf) 3. val lines = sc.textFile(path, 2) 4. val words = lines.flatMap(_.split(" ")) 5. val pairs = words.map(word => (word, 1)) 6. val wordCounts = pairs.reduceByKey(_ + _) 7. wordCounts.print()
- 44. 44 Spark Streaming Example Questions? tiny.cloudera.com/streaming-singapore-questions 1. val conf = new SparkConf().setMaster("local[2]”) 2. val ssc = new StreamingContext(conf, Seconds(1)) 3. val lines = ssc.socketTextStream("localhost", 9999) 4. val words = lines.flatMap(_.split(" ")) 5. val pairs = words.map(word => (word, 1)) 6. val wordCounts = pairs.reduceByKey(_ + _) 7. wordCounts.print() 8. SSC.start()
- 46. 46 Advanced Streaming • In Spark Streaming • A DStream is a collection of RDD with respect to micro batch intervals • If we can access RDDs in Spark Streaming • We can convert to Vectors • KMeans • Principal component analysis • We can convert to LabeledPoint • NaiveBayes • Random Forest • Linear Support Vector Machines • We can convert to a DataFrames • SQL • R Questions? tiny.cloudera.com/streaming-singapore-questions
- 47. 47 Wrap-up
- 48. 48 Understand common use-cases for streaming and their architectures Our original goal
- 49. 49 Common streaming use-cases • Ingestion – Transformation – Decision (e.g. Anomaly detection) • Simple counts – Lambda, etc. • Advanced usage – Machine Learning – Windowing Questions? tiny.cloudera.com/streaming-singapore-questions
- 50. 50 Free books! • Book signings – Wednesday (today), 5:30 PM at O’Reilly booth – Thursday (tomorrow), 3:15 PM at Cloudera booth • Please leave us a review! Questions? tiny.cloudera.com/streaming-singapore-questions
- 51. 51 Stay in touch! Mark Grover | @mark_grover Ted Malaska | @TedMalaska @hadooparchbook tiny.cloudera.com/streaming-singapore tiny.cloudera.com/streaming-singapore-questions hadooparchitecturebook.com