Big Data LDN 2017: Processing Fast Data With Apache Spark: the Tale of Two APIs

1. Processing Fast Data with Apache Spark: The Tale of Two APIs Big Data Lnd Gerard Maas Señor SW Engineer 16/Nov/2017

2. Gerard Maas Señor SW Engineer Computer Engineer Scala Programmer Early Spark Adopter (v0.9) Spark Notebook Contributor Cassandra MVP (2015, 2016) Stack Overflow Top Contributor (Spark, Spark Streaming, Scala) Wannabe { IoT Maker Drone crasher/tinkerer } @maasg https://github.com/maasg https://www.linkedin.com/ in/gerardmaas/ https://stackoverflow.com /users/764040/maasg

4. Streams Everywhere

7. Once upon a time...

8. Apache Spark Core Spark SQL SparkMLLib SparkStreaming Structured Streaming DataFrames DataSets GraphFrames Data Sources

9. Apache Spark Core Spark SQL SparkMLLib SparkStreaming Structured Streaming DataFrames DataSets GraphFrames Data Sources

10. Structured Streaming

11. Structured Streaming Kafka Sockets HDFS/S3 Custom Streaming DataFrame

12. Structured Streaming Kafka Sockets HDFS/S3 Custom Streaming DataFrame

13. Structured Streaming Kafka Sockets HDFS/S3 Custom Streaming DataFrame Query Kafka Files foreachSink console memory Output Mode

14. val rawData = sparkSession.readStream .format("kafka") .option("kafka.bootstrap.servers", kafkaBootstrapServer) .option("subscribe", topic) .option("group.id", "iot-data-consumer") .option("startingOffsets", "earliest") .load() case class SensorData(sensorId: Int, timestamp: Long, reading: Double) def parse(str: String): SensorData = ... val parsedData = rawData.select($"value").as[String].flatMap(record => parse(record)) val validData = parsedData.where($"reading" > 0) val query = validData.writeStream .outputMode("append") .format("parquet") .option("path", "/dest/sensor-data") .option("checkpointLocation", "/tmp/checkpoint") .start()

21. val rawData = sparkSession.readStream .format("kafka") .option("kafka.bootstrap.servers", kafkaBootstrapServer) .option("subscribe", topic) .option("group.id", "iot-data-consumer") .option("startingOffsets", "earliest") .load() case class SensorData(sensorId: Int, timestamp: Long, value: Double) def parse(str: String): SensorData = ... val parsedData = rawData.select($"value").as[String].flatMap(record => parse(record)) val validData = parsedData.where($"value" > 0) val query = validData.writeStream .outputMode("append") .format("parquet") .option("path", "/dest/sensor-data") .option("checkpointLocation", "/tmp/checkpoint") .start()

22. Structured Streaming - Usecases ● ETL ● Stream aggregations, windows ● Event-time oriented analytics ● Join Streams with Fixed Datasets ● Apply Machine Learning Models

24. Spark Streaming Kafka Flume Kinesis Twitter Sockets HDFS/S3 Custom Apache Spark SparkSQL SparkML ... Databases HDFS API Server Streams

26. DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1

27. DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1 RDD[U] RDD[U] RDD[U] RDD[U] RDD[U] Transformation T -> U

28. DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1 RDD[U] RDD[U] RDD[U] RDD[U] RDD[U] Transformation T -> U Actions

29. API: Transformations map, flatmap, filter count, reduce, countByValue, reduceByKey n union, join cogroup

30. API: Transformations mapWithState … …

31. API: Transformations transform val iotDstream = MQTTUtils.createStream(...) val devicePriority = sparkContext.cassandraTable(...) val prioritizedDStream = iotDstream.transform{rdd => rdd.map(d => (d.id, d)).join(devicePriority) }

32. Actions print ------------------------------------------- Time: 1459875469000 ms ------------------------------------------- data1 data2 saveAsTextFiles, saveAsObjectFiles, saveAsHadoopFiles xxx yyy zzz foreachRDD *

33. Actions print ------------------------------------------- Time: 1459875469000 ms ------------------------------------------- data1 data2 saveAsTextFiles, saveAsObjectFiles, saveAsHadoopFiles xxx yyy zzz foreachRDD * Spark SQL Dataframes GraphFrames Any API

34. Structured Streaming - Usecases ● Stream-stream joins ● Complex state management (local + cluster state) ● Streaming Machine Learning ○ Learn ○ Score ● Join Streams with Updatable Datasets ● Event-time oriented analytics ● Continuous processing

35. Structured Streaming +

36. Spark Streaming + Structured Streaming 36 val parse: Dataset[String] => Dataset[Record] = ??? val process: Dataset[Record] => Dataset[Result] = ??? val serialize: Dataset[Result] => Dataset[String] = ??? val kafkaStream = spark.readStream… val f = parse andThen process andThen serialize val result = f(kafkaStream) result.writeStream .format("kafka") .option("kafka.bootstrap.servers",bootstrapServers) .option("topic", writeTopic) .option("checkpointLocation", checkpointLocation) .start() val dstream = KafkaUtils.createDirectStream(...) dstream.map{rdd => val ds = sparkSession.createDataset(rdd) val f = parse andThen process andThen serialize val result = f(ds) result.write.format("kafka") .option("kafka.bootstrap.servers", bootstrapServers) .option("topic", writeTopic) .option("checkpointLocation", checkpointLocation) .save() } Structured StreamingSpark Streaming

37. Streaming Pipelines Structured Streaming Keyword Extraction Keyword Relevance Similarity DB Storage

39. lightbend.com/fast-data-platform

40. Features 1. One-click component installations 2. Automatic dependency checks 3. One-click access to install logs 4. Real-time cluster visualization 5. Access to consolidated production logs Benefits: 1. Easy to get started 2. Ready access to all components 3. Increased developer velocity Fast Data Platform Manager, for Managing Running Clusters

41. Fast Data Platform Visit us at the Booth!

42. Thank You Gerard Maas @maasg

Big Data LDN 2017: Processing Fast Data With Apache Spark: the Tale of Two APIs

Recommended

More Related Content

What's hot (20)

Similar to Big Data LDN 2017: Processing Fast Data With Apache Spark: the Tale of Two APIs (20)

More from Matt Stubbs (20)

Recently uploaded (20)

Big Data LDN 2017: Processing Fast Data With Apache Spark: the Tale of Two APIs