GDG Jakarta Meetup - Streaming Analytics With Apache Beam
0 likes442 views
The document presents an overview of Google Cloud Dataflow, detailing its unified model for stream and batch processing using Apache Beam SDKs. It discusses various components like ETL, data input/output transformations, and windowing concepts essential for stream analytics. Additionally, it covers advanced topics like trigger mechanisms, accumulation modes, and the importance of batch processing for accurate analytics results.
![First trigger firing: [5, 8, 3]
Second trigger firing: [5, 8, 3, 15, 19, 23]
Third trigger firing: [5, 8, 3, 15, 19, 23, 9, 13, 10]
Jakarta
Accumulation Modes](https://image.slidesharecdn.com/gdgjakarta-streaminganalyticswithapachebeam-190210040023/85/GDG-Jakarta-Meetup-Streaming-Analytics-With-Apache-Beam-50-320.jpg)
![First trigger firing: [5, 8, 3]
Second trigger firing: [15, 19, 23]
Third trigger firing: [9, 13, 10]
Jakarta
Discarding Modes](https://image.slidesharecdn.com/gdgjakarta-streaminganalyticswithapachebeam-190210040023/85/GDG-Jakarta-Meetup-Streaming-Analytics-With-Apache-Beam-51-320.jpg)
GDG Jakarta Meetup - Streaming Analytics With Apache Beam
- 1. Imre Nagi Traveloka Data @imrenagi Jakarta Google Cloud Dataflow Unified Model for Stream and Batch processing
- 2. Imre Nagi Ping me @imrenagi Previously: Software Engineer @CERN @eBay Inc Currently: Software Engineer @Traveloka Data Docker Community Leader, Indonesia
- 3. Agenda What is Dataflow? Dataflow Abstraction Dataflow Common Pipeline Stream Analytics
- 5. Apache Beam ... A set of SDK that define programming model that you use to build your stream and batch processing pipeline Cloud Dataflow Fully managed distributed service that runs and optimizes your beam pipeline
- 6. Jakarta Dataflow for .. ● Move ● Filter ● Enrich ETL ● Connecting to Cloud Pub/Sub ● Read and Write to BigQuery, Bigtable, etc. I/O Operation ● Streaming Computing ● Batch Computing ● Machine Learning Analytics
- 7. Jakarta Unified Programming Model Unified: Stream & Batch Pipeline Open Source: Java SDK Python SDK Go SDK (New)
- 8. Jakarta Cloud Pub/Sub Cloud Dataflow (Streaming) Cloud Bigquery Source Processing Data Store Unified Model (Streaming)
- 9. Jakarta Cloud Pub/Sub Cloud Storage Cloud Dataflow (Streaming) Cloud Bigquery Source Processing Data Store Unified Model (Streaming & Backup)
- 10. Jakarta Cloud Storage Cloud Dataflow (Batch) Cloud Bigquery Source Processing Data Store Unified Model (Batch)
- 12. Jakarta Represents graph of data processing transformation PCollection flows through the pipeline Can have multiple I/O in the beginning and end of pipeline Beam Pipeline
- 13. Jakarta
- 14. // Define the pipeline option PipelineOptions options = PipelineOptionsFactory.create(); // Create the pipeline Pipeline p = Pipeline.create(options);
- 15. Jakarta Data Model PCollection<T> is a collection of data type T May be bounded or unbounded in size Element might has implicit or explicit timestamp
- 16. // Create the PCollection 'lines' by applying a 'Read' transform. PCollection<String> lines = p.apply(TextIO.read().from("/path/to/some/inputData.txt")); PCollection<String> linesGCS = p.apply(TextIO.read().from("gs://deeptech/*")); static final List<String> LINES = Arrays.asList( "This is the first line", "You will say this one is the second", "But it's not. "); // Generating PCollection from in memory data PCollection<String> lines = p.apply(Create.of(LINES)).setCoder(StringUtf8Coder.of()) // Generate bounded pcollection PCollection<Long> bounded = p.apply(GenerateSequence.from(0).to(1000)); // Generate unbounded pcollection PCollection<Long> unbounded = p.apply(GenerateSequence.from(0));
- 17. Jakarta PTransform: Transforming the Data
- 18. public class HelloDoFn extends DoFn<String, String> { @ProcessElement public void processElement(ProcessContext context) { String name = context.element(); context.output("Hello, " + name + " ! "); } }
- 19. public class StringToLongDoFn extends DoFn<String, Long> { @ProcessElement public void processElement(ProcessContext context) { String name = context.element(); context.output(name.length()); } }
- 20. PCollection<KV<String, Integer>> scores = input .apply(Sum.integersPerKey());
- 21. Jakarta
- 27. Jakarta Multiple Transformation for a PCollection
- 30. Jakarta data..
- 33. ...maybe infinitely big... 9:008:00 14:0013:0012:0011:0010:00
- 35. Jakarta Lambda Architecture ever says that Stream Processing only CAN’T produce accurate analytics result. Thus, Batch Processing is necessary to fix the inaccuracy of the stream processing.
- 36. Jakarta
- 37. Jakarta 13:00 14:008:00 9:00 10:00 11:00 12:00 Processing Time ∑ ∑ ∑ ∑ ∑ ∑ ∑8:00 8:00 Grouping via Processing-Time Windows
- 38. Jakarta Processing Time 11:0010:00 15:0014:0013:0012:00 Grouping via event-time windows Event Time 11:0010:00 15:0014:0013:0012:00 Input Output ∑ ∑ ∑ ∑ ∑ ∑
- 39. Jakarta What is windowing? Windowing divides data into event-time-based finite chunks. Often required when doing aggregations over unbounded data. Fixed Sliding 1 2 3 54 Key 2 Key 1 Key 3 Time 1 2 3 4 A windowing function computes which window(s) an element belongs to. Temporal functions can be parameterized with duration and frequency.
- 40. Jakarta What about data-dependent windowing? Sessions 2 431 Time Unique per key - you can't know a priori when a session ends, so the windowing function is now also parameterized by state.
- 41. PCollection<KV<String, Integer>> scores = input.apply( Window.into(FixedWindows.of(Duration.standardMinutes(2)))) .apply(Sum.integersPerKey());
- 42. Jakarta
- 43. Jakarta Windowing specifies where events are aggregated in event time, but when are events emitted in processing time?
- 44. Jakarta Trigger Triggers: A trigger is a mechanism for declaring when the output for a window should be materialized relative to some external signal. Triggers provide flexibility in choosing when outputs should be emitted. They also make it possible to observe the output for a window multiple times as it evolves
- 45. Jakarta Windowed summation on a streaming engine with perfect (left) and heuristic (right) watermarks.
- 46. PCollection<KV<String, Integer>> scores = input .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2))) .triggering(AtWatermark() .withEarlyFirings( AtPeriod(Duration.standardMinutes(1))) .withLateFirings(AtCount(1)))) .apply(Sum.integersPerKey());
- 47. Jakarta Windowed summation on a streaming engine with early and late firings.
- 48. PCollection<KV<String, Integer>> scores = input .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2))) .triggering(AtWatermark() .withEarlyFirings( AtPeriod(Duration.standardMinutes(1))) .withLateFirings(AtCount(1)))) .withAllowedLateness(Duration.standardMinutes(1))) .apply(Sum.integersPerKey());
- 49. Jakarta Windowed summation on a streaming engine with early and late firings and allowed lateness
- 50. First trigger firing: [5, 8, 3] Second trigger firing: [5, 8, 3, 15, 19, 23] Third trigger firing: [5, 8, 3, 15, 19, 23, 9, 13, 10] Jakarta Accumulation Modes
- 51. First trigger firing: [5, 8, 3] Second trigger firing: [15, 19, 23] Third trigger firing: [9, 13, 10] Jakarta Discarding Modes
- 52. PCollection<KV<String, Integer>> scores = input .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2))) .triggering(AtWatermark() .withEarlyFirings( AtPeriod(Duration.standardMinutes(1))) .withLateFirings(AtCount(1)))) .discardingFiredPanes()) .apply(Sum.integersPerKey());
- 53. Jakarta Discarding mode version of early/late firings on a streaming engine
- 54. Jakarta 1. http://streamingsystems.org/Presentations/Jelena%20Pjesivac-grbo vic.pdf 2. Stream Analytics with Google Cloud Dataflow: Use Cases & Patterns, Gaurav Anand 3. Streaming 101 & 102, Tyler Akidau 4. https://streamingbook.net 5. Apache Beam Documentation Google Slide version from this slide can be accessed from: https://docs.google.com/presentation/d/1Ws73JxlVH39HiKiYuF3vW 903j8wFzxPQihXz4CQ_HZM/edit?usp=sharing Credits to: