Streaming in the Wild with Apache Flink
Download as PPTX, PDF3 likes2,031 views
This document summarizes a presentation about streaming data processing with Apache Flink. It discusses how Flink enables real-time analysis and continuous applications. Case studies are presented showing how companies like Bouygues Telecom, Zalando, King.com, and Netflix use Flink for applications like monitoring, analytics, and building a stream processing service. Flink performance is discussed through benchmarks, and features like consistent snapshots and dynamic scaling are mentioned.
Ad
More Related Content
What's hot (20)
Viewers also liked (6)
Ad
Similar to Streaming in the Wild with Apache Flink (20)
Ad
More from DataWorks Summit/Hadoop Summit (20)
Recently uploaded (20)
Streaming in the Wild with Apache Flink
- 1. Kostas Tzoumas @kostas_tzoumas Hadoop Summit San Jose June 6, 2016 Streaming in the Wild with Apache FlinkTM
- 2. 2 Streaming technology is enabling the obvious: continuous processing on data that is continuously produced Hint: you are already doing streaming
- 3. Why embrace streaming? Monitor your business and react in real time Implement robust continuous applications Adopt a decentralized architecture Consolidate analytics infrastructure 3
- 4. React in real time 4
- 5. Streaming versus real-time Streaming != Real-time E.g., streaming that is not real time: continuous applications with large windows E.g., real-time that is not streaming: very fast data warehousing queries However: streaming applications can be fast 5 Streaming Real time
- 6. How real-time is Flink? 6 Yahoo! benchmark* data Artisans benchmarks** * https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-computation-engines-at ** http://data-artisans.com/extending-the-yahoo-streaming-benchmark/ and http://data-artisans.com/high-throughput- low-latency-and-exactly-once-stream-processing-with-apache-flink/
- 7. When and why does this matter? Immediate reaction to life • E.g., generate alerts on anomaly/pattern/special event Avoid unnecessary tradeoffs • Even if application is not latency-critical • With Flink you do not pay a price for latency! 7
- 8. Bouygues Telecom – LUX 8 One of the largest telcos in France. System (among others) used for real time diagnostics and alarming. Read more: http://data- artisans.com/flink-at- bouygues-html/
- 10. Continuous application A production data application that needs to be live 24/7 feeding other systems (perhaps customer-facing) Need to be efficient, consistent, correct, and manageable Stream processing is a great way to implement continuous applications robustly 10
- 11. Continuous apps with “batch” 11 file 1 file 2 Job 1 Job 2 time file 3 Job 3 Scheduler Serve&store
- 12. Continuous apps with “lambda” 12 file 1 file 2 Job 1 Job 2 Scheduler Streaming job Serve& store
- 13. Problems with batch and λ Way too many moving parts (and code dup) Implicit treatment of time Out of order event handling Implicit batch boundaries 13
- 14. Continuous apps with streaming 14 Streaming job Serve& store
- 15. Extending the Yahoo! benchmark Work of Jamie Grier, inspired by a real continuous application at Twitter 15 http://data-artisans.com/extending-the-yahoo-streaming-benchmark/
- 16. What is the use case? Counting! • Tweet impressions or ad views Most analytics is continuous counting and aggregations grouped by dimensions • E.g., anomaly detection 16
- 17. Requirements Performance: millions of events/sec, millions of keys Correctness: counts correlated with timestamps Consistency: counts should be correct under failures Manageability: ability to pause & restart, reprocess, change code, etc 17
- 18. Before Flink Performance: 1000s of cores needed to sustain workload Correctness: time handled in application code (or not) Consistency: approximate results during the day, exact results once a day (lambda) Manageability: acceptable 18
- 19. After Flink Performance: 10s of cores needed to sustain workload Correctness: time handled by framework Consistency: correct results on demand Manageability: acceptable 19
- 20. Results (yet to be beaten!) Same program as Yahoo! benchmark 30x over Storm, plus consistent results 20
- 21. Manageability Flink savepoints (Flink 1.0): consistent snapshots of stateful applications • Planned downtime for code upgrades, maintenance, migration, debugging, etc Monitoring (Flink 1.1) Dynamic scaling (Flink 1.2+) 21
- 23. Streaming and microservices 23 App App App local statelocal state Archive A decentralized architecture favors a streaming-based data infrastructure with local application state
- 25. Zalando 25 Transitioning from monolithic architecture to microservices
- 26. New BI stack 26
- 27. Flink @ Zalando (present & future) Business process monitoring • Check if Zalando platform works • Order & delivery velocities • SLAs of related events Continuous ETL • Transformation, combination, pre-aggregation • Data cleansing and validation Complex Event Processing Sales monitoring 27
- 29. Stream Processing as a Service How do we make stream processing more accessible to the data analyst? More familiar interfaces • Flink 1.1 includes the first version of SQL for static data sets and data streams Easier deployment 29
- 30. King.com 30
- 31. King.com - RBEA RBEA – a platform designed to make stream processing available inside King.com Data scientists submit scripts in Groovy Flink backend executes these scripts 31 https://techblog.king.com/rbea-scalable-real-time-analytics-king/
- 32. Netflix Netflix plans to offer Stream Processing as a Service internally in the company Currently testing Flink and Apache Beam 32 http://www.slideshare.net/mdaxini/netflix-keystone-streaming-data-pipeline-scale-in-the-clouddbtb2016-62076009
- 33. Closing 33
- 34. Disclaimer A lot of this presentation is based on the work of very talented engineers building data products with Flink Bouygues Telecom: Amine Abdessemed, ... Zalando: Mihail Vieru, Javier Lopez King.com: Gyula Fora, Mattias Andersson, ... Netflix: Monal Daxini, ... 34
- 35. More Flink tales at Hadoop Summit 35 Xiaowei Jiang Blink−Improved Runtime for Flink and its Application in Alibaba Search Wednesday, June 29, 2016, 2:10PM - 2:50PM 210C Stephan Ewen Turning the Stream Processor into a Database: Building Online Applications on Streams Thursday, June 30, 2016, 12:20PM - 1:00PM 212
- 36. Flink Forward 2016, Berlin Submission deadline: June 30, 2016 (watch website) Early bird deadline: July 15, 2016 www.flink-forward.org
- 38. Appendix
- 39. Batch < Streaming In principle, batch is a special case of streaming (global window) In practice, batch processors can be more efficient than stream processors in batch Flink is a very efficient batch processor (DataSet code path) 39
Editor's Notes
- #14: 3 systems (batch), or 5 systems (streaming), Need to add a new system for millisecond alerts What If I want to count every 5 minutes, not 1 hour? Just ignores out of order What if I wanna do sessions?