Graph Stream Processing : spinning fast, large scale, complex analytics
3 likes1,504 views
The document discusses the need for large-scale, complex, and fast data analysis to answer significant questions quickly, particularly in the context of graph stream processing. It details the evolution of distributed graph processing, the challenges of real-time analytics, and offers insights into solutions like Apache Flink and Gelly Stream. The emphasis is on addressing complex problems through efficient data analysis techniques and frameworks to support real-time decision-making.
Graph Stream Processing : spinning fast, large scale, complex analytics
- 1. Graph Stream Processing spinning fast, large-scale, complex analytics Paris Carbone PhD Candidate @ KTH Committer @ Apache Flink
- 2. We want to analyse….
- 3. We want to analyse…. data
- 4. We want to analyse…. datacomplex
- 5. We want to analyse…. datacomplexlarge-scale
- 6. We want to analyse…. data fastcomplexlarge-scale
- 7. But why do we need large-scale, complex and fast data analysis? >
- 8. But why do we need large-scale, complex and fast data analysis? to answer big complex questions faster>
- 11. Get me the best route to work right now >Hej Siri_ to answer big complex questions faster>
- 12. Get me the best route to work right now >Hej Siri_ …with the fewest human drivers to answer big complex questions faster>
- 13. Get me the best route to work right now >Hej Siri_ Lookup a pizza recipe all of my friends like but did not eat yesterday… …with the fewest human drivers to answer big complex questions faster>
- 14. Get me the best route to work right now >Hej Siri_ Lookup a pizza recipe all of my friends like but did not eat yesterday… or the day before yesterday …with the fewest human drivers to answer big complex questions faster>
- 15. Get me the best route to work right now >Hej Siri_ Lookup a pizza recipe all of my friends like but did not eat yesterday… or the day before yesterday oh! And no kebab pizza! …with the fewest human drivers to answer big complex questions faster>
- 16. Get me the best route to work right now >Hej Siri_ Lookup a pizza recipe all of my friends like but did not eat yesterday… Siri, is it possible to re-unite all data scientists in the world? or the day before yesterday oh! And no kebab pizza! …with the fewest human drivers to answer big complex questions faster>
- 17. no matter if they use Spark or Flink or just ipython Get me the best route to work right now >Hej Siri_ Lookup a pizza recipe all of my friends like but did not eat yesterday… Siri, is it possible to re-unite all data scientists in the world? or the day before yesterday oh! And no kebab pizza! …with the fewest human drivers to answer big complex questions faster>
- 18. no matter if they use Spark or Flink or just ipython Get me the best route to work right now >Hej Siri_ Lookup a pizza recipe all of my friends like but did not eat yesterday… Siri, is it possible to re-unite all data scientists in the world? or the day before yesterday oh! And no kebab pizza! …with the fewest human drivers to answer big complex questions faster>
- 23. to answer big complex questions faster> FIRST EARTH WORLD PROBLEM use Spark or Flink or just ipython best route to work right now re-unite all data scientists in the world? oh! And no kebab pizza! …with the fewest human drivers 30000 AD
- 24. Still, fast analytics might save us some day… • We can access patient movements and fb, twitter and pretty much all social media interactions • Can we stop a pandemic? • Or can we predict fast where the virus can spread?
- 25. Now how do we analyse… data fastcomplexlarge-scale ?
- 26. Now how do we analyse… data graphdistributed streaming
- 27. Now how do we analyse… data graphdistributed streaming everything is a graph
- 28. Now how do we analyse… data graphdistributed streaming everything is many everything is a graph
- 29. Now how do we analyse… data graphdistributed streaming everything is many everything is a graph everything is a stream
- 30. it all started… as a first world problem question
- 31. but then things escalated quickly… …and machinery got cheaper and we suddenly realised that we have big data
- 32. Distributed Graph processing was born Thus,
- 33. Distributed Graph processing was born Thus, Map Reduce 1. Store Partitioned Data 2. Sent Local computation (map) 3. now shuffle it on disks 4. merge the results (reduce) 5. Store the result back DFS : distributed file system
- 34. Distributed Graph processing was born Thus, 1. Store Updates to DFS 2. Load graph snapshot (mem) 3. Compute round~superstep 4. Store updates 5. …repeat Distributed Graph ProcessingMap Reduce 1. Store Partitioned Data 2. Sent Local computation (map) 3. now shuffle it on disks 4. merge the results (reduce) 5. Store the result back DFS : distributed file system
- 35. Distributed Graph processing was born Thus, 1. Store Updates to DFS 2. Load graph snapshot (mem) 3. Compute round~superstep 4. Store updates 5. …repeat Distributed Graph ProcessingMap Reduce 1. Store Partitioned Data 2. Sent Local computation (map) 3. now shuffle it on disks 4. merge the results (reduce) 5. Store the result back DFS : distributed file system
- 36. Distributed Graph processing was born Thus, 1. Store Updates to DFS 2. Load graph snapshot (mem) 3. Compute round~superstep 4. Store updates 5. …repeat Distributed Graph ProcessingMap Reduce 1. Store Partitioned Data 2. Sent Local computation (map) 3. now shuffle it on disks 4. merge the results (reduce) 5. Store the result back DFS : distributed file system
- 37. Distributed Graph processing was born Thus, 1. Store Updates to DFS 2. Load graph snapshot (mem) 3. Compute round~superstep 4. Store updates 5. …repeat Distributed Graph ProcessingMap Reduce 1. Store Partitioned Data 2. Sent Local computation (map) 3. now shuffle it on disks 4. merge the results (reduce) 5. Store the result back DFS : distributed file system
- 38. Distributed Graph processing was born Thus, 1. Store Updates to DFS 2. Load graph snapshot (mem) 3. Compute round~superstep 4. Store updates 5. …repeat Distributed Graph ProcessingMap Reduce 1. Store Partitioned Data 2. Sent Local computation (map) 3. now shuffle it on disks 4. merge the results (reduce) 5. Store the result back DFS : distributed file system
- 39. Distributed Graph processing was born Thus, 1. Store Updates to DFS 2. Load graph snapshot (mem) 3. Compute round~superstep 4. Store updates 5. …repeat Distributed Graph ProcessingMap Reduce 1. Store Partitioned Data 2. Sent Local computation (map) 3. now shuffle it on disks 4. merge the results (reduce) 5. Store the result back DFS : distributed file system
- 40. • We want to compute the Connected Components of a distributed graph. • Basic computation element (map): vertex • Updates : messages to other vertices Distributed Graph processing example
- 41. • We want to compute the Connected Components of a distributed graph. • Basic computation element (map): vertex • Updates : messages to other vertices Distributed Graph processing example 1 2 3
- 42. Distributed Graph processing example 1 43 2 5 6 7 8 ROUND 0
- 43. Distributed Graph processing example 1 43 2 5 ROUND 0 6 7 8 3 1 4 4 5 2 4 2 3 5 7 8 6 8 6 7
- 44. Distributed Graph processing example 1 21 2 2 ROUND 1 6 6 6
- 45. Distributed Graph processing example 1 2 2 2 2 1 2 6 6 6 6 1 21 2 2 ROUND 1 6 6 6 6 6
- 46. Distributed Graph processing example 1 11 2 2 ROUND 2 6 6 6
- 47. Distributed Graph processing example 1 11 2 2 ROUND 2 6 6 6 1 1 1
- 48. Distributed Graph processing example 1 11 1 1 ROUND 3 6 6 6
- 49. Distributed Graph processing example 1 11 1 1 ROUND 3 6 6 6 1 1 1 1
- 50. Distributed Graph processing example 1 11 1 1 ROUND 4 6 6 6 No messages, DONE!
- 51. • Examples of Load-Compute-Store systems: Pregel, Graphx (spark), Graphlab, PowerGraph • Same execution strategy - Same problems • It’s slow • Too much re-computation ($€) for nothing. • Real World Updates anyone? Distributed Graph processing systems
- 53. …and streaming came to mess everything make fast and simple
- 54. …and streaming came to mess everything make fast and simple real world
- 55. …and streaming came to mess everything make fast and simple real world event records • local state stays here • local computation too The Dataflow™
- 57. Streaming is so advanced that… • subsecond latency and high throughput finally coexist • it does fault tolerance without batch writes* • late data** is handled gracefully * https://arxiv.org/abs/1506.08603• ** http://dl.acm.org/citation.cfm?id=2824076
- 58. Streaming is so advanced that… …but what about complex problems? • subsecond latency and high throughput finally coexist • it does fault tolerance without batch writes* • late data** is handled gracefully * https://arxiv.org/abs/1506.08603• ** http://dl.acm.org/citation.cfm?id=2824076
- 60. can we make it happen?
- 61. can we make it happen? • Problem: Can’t keep an infinite graph in- memory and do complex stuff
- 62. can we make it happen? • Problem: Can’t keep an infinite graph in- memory and do complex stuff ?? universe
- 63. can we make it happen? • Problem: Can’t keep an infinite graph in- memory and do complex stuff ?? universe >it was never about the graph silly, it was about answering complex questions, remember?
- 64. can we make it happen? • Problem: Can’t keep an infinite graph in- memory and do complex stuff universe ;) universe summary >it was never about the graph silly, it was about answering complex questions, remember? answers
- 65. Examples of Summaries • Spanners : distance estimation • Sparsifiers : cut estimation • Sketches : homomorphic properties graph summary algorithm algorithm~R1 R2
- 66. Distributed Graph streaming example 54 76 86 42 31 52Connected Components on a stream of edges (additions)
- 67. 31 Distributed Graph streaming example 54 76 86 42 43 31 52 Connected Components on a stream of edges (additions) 1
- 68. 52 Distributed Graph streaming example 54 76 86 42 43 87 52 Connected Components on a stream of edges (additions) 31 1 2
- 69. 52 4 Distributed Graph streaming example 54 76 86 42 43 87 41 Connected Components on a stream of edges (additions) 31 1 2
- 70. 52 4 Distributed Graph streaming example 76 86 42 43 87 41 Connected Components on a stream of edges (additions) 31 1 76 2 6
- 71. 52 4 8 Distributed Graph streaming example 86 42 43 87 41 Connected Components on a stream of edges (additions) 31 1 76 2 6
- 72. 8 52 4 76 Distributed Graph streaming example 42 43 87 41 Connected Components on a stream of edges (additions) 31 1 2 6
- 73. 8 52 4 76 Distributed Graph streaming example 42 43 87 41 Connected Components on a stream of edges (additions) 31 1 2 6
- 74. 8 52 4 76 Distributed Graph streaming example 43 87 41Connected Components on a stream of edges (additions) 31 1 6
- 75. But Is this Efficient? Sure, we can distribute the edges and summaries
- 76. But Is this Efficient? Sure, we can distribute the edges and summaries any systems in mind?
- 77. Gelly Stream Graph stream processing with Apache Flink
- 78. Gelly Stream Oveview DataStreamDataSet Distributed Dataflow Deployment Gelly Gelly- ➤ Static Graphs ➤ Multi-Pass Algorithms ➤ Full Computations ➤ Dynamic Graphs ➤ Single-Pass Algorithms ➤ Approximate Computations DataStream
- 79. Gelly Stream Status ➤ Properties and Metrics ➤ Transformations ➤ Aggregations ➤ Discretization ➤ Neighborhood Aggregations ➤ Graph Streaming Algorithms ➤ Connected Components ➤ Bipartiteness Check ➤ Window Triangle Count ➤ Triangle Count Estimation ➤ Continuous Degree Aggregate
- 81. wait, so now we can detect connected components right away?
- 82. wait, so now we can detect connected components right away?
- 83. wait, so now we can detect connected components right away? Solved! But how about our other issues now?
- 86. Gelly-Stream to the rescue graphStream.filterVertices(DataScientists()) .slice(Time.of(10, MINUTE), EdgeDirection.IN) .applyOnNeighbors(FindPairs()) wendy checked_in glaze steve checked_in glaze tom checked_in joe’s_grill sandra checked_in glaze rafa checked_in joe’s_grill wendy steve sandra glaze tom rafa joe’s grill {wendy, steve} {steve, sandra} {wendy, sandra} {tom, rafa}
- 89. The next step • Iterative model* on streams for deeper analytics • More Summaries • Better Our-Of-Core State Integration • AdHoc Graph Queries Large-scale, Complex, Fast, Deep Analytics * http://dl.acm.org/citation.cfm?id=2983551
- 90. Try out Gelly-Stream* because all questions matter @SenorCarbone *https://github.com/vasia/gelly-streaming