SlideShare a Scribd company logo

Apache Flink & Graph Processing

Vasia Kalavri
Vasia Kalavri

This document discusses batch and stream graph processing with Apache Flink. It provides an overview of distributed graph processing and Flink's graph processing APIs, Gelly for batch graph processing and Gelly-Stream for continuous graph processing on data streams. It describes how Gelly and Gelly-Stream allow for processing large and dynamic graphs in a distributed fashion using Flink's dataflow engine.

Data & Analytics
1 of 97
Downloaded 92 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
Batch & Stream Graph Processing with Apache Flink Vasia Kalavri vasia@apache.org @vkalavri Apache Flink Meetup London October 5th, 2016
2 Graphs capture relationships between data items connections, interactions, purchases, dependencies, friendships, etc. Recommenders Social networks Bioinformatics Web search
Outline • Distributed Graph Processing 101 • Gelly: Batch Graph Processing with Apache Flink • BREAK! • Gelly-Stream: Continuous Graph Processing with Apache Flink
Apache Flink • An open-source, distributed data analysis framework • True streaming at its core • Streaming & Batch API 4 Historic data Kafka, RabbitMQ, ... HDFS, JDBC, ... Event logs ETL, Graphs,
 Machine Learning
 Relational, … Low latency,
 windowing, aggregations, ...
WHEN DO YOU NEED DISTRIBUTED GRAPH PROCESSING?
MY GRAPH IS SO BIG, IT DOESN’T FIT IN A SINGLE MACHINE Big Data Ninja MISCONCEPTION #1
A SOCIAL NETWORK
NAIVE WHO(M)-T0-FOLLOW ▸ Naive Who(m) to Follow: ▸ compute a friends-of-friends list per user ▸ exclude existing friends ▸ rank by common connections
DON’T JUST CONSIDER YOUR INPUT GRAPH SIZE. INTERMEDIATE DATA MATTERS TOO!
DISTRIBUTED PROCESSING IS ALWAYS FASTER THAN SINGLE-NODE Data Science Rockstar MISCONCEPTION #2
Apache Flink & Graph Processing
Apache Flink & Graph Processing
GRAPHS DON’T APPEAR OUT OF THIN AIR Expectation…
GRAPHS DON’T APPEAR OUT OF THIN AIR Reality!
WHEN DO YOU NEED DISTRIBUTED GRAPH PROCESSING? ▸ When you do have really big graphs ▸ When the intermediate data is big ▸ When your data is already distributed ▸ When you want to build end-to-end graph pipelines
HOW DO WE EXPRESS A DISTRIBUTED GRAPH ANALYSIS TASK?
RECENT DISTRIBUTED GRAPH PROCESSING HISTORY 2004 MapReduce Pegasus 2009 Pregel 2010 Signal-Collect PowerGraph 2012 Iterative value propagation Giraph++ 2013 Graph Traversals NScale 2014 Ego-network analysis Arabesque 2015 Pattern Matching Tinkerpop
PREGEL: THINK LIKE A VERTEX 1 5 4 3 2 1 3, 4 2 1, 4 5 3 ...
PREGEL: SUPERSTEPS (Vi+1, outbox) <— compute(Vi, inbox) 1 3, 4 2 1, 4 5 3 .. 1 3, 4 2 1, 4 5 3 .. Superstep i Superstep i+1
PAGERANK: THE WORD COUNT OF GRAPH PROCESSING VertexID Out-degree Transition Probability 1 2 1/2 2 2 1/2 3 0 - 4 3 1/3 5 1 1 1 5 4 3 2
PAGERANK: THE WORD COUNT OF GRAPH PROCESSING VertexID Out-degree Transition Probability 1 2 1/2 2 2 1/2 3 0 - 4 3 1/3 5 1 1 PR(3) = 0.5*PR(1) + 0.33*PR(4) + PR(5) 1 5 4 3 2
1 5 4 3 2 PAGERANK: THE WORD COUNT OF GRAPH PROCESSING VertexID Out-degree Transition Probability 1 2 1/2 2 2 1/2 3 0 - 4 3 1/3 5 1 1 PR(3) = 0.5*PR(1) + 0.33*PR(4) + PR(5)
PAGERANK: THE WORD COUNT OF GRAPH PROCESSING VertexID Out-degree Transition Probability 1 2 1/2 2 2 1/2 3 0 - 4 3 1/3 5 1 1 PR(3) = 0.5*PR(1) + 0.33*PR(4) + PR(5) 1 5 4 3 2
PAGERANK: THE WORD COUNT OF GRAPH PROCESSING VertexID Out-degree Transition Probability 1 2 1/2 2 2 1/2 3 0 - 4 3 1/3 5 1 1 PR(3) = 0.5*PR(1) + 0.33*PR(4) + PR(5) 1 5 4 3 2
PREGEL EXAMPLE: PAGERANK void compute(messages): sum = 0.0 for (m <- messages) do sum = sum + m end for setValue(0.15/numVertices() + 0.85*sum) for (edge <- getOutEdges()) do sendMessageTo( edge.target(), getValue()/numEdges) end for sum up received messages update vertex rank distribute rank to neighbors
SIGNAL-COLLECT outbox <— signal(Vi) 1 3, 4 2 1, 4 5 3 .. 1 3, 4 2 1, 4 5 3 .. Superstep i Vi+1 <— collect(inbox) 1 3, 4 2 1, 4 5 3 .. Signal Collect Superstep i+1
SIGNAL-COLLECT EXAMPLE: PAGERANK void signal(): for (edge <- getOutEdges()) do sendMessageTo( edge.target(), getValue()/numEdges) end for void collect(messages): sum = 0.0 for (m <- messages) do sum = sum + m end for setValue(0.15/numVertices() + 0.85*sum) distribute rank to neighbors sum up messages update vertex rank
GATHER-SUM-APPLY (POWERGRAPH) 1 ... ... Gather Sum 1 2 5 ... Apply 3 1 5 5 3 1 ... Gather 3 1 5 5 3 Superstep i Superstep i+1
GSA EXAMPLE: PAGERANK double gather(source, edge, target): return target.value() / target.numEdges() double sum(rank1, rank2): return rank1 + rank2 double apply(sum, currentRank): return 0.15 + 0.85*sum compute partial rank combine partial ranks update rank
PREGEL VS. SIGNAL-COLLECT VS. GSA Update Function Properties Update Function Logic Communication Scope Communication Logic Pregel arbitrary arbitrary any vertex arbitrary Signal-Collect arbitrary based on received messages any vertex based on vertex state GSA associative & commutative based on neighbors’ values neighborhood based on vertex state
CAN WE HAVE IT ALL? ▸ Data pipeline integration: built on top of an efficient distributed processing engine ▸ Graph ETL: high-level API with abstractions and methods to transform graphs ▸ Familiar programming model: support popular programming abstractions
Gelly the Apache Flink Graph API
Apache Flink Stack Gelly Table/SQL ML SAMOA DataSet (Java/Scala) DataStream (Java/Scala) HadoopM/R Local Remote Yarn Embedded Dataflow Dataflow Table/SQL Cascading Streaming dataflow runtime CEP
Meet Gelly • Java & Scala Graph APIs on top of Flink’s DataSet API Flink Core Scala API (batch and streaming) Java API (batch and streaming) FlinkML GellyTable API ... Transformations and Utilities Iterative Graph Processing Graph Library 34
Gelly is NOT • a graph database • a specialized graph processor 35
Hello, Gelly! ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); DataSet<Edge<Long, NullValue>> edges = getEdgesDataSet(env); Graph<Long, Long, NullValue> graph = Graph.fromDataSet(edges, env); DataSet<Vertex<Long, Long>> verticesWithMinIds = graph.run( new ConnectedComponents(maxIterations)); val env = ExecutionEnvironment.getExecutionEnvironment val edges: DataSet[Edge[Long, NullValue]] = getEdgesDataSet(env) val graph = Graph.fromDataSet(edges, env) val components = graph.run(new ConnectedComponents(maxIterations)) Java Scala
Graph Methods Graph Properties getVertexIds getEdgeIds numberOfVertices numberOfEdges getDegrees Mutations add vertex/edge remove vertex/edge Transformations map, filter, join subgraph, union, difference reverse, undirected getTriplets Generators R-Mat (power-law) Grid Star Complete …
Example: mapVertices // increment each vertex value by one
 val graph = Graph.fromDataSet(...)
 
 // increment each vertex value by one
 val updatedGraph = graph.mapVertices(v => v.getValue + 1) 4 2 8 5 5 3 1 7 4 5
Example: subGraph val graph: Graph[Long, Long, Long] = ...
 
 // keep only vertices with positive values
 // and only edges with negative values
 val subGraph = graph.subgraph( vertex => vertex.getValue > 0, edge => edge.getValue < 0 )
Neighborhood Methods Apply a reduce function to the 1st-hop neighborhood of each vertex in parallel graph.reduceOnNeighbors( new MinValue, EdgeDirection.OUT)
What makes Gelly unique? • Batch graph processing on top of a streaming dataflow engine • Built for end-to-end analytics • Support for multiple iteration abstractions • Graph algorithm building blocks • A large open-source library of graph algorithms
Why streaming dataflow? • Batch engines materialize data… even if they don’t have to • the graph is always loaded and materialized in memory, even if not needed, e.g. mapping, filtering, transformation • Communication and computation overlap • We can do continuous graph processing (more after the break!)
End-to-end analytics • Graphs don’t appear out of thin air… • We need to support pre- and post-processing • Gelly can be easily mixed with the DataSet API: pre-processing, graph analysis, and post- processing in the same Flink program
Iterative Graph Processing • Gelly offers iterative graph processing abstractions on top of Flink’s Delta iterations • vertex-centric • scatter-gather • gather-sum-apply • partition-centric*
Flink Iteration Operators Input Iterative Update Function Result Replace Workset Iterative Update Function Result Solution Set State
Optimization • the runtime is aware of the iterative execution • no scheduling overhead between iterations • caching and state maintenance are handled automatically Push work
 “out of the loop” Maintain state as indexCache Loop-invariant Data
Vertex-Centric SSSP final class SSSPComputeFunction extends ComputeFunction { override def compute(vertex: Vertex, messages: MessageIterator) = { var minDistance = if (vertex.getId == srcId) 0 else Double.MaxValue while (messages.hasNext) { val msg = messages.next if (msg < minDistance) minDistance = msg } if (vertex.getValue > minDistance) { setNewVertexValue(minDistance) for (edge: Edge <- getEdges) sendMessageTo(edge.getTarget, vertex.getValue + edge.getValue) }
Algorithms building blocks • Allow operator re-use across graph algorithms when processing the same input with a similar configuration
Library of Algorithms • PageRank • Single Source Shortest Paths • Label Propagation • Weakly Connected Components • Community Detection • Triangle Count & Enumeration • Local and Global Clustering Coefficient • HITS • Jaccard & Adamic-Adar Similarity • Graph Summarization • val ranks = inputGraph.run(new PageRank(0.85, 20))
Tracker Tracker Ad Server display relevant ads cookie exchange profiling Web Tracking
Can’t we block them? proxy Tracker Tracker Ad Server Legitimate site
• not frequently updated • not sure who or based on what criteria URLs are blacklisted • miss “hidden” trackers or dual-role nodes • blocking requires manual matching against the list • can you buy your way into the whitelist? Available Solutions Crowd-sourced “black lists” of tracker URLs: - AdBlock, DoNotTrack, EasyPrivacy
DataSet • 6 months (Nov 2014 - April 2015) of augmented Apache logs from a web proxy • 80m requests, 2m distinct URLs, 3k users
h2 h3 h4 h5 h6 h8 h7 h1 h3 h4 h5 h6 h1 h2 h7 h8 r1 r2 r3 r5 r6 r7 NT NT T T ? T NT NT r4 r1 r2 r3 r3 r3 r4 r5r6 r7 hosts-projection graph : referer : non-tracker host : tracker host : unlabeled host The Hosts-Projection Graph U: Referers referer-hosts graph V: hosts
Classification via Label Propagation non-tracker tracker unlabeled 55
Data Pipeline raw logs cleaned logs 1: logs pre- processing 2: bipartite graph creation 3: largest connected component extraction 4: hosts- projection graph creation 5: community detection google-analytics.com: T bscored-research.com: T facebook.com: NT github.com: NT cdn.cxense.com: NT ... 6: results DataSet API Gelly DataSet API
Feeling Gelly? • Gelly Guide https://ci.apache.org/projects/flink/flink-docs-master/libs/ gelly_guide.html • To Petascale and Beyond @Flink Forward ‘16 http://flink-forward.org/kb_sessions/to-petascale-and-beyond-apache- flink-in-the-clouds/ • Web Tracker Detection @Flink Forward ’15 https://www.youtube.com/watch?v=ZBCXXiDr3TU paper: Kalavri, Vasiliki, et al. "Like a pack of wolves: Community structure of web trackers." International Conference on Passive and Active Network Measurement, 2016.
Gelly-Stream single-pass stream graph processing with Flink
Real Graphs are dynamic Graphs are created from events happening in real-time
Apache Flink & Graph Processing
How we’ve done graph processing so far 1. Load: read the graph from disk and partition it in memory
2. Compute: read and mutate the graph state How we’ve done graph processing so far 1. Load: read the graph from disk and partition it in memory
3. Store: write the final graph state back to disk How we’ve done graph processing so far 2. Compute: read and mutate the graph state 1. Load: read the graph from disk and partition it in memory
What’s wrong with this model? • It is slow • wait until the computation is over before you see any result • pre-processing and partitioning • It is expensive • lots of memory and CPU required in order to scale • It requires re-computation for graph changes • no efficient way to deal with updates
Can we do graph processing on streams? • Maintain the dynamic graph structure • Provide up-to-date results with low latency • Compute on fresh state only
Single-pass graph streaming • Each event is an edge addition • Maintain only a graph summary • Recent events are grouped in graph windows
Apache Flink & Graph Processing
Graph Summaries • spanners for distance estimation • sparsifiers for cut estimation • sketches for homomorphic properties graph summary algorithm algorithm~R1 R2
1 43 2 5 i=0 Batch Connected Components 6 7 8
1 43 2 5 6 7 8 i=0 Batch Connected Components 1 4 3 4 5 2 3 5 2 4 7 8 6 7 6 8
1 21 2 2 i=1 Batch Connected Components 6 6 6
1 21 1 2 6 6 6 i=1 Batch Connected Components 2 1 2 2 1 1 2 1 2 7 6 6 6
1 11 1 1 i=2 Batch Connected Components 6 6 6
54 76 86 42 31 52 Stream Connected Components Graph Summary: Disjoint Set (Union-Find) • Only store component IDs and vertex IDs
54 76 86 42 43 31 52 1 3 Cid = 1
54 76 86 42 43 87 31 52 1 3 Cid = 1 2 5 Cid = 2
54 76 86 42 43 87 41 31 52 1 3 Cid = 1 2 5 Cid = 2 4
54 76 86 42 43 87 41 31 52 1 3 Cid = 1 2 5 Cid = 2 4 6 7 Cid = 6
54 76 86 42 43 87 41 31 52 1 3 Cid = 1 2 5 Cid = 2 4 6 7 Cid = 6 8
54 76 86 42 43 87 41 52 1 3 Cid = 1 2 5 Cid = 2 4 6 7 Cid = 6 8
54 76 86 42 43 87 41 6 7 Cid = 6 8 1 3 Cid = 1 2 5 Cid = 2 4
54 76 86 42 43 87 41 1 3 Cid = 1 2 5 4 6 7 Cid = 6 8
Distributed Stream Connected Components
Stream Connected Components with Flink DataStream<DisjointSet> cc = edgeStream
 .keyBy(0)
 .timeWindow(Time.of(100, TimeUnit.MILLISECONDS))
 .fold(new DisjointSet(), new UpdateCC())
 .flatMap(new Merger())
 .setParallelism(1);
Stream Connected Components with Flink DataStream<DisjointSet> cc = edgeStream
 .keyBy(0)
 .timeWindow(Time.of(100, TimeUnit.MILLISECONDS))
 .fold(new DisjointSet(), new UpdateCC())
 .flatMap(new Merger())
 .setParallelism(1); Partition the edge stream
Stream Connected Components with Flink DataStream<DisjointSet> cc = edgeStream
 .keyBy(0)
 .timeWindow(Time.of(100, TimeUnit.MILLISECONDS))
 .fold(new DisjointSet(), new UpdateCC())
 .flatMap(new Merger())
 .setParallelism(1); Define the merging frequency
Stream Connected Components with Flink DataStream<DisjointSet> cc = edgeStream
 .keyBy(0)
 .timeWindow(Time.of(100, TimeUnit.MILLISECONDS))
 .fold(new DisjointSet(), new UpdateCC())
 .flatMap(new Merger())
 .setParallelism(1); merge locally
Stream Connected Components with Flink DataStream<DisjointSet> cc = edgeStream
 .keyBy(0)
 .timeWindow(Time.of(100, TimeUnit.MILLISECONDS))
 .fold(new DisjointSet(), new UpdateCC())
 .flatMap(new Merger())
 .setParallelism(1); merge globally
Gelly on Streams DataStreamDataSet Distributed Dataflow Deployment Gelly Gelly-Stream • Static Graphs • Multi-Pass Algorithms • Full Computations • Dynamic Graphs • Single-Pass Algorithms • Approximate Computations DataStream
Introducing Gelly-Stream Gelly-Stream enriches the DataStream API with two new additional ADTs: • GraphStream: • A representation of a data stream of edges. • Edges can have state (e.g. weights). • Supports property streams, transformations and aggregations. • GraphWindow: • A “time-slice” of a graph stream. • It enables neighborhood aggregations
GraphStream Operations .getEdges() .getVertices() .numberOfVertices() .numberOfEdges() .getDegrees() .inDegrees() .outDegrees() GraphStream -> DataStream .mapEdges(); .distinct(); .filterVertices(); .filterEdges(); .reverse(); .undirected(); .union(); GraphStream -> GraphStream Property Streams Transformations
Graph Stream Aggregations result aggregate property streamgraph stream (window) fold combine fold reduce local summaries global summary edges agg global aggregates can be persistent or transient graphStream.aggregate( new MyGraphAggregation(window, fold, combine, transform))
Slicing Graph Streams graphStream.slice(Time.of(1, MINUTE)); 11:40 11:41 11:42 11:43
Aggregating Slices graphStream.slice(Time.of(1, MINUTE), direction) .reduceOnEdges(); .foldNeighbors(); .applyOnNeighbors(); • Slicing collocates edges by vertex information • Neighborhood aggregations on sliced graphs source target Aggregations
Finding Matches Nearby graphStream.filterVertices(GraphGeeks()) .slice(Time.of(15, MINUTE), EdgeDirection.IN) .applyOnNeighbors(FindPairs()) slice GraphStream :: graph geek check-ins wendy checked_in soap_bar steve checked_in soap_bar tom checked_in joe’s_grill sandra checked_in soap_bar rafa checked_in joe’s_grill wendy steve sandra soap bar tom rafa joe’s grill FindPairs {wendy, steve} {steve, sandra} {wendy, sandra} {tom, rafa} GraphWindow :: user-place
Feeling Gelly? • Gelly Guide https://ci.apache.org/projects/flink/flink-docs-master/libs/ gelly_guide.html • Gelly-Stream Repository https://github.com/vasia/gelly-streaming • Gelly-Stream talk @FOSDEM16 https://fosdem.org/2016/schedule/event/graph_processing_apache_flink/ • Related Papers http://www.citeulike.org/user/vasiakalavri/tag/graph-streaming
Batch & Stream Graph Processing with Apache Flink Vasia Kalavri vasia@apache.org @vkalavri Apache Flink Meetup London October 5th, 2016

More Related Content

What's hot (20)

PDF
Airflow introduction
Chandler Huang
 
PPTX
Grafana
NoelMc Grath
 
PDF
Apache Calcite (a tutorial given at BOSS '21)
Julian Hyde
 
PDF
Streaming SQL with Apache Calcite
Julian Hyde
 
PDF
Tame the small files problem and optimize data layout for streaming ingestion...
Flink Forward
 
PDF
Productizing Structured Streaming Jobs
Databricks
 
PDF
Opentracing jaeger
Oracle Korea
 
PDF
Getting Started with Confluent Schema Registry
confluent
 
PDF
Making Apache Spark Better with Delta Lake
Databricks
 
PDF
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022
HostedbyConfluent
 
PDF
Building a fully managed stream processing platform on Flink at scale for Lin...
Flink Forward
 
PPTX
Grafana optimization for Prometheus
Mitsuhiro Tanda
 
PDF
Airflow Intro-1.pdf
BagustTriCahyo1
 
PDF
Data Source API in Spark
Databricks
 
PDF
Beyond SQL: Speeding up Spark with DataFrames
Databricks
 
PDF
Changelog Stream Processing with Apache Flink
Flink Forward
 
PPTX
Apache Airflow Introduction
Liangjun Jiang
 
PDF
Getting Started Monitoring with Prometheus and Grafana
Syah Dwi Prihatmoko
 
PDF
Monitoring Flink with Prometheus
Maximilian Bode
 
PDF
Building robust CDC pipeline with Apache Hudi and Debezium
Tathastu.ai
 
Airflow introduction
Chandler Huang
 
Grafana
NoelMc Grath
 
Apache Calcite (a tutorial given at BOSS '21)
Julian Hyde
 
Streaming SQL with Apache Calcite
Julian Hyde
 
Tame the small files problem and optimize data layout for streaming ingestion...
Flink Forward
 
Productizing Structured Streaming Jobs
Databricks
 
Opentracing jaeger
Oracle Korea
 
Getting Started with Confluent Schema Registry
confluent
 
Making Apache Spark Better with Delta Lake
Databricks
 
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022
HostedbyConfluent
 
Building a fully managed stream processing platform on Flink at scale for Lin...
Flink Forward
 
Grafana optimization for Prometheus
Mitsuhiro Tanda
 
Airflow Intro-1.pdf
BagustTriCahyo1
 
Data Source API in Spark
Databricks
 
Beyond SQL: Speeding up Spark with DataFrames
Databricks
 
Changelog Stream Processing with Apache Flink
Flink Forward
 
Apache Airflow Introduction
Liangjun Jiang
 
Getting Started Monitoring with Prometheus and Grafana
Syah Dwi Prihatmoko
 
Monitoring Flink with Prometheus
Maximilian Bode
 
Building robust CDC pipeline with Apache Hudi and Debezium
Tathastu.ai
 

Viewers also liked (20)

PDF
Gradoop: Scalable Graph Analytics with Apache Flink @ Flink & Neo4j Meetup Be...
Martin Junghanns
 
PDF
The shortest path is not always a straight line
Vasia Kalavri
 
PDF
Graphs as Streams: Rethinking Graph Processing in the Streaming Era
Vasia Kalavri
 
PDF
Large-scale graph processing with Apache Flink @GraphDevroom FOSDEM'15
Vasia Kalavri
 
PDF
Gelly-Stream: Single-Pass Graph Streaming Analytics with Apache Flink
Vasia Kalavri
 
PDF
Gelly in Apache Flink Bay Area Meetup
Vasia Kalavri
 
PPTX
Overview of Apache Flink: Next-Gen Big Data Analytics Framework
Slim Baltagi
 
PDF
Undirected graphs
Nick Lee
 
PDF
Trade-offs in Processing Large Graphs: Representations, Storage, Systems and ...
Deepak Ajwani
 
PDF
Stream Processing with Apache Flink
C4Media
 
PDF
Distributed Graph Analytics with Gradoop
Martin Junghanns
 
PDF
HadoopCon'16, Taipei @myui
Makoto Yui
 
PDF
Computing recommendations at extreme scale with Apache Flink @Buzzwords 2015
Till Rohrmann
 
PPTX
Data Analysis With Apache Flink
DataWorks Summit
 
PDF
Asymmetry in Large-Scale Graph Analysis, Explained
Vasia Kalavri
 
PDF
Big data processing systems research
Vasia Kalavri
 
PDF
m2r2: A Framework for Results Materialization and Reuse
Vasia Kalavri
 
PDF
Block Sampling: Efficient Accurate Online Aggregation in MapReduce
Vasia Kalavri
 
PDF
Like a Pack of Wolves: Community Structure of Web Trackers
Vasia Kalavri
 
PPTX
Graphs in data structure
hamza javed
 
Gradoop: Scalable Graph Analytics with Apache Flink @ Flink & Neo4j Meetup Be...
Martin Junghanns
 
The shortest path is not always a straight line
Vasia Kalavri
 
Graphs as Streams: Rethinking Graph Processing in the Streaming Era
Vasia Kalavri
 
Large-scale graph processing with Apache Flink @GraphDevroom FOSDEM'15
Vasia Kalavri
 
Gelly-Stream: Single-Pass Graph Streaming Analytics with Apache Flink
Vasia Kalavri
 
Gelly in Apache Flink Bay Area Meetup
Vasia Kalavri
 
Overview of Apache Flink: Next-Gen Big Data Analytics Framework
Slim Baltagi
 
Undirected graphs
Nick Lee
 
Trade-offs in Processing Large Graphs: Representations, Storage, Systems and ...
Deepak Ajwani
 
Stream Processing with Apache Flink
C4Media
 
Distributed Graph Analytics with Gradoop
Martin Junghanns
 
HadoopCon'16, Taipei @myui
Makoto Yui
 
Computing recommendations at extreme scale with Apache Flink @Buzzwords 2015
Till Rohrmann
 
Data Analysis With Apache Flink
DataWorks Summit
 
Asymmetry in Large-Scale Graph Analysis, Explained
Vasia Kalavri
 
Big data processing systems research
Vasia Kalavri
 
m2r2: A Framework for Results Materialization and Reuse
Vasia Kalavri
 
Block Sampling: Efficient Accurate Online Aggregation in MapReduce
Vasia Kalavri
 
Like a Pack of Wolves: Community Structure of Web Trackers
Vasia Kalavri
 
Graphs in data structure
hamza javed
 
Ad

Similar to Apache Flink & Graph Processing (20)

PDF
Flink Gelly - Karlsruhe - June 2015
Andra Lungu
 
PDF
Vasia Kalavri – Training: Gelly School
Flink Forward
 
PDF
Greg Hogan – To Petascale and Beyond- Apache Flink in the Clouds
Flink Forward
 
PDF
Ling liu part 02：big graph processing
jins0618
 
PDF
Single-Pass Graph Stream Analytics with Apache Flink
Paris Carbone
 
PDF
Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...
Seattle Apache Flink Meetup
 
PDF
Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...
Bowen Li
 
PDF
Demystifying Distributed Graph Processing
Vasia Kalavri
 
PDF
Processing large-scale graphs with Google(TM) Pregel
ArangoDB Database
 
PDF
Frank Celler – Processing large-scale graphs with Google(TM) Pregel - NoSQL m...
NoSQLmatters
 
PDF
Graph Stream Processing : spinning fast, large scale, complex analytics
Paris Carbone
 
PDF
Introducing Apache Giraph for Large Scale Graph Processing
sscdotopen
 
PDF
Processing large-scale graphs with Google(TM) Pregel by MICHAEL HACKSTEIN at...
Big Data Spain
 
PDF
Large Scale Graph Processing with Apache Giraph
sscdotopen
 
PDF
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
Spark Summit
 
PPTX
First Flink Bay Area meetup
Kostas Tzoumas
 
PPTX
Automatic Scaling Iterative Computations
Guozhang Wang
 
PPTX
2013 06-03 berlin buzzwords
Nitay Joffe
 
PDF
Presentation on "Mizan: A System for Dynamic Load Balancing in Large-scale Gr...
Zuhair khayyat
 
PPTX
Pregel
Weiru Dai
 
Flink Gelly - Karlsruhe - June 2015
Andra Lungu
 
Vasia Kalavri – Training: Gelly School
Flink Forward
 
Greg Hogan – To Petascale and Beyond- Apache Flink in the Clouds
Flink Forward
 
Ling liu part 02：big graph processing
jins0618
 
Single-Pass Graph Stream Analytics with Apache Flink
Paris Carbone
 
Approximate Queries and Graph Streams on Apache Flink - Theodore Vasiloudis -...
Seattle Apache Flink Meetup
 
Approximate queries and graph streams on Flink, theodore vasiloudis, seattle...
Bowen Li
 
Demystifying Distributed Graph Processing
Vasia Kalavri
 
Processing large-scale graphs with Google(TM) Pregel
ArangoDB Database
 
Frank Celler – Processing large-scale graphs with Google(TM) Pregel - NoSQL m...
NoSQLmatters
 
Graph Stream Processing : spinning fast, large scale, complex analytics
Paris Carbone
 
Introducing Apache Giraph for Large Scale Graph Processing
sscdotopen
 
Processing large-scale graphs with Google(TM) Pregel by MICHAEL HACKSTEIN at...
Big Data Spain
 
Large Scale Graph Processing with Apache Giraph
sscdotopen
 
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
Spark Summit
 
First Flink Bay Area meetup
Kostas Tzoumas
 
Automatic Scaling Iterative Computations
Guozhang Wang
 
2013 06-03 berlin buzzwords
Nitay Joffe
 
Presentation on "Mizan: A System for Dynamic Load Balancing in Large-scale Gr...
Zuhair khayyat
 
Pregel
Weiru Dai
 
Ad

More from Vasia Kalavri (7)

PDF
From data stream management to distributed dataflows and beyond
Vasia Kalavri
 
PDF
Self-managed and automatically reconfigurable stream processing
Vasia Kalavri
 
PDF
Predictive Datacenter Analytics with Strymon
Vasia Kalavri
 
PDF
Online performance analysis of distributed dataflow systems (O'Reilly Velocit...
Vasia Kalavri
 
PDF
MapReduce: Optimizations, Limitations, and Open Issues
Vasia Kalavri
 
PDF
A Skype case study (2011)
Vasia Kalavri
 
PDF
Apache Flink Deep Dive
Vasia Kalavri
 
From data stream management to distributed dataflows and beyond
Vasia Kalavri
 
Self-managed and automatically reconfigurable stream processing
Vasia Kalavri
 
Predictive Datacenter Analytics with Strymon
Vasia Kalavri
 
Online performance analysis of distributed dataflow systems (O'Reilly Velocit...
Vasia Kalavri
 
MapReduce: Optimizations, Limitations, and Open Issues
Vasia Kalavri
 
A Skype case study (2011)
Vasia Kalavri
 
Apache Flink Deep Dive
Vasia Kalavri
 

Recently uploaded (20)

PDF
Optimizing Large Language Models with vLLM and Related Tools.pdf
Tamanna36
 
PDF
What does good look like - CRAP Brighton 8 July 2025
Jan Kierzyk
 
PDF
Driving Employee Engagement in a Hybrid World.pdf
Mia scott
 
PDF
Using AI/ML for Space Biology Research
VICTOR MAESTRE RAMIREZ
 
PDF
The Best NVIDIA GPUs for LLM Inference in 2025.pdf
Tamanna36
 
PPTX
apidays Helsinki & North 2025 - Running a Successful API Program: Best Practi...
apidays
 
PDF
Development and validation of the Japanese version of the Organizational Matt...
Yoga Tokuyoshi
 
PDF
Simplifying Document Processing with Docling for AI Applications.pdf
Tamanna36
 
PPTX
apidays Munich 2025 - Building an AWS Serverless Application with Terraform, ...
apidays
 
PDF
NIS2 Compliance for MSPs: Roadmap, Benefits & Cybersecurity Trends (2025 Guide)
GRC Kompas
 
PPTX
apidays Singapore 2025 - From Data to Insights: Building AI-Powered Data APIs...
apidays
 
PPTX
apidays Helsinki & North 2025 - APIs at Scale: Designing for Alignment, Trust...
apidays
 
PPTX
BinarySearchTree in datastructures in detail
kichokuttu
 
PDF
Data Retrieval and Preparation Business Analytics.pdf
kayserrakib80
 
PPTX
apidays Helsinki & North 2025 - From Chaos to Clarity: Designing (AI-Ready) A...
apidays
 
PPTX
Aict presentation on dpplppp sjdhfh.pptx
vabaso5932
 
PDF
A GraphRAG approach for Energy Efficiency Q&A
Marco Brambilla
 
PDF
Product Management in HealthTech (Case Studies from SnappDoctor)
Hamed Shams
 
PDF
apidays Helsinki & North 2025 - API-Powered Journeys: Mobility in an API-Driv...
apidays
 
PPTX
apidays Singapore 2025 - Generative AI Landscape Building a Modern Data Strat...
apidays
 
Optimizing Large Language Models with vLLM and Related Tools.pdf
Tamanna36
 
What does good look like - CRAP Brighton 8 July 2025
Jan Kierzyk
 
Driving Employee Engagement in a Hybrid World.pdf
Mia scott
 
Using AI/ML for Space Biology Research
VICTOR MAESTRE RAMIREZ
 
The Best NVIDIA GPUs for LLM Inference in 2025.pdf
Tamanna36
 
apidays Helsinki & North 2025 - Running a Successful API Program: Best Practi...
apidays
 
Development and validation of the Japanese version of the Organizational Matt...
Yoga Tokuyoshi
 
Simplifying Document Processing with Docling for AI Applications.pdf
Tamanna36
 
apidays Munich 2025 - Building an AWS Serverless Application with Terraform, ...
apidays
 
NIS2 Compliance for MSPs: Roadmap, Benefits & Cybersecurity Trends (2025 Guide)
GRC Kompas
 
apidays Singapore 2025 - From Data to Insights: Building AI-Powered Data APIs...
apidays
 
apidays Helsinki & North 2025 - APIs at Scale: Designing for Alignment, Trust...
apidays
 
BinarySearchTree in datastructures in detail
kichokuttu
 
Data Retrieval and Preparation Business Analytics.pdf
kayserrakib80
 
apidays Helsinki & North 2025 - From Chaos to Clarity: Designing (AI-Ready) A...
apidays
 
Aict presentation on dpplppp sjdhfh.pptx
vabaso5932
 
A GraphRAG approach for Energy Efficiency Q&A
Marco Brambilla
 
Product Management in HealthTech (Case Studies from SnappDoctor)
Hamed Shams
 
apidays Helsinki & North 2025 - API-Powered Journeys: Mobility in an API-Driv...
apidays
 
apidays Singapore 2025 - Generative AI Landscape Building a Modern Data Strat...
apidays
 

Apache Flink & Graph Processing

  • 1. Batch & Stream Graph Processing with Apache Flink Vasia Kalavri [email protected] @vkalavri Apache Flink Meetup London October 5th, 2016
  • 2. 2 Graphs capture relationships between data items connections, interactions, purchases, dependencies, friendships, etc. Recommenders Social networks Bioinformatics Web search
  • 3. Outline • Distributed Graph Processing 101 • Gelly: Batch Graph Processing with Apache Flink • BREAK! • Gelly-Stream: Continuous Graph Processing with Apache Flink
  • 4. Apache Flink • An open-source, distributed data analysis framework • True streaming at its core • Streaming & Batch API 4 Historic data Kafka, RabbitMQ, ... HDFS, JDBC, ... Event logs ETL, Graphs,
 Machine Learning
 Relational, … Low latency,
 windowing, aggregations, ...
  • 5. WHEN DO YOU NEED DISTRIBUTED GRAPH PROCESSING?
  • 6. MY GRAPH IS SO BIG, IT DOESN’T FIT IN A SINGLE MACHINE Big Data Ninja MISCONCEPTION #1
  • 8. NAIVE WHO(M)-T0-FOLLOW ▸ Naive Who(m) to Follow: ▸ compute a friends-of-friends list per user ▸ exclude existing friends ▸ rank by common connections
  • 9. DON’T JUST CONSIDER YOUR INPUT GRAPH SIZE. INTERMEDIATE DATA MATTERS TOO!
  • 10. DISTRIBUTED PROCESSING IS ALWAYS FASTER THAN SINGLE-NODE Data Science Rockstar MISCONCEPTION #2
  • 13. GRAPHS DON’T APPEAR OUT OF THIN AIR Expectation…
  • 14. GRAPHS DON’T APPEAR OUT OF THIN AIR Reality!
  • 15. WHEN DO YOU NEED DISTRIBUTED GRAPH PROCESSING? ▸ When you do have really big graphs ▸ When the intermediate data is big ▸ When your data is already distributed ▸ When you want to build end-to-end graph pipelines
  • 16. HOW DO WE EXPRESS A DISTRIBUTED GRAPH ANALYSIS TASK?
  • 17. RECENT DISTRIBUTED GRAPH PROCESSING HISTORY 2004 MapReduce Pegasus 2009 Pregel 2010 Signal-Collect PowerGraph 2012 Iterative value propagation Giraph++ 2013 Graph Traversals NScale 2014 Ego-network analysis Arabesque 2015 Pattern Matching Tinkerpop
  • 18. PREGEL: THINK LIKE A VERTEX 1 5 4 3 2 1 3, 4 2 1, 4 5 3 ...
  • 19. PREGEL: SUPERSTEPS (Vi+1, outbox) <— compute(Vi, inbox) 1 3, 4 2 1, 4 5 3 .. 1 3, 4 2 1, 4 5 3 .. Superstep i Superstep i+1
  • 20. PAGERANK: THE WORD COUNT OF GRAPH PROCESSING VertexID Out-degree Transition Probability 1 2 1/2 2 2 1/2 3 0 - 4 3 1/3 5 1 1 1 5 4 3 2
  • 21. PAGERANK: THE WORD COUNT OF GRAPH PROCESSING VertexID Out-degree Transition Probability 1 2 1/2 2 2 1/2 3 0 - 4 3 1/3 5 1 1 PR(3) = 0.5*PR(1) + 0.33*PR(4) + PR(5) 1 5 4 3 2
  • 22. 1 5 4 3 2 PAGERANK: THE WORD COUNT OF GRAPH PROCESSING VertexID Out-degree Transition Probability 1 2 1/2 2 2 1/2 3 0 - 4 3 1/3 5 1 1 PR(3) = 0.5*PR(1) + 0.33*PR(4) + PR(5)
  • 23. PAGERANK: THE WORD COUNT OF GRAPH PROCESSING VertexID Out-degree Transition Probability 1 2 1/2 2 2 1/2 3 0 - 4 3 1/3 5 1 1 PR(3) = 0.5*PR(1) + 0.33*PR(4) + PR(5) 1 5 4 3 2
  • 24. PAGERANK: THE WORD COUNT OF GRAPH PROCESSING VertexID Out-degree Transition Probability 1 2 1/2 2 2 1/2 3 0 - 4 3 1/3 5 1 1 PR(3) = 0.5*PR(1) + 0.33*PR(4) + PR(5) 1 5 4 3 2
  • 25. PREGEL EXAMPLE: PAGERANK void compute(messages): sum = 0.0 for (m <- messages) do sum = sum + m end for setValue(0.15/numVertices() + 0.85*sum) for (edge <- getOutEdges()) do sendMessageTo( edge.target(), getValue()/numEdges) end for sum up received messages update vertex rank distribute rank to neighbors
  • 26. SIGNAL-COLLECT outbox <— signal(Vi) 1 3, 4 2 1, 4 5 3 .. 1 3, 4 2 1, 4 5 3 .. Superstep i Vi+1 <— collect(inbox) 1 3, 4 2 1, 4 5 3 .. Signal Collect Superstep i+1
  • 27. SIGNAL-COLLECT EXAMPLE: PAGERANK void signal(): for (edge <- getOutEdges()) do sendMessageTo( edge.target(), getValue()/numEdges) end for void collect(messages): sum = 0.0 for (m <- messages) do sum = sum + m end for setValue(0.15/numVertices() + 0.85*sum) distribute rank to neighbors sum up messages update vertex rank
  • 28. GATHER-SUM-APPLY (POWERGRAPH) 1 ... ... Gather Sum 1 2 5 ... Apply 3 1 5 5 3 1 ... Gather 3 1 5 5 3 Superstep i Superstep i+1
  • 29. GSA EXAMPLE: PAGERANK double gather(source, edge, target): return target.value() / target.numEdges() double sum(rank1, rank2): return rank1 + rank2 double apply(sum, currentRank): return 0.15 + 0.85*sum compute partial rank combine partial ranks update rank
  • 30. PREGEL VS. SIGNAL-COLLECT VS. GSA Update Function Properties Update Function Logic Communication Scope Communication Logic Pregel arbitrary arbitrary any vertex arbitrary Signal-Collect arbitrary based on received messages any vertex based on vertex state GSA associative & commutative based on neighbors’ values neighborhood based on vertex state
  • 31. CAN WE HAVE IT ALL? ▸ Data pipeline integration: built on top of an efficient distributed processing engine ▸ Graph ETL: high-level API with abstractions and methods to transform graphs ▸ Familiar programming model: support popular programming abstractions
  • 33. Apache Flink Stack Gelly Table/SQL ML SAMOA DataSet (Java/Scala) DataStream (Java/Scala) HadoopM/R Local Remote Yarn Embedded Dataflow Dataflow Table/SQL Cascading Streaming dataflow runtime CEP
  • 34. Meet Gelly • Java & Scala Graph APIs on top of Flink’s DataSet API Flink Core Scala API (batch and streaming) Java API (batch and streaming) FlinkML GellyTable API ... Transformations and Utilities Iterative Graph Processing Graph Library 34
  • 35. Gelly is NOT • a graph database • a specialized graph processor 35
  • 36. Hello, Gelly! ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); DataSet<Edge<Long, NullValue>> edges = getEdgesDataSet(env); Graph<Long, Long, NullValue> graph = Graph.fromDataSet(edges, env); DataSet<Vertex<Long, Long>> verticesWithMinIds = graph.run( new ConnectedComponents(maxIterations)); val env = ExecutionEnvironment.getExecutionEnvironment val edges: DataSet[Edge[Long, NullValue]] = getEdgesDataSet(env) val graph = Graph.fromDataSet(edges, env) val components = graph.run(new ConnectedComponents(maxIterations)) Java Scala
  • 37. Graph Methods Graph Properties getVertexIds getEdgeIds numberOfVertices numberOfEdges getDegrees Mutations add vertex/edge remove vertex/edge Transformations map, filter, join subgraph, union, difference reverse, undirected getTriplets Generators R-Mat (power-law) Grid Star Complete …
  • 38. Example: mapVertices // increment each vertex value by one
 val graph = Graph.fromDataSet(...)
 
 // increment each vertex value by one
 val updatedGraph = graph.mapVertices(v => v.getValue + 1) 4 2 8 5 5 3 1 7 4 5
  • 39. Example: subGraph val graph: Graph[Long, Long, Long] = ...
 
 // keep only vertices with positive values
 // and only edges with negative values
 val subGraph = graph.subgraph( vertex => vertex.getValue > 0, edge => edge.getValue < 0 )
  • 40. Neighborhood Methods Apply a reduce function to the 1st-hop neighborhood of each vertex in parallel graph.reduceOnNeighbors( new MinValue, EdgeDirection.OUT)
  • 41. What makes Gelly unique? • Batch graph processing on top of a streaming dataflow engine • Built for end-to-end analytics • Support for multiple iteration abstractions • Graph algorithm building blocks • A large open-source library of graph algorithms
  • 42. Why streaming dataflow? • Batch engines materialize data… even if they don’t have to • the graph is always loaded and materialized in memory, even if not needed, e.g. mapping, filtering, transformation • Communication and computation overlap • We can do continuous graph processing (more after the break!)
  • 43. End-to-end analytics • Graphs don’t appear out of thin air… • We need to support pre- and post-processing • Gelly can be easily mixed with the DataSet API: pre-processing, graph analysis, and post- processing in the same Flink program
  • 44. Iterative Graph Processing • Gelly offers iterative graph processing abstractions on top of Flink’s Delta iterations • vertex-centric • scatter-gather • gather-sum-apply • partition-centric*
  • 45. Flink Iteration Operators Input Iterative Update Function Result Replace Workset Iterative Update Function Result Solution Set State
  • 46. Optimization • the runtime is aware of the iterative execution • no scheduling overhead between iterations • caching and state maintenance are handled automatically Push work
 “out of the loop” Maintain state as indexCache Loop-invariant Data
  • 47. Vertex-Centric SSSP final class SSSPComputeFunction extends ComputeFunction { override def compute(vertex: Vertex, messages: MessageIterator) = { var minDistance = if (vertex.getId == srcId) 0 else Double.MaxValue while (messages.hasNext) { val msg = messages.next if (msg < minDistance) minDistance = msg } if (vertex.getValue > minDistance) { setNewVertexValue(minDistance) for (edge: Edge <- getEdges) sendMessageTo(edge.getTarget, vertex.getValue + edge.getValue) }
  • 48. Algorithms building blocks • Allow operator re-use across graph algorithms when processing the same input with a similar configuration
  • 49. Library of Algorithms • PageRank • Single Source Shortest Paths • Label Propagation • Weakly Connected Components • Community Detection • Triangle Count & Enumeration • Local and Global Clustering Coefficient • HITS • Jaccard & Adamic-Adar Similarity • Graph Summarization • val ranks = inputGraph.run(new PageRank(0.85, 20))
  • 51. Can’t we block them? proxy Tracker Tracker Ad Server Legitimate site
  • 52. • not frequently updated • not sure who or based on what criteria URLs are blacklisted • miss “hidden” trackers or dual-role nodes • blocking requires manual matching against the list • can you buy your way into the whitelist? Available Solutions Crowd-sourced “black lists” of tracker URLs: - AdBlock, DoNotTrack, EasyPrivacy
  • 53. DataSet • 6 months (Nov 2014 - April 2015) of augmented Apache logs from a web proxy • 80m requests, 2m distinct URLs, 3k users
  • 54. h2 h3 h4 h5 h6 h8 h7 h1 h3 h4 h5 h6 h1 h2 h7 h8 r1 r2 r3 r5 r6 r7 NT NT T T ? T NT NT r4 r1 r2 r3 r3 r3 r4 r5r6 r7 hosts-projection graph : referer : non-tracker host : tracker host : unlabeled host The Hosts-Projection Graph U: Referers referer-hosts graph V: hosts
  • 55. Classification via Label Propagation non-tracker tracker unlabeled 55
  • 56. Data Pipeline raw logs cleaned logs 1: logs pre- processing 2: bipartite graph creation 3: largest connected component extraction 4: hosts- projection graph creation 5: community detection google-analytics.com: T bscored-research.com: T facebook.com: NT github.com: NT cdn.cxense.com: NT ... 6: results DataSet API Gelly DataSet API
  • 57. Feeling Gelly? • Gelly Guide https://ci.apache.org/projects/flink/flink-docs-master/libs/ gelly_guide.html • To Petascale and Beyond @Flink Forward ‘16 http://flink-forward.org/kb_sessions/to-petascale-and-beyond-apache- flink-in-the-clouds/ • Web Tracker Detection @Flink Forward ’15 https://www.youtube.com/watch?v=ZBCXXiDr3TU paper: Kalavri, Vasiliki, et al. "Like a pack of wolves: Community structure of web trackers." International Conference on Passive and Active Network Measurement, 2016.
  • 59. Real Graphs are dynamic Graphs are created from events happening in real-time
  • 61. How we’ve done graph processing so far 1. Load: read the graph from disk and partition it in memory
  • 62. 2. Compute: read and mutate the graph state How we’ve done graph processing so far 1. Load: read the graph from disk and partition it in memory
  • 63. 3. Store: write the final graph state back to disk How we’ve done graph processing so far 2. Compute: read and mutate the graph state 1. Load: read the graph from disk and partition it in memory
  • 64. What’s wrong with this model? • It is slow • wait until the computation is over before you see any result • pre-processing and partitioning • It is expensive • lots of memory and CPU required in order to scale • It requires re-computation for graph changes • no efficient way to deal with updates
  • 65. Can we do graph processing on streams? • Maintain the dynamic graph structure • Provide up-to-date results with low latency • Compute on fresh state only
  • 66. Single-pass graph streaming • Each event is an edge addition • Maintain only a graph summary • Recent events are grouped in graph windows
  • 68. Graph Summaries • spanners for distance estimation • sparsifiers for cut estimation • sketches for homomorphic properties graph summary algorithm algorithm~R1 R2
  • 74. 54 76 86 42 31 52 Stream Connected Components Graph Summary: Disjoint Set (Union-Find) • Only store component IDs and vertex IDs
  • 84. Stream Connected Components with Flink DataStream<DisjointSet> cc = edgeStream
 .keyBy(0)
 .timeWindow(Time.of(100, TimeUnit.MILLISECONDS))
 .fold(new DisjointSet(), new UpdateCC())
 .flatMap(new Merger())
 .setParallelism(1);
  • 85. Stream Connected Components with Flink DataStream<DisjointSet> cc = edgeStream
 .keyBy(0)
 .timeWindow(Time.of(100, TimeUnit.MILLISECONDS))
 .fold(new DisjointSet(), new UpdateCC())
 .flatMap(new Merger())
 .setParallelism(1); Partition the edge stream
  • 86. Stream Connected Components with Flink DataStream<DisjointSet> cc = edgeStream
 .keyBy(0)
 .timeWindow(Time.of(100, TimeUnit.MILLISECONDS))
 .fold(new DisjointSet(), new UpdateCC())
 .flatMap(new Merger())
 .setParallelism(1); Define the merging frequency
  • 87. Stream Connected Components with Flink DataStream<DisjointSet> cc = edgeStream
 .keyBy(0)
 .timeWindow(Time.of(100, TimeUnit.MILLISECONDS))
 .fold(new DisjointSet(), new UpdateCC())
 .flatMap(new Merger())
 .setParallelism(1); merge locally
  • 88. Stream Connected Components with Flink DataStream<DisjointSet> cc = edgeStream
 .keyBy(0)
 .timeWindow(Time.of(100, TimeUnit.MILLISECONDS))
 .fold(new DisjointSet(), new UpdateCC())
 .flatMap(new Merger())
 .setParallelism(1); merge globally
  • 89. Gelly on Streams DataStreamDataSet Distributed Dataflow Deployment Gelly Gelly-Stream • Static Graphs • Multi-Pass Algorithms • Full Computations • Dynamic Graphs • Single-Pass Algorithms • Approximate Computations DataStream
  • 90. Introducing Gelly-Stream Gelly-Stream enriches the DataStream API with two new additional ADTs: • GraphStream: • A representation of a data stream of edges. • Edges can have state (e.g. weights). • Supports property streams, transformations and aggregations. • GraphWindow: • A “time-slice” of a graph stream. • It enables neighborhood aggregations
  • 91. GraphStream Operations .getEdges() .getVertices() .numberOfVertices() .numberOfEdges() .getDegrees() .inDegrees() .outDegrees() GraphStream -> DataStream .mapEdges(); .distinct(); .filterVertices(); .filterEdges(); .reverse(); .undirected(); .union(); GraphStream -> GraphStream Property Streams Transformations
  • 92. Graph Stream Aggregations result aggregate property streamgraph stream (window) fold combine fold reduce local summaries global summary edges agg global aggregates can be persistent or transient graphStream.aggregate( new MyGraphAggregation(window, fold, combine, transform))
  • 93. Slicing Graph Streams graphStream.slice(Time.of(1, MINUTE)); 11:40 11:41 11:42 11:43
  • 94. Aggregating Slices graphStream.slice(Time.of(1, MINUTE), direction) .reduceOnEdges(); .foldNeighbors(); .applyOnNeighbors(); • Slicing collocates edges by vertex information • Neighborhood aggregations on sliced graphs source target Aggregations
  • 95. Finding Matches Nearby graphStream.filterVertices(GraphGeeks()) .slice(Time.of(15, MINUTE), EdgeDirection.IN) .applyOnNeighbors(FindPairs()) slice GraphStream :: graph geek check-ins wendy checked_in soap_bar steve checked_in soap_bar tom checked_in joe’s_grill sandra checked_in soap_bar rafa checked_in joe’s_grill wendy steve sandra soap bar tom rafa joe’s grill FindPairs {wendy, steve} {steve, sandra} {wendy, sandra} {tom, rafa} GraphWindow :: user-place
  • 96. Feeling Gelly? • Gelly Guide https://ci.apache.org/projects/flink/flink-docs-master/libs/ gelly_guide.html • Gelly-Stream Repository https://github.com/vasia/gelly-streaming • Gelly-Stream talk @FOSDEM16 https://fosdem.org/2016/schedule/event/graph_processing_apache_flink/ • Related Papers http://www.citeulike.org/user/vasiakalavri/tag/graph-streaming
  • 97. Batch & Stream Graph Processing with Apache Flink Vasia Kalavri [email protected] @vkalavri Apache Flink Meetup London October 5th, 2016