Data Analysis With Apache Flink
1 like2,375 views
Apache Flink is a distributed stream and batch data processing framework. It provides a functional API, relational API, and machine learning capabilities. Flink executes jobs as dataflow graphs on a distributed runtime. An example use case demonstrates log analysis to extract click data and combine it with user information to find interesting URLs. Advanced analysis like collaborative filtering for website recommendations is also supported.
![Getting at Those Clicks
6
DataSet<String> log = env.readTextFile("hdfs:///log");
DataSet<Tuple2<String, Integer>> clicks = log.flatMap(
(String line, Collector<Tuple2<String, Integer>> out) ->
String[] parts = in.split("*magic regex*");
if (isClick(parts)) {
out.collect(new Tuple2<>(parts[1],Integer.parseInt(parts[2])));
}
}
)
post /foo/bar… 313
get /data/pic.jpg 128
post /bar/baz… 128
post /hello/there… 42](https://image.slidesharecdn.com/thurs1440dataartisanskrettekrohrmann-150427165644-conversion-gate02/85/Data-Analysis-With-Apache-Flink-7-320.jpg)
val parameters = ParameterMap()
.add(HashingFT.NumFeatures, 1000000)
.add(ALS.Iterations, 10)
.add(ALS.NumFactors, 50)
.add(ALS.Lambda, 1.5)
val factorization = pipeline.fit(clickstreamDS, parameters)
Clickstream
Data
Hashing
Feature
Extractor
ALS
Matrix
factorization](https://image.slidesharecdn.com/thurs1440dataartisanskrettekrohrmann-150427165644-conversion-gate02/85/Data-Analysis-With-Apache-Flink-21-320.jpg)
![Log Analysis Code
30
ExecutionEnvironment env = TableEnvironment tableEnv = new TableEnvironment();
TableEnvironment tableEnv = new TableEnvironment();
DataSet<String> log = env.readTextFile("hdfs:///log");
DataSet<Tuple2<String, Integer>> clicks = log.flatMap(
new FlatMapFunction<String, Tuple2<String, Integer>>() {
public void flatMap(String in, Collector<Tuple2<>> out) {
String[] parts = in.split("*magic regex*");
if (parts[0].equals("click")) {
out.collect(new Tuple2<>(parts[1], Integer.parseInt(parts[4])));
}
}
});
Table clicksTable = tableEnv.toTable(clicks, "url, userId");
Table urlClickCounts = clicksTable
.groupBy("url, userId")
.select("url, userId, url.count as count");
Table userInfo = tableEnv.toTable(…, "name, id, …");
Table resultTable = urlClickCounts.join(userInfo)
.where("userId = id && count > 10")
.select("url, count, name, …");
DataSet<Result> result = tableEnv.toSet(resultTable, Result.class);
result.writeAsText("hdfs:///result");
env.execute();](https://image.slidesharecdn.com/thurs1440dataartisanskrettekrohrmann-150427165644-conversion-gate02/85/Data-Analysis-With-Apache-Flink-31-320.jpg)
Data Analysis With Apache Flink
- 1. Aljoscha Krettek / Till Rohrmann Flink committers Co-founders @ data Artisans [email protected] / [email protected] Data Analysis With Apache Flink
- 2. What is Apache Flink? 1 Functional API Relational API Graph API Machine Learning … Iterative Dataflow Engine
- 3. Apache Flink Stack 2 Python Gelly Table FlinkML SAMOA Batch Optimizer DataSet (Java/Scala) DataStream (Java/Scala) Stream Builder Hadoop M/R Distributed Runtime Local Remote Yarn Tez Embedded Dataflow Dataflow *current Flink master + few PRs Table
- 4. Example Use Case: Log Analysis 3
- 5. What Seems to be the Problem? Collect clicks from a webserver log Find interesting URLs Combine with user data 4 Web server log user data base Interesting User Data Extract Clicks Combine Massage
- 6. The Execution Environment Entry point for all Flink programs Creates DataSets from data sources 5 ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
- 7. Getting at Those Clicks 6 DataSet<String> log = env.readTextFile("hdfs:///log"); DataSet<Tuple2<String, Integer>> clicks = log.flatMap( (String line, Collector<Tuple2<String, Integer>> out) -> String[] parts = in.split("*magic regex*"); if (isClick(parts)) { out.collect(new Tuple2<>(parts[1],Integer.parseInt(parts[2]))); } } ) post /foo/bar… 313 get /data/pic.jpg 128 post /bar/baz… 128 post /hello/there… 42
- 8. The Table Environment Environment for dealing with Tables Converts between DataSet and Table 7 TableEnvironment tableEnv = new TableEnvironment();
- 9. Counting those Clicks 8 Table clicksTable = tableEnv.toTable(clicks, "url, userId"); Table urlClickCounts = clicksTable .groupBy("url, userId") .select("url, userId, url.count as count");
- 10. Getting the User Information 9 Table userInfo = tableEnv.toTable(…, "name, id, …"); Table resultTable = urlClickCounts.join(userInfo) .where("userId = id && count > 10") .select("url, count, name, …");
- 11. The Final Step 10 class Result { public String url; public int count; public String name; … } DataSet<Result> set = tableEnv.toSet(resultTable, Result.class); DataSet<Result> result = set.groupBy("url").reduceGroup(new ComplexOperation()); result.writeAsText("hdfs:///result"); env.execute();
- 12. API in a Nutshell Element-wise • map, flatMap, filter Group-wise • groupBy, reduce, reduceGroup, combineGroup, mapPartition, aggregate, distinct Binary • join, coGroup, union, cross Iterations • iterate, iterateDelta Physical re-organization • rebalance, partitionByHash, sortPartition Streaming • window, windowMap, coMap, ... 11
- 13. What happens under the hood? 12
- 14. From Program to Dataflow 13 Flink Program Dataflow Plan Optimized Plan
- 17. Going Further Log analysis result: Which user visited how often which web site Which other websites might they like? Recommendation by collaborative filtering 16
- 18. Collaborative Filtering Recommend items based on users with similar preferences Latent factor models capture underlying characteristics of items and preferences of user Predicted preference: 17 ˆru,i = xu T yi
- 19. Matrix Factorization 18 minX,Y ru,i - xu T yi( ) 2 +l nu xu 2 + ni yi 2 i å u å æ è ç ö ø ÷ ru,i¹0 å R » XT Y
- 20. Alternating least squares Iterative approximation 1. Fix X and optimize Y 2. Fix Y and optimize X Communication and computation intensive 19 R=YX x R=YX x
- 21. Matrix Factorization Pipeline 20 val featureExtractor = HashingFT() val factorizer = ALS() val pipeline = featureExtractor.chain(factorizer) val clickstreamDS = env.readCsvFile[(String, String, Int)](clickStreamData) val parameters = ParameterMap() .add(HashingFT.NumFeatures, 1000000) .add(ALS.Iterations, 10) .add(ALS.NumFactors, 50) .add(ALS.Lambda, 1.5) val factorization = pipeline.fit(clickstreamDS, parameters) Clickstream Data Hashing Feature Extractor ALS Matrix factorization
- 22. Does it Scale? 21 • 40 node GCE cluster, highmem-8 • 10 ALS iteration with 50 latent factors • Based on Spark MLlib’s implementation Scale of Netflix or Spotify
- 23. What Else Can You Do? Classification using SVMs • Conversion goal prediction Clustering • Visitor segmentation Multiple linear regression • Visitor prediction 22
- 24. Closing 23
- 25. What Have You Seen? Flink is a general-purpose analytics system Highly expressive Table API Advanced analysis with Flink’s machine learning library Jobs are executed on powerful distributed dataflow engine 24
- 26. Flink Roadmap for 2015 Additions to Machine Learning library Streaming Machine Learning Support for interactive programs Optimization for Table API queries SQL on top of Table API 25
- 27. 26
- 29. Backup Slides 28
- 30. WordCount in DataSet API 29 case class Word (word: String, frequency: Int) val env = ExecutionEnvironment.getExecutionEnvironment() val lines = env.readTextFile(...) lines .flatMap {line => line.split(" ").map(word => Word(word,1))} .groupBy("word").sum("frequency”) .print() env.execute() Java and Scala APIs offer the same functionality.
- 31. Log Analysis Code 30 ExecutionEnvironment env = TableEnvironment tableEnv = new TableEnvironment(); TableEnvironment tableEnv = new TableEnvironment(); DataSet<String> log = env.readTextFile("hdfs:///log"); DataSet<Tuple2<String, Integer>> clicks = log.flatMap( new FlatMapFunction<String, Tuple2<String, Integer>>() { public void flatMap(String in, Collector<Tuple2<>> out) { String[] parts = in.split("*magic regex*"); if (parts[0].equals("click")) { out.collect(new Tuple2<>(parts[1], Integer.parseInt(parts[4]))); } } }); Table clicksTable = tableEnv.toTable(clicks, "url, userId"); Table urlClickCounts = clicksTable .groupBy("url, userId") .select("url, userId, url.count as count"); Table userInfo = tableEnv.toTable(…, "name, id, …"); Table resultTable = urlClickCounts.join(userInfo) .where("userId = id && count > 10") .select("url, count, name, …"); DataSet<Result> result = tableEnv.toSet(resultTable, Result.class); result.writeAsText("hdfs:///result"); env.execute();
- 32. Log Analysis Dataflow Graph 31 Log Map AggUsers Join Result Group Log Map AggUsers Join combine partition sort merge sort Result Group partition sort
- 33. Pipelined Execution 32 Only 1 Stage (depending on join strategy) Data transfer in-memory and disk if needed Note: Intermediate DataSets are not necessarily “created”!
Editor's Notes
- #3: Engine is Batch or Streaming
- #7: Works also with Scala API
- #15: Visualization of program to plan to optimized plan to JobGraph What you see is not what you get.
- #16: Pipelined Execution
- #27: Algorithms: Decision trees and random forests PCA CCA More transformers: Scaler, Centering, Whitening Feature extractor Count vectorizer Outlier detector Support for cross validation Improved pipeline support Automatic pre- and post-processing pipeline SAMOA support Pending PR which will be merged with the upcoming milestone release Integration with Zeppelin, a IPython Notebook-like web interface for explorative data analysis
- #34: Visualization of JobGraph to ExecutionGraph