Working with Skewed Data: The Iterative Broadcast with Fokko Driesprong Rob Keevil
2 likes2,825 views
Skewed data is the enemy when joining tables using Spark. It shuffles a large proportion of the data onto a few overloaded nodes, bottlenecking Spark’s parallelism and resulting in out of memory errors. The go-to answer is to use broadcast joins; leaving the large, skewed dataset in place and transmitting a smaller table to every machine in the cluster for joining. But what happens when your second table is too large to broadcast, and does not fit into memory? Or even worse, when a single key is bigger than the total size of your executor? Firstly, we will give an introduction into the problem. Secondly, the current ways of fighting the problem will be explained, including why these solutions are limited. Finally, we will demonstrate a new technique – the iterative broadcast join – developed while processing ING Bank’s global transaction data. This technique, implemented on top of the Spark SQL API, allows multiple large and highly skewed datasets to be joined successfully, while retaining a high level of parallelism. This is something that is not possible with existing Spark join types.
Ad
More Related Content
What's hot (20)
Viewers also liked (6)
Ad
More from Spark Summit (20)
Ad
Recently uploaded (20)
Working with Skewed Data: The Iterative Broadcast with Fokko Driesprong Rob Keevil
- 1. Rob Keevil - KnowIT Fokko Driesprong- GoDataDriven Working with Skewed Data: The Iterative Broadcast #EUde11
- 2. #EUde11 About Rob Keevil • Philosophy, Politics & Economics • Big Data Architect • Financial and Anti-Fraud Domains • Scala Programmer
- 3. #EUde11 About Fokko Driesprong • Software Engineering and Distributed Systems • Data Engineer at GoDataDriven • Committer & PPMC Member, Apache Airflow • Open source enthusiast https://godatadriven.com/players/fokko-driesprong
- 4. #EUde11 Overview • The Skew Problem at ING • Join Strategies in Spark • Our Solution • Performance Analysis • Considerations • Future Work
- 5. #EUde11 The Skew Problem at • Billions of transactions… • Billions of accounts… • Millions of companies… • …with a very uneven distribution!
- 7. #EUde11 Join Types Native to Spark • Sort Merge Join • Broadcast Hash Join • Shuffled Hash Join • Broadcast Nested Loop Join • Cartesian Product
- 8. Key Column A 1 HELLO The Sort Merge Join 8#EUde11 Stage 1: Determine partitions by hashing the key(s) 3 GOOD Key Column A Key Column BKey Column B 1 WORLD! 2 SUMMIT 1 DUBLIN 2 STREAMING 1 EVERYBODY 2 SQL 3 MORNING 3 EVENING 3 NIGHT 4 SPAM 4 SPAM 4 SPAM 2 SPARK 4 SPAM
- 9. Executor 1 Key Column B 1 WORLD! 1 DUBLIN 1 EVERYBODY 3 MORNING 3 EVENING 3 NIGHT Key Column A Column B 1 WORLD! 1 DUBLIN 1 EVERYBODY 3 MORNING 3 EVENING 3 NIGHT 3 GOOD3 GOOD3 GOOD3 GOOD Key Column A Executor 2 9#EUde11 Stage 2: Merge Key Column B 2 SUMMIT 2 STREAMING 2 SQL 4 SPAM 4 SPAM 4 SPAM Key Column AKey Column A 1 HELLO1 HELLO1 HELLO1 HELLO Key Column A Column B 2 SUMMIT 2 STREAMING 2 SQL 4 SPAM 4 SPAM 4 SPAM 2 SPARK2 SPARK2 SPARK 4 SPAM4 SPAM4 SPAM 2 SPARK 4 SPAM
- 10. Executor 1 Executor 2 10#EUde11 Key Column A Column B 1 HELLO WORLD! 1 HELLO DUBLIN 1 HELLO EVERYBODY 3 GOOD MORNING 3 GOOD EVENING 3 GOOD NIGHT Key Column A Column B 2 SPARK SUMMIT 2 SPARK STREAMING 2 SPARK SQL 4 SPAM SPAM 4 SPAM SPAM 4 SPAM SPAM
- 11. 11
- 12. Key Column A 1 HELLO 12#EUde11 Stage 1: Determine partitions by hashing the key(s) 3 GOOD Key Column A 2 SPARK 4 SPAM Key Column BKey Column B 4 SPAM 2 SUMMIT 3 MORNING 4 SPAM 1 WORLD! 4 SPAM 4 SPAM 4 SPAM 4 SPAM 4 SPAM 4 SPAM 4 SPAM Regular Table Skewed Table
- 13. Executor 2Executor 1 Key Column B 1 WORLD! 3 MORNING Key Column A Column B 1 WORLD! 3 MORNING 3 GOOD Key Column A 13#EUde11 Stage 2: Merge Key Column B 2 SUMMIT 4 SPAM 4 SPAM 4 SPAM 4 SPAM 4 SPAM 4 SPAM 4 SPAM 4 SPAM 4 SPAM Key Column AKey Column A 1 HELLO Key Column A Column B 2 SUMMIT 4 SPAM 4 SPAM 4 SPAM 4 SPAM 4 SPAM 4 SPAM 4 SPAM 4 SPAM 4 SPAM 2 SPARK 4 SPAM 1 HELLO 3 GOOD 4 SPAM4 SPAM4 SPAM4 SPAM4 SPAM4 SPAM4 SPAM4 SPAM 2 SPARK 4 SPAM
- 14. Executor 1 Executor 2 14#EUde11 Key Column A Column B 1 HELLO WORLD! 3 GOOD MORNING Key Column A Column B 2 SPARK SUMMIT 4 SPAM SPAM 4 SPAM SPAM 4 SPAM SPAM 4 SPAM SPAM 4 SPAM SPAM 4 SPAM SPAM 4 SPAM SPAM 4 SPAM SPAM 4 SPAM SPAM
- 16. What about Broadcast Hash Join? • Leaves the partitions larger table untouched • Copies the entire smaller table to every executor • Iterate over the large table, and join using a HashMap 16#EUde11
- 17. Key Column B 4 SPAM 4 SPAM 4 SPAM 2 SUMMIT 3 MORNING 4 SPAM Key Column A 1 HELLO 4 SPAM 2 SPARK 3 GOOD Key Column A 1 HELLO 4 SPAM 2 SPARK 3 GOOD Key Column A 1 HELLO 4 SPAM 2 SPARK 3 GOOD Key Column B 4 SPAM 4 SPAM 1 WORLD 4 SPAM 4 SPAM 4 SPAM 17#EUde11 Smaller Table Larger Table Stage 1: Broadcast smaller dataset to all executors Key Column B 4 SPAM 4 SPAM 4 SPAM 2 SUMMIT 3 MORNING 4 SPAM 4 SPAM 4 SPAM 1 WORLD! 4 SPAM 4 SPAM 4 SPAM
- 18. Executor 2 3 GOOD 2 SPARK Executor 1 Key Column B 4 SPAM 4 SPAM 4 SPAM 2 SUMMIT 3 MORNING 4 SPAM Key Column A Column B 4 SPAM 4 SPAM 4 SPAM 2 SUMMIT 3 MORNING 4 SPAM 3 GOOD Key Column A 18#EUde11 Stage 2: Left join Key Column B 4 SPAM 4 SPAM 1 WORLD! 4 SPAM 4 SPAM 4 SPAM Key Column A Key Column A Column B 4 SPAM 4 SPAM 1 WORLD! 4 SPAM 4 SPAM 4 SPAM 2 SPARK 4 SPAM4 SPAM4 SPAM4 SPAM2 SPARK 4 SPAM 3 GOOD Key Column AKey Column A 1 HELLO 4 SPAM4 SPAM4 SPAM4 SPAM4 SPAM 1 HELLO 4 SPAM 1 HELLO
- 19. Executor 1 Executor 2 19#EUde11 Key Column A Column B 4 SPAM SPAM 4 SPAM SPAM 4 SPAM SPAM 2 SPARK SUMMIT 3 GOOD MORNING 4 SPAM SPAM Key Column A Column B 4 SPAM SPAM 4 SPAM SPAM 1 HELLO WORLD! 4 SPAM SPAM 4 SPAM SPAM 4 SPAM SPAM
- 21. Introducing: The Iterative Broadcast! • Divide the smaller table into “passes” • Broadcast a pass & left join to the larger dataset • Clear the broadcast partition from memory • Repeat until all passes are processed 21#EUde11
- 22. 22#EUde11 Key Column A 1 HELLO 4 SPAM 2 SPARK 3 GOOD Stage 1: Assign Pass Number to Smaller Table Key Column A Pass 1 HELLO 1 4 SPAM 2 2 SPARK 2 3 GOOD 1
- 23. Executor 2Executor 1 23#EUde11 Stage 2: Broadcast first pass Key Column B 4 SPAM 4 SPAM 1 WORLD! 4 SPAM 4 SPAM 4 SPAM Key Column B 4 SPAM 4 SPAM 4 SPAM 2 SUMMIT 3 MORNING 4 SPAM 3 GOOD 1 HELLO Key Column A Pass 1 HELLO 1 3 GOOD 1 Key Column A Pass 1 HELLO 1 3 GOOD 1
- 24. Executor 2Executor 1 24#EUde11 Stage 3: Left join first pass Key Column B 4 SPAM 4 SPAM 1 WORLD! 4 SPAM 4 SPAM 4 SPAM Key Column B 4 SPAM 4 SPAM 4 SPAM 2 SUMMIT 3 MORNING 4 SPAM Key Column A Column B 4 SPAM 4 SPAM 4 SPAM 2 SUMMIT 3 MORNING 4 SPAM Key Column A Column B 4 SPAM 4 SPAM 1 WORLD! 4 SPAM 4 SPAM 4 SPAM 3 GOOD 1 HELLO Key Column A Pass 1 HELLO 1 3 GOOD 1 Key Column A Pass 1 HELLO 1 3 GOOD 1
- 25. Executor 2Executor 1 25#EUde11 Key Column B 4 SPAM 4 SPAM 1 WORLD! 4 SPAM 4 SPAM 4 SPAM Key Column B 4 SPAM 4 SPAM 4 SPAM 2 SUMMIT 3 MORNING 4 SPAM Key Column A Column B 4 SPAM 4 SPAM 4 SPAM 2 SUMMIT 3 GOOD MORNING 4 SPAM Key Column A Column B 4 SPAM 4 SPAM 1 HELLO WORLD! 4 SPAM 4 SPAM 4 SPAM 1 HELLO Stage 4: Broadcast second pass Key Column A Pass 1 HELLO 1 3 GOOD 1 Key Column A Pass 1 HELLO 1 3 GOOD 1 Key Column A Pass 2 SPARK 2 4 SPAM 2 Key Column A Pass 2 SPARK 2 4 SPAM 2
- 26. Executor 2Executor 1 26#EUde11 Key Column B 4 SPAM 4 SPAM 1 WORLD! 4 SPAM 4 SPAM 4 SPAM Key Column B 4 SPAM 4 SPAM 4 SPAM 2 SUMMIT 3 MORNING 4 SPAM Key Column A Column B 4 SPAM 4 SPAM 4 SPAM 2 SUMMIT 3 GOOD MORNING 4 SPAM Key Column A Column B 4 SPAM 4 SPAM 1 HELLO WORLD! 4 SPAM 4 SPAM 4 SPAM 3 GOOD 1 HELLO Stage 5: Left join second pass Key Column A Pass 1 HELLO 1 3 GOOD 1 3 SPARK 3 GOOD3 GOOD3 GOOD3 GOOD 3 GOOD3 GOOD3 GOOD3 GOOD3 GOOD3 GOOD Key Column A Pass 2 SPARK 2 4 SPAM 2 Key Column A Pass 2 SPARK 2 4 SPAM 2
- 27. 27#EUde11 Performance Analysis • Setup notes – Tests on Amazon EMR using 4x m4.4xlarge workers – 5 cores and 18gb memory per executor – Spark 2.1 – Benchmark on GitHub – Skewed dataset, 10000 keys • Largest key 10000 rows, 2nd key 5000 rows, 3rd key 2500 rows … 10000th key 1 row – Uniform dataset, evenly spread
- 30. Considerations • Hyper parameters – Size of the broadcast variable – The number of passes • Additional complexity 30#EUde11
- 31. Future Work 31#EUde11 • In-memory iterations • Native to Spark
- 32. Recap 32#EUde11 • Skew hurts Spark parallelism and stability • Default join types have issues with skewed data • Iterative Broadcast Join can be used to process skewed data while maintaining parallelism