Map reduce vs spark
- 1. MapReduce vs/and Spark Tudor Lapusan BigData Romanian Tour - Timisoara
- 2. History
- 3. MapReduce basic functionalities ● Fault tolerance ● Monitoring & status updates ● Scalability
- 4. Hadoop MapReduce Input Map Reduce Output
- 5. Hadoop MapReduce Input Map Shuffle Reduce Output
- 7. Spark ● RDD ● Operations : Transformations and Actions
- 8. RDD - Resilient Distributed Dataset RDD is fault-tolerant collection of elements distributed across many servers on which we can perform parallel operations.
- 9. RDD Scala code val data = Array(1, 2, 3, 4, 5, 6, 7, 8) val rddData = sc.parallelize(data)
- 10. RDD Scala code val rddFile = sc.textFile("data.txt")
- 12. Transformations RDD 1 RDD 2 Transformations are operations on RDDs that return new RDDs
- 13. Transformations RDD 1 InputRDD {1,2,3,4,5,6} MapRDD {2,3,4,5,6,7} FilterRDD {1,2,3,5,6} map x => x +1 filter x => x != 4
- 14. Actions RDD 1 Actions are the operations on RDD which return a final value or write the data to an external storage system. RDD 1
- 15. Actions RDD 1 InputRDD {1,2,3,4,5,6} MapRDD {2,3,4,5,6,7} FilterRDD {1,2,3,5,6} map x => x +1 filter x => x != 4 count()=6 take(2)={1,2} saveAsTextFile()
- 16. Spark DAG RDD 1 RDD 2 RDD 4 RDD 6 RDD 3 RDD 5 Action Transformation Stage
- 17. Spark DAG vs MapReduce DAG RDD 1 RDD 2 RDD 4 RDD 6 RDD 3 RDD 5 A B D C E F
- 19. Easy of use - Spark is easier to program and include an interactive mode. - Hadoop MapReduce is harder to program but many tools are available to make it easier.
- 20. Performance : Sort Benchmark 2013
- 21. Performance : Sort Benchmark 2014
- 22. Costs
- 23. Costs : hardware recommendation Spark MapReduce Hadoop Cores 8-16 4 Memory 8GB to hundreds of GB 24GB Disks 4-8 4-6 one-TB disks Network 10GB or more 1GB Ethernet Spark recommendation Hortonworks recommendation