State of Spark in the cloud (Spark Summit EU 2017)
Originally presented at: https://spark-summit.org/eu-2017/events/the-state-of-apache-spark-in-the-cloud/ Cloud providers currently offer convenient on-demand managed big data clusters (PaaS) with a pay-as-you-go model. In PaaS, analytical engines such as Spark and Hive come ready to use, with a general-purpose configuration and upgrade management. Over the last year, the Spark framework and APIs have been evolving very rapidly, with major improvements on performance and the release of v2, making it challenging to keep up-to-date production services both on-premises and in the cloud for compatibility and stability. Nicolas Poggi evaluates the out-of-the-box support for Spark and compares the offerings, reliability, scalability, and price-performance from major PaaS providers, including Azure HDinsight, Amazon Web Services EMR, Google Dataproc with an on-premises commodity cluster as baseline. Nicolas uses BigBench, the brand new standard (TPCx-BB) for big data systems, with both Spark and Hive implementations for benchmarking the systems. BigBench combines SQL queries, MapReduce, user code (UDF), and machine learning, which makes it ideal to stress Spark libraries (SparkSQL, DataFrames, MLlib, etc.). The work is framed within the ALOJA research project, which features an open source benchmarking and analysis platform that has been recently extended to support SQL-on-Hadoop engines and BigBench. The ALOJA project aims to lower the total cost of ownership (TCO) of big data deployments and study their performance characteristics for optimization. Nicolas highlights how to easily repeat the benchmarks through ALOJA and benefit from BigBench to optimize your Spark cluster for advanced users. The work is a continuation of a paper to be published at the IEEE Big Data 16 conference.
![Conclusions
• All providers have up to date (2.1.0) and well tuned versions of Spark
• They could run BigBench up to 1TB on medium-sized cluster,
• [Almost] Out-of-the box
• Performance similar among providers for similar cluster types and disk configs
• Difference according to scale (and pricing)
• Spark 2.1.0 is faster than previous versions
• Also MLlib 2 with dataframes
• But improvements within the 30% range
• Hive (+Tez + MLlib) are still slightly faster than Spark at lower scales for sequential
• But Spark significantly faster at high data scales and concurrency
• BigBench has been useful to stress a cluster with different workloads
• Highlights config problems fast and stresses scale limits
• Helpful for tuning the clusters
23](https://image.slidesharecdn.com/cloud-bigbenchsparksummiteu2017-171025161500/85/State-of-Spark-in-the-cloud-Spark-Summit-EU-2017-23-320.jpg)
More Related Content
What's hot (20)
Similar to State of Spark in the cloud (Spark Summit EU 2017) (20)
More from Nicolas Poggi (9)
![Vagrant + Docker provider [+Puppet]](https://cdn.slidesharecdn.com/ss_thumbnails/npoggivagrant-docker-140725100147-phpapp02-thumbnail.jpg?width=560&fit=bounds)
Recently uploaded (20)
State of Spark in the cloud (Spark Summit EU 2017)
- 1. The state of in the cloud Nicolas Poggi Oct 2017 ____ __ / __/__ ___ _____/ /__ _ / _ / _ `/ __/ '_/ /___/ .__/_,_/_/ /_/_ /_/
- 2. Outline 1. Intro 1. PaaS Cloud 2. BigBench 2. Part I – Scalability 1. Hive vs. Spark 3. Part II – Additional experiments 1. Versions, Concurrency, 10TB 4. Summary 2
- 3. Motivation • 2016 SQL-on-Hadoop paper and presentations • Focused on Hive, due to SparkSQL not being ready to use in PaaS • Used benchmark (TPC-H) • Early 2017, BigBench testing Hive and Spark and TPC-TC paper • New code available in May for MLlib2 compatibility • Goals: Evaluate the current out-of-the-box experience of Spark v2 in PaaS cloud • Using Hive as baseline 3
- 4. Platform-as-a-Service Spark • Simplified management • Cloud-based managed Hadoop services • Ready to use Spark, Hive, … • Deploys in minutes, on-demand, elastic • Pay-as-you-go pricing model • Decoupled compute and storage • Optimized for general purpose • Fined tuned to the cloud provider architecture 4
- 5. Surveyed PaaS services • Amazon Elastic Map Reduce (EMR) • Released: Apr 2009 • OS: Amazon Linux AMI (RHEL-like) • Spark 2.1.0 and Hive 2.1 (Tez) • VM: M4.2xlarge (32GB RAM) • Google Cloud DataProc (GCD) • Released: Feb 2016 • OS: Debian 8.4 • Spark 2.1.0 (preview), Hive 2.1 (M/R) • VM: n1-standard-8 (30GB RAM) • Azure HDInsight (HDI) • Released: Oct 2013 • OS: Ubuntu 16.04 (HDP-based) • Spark 2.1.0 and 1.6.3, Hive 1.2 (Tez) • VM: D4v2 (28GB RAM) • Target deployment 128-cores: • 16 data nodes with 8-cores each • Master node with 16-cores • Decoupled storage only • EBS, WASB, GCS 5
- 6. What is BigBench (TPCx-BB) • End-to-end application level benchmark specification • result of many years of collaboration of industry and academia • Covers most Big Data Analytical properties (3Vs) • 30 business use cases for a retailer company • Merchandizing, • pricing, • customers … • Defines data scale factors • 1GB to PBs 6 Retailer database
- 7. Sequential Hive vs Spark 2.1 Queries 1-30 on Spark 2.1 (power runs) Query 1 Query 2 …. Query 30 Welcome to ____ __ / __/__ ___ _____/ /__ _ / _ / _ `/ __/ '_/ /___/ .__/_,_/_/ /_/_ version 2.1.0 /_/
- 8. BB 1GB-1TB Scalability Dataproc: Hive 2.1 (M/R) vs Spark 2.1
- 9. BB 1GB-1TB Scalability EMR: Hive 2.1 (Tez) vs Spark 2.1
- 10. BB 1GB-1TB Scalability HDI: Hive 1.2 (Tez) vs Spark 2.1
- 11. BB 1GB-1TB Scalability: Hive vs Spark 2.1 All providers
- 12. BB 1TB Power runs : Hive vs Spark 2.1 (ALL)
- 13. CPU % Q5 (ML) in Hive and Spark (HDI) 13 • Hive (MLlib2) • Spark (MLlib2) Time (s) Time (s) - 2X faster
- 14. Radar charts – query characterization • Useful for displaying multivariate data (5 resources) • Quickly identify similarities and differences. • From example • Hive and Spark • Only Disk Write is similar • Hive consumes more MEM and CPU • Spark read more from disk (DISK_R) • And moderately more network Sample radar chart for Q7 in EMR at 1TB 14
- 15. BB 1TB Query 5 (ML) providers comparison Hive (MLlib2) Spark (MLlib2) 15
- 16. Other comparisons: 10TB SQL-Only 2.0.2 vs 2.1.0 1.6.3 vs 2.1.0 MLlib v1 vs v2 16
- 17. BB 1GB-10TB Scalability SQL-only queries Hive Spark
- 18. BigBench 1GB-1TB: Spark 2.0.2 vs 2.1.0 (CDP) Notes: Spark 2.1 a bit faster at small scales, slower at 100 GB and 1 TB on the UDF/NLP queries 2.1 faster up to 100GB Slower at 1TB
- 19. BigBench 1GB-1TB: Spark 1.6.3 vs 2.1.0 MLlib 1 vs 2.1 MLlib 2(HDI) Notes: • Spark 2.1 is always faster than 1.6.3 in HDI • MLlilb 2 using dataframes over RDDs is only slightly faster than V1.
- 20. Concurrency runs (throughput) SQL-only: 100GB – 1TB 512-core cluster 2020
- 21. BB Throughput at 100GB 8 streams SQL-only (512-cores)
- 22. BB Throughput at 1TB 4 streams SQL-only (512-cores)
- 23. Conclusions • All providers have up to date (2.1.0) and well tuned versions of Spark • They could run BigBench up to 1TB on medium-sized cluster, • [Almost] Out-of-the box • Performance similar among providers for similar cluster types and disk configs • Difference according to scale (and pricing) • Spark 2.1.0 is faster than previous versions • Also MLlib 2 with dataframes • But improvements within the 30% range • Hive (+Tez + MLlib) are still slightly faster than Spark at lower scales for sequential • But Spark significantly faster at high data scales and concurrency • BigBench has been useful to stress a cluster with different workloads • Highlights config problems fast and stresses scale limits • Helpful for tuning the clusters 23
- 24. Resources and references BigBench and ALOJA • BigBench Spark 2 branch (thanks Christoph and Michael from bankmark.de): • https://github.com/carabolic/Big-Data- Benchmark-for-Big-Bench/tree/spark2 • Original BigBench Implementation repository • https://github.com/intel-hadoop/Big-Data- Benchmark-for-Big-Bench • ALOJA benchmarking platform • https://github.com/Aloja/aloja • ALOJA fork of BigBench (adds support for HDI and fixes spark) • https://github.com/Aloja/Big-Data-Benchmark- for-Big-Bench Papers and slides • https://www.slideshare.net/ni_po • Characterizing TPCx-BB Queries, Hive, and Spark in Multi-Cloud Environments – N. Poggi et. Al • TPC-TC 2017 • The State of SQL-on-Hadoop in the Cloud – N. Poggi et. al. • IEEE Big Data 2016 • https://doi.org/10.1109/BigData.2016 .7840751 24
- 25. Thanks, questions? Follow up / feedback : [email protected] Twitter: ni_po The state of in the cloud ____ __ / __/__ ___ _____/ /__ _ / _ / _ `/ __/ '_/ /___/ .__/_,_/_/ /_/_ /_/
- 26. Extra slides 26
- 27. BB 1TB Query 2 (M/R) providers comparison Hive Spark 27
- 28. Spark config EMR CDP HDI Java version OpenJDK 1.8.0_121 OpenJDK 1.8.0_121 OpenJDK 1.8.0_131 Spark version 2.1.0 2.1 2.1.0.2.6.0.2-76 Driver memory 5G 5G 5G Executor memory 5G 10G 4G Executor cores 4 4 3 Executor instances Dynamic Dynamic 20 dynamicAllocation enabled TRUE TRUE FALSE Executor memoryOverhead Default (384MB) 1,117 MB Default (384MB) 28