Alluxio on AWS EMR Fast Storage Access & Sharing for Spark
2 likes857 views
The document discusses the integration of Alluxio with AWS EMR for enhanced data processing with Spark, highlighting key features such as data locality, caching, and tiered storage. It covers the setup process for Alluxio on EMR and includes a demo showcasing hardware and configurations used. Additionally, it emphasizes challenges and solutions related to data sharing and job execution across different storage systems.
![Setup Alluxio on EMR
aws emr create-cluster
--applications Name=Spark
--release-label emr-5.23.0
--configurations ...
--bootstrap-actions '[{"Path":"s3://test/bootstrap_cz.sh"]'
--region us-east-1
core-site
"fs.alluxio.impl": "alluxio.hadoop.FileSystem"
"fs.AbstractFileSystem.alluxio.impl": "alluxio.hadoop.AlluxioFileSystem"
spark-defaults
"spark.driver.extraClassPath":
":/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrf
s/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/sh
are/aws/emr/security/lib/*:/opt/alluxio-1.8.1-hadoop-2.8/client/alluxio-1.8.1-client.jar",
"spark.executor.extraClassPath": ":/opt/alluxio-1.8.1-hadoop-2.8/client/alluxio-1.8.1-client.jar"](https://image.slidesharecdn.com/alluxioonawsemrfaststorageaccessandsharingforspark-190722224418/85/Alluxio-on-AWS-EMR-Fast-Storage-Access-Sharing-for-Spark-19-320.jpg)
![Bootstrap
ismaster=`cat /mnt/var/lib/info/instance.json | jq -r '.isMaster'`
masterdns=`cat /mnt/var/lib/info/job-flow.json | jq -r '.masterPrivateDnsName'`
….
if [[ ${ismaster} == "true" ]]; then
sudo ./bin/alluxio bootstrapConf ${masterdns}
sudo ./bin/alluxio format
sudo ./bin/alluxio-start.sh master
else
sudo ./bin/alluxio bootstrapConf ${masterdns}
initialize_alluxio
sudo ./bin/alluxio format
sudo ./bin/alluxio-start.sh worker Mount
fi
….](https://image.slidesharecdn.com/alluxioonawsemrfaststorageaccessandsharingforspark-190722224418/85/Alluxio-on-AWS-EMR-Fast-Storage-Access-Sharing-for-Spark-20-320.jpg)
Alluxio on AWS EMR Fast Storage Access & Sharing for Spark
- 1. Alluxio Open Source New York Meetup Alluxio on AWS EMR Fast storage access and sharing for Spark
- 2. Table of Content ● Spark and Alluxio ● AWS EMR and deployment of Alluxio ● Demo
- 3. About me Chengzhi Zhao ● Data Platform Engineer at Meetup 2.5 years, 4 years in retail and healthcare ● Area of Interest: Build scalable, reliable and maintainable data infrastructure and tools
- 4. How do we run spark jobs? EMR Dataproc HDInsight on premise S3 GCS DB HDFS Compute Storage
- 5. To Summarize... ● Multiple cloud providers ● Storage layer is separated from compute layer ● Data skew/Data Format ● Failure/Restart ● Data sharing is difficult
- 7. Alluxio
- 8. Architecture
- 9. DATA LOCALITY Spark tries to execute tasks as close to the data as possible to minimize data transfer (over the wire). Preferred Locality Order: PROCESS_LOCAL, NODE_LOCAL, RACK_LOCAL, or ANY
- 12. Data Sharing
- 13. Spark Read & Write spark.read.parquet("alluxio://") val rdd = sc.textFile("alluxio://") df.spark.write.parquet("alluxio://") rdd.saveAsTextFile("alluxio://")
- 14. Caching ● Avoid duplicated cache ● Crash/Restart
- 15. Tiered Storage By default, Alluxio only enables a single, memory tier. ● MEM (Memory) ● SSD (Solid State Drives) ● HDD (Hard Disk Drives)
- 17. Preload data into Alluxio Uniformly $ bin/alluxio fs load /path/to/load Data Warehouse
- 18. EMR (Elastic MapReduce) Amazon EMR simplifies building and operating big data environments and applications. EMR features include easy provisioning, scaling, and reconfiguring of clusters, and notebooks for collaborative development. Applications: Flink, Ganglia, Hadoop, HBase, HCatalog, Hive, Hue, JupyterHub, Livy, Mahout, MXNet, Oozie, Phoenix, Pig, Presto, Spark, Sqoop, TensorFlow, Tez, Zeppelin, and ZooKeeper.
- 19. Setup Alluxio on EMR aws emr create-cluster --applications Name=Spark --release-label emr-5.23.0 --configurations ... --bootstrap-actions '[{"Path":"s3://test/bootstrap_cz.sh"]' --region us-east-1 core-site "fs.alluxio.impl": "alluxio.hadoop.FileSystem" "fs.AbstractFileSystem.alluxio.impl": "alluxio.hadoop.AlluxioFileSystem" spark-defaults "spark.driver.extraClassPath": ":/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrf s/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/sh are/aws/emr/security/lib/*:/opt/alluxio-1.8.1-hadoop-2.8/client/alluxio-1.8.1-client.jar", "spark.executor.extraClassPath": ":/opt/alluxio-1.8.1-hadoop-2.8/client/alluxio-1.8.1-client.jar"
- 20. Bootstrap ismaster=`cat /mnt/var/lib/info/instance.json | jq -r '.isMaster'` masterdns=`cat /mnt/var/lib/info/job-flow.json | jq -r '.masterPrivateDnsName'` …. if [[ ${ismaster} == "true" ]]; then sudo ./bin/alluxio bootstrapConf ${masterdns} sudo ./bin/alluxio format sudo ./bin/alluxio-start.sh master else sudo ./bin/alluxio bootstrapConf ${masterdns} initialize_alluxio sudo ./bin/alluxio format sudo ./bin/alluxio-start.sh worker Mount fi ….
- 21. DEMO
- 22. Hardware ● r5.2xlarge ● 8 vCore ● 64 GiB memory ● 3 instances
- 23. Size: 51G Parquet Non Alluxio Alluxio
- 24. Size: 51G Parquet Non Alluxio Alluxio