Amazon Elastic Map Reduce - Ian Meyers
1 like2,066 views
In this talk, Ian will talk about Amazon Elastic MapReduce and how it integrates with other AWS services in a big data stack.
![Lower TCO
June
2013
Study
by
Accenture
Technology
Labs
Not
Sponsored
or
Funded
by
Amazon
“Accenture
assessed
the
price-‐
performance
raJo
between
bare-‐metal
Hadoop
clusters
and
Hadoop-‐as-‐a-‐Service
on
Amazon
Web
Services…[and]
revealed
that
Hadoop-‐as-‐a-‐Service
offers
bePer
price-‐performance
raJo…”
hkp://www.accenture.com/us-‐en/Pages/insight-‐hadoop-‐
deployment-‐comparison.aspx](https://image.slidesharecdn.com/july2013-130722153651-phpapp02/85/Amazon-Elastic-Map-Reduce-Ian-Meyers-40-320.jpg)
![[よくわかるAmazon Redshift in 大阪]Amazon Redshift最新情報と導入事例のご紹介](https://cdn.slidesharecdn.com/ss_thumbnails/20140221redshiftupdatesv2osaka-140224010309-phpapp02-thumbnail.jpg?width=560&fit=bounds)
![[よくわかるAmazon Redshift]Amazon Redshift最新情報と導入事例のご紹介](https://cdn.slidesharecdn.com/ss_thumbnails/20140219redshiftupdatesv1tokyo-140224010117-phpapp02-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (10)
Similar to Amazon Elastic Map Reduce - Ian Meyers (9)
Ad
More from huguk (20)
Ad
Recently uploaded (20)
Amazon Elastic Map Reduce - Ian Meyers
- 2. Deep experience in building and operating global web scale systems About Amazon Web Services ? …get into cloud computing? How did Amazon…
- 3. Utility computing On demand Pay as you go Uniform Available
- 4. Utility computing On demand Pay as you go Uniform Available
- 6. Utility computing On demand Pay as you go Uniform Available Compute Storage Security Scaling Database Networking Monitoring Messaging Workflow DNS Load Balancing Backup CDN
- 7. No Up-‐Front Capital Expense Pay Only for What You Use Self-‐Service Infrastructure Easily Scale Up and Down Improve Agility & Time-‐to-‐Market Low Cost Deploy Cloud computing benefits
- 8. Traditional IT capacity ElasNc capacity Capacity Time Your IT needs
- 9. On and Off Fast Growth Variable peaks Predictable peaks ElasNc capacity
- 10. ElasNc capacity On and Off Fast Growth Predictable peaks Variable peaks WASTE CUSTOMER DISSATISFACTION
- 11. ElasNc capacity Fast Growth On and Off Predictable peaks Variable peaks
- 12. NumberofEC2Instances 4/12/2008 4/14/2008 4/15/2008 4/16/2008 4/18/2008 4/19/2008 4/20/20084/17/20084/13/2008 40 servers to 5000 in 3 days EC2 scaled to peak of 5000 instances “Techcrunched” Launch of Facebook modification Steady state of ~40 instances
- 13. Compute Storage AWS Global Infrastructure Database App Services Deployment & AdministraNon Networking Global Infrastructure
- 14. Global Infrastructure Region US-WEST (N. California) EU-WEST (Ireland) ASIA PAC (Tokyo) ASIA PAC (Singapore) US-WEST (Oregon) SOUTH AMERICA (Sao Paulo) US-EAST (Virginia) GOV CLOUD ASIA PAC (Sydney)
- 16. Customer Needs • Store Any Amount of Data – Without Capacity Planning • Perform Complex Analysis on Any Data – Scale on Demand • Store Data Securely • Decrease Time to Market – Build Environments Quickly • Reduce Costs – Reduce Capital Expenditure • Enable Global Reach
- 18. ElasNc Block Store High performance block storage device 1GB to 1TB in size Mount as drives to instances with snapshot/cloning functionalities IMAGE Availability 99.99% Durability 99.999999999% Is a Web Store Not a file system No Single Points of Failure Eventually consistent Paradigm Object store Performance Very Fast Redundancy Across Availability Zones Security Public Key / Private Key Pricing $0.095/GB/month Typical use case Write once, read many Limits 100 Buckets, Unlimited Storage, 5TB Objects Simple Storage Service Highly scalable object storage for the internet 1 byte to 5TB in size 99.999999999% durability
- 19. Peak Requests: 830,000+ per second Total Number of Objects Stored in Amazon S3 14 Billion 40 Billion 102 Billion 762 Billion 262 Billion 1.3 Trillion Q4 2006 Q4 2007 Q4 2008 Q4 2009 Q4 2010 Q4 2011 Q4 2012 Objects in S3
- 20. Glacier Long term object archive Extremely low cost per gigabyte 99.999999999% durability ElasNc Block Store High performance block storage device 1GB to 1TB in size Mount as drives to instances with snapshot/cloning functionalities IMAGE Durability 99.999999999% Designed for Archival Not a file system Vaults & Archives 3-5 Hour Retrieval Time Paradigm Archive Store Performance Configurable - Low Redundancy Across Availability Zones Security Public Key / Private Key Pricing $0.011/GB/month Typical use case Write once, read infrequently < 10% / Month
- 21. Simple Storage Service Highly scalable object storage 1 byte to 5TB in size 99.999999999% durability Glacier Long term object archive Extremely low cost per gigabyte 99.999999999% durability Storage Lifecycle IntegraNon
- 23. Compute Storage AWS Global Infrastructure Database App Services Deployment & AdministraNon Networking Database Relational Database Service Managed Oracle, MySQL & SQL Server Dynamo DB Managed NOSQL Database Amazon Redshift Massively Parallel Petabyte Scale Data Warehouse RDS Dynamo DB Redshift
- 24. Compute Storage AWS Global Infrastructure Database App Services Deployment & AdministraNon Networking Database Relational Database Service Database-as-a-Service No need to install or manage database instances Scalable and fault tolerant configurations Integration with Data Pipeline RDS Dynamo DB Redshift
- 25. Compute Storage AWS Global Infrastructure Database App Services Deployment & AdministraNon Networking Database DynamoDB Provisioned throughput NoSQL database Fast, predictable, configurable performance Fully distributed, fault tolerant HA architecture Integration with EMR & Hive RDS Dynamo DB Redshift
- 26. Compute Storage AWS Global Infrastructure Database App Services Deployment & AdministraNon Networking Database Redshift Managed Massively Parallel Petabyte Scale Data Warehouse Streaming Backup/Restore to S3 Extensive Security 2 TB -> 1.6 PB RDS Dynamo DB Redshift
- 27. Unstructured Data … Parallel ETL
- 28. Elastic MapReduce Managed, elastic Hadoop cluster Integrates with S3 & DynamoDB Leverage Hive & Pig analytics scripts Support for Spot Instances Integrated HBase NOSQL Database Compute Storage AWS Global Infrastructure Database App Services Deployment & AdministraNon Networking Application Services Elastic MapReduce
- 29. • AWS Web Console • Command Line elastic-‐mapreduce -‐-‐create -‐-‐key-‐pair micro -‐-‐region eu-‐ west-‐1 -‐-‐name IanMM-‐Test1 -‐-‐num-‐instances 5 -‐-‐instance-‐ type m2.4xlarge –alive -‐-‐log-‐uri s3n://meyersi-‐ire/EMR/ log Launching Clusters
- 30. • Enabling Tools elastic-‐mapreduce -‐-‐create -‐-‐key-‐pair micro -‐-‐region eu-‐west-‐1 -‐-‐ name IanMM-‐Test1 -‐-‐num-‐instances 5 -‐-‐instance-‐type m2.4xlarge -‐-‐ alive -‐-‐pig-‐interactive -‐-‐pig-‐versions latest -‐-‐hive-‐interactive –-‐hive-‐versions latest -‐-‐hbase -‐-‐log-‐uri s3n://meyersi-‐ire/EMR/log Launching Clusters
- 31. • Hadoop Configuration Bootstrap Action elastic-‐mapreduce -‐-‐create -‐-‐bootstrap-‐action s3://elasticmapreduce/bootstrap-‐ actions/configure-‐hadoop -‐-‐args "-‐ s,dfs.block.size=1048576” -‐-‐key-‐pair micro -‐-‐region eu-‐west-‐1 -‐-‐name IanMM-‐Test-‐3 -‐-‐instance-‐group core -‐-‐instance-‐count 2 -‐-‐instance-‐type m2.4xlarge -‐-‐ instance-‐group task -‐-‐instance-‐count 2 -‐-‐instance-‐type m2.4xlarge -‐-‐alive -‐-‐pig-‐interactive -‐-‐hive-‐interactive -‐-‐log-‐uri s3n://meyersi-‐ire/EMR/log Launching Clusters
- 32. Input Datanode: This could be a S3 bucket, RDS table, EMR Hive table, etc. Activity: This is a data aggregation, manipulation, or copy that runs on a user- configured schedule. Output Datanode: This supports all the same datasources as the input datanode, but they don’t have to be the same type. Amazon Data Pipeline
- 33. Output: S3 file Path: s3://trend-‐data/#{year-‐month-‐day}.csv AcNvity: EMR Transform Hive Query: user-‐metrics.hql Frequency: Daily Input: RDS Table Table: User-‐Demographics SQL PrecondiNon: “Select last_update from table“ > #{YY-‐MM-‐DD} Input: DynamoDB Table Table: User-‐Event-‐Data-‐#{year-‐month} Success NoNficaNon: [email protected] Failure NoNficaNon: emr-‐[email protected] Delay NoNficaNon: : emr-‐[email protected] Orchestration with Data Pipeline
- 34. Analytics Pipeline Redshift S3 RDS EMR Data Pipeline …collect & store …orchestrate …process & analyse Dynamo DB
- 35. Benefits only possible in the Cloud Pay as you Go Lower Overall Costs Stop Guessing Capacity Agility / Speed / Innovation Avoid Undifferentiated Heavy Lifting Go Global in Minutes ✔ ✔ ✔ ✔ ✔ ✔ “Private Cloud” / On Premises X X X X X X
- 36. Agility & Global Reach at the Core of EMR
- 37. Ease of Operation Compute Infrastructure Hadoop ConfiguraNon Local Disk OperaNng System Config HDFS Networking Hive Pig HBase User Defined Sogware InstallaNon
- 38. Ease of Operation Compute Infrastructure Hadoop ConfiguraNon Local Disk OperaNng System Config HDFS Networking Hive Pig HBase User Defined Sogware InstallaNon Multiple Hadoop Distributions - Open Source & MapR Clusters Launched with 1 Command Up in 5 Minutes Hard Partitioned per Customer on CPU, Memory and Disk Dynamic Cluster Resizing In any of 8 Regions around the Globe
- 39. Lower Overall Costs Cheaper | Spot Market Management
- 40. Lower TCO June 2013 Study by Accenture Technology Labs Not Sponsored or Funded by Amazon “Accenture assessed the price-‐ performance raJo between bare-‐metal Hadoop clusters and Hadoop-‐as-‐a-‐Service on Amazon Web Services…[and] revealed that Hadoop-‐as-‐a-‐Service offers bePer price-‐performance raJo…” hkp://www.accenture.com/us-‐en/Pages/insight-‐hadoop-‐ deployment-‐comparison.aspx
- 41. • Spot allows customers to bid on unused EC2 capacity • Spot price based on supply/demand of instance types in an Availability Zone • Customers are fulfilled when their bid price is higher than the Spot Price • Instances will be interrupted when the Spot price exceed the bid price Spot 101 - What are Spot Instances
- 42. elastic-mapreduce --add-instance-group TASK --instance-count 100 --bid-price .4
- 43. Mix Spot and On-Demand instances to reduce cost and accelerate computation while protecting against interruption #1: Cost without Spot 4 instances *14 hrs * $0.50 = $28 Job Flow 14 Hours Duration: Other EMR + Spot Use Cases § Run entire cluster on Spot for biggest cost savings § Reduce the cost of application testing #2: Cost with Spot 4 instances *7 hrs * $0.50 = $14 + 5 instances * 7 hrs * $0.25 = $8.75 Total = $22.75 Scenario #1 Duration: Job Flow 7 Hours Scenario #2 Time Savings: 50% Cost Savings: ~20% Reducing Hadoop Costs with Spot
- 44. Stop Guessing Capacity Dynamic Clusters
- 46. with Amazon VPC…
- 47. Populate as demand dictates…
- 48. Connect over dedicated links…
- 49. And turn it off when you are done
- 50. EMR is Hadoop… …cheaper, easier, and more agile
- 51. What’s New? • MapR M7 Introduction • Optimised for HBase Clusters • Failure Recovery • Point in Time Recovery Snapshotting • Low Latency Hadoop Optimisations • HBase Mirroring • NFS + HDFS • MapR M5 Price Drop • Support for Pig 0.11.1 • RANK, CUBE & ROLLUP capability • Groovy UDF’s • Support for Guava Functions • Performance Improvements • Spark/Shark Bootstrap Action • In Memory Hadoop • Spark Scripting (similar to Pig) • Shark Shell with Hive Interoperability