Effective use of cloud resources for Data Engineering - Johnson Darkwah
- 1. Effective use of cloud resources for Data Engineering A guide to Cloud Computing & Big Data 12/2017
- 2. Obsessed With Creating Value From Data
- 4. Where are you running your Big Data ? The vast majority has been running on premise, on private infrastructure
- 5. 33% of enterprises will take their data lakes off life support. Forrester Predictions 2018: The Honeymoon For AI Is Over
- 6. Initial Cost of a Production Big Data Management Platform running On Premise 200,000 EUR 5,092,000 CZK
- 7. Challenges with the Cloud • Security remains a concern for early adopters • Experienced companies focus on optimizing cost • Data integration poses a challenge • Lack of resources
- 8. Why use the cloud today for analytics It’s never been cheaper to process and store data in the cloud It’s never been easier to manage cloud services Cost Flexibility
- 9. Master m4.xlarge 1x 1TB EBS Worker m4.2xlarge 4x 1TB EBS Worker m4.2xlarge 4x 1TB EBS Worker m4.2xlarge 4x 1TB EBS Worker m4.2xlarge 4x 1TB EBS Worker m4.2xlarge 4x 1TB EBS Data processing power of 32 CPU cores, 120GB RAM and the ability to store up to 6,5 TB of data. When you treat cloud the same way as on-premise
- 10. Monthly Costs on AWS AWS EC2 (Europe Central) - Compute: 1932.48 USD - EBS Volumes: 1134.00 USD $ 3,066.49 USD 65,371 CZK AWS EC2 (Europe Central) - Compute: 8725.44 USD - EBS Volumes: 4860.00 USD $ 13,585.44 USD 289,605 CZK Reasonable minimum spec Comparable to production on-prem. https://calculator.s3.amazonaws.com/index.html
- 11. ü Utilize Object Stores ü Pay only for what we really use ü Get discounts ü Optimize the Data Architecture ü Automate Everything Cloud Data Engineering Checklist
- 13. ü Utilize Object Stores ü Pay only for what we really use ü Get discounts ü Optimize the Data Architecture ü Automate Everything Cloud Data Engineering Checklist
- 15. ü Utilize Object Stores ü Pay only for what we really use ü Get discounts ü Optimize the Data Architecture ü Automate Everything Cloud Data Engineering Checklist
- 18. ü Utilize Object Stores ü Pay only for what we really use ü Get discounts ü Optimize the Data Architecture ü Automate Everything Cloud Data Engineering Checklist
- 19. Realtime System 24x7 S3/ADLS M W W W W W W SQL Engine Terraform / Cloudera Altus Existing Enterprise Logs, CRM, RDBMS Gauss Data Tool Clicks IoT Trans. Temp cluster Client Job ML, SQL, Enrichment, … Spot On-premise Cloud RIOSOD
- 20. Show me the money … AWS EC2 (Europe Central) Realtime System (RI – 24x7) - Compute (3x): 352.52 USD - EBS Vol. (3x1TB): 54.00 USD Temp - Compute (OD 8x7): - 1x m4.xlarge: 58.56 USD - 3x m4.2xlarge: 351.36 USD - EBS Vol. (9x1TB): 162.00 USD $ 1,165.78 USD 24,852.10 CZK Spot support (3x – m4.2xlarge - $0.2/h) - Compute(): 34.34 USD - EBS Vol. (9x1TB): 162.00 USD https://calculator.s3.amazonaws.com/index.html 8 Hours Compute Daily
- 21. ü Utilize Object Stores ü Pay only for what we really use ü Get discounts ü Optimize the Data Architecture ü Automate Everything Cloud Data Engineering Checklist
- 22. Terraform • Enables simple management and automation of cloud environments • AWS, Azure, GCP, OpenStack, … • Many provisioning options # Configure the AWS Provider provider "aws" { access_key = "${var.aws_access_key}" secret_key = "${var.aws_secret_key}" region = "us-east-1" } # Create a datanode resource "aws_instance" ”datanode" { # ... }
- 23. Cloudera Altus • Enterprise ready & fully managed cluster • Spark, MRv2 and Hive supported • Support for spot instances • Uses S3 or ADLS • Clusters can run only when executing jobs • Custom pre-built image (AMI) support • Per hour pricing with support included
- 24. ü Utilize Object Stores ü Pay only for what we really use ü Get discounts ü Optimize the Data Architecture ü Automate Everything Cloud Data Engineering Checklist
- 25. Twitter: @darkwahj LinkedIn: Johnson Darkwah Do more with data & save money. Questions ? www.gaussalgo.com