Testing Big Data in AWS - Sept 2021
- 1. Tonight’s Meetup Michael Sexton "Testing Big Data in AWS" 23rd September 2021
- 2. Travel Plan What is Big Data in AWS What is Testing Big Data in AWS Cloud Types of Testing Deployment Testing and Deployments Tools & Automation Challenges Conclusion Q & A
- 4. What is Big Data? - IoT, Netflix, Cybersecurity, Social Media, EuroVision, Traffic on GoogleMaps, Machine Learning, Video, Communications, etc. https://bigdatapath.wordpress.com/
- 5. What Work is There? Senior QA Engineer - Data Analytics Software - Dublin - Selenium/Python - 65k - 75k Senior SDET - Big Data/Analytics - Dublin/Remote - Python, System Testing, Performance Testing - 80k - 90k Lead SDET - Big Data/Analytics - Dublin/Remote - Python, System Testing, Performance Testing - 90k - 100k Senior QA Automation Engineer - Data Management Software - Galway/Remote - Python/Selenium 60k - 70k Reperio Human Capital
- 6. Who Are the Cloud Providers? - Various cloud providers: AWS, Azure, Google Cloud, Alibaba, IBM, Dell, Tencent
- 7. What is Big Data in AWS?
- 8. Services Provided By AWS
- 9. What is Testing Big Data in AWS? - Testing application that carries data works well (no anomalies) - Functional, Performance testing - Testing of migration from on-prem to the cloud - Verifying resultant big data analysis is correct
- 10. Functional Testing - Testing with varied valid and invalid input - Boundary cases, Calculations - Scripts - Latin, Sanskrit, Arabic, encoded - Testing against existing on-prem results - Failure cases
- 11. Performance and SEcurity Testing - Data ingestion (many different sources of data e.g. v1 and v3) - Data processing and throughput - soak - Sub-component performance - Security of pipeline and stored data - Robustness (no data arrives - what then?)
- 12. Pythagorean Cup
- 13. Deployment Testing and DEployments - Deployment Testing (upgrades, timings) - Code merged to Master branch, deploy scripts & documentation written & tested - Are go-to person for devops during deployment and upgrades - Staging Environment, Production Environment - Integration testing with other teams.
- 14. Tools & Automation - AWS EC2 machine with linux/python scripts - Pytest/Robotframework for automation & regression testing - AWS eco system (Athena, CloudWatch, X-Ray, QuickSight) - EXCEL (max=1048576 rows) & VLOOKUP - Pyspark
- 15. Challenges - Large varied dataset - Automation and scripting skills - Costs & AWS Knowledge - Knowing the big picture - Deployments to staging and production - Communicating with other teams
- 16. Conclusion - What is Big Data … in AWS? - What is Big Data Testing in AWS - Types of Testing (Functional & Performance) - Deployment Testing & Deployments - Tools & Automation - Challenges
- 17. Any Questions?
- 18. Thanks Ministry of Testing Poppulo Twitter: @MinistryCork
Editor's Notes
- #19: The Ministry of Testing aims to change and lead within the software testing world. We are doing this through a strong focus on learning, collaboration and resources. You are part of the story too, we hope you can join us along the way. We run several conferences around the world under the name of TestBash. We also over an e-learning platform called the Dojo. The Dojo offer numerous online courses for you to improve your testing skills.