Run tests at scale with on-demand Selenium Grid using AWS Fargate
Download as PPTX, PDF•5 likes•4,248 views
The document discusses a solution for running tests at scale using an on-demand Selenium grid on AWS Fargate, aiming to address issues like slower feedback, hardware provisioning challenges, and high costs associated with traditional infrastructure. It highlights the benefits of a Dockerized Selenium grid that auto-scales, facilitates quick testing, and reduces execution time significantly while offering substantial cost savings compared to on-premises solutions. Key technical details include the setup of a high-capacity Selenium hub and nodes, container orchestration, cost analysis, and the use of AWS services like Elastic Container Registry and CloudFormation for efficient infrastructure management.
![• ECR – AWS Elastic Container Registry to
store Docker images
• Official selenium-node images
modified to expose container IP to hub
for registration
CONTAINER_IP=$(curl -s http://169.254.170.2/v2/metadata
| jq -r '.Containers[1].Networks[0].IPv4Addresses[0]')
ECR – REPOSITORY FOR DOCKER
IMAGES
AWS ECR](https://image.slidesharecdn.com/runtestsatscalewithon-demandseleniumgridusingawsfargate-190428140951/85/Run-tests-at-scale-with-on-demand-Selenium-Grid-using-AWS-Fargate-18-320.jpg)
![VPC
Application
Load Balancer
Node containers - Chrome
Node containers -
Firefox
Autoscaling Group
AWS Fargate
Selenium Hub
Selenium Grid
Amazon ECR
Jenkins
Node
Jenkins
node
Jenkins
Node
CloudWatch
logs
Cloudwatch
Alarm [Triggered
on CPU
utilization >
80%]](https://image.slidesharecdn.com/runtestsatscalewithon-demandseleniumgridusingawsfargate-190428140951/85/Run-tests-at-scale-with-on-demand-Selenium-Grid-using-AWS-Fargate-21-320.jpg)
Run tests at scale with on-demand Selenium Grid using AWS Fargate
- 1. RUN TESTS AT SCALE WITH ON-DEMAND SELENIUM GRID USING AWS FARGATE Presenter: Megha Mehta
- 2. Megha Mehta Senior Automation Engineer S&P Global Market Intelligence, Gurgaon https://www.linkedin.com/in/m-mehta/ SPEAKER [email protected] @meghamehtatech
- 3. PROBLEM DESCRIPTION • Slower feedback • Hardware provisioning challenges • Additional maintenance effort • Challenges with cross-browser testing
- 4. • UI tests take long to run • More expensive • Longer waits for feedback without parallelization SLOWER FEEDBACK
- 5. HARDWARE PROVISIONING CHALLENGES • Performance of under-provisioned hardware suffers under peak loads • Avoid wastage due to over-provisioning by optimizing capacity utilization Application stops responding Infrastructure slow under load Tests failed
- 6. • Frequent software upgrades and patches for system efficiency -Typical maintenance activities – browser upgrades, OS hotfixes, security patches, system restart, disk cleanup etc. • This takes focus away from testing ADDITIONAL MAINTENANCE EFFORT
- 7. CHALLENGES WITH CROSS- BROWSER TESTING CHROME 75 FIREFOX 60 FIREFOX 65 On different versions of same browser On different browsers
- 8. SOLUTION Auto- provisioned with minimal maintenance requirement s • Auto- scalable in response to varying load • Reliable and cost- effective Run tests in hyper-parallel mode for faster feedback Dynamic and quick to change
- 9. OUR SOLUTION A dockerized Selenium Grid running on AWS
- 10. TAKEAWAYS • Designing your own high-capacity Selenium Grid using Docker and AWS • A scalable yet disposable grid • Faster feedback – reduce execution time from hours to minutes • Enable Continuous Integration • Major cost savings compared to traditional on-prem infrastructure
- 12. |WHY DOCKER CONTAINERS? VMs heavyweight Take up lot of memory resources Startup time in minutes Containers lighweight Require lower memory Startup time in seconds
- 13. SELENIUM GRID – HUB SETUP • Hub should have High Bandwidth to support large number of requests from test clients and nodes • High CPU needed for 1000’s of tests but lower memory is fine • In our case, hub is single container of 16GB RAM, 4 vCPU
- 14. SELENIUM GRID – NODE SETUP • Containers of 16 GB RAM, 4 vCPU • Each container can run up to 20 browsers without any performance hit
- 15. Docker Swarm AWS EKS (Kubernetes) AWS ECS/Fargate Do we need to provision instances in advance? Yes Yes No, automatically done by AWS Is it suitable for short workloads? Yes No, master node always up Yes How cost-effective is the solution? * Charged for provisioned EC2’s Additional charges for cluster management Charged only for duration that containers are running *Considering the container cluster only runs for a few hours HOW TO MANAGE SO MANY CONTAINERS? Deciding factors ↓ Orchestration solutions
- 16. CONTAINER ORCHESTRATION USING FARGATE • No need to provision, configure, or scale clusters of machines to run containers – unlike other orchestration solutions • Focus on designing and building Selenium Grid instead of managing the infrastructure that runs it AWS Fargate
- 17. LOAD BALANCER • Selenium Hub is behind an AWS LB which listens to incoming requests from clients and responses from nodes on port 4444 • Domain Name System (DNS) name received by the LB can be quite complex and lengthy - Use a friendly DNS name for load balancer using AWS Route53 AWS Application Load Balancer AWS Route53
- 18. • ECR – AWS Elastic Container Registry to store Docker images • Official selenium-node images modified to expose container IP to hub for registration CONTAINER_IP=$(curl -s http://169.254.170.2/v2/metadata | jq -r '.Containers[1].Networks[0].IPv4Addresses[0]') ECR – REPOSITORY FOR DOCKER IMAGES AWS ECR
- 19. VPC Application Load Balancer Node containers - Chrome Node containers - Firefox Autoscaling Group Amazon ECR Jenkins Node Jenkins node Jenkins Node AWS Fargate Selenium Hub Selenium Grid
- 20. • Monitor container metrics like CPU and memory utilization during tests • Scale out – add 2 node containers whenever CPU Utilization > 80% for 2 min • Scale down to ZERO if idle > 5 mins AUTO-SCALING USING CLOUDWATCH
- 21. VPC Application Load Balancer Node containers - Chrome Node containers - Firefox Autoscaling Group AWS Fargate Selenium Hub Selenium Grid Amazon ECR Jenkins Node Jenkins node Jenkins Node CloudWatch logs Cloudwatch Alarm [Triggered on CPU utilization > 80%]
- 22. CLOUDFORMATION – BRINGING IT ALL TOGETHER • Describe and provision all the infrastructure resources using CloudFormation template • Version control/replicate/update the templates like application code AWS CloudFormation
- 23. *Calculations shown for US East (N. Virginia) Region. Also, costs of other services like ECR, AWS ALB etc. are not included. COST BENEFITS 41 Fargate Containers (1 Hub + 40 Nodes, each of 4vCPU and 16GB configuration) x 10 minutes to complete 800 tests x 5 times a day x 30 days in a month = 1025 hours ~ $238.59 per month* 41 VMs x ~ $80 per VM per month ~ $3280 per month
- 24. COST BENEFITS $39360 $2856 Annual costs Fargate containers On-prem VM That’s a saving of ~93%!
- 25. SUMMARY • Designing your own high-capacity Selenium Grid using Docker and AWS • A scalable yet disposable grid • Faster feedback – reduce execution time from hours to minutes • Enable Continuous Integration • Major cost savings compared to traditional on-prem infrastructure
- 26. • Task networking in Fargate https://aws.amazon.com/blogs/compute/task-networking-in-aws-fargate/ • Comparison between clusters https://www.trinimbus.com/blog/fresh-reinvent-2017-aws-introduces- elastic-container-service-kubernetes-eks-fargate • FARGATE pricing calculator http://fargate-pricing-calculator.site.s3-website-us-east-1.amazonaws.com/ REFERENCES
- 28. THANK YOU!